Ranking under each sentiment pathway
Cell = #rank · score · change vs live (↑ moves up) · bucket · R = Reddit community score. Colour = bucket (Watch Depends Skip). Click a cell for that movie's comments with each pathway's verdict and rationale. The live shortlist on the home page uses GPT-6.1 Sol (since 2026-10-25; previously siebert+opinion). Buckets use rule v2 for every pathway: Watch needs score ≥ max(75, Q3 of that pathway's own slate); per-pathway bars: VADER (old v1) 80.05, siebert+opinion (previous live) 77.25, Local NLI stance (deberta-v3-large) 75, GPT-6.1 Sol (LIVE) 90.45, Claude Opus 5.5 91.05, Claude Sonnet 5.5 90, Gemini 4 Argon 91.3, Grok 4.7 90.55, GPT-6 Luna (cheap) 89.05, Claude Haiku 5.5 (cheap) 89.45, Gemini 3.5 Flash-Lite (cheap) 88.55. Skip < 55 (see Formula).
Stance pathways ask: did the commenter like or recommend the film? (positive / negative / mixed / no verdict, JSON, temperature 0). Mixed and no-verdict count as neutral. OpenRouter spend for these evaluations and runs: $14.47. Not run: Local Qwen2.5-3B stance
Stance prompt (shared by every LLM pathway, stance-v1)
You classify a Reddit comment from a movie's discussion thread by the commenter's VERDICT ON THE FILM ITSELF: did they like it / would they recommend it?
Judge the verdict, not the tone or vocabulary.
- Praise of dark, scary, violent, gross, sad or disturbing content is POSITIVE ("the villain is terrifying", "that scene was brutal, incredible", "I cried the whole time, beautiful").
- Hating a character, calling a character evil/a psycho, or describing a horrifying scene is NOT a negative verdict unless they criticise the film.
- Polite or soft disappointment is NEGATIVE ("I really wanted to like it", "it fell flat for me", "beautiful but I didn't care").
- Clear praise and clear criticism with no overall lean is MIXED. If they lean one way overall ("I liked it despite flaws"), pick that side.
- Plot discussion, theories, quotes, jokes, trivia, comments about the book/director/marketing without a judgement of this film, or not having seen it: NO_VERDICT.
Reply with JSON only: {"verdict": "positive" | "negative" | "mixed" | "no_verdict", "rationale": "<max 20 words>"}NLI hypotheses
- positive: The writer liked this movie or recommends it, even if they describe dark, scary or violent scenes.
- negative: The writer thinks this movie is bad or was disappointed by it.
- mixed: The writer has mixed feelings about this movie, praising some parts and criticising others.
- no_verdict: The writer is only discussing the plot, a character, a scene or trivia, without judging the movie.
Challenge set by type (correct / n)
| pathway | dark words, positive verdict | polite words, negative verdict | dark description, no verdict | mixed |
|---|---|---|---|---|
| VADER (old v1) | 28/39 | 10/19 | 1/11 | 0/2 |
| siebert+opinion (previous live) | 35/39 | 17/19 | 2/11 | 0/2 |
| Local NLI stance (deberta-v3-large) | 26/39 | 15/19 | 5/11 | 1/2 |
| GPT-6.1 Sol (LIVE) | 36/39 | 18/19 | 11/11 | 1/2 |
| Claude Opus 5.5 | 36/39 | 16/19 | 10/11 | 2/2 |
| Claude Sonnet 5.5 | 35/39 | 15/19 | 11/11 | 2/2 |
| Gemini 4 Argon | 36/39 | 15/19 | 10/11 | 2/2 |
| Grok 4.7 | 36/39 | 16/19 | 10/11 | 1/2 |
| GPT-6 Luna (cheap) | 36/39 | 18/19 | 10/11 | 2/2 |
| Claude Haiku 5.5 (cheap) | 36/39 | 16/19 | 10/11 | 2/2 |
| Gemini 3.5 Flash-Lite (cheap) | 35/39 | 14/19 | 10/11 | 2/2 |
Data: compare_pathways.json · stance_eval.json (every labelled comment, every pathway) · challenge labels