How the decision is formed
0. Comment sentiment model
| model | openrouter:openai/gpt-6.1-sol via OpenRouter, prompt stance-v1, temperature 0, JSON output. Asks: did the commenter like or recommend the film? → positive / negative / mixed / no verdict. positive = +1, negative = −1, mixed and no verdict = neutral (don't count). |
| why | Most accurate pathway on my labelled sets: held-out 87% (n=112), challenge 93% (n=71), ~$1.47 per full cold run. Previous live model siebert+opinion: held-out 59% (n=112), challenge 76% (n=71), $0, runs locally per full cold run. All pathways side by side on Compare. |
| cache | Each comment's verdict is cached in cache/stance/ (key = sha1(prompt version | film | comment text)). This run: 1381 cached verdicts, 0 API calls (built entirely from cache). A new comment, film or prompt version is a cache miss and is scored once. |
| cheap rerun | MR_MODEL=openrouter:openai/gpt-6-luna (GPT-6 Luna): held-out 80% (n=112), challenge 93% (n=71), ~$0.10 per full cold run. Same prompt and buckets; ranks within ~1 place of Sol this week. |
| other options | MR_SENTIMENT=siebert+opinion (previous live: local open-weights siebert/roberta-large + zero-shot opinion gate, no API key) and MR_SENTIMENT=vader (v1) still work and are shown on Compare. VADER compound is still computed per comment for comparison and for theme sentences, but not used for the score. |
Prompt
You classify a Reddit comment from a movie's discussion thread by the commenter's VERDICT ON THE FILM ITSELF: did they like it / would they recommend it?
Judge the verdict, not the tone or vocabulary.
- Praise of dark, scary, violent, gross, sad or disturbing content is POSITIVE ("the villain is terrifying", "that scene was brutal, incredible", "I cried the whole time, beautiful").
- Hating a character, calling a character evil/a psycho, or describing a horrifying scene is NOT a negative verdict unless they criticise the film.
- Polite or soft disappointment is NEGATIVE ("I really wanted to like it", "it fell flat for me", "beautiful but I didn't care").
- Clear praise and clear criticism with no overall lean is MIXED. If they lean one way overall ("I liked it despite flaws"), pick that side.
- Plot discussion, theories, quotes, jokes, trivia, comments about the book/director/marketing without a judgement of this film, or not having seen it: NO_VERDICT.
Reply with JSON only: {"verdict": "positive" | "negative" | "mixed" | "no_verdict", "rationale": "<max 20 words>"}1. Signals, each normalised to 0-100
| signal | weight | normalisation | source |
|---|---|---|---|
| 0.45 | community score = 100 × pos_share; pos_share = Σw(positive comments) / Σw(positive + negative comments). Neutral comments don't count. Comment labels come from the sentiment model in section 0. Comment weight = base / √(rank+1) where rank is the position in the thread's top-sorted RSS feed (RSS has no vote counts). base = 1 for r/movies Official Discussion comments, 0.5 for r/MovieSuggestions / r/TrueFilm posts whose title names the film. Comments <15 chars and moderator/AMA notices are excluded. Needs ≥5 scored comments. | reddit.com public RSS | |
| audience | 0.2 | Audience score (audience aggregate; verified audience when available), as-is | critics/audience source |
| critics | 0.2 | Critics score (critics aggregate), as-is | critics/audience source |
| cinemascore | 0.05 | A+=100, A=92, A-=85, B+=77, B=70, B-=62, C+=55, C=47, C-=40, D+=30, D=20, D-=10, F=0 | r/boxoffice post titles that contain the grade and the film's title |
| boxoffice | 0.1 | New (≤7 days): 25·log10(daily $/theater) − 25, clamped 0-100. Holdover: 50 + (pct_lw + 50)·1.25, clamped; pct_lw = daily gross vs same weekday last week. Not scored if theater count moved more than ±40% vs last week. | boxofficemojo.com daily chart |
2. Composite
score = Σ weight·value / Σ weight over available signals only. A missing signal is dropped and the others are re-weighted; it is not counted as zero.
3. Confidence
confidence = Σ weight(available) × min(1, 0.5 + n_reddit/120). The thread factor applies only when Reddit data exists, so a movie without a thread can reach at most 0.55.
4. Bucket
| bucket | rule (all must hold) |
|---|---|
| Watch this | score ≥ max(75, Q3 of this week's scores) = 90.45 this run · confidence ≥ 0.55 · Reddit thread data present · |Audience − Critics score| < 30 · Reddit consensus ≥ 0.15 (consensus = |pos − neg| / (pos + neg), weighted; v1 used 1 − sd(VADER)/0.8, which is scale-dependent and collapses to 0 for model probabilities) |
| Skip | score < 55 · confidence ≥ 0.45 |
| Depends on your taste | everything else (high score but low confidence or polarized, or mid-range) |
Within each bucket, movies are sorted by score.
4a. Why the Watch bar is relative (rule v2, 2026-10-25)
Rule. Q3 = 75th percentile of the 15 ranked films' scores this run (statistics.quantiles(scores, n=4, method="inclusive")[2], the same as numpy's default “linear”). This run Q3 = 90.45, so the bar is 90.45 (Q3 binds). Watch still also needs every other gate above. Skip is unchanged.
Why. The Reddit signal is the share of opinionated commenters who liked the film, in that film's own discussion thread, and people who bother to post there mostly liked it. With a stance model that reads verdicts correctly, that share is 93–98% for most films, so the composite crowds into the high 80s and 90s. Under the old absolute bar (score ≥ 75) 8 of 15 films were “Watch”, which stopped being a recommendation. “Watch this” is meant as the best of what's playing now, so the bar is the top quarter of this week's slate, and never below the old floor of 75 (in a weak week nothing under 75 gets recommended). Because it is relative, the same rule works for every pathway: a model that rates everything higher moves Q3 up with it (VADER, siebert and all 8 LLM pathways each get 4 Watch this week; the large NLI model, which scores lower overall, hits the 75 floor and gets 1). Skip stays absolute (score < 55): it already picked the same 4 films under every pathway, and a fixed share of Skips would force a film into Skip in a good week.
Weights are unchanged (reddit 0.45, audience 0.2, critics 0.2, cinemascore 0.05, boxoffice 0.1).
| v1 (until 2026-10-25) | v2 (now) | |
|---|---|---|
| Watch score bar | score ≥ 75 | score ≥ max(75, Q3 of slate) = 90.45 |
| other Watch gates | confidence ≥ 0.55 · Reddit present · |aud − crit| < 30 · consensus ≥ 0.15 (unchanged) | |
| Skip | score < 55 · confidence ≥ 0.45 (unchanged) | |
| consensus | redefined earlier (commit 501f2c3, with the move off VADER): |pos − neg| / (pos + neg), weighted. v1 was 1 − sd(VADER compound)/0.8, which is scale-dependent and collapses to 0 for model probabilities. | |
Edge: a film within a point of the bar can flip with small changes (this week Primetime is 89.8 vs a bar of 90.45). The bar is in every film's bucket trace under “Under the hood”. Change history: Run log → Changes.
5. Themes (praised / gripes)
Each non-spoiler sentence with |VADER| ≥ 0.2 (themes still use per-sentence VADER; the model is only used for whole-comment polarity) is tagged with aspects by keyword (performances, visuals, story, pacing, …). Sentences containing spoiler words (ending, twist, dies, reveal, …) are dropped. Praised: ≥3 hits and ≥65% of the weight positive. Gripe: ≥3 hits, ≥45% negative and negative weight > 0.3. Themes don't affect the score.