Best LLMs for Social Post Relevance Scoring

Evaluation rubric

The output under evaluation is a relevance assessment of retrieved posts/
articles against an editorial scoring context (the analysis brief): per item,
an id and a relevance score, optionally with brief reasoning.

Judge ONLY the correctness of the relevance assessments themselves. For each
assessed item the question is: is this relevance score a defensible reflection
of how well that item's content matches the scoring context?

Correctness checks:
- Clearly on-topic items must carry high scores; clearly off-topic items low
  scores. Wrong-direction assignments are the primary quality failure.
- Scores should discriminate between items that visibly differ in topical
  fit. A run of identical scores across plainly-different items is degenerate
  output, not calibrated judgement.
- Every assessed id must exist in the input — hallucinated ids are a failure.
- Each item must receive ONE unambiguous verdict. The same item assessed as
  both relevant and irrelevant is an assessment defect — penalize it.

OUT OF SCOPE — do not reward or penalize:
- Output structure, schema, field naming, JSON shape, or formatting.
  Structural deviations are repaired by downstream validation before the
  output is consumed; a non-conforming layout carrying correct relevance
  judgments is still good output. Grade content only.
- Response completeness. If some input items are missing from the output,
  grade only the entries that are present, each on its own merit.
- Verbosity, phrasing, or style of optional reasoning text; decimal
  precision; field ordering.

Penalize:
- Demonstrably wrong assignments (high score on clearly off-topic content,
  low score on clearly on-topic content).
- Degenerate batches: all-zero, all-one, or all-identical scores when the
  input items visibly differ in topical fit.
- Hallucinated ids, or contradictory duplicate verdicts for the same item.