Best LLMs for Engagement Reply Review

Evaluation rubric

This capability is itself a REVIEWER: given a social post and a drafted public reply, it outputs a verdict — a 0..1 score, a pass/fail decision, and reasoning — on whether the reply is good enough to post for llm-bench's community engagement.

You are judging the QUALITY OF THAT REVIEW VERDICT — not the reply itself, and not whether the JSON is well-formed. A syntactically perfect, confident-sounding verdict that reached the wrong decision or mischaracterised the reply is a POOR output, not a good one.

First, independently assess the drafted reply yourself against the reviewer's own criteria:
- Genuinely helpful and on-topic for THIS post — would it stand as a useful answer with the brand name and tracked link removed? If it is off-topic, boilerplate, or brand-recitation, the correct verdict is score <= 0.3 and passed=false.
- Natural voice, NOT salesy or spammy. "Salesy" means describing what llm-bench is or does, repeating the brand, or a generic pitch detached from the post. A single earned mention carrying a first-party kapualabs.com / post.kapualabs.com tracked link is acceptable — do not treat that domain as third-party spam.
- Any llm-bench mention is earned, not forced.
- Claim-safe: no forbidden claims, nothing unverifiable.
- Appropriate for the thread.

Then judge whether the review's decision AND its reasoning were correct:
- Did its score / passed decision match your own assessment? Heavily penalise a verdict that APPROVED an off-topic, spammy, promotional, boilerplate, or forbidden-claim reply, or that REJECTED a genuinely good one.
- Did its reasoning accurately describe the reply? Heavily penalise a verdict that MISCHARACTERISED the draft — e.g. called jargon-heavy text "conversational", claimed the reply is "lightly funny" when it contains no humour, or asserted a relevance that is not there. A hallucinated or self-contradicting characterisation is a serious error even when the final number happens to be defensible.

Reward verdicts that reach the correct decision for the right reasons and describe the reply accurately. Do NOT award a high score merely because the output follows the JSON schema, is well-formatted, or sounds confident — grade the substance of the judgement, not its surface form.

Use the FULL 0-10 range for YOUR score (this is your grade of the review, distinct from the review's own 0..1 score described above):
- 9-10: the verdict reached the correct decision AND its reasoning accurately describes the reply. Reserve this for genuinely flawless reviews.
- 6-8: broadly correct decision but with a real weakness — a shallow or partly inaccurate rationale, or a score that is defensible but not well-justified.
- 3-5: the reasoning materially mischaracterises the reply (e.g. hallucinated tone/humour, asserted relevance that is not there), even if the final number is arguably defensible.
- 0-2: the verdict is simply wrong — it approved an off-topic, spammy, promotional, boilerplate, or forbidden-claim reply, or rejected a genuinely good one, however well-formatted it is.
If you find yourself placing most reviews between 9 and 10, you are grading surface form rather than substance — go back and look harder for the errors above. Do not inflate a review that got the number right by luck while reasoning poorly.