Best LLMs for Subreddit Quality Vetting
Decides whether a subreddit permits automated posting via the Reddit API for substantive analytical content. Fails closed only on explicit prohibitions; topic restrictions, flair requirements, and karma thresholds are not blockers.
Models
Frontier on this task: GPT-5.6 Luna at 9.48 / 10. Quality bar at 90%: 8.53.
point-estimate floor (CI low) · upper CI (less certain) · Bars sorted by blended cost; best-value model first.
| Model | Quality score | CI low | Cost / 1k runs | vs best value |
|---|---|---|---|---|
| MiniMax M3 | 8.57 / 10 | 8.35 | $0.53 | best value |
| GPT-5.6 Luna | 9.48 / 10 | 9.12 | $1.31 | 2.5x more expensive |
| GPT-5.6 Terra | 9.27 / 10 | 8.77 | $2.41 | 4.6x more expensive |
| Gemini 3.5 Flash | 8.70 / 10 | 8.22 | $3.84 | 7.3x more expensive |
| Claude Sonnet 5 | 9.28 / 10 | 9.03 | $4.04 | 7.7x more expensive |
| GPT-5.6 Sol | 9.28 / 10 | 8.80 | $5.19 | 9.8x more expensive |
Cost breakdown
| Model | Quality | Confidence | Cost / 1k runs | Overpay | Mode |
|---|---|---|---|---|---|
| MiniMax M3 ★ MiniMax | 8.57 / 10 CI [8.35, 8.79] | HIGH | $0.53 | best value | batch |
| GPT-5.6 Luna best OpenAI | 9.48 / 10 CI [9.12, 9.84] | MEDIUM | $1.31 | 2.5x | batch |
| GPT-5.6 Terra OpenAI | 9.27 / 10 CI [8.77, 9.76] | MEDIUM | $2.41 | 4.6x | batch |
| Gemini 3.5 Flash Gemini | 8.70 / 10 CI [8.22, 9.19] | MEDIUM | $3.84 | 7.3x | batch |
| Claude Sonnet 5 Anthropic | 9.28 / 10 CI [9.03, 9.52] | HIGH | $4.04 | 7.7x | batch |
| GPT-5.6 Sol OpenAI | 9.28 / 10 CI [8.80, 9.77] | MEDIUM | $5.19 | 9.8x | batch |
Overpay shows how much more you pay than the best-value model that clears the quality bar (marked ★) — the best-value good-enough option. "16x" means you overpay 16× — 16× that reference for no quality benefit above the bar. Typical call shape for this task: 1291 input tokens → 694 output tokens, EMA-tracked from production traffic. Cost is the observed, all-in $ per 1,000 task runs: each model's own measured usage on this task — output verbosity, thinking/reasoning tokens, cache reads and writes, and the spend on its billed failures — priced at current list rates and adjusted by the billing overhead we actually reconcile against provider invoices. Models that answer tersely cost what they actually cost; models that think at length pay for it. Not comparable to providers' advertised $/1M list rates — this is what running the task costs, not a per-token price.
Evaluation rubric
The output under evaluation is a subreddit vetting verdict: whether automated promotional posting is allowed on a given subreddit, with a summary of the subreddit's rules and reasoning. Judge the correctness of the verdict against the rules presented: - An EXPLICIT prohibition of bots, automation, self-promotion, or advertising in the subreddit's rules means `is_posting_allowed` MUST be false. The poster is an automated system — "default to TRUE when ambiguous" applies ONLY when the rules are genuinely silent on bots/automation/promotion, not when an explicit rule is present. Penalize TRUE verdicts that contradict an explicit prohibition; reward FALSE verdicts grounded in one. - The rule summary must be faithful: penalize verdicts whose reasoning misstates, omits, or invents subreddit rules. - Reasoning should connect the cited rules to the verdict. OUT OF SCOPE — do not reward or penalize: - Output structure, schema, field naming, or formatting details. - Verbosity or style of the reasoning text.
Prompt templates
The system + user template pair used for this task.
SUBREDDIT_VETTING_SYSTEM_PROMPT +
SUBREDDIT_VETTING_USER_PROMPT
(892 calls in window)
System prompt
You are an expert at evaluating Reddit subreddits for whether automated posting via the API is allowed.
You will be given a subreddit's rules, description, and metadata. Your job is to determine whether this subreddit permits posting via the Reddit API. Our posts are well-written, substantive analytical content with sourced claims — not spam or low-effort promotional links.
Set is_posting_allowed to FALSE only if:
- The subreddit explicitly prohibits bot/automated posts
- The subreddit requires manual moderator approval for all posts
- The subreddit is restricted to approved submitters only
- The subreddit explicitly bans all external links or self-promotion of any kind
Set is_posting_allowed to TRUE if:
- The subreddit allows self-posts (text posts)
- There are no rules explicitly blocking API/bot posting
- The subreddit has topic restrictions (e.g., "must be about geopolitics") — this is fine, our system already selects subreddits based on content relevance
Do NOT set is_posting_allowed to FALSE just because:
- The subreddit has a specific topic focus (we only post relevant content)
- The subreddit has flair requirements (we can handle those)
- The subreddit has karma requirements (informational only)
- Rules are ambiguous — default to TRUE unless there is a clear prohibition
## Required Output Format
Your response MUST be a single, valid JSON object conforming to this schema:
```json
{schema_json_string}
```User prompt
Evaluate whether the following subreddit allows automated posting via the Reddit API.
Subreddit: r/{subreddit_name}
Subscribers: {subscriber_count}
Subreddit type: {subreddit_type}
Description:
{subreddit_description}
Rules:
{subreddit_rules}
Respond with the following JSON structure:
The required JSON output schema is provided in the system prompt.