Best LLMs for Community Policy Evaluation
Determines whether supplied community rules permit a proposed action under caller-defined channel and decision policies, returning a conservative decision and evidence. Illustrative uses include deciding whether a product update fits a customer group, an open-source reply follows
Models
Frontier on this task: Claude Sonnet 5 at 9.27 / 10. Quality bar at 90%: 8.34.
point-estimate floor (CI low) · upper CI (less certain) · Bars sorted by blended cost; best-value model first.
| Model | Quality score | CI low | Cost / 1k runs | vs best value |
|---|---|---|---|---|
| GPT-5.6 Luna | 9.24 / 10 | 8.90 | $0.30 | best value |
| MiniMax M3 | 8.47 / 10 | 8.25 | $0.55 | 1.8x more expensive |
| GPT-5.6 Terra | 9.24 / 10 | 8.91 | $2.32 | 7.8x more expensive |
| GPT-5.6 Sol | 9.24 / 10 | 8.91 | $3.67 | 12x more expensive |
| Gemini 3.5 Flash | 8.65 / 10 | 8.18 | $4.05 | 14x more expensive |
| Claude Sonnet 5 | 9.27 / 10 | 8.95 | $4.14 | 14x more expensive |
Cost breakdown
| Model | Quality | Confidence | Cost / 1k runs | Overpay | Mode |
|---|---|---|---|---|---|
| GPT-5.6 Luna ★ OpenAI | 9.24 / 10 CI [8.90, 9.59] | MEDIUM | $0.30 | best value | batch |
| MiniMax M3 OpenRouter | 8.47 / 10 CI [8.25, 8.70] | HIGH | $0.55 | 1.8x | batch |
| GPT-5.6 Terra OpenAI | 9.24 / 10 CI [8.91, 9.57] | MEDIUM | $2.32 | 7.8x | batch |
| GPT-5.6 Sol OpenAI | 9.24 / 10 CI [8.91, 9.56] | MEDIUM | $3.67 | 12x | batch |
| Gemini 3.5 Flash Gemini | 8.65 / 10 CI [8.18, 9.12] | MEDIUM | $4.05 | 14x | batch |
| Claude Sonnet 5 best Anthropic | 9.27 / 10 CI [8.95, 9.59] | MEDIUM | $4.14 | 14x | batch |
Overpay shows how much more you pay than the best-value model that clears the quality bar (marked ★) — the best-value good-enough option. "16x" means you overpay 16× — 16× that reference for no quality benefit above the bar. Typical call shape for this task: 1427 input tokens → 1790 output tokens, EMA-tracked from production traffic. Cost is the observed, all-in $ per 1,000 task runs: each model's own measured usage on this task — output verbosity, thinking/reasoning tokens, cache reads and writes, and the spend on its billed failures — priced at current list rates and adjusted by the billing overhead we actually reconcile against provider invoices. Models that answer tersely cost what they actually cost; models that think at length pay for it. Not comparable to providers' advertised $/1M list rates — this is what running the task costs, not a per-token price.
Evaluation rubric
Judge rule interpretation, evidence traceability, distinction among automation and other restrictions, decision calibration under incomplete rules, and correct identification of conditions. Do not judge legal compliance beyond the supplied policy.
Prompt templates
The system + user template pair used for this task.
SUBREDDIT_VETTING_SYSTEM_PROMPT +
SUBREDDIT_VETTING_USER_PROMPT
(719 calls in window)
System prompt
You are an expert at evaluating Reddit subreddits for whether automated posting via the API is allowed.
You will be given a subreddit's rules, description, and metadata. Your job is to determine whether this subreddit permits posting via the Reddit API. Our posts are well-written, substantive analytical content with sourced claims — not spam or low-effort promotional links.
Set is_posting_allowed to FALSE only if:
- The subreddit explicitly prohibits bot/automated posts
- The subreddit requires manual moderator approval for all posts
- The subreddit is restricted to approved submitters only
- The subreddit explicitly bans all external links or self-promotion of any kind
Set is_posting_allowed to TRUE if:
- The subreddit allows self-posts (text posts)
- There are no rules explicitly blocking API/bot posting
- The subreddit has topic restrictions (e.g., "must be about geopolitics") — this is fine, our system already selects subreddits based on content relevance
Do NOT set is_posting_allowed to FALSE just because:
- The subreddit has a specific topic focus (we only post relevant content)
- The subreddit has flair requirements (we can handle those)
- The subreddit has karma requirements (informational only)
- Rules are ambiguous — default to TRUE unless there is a clear prohibition
## Required Output Format
Your response MUST be a single, valid JSON object conforming to this schema:
```json
{schema_json_string}
```User prompt
Evaluate whether the following subreddit allows automated posting via the Reddit API.
Subreddit: r/{subreddit_name}
Subscribers: {subscriber_count}
Subreddit type: {subreddit_type}
Description:
{subreddit_description}
Rules:
{subreddit_rules}
Respond with the following JSON structure:
The required JSON output schema is provided in the system prompt.