Best LLMs for Content Domain Suggestion

Category: Relevance, Classification & Matching · Rail: absolute · Typical I/O: 5201→1289 tokens

Models

Frontier on this task: Claude Opus 5 at 9.18 / 10. Quality bar at 90%: 8.26.

point-estimate floor (CI low) · upper CI (less certain) · Bars sorted by blended cost; best-value model first. Greyed rows are MEDIUM+ models whose point estimate clears the bar but whose CI low does not.

Model	Quality score	CI low	Cost / 1k runs	vs best value
Claude Opus 5	9.18 / 10	8.75	$23.83	best value
Qwen 3.8 Flash	4.16 / 10	3.93	$1.24	95% cheaper
Gemini 3.8 Flash	4.51 / 10	4.05	$4.53	81% cheaper
Meta Muse Spark 1.3	4.13 / 10	3.97	$10.42	56% cheaper
Tencent Hy4 Preview	4.55 / 10	4.07	$18.84	21% cheaper
NVIDIA Nemotron 3.5 Lightning	4.27 / 10	3.82	$0.90	96% cheaper
Grok 4.6	4.11 / 10	3.94	$17.23	28% cheaper
Qwen 3.8 Max	4.16 / 10	3.96	$15.36	36% cheaper
Claude Sonnet 5	5.92 / 10	5.68	$9.36	61% cheaper
Gemini 3.5 Flash Lite	4.04 / 10	3.93	$0.79	97% cheaper
GPT-5.6 Luna	4.78 / 10	4.38	$0.23	99% cheaper
GPT-5.6 Sol	3.84 / 10	3.74	$2.59	89% cheaper
GPT-5.6 Terra	3.87 / 10	3.67	$1.24	95% cheaper
NVIDIA Nemotron-3 Super 120B	4.36 / 10	3.96	$1.76	93% cheaper
Tencent Hy3	5.15 / 10	4.74	$0.32	99% cheaper
Thinking Machines Inkling	4.33 / 10	3.88	$8.39	65% cheaper
Thinking Machines Inkling Small	7.59 / 10	7.14	$4.94	79% cheaper

Cost breakdown

Model	Quality	Confidence	Cost / 1k runs	Overpay	Mode
Claude Opus 5 ★ best Anthropic	9.18 / 10 CI [8.75, 9.61]	MEDIUM	$23.83	best value	batch

Overpay shows how much more you pay than the best-value model that clears the quality bar (marked ★) — the best-value good-enough option. "16x" means you overpay 16× — 16× that reference for no quality benefit above the bar. Typical call shape for this task: 5201 input tokens → 1289 output tokens, EMA-tracked from production traffic. Cost is the observed, all-in $ per 1,000 task runs: each model's own measured usage on this task — output verbosity, thinking/reasoning tokens, cache reads and writes, and the spend on its billed failures — priced at current list rates and adjusted by the billing overhead we actually reconcile against provider invoices. Models that answer tersely cost what they actually cost; models that think at length pay for it. Not comparable to providers' advertised $/1M list rates — this is what running the task costs, not a per-token price.

Evaluation rubric

Judge semantic fit, coverage of the material’s principal subjects, label clarity, useful granularity, and reuse of equivalent existing labels. Penalize method/format labels, redundant synonyms, vague umbrella terms, and unsupported domains. Schema and item-count enforcement are out of scope.

Prompt templates

The system + user template pair used for this task.

AT_CONTENT_DOMAIN_SUGGEST_SYSTEM + AT_CONTENT_DOMAIN_SUGGEST_USER (407 calls in window)

System prompt

You are an editor configuring content domains for an analysis template.

Domains must name the SUBJECT or VERTICAL the template is about (the "what"), NOT the business function or report lens (the "how"). Extract the concrete topic from the template name, description, and chapter content.

**What we want (subject/vertical domains):** Menopause, Femtech, Supplements, Women's Health, Biotech, Consumer Health, Life Sciences, Wellness — i.e. the actual area (menopause, supplements, femtech, health…).

**What we do NOT want (generic business/functional domains):** Business Strategy, Operations, Regulatory Affairs, Supply Chain, Market Analysis, Competitive Analysis, Innovation, Consumer Insights — these describe how the report is structured, not what it is about. A chapter on "regulatory for menopause supplements" should yield domains like "Menopause" or "Supplements", not "Regulatory Affairs". A chapter on "supply chain for menopause supplements" → "Menopause" or "Supplements", not "Supply Chain".

- When the template and chapters clearly focus on a vertical (e.g. menopause, supplements, femtech, women's health): assign 1–3 specific vertical domains such as "Menopause", "Femtech", "Supplements", "Women's Health". Create them in new_domains_to_create if they are missing from existing domains. Prefer these over any generic "Business Strategy", "Operations", or "Regulatory" style domain.
- Finance and Economics: only for financial markets, corporate finance, trading, macroeconomics. Do NOT use when the substance is health, supplements, ingredients, or wellness.
- If existing domains already include the right vertical names, use them and set new_domains_to_create to [].
- New names must NOT duplicate existing domain names.

Output valid JSON in exactly the structure of the schema provided.

## Required Output Format
Your response MUST be a single, valid JSON object conforming to this schema:
```json
{schema_json_string}
```

User prompt

## Report structure

{report_structure_summary}

## Existing domains

{existing_domains_json}

## Template

name: {at_name}
description: {at_description}
subject: {at_subject_name} ({at_subject_code})

## Instructions

Pick domains that name the SUBJECT/VERTICAL (the "what"): e.g. Menopause, Femtech, Supplements, Women's Health. Do NOT pick generic business/functional domains like Business Strategy, Operations, Regulatory Affairs, Supply Chain, Market Analysis, Competitive Analysis, Innovation, Consumer Insights — those are report angles, not the topic. If the template and chapters are about menopause, supplements, femtech, or women's health, assign domains such as "Menopause", "Femtech", "Supplements", "Women's Health" (create them in new_domains_to_create if missing). Prefer 1–3 specific vertical domains over one broad "Business" or "Operations" style domain.

## Output

Respond with valid JSON in exactly this structure:

The required JSON output schema is provided in the system prompt.