Scores You Can—and Cannot—Trust Yet

A model is good enough for production only when it clears the required quality bar and produces an accepted result reliably enough for the workflow. The measurements here show why those conditions must be evaluated together: a high score describes the quality of returned work, but not necessarily the proportion of requests that receive an answer, the cost of serving it, or whether the result can be used without retries, fallbacks, or repair.

Strong scores do not guarantee dependable completion

For engagement triage, gpt-5.5 scores 8.5 out of 10 with an uncertainty of ±0.18 after 164 independent judgments. That falls within the ±0.2 threshold used for a ranked result, so the quality measurement clears that stated ranking threshold [^26] [^9]. It supports considering gpt-5.5 when quality is the primary requirement, but the supplied evidence does not establish that it is cheaper, faster, or better than another model for this task.

Answer availability can matter more than a higher conditional score. For generating themes, Minimax-served minimax/m3 scores 8.63 out of 10 but answers on 0% of attempts, while NVIDIA-served nvidia/nemotron-3-super-120b-a12b scores 7.79 out of 10 and answers on 73% of attempts [^17]. Claude Sonnet 5 scores 9.51 out of 10 for section generation but returns an answer on only 5% of attempts [^11]. Those results make the operational requirement explicit: a conditional quality score is not evidence of dependable automated completion when unanswered requests are unacceptable.

The same pattern appears in other measured tasks. gpt-5.6-sol scores 8.88 out of 10 for grouping related topics while answering 73% of attempts, 8.94 out of 10 for analyzing prospective customers while answering 14% of attempts, and 9.17 out of 10 for newsletter drafting while answering 0% of attempts [^13] [^25] [^8]. Qwen3.7-plus scores 9.67 out of 10 for structured extraction but answers only 40% of attempts, while NVIDIA-served nvidia/nemotron-3-super-120b-a12b scores 9.09 out of 10 for generating onboarding chapters and answers on 0% of attempts [^16] [^21]. These are not safe defaults for workflows that require every request to complete without fallback handling, regardless of their headline quality scores.

Summarization provides a more favorable comparison. gpt-5.4-nano scores 9.33 out of 10 with an 87% answer rate, compared with gpt-5.6-sol at 8.95 out of 10 and a 54% answer rate; the gpt-5.4-nano result is also reported with ±0.1 after 374 independent judgments [^18]. Among those results, gpt-5.4-nano is the better-supported choice for a workflow in which completion matters, although the evidence does not establish that it is the cheapest or fastest option.

Route by task, not by model reputation

The scores apply to the measured work, not to a model’s general ability. Kimi-k3 scores 8.47 out of 10 for determining topic order, 8.98 out of 10 for naming topic clusters, and 8.35 out of 10 for matching topics to clients, but only 3.82 out of 10 for suggesting a content domain [^20] [^7] [^3] [^10]. Tencent/hy3 is stronger on content-domain suggestion at 5.21 out of 10, yet scores 6.68 out of 10 for topic-to-client matching [^10] [^3]. Claude Sonnet 5 scores 1.46 out of 10 for discovering and clustering topics, 9.7 out of 10 for generating themes, and 8.72 out of 10 for synthesis and analysis [^13] [^17] [^4][^5]. Claude Haiku 4.5 is better supported for selecting X posts, at 8.08 out of 10, than for scoring X-post relevance, at 4.61 out of 10 [^15] [^2][^12][^22].

The practical implication is to split routing by stage. A model that is strong at downstream writing or analysis may be a poor choice for discovery, while a model that performs well at selecting material may be unsuitable for judging its relevance. The evidence supports choosing a model for each measured task rather than assigning one model to an entire workflow.

Where the quality comparisons are inconclusive

Not every numerical gap supports a meaningful winner. For naming topic clusters, claude-opus-4-8 scores 8.75 out of 10 with ±0.36, versus qwen3.7-plus at 8.09 out of 10 with ±0.45; the intervals overlap [^7]. The point estimates differ, but the reported uncertainty does not justify treating the higher score as a settled advantage.

The same caution applies to content-domain suggestion. qwen3.7-flash scores 3.63 out of 10 with ±0.12, kimi-k3 scores 3.82 out of 10 with ±0.32, and gpt-5.6-sol scores 3.84 out of 10 with ±0.18. Their uncertainty ranges overlap, making the quality results difficult to distinguish [^10]. DeepSeek-V4-Pro and DeepSeek-V4-Flash are likewise indistinguishable for subreddit selection, at 4.82 out of 10 with ±0.25 and 4.74 out of 10 with ±0.21, respectively [^19].

For theme generation, qwen3.7-flash and qwen3.7-plus are effectively tied at 5.83 out of 10 with ±0.41 and 5.89 out of 10 with ±0.13 [^17]. In that case, qwen3.7-flash costs $0.0014 per task run and qwen3.7-plus costs $0.0054 per task run [^17]. The lower-cost option is the practical choice only if its serving conditions and answer availability also meet the application’s requirements; the overlapping intervals do not support paying more solely for a quality advantage that the evidence cannot clearly establish.

Cost comparisons require matching the serving mode

Cost is a production constraint, but figures from different serving modes should not be treated as direct price rankings. For generating an auto Reddit post, Qwen3.7-plus scores 8.04 out of 10 with ±0.26 at $0.0098 per task run in sync mode, while Claude Sonnet 5 scores 8.37 out of 10 with ±0.16 at $0.08 per task run in batch mode [^14]. Claude Sonnet 5 has the higher measured score, and Qwen3.7-plus the lower listed cost, but the different modes prevent a like-for-like price comparison.

The same distinction matters for gpt-5.5. Its trading-recommendation task costs $0.27 per task in batch mode versus $0.56 synchronously, while claim extraction costs $0.02 in batch mode versus $0.06 synchronously [^24] [^6]. Gemini batch serving is cheaper than synchronous serving for the listed workflows, but the supplied results do not establish that the cheaper mode meets a higher quality bar [^1] [^23] [^6]. Batch is therefore a cost option when latency can be deferred, not an automatic quality or deployment recommendation.

What the uncertainty can and cannot settle

The reported intervals show that confidence matters, but the supplied findings do not provide a distribution of measured cells across confidence-level categories. They also do not provide enough information to characterize uncertainty for every model-task cell. The explicit judgment counts are 164 for gpt-5.5’s engagement-triage result and 374 for gpt-5.4-nano’s summarization result [^26] [^18]; those counts should not be generalized into a claim about the broader measurement set.

Some apparently impressive results are therefore not safe to act on without confirming the operational path. Minimax-served minimax/m3’s 8.63 for theme generation is paired with a 0% answer rate, Claude Sonnet 5’s 9.51 for section generation with a 5% answer rate, gpt-5.6-sol’s 9.17 for newsletter drafting with a 0% answer rate, and NVIDIA-served nvidia/nemotron-3-super-120b-a12b’s 9.09 for onboarding-chapter generation with a 0% answer rate [^17] [^11] [^8] [^21]. These results could become actionable only if repeated measurement shows materially better answer availability under the intended serving conditions and the resulting outputs meet the workflow’s acceptance requirements.

Likewise, the apparent advantage of claude-opus-4-8 over qwen3.7-plus for naming topic clusters, or of qwen3.7-plus over qwen3.7-flash for theme generation, is not yet a safe basis for quality-first routing when the intervals overlap [^7] [^17]. A stronger decision would require uncertainty narrow enough to separate the alternatives, or an operational difference in answer rate, cost, latency, or accepted-output rate that is large enough to decide the route independently of the quality tie.

Routing conclusion

Set the quality bar per task, then require the model to meet the workflow’s completion requirement before comparing cost. For workflows where every request must complete, favor gpt-5.4-nano over gpt-5.6-sol for the measured summarization comparison because it combines the higher score with the higher answer rate [^18]. For theme generation, do not route automatically to Minimax-served minimax/m3 or Claude Sonnet 5 merely because their conditional scores are high; their 0% and 5% answer rates make them unsuitable defaults when retries and fallback paths are unacceptable [^17] [^11].

Use task-specific routing for Kimi-k3, Tencent/hy3, Claude Sonnet 5, and Claude Haiku 4.5 rather than treating any of them as a universal choice, because their measured strengths vary sharply by task [^20] [^7] [^3] [^10] [^3] [^13] [^17] [^4][^5] [^15] [^2][^12][^22]. Where quality intervals overlap, treat the models as tied and choose on answer availability, matched serving cost, latency, and the cost of recovery. In all cases, compare the cost of a clean, accepted output—not merely the nominal cost of an attempted call—and do not promote a strong but low-answer-rate result to production until its completion path has been demonstrated under the intended conditions.

The live benchmark is available at https://llm-bench.kapualabs.com/, where you can set the quality bar your workflow actually needs. Use it to compare the relevant task cells and check batch against sync pricing yourself before routing production traffic.