Open-Weight LLMs Benchmarked (2026): Llama 4 vs DeepSeek V3 vs Qwen 3 vs Mistral Large 3
We benchmarked 6 open-weight LLMs on identical H200 infrastructure across 1,400 prompts. DeepSeek V3 leads on coding and math, Qwen 3 on long-context — both now rival mid-tier proprietary models at a fraction of inference cost.
Dr. Elena Vasquez
Technical Research Director
In this report
Executive Summary
The open-weight frontier closed much of the gap to proprietary models in the first half of 2026. We benchmarked 6 open-weight LLMs — Llama 4 Maverick, DeepSeek V3, Qwen 3 235B, Mistral Large 3, Gemma 3 27B, and Command R+ 2 — across 1,400 prompts in reasoning, coding, math, and long-context retrieval.
All models were run on identical H200 infrastructure to control for serving differences, measuring quality, throughput (tokens/sec), and cost per million tokens at self-hosted scale.
Key finding: DeepSeek V3 and Qwen 3 trade the top spot depending on task — DeepSeek leads on math and coding, Qwen 3 on multilingual and long-context. Both now rival mid-tier proprietary models at a fraction of the inference cost.
Comparative Rankings
| Model | Reasoning | Coding | Long-Context | Throughput (tok/s) | Overall |
|---|---|---|---|---|---|
| DeepSeek V3 | 8.9 | 9.0 | 8.5 | 71 | 8.8 |
| Qwen 3 235B | 8.8 | 8.6 | 9.0 | 64 | 8.7 |
| Llama 4 Maverick | 8.6 | 8.4 | 8.7 | 88 | 8.6 |
| Mistral Large 3 | 8.4 | 8.3 | 8.1 | 79 | 8.3 |
| Gemma 3 27B | 8.0 | 7.9 | 7.6 | 112 | 7.9 |
Key Findings
1. The Open/Closed Quality Gap Is Now ~6 Months, Not 18
On our reasoning and coding suites, the best open-weight models now match proprietary releases from roughly two quarters prior. For many production workloads, that gap no longer justifies frontier API pricing.
2. Long-Context Retrieval Favors Qwen
At 128K+ context, Qwen 3 maintained the highest retrieval accuracy in our needle-in-a-haystack tests, with graceful degradation rather than the sharp cliffs we observed in smaller models.
3. Self-Hosting Economics Have Shifted
At sustained utilization above ~40%, self-hosting DeepSeek V3 or Llama 4 on rented H200 capacity undercut equivalent-quality API calls by 60–80%. Below that utilization, serverless API pricing still wins.
Recommendations
For coding and math workloads: DeepSeek V3 delivers the strongest quality-per-dollar of any open-weight model tested.
For multilingual and long-document tasks: Qwen 3 235B is the most capable open option.
For latency-sensitive, high-throughput serving: Llama 4 Maverick and Gemma 3 offer the best tokens-per-second on commodity hardware.
Dr. Elena Vasquez
Technical Research Director
Former ML engineer at DeepMind. Leads Aldric's technical benchmarking methodology. PhD in Machine Learning from MIT.
Get the Full Dataset
Subscribe for access to our complete research data, methodology documentation, and weekly intelligence briefings.
Subscribe to Aldric Research