LM Benchmarks
Beta| Model | Size | Release | Updated | LBCode | Aider | HumEval | HumEval+ | ArenaCode | LBReason | MMLU Pro | MMLU | GPQA | MATH | LBMath | Arena | AA Score | In Cost | Out Cost | Cutoff | Context | Org | Type | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
  | o1 | Unknown | 69.7% | 84.2% | 91.6% | 76.0% | 67.3% | OpenAI | Proprietary | ||||||||||||||
  | Claude 3.5 Sonnet | Unknown | 67.1% | 84.2% | 93.7% | 85.9% | 1321 | 58.7% | 78.0% | 88.0% | 65.0% | 78.3% | 51.3% | 1282 | 80 | $3 | $15 | Apr 24 | 200K | Anthropic | Proprietary | ||
 | Gemini Exp 1206 | Unknown | 63.4% | 80.5% | 1385 | 57.0% | 70.0% | 1379 | 2M | Proprietary | |||||||||||||
| | Claude 3.5 Sonnet | Unknown | 60.9% | 77.4% | 92.0% | 1295 | 58.7% | 76.1% | 88.7% | 59.4% | 71.1% | 53.3% | 1268 | 76 | $3 | $15 | Apr 24 | 200K | Anthropic | Proprietary | |||
 | Qwen 2.5 | 72B | Unknown | 57.6% | 55.6% | 86.6% | 51.2% | 1283 | 46.0% | 71.1% | 86.8% | 49.0% | 83.1% | 52.4% | 1259 | 75 | $0.38 | $0.57 | Sep 24 | 128K | Alibaba | Open Source | |
| Qwen 2.5 Coder | 32B | Unknown | 56.9% | 72.9% | 92.7% | 86.3% | 1264 | 47.3% | 79.0% | 41.0% | 72.0% | 46.0% | 1220 | 70 | $0.2 | $0.2 | Sep 24 | 128K | Alibaba | Open Source | ||
| Gemini 2.0 Flash Exp | Unknown | 54.4% | 69.9% | 91.0% | 1341 | 59.1% | 76.4% | 87.0% | 62.1% | 89.7% | 59.4% | 1356 | 82 | Aug 24 | 1M | Proprietary | ||||||
| Gemini 2 Flash Thinking | Unknown | 53.1% | 1343 | 64.6% | 69.0% | 1369 | 32K | Proprietary | ||||||||||||||
| Gemini Exp 1114 | Unknown | 52.4% | 60.9% | 1324 | 54.7% | 54.9% | 1343 | 32K | Proprietary | |||||||||||||
| | GPT 4o | Unknown | 51.4% | 71.4% | 90.0% | 87.2% | 1274 | 54.7% | 74.7% | 89.0% | 51.0% | 78.0% | 48.2% | 1265 | 77 | $2.5 | $10 | Oct 23 | 128K | OpenAI | Proprietary | ||
| | Claude 3.5 Haiku | Unknown | 51.4% | 75.2% | 88.1% | 1269 | 29.3% | 65.0% | 81.0% | 41.6% | 73.0% | 35.5% | 1239 | 69 | $1 | $5 | Apr 24 | 200K | Anthropic | Proprietary | |||
| | o1 Preview | Unknown | 50.8% | 79.7% | 92.4% | 89.0% | 1355 | 68.0% | 91.0% | 73.3% | 85.0% | 62.9% | 1333 | 85 | $15 | $60 | Oct 23 | 128K | OpenAI | Proprietary | |||
 | Deepseek 3 | 671B | Unknown | 79.7% | 82.6% | 75.9% | 88.5% | 59.1% | $0.07 | $1.1 | 128K | DeepSeek | Open Source | ||||||||||
| Gemini Exp 1121 | Unknown | 50.4% | 57.9% | 1341 | 45.3% | 62.7% | 1365 | 32K | Proprietary | |||||||||||||
| Gemini 1.5 Pro 002 | Unknown | 48.8% | 66.9% | 84.1% | 79.3% | 1290 | 46.0% | 75.8% | 86.0% | 59.1% | 86.5% | 57.4% | 1301 | 80 | $1.25 | $2.5 | Nov 23 | 2M | Proprietary | |||
| | o1 mini | Unknown | 48.0% | 70.7% | 92.4% | 89.0% | 1362 | 77.3% | 85.0% | 60.0% | 90.0% | 59.2% | 1308 | 82 | $3 | $12 | Oct 23 | 128K | OpenAI | Proprietary | |||
 | Mistral Large | 123B | Unknown | 47.1% | 65.4% | 90.0% | 1277 | 43.4% | 84.0% | 47.0% | 72.0% | 32.6% | 1251 | 74 | $2 | $6 | Jul 24 | 128K | Mistral | Proprietary | |||
| Mistral Large | 123B | Unknown | 47.1% | 60.2% | 91.3% | 62.2% | 1269 | 42.0% | 65.9% | 85.0% | 48.0% | 72.0% | 43.7% | 1250 | 74 | $2 | $6 | Jul 24 | 128K | Mistral | Proprietary | |
| DeepSeek 2.5 | 236B | Unknown | 46.3% | 72.2% | 89.0% | 83.5% | 1288 | 39.3% | 65.8% | 81.0% | 42.0% | 55.0% | 1221 | 66 | $0.14 | $0.28 | Nov 23 | 128K | DeepSeek | Proprietary | ||
| | GPT 4o | Unknown | 46.1% | 72.9% | 90.0% | 1351 | 53.3% | 74.7% | 86.0% | 39.0% | 69.0% | 42.5% | 1361 | 71 | $2.5 | $10 | Oct 23 | 128K | OpenAI | Proprietary | |||
  | Grok 2 | Unknown | 45.1% | 58.6% | 88.4% | 80.5% | 1287 | 36.3% | 75.5% | 87.5% | 56.0% | 76.1% | 45.8% | 1290 | 70 | $5 | $15 | Aug 24 | 128K | xAI | Proprietary | ||
| | GPT 4o mini | Unknown | 43.1% | 55.6% | 87.2% | 83.5% | 1284 | 35.3% | 63.1% | 82.0% | 43.0% | 75.0% | 35.6% | 1272 | 71 | $0.15 | $0.6 | Oct 23 | 128K | OpenAI | Proprietary | ||
 | Llama 3.1 | 405B | Unknown | 42.6% | 66.2% | 89.0% | 1284 | 53.3% | 73.3% | 87.0% | 51.1% | 73.8% | 40.5% | 1266 | 72 | $5 | $15 | Dec 23 | 128K | Meta | Open Source | ||
| Gemini 1.5 Flash 002 | Unknown | 41.9% | 52.6% | 84.0% | 75.6% | 1254 | 50.0% | 67.3% | 81.0% | 51.0% | 77.9% | 47.2% | 1269 | 73 | $0.075 | $0.3 | Nov 23 | 1M | Proprietary | |||
| DeepSeek Coder 2 | 236B | Unknown | 41.5% | 72.9% | 90.2% | 82.3% | 1267 | 45.3% | 63.6% | 80.0% | 42.0% | 58.0% | 1213 | 67 | $0.14 | $0.28 | Nov 23 | 128K | DeepSeek | Proprietary | ||
| Gemini 1.5 Pro 001 | Unknown | 40.9% | 66.9% | 1262 | 49.3% | 69.0% | 56.1% | 1260 | Nov 23 | 2M | Proprietary | |||||||||||
| Gemini 1.5 Flash 001 | Unknown | 40.6% | 52.6% | 1232 | 47.3% | 59.1% | 1227 | Nov 23 | 1M | Proprietary | ||||||||||||
| | Claude 3 Opus | Unknown | 38.6% | 68.4% | 84.9% | 77.4% | 1250 | 41.3% | 68.5% | 84.0% | 50.4% | 60.1% | 43.4% | 1248 | 70 | $15 | $75 | Aug 23 | 200K | Anthropic | Proprietary | ||
| Qwen 2.5 | 7B | Unknown | 38.4% | 63.9% | 26.2% | 38.2% | Sep 24 | 128K | Alibaba | Open Source | ||||||||||||
| | Grok-2 mini | Unknown | 37.5% | 54.9% | 85.7% | 1262 | 42.0% | 72.0% | 86.2% | 51.0% | 73.0% | 1266 | Aug 24 | 128K | xAI | Proprietary | |||||||
| QwQ Preview | 32B | Unknown | 37.2% | 57.7% | 75.0% | 65.2% | 50.8% | Sep 24 | 32K | Alibaba | Open Source | |||||||||||
  | Nova Pro | Unknown | 37.1% | 54.1% | 89.0% | 32.6% | 84.0% | 46.9% | 76.6% | 39.1% | 75 | $0.8 | $3.2 | 300K | Amazon | Proprietary | |||||||
| Llama 3.3 | 70B | Unknown | 36.6% | 59.4% | 88.4% | 87.6% | 1385 | 50.7% | 68.9% | 86.0% | 50.5% | 77.0% | 41.1% | 74 | $0.1 | $0.4 | Meta | Open Source | ||||
| Gemma 2 | 27B | Unknown | 35.9% | 51.8% | 28.1% | 56.5% | 75.2% | 42.3% | 26.2% | Jun 24 | 8K | Open Source | ||||||||||
| Llama 3.1 | 70B | Unknown | 32.7% | 58.6% | 80.5% | 1244 | 40.7% | 66.4% | 86.0% | 43.0% | 60.0% | 34.4% | 1242 | 65 | $1 | $3 | Dec 23 | 128K | Meta | Open Source | ||
| Gemini 1.5 Flash | 8B | Unknown | 28.7% | 38.3% | 1163 | 33.3% | 58.7% | 38.4% | 58.7% | 1177 | $0.0375 | $0.15 | Nov 23 | 1M | Proprietary | |||||||
| Mistral Nemo | 12B | Unknown | 28.7% | 33.1% | 25.3% | 44.8% | 68.0% | 33.0% | 40.0% | 16.9% | 52 | $0.15 | $0.15 | Apr 24 | 128K | Mistral | Proprietary | |||||
| | Nova Lite | Unknown | 27.5% | 85.4% | 34.7% | 79.0% | 42.0% | 73.3% | 30.8% | 70 | $0.06 | $0.24 | 300K | Amazon | Proprietary | ||||||||
| Qwen 2.5 | 32B | Unknown | 27.0% | 58.6% | 88.4% | 69.2% | 83.9% | 49.5% | 83.1% | Sep 24 | 128K | Alibaba | Open Source | |||||||||
| | Claude 3 Haiku | Unknown | 24.5% | 47.4% | 75.9% | 1189 | 25.9% | 42.3% | 75.2% | 33.3% | 40.9% | 22.9% | 1179 | 54 | $0.25 | $1.25 | Aug 23 | 200K | Anthropic | Proprietary | |||
| Gemma 2 | 9B | Unknown | 22.5% | 40.2% | 15.2% | 52.1% | 71.3% | 36.6% | 19.5% | Jun 24 | 8K | Open Source | ||||||||||
| Qwen 2.5 | 14B | Unknown | 20.9% | 83.5% | 63.7% | 80.0% | 45.5% | 80.0% | Sep 24 | 128K | Alibaba | Open Source | ||||||||||
| Qwen 2.5 Coder | 14B | Unknown | 23.4% | 69.2% | 89.6% | 84.0% | Sep 24 | 128K | Alibaba | Open Source | ||||||||||||
| Qwen 2.5 Coder | 7B | Unknown | 20.3% | 63.9% | 88.4% | 81.9% | Sep 24 | 128K | Alibaba | Open Source | ||||||||||||
| | Nova Micro | Unknown | 20.2% | 81.1% | 25.1% | 76.0% | 40.0% | 69.3% | 33.7% | 66 | $0.04 | $0.14 | 128K | Amazon | Proprietary | ||||||||
| Qwen 2.5 Coder | 3B | Unknown | 10.8% | 39.1% | 84.1% | 75.2% | Sep 24 | 32K | Alibaba | Open Source | ||||||||||||
| Llama 3.1 | 8B | Unknown | 19.7% | 37.6% | 72.6% | 62.8% | 1174 | 15.3% | 44.3% | 71.0% | 27.0% | 64.0% | 16.6% | 1171 | 53 | $0.3 | $0.6 | Dec 23 | 128K | Meta | Open Source | |
| Qwen 2.5 Coder | 1.5B | Unknown | 6.1% | 31.6% | 70.7% | 66.5% | Sep 24 | 32K | Alibaba | Open Source | ||||||||||||
| Qwen 2.5 | 3B | Unknown | 43.7% | Sep 24 | 32K | Alibaba | Open Source | |||||||||||||||
| Qwen 2.5 | 1.5B | Unknown | 4.7% | 32.1% | Sep 24 | 32K | Alibaba | Open Source | ||||||||||||||
| Qwen 2.5 | 0.5B | Unknown | 2.0% | 14.9% | Sep 24 | 32K | Alibaba | Open Source | ||||||||||||||
| Qwen 2.5 Coder | 0.5B | Unknown | 2.0% | 14.3% | 61.6% | 53.8% | Sep 24 | 32K | Alibaba | Open Source | ||||||||||||
| Pixtral Large | 123B | Unknown | 82.0% | 85.0% | 52.0% | 72.0% | 73 | $2 | $6 | 128K | Mistral | Proprietary | ||||||||||
| Llama 3.2 | 3B | Unknown | 26.3% | 1048 | 22.2% | 63.4% | 21.0% | 49.0% | 1093 | 46 | $0.08 | $0.1 | Dec 23 | 128K | Meta | Open Source | ||||||
| Llama 3.2 | 1B | Unknown | 1009 | 11.9% | 49.3% | 14.0% | 27.0% | 1044 | 28 | $0.04 | $0.08 | Dec 23 | 128K | Meta | Open Source | |||||||
| Llama 3.2 (Vision) | 90B | Unknown | 86.0% | 43.0% | 68.0% | 67 | $0.9 | $0.9 | Dec 23 | 128K | Meta | Open Source | ||||||||||
| Llama 3.2 (Vision) | 11B | Unknown | 73.0% | 26.0% | 51.9% | 54 | $0.19 | $0.19 | Dec 23 | 128K | Meta | Open Source | ||||||||||
| R1 Lite Preview | Unknown | 58.5% | 91.6% | DeepSeek | Proprietary | |||||||||||||||||
| | o1 Pro mode | Unknown | 79.0% | OpenAI | Proprietary | ||||||||||||||||||
| Gemma 2 | 2B | Unknown | 17.7% | 51.3% | 15.0% | Jun 24 | 8K | Open Source | ||||||||||||||
| Qwen 2.5 Turbo | Unknown | Sep 24 | 1M | Alibaba | Proprietary | |||||||||||||||||
| Deepseek 2.5 | Unknown | Nov 23 | 128K | DeepSeek | Proprietary | |||||||||||||||||
| Phi 4 | 14B | Unknown | 82.6% | 82.8% | 70.4% | 84.8% | 56.1% | 80.4% | Microsoft | Open Source | |||||||||||||
| | o3 | Unknown | 87.7% | OpenAI | Proprietary | ||||||||||||||||||
| | o3 mini high | Unknown | OpenAI | Proprietary | |||||||||||||||||||
| | o3 mini medium | Unknown | OpenAI | Proprietary | |||||||||||||||||||
| | o3 mini low | Unknown | OpenAI | Proprietary |
67 models