AI Intelligence

Benchmarks

Each chart is one test, best score first. Different versions of a test are charted separately because their scores can't be compared. means the model's maker reported the score. Dark bars are closed models; light bars are open-weight models.

Coding

Terminal-Bench 4.0

Coding · higher is better · 5 models

CursorBench 4.0

Coding · higher is better · 4 models

HumanEval

Coding · higher is better · 3 models

PaLM 540B (paper model)76.2%*
GPT-4 pretrained base (paper model)67.0%*
Mistral-7B-v0.130.5%*

All 4 results

DeepSWE 1.1

Coding · higher is better · 2 models

Gemini 4 Argon77.9%*
Gemini 3.8 Flash73.7%*

Reasoning and knowledge

Humanity's Last Exam

Reasoning and knowledge · higher is better · 4 models

HellaSwag

Reasoning and knowledge · higher is better · 4 models

GPT-4 pretrained base (paper model)95.3%*
PaLM 540B (paper model)83.8%*
Mistral-7B-v0.181.3%*
Llama 2 7B pretrained (July 2023 release)77.1%*

All 5 results

ARC-Challenge

Reasoning and knowledge · higher is better · 2 models

Mistral-7B-v0.155.5%*
Llama 2 7B pretrained (July 2023 release)43.2%*

GSM8K

Reasoning and knowledge · higher is better · 2 models

Mistral-7B-v0.152.2%*
Llama 2 7B pretrained (July 2023 release)16.0%*

Embodied Reasoning (ERQA)

Reasoning and knowledge · higher is better · 1 model

Gemini 4 Argon87%*

Multimodal

LVBench

Multimodal · higher is better · 1 model

Gemini 4 Argon91.7%*

Other

MMLU

Other · higher is better · 2 models

GPT-4 pretrained base (paper model)86.4%*
Mistral-7B-v0.160.1%*

LAMBADA

Other · higher is better · 2 models

PaLM 540B (paper model)89.7%*
GPT-3 175B (paper model)86.4%*

All 4 results