HellaSwag

Every result

• GPT-4 pretrained base (paper model)95.3%Reported by the model's maker (OpenAI), Mar 15, 2023

  OpenAI reported 95.3% on a private HellaSwag holdout for pretrained-base GPT-4 using ten-shot prompting.

  Tasks: Privately held secret holdout, not validation split

  Limits: Provider-reported result, not independently reproduced or a served API snapshot. This private holdout is distinct from the separately reported validation score. Unknown run settings remain null; no unqualified cross-model rank is established.

  OpenAI; hosted by arXiv

• PaLM 540B (paper model)83.8%Reported by the model's maker (Google Research / Chowdhery et al.), Apr 5, 2022

  Google reported 83.8% on-task accuracy for PaLM 540B on HellaSwag (5-shot) in its original April 2022 paper.

  Tasks: HellaSwag; Table 4 says splits follow Du et al. (2021) and Brown et al. (2020); exact dataset artifact/revision unspecified.

  Limits: Provider-reported result; no run was reproduced by Spellwright. This is the original PaLM 540B pretrained paper model, not PaLM-Coder, an instruction-tuned variant or an API service. Dataset revision, harness version, execution date and exact checkpoint hash remain unknown. No cross-paper or overall model rank is established. Zero-shot and few-shot rows differ in prompt examples; they are a protocol contrast for the same model, not identical-budget model evaluations. The source identifies dataset splits by prior-work references, without an exact artifact revision. No sample count is inferred from benchmark convention.

  Chowdhery et al. / Google Research; hosted by arXiv

• PaLM 540B (paper model)83.4%Reported by the model's maker (Google Research / Chowdhery et al.), Apr 5, 2022

  Google reported 83.4% on-task accuracy for PaLM 540B on HellaSwag (0-shot) in its original April 2022 paper.

  Tasks: HellaSwag; Table 4 says splits follow Du et al. (2021) and Brown et al. (2020); exact dataset artifact/revision unspecified.

  Limits: Provider-reported result; no run was reproduced by Spellwright. This is the original PaLM 540B pretrained paper model, not PaLM-Coder, an instruction-tuned variant or an API service. Dataset revision, harness version, execution date and exact checkpoint hash remain unknown. No cross-paper or overall model rank is established. Zero-shot and few-shot rows differ in prompt examples; they are a protocol contrast for the same model, not identical-budget model evaluations. The source identifies dataset splits by prior-work references, without an exact artifact revision. No sample count is inferred from benchmark convention.

  Chowdhery et al. / Google Research; hosted by arXiv

• Mistral-7B-v0.181.3%Reported by the model's maker (Mistral AI / Jiang et al.), Oct 10, 2023

  Mistral reported 81.3% on HellaSwag for Mistral 7B v0.1 base in its October 2023 paper.

  Tasks: Exact dataset split/revision unspecified in the inspected paper

  Limits: Mistral reported these results in its own common evaluation pipeline; we did not reproduce the runs. The table identifies the pretrained base variant, without a checkpoint hash. No Chat/Instruct or served API model is implied. Missing dataset/run metadata remain unknown. Compare only this paper’s paired rows for the same benchmark; no cross-paper or overall model ranking is established.

  Jiang et al. / Mistral AI; hosted by arXiv

• Llama 2 7B pretrained (July 2023 release)77.1%Reported by the model's maker (Mistral AI / Jiang et al.), Oct 10, 2023

  Mistral reported 77.1% on HellaSwag for Llama 2 7B pretrained in its October 2023 paper.

  Tasks: Exact dataset split/revision unspecified in the inspected paper

  Limits: Mistral reported these results in its own common evaluation pipeline; we did not reproduce the runs. The table identifies the pretrained base variant, without a checkpoint hash. No Chat/Instruct or served API model is implied. Missing dataset/run metadata remain unknown. Compare only this paper’s paired rows for the same benchmark; no cross-paper or overall model ranking is established. Mistral is a competing model provider evaluating Meta’s model; this is a provider-run comparison, not an independent external laboratory test.

  Jiang et al. / Mistral AI; hosted by arXiv