HumanEval

Every result

• PaLM 540B (paper model)76.2%Reported by the model's maker (Google Research / Chowdhery et al.), Apr 5, 2022

  Google reported 76.2% pass@100 for PaLM 540B on HumanEval (0-shot) in its original April 2022 paper.

  Tasks: HumanEval Python code synthesis; exact dataset revision/split unspecified in the inspected paper.

  Limits: Provider-reported result; no run was reproduced by Spellwright. This is the original PaLM 540B pretrained paper model, not PaLM-Coder, an instruction-tuned variant or an API service. Dataset revision, harness version, execution date and exact checkpoint hash remain unknown. No cross-paper or overall model rank is established. pass@1 and pass@100 use different sampling/attempt conditions and cannot be pooled into an equal-budget ranking. The paper’s direct sampled pass@k method is not assumed equivalent to an pass@k estimator in another report.

  Chowdhery et al. / Google Research; hosted by arXiv

• GPT-4 pretrained base (paper model)67.0%Reported by the model's maker (OpenAI), Mar 15, 2023

  OpenAI reported a 67.0% HumanEval score for pretrained-base GPT-4 under zero-shot evaluation.

  Tasks: Python coding tasks; exact split/run subset unspecified

  Limits: Provider-reported result, not independently reproduced or a served API snapshot. Table2 does not establish pass@k; passK remains null. Unknown run settings remain null; no unqualified cross-model rank is established. The report estimated 25% training-data overlap for HumanEval; the separately reported clean-subset result is a different population.

  OpenAI; hosted by arXiv

• Mistral-7B-v0.130.5%Reported by the model's maker (Mistral AI / Jiang et al.), Oct 10, 2023

  Mistral AI reported a 30.5% HumanEval score for Mistral 7B v0.1 base with zero-shot evaluation.

  Tasks: Exact dataset split/revision unspecified in the inspected paper

  Limits: Provider-reported result from its own evaluation pipeline; not independently reproduced. This is the pretrained base, not Mistral 7B Instruct. Missing run metadata remain unknown; no cross-paper or universal model rank is established.

  Jiang et al. / Mistral AI; hosted by arXiv

• PaLM 540B (paper model)26.2%Reported by the model's maker (Google Research / Chowdhery et al.), Apr 5, 2022

  Google reported 26.2% pass@1 for PaLM 540B on HumanEval (0-shot) in its original April 2022 paper.

  Tasks: HumanEval Python code synthesis; exact dataset revision/split unspecified in the inspected paper.

  Limits: Provider-reported result; no run was reproduced by Spellwright. This is the original PaLM 540B pretrained paper model, not PaLM-Coder, an instruction-tuned variant or an API service. Dataset revision, harness version, execution date and exact checkpoint hash remain unknown. No cross-paper or overall model rank is established. pass@1 and pass@100 use different sampling/attempt conditions and cannot be pooled into an equal-budget ranking. The paper’s direct sampled pass@k method is not assumed equivalent to an pass@k estimator in another report.

  Chowdhery et al. / Google Research; hosted by arXiv