Google reported 26.2% pass@1 for PaLM 540B on HumanEval (0-shot) in its original April 2022 paper.
Tasks: HumanEval Python code synthesis; exact dataset revision or split not specified in the paper.
Limits: Provider-reported result; no run was reproduced by Spellwright. This is the original PaLM 540B pretrained paper model, not PaLM-Coder, an instruction-tuned variant or an API service. Dataset revision, harness version, execution date and exact checkpoint hash remain unknown. No cross-paper or overall model rank is established. pass@1 and pass@100 use different sampling/attempt conditions and cannot be pooled into an equal-budget ranking. The paper’s direct sampled pass@k method is not assumed equivalent to an pass@k estimator in another report.
Chowdhery et al. / Google Research; hosted by arXiv
