LAMBADA

Every result

• PaLM 540B (paper model)89.7%Reported by the model's maker (Google Research / Chowdhery et al.), Apr 5, 2022

  Google reported 89.7% exact match for PaLM 540B on LAMBADA (8-shot) in its original April 2022 paper.

  Tasks: LAMBADA; Table 4 says splits follow Du et al. (2021) and Brown et al. (2020); exact dataset artifact/revision unspecified.

  Limits: Provider-reported result; no run was reproduced by Spellwright. This is the original PaLM 540B pretrained paper model, not PaLM-Coder, an instruction-tuned variant or an API service. Dataset revision, harness version, execution date and exact checkpoint hash remain unknown. No cross-paper or overall model rank is established. Zero-shot and few-shot rows differ in prompt examples; they are a protocol contrast for the same model, not identical-budget model evaluations. The source identifies dataset splits by prior-work references, without an exact artifact revision. No sample count is inferred from benchmark convention.

  Chowdhery et al. / Google Research; hosted by arXiv

• GPT-3 175B (paper model)86.4%Reported by the model's maker (OpenAI / Brown et al.), Jul 22, 2020

  OpenAI reported 86.4% LAMBADA accuracy for its 175B-parameter GPT-3 with 15 demonstrations and a cloze prompt.

  Tasks: Test split; final-word prediction

  Limits: Reported result, not independently reproduced. Prompt format differs across zero-shot and few-shot settings; no controlled shot-count effect or overall model rank. The paper discusses training/test overlap; missing run metadata remain unknown.

  Brown et al. / OpenAI; hosted by arXiv

• PaLM 540B (paper model)77.9%Reported by the model's maker (Google Research / Chowdhery et al.), Apr 5, 2022

  Google reported 77.9% exact match for PaLM 540B on LAMBADA (0-shot) in its original April 2022 paper.

  Tasks: LAMBADA; Table 4 says splits follow Du et al. (2021) and Brown et al. (2020); exact dataset artifact/revision unspecified.

  Limits: Provider-reported result; no run was reproduced by Spellwright. This is the original PaLM 540B pretrained paper model, not PaLM-Coder, an instruction-tuned variant or an API service. Dataset revision, harness version, execution date and exact checkpoint hash remain unknown. No cross-paper or overall model rank is established. Zero-shot and few-shot rows differ in prompt examples; they are a protocol contrast for the same model, not identical-budget model evaluations. The source identifies dataset splits by prior-work references, without an exact artifact revision. No sample count is inferred from benchmark convention.

  Chowdhery et al. / Google Research; hosted by arXiv

• GPT-3 175B (paper model)76.2%Reported by the model's maker (OpenAI / Brown et al.), Jul 22, 2020

  OpenAI reported 76.2% LAMBADA accuracy for its 175B-parameter GPT-3 under zero-shot evaluation.

  Tasks: Test split; final-word prediction

  Limits: Reported result, not independently reproduced. Prompt format differs across zero-shot and few-shot settings; no controlled shot-count effect or overall model rank. The paper discusses training/test overlap; missing run metadata remain unknown.

  Brown et al. / OpenAI; hosted by arXiv