ARC-Challenge

Every result

• Mistral-7B-v0.155.5%Reported by the model's maker (Mistral AI / Jiang et al.), Oct 10, 2023

  Mistral reported 55.5% on ARC-Challenge for Mistral 7B v0.1 base in its October 2023 paper.

  Tasks: ARC-Challenge; exact dataset split/revision unspecified in the inspected paper

  Limits: Mistral reported these results in its own common evaluation pipeline; we did not reproduce the runs. The table identifies the pretrained base variant, without a checkpoint hash. No Chat/Instruct or served API model is implied. Missing dataset/run metadata remain unknown. Compare only this paper’s paired rows for the same benchmark; no cross-paper or overall model ranking is established.

  Jiang et al. / Mistral AI; hosted by arXiv

• Llama 2 7B pretrained (July 2023 release)43.2%Reported by the model's maker (Mistral AI / Jiang et al.), Oct 10, 2023

  Mistral reported 43.2% on ARC-Challenge for Llama 2 7B pretrained in its October 2023 paper.

  Tasks: ARC-Challenge; exact dataset split/revision unspecified in the inspected paper

  Limits: Mistral reported these results in its own common evaluation pipeline; we did not reproduce the runs. The table identifies the pretrained base variant, without a checkpoint hash. No Chat/Instruct or served API model is implied. Missing dataset/run metadata remain unknown. Compare only this paper’s paired rows for the same benchmark; no cross-paper or overall model ranking is established. Mistral is a competing model provider evaluating Meta’s model; this is a provider-run comparison, not an independent external laboratory test.

  Jiang et al. / Mistral AI; hosted by arXiv