Hudson / llemma

EleutherAI LLEMMA 7b

7b

14 6 weeks ago

1 Tag

Readme

ArXiv | Models | Data | Code | Blog | Sample Explorer

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, Sean Welleck

Llemma 7B is a language model for mathematics. It was initialized with Code Llama 7B weights, and trained on the Proof-Pile-2 for 200B tokens.

This model also comes in a 34B parameter version: Llemma 34B.

Evaluations

Llemma models are particularly strong at chain-of-thought mathematical reasoning and using computational tools for mathematics, such as Python and formal theorem provers.

Chain-of-thought Math

On chain-of-thought mathematics tasks, Llemma models outperform Llama-2, Code Llama, and when controlled for model size, outperform Minerva.

Model Size GSM8k OCW MMLU-STEM SAT MATH
Llama 2 7B 11.8% 3.7% 29.9% 25% 3.2%
Code Llama 7B 10.5% 4.4% 25.1% 9.4% 4.5%
LLEMMA 7B 36.4% 7.7% 37.7% 53.1% 18.0%
Minerva 8B 16.2% 7.7% 35.6% - 14.1%
———— —— ——– ——- ———– ——- ——-
Code Llama 34B 29.6% 7.0% 40.5% 40.6% 12.2%
LLEMMA 34B 51.5% 11.8% 49.0% 71.9% 25.0%
———— —— ——– ——- ———– ——- ——-
Minerva 62B 52.4% 12.0% 53.9% - 27.6%
Minerva 540B 58.8% 17.6% 63.9% - 33.6%

Further performance can be extracted by using majority voting:

Model Size GSM8k maj@100 OCW maj@100 MMLU-STEM maj@16 SAT maj@16 MATH maj@256
LLEMMA 7B 54.0% 14.3% 49.9% 78.1% 33.5
Minerva 8B 28.4% 12.5% 43.4% - 25.4%
——— —— ————- ———– —————– ———– ————
LLEMMA 34B 69.3% 18.4% 59.7% 81.3% 43.1%
——— —— ————- ———– —————– ———– ————
Minerva 62B 68.5% 23.5% 63.5% - 43.4%
Minerva 540B 78.5% 30.8% 75.0% - 50.3%

Tool Use and Theorem Proving

In addition to chain-of-thought reasoning, Llemma has strong capabilities in computational mathematics tasks. For tool use and formal theorem proving evaluations, see our paper.

Citation

@misc{azerbayev2023llemma,
      title={Llemma: An Open Language Model For Mathematics}, 
      author={Zhangir Azerbayev and Hailey Schoelkopf and Keiran Paster and Marco Dos Santos and Stephen McAleer and Albert Q. Jiang and Jia Deng and Stella Biderman and Sean Welleck},
      year={2023},
      eprint={2310.10631},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}