This model extends LLama-3 70B's context length from 8k to over 1m tokens. [I-Quants]
103 Pulls Updated 5 months ago
Updated 5 months ago
5 months ago
0087ebdee223 · 22GB
Readme
These iq versions can be useful to fit the 70b model on 1x or 2x 24GB or a single 48GB VRAM GPU with a small or large context.
Mind that at 2 or 3-bit the quality is severely impacted, running the 8b model is probably better for most use cases.
Recommended for single 24GB card: 70b-instruct-1048k-iq2_xs
Runs at 13 t/s with a small offloading on CPU.
This model extends LLama-3 8B’s context length from 8k to > 1040K, developed by Gradient, sponsored by compute from Crusoe Energy. It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. We trained on 830M tokens for this stage, and 1.4B tokens total for all stages, which is < 0.01% of Llama-3’s original pre-training data.
Large Context Window
Note: using a 256k context window requires at least 64GB of memory. Using a 1M+ context window requires significantly more (100GB+).
To extend the context window (to 256k for example) use:
API
curl http://localhost:11434/api/generate -d '{
"model": "llama3-gradient",
"prompt": "Why is the sky blue?",
"options": {
"num_ctx": 256000
}
}'
CLI
ollama run llama3-gradient
>>> /set parameter num_ctx 256000
References