falcon3:7b

999.2K 10 months ago

A family of efficient AI models under 10B parameters performant in science, math, and coding through innovative training techniques.

1b 3b 7b 10b

10 months ago

472ea1c89f64 · 4.6GB ·

model
llama
·
7.46B
·
Q4_K_M
template
{{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1 -}} <|{{ .Role }}|> {
license
Falcon 3 TII Falcon License December 2024 FalconLLM.tii.ae Introductory note This license is, in par
params
{ "stop": [ "<|system|>", "<|user|>", "<|end|>", "<|assistant|>"

Readme

Falcon3 represents TII’s latest advancement in efficient language models under 10B parameters, focused on enhancing science, math, and code capabilities while maintaining training efficiency.

Key Features

  • Four sizes: 1B, 3B, 7B, 10B
  • Depth up-scaling technique used to create 10B model from 7B
  • Knowledge distillation for smaller models (1B, 3B)

Performance Highlights

  • falcon3:1b outperforms smollm2:1.7b, matches gemma2:2b
  • falcon3:10b achieves SOTA in under-13B category
  • Extended context length up to 32K tokens (8K for 1B model)

References

Hugging Face