Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Their "model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models."
{{ if .System }}<|im_system|>system<|im_middle|>{{ .System }}<|im_end|>{{ end }}{{ if .Prompt }}<|im_user|>user<|im_middle|>{{ .Prompt }}<|im_end|>{{ end }}<|im_assistant|>assistant<|im_middle|>{{ .Response }}<|im_end|>