lgkt / llama3-chinese-wangshenzhi

[May 6, 2024] We now introduce Llama3-8B-Chinese-Chat-v2.1! Compared to v1, the training dataset of v2.1 is 5x larger (~100K preference pairs), and it exhibits significant enhancements, especially in roleplay, function calling, and math capabilities! Compared to v2, v2.1 surpasses v2 in math and is less prone to including English words in Chinese responses. The training dataset of Llama3-8B-Chinese-Chat-v2.1 will be released soon. If you love our Llama3-8B-Chinese-Chat-v1 or v2, you won’t want to miss out on Llama3-8B-Chinese-Chat-v2.1!

Model Summary Llama3-8B-Chinese-Chat is an instruction-tuned language model for Chinese & English users with various abilities such as roleplaying & tool-using built upon the Meta-Llama-3-8B-Instruct model.

Developed by: Shenzhi Wang (王慎执) and Yaowei Zheng (郑耀威)

License: Llama-3 License Base Model: Meta-Llama-3-8B-Instruct Model Size: 8.03B Context length: 8K

This is the first model specifically fine-tuned for Chinese & English user through ORPO [1] based on the Meta-Llama-3-8B-Instruct model.

Compared to the original Meta-Llama-3-8B-Instruct model, our Llama3-8B-Chinese-Chat-v1 model significantly reduces the issues of “Chinese questions with English answers” and the mixing of Chinese and English in responses.

Compared to Llama3-8B-Chinese-Chat-v1, our Llama3-8B-Chinese-Chat-v2 model significantly increases the training data size (from 20K to 100K), which introduces great performance enhancement, especially in roleplay, tool using, and math.

[1] Hong, Jiwoo, Noah Lee, and James Thorne. “Reference-free Monolithic Preference Optimization with Odds Ratio.” arXiv preprint arXiv:2403.07691 (2024).

Training framework: LLaMA-Factory.

Training details:

epochs: 2 learning rate: 5e-6 learning rate scheduler type: cosine Warmup ratio: 0.1 cutoff len (i.e. context length): 8192 orpo beta (i.e. $\lambda$ in the ORPO paper): 0.05 global batch size: 128 fine-tuning type: full parameters optimizer: paged_adamw_32bit

模型来源：https://hf-mirror.com/shenzhi-wang/Llama3-8B-Chinese-Chat-GGUF-4bit

QQ: 83649263

### Llama3-8B-Chinese-Chat-v2.1-Q4_0（from wangshenzhi）
2024-5-19

以下来自wangshenzhi

[May 6, 2024] We now introduce **Llama3-8B-Chinese-Chat-v2.1**! Compared to v1, the training dataset of v2.1 is 5x larger (~100K preference pairs), and it exhibits significant enhancements, especially in roleplay, function calling, and math capabilities! Compared to v2, v2.1 surpasses v2 in math and is less prone to including English words in Chinese responses. The training dataset of Llama3-8B-Chinese-Chat-v2.1 will be released soon. If you love our Llama3-8B-Chinese-Chat-v1 or v2, you won't want to miss out on Llama3-8B-Chinese-Chat-v2.1!

**Model Summary**
**Llama3-8B-Chinese-Chat** is an instruction-tuned language model for Chinese & English users with various abilities such as roleplaying & tool-using built upon the **Meta-Llama-3-8B-Instruct** model.

Developed **by:** Shenzhi Wang (王慎执) and Yaowei Zheng (郑耀威)

License: Llama-3 License
Base Model: Meta-Llama-3-8B-Instruct
Model Size: 8.03B
Context length: 8K

This is the first model specifically fine-tuned for Chinese & English user through ORPO [1] based on the Meta-Llama-3-8B-Instruct model.

Compared to the original Meta-Llama-3-8B-Instruct model, our Llama3-8B-Chinese-Chat-v1 model significantly reduces the issues of "Chinese questions with English answers" and the mixing of Chinese and English in responses.

[1] Hong, Jiwoo, Noah Lee, and James Thorne. "Reference-free Monolithic Preference Optimization with Odds Ratio." arXiv preprint arXiv:2403.07691 (2024).

Training framework: LLaMA-Factory.

Training details:

epochs: 2
learning rate: 5e-6
learning rate scheduler type: cosine
Warmup ratio: 0.1
cutoff len (i.e. context length): 8192
orpo beta (i.e. $\lambda$ in the ORPO paper): 0.05
global batch size: 128
fine-tuning type: full parameters
optimizer: paged_adamw_32bit

模型来源：https://hf-mirror.com/shenzhi-wang/Llama3-8B-Chinese-Chat-GGUF-4bit

QQ: 83649263

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)