4,737 Pulls Updated 4 months ago
Updated 4 months ago
4 months ago
90ba362f8207 · 3.6GB
Readme
gte-Qwen2-1.5B-instruct
gte-Qwen2-1.5B-instruct is the latest model in the gte (General Text Embedding) model family. The model is built on Qwen2-1.5B LLM model and use the same training data and strategies as the gte-Qwen2-7B-instruct model.
The model incorporates several key advancements:
- Integration of bidirectional attention mechanisms, enriching its contextual understanding.
- Instruction tuning, applied solely on the query side for streamlined efficiency
- Comprehensive training across a vast, multilingual text corpus spanning diverse domains and scenarios. This training leverages both weakly supervised and supervised data, ensuring the model’s applicability across numerous languages and a wide array of downstream tasks.
Model Information
- Model Size: 1.5B
- Embedding Dimension: 1536
- Max Input Tokens: 32k
Evaluation
MTEB & C-MTEB
You can use the scripts/eval_mteb.py to reproduce the following result of gte-Qwen2-1.5B-instruct on MTEB(English)/C-MTEB(Chinese):
| Model Name | MTEB(56) | C-MTEB(35) | MTEB-fr(26) | MTEB-pl(26) |
|---|---|---|---|---|
| bge-base-en-1.5 | 64.23 | - | - | - |
| bge-large-en-1.5 | 63.55 | - | - | - |
| gte-large-en-v1.5 | 65.39 | - | - | - |
| gte-base-en-v1.5 | 64.11 | - | - | - |
| mxbai-embed-large-v1 | 64.68 | - | - | - |
| acge_text_embedding | - | 69.07 | - | - |
| stella-mrl-large-zh-v3.5-1792d | - | 68.55 | - | - |
| gte-large-zh | - | 66.72 | - | - |
| multilingual-e5-base | 59.45 | 56.21 | - | - |
| multilingual-e5-large | 61.50 | 58.81 | - | - |
| e5-mistral-7b-instruct | 66.63 | 60.81 | - | - |
| gte-Qwen1.5-7B-instruct | 67.34 | 69.52 | - | - |
| NV-Embed-v1 | 69.32 | - | - | - |
| gte-Qwen2-7B-instruct | 70.24 | 72.05 | 68.25 | 67.86 |
| gte-Qwen2-1.5B-instruct | 67.16 | 67.65 | 66.60 | 64.04 |
GTE Models
The gte series models have consistently released two types of models: encoder-only models (based on the BERT architecture) and decode-only models (based on the LLM architecture).
| Models | Language | Max Sequence Length | Dimension | Model Size (Memory Usage, fp32) |
|---|---|---|---|---|
| GTE-large-zh | Chinese | 512 | 1024 | 1.25GB |
| GTE-base-zh | Chinese | 512 | 512 | 0.41GB |
| GTE-small-zh | Chinese | 512 | 512 | 0.12GB |
| GTE-large | English | 512 | 1024 | 1.25GB |
| GTE-base | English | 512 | 512 | 0.21GB |
| GTE-small | English | 512 | 384 | 0.10GB |
| GTE-large-en-v1.5 | English | 8192 | 1024 | 1.74GB |
| GTE-base-en-v1.5 | English | 8192 | 768 | 0.51GB |
| GTE-Qwen1.5-7B-instruct | Multilingual | 32000 | 4096 | 26.45GB |
| GTE-Qwen2-7B-instruct | Multilingual | 32000 | 3584 | 26.45GB |
| GTE-Qwen2-1.5B-instruct | Multilingual | 32000 | 1536 | 6.62GB |
Citation
If you find our paper or models helpful, please consider cite: