Quantization Details

Both the 2B Base and 2B Instruct models were quantized using llama.cpp with a similar importance matrix calculated from 34 languages (1000 samples each) using the OSCAR dataset. Here we combine the details of quantization types for both models and provide a comparison of their perplexity (PPL) values, file sizes, and notes on each quantization scheme.

2b Quantization Summary

The selection was driven by achieving the lowest possible perplexity while reducing model file size. Quantization types below Q4 were experimented with, but Q4 and above provide the best performance-to-size ratio. The recommended quantizations span IQ3_M to Q6_K, covering a variety of sizes for different use cases.

Perplexity Comparison Table

Notes on Quantization Selection:

• Recommended Quantizations:

  • IQ4_XS (2b Instruct): Offers a good size reduction with minimal PPL increase. File size remains very close to Q4_K_S at 1.8GB.
  • Q4_K_S: Strong size-to-performance ratio for both models, providing minimal PPL increase while maintaining a small size.
  • Q5_K_M: Excellent balance between low perplexity and a modest file size, recommended for most applications.

• Non-recommended Quantizations:

  • IQ3_M: The best of the highly compressed quantizations, offering good size efficiency with a manageable PPL increase. Suitable for sub-2GB requirements.
  • Q3_K_L: Modest PPL increase but provides a slightly larger file size compared to IQ3_M.
  • Q6_K: Delivers nearly bf16 performance with significantly smaller file size. Suitable for those needing near-lossless accuracy.

Observations on Differences:

• The 2B Instruct model shows a slight upward shift in PPL values across quantization types compared to the 2B Base model. This is likely due to the additional instruction-tuning step, which introduces slightly higher loss.
• Despite these differences, the overall pattern of quantization performance remains consistent across both models, reinforcing the reliability of these schemes irrespective of fine-tuning.

Defending the Selection:

1. Diversity in Quantization Types:

   • I Quantization Below Q4: IQ3_M offers a solid sub-Q4 option for highly compressed models with a good size-to-performance trade-off.
   • K Quantization At and Above Q4: Q4_K_S, Q5_K_M, and Q6_K offer a range of options with strong PPL performance and reduced file sizes.

2. Selection Criteria:

   • Log PPL Difference <0.3: Only models with a log PPL difference of less than 0.3 are included to ensure acceptable performance even in highly quantized forms.
   • No Redundancy in File Size: Each quantization is chosen to provide unique value without overlap in file sizes and performance, ensuring a comprehensive but concise selection.

• For 2B Base: In an effort to achieve a model under 1.5GB, we attempted quantizing with IQ2_XS. However, the resulting file size slightly exceeded this target, and the perplexity was significantly higher than acceptable (more than double the 0.3 selection threshold). For models under 1.7GB, Richard Erkhov’s static quantizations are a better option as they don’t use the importance matrix, resulting in smaller sizes.
• For 2B Instruct: An attempt to reduce the file size even further below IQ3_M was made with IQ2_M, but the perplexity remained unacceptable and exceeded the log PPL difference threshold of 0.3. As with the base model, those needing a model smaller than 1.7GB might prefer Richard Erkhov’s static quantizations, which sacrifice the importance matrix for size reduction.

PPL is measured (with llama-perplexity) from a sample of 50 of each language from the same dataset used to calculate the importance matrix.

Salamandra Model Card

Salamandra is a highly multilingual model pre-trained from scratch that comes in three different sizes — 2B, 7B, and 40B parameters — with their respective base and instruction-tuned variants. This model card corresponds to the 7B instructed and base versions.

The entire Salamandra family is released under a permissive Apache 2.0 license. Along with the open weights, all training scripts and configuration files are made publicly available in this GitHub repository.

[!WARNING] DISCLAIMER: This warning applies to the 2B-instruct version. The model is a first proof-of-concept designed to demonstrate the instruction-following capabilities of recently released base models. It has been optimized to engage in conversation but has NOT been aligned through RLHF to filter or avoid sensitive topics. As a result, it may generate harmful or inappropriate content. The team is actively working to enhance its performance through further instruction and alignment with RL techniques.

Model Details (2B Version)

Description

Transformer-based decoder-only language model that has been pre-trained from scratch on 7.8 trillion tokens of highly curated data. The pre-training corpus contains text in 35 European languages and code.

Hyperparameters

The full list of hyperparameters for each model can be found here.

Architecture

Intended Use

Direct Use

The models are intended for both research and commercial use in any of the languages included in the training data. The base models are intended either for language generation or to be further fine-tuned for specific use-cases. The instruction-tuned variants can be used as general-purpose assistants, as long as the user is fully aware of the model’s limitations.

Out-of-scope Use

The model is not intended for malicious activities, such as harming others or violating human rights. Any downstream application must comply with current laws and regulations. Irresponsible usage in production environments without proper risk assessment and mitigation is also discouraged.

Hardware and Software

Training Framework

Pre-training was conducted using NVIDIA’s NeMo Framework, which leverages PyTorch Lightning for efficient model training in highly distributed settings.

The instruction-tuned versions were produced with FastChat.

Compute Infrastructure

All models were trained on MareNostrum 5, a pre-exascale EuroHPC supercomputer hosted and operated by Barcelona Supercomputing Center.

The accelerated partition is composed of 1,120 nodes with the following specifications: - 4x Nvidia Hopper GPUs with 64 HBM2 memory - 2x Intel Sapphire Rapids 8460Y+ at 2.3Ghz and 32c each (64 cores) - 4x NDR200 (BW per node 800Gb/s) - 512 GB of Main memory (DDR5) - 460GB on NVMe storage

Data

Pretraining Data

The training corpus consists of 2.4 trillion tokens, including 35 European languages and 92 programming languages. It amounts to a total of 33TB of pre-processed text. Languages were sampled manually by giving x2 oversampling to Spain’s co-official languages (Spanish, Catalan, Galician and Basque), code was undersampled by half, and the rest of the languages were kept as is, resulting in the following distribution:

This highly multilingual corpus is predominantly composed of data from Colossal OSCAR, which contributes a significant 66.06% of the total tokens. Following this, Starcoder provides 11.91%, and Spanish Crawling adds 3.34%. The next largest sources are French FR at 3.12% and Proof Pile at 1.98%. Other notable contributions include Macocu, Pile of Law, and Eurlex, each contributing around 1.5% to 1.3%. These major sources collectively form the bulk of the corpus, ensuring a rich and diverse dataset for training the language model. The remaining 10% comes from smaller sources in various languages.

Feel free to click the expand button below to see the full list of sources.

The model was trained for 3 epochs, with two final rounds of 0.3B higher-quality tokens each, meaning that the total number of tokens seen during pre-training amounts to roughly 7.8 trillion tokens.

We provide an extense Datasheet section following the best practices defined by (Gebru et al., 2021).

2b-instruct Finetuning Data

This instruction-tuned variant has been trained with a mixture of 276k English, Spanish, and Catalan multi-turn instructions gathered from open datasets:

Evaluation

Gold-standard benchmarks

Evaluation is done using the Language Model Evaluation Harness (Gao et al., 2024). We evaluate on a set of tasks taken from SpanishBench, CatalanBench, BasqueBench, and GalicianBench.

2B Instruct Model:

For the 2B instruct model, we evaluated tasks using the chat-template feature of the LM Evaluation Harness. We focused on tasks from human-generated or human-in-the-loop sources, ensuring high quality and accuracy. The model was evaluated in a 0-shot setting, where no prior examples were given to the model for prediction.

We observed ≈1.5% variances in performance for some tasks based on the version of the transformers library used and whether tensor parallelism was employed when loading the model. Replicating these results in other Harness implementations may yield different outcomes due to these factors. It’s important to note that gold-standard evaluations do not fully capture the model’s capabilities or potential, so caution is advised when interpreting the results.

2B Base Model:

The 2B base model was evaluated using the same benchmarks, including additional English tasks from the LM Evaluation Harness. We used a 5-shot setting for these evaluations, meaning that the model was provided with a few examples before making predictions. This slightly different setup allows for comparison between the instructed and non-instructed (base) versions of the model.

Similar issues with replicating results apply to the 2B base model, particularly variances in performance based on the library version and tensor parallelism settings.

Differences between 2B instruct and 2B base models:

• 2B instruct was evaluated in a 0-shot setting, where it receives no prior examples.
• 2B base was evaluated in a 5-shot setting, providing the model with examples before predicting.
• 2B instruct leverages chat-template for instruction following tasks, making it more suited for conversational or instruction-driven tasks, while 2B base is a more general model without that specialization.

Spanish

Catalan

Basque

Galician

English

Ethical Considerations and Limitations

We examine the presence of undesired societal and cognitive biases in the 2B Base and 2B Instruct models using different benchmarks.

Societal Biases

• 2B Instruct: We use the BBQ dataset (Parrish et al., 2022) and the Regard dataset (Sheng et al., 2019). Results indicate moderate accuracies (between 0.5 and 0.6 depending on the social groups) in disambiguated settings, while the model performs poorly in ambiguous settings. These results suggest pervasive social biases that may affect task performance.
• 2B Base: Similar tests reveal inadequate accuracies in both ambiguous and disambiguated contexts, reflecting the presence of societal biases. These biases should be addressed in the post-training phases.

Cognitive Biases

• 2B Instruct: Positional effects in 0-shot settings and majority class bias in few-shot settings were analyzed. Moderate weak primacy effects were observed (a preference for earlier answers in multiple-choice questions), and we detected small majority class effects. This suggests some robustness against cognitive biases.
• 2B Base: We observed moderate to very strong primacy effects and significant majority class effects, implying that outputs are influenced by prompts and model settings.

Limitations and Next Steps

• Both models’ analyses are limited due to the scarcity of resources in many languages in the training data. Future work will expand analyses and testing.
• The observed biases are expected from models with limited instruction tuning. Developers should account for these biases and perform safety testing tailored to their specific applications.

Additional Information

Author

The Language Technologies Unit from Barcelona Supercomputing Center.

Contact

For further information, please send an email to langtech@bsc.es.

Copyright

Copyright© 2024 by Language Technologies Unit, Barcelona Supercomputing Center.

Funding

This work has been promoted and financed by the Government of Catalonia through the Aina Project.

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of ILENIA Project with reference 2022/TL22/00215337.

Acknowledgements

This project has benefited from the contributions of numerous teams and institutions, mainly through data contributions, knowledge transfer, or technical support.

• Catalonia: Thanks to Òmnium Cultural, Parlament de Catalunya, Institut d’Estudis Aranesos, Racó Català, Vilaweb, ACN, Nació Digital, El món, and Aquí Berguedà.
• National level: Special thanks to our ILENIA project partners: CENID, HiTZ, and CiTIUS. Also, thanks to the Spanish Senate and Congress, Fundación Dialnet, Fundación Elcano, and the Instituto Universitario de Sistemas Inteligentes y Aplicaciones Numéricas en Ingeniería (SIANI).
• International level: Gratitude to the Welsh government, DFKI, Occiglot project, and Common Crawl Foundation. Special thanks to the NVIDIA team, including Ignacio Sarasua, Adam Henryk Grzywaczewski, Oleg Sudakov, Sergio Perez, Miguel Martinez, Felipes Soares, and Meriem Bendris for their support.

Disclaimer

Be aware that the model may contain biases or other unintended distortions. Third-party developers deploying systems or using the model must take responsibility for mitigating risks and ensuring compliance with AI regulations. The Barcelona Supercomputing Center is not liable for outcomes resulting from third-party use.

Citation

Technical report and paper coming soon.

License

Apache License, Version 2.0

Model Index