--- license: mit language: - en - ja - ko - vi base_model: - Qwen/Qwen2.5-1.5B pipeline_tag: text-generation library_name: transformers --- # NxMobileLM-1.5B-SFT ## Model Description `NxMobileLM-1.5B-SFT` is a fine-tuned version of the base model `Qwen2.5-1.5B`, optimized for mobile and edge applications. This model has been trained on proprietary instruction datasets curated to enhance performance in natural language understanding and generation tasks tailored to specific applications. ### Key Features: - **Base Model:** Qwen2.5-1.5B - **Parameter Count:** 1.5 billion - **Fine-tuning Objective:** Supervised fine-tuning (SFT) on instruction datasets. - **Specialization:** Lightweight and efficient performance for mobile environments. - **Multilingual Support:** Designed to handle multiple languages effectively, enabling robust cross-lingual capabilities for diverse applications. ## Model Details ### Training Data The model was fine-tuned using a proprietary dataset designed for diverse instruction-following tasks, including question answering, summarization, and dialogue. The dataset emphasizes: - Multi-domain generalization - Task-specific instruction understanding - Multilingual Coverage: Training data includes samples from several major languages to enhance cross-lingual understanding. ### Training Configuration - **Framework:** PyTorch - **Optimizer:** AdamW - **Learning Rate:** 5e-5 - **Batch Size:** 128 - **Epochs:** 3 - **Mixed Precision:** FP16 ### Evaluation The model was evaluated on a variety of benchmarks, demonstrating: - **High Accuracy:** Achieves strong performance across general natural language tasks. - **Efficiency:** Optimized for low-latency inference on edge devices. - **Multilingual Competence:** Strong performance across multiple languages, making it suitable for global applications. ### Performance Comparison #### Open-LLM Leaderboard On January 15, 2025, NxMobileLM-1.5B-SFT was ranked among the top 10 edge device models with fewer than 3 billion parameters and achieved the first rank for models with under 2 billion parameters, according to the [OpenLLM leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?params=0%2C3). ![image/png](https://cdn-uploads.huggingface.co/production/uploads/5ee1b417636bdb3834e2da19/6877IlKJN7agg-6Zt--ZY.png) #### P-MMEval To evaluate the multilingual capabilities of the model, We conducted evaluations on several benchmarks across three languages: English (en), Japanese (ja), and Vietnamese (vi). For detailed benchmark information, refer to [P-MMEval](https://huggingface.co/datasets/Qwen/P-MMEval). | Benchmark | Llama-3.2-1B-Instruct | SmolLM2-1.7B-Instruct | Qwen2.5-1.5B-Instruct | NxMobileLM-1.5B-SFT | |------------------|------------------------|------------------------|------------------------|---------------------| | mifeval-en | 44.79 | 43.75 | 50 | **57.29** | | mifeval-ja | 22.92 | 23.96 | 29.17 | **30.21** | | mifeval-vi | 30.21 | 25 | 28.12 | **46.88** | | mmmlu-EN-US | 35.25 | 42.5 | **45.5** | 45.25 | | mmmlu-JA-JP | 31.5 | 26.25 | 36.00 | **41.00** | | mmmlu-VI-VT | 22.75 | 22.25 | **39.00** | 38.00 | | xnli-en | 35.83 | 35.83 | 59.17 | **66.67** | | xnli-ja | 34.17 | 35.83 | 52.5 | **57.5** | | xnli-vi | 37.5 | 34.17 | 45.83 | **55.83** | | **Average** | 32.21 | 31.61 | 42.93 | **48.07** | #### LightEval The table below compares `NxMobileLM-1.5B-SFT` with other instruction-tuned models using various benchmarks. Results were obtained using the [lighteval](https://github.com/huggingface/lighteval) evaluation framework and are referenced from [Hugging Face TB](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct): | Metric | SmolLM2-1.7B-Instruct | Llama-1B-Instruct | Qwen2.5-1.5B-Instruct | SmolLM1-1.7B-Instruct | NxMobileLM-1.5B-SFT | |------------------------------|-----------------------|-------------------|-----------------------|-----------------------|-----------------| | IFEval (Average prompt/inst) | 56.7 | 53.5 | 47.4 | 23.1 | **64.2** | | HellaSwag | **66.1** | 56.1 | 60.9 | 55.5 | 63.57 | | ARC (Average) | **51.7** | 41.6 | 46.2 | 43.7 | 45.21 | | PIQA | **74.4** | 72.3 | 73.2 | 71.6 | 72.91 | | MMLU-Pro (MCF) | 19.3 | 12.7 | **24.2** | 11.7 | 15.43 | | BBH (3-shot) | 32.2 | 27.6 | **35.3** | 25.7 | 31.44 | | GSM8K (5-shot) | 48.2 | 26.8 | 42.8 | 4.62 | **59.51** | | **Average** | 49.8 | 41.5 | 47.1 | 33.7 | **50.3** | ### Limitations While `NxMobileLM-1.5B-SFT` excels in many areas, it may not perform well on tasks outside the scope of the fine-tuned dataset. Biases inherent in the training data may also affect outcomes. ## Intended Use `NxMobileLM-1.5B-SFT` is designed for use in: - Mobile virtual assistants - Real-time language-based applications - Compact edge AI solutions - Multilingual Scenarios: Supporting applications that require cross-lingual communication and understanding. **Misuse Warning:** The model is not intended for use in generating harmful, biased, or illegal content. ## How to Use Here is a sample code snippet to load and use the model: ```python from transformers import AutoModelForCausalLM, AutoTokenizer # Load the model and tokenizer model_name = "NTQAI/NxMobileLM-1.5B-SFT" model = AutoModelForCausalLM.from_pretrained(model_name) tokenizer = AutoTokenizer.from_pretrained(model_name) # Example usage inputs = tokenizer("What is the capital of Vietnam?", return_tensors="pt") outputs = model.generate(**inputs, max_length=50) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` ## Citation If you use this model in your research, please cite it as: ``` @misc{NxMobileLM-1.5B-SFT, title={NxMobileLM-1.5B-SFT}, author={NTQAI}, year={2025}, url={https://huggingface.co/NTQAI/NxMobileLM-1.5B-SFT}, } ``` ## License This model is licensed under MIT. ## Contact For questions or issues, please contact us via website: https://ntq.ai