NxMobileLM-1.5B-SFT

Model Description

NxMobileLM-1.5B-SFT is a fine-tuned version of the base model Qwen2.5-1.5B, optimized for mobile and edge applications. This model has been trained on proprietary instruction datasets curated to enhance performance in natural language understanding and generation tasks tailored to specific applications.

Key Features:

• Base Model: Qwen2.5-1.5B
• Parameter Count: 1.5 billion
• Fine-tuning Objective: Supervised fine-tuning (SFT) on instruction datasets.
• Specialization: Lightweight and efficient performance for mobile environments.
• Multilingual Support: Designed to handle multiple languages effectively, enabling robust cross-lingual capabilities for diverse applications.

Model Details

Training Data

The model was fine-tuned using a proprietary dataset designed for diverse instruction-following tasks, including question answering, summarization, and dialogue. The dataset emphasizes:

• Multi-domain generalization
• Task-specific instruction understanding
• Multilingual Coverage: Training data includes samples from several major languages to enhance cross-lingual understanding.

Training Configuration

• Framework: PyTorch
• Optimizer: AdamW
• Learning Rate: 5e-5
• Batch Size: 128
• Epochs: 3
• Mixed Precision: FP16

Evaluation

The model was evaluated on a variety of benchmarks, demonstrating:

• High Accuracy: Achieves strong performance across general natural language tasks.
• Efficiency: Optimized for low-latency inference on edge devices.
• Multilingual Competence: Strong performance across multiple languages, making it suitable for global applications.

Performance Comparison

Open-LLM Leaderboard

On January 15, 2025, NxMobileLM-1.5B-SFT was ranked among the top 10 edge device models with fewer than 3 billion parameters and achieved the first rank for models with under 2 billion parameters, according to the OpenLLM leaderboard.

P-MMEval

To evaluate the multilingual capabilities of the model, We conducted evaluations on several benchmarks across three languages: English (en), Japanese (ja), and Vietnamese (vi). For detailed benchmark information, refer to P-MMEval.

Benchmark	Llama-3.2-1B-Instruct	SmolLM2-1.7B-Instruct	Qwen2.5-1.5B-Instruct	NxMobileLM-1.5B-SFT
mifeval-en	44.79	43.75	50	57.29
mifeval-ja	22.92	23.96	29.17	30.21
mifeval-vi	30.21	25	28.12	46.88
mmmlu-EN-US	35.25	42.5	45.5	45.25
mmmlu-JA-JP	31.5	26.25	36.00	41.00
mmmlu-VI-VT	22.75	22.25	39.00	38.00
xnli-en	35.83	35.83	59.17	66.67
xnli-ja	34.17	35.83	52.5	57.5
xnli-vi	37.5	34.17	45.83	55.83
Average	32.21	31.61	42.93	48.07

LightEval

The table below compares NxMobileLM-1.5B-SFT with other instruction-tuned models using various benchmarks. Results were obtained using the lighteval evaluation framework and are referenced from Hugging Face TB:

Metric	SmolLM2-1.7B-Instruct	Llama-1B-Instruct	Qwen2.5-1.5B-Instruct	SmolLM1-1.7B-Instruct	NxMobileLM-1.5B-SFT
IFEval (Average prompt/inst)	56.7	53.5	47.4	23.1	64.2
HellaSwag	66.1	56.1	60.9	55.5	63.57
ARC (Average)	51.7	41.6	46.2	43.7	45.21
PIQA	74.4	72.3	73.2	71.6	72.91
MMLU-Pro (MCF)	19.3	12.7	24.2	11.7	15.43
BBH (3-shot)	32.2	27.6	35.3	25.7	31.44
GSM8K (5-shot)	48.2	26.8	42.8	4.62	59.51
Average	49.8	41.5	47.1	33.7	50.3

Limitations

While NxMobileLM-1.5B-SFT excels in many areas, it may not perform well on tasks outside the scope of the fine-tuned dataset. Biases inherent in the training data may also affect outcomes.

Intended Use

NxMobileLM-1.5B-SFT is designed for use in:

• Mobile virtual assistants
• Real-time language-based applications
• Compact edge AI solutions
• Multilingual Scenarios: Supporting applications that require cross-lingual communication and understanding.

Misuse Warning: The model is not intended for use in generating harmful, biased, or illegal content.

How to Use

Here is a sample code snippet to load and use the model:

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model and tokenizer
model_name = "NTQAI/NxMobileLM-1.5B-SFT"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Example usage
inputs = tokenizer("What is the capital of Vietnam?", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

If you use this model in your research, please cite it as:

@misc{NxMobileLM-1.5B-SFT,
  title={NxMobileLM-1.5B-SFT},
  author={NTQAI},
  year={2025},
  url={https://huggingface.co/NTQAI/NxMobileLM-1.5B-SFT},
}

License

This model is licensed under MIT.

Contact

For questions or issues, please contact us via website: https://ntq.ai