README.md

---
base_model:
- alpindale/Llama-3.2-3B
- belyakoff/llama-3.2-3b-instruct-fine-tuned
- unsloth/Llama-3.2-3B-Instruct
- Medragondot/llama-3.2-3b-thinking
library_name: transformers
tags:
- mergekit
- merge

---
# merge

This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).

## Merge Details
### Merge Method

This model was merged using the [DARE TIES](https://arxiv.org/abs/2311.03099) merge method using [alpindale/Llama-3.2-3B](https://huggingface.co/alpindale/Llama-3.2-3B) as a base.

### Models Merged

The following models were included in the merge:
* [belyakoff/llama-3.2-3b-instruct-fine-tuned](https://huggingface.co/belyakoff/llama-3.2-3b-instruct-fine-tuned)
* [unsloth/Llama-3.2-3B-Instruct](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct)
* [Medragondot/llama-3.2-3b-thinking](https://huggingface.co/Medragondot/llama-3.2-3b-thinking)

### Configuration

The following YAML configuration was used to produce this model:

```yaml
# Define the models to be used in the merge, along with their respective weight and density parameters.
models:
  - model: alpindale/Llama-3.2-3B  # The first model in the merge.
    parameters:
      weight: 0.3  # Weight determines how much influence this model will have in the final output. Higher weight means more influence.
      density: 0.35  # Density specifies how many of the model's parameters are retained during the merging process. Higher density keeps more parameters.

  - model: unsloth/Llama-3.2-3B-Instruct  # The second model in the merge.
    parameters:
      weight: 0.25  # A slightly smaller weight indicates this model will have less impact on the final merged model.
      density: 0.25  # A moderate density ensures this model's parameters are included, but not as heavily as others.

  - model: belyakoff/llama-3.2-3b-instruct-fine-tuned  # The third model in the merge.
    parameters:
      weight: 0.25  # Similar to the second model, this model contributes less to the final model.
      density: 0.25  # Keeps a balanced contribution of parameters for the merge.

  - model: Medragondot/llama-3.2-3b-thinking  # The fourth model in the merge.
    parameters:
      weight: 0.2  # This model will have the least influence on the merged output.
      density: 0.15  # The lowest density means fewer of this model’s parameters will contribute to the final merge.

# Specify the merge method to be used.
merge_method: dare_ties  # The DARE-TIES method is used to merge models by estimating residuals between them. This allows for fine-tuning and adjusting contributions for each model layer.

# Set the base model for the merge. This model serves as the foundation for blending the other models.
base_model: alpindale/Llama-3.2-3B  # The base model is typically the one that will retain the highest influence in the final merged model.

# Define additional parameters to customize the merging behavior.
parameters:
  normalize: true  # Normalization ensures the weights across models are balanced so the merge remains stable and well-scaled.
  int8_mask: true  # Enables int8 masking, which optimizes performance by using 8-bit integers for certain computations, reducing memory usage.
  interpolation_factor: 0.7  # Controls the blending strength between the models. Values closer to 1 will favor the base model, while values closer to 0 distribute more influence evenly among models.
dtype: bfloat16  # Uses bfloat16 (brain floating point 16) format to store weights, offering a good balance between numerical precision and memory efficiency for model merging.
```