Novaciano's picture
Adding Evaluation Results (#1)
ad7f283 verified
metadata
base_model:
  - Novaciano/LAMED
  - Novaciano/VAV
  - Novaciano/TAV
  - Novaciano/YOD
  - Novaciano/NUN-FINAL
  - Novaciano/BAPHOMET
library_name: transformers
tags:
  - mergekit
  - merge
  - abliterated
  - uncensored
  - llama
  - llama3.2
  - not-for-all-audiences
language:
  - en
  - es
model-index:
  - name: Sigil-Of-Satan-3.2-1B
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: IFEval (0-Shot)
          type: wis-k/instruction-following-eval
          split: train
          args:
            num_few_shot: 0
        metrics:
          - type: inst_level_strict_acc and prompt_level_strict_acc
            value: 54.94
            name: averaged accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Novaciano%2FSigil-Of-Satan-3.2-1B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: BBH (3-Shot)
          type: SaylorTwift/bbh
          split: test
          args:
            num_few_shot: 3
        metrics:
          - type: acc_norm
            value: 9.4
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Novaciano%2FSigil-Of-Satan-3.2-1B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MATH Lvl 5 (4-Shot)
          type: lighteval/MATH-Hard
          split: test
          args:
            num_few_shot: 4
        metrics:
          - type: exact_match
            value: 5.44
            name: exact match
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Novaciano%2FSigil-Of-Satan-3.2-1B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GPQA (0-shot)
          type: Idavidrein/gpqa
          split: train
          args:
            num_few_shot: 0
        metrics:
          - type: acc_norm
            value: 1.45
            name: acc_norm
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Novaciano%2FSigil-Of-Satan-3.2-1B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MuSR (0-shot)
          type: TAUR-Lab/MuSR
          args:
            num_few_shot: 0
        metrics:
          - type: acc_norm
            value: 1.42
            name: acc_norm
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Novaciano%2FSigil-Of-Satan-3.2-1B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU-PRO (5-shot)
          type: TIGER-Lab/MMLU-Pro
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 9.5
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Novaciano%2FSigil-Of-Satan-3.2-1B
          name: Open LLM Leaderboard

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

pngimg-com-pentagram-PNG23

🇬🇧 English

Remixed version of HarmfulProject-3.2-1B model.

I'll note that there's nothing unusual about it other than being a mix of mixes of uncensored and abliterated models.

The difference with HarmfulProject-3.2-1B is that instead of being mixed in one model, the same models have been mixed in smaller mixes in this model.

Author's Note:

I'm not responsible for the content of the model since I only made the mix, I didn't inject any dataset into it... yet.

Others models created with this merge


🇪🇦 Español

Versión remezclada / remixada del modelo HarmfulProject-3.2-1B.

Aviso desde ya que no tiene nada raro mas allá de ser una mezcla de mezclas de modelos abliterados y sin censura.

La diferencia con HarmfulProject-3.2-1B es que en vez de haber sido mezclados en un modelo en este modelo se ha mezclado los mismos modelos pero en mezclas más pequeñas.

AVISO: No me hago cargo del contenido del modelo ya que solo hice la mezcla, no le inyecté dataset alguno... aún.

Otros modelos creados con esta mezclas


Quants / Cuantizaciones


Merge Method

This model was merged using the Model Stock merge method using Novaciano/BAPHOMET as a base.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

models:
- model: Novaciano/LAMED
- model: Novaciano/VAV
- model: Novaciano/YOD
- model: Novaciano/TAV
- model: Novaciano/NUN-FINAL

merge_method: model_stock
base_model: Novaciano/BAPHOMET
dtype: bfloat16
parameters:
  t: [0, 0.5, 1, 0.5, 0]

Open LLM Leaderboard Evaluation Results

Detailed results can be found here! Summarized results can be found here!

Metric Value (%)
Average 13.69
IFEval (0-Shot) 54.94
BBH (3-Shot) 9.40
MATH Lvl 5 (4-Shot) 5.44
GPQA (0-shot) 1.45
MuSR (0-shot) 1.42
MMLU-PRO (5-shot) 9.50