Blackroot
/

Mirai-3.0-70B

@@ -54,16 +54,29 @@ Alright. We're still doing the evolutionary thing, take the previous generation,
 # Base Folding
 We have competing objectives for the model. We want model diversity for interesting storytelling. That's really great. However, increasing model diversity also increases EOS diversity. This is a problem. We also want the model to be able to shut up when it wants to. There's a problem, modelstock is using geometric interpolation. In the case of lots of EOS tokens, it's going to act a bit like averaging them. This is a huge problem. We can't disagree about which EOS to use. Hence, we're going to fold the bases together, which all have the same EOS tokens. Specifically, we're going to do a modelstock merge like this:
 - Bllossom/llama-3-Korean-Bllossom-70B
 - rinna/llama-3-youko-70B
 - hitachi-nlp/Llama-3.1-70B-FLDx2
-- (MergedHistLlama -- A custom model stock merge itself of multiple See [Version 0.2 For the details](https://huggingface.co/Blackroot/Mirai-70B-0.2))
 - tokyotech-llm/Llama-3.1-Swallow-70B-v0.1
 Base model:
 - meta-llama/Meta-Llama-3-70B
 All of these were folded into the base model, and they critically all agreed about the EOS token.
 # Model Stock is Really bad at this EOS thing
 I love the interest and diversity of model stock, but after trying many different merges, I realized that I had to try something else. Specifically, TIES. Model stock and TIES are almost polar opposites. TIES acts as an amplifier, when models agree, the task vectors align, TIES strengthens those underlying weights, this means things that are good about the model get amplified, things that are bad get amplified. Model stock smoothes things out, smoothing the weights out between models. If smoothing is an issue, lets try amplifying. I've avoided TIES merges, because I'm specifically trying to avoid some of the bad mannerisms of the base ablation nemo model. However, I tried it anyways. Wouldn't you know it, TIES preserved the EOS, and can actually shut up most of the time. Not only that, but the model result is good. Quite good. The instruct is simply better than prior Mirai's and I don't think it's by a small margin either. There's some quirks, but I'm still able to run inference without any penalties and with the same sampling settings I've been running with. This was really surprising to me, I had not anticipated good results with TIES merging, but I'll eat my shoes now, it's good. The model is by no means perfect, there's some edge areas that end up in strange outputs, and the model occassionaly will insert phrases that appeared commonly into the end of responses. However, overall, I like the result.

 # Base Folding
 We have competing objectives for the model. We want model diversity for interesting storytelling. That's really great. However, increasing model diversity also increases EOS diversity. This is a problem. We also want the model to be able to shut up when it wants to. There's a problem, modelstock is using geometric interpolation. In the case of lots of EOS tokens, it's going to act a bit like averaging them. This is a huge problem. We can't disagree about which EOS to use. Hence, we're going to fold the bases together, which all have the same EOS tokens. Specifically, we're going to do a modelstock merge like this:
+Model Stock:
 - Bllossom/llama-3-Korean-Bllossom-70B
 - rinna/llama-3-youko-70B
 - hitachi-nlp/Llama-3.1-70B-FLDx2
+- (MergedHistLlama -- A custom model stock merge itself of multiple See below)
 - tokyotech-llm/Llama-3.1-Swallow-70B-v0.1
 Base model:
 - meta-llama/Meta-Llama-3-70B
 All of these were folded into the base model, and they critically all agreed about the EOS token.
+# What is the MergedHistLlama?
+Well, it's more merging, Specifically of six base models from PKU-Alignment. The selection of the particular base model was done at random.
+Model Stock:
+- PKU-Alignment/ProgressGym-HistLlama3-70B-C013-pretrain-v0.1
+- PKU-Alignment/ProgressGym-HistLlama3-70B-C020-pretrain-v0.1
+- PKU-Alignment/ProgressGym-HistLlama3-70B-C017-pretrain-v0.1
+- PKU-Alignment/ProgressGym-HistLlama3-70B-C014-pretrain-v0.1
+- PKU-Alignment/ProgressGym-HistLlama3-70B-C021-pretrain-v0.1
+Base Model:
+- PKU-Alignment/ProgressGym-HistLlama3-70B-C018-pretrain-v0.1
 # Model Stock is Really bad at this EOS thing
 I love the interest and diversity of model stock, but after trying many different merges, I realized that I had to try something else. Specifically, TIES. Model stock and TIES are almost polar opposites. TIES acts as an amplifier, when models agree, the task vectors align, TIES strengthens those underlying weights, this means things that are good about the model get amplified, things that are bad get amplified. Model stock smoothes things out, smoothing the weights out between models. If smoothing is an issue, lets try amplifying. I've avoided TIES merges, because I'm specifically trying to avoid some of the bad mannerisms of the base ablation nemo model. However, I tried it anyways. Wouldn't you know it, TIES preserved the EOS, and can actually shut up most of the time. Not only that, but the model result is good. Quite good. The instruct is simply better than prior Mirai's and I don't think it's by a small margin either. There's some quirks, but I'm still able to run inference without any penalties and with the same sampling settings I've been running with. This was really surprising to me, I had not anticipated good results with TIES merging, but I'll eat my shoes now, it's good. The model is by no means perfect, there's some edge areas that end up in strange outputs, and the model occassionaly will insert phrases that appeared commonly into the end of responses. However, overall, I like the result.