Hugging Face's logo

schnapper79
/
lumikabra_behemoth_195b

Text Generation
Transformers
Safetensors
mistral
mergekit
Merge
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
2

Upload folder using huggingface_hub

#1
by schnapper79 - opened
base: refs/heads/main
←
from: refs/pr/1
Discussion Files changed
+378449
-0
This view is limited to 50 files because it contains too many changes.  See the raw diff here.
Files changed (50) hide show
  1. README.md +39 -0
  2. config.json +27 -0
  3. measurement.json +0 -0
  4. mergekit_config.yml +9 -0
  5. model-00001-of-00081.safetensors +3 -0
  6. model-00002-of-00081.safetensors +3 -0
  7. model-00003-of-00081.safetensors +3 -0
  8. model-00004-of-00081.safetensors +3 -0
  9. model-00005-of-00081.safetensors +3 -0
  10. model-00006-of-00081.safetensors +3 -0
  11. model-00007-of-00081.safetensors +3 -0
  12. model-00008-of-00081.safetensors +3 -0
  13. model-00009-of-00081.safetensors +3 -0
  14. model-00010-of-00081.safetensors +3 -0
  15. model-00011-of-00081.safetensors +3 -0
  16. model-00012-of-00081.safetensors +3 -0
  17. model-00013-of-00081.safetensors +3 -0
  18. model-00014-of-00081.safetensors +3 -0
  19. model-00015-of-00081.safetensors +3 -0
  20. model-00016-of-00081.safetensors +3 -0
  21. model-00017-of-00081.safetensors +3 -0
  22. model-00018-of-00081.safetensors +3 -0
  23. model-00019-of-00081.safetensors +3 -0
  24. model-00020-of-00081.safetensors +3 -0
  25. model-00021-of-00081.safetensors +3 -0
  26. model-00022-of-00081.safetensors +3 -0
  27. model-00023-of-00081.safetensors +3 -0
  28. model-00024-of-00081.safetensors +3 -0
  29. model-00025-of-00081.safetensors +3 -0
  30. model-00026-of-00081.safetensors +3 -0
  31. model-00027-of-00081.safetensors +3 -0
  32. model-00028-of-00081.safetensors +3 -0
  33. model-00029-of-00081.safetensors +3 -0
  34. model-00030-of-00081.safetensors +3 -0
  35. model-00031-of-00081.safetensors +3 -0
  36. model-00032-of-00081.safetensors +3 -0
  37. model-00033-of-00081.safetensors +3 -0
  38. model-00034-of-00081.safetensors +3 -0
  39. model-00035-of-00081.safetensors +3 -0
  40. model-00036-of-00081.safetensors +3 -0
  41. model-00037-of-00081.safetensors +3 -0
  42. model-00038-of-00081.safetensors +3 -0
  43. model-00039-of-00081.safetensors +3 -0
  44. model-00040-of-00081.safetensors +3 -0
  45. model-00041-of-00081.safetensors +3 -0
  46. model-00042-of-00081.safetensors +3 -0
  47. model-00043-of-00081.safetensors +3 -0
  48. model-00044-of-00081.safetensors +3 -0
  49. model-00045-of-00081.safetensors +3 -0
  50. model-00046-of-00081.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: []
3
+ library_name: transformers
4
+ tags:
5
+ - mergekit
6
+ - merge
7
+
8
+ ---
9
+ # lumikabra_behemoth_195b
10
+
11
+ This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
12
+
13
+ ## Merge Details
14
+ ### Merge Method
15
+
16
+ This model was merged using the passthrough merge method.
17
+
18
+ ### Models Merged
19
+
20
+ The following models were included in the merge:
21
+ * /workspace/models/schnapper79_lumikabra-123B_v0.3
22
+ * /workspace/models/TheDrummer_Behemoth-123B-v1
23
+
24
+ ### Configuration
25
+
26
+ The following YAML configuration was used to produce this model:
27
+
28
+ ```yaml
29
+ dtype: bfloat16
30
+ merge_method: passthrough
31
+ slices:
32
+ - sources:
33
+ - layer_range: [0, 70]
34
+ model: /workspace/models/schnapper79_lumikabra-123B_v0.3
35
+ - sources:
36
+ - layer_range: [18, 88]
37
+ model: /workspace/models/TheDrummer_Behemoth-123B-v1
38
+
39
+ ```
config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "/workspace/models/schnapper79_lumikabra-123B_v0.3",
3
+ "architectures": [
4
+ "MistralForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "head_dim": 128,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 12288,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 28672,
14
+ "max_position_embeddings": 131072,
15
+ "model_type": "mistral",
16
+ "num_attention_heads": 96,
17
+ "num_hidden_layers": 140,
18
+ "num_key_value_heads": 8,
19
+ "rms_norm_eps": 1e-05,
20
+ "rope_theta": 1000000.0,
21
+ "sliding_window": null,
22
+ "tie_word_embeddings": false,
23
+ "torch_dtype": "bfloat16",
24
+ "transformers_version": "4.45.2",
25
+ "use_cache": true,
26
+ "vocab_size": 32768
27
+ }
measurement.json ADDED
The diff for this file is too large to render. See raw diff
 
mergekit_config.yml ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ dtype: bfloat16
2
+ merge_method: passthrough
3
+ slices:
4
+ - sources:
5
+ - layer_range: [0, 70]
6
+ model: /workspace/models/schnapper79_lumikabra-123B_v0.3
7
+ - sources:
8
+ - layer_range: [18, 88]
9
+ model: /workspace/models/TheDrummer_Behemoth-123B-v1
model-00001-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:25eb0d85319381a1d524a6d1956b86b126a52a1ea953382cc2888696588124af
3
+ size 4378928504
model-00002-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:95a1be3519207a56f51deb85a788013c6081794d2af03b58a60c96bfe93735e5
3
+ size 4907411088
model-00003-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:561bef5b2fdd3d814639c55b761d2f5d6f248acbf06279820b79295d13e6f7e1
3
+ size 4806747904
model-00004-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f3aea6320a7254f1e10b0b1f68d20429a0bb7068882d1e90de125192762d3546
3
+ size 4831938544
model-00005-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:58324e621dec2ae32a2d9cc18d91f29e044d5dc258c9e7f572bab47f64dd6aed
3
+ size 4831938552
model-00006-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8b8320b63a92bba5db3d3855737f5f8eda778d79763328c00012b3fcbac26d85
3
+ size 4907411096
model-00007-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f7fe3c88a80443711cc319ffd01abe5c5273dce3e0ffd5080b442519e59f14bc
3
+ size 4907435776
model-00008-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e8f0fd41ee65cfbe601e6463fbd366615bf0ea6d6f3af43095250b2e9ad897a
3
+ size 4781557264
model-00009-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:683f2a72d96cc3604828d4ac22f0950ae93edc1d259f318c5bb2b5359039a99d
3
+ size 4781656640
model-00010-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:472687fa8a4ef497b08bc5f3bf51aaf5c8694e4cc4c628684ad5dc6795e9d49c
3
+ size 4882220464
model-00011-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5136489ea6a90aee0683af980650bb7a4b1eb479b915bcecb21d6906ac677c76
3
+ size 4932601720
model-00012-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:52c88b1a3fa840484907de2b53b948ef7cbc6c9ccfbb4c5723f8bb016c8bf47d
3
+ size 4781557264
model-00013-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7f712eb7a0d9a62abf1a57311e1ac5a810c3c3b830503c6c31c5d89b5064bbce
3
+ size 4831938552
model-00014-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:342143428e6d5eac3bfe91394eaf0278a51fd5aed1a4acfbc9236e982640ac28
3
+ size 4831938544
model-00015-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ed501ad3b4b59a52b43a164de33fdb8cb74ee717cb83a9c94a8f1282aa3d362a
3
+ size 4831938552
model-00016-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f793cead76c80c964447bcb57cdd10ffd483a60d7fcae8d22195c7e006b8d67a
3
+ size 4831938552
model-00017-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:690c054a9fdcd4ee4494f4792c24fd44380456c2c65789e9929fdd427e5d7bf3
3
+ size 4831938552
model-00018-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:81c53e1ed642845199ae464a217b4fab7e6d3775dd632ea57f569ae813cbb19f
3
+ size 4882220464
model-00019-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:19d8826e11b398c794637f87b3c0bbe4d01bcaec178765257ddc7567619b7405
3
+ size 4932601720
model-00020-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5440ba460331e98d5f4c8f2436d3e964b09003635c62ee104093c9f614917d00
3
+ size 4781557264
model-00021-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a47db702cf4ba581f086b53e94572ddc8235d0dce7ae73029d54263affb8a02d
3
+ size 4781656640
model-00022-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9fd64f0314b71264bc523676cefb7b8ba1cb6e28b25fa77a895dda3ef2f228b3
3
+ size 4882220464
model-00023-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3f34528e31089fceb091aa28c3519e85bf642311d9efa59720b008f26dcf375b
3
+ size 4932601720
model-00024-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:174333572045d8c780c9e39de362c81c3b28336d4581fffc4183f586f821dde6
3
+ size 4781557264
model-00025-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ba66d1667e61fefe57dc16a972e29e60f2671fb6536b1c8ac93f21ff155fa2e1
3
+ size 4831938552
model-00026-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9ee69f530c6fa44b607e309fa820b6802ec61a69fb20cf92616d34dcebbf7e36
3
+ size 4831938544
model-00027-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b20fd06db5ed950e031835110f76d7ec3e09990346cb6654464dd1419f5c973d
3
+ size 4831938552
model-00028-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e630e2df31c26b52813354e9326783b6c98d82bbe66dc6bcef22b1c66797739e
3
+ size 4831938552
model-00029-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4c6728cbfdd3b4c146757dbbf62ec6d05a5043d8eef2455c020edad424159d4e
3
+ size 4831938552
model-00030-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cd22a96129108a3aaf9c662474dc8f71d906200d1b83c20b578cb82f6a0bea68
3
+ size 4882220464
model-00031-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:583262fd6e22d5150902c03fa7537e2585c716e9e9ef968cd2a74211f84b2a1a
3
+ size 4932601720
model-00032-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:78cddc0ce21a8f0db3ee362678acbf282c81d26e5061ab228048d846039d7ac9
3
+ size 4781557264
model-00033-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0980dc1e921315fefdb2b3a65b468ea91fc3d56b56f6880c29d79de980d46d8f
3
+ size 4781656640
model-00034-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0ca1ff07f107600bd06187e0ade715327d6d6e9f335ac37a0af22f4c60b2b9ce
3
+ size 4882220464
model-00035-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dd045d2310f836b33795f62575a81242aeca32944a9d3391442a18f844fadefc
3
+ size 4932601720
model-00036-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e0733ed83dfb564cf1c8c882645616ef65825b02a178735ff39fb58c1cfdfc64
3
+ size 4781557264
model-00037-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:957563de5dd1302df15d53f1dbe7a0b01b8feb48a02273fd7f133418d3f1f22f
3
+ size 4831938552
model-00038-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:98404bf2e8d7be521d147394ffd7f8f67d4e07b8ff4f8b99a13e3678cf470761
3
+ size 4831938544
model-00039-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1dbc1f28339935222d212f1ce370476a2d427700bf877242f89875c6904690e3
3
+ size 4831938552
model-00040-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:13ea93e5b7b9b4dabdbfaccf04f15b4a6ab983892ce8ae950caa97a27a85d970
3
+ size 4831938552
model-00041-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:828b5608aac4c9a465331a0c9304a2038fef61ab6a5e309dc4d90049d0449c71
3
+ size 4831938552
model-00042-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a3516ab7579f6be1e00d8f18745a3f1cf18be353a98c8b59c90dc9a5baf315c5
3
+ size 4882220464
model-00043-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2d701ed34d0f7534e1163005ca2691d480853d1d8a8f95877e747f7a5e08e7a9
3
+ size 4932601720
model-00044-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b3dd4b0e40606f40d44eced9049ee623991a6e5f69984b6c4ff56085e78f3f1d
3
+ size 4781557272
model-00045-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:01a35ae46c01950f0763462911d036a73801c7594dca4451c7c39d1a350c4f56
3
+ size 4781656632
model-00046-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ea5bb82235140b61cf40b485d1090636708d453d2482cbb03537c6e919e1b6d4
3
+ size 4882220464