eaddario/imatrix-calibration
Viewer
•
Updated
•
907k
•
13.6k
•
2
SuperNintendoChalmers
SuperNintendoChalmers
AI & ML interests
None yet
Recent Activity
liked
a dataset
2 days ago
eaddario/imatrix-calibration
new activity
19 days ago
mradermacher/model_requests:allura-org/L3.1-8b-RP-Ink
new activity
20 days ago
mradermacher/model_requests:FreedomIntelligence/Apollo-MoE-7B
Organizations
SuperNintendoChalmers's activity
liked
a
dataset
2 days ago
allura-org/L3.1-8b-RP-Ink
1
#713 opened 19 days ago
by
SuperNintendoChalmers
FreedomIntelligence/Apollo-MoE-7B
3
#709 opened 20 days ago
by
SuperNintendoChalmers
Tulu 3.1
2
#687 opened 25 days ago
by
SuperNintendoChalmers
liked
2
models
23 days ago
Any value in static quants?
2
#688 opened 25 days ago
by
SuperNintendoChalmers
Very cool! Also how?
1
#2 opened about 1 month ago
by
SuperNintendoChalmers
motexture's cData models
3
#654 opened about 1 month ago
by
SuperNintendoChalmers
liked
a
dataset
about 1 month ago
Regenerate quants to include bufixes?
2
#5 opened about 1 month ago
by
SuperNintendoChalmers
Re-generate quants for new model with same name
2
#3 opened about 1 month ago
by
SuperNintendoChalmers
Request to fix bugs quicker next time
#37 opened about 1 month ago
by
SuperNintendoChalmers
nvidia/Hymba-1.5B-Instruct
1
#557 opened 2 months ago
by
SuperNintendoChalmers
liked
2
datasets
3 months ago
Granite 3 MoE 1B returns all 4's
3
#2 opened 5 months ago
by
AMead10
Falcon 3 10B broken?
1
#5 opened 3 months ago
by
SuperNintendoChalmers
reacted to
bartowski's
post with 👍
3 months ago
Post
64756
Looks like Q4_0_N_M file types are going away
Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable)
You can see the reference PR here:
https://github.com/ggerganov/llama.cpp/pull/10446
So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms)
As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those !
Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: https://github.com/ggerganov/llama.cpp/pull/10541
Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights
Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable)
You can see the reference PR here:
https://github.com/ggerganov/llama.cpp/pull/10446
So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms)
As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those !
Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: https://github.com/ggerganov/llama.cpp/pull/10541
Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights
- 16 replies
liked
a
Space
3 months ago