Pruned MoEs (Mixtral-8x7B-Instruct-v0.1)

Na0s 's Collections

Depth pruned and fine tuned Llama-3.1-8B

updated Nov 18, 2024

Pruned experts from Mixtral-8x7B-Instruct-v0.1 with respect to the paper "A Provably Effective Method for Pruning Experts in Fine-tuned Sparse MoEs"

Upvote

mistralai/Mixtral-8x7B-Instruct-v0.1

Text Generation • Updated Aug 19, 2024 • 494k • • 4.3k
Na0s/Mixtral-8x7B-Instruct-v0.1-LoRA-on-Gates

Text Generation • Updated Sep 26, 2024 • 4
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA

Text Generation • Updated Nov 12, 2024 • 12
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-1-experts

Text Generation • Updated Oct 6, 2024 • 5
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-2-experts

Text Generation • Updated Oct 6, 2024 • 5
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-3-experts

Text Generation • Updated Oct 6, 2024 • 6
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-4-experts

Text Generation • Updated Oct 6, 2024 • 4
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-1-expert

Text Generation • Updated Oct 6, 2024 • 5
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-2-experts

Text Generation • Updated Oct 5, 2024 • 4
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-3-experts

Text Generation • Updated Oct 6, 2024 • 7
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-4-experts

Text Generation • Updated Oct 6, 2024 • 16
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-1-expert

Text Generation • Updated Nov 18, 2024 • 4
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-2-experts

Text Generation • Updated Nov 18, 2024 • 4
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-3-experts

Text Generation • Updated Nov 18, 2024 • 8
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-4-experts

Text Generation • Updated Nov 18, 2024 • 4

Upvote