MISHANM/meta-Llama-3.1-405B-Instruct.gguf

This model is a GGUF version of the meta-llama/Llama-3.1-405B-Instruct, optimized for use with the llama.cpp framework. It is designed to run efficiently on CPUs, specifically tested on the AMD EPYC™ 9755, and can be used for various natural language processing tasks.

Model Details

1. Language: English
2. Tasks: Text generation
3. Base Model: meta-llama/Llama-3.1-405B-Instruct

Building and Running the Model

To build and run the model using llama.cpp, follow these steps:

Model

Steps to Download the Model:

1. Go to the "Files and Versions" section.
2. Click on the model.
3. Copy the download link.
4. Create a directory (e.g., for Linux: mkdir Llama-3.1-405B-Instruct).
5. Navigate to that directory (cd Llama-3.1-405B-Instruct).
6. Download the model parts: Llama-3.1-405B-Instruct.gguf.part_01, Llama-3.1-405B-Instruct.gguf.part_02, Llama-3.1-405B-Instruct.gguf.part_03, Llama-3.1-405B-Instruct.gguf.part_04, Llama-3.1-405B-Instruct.gguf.part_05, Llama-3.1-405B-Instruct.gguf.part_06, Llama-3.1-405B-Instruct.gguf.part_07, Llama-3.1-405B-Instruct.gguf.part_08, Llama-3.1-405B-Instruct.gguf.part_09, Llama-3.1-405B-Instruct.gguf.part_10, Llama-3.1-405B-Instruct.gguf.part_11, Llama-3.1-405B-Instruct.gguf.part_12, Llama-3.1-405B-Instruct.gguf.part_13, Llama-3.1-405B-Instruct.gguf.part_14, Llama-3.1-405B-Instruct.gguf.part_15, Llama-3.1-405B-Instruct.gguf.part_16, Llama-3.1-405B-Instruct.gguf.part_17 (e.g., using wget with the copied link).

After downloading the model parts, use the following command to combine them into a complete model:

cat Llama-3.1-405B-Instruct.gguf.part_01 Llama-3.1-405B-Instruct.gguf.part_02 Llama-3.1-405B-Instruct.gguf.part_03 Llama-3.1-405B-Instruct.gguf.part_04 Llama-3.1-405B-Instruct.gguf.part_05 Llama-3.1-405B-Instruct.gguf.part_06 Llama-3.1-405B-Instruct.gguf.part_07 Llama-3.1-405B-Instruct.gguf.part_08 Llama-3.1-405B-Instruct.gguf.part_09 Llama-3.1-405B-Instruct.gguf.part_10 Llama-3.1-405B-Instruct.gguf.part_11 Llama-3.1-405B-Instruct.gguf.part_12 Llama-3.1-405B-Instruct.gguf.part_13 Llama-3.1-405B-Instruct.gguf.part_14 Llama-3.1-405B-Instruct.gguf.part_15 Llama-3.1-405B-Instruct.gguf.part_16 Llama-3.1-405B-Instruct.gguf.part_17  > meta-Llama-3.1-405B-Instruct.gguf

Build llama.cpp Locally

git clone https://github.com/ggerganov/llama.cpp  
cd llama.cpp  
cmake -B build  
cmake --build build --config Release  

Run the Model

Navigate to the build directory and run the model with a prompt:

cd llama.cpp/build/bin   

Inference with llama.cpp

./llama-cli -m /path/to/meta-Llama-3.1-405B-Instruct.gguf/ -p "Your prompt here" -n 500 --ctx-size 8192 --temp 0.6 --seed 3407

Citation Information

@misc{MISHANM/meta-Llama-3.1-405B-Instruct.gguf,
  author = {Mishan Maurya},
  title = {Introducing meta-llama/Llama-3.1-405B-Instruct GGUF Model},
  year = {2025},
  publisher = {Hugging Face},
  journal = {Hugging Face repository},
  
}