👋 join us on Discord and WeChat
## Introduction The `Intern-S1-mini` model in GGUF format can be utilized by [llama.cpp](https://github.com/ggerganov/llama.cpp), a highly popular open-source framework for Large Language Model (LLM) inference, across a variety of hardware platforms, both locally and in the cloud. This repository offers `Intern-S1-mini` models in GGUF format in both half precision and various low-bit quantized versions, including `q8_0`. In the subsequent sections, we will first present the installation procedure, followed by an explanation of the model download process. And finally we will illustrate the methods for model inference and service deployment through specific examples. ## Installation We recommend building `llama.cpp` from source. The following code snippet provides an example for the Linux CUDA platform. For instructions on other platforms, please refer to the [official guide](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#build). - Step 1: create a conda environment and install cmake ```shell conda create --name interns1 python=3.10 -y conda activate interns1 pip install cmake ``` - Step 2: clone the source code and build the project ```shell git clone --depth=1 https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build -DGGML_CUDA=ON cmake --build build --config Release -j ``` All the built targets can be found in the sub directory `build/bin` In the following sections, we assume that the working directory is at the root directory of `llama.cpp`. ## Download models In the [introduction section](#introduction), we mentioned that this repository includes several models with varying levels of computational precision. You can download the appropriate model based on your requirements. For instance, fp16 gguf files can be downloaded as below： ```shell pip install huggingface-hub huggingface-cli download internlm/Intern-S1-mini-GGUF --include *-f16.gguf --local-dir Intern-S1-mini-GGUF --local-dir-use-symlinks False ``` ## Inference You can use `build/bin/llama-mtmd-cli` for conducting inference. For a detailed explanation of `build/bin/llama-mtmd-cli`, please refer to [this guide](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md) ### chat example Here is an example of using the thinking system prompt. ```shell system_prompt="<|im_start|>system\nYou are an expert reasoner with extensive experience in all areas. You approach problems through systematic thinking and rigorous reasoning. Your response should reflect deep understanding and precise logical thinking, making your solution path and reasoning clear to others. Please put your thinking process within