Upload folder using huggingface_hub

README.md

@@ -16,7 +16,9 @@ For more info see our [End-to-end development service for custom LLMs and AI sys
 
 This model extends LLama-3 8B's context length from 8k to > 1040K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. We trained on 830M tokens for this stage, and 1.4B tokens total for all stages, which is < 0.01% of Llama-3's original pre-training data.
 
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/6MKLoX2ruLIaREiyb6coO.png)
+**Update (5/3): We further fine-tuned our model to strengthen its assistant-like chat ability as well. The NIAH result is updated.** 
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/-qaI__83ksClzoJzlqZjq.png)
 
 **Approach:**
 
@@ -32,7 +34,7 @@ Notably, we layered parallelism on top of Ring Attention with a custom network t
 
 **Data:**
 
-For training data, we generate long contexts by augmenting [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B).
+For training data, we generate long contexts by augmenting [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B). We also fine-tune on a chat dataset based on UltraChat [4], following a similar recipe for data augmentation to [2].
 
 **Progressive Training Details:**
 
@@ -50,9 +52,37 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
 | GPU Type               | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |
 | Minutes to Train (Wall)| 202       | 555       | 61        | 87        |
 
-**Quants**:
-- [GGUF](https://huggingface.co/crusoeai/Llama-3-8B-Instruct-1048k-GGUF)
+
+**Evaluation:**
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/mWxIGZNi3ejlmeIDWafKu.png)
+
+```
+EVAL_MAX_CONTEXT_LENGTH=1040200
+EVAL_MIN_CONTEXT_LENGTH=100
+EVAL_CONTEXT_INTERVAL=86675
+EVAL_DEPTH_INTERVAL=0.2
+EVAL_RND_NUMBER_DIGITS=8
+
+HAYSTACK1:
+    EVAL_GENERATOR_TOKENS=25
+
+HAYSTACK2:
+    EVAL_CONTEXT_INTERVAL=173350
+    EVAL_GENERATOR_TOKENS=150000
+
+HAYSTACK3:
+    EVAL_GENERATOR_TOKENS=925000
+```
+
+All boxes not pictured for Haystack 1 and 3 are 100% accurate. Haystacks 1,2 and 3 are further detailed in this [blog post](https://gradient.ai/blog/the-haystack-matters-for-niah-evals).
+
+**Quants:**
+- [GGUF by Crusoe](https://huggingface.co/crusoeai/Llama-3-8B-Instruct-1048k-GGUF). Note that you need to add 128009 as [special token with llama.cpp](https://huggingface.co/gradientai/Llama-3-8B-Instruct-262k/discussions/13).
 - [MLX-4bit](https://huggingface.co/mlx-community/Llama-3-8B-Instruct-1048k-4bit)
+- [Ollama](https://ollama.com/library/llama3-gradient)
+- vLLM docker image, recommended to load via `--max-model-len 32768`
+- If you are interested in a hosted version, drop us a mail below.
 
 ## The Gradient AI Team
 
@@ -72,6 +102,9 @@ Drop an email to [[email protected]](mailto:[email protected])
 
 [3] https://github.com/jzhang38/EasyContext
 
+[3] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan
+Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling
+high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023.
 
 ----
 

output.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:99270dfbd682a25a819d6bd92e13f0fbba046745ef0543d89126f110182e747f
-size 4956801808
+oid sha256:dbcb8866a8160dd042d8d401b668842b1bf40189f39b635fa5f65d9bca5d9905
+size 4956183188