section_path: Development and release history
section_name: Development and release history
paragraph_text: Development and release history
  
  The first DeepSeek models were essentially the same as Llama, which were dense decoder-only transformers. Later models incorporated the multi-head latent attention (MLA), Mixture of Experts (MoE), and KV caching.
  
  A decoder-only transformer consists of multiple identical decoder layers. Each of these layers features two main components: an attention layer and a FeedForward network (FFN) layer. In the attention layer, the traditional multi-head attention mechanism has been enhanced with multi-head latent attention. This update introduces compressed latent vectors to boost performance and reduce memory usage during inference.
  
  Meanwhile, the FFN layer adopts a variant of the mixture of experts (MoE) approach, effectively doubling the number of experts compared to standard implementations. It distinguishes between two types of experts: shared experts, which are always active to encapsulate general knowledge, and routed experts, only a select few of which are activated to capture specialized information.
  
  Consider the current sequence of n tokens as input. To predict the next token based on the current input, the attention mechanism involves extensive calculations of matrices, including query (Q), key (K), and value (V) matrices. The dimensions of Q, K, and V are determined by the current number of tokens and the model's embedding size. Once the new token is generated, the autoregressive procedure appends it to the end of the input sequence, and the transformer layers repeat the matrix calculation for the next token. A mathematical analysis reveals that the new token introduces a new query, key, and value vector, appended to Q, K, and V, respectively. Appending these new vectors to the K and V matrices is sufficient for calculating the next token prediction. Consequently, storing the current K and V matrices in memory saves time by avoiding the recalculation of the attention matrix. This feature is known as KV (key–value) caching. This technique effectively reduces computational cost during inference.