initial commit

.ipynb_checkpoints/modules-checkpoint.json

@@ -0,0 +1,20 @@
+[
+    {
+      "idx": 0,
+      "name": "0",
+      "path": "",
+      "type": "sentence_transformers.models.Transformer"
+    },
+    {
+      "idx": 1,
+      "name": "1",
+      "path": "1_Pooling",
+      "type": "sentence_transformers.models.Pooling"
+    },
+    {
+      "idx": 2,
+      "name": "2",
+      "path": "2_Normalize",
+      "type": "sentence_transformers.models.Normalize"
+    }
+]

.ipynb_checkpoints/sentence_bert_config-checkpoint.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 1024,
+  "do_lower_case": false
+}

1_Pooling/.ipynb_checkpoints/config-checkpoint.json

@@ -0,0 +1,7 @@
+{
+    "word_embedding_dimension": 1024,
+    "pooling_mode_cls_token": false,
+    "pooling_mode_mean_tokens": true,
+    "pooling_mode_max_tokens": false,
+    "pooling_mode_mean_sqrt_len_tokens": false
+}

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+    "word_embedding_dimension": 1536,
+    "pooling_mode_cls_token": false,
+    "pooling_mode_mean_tokens": true,
+    "pooling_mode_max_tokens": false,
+    "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -1,3 +1,113 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- bigcode/the-stack-v2
+- tiiuae/falcon-refinedweb 
+
+library_name: transformers
+language:
+- code
+- en
+---
+
+## SageLite-l
+
+### Model Description
+SageLite is a new family of open embedding models with an encoder architecture that supports a wide range of tasks in both code and text. SageLite went through three stages of training:
+1. **MLM Pretraining**: Standard masked language model (MLM) pretraining on mixed code and text data ([The-Stack-v2](https://huggingface.co/datasets/bigcode/the-stack-v2) and [Falcon-refinedweb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)).
+2. **Contrastive Pre-Finetuning**: Learning from a large amount of positive pairs mined from web data and GitHub.
+3. **Contrastive Fine-Tuning**: Fine-tuning on a small amount of synthetic data.
+
+---
+
+### **Code Retrieval Performance**
+
+#### 1. Code2Code Search
+
+| Model Name          | # Params | Embd Dim | Python | Java  | JS    | TS     | C#     | C      | Ruby   | PhP    | GO     | AVG    |
+|---------------------|----------|----------|--------|-------|-------|--------|--------|--------|--------|--------|--------|--------|
+| OpenAI-Code-01      | NA       | 3072     | 21.92  | 8.90  | 4.90  | 5.70   | 3.15   | 11.58  | 26.25  | 16.60  | 9.40   | 12.04  |
+| OpenAI-Text-3-Small | NA       | 1536     | 25.18  | 12.61 | 8.00  | 9.44   | 5.46   | 15.86  | 30.70  | 23.33  | 11.20  | 15.57  |
+| OpenAI-Text-3-Large | NA       | 3072     | 40.57  | 25.33 | 20.09 | 22.00  | 11.84  | 31.90  | 42.54  | 41.84  | 21.75  | 28.65  |
+| CodeSage-v2-Small   | 130M     | 1024     | 45.60  | 33.65 | 39.96 | 47.78  | 19.19  | 30.55  | 40.12  | 55.39  | 30.96  | 38.13  |
+| CodeSage-v2-Base    | 356M     | 1024     | 55.86  | 42.89 | 45.29 | 54.58  | 23.90  | 38.52  | 56.02  | 64.56  | 42.88  | 47.17  |
+| CodeSage-v2-Large   | 1.3B     | 2048     | 61.11  | 47.09 | 51.18 | 60.67  | 28.04  | 43.40  | 60.74  | 67.87  | 43.86  | 51.55  |
+| SageLite-s          | 80M      | 768      | 47.93  | 30.83 | 35.15 | 37.64  | 18.14  | 30.53  | 42.89  | 50.70  | 21.69  | 35.06  |
+| SageLite-l          | 850M     | 1536     | 64.46  | 45.53 | 50.80 | 54.71  | 30.66  | 47.46  | 61.01  | 68.68  | 39.25  | 51.40  |
+
+#### 2. NL2Code Search
+
+| Model Name          | # Params | CoSQA | AdvTest | Python | Java  | JS    | PhP    | GO     | Ruby   | Avg    |
+|---------------------|----------|-------|---------|--------|-------|-------|--------|--------|--------|--------|
+| OpenAI-Code-01      | NA       | 52.20 | 36.03   | 63.13  | 67.85 | 62.30 | 57.47  | 85.22  | 69.28  | 61.69  |
+| OpenAI-Text-3-Small | NA       | 52.48 | 34.10   | 62.62  | 65.87 | 60.28 | 54.85  | 81.96  | 67.57  | 59.97  |
+| OpenAI-Text-3-Large | NA       | 55.21 | 46.83   | 70.81  | 72.89 | 68.12 | 59.58  | 87.60  | 75.22  | 67.03  |
+| CodeSage-v2-Small   | 130M     | 52.39 | 47.28   | 68.79  | 68.13 | 65.77 | 60.20  | 80.26  | 72.46  | 64.41  |
+| CodeSage-v2-Base    | 356M     | 50.74 | 52.00   | 70.46  | 70.89 | 69.61 | 62.81  | 82.37  | 73.71  | 66.57  |
+| CodeSage-v2-Large   | 1.3B     | 53.18 | 56.31   | 74.18  | 72.33 | 72.49 | 65.26  | 84.67  | 76.61  | 69.38  |
+| SageLite-s          | 80M      | 56.49 | 42.32   | 67.59  | 66.62 | 62.32 | 58.87  | 79.36  | 70.75  | 63.04  |
+| SageLite-l          | 850M     | 59.76 | 55.55   | 74.25  | 71.76 | 69.35 | 61.62  | 84.09  | 77.14  | 69.19  |
+
+---
+
+### **Text Retrieval Performance ([MTEB Retrieval](https://huggingface.co/spaces/mteb/leaderboard))**
+
+| Metric                        | SageLite-s | SageLite-l |
+|-------------------------------|------------|------------|
+| ArguAna                       | 57.75      | 60.706     |
+| CQADupstackWordpressRetrieval | 32.42      | 38.625     |
+| FiQA2018                      | 34.85      | 46.729     |
+| NFCorpus                      | 29.97      | 33.698     |
+| QuoraRetrieval                | 85.35      | 87.497     |
+| SCIDOCS                       | 18.99      | 21.379     |
+| SciFact                       | 68.43      | 69.050     |
+| Touche2020                    | 24.41      | 21.425     |
+| TRECCOVID                     | 70.88      | 76.078     |
+| FEVER                         | 71.72      | 73.644     |
+| HotpotQA                      | 58.81      | 62.955     |
+| NQ                            | 48.26      | 54.478     |
+| DBPedia                       | 34.83      | 40.689     |
+| ClimateFEVER                  | 25.69      | 26.198     |
+| MSMARCO                       | 35.01      | 36.546     |
+| average                       | 46.49      | 49.980     |
+
+---
+
+### **Training Data**
+This checkpoint is trained on both [The-Stack-v2](https://huggingface.co/datasets/bigcode/the-stack-v2) and [Falcon-refinedweb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb). Supported languages (15 in total) are: English, C, C#, Go, Java, JavaScript, TypeScript, PHP, Python, and Ruby.
+
+---
+
+### **Training Procedure**
+This checkpoint was trained using the following procedure:
+1. **MLM Pretraining**: Masked language modeling on code data.
+2. **Contrastive Pre-Finetuning**: Using large-scale positive pairs mined from web and GitHub data.
+3. **Contrastive Fine-Tuning**: Using a small amount of synthetic data.
+
+---
+
+### **How to Use**
+This checkpoint consists of an encoder (850M model) that extracts code embeddings of 768 dimensions. It can be loaded using the Hugging Face Transformers library and employs the [Starcoder Tokenizer](https://arxiv.org/pdf/2305.06161.pdf).
+
+#### Pre-requisite
+Please install OpenAI tiktoken for the tokenizer.
+
+```
+pip install tiktoken>=0.4.0 
+```
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+# Specify the checkpoint
+checkpoint = "SageLite/SageLite-l"
+device = "cuda"  # Use "cpu" if GPU is unavailable
+
+# Load tokenizer and model
+tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True, add_eos_token=True)
+model = AutoModel.from_pretrained(checkpoint, trust_remote_code=True).to(device)
+
+# Example usage
+code_snippet = "def print_hello_world():\tprint('Hello World!')"
+inputs = tokenizer.encode(code_snippet, return_tensors="pt").to(device)
+embedding = model(inputs)[0]  # Extract the embedding

config.json

@@ -0,0 +1,26 @@
+{
+    "_name_or_path": "SageLite/SageLite-l",
+    "architectures": [
+        "SageLite"
+    ],
+    "auto_map": {
+        "AutoConfig": "config_sagelite.SageLiteConfig",
+        "AutoTokenizer": "tokenization_sagelite.SageLiteTokenizer",
+        "AutoModel": "modeling_sagelite.SageLiteModel",
+        "AutoModelForMaskedLM": "modeling_sagelite.SageLiteForMaskedLM",
+        "AutoModelForSequenceClassification": "modeling_sagelite.SageLiteForSequenceClassification"
+    },
+    "activation_function": "gelu_new",
+    "attention_dropout_prob": 0.1,
+    "embedding_dropout_prob": 0.1,
+    "initializer_range": 0.02,
+    "layer_norm_epsilon": 1e-05,
+    "hidden_size": 1536,
+    "num_attention_heads": 12,
+    "num_hidden_layers": 24,
+    "intermediate_size": 6144,
+    "max_position_embeddings": 2048,
+    "residual_dropout_prob": 0.1,
+    "vocab_size": 100318
+}
+

config_codext.py

@@ -0,0 +1,51 @@
+#!/usr/bin/env python
+# coding=utf-8
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+from transformers.configuration_utils import PretrainedConfig
+
+CODESAGE_PRETRAINED_CONFIG_ARCHIVE_MAP = {
+    "SageLite/SageLite-s": "https://huggingface.co/SageLite/SageLite-s/resolve/main/config.json",
+    "SageLite/SageLite-l": "https://huggingface.co/SageLite/SageLite-l/resolve/main/config.json",
+}
+
+
+class SageLiteConfig(PretrainedConfig):
+    model_type = "SageLite"
+
+    def __init__(
+            self,
+            vocab_size=100318,
+            max_position_embeddings=2048,
+            hidden_size=1536,
+            num_hidden_layers=24,
+            num_attention_heads=12,
+            intermediate_size=6144,
+            activation_function="gelu_new",
+            residual_dropout_prob=0.1,
+            embedding_dropout_prob=0.1,
+            attention_dropout_prob=0.1,
+            layer_norm_epsilon=1e-5,
+            initializer_range=0.02,
+            position_embedding_type='absolute',
+            bos_token_id=100257,
+            eos_token_id=100257,
+            pad_token_id=100317,
+            **kwargs
+    ):
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.intermediate_size = intermediate_size
+        assert 'gelu' in activation_function
+        self.activation_function = activation_function
+        self.residual_dropout_prob = residual_dropout_prob
+        self.embedding_dropout_prob = embedding_dropout_prob
+        self.attention_dropout_prob = attention_dropout_prob
+        self.layer_norm_epsilon = layer_norm_epsilon
+        self.initializer_range = initializer_range
+        self.position_embedding_type = position_embedding_type
+
+        super().__init__(pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs)

modeling_codext.py

@@ -0,0 +1,425 @@
+#!/usr/bin/env python
+# coding=utf-8
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+import math
+import torch
+import torch.utils.checkpoint
+from torch import nn
+from torch.nn import CrossEntropyLoss, MSELoss, BCEWithLogitsLoss
+from transformers.activations import ACT2FN
+from transformers.modeling_utils import Conv1D, PreTrainedModel
+from transformers.utils import logging
+from .config_sagelite import SageLiteConfig
+from transformers.modeling_outputs import (
+    BaseModelOutputWithPooling,
+    MaskedLMOutput,
+    SequenceClassifierOutput
+)
+
+logger = logging.get_logger(__name__)
+
+SAGELITE_PRETRAINED_MODEL_ARCHIVE_LIST = [
+    "SageLite/SageLite-s",
+    "SageLite/SageLite-l",
+    # See all SageLite models at https://huggingface.co/models?filter=SageLite
+]
+
+
+class SageLiteAttention(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+
+        self.hidden_size = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.head_dim = config.hidden_size // self.num_heads
+        if self.head_dim * self.num_heads != config.hidden_size:
+            raise ValueError(
+                f"`hidden_size` must be divisible by num_heads "
+                f"(got `hidden_size`: {config.hidden_size} and `num_heads`: {self.num_heads})."
+            )
+
+        self.c_attn = Conv1D(3 * self.hidden_size, self.hidden_size)
+        self.c_proj = Conv1D(self.hidden_size, self.hidden_size)
+
+        self.attention_dropout = nn.Dropout(config.attention_dropout_prob)
+        self.residual_dropout = nn.Dropout(config.residual_dropout_prob)
+
+    def attn(self, query, key, value, attention_mask=None, head_mask=None):
+        attn_weights = torch.matmul(query, key.transpose(-1, -2))
+        attn_weights = attn_weights / math.sqrt(self.head_dim)
+        if attention_mask is not None:
+            attn_weights = attn_weights + attention_mask
+
+        attn_weights = nn.Softmax(dim=-1)(attn_weights)
+        attn_weights = self.attention_dropout(attn_weights)
+        if head_mask is not None:
+            attn_weights = attn_weights * head_mask
+
+        attn_output = torch.matmul(attn_weights, value)
+        return attn_output, attn_weights
+
+    def split_heads(self, tensor, num_heads, attn_head_size):
+        """
+        Splits hidden_size dim into attn_head_size and num_heads
+        """
+        new_shape = tensor.size()[:-1] + (num_heads, attn_head_size)
+        tensor = tensor.view(*new_shape)
+        return tensor.permute(0, 2, 1, 3)  # (batch, head, seq_length, head_features)
+
+    def merge_heads(self, tensor, num_heads, attn_head_size):
+        """
+        Merges attn_head_size dim and num_attn_heads dim into hidden_size
+        """
+        tensor = tensor.permute(0, 2, 1, 3).contiguous()
+        new_shape = tensor.size()[:-2] + (num_heads * attn_head_size,)
+        return tensor.view(new_shape)
+
+    def forward(
+            self,
+            hidden_states,
+            attention_mask=None,
+            head_mask=None,
+            output_attentions=False,
+    ):
+        query, key, value = self.c_attn(hidden_states).split(self.hidden_size, dim=2)
+        query = self.split_heads(query, self.num_heads, self.head_dim)
+        key = self.split_heads(key, self.num_heads, self.head_dim)
+        value = self.split_heads(value, self.num_heads, self.head_dim)
+
+        attn_output, attn_weights = self.attn(query, key, value, attention_mask, head_mask)
+
+        attn_output = self.merge_heads(attn_output, self.num_heads, self.head_dim)
+        attn_output = self.c_proj(attn_output)
+        attn_output = self.residual_dropout(attn_output)
+
+        outputs = (attn_output, attn_weights) if output_attentions else (attn_output,)
+        return outputs  # a, present, (attentions)
+
+
+class SageLiteMLP(nn.Module):
+    def __init__(self, intermediate_size, config):
+        super().__init__()
+
+        self.c_fc = Conv1D(intermediate_size, config.hidden_size)
+        self.act = ACT2FN[config.activation_function]
+        self.c_proj = Conv1D(config.hidden_size, intermediate_size)
+        self.dropout = nn.Dropout(config.residual_dropout_prob)
+
+    def forward(self, hidden_states):
+        hidden_states = self.c_fc(hidden_states)
+        hidden_states = self.act(hidden_states)
+        hidden_states = self.c_proj(hidden_states)
+        hidden_states = self.dropout(hidden_states)
+        return hidden_states
+
+
+class SageLiteBlock(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        hidden_size = config.hidden_size
+        inner_dim = config.intermediate_size if config.intermediate_size is not None else 4 * hidden_size
+        self.ln_1 = nn.LayerNorm(hidden_size, eps=config.layer_norm_epsilon)
+        self.attn = SageLiteAttention(config)
+        self.ln_2 = nn.LayerNorm(hidden_size, eps=config.layer_norm_epsilon)
+        self.mlp = SageLiteMLP(inner_dim, config)
+
+    def forward(
+            self,
+            hidden_states,
+            attention_mask=None,
+            head_mask=None,
+            output_attentions=False,
+    ):
+        residual = hidden_states
+        hidden_states = self.ln_1(hidden_states)
+        attn_outputs = self.attn(
+            hidden_states,
+            attention_mask=attention_mask,
+            head_mask=head_mask,
+            output_attentions=output_attentions
+        )
+        attn_output = attn_outputs[0]  # output_attn: a, present, (attentions)
+        outputs = attn_outputs[1:]
+        hidden_states = attn_output + residual
+
+        residual = hidden_states
+        hidden_states = self.ln_2(hidden_states)
+        feed_forward_hidden_states = self.mlp(hidden_states)
+        hidden_states = residual + feed_forward_hidden_states
+
+        outputs = (hidden_states,) + outputs[1:]
+        return outputs  # hidden_states, present, (attentions)
+
+
+class SageLitePreTrainedModel(PreTrainedModel):
+    config_class = SageLiteConfig
+    base_model_prefix = "transformer"
+
+    def _init_weights(self, module):
+        """Initialize the weights."""
+        if isinstance(module, (nn.Linear, Conv1D)):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+        elif isinstance(module, nn.LayerNorm):
+            module.bias.data.zero_()
+            module.weight.data.fill_(1.0)
+
+
+class SageLiteModel(SageLitePreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+
+        self.wte = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.wpe = nn.Embedding(config.max_position_embeddings, config.hidden_size)
+
+        self.drop = nn.Dropout(config.embedding_dropout_prob)
+        self.h = nn.ModuleList([SageLiteBlock(config) for _ in range(config.num_hidden_layers)])
+        self.ln_f = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_epsilon)
+
+        self.init_weights()
+
+    def get_input_embeddings(self):
+        return self.wte
+
+    def set_input_embeddings(self, new_embeddings: torch.Tensor):
+        self.wte = new_embeddings
+
+    def forward(
+            self,
+            input_ids=None,
+            attention_mask=None,
+            position_ids=None,
+            head_mask=None,
+            inputs_embeds=None,
+            output_attentions=None,
+            output_hidden_states=None,
+            return_dict=None
+    ):
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
+        if input_ids is not None:
+            input_shape = input_ids.size()
+        elif inputs_embeds is not None:
+            input_shape = inputs_embeds.size()[:-1]
+        else:
+            raise ValueError("You have to specify either input_ids or inputs_embeds")
+
+        device = input_ids.device if input_ids is not None else inputs_embeds.device
+        if position_ids is None:
+            position_ids = torch.arange(input_shape[-1], dtype=torch.long, device=device)
+            position_ids = position_ids.unsqueeze(0).view(-1, input_shape[-1])
+        else:
+            position_ids = position_ids.view(-1, input_shape[-1])
+
+        extended_attention_mask = None
+        if attention_mask is not None:
+            assert attention_mask.dim() == 2
+            extended_attention_mask = attention_mask[:, None, None, :]
+            extended_attention_mask = extended_attention_mask.to(dtype=self.dtype)  # fp16 compatibility
+            extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
+
+        head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
+        if inputs_embeds is None:
+            inputs_embeds = self.wte(input_ids)
+
+        position_embeds = self.wpe(position_ids)
+        hidden_states = inputs_embeds + position_embeds
+
+        hidden_states = self.drop(hidden_states)
+        output_shape = input_shape + (hidden_states.size(-1),)
+
+        all_self_attentions = () if output_attentions else None
+        all_hidden_states = () if output_hidden_states else None
+        for i, block in enumerate(self.h):
+            if output_hidden_states:
+                all_hidden_states = all_hidden_states + (hidden_states,)
+
+            outputs = block(
+                hidden_states,
+                attention_mask=extended_attention_mask,
+                head_mask=head_mask[i],
+                output_attentions=output_attentions,
+            )
+
+            hidden_states = outputs[0]
+            if output_attentions:
+                all_self_attentions = all_self_attentions + (outputs[1],)
+
+        hidden_states = self.ln_f(hidden_states)
+        hidden_states = hidden_states.view(*output_shape)
+        if output_hidden_states:
+            all_hidden_states = all_hidden_states + (hidden_states,)
+
+        pooled_output = None  # max-pooled output
+        if attention_mask is not None:
+            pooled_output = (hidden_states * attention_mask[:, :, None]).sum(1) / attention_mask.sum(1)[:, None]
+
+        if not return_dict:
+            return tuple(
+                v
+                for v in [hidden_states, pooled_output, all_hidden_states, all_self_attentions]
+                if v is not None
+            )
+
+        return BaseModelOutputWithPooling(
+            last_hidden_state=hidden_states,
+            pooler_output=pooled_output,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attentions
+        )
+
+
+class SageLiteForMaskedLM(SageLitePreTrainedModel):
+    _tied_weights_keys = ["lm_head.weight"]
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.transformer = SageLiteModel(config)
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        self.init_weights()
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def forward(
+            self,
+            input_ids=None,
+            attention_mask=None,
+            position_ids=None,
+            head_mask=None,
+            inputs_embeds=None,
+            labels=None,
+            output_attentions=None,
+            output_hidden_states=None,
+            return_dict=None
+    ):
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        transformer_outputs = self.transformer(
+            input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict
+        )
+        hidden_states = transformer_outputs[0]
+        lm_logits = self.lm_head(hidden_states)
+
+        masked_lm_loss = None
+        if labels is not None:
+            loss_fct = CrossEntropyLoss()
+            masked_lm_loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1))
+
+        if not return_dict:
+            output = (lm_logits,) + transformer_outputs[1:]
+            return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output
+
+        return MaskedLMOutput(
+            loss=masked_lm_loss,
+            logits=lm_logits,
+            hidden_states=transformer_outputs.hidden_states,
+            attentions=transformer_outputs.attentions,
+        )
+
+
+class SageLiteForSequenceClassification(SageLitePreTrainedModel):
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+        self.config = config
+
+        self.transformer = SageLiteModel(config)
+        classifier_dropout = (
+            config.classifier_dropout 
+            if hasattr(config, 'classifier_dropout') and config.classifier_dropout is not None 
+            else config.residual_dropout_prob
+        )
+        self.dropout = nn.Dropout(classifier_dropout)
+        self.classifier = nn.Linear(config.hidden_size, config.num_labels)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def forward(
+            self,
+            input_ids=None,
+            attention_mask=None,
+            position_ids=None,
+            head_mask=None,
+            inputs_embeds=None,
+            labels=None,
+            output_attentions=None,
+            output_hidden_states=None,
+            return_dict=None,
+    ):
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+        assert attention_mask is not None, "attention_mask is needed to perform max-pooling"
+
+        outputs = self.transformer(
+            input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        pooled_output = outputs[1]
+        pooled_output = self.dropout(pooled_output)
+        logits = self.classifier(pooled_output)
+
+        loss = None
+        if labels is not None:
+            if self.config.problem_type is None:
+                if self.num_labels == 1:
+                    self.config.problem_type = "regression"
+                elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
+                    self.config.problem_type = "single_label_classification"
+                else:
+                    self.config.problem_type = "multi_label_classification"
+
+            if self.config.problem_type == "regression":
+                loss_fct = MSELoss()
+                if self.num_labels == 1:
+                    loss = loss_fct(logits.squeeze(), labels.squeeze())
+                else:
+                    loss = loss_fct(logits, labels)
+            elif self.config.problem_type == "single_label_classification":
+                loss_fct = CrossEntropyLoss()
+                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
+            elif self.config.problem_type == "multi_label_classification":
+                loss_fct = BCEWithLogitsLoss()
+                loss = loss_fct(logits, labels)
+
+        if not return_dict:
+            output = (logits,) + outputs[2:]
+            return ((loss,) + output) if loss is not None else output
+
+        return SequenceClassifierOutput(
+            loss=loss,
+            logits=logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )

modules.json

@@ -0,0 +1,20 @@
+[
+    {
+      "idx": 0,
+      "name": "0",
+      "path": "",
+      "type": "sentence_transformers.models.Transformer"
+    },
+    {
+      "idx": 1,
+      "name": "1",
+      "path": "1_Pooling",
+      "type": "sentence_transformers.models.Pooling"
+    },
+    {
+      "idx": 2,
+      "name": "2",
+      "path": "2_Normalize",
+      "type": "sentence_transformers.models.Normalize"
+    }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66d18ff9e119be7ed2c0f39d041e4ff06744b371a88b33032f070dc0d06f0ed9
+size 1674472633

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 1024,
+  "do_lower_case": false
+}

tokenization_codext.py

@@ -0,0 +1,337 @@
+from functools import lru_cache
+from typing import Any, Dict, List, Optional, Tuple
+
+from transformers import PreTrainedTokenizer
+import tiktoken
+
+
+# Taken from
+# https://github.com/huggingface/transformers/blob/8aca43bdb3cb9a5020f6d57589d85679dc873b1c/src/transformers/models/gpt2/tokenization_gpt2.py#L62-L84
+@lru_cache()
+def bytes_to_unicode():
+    """Returns list of utf-8 byte and a mapping to unicode strings.
+    We specifically avoids mapping to whitespace/control characters the bpe code
+    barfs on.
+    The reversible bpe codes work on unicode strings. This means you need a
+    large # of unicode characters in your vocab if you want to avoid UNKs. When
+    you're at something like a 10B token dataset you end up needing around 5K
+    for decent coverage. This is a significant percentage of your normal, say,
+    32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and
+    unicode strings.
+    """
+    bs = (list(range(ord('!'),
+                     ord('~') + 1)) + list(range(ord('¡'),
+                                                 ord('¬') + 1)) +
+          list(range(ord('®'),
+                     ord('ÿ') + 1)))
+    cs = bs[:]
+    n = 0
+    for b in range(2**8):
+        if b not in bs:
+            bs.append(b)
+            cs.append(2**8 + n)
+            n += 1
+    cs = [chr(n) for n in cs]
+    return dict(zip(bs, cs))
+
+
+def add_special_tokens_to_tiktoken(base="cl100k_base", eos_token=None, pad_token=None):
+    def include_dobf_tokens():
+        dobf_tokens = [f"<dobf_special_{i}>" for i in range(18)]
+        return dobf_tokens
+
+    def include_vector_tokens():
+        tokens = []
+        tokens.append("<sep>")
+        tokens.append("<mask>")
+        tokens += [f"<dummy_{i}>" for i in reversed(range(20))]
+        return tokens
+
+    dobf_tokens = include_dobf_tokens()
+    vector_tokens = include_vector_tokens()
+
+    tokenizer = tiktoken.get_encoding(base)
+    idx = tokenizer.n_vocab
+    bpe_ranks = tokenizer._mergeable_ranks
+    special_tokens = dict()
+
+    # print(f"INIT TOKEN SIZE: {idx}, EOS TOKEN: {tokenizer._special_tokens[eos_token]}")
+    if eos_token and eos_token not in tokenizer._special_tokens and eos_token not in special_tokens:
+        special_tokens[eos_token] = idx
+        idx += 1
+
+    for sp in dobf_tokens:
+        special_tokens[sp] = idx
+        idx += 1
+    for sp in vector_tokens:
+        special_tokens[sp] = idx
+        idx += 1
+
+    if pad_token and pad_token not in tokenizer._special_tokens and pad_token not in special_tokens:
+        special_tokens[pad_token] = idx
+        idx += 1
+        # print(f"PAD TOKEN ADDED: {pad_token}")
+    # In production, load the arguments directly instead of accessing private attributes
+    # See openai_public.py for examples of arguments for specific encodings
+    enc = tiktoken.Encoding(
+        # If you're changing the set of special tokens, make sure to use a different name
+        # It should be clear from the name what behaviour to expect.
+        name=base.replace("base", "im"),
+        pat_str=tokenizer._pat_str,
+        mergeable_ranks=bpe_ranks,
+        special_tokens={
+            **tokenizer._special_tokens,
+            **special_tokens
+        }
+    )
+    return enc
+
+
+class SageLiteTokenizer(PreTrainedTokenizer):
+    """A thin wrapper around tiktoken to make it compatible with Hugging Face.
+    tokenizers.
+    See HuggingFace for further documentation on general tokenizer methods.
+    """
+
+    model_input_names = ['input_ids', 'attention_mask']
+
+    def __init__(self,
+                 model_name: Optional[str] = None,
+                 encoding_name: Optional[str] = "cl100k_base",
+                 add_bos_token: bool = False,
+                 add_eos_token: bool = False,
+                 unk_token: Optional[str] = '<|endoftext|>',
+                 eos_token: Optional[str] = '<|endoftext|>',
+                 bos_token: Optional[str] = '<|endoftext|>',
+                 pad_token: Optional[str] = '<pad>',
+                 errors: str = 'replace',
+                 **kwargs: Any):
+        """Constructor creates a tiktoken tokenizer to use as the underlying.
+        tokenizer.
+        Args:
+            model_name (Optional[str], optional): The name of the model to load from tiktoken. Defaults to None.
+                Either model_name or encoding_name must be set, but not both.
+            encoding_name (Optional[str], optional): The name of the encoding to load from tiktoken. Defaults to None.
+                Either model_name or encoding_name must be set, but not both.
+            add_bos_token (bool, optional): Whether to add bos tokens. Defaults to False.
+            add_eos_token (bool, optional): Whether to add eos tokens. Defaults to False.
+            use_default_system_prompt (bool, optional): Use the default system prompt or not. Defaults to False.
+            unk_token (Optional[str], optional): The unk token. Defaults to '<|endoftext|>'.
+            eos_token (Optional[str], optional): The eos token. Defaults to '<|endoftext|>'.
+            bos_token (Optional[str], optional): The bos token. Defaults to '<|endoftext|>'.
+            pad_token (Optional[str], optional): The pad token. Defaults to None.
+            errors (str, optional): Paradigm to follow when decoding bytes to UTF-8. See
+                [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.
+                Defaults to `"replace"`.
+        """
+        try:
+            import tiktoken
+        except:
+            raise ImportError(
+                'You need to install tiktoken to use TiktokenTokenizerWrapper.')
+
+        # Workaround to make tiktokenizer picklable.
+        # https://github.com/huggingface/datasets/issues/5536#issuecomment-1682309347
+        # There is an open PR from HF to add this to tiktoken: https://github.com/openai/tiktoken/pull/181
+        import copyreg
+        import functools
+
+        from tiktoken import Encoding  # type: ignore (thirdParty)
+
+        def pickle_Encoding(enc: Encoding):
+            return (functools.partial(Encoding,
+                                      enc.name,
+                                      pat_str=enc._pat_str,
+                                      mergeable_ranks=enc._mergeable_ranks,
+                                      special_tokens=enc._special_tokens), ())
+
+        copyreg.pickle(Encoding, pickle_Encoding)
+
+
+        self.encoding = add_special_tokens_to_tiktoken(base=encoding_name, eos_token=eos_token, pad_token=pad_token)
+
+        self.add_bos_token = add_bos_token
+        self.add_eos_token = add_eos_token
+
+        self.byte_encoder = bytes_to_unicode()
+        self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
+        self.errors = errors
+
+        self.decoder: Dict[int, str] = {}
+        for i in range(self.encoding.n_vocab):
+            try:
+                self.encoding.decode_single_token_bytes(i)
+            except KeyError:
+                continue
+            # Taken from
+            # https://gist.github.com/xenova/a452a6474428de0182b17605a98631ee
+            decoding = ''.join([
+                bytes_to_unicode()[ord(char)] for char in
+                self.encoding.decode_single_token_bytes(i).decode('latin-1')
+            ])
+            self.decoder[i] = decoding
+
+        self.encoder: Dict[str, int] = {}
+        for i in range(self.encoding.n_vocab):
+            if i in self.decoder:
+                self.encoder[self.decoder[i]] = i
+
+        super().__init__(model_name=model_name,
+                         encoding_name=encoding_name,
+                         add_bos_token=add_bos_token,
+                         add_eos_token=add_eos_token,
+                         unk_token=unk_token,
+                         eos_token=eos_token,
+                         bos_token=bos_token,
+                         pad_token=pad_token,
+                         errors=errors,
+                         **kwargs)
+
+    @property
+    def vocab_size(self) -> int:
+        """Returns vocab size."""
+        return self.encoding.n_vocab
+
+    @property
+    def is_fast(self) -> bool:
+        return False
+
+    def get_vocab(self) -> Dict[str, int]:
+        """Returns vocab as a dict."""
+        # As far as I can tell, we don't require get_vocab to completely work,
+        # but when using additional_special_tokens, Hugging Face determines the next
+        # token index to add with len(self.get_vocab()) so we need the _size_ of this dictionary to be correct.
+        vocab_clone = self.encoder.copy()
+        extra_id_index = 0
+        candidate_extra_id = f'<extra_id_{extra_id_index}>'
+        indices_to_fill_in = {i for i in range(self.vocab_size)} - set(
+            vocab_clone.values())
+
+        # Add enough indices to make get_vocab() the right length
+        for index_to_add in indices_to_fill_in:
+            # Make sure we don't overwrite a token that already exists
+            while candidate_extra_id in vocab_clone:
+                extra_id_index += 1
+                candidate_extra_id = f'<extra_id_{extra_id_index}>'
+
+            # Get an index to add and add the item
+            vocab_clone[candidate_extra_id] = index_to_add
+
+        return vocab_clone
+
+    def _tokenize(self, text: str) -> List[str]:
+        """Returns a tokenized string."""
+        if not isinstance(text, str):
+            raise ValueError(
+                f'Expected a string input to _tokenize but got {type(text)}.')
+
+        tokens = [
+            self.decoder[t]
+            for t in self.encoding.encode(text, allowed_special='all')
+        ]
+
+        return tokens
+
+    def _convert_token_to_id(self, token: str) -> Optional[int]:
+        """Converts a token (str) in an id using the vocab."""
+        return self.encoder.get(token, self.encoder.get(self.unk_token))
+
+    def _convert_id_to_token(self, index: int) -> Optional[str]:
+        """Converts an index (integer) in a token (str) using the vocab."""
+        # For tokens in either the gap in ids in the tokenizer, or beyond the range of the tokenizer,
+        # we return empty string. This matches the behavior of Hugging Face fast tokenizers,
+        # but not slow tokenizers.
+        return self.decoder.get(index, '')
+
+    def convert_tokens_to_string(self, tokens: List[str]) -> str:
+        """Converts a sequence of tokens (string) in a single string."""
+        text = ''.join(tokens)
+        text = bytearray([self.byte_decoder[c] for c in text
+                         ]).decode('utf-8', errors=self.errors)
+        return text
+
+    def build_inputs_with_special_tokens(
+            self,
+            token_ids_0: List[int],
+            token_ids_1: Optional[List[int]] = None) -> List[int]:
+        bos_token_id = [self.bos_token_id] if self.add_bos_token else []
+        eos_token_id = [self.eos_token_id] if self.add_eos_token else []
+
+        output = bos_token_id + token_ids_0 + eos_token_id
+
+        if token_ids_1 is not None:
+            output = output + bos_token_id + token_ids_1 + eos_token_id
+
+        return output
+
+    def get_special_tokens_mask(
+            self,
+            token_ids_0: List[int],
+            token_ids_1: Optional[List[int]] = None,
+            already_has_special_tokens: bool = False) -> List[int]:
+        """Retrieves sequence ids from a token list that has no special tokens.
+        Function copied from
+        https://github.com/huggingface/transformers/blob/e3a4bd2bee212a2d0fd9f03b27fe7bfc1debe42d/src/transformers/models/gpt2/tokenization_gpt2.py#L265-L295
+        added. This method is called when adding special tokens using the
+        tokenizer `prepare_for_model` or `encode_plus` methods.
+        Args:
+            token_ids_0 (`List[int]`):
+                List of IDs.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+            already_has_special_tokens (`bool`, *optional*, defaults to `False`):
+                Whether or not the token list is already formatted with special tokens for the model.
+        Returns:
+            `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
+        """
+        if already_has_special_tokens:
+            return super().get_special_tokens_mask(
+                token_ids_0=token_ids_0,
+                token_ids_1=token_ids_1,
+                already_has_special_tokens=True)
+
+        bos_token_id = [1] if self.add_bos_token else []
+        eos_token_id = [1] if self.add_eos_token else []
+
+        if token_ids_1 is None:
+            return bos_token_id + ([0] * len(token_ids_0)) + eos_token_id
+        return (bos_token_id + ([0] * len(token_ids_0)) + eos_token_id +
+                bos_token_id + ([0] * len(token_ids_1)) + eos_token_id)
+
+    def create_token_type_ids_from_sequences(
+            self,
+            token_ids_0: List[int],
+            token_ids_1: Optional[List[int]] = None) -> List[int]:
+        sep = [self.sep_token_id]
+
+        if token_ids_1 is None:
+            return len(token_ids_0 + sep) * [0]
+        return len(token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1]
+
+    def save_vocabulary(self,
+                        save_directory: str,
+                        filename_prefix: Optional[str] = None) -> Tuple[str]:
+
+        # ignore the below type to keep the original signature
+        # we are knowingly breaking the signature here, although not 100% certain
+        # it doesn't have side effects
+        # There is some code in huggingface that calls this function to get the vocab files,
+        # but it doesn't seem to access them (or at least checks for their existence
+        # before accessing them)
+        return (None, None)  # type: ignore
+
+    def sanitize_special_tokens(self) -> int:
+        """Make sure that all the special tokens attributes of the tokenizer.
+        (`tokenizer.mask_token`, `tokenizer.cls_token`, etc.) are in the
+        vocabulary.
+        Add the missing ones to the vocabulary if needed.
+        Return:
+            `int`: The number of tokens added in the vocabulary during the operation.
+        """
+        actual_new_tokens = []
+        for token in self.all_special_tokens_extended:
+            encoded = self.encoding.encode(token, allowed_special='all')
+            if len(encoded) > 1:
+                actual_new_tokens.append(token)
+
+        return self.add_tokens(actual_new_tokens, special_tokens=True)

tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "add_eos_token": true,
+  "add_special_tokens": true,
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "SageLiteTokenizer",
+  "auto_map": {
+    "AutoTokenizer": ["tokenization_sagelite.SageLiteTokenizer", null]
+  }
+}