Upload folder using huggingface_hub

FOREcasT_model/README.md

@@ -0,0 +1,88 @@
+---
+library_name: transformers
+tags:
+- generated_from_trainer
+datasets:
+- crispr_data
+model-index:
+- name: SX_ispymac_FOREcasT
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# SX_ispymac_FOREcasT
+
+This model is a fine-tuned version of [](https://huggingface.co/) on the crispr_data dataset.
+It achieves the following results on the evaluation set:
+- Loss: 50.9107
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.001
+- train_batch_size: 100
+- eval_batch_size: 100
+- seed: 63036
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.05
+- num_epochs: 30.0
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 5348.4985     | 1.0   | 326  | 4421.2725       |
+| 3199.7485     | 2.0   | 652  | 2104.5654       |
+| 1490.7015     | 3.0   | 978  | 1002.8325       |
+| 713.0995      | 4.0   | 1304 | 480.3265        |
+| 344.2885      | 5.0   | 1630 | 238.0577        |
+| 178.0734      | 6.0   | 1956 | 131.9894        |
+| 106.8135      | 7.0   | 2282 | 87.4150         |
+| 76.4049       | 8.0   | 2608 | 67.8651         |
+| 62.8691       | 9.0   | 2934 | 58.8372         |
+| 56.6853       | 10.0  | 3260 | 54.8777         |
+| 53.8423       | 11.0  | 3586 | 53.1428         |
+| 52.4774       | 12.0  | 3912 | 51.9998         |
+| 51.8337       | 13.0  | 4238 | 51.5273         |
+| 51.4675       | 14.0  | 4564 | 51.2515         |
+| 51.3089       | 15.0  | 4890 | 51.1025         |
+| 51.2186       | 16.0  | 5216 | 51.0217         |
+| 51.1556       | 17.0  | 5542 | 51.0496         |
+| 51.1167       | 18.0  | 5868 | 50.9584         |
+| 51.1001       | 19.0  | 6194 | 50.9669         |
+| 51.0948       | 20.0  | 6520 | 50.9908         |
+| 51.0668       | 21.0  | 6846 | 50.9851         |
+| 51.0647       | 22.0  | 7172 | 50.9715         |
+| 51.0496       | 23.0  | 7498 | 50.9685         |
+| 51.0475       | 24.0  | 7824 | 50.9484         |
+| 51.0267       | 25.0  | 8150 | 50.9452         |
+| 51.0162       | 26.0  | 8476 | 50.9290         |
+| 51.0045       | 27.0  | 8802 | 50.9395         |
+| 50.9944       | 28.0  | 9128 | 50.9163         |
+| 50.9849       | 29.0  | 9454 | 50.9197         |
+| 50.9731       | 30.0  | 9780 | 50.9107         |
+
+
+### Framework versions
+
+- Transformers 4.44.2
+- Pytorch 2.4.0+cu124
+- Datasets 2.21.0
+- Tokenizers 0.19.1

FOREcasT_model/config.json

@@ -1,5 +1,4 @@
 {
-  "_name_or_path": "/home/ljw/sdc1/CRISPR_results/FOREcasT/SX_ispymac_FOREcasT",
   "architectures": [
     "FOREcasTModel"
   ],

FOREcasT_model/model.py

@@ -0,0 +1,110 @@
+from transformers import PretrainedConfig, PreTrainedModel
+import torch.nn as nn
+import torch
+import torch.nn.functional as F
+
+class FOREcasTConfig(PretrainedConfig):
+    model_type = "FOREcasT"
+    label_names = ["count"]
+
+    def __init__(
+        self,
+        reg_const = 0.01, # regularization coefficient for insertion
+        i1_reg_const = 0.01, # regularization coefficient for deletion
+        seed = 63036, # random seed for intialization
+        **kwargs
+    ):
+        self.reg_const = reg_const
+        self.i1_reg_const = i1_reg_const
+        self.seed = seed
+        super().__init__(**kwargs)
+
+class FOREcasTModel(PreTrainedModel):
+    config_class = FOREcasTConfig
+
+    @staticmethod
+    def get_feature_label():
+        def features_pairwise_label(features1_label, features2_label):
+            features_label = []
+            for label1 in features1_label:
+                for label2 in features2_label:
+                    features_label.append(f'PW_{label1}_vs_{label2}')
+            return features_label
+        feature_DelSize_label = ["Any Deletion", "D1", "D2-3", "D4-7", "D8-12", "D>12"]
+        feature_InsSize_label = ["Any Insertion", "I1", "I2"]
+        feature_DelLoc_label = ['DL-1--1', 'DL-2--2', 'DL-3--3', 'DL-4--6', 'DL-7--10', 'DL-11--15', 'DL-16--30', 'DL<-30', 'DL>=0', 'DR0-0', 'DR1-1', 'DR2-2', 'DR3-5', 'DR6-9', 'DR10-14', 'DR15-29', 'DR<0', 'DR>=30']
+        feature_InsSeq_label = ["I1_A", "I1_C", "I1_G", "I1_T", "I2_AA", "I2_AC", "I2_AG", "I2_AT", "I2_CA", "I2_CC", "I2_CG", "I2_CT", "I2_GA", "I2_GC", "I2_GG", "I2_GT", "I2_TA", "I2_TC", "I2_TG", "I2_TT"]
+        feature_InsLoc_label = ["IL-1--1", "IL-2--2", "IL-3--3", "IL<-3", "IL>=0"]
+        feature_LocalCutSiteSequence_label = []
+        for offset in range(-5, 4):
+            for nt in ["A", "G", "C", "T"]:
+                feature_LocalCutSiteSequence_label.append(f"CS{offset}_NT={nt}")
+        feature_LocalCutSiteSeqMatches_label = []
+        for offset1 in range(-3, 2):
+            for offset2 in range(-3, offset1):
+                for nt in ["A", "G", "C", "T"]:
+                    feature_LocalCutSiteSeqMatches_label.append(f"M_CS{offset1}_{offset2}_NT={nt}")
+        feature_LocalRelativeSequence_label = []
+        for offset in range(-3, 3):
+            for nt in ["A", "G", "C", "T"]:
+                feature_LocalRelativeSequence_label.append(f'L{offset}_NT={nt}')
+        for offset in range(-3, 3):
+            for nt in ["A", "G", "C", "T"]:
+                feature_LocalRelativeSequence_label.append(f'R{offset}_NT={nt}')
+        feature_SeqMatches_label = []
+        for loffset in range(-3, 3):
+            for roffset in range(-3, 3):
+                feature_SeqMatches_label.append(f'X_L{loffset}_R{roffset}')
+                feature_SeqMatches_label.append(f'M_L{loffset}_R{roffset}')
+        feature_I1or2Rpt_label = ['I1Rpt', 'I1NonRpt', 'I2Rpt', 'I2NonRpt']
+        feature_microhomology_label = ['L_MH1-1', 'R_MH1-1', 'L_MH2-2', 'R_MH2-2', 'L_MH3-3', 'R_MH3-3', 'L_MM1_MH3-3', 'R_MM1_MH3-3', 'L_MH4-6', 'R_MH4-6', 'L_MM1_MH4-6', 'R_MM1_MH4-6', 'L_MH7-10', 'R_MH7-10', 'L_MM1_MH7-10', 'R_MM1_MH7-10', 'L_MH11-15', 'R_MH11-15', 'L_MM1_MH11-15', 'R_MM1_MH11-15', 'No MH']
+        return (
+            features_pairwise_label(feature_DelSize_label, feature_DelLoc_label) +
+            feature_InsSize_label +
+            feature_DelSize_label +
+            feature_DelLoc_label +
+            feature_InsLoc_label +
+            feature_InsSeq_label +
+            features_pairwise_label(feature_LocalCutSiteSequence_label, feature_InsSize_label + feature_DelSize_label) +
+            features_pairwise_label(feature_microhomology_label + feature_LocalRelativeSequence_label, feature_DelSize_label + feature_DelLoc_label) +
+            features_pairwise_label(feature_LocalCutSiteSeqMatches_label + feature_SeqMatches_label, feature_DelSize_label) +
+            features_pairwise_label(feature_InsSeq_label + feature_LocalCutSiteSequence_label + feature_LocalCutSiteSeqMatches_label, feature_I1or2Rpt_label) +
+            feature_I1or2Rpt_label +
+            feature_LocalCutSiteSequence_label +
+            feature_LocalCutSiteSeqMatches_label +
+            feature_LocalRelativeSequence_label +
+            feature_SeqMatches_label +
+            feature_microhomology_label
+        )
+
+    def __init__(self, config) -> None:
+        super().__init__(config)
+        # In more recent versions of PyTorch, you no longer need to explicitly register_parameter, it's enough to set a member of your nn.Module with nn.Parameter to "notify" pytorch that this variable should be treated as a trainable parameter (https://stackoverflow.com/questions/59234238/how-to-add-parameters-in-module-class-in-pytorch-custom-model).
+        self.generator = torch.Generator().manual_seed(config.seed)
+        is_delete = torch.tensor(['I' not in label for label in FOREcasTModel.get_feature_label()])
+        self.register_buffer('reg_coff', (is_delete * config.reg_const + ~is_delete * config.i1_reg_const))
+        self.linear = nn.Linear(in_features=len(self.reg_coff), out_features=1, bias=False)
+        self.initialize_weights()
+
+    def initialize_weights(self):
+        for m in self.modules():
+            if isinstance(m, nn.Linear):
+                nn.init.normal_(m.weight, mean=0, std=1, generator=self.generator)
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+
+    def forward(self, feature, count=None) -> torch.Tensor:
+        logit = self.linear(feature).squeeze()
+        if count is not None:
+            return {
+                "logit": logit,
+                "loss": self.kl_divergence(logit, count)
+            }
+        return {"logit": logit}
+
+    def kl_divergence(self, logit, count):
+        return F.kl_div(
+            F.log_softmax(logit, dim=-1),
+            F.normalize(count + 0.5, p=1.0, dim=-1), # add 0.5 to prevent log(0), see loadOligoFeaturesAndReadCounts
+            reduction='sum'
+        ) + logit.shape[0] * (self.reg_coff * (self.linear.weight ** 2)).sum()

FOREcasT_model/runs/Nov22_09-23-55_ljw-System-Product-Name/events.out.tfevents.1732238639.ljw-System-Product-Name.7114.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed325373d7b951d73e184d2af92c077947a22c7e715472ab8f68d3f946c22c35
+size 19361

FOREcasT_model/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f40f824eac3f47aa7f2c23d3f36d4ec4956c4b4b4149c8d4f1f110f5d33d1396
+size 5304