End of training

README.md

@@ -78,7 +78,7 @@ GPT2LMHeadModel(
 
 # Resource Usage
 
-- Max Train VRAM Use: 20.5045 GB
+- Max Train VRAM Use: 13.7815 GB
 - Available VRAM: 23.4329 GB
 - GPUs: 
   - 1x NVIDIA GeForce RTX 4090
@@ -115,7 +115,7 @@ GPT2LMHeadModel(
 <br/>
 
 # Train Dataset
-Trained on 525,563,160 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
+Trained on 525,579,616 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
 - Num Samples: `998,000`
 - Subset: `20231101.en`
@@ -134,11 +134,7 @@ DistillationObjective(
         weight=0
     ),
     attn_loss_component=LossComponent(
-        weight=5.0,
-        loss_fn='raw_mse',
-        layer_mapper='layer-2',
-        norm='layernorm_teacher_only_affine',
-        projector='orthogonal'
+        weight=0
     )
 )
 ```
@@ -165,14 +161,10 @@ The following hyperparameters were used during training:
         weight=0
     ),
     attn_loss_component=LossComponent(
-        weight=5.0,
-        loss_fn='raw_mse',
-        layer_mapper='layer-2',
-        norm='layernorm_teacher_only_affine',
-        projector='orthogonal'
+        weight=0
     )
 )`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7a9e082a8370>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x710a3dbe03a0>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `distilbert/distilgpt2`
 - student_model_config: `None`

logs/attn_weight=0.0/events.out.tfevents.1726105148.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c99482e6c95932d020f3af668376ea7268eb59d8a81ae92be8f4886be03ead19
+size 529