Upload 17 files

README.md

@@ -1,139 +1,104 @@
 ---
 base_model: meta-llama/Llama-3.2-1B-Instruct
 library_name: peft
-license: apache-2.0
-language:
-- en
-metrics:
-- accuracy
-tags:
-- finance
-- relation_extraction
-- relation_types
 ---
 
+# Model Card for Model ID
 
-## Model Description
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
 
 <!-- Provide a longer summary of what this model is. -->
 
 
-- **Finetuned from model:** Llama-3.2-1B-Instruct
 
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
 
+### Model Sources [optional]
 
-## Downstream Use
+<!-- Provide the basic links for the model. -->
 
-Model for predicting relations between entities in the financial documents. 
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
 
-### Relation Types
-- no_relation
-- title
-- operations_in
-- employee_of
-- agreement_with
-- formed_on
-- member_of
-- subsidiary_of
-- shares_of
-- revenue_of
-- loss_of
-- headquartered_in
-- acquired_on
-- founder_of
-- formed_in
+## Uses
 
-## How to Get Started with the Model
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
-```python
-import torch
-from peft import AutoPeftModelForCausalLM
-from transformers import AutoTokenizer, pipeline
+### Direct Use
 
-# Load Model with PEFT adapter
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
-finetune_name = 'Askinkaty/llama-finance-relations'
+[More Information Needed]
 
-finetined_model = AutoPeftModelForCausalLM.from_pretrained(
-    pretrained_model_name_or_path=finetune_name,
-    torch_dtype=torch.float16,
-    low_cpu_mem_usage=True,
-)
+### Downstream Use [optional]
 
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
-base_model = "meta-llama/Llama-3.2-1B-Instruct"
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-base_model.config.pad_token_id = base_model.config.eos_token_id
+[More Information Needed]
 
+### Out-of-Scope Use
 
-pipeline = pipeline('text-generation', model=base_model, tokenizer=tokenizer)
-pipeline.model = model.to(device) 
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
+[More Information Needed]
 
-```
+## Bias, Risks, and Limitations
 
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-## Training Details
+[More Information Needed]
 
-### Training Data
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
 
-Samples from [ReFinD dataset](https://refind-re.github.io/). 100 examples for each relation type were used, least frequent relation types are omitted.
+## Training Details
 
+### Training Data
 
-#### Preprocessing
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-Dataset is converted into a message format as in the code snippet below: 
+[More Information Needed]
 
-```python
-def batch_convert_to_messages(data):
-        
-    questions = data.apply(
-        lambda x: f"Entity 1: {' '.join(x['token'][x['e1_start']:x['e1_end']])}. "
-                  f"Entity 2: {' '.join(x['token'][x['e2_start']:x['e2_end']])}. "
-                  f"Input sentence: {' '.join(x['token'])}",
-        axis=1
-    )
+### Training Procedure
 
-    relations = data['relation'].apply(lambda relation: relation.split(':')[-1])
-    
-    messages = [
-        [
-            {
-                "role": "system",
-                "content": "You are an expert in financial documentation and market analysis. Define relations between two specified entities: entity 1 [E1] and entity 2 [E2] in a sentence. Return a short response of the required format. "
-            },
-            {"role": "user", "content": question},
-            {"role": "assistant", "content": relation},
-        ]
-        for question, relation in zip(questions, relations)
-    ]
-    
-    return messages
-```
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
+#### Preprocessing [optional]
 
+[More Information Needed]
 
 
 #### Training Hyperparameters
 
-SFT parameters:
-- num_train_epochs=1
-- per_device_train_batch_size=2
-- gradient_accumulation_steps=2
-- gradient_checkpointing=True
-- optim="adamw_torch_fused"
-- learning_rate=2e-4
-- max_grad_norm=0.3
-- warmup_ratio=0.01
-- lr_scheduler_type="cosine"
-- bf16=True
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
 
-LORA parameters:
-- rank_dimension = 6
-- lora_alpha = 8
-- lora_dropout = 0.05
+#### Speeds, Sizes, Times [optional]
 
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
 
+[More Information Needed]
 
 ## Evaluation
 
@@ -143,18 +108,95 @@ LORA parameters:
 
 #### Testing Data
 
-Test set sampled from Samples from [ReFinD dataset](https://refind-re.github.io/).
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
 
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
 
 #### Metrics
 
-Accuracy. Other metrics: work in progress.
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
 
 ### Results
 
-Accuracy before clearning the output: 0.41
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
 
-Accuracy after clearning the output: 0.62
+## Model Card Contact
 
+[More Information Needed]
+### Framework versions
 
 - PEFT 0.14.0

adapter_config.json

@@ -23,13 +23,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "k_proj",
-    "up_proj",
-    "o_proj",
-    "gate_proj",
     "v_proj",
+    "up_proj",
     "q_proj",
-    "down_proj"
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "o_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:93a96bd9a22e1bc9ba99b9f303bb95318c937238d6217e06707c1d5ffff21001
+oid sha256:bbfc469fc0f359611284b9a33ffea89a4c2cd108a6411a23c605797eac90d6ef
 size 2118301632

model-00001-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:81b62127279b14b81dc5ba337f179c0eff707e894f824c7b9a1bf889c671bc2b
+oid sha256:e876dadb94eaf20db78c2ca1778580419eacb10c8032d42f5396d446ce18a1c2
 size 1990270808

model-00002-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d1b19c21a12ba689d7772537834f3a206f26546166543d64fcbe078c5315f950
+oid sha256:7c2098a54b6104c08f791ffd311b2c8b264f47830750fc2f261e3f9f8efab622
 size 1006719368

optimizer.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3b5a53107f741c6fba66c4467e45596e9a583177380e53e494035d8463be6b38
+oid sha256:60ec98f4495156924b88229bdbe56f8bb1202e5a8203e76105d5d2b0acc01442
 size 34007674

rng_state.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0025193221626e8e44624e550369b44ae92802c57be84c04b10bc0ca73e990e8
+oid sha256:e0f3a6090e681048a6c2caa8a2f14f616e5a16d68b188a93348c4482d6393a0c
 size 14244

scheduler.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b799fc6e89913f6020bade7e21eca60ed0da11a47da89511a1092cf00a63e990
+oid sha256:3c92fc546447cfc1151596b19e46f7088e39100f9de60871cd8a0f9b1a8d1df9
 size 1064

trainer_state.json

@@ -1,268 +1,142 @@
 {
   "best_metric": null,
   "best_model_checkpoint": null,
-  "epoch": 0.9986431478968792,
+  "epoch": 0.9993211133740665,
   "eval_steps": 500,
-  "global_step": 368,
+  "global_step": 184,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
-      "epoch": 0.027137042062415198,
-      "grad_norm": 1.4497582912445068,
-      "learning_rate": 0.0001998659482680456,
-      "loss": 3.4336,
-      "step": 10
-    },
-    {
-      "epoch": 0.054274084124830396,
-      "grad_norm": 1.846526861190796,
+      "epoch": 0.05431093007467753,
+      "grad_norm": 1.5025734901428223,
       "learning_rate": 0.00019904804439875633,
-      "loss": 1.9601,
-      "step": 20
-    },
-    {
-      "epoch": 0.0814111261872456,
-      "grad_norm": 0.9556183815002441,
-      "learning_rate": 0.00019749279121818235,
-      "loss": 1.202,
-      "step": 30
+      "loss": 3.2617,
+      "step": 10
     },
     {
-      "epoch": 0.10854816824966079,
-      "grad_norm": 1.0818341970443726,
+      "epoch": 0.10862186014935506,
+      "grad_norm": 1.5902663469314575,
       "learning_rate": 0.00019521176659107142,
-      "loss": 1.1366,
-      "step": 40
-    },
-    {
-      "epoch": 0.13568521031207598,
-      "grad_norm": 0.9879324436187744,
-      "learning_rate": 0.00019222195128618106,
-      "loss": 1.0269,
-      "step": 50
+      "loss": 1.7882,
+      "step": 20
     },
     {
-      "epoch": 0.1628222523744912,
-      "grad_norm": 1.1216270923614502,
+      "epoch": 0.1629327902240326,
+      "grad_norm": 0.7277324795722961,
       "learning_rate": 0.000188545602565321,
-      "loss": 1.0766,
-      "step": 60
-    },
-    {
-      "epoch": 0.18995929443690637,
-      "grad_norm": 1.1110620498657227,
-      "learning_rate": 0.00018421008849228118,
-      "loss": 0.9947,
-      "step": 70
+      "loss": 1.1819,
+      "step": 30
     },
     {
-      "epoch": 0.21709633649932158,
-      "grad_norm": 0.8764580488204956,
+      "epoch": 0.2172437202987101,
+      "grad_norm": 0.6814110279083252,
       "learning_rate": 0.00017924768419510904,
-      "loss": 1.0024,
-      "step": 80
-    },
-    {
-      "epoch": 0.24423337856173677,
-      "grad_norm": 1.2004687786102295,
-      "learning_rate": 0.00017369533159843369,
-      "loss": 1.04,
-      "step": 90
+      "loss": 1.0805,
+      "step": 40
     },
     {
-      "epoch": 0.27137042062415195,
-      "grad_norm": 0.9665369391441345,
+      "epoch": 0.27155465037338766,
+      "grad_norm": 0.6621173620223999,
       "learning_rate": 0.00016759436441447545,
-      "loss": 0.9809,
-      "step": 100
-    },
-    {
-      "epoch": 0.29850746268656714,
-      "grad_norm": 1.5474181175231934,
-      "learning_rate": 0.00016099020044000727,
-      "loss": 0.9405,
-      "step": 110
+      "loss": 1.0806,
+      "step": 50
     },
     {
-      "epoch": 0.3256445047489824,
-      "grad_norm": 1.014674186706543,
+      "epoch": 0.3258655804480652,
+      "grad_norm": 0.717393696308136,
       "learning_rate": 0.00015393200344991995,
-      "loss": 0.9054,
-      "step": 120
-    },
-    {
-      "epoch": 0.35278154681139756,
-      "grad_norm": 1.013979434967041,
-      "learning_rate": 0.00014647231720437686,
-      "loss": 0.9754,
-      "step": 130
+      "loss": 1.0157,
+      "step": 60
     },
     {
-      "epoch": 0.37991858887381275,
-      "grad_norm": 1.0418416261672974,
+      "epoch": 0.3801765105227427,
+      "grad_norm": 0.6353682279586792,
       "learning_rate": 0.0001386666742941419,
-      "loss": 0.8711,
-      "step": 140
-    },
-    {
-      "epoch": 0.40705563093622793,
-      "grad_norm": 0.9280593395233154,
-      "learning_rate": 0.0001305731827359753,
-      "loss": 0.8358,
-      "step": 150
+      "loss": 1.0388,
+      "step": 70
     },
     {
-      "epoch": 0.43419267299864317,
-      "grad_norm": 1.0268974304199219,
+      "epoch": 0.4344874405974202,
+      "grad_norm": 0.7220941185951233,
       "learning_rate": 0.00012225209339563145,
-      "loss": 0.8568,
-      "step": 160
-    },
-    {
-      "epoch": 0.46132971506105835,
-      "grad_norm": 0.9812881350517273,
-      "learning_rate": 0.00011376535145871684,
-      "loss": 0.888,
-      "step": 170
+      "loss": 0.9547,
+      "step": 80
     },
     {
-      "epoch": 0.48846675712347354,
-      "grad_norm": 1.2109171152114868,
+      "epoch": 0.48879837067209775,
+      "grad_norm": 0.7532466650009155,
       "learning_rate": 0.00010517613528842097,
-      "loss": 0.9166,
-      "step": 180
-    },
-    {
-      "epoch": 0.5156037991858887,
-      "grad_norm": 0.9860134124755859,
-      "learning_rate": 9.654838610302923e-05,
-      "loss": 0.8357,
-      "step": 190
+      "loss": 1.0116,
+      "step": 90
     },
     {
-      "epoch": 0.5427408412483039,
-      "grad_norm": 1.1094295978546143,
+      "epoch": 0.5431093007467753,
+      "grad_norm": 0.8198474645614624,
       "learning_rate": 8.79463319744677e-05,
-      "loss": 0.928,
-      "step": 200
-    },
-    {
-      "epoch": 0.5698778833107191,
-      "grad_norm": 0.9386014342308044,
-      "learning_rate": 7.943400969140635e-05,
-      "loss": 0.9522,
-      "step": 210
+      "loss": 0.9763,
+      "step": 100
     },
     {
-      "epoch": 0.5970149253731343,
-      "grad_norm": 1.0436668395996094,
+      "epoch": 0.5974202308214528,
+      "grad_norm": 0.79359370470047,
       "learning_rate": 7.107478804634325e-05,
-      "loss": 0.8619,
-      "step": 220
-    },
-    {
-      "epoch": 0.6241519674355496,
-      "grad_norm": 1.0143946409225464,
-      "learning_rate": 6.293089609549325e-05,
-      "loss": 0.8046,
-      "step": 230
+      "loss": 1.0034,
+      "step": 110
     },
     {
-      "epoch": 0.6512890094979648,
-      "grad_norm": 1.107064127922058,
+      "epoch": 0.6517311608961304,
+      "grad_norm": 1.1620718240737915,
       "learning_rate": 5.506295990328385e-05,
-      "loss": 0.8532,
-      "step": 240
-    },
-    {
-      "epoch": 0.6784260515603799,
-      "grad_norm": 0.8748170137405396,
-      "learning_rate": 4.75295512200992e-05,
-      "loss": 0.8276,
-      "step": 250
+      "loss": 0.9125,
+      "step": 120
     },
     {
-      "epoch": 0.7055630936227951,
-      "grad_norm": 1.0721319913864136,
+      "epoch": 0.7060420909708078,
+      "grad_norm": 1.5092897415161133,
       "learning_rate": 4.038675145307747e-05,
-      "loss": 0.7517,
-      "step": 260
-    },
-    {
-      "epoch": 0.7327001356852103,
-      "grad_norm": 1.1542731523513794,
-      "learning_rate": 3.36877341759205e-05,
-      "loss": 0.8542,
-      "step": 270
+      "loss": 0.8748,
+      "step": 130
     },
     {
-      "epoch": 0.7598371777476255,
-      "grad_norm": 0.89125657081604,
+      "epoch": 0.7603530210454854,
+      "grad_norm": 1.0216153860092163,
       "learning_rate": 2.7482369285662378e-05,
-      "loss": 0.8601,
-      "step": 280
-    },
-    {
-      "epoch": 0.7869742198100407,
-      "grad_norm": 0.9904446601867676,
-      "learning_rate": 2.181685175319702e-05,
-      "loss": 0.8787,
-      "step": 290
+      "loss": 0.9131,
+      "step": 140
     },
     {
-      "epoch": 0.8141112618724559,
-      "grad_norm": 0.9311710596084595,
+      "epoch": 0.814663951120163,
+      "grad_norm": 1.5323857069015503,
       "learning_rate": 1.6733357731279377e-05,
-      "loss": 0.7493,
-      "step": 300
-    },
-    {
-      "epoch": 0.841248303934871,
-      "grad_norm": 1.0216141939163208,
-      "learning_rate": 1.2269730580055805e-05,
-      "loss": 0.7497,
-      "step": 310
+      "loss": 0.8702,
+      "step": 150
     },
     {
-      "epoch": 0.8683853459972863,
-      "grad_norm": 0.9951556324958801,
+      "epoch": 0.8689748811948405,
+      "grad_norm": 0.962648332118988,
       "learning_rate": 8.45919914746337e-06,
-      "loss": 0.8324,
-      "step": 320
-    },
-    {
-      "epoch": 0.8955223880597015,
-      "grad_norm": 1.0728198289871216,
-      "learning_rate": 5.3301304017194135e-06,
-      "loss": 0.7964,
-      "step": 330
+      "loss": 0.8396,
+      "step": 160
     },
     {
-      "epoch": 0.9226594301221167,
-      "grad_norm": 1.1030397415161133,
+      "epoch": 0.923285811269518,
+      "grad_norm": 0.7791914939880371,
       "learning_rate": 2.905818257394799e-06,
-      "loss": 0.8166,
-      "step": 340
-    },
-    {
-      "epoch": 0.9497964721845319,
-      "grad_norm": 0.974777102470398,
-      "learning_rate": 1.2043101671253554e-06,
-      "loss": 0.7952,
-      "step": 350
+      "loss": 0.8633,
+      "step": 170
     },
     {
-      "epoch": 0.9769335142469471,
-      "grad_norm": 1.078715443611145,
+      "epoch": 0.9775967413441955,
+      "grad_norm": 0.8390964865684509,
       "learning_rate": 2.382727698752474e-07,
-      "loss": 0.8726,
-      "step": 360
+      "loss": 0.8817,
+      "step": 180
     }
   ],
   "logging_steps": 10,
-  "max_steps": 368,
+  "max_steps": 184,
   "num_input_tokens_seen": 0,
   "num_train_epochs": 1,
   "save_steps": 500,
@@ -278,8 +152,8 @@
       "attributes": {}
     }
   },
-  "total_flos": 1620605933346816.0,
-  "train_batch_size": 2,
+  "total_flos": 1445464568266752.0,
+  "train_batch_size": 1,
   "trial_name": null,
   "trial_params": null
 }

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3ad9d3a4e9a5aecdee049eba346c401401fe901b8db3ce260bbc237b3b3e9243
+oid sha256:0d4a4dcbea433b28b388dcc17c84657d36933b48455101d1610f7855b760e64f
 size 5624