jebish7 commited on
Commit
04c2d6b
·
verified ·
1 Parent(s): 9c464c3

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +22 -16
README.md CHANGED
@@ -107,7 +107,7 @@ model-index:
107
  ---
108
  # Phi-4-Hindi
109
 
110
- <strong>Phi-4-Hindi</strong> is a 14.7B multilingual large language model, instruction-tuned to achieve state-of-the-art performance in Hindi and English. Built on a pre-trained foundation, it is optimized for bilingual tasks with a diverse mixed-language dataset.
111
  - <strong>~1%</strong> better performance on <strong>English</strong> Tasks compared to the original (average benchmark scores)
112
  - <strong>~4%</strong> better performance on <strong>Hindi</strong> Tasks compared to the original (average benchmark scores)
113
  - <strong>~10% less emissions</strong> than the original (as reported on benchmark evaluations like open-llm-leaderboard)
@@ -123,8 +123,8 @@ model-index:
123
  ## Intended Use
124
 
125
  <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
126
- We release Phi-4-Hindi under the Apache 2.0 license, encouraging researchers, developers, and enterprises to experiment with and build upon the model, particularly for bilingual, multilingual and non-English applications.
127
- At the time of release, the model demonstrated state-of-the-art performance across an extensive English and Hindi evaluation suite.
128
 
129
  Some potential downstream applications are as follows:
130
  - *Research*: This model serves as a valuable tool for researchers and developers working in NLP.
@@ -156,22 +156,22 @@ Target audiences who may benefit from our model:
156
  | Translation to specified language | "`Input ### TRANSLATION [lang] ###`" |
157
  | Text Simplification/ELI5 | "`Input ### SIMPLIFY ###`" |
158
 
159
- The following prompt formats were used during training and are better suited for usage, however the model works well even without such formatting
160
 
161
  ### Out-of-Scope Use
162
 
163
  <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
164
 
165
- While Phi-4-Hindi is a powerful bilingual model designed for Hindi and English, it is crucial to acknowledge its limitations and the potential for misuse. The model must not be used in ways that violate any applicable laws or regulations. Below are specific scenarios where its use is restricted:
166
 
167
  - *Harmful or Malicious Use*: The model should not be employed to create or distribute harmful, misleading, or inappropriate content, including but not limited to:
168
  - Encouraging hate speech, violence, or discrimination
169
  - Spreading misinformation or false narratives
170
  - Facilitating or promoting illegal activities
171
 
172
- - *Sensitive Data Handling*: The model is not designed to process or generate personal, confidential, or sensitive information.
173
 
174
- - *Language Constraints*: While optimized for Hindi and English, the model should not be assumed to have the same proficiency in other languages.
175
 
176
  - *High-Risk Decision-Making*: It should not be used for critical decision-making without human oversight, especially in medical, legal, financial, or safety-related contexts.
177
 
@@ -180,14 +180,14 @@ While Phi-4-Hindi is a powerful bilingual model designed for Hindi and English,
180
  <!-- This section is meant to convey both technical and sociotechnical limitations. -->
181
 
182
  <!-- The model is trained on publicly available data which was in part curated by Inception. -->
183
- While efforts have been made to minimize biases, it is likely that the model, as with all LLM models, will exhibit some bias.
184
 
185
- The model is trained as an AI assistant for Hindi and English speakers. The model is limited to produce responses for queries in these two languages
186
  and may not produce appropriate responses to other language queries.
187
 
188
- By using this model, you acknowledge and accept that, as with any large language model, it may generate incorrect, misleading and/or offensive information or content.
189
- The information is not intended as advice and should not be relied upon in any way, nor are we responsible for any of the content or consequences resulting from its use.
190
- We are continuously working to develop models with greater capabilities, and as such, welcome any feedback on the model~~
191
 
192
  ## Evaluation:
193
  We evaluated our models on multiple well-known benchmarks to measure their effectiveness against other leading models, and the results are as follows:
@@ -261,16 +261,22 @@ It is advisable for users to:
261
 
262
  ### Emissions
263
 
264
- We belive our usage of shorter and compressed instruction-reponse pairs in training resulted in the model responding in simplified manner while meeting the requirements/ arriving at the correct answers. Hence the better scores while reducing emissions.
265
 
266
- Unlike distillation from reasoining or CoT models which produced unnecessarily long responses like "Next we proceed with...", "Ok lets do this...." during generation of step by step solutions of a math problem, we use only the step by step math part ignoring such fillers, for datasets with multiple step-by-step solutions which are correct, we chose the shortest one to train our models.
 
 
267
 
268
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/645c60dd7d655680b57ddbff/vgNk0bKthxNsxO0oAdaPD.png)
269
 
270
  ### Model Responses vs Order of Choices in MCQs
271
 
272
- As benchmarks like MMLU-Pro have upto 10 choices, while most training datasets consist of typically 4-5 choices, we modified the ordering and labelling of choices i.e re-ordering choices to create an imbalance opposing the original model's choice distribution, replacement of labels from A/B/C/D to a/b/c/d or 1/2/3/4 or w/x/y/z etc.. in 5% of the MCQ samples for better robustness
273
- This resulted in less bias towards the earlier choices among MCQs as compared to the original phi-4. The below images are a distution of choices selected by the model while being evaluated over MMLU-pro
 
 
 
 
274
 
275
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/645c60dd7d655680b57ddbff/5DYCkLHpdk2jaTsALcwN8.png)
276
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/645c60dd7d655680b57ddbff/hhNNE4s8mALYsxdVf-UCq.png)
 
107
  ---
108
  # Phi-4-Hindi
109
 
110
+ Phi-4-Hindi is a 14.7B multilingual large language model, instruction-tuned to achieve <strong>state-of-the-art</strong> performance in <strong>Hindi</strong> and <strong>English</strong>. Built on a pre-trained foundation, it is optimized for bilingual tasks with a diverse mixed-language dataset.
111
  - <strong>~1%</strong> better performance on <strong>English</strong> Tasks compared to the original (average benchmark scores)
112
  - <strong>~4%</strong> better performance on <strong>Hindi</strong> Tasks compared to the original (average benchmark scores)
113
  - <strong>~10% less emissions</strong> than the original (as reported on benchmark evaluations like open-llm-leaderboard)
 
123
  ## Intended Use
124
 
125
  <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
126
+ We release Phi-4-Hindi under the Apache 2.0 license, encouraging researchers, developers, and enterprises to experiment with and build upon the model, particularly for bilingual, multilingual, and non-English applications.
127
+ At the time of release, the model demonstrated state-of-the-art performance across an extensive English and Hindi evaluation suite.
128
 
129
  Some potential downstream applications are as follows:
130
  - *Research*: This model serves as a valuable tool for researchers and developers working in NLP.
 
156
  | Translation to specified language | "`Input ### TRANSLATION [lang] ###`" |
157
  | Text Simplification/ELI5 | "`Input ### SIMPLIFY ###`" |
158
 
159
+ The following prompt formats were used during training and are better suited for usage; however, the model works well even without such formatting.
160
 
161
  ### Out-of-Scope Use
162
 
163
  <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
164
 
165
+ While Phi-4-Hindi is a powerful model designed for Hindi and English, usage must adhere to any applicable laws or regulations with its limitations in mind. The model must not be misused in ways that violate any applicable laws or regulations. Below are specific scenarios where its use is restricted:
166
 
167
  - *Harmful or Malicious Use*: The model should not be employed to create or distribute harmful, misleading, or inappropriate content, including but not limited to:
168
  - Encouraging hate speech, violence, or discrimination
169
  - Spreading misinformation or false narratives
170
  - Facilitating or promoting illegal activities
171
 
172
+ - *Sensitive Data Handling*: The model is not designed to process or generate personal, confidential, or sensitive information. Users must implement their own privacy policy and pipelines for handling sensitive contents.
173
 
174
+ - *Language Constraints*: Phi-4-Hindi is a multilingual model built on the Phi-4 foundation but has been instruction-tuned for enhanced performance in Hindi and English. As a result, its proficiency in other languages may differ from the base Phi-4 model. The potential performance impact on other languages has not been evaluated, as this model is specifically optimized for bilingual (Hindi-English) tasks.
175
 
176
  - *High-Risk Decision-Making*: It should not be used for critical decision-making without human oversight, especially in medical, legal, financial, or safety-related contexts.
177
 
 
180
  <!-- This section is meant to convey both technical and sociotechnical limitations. -->
181
 
182
  <!-- The model is trained on publicly available data which was in part curated by Inception. -->
183
+ While efforts have been made to minimize biases, it is likely that the model, as with all LLM models, will exhibit some bias.
184
 
185
+ The model is trained as an AI assistant for Hindi and English speakers. The model is limited to producing responses for queries in these two languages
186
  and may not produce appropriate responses to other language queries.
187
 
188
+ By using this model, you acknowledge and accept that, as with any large language model, it may generate incorrect, misleading, and/or offensive information or content.
189
+ The information is not intended as advice and should not be relied upon in any way, nor are we responsible for any of the content or consequences resulting from its use.
190
+ We are continuously working to develop models with greater capabilities, and as such, welcome any feedback on the model.~~
191
 
192
  ## Evaluation:
193
  We evaluated our models on multiple well-known benchmarks to measure their effectiveness against other leading models, and the results are as follows:
 
261
 
262
  ### Emissions
263
 
264
+ We believe that our use of shorter, more compressed instruction-response pairs during training has led to a model that generates concise yet accurate responses. This approach allows the model to meet requirements and arrive at correct answers while also improving efficiency and reducing emissions.
265
 
266
+ Unlike distillation from reasoning or Chain-of-Thought (CoT) models, which often produce unnecessarily long responses—such as "Next, we proceed with..." or "Okay, let's do this..."—our method focuses purely on the essential step-by-step reasoning. For datasets containing multiple correct multi-step solutions, we prioritized training on the shortest valid solution, eliminating filler content.
267
+
268
+ This ensures that Phi-4-Hindi remains both effective and efficient, delivering high-quality results without unnecessary verbosity.
269
 
270
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/645c60dd7d655680b57ddbff/vgNk0bKthxNsxO0oAdaPD.png)
271
 
272
  ### Model Responses vs Order of Choices in MCQs
273
 
274
+ Benchmarks like MMLU-Pro include up to 10 answer choices, whereas most training datasets typically contain only 4-5 choices. To improve robustness and reduce bias, we introduced modifications in the ordering and labeling of answer choices, including:
275
+
276
+ - Reordering answer choices to create an imbalance that opposes the original model's choice distribution.
277
+ - Altering labels in 5% of MCQ samples, replacing standard A/B/C/D labels with variations like a/b/c/d, 1/2/3/4, or w/x/y/z.
278
+ -
279
+ These adjustments resulted in reduced bias toward earlier answer choices, leading to a more balanced selection distribution compared to the original Phi-4. The images below illustrate the distribution of choices selected by the model during MMLU-Pro evaluation.
280
 
281
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/645c60dd7d655680b57ddbff/5DYCkLHpdk2jaTsALcwN8.png)
282
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/645c60dd7d655680b57ddbff/hhNNE4s8mALYsxdVf-UCq.png)