Update README.md

README.md

@@ -8,192 +8,183 @@ tags: []
 <!-- Provide a quick summary of what the model is/does. -->
 
 
-
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+from typing import List, Optional
+import numpy as np
+from datetime import datetime
+
+class TextStreamer:
+    def __init__(self, tokenizer, output_file=None):
+        self.tokenizer = tokenizer
+        self.current_tokens = []
+        self.output_file = output_file
+        self.full_response = ""
+        
+    def put(self, value):
+        if isinstance(value, torch.Tensor):
+            value = value.cpu().numpy()
+            if len(value.shape) > 1:
+                value = value[0]
+            
+        if isinstance(value, np.ndarray):
+            value = value.tolist()
+            
+        if isinstance(value, list):
+            if isinstance(value[0], list):
+                value = value[0]
+            self.current_tokens.extend(value)
+        else:
+            self.current_tokens.append(value)
+            
+        tokens_to_decode = [int(token) for token in self.current_tokens]
+        text = self.tokenizer.decode(tokens_to_decode, skip_special_tokens=True)
+        
+        if len(self.current_tokens) > len(value):
+            previous_text = self.tokenizer.decode(
+                [int(token) for token in self.current_tokens[:-len(value) if isinstance(value, list) else -1]], 
+                skip_special_tokens=True
+            )
+            new_text = text[len(previous_text):]
+        else:
+            new_text = text
+            
+        if new_text:
+            print(new_text, end="", flush=True)
+            self.full_response += new_text
+
+    def end(self):
+        print("")
+        if self.output_file:
+            with open(self.output_file, 'a', encoding='utf-8') as f:
+                f.write(f"\n--- Response generated at {datetime.now().strftime('%Y-%m-%d %H:%M:%S')} ---\n")
+                f.write(self.full_response)
+                f.write("\n\n")
+        return self.full_response
+
+class Translator:
+    DEFAULT_SYSTEM_PROMPT = """
+    You are a skilled linguistic expert specializing in cross-lingual translation. Your task is to perform accurate and detailed translations, moving from a given source language to a specified destination language. 
+    You will perform the translation by thinking and reasoning step-by-step by and demonstrating the linguistic transformation process while maintaining the source context.
+
+    # Output Format
+
+    Your translation responses should be structured as follows:
+
+    ```
+    <think>
+    [Detailed thinking and reasoning process, including the analysis and breakdown of the sentence]
+    </think>
+    <translation>
+    [Final translated sentence based on the step-by-step reasoning]
+    </translation>
+    ```
+
+    Stick to the above formate and exclose the final translation in <translation>{translated sentence}</translation>
+    """
+
+    def __init__(
+        self,
+        model_name: str,
+        system_prompt: Optional[str] = None,
+        device_map: str = "auto",
+        torch_dtype: str = "auto"
+    ):
+        """
+        Initialize the translator with a model and tokenizer.
+        
+        Args:
+            model_name: Path or name of the model to load
+            system_prompt: Optional custom system prompt
+            device_map: Device mapping strategy for model loading
+            torch_dtype: Torch data type for model
+        """
+        self.model = AutoModelForCausalLM.from_pretrained(
+            model_name,
+            torch_dtype=torch_dtype,
+            device_map=device_map
+        )
+        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
+        self.system_prompt = system_prompt or self.DEFAULT_SYSTEM_PROMPT
+
+    def translate(
+        self,
+        text: str,
+        max_new_tokens: int = 2048,
+        temperature: float = 0.1,
+        top_p: float = 0.7,
+        output_file: Optional[str] = None
+    ) -> str:
+        """
+        Translate the given text using the loaded model.
+        
+        Args:
+            text: Text to translate
+            max_new_tokens: Maximum number of tokens to generate
+            temperature: Temperature for generation
+            top_p: Top-p sampling parameter
+            output_file: Optional file to save the translation
+            
+        Returns:
+            str: The translated text
+        """
+        # Prepare messages
+        messages = [
+            {"role": "system", "content": self.system_prompt},
+            {"role": "user", "content": text}
+        ]
+
+        # Apply chat template
+        prompt = self.tokenizer.apply_chat_template(
+            messages,
+            tokenize=False,
+            add_generation_prompt=True
+        )
+
+        # Tokenize input
+        model_inputs = self.tokenizer([prompt], return_tensors="pt").to(self.model.device)
+
+        # Create streamer
+        streamer = TextStreamer(self.tokenizer, output_file=output_file)
+        
+        # Generate with streaming
+        self.model.generate(
+            **model_inputs,
+            max_new_tokens=max_new_tokens,
+            temperature=temperature,
+            top_p=top_p,
+            streamer=streamer
+        )
+
+        return streamer.end()
+
+    def __call__(self, *args, **kwargs) -> str:
+        """
+        Make the translator callable directly.
+        """
+        return self.translate(*args, **kwargs)
+
+# %%
+translator = Translator(
+    model_name="CoT-Translator/Llama-3b-Reasoning-Translate"
+)
+
+# %%
+# Use it multiple times
+texts = [
+    # "संक्रमित चमगादड़ निपाह विषाणु को सूअरों जैसे अन्य जानवरों में भी फैला सकते हैं। .translate from hindi to english",
+    "how are you doing today and what is your name .translate from english to hindi",
+    # "सफरचंदसाठी आजचा दिवस खरोखर चांगला आहे आणि मला खूप मजा येत आहे. translate from marathi to english"
+    # "Today's day is really good for Safar Chand and I'm having a lot of fun. translate from english to marathi"
+]
+
+for text in texts:
+    print(f"\nTranslating: {text}")
+    translation = translator(
+        text,
+        output_file="translation_responses_llama_translate.txt"
+    )
+    print(f"Complete translation: {translation}\n")
+
+```