Create README.md

README.md

@@ -0,0 +1,112 @@
+---
+license: mit
+datasets:
+- flytech/python-codes-25k
+tags:
+- code
+---
+# GPT2 PyCode
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+This model is a fine-tuned version of the GPT 124M model, specifically adapted for testing purposes in Python code generation. It was trained on a small corpus of 25,000 Python code samples.
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This project features a GPT (Generative Pre-trained Transformer) language model with 124 million parameters that has been fine-tuned for Python code generation. Unlike larger models like GPT-2 or GPT-3, this is a smaller-scale model designed primarily for testing and experimental purposes.
+
+- **Developed by:** Maharnab Saikia
+- **Model type:** Language model
+- **Language(s) (NLP):** English
+- **License:** MIT
+- **Finetuned from model:** GPT2 124M
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+ - **Research:** Studying the behavior of small-scale language models in code generation tasks
+ - **Benchmarking:** Providing a baseline for comparing different model architectures or training strategies
+ - **Rapid Prototyping:** Quick tests of code generation ideas without the overhead of larger models
+ - **Education:** Demonstrating the principles of fine-tuning language models for specific tasks
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+It's crucial to understand the limitations of this model:
+
+ - Limited knowledge base due to the small training corpus
+ - May struggle with complex or specialized Python code
+ - Not suitable for production-level code generation tasks
+ - Performance will likely be significantly lower than larger, more comprehensively trained models
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+```python
+from transformers import GPT2LMHeadModel, GPT2Tokenizer
+import re
+
+tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
+model = GPT2Model.from_pretrained('gpt2')
+
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer.encode_plus(f"<sos><user>{prompt}</user><assistant>", max_length=20, truncation=True, return_tensors="pt")
+
+input_ids = encoded_input['input_ids']
+attention_mask = encoded_input['attention_mask']
+
+output = model.generate(
+    input_ids, 
+    max_length=512, 
+    num_return_sequences=1, 
+    no_repeat_ngram_size=2,
+    temperature=0.7,
+    do_sample=True,
+    top_k=50,
+    top_p=0.95,
+    attention_mask=attention_mask,
+    pad_token_id=tokenizer.pad_token_id
+)
+
+generated_code = tokenizer.decode(output[0])
+generated_code = re.search(r'<assistant>(.*?)</assistant>', generated_code, re.DOTALL).group(1)
+
+print(f"Prompt: {prompt}\nGenerated Code:\n{generated_code}")
+```
+
+## Training Details
+
+### Training Data
+
+ - **Model:** GPT with 124 million parameters
+ - **Training Data:** 25,000 Python code samples
+ - **Fine-tuning:** Adapted specifically for Python code generation tasks
+
+
+#### Training Hyperparameters
+
+ - **Epochs:** 5
+ - **Batch Size:** 8
+ - **Learning Rate:** 5e-5
+ - **Contex Window:** 512
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions was estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** P100 GPU
+- **Hours used:** 5
+- **Cloud Provider:** Kaggle
+- **Compute Region:** South Asia
+- **Carbon Emitted:** 1.15
+
+## Acknowledgements
+
+This project builds upon the GPT-2 model developed by OpenAI. We acknowledge their groundbreaking work in the field of natural language processing.