Model Card for ZetaQA-1.1B-XML

Model Overview

ZetaQA-1.1B-XML is a fine-tuned version of TinyLlama-1.1B, designed for structured question answering with XML-formatted reasoning. It specializes in breaking down complex questions into logical steps and providing detailed, structured responses. The model is particularly adept at tasks requiring multi-step reasoning and explainable outputs.

---

Model Details

• Architecture: Transformer-based causal language model
• Base Model: TinyLlama-1.1B
• Fine-Tuning Dataset: StrategyQA (2,290 examples)
• Training Framework: Hugging Face Transformers + TRL
• Parameter Count: 1.1 billion
• License: Apache 2.0

---

What This Model Does

The model takes a question as input and generates a structured response that includes:

1. Key Terms: Identifies important concepts in the question.
2. Description: Provides context or definitions for key terms.
3. Decomposition Steps: Breaks the question into logical sub-steps.
4. Relevant Facts: Lists supporting evidence or facts.
5. Final Verdict: Answers the question with "Yes" or "No."
6. Reasoning: Explains the final answer based on the decomposition and facts.

Example:

Question: Can penguins fly?

Response:

• Key Term: Penguins
• Description: Flightless birds native to the southern hemisphere.
• Decomposition Steps:
  1. Penguins are flightless birds.
  2. Their wings evolved into flippers.
• Relevant Facts:
  • All penguin species are flightless.
  • They use wings for swimming.
• Final Verdict: No
• Reasoning: Penguins' wings have evolved into flippers for swimming, making them physically incapable of flight.

---

Training Process

1. Dataset:

   • StrategyQA: A dataset of complex questions requiring multi-step reasoning.
   • Fine-tuned on 2,290 examples with XML-formatted responses.

2. Preprocessing:

   • Questions and answers were formatted into XML structures.
   • Special tokens (<think>, </think>, <answer>, </answer>) were added to the tokenizer.

3. Fine-Tuning:

   • Framework: Hugging Face Transformers + TRL (Transformer Reinforcement Learning).
   • Hardware: 1x NVIDIA T4 GPU (Google Colab).
   • Batch Size: 2 (with gradient accumulation steps of 4).
   • Learning Rate: 3e-5.
   • Epochs: 3.
   • Sequence Length: 1024 tokens.

4. Evaluation:

   • Evaluated on a held-out validation set from StrategyQA.
   • Focused on response quality, reasoning accuracy, and XML structure adherence.

---

Intended Use

• Primary Use: Answering complex questions with structured, explainable reasoning.
• Target Audience:
  • Researchers studying explainable AI.
  • Developers building educational or decision-support tools.
  • Enterprises needing transparent AI systems.
• Applications:
  • Educational platforms (e.g., tutoring systems).
  • Decision support systems (e.g., medical or legal reasoning).
  • Explainable AI pipelines.

---

Performance

• Strengths:
  • Handles multi-step reasoning effectively.
  • Produces human-readable, structured outputs.
  • Lightweight (1.1B parameters) for efficient deployment.
• Limitations:
  • May struggle with highly domain-specific questions.
  • Limited by the 1.1B parameter size for extremely complex reasoning.

---

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("Kuldeep08/ZetaQA-1.1B-XML")
tokenizer = AutoTokenizer.from_pretrained("Kuldeep08/ZetaQA-1.1B-XML")

# Generate response
question = "Are strawberries a fruit?"
inputs = tokenizer(question, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)

# Decode and print response
response = tokenizer.decode(outputs[0], skip_special_tokens=False)
print(response)

Ethical Considerations

Bias: May inherit biases from the base model and training data.

Transparency: XML outputs improve explainability but should be validated for accuracy.

Deployment: Suitable for non-critical applications where errors can be tolerated.