README_HF.md

MechVis: GPT-2 Attention Head Visualization

This interactive web app allows you to visualize how different attention heads in GPT-2 small contribute to next token predictions.

How to Use

1. Enter text in the input field (e.g., "When John and Mary went to the store, John gave a drink to")
2. Click "Predict Next Word"
3. See what token GPT-2 predicts next and explore how each attention head contributes to that prediction

Features

• Next token prediction with GPT-2 small
• Interactive heatmap showing attention head contributions
• Layer contribution analysis
• Hover over cells to see exact contribution values

Examples to Try

• Indirect Object Identification: "When John and Mary went to the store, John gave a drink to" (likely predicts "Mary")
• Induction Pattern: "The capital of France is Paris. The capital of Germany is" (likely predicts "Berlin")

About

This project uses TransformerLens to access internal model activations and calculate how each attention head contributes to the final logit score of the predicted token.

GitHub Repository