YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
NOT IN WORKING STATE, PLS WAIT
Custom GPT Model
This is a custom GPT model with:
- RMS normalization
- Rotary positional embeddings (RoPE)
- Separate Q,K,V projections
- Squared ReLU activation in MLP
- QK normalization in attention
- Zero initialization for projection layers
Architecture
- Vocabulary Size: 50304
- Context Length: 1024
- Number of Layers: 12
- Number of Heads: 6
- Embedding Dimension: 768
Usage
from transformers import AutoModel
model = AutoModel.from_pretrained("Arjun-G-Ravi/Custom-GPT-555k")
- Downloads last month
- 21
Inference Providers
NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API:
The model has no library tag.