|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
# Linear_Tiny_87M |
|
|
|
## Introduction |
|
|
|
Linear transformers have emerged as a subquadratic-time alternative to softmax attention and have garnered significant interest due to their fixed-size recurrent state that lowers inference cost. However, their original formulation suffers from poor scaling and underperforms compute-matched transformers. Recent linear models such as RWKV and Mamba have attempted to address these shortcomings by proposing novel time-mixing and gating architectures, but pre-training large language models requires significant data and compute investments. Thus, the search for subquadratic architectures is limited by the availability of compute and quality pre-training datasets. |
|
As a cost-effective alternative to pre-training linear transformers, we propose Scalable UPtraining for Recurrent Attention (SUPRA). For more detail, refer to the |
|
[paper](https://arxiv.org/abs/2405.06640) |
|
|
|
|
|
Linear_Tiny_87M is a linear model that has been trained on a subset of redpajama dataset for 1 epoch on **1x A4000**. It took almost 4 hours for training to be completed. |
|
|
|
## Usage |
|
|
|
Just download the checkpoint and afterwards run the following code snippet: |
|
|
|
```python |
|
cd scripts |
|
|
|
python generate.py \ |
|
--model open_lm_87m \ |
|
--checkpoint /path/to/checkpoint.pt \ |
|
--positional-embedding-type head_rotary \ |
|
--input-text "Machine Learning is a" |
|
|
|
``` |
|
|
