This model was contributed by [vasudevgupta](https://huggingface.co/vasudevgupta). The original code can be found
[here](https://github.com/google-research/bigbird).

## Usage tips

- For an in-detail explanation on how BigBird's attention works, see [this blog post](https://huggingface.co/blog/big-bird).
- BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using
  **original_full** is advised as there is no benefit in using **block_sparse** attention.
- The code currently uses window size of 3 blocks and 2 global blocks.
- Sequence length must be divisible by block size.
- Current implementation supports only **ITC**.
- Current implementation doesn't support **num_random_blocks = 0**
- BigBird is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.


## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## BigBirdConfig

[[autodoc]] BigBirdConfig

## BigBirdTokenizer

[[autodoc]] BigBirdTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## BigBirdTokenizerFast

[[autodoc]] BigBirdTokenizerFast

## BigBird specific outputs

[[autodoc]] models.big_bird.modeling_big_bird.BigBirdForPreTrainingOutput

<frameworkcontent>
<pt>

## BigBirdModel

[[autodoc]] BigBirdModel
    - forward

## BigBirdForPreTraining

[[autodoc]] BigBirdForPreTraining
    - forward

## BigBirdForCausalLM

[[autodoc]] BigBirdForCausalLM
    - forward

## BigBirdForMaskedLM

[[autodoc]] BigBirdForMaskedLM
    - forward
The context provides a clear and direct answer to the question. It states that for sequence lengths less than 1024, the advised implementation of BigBird is "original_full" because there is no benefit in using "block_sparse" attention for these shorter sequences. This information is explicitly mentioned in the usage tips section of the context.


This question is quite useful for machine learning developers working with NLP applications in the Hugging Face ecosystem. BigBird is a transformer model designed to handle long sequences efficiently, and understanding its implementation for shorter sequences can help developers optimize their models for different use cases. The question addresses practical implementation details, which are crucial for developers aiming to leverage BigBird's capabilities effectively. However, it is somewhat specific and may not apply to all developers, which slightly limits its general usefulness.


The question is asking about the advised implementation of BigBird for sequence lengths less than 1024. It is clear and specific, referring to a particular model (BigBird) and a specific condition (sequence lengths less than 1024). An operator with access to documentation or knowledge about BigBird would understand what is being asked without needing additional context. Therefore, the question is context-independent.

