README.md

---
license: mit
---

Converted [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) model in onnx fp16/int8 format for use with [Vespa Embedding](https://docs.vespa.ai/en/embedding.html).

- intfloat-multilingual-e5-large_fp16.onnx (fp16)
- intfloat-multilingual-e5-large_quantized.onnx (int8 quantized)

The model was quantized using the [optimum](https://github.com/huggingface/optimum) toolkit.

## Example of vespa services.xml:

**Notice**: FP16 works well with Vespa versions `8.325.46` and above.

```
<component id="me5_large" type="hugging-face-embedder">
    <transformer-model
        url="https://huggingface.co/hotchpotch/vespa-onnx-intfloat-multilingual-e5-large/resolve/main/intfloat-multilingual-e5-large_fp16.onnx" />
    <!-- or int8 quantization model
    <transformer-model
    url="https://huggingface.co/hotchpotch/vespa-onnx-intfloat-multilingual-e5-large/resolve/main/intfloat-multilingual-e5-large_quantized.onnx"
    />
    -->
    <tokenizer-model
        url="https://huggingface.co/hotchpotch/vespa-onnx-intfloat-multilingual-e5-large/resolve/main/tokenizer.json" />
    <normalize>true</normalize>
    <pooling-strategy>mean</pooling-strategy>
</component>
```

### deploy

```
# FP16 model has a larger file size, which can result in longer deployment times.
vespa deploy --wait 1800 .
```


## Tips: conver to int8 quantized

```
# https://github.com/vespa-engine/sample-apps/blob/master/simple-semantic-search/export_hf_model_from_hf.py
./export_hf_model_from_hf.py --hf_model intfloat/multilingual-e5-large --output_dir me5-large
```

```
optimum-cli onnxruntime quantize --onnx_model ./me5-large  -o me5-large-large_quantized --avx512_vnni
```


## Tips: convert to fp16

```
# https://github.com/vespa-engine/sample-apps/blob/master/simple-semantic-search/export_hf_model_from_hf.py
./export_hf_model_from_hf.py --hf_model intfloat/multilingual-e5-large --output_dir me5-large
```

- https://gist.github.com/hotchpotch/64fa52d32886fe61cc1d110066afef38

```
# https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/float16.py

import onnx
from onnxruntime.transformers.float16 import convert_float_to_float16

onnx_model = onnx.load("me5-large/intfloat-multilingual-e5-large.onnx")
model_fp16 = convert_float_to_float16(onnx_model, disable_shape_infer=True)
onnx.save(model_fp16, "me5-large/intfloat-multilingual-e5-large_fp16.onnx")
```

## License

The license for this model is based on the original license (found in the LICENSE file in the project's root directory), which is the MIT License.

- https://huggingface.co/intfloat/multilingual-e5-large

## Attribution

All credits for this model go to the authors of Multilingual-E5-large and the associated researchers and organizations. When using this model, please be sure to attribute the original authors.