COMET-instant-confidence

This model is based on COMET-early-exit, which is a fork but not compatible with original Unbabel's COMET. To run the model, you need to first install this version of COMET either with:

pip install "git+https://github.com/zouharvi/COMET-early-exit#egg=comet-early-exit&subdirectory=comet_early_exit"

or in editable mode:

git clone https://github.com/zouharvi/COMET-early-exit.git
cd COMET-early-exit
pip3 install -e comet_early_exit

This model specifically behaves like standard quality estimation, but outputs two numbers: scores (as usual) and confidences, which is the estimated absolute error from the human score. Thus, contrary to expectations, higher "confidence" correponds to less correct QE estimation.

model = comet_early_exit.load_from_checkpoint(comet_early_exit.download_model("zouharvi/COMET-instant-confidence"))
data = [
    {
        "src": "Can I receive my food in 10 to 15 minutes?",
        "mt": "Moh bych obdržet jídlo v 10 do 15 minut?",
    },
    {
        "src": "Can I receive my food in 10 to 15 minutes?",
        "mt": "Mohl bych dostat jídlo během 10 či 15 minut?",
    }
]
model_output = model.predict(data, batch_size=8, gpus=1)
print("scores", model_output["scores"])
print("estimated errors", model_output["confidences"])

assert len(model_output["scores"]) == 2 and len(model_output["confidences"]) == 2

Outputs (formatted):

scores            72.71  88.56
estimated errors  15.63   9.74

This model is based on the work Early-Exit and Instant Confidence Translation Quality Estimation which can be cited as:

@misc{zouhar2025earlyexitinstantconfidencetranslation,
      title={Early-Exit and Instant Confidence Translation Quality Estimation}, 
      author={Vilém Zouhar and Maike Züfle and Beni Egressy and Julius Cheng and Jan Niehues},
      year={2025},
      eprint={2502.14429},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.14429}, 
}