| | --- |
| | pipeline_tag: translation |
| | library_name: comet |
| | language: |
| | - multilingual |
| | - af |
| | - am |
| | - ar |
| | - as |
| | - az |
| | - be |
| | - bg |
| | - bn |
| | - br |
| | - bs |
| | - ca |
| | - cs |
| | - cy |
| | - da |
| | - de |
| | - el |
| | - en |
| | - eo |
| | - es |
| | - et |
| | - eu |
| | - fa |
| | - fi |
| | - fr |
| | - fy |
| | - ga |
| | - gd |
| | - gl |
| | - gu |
| | - ha |
| | - he |
| | - hi |
| | - hr |
| | - hu |
| | - hy |
| | - id |
| | - is |
| | - it |
| | - ja |
| | - jv |
| | - ka |
| | - kk |
| | - km |
| | - kn |
| | - ko |
| | - ku |
| | - ky |
| | - la |
| | - lo |
| | - lt |
| | - lv |
| | - mg |
| | - mk |
| | - ml |
| | - mn |
| | - mr |
| | - ms |
| | - my |
| | - ne |
| | - nl |
| | - 'no' |
| | - om |
| | - or |
| | - pa |
| | - pl |
| | - ps |
| | - pt |
| | - ro |
| | - ru |
| | - sa |
| | - sd |
| | - si |
| | - sk |
| | - sl |
| | - so |
| | - sq |
| | - sr |
| | - su |
| | - sv |
| | - sw |
| | - ta |
| | - te |
| | - th |
| | - tl |
| | - tr |
| | - ug |
| | - uk |
| | - ur |
| | - uz |
| | - vi |
| | - xh |
| | - yi |
| | - zh |
| | license: apache-2.0 |
| | base_model: |
| | - FacebookAI/xlm-roberta-large |
| | --- |
| | |
| | # PreCOMET-diff [](https://arxiv.org/abs/2501.18251) |
| |
|
| | This is a source-only COMET model used for efficient evaluation subset selection. |
| | Specifically this model predicts `difficulty` distilled from an IRT model from up to WMT2022 (inclusive). |
| | The higher the scores, the better it is for evaluation because models will likely fail to translate the segment. |
| | It is not compatible with the original Unbabel's COMET and to run it you have to install [github.com/zouharvi/PreCOMET](https://github.com/zouharvi/PreCOMET): |
| | ```bash |
| | pip install pip3 install git+https://github.com/zouharvi/PreCOMET.git |
| | ``` |
| |
|
| | You can then use it in Python: |
| | ```python |
| | import precomet |
| | model = precomet.load_from_checkpoint(precomet.download_model("zouharvi/PreCOMET-diff")) |
| | model.predict([ |
| | {"src": "This is an easy source sentence."}, |
| | {"src": "this is a much more complicated source sen-tence that will pro·bably lead to loww scores 🤪"} |
| | ])["scores"] |
| | > [-0.3407433331012726, 0.6234546899795532] |
| | ``` |
| |
|
| | The primary use of this model is from the [subset2evaluate](https://github.com/zouharvi/subset2evaluate) package: |
| |
|
| | ```python |
| | import subset2evaluate |
| | |
| | data_full = subset2evaluate.utils.load_data("wmt23/en-cs") |
| | data_random = subset2evaluate.select_subset.basic(data_full, method="random") |
| | subset2evaluate.evaluate.eval_subset_clusters(data_random[:100]) |
| | > 1 |
| | subset2evaluate.evaluate.eval_subset_correlation(data_random[:100], data_full) |
| | > 0.71 |
| | ``` |
| | Random selection gives us only one cluster and system-level Spearman correlation of 0.71 when we have a budget for only 100 segments. However, by using this model: |
| | ```python |
| | data_precomet = subset2evaluate.select_subset.basic(data_full, method="precomet_diff") |
| | subset2evaluate.evaluate.eval_subset_clusters(data_precomet[:100]) |
| | > 1 |
| | subset2evaluate.evaluate.eval_subset_correlation(data_precomet[:100], data_full) |
| | > 0.93 |
| | ``` |
| | we get higher correlation. |
| | Note that this is not the best PreCOMET model and you can expect a bigger effect on a larger scale, as described in the paper. |
| |
|
| |
|
| | This work is described in [How to Select Datapoints for Efficient Human Evaluation of NLG Models?](https://arxiv.org/abs/2501.18251). |
| | Cite as: |
| | ``` |
| | @misc{zouhar2025selectdatapointsefficienthuman, |
| | title={How to Select Datapoints for Efficient Human Evaluation of NLG Models?}, |
| | author={Vilém Zouhar and Peng Cui and Mrinmaya Sachan}, |
| | year={2025}, |
| | eprint={2501.18251}, |
| | archivePrefix={arXiv}, |
| | primaryClass={cs.CL}, |
| | url={https://arxiv.org/abs/2501.18251}, |
| | } |
| | ``` |
