README.md

---
license: apache-2.0
datasets:
- Shuu12121/rust-codesearch-dataset-open
- Shuu12121/java-codesearch-dataset-open
language:
- en
pipeline_tag: sentence-similarity
tags:
- code
- code-search
- retrieval
- sentence-similarity
- bert
- transformers
- deep-learning
- machine-learning
- nlp
- programming
- multi-language
- rust
- python
- java
- javascript
- php
- ruby
- go
---


# **CodeModernBERT-Owl**

## **概要 / Overview**

### **🦉 CodeModernBERT-Owl: 高精度なコード検索 & コード理解モデル**  
**CodeModernBERT-Owl** is a **pretrained model** designed from scratch for **code search and code understanding tasks**.  

Compared to previous versions such as **CodeHawks-ModernBERT** and **CodeMorph-ModernBERT**, this model **now supports Rust** and **improves search accuracy** in Python, PHP, Java, JavaScript, Go, and Ruby.  

### **🛠 主な特徴 / Key Features**
✅ **Supports long sequences up to 2048 tokens** (compared to Microsoft's 512-token models)  
✅ **Optimized for code search, code understanding, and code clone detection**  
✅ **Fine-tuned on GitHub open-source repositories (Java, Rust)**  
✅ **Achieves the highest accuracy among the CodeHawks/CodeMorph series**  
✅ **Multi-language support**: **Python, PHP, Java, JavaScript, Go, Ruby, and Rust**  

---

## **📊 モデルパラメータ / Model Parameters**
| パラメータ / Parameter   | 値 / Value |
|-------------------------|------------|
| **vocab_size**         | 50,000      |
| **hidden_size**        | 768        |
| **num_hidden_layers**  | 12         |
| **num_attention_heads**| 12         |
| **intermediate_size**  | 3,072      |
| **max_position_embeddings** | 2,048 |
| **type_vocab_size**    | 2          |
| **hidden_dropout_prob**| 0.1        |
| **attention_probs_dropout_prob** | 0.1 |
| **local_attention_window** | 128 |
| **rope_theta**         | 160,000    |
| **local_attention_rope_theta** | 10,000 |

---

## **💻 モデルの使用方法 / How to Use**
This model can be easily loaded using the **Hugging Face Transformers** library.  

⚠️ **Requires `transformers >= 4.48.0`**  

🔗 **[Colab Demo (Replace with "CodeModernBERT-Owl")](https://github.com/Shun0212/CodeBERTPretrained/blob/main/UseMyCodeMorph_ModernBERT.ipynb)**

### **モデルのロード / Load the Model**
```python
from transformers import AutoModelForMaskedLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Shuu12121/CodeModernBERT-Owl")
model = AutoModelForMaskedLM.from_pretrained("Shuu12121/CodeModernBERT-Owl")
```

### **コード埋め込みの取得 / Get Code Embeddings**
```python
import torch

def get_embedding(text, model, tokenizer, device="cuda"):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
    if "token_type_ids" in inputs:
        inputs.pop("token_type_ids")
    inputs = {k: v.to(device) for k, v in inputs.items()}
    outputs = model.model(**inputs)
    embedding = outputs.last_hidden_state[:, 0, :]
    return embedding

embedding = get_embedding("def my_function(): pass", model, tokenizer)
print(embedding.shape)
```

---

# **🔍 評価結果 / Evaluation Results**

### **データセット / Dataset**
📌 **Tested on `code_x_glue_ct_code_to_text` with a candidate pool size of 100.**  
📌 **Rust-specific evaluations were conducted using `Shuu12121/rust-codesearch-dataset-open`.**  

---

## **📈 主要な評価指標の比較（同一シード値）/ Key Evaluation Metrics (Same Seed)**
| 言語 / Language | **CodeModernBERT-Owl** | **CodeHawks-ModernBERT** | **Salesforce CodeT5+** | **Microsoft CodeBERT** | **GraphCodeBERT** |
|-----------|-----------------|----------------------|-----------------|------------------|------------------|
| **Python**     | **0.8793**  | 0.8551  | 0.8266  | 0.5243  | 0.5493  |
| **Java**       | **0.8880**  | 0.7971  | **0.8867**  | 0.3134  | 0.5879  |
| **JavaScript** | **0.8423**  | 0.7634  | 0.7628  | 0.2694  | 0.5051  |
| **PHP**        | **0.9129**  | 0.8578  | **0.9027**  | 0.2642  | 0.6225  |
| **Ruby**       | **0.8038**  | 0.7469  | **0.7568**  | 0.3318  | 0.5876  |
| **Go**         | **0.9386**  | 0.9043  | 0.8117  | 0.3262  | 0.4243  |

✅ **Achieves the highest accuracy in all target languages.**  
✅ **Significantly improved Java accuracy using additional fine-tuned GitHub data.**  
✅ **Outperforms previous models, especially in PHP and Go.**  

---

## **📊 Rust (独自データセット) / Rust Performance**
| 指標 / Metric | **CodeModernBERT-Owl** |
|--------------|----------------|
| **MRR**      | 0.7940 |
| **MAP**      | 0.7940 |
| **R-Precision** | 0.7173 |

### **📌 K別評価指標 / Evaluation Metrics by K**
| K  | **Recall@K** | **Precision@K** | **NDCG@K** | **F1@K** | **Success Rate@K** | **Query Coverage@K** |
|----|-------------|---------------|------------|--------|-----------------|-----------------|
| **1**   | 0.7173  | 0.7173  | 0.7173  | 0.7173  | 0.7173  | 0.7173  |
| **5**   | 0.8913  | 0.7852  | 0.8118  | 0.8132  | 0.8913  | 0.8913  |
| **10**  | 0.9333  | 0.7908  | 0.8254  | 0.8230  | 0.9333  | 0.9333  |
| **50**  | 0.9887  | 0.7938  | 0.8383  | 0.8288  | 0.9887  | 0.9887  |
| **100** | 1.0000  | 0.7940  | 0.8401  | 0.8291  | 1.0000  | 1.0000  |

---

## **📝 結論 / Conclusion**
✅ **Top performance in all languages**  
✅ **Rust support successfully added through dataset augmentation**  
✅ **Further performance improvements possible with better datasets**  

---

## **📜 ライセンス / License**
📄 **Apache 2.0**  

## **📧 連絡先 / Contact**
📩 **For any questions, please contact:**  
📧 **[email protected]**