File size: 19,138 Bytes
a37ced9
 
 
 
 
a9077b6
a37ced9
 
 
65555b5
a37ced9
8c5bcdc
 
 
 
 
 
a37ced9
 
 
 
 
8c5bcdc
 
a37ced9
a9077b6
a37ced9
8c5bcdc
a37ced9
8c5bcdc
 
 
 
 
 
 
a37ced9
85cc80e
8c5bcdc
a37ced9
8c5bcdc
 
 
a37ced9
8c5bcdc
a37ced9
8c5bcdc
 
 
 
 
a37ced9
8c5bcdc
 
a37ced9
8c5bcdc
 
a37ced9
8c5bcdc
a942142
a37ced9
8c5bcdc
a37ced9
8c5bcdc
a37ced9
74201a3
a37ced9
a9077b6
a37ced9
 
 
 
 
dda4e58
65555b5
8c5bcdc
 
 
 
 
 
a9077b6
8c5bcdc
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
a37ced9
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
---
library_name: transformers
tags: []
---

# Model Details

### Model Description

MiewID-msv2 is a feature extractor trained for re-identification using contrastive learning on a large, high-quality dataset of 54 wildlife species - terrestrial and aquatic - including fins, flukes, flanks, faces.

- **Model Type:** Wildlife re-identification feature backbone
- **Model Stats:**
  - Params (M): 51.11
  - GMACs: 24.38
  - Activations (M): 91.11
  - Image size: 440 x 440

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** https://github.com/WildMeOrg/wbia-plugin-miew-id
- **Backbone:** https://huggingface.co/timm/efficientnetv2_rw_m.agc_in1k

# Usage

Intended use is re-identification of individuals from different species by matching against a database of grount-truth samples. Model features can also be used for species classification by retrieval.

### Embedding Extraction
```
import numpy as np
from PIL import Image
import torch
import torchvision.transforms as transforms
from transformers import AutoModel

model_tag = f"conservationxlabs/miewid-msv2"
model = AutoModel.from_pretrained(model_tag, trust_remote_code=True)

def generate_random_image(height=440, width=440, channels=3):
    random_image = np.random.randint(0, 256, (height, width, channels), dtype=np.uint8)
    return Image.fromarray(random_image)

random_image = generate_random_image()

preprocess = transforms.Compose([
    transforms.Resize((440, 440)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

input_tensor = preprocess(random_image)
input_batch = input_tensor.unsqueeze(0) 

with torch.no_grad():
    output = model(input_batch)

print(output)
print(output.shape)

```

### More Examples

View more usage examples at https://github.com/WildMeOrg/wbia-plugin-miew-id/tree/main/wbia_miew_id/examples

# Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The dataset used for these experiments was a combination of data from Wildbook platforms (multiple users), Happywhale Kaggle competitions multi-species dataset and multiple publicly available datasets. A small subset of data from Wildbook platforms is available at https://lila.science/datasets. 


### Example images


![image/jpg](https://cthulhu.dyn.wildme.io/public/datasets/msv2_dataset_example.jpg)

# Evaluation Results

The multi-species model shows improvements over models trained on single species. Moreover the model shows strong generalization ability for majority of the species when trained and evaluated in a leave-one-out fashion.

### Detailed results

|Source        |group                           |map  |rank-1|rank-5|rank-10|rank-20|viewpoints                                                                                                          |n_test_annots|n_test_names|n_train_annots|n_train_names|
|--------------|--------------------------------|-----|------|------|-------|-------|--------------------------------------------------------------------------------------------------------------------|-------------|------------|--------------|-------------|
|Wildbook      |amur_tiger                      |91.8 |100.0 |100.0 |100.0  |100.0  |['left', 'right']                                                                                                   |233          |47          |691           |75           |
|Wildbook      |beluga_whale                    |61.36|71.85 |81.63 |85.51  |89.28  |['up']                                                                                                              |849          |228         |2767          |354          |
|Happywhale    |blue_whale                      |36.98|38.68 |55.11 |61.52  |69.04  |['unknown']                                                                                                         |998          |392         |1185          |339          |
|Wildbook      |bottlenose_dolphin              |85.69|93.99 |96.18 |96.85  |97.52  |['right', 'left']                                                                                                   |4911         |838         |11607         |1062         |
|Happywhale    |brydes_whale                    |69.96|80.95 |92.86 |95.24  |100.0  |['unknown']                                                                                                         |42           |9           |81            |15           |
|Lomas Capuchin|capuchin                        |34.8 |47.97 |68.58 |76.69  |85.14  |['front']                                                                                                           |296          |35          |1708          |29           |
|Wildbook      |cheetah                         |57.37|71.15 |84.92 |87.87  |92.13  |['left', 'right']                                                                                                   |624          |139         |1522          |144          |
|Wildbook      |chimpanzee                      |94.7 |100.0 |100.0 |100.0  |100.0  |['unknown']                                                                                                         |52           |8           |138           |9            |
|C-Tai         |chimpanzee_ctai                 |51.67|69.64 |83.68 |89.18  |94.12  |['front']                                                                                                           |527          |63          |2680          |44           |
|C-Zoo         |chimpanzee_czoo                 |77.66|86.67 |93.33 |97.5   |99.17  |['front']                                                                                                           |240          |24          |1258          |17           |
|Happywhale    |commersons_dolphin              |0.0  |0.0   |0.0   |0.0    |0.0    |['unknown']                                                                                                         |10           |5           |7             |2            |
|Happywhale    |cuviers_beaked_whale            |52.89|55.71 |75.71 |82.86  |90.0   |['unknown']                                                                                                         |70           |28          |89            |21           |
|DogFaceNet    |dog                             |82.96|87.38 |94.78 |96.74  |97.99  |['unknown']                                                                                                         |1838         |544         |5750          |1088         |
|Happywhale    |dusky_dolphin                   |85.82|83.54 |93.04 |94.3   |94.94  |['unknown']                                                                                                         |158          |68          |154           |41           |
|Wildbook      |eurasianlynx                    |58.8 |67.82 |83.78 |88.83  |92.02  |['left', 'right']                                                                                                   |726          |199         |2251          |285          |
|Happywhale    |fin_whale                       |67.65|76.92 |87.18 |90.11  |93.04  |['unknown']                                                                                                         |273          |71          |533           |55           |
|Happywhale    |frasiers_dolphin                |0.0  |0.0   |0.0   |0.0    |0.0    |                                                                                                                    |-1           |-1          |-1            |-1           |
|Wildbook      |giraffe                         |99.18|99.08 |99.08 |100.0  |100.0  |['right', 'front', 'left', 'back']                                                                                  |423          |162         |2540          |426          |
|Wildbook      |horse_wild+face                 |0.0  |0.0   |0.0   |0.0    |0.0    |['front']                                                                                                           |17           |7           |174           |53           |
|Wildbook      |horse_wild_tunisian+face        |79.59|98.53 |99.51 |100.0  |100.0  |['unknown']                                                                                                         |204          |39          |999           |37           |
|Wildbook      |hyena                           |64.17|79.32 |89.22 |93.14  |96.41  |['right', 'left']                                                                                                   |1231         |301         |1154          |201          |
|Wildbook      |hyperoodon_ampullatus           |91.14|96.11 |98.12 |98.34  |98.84  |['left', 'right']                                                                                                   |1930         |354         |24952         |759          |
|Wildbook      |jaguar                          |67.45|81.72 |91.04 |92.91  |94.4   |['left', 'right']                                                                                                   |540          |124         |980           |103          |
|PrimFace      |japanese_monkey                 |89.05|93.85 |98.46 |98.46  |100.0  |['unknown']                                                                                                         |65           |12          |219           |14           |
|Happywhale    |killer_whale                    |62.82|77.13 |85.48 |88.81  |91.82  |['unknown']                                                                                                         |599          |145         |1379          |177          |
|Wildbook      |leopard                         |65.03|82.64 |90.23 |92.83  |94.46  |['right', 'left']                                                                                                   |595          |108         |4261          |381          |
|Wildbook      |lion                            |87.12|93.86 |100.0 |100.0  |100.0  |['front', 'unknown', 'frontright', 'frontleft', 'upfront', 'right', 'upleft', 'left', 'upright', 'backleft', 'back']|1364         |242         |3981          |241          |
|Happywhale    |long_finned_pilot_whale         |99.06|98.11 |100.0 |100.0  |100.0  |['unknown']                                                                                                         |53           |21          |33            |10           |
|Wildbook      |lynx_pardinus                   |48.22|57.68 |73.44 |77.18  |84.96  |['left', 'right']                                                                                                   |467          |142         |1376          |259          |
|MacaqueFaces  |macaque_face                    |89.87|95.88 |97.94 |99.12  |100.0  |['front']                                                                                                           |340          |34          |3480          |25           |
|Happywhale    |melon_headed_whale              |90.8 |93.42 |96.05 |96.71  |98.68  |['unknown']                                                                                                         |152          |55          |180           |43           |
|Wildbook      |nyala                           |64.64|70.65 |90.05 |94.03  |97.01  |['left', 'right']                                                                                                   |418          |99          |1303          |149          |
|Happywhale    |pantropic_spotted_dolphin       |78.15|85.19 |90.74 |94.44  |96.3   |['unknown']                                                                                                         |54           |11          |55            |8            |
|Happywhale    |pygmy_killer_whale              |89.07|84.62 |100.0 |100.0  |100.0  |['unknown']                                                                                                         |26           |10          |32            |5            |
|PrimFace      |rhesus_monkey                   |78.12|89.13 |97.28 |98.91  |100.0  |['unknown']                                                                                                         |184          |28          |478           |24           |
|Happywhale    |rough_toothed_dolphin           |0.0  |0.0   |0.0   |0.0    |0.0    |['unknown']                                                                                                         |6            |3           |7             |2            |
|SealID        |seal                            |37.93|59.51 |80.0  |86.1   |91.71  |['unknown']                                                                                                         |410          |41          |1448          |36           |
|Happywhale    |sei_whale                       |67.86|71.55 |87.07 |93.1   |95.69  |['unknown']                                                                                                         |116          |35          |106           |29           |
|Wildbook      |short_fin_pilot_whale+fin_dorsal|92.49|92.31 |98.08 |100.0  |100.0  |['right', 'left']                                                                                                   |272          |64          |1290          |137          |
|Happywhale    |short_finned_pilot_whale        |94.39|94.96 |95.8  |95.8   |97.48  |['unknown']                                                                                                         |119          |57          |67            |17           |
|Wildbook      |snow_leopard                    |62.43|82.45 |88.91 |92.15  |94.53  |['left', 'right']                                                                                                   |927          |144         |2307          |99           |
|Happywhale    |spinner_dolphin                 |99.88|100.0 |100.0 |100.0  |100.0  |['left', 'right']                                                                                                   |335          |97          |1562          |251          |
|Happywhale    |spotted_dolphin                 |84.92|83.67 |89.8  |95.92  |97.96  |['unknown']                                                                                                         |49           |22          |128           |28           |
|Wildbook      |turtle_green                    |100.0|100.0 |100.0 |100.0  |100.0  |['left', 'right', 'up', 'front']                                                                                    |543          |109         |5097          |76           |
|Wildbook      |turtle_green+head               |95.1 |96.02 |100.0 |100.0  |100.0  |['left', 'right', 'up', 'front']                                                                                    |544          |108         |5737          |76           |
|Wildbook      |turtle_hawksbill                |78.84|81.15 |88.37 |92.56  |95.81  |['up', 'left', 'right', 'front']                                                                                    |506          |106         |5397          |71           |
|Wildbook      |turtle_hawksbill+head           |86.78|91.74 |95.45 |96.69  |98.35  |['right', 'left', 'up', 'front']                                                                                    |589          |109         |7033          |71           |
|SeaTurtleID   |turtle_loggerhead_ext           |63.67|75.43 |86.61 |90.17  |92.87  |['unknown', 'left', 'top', 'right']                                                                                 |815          |217         |6573          |355          |
|SeaTurtleID   |turtle_loggerhead_ext+head      |94.14|94.63 |100.0 |100.0  |100.0  |['left', 'right', 'topright', 'top', 'topleft', 'below', 'front']                                                   |778          |209         |6361          |348          |
|Wildbook      |whale_fin+fin_dorsal            |82.12|85.12 |93.87 |95.04  |95.75  |['left', 'right']                                                                                                   |394          |110         |1629          |186          |
|Wildbook      |whale_grey                      |90.97|91.55 |95.54 |96.18  |96.82  |['right', 'left']                                                                                                   |381          |103         |5662          |758          |
|Wildbook      |whale_humpback+fin_dorsal       |80.3 |84.07 |90.08 |91.6   |95.42  |['right', 'left']                                                                                                   |509          |194         |3734          |541          |
|Wildbook      |whale_humpback+fluke            |97.11|96.67 |100.0 |100.0  |100.0  |['back']                                                                                                            |150          |64          |1151          |356          |
|Wildbook      |whale_orca                      |97.01|98.36 |99.18 |100.0  |100.0  |['right', 'left']                                                                                                   |249          |49          |2790          |435          |
|Wildbook      |whale_orca+fin_dorsal           |95.41|95.87 |97.52 |97.52  |99.17  |['right', 'left']                                                                                                   |623          |187         |3978          |430          |
|Wildbook      |whale_sperm+fluke               |96.14|97.87 |98.33 |98.48  |99.09  |['back']                                                                                                            |658          |125         |25849         |558          |
|Wildbook      |whaleshark                      |57.61|72.25 |79.96 |82.27  |87.21  |['left', 'right']                                                                                                   |997          |212         |5713          |376          |
|Wildbook      |white_shark+fin_dorsal          |91.4 |94.89 |97.87 |98.54  |99.27  |['left', 'right']                                                                                                   |322          |88          |1067          |162          |
|Happywhale    |white_sided_dolphin             |81.82|76.0  |88.0  |88.0   |100.0  |['unknown']                                                                                                         |25           |11          |13            |4            |
|Wildbook      |wilddog                         |73.4 |85.11 |90.93 |92.38  |94.07  |['left', 'right']                                                                                                   |3460         |848         |4217          |656          |
|Wildbook      |zebra_grevys                    |91.79|96.79 |97.57 |97.74  |98.04  |['right']                                                                                                           |1684         |332         |8927          |362          |