## Computer Vision Interpret

[`vision.interpret`](/vision.interpret.html#vision.interpret) is the module that implements custom [`Interpretation`](/train.html#Interpretation) classes for different vision tasks by inheriting from it.


```python
from fastai.gen_doc.nbdoc import *
from fastai.vision import *
from fastai.vision.interpret import *
```


```python
show_doc(SegmentationInterpretation)
```


<h2 id="SegmentationInterpretation" class="doc_header"><code>class</code> <code>SegmentationInterpretation</code><a href="https://github.com/fastai/fastai/blob/master/fastai/vision/interpret.py#L10" class="source_link" style="float:right">[source]</a><a class="source_link" data-toggle="collapse" data-target="#SegmentationInterpretation-pytest" style="float:right; padding-right:10px">[test]</a></h2>

> <code>SegmentationInterpretation</code>(**`learn`**:[`Learner`](/basic_train.html#Learner), **`preds`**:`Tensor`, **`y_true`**:`Tensor`, **`losses`**:`Tensor`, **`ds_type`**:[`DatasetType`](/basic_data.html#DatasetType)=***`<DatasetType.Valid: 2>`***) :: [`Interpretation`](/train.html#Interpretation)

<div class="collapse" id="SegmentationInterpretation-pytest"><div class="card card-body pytest_card"><a type="button" data-toggle="collapse" data-target="#SegmentationInterpretation-pytest" class="close" aria-label="Close"><span aria-hidden="true">&times;</span></a><p>No tests found for <code>SegmentationInterpretation</code>. To contribute a test please refer to <a href="/dev/test.html">this guide</a> and <a href="https://forums.fast.ai/t/improving-expanding-functional-tests/32929">this discussion</a>.</p></div></div>

Interpretation methods for segmenatation models.  



```python
show_doc(SegmentationInterpretation.top_losses)
```


<h4 id="SegmentationInterpretation.top_losses" class="doc_header"><code>top_losses</code><a href="https://github.com/fastai/fastai/blob/master/fastai/vision/interpret.py#L19" class="source_link" style="float:right">[source]</a><a class="source_link" data-toggle="collapse" data-target="#SegmentationInterpretation-top_losses-pytest" style="float:right; padding-right:10px">[test]</a></h4>

> <code>top_losses</code>(**`sizes`**:`Tuple`, **`k`**:`int`=***`None`***, **`largest`**=***`True`***)

<div class="collapse" id="SegmentationInterpretation-top_losses-pytest"><div class="card card-body pytest_card"><a type="button" data-toggle="collapse" data-target="#SegmentationInterpretation-top_losses-pytest" class="close" aria-label="Close"><span aria-hidden="true">&times;</span></a><p>No tests found for <code>top_losses</code>. To contribute a test please refer to <a href="/dev/test.html">this guide</a> and <a href="https://forums.fast.ai/t/improving-expanding-functional-tests/32929">this discussion</a>.</p></div></div>

Reduce flatten loss to give a single loss value for each image  



```python
show_doc(SegmentationInterpretation._interp_show)
```


<h4 id="SegmentationInterpretation._interp_show" class="doc_header"><code>_interp_show</code><a href="https://github.com/fastai/fastai/blob/master/fastai/vision/interpret.py#L24" class="source_link" style="float:right">[source]</a><a class="source_link" data-toggle="collapse" data-target="#SegmentationInterpretation-_interp_show-pytest" style="float:right; padding-right:10px">[test]</a></h4>

> <code>_interp_show</code>(**`ims`**:[`ImageSegment`](/vision.image.html#ImageSegment), **`classes`**:`Collection`\[`T_co`\]=***`None`***, **`sz`**:`int`=***`20`***, **`cmap`**=***`'tab20'`***, **`title_suffix`**:`str`=***`None`***)

<div class="collapse" id="SegmentationInterpretation-_interp_show-pytest"><div class="card card-body pytest_card"><a type="button" data-toggle="collapse" data-target="#SegmentationInterpretation-_interp_show-pytest" class="close" aria-label="Close"><span aria-hidden="true">&times;</span></a><p>No tests found for <code>_interp_show</code>. To contribute a test please refer to <a href="/dev/test.html">this guide</a> and <a href="https://forums.fast.ai/t/improving-expanding-functional-tests/32929">this discussion</a>.</p></div></div>

Show ImageSegment with color mapping labels  



```python
show_doc(SegmentationInterpretation.show_xyz)
```


<h4 id="SegmentationInterpretation.show_xyz" class="doc_header"><code>show_xyz</code><a href="https://github.com/fastai/fastai/blob/master/fastai/vision/interpret.py#L53" class="source_link" style="float:right">[source]</a><a class="source_link" data-toggle="collapse" data-target="#SegmentationInterpretation-show_xyz-pytest" style="float:right; padding-right:10px">[test]</a></h4>

> <code>show_xyz</code>(**`i`**, **`classes`**:`list`=***`None`***, **`sz`**=***`10`***)

<div class="collapse" id="SegmentationInterpretation-show_xyz-pytest"><div class="card card-body pytest_card"><a type="button" data-toggle="collapse" data-target="#SegmentationInterpretation-show_xyz-pytest" class="close" aria-label="Close"><span aria-hidden="true">&times;</span></a><p>No tests found for <code>show_xyz</code>. To contribute a test please refer to <a href="/dev/test.html">this guide</a> and <a href="https://forums.fast.ai/t/improving-expanding-functional-tests/32929">this discussion</a>.</p></div></div>

show (image, true and pred) from self.ds with color mappings, optionally only plot  



```python
show_doc(SegmentationInterpretation._generate_confusion)
```


<h4 id="SegmentationInterpretation._generate_confusion" class="doc_header"><code>_generate_confusion</code><a href="https://github.com/fastai/fastai/blob/master/fastai/vision/interpret.py#L60" class="source_link" style="float:right">[source]</a><a class="source_link" data-toggle="collapse" data-target="#SegmentationInterpretation-_generate_confusion-pytest" style="float:right; padding-right:10px">[test]</a></h4>

> <code>_generate_confusion</code>()

<div class="collapse" id="SegmentationInterpretation-_generate_confusion-pytest"><div class="card card-body pytest_card"><a type="button" data-toggle="collapse" data-target="#SegmentationInterpretation-_generate_confusion-pytest" class="close" aria-label="Close"><span aria-hidden="true">&times;</span></a><p>No tests found for <code>_generate_confusion</code>. To contribute a test please refer to <a href="/dev/test.html">this guide</a> and <a href="https://forums.fast.ai/t/improving-expanding-functional-tests/32929">this discussion</a>.</p></div></div>

Average and Per Image Confusion: intersection of pixels given a true label, true label sums to 1  



```python
show_doc(SegmentationInterpretation._plot_intersect_cm)
```


<h4 id="SegmentationInterpretation._plot_intersect_cm" class="doc_header"><code>_plot_intersect_cm</code><a href="https://github.com/fastai/fastai/blob/master/fastai/vision/interpret.py#L79" class="source_link" style="float:right">[source]</a><a class="source_link" data-toggle="collapse" data-target="#SegmentationInterpretation-_plot_intersect_cm-pytest" style="float:right; padding-right:10px">[test]</a></h4>

> <code>_plot_intersect_cm</code>(**`cm`**, **`title`**=***`'Intersection with Predict given True'`***)

<div class="collapse" id="SegmentationInterpretation-_plot_intersect_cm-pytest"><div class="card card-body pytest_card"><a type="button" data-toggle="collapse" data-target="#SegmentationInterpretation-_plot_intersect_cm-pytest" class="close" aria-label="Close"><span aria-hidden="true">&times;</span></a><p>No tests found for <code>_plot_intersect_cm</code>. To contribute a test please refer to <a href="/dev/test.html">this guide</a> and <a href="https://forums.fast.ai/t/improving-expanding-functional-tests/32929">this discussion</a>.</p></div></div>

Plot confusion matrices: self.mean_cm or self.single_img_cm generated by `_generate_confusion`  


Let's show how [`SegmentationInterpretation`](/vision.interpret.html#SegmentationInterpretation) can be used once we train a segmentation model.

### train 


```python
camvid = untar_data(URLs.CAMVID_TINY)
path_lbl = camvid/'labels'
path_img = camvid/'images'
```


```python
codes = np.loadtxt(camvid/'codes.txt', dtype=str)
get_y_fn = lambda x: path_lbl/f'{x.stem}_P{x.suffix}'
```


```python
data = (SegmentationItemList.from_folder(path_img)
        .split_by_rand_pct()
        .label_from_func(get_y_fn, classes=codes)
        .transform(get_transforms(), tfm_y=True, size=128)
        .databunch(bs=16, path=camvid)
        .normalize(imagenet_stats))
```


```python
data.show_batch(rows=2, figsize=(7,5))
```


```python
learn = unet_learner(data, models.resnet18)
learn.fit_one_cycle(3,1e-2)
learn.save('mini_train')
```


<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: left;">
      <th>epoch</th>
      <th>train_loss</th>
      <th>valid_loss</th>
      <th>time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>0</td>
      <td>10.024513</td>
      <td>3.442348</td>
      <td>00:15</td>
    </tr>
    <tr>
      <td>1</td>
      <td>6.325253</td>
      <td>2.343699</td>
      <td>00:03</td>
    </tr>
    <tr>
      <td>2</td>
      <td>4.759998</td>
      <td>2.108100</td>
      <td>00:02</td>
    </tr>
  </tbody>
</table>



```python
jekyll_warn("Following results will not make much sense with this underperforming model but functionality will be explained with ease")
```


<div markdown="span" class="alert alert-danger" role="alert"><i class="fa fa-danger-circle"></i> <b>Warning: </b>Following results will not make much sense with this underperforming model but functionality will be explained with ease</div>


### interpret


```python
interp = SegmentationInterpretation.from_learner(learn)
```

Since `FlattenedLoss of CrossEntropyLoss()` is used we reshape and then take the mean of pixel losses per image. In order to do so we need to pass `sizes:tuple` to `top_losses()`


```python
top_losses, top_idxs = interp.top_losses(sizes=(128,128))
```


```python
top_losses, top_idxs
```




    (tensor([3.3195, 3.1692, 2.6574, 2.5976, 2.4910, 2.3759, 2.3710, 2.2064, 2.0871,
             2.0834, 2.0479, 1.8645, 1.8412, 1.7956, 1.7013, 1.6126, 1.6015, 1.5470,
             1.4495, 1.3423]),
     tensor([12,  4, 17, 13, 19, 18,  7,  8, 10,  1, 15,  0,  2,  9, 16, 11, 14,  5,
              6,  3]))




```python
plt.hist(to_np(top_losses), bins=20);plt.title("Loss Distribution");
```

Next, we can generate a confusion matrix similar to what we usually have for classification. Two confusion matrices are generated: `mean_cm` which represents the global label performance and `single_img_cm` which represents the same thing but for each individual image in dataset.

Values in the matrix are calculated as:

\begin{align}
\ CM_{ij} & = IOU(Predicted , True | True) \\
\end{align}

Or in plain english: ratio of pixels of predicted label given the true pixels


```python
learn.data.classes
```




    array(['Animal', 'Archway', 'Bicyclist', 'Bridge', 'Building', 'Car', 'CartLuggagePram', 'Child', 'Column_Pole',
           'Fence', 'LaneMkgsDriv', 'LaneMkgsNonDriv', 'Misc_Text', 'MotorcycleScooter', 'OtherMoving', 'ParkingBlock',
           'Pedestrian', 'Road', 'RoadShoulder', 'Sidewalk', 'SignSymbol', 'Sky', 'SUVPickupTruck', 'TrafficCone',
           'TrafficLight', 'Train', 'Tree', 'Truck_Bus', 'Tunnel', 'VegetationMisc', 'Void', 'Wall'], dtype='<U17')




```python
mean_cm, single_img_cm = interp._generate_confusion()
```


```python
mean_cm.shape, single_img_cm.shape
```




    ((32, 32), (20, 32, 32))



`_plot_intersect_cm` first displays a dataframe showing per class score using the IOU definition we made earlier. These are the diagonal values from the confusion matrix which is displayed after. 

`NaN` indicate that these labels were not present in our dataset, in this case validation set. As you can imagine it also helps you to maybe construct a better representing validation set.


```python
df = interp._plot_intersect_cm(mean_cm, "Mean of Ratio of Intersection given True Label")
```

Next let's look at the single worst prediction in our dataset. It looks like this dummy model just predicts everything as `Road` :)


```python
i = top_idxs[0]
df = interp._plot_intersect_cm(single_img_cm[i], f"Ratio of Intersection given True Label, Image:{i}")
```

Finally we will visually inspect this single prediction


```python
interp.show_xyz(i, sz=15)
```


```python
jekyll_warn("""With matplotlib colormaps the max number of unique qualitative colors is 20. 
So if len(classes) > 20 then close class indexes may be plotted with the same color.
Let's fix this together :)""")
```


<div markdown="span" class="alert alert-danger" role="alert"><i class="fa fa-danger-circle"></i> <b>Warning: </b>With matplotlib colormaps the max number of unique qualitative colors is 20. 
So if len(classes) > 20 then close class indexes may be plotted with the same color.
Let's fix this together :)</div>



```python
interp.c2i
```




    {'Animal': 0,
     'Archway': 1,
     'Bicyclist': 2,
     'Bridge': 3,
     'Building': 4,
     'Car': 5,
     'CartLuggagePram': 6,
     'Child': 7,
     'Column_Pole': 8,
     'Fence': 9,
     'LaneMkgsDriv': 10,
     'LaneMkgsNonDriv': 11,
     'Misc_Text': 12,
     'MotorcycleScooter': 13,
     'OtherMoving': 14,
     'ParkingBlock': 15,
     'Pedestrian': 16,
     'Road': 17,
     'RoadShoulder': 18,
     'Sidewalk': 19,
     'SignSymbol': 20,
     'Sky': 21,
     'SUVPickupTruck': 22,
     'TrafficCone': 23,
     'TrafficLight': 24,
     'Train': 25,
     'Tree': 26,
     'Truck_Bus': 27,
     'Tunnel': 28,
     'VegetationMisc': 29,
     'Void': 30,
     'Wall': 31}




```python
show_doc(ObjectDetectionInterpretation)
```


<h2 id="ObjectDetectionInterpretation" class="doc_header"><code>class</code> <code>ObjectDetectionInterpretation</code><a href="https://github.com/fastai/fastai/blob/master/fastai/vision/interpret.py#L101" class="source_link" style="float:right">[source]</a><a class="source_link" data-toggle="collapse" data-target="#ObjectDetectionInterpretation-pytest" style="float:right; padding-right:10px">[test]</a></h2>

> <code>ObjectDetectionInterpretation</code>(**`learn`**:[`Learner`](/basic_train.html#Learner), **`preds`**:`Tensor`, **`y_true`**:`Tensor`, **`losses`**:`Tensor`, **`ds_type`**:[`DatasetType`](/basic_data.html#DatasetType)=***`<DatasetType.Valid: 2>`***) :: [`Interpretation`](/train.html#Interpretation)

<div class="collapse" id="ObjectDetectionInterpretation-pytest"><div class="card card-body pytest_card"><a type="button" data-toggle="collapse" data-target="#ObjectDetectionInterpretation-pytest" class="close" aria-label="Close"><span aria-hidden="true">&times;</span></a><p>No tests found for <code>ObjectDetectionInterpretation</code>. To contribute a test please refer to <a href="/dev/test.html">this guide</a> and <a href="https://forums.fast.ai/t/improving-expanding-functional-tests/32929">this discussion</a>.</p></div></div>

Interpretation methods for classification models.  



```python
jekyll_warn("ObjectDetectionInterpretation is not implemented yet. Feel free to implement it :)")
```


<div markdown="span" class="alert alert-danger" role="alert"><i class="fa fa-danger-circle"></i> <b>Warning: </b>ObjectDetectionInterpretation is not implemented yet. Feel free to implement it :)</div>


## Undocumented Methods - Methods moved below this line will intentionally be hidden

## New Methods - Please document or move to the undocumented section
