title: Build confusion matrix for instance segmantation (mask r-cnn from detectron2)
body: <p>I've trained a mask r-cnn on corn images (I cannot show examples because they are confidential), but they are basically pictures of corn kernels scattered over a flat surface.</p>
  <p>There are different kinds of corn kernels I want to be able to segment and classify. I understand the AP metrics are the best way of measuring the performance of an instance segmentation algorithm and I know a confusion matrix for this kind of algorithm doesn't usually make sense.</p>
  <p>But for his specific case, where I have 4 classes of very similar objects, I would like to be able to set a fixed AP value, like AP50/AP75 and build a confusion matrix for that.</p>
  <p>Would it be possible? How would I do it?</p>
  <p>I used detectron2 library to train and get predictions. Here is the code I use to load my trained model from disk, generate predictions in the validation set, and visualize the results:</p>
  <pre><code>import detectron2
  from detectron2.utils.logger import setup_logger
  setup_logger()
  
  import numpy as np
  import matplotlib.pyplot as plt
  import os, json, cv2, random, gc
  
  from detectron2 import model_zoo
  from detectron2.data.datasets import register_coco_instances
  from detectron2.checkpoint import DetectionCheckpointer, Checkpointer
  from detectron2.data import MetadataCatalog, DatasetCatalog, build_detection_test_loader
  from detectron2.engine import DefaultTrainer, DefaultPredictor
  from detectron2.config import get_cfg
  from detectron2.utils.visualizer import Visualizer, ColorMode
  from detectron2.modeling import build_model
  from detectron2.evaluation import COCOEvaluator, inference_on_dataset
  
  train_annotations_path = &quot;./data/cvat-corn-train-coco-1.0/annotations/instances_default.json&quot;
  train_images_path = &quot;./data/cvat-corn-train-coco-1.0/images&quot;
  validation_annotations_path = &quot;./data/cvat-corn-validation-coco-1.0/annotations/instances_default.json&quot;
  validation_images_path = &quot;./data/cvat-corn-validation-coco-1.0/images&quot;
  
  cfg = get_cfg()
  cfg.merge_from_file(model_zoo.get_config_file(&quot;COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml&quot;))
  cfg.DATASETS.TRAIN = (&quot;train-corn&quot;,)
  cfg.DATASETS.TEST = (&quot;validation-corn&quot;,)
  cfg.DATALOADER.NUM_WORKERS = 2
  cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(&quot;COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml&quot;)  # Let training initialize from model zoo
  cfg.SOLVER.IMS_PER_BATCH = 2
  cfg.SOLVER.BASE_LR = 0.00025
  cfg.SOLVER.MAX_ITER = 10000
  cfg.SOLVER.STEPS = []
  cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128 
  cfg.MODEL.ROI_HEADS.NUM_CLASSES = 4
  cfg.OUTPUT_DIR = &quot;./output&quot;
  cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7
  
  register_coco_instances(
      &quot;train-corn&quot;,
      {},
      train_annotations_path,
      train_images_path
  )
  register_coco_instances(
      &quot;validation-corn&quot;,
      {},
      validation_annotations_path,
      validation_images_path
  )
  metadata_train = MetadataCatalog.get(&quot;train-corn&quot;)
  dataset_dicts = DatasetCatalog.get(&quot;train-corn&quot;)
  
  cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, &quot;model_final.pth&quot;)
  cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7
  predictor = DefaultPredictor(cfg)
  
  predicted_images_path = os.path.abspath(&quot;./predicted/&quot;)
  
  dataset_dicts_validation = DatasetCatalog.get(&quot;validation-corn&quot;)
  for d in dataset_dicts_validation:    
      im = cv2.imread(d[&quot;file_name&quot;])
      outputs = predictor(im)
      v = Visualizer(im[:, :, ::-1],
                     metadata=metadata_train, 
                     scale=0.5, 
                     instance_mode=ColorMode.IMAGE_BW
      )
      out = v.draw_instance_predictions(outputs[&quot;instances&quot;].to(&quot;cpu&quot;))
      fig = plt.figure(frameon=False, dpi=1)
      fig.set_size_inches(1024,1024)
      ax = plt.Axes(fig, [0., 0., 1., 1.])
      ax.set_axis_off()
      fig.add_axes(ax)
      ax.imshow(cv2.cvtColor(out.get_image()[:, :, ::-1], cv2.COLOR_BGR2RGB), aspect='auto')
      fig.savefig(f&quot;{predicted_images_path}/{d['file_name'].split('/')[-1]}&quot;)
  </code></pre>
  <p>That is what my output for a given image looks like:
  <a href="https://i.stack.imgur.com/om4W6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/om4W6.png" alt="enter image description here" /></a></p>
  <p>It is a dictionary with an Instances object as its only value, the Instances object has four lists: pred_boxes, scores, pred_classes and pred_masks. And can be visualized using the detectron2 visualizer, but I can't show the visualization for confidentiality reasons.</p>
  <p>Those are the metrics I have for the model right now:</p>
  <p><a href="https://i.stack.imgur.com/ebnw3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ebnw3.png" alt="enter image description here" /></a></p>
  <p>And for each class:
  <a href="https://i.stack.imgur.com/7qfwF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7qfwF.png" alt="enter image description here" /></a></p>
  <p>And I noticed visually that some of the kernels are being confused for other classes, specially between classes ardido and fermentado, that is why I want to somehow be able to build a confusion matrix.</p>
  <p>I expect the confusion matrix would look something like this:
  <a href="https://i.stack.imgur.com/bIHg6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bIHg6.png" alt="enter image description here" /></a></p>
  <p>EDIT:
  I found this repository:</p>
  <p><a href="https://github.com/kaanakan/object_detection_confusion_matrix" rel="nofollow noreferrer">https://github.com/kaanakan/object_detection_confusion_matrix</a></p>
  <p>And tried to use it:</p>
  <pre><code>from confusion_matrix import ConfusionMatrix
  cm = ConfusionMatrix(4, CONF_THRESHOLD=0.3, IOU_THRESHOLD=0.3)
  
  for d in dataset_dicts_validation:
      img = cv2.imread(d[&quot;file_name&quot;])
      outputs = predictor(img)
      labels = list()
      detections = list()
      for ann in d[&quot;annotations&quot;]:
          labels.append([ann[&quot;category_id&quot;]] + ann[&quot;bbox&quot;])
      for coord, conf, cls in zip(
          outputs[&quot;instances&quot;].get(&quot;pred_boxes&quot;).tensor.cpu().numpy(), 
          outputs[&quot;instances&quot;].get(&quot;scores&quot;).cpu().numpy(), 
          outputs[&quot;instances&quot;].get(&quot;pred_classes&quot;).cpu().numpy()
      ):
          detections.append(list(coord) + [conf] + [cls])
      cm.process_batch(np.array(detections), np.array(labels))
  </code></pre>
  <p>But the matrix I got is clearly wrong, and I'm having a hard time trying to fix it.
  <a href="https://i.stack.imgur.com/jsLEB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jsLEB.png" alt="Confusion Matrix" /></a></p>