title: Custom CoreML output layer that sums multiArray output
body: <p>Please bear with me. I'm new to CoreML and machine learning. I have a CoreML model that I was able to convert from a research paper implementation that used Caffe. It's a CSRNet, the objective being crowd-counting. After much wrangling, I'm able to load the MLmodel into Python using Coremltools, pre-process an image using Pillow and predict an output. The result is a MultiArray (from a density map), which I've then processed further to derive the actual numerical prediction.</p>
  <p>How do I add a custom layer as an output to the model that takes the current output and performs the following functionality? I've read numerous articles, and am still at a loss. (Essentially, it sums the values all the values in the MultiArray) I'd like to be able to save the model/ layer and import it into Xcode so that the MLModel result is a single numerical value, and not a MultiArray.</p>
  <p>This is the code I'm currently using to convert the output from the model into a number (in Python):</p>
  <pre><code># predict output
  
  output = model.predict({'data': img})
  summed_output = sum(output.values())
  prediction = np.sum(summed_output)
  print(&quot;prediction: &quot;, prediction)
  </code></pre>
  <p>Full (abbreviated) code:</p>
  <pre><code>import coremltools as ct
  from PIL import Image
  import numpy as np
  
  # instantiate model (CSRNet)
  model = ct.models.MLModel('shanghai_b.mlmodel')
  
  # function to resize image
  def load_image(path, resize_to=None):
      img = PIL.Image.open(path)
      if resize_to is not None:
          img = img.resize(resize_to, PIL.Image.ANTIALIAS)
      img_np = np.array(img).astype(np.float32)
      return img_np, img
  
  # select image 
  image = 'IMG_173.jpg'
  
  #resize image
  _, img = load_image(image, resize_to=(900, 675)) 
  
  # predict output
  output = model.predict({'data': img})
  summed_output = sum(output.values())
  prediction = np.sum(summed_output)
  print(&quot;prediction: &quot;, prediction)
  </code></pre>
  <p>Xcode shows the output for the MLModel as being: &quot;MultiArray (Double 1 x 168 x 225)&quot;. The spec description for the same model as it currently stands when I import it into python using Coremltools is as follows:</p>
  <pre><code>&lt;bound method MLModel.predict of input {
    name: &quot;data&quot;
    type {
      imageType {
        width: 224
        height: 224
        colorSpace: RGB
      }
    }
  }
  output {
    name: &quot;estdmap&quot;
    type {
      multiArrayType {
        dataType: DOUBLE
      }
    }
  }
  &gt;
  </code></pre>
  <p>Thanks for any help! I'm happy to post any other code in the process if it's useful.</p>
  <p>P.S. I'm adding the code I have inside my Xcode project as a reference as well.</p>
  <pre><code>private func detectImage(_ image: CIImage) {
  
          guard let model = try? VNCoreMLModel(for: HundredsPredictor().model) else {
              fatalError(&quot;Loading to CoreML failed&quot;) }
          
          let modelRequest = VNCoreMLRequest(model: model) { (request, error) in
              if error != nil {
                  print(error?.localizedDescription ?? &quot;Error&quot;)
              } else {
                  guard let result = request.results as? [VNObservation] else {fatalError(&quot;Error&quot;)}
                  
                  if #available(iOS 14.0, *) {
                      print(result)
                      
                      // output: [&lt;VNCoreMLFeatureValueObservation: 0x282069da0&gt; 344A87BC-B13E-4195-922E-7381694C91FF requestRevision=1 confidence=1.000000 timeRange={{0/1 = 0.000}, {0/1 = 0.000}} &quot;density_map&quot; - &quot;MultiArray: Double 1 × 168 × 225 array&quot; (1.000000)]
                      
                  } else {
                      // Fallback on earlier versions
                  }
                  if let firstResult = result.first {
                      print(firstResult)
  
                      // output: [&lt;VNCoreMLFeatureValueObservation: 0x282069da0&gt; 344A87BC-B13E-4195-922E-7381694C91FF requestRevision=1 confidence=1.000000 timeRange={{0/1 = 0.000}, {0/1 = 0.000}} &quot;density_map&quot; - &quot;MultiArray : Double 1 × 168 × 225 array&quot; (1.000000)]
  
                  }
              }
          }
  
          let handler = VNImageRequestHandler(ciImage: image)
          do {
              try handler.perform([modelRequest])
              print(handler)
          }
          catch let error as NSError {
              print(error)
          }
      }
  </code></pre>
  <h1><strong>Update: Solution</strong></h1>
  <p>In python:</p>
  <pre><code>from helpers import get_nn
  # helper file sourced from Matthijs Hollemans github
  # url: https://github.com/hollance/coreml-survival-guide/blob/master/Scripts/helpers.py
  
  # load original model
  spec = ct.utils.load_spec(&quot;HundredsPredictor.mlmodel&quot;)
  
  nn = get_nn(spec)
  
  #construct new layer
  new_layer = nn.layers.add()
  new_layer.name = &quot;summingLayer&quot;
  
  params = ct.proto.NeuralNetwork_pb2.ReduceLayerParams
  new_layer.reduce.mode = params.SUM
  new_layer.reduce.axis = params.CHW
  
  # append new layer to model
  new_layer.output.append(nn.layers[-2].output[0])
  nn.layers[-2].output[0] = nn.layers[-2].name + &quot;_output&quot;
  new_layer.input.append(nn.layers[-2].output[0])
  
  spec.description.output[0].type.multiArrayType.shape[0] = 1
  
  # save new model
  ct.models.utils.save_spec(spec, &quot;HundredPredictorSummed.mlmodel&quot;)
  
  </code></pre>
  <p>In Swift, after importing the new updated model:</p>
  <pre><code>private func detectImage(_ image: CIImage) {
          
          guard let model = try? VNCoreMLModel(for: HundredPredictorSummed().model) else {
              fatalError(&quot;Loading to CoreML failed&quot;) }
          
          let request = VNCoreMLRequest(model: model) { [weak self] request, error in
              guard let results = request.results as? [VNCoreMLFeatureValueObservation],
                    let topResult = results.first else {
                  fatalError(&quot;Unexpected result type from VNCoreMLRequest&quot;)}
              
              DispatchQueue.main.async {
  
                  guard let data = topResult.featureValue.multiArrayValue else { return }
  
                  let ptr = data.dataPointer.assumingMemoryBound(to: Double.self)
                  let sum = ptr[0]
                  print(&quot;SUM: &quot;, sum)
                  
                  self?.detectLabel.text = &quot;~\(String(Int(round(sum)))) ppl&quot;
  
              }
              
          }
          
          let handler = VNImageRequestHandler(ciImage: image)
          
          DispatchQueue.global(qos: .userInteractive).async {
              do {
                  try handler.perform([request])
              } catch {
                  print(error)
          
              }
          }
          
      }
  </code></pre>