title: Using CoreML to classify NSImages
body: <p>I'm trying to work with Xcode CoreML to classify images that are simply single digits or letters. To start out with I'm just usiing .png images of digits. Using Create ML tool, I built an image classifier (NOT including any Vision support stuff) and provided a set of about 300 training images and separate set of 50 testing images. When I run this model, it trains and tests successfully and generates a model. Still within the tool I access the model and feed it another set of 100 images to classify. It works properly, identifying 98 of them corrrectly.</p>
  <p>Then I created a Swift sample program to access the model (from the Mac OS X single view template); it's set up to accept a dropped image file and then access the model's prediction method and print the result. The problem is that the model expects an object of type CVPixelBuffer and I'm not sure how to properly create this from NSImage. I found some reference code and incorported but when I actually drag my classification images to the app it's only about 50% accurate. So I'm wondering if anyone has any experience with this type of model. It would be nice if there were a way to look at the &quot;Create ML&quot; source code to see how it processes a dropped image when predicting from the model.</p>
  <p>The code for processing the image and invoking model prediction method is:</p>
  <pre><code>// initialize the model
  mlModel2 = MLSample()   //MLSample is model generated by ML Create tool and imported to project
  // prediction logic for the image
  // (included in a func)
  //
  let fimage = NSImage.init(contentsOfFile: fname)    //fname is obtained from dropped file
  do {
      let fcgImage = fimage.cgImage(forProposedRect: nil, context: nil, hints: nil)
      let imageConstraint = mlModel2?.model.modelDescription.inputDescriptionsByName[&quot;image&quot;]?.imageConstraint
      let featureValue = try MLFeatureValue(cgImage: fcgImage!, constraint: imageConstraint!, options: nil)
      let pxbuf = featureValue.imageBufferValue
      let mro = try mlModel2?.prediction(image: pxbuf!)
      if mro != nil {
             let mroLbl = mro!.classLabel
             let mroProb = mro!.classLabelProbs[mroLbl] ?? 0.0
             print(String.init(format: &quot;M2 MLFeature: %@ %5.2f&quot;, mroLbl, mroProb))
             return
      }
  }
  catch {
       print(error.localizedDescription)
  }
  return
  </code></pre>