title: Unity3D and OpenCV integration and passing each other camera frames
body: <p><strong>Background:</strong></p>
  
  <p>I've been working on an AR project using Unity3D and OpenCV. The integration has been done in the following way.</p>
  
  <p>Our OpenCV program and the OpenCV library itself compiles into a single dynamic library (<code>so</code> file Linux). (OpenCV library is linked to statically)</p>
  
  <p>Then, in Unity3D, in a C# script we import it (<code>DllImport</code>) and use its functionalities.</p>
  
  <p>So far everything looks clear, but the problem is here: Both parts (Unity3D and OpenCV) need every frames of the camera, so there are multiple ways to give them camera frames:</p>
  
  <ol>
  <li><p>Unity3D reads frames (<code>WebCamTexture</code>) and sends them via converting to three 2D arrays and passing to OpenCV part. (In this way, OpenCV part does not have access to camera)</p></li>
  <li><p>OpenCV part reads frames (<code>VideoCapture::open</code>) and sends back to Unity3D. (In this way, Unity3D does not have access to camera)</p></li>
  <li><p>Both parts are allowed to open the camera.</p></li>
  </ol>
  
  <p>Note that the major processing is done in the OpenCV part and Unity3D only needs frames to show a camera preview.</p>
  
  <p><strong>Question:</strong></p>
  
  <p>It's all about performance! Which approach is the best way in performance and being the best-practice point of view?</p>