{"action":"created","issue":{"url":"https://api.github.com/repos/PDAL/PDAL/issues/1332","repository_url":"https://api.github.com/repos/PDAL/PDAL","labels_url":"https://api.github.com/repos/PDAL/PDAL/issues/1332/labels{/name}","comments_url":"https://api.github.com/repos/PDAL/PDAL/issues/1332/comments","events_url":"https://api.github.com/repos/PDAL/PDAL/issues/1332/events","html_url":"https://github.com/PDAL/PDAL/issues/1332","id":183574932,"number":1332,"title":"Feature: PDAL JVM bindings","user":{"login":"lossyrob","id":2320142,"avatar_url":"https://avatars.githubusercontent.com/u/2320142?v=3","gravatar_id":"","url":"https://api.github.com/users/lossyrob","html_url":"https://github.com/lossyrob","followers_url":"https://api.github.com/users/lossyrob/followers","following_url":"https://api.github.com/users/lossyrob/following{/other_user}","gists_url":"https://api.github.com/users/lossyrob/gists{/gist_id}","starred_url":"https://api.github.com/users/lossyrob/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lossyrob/subscriptions","organizations_url":"https://api.github.com/users/lossyrob/orgs","repos_url":"https://api.github.com/users/lossyrob/repos","events_url":"https://api.github.com/users/lossyrob/events{/privacy}","received_events_url":"https://api.github.com/users/lossyrob/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-10-18T02:27:54Z","updated_at":"2016-10-18T14:56:34Z","closed_at":null,"body":"There is currently not much support for reading point cloud data into and writing point cloud data out of JVM languages. By JVM languages I mean programming languages that compile down to JVM bytecode and execute via a Java Virtual Machine (this includes Java, Scala, Clojure, Groovy, and more). JVM bindings for PDAL would fill this gap in the open source ecosystem, and allow JVM-based projects to utilize not only it's point cloud io capabilities, but also it's extensive Pipeline architecture.\r\n\r\nPersonally, as a developer on the [GeoTrellis](https://github.com/geotrellis/geotrellis) project, I would like to take advantage of PDAL for reading point cloud data into [Apache Spark](http://spark.apache.org/) [RDDs](http://spark.apache.org/docs/latest/programming-guide.html#resilient-distributed-datasets-rdds) for distributed processing. The first step to a point-cloud-on-spark story is data ingestion, and I see utilizing PDAL to be the best path forward for that purpose.\r\n\r\nThis issue intends to track the creation of JVM bindings for PDAL.\r\n\r\n### Proposed Architecture\r\n\r\nThere are a variety of ways for the JVM to interact with native code out there (e.g. JNI, JNA, JavaCPP). After considering the options, I believe that implementing JNI bindings is the most clear way forward. If anyone has opinions on the matter, let's hear them.\r\n\r\nWe will need to expose a PDAL pipeline, very similarly to how the python bindings do this (see [Pipeline.cpp](https://github.com/PDAL/PDAL/blob/ff4b63e24ea7aebe2adb7c20dbf024f9191d5b2a/python/pdal/Pipeline.cpp))\r\n\r\nWe will not be able to directly translate the python binding code to JNI, however, since as we can see from the [return type of the getArrays method](https://github.com/PDAL/PDAL/blob/ff4b63e24ea7aebe2adb7c20dbf024f9191d5b2a/python/pdal/Pipeline.cpp#L44), the Arrays which are referenced are [actually NumPy arrays](https://github.com/PDAL/PDAL/blob/ff4b63e24ea7aebe2adb7c20dbf024f9191d5b2a/src/plang/Array.cpp#L145). Since NumPy is not accessible to the JVM, we will have to construct something separately.\r\n\r\n### Issue of language\r\n\r\nIt is possible to create these bindings in any JVM language. One might argue that it makes sense to write them in Java, as it's the most basic of the JVM languages. I mean \"basic\" not as some dig on Java, but because it's the base language for the JVM, and would not require any additional language-specific library dependencies to run the code. For instance, Scala code requires the `scala-lang` dependency in order to run on the JVM. Scala or other languages can also can create code that is not able to be called (without doing backflips) from Java.\r\n\r\nHowever, there is something to be said about coding this feature in another JVM language. If we code in another JVM language it is possible to make the interface into this feature in such a way that calling it from Java would be as simple as calling the same code written in Java. There's also the potential benefit of having the author of the feature write in a language other than Java, either because they are more used to the other language, or because Java can be a verbose and unexciting language to code in (that's an actual dig on Java). For instance, if I were to code this feature, coding it in Scala would be more familiar to me, much more interesting, and would expose a set of functionality that may be useful (for instance I believe we might be able to leverage [scodec](http://scodec.org/) for unpacking point data).\r\n\r\nTalking about the language issue is an attempt to preempt objections to a possible future decision to code this feature in something other than Java; if there are still strong objections to this, let's hash them out.\r\n\r\n### Rough draft of types required from the JVM side\r\n\r\nAfter surveying the codebase, I have a rough draft of what the classes/traits from the JVM side that will be sufficient to access PDAL pipeline functionality from the JVM (written in Scala).\r\n\r\n```scala\r\nabstract class Pipeline(json: String) {\r\n  def execute(): Unit\r\n  def pointViews(): PointViewIterator\r\n  /** Create an iterator of point views, with each point\r\n    * only containing the data contained in the supplied layout\r\n    */\r\n  def pointViews(layout: PointLayout): PointViewIterator\r\n  def dispose(): Unit\r\n}\r\n\r\ntrait PointViewIterator extends Iterator[PointView] {\r\n  def layout: PointLayout\r\n}\r\n\r\ntrait PointView {\r\n  // This grabs the points into some\r\n  // class that contains members which represent\r\n  // the data contained in the layout.\r\n  def points[T]: Iterable[T]\r\n}\r\n\r\ntrait PointLayout\r\n```\r\n\r\nThat's all we'll have to really code up in the bindings based on what I see: a way to execute a pipeline from some JSON, and a sane way to read points from the PointViews it returns.\r\n\r\nThere's a lot to be worked out as far as how the PointLayout translates to\r\nsome type that represents all the data of the point in a type safe way. Also we need to consider access patterns that allow the client to skirt performance killers\r\nlike creating an Object in memory for each point if all they want is (x,y,z)\r\ncoordinates. However this rough draft shows some of the components that are\r\nnecessary for the JVM bindings to be successful.\r\n"},"comment":{"url":"https://api.github.com/repos/PDAL/PDAL/issues/comments/254533931","html_url":"https://github.com/PDAL/PDAL/issues/1332#issuecomment-254533931","issue_url":"https://api.github.com/repos/PDAL/PDAL/issues/1332","id":254533931,"user":{"login":"abellgithub","id":1316400,"avatar_url":"https://avatars.githubusercontent.com/u/1316400?v=3","gravatar_id":"","url":"https://api.github.com/users/abellgithub","html_url":"https://github.com/abellgithub","followers_url":"https://api.github.com/users/abellgithub/followers","following_url":"https://api.github.com/users/abellgithub/following{/other_user}","gists_url":"https://api.github.com/users/abellgithub/gists{/gist_id}","starred_url":"https://api.github.com/users/abellgithub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abellgithub/subscriptions","organizations_url":"https://api.github.com/users/abellgithub/orgs","repos_url":"https://api.github.com/users/abellgithub/repos","events_url":"https://api.github.com/users/abellgithub/events{/privacy}","received_events_url":"https://api.github.com/users/abellgithub/received_events","type":"User","site_admin":false},"created_at":"2016-10-18T14:56:34Z","updated_at":"2016-10-18T14:56:34Z","body":"I know very little about JNI interfaces, but my gut reaction is that the best way to do such a thing is to iteratively add what's necessary in the most simple way possible that works.  Experience tells me that assuming a necessary interface will be insufficient for everyone except the person implementing the interface.  That said, it seems like the Scala/JAVA/whatever interface is immaterial.  What really needs to be defined is the JNI interface, which unless I misunderstand, maps to the high-level language, but is also not tied to it. (A Scala class and Java class can map to the same JNI stubs, right?)\r\n\r\nPerhaps the best way to start would be a simple program in JAVA (or Scala, but JAVA seems a natural baseline) that would represent a first/base use-case?\r\n\r\nAlso, the PDAL model makes a point's data independent of any sort of point object, generally speaking.  This allows users to use/discard data read from a point cloud source and to retrieve in a format they desire.  I don't know Scala so I'm not sure what's represented by the type `layout`or whether this makes sense given the PDAL architecture.  A small notional working program would be helpful for clarification."}}
{"actor":{"display_login":"abellgithub"}}