{"action":"created","issue":{"url":"https://api.github.com/repos/Stratio/Spark-MongoDB/issues/113","repository_url":"https://api.github.com/repos/Stratio/Spark-MongoDB","labels_url":"https://api.github.com/repos/Stratio/Spark-MongoDB/issues/113/labels{/name}","comments_url":"https://api.github.com/repos/Stratio/Spark-MongoDB/issues/113/comments","events_url":"https://api.github.com/repos/Stratio/Spark-MongoDB/issues/113/events","html_url":"https://github.com/Stratio/Spark-MongoDB/issues/113","id":147086692,"number":113,"title":"Broken schema discovery for heterogeneous arrays","user":{"login":"ssimeonov","id":491953,"avatar_url":"https://avatars.githubusercontent.com/u/491953?v=3","gravatar_id":"","url":"https://api.github.com/users/ssimeonov","html_url":"https://github.com/ssimeonov","followers_url":"https://api.github.com/users/ssimeonov/followers","following_url":"https://api.github.com/users/ssimeonov/following{/other_user}","gists_url":"https://api.github.com/users/ssimeonov/gists{/gist_id}","starred_url":"https://api.github.com/users/ssimeonov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssimeonov/subscriptions","organizations_url":"https://api.github.com/users/ssimeonov/orgs","repos_url":"https://api.github.com/users/ssimeonov/repos","events_url":"https://api.github.com/users/ssimeonov/events{/privacy}","received_events_url":"https://api.github.com/users/ssimeonov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-04-09T04:10:28Z","updated_at":"2016-04-25T16:21:52Z","closed_at":null,"body":"It seems that schema discovery for documents with heterogeneous arrays just picks the type of the first element or, perhaps, maps everything to string. For example, a field value such as `[\"props\", {a: 1, b: 2}]` is mapped to `array<string>`."},"comment":{"url":"https://api.github.com/repos/Stratio/Spark-MongoDB/issues/comments/214427841","html_url":"https://github.com/Stratio/Spark-MongoDB/issues/113#issuecomment-214427841","issue_url":"https://api.github.com/repos/Stratio/Spark-MongoDB/issues/113","id":214427841,"user":{"login":"pfcoperez","id":273379,"avatar_url":"https://avatars.githubusercontent.com/u/273379?v=3","gravatar_id":"","url":"https://api.github.com/users/pfcoperez","html_url":"https://github.com/pfcoperez","followers_url":"https://api.github.com/users/pfcoperez/followers","following_url":"https://api.github.com/users/pfcoperez/following{/other_user}","gists_url":"https://api.github.com/users/pfcoperez/gists{/gist_id}","starred_url":"https://api.github.com/users/pfcoperez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pfcoperez/subscriptions","organizations_url":"https://api.github.com/users/pfcoperez/orgs","repos_url":"https://api.github.com/users/pfcoperez/repos","events_url":"https://api.github.com/users/pfcoperez/events{/privacy}","received_events_url":"https://api.github.com/users/pfcoperez/received_events","type":"User","site_admin":false},"created_at":"2016-04-25T16:21:52Z","updated_at":"2016-04-25T16:21:52Z","body":"Hi @ssimeonov , Spark's type for arrays is [ArrayType](http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.types.ArrayType). As you can see from the documentation, these arrays have an unique element type:\r\n\r\n```scala\r\nArrayType(**elementType**: DataType, ....)\r\n```\r\n\r\nThe connector iterates over each heterogeneous array element finding a compatible common type ancestor: For example, it can deduce `DoubleType` can represent both real and integer numbers.\r\nIf it finds two different structures such as `{a: 1, b: 2}` and `{c: \"hello\"}` it infers a common structure type: `{a: 1, b: 2, c: \"hello\"}`.  These \"type merging policies are expressed by the [compatibleType](https://github.com/Stratio/Spark-MongoDB/blob/506be658d99513bfb423d548a316286fc858142f/spark-mongodb/src/main/scala/com/stratio/datasource/mongodb/schema/MongodbSchema.scala#L80).\r\n\r\nAt the end of the day, the connector need to return a Spark compatible type, unfortunately, there is not a heterogeneous `ArrayType` in Spark yet. We could invest some time in implementing that type but, is that the right thing to do?\r\n\r\nFrom [Learning Spark,](http://shop.oreilly.com/product/0636920028512.do) Chapter 9: \r\n\r\n> Structured data is any data that has a schema—that is, a known\r\n> set of fields for each record\r\n\r\n...\r\n...\r\n...\r\n\r\n> Under the hood, Spark SQL is based on an extension of the RDD model called a\r\n> DataFrame. A DataFrame contains an RDD of Row objects, each representing a\r\n> record. A DataFrame also knows the schema (i.e., data fields) of its rows. DataFrames\r\n> store data in a more efficient manner than native RDDs, taking advantage of their\r\n> schema\r\n\r\nFrom those paragraphs I understand that schema inference implies passing from semi structured to structured data. The latter being conceptually incompatible with heterogeneous arrays."}}