{"action":"created","issue":{"url":"https://api.github.com/repos/filodb/FiloDB/issues/150","repository_url":"https://api.github.com/repos/filodb/FiloDB","labels_url":"https://api.github.com/repos/filodb/FiloDB/issues/150/labels{/name}","comments_url":"https://api.github.com/repos/filodb/FiloDB/issues/150/comments","events_url":"https://api.github.com/repos/filodb/FiloDB/issues/150/events","html_url":"https://github.com/filodb/FiloDB/issues/150","id":311983539,"number":150,"title":"FiloDB Write format filodb.spark giving errors","user":{"login":"PallaviPersistent","id":38134681,"avatar_url":"https://avatars3.githubusercontent.com/u/38134681?v=4","gravatar_id":"","url":"https://api.github.com/users/PallaviPersistent","html_url":"https://github.com/PallaviPersistent","followers_url":"https://api.github.com/users/PallaviPersistent/followers","following_url":"https://api.github.com/users/PallaviPersistent/following{/other_user}","gists_url":"https://api.github.com/users/PallaviPersistent/gists{/gist_id}","starred_url":"https://api.github.com/users/PallaviPersistent/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PallaviPersistent/subscriptions","organizations_url":"https://api.github.com/users/PallaviPersistent/orgs","repos_url":"https://api.github.com/users/PallaviPersistent/repos","events_url":"https://api.github.com/users/PallaviPersistent/events{/privacy}","received_events_url":"https://api.github.com/users/PallaviPersistent/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-06T13:50:16Z","updated_at":"2018-04-06T15:59:38Z","closed_at":null,"author_association":"NONE","body":"**Branch, version, commit** \r\n\r\n**OS and Environment**Red Hat Enterprise Linux Server release 7.2\r\n\r\n**JVM version**java -version\r\njava version \"1.8.0_161\"\r\nJava(TM) SE Runtime Environment (build 1.8.0_161-b12)\r\nJava HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)\r\n\r\n\r\n**Scala version**\r\nScala version 2.11.8\r\n**Kafka and Cassandra versions and setup**\r\n\r\n**Spark version if used**\r\nSpark version 2.0.0\r\n**Deployed mode**\r\n(client/cluster on Spark Standalone/YARN/Mesos/EMR or default)\r\nSpark Standalone\r\n**Actual (wrong) behavior**\r\nscala> test.write.format(\"filodb.spark\").option(\"path\",\"/root/FiloDB/JSON\").option(\"dataset\",\"jsontest\").option(\"row_keys\",\"action\").option(\"partition_keys\",\"timestamp\").mode(SaveMode.Overwrite).save()\r\njava.util.NoSuchElementException: key not found: ArrayType(StringType,true)\r\n  at scala.collection.MapLike$class.default(MapLike.scala:228)\r\n  at scala.collection.AbstractMap.default(Map.scala:59)\r\n  at scala.collection.MapLike$class.apply(MapLike.scala:141)\r\n  at scala.collection.AbstractMap.apply(Map.scala:59)\r\n  at filodb.spark.package$$anonfun$dfToFiloColumns$1.apply(package.scala:132)\r\n  at filodb.spark.package$$anonfun$dfToFiloColumns$1.apply(package.scala:131)\r\n  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\r\n  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\r\n  at scala.collection.Iterator$class.foreach(Iterator.scala:893)\r\n  at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)\r\n  at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)\r\n  at org.apache.spark.sql.types.StructType.foreach(StructType.scala:95)\r\n  at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)\r\n  at org.apache.spark.sql.types.StructType.map(StructType.scala:95)\r\n  at filodb.spark.package$.dfToFiloColumns(package.scala:131)\r\n  at filodb.spark.FiloContext$.createOrUpdateDataset$extension(FiloContext.scala:57)\r\n  at filodb.spark.FiloContext$.saveAsFilo$extension(FiloContext.scala:117)\r\n  at filodb.spark.DefaultSource.createRelation(DefaultSource.scala:63)\r\n  at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:429)\r\n  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)\r\n  ... 50 elided\r\n\r\n**Steps to reproduce**\r\nscala> val events = sc.parallelize(\r\n     | \"\"\"{\"action\":[\"create\",\"drop\"],\"timestamp\":1452121277}\"\"\" ::\r\n     | \"\"\"{\"action\":[\"create\",\"drop\"],\"timestamp\":\"1452121277\"}\"\"\" ::\r\n     | \"\"\"{\"action\":[\"create\",\"drop\"],\"timestamp\":\"\"}\"\"\" ::\r\n     | \"\"\"{\"action\":[\"create\",\"drop\"],\"timestamp\":null}\"\"\" ::\r\n     | \"\"\"{\"action\":[\"create\",\"drop\"],\"timestamp\":\"null\"}\"\"\" ::\r\n     | Nil\r\n     | )\r\nevents: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[27] at parallelize at <console>:28\r\n\r\nscala> val test= sqlContext.read.json(events)\r\ntest: org.apache.spark.sql.DataFrame = [action: array<string>, timestamp: string]\r\n\r\nscala> test.write.format(\"filodb.spark\").option(\"path\",\"/root/FiloDB/JSON\").option(\"dataset\",\"jsontest\").option(\"row_keys\",\"action\").option(\"partition_keys\",\"timestamp\").mode(SaveMode.Overwrite).save()\r\n\r\n**Logs**\r\n\r\n```\r\nsome log\r\n```\r\n\r\nor as attached file (see below)\r\n\r\n\r\nUnused parts of this template should be removed (including this line).\r\n"},"comment":{"url":"https://api.github.com/repos/filodb/FiloDB/issues/comments/379298397","html_url":"https://github.com/filodb/FiloDB/issues/150#issuecomment-379298397","issue_url":"https://api.github.com/repos/filodb/FiloDB/issues/150","id":379298397,"user":{"login":"velvia","id":1062875,"avatar_url":"https://avatars3.githubusercontent.com/u/1062875?v=4","gravatar_id":"","url":"https://api.github.com/users/velvia","html_url":"https://github.com/velvia","followers_url":"https://api.github.com/users/velvia/followers","following_url":"https://api.github.com/users/velvia/following{/other_user}","gists_url":"https://api.github.com/users/velvia/gists{/gist_id}","starred_url":"https://api.github.com/users/velvia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/velvia/subscriptions","organizations_url":"https://api.github.com/users/velvia/orgs","repos_url":"https://api.github.com/users/velvia/repos","events_url":"https://api.github.com/users/velvia/events{/privacy}","received_events_url":"https://api.github.com/users/velvia/received_events","type":"User","site_admin":false},"created_at":"2018-04-06T15:59:38Z","updated_at":"2018-04-06T15:59:38Z","author_association":"CONTRIBUTOR","body":"Sorry, for now we don't support Array types or nested lists in the Spark schema.   You need to flatten the data.   Something like `test.withColumn(\"firstAction\", .....).select(....)` and leave out the nested list when you select.  Or maybe turn each JSON into two rows?"}}
{"actor":{"display_login":"velvia"}}