{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/ecosystem/issues/67","repository_url":"https://api.github.com/repos/tensorflow/ecosystem","labels_url":"https://api.github.com/repos/tensorflow/ecosystem/issues/67/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/ecosystem/issues/67/comments","events_url":"https://api.github.com/repos/tensorflow/ecosystem/issues/67/events","html_url":"https://github.com/tensorflow/ecosystem/issues/67","id":269321779,"node_id":"MDU6SXNzdWUyNjkzMjE3Nzk=","number":67,"title":"Failed to find data source: tensorflow","user":{"login":"rolanyan","id":33174745,"node_id":"MDQ6VXNlcjMzMTc0NzQ1","avatar_url":"https://avatars0.githubusercontent.com/u/33174745?v=4","gravatar_id":"","url":"https://api.github.com/users/rolanyan","html_url":"https://github.com/rolanyan","followers_url":"https://api.github.com/users/rolanyan/followers","following_url":"https://api.github.com/users/rolanyan/following{/other_user}","gists_url":"https://api.github.com/users/rolanyan/gists{/gist_id}","starred_url":"https://api.github.com/users/rolanyan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rolanyan/subscriptions","organizations_url":"https://api.github.com/users/rolanyan/orgs","repos_url":"https://api.github.com/users/rolanyan/repos","events_url":"https://api.github.com/users/rolanyan/events{/privacy}","received_events_url":"https://api.github.com/users/rolanyan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-10-28T14:30:10Z","updated_at":"2018-06-06T12:39:02Z","closed_at":null,"author_association":"NONE","body":"\r\n    import com.tencent.mmsearch_x.SparkTool._\r\n    import org.apache.spark.sql.{DataFrame, SparkSession}\r\n    import org.apache.commons.io.FileUtils\r\n    import org.apache.spark.sql.{ DataFrame, Row }\r\n    import org.apache.spark.sql.catalyst.expressions.GenericRow\r\n    import org.apache.spark.sql.types._\r\n    val path = \"./output/test-output.tfrecord\"\r\n    val testRows: Array[Row] = Array(\r\n      new GenericRow(Array[Any](11, 1, 23L, 10.0F, 14.0, List(1.0, 2.0), \"r1\")),\r\n      new GenericRow(Array[Any](21, 2, 24L, 12.0F, 15.0, List(2.0, 2.0), \"r2\")))\r\n    val schema = StructType(List(StructField(\"id\", IntegerType),\r\n      StructField(\"IntegerTypeLabel\", IntegerType),\r\n      StructField(\"LongTypeLabel\", LongType),\r\n      StructField(\"FloatTypeLabel\", FloatType),\r\n      StructField(\"DoubleTypeLabel\", DoubleType),\r\n      StructField(\"VectorLabel\", ArrayType(DoubleType, true)),\r\n      StructField(\"name\", StringType)))\r\n\r\n    val rdd = spark.sparkContext.parallelize(testRows)\r\n\r\n    //Save DataFrame as TFRecords\r\n    val df: DataFrame = spark.createDataFrame(rdd, schema)\r\n    df.show()\r\n    df.printSchema()\r\n\r\n    //.option(\"recordType\", \"Example\")\r\n    df.write.format(\"tensorflow\").save(path)  //or df.write.format(\"tfrecords\").save(path)\r\n\r\nboth format \"tensorflow\" and \"tfrecords\" result to { Exception in thread \"main\" java.lang.ClassNotFoundException: Failed to find data source: tensorflow }"},"comment":{"url":"https://api.github.com/repos/tensorflow/ecosystem/issues/comments/395053620","html_url":"https://github.com/tensorflow/ecosystem/issues/67#issuecomment-395053620","issue_url":"https://api.github.com/repos/tensorflow/ecosystem/issues/67","id":395053620,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NTA1MzYyMA==","user":{"login":"skavulya","id":6307926,"node_id":"MDQ6VXNlcjYzMDc5MjY=","avatar_url":"https://avatars2.githubusercontent.com/u/6307926?v=4","gravatar_id":"","url":"https://api.github.com/users/skavulya","html_url":"https://github.com/skavulya","followers_url":"https://api.github.com/users/skavulya/followers","following_url":"https://api.github.com/users/skavulya/following{/other_user}","gists_url":"https://api.github.com/users/skavulya/gists{/gist_id}","starred_url":"https://api.github.com/users/skavulya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/skavulya/subscriptions","organizations_url":"https://api.github.com/users/skavulya/orgs","repos_url":"https://api.github.com/users/skavulya/repos","events_url":"https://api.github.com/users/skavulya/events{/privacy}","received_events_url":"https://api.github.com/users/skavulya/received_events","type":"User","site_admin":false},"created_at":"2018-06-06T12:39:02Z","updated_at":"2018-06-06T12:39:02Z","author_association":"CONTRIBUTOR","body":"@lei If you already have the jar, you don't need to recompile it. At what point do you get this error? At compile time or runtime? If it occurs at runtime, you can either need to include the `--jars` argument when running `spark-submit` or `spark-shell` as described in the README.  The other option is to build an assembly jar with sbt for your application that includes your dependencies."}}
{"actor":{"display_login":"skavulya"}}