{"action":"created","issue":{"url":"https://api.github.com/repos/hortonworks-spark/shc/issues/205","repository_url":"https://api.github.com/repos/hortonworks-spark/shc","labels_url":"https://api.github.com/repos/hortonworks-spark/shc/issues/205/labels{/name}","comments_url":"https://api.github.com/repos/hortonworks-spark/shc/issues/205/comments","events_url":"https://api.github.com/repos/hortonworks-spark/shc/issues/205/events","html_url":"https://github.com/hortonworks-spark/shc/issues/205","id":280077418,"node_id":"MDU6SXNzdWUyODAwNzc0MTg=","number":205,"title":"SHC with Spark Structured Streaming","user":{"login":"EDALJO","id":22967657,"node_id":"MDQ6VXNlcjIyOTY3NjU3","avatar_url":"https://avatars2.githubusercontent.com/u/22967657?v=4","gravatar_id":"","url":"https://api.github.com/users/EDALJO","html_url":"https://github.com/EDALJO","followers_url":"https://api.github.com/users/EDALJO/followers","following_url":"https://api.github.com/users/EDALJO/following{/other_user}","gists_url":"https://api.github.com/users/EDALJO/gists{/gist_id}","starred_url":"https://api.github.com/users/EDALJO/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EDALJO/subscriptions","organizations_url":"https://api.github.com/users/EDALJO/orgs","repos_url":"https://api.github.com/users/EDALJO/repos","events_url":"https://api.github.com/users/EDALJO/events{/privacy}","received_events_url":"https://api.github.com/users/EDALJO/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2017-12-07T10:29:54Z","updated_at":"2018-09-18T04:09:38Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nI have a Spark Structured Streaming application where I'd like to write streaming data to HBase using SHC. It reads data from a location where new csv files continuously are being created. The defined catalog works for writing a DataFrame with identical data into HBase.\r\nThe key components of my streaming application are a DataStreamReader and a DataStreamWriter.\r\n\r\n```\r\nval inputDataStream = spark\r\n      .readStream\r\n      .option(\"sep\", \",\")\r\n      .schema(schema)\r\n      .csv(\"/path/to/data/*.csv\")\r\n\r\ninputDataStream\r\n      .writeStream\r\n      .outputMode(\"append\")\r\n      .options(\r\n        Map(HBaseTableCatalog.tableCatalog -> catalog, HBaseTableCatalog.newTable -> \"2\"))\r\n      .format(\"org.apache.spark.sql.execution.datasources.hbase\")\r\n      .start\r\n```\r\n\r\nWhen running the application I'm getting the following message:\r\n\r\n`Exception in thread \"main\" java.lang.UnsupportedOperationException: Data source org.apache.spark.sql.execution.datasources.hbase does not support streamed writing\r\n        at org.apache.spark.sql.execution.datasources.DataSource.createSink(DataSource.scala:285)\r\n        at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:286)\r\n        at my.package.SHCStreamingApplication$.main(SHCStreamingApplication.scala:153)\r\n        at my.package.SHCStreamingApplication.main(SHCStreamingApplication.scala)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)\r\n        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)\r\n        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)\r\n        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)\r\n        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)`\r\n\r\nDoes anyone know a solution or way/workaround to still use the SHC for writing structured streaming data to HBase?\r\nThanks in advance!"},"comment":{"url":"https://api.github.com/repos/hortonworks-spark/shc/issues/comments/422248964","html_url":"https://github.com/hortonworks-spark/shc/issues/205#issuecomment-422248964","issue_url":"https://api.github.com/repos/hortonworks-spark/shc/issues/205","id":422248964,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMjI0ODk2NA==","user":{"login":"sutugin","id":7840585,"node_id":"MDQ6VXNlcjc4NDA1ODU=","avatar_url":"https://avatars2.githubusercontent.com/u/7840585?v=4","gravatar_id":"","url":"https://api.github.com/users/sutugin","html_url":"https://github.com/sutugin","followers_url":"https://api.github.com/users/sutugin/followers","following_url":"https://api.github.com/users/sutugin/following{/other_user}","gists_url":"https://api.github.com/users/sutugin/gists{/gist_id}","starred_url":"https://api.github.com/users/sutugin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sutugin/subscriptions","organizations_url":"https://api.github.com/users/sutugin/orgs","repos_url":"https://api.github.com/users/sutugin/repos","events_url":"https://api.github.com/users/sutugin/events{/privacy}","received_events_url":"https://api.github.com/users/sutugin/received_events","type":"User","site_admin":false},"created_at":"2018-09-18T04:09:38Z","updated_at":"2018-09-18T04:09:38Z","author_association":"NONE","body":"Hi @swarup5s ,if you give me the implementation code and how you use it, show me the logs, maybe we can find the problem together.\r\n\r\n"}}
{"actor":{"display_login":"sutugin"}}