{"action":"created","issue":{"url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/1195","repository_url":"https://api.github.com/repos/TresAmigosSD/SMV","labels_url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/1195/labels{/name}","comments_url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/1195/comments","events_url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/1195/events","html_url":"https://github.com/TresAmigosSD/SMV/issues/1195","id":314740363,"number":1195,"title":"Make SmvInput python class more user friendly so user can easily define their own inputs","user":{"login":"ninjapapa","id":8062822,"avatar_url":"https://avatars1.githubusercontent.com/u/8062822?v=4","gravatar_id":"","url":"https://api.github.com/users/ninjapapa","html_url":"https://github.com/ninjapapa","followers_url":"https://api.github.com/users/ninjapapa/followers","following_url":"https://api.github.com/users/ninjapapa/following{/other_user}","gists_url":"https://api.github.com/users/ninjapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ninjapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ninjapapa/subscriptions","organizations_url":"https://api.github.com/users/ninjapapa/orgs","repos_url":"https://api.github.com/users/ninjapapa/repos","events_url":"https://api.github.com/users/ninjapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ninjapapa/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"ninjapapa","id":8062822,"avatar_url":"https://avatars1.githubusercontent.com/u/8062822?v=4","gravatar_id":"","url":"https://api.github.com/users/ninjapapa","html_url":"https://github.com/ninjapapa","followers_url":"https://api.github.com/users/ninjapapa/followers","following_url":"https://api.github.com/users/ninjapapa/following{/other_user}","gists_url":"https://api.github.com/users/ninjapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ninjapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ninjapapa/subscriptions","organizations_url":"https://api.github.com/users/ninjapapa/orgs","repos_url":"https://api.github.com/users/ninjapapa/repos","events_url":"https://api.github.com/users/ninjapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ninjapapa/received_events","type":"User","site_admin":false},"assignees":[{"login":"ninjapapa","id":8062822,"avatar_url":"https://avatars1.githubusercontent.com/u/8062822?v=4","gravatar_id":"","url":"https://api.github.com/users/ninjapapa","html_url":"https://github.com/ninjapapa","followers_url":"https://api.github.com/users/ninjapapa/followers","following_url":"https://api.github.com/users/ninjapapa/following{/other_user}","gists_url":"https://api.github.com/users/ninjapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ninjapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ninjapapa/subscriptions","organizations_url":"https://api.github.com/users/ninjapapa/orgs","repos_url":"https://api.github.com/users/ninjapapa/repos","events_url":"https://api.github.com/users/ninjapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ninjapapa/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-04-16T16:59:31Z","updated_at":"2018-04-16T17:51:00Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"The use case is that other that the input connectors SMV is supporting, there might be all types of input sources which are not covered and can't be fully covered anyhow. The best way to address this could be providing an easier `SmvInput` ABC, so that user can easily extend for their own use cases.\r\n\r\nAs an example, we will use XML reader to test the design of the simpler `SmvInout` ABC.\r\n\r\nThis is a Type2 issue. Will design it within issue comments."},"comment":{"url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/comments/381691818","html_url":"https://github.com/TresAmigosSD/SMV/issues/1195#issuecomment-381691818","issue_url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/1195","id":381691818,"user":{"login":"ninjapapa","id":8062822,"avatar_url":"https://avatars1.githubusercontent.com/u/8062822?v=4","gravatar_id":"","url":"https://api.github.com/users/ninjapapa","html_url":"https://github.com/ninjapapa","followers_url":"https://api.github.com/users/ninjapapa/followers","following_url":"https://api.github.com/users/ninjapapa/following{/other_user}","gists_url":"https://api.github.com/users/ninjapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ninjapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ninjapapa/subscriptions","organizations_url":"https://api.github.com/users/ninjapapa/orgs","repos_url":"https://api.github.com/users/ninjapapa/repos","events_url":"https://api.github.com/users/ninjapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ninjapapa/received_events","type":"User","site_admin":false},"created_at":"2018-04-16T17:51:00Z","updated_at":"2018-04-16T17:51:00Z","author_association":"CONTRIBUTOR","body":"# Requirement\r\n\r\nFor the use case that an end-user to use `SmvInout` directly, target client code for reading in an XML file:\r\n```python\r\nclass MyXmlInput(SmvInput):\r\n    def reader(self):\r\n        return self.smvApp.sqlContext\\\r\n            .read.format('com.databricks.spark.xml')\\\r\n            .options(rowTag='MedlineCitation')\\\r\n            .load('path/to/xmlfile.xml')\r\n```\r\n\r\nFor use case that a \"library-creator\" using `SmvInout` to create an `SmvXmlInput` class for end-user,\r\nThe end-user code will look like:\r\n```python\r\nclass MyXmlInput(SmvXmlInput):\r\n    def path(self):\r\n        return \"path/to/xmlfile.xml\" # could be dir or xmlfiles.*.xml\r\n    def rowTag(self):\r\n        return \"MedlineCitation\"\r\n    def userSchema(self):\r\n        with open (\"path/to/schema.json\", \"r\") as sj:\r\n            schema_st = sj.read()\r\n        return schema_st\r\n```\r\n\r\n# Design\r\n\r\nCurrent `SmvInput` assumes there is a Scala implementation of the concrete class. That assumption should be separated to another class, and can be mixed in when the concrete class does have a Scala backend.\r\n\r\nThe `instanceValHash` logic for `SmvFile` on Scala side should be made available out of the class definition so that the Python side can call it even in the case that there are no concrete class level Scala backend.\r\n"}}
{"actor":{"display_login":"ninjapapa"}}