{"action":"created","issue":{"url":"https://api.github.com/repos/lsds/SEEPng/issues/3","repository_url":"https://api.github.com/repos/lsds/SEEPng","labels_url":"https://api.github.com/repos/lsds/SEEPng/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/lsds/SEEPng/issues/3/comments","events_url":"https://api.github.com/repos/lsds/SEEPng/issues/3/events","html_url":"https://github.com/lsds/SEEPng/pull/3","id":133257345,"number":3,"title":"Implemented HDFS as an InputAdapter","user":{"login":"JamesDavidCarr","id":4114498,"avatar_url":"https://avatars.githubusercontent.com/u/4114498?v=3","gravatar_id":"","url":"https://api.github.com/users/JamesDavidCarr","html_url":"https://github.com/JamesDavidCarr","followers_url":"https://api.github.com/users/JamesDavidCarr/followers","following_url":"https://api.github.com/users/JamesDavidCarr/following{/other_user}","gists_url":"https://api.github.com/users/JamesDavidCarr/gists{/gist_id}","starred_url":"https://api.github.com/users/JamesDavidCarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JamesDavidCarr/subscriptions","organizations_url":"https://api.github.com/users/JamesDavidCarr/orgs","repos_url":"https://api.github.com/users/JamesDavidCarr/repos","events_url":"https://api.github.com/users/JamesDavidCarr/events{/privacy}","received_events_url":"https://api.github.com/users/JamesDavidCarr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-02-12T14:38:44Z","updated_at":"2016-02-23T16:33:38Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/lsds/SEEPng/pulls/3","html_url":"https://github.com/lsds/SEEPng/pull/3","diff_url":"https://github.com/lsds/SEEPng/pull/3.diff","patch_url":"https://github.com/lsds/SEEPng/pull/3.patch"},"body":"This is based off of the Kafka InputAdapter implementation.\r\n\r\nDepending on the size and number of files stored on HDFS this implementation may change."},"comment":{"url":"https://api.github.com/repos/lsds/SEEPng/issues/comments/187776407","html_url":"https://github.com/lsds/SEEPng/pull/3#issuecomment-187776407","issue_url":"https://api.github.com/repos/lsds/SEEPng/issues/3","id":187776407,"user":{"login":"WJCIV","id":15984878,"avatar_url":"https://avatars.githubusercontent.com/u/15984878?v=3","gravatar_id":"","url":"https://api.github.com/users/WJCIV","html_url":"https://github.com/WJCIV","followers_url":"https://api.github.com/users/WJCIV/followers","following_url":"https://api.github.com/users/WJCIV/following{/other_user}","gists_url":"https://api.github.com/users/WJCIV/gists{/gist_id}","starred_url":"https://api.github.com/users/WJCIV/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/WJCIV/subscriptions","organizations_url":"https://api.github.com/users/WJCIV/orgs","repos_url":"https://api.github.com/users/WJCIV/repos","events_url":"https://api.github.com/users/WJCIV/events{/privacy}","received_events_url":"https://api.github.com/users/WJCIV/received_events","type":"User","site_admin":false},"created_at":"2016-02-23T16:33:38Z","updated_at":"2016-02-23T16:33:38Z","body":"If you take a look in the example folder you can see how users actually interface with the system. The basic idea is to create a set of tasks which process input and turn it into output. So if you have one operation which processes the data you define the input (source), the task, and the output (sink), then the connections between them. If you have two operations the flow is source -> task 1 -> task 2 -> sink where the output from task 1 placed in a Dataset and used as input to task 2.\r\n\r\nYour code can read data from HDFS and place it in a Dataset, but that is one step shy of creating the source abstraction the user can actually use. Without this a user cannot say say that the initial Dataset for a job resides in HDFS.\r\n\r\nThe options are:\r\n1. Create an HDFSSource. This is probably easier, although likely not by much.\r\n2. Add a configuration to FileConfig (used by FileSource) that marks it as residing on HDFS and update the FileSource to use your new code when this is marked. While this might be a bit more complex it will be much easier for users to understand (a file is a file, regardless of which file system is used) and will allow them to move to HDFS with a simple configuration setting instead of replacing their source classes.\r\n\r\nI'd recommend you look at the Base class in the simple-sourcefile-stateless-query example to see how a user actually programs source -> task -> sink."}}