{"action":"created","issue":{"url":"https://api.github.com/repos/cantino/huginn/issues/1301","repository_url":"https://api.github.com/repos/cantino/huginn","labels_url":"https://api.github.com/repos/cantino/huginn/issues/1301/labels{/name}","comments_url":"https://api.github.com/repos/cantino/huginn/issues/1301/comments","events_url":"https://api.github.com/repos/cantino/huginn/issues/1301/events","html_url":"https://github.com/cantino/huginn/pull/1301","id":134322609,"number":1301,"title":"Introduce concept to handle files","user":{"login":"dsander","id":20943,"avatar_url":"https://avatars.githubusercontent.com/u/20943?v=3","gravatar_id":"","url":"https://api.github.com/users/dsander","html_url":"https://github.com/dsander","followers_url":"https://api.github.com/users/dsander/followers","following_url":"https://api.github.com/users/dsander/following{/other_user}","gists_url":"https://api.github.com/users/dsander/gists{/gist_id}","starred_url":"https://api.github.com/users/dsander/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dsander/subscriptions","organizations_url":"https://api.github.com/users/dsander/orgs","repos_url":"https://api.github.com/users/dsander/repos","events_url":"https://api.github.com/users/dsander/events{/privacy}","received_events_url":"https://api.github.com/users/dsander/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-02-17T16:04:53Z","updated_at":"2016-03-07T02:04:43Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/cantino/huginn/pulls/1301","html_url":"https://github.com/cantino/huginn/pull/1301","diff_url":"https://github.com/cantino/huginn/pull/1301.diff","patch_url":"https://github.com/cantino/huginn/pull/1301.patch"},"body":"This PR introduces a concept for Huginn to work with files. The idea is to have source agents (`LocalFile` and `S3Agent` for now), which just emit a \"file_pointer\" and consuming agents (`ReadFile` and `CsvAgent`) which get access to an `IO` object to actually access the file contents. This way we do not have to pass megabytes of data through the database and the consuming agents can work with relatively unlimited file sizes.\r\nIt should be extensible to all possible sources as long as the source client for the file or data supports to return a `IO` object (or can be wrapped into one). For example the `DropboxWatchAgent` could send the HTTP links to an `HTTPFileAgent` which would trigger the `S3Agent` to back up the Dropbox files to S3.\r\n\r\nIntroduces four new agents:\r\n\r\n* `LocalFileAgent` (source), can watch for changes of a file/directory, when schedules just emits an event for every file\r\n* `S3Agent` (source), watches an S3 bucket for changes or emits the files on interval\r\n* `ReadFileAgent` (birdge/consumer), takes the file pointer from one of the source agents, reads the file and emits to contents so that other agents that do not yet support the file pointers can work with the data.\r\n* `CsvAgent` (consumer),parses CSV data, it can consume a file pointer or with with data supplied via the event payload\r\n\r\nTo be done/decided:\r\n\r\n* [x] `S3Agent` should be able to write files\r\n* [x] `FtpSiteAgent` should be able to write files\r\n* [x] `LocalFileAgent` should be able to write files\r\n* [x] `CSVAgent` should be able to serialize data to CSV\r\n* [x] ~~should the `ReadFileAgent` be able to emit an event per line?~~ (Not needed by us)\r\n* [x] Introduce `FileHandling` concern to the `FtpSiteAgent`, or add a new which could be more similar to the `LocalFileAgent` and `S3Agent`? (could also be done in a different PR)\r\n* [x] ~~Should the `FileHandling` agents be able to consume a file pointer when writing? This would allow efficient mirroring from one storage to another, but only be useful in that use case~~ (Not needed by us, should be easy to add when needed)\r\n\r\nThanks to [kreuzwerker.de](http://kreuzwerker.de) who are paying me to work on this in the context of the research project '[Digitale Kuratierungstechnologien](http://digitale-kuratierung.de/ )' of which they are a part of."},"comment":{"url":"https://api.github.com/repos/cantino/huginn/issues/comments/193049068","html_url":"https://github.com/cantino/huginn/pull/1301#issuecomment-193049068","issue_url":"https://api.github.com/repos/cantino/huginn/issues/1301","id":193049068,"user":{"login":"cantino","id":83835,"avatar_url":"https://avatars.githubusercontent.com/u/83835?v=3","gravatar_id":"","url":"https://api.github.com/users/cantino","html_url":"https://github.com/cantino","followers_url":"https://api.github.com/users/cantino/followers","following_url":"https://api.github.com/users/cantino/following{/other_user}","gists_url":"https://api.github.com/users/cantino/gists{/gist_id}","starred_url":"https://api.github.com/users/cantino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cantino/subscriptions","organizations_url":"https://api.github.com/users/cantino/orgs","repos_url":"https://api.github.com/users/cantino/repos","events_url":"https://api.github.com/users/cantino/events{/privacy}","received_events_url":"https://api.github.com/users/cantino/received_events","type":"User","site_admin":false},"created_at":"2016-03-07T02:04:43Z","updated_at":"2016-03-07T02:04:43Z","body":"Quite an effort!"}}