{"action":"created","issue":{"url":"https://api.github.com/repos/frictionlessdata/goodtables.io/issues/2","repository_url":"https://api.github.com/repos/frictionlessdata/goodtables.io","labels_url":"https://api.github.com/repos/frictionlessdata/goodtables.io/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/frictionlessdata/goodtables.io/issues/2/comments","events_url":"https://api.github.com/repos/frictionlessdata/goodtables.io/issues/2/events","html_url":"https://github.com/frictionlessdata/goodtables.io/issues/2","id":184219412,"number":2,"title":"Thoughts on architecture","user":{"login":"amercader","id":200230,"avatar_url":"https://avatars.githubusercontent.com/u/200230?v=3","gravatar_id":"","url":"https://api.github.com/users/amercader","html_url":"https://github.com/amercader","followers_url":"https://api.github.com/users/amercader/followers","following_url":"https://api.github.com/users/amercader/following{/other_user}","gists_url":"https://api.github.com/users/amercader/gists{/gist_id}","starred_url":"https://api.github.com/users/amercader/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amercader/subscriptions","organizations_url":"https://api.github.com/users/amercader/orgs","repos_url":"https://api.github.com/users/amercader/repos","events_url":"https://api.github.com/users/amercader/events{/privacy}","received_events_url":"https://api.github.com/users/amercader/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-10-20T12:45:12Z","updated_at":"2016-10-31T13:28:30Z","closed_at":null,"body":"@roll @pwalsh Apologies for the long post and what might be seen as obvious stuff. This is mainly me thinking out loud with the help of some pictures, but would appreciate your input.\n\nI'm trying to think of the simplest architecture for the data validation service and working up from there based on the requirements and user stories.\n\nAt the most basic level we want to run a validation job:\n1. We somehow got hold of a Tabular data file (let's not get into how we got it for now)\n2. We run a validation command against it (based on one of the goodtables libraries)\n3. The validation command generates some kind of output\n\n![goodtables io architecture1](https://cloud.githubusercontent.com/assets/200230/19557159/6ffdacaa-96bc-11e6-8c4c-8ab74be73f8a.png)\n\nLet's expand on this for a bit. Although we might have a default behaviour with no defined configuration, we will probably want to customize the validation with configuration options. These could control things like what files to validate in case there are many of them, whether there is a schema, what tests to run, etc.\n\nNote that I'm running validation on multiple related files (eg same GitHub repo) on the same job just because it seems easier than to run them separately and combine the outputs later but this is not a hard requirement.\n\n![goodtables io architecture2](https://cloud.githubusercontent.com/assets/200230/19557158/6ffc3cd0-96bc-11e6-8943-d4c443a0f295.png)\n\nThese jobs will be started from a pool or queue of pending ones. This should be able to handle parallelization, scaling of resources, etc\n\n![goodtables io architecture3](https://cloud.githubusercontent.com/assets/200230/19557157/6ffbcd7c-96bc-11e6-9532-8c5e73e85472.png)\n\nOn top of that there is an API that gets requests for validation, starts jobs and returns status and outputs (and handles authorization, of course). Even higher than that there will be the Web UI for integration and reporting but I'm less concerned about that now as it could be just a client on top of the API.\n\n![goodtables io architecture4](https://cloud.githubusercontent.com/assets/200230/19557156/6ff9ff24-96bc-11e6-80de-0a3df27981fc.png)\n\nI know we've been focusing on GitHub and I agree that's the most valuable integration to begin with but I think having this generic API on top would allow much more flexibility with other third-party integrations in my view. GitHub would be just another \"integration\" or plugin on top of the API, which would handle the specifics of each source, eg:\n- GitHub: respond to GitHub's webhooks (preconfigured by the user by \"switching on\" their repo on the GT.io web UI), trigger a new job if necessary, passing the config options translated from the `.goodtables.yml` file and a list of files to validate (and maybe the necessary credentials if it's a private repo?)\n- S3: Regardless of how we get notified of changes (either we get [notified](http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html), but not sure how easy to set this up it will be, or we poll regularly the bucket) we also pass the config options translated from the `.goodtables.yml` file and a list of files to validate (and maybe the necessary credentials if it's a private bucket?)\n\nOther sources could interact with the API directly:\n- CKAN: Whenever a new tabular file gets uploaded or linked an extension pings the API directly with a link to the file and some config options\n- Any third party app with files available online: whenever they need to validate a file, they ping the API with a link to their file(s) and config options (or we create an integration that hooks into whatever notification system they have)\n- Any third-party desktop app (Comma Chamelon, Excel, gt.io CLI client, ...): whenever you want to validate a file, they POST it to the API, along with a schema and config options if needed.\n\n![goodtables io architecture 4](https://cloud.githubusercontent.com/assets/200230/19559830/97afe2ba-96c9-11e6-8ac8-63b18eff0607.png)\n\nI haven't given much though to the actual tech involved in this, but I'm definitely thinking that GitLab CI might not be the best fit for this. It does give us a lot in terms of all the CI infrastucture as in runners, etc but at the expense of having to fit each of the different sources into a git based workflow. And also having to design any higher level API with the constraints of what GitLab offers. Perhaps something simpler based on a queue and job runs gives us more flexibility to start with.\n\nHow does this sound as a general approach? Can you see any major drawbacks?\n\nIf that sounds good, any thoughts on technology that could fit this scenario? Jenkins, Celery?\n\nWould love your feedback on this.\n\nDrawings source is [here](https://docs.google.com/drawings/d/1emknr4Qy74lx9uQ-YSoJB_v9bLAwq6msUBmxhhnxwig/edit?usp=sharing)\n"},"comment":{"url":"https://api.github.com/repos/frictionlessdata/goodtables.io/issues/comments/257293297","html_url":"https://github.com/frictionlessdata/goodtables.io/issues/2#issuecomment-257293297","issue_url":"https://api.github.com/repos/frictionlessdata/goodtables.io/issues/2","id":257293297,"user":{"login":"pwalsh","id":791678,"avatar_url":"https://avatars.githubusercontent.com/u/791678?v=3","gravatar_id":"","url":"https://api.github.com/users/pwalsh","html_url":"https://github.com/pwalsh","followers_url":"https://api.github.com/users/pwalsh/followers","following_url":"https://api.github.com/users/pwalsh/following{/other_user}","gists_url":"https://api.github.com/users/pwalsh/gists{/gist_id}","starred_url":"https://api.github.com/users/pwalsh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pwalsh/subscriptions","organizations_url":"https://api.github.com/users/pwalsh/orgs","repos_url":"https://api.github.com/users/pwalsh/repos","events_url":"https://api.github.com/users/pwalsh/events{/privacy}","received_events_url":"https://api.github.com/users/pwalsh/received_events","type":"User","site_admin":false},"created_at":"2016-10-31T13:28:30Z","updated_at":"2016-10-31T13:28:30Z","body":"@amercader \r\n\r\nSo, we are all agreed. Can you take these ideas and modify the requirements as current specified in the README, to reflect these changes, and any other changes that are relevant, in comparison to what I wrote originally."}}
{"actor":{"display_login":"pwalsh"}}