{"action":"created","issue":{"url":"https://api.github.com/repos/azavea/raster-foundry/issues/370","repository_url":"https://api.github.com/repos/azavea/raster-foundry","labels_url":"https://api.github.com/repos/azavea/raster-foundry/issues/370/labels{/name}","comments_url":"https://api.github.com/repos/azavea/raster-foundry/issues/370/comments","events_url":"https://api.github.com/repos/azavea/raster-foundry/issues/370/events","html_url":"https://github.com/azavea/raster-foundry/issues/370","id":165548035,"number":370,"title":"Write ADR to choose workflow/asynchronous process framework","user":{"login":"notthatbreezy","id":898060,"avatar_url":"https://avatars.githubusercontent.com/u/898060?v=3","gravatar_id":"","url":"https://api.github.com/users/notthatbreezy","html_url":"https://github.com/notthatbreezy","followers_url":"https://api.github.com/users/notthatbreezy/followers","following_url":"https://api.github.com/users/notthatbreezy/following{/other_user}","gists_url":"https://api.github.com/users/notthatbreezy/gists{/gist_id}","starred_url":"https://api.github.com/users/notthatbreezy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/notthatbreezy/subscriptions","organizations_url":"https://api.github.com/users/notthatbreezy/orgs","repos_url":"https://api.github.com/users/notthatbreezy/repos","events_url":"https://api.github.com/users/notthatbreezy/events{/privacy}","received_events_url":"https://api.github.com/users/notthatbreezy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/azavea/raster-foundry/labels/geospatial-insights","name":"geospatial-insights","color":"0e8a16"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-07-14T12:16:03Z","updated_at":"2016-07-14T13:46:35Z","closed_at":null,"body":"At least a few parts of this project require asynchronous process/task running, including but not limited to the following:\r\n - [initial thumbnail generation to run on new layers](https://github.com/azavea/raster-foundry/blob/65f9050f03b3f0ec5b063549d763bf0c6b169468/src/rf/apps/workers/thumbnail.py) (likely python, shelling out to gdal)\r\n - initial polygon mask generation to run on new layers (likely python, shelling out to gdal)\r\n - kicking off a batch spark job that does not need to be interactive (e.g. via spark-submit)\r\n\r\nMost of our familiarity as an organization and team in this area is focused on primarily two frameworks:\r\n 1. [Amazon Simple Workflow Service](https://aws.amazon.com/swf/) (SWF)\r\n 2. [Celery](http://www.celeryproject.org/)\r\n\r\nOur experience has been mixed at best with these two frameworks, both of which can require significant work to get much useful done depending on the use-case. Additionally, tools for tracking and visualizing workflows are very much lacking with these frameworks.\r\n\r\nBefore we decide on a framework there are probably a few frameworks that we should evaluate:\r\n - [Luigi](https://github.com/spotify/luigi) -- framework written by Spotify and used at a number of businesses, with nice visualization/job-tracking and out-of-the-box integration with tools we need to use (S3, postgres, hadoop/spark)\r\n - [Airflow](http://nerds.airbnb.com/airflow/) -- similar to Luigi, but built by AirBnB\r\n\r\nThere may be others, but at first glance these seem like possibly the best-suited to our use-case"},"comment":{"url":"https://api.github.com/repos/azavea/raster-foundry/issues/comments/232669901","html_url":"https://github.com/azavea/raster-foundry/issues/370#issuecomment-232669901","issue_url":"https://api.github.com/repos/azavea/raster-foundry/issues/370","id":232669901,"user":{"login":"notthatbreezy","id":898060,"avatar_url":"https://avatars.githubusercontent.com/u/898060?v=3","gravatar_id":"","url":"https://api.github.com/users/notthatbreezy","html_url":"https://github.com/notthatbreezy","followers_url":"https://api.github.com/users/notthatbreezy/followers","following_url":"https://api.github.com/users/notthatbreezy/following{/other_user}","gists_url":"https://api.github.com/users/notthatbreezy/gists{/gist_id}","starred_url":"https://api.github.com/users/notthatbreezy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/notthatbreezy/subscriptions","organizations_url":"https://api.github.com/users/notthatbreezy/orgs","repos_url":"https://api.github.com/users/notthatbreezy/repos","events_url":"https://api.github.com/users/notthatbreezy/events{/privacy}","received_events_url":"https://api.github.com/users/notthatbreezy/received_events","type":"User","site_admin":false},"created_at":"2016-07-14T13:46:35Z","updated_at":"2016-07-14T13:46:35Z","body":"Other things to consider when evaluating workflow frameworks:\r\n - can they version jobs/workflows to enhance blue/green deployments\r\n - is there a way to pin a set of jobs to a single worker -- for instance, to prevent having to re-download a file onto multiple workers if a set of jobs uses the same file"}}
{"actor":{"display_login":"notthatbreezy"}}