{"action":"created","issue":{"url":"https://api.github.com/repos/biggis-project/biggis-infrastructure/issues/12","repository_url":"https://api.github.com/repos/biggis-project/biggis-infrastructure","labels_url":"https://api.github.com/repos/biggis-project/biggis-infrastructure/issues/12/labels{/name}","comments_url":"https://api.github.com/repos/biggis-project/biggis-infrastructure/issues/12/comments","events_url":"https://api.github.com/repos/biggis-project/biggis-infrastructure/issues/12/events","html_url":"https://github.com/biggis-project/biggis-infrastructure/issues/12","id":167627264,"number":12,"title":"M3 Workflow description ","user":{"login":"wipatrick","id":15153294,"avatar_url":"https://avatars.githubusercontent.com/u/15153294?v=3","gravatar_id":"","url":"https://api.github.com/users/wipatrick","html_url":"https://github.com/wipatrick","followers_url":"https://api.github.com/users/wipatrick/followers","following_url":"https://api.github.com/users/wipatrick/following{/other_user}","gists_url":"https://api.github.com/users/wipatrick/gists{/gist_id}","starred_url":"https://api.github.com/users/wipatrick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wipatrick/subscriptions","organizations_url":"https://api.github.com/users/wipatrick/orgs","repos_url":"https://api.github.com/users/wipatrick/repos","events_url":"https://api.github.com/users/wipatrick/events{/privacy}","received_events_url":"https://api.github.com/users/wipatrick/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/biggis-project/biggis-infrastructure/labels/help%20wanted","name":"help wanted","color":"159818"}],"state":"open","locked":false,"assignee":{"login":"vsimko","id":7080773,"avatar_url":"https://avatars.githubusercontent.com/u/7080773?v=3","gravatar_id":"","url":"https://api.github.com/users/vsimko","html_url":"https://github.com/vsimko","followers_url":"https://api.github.com/users/vsimko/followers","following_url":"https://api.github.com/users/vsimko/following{/other_user}","gists_url":"https://api.github.com/users/vsimko/gists{/gist_id}","starred_url":"https://api.github.com/users/vsimko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vsimko/subscriptions","organizations_url":"https://api.github.com/users/vsimko/orgs","repos_url":"https://api.github.com/users/vsimko/repos","events_url":"https://api.github.com/users/vsimko/events{/privacy}","received_events_url":"https://api.github.com/users/vsimko/received_events","type":"User","site_admin":false},"assignees":[{"login":"vsimko","id":7080773,"avatar_url":"https://avatars.githubusercontent.com/u/7080773?v=3","gravatar_id":"","url":"https://api.github.com/users/vsimko","html_url":"https://github.com/vsimko","followers_url":"https://api.github.com/users/vsimko/followers","following_url":"https://api.github.com/users/vsimko/following{/other_user}","gists_url":"https://api.github.com/users/vsimko/gists{/gist_id}","starred_url":"https://api.github.com/users/vsimko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vsimko/subscriptions","organizations_url":"https://api.github.com/users/vsimko/orgs","repos_url":"https://api.github.com/users/vsimko/repos","events_url":"https://api.github.com/users/vsimko/events{/privacy}","received_events_url":"https://api.github.com/users/vsimko/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-07-26T14:51:16Z","updated_at":"2016-07-26T15:03:24Z","closed_at":null,"body":"Description of M3 raster pipeline workflow needs to be reviewed for README.md. @vsimko could you please add missing points?\r\n\r\n> As a first PoC, we want to demonstrate an integrated, end-to-end, analytical BigGIS pipeline for a specific use case. Therefore, we are considering performing a [hot spot analysis](https://pro.arcgis.com/de/pro-app/tool-reference/spatial-statistics/h-how-hot-spot-analysis-getis-ord-gi-spatial-stati.htm) on a thermal flight dataset. The workflow of our M3 Raster-Pipeline is as follows:\r\n> \r\n> 1. Pre-chopped tiles of thermal flight dataset are dropped sequentially to specific data directory on the Docker Host, which the ```Collector``` container is monitoring via a simple _inotify_ script. Detected changes are fetched, such that when a new tile arrives, a new _collect-event_ is pushed to ```Kafka```. This event contains some metadata information about the file, e.g. _path/to/file_.\r\n> 2. ```Flink``` consists of several individual jobs: the first job consumes the _collect-event_, loads in the raster file and normalizes. Normalized tiles are then stored under _/normtiles_ volume on the host and a new event _norm-event_ is pushed out to ```Kafka```. The second job consumes the _norm-event_, takes the normalized tiles and indexes them. These indices are stored in ```DB``` and a new event _idx-event_ is pushed out to ```Kafka```. The third job consumes the _idx-event_, loads the normalized tiles and the corresponding indices and performs a hot spot analysis, using the neighborhood information. The results (z-scores) are pushed in ```DB```, new hot spot raster tiles are stored under _/hstiles_ and a new event _hs-event_ is pushed to ```Kafka```. The last job consumes the _hs-event_, uses the hot spot raster tiles as well as the z-scores and ranks them accordingly. The resulting dataset contains TopK hot spot (cold spot) locations in the given area, which is again stored in the ```DB```. Eventually, a _topk-event_ is committed to ```Kafka```.\r\n> 3. Finally, the creation of a new (part) of raster image is initialized by receiving the _topk-event_, loading the geolocations from ```DB``` and the hot spot tiles from _/hstiles_.\r\n> \r\n> \r\n> ![pipeline-workflow](https://cloud.githubusercontent.com/assets/15153294/17138347/1da64562-5340-11e6-9c50-9cd13d532b57.png)"},"comment":{"url":"https://api.github.com/repos/biggis-project/biggis-infrastructure/issues/comments/235296583","html_url":"https://github.com/biggis-project/biggis-infrastructure/issues/12#issuecomment-235296583","issue_url":"https://api.github.com/repos/biggis-project/biggis-infrastructure/issues/12","id":235296583,"user":{"login":"vsimko","id":7080773,"avatar_url":"https://avatars.githubusercontent.com/u/7080773?v=3","gravatar_id":"","url":"https://api.github.com/users/vsimko","html_url":"https://github.com/vsimko","followers_url":"https://api.github.com/users/vsimko/followers","following_url":"https://api.github.com/users/vsimko/following{/other_user}","gists_url":"https://api.github.com/users/vsimko/gists{/gist_id}","starred_url":"https://api.github.com/users/vsimko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vsimko/subscriptions","organizations_url":"https://api.github.com/users/vsimko/orgs","repos_url":"https://api.github.com/users/vsimko/repos","events_url":"https://api.github.com/users/vsimko/events{/privacy}","received_events_url":"https://api.github.com/users/vsimko/received_events","type":"User","site_admin":false},"created_at":"2016-07-26T15:03:24Z","updated_at":"2016-07-26T15:03:24Z","body":"Ok, I'll draw something"}}
{"actor":{"display_login":"vsimko"}}