{"action":"created","issue":{"url":"https://api.github.com/repos/developmentseed/planet-stream/issues/18","repository_url":"https://api.github.com/repos/developmentseed/planet-stream","labels_url":"https://api.github.com/repos/developmentseed/planet-stream/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/developmentseed/planet-stream/issues/18/comments","events_url":"https://api.github.com/repos/developmentseed/planet-stream/issues/18/events","html_url":"https://github.com/developmentseed/planet-stream/issues/18","id":159736991,"number":18,"title":"Changeset metadata intermittently fails to save to redis","user":{"login":"kriscarle","id":2431570,"avatar_url":"https://avatars.githubusercontent.com/u/2431570?v=3","gravatar_id":"","url":"https://api.github.com/users/kriscarle","html_url":"https://github.com/kriscarle","followers_url":"https://api.github.com/users/kriscarle/followers","following_url":"https://api.github.com/users/kriscarle/following{/other_user}","gists_url":"https://api.github.com/users/kriscarle/gists{/gist_id}","starred_url":"https://api.github.com/users/kriscarle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kriscarle/subscriptions","organizations_url":"https://api.github.com/users/kriscarle/orgs","repos_url":"https://api.github.com/users/kriscarle/repos","events_url":"https://api.github.com/users/kriscarle/events{/privacy}","received_events_url":"https://api.github.com/users/kriscarle/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-06-10T22:42:18Z","updated_at":"2016-07-18T19:47:55Z","closed_at":null,"body":"I'm seeing a large number of \"not found\" messages for metadata that is missing from redis so it never gets joined and sent out. I've confirmed that planet stream is loading the changeset metadata file from OSM, but occasionally it fails to  store an entire file's worth of changesets in redis. \r\n\r\nI'm going to turn on DEBUG=* and see if I can get and details from the redis client logs.\r\n\r\nI'm thinking it might happen if overpass also comes back at the same time. Here are the logs around a file that didn't save:\r\n\r\n```\r\nFri Jun 10 2016 21:39:16 GMT+0000 (UTC)] INFO Polling OSM planet\r\n[Fri Jun 10 2016 21:39:19 GMT+0000 (UTC)] INFO Retrieving http://planet.osm.org/replication/changesets/001/880/581.osm.gz from planet\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] INFO Polling Overpass\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] INFO Matching 24 members\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]7085924\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]7166210\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]13897298\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]21938396\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]22146408\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39591061\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39938357\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39941586\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39941739\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942584\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942654\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942703\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942704\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942705\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942706\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942707\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942708\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942709\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942710\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942711\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942712\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942713\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942714\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942715\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] INFO Polling OSM planet\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 7085924 not found\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 7166210 not found\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 13897298 not found\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 21938396 not found\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 22146408 not found\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 39591061 not found\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 39938357 not found\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 39941586 not found\r\n[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 39941739 not found\r\n```\r\n\r\nHere is an example of all the not found errors I'm seeing in AWS.\r\n\r\n![image](https://cloud.githubusercontent.com/assets/2431570/15980513/35a159ce-2f3a-11e6-96d5-2b91d23a943f.png)\r\n"},"comment":{"url":"https://api.github.com/repos/developmentseed/planet-stream/issues/comments/233437611","html_url":"https://github.com/developmentseed/planet-stream/issues/18#issuecomment-233437611","issue_url":"https://api.github.com/repos/developmentseed/planet-stream/issues/18","id":233437611,"user":{"login":"kamicut","id":719357,"avatar_url":"https://avatars.githubusercontent.com/u/719357?v=3","gravatar_id":"","url":"https://api.github.com/users/kamicut","html_url":"https://github.com/kamicut","followers_url":"https://api.github.com/users/kamicut/followers","following_url":"https://api.github.com/users/kamicut/following{/other_user}","gists_url":"https://api.github.com/users/kamicut/gists{/gist_id}","starred_url":"https://api.github.com/users/kamicut/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kamicut/subscriptions","organizations_url":"https://api.github.com/users/kamicut/orgs","repos_url":"https://api.github.com/users/kamicut/repos","events_url":"https://api.github.com/users/kamicut/events{/privacy}","received_events_url":"https://api.github.com/users/kamicut/received_events","type":"User","site_admin":false},"created_at":"2016-07-18T19:47:54Z","updated_at":"2016-07-18T19:47:54Z","body":"@kriscarle I definitely echo your sentiment for looking at a dockerized version of `osm-stats`, we would need to find alternatives to kinesis and lambda workers. As for planet-stream, I started work on a branch using `kue`, a node.js task manager backed by redis. It has support for task retry, and delaying tasks. That could structure planet-stream's logging of missed syncs between the data and metadata. \r\n\r\nOnce it's at a good point, it would be great if you could take a look at it. I think it could solve this issue, or at the very least have better logs.  "}}
{"actor":{"display_login":"kamicut"}}