{"action":"opened","issue":{"url":"https://api.github.com/repos/NextGeoss/ckanext-nextgeossharvest/issues/25","repository_url":"https://api.github.com/repos/NextGeoss/ckanext-nextgeossharvest","labels_url":"https://api.github.com/repos/NextGeoss/ckanext-nextgeossharvest/issues/25/labels{/name}","comments_url":"https://api.github.com/repos/NextGeoss/ckanext-nextgeossharvest/issues/25/comments","events_url":"https://api.github.com/repos/NextGeoss/ckanext-nextgeossharvest/issues/25/events","html_url":"https://github.com/NextGeoss/ckanext-nextgeossharvest/issues/25","id":305935182,"number":25,"title":"CMEMS needs a mechanism for recovering from outages","user":{"login":"kmbn","id":5577223,"avatar_url":"https://avatars3.githubusercontent.com/u/5577223?v=4","gravatar_id":"","url":"https://api.github.com/users/kmbn","html_url":"https://github.com/kmbn","followers_url":"https://api.github.com/users/kmbn/followers","following_url":"https://api.github.com/users/kmbn/following{/other_user}","gists_url":"https://api.github.com/users/kmbn/gists{/gist_id}","starred_url":"https://api.github.com/users/kmbn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmbn/subscriptions","organizations_url":"https://api.github.com/users/kmbn/orgs","repos_url":"https://api.github.com/users/kmbn/repos","events_url":"https://api.github.com/users/kmbn/events{/privacy}","received_events_url":"https://api.github.com/users/kmbn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-16T13:51:22Z","updated_at":"2018-03-16T13:51:22Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Consider the following two scenarios:\r\n1. The data hub is harvesting daily using the CMEMS harvester. The data hub suffers an outage that lasts one day. When the data hub comes back online and resumes harvesting, it will not harvest the datasets that it should have harvested during the outage. There will be a one-day gap in the database. The longer the outage, the longer the gap. The harvester has no way of going back and harvesting from the day of the outage rather than from the day the data hub came back online.\r\n2. One of the sources that the CMEMS harvester harvests suffers an outage that lasts one day. The harvester does not harvest the datasets that it should have harvested that day. The next day, the source is back online. The harvester continues harvesting as before, and it will not harvest the datasets that it should have harvested during the outage. There will be a one-day gap in the database. The longer the outage, the longer the gap. The harvester has no way of going back and harvesting from the day of the outage rather than from the day the data hub came back online. Furthermore, since the CMEMS harvester harvests from multiple sources rather than from a single source, it won't be immediately apparent which source needs to be reharvested. And there is no easy way to have the harvester restart from the day of the outage because the sources will be out sync—the harvester would need to go back a day for the source that had the outage and also continue harvesting normally from the sources that didn't suffer outages."}}
{"actor":{"display_login":"kmbn"}}