{"action":"created","issue":{"url":"https://api.github.com/repos/openbmap/android/issues/170","repository_url":"https://api.github.com/repos/openbmap/android","labels_url":"https://api.github.com/repos/openbmap/android/issues/170/labels{/name}","comments_url":"https://api.github.com/repos/openbmap/android/issues/170/comments","events_url":"https://api.github.com/repos/openbmap/android/issues/170/events","html_url":"https://github.com/openbmap/android/issues/170","id":187206050,"number":170,"title":"Re-add download of full cell/wifi catalog","user":{"login":"mvglasow","id":2377496,"avatar_url":"https://avatars.githubusercontent.com/u/2377496?v=3","gravatar_id":"","url":"https://api.github.com/users/mvglasow","html_url":"https://github.com/mvglasow","followers_url":"https://api.github.com/users/mvglasow/followers","following_url":"https://api.github.com/users/mvglasow/following{/other_user}","gists_url":"https://api.github.com/users/mvglasow/gists{/gist_id}","starred_url":"https://api.github.com/users/mvglasow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mvglasow/subscriptions","organizations_url":"https://api.github.com/users/mvglasow/orgs","repos_url":"https://api.github.com/users/mvglasow/repos","events_url":"https://api.github.com/users/mvglasow/events{/privacy}","received_events_url":"https://api.github.com/users/mvglasow/received_events","type":"User","site_admin":false},"labels":[{"id":243082874,"url":"https://api.github.com/repos/openbmap/android/labels/bug","name":"bug","color":"fc2929","default":true},{"id":330905697,"url":"https://api.github.com/repos/openbmap/android/labels/website","name":"website","color":"bfdadc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-11-03T22:19:02Z","updated_at":"2016-11-04T17:35:53Z","closed_at":null,"body":"What steps will reproduce the problem?\r\nGo to the web site, navigate to *Downloads > Cell and WiFi data* and choose *Worldwide dataset*.\r\n\r\nWhat is the expected output? What do you see instead?\r\nI'd expect the full cell/wifi catalog as it used to be. Instead, I get a *404 Not Found*.\r\n\r\nWhat version are you using? On what operating system?\r\nDoes not apply.\r\n\r\nPlease provide any additional information below.\r\nSimilar issues arise when trying to download the wifi catalog from Radiobeacon or the NLP backend – only individual countries are selectable but there's no way to download the full set.\r\n\r\nI understand that this change was probably introduced because of bandwidth issues caused by many users downloading the whole thing over and over again, but limiting downloads to one country comes with other side effects. For example, I need to download multiple catalogs and switch manually whenever I cross a border.\r\n\r\nI see three possible approaches to tackle this:\r\n\r\n**1. Simply restore the option to download the full catalog**\r\nWhat probably contributed to the bandwidth issues was that originally one needed to download the whole catalog or nothing at all, because nothing else was available. Now that users can limit their downloads to particular areas of interest, this may be less of an issue.\r\n\r\n**2. Host the catalogs on a public cloud service**\r\nMaybe Google Drive, maybe even Github, if their terms of service permit that. Anything that permits sharing (somewhat) arbitrary files over 1 GB in size and allows downloads by anonymous users would work.\r\n\r\n**3. Implement differential updates**\r\nAbandon the idea of the catalog being a file of which clients periodically download a new version. Rather, start with a blank database on the client into which the client loads the data it needs. (This can still be in the form of *.sqlite* databases, of course.) Downloading the whole set of data is only necessary the first time. After that, clients/users just fetch those records which were added or changed since the last update. And since the catalog already has a \"last updated\" timestamp column, clients have all the information they need to merge two databases – it would just need to be implemented.\r\n\r\nOn the server side, this could be implemented in the following way:\r\n* The full catalog (as well as any per-country catalogs) remains available as it is.\r\n* Additionally, diffs are offered – for the whole world and optionally for individual countries. Structure could be something like:\r\n    * Yearly diffs since the start of the project, until the end of last year\r\n    * Monthly diffs for the current year (all that's not in the last yearly diff), until the end of last month\r\n    * Weekly diffs for the current month (all that's not in the last monthly diff), until the end of last week\r\n    * Daily diffs for the current week\r\n* Each diff is a *.sqlite* file, just like a full catalog – the only difference being the records it contains.\r\n* Timestamps for a catalog entry must increase with each update.\r\n* To make generating diffs easier, timestamps should reflect the time the entry was added to/updated in the central catalog. If I've scanned a few wifis on Monday but don't upload them until Wednesday, they'll have Wednesday's timestamp. If this is satisfied, diffs can be generated from the full catalog any time.\r\n* Diffs could either be pre-generated and stored on the server, or generated on the fly. That's basically a trade-off between storage vs. processing power on the server.\r\n\r\nWith client-side support, databases (full and diffs) could then be imported in any order. It comes with some coding but is by far the most elegant solution – and the most efficient one in terms of bandwidth."},"comment":{"url":"https://api.github.com/repos/openbmap/android/issues/comments/258498062","html_url":"https://github.com/openbmap/android/issues/170#issuecomment-258498062","issue_url":"https://api.github.com/repos/openbmap/android/issues/170","id":258498062,"user":{"login":"mvglasow","id":2377496,"avatar_url":"https://avatars.githubusercontent.com/u/2377496?v=3","gravatar_id":"","url":"https://api.github.com/users/mvglasow","html_url":"https://github.com/mvglasow","followers_url":"https://api.github.com/users/mvglasow/followers","following_url":"https://api.github.com/users/mvglasow/following{/other_user}","gists_url":"https://api.github.com/users/mvglasow/gists{/gist_id}","starred_url":"https://api.github.com/users/mvglasow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mvglasow/subscriptions","organizations_url":"https://api.github.com/users/mvglasow/orgs","repos_url":"https://api.github.com/users/mvglasow/repos","events_url":"https://api.github.com/users/mvglasow/events{/privacy}","received_events_url":"https://api.github.com/users/mvglasow/received_events","type":"User","site_admin":false},"created_at":"2016-11-04T17:35:53Z","updated_at":"2016-11-04T17:35:53Z","body":"Thanks, looks like the DB is up again. There used to be a JSON with version information at http://radiocells.org/default/database_version.json – where did that end up?"}}
{"actor":{"display_login":"mvglasow"}}