{"action":"created","issue":{"url":"https://api.github.com/repos/m-lab/mlab-ns/issues/137","repository_url":"https://api.github.com/repos/m-lab/mlab-ns","labels_url":"https://api.github.com/repos/m-lab/mlab-ns/issues/137/labels{/name}","comments_url":"https://api.github.com/repos/m-lab/mlab-ns/issues/137/comments","events_url":"https://api.github.com/repos/m-lab/mlab-ns/issues/137/events","html_url":"https://github.com/m-lab/mlab-ns/pull/137","id":321374633,"number":137,"title":"Use MaxMind GeoLite2 databases instead of the deprecated \"legacy\" ones.","user":{"login":"nkinkade","id":1392825,"avatar_url":"https://avatars2.githubusercontent.com/u/1392825?v=4","gravatar_id":"","url":"https://api.github.com/users/nkinkade","html_url":"https://github.com/nkinkade","followers_url":"https://api.github.com/users/nkinkade/followers","following_url":"https://api.github.com/users/nkinkade/following{/other_user}","gists_url":"https://api.github.com/users/nkinkade/gists{/gist_id}","starred_url":"https://api.github.com/users/nkinkade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nkinkade/subscriptions","organizations_url":"https://api.github.com/users/nkinkade/orgs","repos_url":"https://api.github.com/users/nkinkade/repos","events_url":"https://api.github.com/users/nkinkade/events{/privacy}","received_events_url":"https://api.github.com/users/nkinkade/received_events","type":"User","site_admin":false},"labels":[{"id":568586385,"url":"https://api.github.com/repos/m-lab/mlab-ns/labels/in%20progress","name":"in progress","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-05-08T22:56:04Z","updated_at":"2018-05-14T17:34:53Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/m-lab/mlab-ns/pulls/137","html_url":"https://github.com/m-lab/mlab-ns/pull/137","diff_url":"https://github.com/m-lab/mlab-ns/pull/137.diff","patch_url":"https://github.com/m-lab/mlab-ns/pull/137.patch"},"body":"Currently, we are using an outdated, deprecated, legacy MaxMind database that will not receive many updates going forward, if any, and will [completely removed from MaxMind's website on January 2, 2019](https://support.maxmind.com/geolite-legacy-discontinuation-notice/).\r\n\r\nThis PR implements the changes necessary to start using the current [GeoLite2 databases](https://dev.maxmind.com/geoip/geoip2/geolite2/).\r\n\r\n**NOTE**: This PR is not at all as large as it seems. All changes in the `mlabns/third_party/*` directory, which is the overwhelming vast majority of apparent changes in this PR, can be ignored, since it is all purely 3rd party vendor code fetched by pip. However, it should be noted that there *are* changes to the `geoip2` code, for which I'll be submitting a PR to MaxMind for, and is separate from this PR.\r\n\r\nOf note in the PR:\r\n* The MaxMind database file is no longer included in this repository. GAE has a limit of 32MB for static code files, and the new GeoLite2 database is 55MB. Instead, the file is stored in GCS and fetched from there (location of this file is in `util.constants`.\r\n* The new GeoLite2 database contain information for both IPv4 and IPv6 lookups, thus we no longer need to distinguish between the two when doing lookups.\r\n* There is a new `util.handler` class which will allow us to update the running version of the MaxMind database by simply fetching a URL from the GAE app (`/reload_maxminddb`).\r\n\r\n<!-- Reviewable:start -->\r\n---\r\nThis change is [<img src=\"https://reviewable.io/review_button.svg\" height=\"34\" align=\"absmiddle\" alt=\"Reviewable\"/>](https://reviewable.io/reviews/m-lab/mlab-ns/137)\r\n<!-- Reviewable:end -->\r\n"},"comment":{"url":"https://api.github.com/repos/m-lab/mlab-ns/issues/comments/388900091","html_url":"https://github.com/m-lab/mlab-ns/pull/137#issuecomment-388900091","issue_url":"https://api.github.com/repos/m-lab/mlab-ns/issues/137","id":388900091,"user":{"login":"nkinkade","id":1392825,"avatar_url":"https://avatars2.githubusercontent.com/u/1392825?v=4","gravatar_id":"","url":"https://api.github.com/users/nkinkade","html_url":"https://github.com/nkinkade","followers_url":"https://api.github.com/users/nkinkade/followers","following_url":"https://api.github.com/users/nkinkade/following{/other_user}","gists_url":"https://api.github.com/users/nkinkade/gists{/gist_id}","starred_url":"https://api.github.com/users/nkinkade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nkinkade/subscriptions","organizations_url":"https://api.github.com/users/nkinkade/orgs","repos_url":"https://api.github.com/users/nkinkade/repos","events_url":"https://api.github.com/users/nkinkade/events{/privacy}","received_events_url":"https://api.github.com/users/nkinkade/received_events","type":"User","site_admin":false},"created_at":"2018-05-14T17:34:53Z","updated_at":"2018-05-14T17:34:53Z","author_association":"CONTRIBUTOR","body":"@yachang: I think I've addressed your comment and concerns. PTAL?\n\n---\n\nReview status: 0 of 14 files reviewed at latest revision, 4 unresolved discussions, some commit checks failed.\n\n---\n\n*[server/app.yaml, line 4 at r4](https://reviewable.io/reviews/m-lab/mlab-ns/137#-LC0y1QrRYyLyoa4LhLx-r4-4:-LCUhr_kE0cBGM_cRtCd:bwoo150) ([raw file](https://github.com/m-lab/mlab-ns/blob/be500240a8d258e36184ec82df70e53408e24c2f/server/app.yaml#L4)):*\n<details><summary><i>Previously, yachang wrote…</i></summary><blockquote>\n\nResolve with master branch. I have modified that in the roundrobin PR.\n</blockquote></details>\n\nI've removed this change from this PR.\n\n---\n\n*[server/mlabns/tests/test_maxmind.py, line 62 at r3](https://reviewable.io/reviews/m-lab/mlab-ns/137#-LC0y1QoOKE8AQuW6fiX-r3-62:-LCUha4C4QqgEYbnksLn:b-gut99s) ([raw file](https://github.com/m-lab/mlab-ns/blob/66e7c6b50e679e75a33fbf796c2ec7eedb416681/server/mlabns/tests/test_maxmind.py#L62)):*\n<details><summary><i>Previously, yachang wrote…</i></summary><blockquote>\n\nThanks for the explanation. Can you add a TODO for \"create a fake database (same format but much smaller, maybe just several K), then check the validity of an entry using the fake database\" ? Thanks!\n</blockquote></details>\n\nDone. I've add a new TODO.\n\n---\n\n*[server/mlabns/util/constants.py, line 23 at r3](https://reviewable.io/reviews/m-lab/mlab-ns/137#-LC0y1QoOKE8AQuW6fiZ-r3-23:-LCUh-6L36k9DlOVLwh5:bdkbqwr) ([raw file](https://github.com/m-lab/mlab-ns/blob/66e7c6b50e679e75a33fbf796c2ec7eedb416681/server/mlabns/util/constants.py#L23)):*\n<details><summary><i>Previously, yachang wrote…</i></summary><blockquote>\n\nWhich project does the bucket belong to? Could you add documentation for that?\n</blockquote></details>\n\nI'm hesitant to document which GCP project owns the bucket since the bucket name is going to change, and when it changes someone will need to remember to also update the note, and they may get out of sync. Also, because GCS bucket names are global (across every project), which project owns it may be less relevant. I've added a comment on how to use `gsutil` to determine which GCP project owns the bucket. Does that address your concern?\n\n---\n\n\n*Comments from [Reviewable](https://reviewable.io/reviews/m-lab/mlab-ns/137#-:-LCUhwKLEo8kpd8W3d_5:bxsk8t3)*\n<!-- Sent from Reviewable.io -->\n"}}
{"actor":{"display_login":"nkinkade"}}