{"action":"created","issue":{"url":"https://api.github.com/repos/openvenues/libpostal/issues/132","repository_url":"https://api.github.com/repos/openvenues/libpostal","labels_url":"https://api.github.com/repos/openvenues/libpostal/issues/132/labels{/name}","comments_url":"https://api.github.com/repos/openvenues/libpostal/issues/132/comments","events_url":"https://api.github.com/repos/openvenues/libpostal/issues/132/events","html_url":"https://github.com/openvenues/libpostal/issues/132","id":192402811,"number":132,"title":"Splitting data files by country and language","user":{"login":"rinigus","id":15706605,"avatar_url":"https://avatars.githubusercontent.com/u/15706605?v=3","gravatar_id":"","url":"https://api.github.com/users/rinigus","html_url":"https://github.com/rinigus","followers_url":"https://api.github.com/users/rinigus/followers","following_url":"https://api.github.com/users/rinigus/following{/other_user}","gists_url":"https://api.github.com/users/rinigus/gists{/gist_id}","starred_url":"https://api.github.com/users/rinigus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rinigus/subscriptions","organizations_url":"https://api.github.com/users/rinigus/orgs","repos_url":"https://api.github.com/users/rinigus/repos","events_url":"https://api.github.com/users/rinigus/events{/privacy}","received_events_url":"https://api.github.com/users/rinigus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2016-11-29T21:21:12Z","updated_at":"2016-12-08T00:38:06Z","closed_at":null,"body":"To be able to use libpostal on mobile, it would be advantageous to split data files by language and country. Is such split possible?"},"comment":{"url":"https://api.github.com/repos/openvenues/libpostal/issues/comments/265618685","html_url":"https://github.com/openvenues/libpostal/issues/132#issuecomment-265618685","issue_url":"https://api.github.com/repos/openvenues/libpostal/issues/132","id":265618685,"user":{"login":"thatdatabaseguy","id":238455,"avatar_url":"https://avatars.githubusercontent.com/u/238455?v=3","gravatar_id":"","url":"https://api.github.com/users/thatdatabaseguy","html_url":"https://github.com/thatdatabaseguy","followers_url":"https://api.github.com/users/thatdatabaseguy/followers","following_url":"https://api.github.com/users/thatdatabaseguy/following{/other_user}","gists_url":"https://api.github.com/users/thatdatabaseguy/gists{/gist_id}","starred_url":"https://api.github.com/users/thatdatabaseguy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatdatabaseguy/subscriptions","organizations_url":"https://api.github.com/users/thatdatabaseguy/orgs","repos_url":"https://api.github.com/users/thatdatabaseguy/repos","events_url":"https://api.github.com/users/thatdatabaseguy/events{/privacy}","received_events_url":"https://api.github.com/users/thatdatabaseguy/received_events","type":"User","site_admin":false},"created_at":"2016-12-08T00:38:06Z","updated_at":"2016-12-08T00:38:06Z","body":"Ok, so generating the data sets from a raw OSM dump is possible, but may require a very large machine (like one with ~64GB of memory) and about 7-8 days of compute time for planet, and there are several indexes that need to be built first. In an all EU model on some of the smaller OSM dumps coming from geofabrik or something like that, maybe cut those numbers in half.\r\n\r\nIt may be tempting to try forgoing all of the normalization and just mapping the values of OSM addr:* tags to that libpostal/address-formatting schema. That unfortunately is fraught with problems, many of which I've detailed in the libpostal blog post. One of the big ones is that libpostal relies heavily on gazetteers (memorization) for city names, etc. and at training time it needs to have seen basically every city that it's expected to parse at runtime. This includes name variations, so if for instance only München is seen at training time and the user types Munich, the parser almost certainly would get that wrong, whereas a traditional TFIDF-based search system might not. The addr:city tags can be pretty sparse. Around 76% of OSM tags that include addr:street also include addr:city, which is not bad overall, but isn't applied consistently across cities, and non-local names like Munich are usually not included, so it might miss some conspicuous examples, especially in a general routing/search application where you need to be able to find the tiny cities, hamlets, etc. or need it to work for tourists as well as locals.\r\n\r\nThe other area that doesn't work well using vanilla OSM tags is venue search due to the fact that only 9-10% of OSM venues have any address tags, so the parser won't see many examples of \"venue + street address\", \"venue + city\", etc. Usually it will just see venue by itself, and, because the model predicts from left to right and uses its previous predicted tag to inform the next, it will often lump the entire string into the venue name. For OSM we combat this by actually reverse geocoding venues to both admin boundaries and building polygons, which often do have address tags, and adding addresses that way.\r\n\r\nReverse geocoding to the admins is what takes up the majority of the memory and time in building the training set, but it's also one of the largest contributors to the parser's accuracy and robustness. Much of this \"hydration\" is quite similar to what a geocoder like Nominatim does in the results you see when searching OSM.\r\n\r\nThere are also dozens of different random normalizations that we apply so variations of abbreviated/hyphenated city names (Saint Pierre, Saint-Pierre, St.-Pierre, St. Pierre), variations with and without accents (München, Munchen, Muenchen), etc. are seen in the training data.\r\n\r\nBecause it's fairly resource-intensive and a reasonably heavy cognitive load, I thought it might be better if I just push out the resulting planet training data files when the next batch finishes building and let people slice and dice it from there using grep/awk, etc. It would be great to have more eyes on the training data to spot any systematic issues. The process of training the model on a given segment of the input is just ```./src/address_parser_train input_file output_dir```, which is substantially easier and can be run in parallel for different models.\r\n\r\nJust checked and the geonames/postal_codes TSVs are indeed splittable by country as well. Those can be downloaded from S3 now: [geonames.tsv](https://libpostal.s3.amazonaws.com/geodb/geonames.tsv) and [postal_codes.tsv](https://libpostal.s3.amazonaws.com/geodb/postal_codes.tsv). License there is CC-BY, same as GeoNames.\r\n\r\nThe latest batch of training data is still being generated, but will publish it to S3 sometime next week most likely."}}
{"actor":{"display_login":"thatdatabaseguy"}}