{"action":"created","issue":{"url":"https://api.github.com/repos/OpenSextant/Gazetteer/issues/1","repository_url":"https://api.github.com/repos/OpenSextant/Gazetteer","labels_url":"https://api.github.com/repos/OpenSextant/Gazetteer/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/OpenSextant/Gazetteer/issues/1/comments","events_url":"https://api.github.com/repos/OpenSextant/Gazetteer/issues/1/events","html_url":"https://github.com/OpenSextant/Gazetteer/issues/1","id":168986509,"number":1,"title":"County missing from Merged.","user":{"login":"mubaldino","id":4236378,"avatar_url":"https://avatars.githubusercontent.com/u/4236378?v=3","gravatar_id":"","url":"https://api.github.com/users/mubaldino","html_url":"https://github.com/mubaldino","followers_url":"https://api.github.com/users/mubaldino/followers","following_url":"https://api.github.com/users/mubaldino/following{/other_user}","gists_url":"https://api.github.com/users/mubaldino/gists{/gist_id}","starred_url":"https://api.github.com/users/mubaldino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mubaldino/subscriptions","organizations_url":"https://api.github.com/users/mubaldino/orgs","repos_url":"https://api.github.com/users/mubaldino/repos","events_url":"https://api.github.com/users/mubaldino/events{/privacy}","received_events_url":"https://api.github.com/users/mubaldino/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2016-08-02T20:56:07Z","updated_at":"2016-08-20T18:54:09Z","closed_at":null,"body":"We ran a doc through Xponents;  Found that many \"Lee County\" entries were missing. \r\nI went back to my Gazetteer run and found Lee County indeed exists in USGS transformed, but not at all in the final GeoData/Merged/ final output.\r\n \r\nUSGS1496538\tLee County\t\t36.7054481\t-83.1285285\tA\tADM2\tUS\tUS\tUSA\tUS51\tUS51.0105\t\t\t\tUSGS\t1496538\t\tname\t\tF\r\n\r\nI wonder if there is something too aggressive with the conflation of entries.\r\n\r\n205 entries in transformed USGS intermediate gazetteer for grep \"lee county\"\r\n189 entries in Merged.   In particular, the one above is missing.\r\n\r\nThe concern is that this may be wider spread than just the one above.\r\n\r\nMarc\r\n"},"comment":{"url":"https://api.github.com/repos/OpenSextant/Gazetteer/issues/comments/241217237","html_url":"https://github.com/OpenSextant/Gazetteer/issues/1#issuecomment-241217237","issue_url":"https://api.github.com/repos/OpenSextant/Gazetteer/issues/1","id":241217237,"user":{"login":"dsmiley","id":377295,"avatar_url":"https://avatars.githubusercontent.com/u/377295?v=3","gravatar_id":"","url":"https://api.github.com/users/dsmiley","html_url":"https://github.com/dsmiley","followers_url":"https://api.github.com/users/dsmiley/followers","following_url":"https://api.github.com/users/dsmiley/following{/other_user}","gists_url":"https://api.github.com/users/dsmiley/gists{/gist_id}","starred_url":"https://api.github.com/users/dsmiley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dsmiley/subscriptions","organizations_url":"https://api.github.com/users/dsmiley/orgs","repos_url":"https://api.github.com/users/dsmiley/repos","events_url":"https://api.github.com/users/dsmiley/events{/privacy}","received_events_url":"https://api.github.com/users/dsmiley/received_events","type":"User","site_admin":false},"created_at":"2016-08-20T18:54:09Z","updated_at":"2016-08-20T18:54:09Z","body":"Ouch; who's bug?  I'm guessing this was done because maybe Kettle doesn't have a disk-resident de-dup'er and thus it won't scale (i.e. would use gobs of memory)?  Just guessing.  That's the kind of thing I would expect/want out of an ETL tool.\r\n\r\np.s. Lucene has to tackle this internally -- it has a disk based sorter (similar requirement to dedup) when the sort task would use too much memory.  \r\n\r\nIf we wanted to develop a disk based de-duper, you might use a mini Lucene index with one indexed StringField (no tokenization). To avoid committing after every single document (anti-pattern -- slow), you'd have a simple HashSet capped at maybe 1000 before flushing to Lucene that is consulted in conjunction with the Lucene index."}}
{"actor":{"display_login":"dsmiley"}}