{"action":"created","issue":{"url":"https://api.github.com/repos/mapbox/carmen/issues/445","repository_url":"https://api.github.com/repos/mapbox/carmen","labels_url":"https://api.github.com/repos/mapbox/carmen/issues/445/labels{/name}","comments_url":"https://api.github.com/repos/mapbox/carmen/issues/445/comments","events_url":"https://api.github.com/repos/mapbox/carmen/issues/445/events","html_url":"https://github.com/mapbox/carmen/issues/445","id":139098317,"number":445,"title":"Setup tests for multipart-indexing","user":{"login":"yhahn","id":83384,"avatar_url":"https://avatars.githubusercontent.com/u/83384?v=3","gravatar_id":"","url":"https://api.github.com/users/yhahn","html_url":"https://github.com/yhahn","followers_url":"https://api.github.com/users/yhahn/followers","following_url":"https://api.github.com/users/yhahn/following{/other_user}","gists_url":"https://api.github.com/users/yhahn/gists{/gist_id}","starred_url":"https://api.github.com/users/yhahn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yhahn/subscriptions","organizations_url":"https://api.github.com/users/yhahn/orgs","repos_url":"https://api.github.com/users/yhahn/repos","events_url":"https://api.github.com/users/yhahn/events{/privacy}","received_events_url":"https://api.github.com/users/yhahn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-03-07T21:22:24Z","updated_at":"2016-03-11T18:26:16Z","closed_at":null,"body":"Let's walk through this backlog together and I can teach you about parts of carmen as we go along.\r\n\r\n- [x] create new branch from master\r\n- [x] add `test/index.merge.test.js` -- we can use the first parts of `index.test.js` as a guide to how to create and index some features\r\n- [x] we'll add tests for 3 different mbtiles indexes. Each test will\r\n  - index A: lines 1-100\r\n  - index B: lines 101-200,\r\n  - index C: lines 201-254\r\n  - of the indexing datasource (https://github.com/mapbox/carmen/blob/master/test/fixtures/docs.jsonl)\r\n  - query each index and confirm it has some of the features from its lines (but not features from the other lines)\r\n- [ ] **SKIP** - merge the indexes and query for any of the features. We'll add this once @apendleton's branch is up + merged into ours.\r\n- [ ] cleanup the indexes\r\n\r\ncc @aarthykc"},"comment":{"url":"https://api.github.com/repos/mapbox/carmen/issues/comments/195488387","html_url":"https://github.com/mapbox/carmen/issues/445#issuecomment-195488387","issue_url":"https://api.github.com/repos/mapbox/carmen/issues/445","id":195488387,"user":{"login":"apendleton","id":78930,"avatar_url":"https://avatars.githubusercontent.com/u/78930?v=3","gravatar_id":"","url":"https://api.github.com/users/apendleton","html_url":"https://github.com/apendleton","followers_url":"https://api.github.com/users/apendleton/followers","following_url":"https://api.github.com/users/apendleton/following{/other_user}","gists_url":"https://api.github.com/users/apendleton/gists{/gist_id}","starred_url":"https://api.github.com/users/apendleton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apendleton/subscriptions","organizations_url":"https://api.github.com/users/apendleton/orgs","repos_url":"https://api.github.com/users/apendleton/repos","events_url":"https://api.github.com/users/apendleton/events{/privacy}","received_events_url":"https://api.github.com/users/apendleton/received_events","type":"User","site_admin":false},"created_at":"2016-03-11T18:26:16Z","updated_at":"2016-03-11T18:26:16Z","body":"The current state of affairs: the following things are being merged in the merge-redux branch:\r\n* freq\r\n* stat (so: the DAWG cache)\r\n* feature\r\n* grid\r\n\r\nMissing from this list is vector tiles, which are not being merged in carmen at present. We don't have a good plan for merging vector tiles, and for production use our current plan is to not try, and instead stash feature geojson during the indexing process, and then concatenate that and build vector tiles out of it outside of carmen. Preliminarily this looks like it will work fine, but leaves us with the question of how to test merging within carmen. The testing strategy so far anticipates being able to perform geocodes on merged indexes, but without vector tiles in the merged indexes, this won't work. At this point, I see a few paths forward for testing:\r\n* **make tests lower-level:** rather than attempting to perform geocodes, test the contents of individual types in the merged index (e.g., contents of individual freq, feature, or grid shards) against what we expect the values to be\r\n  * *harder variant:* rather than testing against what we think the values should be, separately index the combined data in a single pass, so that we have two indexes, one the result of a merge, and one from an all-at-once build, that should contain the same data, and then compare the contents of the index. This will be challenging because carmen's decision-making about dropping low-value features is based on the frequency distribution of the terms it's indexing at any given time, and it may make slightly different decisions when building sub-indexes than when building the full index, which might lead to small differences in, at a minimum, the DAWG contents and the feature index, and probably also freq and grid. We'd need to build in a fudge factor to make this work right.\r\n* **mimic the production vector tile merge strategy:** store the geojson of the features we're indexing in tests as we index them, and manually combine them after merge and add them to tiles in the merged index. This would let us run geocodes to confirm that the other datatypes in the index are correct.\r\n* **use mbtiles instead of api-mem for the merge tests:** this would let us use similar tooling for tests as we intend to use in production, rather than having to reinvent it on top of api-mem\r\n\r\nI'm not sure which of these is the best way forward."}}