{"action":"created","issue":{"url":"https://api.github.com/repos/mapbox/cardboard/issues/178","repository_url":"https://api.github.com/repos/mapbox/cardboard","labels_url":"https://api.github.com/repos/mapbox/cardboard/issues/178/labels{/name}","comments_url":"https://api.github.com/repos/mapbox/cardboard/issues/178/comments","events_url":"https://api.github.com/repos/mapbox/cardboard/issues/178/events","html_url":"https://github.com/mapbox/cardboard/issues/178","id":167938557,"number":178,"title":"Full featured bounding box options","user":{"login":"mcwhittemore","id":1551510,"avatar_url":"https://avatars.githubusercontent.com/u/1551510?v=3","gravatar_id":"","url":"https://api.github.com/users/mcwhittemore","html_url":"https://github.com/mcwhittemore","followers_url":"https://api.github.com/users/mcwhittemore/followers","following_url":"https://api.github.com/users/mcwhittemore/following{/other_user}","gists_url":"https://api.github.com/users/mcwhittemore/gists{/gist_id}","starred_url":"https://api.github.com/users/mcwhittemore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mcwhittemore/subscriptions","organizations_url":"https://api.github.com/users/mcwhittemore/orgs","repos_url":"https://api.github.com/users/mcwhittemore/repos","events_url":"https://api.github.com/users/mcwhittemore/events{/privacy}","received_events_url":"https://api.github.com/users/mcwhittemore/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-07-27T19:41:35Z","updated_at":"2016-09-01T18:49:49Z","closed_at":null,"body":"@mick recently wrote up some options for how we can fix bounding box. I'm posting it here to help keep the convo going.\r\n\r\n- *multicell in dynamo* \r\n  - we delete the cell secondary index.\r\n  - our spatial index in another table, its multicell. This is outlined in more detail below \r\n  - a lambda following the main table keeps the cardboard index table up to date.\r\n- *multicell outside of dynamo*\r\n  - delete the cell secondary index.\r\n  - All indexing would happen in elasticsearch both for spatial and on other attributes. \r\n  - a lambda following the main table keeps the cardboard index table up to date.\r\n- *something else*\r\n  - another storage option? could we do better then our core limitations we already have with dynamo (write speed, full dataset reads, etc)\r\n  - this could be time sink, I think at least worth considering since it was over 2 years ago we made that choice. "},"comment":{"url":"https://api.github.com/repos/mapbox/cardboard/issues/comments/244175159","html_url":"https://github.com/mapbox/cardboard/issues/178#issuecomment-244175159","issue_url":"https://api.github.com/repos/mapbox/cardboard/issues/178","id":244175159,"user":{"login":"rclark","id":515424,"avatar_url":"https://avatars.githubusercontent.com/u/515424?v=3","gravatar_id":"","url":"https://api.github.com/users/rclark","html_url":"https://github.com/rclark","followers_url":"https://api.github.com/users/rclark/followers","following_url":"https://api.github.com/users/rclark/following{/other_user}","gists_url":"https://api.github.com/users/rclark/gists{/gist_id}","starred_url":"https://api.github.com/users/rclark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rclark/subscriptions","organizations_url":"https://api.github.com/users/rclark/orgs","repos_url":"https://api.github.com/users/rclark/repos","events_url":"https://api.github.com/users/rclark/events{/privacy}","received_events_url":"https://api.github.com/users/rclark/received_events","type":"User","site_admin":false},"created_at":"2016-09-01T18:49:49Z","updated_at":"2016-09-01T18:49:49Z","body":"Cardboard already has serious partitioning issues with our aggregation of an entire dataset in a single HASH key. \r\n\r\n_Average usage nowhere near provisioned capacity_\r\n<img width=\"662\" alt=\"screen shot 2016-09-01 at 1 38 19 pm\" src=\"https://cloud.githubusercontent.com/assets/515424/18179713/67058b70-7049-11e6-89e6-fe098d29ed1c.png\">\r\n\r\n_But we're still getting throttled requests_\r\n<img width=\"724\" alt=\"screen shot 2016-09-01 at 1 38 05 pm\" src=\"https://cloud.githubusercontent.com/assets/515424/18179736/788a2e46-7049-11e6-9d41-8ecd71617537.png\">\r\n\r\nAdding more records per feature with the same HASH key (datasetid) is going to exacerbate this problem.\r\n\r\nAlso, keep in mind that a BBOX query via...\r\n\r\n```\r\n#dataset = :dataset and begins_with(#id, 'bbox!quadkey!021')\r\n```\r\n\r\n... only gets you features that are indexed at `021` or smaller. To find features that are indexed by a larger cell, you either need to follow that up with more searches:\r\n\r\n```\r\n#dataset = :dataset and begins_with(#id, 'bbox!quadkey!02!')\r\n#dataset = :dataset and begins_with(#id, 'bbox!quadkey!0!')\r\n```\r\n\r\n... or you need to index large features at all the sub-cells down to some max zoom. The former approach (more queries) leads to some convoluted pagination, and the latter can lead to exponential increase is dynamodb write needs.\r\n\r\nI'm very gungho for trying out the DynamoDB --> ElasticSearch pipeline. At the end of the day _DynamoDB loses usefulness the more important aggregation is to your application_. And it gets worse when you want to be able to locate the extremes in any aggregation. There have been a number of discussions in cardboard and other systems where I've felt that DynamoDB can make a lot of sense as scalable write storage, but the details of application querying demand some other store for aggregation and indexing on more axes that DynamoDB can realistically ($$$) support."}}
{"actor":{"display_login":"rclark"}}