{"action":"created","issue":{"url":"https://api.github.com/repos/Project-OSRM/osrm-backend/issues/1947","repository_url":"https://api.github.com/repos/Project-OSRM/osrm-backend","labels_url":"https://api.github.com/repos/Project-OSRM/osrm-backend/issues/1947/labels{/name}","comments_url":"https://api.github.com/repos/Project-OSRM/osrm-backend/issues/1947/comments","events_url":"https://api.github.com/repos/Project-OSRM/osrm-backend/issues/1947/events","html_url":"https://github.com/Project-OSRM/osrm-backend/issues/1947","id":130724158,"number":1947,"title":"Implement mmapDataFacade","user":{"login":"danpat","id":1892250,"avatar_url":"https://avatars.githubusercontent.com/u/1892250?v=3","gravatar_id":"","url":"https://api.github.com/users/danpat","html_url":"https://github.com/danpat","followers_url":"https://api.github.com/users/danpat/followers","following_url":"https://api.github.com/users/danpat/following{/other_user}","gists_url":"https://api.github.com/users/danpat/gists{/gist_id}","starred_url":"https://api.github.com/users/danpat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danpat/subscriptions","organizations_url":"https://api.github.com/users/danpat/orgs","repos_url":"https://api.github.com/users/danpat/repos","events_url":"https://api.github.com/users/danpat/events{/privacy}","received_events_url":"https://api.github.com/users/danpat/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Project-OSRM/osrm-backend/labels/Feature%20Request","name":"Feature Request","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-02-02T15:51:37Z","updated_at":"2016-05-29T15:54:40Z","closed_at":null,"body":"OSRM currently supports reading data from files into heap memory (`InternalDataFacade`), or pre-loading data into shared memory using IPC shared memory blocks (`SharedDataFacade`+`osrm-datastore`).\r\n\r\nWe can consolidate the behaviour of both of these by using `mmap`.  Instead of reading files into memory explicitly, we should be able to `mmap` the data files, and immediately begin using them.\r\n\r\nThere are a few changes that need to be made to get us there:\r\n\r\n   - [x] Benchmark `mmap`d data access vs heap - what, if any, penalty is there? How does this change when the file we `mmap` is on a ramdisk?\r\n   - [ ] Identify data structures that can't be `mmap`ed and fix them - basically anything in `osrm-datastore` (`src/storage/storage.cpp`) that isn't just loaded into memory in one big blob. Problem here is `vector<bool>` and its proxy behavior; we need a contiguous container we can `memcpy` to.\r\n   - [ ] Clone the `SharedDataFacade` and perform similar `.swap` operations against `mmap`ed memory addresses rather than `shm` addresses.\r\n   - [ ] Figure out IPC signalling for swapping out `mmap`ed files on-the-fly\r\n   - [x] Investigate using `mmap` instead of explicit `read` disk files for leaf nodes in the StaticRTree to boost performance (coordinate lookups represent the largest part of any given routing query because of the I/O in the rtree).\r\n   - [ ] Make sure this works on Windows too.\r\n\r\nThe main goal here is to minimize double-reads of data.  In situations where we are constantly cycling out data sets (in the case of traffic updates), we want to minimize I/O and the number of times any single bit of data gets touched.  By using `mmap` and `tmpfs`, we can emulate the current share-memory behavior, but avoid an extra pass over the data.\r\n\r\nFor normal `osrm-routed` use, we would essentially get lazy-loading of data - `osrm-routed` would start up faster, but queries would be slower since pages are loaded from disk on demand until data is touched and lives in the filesystem cache. This initial slowness could be avoided by pre-seeding the data files into the filesystem cache or via `MAP_POPULATE` (Linux 2.5.46+), and this could be done in parallel to `osrm-routed` already starting up and answering queries.\r\n\r\n/cc @daniel-j-h @TheMarex "},"comment":{"url":"https://api.github.com/repos/Project-OSRM/osrm-backend/issues/comments/222367824","html_url":"https://github.com/Project-OSRM/osrm-backend/issues/1947#issuecomment-222367824","issue_url":"https://api.github.com/repos/Project-OSRM/osrm-backend/issues/1947","id":222367824,"user":{"login":"daniel-j-h","id":527241,"avatar_url":"https://avatars.githubusercontent.com/u/527241?v=3","gravatar_id":"","url":"https://api.github.com/users/daniel-j-h","html_url":"https://github.com/daniel-j-h","followers_url":"https://api.github.com/users/daniel-j-h/followers","following_url":"https://api.github.com/users/daniel-j-h/following{/other_user}","gists_url":"https://api.github.com/users/daniel-j-h/gists{/gist_id}","starred_url":"https://api.github.com/users/daniel-j-h/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/daniel-j-h/subscriptions","organizations_url":"https://api.github.com/users/daniel-j-h/orgs","repos_url":"https://api.github.com/users/daniel-j-h/repos","events_url":"https://api.github.com/users/daniel-j-h/events{/privacy}","received_events_url":"https://api.github.com/users/daniel-j-h/received_events","type":"User","site_admin":false},"created_at":"2016-05-29T15:54:40Z","updated_at":"2016-05-29T15:54:40Z","body":"https://github.com/mapbox/tmpfs-mmap-zero-copy"}}
{"actor":{"display_login":"daniel-j-h"}}