{"action":"created","issue":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/5318","labels_url":"https://api.github.com/repos/influxdata/influxdb/issues/5318/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/influxdb/issues/5318/comments","events_url":"https://api.github.com/repos/influxdata/influxdb/issues/5318/events","html_url":"https://github.com/influxdata/influxdb/pull/5318","id":125638254,"number":5318,"title":"Remove MAP_POPULATE","user":{"login":"jwilder","id":219935,"avatar_url":"https://avatars.githubusercontent.com/u/219935?v=3","gravatar_id":"","url":"https://api.github.com/users/jwilder","html_url":"https://github.com/jwilder","followers_url":"https://api.github.com/users/jwilder/followers","following_url":"https://api.github.com/users/jwilder/following{/other_user}","gists_url":"https://api.github.com/users/jwilder/gists{/gist_id}","starred_url":"https://api.github.com/users/jwilder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwilder/subscriptions","organizations_url":"https://api.github.com/users/jwilder/orgs","repos_url":"https://api.github.com/users/jwilder/repos","events_url":"https://api.github.com/users/jwilder/events{/privacy}","received_events_url":"https://api.github.com/users/jwilder/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-01-08T16:01:12Z","updated_at":"2016-01-08T16:17:07Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/influxdata/influxdb/pulls/5318","html_url":"https://github.com/influxdata/influxdb/pull/5318","diff_url":"https://github.com/influxdata/influxdb/pull/5318.diff","patch_url":"https://github.com/influxdata/influxdb/pull/5318.patch"},"body":"This may be causing slow restart times for systems with many large TSM files.\r\nWhat I believe is happening at startup in these cases is that multiple goroutines\r\nare started to load each TSM file concurrently.  The kernel appears to serialize\r\nmmap calls from the same process so all of the goroutines end up getting blocked\r\non the actual mmap system call.  MAP_POPULATE instruct the kernel to pre-fault the\r\npage table for the files and triggers read-ahead of the pages.  For larger, 2GB files,\r\nthis makes the mmap call more expensive and slower.  When there are many of these files\r\nand calls it is possible to fill all available memory with pagecache.  In this case,\r\nthe OS will end up pre-faulting pages from one file and have to remove pages that it just\r\nloaded from another files causing slowness.  MAP_POPULATE may also be cause much more data\r\nto be pre-faulted than necessary.  To load a file, we just need to scan the index at the end\r\nof the file.  MAP_POPULATE is likely causing the whole file to be loaded when it won't actually\r\nbe accessed for a while (or at all).\r\n\r\nMight fix issue #5311."},"comment":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/comments/170044551","html_url":"https://github.com/influxdata/influxdb/pull/5318#issuecomment-170044551","issue_url":"https://api.github.com/repos/influxdata/influxdb/issues/5318","id":170044551,"user":{"login":"pauldix","id":4331,"avatar_url":"https://avatars.githubusercontent.com/u/4331?v=3","gravatar_id":"","url":"https://api.github.com/users/pauldix","html_url":"https://github.com/pauldix","followers_url":"https://api.github.com/users/pauldix/followers","following_url":"https://api.github.com/users/pauldix/following{/other_user}","gists_url":"https://api.github.com/users/pauldix/gists{/gist_id}","starred_url":"https://api.github.com/users/pauldix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pauldix/subscriptions","organizations_url":"https://api.github.com/users/pauldix/orgs","repos_url":"https://api.github.com/users/pauldix/repos","events_url":"https://api.github.com/users/pauldix/events{/privacy}","received_events_url":"https://api.github.com/users/pauldix/received_events","type":"User","site_admin":false},"created_at":"2016-01-08T16:17:07Z","updated_at":"2016-01-08T16:17:07Z","body":"@jwilder sounds good, +1"}}