{"action":"created","issue":{"url":"https://api.github.com/repos/apache/couchdb-couch/issues/185","repository_url":"https://api.github.com/repos/apache/couchdb-couch","labels_url":"https://api.github.com/repos/apache/couchdb-couch/issues/185/labels{/name}","comments_url":"https://api.github.com/repos/apache/couchdb-couch/issues/185/comments","events_url":"https://api.github.com/repos/apache/couchdb-couch/issues/185/events","html_url":"https://github.com/apache/couchdb-couch/pull/185","id":165897853,"number":185,"title":"3061 adaptive header search","user":{"login":"jaydoane","id":51209,"avatar_url":"https://avatars.githubusercontent.com/u/51209?v=3","gravatar_id":"","url":"https://api.github.com/users/jaydoane","html_url":"https://github.com/jaydoane","followers_url":"https://api.github.com/users/jaydoane/followers","following_url":"https://api.github.com/users/jaydoane/following{/other_user}","gists_url":"https://api.github.com/users/jaydoane/gists{/gist_id}","starred_url":"https://api.github.com/users/jaydoane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaydoane/subscriptions","organizations_url":"https://api.github.com/users/jaydoane/orgs","repos_url":"https://api.github.com/users/jaydoane/repos","events_url":"https://api.github.com/users/jaydoane/events{/privacy}","received_events_url":"https://api.github.com/users/jaydoane/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2016-07-16T00:06:03Z","updated_at":"2016-09-22T15:10:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/couchdb-couch/pulls/185","html_url":"https://github.com/apache/couchdb-couch/pull/185","diff_url":"https://github.com/apache/couchdb-couch/pull/185.diff","patch_url":"https://github.com/apache/couchdb-couch/pull/185.patch"},"body":"The current find_header algorithm performs one file read per block,\r\nwhich can be inefficient when the most recent header is buried deeply in\r\na large file.\r\n\r\nWith default config params, this change essentially keeps current\r\nbehavior, reading each block into memory one at a time until a header is\r\nfound, or beginning of file is reached. However, by setting new config\r\nparams, it's possible to exponentially increase the number of blocks\r\nread into a \"chunk\" of memory with a single read, up to a configurable\r\nlimit.\r\n\r\nFor example, the following settings begin with a chunk size of one\r\nblock, and then double in size each additional search step backward to a\r\nmaximum limit of 16GB:\r\n\r\n[couchdb]\r\nchunk_max_size = 4096*4096\r\nchunk_exponent_base = 2\r\n\r\nMeasurements for a 12GB .couch.meta file with its last header at 4GB\r\nshows a speed improvement of from 17x (server) to 27x (laptop).\r\n\r\nCOUCHDB-3061"},"comment":{"url":"https://api.github.com/repos/apache/couchdb-couch/issues/comments/248932406","html_url":"https://github.com/apache/couchdb-couch/pull/185#issuecomment-248932406","issue_url":"https://api.github.com/repos/apache/couchdb-couch/issues/185","id":248932406,"user":{"login":"theburge","id":1048220,"avatar_url":"https://avatars.githubusercontent.com/u/1048220?v=3","gravatar_id":"","url":"https://api.github.com/users/theburge","html_url":"https://github.com/theburge","followers_url":"https://api.github.com/users/theburge/followers","following_url":"https://api.github.com/users/theburge/following{/other_user}","gists_url":"https://api.github.com/users/theburge/gists{/gist_id}","starred_url":"https://api.github.com/users/theburge/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/theburge/subscriptions","organizations_url":"https://api.github.com/users/theburge/orgs","repos_url":"https://api.github.com/users/theburge/repos","events_url":"https://api.github.com/users/theburge/events{/privacy}","received_events_url":"https://api.github.com/users/theburge/received_events","type":"User","site_admin":false},"created_at":"2016-09-22T15:10:18Z","updated_at":"2016-09-22T15:10:18Z","body":"Thanks for doing the additional investigation @jaydoane. I'm glad the performance remains consistently improved.\r\n\r\nFrom memory, I've observed deeply buried headers frequently in MT, although admittedly not lots at once (usually a DB or two). However, they're not only on .meta files if memory serves, but also on view files (and I'm fairly sure on primary database files, although IIRC usually in correlation with some other odd behaviour -- so perhaps it shouldn't be considered _normal_ but it's certainly possible).\r\n\r\nIn any case, it's reassuring that the vectored approach is only tried if the basic approach fails, given the striking difference in page-in rate."}}
{"actor":{"display_login":"theburge"}}