{"push_id":1114036303,"size":17,"distinct_size":17,"ref":"refs/heads/dturner/pclouds-watchman-noshm","head":"74f098c413ebb0a490f4bc3fe825b9136e9260c3","before":"c9af71a6c3a4ee62875da68786f2e6be253413dd","commits":[{"sha":"dd8ebba0c31a088f3d37db0ee3173e13822a3102","author":{"email":"a13b1135e6f2f8530e85bc2896cd5ed89a6ed863@twopensource.com","name":"David Turner"},"message":"pkt-line: add gentle version of packet_write\n\npacket_write calls write_or_die, which dies with a sigpipe even if\ncalling code has explicitly blocked that signal.\n\nAdd packet_write_gently and packet_flush_gently, which don't.  Soon,\nwe will use this for communication with git index-helper, which, being\nmerely an optimization, should be permitted to die without disrupting\nclients.\n\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/dd8ebba0c31a088f3d37db0ee3173e13822a3102"},{"sha":"5f144e75805e344e981433b99c43cc19e3b6c638","author":{"email":"024f400cd820cb2ddbe54c7fecfd01e45f30bb96@gmail.com","name":"Nguyễn Thái Ngọc Duy"},"message":"index-helper: new daemon for caching index and related stuff\n\nInstead of reading the index from disk and worrying about disk\ncorruption, the index is cached in memory (memory bit-flips happen\ntoo, but hopefully less often). The result is faster read. Read time\nis reduced by 70%.\n\nThe biggest gain is not having to verify the trailing SHA-1, which\ntakes lots of time especially on large index files. But this also\nopens doors for further optimizations:\n\n - we could create an in-memory format that's essentially the memory\n   dump of the index to eliminate most of parsing/allocation\n   overhead. The mmap'd memory can be used straight away. Experiment\n   [1] shows we could reduce read time by 88%.\n\n - we could cache non-index info such as name hash\n\nShared memory is done by storing files in a per-repository temporary\ndirectory.  This is more portable than shm (which requires\nposix-realtime and has various quirks on OS X).  It might even work on\nWindows, although this has not been tested. The shared memory file's\nname follows the template \"shm-<object>-<SHA1>\" where <SHA1> is the\ntrailing SHA-1 of the index file. <object> is \"index\" for cached index\nfiles (and might later be \"name-hash\" for name-hash cache). If such\nshared memory exists, it contains the same index content as on\ndisk. The content is already validated by the daemon and git won't\nvalidate it again (except comparing the trailing SHA-1s).\n\nWe also add some bits to the index (to_shm and from_shm) to track\nwhen an index came from shared memory or is going to shared memory.\n\nWe keep this daemon's logic as thin as possible. The \"brain\" stays in\ngit. So the daemon can read and validate stuff, but that's all it's\nallowed to do. It does not add/create new information. It doesn't even\naccept direct updates from git.\n\nGit can poke the daemon via unix domain sockets to tell it to refresh\nthe index cache, or to keep it alive some more minutes. It can't give\nany real index data directly to the daemon. Real data goes to disk\nfirst, then the daemon reads and verifies it from there. The daemon\nonly handles $GIT_DIR/index, not temporary index files; it only gets\npoked for the former.\n\n$GIT_DIR/index-helper.sock is the socket for the daemon process. The\ndaemon reads from the socket and executes commands.\n\nNamed pipes were considered for portability reasons, but then commands\nthat need replies from the daemon would have to open their own pipes,\nsince a named pipe should only have one reader.  Unix domain sockets\ndon't have this problem.\n\nOn webkit.git with index format v2, duplicating 8 times to 1.5m\nentries and 236MB in size:\n\n(vanilla)      0.50 s: read_index_from .git/index\n(index-helper) 0.18 s: read_index_from .git/index\n\nInterestingly with index v4, we get less out of index-helper. It makes\nsense as v4 requires more processing after loading the index:\n\n(vanilla)      0.37 s: read_index_from .git/index\n(index-helper) 0.22 s: read_index_from .git/index\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/247268/focus=248771\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\nSigned-off-by: David Turner <dturner@twopensource.com>\nSigned-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/5f144e75805e344e981433b99c43cc19e3b6c638"},{"sha":"cd716b47465f7a056167c34d07c70251e329397e","author":{"email":"024f400cd820cb2ddbe54c7fecfd01e45f30bb96@gmail.com","name":"Nguyễn Thái Ngọc Duy"},"message":"index-helper: add --strict\n\nThere are \"holes\" in the index-helper approach because the shared\nmemory is not verified again by git. If $USER is compromised, shared\nmemory could be modified. But anyone who could do this could already\nmodify $GIT_DIR/index. A more realistic risk is some bugs in\nindex-helper that produce corrupt shared memory. --strict is added to\navoid that.\n\nStrictly speaking there's still a very small gap where corrupt shared\nmemory could still be read by git: after we write the trailing SHA-1 in\nthe shared memory (thus signaling \"this shm is ready\") and before\nverify_shm() detects an error.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/cd716b47465f7a056167c34d07c70251e329397e"},{"sha":"94e8be8969449d74b96119f8bc756e8d43317b87","author":{"email":"024f400cd820cb2ddbe54c7fecfd01e45f30bb96@gmail.com","name":"Nguyễn Thái Ngọc Duy"},"message":"daemonize(): set a flag before exiting the main process\n\nThis allows signal handlers and atexit functions to realize this\nsituation and not clean up.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/94e8be8969449d74b96119f8bc756e8d43317b87"},{"sha":"7208c7c9462875facdd130a1e8950f8ba70f0fd9","author":{"email":"024f400cd820cb2ddbe54c7fecfd01e45f30bb96@gmail.com","name":"Nguyễn Thái Ngọc Duy"},"message":"index-helper: add --detach\n\nWe detach after creating and opening the socket, because otherwise\nwe might return control to the shell before index-helper is ready to\naccept commands.  This might lead to flaky tests.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/7208c7c9462875facdd130a1e8950f8ba70f0fd9"},{"sha":"bcd6a30363a81863cd8f842105613fa3da6ef7f2","author":{"email":"a13b1135e6f2f8530e85bc2896cd5ed89a6ed863@twopensource.com","name":"David Turner"},"message":"index-helper: log warnings\n\nInstead of writing warnings to stderr, write them to a log.  Later, we'll\nprobably be daemonized, so writing to stderr will be pointless.\n\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/bcd6a30363a81863cd8f842105613fa3da6ef7f2"},{"sha":"efbc0c3074f96fac9368d7fefe76e2de675d0744","author":{"email":"024f400cd820cb2ddbe54c7fecfd01e45f30bb96@gmail.com","name":"Nguyễn Thái Ngọc Duy"},"message":"read-cache: add watchman 'WAMA' extension\n\nThe extension contains a bitmap, one bit for each entry in the\nindex. If the n-th bit is zero, the n-th entry is considered\nunchanged, we can ce_mark_uptodate() it without refreshing. If the bit\nis non-zero and we found out the corresponding file is clean after\nrefresh, we can clear the bit.\n\nIn addition, there's a list of directories in the untracked-cache\nto invalidate (because they have new or modified entries).\n\nThe 'skipping refresh' bit is not in this patch yet as we would need\nwatchman. More details in later patches.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/efbc0c3074f96fac9368d7fefe76e2de675d0744"},{"sha":"519a644d7a057a7ed12d3c16160860dfef29fd0b","author":{"email":"024f400cd820cb2ddbe54c7fecfd01e45f30bb96@gmail.com","name":"Nguyễn Thái Ngọc Duy"},"message":"watchman: support watchman to reduce index refresh cost\n\nThe previous patch has the logic to clear bits in 'WAMA' bitmap. This\npatch has logic to set bits as told by watchman. The missing bit,\n_using_ these bits, are not here yet.\n\nA lot of this code is written by David Turner originally, mostly from\n[1]. I'm just copying and polishing it a bit.\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/248006\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/519a644d7a057a7ed12d3c16160860dfef29fd0b"},{"sha":"54a1a88e2049b17ac141b32dce40d7034416381c","author":{"email":"024f400cd820cb2ddbe54c7fecfd01e45f30bb96@gmail.com","name":"Nguyễn Thái Ngọc Duy"},"message":"index-helper: use watchman to avoid refreshing index with lstat()\n\nWatchman is hidden behind index-helper. Before git tries to read the\nindex from shm, it notifies index-helper through the socket and waits\nfor index-helper to prepare a file for sharing memory (with\nMAP_SHARED). index-helper then contacts watchman, updates 'WAMA'\nextension and put it in a separate file and wakes git up with a reply\nto git's socket.\n\nGit uses this extension to not lstat unchanged entries. Git only\ntrusts the 'WAMA' extension when it's received from the separate file,\nnot from disk. Unmarked entries are \"clean\". Marked entries are dirty\nfrom watchman point of view. If it finds out some entries are\n'watchman-dirty', but are really unchanged (e.g. the file was changed,\nthen reverted back), then Git will clear the marking in 'WAMA' before\nwriting it down.\n\nHiding watchman behind index-helper means you need both daemons. You\ncan't run watchman alone. Not so good. But on the other hand, 'git'\nbinary is not linked to watchman/json libraries, which is good for\npackaging. Core git package will run fine without watchman-related\npackages. If they need watchman, they can install git-index-helper and\ndependencies.\n\nThis also lets us trust anything in the untracked cache that we haven't\nmarked invalid, saving those stat() calls.\n\nAnother reason for tying watchman to index-helper is, when used with\nuntracked cache, we need to keep track of $GIT_WORK_TREE file\nlisting. That kind of list can be kept in index-helper.\n\nHelped-by: Ramsay Jones <ramsay@ramsayjones.plus.com>\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/54a1a88e2049b17ac141b32dce40d7034416381c"},{"sha":"582454e470702255c2896e7fe56f3c72f7ecfa1f","author":{"email":"024f400cd820cb2ddbe54c7fecfd01e45f30bb96@gmail.com","name":"Nguyễn Thái Ngọc Duy"},"message":"update-index: enable/disable watchman support\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/582454e470702255c2896e7fe56f3c72f7ecfa1f"},{"sha":"6a194b8ac709c5601258b38650311ba172d8fcff","author":{"email":"a13b1135e6f2f8530e85bc2896cd5ed89a6ed863@twopensource.com","name":"David Turner"},"message":"unpack-trees: preserve index extensions\n\nMake git checkout (and other unpack_tree operations) preserve the\nuntracked cache and watchman status. This is valuable for two reasons:\n\n1. Often, an unpack_tree operation will not touch large parts of the\nworking tree, and thus most of the untracked cache will continue to be\nvalid.\n\n2. Even if the untracked cache were entirely invalidated by such an\noperation, the user has signaled their intention to have such a cache,\nand we don't want to throw it away.\n\nThe same logic applies to the watchman state.\n\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/6a194b8ac709c5601258b38650311ba172d8fcff"},{"sha":"2e843a2bc623d8a71965a69ee8e086fd3cdd9cd4","author":{"email":"a13b1135e6f2f8530e85bc2896cd5ed89a6ed863@twopensource.com","name":"David Turner"},"message":"watchman: add a config option to enable the extension\n\nFor installations that have centrally-managed configuration, it's\neasier to set a config once than to run update-index on every\nrepository.\n\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/2e843a2bc623d8a71965a69ee8e086fd3cdd9cd4"},{"sha":"5e441f768f35657d409c26814b0b5a720ca36336","author":{"email":"a13b1135e6f2f8530e85bc2896cd5ed89a6ed863@twopensource.com","name":"David Turner"},"message":"index-helper: kill mode\n\nAdd a new command (and command-line arg) to allow index-helpers to\nexit cleanly.\n\nThis is mainly useful for tests.\n\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/5e441f768f35657d409c26814b0b5a720ca36336"},{"sha":"7ee7382213df1ca239db71d9184b05e1c298d02a","author":{"email":"a13b1135e6f2f8530e85bc2896cd5ed89a6ed863@twopensource.com","name":"David Turner"},"message":"index-helper: don't run if already running\n\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/7ee7382213df1ca239db71d9184b05e1c298d02a"},{"sha":"12beaf6dc5f9a8f152213ca63e5a370d646dfe1a","author":{"email":"a13b1135e6f2f8530e85bc2896cd5ed89a6ed863@twopensource.com","name":"David Turner"},"message":"index-helper: autorun mode\n\nSoon, we'll want to automatically start index-helper, so we need\na mode that silently exits if it can't start up (either because\nit's not in a git dir, or because another one is already running).\n\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/12beaf6dc5f9a8f152213ca63e5a370d646dfe1a"},{"sha":"af5a7215ea53c04cb45a1e90c9bdf0c31537973c","author":{"email":"a13b1135e6f2f8530e85bc2896cd5ed89a6ed863@twopensource.com","name":"David Turner"},"message":"index-helper: optionally automatically run\n\nIntroduce a new config option, indexhelper.autorun, to automatically\nrun git index-helper before starting up a builtin git command.  This\nenables users to keep index-helper running without manual\nintervention.\n\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/af5a7215ea53c04cb45a1e90c9bdf0c31537973c"},{"sha":"74f098c413ebb0a490f4bc3fe825b9136e9260c3","author":{"email":"024f400cd820cb2ddbe54c7fecfd01e45f30bb96@gmail.com","name":"Nguyễn Thái Ngọc Duy"},"message":"trace: measure where the time is spent in the index-heavy operations\n\nAll the known heavy code blocks are measured (except object database\naccess). This should help identify if an optimization is effective or\nnot. An unoptimized git-status would give something like below (92% of\ntime is accounted). To sum up the effort of making git scale better:\n\n - read cache line is being addressed by index-helper\n - preload/refresh index by watchman\n - read packed refs by lmdb backend\n - diff-index by rebuilding cache-tree\n - read directory by untracked cache and watchman\n - write index by split index\n - name hash potientially by index-helper\n\nread-cache.c:2075         performance: 0.004058570 s: read cache .../index\npreload-index.c:104       performance: 0.004419864 s: preload index\nread-cache.c:1265         performance: 0.000185224 s: refresh index\nrefs/files-backend.c:1100 performance: 0.001935788 s: read packed refs\ndiff-lib.c:240            performance: 0.000144132 s: diff-files\ndiff-lib.c:506            performance: 0.013592000 s: diff-index\nname-hash.c:128           performance: 0.000614177 s: initialize name hash\ndir.c:2030                performance: 0.015814103 s: read directory\nread-cache.c:2565         performance: 0.004052343 s: write index, changed mask = 2\ntrace.c:420               performance: 0.048365509 s: git command: './git' 'status'\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\nSigned-off-by: David Turner <dturner@twopensource.com>","distinct":true,"url":"https://api.github.com/repos/dturner-tw/git/commits/74f098c413ebb0a490f4bc3fe825b9136e9260c3"}]}