{"push_id":987741382,"size":15,"distinct_size":15,"ref":"refs/heads/master","head":"a56cbb4409dbc3faecf4fbb2edea81d605d757de","before":"1a28d81c2bde64769e6017cd7d158dc71dc6cbb4","commits":[{"sha":"a42502e205597b00d310631ae5b3c824f22d848f","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Implement debugging infrastructure\n\nThis is meant to be very simple as to avoid doing any real work in the\ninitialization handler. Debugging messages were proposed and inserted by\nMoo Kim <Moo.Kim@Teradata.com>; I have taken the liberty to expand\nthe #ifdef-based approach that opened a file to this version. Rationale\nfrom the discussion with Moo:\n\n> Second, about the debugging. It’s really fragile to do file open\n> operations in a library designed for intercepting file open\n> operations, especially if it’s in the __init__ handler. Instead I have\n> the following idea: In the nocache wrapper, we add a log file argument\n> e.g. -D for debugging, which takes a filename. Before starting the\n> binary, we open the file for writing via\n>\n>     exec 3>${FILENAME}\n>\n> And we export the env variable NOCACHE_DEBUG_FD=3. The executed\n> process will inherit this already opened FD and will just use\n> something like\n>\n>     debugfd = atoi(getenv(\"NOCACHE_DEBUG_FD\"));\n>     debugfp = fdopen(debugfd);\n>     ...\n>     fprintf(debugfp, \"DEBUG: ....\\n\");\n>\n> We simultaneously get rid of the problem of permissions (maybe the\n> process raises/drops its privileges?), race conditions (file is\n> already open and need not be closed at any time) and the ugliness of\n> doing something with actual files that could fail somehow. It’s\n> trivial to implement and clean in my eyes. What do you think about it?\n>\n> In Bash/Zsh, we could actually do better by doing:\n>\n>     exec {debugfd}>${FILENAME}\n>     export NOCACHE_DEBUG_FD=$debugfd\n>\n> (Because fd 3 might be in use, the exec above can fail; this one picks\n> a new fd greater than 10 which is not in use yet.)","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/a42502e205597b00d310631ae5b3c824f22d848f"},{"sha":"c7408da8c91b6d1b1eb6ffd5234d465866bd6875","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Make the debugging easier to handle for end users\n\nThis REQUIRES a Bash (or Zsh) now; POSIX does not guarantee the\navailability of the {var}>/file syntax!","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/c7408da8c91b6d1b1eb6ffd5234d465866bd6875"},{"sha":"b631050ebd4e57a8f780ba0fbf949a9ce2eb6191","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Merge branch 'debug-infrastructure'\n\n* debug-infrastructure:\n  Make the debugging easier to handle for end users\n  Implement debugging infrastructure","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/b631050ebd4e57a8f780ba0fbf949a9ce2eb6191"},{"sha":"f8b75e8ec6297157a2555b7a0dfd91a850fc6927","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Debug statements for the fadv_dontneed() calls","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/f8b75e8ec6297157a2555b7a0dfd91a850fc6927"},{"sha":"566f37e433665b52b88f9acf10bd431fb97d4c6f","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Better debug statement for open(), spit out fd #","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/566f37e433665b52b88f9acf10bd431fb97d4c6f"},{"sha":"f7f4e890b6529f554f8227cc60ce66a35edda52e","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"README: Give hint about /dev/stderr","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/f7f4e890b6529f554f8227cc60ce66a35edda52e"},{"sha":"605fb4790985f43d99a059d30f1086c049d7eaf1","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Shell wrapper: Only write to debugfd if it was opened","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/605fb4790985f43d99a059d30f1086c049d7eaf1"},{"sha":"dec6b6eb9c8f20f133172e09908987362b1c7df1","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"No need for this code","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/dec6b6eb9c8f20f133172e09908987362b1c7df1"},{"sha":"8d26d64533157a41a5913c2fe6876443c4c64e3a","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Free fd info array in destructor\n\nThe destructor should only be called on program exit so technically the\nOS should take care of this. But for cleanliness we’ll do it anyway!","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/8d26d64533157a41a5913c2fe6876443c4c64e3a"},{"sha":"87d5b0193f8df6d36eb8b132dd7dce6ce4cba578","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Compute not-cached intervals on file open\n\nThis is probably a huge memory saver, and pushes the computation\ncomplexity in the open() part, not the close() part.\n\nInstead of storing the whole info vector from mincore, we store it\ntemporarily, compute which parts of the file are not cached, and then\nfree the vector again, retaining only a singly linked list of byte\nranges we’ll want to fadvise() on when closing the file.\n\nFor example, consider an MP3 file which I played at 0:00, skipped to\n10:00, then skipped to 20:00. It has three small cached regions (plus\nthe last page is cached), hence we store three “uncached intervals”.\nOn close(), these intervals will be marked “don’t need”. Additionally,\nif the file size changes, everything newly added will be marked “don’t\nneed”, as happens with the new file (fd = 4).\n\n    $ ./nocache -D /dev/stderr cp ~/musik/fernanda.mp3 /tmp/x\n    [nocache] DEBUG: Executing: cp /home/feh/musik/fernanda.mp3 /tmp/x\n    [nocache] DEBUG: open(pathname=/home/feh/musik/fernanda.mp3, flags=0x0, mode=00) = 3\n    [nocache] DEBUG: fd_get_pageinfo(fd=3): st.st_size=80695431, nr_pages=19702\n    [nocache] DEBUG: store_pageinfo(fd=3): pages in cache: 183/19702 (0.9%)  [filesize=78804.1K, pagesize=4K]\n    [nocache] DEBUG: open(pathname=/tmp/x, flags=0x201, mode=00) = 4\n    [nocache] DEBUG: fd_get_pageinfo(fd=4): st.st_size=0, nr_pages=0\n    [nocache] DEBUG: store_pageinfo(fd=4): pages in cache: 0/0 (0.0%)  [filesize=0.0K, pagesize=4K]\n    [nocache] DEBUG: fadv_dontneed(fd=4, from=0, len=0 [till new end, file has grown])\n    [nocache] DEBUG: close(4)\n    [nocache] DEBUG: fadv_dontneed(fd=3, from=245760, len=14266368)\n    [nocache] DEBUG: fadv_dontneed(fd=3, from=14761984, len=14258176)\n    [nocache] DEBUG: fadv_dontneed(fd=3, from=29270016, len=51425280)\n    [nocache] DEBUG: close(3)\n\nIt might be beneficial to “chunk” the mincore() call, since one call\nneeds filesize/PAGESIZE bytes of memory, so we can fit into 16MB of heap\nmemory only the map for a file of size 16*1024^2*4096/1024^3 = 64 GB.\nOn terabyte-sized file, we would allocate a tremendous amount of memory,\nand by repeatedly calling mincore() on 64GB-chunks of the file and\nreusing the allocated heap space, we could probably improve performance\nand waste less space. This is for a future commit.","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/87d5b0193f8df6d36eb8b132dd7dce6ce4cba578"},{"sha":"91a5da8c8c73b667862b2df3040ae3c644b7a8e7","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Free page_vec in case of error","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/91a5da8c8c73b667862b2df3040ae3c644b7a8e7"},{"sha":"40eba043cd3b894cc31f597aab43758ea0c234ea","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Makefile: Let *.o depend on corresponding *.c file","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/40eba043cd3b894cc31f597aab43758ea0c234ea"},{"sha":"dfdfd97dcc36bd4cd5db472d45115210baba6667","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"fix edgecase","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/dfdfd97dcc36bd4cd5db472d45115210baba6667"},{"sha":"6364e81088ccb3759ebf847ca277a1530963e6d6","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Fix option parsing","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/6364e81088ccb3759ebf847ca277a1530963e6d6"},{"sha":"a56cbb4409dbc3faecf4fbb2edea81d605d757de","author":{"email":"e598c132c8fc497ee05802af14ba7ec60da13760@plenz.com","name":"Julius Plenz"},"message":"Makefile: Fix more dependencies","distinct":true,"url":"https://api.github.com/repos/hhoffstaette/nocache/commits/a56cbb4409dbc3faecf4fbb2edea81d605d757de"}]}