{"push_id":1228953487,"size":43,"distinct_size":43,"ref":"refs/heads/threading_pool","head":"5c591725174f70f4cd6857badf6e672ff3f6b469","before":"8fd420ec9f297fccc3da15568408b786167a07f2","commits":[{"sha":"519f75a987530aa1a66180a7a8dc883a54d583ef","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Prompoted cram/thread_pool code to the top-level and use it within\nbgzf writing instead of its own multi-threading.\n\nThis can be a very significant multi-threading speed improvement in\nsome situations (depending on what other bottlenecks exist).","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/519f75a987530aa1a66180a7a8dc883a54d583ef"},{"sha":"aaf998a97a8b26d7d800076aa496ef89daf62c66","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Tidied up the mass of #ifdefs.\n\nAll the #ifdef DEBUG + fprintf are now DBG_OUT which expands to fprintf\nor nothing.\n\nThe #ifdef IN_ORDER is now mandatory with the old code culled, as we\nfound it to generally be a win on machines with CPU frequency scaling\nturned on.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/aaf998a97a8b26d7d800076aa496ef89daf62c66"},{"sha":"f72add4a2c7dc75460a8558ebc41faeb99888fa2","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Further updates to multi-threading.\n\nThe BGZF writing code now runs in its own dedicated thread.  This\nsimply permits it to interleave writing time with job dispatching and\nother main loop bits.\n\nThe writing thread also does periodic hflush, which in turn calls\nfdatasync is operating on a local file.  This avoids a much larger\namount to sync at the end when we close the file.\n\nA consequence of the separate write thread is that the thread pool\noutput queue (aka \"results queue\") now has a size limit.  Otherwise\nfailure to write fast enough would build up more and more compressed\nblocks in memory.\n\nThe net effect of these changes are further speed increases.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/f72add4a2c7dc75460a8558ebc41faeb99888fa2"},{"sha":"2cb33f9934e77947f26a4d8aa544c3a04d88e556","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Fixed dependencies & .mk defs so Samtools links too (threading change).","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/2cb33f9934e77947f26a4d8aa544c3a04d88e556"},{"sha":"70f1faeca8a524eb95b6114f76c967f02de2ba36","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Fixed uninitialised memory (queue shutdown).","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/70f1faeca8a524eb95b6114f76c967f02de2ba36"},{"sha":"242e03ee01ed9fa348a228572aca64283bd5872c","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Tidied up pointless while/if duplication.\n(Left over from a more complex structure.)","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/242e03ee01ed9fa348a228572aca64283bd5872c"},{"sha":"6717817dcda674885b1e55d3ff241e9dad050b5e","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Draft of the multi-threaded decoder.\n\nStill to-do: fix the termination cases.  It works, but sometimes fails\nat the last block.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/6717817dcda674885b1e55d3ff241e9dad050b5e"},{"sha":"2ef8e3e32a1194d473da7fdb5feb159e9bdf7904","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Committing as it's a working MT decoder now.\n\nIt isn't complete and can't handle uncompressed files, but this is a\nhandy checkpoint to go back to incase I bork it.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/2ef8e3e32a1194d473da7fdb5feb159e9bdf7904"},{"sha":"41a714a48d1411f24607f428a571fde602ec693c","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Attempt to use shared pool.\n\nERROR: this isn't complete, but it's our best try so far so\ncheck-pointing for ease of backing out.  Subsequent reorganisations\nwill be messy. :)","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/41a714a48d1411f24607f428a571fde602ec693c"},{"sha":"121bab9d3ad1a498fcefd0b49e73b54e2bff60fc","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Major revamp of thread pool and associated tests.  In particular the\ninput queue is now per output queue rather than shared between all\noutput queues.  This helps resolve deadlocks and permits more advanced\ntask sharing.\n\nNote: the rest of htslib using this has still to be updated.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/121bab9d3ad1a498fcefd0b49e73b54e2bff60fc"},{"sha":"5142d24e7091db54cdd0da81f8ade79a292eac34","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Bumped the sample size for the tests, and made all tests share a\ncommon size.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/5142d24e7091db54cdd0da81f8ade79a292eac34"},{"sha":"8e9bd8010395971fe8f3a1beeb716130d8a9d4e0","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"White space updates (tabs -> spaces).\n\nAlso culled the deprecated flush method.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/8e9bd8010395971fe8f3a1beeb716130d8a9d4e0"},{"sha":"7d50ac69c80fdba6f2cf3edfe59a370b94a5715e","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Sorry, updates to the previous commits too; forgetting to add this file!","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/7d50ac69c80fdba6f2cf3edfe59a370b94a5715e"},{"sha":"0ba563648b064331c5f3097d54113a253ae7ad7b","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Updates to cope with the new thread pool API.\n\nNow passes minimal testing.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/0ba563648b064331c5f3097d54113a253ae7ad7b"},{"sha":"2af39f8fbff68e4847682bf4d1dcf0d64f03eeb1","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Removed some debugging and changed the default queue sizes to be smaller.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/2af39f8fbff68e4847682bf4d1dcf0d64f03eeb1"},{"sha":"bd32ec6cdbb854ee88f32fab21c8d7a609dce448","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Improvements to avoid boom & bust scenarios.\n\nWe now continually check how many processing jobs we have vs the\noutput queue size so we don't launch more jobs than fit in the output\nqueue, causing subsequent waiting in theads later on as we over-egged\nthe pudding.  This caused a yo-yo effect that overall harms\nperformance.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/bd32ec6cdbb854ee88f32fab21c8d7a609dce448"},{"sha":"8a8fac27453dcd892c0896becf2ecf337c1ddb7a","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Culled the duplicate is_compressed assignment.\n\nThis appeared in bf909d669d45cb137e7cab5b56db0fc60be48a1c, but perhaps\na failed merge or just forgetting to delete the earlier variant.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/8a8fac27453dcd892c0896becf2ecf337c1ddb7a"},{"sha":"2c2330b3cd42104dd3839a21adb16c48db6da4e2","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Various memory leaks fixed in bgzf multi-threading.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/2c2330b3cd42104dd3839a21adb16c48db6da4e2"},{"sha":"bedecccc94778800c3c6cdbaf817c6400f53eb3f","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"First apparent working bgzf_seek implementation.\n\nThis also adds new functions to thread_pool, most importantly the\nability to reset a queue.  There is now also a requirement to shutdown\nqueues before destroying the pool itself.  (You can destroy the pool,\nbut the queue shutdown will then wedge if it has jobs in flight as\nthey'll never be processed.)","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/bedecccc94778800c3c6cdbaf817c6400f53eb3f"},{"sha":"d6dacc7bf6ccc39a6d6e486705570a7ea5e9b1b6","author":{"name":"James Bonfield","email":"6eeed454fb58375f4e96af8c413b4be6126267a8@sanger.ac.uk"},"message":"Added HTS_OPT_CACHE_SIZE option to specify the bgzf cache size.  This\ncan speed up decoding considerably in certain circumstances.","distinct":true,"url":"https://api.github.com/repos/jkbonfield/htslib/commits/d6dacc7bf6ccc39a6d6e486705570a7ea5e9b1b6"}]}
{"actor":{"display_login":"jkbonfield"}}