{"push_id":939222655,"size":12,"distinct_size":6,"ref":"refs/heads/fix/2.14/rb8934","head":"2e068a5e7c59e5f8eb3a86804566b9af68f2b75d","before":"4f777e53781560e81da710fb37027606717fbf67","commits":[{"sha":"063afcc38568afa9c5a912e451e1d362198f7341","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"pool: Simplify synchronization during repository setup\n\nMotivation:\n\nSimplify the various synchronization logic of CacheRepositoryV5.\n\nModification:\n\nDo not synchronize setters for parameters that must be called\nbefore initializing the repository. This is often referred to\nas threadsafe once configured.\n\nResult:\n\nSynchronization on the repository is limited to state that\nactually changes during the operation of the repository.\n\nTarget: trunk\nRequire-notes: no\nRequire-book: no\nRequest: 2.14\nRequest: 2.13\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/8928\n(cherry picked from commit 0f18c8bdc8cc36b16cd8a42b6f38374357e971be)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/063afcc38568afa9c5a912e451e1d362198f7341"},{"sha":"0a48e12f67fa16b237b5a2d5c97eb7f1fd2659c7","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"pool: Avoid lock contention when opening files and setting sticky flags\n\nMotivation:\n\nTo avoid races between the change notifications during file opening/setting\nsticky flags and repository initialization, the repository acquires an\nexclusive lock around the notification logic. This is a contention point\nwhen opening several files concurrently, in particular when opening the\nfile requires reads from the meta DB.\n\nModification:\n\nReplace the lock with a read write lock to allow open/set sticky operations to\nbe executed concurrently.\n\nThe patch is deliberately minimalistic to allow backporting to 2.13 and 2.14.\nBigger changes may be made on master in separate patches.\n\nResult:\n\nResolved lock contention when opening files or changing sticky flags on pools.\n\nTarget: trunk\nRequest: 2.14\nRequest: 2.13\nRequire-notes: yes\nRequire-book: no\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nPatch: https://rb.dcache.org/r/8924/\n(cherry picked from commit 0d9b3b1267c6c41f9c2f5532a02004d6c3a575a6)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/0a48e12f67fa16b237b5a2d5c97eb7f1fd2659c7"},{"sha":"6ba57be8a456ccab184250869adb738020fbf080","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"pool: Fix accounting error in repository statistics\n\nMotivation:\n\nWe pre-calculate the output of 'rep ls -s' by listening for repository\nchange events. We have observed this pre-calculation to claim that we\nhave a negative number of files with a sticky flag.\n\nThe repository generates change events when sticky flags expire. The change\nevent is generated for each expired sticky flag, however the before and after\nstate inluded in the event is the same for all these notifications. For the\ncode maintaining the repository statistics this is fatal as it will observe the\nsame file changing from sticky to non-sticky several times even when reality\ntwo sticky flags were expired in one operation.\n\nModification:\n\nDo not include the sticky flag that changed in the notification; it was not\nused by any listener and if needed it can be extracted by calculsting the\ndifference of the before and after state.\n\nResult:\n\nFixed an accounting bug that could cause the output of 'rep ls -s' to be\nwrong.\n\nTarget: trunk\nRequest: 2.14\nRequest: 2.13\nRequest: 2.12\nRequire-notes: yes\nRequire-book: no\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nPatch: https://rb.dcache.org/r/8925/\n(cherry picked from commit 830070e6630db10ae2e35fff3998f66de3cfe3d1)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/6ba57be8a456ccab184250869adb738020fbf080"},{"sha":"935649ad55267150cca2e2ad736369f20ff78b51","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"pool: Fix lock starvation in migration module\n\nMotivation:\n\nA migration job synchronizes on the job monitor to prevent race conditions. In\nparticular during initialization it acquires the monitor lock to prevent a race\nbetween loading an entry and that entry being modified by the pool at the\nchange notification overtaking the initialization thread. To allow concurrent\nqueries to the job object, such as when using the 'migration info' command or\ncollecting the pool info with 'info', the initialization thread releases and\nreacquires the lock for every pool entry.\n\nJava monitor locks have no concept of fairness and thus it is unlikely that\nanother thread can acquire the lock while the job initializes. This leads\nto unresponsive pools. The problem is worst on very large pools or pools\nwith slow meta data access as job initialization takes longer.\n\nModification:\n\nReplace the monitor lock with a fair reentrant lock. I did not use a read-write\nlock as most synchronization would need a write lock anyway.\n\nResult:\n\nResolved a lock starvation issue during migration job initialization that\ncould lead to unresponsive pools.\n\nTarget: trunk\nRequest: 2.14\nRequest: 2.13\nRequire-notes: yes\nRequire-book: no\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nPatch: https://rb.dcache.org/r/8927/\n(cherry picked from commit 9cc9afe0d760db7e7d186ca16214233b43dcdd75)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/935649ad55267150cca2e2ad736369f20ff78b51"},{"sha":"966ca5a830ef02c2ea87b053b08ac3977ff6008b","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"xrootd: Roll back asynchronous reply on open\n\nMotivation:\n\nUpon opening a file at the xrootd door, dCache generates an await-async\nresponse.  It does this because pool selection and mover startup may take a\nlong time (staging, p2p, queing) and this way we tell the client that it\nmay have to wait. The actual response to the open request is an asynchronous\nredirect.\n\nAn asynchronous response allowed by the xrootd protocol and both the old (3.x)\nand new (4.x) xrootd clients handle it. The implementation in the old client\nis however a hack:\n\n- The asynchronous response processing code recognizes the redirect and\n  implements it by: first setting the remote host to that of the redirection\n  target; then forcefully closing the logical connection to the door; then\n  injecting a fake kXR_wait response to wake up the thread doing the kXR_open.\n\n- kXR_wait is a response the server may use to force the client into a polling\n  behaviour. It instructs the client to sleep for some time and then retry the\n  request. dCache doesn't generate the kXR_wait response - it is faked by the\n  asynchronuos response processing code in the client.\n\n- The result is that the client's thread doing kXR_open wakes up, sees the\n  faked kXR_wait response, waits for 1 second (as specified in the asynchronous\n  response processing code), retries the request to the door, fails because\n  the asynchronous response processing code has closed the logial connection,\n  reestablishes the connection but this time to the redirection target (because\n  the asynchronous response processing code changed the remote host) and then\n  resubmits the open request.\n\nThe result is an artifical 1 second delay on redirect. When used from within\nROOT, the intial retry induced by kXR_wait results in an error message being\noutput (\"Unknown logical conn\").\n\nThe 4.x client doesn't suffer from this (but the old client still shipped with\nthe 4.x package does).\n\nModification:\n\nEven though it isn't our fault, the prevelance of the old client forces us\nto change back to the old behaviour and generate a synchronous reply.\n\nResult:\n\nWorked around an xrootd 3.x client misbehaviour that caused the client to\nartificially sleep for 1 secon when opening files and in some cases output\nthe error \"Unkown logical conn X\".\n\nTarget: trunk\nRequest: 2.14\nRequest: 2.13\nRequire-notes: yes\nRequire-book: no\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/8941/\n(cherry picked from commit 1baaf2f88e7c912249be0ef4dc16886a4bdfa349)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/966ca5a830ef02c2ea87b053b08ac3977ff6008b"},{"sha":"c0f8ae6eb83b45afb577a71666b62b75e34cb9ac","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"xrootd: Fix classification of uploads\n\nMotivation:\n\nWe have observed upload failures from Alice jobs. These jobs specify the options\ndelete, mkpath and retstat, but neither new or open_updt. dCache classifies such\nrequests as reads and subsequently fails stating that the file does not exist.\nClearly the delete option only makes sense for an upload, thus this is a bug\nin dCache.\n\nModification:\n\nUpgrade xrootd4j and rely on the fixed classification provided by the new version.\n\nResult:\n\nFixes an xrootd protocol incompatibility that caused uploads to be considered\nas downloads.\n\nTarget: trunk\nRequest: 2.14\nRequest: 2.13\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nRequire-notes: yes\nRequire-book: no\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/8932/\n(cherry picked from commit ac00962decb5d1866484125ec36183d77175387f)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/c0f8ae6eb83b45afb577a71666b62b75e34cb9ac"},{"sha":"c8a7bf719b6c78b758737e97f1ee2f58d5a9bd5f","author":{"email":"e260d3bce7ebdf2b872b611fd23ca302a2bc8a49@desy.de","name":"Paul Millar"},"message":"Merge pull request #2104 from gbehrmann/fix/2.14/rb8928\n\npool: Avoid lock contention when opening files and setting sticky flags","distinct":true,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/c8a7bf719b6c78b758737e97f1ee2f58d5a9bd5f"},{"sha":"8b5a28368a47e6fd1f4c0ff8e11253c3f0c569eb","author":{"email":"e260d3bce7ebdf2b872b611fd23ca302a2bc8a49@desy.de","name":"Paul Millar"},"message":"Merge pull request #2107 from gbehrmann/fix/2.14/rb8927\n\npool: Fix lock starvation in migration module","distinct":true,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/8b5a28368a47e6fd1f4c0ff8e11253c3f0c569eb"},{"sha":"94f26951f4ca93173c691696d9e7a3b0bc1cf442","author":{"email":"e260d3bce7ebdf2b872b611fd23ca302a2bc8a49@desy.de","name":"Paul Millar"},"message":"Merge pull request #2112 from gbehrmann/fix/2.14/rb8934\n\npool: Expose Berkeley DB configuration as dCache properties","distinct":true,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/94f26951f4ca93173c691696d9e7a3b0bc1cf442"},{"sha":"7e2e0056a9c7ec55a5021afa8ccdcab188526493","author":{"email":"e260d3bce7ebdf2b872b611fd23ca302a2bc8a49@desy.de","name":"Paul Millar"},"message":"Merge pull request #2110 from gbehrmann/fix/2.14/rb8941\n\nxrootd: Roll back asynchronous reply on open","distinct":true,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/7e2e0056a9c7ec55a5021afa8ccdcab188526493"},{"sha":"d54b312c2efb70d9ebfe625441ee3b41aad91ac7","author":{"email":"e260d3bce7ebdf2b872b611fd23ca302a2bc8a49@desy.de","name":"Paul Millar"},"message":"Merge pull request #2115 from gbehrmann/fix/2.14/rb8932\n\nxrootd: Fix classification of uploads","distinct":true,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/d54b312c2efb70d9ebfe625441ee3b41aad91ac7"},{"sha":"2e068a5e7c59e5f8eb3a86804566b9af68f2b75d","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"pool: Fix buffer leak in HTTP mover\n\nMotivation:\n\nNetty 4 buffers have to be released when no longer used to allow pooling.\n\nModification:\n\nRelease the buffer if the read in ReuseableChunkedNioFile fails with an\nexception.\n\nResult:\n\nFixed a buffer leak in the HTTP mover.\n\nTarget: trunk\nRequest: 2.14\nRequest: 2.13\nRequest: 2.12\nRequire-notes: yes\nRequire-book: no\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/8934/\n(cherry picked from commit 42fa54bbd14e938f00c26ad43687589d3445fe53)","distinct":true,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/2e068a5e7c59e5f8eb3a86804566b9af68f2b75d"}]}