{"action":"created","issue":{"url":"https://api.github.com/repos/xrootd/xrootd-python/issues/13","labels_url":"https://api.github.com/repos/xrootd/xrootd-python/issues/13/labels{/name}","comments_url":"https://api.github.com/repos/xrootd/xrootd-python/issues/13/comments","events_url":"https://api.github.com/repos/xrootd/xrootd-python/issues/13/events","html_url":"https://github.com/xrootd/xrootd-python/issues/13","id":124391677,"number":13,"title":"Asynchronous callbacks ridiculously hard to use","user":{"login":"bbockelm","id":1093447,"avatar_url":"https://avatars.githubusercontent.com/u/1093447?v=3","gravatar_id":"","url":"https://api.github.com/users/bbockelm","html_url":"https://github.com/bbockelm","followers_url":"https://api.github.com/users/bbockelm/followers","following_url":"https://api.github.com/users/bbockelm/following{/other_user}","gists_url":"https://api.github.com/users/bbockelm/gists{/gist_id}","starred_url":"https://api.github.com/users/bbockelm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bbockelm/subscriptions","organizations_url":"https://api.github.com/users/bbockelm/orgs","repos_url":"https://api.github.com/users/bbockelm/repos","events_url":"https://api.github.com/users/bbockelm/events{/privacy}","received_events_url":"https://api.github.com/users/bbockelm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-12-30T19:43:44Z","updated_at":"2016-01-11T13:54:18Z","closed_at":null,"body":"It appears that it's incredibly difficult to use the asynchronous callback mechanism due to lock ordering issues:\r\n\r\n- The callback thread pool holds various Xrootd locks when it invokes python, which needs the python GIL.\r\n- Python-owned threads have the python GIL and needs various Xrootd locks when invoking operations in the Xrootd module.\r\n\r\nHence, it's quite straightforward to deadlock.\r\n\r\nIn order to actually use the asynchronous callbacks, you must be able to guarantee there is no call to the Xrootd module from any python threads as long as there are pending callbacks.  It's a pretty high barrier.\r\n\r\nWould it be possible to invoke callbacks from a separate thread pool which takes the python GIL first?"},"comment":{"url":"https://api.github.com/repos/xrootd/xrootd-python/issues/comments/170556379","html_url":"https://github.com/xrootd/xrootd-python/issues/13#issuecomment-170556379","issue_url":"https://api.github.com/repos/xrootd/xrootd-python/issues/13","id":170556379,"user":{"login":"bbockelm","id":1093447,"avatar_url":"https://avatars.githubusercontent.com/u/1093447?v=3","gravatar_id":"","url":"https://api.github.com/users/bbockelm","html_url":"https://github.com/bbockelm","followers_url":"https://api.github.com/users/bbockelm/followers","following_url":"https://api.github.com/users/bbockelm/following{/other_user}","gists_url":"https://api.github.com/users/bbockelm/gists{/gist_id}","starred_url":"https://api.github.com/users/bbockelm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bbockelm/subscriptions","organizations_url":"https://api.github.com/users/bbockelm/orgs","repos_url":"https://api.github.com/users/bbockelm/repos","events_url":"https://api.github.com/users/bbockelm/events{/privacy}","received_events_url":"https://api.github.com/users/bbockelm/received_events","type":"User","site_admin":false},"created_at":"2016-01-11T13:54:18Z","updated_at":"2016-01-11T13:54:18Z","body":"@ljanyst - Yeah, we've hit this a few times in CMS, so this part was unremarkable.  Here's the issues I know of:\r\n- If a failure occurs, the callback may not be invoked from the callback threadpool.  Thus, the mutexes held can be problematic to predict, meaning the ordering is difficult.  (Haven't hit this with the python bindings, but hit it quite a few times in CMSSW.)\r\n- There's some overlap between the `XrdCl::File::~File` callpath and the callback threadpool.\r\n- If the main thread tries to exit while a callback is executing, you'll get a deadlock on pthread_join of the callback threadpool."}}