{"action":"created","issue":{"url":"https://api.github.com/repos/ganga-devs/ganga/issues/182","repository_url":"https://api.github.com/repos/ganga-devs/ganga","labels_url":"https://api.github.com/repos/ganga-devs/ganga/issues/182/labels{/name}","comments_url":"https://api.github.com/repos/ganga-devs/ganga/issues/182/comments","events_url":"https://api.github.com/repos/ganga-devs/ganga/issues/182/events","html_url":"https://github.com/ganga-devs/ganga/pull/182","id":132371558,"number":182,"title":"Bugfix/fix repo lockup bug","user":{"login":"rob-c","id":1102021,"avatar_url":"https://avatars.githubusercontent.com/u/1102021?v=3","gravatar_id":"","url":"https://api.github.com/users/rob-c","html_url":"https://github.com/rob-c","followers_url":"https://api.github.com/users/rob-c/followers","following_url":"https://api.github.com/users/rob-c/following{/other_user}","gists_url":"https://api.github.com/users/rob-c/gists{/gist_id}","starred_url":"https://api.github.com/users/rob-c/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rob-c/subscriptions","organizations_url":"https://api.github.com/users/rob-c/orgs","repos_url":"https://api.github.com/users/rob-c/repos","events_url":"https://api.github.com/users/rob-c/events{/privacy}","received_events_url":"https://api.github.com/users/rob-c/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ganga-devs/ganga/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/ganga-devs/ganga/labels/Core","name":"Core","color":"d4c5f9"},{"url":"https://api.github.com/repos/ganga-devs/ganga/labels/in%20progress","name":"in progress","color":"bfe5bf"}],"state":"open","locked":false,"assignee":{"login":"rob-c","id":1102021,"avatar_url":"https://avatars.githubusercontent.com/u/1102021?v=3","gravatar_id":"","url":"https://api.github.com/users/rob-c","html_url":"https://github.com/rob-c","followers_url":"https://api.github.com/users/rob-c/followers","following_url":"https://api.github.com/users/rob-c/following{/other_user}","gists_url":"https://api.github.com/users/rob-c/gists{/gist_id}","starred_url":"https://api.github.com/users/rob-c/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rob-c/subscriptions","organizations_url":"https://api.github.com/users/rob-c/orgs","repos_url":"https://api.github.com/users/rob-c/repos","events_url":"https://api.github.com/users/rob-c/events{/privacy}","received_events_url":"https://api.github.com/users/rob-c/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/ganga-devs/ganga/milestones/4","html_url":"https://github.com/ganga-devs/ganga/milestones/6.1.16","labels_url":"https://api.github.com/repos/ganga-devs/ganga/milestones/4/labels","id":1489242,"number":4,"title":"6.1.16","description":"","creator":{"login":"egede","id":4319174,"avatar_url":"https://avatars.githubusercontent.com/u/4319174?v=3","gravatar_id":"","url":"https://api.github.com/users/egede","html_url":"https://github.com/egede","followers_url":"https://api.github.com/users/egede/followers","following_url":"https://api.github.com/users/egede/following{/other_user}","gists_url":"https://api.github.com/users/egede/gists{/gist_id}","starred_url":"https://api.github.com/users/egede/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/egede/subscriptions","organizations_url":"https://api.github.com/users/egede/orgs","repos_url":"https://api.github.com/users/egede/repos","events_url":"https://api.github.com/users/egede/events{/privacy}","received_events_url":"https://api.github.com/users/egede/received_events","type":"User","site_admin":false},"open_issues":6,"closed_issues":5,"state":"open","created_at":"2016-01-07T10:47:56Z","updated_at":"2016-02-09T09:36:52Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2016-02-09T09:36:51Z","updated_at":"2016-02-09T17:04:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ganga-devs/ganga/pulls/182","html_url":"https://github.com/ganga-devs/ganga/pull/182","diff_url":"https://github.com/ganga-devs/ganga/pull/182.diff","patch_url":"https://github.com/ganga-devs/ganga/pull/182.patch"},"body":"This PR addresses the issue where ```_checkBackend``` method within the Monitoring Loop would randomly lock up.\r\n\r\nIt appears that a combination of this method on Monitoring thread, the JobsRepisotory and the Registry access on a ```MonitoringWorkerThread``` was causing the repository to enter a state where the re-entrant locking stopped behaving as expected.\r\nIt's unclear if this is actually a re-entrant lock being caught between threads requiring a task to be completed but it appears that the registry is performing a task on the worker thread and then gets stuck and unable to proceed.\r\n\r\nI've traced the full monitoring loop through the tests with the current thread name on in the debug logging. (this was difficult to parse but hopefully worth the effort). This showed the worker threads to stall at a lock statement in the registry and never seem to progress any further even though it looks like all other threads (and the main thread) have moved onto other tasks.\r\n\r\nAs it stands I've since implemented hard non-rentrant locks around methods such as flush and load to avoid in-thread corruption issues so these older locks aren't needed here. Other than that these RLocks are mainly around _relatively_ safe commands such as ```__len__``` etc, which if we knew them to be particularly unsafe i.e. length of a repo when jobs are being added, we can in future add a 'better' locking mechanism at the registry level.\r\n\r\nGiven this was causing the backend monitoring JobAction in the monitoring loop to lock and only seems to effect this I'm hoping this _might_ have actually fixed some of the mystery behaviour in the Monitoring.\r\n\r\n\r\nEdit:\r\nIn addition I've included some minor changes in the monitoring loop which were helpful as well as some minor changes to the GangaUnitTest to allow for unittests to run in their own directory(registry) to isolate them from problems or corruptions from other jobs."},"comment":{"url":"https://api.github.com/repos/ganga-devs/ganga/issues/comments/181958838","html_url":"https://github.com/ganga-devs/ganga/pull/182#issuecomment-181958838","issue_url":"https://api.github.com/repos/ganga-devs/ganga/issues/182","id":181958838,"user":{"login":"rob-c","id":1102021,"avatar_url":"https://avatars.githubusercontent.com/u/1102021?v=3","gravatar_id":"","url":"https://api.github.com/users/rob-c","html_url":"https://github.com/rob-c","followers_url":"https://api.github.com/users/rob-c/followers","following_url":"https://api.github.com/users/rob-c/following{/other_user}","gists_url":"https://api.github.com/users/rob-c/gists{/gist_id}","starred_url":"https://api.github.com/users/rob-c/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rob-c/subscriptions","organizations_url":"https://api.github.com/users/rob-c/orgs","repos_url":"https://api.github.com/users/rob-c/repos","events_url":"https://api.github.com/users/rob-c/events{/privacy}","received_events_url":"https://api.github.com/users/rob-c/received_events","type":"User","site_admin":false},"created_at":"2016-02-09T17:04:19Z","updated_at":"2016-02-09T17:04:19Z","body":"I'll try and do some more testing this-evening/tomorrow and see if this is good enough to ask for merger."}}