{"action":"created","issue":{"url":"https://api.github.com/repos/edx/edx-platform/issues/14061","repository_url":"https://api.github.com/repos/edx/edx-platform","labels_url":"https://api.github.com/repos/edx/edx-platform/issues/14061/labels{/name}","comments_url":"https://api.github.com/repos/edx/edx-platform/issues/14061/comments","events_url":"https://api.github.com/repos/edx/edx-platform/issues/14061/events","html_url":"https://github.com/edx/edx-platform/pull/14061","id":192132843,"number":14061,"title":"Add logging around problematically queued task","user":{"login":"efischer19","id":7373924,"avatar_url":"https://avatars.githubusercontent.com/u/7373924?v=3","gravatar_id":"","url":"https://api.github.com/users/efischer19","html_url":"https://github.com/efischer19","followers_url":"https://api.github.com/users/efischer19/followers","following_url":"https://api.github.com/users/efischer19/following{/other_user}","gists_url":"https://api.github.com/users/efischer19/gists{/gist_id}","starred_url":"https://api.github.com/users/efischer19/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/efischer19/subscriptions","organizations_url":"https://api.github.com/users/efischer19/orgs","repos_url":"https://api.github.com/users/efischer19/repos","events_url":"https://api.github.com/users/efischer19/events{/privacy}","received_events_url":"https://api.github.com/users/efischer19/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-11-28T22:19:23Z","updated_at":"2016-11-28T22:23:20Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/edx/edx-platform/pulls/14061","html_url":"https://github.com/edx/edx-platform/pull/14061","diff_url":"https://github.com/edx/edx-platform/pull/14061.diff","patch_url":"https://github.com/edx/edx-platform/pull/14061.patch"},"body":"# [TNL-5408](https://openedx.atlassian.net/browse/TNL-5408)\r\n\r\nThus far, I've been completely unable to discern any rhyme or reason as to why this fails sometimes. It kind of correlates with production deployments if you squint a bit, but there exist problem cases nowhere near any deploys as well.\r\n\r\nAll I can say for sure is that this is a problem with the task being queued, not the execution of that task on picked up by a queue. A CMS worker will always fail to runs these tasks (And as a fun addition - there are 2 different ways it can fail, and I have no idea why that's not deterministic - see screenshot below). To that end, I've beefed up logging of this particular task being enqueued. Hopefully running this for a few days will provide some additional data/insight into why this is happening.\r\n\r\nWhat do you think, @jibsheet and @nasthagiri? Does this look like something we could sneak onto the ongoing RC/push out a patch release immediately after? From splunk/new relic, it looks like there were some ~5000 instances of this task (0.55 rpm) over the past week, so I don't think the additional volume will be overwhelming."},"comment":{"url":"https://api.github.com/repos/edx/edx-platform/issues/comments/263413780","html_url":"https://github.com/edx/edx-platform/pull/14061#issuecomment-263413780","issue_url":"https://api.github.com/repos/edx/edx-platform/issues/14061","id":263413780,"user":{"login":"jibsheet","id":57604,"avatar_url":"https://avatars.githubusercontent.com/u/57604?v=3","gravatar_id":"","url":"https://api.github.com/users/jibsheet","html_url":"https://github.com/jibsheet","followers_url":"https://api.github.com/users/jibsheet/followers","following_url":"https://api.github.com/users/jibsheet/following{/other_user}","gists_url":"https://api.github.com/users/jibsheet/gists{/gist_id}","starred_url":"https://api.github.com/users/jibsheet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jibsheet/subscriptions","organizations_url":"https://api.github.com/users/jibsheet/orgs","repos_url":"https://api.github.com/users/jibsheet/repos","events_url":"https://api.github.com/users/jibsheet/events{/privacy}","received_events_url":"https://api.github.com/users/jibsheet/received_events","type":"User","site_admin":false},"created_at":"2016-11-28T22:23:20Z","updated_at":"2016-11-28T22:23:20Z","body":"I'm generally fine with this, and need to run before looking very hard, but admit I'm confused why something is trying to get an XQUEUE setting for that task."}}
{"actor":{"display_login":"jibsheet"}}