{"action":"created","issue":{"url":"https://api.github.com/repos/malthe/pq/issues/15","repository_url":"https://api.github.com/repos/malthe/pq","labels_url":"https://api.github.com/repos/malthe/pq/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/malthe/pq/issues/15/comments","events_url":"https://api.github.com/repos/malthe/pq/issues/15/events","html_url":"https://github.com/malthe/pq/pull/15","id":174990161,"number":15,"title":"get() handles non-existent q_name cost-effectively","user":{"login":"mhjohnson","id":831487,"avatar_url":"https://avatars.githubusercontent.com/u/831487?v=3","gravatar_id":"","url":"https://api.github.com/users/mhjohnson","html_url":"https://github.com/mhjohnson","followers_url":"https://api.github.com/users/mhjohnson/followers","following_url":"https://api.github.com/users/mhjohnson/following{/other_user}","gists_url":"https://api.github.com/users/mhjohnson/gists{/gist_id}","starred_url":"https://api.github.com/users/mhjohnson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mhjohnson/subscriptions","organizations_url":"https://api.github.com/users/mhjohnson/orgs","repos_url":"https://api.github.com/users/mhjohnson/repos","events_url":"https://api.github.com/users/mhjohnson/events{/privacy}","received_events_url":"https://api.github.com/users/mhjohnson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-09-05T03:42:16Z","updated_at":"2016-09-08T14:04:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/malthe/pq/pulls/15","html_url":"https://github.com/malthe/pq/pull/15","diff_url":"https://github.com/malthe/pq/pull/15.diff","patch_url":"https://github.com/malthe/pq/pull/15.patch"},"body":"Hi @malthe - \r\nThis is a simple fix intended to address Issue #14 that @3manuek posted.  \r\n\r\nWe discovered some very long running queries on our database today that were due to a worker trying to `get()` a task for a `q_name` that did not exist in our table.  This situation can occur when the rows of a `q_name` are purged from the tasks table and then`put()` occurs very infrequently.  If the worker is trying to `get()` repetitively in this scenario, it results in the costly queries that @3manuek described (especially on a big table with many other q_names).  \r\n\r\nI'm no wizard, but these changes seem safe and introduce little need to refactor the tests.  I used `time.sleep(timeout)` in lieu of the unix `select.select(timeout)`, since it didn't seem appropriate to monitor the file descriptor in this specific context.\r\n\r\nI hope it works out!  If not, I'd love to hear your thoughts on a better approach.\r\n\r\nThanks,\r\n-Matt"},"comment":{"url":"https://api.github.com/repos/malthe/pq/issues/comments/245607742","html_url":"https://github.com/malthe/pq/pull/15#issuecomment-245607742","issue_url":"https://api.github.com/repos/malthe/pq/issues/15","id":245607742,"user":{"login":"3manuek","id":1935149,"avatar_url":"https://avatars.githubusercontent.com/u/1935149?v=3","gravatar_id":"","url":"https://api.github.com/users/3manuek","html_url":"https://github.com/3manuek","followers_url":"https://api.github.com/users/3manuek/followers","following_url":"https://api.github.com/users/3manuek/following{/other_user}","gists_url":"https://api.github.com/users/3manuek/gists{/gist_id}","starred_url":"https://api.github.com/users/3manuek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/3manuek/subscriptions","organizations_url":"https://api.github.com/users/3manuek/orgs","repos_url":"https://api.github.com/users/3manuek/repos","events_url":"https://api.github.com/users/3manuek/events{/privacy}","received_events_url":"https://api.github.com/users/3manuek/received_events","type":"User","site_admin":false},"created_at":"2016-09-08T14:04:22Z","updated_at":"2016-09-08T14:04:22Z","body":"@malthe The problem is that the filtering is not done over only 1 backend, _every backend that is listening on at least one channel_ is an entry on the async array. We know that the design would need to step away from  _transactional_ to an app-level queue system in a future, but we need to keep it as it is until then. \r\n\r\nWith the change purposed above (splitting the listeners), I'm pretty confident that the CPU usage will reduce considerably in certain scenarios (when you split the queues in the same database).   "}}
{"actor":{"display_login":"3manuek"}}