{"action":"closed","issue":{"url":"https://api.github.com/repos/AlexsLemonade/refinebio/issues/789","repository_url":"https://api.github.com/repos/AlexsLemonade/refinebio","labels_url":"https://api.github.com/repos/AlexsLemonade/refinebio/issues/789/labels{/name}","comments_url":"https://api.github.com/repos/AlexsLemonade/refinebio/issues/789/comments","events_url":"https://api.github.com/repos/AlexsLemonade/refinebio/issues/789/events","html_url":"https://github.com/AlexsLemonade/refinebio/issues/789","id":378475318,"node_id":"MDU6SXNzdWUzNzg0NzUzMTg=","number":789,"title":"Foreman can't re-queue any jobs because of the database connection closing","user":{"login":"kurtwheeler","id":4026833,"node_id":"MDQ6VXNlcjQwMjY4MzM=","avatar_url":"https://avatars1.githubusercontent.com/u/4026833?v=4","gravatar_id":"","url":"https://api.github.com/users/kurtwheeler","html_url":"https://github.com/kurtwheeler","followers_url":"https://api.github.com/users/kurtwheeler/followers","following_url":"https://api.github.com/users/kurtwheeler/following{/other_user}","gists_url":"https://api.github.com/users/kurtwheeler/gists{/gist_id}","starred_url":"https://api.github.com/users/kurtwheeler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kurtwheeler/subscriptions","organizations_url":"https://api.github.com/users/kurtwheeler/orgs","repos_url":"https://api.github.com/users/kurtwheeler/repos","events_url":"https://api.github.com/users/kurtwheeler/events{/privacy}","received_events_url":"https://api.github.com/users/kurtwheeler/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-11-07T21:24:33Z","updated_at":"2018-11-09T00:51:57Z","closed_at":"2018-11-09T00:51:57Z","author_association":"COLLABORATOR","body":"### Context\r\n\r\nThe Foreman logs are full of this:\r\n```\r\n2018-11-07 20:02:40,247 i-0d4a6a561fd82091c data_refinery_foreman.foreman.main ERROR: Exception caught by Foreman while running retry_hung_survey_jobs\r\n Traceback (most recent call last):\r\n   File \"/usr/local/lib/python3.5/dist-packages/django/db/backends/base/base.py\", line 234, in _cursor\r\n     return self._prepare_cursor(self.create_cursor(name))\r\n   File \"/usr/local/lib/python3.5/dist-packages/django/db/backends/postgresql/base.py\", line 222, in create_cursor\r\n     cursor = self.connection.cursor()\r\n psycopg2.InterfaceError: connection already closed\r\n \r\n The above exception was the direct cause of the following exception:\r\n \r\n Traceback (most recent call last):\r\n   File \"/home/user/data_refinery_foreman/foreman/main.py\", line 65, in wrapper\r\n     function(*args, **kwargs)\r\n   File \"/home/user/data_refinery_foreman/foreman/main.py\", line 508, in retry_hung_survey_jobs\r\n     for job in potentially_hung_jobs:\r\n   File \"/usr/local/lib/python3.5/dist-packages/django/db/models/query.py\", line 268, in __iter__\r\n     self._fetch_all()\r\n   File \"/usr/local/lib/python3.5/dist-packages/django/db/models/query.py\", line 1186, in _fetch_all\r\n     self._result_cache = list(self._iterable_class(self))\r\n   File \"/usr/local/lib/python3.5/dist-packages/django/db/models/query.py\", line 54, in __iter__\r\n     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)\r\n   File \"/usr/local/lib/python3.5/dist-packages/django/db/models/sql/compiler.py\", line 1063, in execute_sql\r\n     cursor = self.connection.cursor()\r\n   File \"/usr/local/lib/python3.5/dist-packages/django/db/backends/base/base.py\", line 255, in cursor\r\n     return self._cursor()\r\n   File \"/usr/local/lib/python3.5/dist-packages/django/db/backends/base/base.py\", line 234, in _cursor\r\n     return self._prepare_cursor(self.create_cursor(name))\r\n   File \"/usr/local/lib/python3.5/dist-packages/django/db/utils.py\", line 89, in __exit__\r\n     raise dj_exc_value.with_traceback(traceback) from exc_value\r\n   File \"/usr/local/lib/python3.5/dist-packages/django/db/backends/base/base.py\", line 234, in _cursor\r\n     return self._prepare_cursor(self.create_cursor(name))\r\n   File \"/usr/local/lib/python3.5/dist-packages/django/db/backends/postgresql/base.py\", line 222, in create_cursor\r\n     cursor = self.connection.cursor()\r\n django.db.utils.InterfaceError: connection already closed\r\n```\r\n\r\n### Problem or idea\r\n\r\nI am not sure what is going on, but there are >40k downloader jobs that need to be requeued (they hit their retry limit because of getting cancelled and requeued too many times before we added a fix to prevent that issue). None of them are getting requeued because of that error, so we aren't processing much.\r\n\r\n### Solution or next step\r\n\r\nI am not yet sure what the solution is, but we gotta get the Foreman back up and running so we can requeue the work and process lots of stuff!\r\n\r\n### New Issue Checklist\r\n\r\n- [x] The title is short and descriptive.\r\n- [x] You have explained the context that led you to write this issue.\r\n- [x] You have reported a problem or idea.\r\n- [x] You have proposed a solution or next step.\r\n"}}
{"actor":{"display_login":"kurtwheeler"}}