{"action":"closed","issue":{"url":"https://api.github.com/repos/buildkite/agent/issues/775","repository_url":"https://api.github.com/repos/buildkite/agent","labels_url":"https://api.github.com/repos/buildkite/agent/issues/775/labels{/name}","comments_url":"https://api.github.com/repos/buildkite/agent/issues/775/comments","events_url":"https://api.github.com/repos/buildkite/agent/issues/775/events","html_url":"https://github.com/buildkite/agent/issues/775","id":326163862,"number":775,"title":"Agent jobs hanging after process has finished","user":{"login":"avsm","id":53164,"avatar_url":"https://avatars3.githubusercontent.com/u/53164?v=4","gravatar_id":"","url":"https://api.github.com/users/avsm","html_url":"https://github.com/avsm","followers_url":"https://api.github.com/users/avsm/followers","following_url":"https://api.github.com/users/avsm/following{/other_user}","gists_url":"https://api.github.com/users/avsm/gists{/gist_id}","starred_url":"https://api.github.com/users/avsm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/avsm/subscriptions","organizations_url":"https://api.github.com/users/avsm/orgs","repos_url":"https://api.github.com/users/avsm/repos","events_url":"https://api.github.com/users/avsm/events{/privacy}","received_events_url":"https://api.github.com/users/avsm/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-05-24T15:03:35Z","updated_at":"2018-05-25T01:02:34Z","closed_at":"2018-05-25T01:02:34Z","author_association":"NONE","body":"Agent version: tried master branch as of today, and v3.1.2\r\n\r\nI have a step that is building and pushing a docker container to the Hub.  It works fine 99% of the time, but one particular combination is triggering a hang that is causing the job to remain marked as running (and never terminating) after the process spawned has gone.  I see this when running the agent in debug mode on this particular job:\r\n\r\n```\r\n2018-05-24 15:07:00 DEBUG  [LineScanner] Error from scanner: bufio.Scanner: token too long\r\n2018-05-24 15:07:00 DEBUG  [LineScanner] Waiting for callbacks to finish\r\n2018-05-24 15:07:00 DEBUG  [LineScanner] Finished\r\n```\r\n\r\nand then when ctrl-C the agent, a clean stop fails:\r\n\r\n```\r\n^C2018-05-24 15:58:13 DEBUG  Received signal `INT`\r\n2018-05-24 15:58:13 INFO   Forcefully stopping running jobs and stopping the agent\r\n2018-05-24 15:58:13 INFO   Forcefully stopping agent. The current job will be canceled before disconnecting...\r\n2018-05-24 15:58:13 INFO   Canceling job c68a303f-01ad-439f-8310-3c2ab3265779\r\n2018-05-24 15:58:13 DEBUG  [Process] Sending signal: terminated to PID: 45791\r\n2018-05-24 15:58:13 ERROR  [Process] Failed to send signal: terminated to PID: 45791 (*errors.errorString: os: process already finished)\r\n2018-05-24 15:58:14 DEBUG  GET https://agent.buildkite.com/v3/jobs/c68a303f-01ad-439f-8310-3c2ab3265779\r\n2018-05-24 15:58:14 DEBUG  _ GET https://agent.buildkite.com/v3/jobs/c68a303f-01ad-439f-8310-3c2ab3265779 (200 OK 93.440879ms)\r\n```\r\n\r\nctrl-c again just leaves us in a loop of:\r\n\r\n```\r\n2018-05-24 16:02:23 INFO   Forcefully stopping running jobs and stopping the agent\r\n2018-05-24 16:02:23 INFO   Forcefully stopping agent. The current job will be canceled before disconnecting...\r\n2018-05-24 16:02:23 DEBUG  GET https://agent.buildkite.com/v3/jobs/c68a303f-01ad-439f-8310-3c2ab3265779\r\n2018-05-24 16:02:24 DEBUG  _ GET https://agent.buildkite.com/v3/jobs/c68a303f-01ad-439f-8310-3c2ab3265779 (200 OK 99.930666ms)\r\n```\r\n\r\nA kill -9 works, but the job has to be manually cancelled from the web ui even though the agent has gone.\r\n\r\nMysteriously, this only happens on one job of thousands, and is on ppc64le if that is relevant (but other jobs complete just fine there)"}}
{"actor":{"display_login":"lox"}}