{"action":"closed","issue":{"url":"https://api.github.com/repos/buildbot/sandbox/issues/6","repository_url":"https://api.github.com/repos/buildbot/sandbox","labels_url":"https://api.github.com/repos/buildbot/sandbox/issues/6/labels{/name}","comments_url":"https://api.github.com/repos/buildbot/sandbox/issues/6/comments","events_url":"https://api.github.com/repos/buildbot/sandbox/issues/6/events","html_url":"https://github.com/buildbot/sandbox/issues/6","id":188595237,"number":6,"title":"Latent build slaves shut down uncleanly and get forgotten by the master","user":{"login":"bb-bot","id":13008127,"avatar_url":"https://avatars.githubusercontent.com/u/13008127?v=3","gravatar_id":"","url":"https://api.github.com/users/bb-bot","html_url":"https://github.com/bb-bot","followers_url":"https://api.github.com/users/bb-bot/followers","following_url":"https://api.github.com/users/bb-bot/following{/other_user}","gists_url":"https://api.github.com/users/bb-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/bb-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bb-bot/subscriptions","organizations_url":"https://api.github.com/users/bb-bot/orgs","repos_url":"https://api.github.com/users/bb-bot/repos","events_url":"https://api.github.com/users/bb-bot/events{/privacy}","received_events_url":"https://api.github.com/users/bb-bot/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/buildbot/sandbox/milestones/1","html_url":"https://github.com/buildbot/sandbox/milestones/0.9.+","labels_url":"https://api.github.com/repos/buildbot/sandbox/milestones/1/labels","id":2127861,"number":1,"title":"0.9.+","description":null,"creator":{"login":"bb-bot","id":13008127,"avatar_url":"https://avatars.githubusercontent.com/u/13008127?v=3","gravatar_id":"","url":"https://api.github.com/users/bb-bot","html_url":"https://github.com/bb-bot","followers_url":"https://api.github.com/users/bb-bot/followers","following_url":"https://api.github.com/users/bb-bot/following{/other_user}","gists_url":"https://api.github.com/users/bb-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/bb-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bb-bot/subscriptions","organizations_url":"https://api.github.com/users/bb-bot/orgs","repos_url":"https://api.github.com/users/bb-bot/repos","events_url":"https://api.github.com/users/bb-bot/events{/privacy}","received_events_url":"https://api.github.com/users/bb-bot/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":23,"state":"open","created_at":"2016-11-10T19:23:04Z","updated_at":"2016-11-11T05:15:03Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2016-11-10T19:23:05Z","updated_at":"2016-11-11T05:15:03Z","closed_at":"2016-11-11T05:15:03Z","body":"This ticket is a migrated Trac ticket [1780](http://trac.buildbot.net/ticket/1780)\n\nPeople contributed to the original ticket: None\n\n---\n\nOccasionally when the master shuts down a latent buildslave it'll fail weirdly, and the master decides that the latent build slave is broken and never tries to reboot it.\n\nUnfortunately I don't have a lot of insight into what's actually happening, but I'll provide as much detail as a I can:\n\nThe buildmaster is http://buildbot.djangoproject.com/. All the code running there lives at https://github.com/jacobian/django-buildmaster, and you can see the specific latent buildslave implementation at https://github.com/jacobian/django-buildmaster/blob/master/djangobotcfg/rsc_slave.py.\n\nHere's what I see in the logs when this error occurs:\n\n\n    2011-01-26 10:22:41-0800 [-] disconnecting old slave bs1.jacobian.org now\n    2011-01-26 10:22:41-0800 [-] waiting for slave to finish disconnecting\n    2011-01-26 10:22:41-0800 [-] [[DjangoCloudserversBuildSlave]] bs1.jacobian.org deleting instance 572258\n    2011-01-26 10:22:41-0800 [Broker,2,204.232.209.196] [[BuildSlave]].detached(bs1.jacobian.org)\n    2011-01-26 10:22:45-0800 [Broker,3,204.232.209.196] slave 'bs1.jacobian.org' attaching from I[[Pv4Address]](TCP, '204.232.209.196', 53732)\n    2011-01-26 10:22:45-0800 [Broker,3,204.232.209.196] Slave bs1.jacobian.org received connection while not trying to substantiate.  Disconnecting.\n    2011-01-26 10:22:45-0800 [Broker,3,204.232.209.196] waiting for slave to finish disconnecting\n    2011-01-26 10:22:45-0800 [Broker,3,204.232.209.196] Peer will receive following PB traceback:\n    2011-01-26 10:22:45-0800 [Broker,3,204.232.209.196] Unhandled Error\n            Traceback (most recent call last):\n            Failure: exceptions.[[RuntimeError]]: Slave bs1.jacobian.org received connection while not trying to substantiate.  Disconnecting.\n            \n    2011-01-26 10:22:45-0800 [-] [[DjangoCloudserversBuildSlave]] bs1.jacobian.org deleted instance 572258\n\n\nThe lines \"Django[[CloudserversBuildSlave]] bs1.jacobian.org deleting instance 572258\" and \"Django[[CloudserversBuildSlave]] bs1.jacobian.org deleted instance 572258\" are coming from my code; the rest are logged by Buildbot itself.\n\nThe problem isn't the connection error: the slave gets shut down just a few seconds later. But when this happens the master decides the slave is somehow broken and never boots another instance. The only way to get it working again is to restart the buildmaster.\n\nThat's all I know for sure, but here's my speculation on what I *think* might be happening: it appears that the build master disconnects my latent slave, then calls `stop_instance()` to shut it down. The master then detaches the slave. If the shutdown hasn't finished quickly enough, though, it looks like the slave tries to reconnect -- it's been kicked off by the master, and not yet killed as part of the shutdown process. So it looks like the master freaks out and decides that the slave's misbehaving and never tries to boot it again.\n\nIt seems that the master should just ignore connections from the slave while it's trying to unsubstantiate the slave. Otherwise unless the slave shuts down immediately upon the `stop_instance()` call it seems like this'll happen again and again.\n\n\n---\n"}}
{"actor":{"display_login":"sa2ajj"}}