{"action":"created","issue":{"url":"https://api.github.com/repos/flynn/flynn/issues/4334","repository_url":"https://api.github.com/repos/flynn/flynn","labels_url":"https://api.github.com/repos/flynn/flynn/issues/4334/labels{/name}","comments_url":"https://api.github.com/repos/flynn/flynn/issues/4334/comments","events_url":"https://api.github.com/repos/flynn/flynn/issues/4334/events","html_url":"https://github.com/flynn/flynn/issues/4334","id":298094494,"number":4334,"title":"Scheduler keeps crashing","user":{"login":"reneweteling","id":3830933,"avatar_url":"https://avatars2.githubusercontent.com/u/3830933?v=4","gravatar_id":"","url":"https://api.github.com/users/reneweteling","html_url":"https://github.com/reneweteling","followers_url":"https://api.github.com/users/reneweteling/followers","following_url":"https://api.github.com/users/reneweteling/following{/other_user}","gists_url":"https://api.github.com/users/reneweteling/gists{/gist_id}","starred_url":"https://api.github.com/users/reneweteling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reneweteling/subscriptions","organizations_url":"https://api.github.com/users/reneweteling/orgs","repos_url":"https://api.github.com/users/reneweteling/repos","events_url":"https://api.github.com/users/reneweteling/events{/privacy}","received_events_url":"https://api.github.com/users/reneweteling/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-02-18T15:46:22Z","updated_at":"2018-02-19T08:30:01Z","closed_at":null,"author_association":"NONE","body":"Hi Guys,\r\n\r\nMy server wanst responding, so ive executed a hard reset, then i found that my host-state.bolt file is over 6GB, so i started running the script ive got a while back from one of you guys, the `flynn-host-state-clean.sh` and it cleand for about 5 minutes more than 900.000 jobs. \r\n\r\nAfther that i tried rebooting the machine, but when i run `flynn-host ps` a couple of times i see that the scheduler keeps crashing (startting time keeps resetting)\r\n\r\nSo i checked the log, end this is wat i saw. \r\n\r\n```\r\nt=2018-02-18T15:34:23+0000 lvl=warn msg=\"ignoring unknown volume\" component=scheduler job.id=cc5de76b-6be4-4bf5-9c56-97623132fe15 vol.id=a871327c-5978-4aa4-8534-ff05d7bbf295\r\nt=2018-02-18T15:34:23+0000 lvl=info msg=\"handling job status change\" component=scheduler fn=handleJobStatus job.id=Ubuntu1604xenial64minimal-cc5de76b-6be4-4bf5-9c56-97623132fe15 app.id=bf8e8f54-f323-45ba-8b94-b83a9ac83845 release.id=3e3a505e-5230-46cd-8ffa-f5b121ff7ad7 job.type=redis from= to=stopped\r\nt=2018-02-18T15:34:23+0000 lvl=eror msg=\"error looking up formation for job\" component=scheduler fn=handleJobStatus job.id=Ubuntu1604xenial64minimal-cc5de76b-6be4-4bf5-9c56-97623132fe15 app.id=bf8e8f54-f323-45ba-8b94-b83a9ac83845 release.id=3e3a505e-5230-46cd-8ffa-f5b121ff7ad7 job.type=redis err=\"controller: resource not found\"\r\nt=2018-02-18T15:34:23+0000 lvl=warn msg=\"ignoring unknown volume\" component=scheduler job.id=08e3ac00-4fda-453b-8a53-cbd86c71e84f vol.id=a871327c-5978-4aa4-8534-ff05d7bbf295\r\nt=2018-02-18T15:34:23+0000 lvl=info msg=\"handling job status change\" component=scheduler fn=handleJobStatus job.id=Ubuntu1604xenial64minimal-08e3ac00-4fda-453b-8a53-cbd86c71e84f app.id=bf8e8f54-f323-45ba-8b94-b83a9ac83845 release.id=3e3a505e-5230-46cd-8ffa-f5b121ff7ad7 job.type=redis from= to=stopped\r\nt=2018-02-18T15:34:23+0000 lvl=eror msg=\"error looking up formation for job\" component=scheduler fn=handleJobStatus job.id=Ubuntu1604xenial64minimal-08e3ac00-4fda-453b-8a53-cbd86c71e84f app.id=bf8e8f54-f323-45ba-8b94-b83a9ac83845 release.id=3e3a505e-5230-46cd-8ffa-f5b121ff7ad7 job.type=redis err=\"controller: resource not found\"\r\nt=2018-02-18T15:34:23+0000 lvl=warn msg=\"ignoring unknown volume\" component=scheduler job.id=bbc513b7-1018-46cb-8a26-4e5806f78b2a vol.id=6bf0089a-ab87-4a06-af70-8fcff80c2f91\r\n```\r\n\r\nIm a bit at a loss here, im tryting to get the debug info, but the server just hangs here for 5 min and e the job is taking over 32GB of internal memory\r\n\r\n```\r\nroot@Ubuntu-1604-xenial-64-minimal ~ # flynn-host collect-debug-info\r\nINFO[02-18|16:40:29] uploading logs and debug information to a private, anonymous gist\r\nINFO[02-18|16:40:29] this may take a while depending on the size of your logs\r\nINFO[02-18|16:40:29] getting flynn-host logs\r\nINFO[02-18|16:41:50] getting sirenia metadata\r\nINFO[02-18|16:41:50] getting scheduler state\r\nEROR[02-18|16:41:50] error getting scheduler state            err=\"object_not_found: no leader found\"\r\nINFO[02-18|16:41:50] getting job logs\r\n```\r\n\r\nThe server is pretty decent, 64GB ram, SSD disks and 8 processors. But its maxing out.... so something is quite wrong. \r\n\r\nThanks again!!!"},"comment":{"url":"https://api.github.com/repos/flynn/flynn/issues/comments/366618865","html_url":"https://github.com/flynn/flynn/issues/4334#issuecomment-366618865","issue_url":"https://api.github.com/repos/flynn/flynn/issues/4334","id":366618865,"user":{"login":"reneweteling","id":3830933,"avatar_url":"https://avatars2.githubusercontent.com/u/3830933?v=4","gravatar_id":"","url":"https://api.github.com/users/reneweteling","html_url":"https://github.com/reneweteling","followers_url":"https://api.github.com/users/reneweteling/followers","following_url":"https://api.github.com/users/reneweteling/following{/other_user}","gists_url":"https://api.github.com/users/reneweteling/gists{/gist_id}","starred_url":"https://api.github.com/users/reneweteling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reneweteling/subscriptions","organizations_url":"https://api.github.com/users/reneweteling/orgs","repos_url":"https://api.github.com/users/reneweteling/repos","events_url":"https://api.github.com/users/reneweteling/events{/privacy}","received_events_url":"https://api.github.com/users/reneweteling/received_events","type":"User","site_admin":false},"created_at":"2018-02-19T08:30:01Z","updated_at":"2018-02-19T08:30:01Z","author_association":"NONE","body":"The size of the host-state.bolt is just rediculous\r\n\r\n![image](https://user-images.githubusercontent.com/3830933/36368427-6fb85e14-1557-11e8-82e8-93eed67326bb.png)\r\n\r\ni think that has something to do with it."}}
{"actor":{"display_login":"reneweteling"}}