{"action":"created","issue":{"url":"https://api.github.com/repos/openshift/origin-metrics/issues/89","repository_url":"https://api.github.com/repos/openshift/origin-metrics","labels_url":"https://api.github.com/repos/openshift/origin-metrics/issues/89/labels{/name}","comments_url":"https://api.github.com/repos/openshift/origin-metrics/issues/89/comments","events_url":"https://api.github.com/repos/openshift/origin-metrics/issues/89/events","html_url":"https://github.com/openshift/origin-metrics/issues/89","id":137681394,"number":89,"title":"Heapster keeps restarting with no reason","user":{"login":"gravis","id":12752,"avatar_url":"https://avatars.githubusercontent.com/u/12752?v=3","gravatar_id":"","url":"https://api.github.com/users/gravis","html_url":"https://github.com/gravis","followers_url":"https://api.github.com/users/gravis/followers","following_url":"https://api.github.com/users/gravis/following{/other_user}","gists_url":"https://api.github.com/users/gravis/gists{/gist_id}","starred_url":"https://api.github.com/users/gravis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gravis/subscriptions","organizations_url":"https://api.github.com/users/gravis/orgs","repos_url":"https://api.github.com/users/gravis/repos","events_url":"https://api.github.com/users/gravis/events{/privacy}","received_events_url":"https://api.github.com/users/gravis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-03-01T20:45:11Z","updated_at":"2016-03-02T14:30:08Z","closed_at":null,"body":"Following discussion on #62 and kubernetes/heapster#925:\r\n\r\nWe have updated 2 clusters from OS 1.1.0 to 1.1.3 directly. One of them is working correctly, but on the last one, metrics have \"holes\":\r\n\r\n<img width=\"865\" alt=\"screenshot 2016-02-27 16 31 18\" src=\"https://cloud.githubusercontent.com/assets/12752/13441175/f472d556-dfc3-11e5-9c48-f0c238b3b535.png\">\r\n\r\nApparently, the heapster pod is being deleted on a regular basis (but not a stable frequency). Every 2-5 minutes, I can see these events:\r\n\r\n```\r\n3:41:08 PM\theapster-a1hjp\tPod\tNormal\tStarted \tStarted container with docker id 42d99f4118df\r\n3:41:07 PM\theapster-a1hjp\tPod\tNormal\tCreated \tCreated container with docker id 42d99f4118df\r\n3:41:06 PM\theapster-a1hjp\tPod\tNormal\tPulled \tSuccessfully pulled image \"docker.io/openshift/origin-metrics-heapster:latest\"\r\n3:41:05 PM\thawkular-metrics-41gx2\tPod\tNormal\tStarted \tStarted container with docker id 75c76cd6921b\r\n3:41:05 PM\thawkular-metrics-41gx2\tPod\tNormal\tCreated \tCreated container with docker id 75c76cd6921b\r\n3:41:04 PM\theapster-a1hjp\tPod\tNormal\tPulling \tpulling image \"docker.io/openshift/origin-metrics-heapster:latest\"\r\n3:41:04 PM\thawkular-metrics-41gx2\tPod\tNormal\tPulled \tSuccessfully pulled image \"docker.io/openshift/origin-metrics-hawkular-metrics:latest\"\r\n3:41:03 PM\thawkular-metrics-41gx2\tPod\tNormal\tPulling \tpulling image \"docker.io/openshift/origin-metrics-hawkular-metrics:latest\"\r\n3:41:02 PM\thawkular-metrics-paek6\tPod\tNormal\tKilling \tKilling container with docker id 8e5ac35cd912: Need to kill pod.\r\n3:41:00 PM\theapster-a1hjp\tPod\tNormal\tScheduled \tSuccessfully assigned heapster-a1hjp to node-1\r\n3:41:00 PM\theapster-ifp3k\tPod\tNormal\tKilling \tKilling container with docker id 968b35a9fc16: Need to kill pod.\r\n3:41:00 PM\thawkular-metrics\tReplicationController\tNormal\tSuccessfulCreate \t(events with common reason combined) (142 times in the last 4 hours, 31 minutes)\r\n3:41:00 PM\thawkular-metrics-41gx2\tPod\tNormal\tScheduled \tSuccessfully assigned hawkular-metrics-41gx2 to node-1\r\n3:40:59 PM\theapster\tReplicationController\tNormal\tSuccessfulCreate \t(events with common reason combined) (144 times in the last 4 hours, 34 minutes)\r\n```\r\n\r\nI have deployed (and redeployed) metrics with a single cassandra, a 20Gb pv, and a signed certificate.\r\nThe same configuration is working well on the first cluster. \r\n\r\nThere is no error in the node's log, nor docker container logs."},"comment":{"url":"https://api.github.com/repos/openshift/origin-metrics/issues/comments/191260018","html_url":"https://github.com/openshift/origin-metrics/issues/89#issuecomment-191260018","issue_url":"https://api.github.com/repos/openshift/origin-metrics/issues/89","id":191260018,"user":{"login":"gravis","id":12752,"avatar_url":"https://avatars.githubusercontent.com/u/12752?v=3","gravatar_id":"","url":"https://api.github.com/users/gravis","html_url":"https://github.com/gravis","followers_url":"https://api.github.com/users/gravis/followers","following_url":"https://api.github.com/users/gravis/following{/other_user}","gists_url":"https://api.github.com/users/gravis/gists{/gist_id}","starred_url":"https://api.github.com/users/gravis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gravis/subscriptions","organizations_url":"https://api.github.com/users/gravis/orgs","repos_url":"https://api.github.com/users/gravis/repos","events_url":"https://api.github.com/users/gravis/events{/privacy}","received_events_url":"https://api.github.com/users/gravis/received_events","type":"User","site_admin":false},"created_at":"2016-03-02T14:30:08Z","updated_at":"2016-03-02T14:30:08Z","body":"I thought it was the livenessProbe, but even with that part removed, I can see pod restarts.\r\nWeird thing, when heapster is restarting, I can see 2 heapster pods starting, then only one is kept:\r\n\r\n<img width=\"589\" alt=\"screenshot 2016-03-02 09 28 51\" src=\"https://cloud.githubusercontent.com/assets/12752/13463214/43c8a442-e059-11e5-8f19-100e873fcf26.png\">\r\n\r\n\r\n<img width=\"451\" alt=\"screenshot 2016-03-02 09 29 38\" src=\"https://cloud.githubusercontent.com/assets/12752/13463230/4f8df9ee-e059-11e5-95d2-61a8d6359459.png\">\r\n\r\nI really wonder while the RC would create 2 pods for 1 replica only..."}}