{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kops/issues/4161","repository_url":"https://api.github.com/repos/kubernetes/kops","labels_url":"https://api.github.com/repos/kubernetes/kops/issues/4161/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kops/issues/4161/comments","events_url":"https://api.github.com/repos/kubernetes/kops/issues/4161/events","html_url":"https://github.com/kubernetes/kops/issues/4161","id":284913192,"node_id":"MDU6SXNzdWUyODQ5MTMxOTI=","number":4161,"title":"Protokube fails when restarting journald","user":{"login":"mzahn86","id":8417934,"node_id":"MDQ6VXNlcjg0MTc5MzQ=","avatar_url":"https://avatars3.githubusercontent.com/u/8417934?v=4","gravatar_id":"","url":"https://api.github.com/users/mzahn86","html_url":"https://github.com/mzahn86","followers_url":"https://api.github.com/users/mzahn86/followers","following_url":"https://api.github.com/users/mzahn86/following{/other_user}","gists_url":"https://api.github.com/users/mzahn86/gists{/gist_id}","starred_url":"https://api.github.com/users/mzahn86/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mzahn86/subscriptions","organizations_url":"https://api.github.com/users/mzahn86/orgs","repos_url":"https://api.github.com/users/mzahn86/repos","events_url":"https://api.github.com/users/mzahn86/events{/privacy}","received_events_url":"https://api.github.com/users/mzahn86/received_events","type":"User","site_admin":false},"labels":[{"id":778139307,"node_id":"MDU6TGFiZWw3NzgxMzkzMDc=","url":"https://api.github.com/repos/kubernetes/kops/labels/lifecycle/stale","name":"lifecycle/stale","color":"795548","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-12-28T13:19:31Z","updated_at":"2018-08-14T17:53:15Z","closed_at":null,"author_association":"NONE","body":"1. What `kops` version are you running? \r\n> Version 1.8.0 (git-5099bc5)\r\n\r\n2. What Kubernetes version are you running? \r\n\r\n> Client Version: version.Info{Major:\"1\", Minor:\"8\", GitVersion:\"v1.8.4\", GitCommit:\"9befc2b8928a9426501d3bf62f72849d5cbcd5a3\", GitTreeState:\"clean\", BuildDate:\"2017-11-20T05:28:34Z\", GoVersion:\"go1.8.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n\r\n> Server Version: version.Info{Major:\"1\", Minor:\"8\", GitVersion:\"v1.8.4\", GitCommit:\"9befc2b8928a9426501d3bf62f72849d5cbcd5a3\", GitTreeState:\"clean\", BuildDate:\"2017-11-20T05:17:43Z\", GoVersion:\"go1.8.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n\r\n\r\n3. What cloud provider are you using?\r\n> aws\r\n\r\n4. What commands did you run?  What is the simplest way to reproduce this issue?\r\n```\r\nkops create cluster \\        \r\n--zones eu-central-1a,eu-central-1b,eu-central-1c \\\r\n--master-zones eu-central-1a,eu-central-1b,eu-central-1c \\\r\n--networking weave \\\r\n--vpc ${My_Existing_VPC_ID} \\\r\n--node-count 3 \\\r\n--master-count 3 \\\r\n--node-size m4.large \\\r\n--master-size m4.large \\\r\n--ssh-public-key /path/to/my/public/key \\\r\nprotokube.test.k8s.local\r\n\r\n```\r\nThen restart journald service on a node/master:\r\n> systemctl restart systemd-journald \r\n\r\n5. What happened after the commands executed?\r\nProtokube.service tries to restart every second.\r\nIt fails since it is already running and the port is already used.\r\nEvery second multiple ec2-requests are made which causes followup issues like ec2 RequestLimitExceeded responses.\r\nThe cluster itself stays operational.\r\n\r\nExcerpt of journalctl\r\n```\r\nDec 28 11:54:43 ip-172-31-93-175 systemd[1]: Unit protokube.service entered failed state.\r\nDec 28 11:54:43 ip-172-31-93-175 systemd[1]: protokube.service: main process exited, code=exited, status=2/INVALIDARGUMENT\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: /go/src/k8s.io/kops/protokube/cmd/protokube/main.go:223 +0x23eb\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: created by main.run\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: /go/src/k8s.io/kops/protokube/cmd/protokube/main.go:217 +0x2f\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: main.run.func1(0xc4203e1110)\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: /go/src/k8s.io/kops/protokube/pkg/gossip/mesh/gossip.go:86 +0x45\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: k8s.io/kops/protokube/pkg/gossip/mesh.(*MeshGossiper).Start(0xc4203e1110, 0x0, 0x0)\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: /go/src/k8s.io/kops/vendor/github.com/weaveworks/mesh/router.go:84 +0x2b\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: k8s.io/kops/vendor/github.com/weaveworks/mesh.(*Router).Start(0xc420136000)\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: /go/src/k8s.io/kops/vendor/github.com/weaveworks/mesh/router.go:105 +0x188\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: k8s.io/kops/vendor/github.com/weaveworks/mesh.(*Router).listenTCP(0xc420136000)\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: goroutine 59 [running]:\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: panic: listen tcp4 0.0.0.0:3999: bind: address already in use\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: I1228 11:54:43.206721   16891 aws_volume.go:72] AWS API Request: ec2/DescribeInstances\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: I1228 11:54:43.203855   16891 aws_volume.go:72] AWS API Request: ec2metadata/GetMetadata\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: I1228 11:54:43.202792   16891 aws_volume.go:72] AWS API Request: ec2metadata/GetMetadata\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: I1228 11:54:43.201643   16891 aws_volume.go:72] AWS API Request: ec2metadata/GetMetadata\r\nDec 28 11:54:43 ip-172-31-93-175 docker[16865]: protokube version 0.1\r\nDec 28 11:54:42 ip-172-31-93-175 systemd[1]: Started Kubernetes Protokube Service.\r\nDec 28 11:54:42 ip-172-31-93-175 systemd[1]: Starting Kubernetes Protokube Service...\r\nDec 28 11:54:42 ip-172-31-93-175 systemd[1]: Stopping Kubernetes Protokube Service...\r\nDec 28 11:54:42 ip-172-31-93-175 systemd[1]: protokube.service holdoff time over, scheduling restart.\r\n```\r\n\r\n\r\n6. What did you expect to happen?\r\nProtokube should not be affected by journald restarts\r\n\r\n7. Please provide your cluster manifest.\r\n```\r\napiVersion: kops/v1alpha2\r\nkind: Cluster\r\nmetadata:\r\n  creationTimestamp: 2017-12-28T09:04:29Z\r\n  name: protokube.test.k8s.local\r\nspec:\r\n  api:\r\n    loadBalancer:\r\n      type: Public\r\n  authorization:\r\n    alwaysAllow: {}\r\n  channel: stable\r\n  cloudProvider: aws\r\n  configBase: s3://test.kops/protokube.test.k8s.local\r\n  etcdClusters:\r\n  - etcdMembers:\r\n    - instanceGroup: master-eu-central-1a\r\n      name: a\r\n    - instanceGroup: master-eu-central-1b\r\n      name: b\r\n    - instanceGroup: master-eu-central-1c\r\n      name: c\r\n    name: main\r\n  - etcdMembers:\r\n    - instanceGroup: master-eu-central-1a\r\n      name: a\r\n    - instanceGroup: master-eu-central-1b\r\n      name: b\r\n    - instanceGroup: master-eu-central-1c\r\n      name: c\r\n    name: events\r\n  iam:\r\n    allowContainerRegistry: true\r\n    legacy: false\r\n  kubernetesApiAccess:\r\n  - 0.0.0.0/0\r\n  kubernetesVersion: 1.8.4\r\n  masterInternalName: api.internal.protokube.test.k8s.local\r\n  masterPublicName: api.protokube.test.k8s.local\r\n  networkCIDR: 172.31.0.0/16\r\n  networkID: MY-VPC\r\n  networking:\r\n    weave:\r\n      mtu: 8912\r\n  nonMasqueradeCIDR: 100.64.0.0/10\r\n  sshAccess:\r\n  - 0.0.0.0/0\r\n  subnets:\r\n  - cidr: 172.31.91.0/24\r\n    name: eu-central-1a\r\n    type: Public\r\n    zone: eu-central-1a\r\n  - cidr: 172.31.92.0/24\r\n    name: eu-central-1b\r\n    type: Public\r\n    zone: eu-central-1b\r\n  - cidr: 172.31.93.0/24\r\n    name: eu-central-1c\r\n    type: Public\r\n    zone: eu-central-1c\r\n  topology:\r\n    dns:\r\n      type: Public\r\n    masters: public\r\n    nodes: public\r\n\r\n---\r\n\r\napiVersion: kops/v1alpha2\r\nkind: InstanceGroup\r\nmetadata:\r\n  creationTimestamp: 2017-12-28T09:04:29Z\r\n  labels:\r\n    kops.k8s.io/cluster: protokube.test.k8s.local\r\n  name: master-eu-central-1a\r\nspec:\r\n  image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02\r\n  machineType: m4.large\r\n  maxSize: 1\r\n  minSize: 1\r\n  nodeLabels:\r\n    kops.k8s.io/instancegroup: master-eu-central-1a\r\n  role: Master\r\n  subnets:\r\n  - eu-central-1a\r\n\r\n---\r\n\r\napiVersion: kops/v1alpha2\r\nkind: InstanceGroup\r\nmetadata:\r\n  creationTimestamp: 2017-12-28T09:04:30Z\r\n  labels:\r\n    kops.k8s.io/cluster: protokube.test.k8s.local\r\n  name: master-eu-central-1b\r\nspec:\r\n  image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02\r\n  machineType: m4.large\r\n  maxSize: 1\r\n  minSize: 1\r\n  nodeLabels:\r\n    kops.k8s.io/instancegroup: master-eu-central-1b\r\n  role: Master\r\n  subnets:\r\n  - eu-central-1b\r\n\r\n---\r\n\r\napiVersion: kops/v1alpha2\r\nkind: InstanceGroup\r\nmetadata:\r\n  creationTimestamp: 2017-12-28T09:04:30Z\r\n  labels:\r\n    kops.k8s.io/cluster: protokube.test.k8s.local\r\n  name: master-eu-central-1c\r\nspec:\r\n  image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02\r\n  machineType: m4.large\r\n  maxSize: 1\r\n  minSize: 1\r\n  nodeLabels:\r\n    kops.k8s.io/instancegroup: master-eu-central-1c\r\n  role: Master\r\n  subnets:\r\n  - eu-central-1c\r\n\r\n---\r\n\r\napiVersion: kops/v1alpha2\r\nkind: InstanceGroup\r\nmetadata:\r\n  creationTimestamp: 2017-12-28T09:04:30Z\r\n  labels:\r\n    kops.k8s.io/cluster: protokube.test.k8s.local\r\n  name: nodes\r\nspec:\r\n  image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02\r\n  machineType: m4.large\r\n  maxSize: 3\r\n  minSize: 3\r\n  nodeLabels:\r\n    kops.k8s.io/instancegroup: nodes\r\n  role: Node\r\n  subnets:\r\n  - eu-central-1a\r\n  - eu-central-1b\r\n  - eu-central-1c\r\n\r\n```\r\n\r\n8. Please run the commands with most verbose logging by adding the `-v 10` flag.\r\nN/A\r\n\r\n9. Anything else do we need to know?\r\nWe limit forwarding of journald for preventing full hard disk drives on the nodes/masters.\r\nTherefore, we restart journald once after updating its config.\r\n\r\nMy workaround: kill the existing protokube process and then the service can restart successfully:\r\n```\r\nfuser -k 3999/tcp\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kops/issues/comments/412959877","html_url":"https://github.com/kubernetes/kops/issues/4161#issuecomment-412959877","issue_url":"https://api.github.com/repos/kubernetes/kops/issues/4161","id":412959877,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMjk1OTg3Nw==","user":{"login":"justinsb","id":100893,"node_id":"MDQ6VXNlcjEwMDg5Mw==","avatar_url":"https://avatars2.githubusercontent.com/u/100893?v=4","gravatar_id":"","url":"https://api.github.com/users/justinsb","html_url":"https://github.com/justinsb","followers_url":"https://api.github.com/users/justinsb/followers","following_url":"https://api.github.com/users/justinsb/following{/other_user}","gists_url":"https://api.github.com/users/justinsb/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsb/subscriptions","organizations_url":"https://api.github.com/users/justinsb/orgs","repos_url":"https://api.github.com/users/justinsb/repos","events_url":"https://api.github.com/users/justinsb/events{/privacy}","received_events_url":"https://api.github.com/users/justinsb/received_events","type":"User","site_admin":false},"created_at":"2018-08-14T17:53:15Z","updated_at":"2018-08-14T17:53:15Z","author_association":"MEMBER","body":"I tried reproducing this with latest and wasn't able to - when I restarted systemd-journald protokube continued uninterrupted; I tried restarting the protokube service and it killed the docker container as well.\r\n\r\nI'll try again with kops 1.8, but going to mark this for the 1.11 milestone as it isn't obviously still happening."}}
{"actor":{"display_login":"justinsb"}}