{"action":"created","issue":{"url":"https://api.github.com/repos/moby/moby/issues/35508","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/35508/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/35508/comments","events_url":"https://api.github.com/repos/moby/moby/issues/35508/events","html_url":"https://github.com/moby/moby/issues/35508","id":274160910,"node_id":"MDU6SXNzdWUyNzQxNjA5MTA=","number":35508,"title":"Swarm service not registered in internal DNS","user":{"login":"kostrzewa9ld","id":29654749,"node_id":"MDQ6VXNlcjI5NjU0NzQ5","avatar_url":"https://avatars1.githubusercontent.com/u/29654749?v=4","gravatar_id":"","url":"https://api.github.com/users/kostrzewa9ld","html_url":"https://github.com/kostrzewa9ld","followers_url":"https://api.github.com/users/kostrzewa9ld/followers","following_url":"https://api.github.com/users/kostrzewa9ld/following{/other_user}","gists_url":"https://api.github.com/users/kostrzewa9ld/gists{/gist_id}","starred_url":"https://api.github.com/users/kostrzewa9ld/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kostrzewa9ld/subscriptions","organizations_url":"https://api.github.com/users/kostrzewa9ld/orgs","repos_url":"https://api.github.com/users/kostrzewa9ld/repos","events_url":"https://api.github.com/users/kostrzewa9ld/events{/privacy}","received_events_url":"https://api.github.com/users/kostrzewa9ld/received_events","type":"User","site_admin":false},"labels":[{"id":237472034,"node_id":"MDU6TGFiZWwyMzc0NzIwMzQ=","url":"https://api.github.com/repos/moby/moby/labels/area/networking","name":"area/networking","color":"fbca04","default":false},{"id":392678049,"node_id":"MDU6TGFiZWwzOTI2NzgwNDk=","url":"https://api.github.com/repos/moby/moby/labels/area/swarm","name":"area/swarm","color":"fbca04","default":false},{"id":616332869,"node_id":"MDU6TGFiZWw2MTYzMzI4Njk=","url":"https://api.github.com/repos/moby/moby/labels/version/17.06","name":"version/17.06","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2017-11-15T13:44:17Z","updated_at":"2018-11-27T16:03:38Z","closed_at":null,"author_association":"NONE","body":"I have a docker swarm setup on 4 manager nodes (not 3 by accident). At some point one of the nodes got out of space on root partition. After a while (around 2 hours) swarm run leader election and all the services were restarted. However, restart of the services didn't go as expected. For example, some services stabilized after around 24 hours(!) and some didn't stabilize till now and in a result they are no longer resolvable by its name (dns query go to our external DNS).\r\n\r\n**Steps to reproduce the issue:**\r\nI don't have exact reproduction steps, but the services restart might have something to do with one of the nodes getting out of space and election of a new leader.\r\n\r\n**Describe the results you received:**\r\nSome services are not resolvable by their name.\r\n\r\n**Describe the results you expected:**\r\nWhen service is restarted it should be properly registered in swarm.\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\nServices were deployed from docker compose files using `docker stack deploy` so the issue might be to some extent related to #34904.\r\n\r\n**Logs**\r\nLogs related to the service that didn't stabilize at all (i.e. wasn't registered in swarm's internal DNS).\r\n```\r\nNov 12 22:05:53 node-10-9-4-103 dockerd: time=\"2017-11-12T22:05:53.729702654+01:00\" level=debug msg=\"Name To resolve: sa_openldap_replica.\"\r\nNov 12 22:05:53 node-10-9-4-103 dockerd: time=\"2017-11-12T22:05:53.729821797+01:00\" level=debug msg=\"Lookup for sa_openldap_replica.: IP [10.0.0.22]\"\r\nNov 12 22:05:53 node-10-9-4-103 dockerd: time=\"2017-11-12T22:05:53.729702787+01:00\" level=debug msg=\"Name To resolve: sa_openldap_replica.\"\r\nNov 12 22:05:53 node-10-9-4-103 dockerd: time=\"2017-11-12T22:05:53.729920814+01:00\" level=debug msg=\"Lookup name sa_openldap_replica. present without IPv6 address\"\r\nNov 12 22:06:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:00.680899703+01:00\" level=debug msg=\"Name To resolve: sa_openldap_replica.\"\r\nNov 12 22:06:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:00.681042598+01:00\" level=debug msg=\"Lookup name sa_openldap_replica. present without IPv6 address\"\r\nNov 12 22:06:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:00.684198454+01:00\" level=debug msg=\"Name To resolve: sa_openldap_replica.\"\r\nNov 12 22:06:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:00.684247545+01:00\" level=debug msg=\"Lookup for sa_openldap_replica.: IP [10.0.0.22]\"\r\nNov 12 22:06:11 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:11.735412966+01:00\" level=debug msg=\"deleteServiceInfoFromCluster from sbLeave START for sa_openldap_replica 426a580e53fd721d410d3a2b087bf8cb998b9ecb6a0cd8da5c8b00f87cb93a90\"\r\nNov 12 22:06:11 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:11.735571015+01:00\" level=debug msg=\"rmServiceBinding from deleteServiceInfoFromCluster START for sa_openldap_replica 426a580e53fd721d410d3a2b087bf8cb998b9ecb6a0cd8da5c8b00f87\r\ncb93a90 p:0xc4219f5480 nid:ai6hs0uuezm9hlqpi44aw0w22 sKey:{ljrvybzqa6b5z2m67643einto } deleteSvc:true\"\r\nNov 12 22:06:12 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:12.000830296+01:00\" level=debug msg=\"deleteEndpointNameResolution 426a580e53fd721d410d3a2b087bf8cb998b9ecb6a0cd8da5c8b00f87cb93a90 sa_openldap_replica rm_service:true suppress\r\n:false sAliases:[openldap_replica] tAliases:[1951c4f0d3ce]\"\r\nNov 12 22:06:12 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:12.000902728+01:00\" level=debug msg=\"delContainerNameResolution 426a580e53fd721d410d3a2b087bf8cb998b9ecb6a0cd8da5c8b00f87cb93a90 sa_openldap_replica.1.kq869yybtx4bq9437m5cqh57\r\nf\"\r\nNov 12 22:06:12 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:12.000931567+01:00\" level=debug msg=\"426a580e53fd721d410d3a2b087bf8cb998b9ecb6a0cd8da5c8b00f87cb93a90 (ai6hs0u).deleteSvcRecords(sa_openldap_replica.1.kq869yybtx4bq9437m5cqh57\r\nf, 10.0.0.23, <nil>, true) rmServiceBinding sid:426a580e53fd721d410d3a2b087bf8cb998b9ecb6a0cd8da5c8b00f87cb93a90 \"\r\nNov 12 22:06:12 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:12.001064630+01:00\" level=debug msg=\"426a580e53fd721d410d3a2b087bf8cb998b9ecb6a0cd8da5c8b00f87cb93a90 (ai6hs0u).deleteSvcRecords(tasks.sa_openldap_replica, 10.0.0.23, <nil>, f\r\nalse) rmServiceBinding sid:ljrvybzqa6b5z2m67643einto \"\r\nNov 12 22:06:12 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:12.001128031+01:00\" level=debug msg=\"426a580e53fd721d410d3a2b087bf8cb998b9ecb6a0cd8da5c8b00f87cb93a90 (ai6hs0u).deleteSvcRecords(sa_openldap_replica, 10.0.0.22, <nil>, false) \r\nrmServiceBinding sid:ljrvybzqa6b5z2m67643einto \"\r\nNov 12 22:06:12 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:12.001178124+01:00\" level=debug msg=\"rmServiceBinding from deleteServiceInfoFromCluster END for sa_openldap_replica 426a580e53fd721d410d3a2b087bf8cb998b9ecb6a0cd8da5c8b00f87cb\r\n93a90\"\r\nNov 12 22:06:12 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:12.001199826+01:00\" level=debug msg=\"deleteServiceInfoFromCluster from sbLeave END for sa_openldap_replica 426a580e53fd721d410d3a2b087bf8cb998b9ecb6a0cd8da5c8b00f87cb93a90\"\r\nNov 12 22:06:17 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:17.878801373+01:00\" level=debug msg=\"Releasing addresses for endpoint sa_openldap_replica.1.kq869yybtx4bq9437m5cqh57f's interface on network sa\"\r\nNov 12 22:07:33 node-10-9-4-103 dockerd: time=\"2017-11-12T22:07:33.407972442+01:00\" level=debug msg=\"copying image data from 3d18bae8e135b131302e67e6b723664c71e69a49a403516800ac298bd4e7dd2b:/var/lib/ldap-single, to sa_openldap_replica\"\r\nNov 12 22:07:33 node-10-9-4-103 dockerd: time=\"2017-11-12T22:07:33.526664394+01:00\" level=debug msg=\"Creating service config in agent for t = &Task{ID:xxnkynuwa0xroymvzjcqsr2wc,Meta:Meta{Version:Version{Index:596,},CreatedAt:2017-11-12T21:\r\n05:53.981511233Z,UpdatedAt:2017-11-12T21:05:54.137337895Z,},Spec:TaskSpec{Runtime:&TaskSpec_Container{Container:&ContainerSpec{Image:ramencloud/fusiondirectory-openldap:0.3.0@sha256:141a737f228a67aee03b8ac3fb2e3d7e2fb2e8596e9cdbc0ee60a26ab\r\n5ea8d25,Labels:map[string]string{com.docker.stack.namespace: sa,},Command:[],Args:[],Env:[<CAN'T_SHARE_THESE>],Dir:,User:,Mounts:[{VOLUME sa_openld\r\nap_replica /var/lib/ldap-single false nil Mount_VolumeOptions{NoCopy:false,Labels:map[string]string{com.docker.stack.namespace: sa,},DriverConfig:&Driver{Name:local,Options:map[string]string{},},} nil}],StopGracePeriod:(duration: nil googl\r\ne_protobuf1.Duration),PullOptions:nil,Groups:[],Secrets:[],TTY:false,Hostname:,DNSConfig:nil,Healthcheck:&HealthConfig{Test:[CMD-SHELL ldapsearch -Q -Y EXTERNAL -H ldapi:/// -b \\\"cn=config\\\" \\\"cn=config\\\"],Interval:10s,Timeout:5s,Retries:3\r\n0,StartPeriod:0s,},Hosts:[],OpenStdin:false,ReadOnly:false,StopSignal:,Configs:[],Privileges:&Privileges{CredentialSpec:nil,SELinuxContext:nil,},},},Resources:&ResourceRequirements{Limits:nil,Reservations:nil,},Restart:nil,Placement:&Place\r\nment{Constraints:[node.labels.openldap.role == replica],Preferences:[],Platforms:[&Platform{Architecture:amd64,OS:linux,}],},LogDriver:nil,Networks:[&NetworkAttachmentConfig{Target:ai6hs0uuezm9hlqpi44aw0w22,Aliases:[openldap_replica],Addre\r\nsses:[],DriverAttachmentOpts:map[string]string{},}],ForceUpdate:0,},ServiceID:ljrvybzqa6b5z2m67643einto,Slot:1,NodeID:jvxg46j8bcwa8o5qa96psner9,Annotations:Annotations{Name:,Lab\r\nNov 12 22:07:33 node-10-9-4-103 dockerd: els:map[string]string{},Indices:[],},ServiceAnnotations:Annotations{Name:sa_openldap_replica,Labels:map[string]string{com.docker.stack.image: ramencloud/fusiondirectory-openldap:0.3.0,com.docker.sta\r\nck.namespace: sa,},Indices:[],},Status:TaskStatus{Timestamp:2017-11-12T21:05:54.136957131Z,State:ASSIGNED,Message:scheduler assigned task to node,Err:,RuntimeStatus:<nil>,PortStatus:nil,},DesiredState:READY,Networks:[&NetworkAttachment{Net\r\nwork:&Network{ID:ai6hs0uuezm9hlqpi44aw0w22,Meta:Meta{Version:Version{Index:582,},CreatedAt:2017-10-30T15:01:11.480607346Z,UpdatedAt:2017-11-12T21:05:45.11102014Z,},Spec:NetworkSpec{Annotations:Annotations{Name:sa,Labels:map[string]string{}\r\n,Indices:[],},DriverConfig:&Driver{Name:overlay,Options:map[string]string{},},Ipv6Enabled:false,Internal:false,IPAM:nil,Attachable:true,Ingress:false,ConfigFrom:,},DriverState:&Driver{Name:overlay,Options:map[string]string{com.docker.netwo\r\nrk.driver.overlay.vxlanid_list: 4097,},},IPAM:&IPAMOptions{Driver:&Driver{Name:default,Options:map[string]string{},},Configs:[&IPAMConfig{Family:IPV4,Subnet:10.0.0.0/24,Range:,Gateway:10.0.0.1,Reserved:map[string]string{},}],},},Addresses:\r\n[10.0.0.35/24],Aliases:[openldap_replica],DriverAttachmentOpts:map[string]string{},}],Endpoint:&Endpoint{Spec:&EndpointSpec{Mode:VIP,Ports:[],},Ports:[],VirtualIPs:[&Endpoint_VirtualIP{NetworkID:ai6hs0uuezm9hlqpi44aw0w22,Addr:10.0.0.22/24,\r\n}],},LogDriver:nil,SpecVersion:&Version{Index:0,},}\"\r\nNov 12 22:07:33 node-10-9-4-103 dockerd: time=\"2017-11-12T22:07:33.594981939+01:00\" level=debug msg=\"Assigning addresses for endpoint sa_openldap_replica.1.xxnkynuwa0xroymvzjcqsr2wc's interface on network sa\"\r\nNov 12 22:07:33 node-10-9-4-103 dockerd: time=\"2017-11-12T22:07:33.597121564+01:00\" level=debug msg=\"Assigning addresses for endpoint sa_openldap_replica.1.xxnkynuwa0xroymvzjcqsr2wc's interface on network sa\"\r\nNov 12 22:10:28 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:28.709341955+01:00\" level=debug msg=\"Name To resolve: sa_openldap_replica.\"\r\nNov 12 22:10:28 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:28.709474549+01:00\" level=debug msg=\"Name To resolve: sa_openldap_replica.\"\r\nNov 12 22:10:28 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:28.726655795+01:00\" level=debug msg=\"Query sa_openldap_replica.[1] from 172.18.0.5:59602, forwarding to udp:10.9.4.4\"\r\nNov 12 22:10:28 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:28.732668381+01:00\" level=debug msg=\"Query sa_openldap_replica.[28] from 172.18.0.5:40307, forwarding to udp:10.9.4.4\"\r\nNov 12 22:10:28 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:28.774710890+01:00\" level=debug msg=\"Name To resolve: sa_openldap_replica.bdcluster.\"\r\nNov 12 22:10:28 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:28.774792259+01:00\" level=debug msg=\"Name To resolve: sa_openldap_replica.bdcluster.\"\r\n```\r\nThe last messages - \"Query\" and \"Name To resolve\" - keep repeating until now.\r\n\r\nLogs related to service that was restarted after around 24hours:\r\n```\r\nNov 12 22:06:17 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:17.398920379+01:00\" level=debug msg=\"deleteServiceInfoFromCluster from sbLeave START for sa_postgres_replica b2cfde017504b180d1bc02a1d90725451b48efaaa3185eec4bc70a3ece61c582\"\r\nNov 12 22:06:17 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:17.399015314+01:00\" level=debug msg=\"rmServiceBinding from deleteServiceInfoFromCluster START for sa_postgres_replica b2cfde017504b180d1bc02a1d90725451b48efaaa3185eec4bc70a3ece61c582 p:0xc420389780 nid:ai6hs0uuezm9hlqpi44aw0w22 sKey:{lgczmoowxriiseuxksl7fibzw } deleteSvc:true\"\r\nNov 12 22:06:19 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:19.538822724+01:00\" level=debug msg=\"deleteEndpointNameResolution b2cfde017504b180d1bc02a1d90725451b48efaaa3185eec4bc70a3ece61c582 sa_postgres_replica rm_service:true suppress:false sAliases:[postgres_replica] tAliases:[f959938f9f67]\"\r\nNov 12 22:06:19 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:19.538877237+01:00\" level=debug msg=\"delContainerNameResolution b2cfde017504b180d1bc02a1d90725451b48efaaa3185eec4bc70a3ece61c582 sa_postgres_replica.1.xi6oi1yy6kfp5nkqdfit3ypzd\"\r\nNov 12 22:06:19 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:19.538904963+01:00\" level=debug msg=\"b2cfde017504b180d1bc02a1d90725451b48efaaa3185eec4bc70a3ece61c582 (ai6hs0u).deleteSvcRecords(sa_postgres_replica.1.xi6oi1yy6kfp5nkqdfit3ypzd, 10.0.0.15, <nil>, true) rmServiceBinding sid:b2cfde017504b180d1bc02a1d90725451b48efaaa3185eec4bc70a3ece61c582 \"\r\nNov 12 22:06:19 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:19.539002002+01:00\" level=debug msg=\"b2cfde017504b180d1bc02a1d90725451b48efaaa3185eec4bc70a3ece61c582 (ai6hs0u).deleteSvcRecords(tasks.sa_postgres_replica, 10.0.0.15, <nil>, false) rmServiceBinding sid:lgczmoowxriiseuxksl7fibzw \"\r\nNov 12 22:06:19 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:19.539090778+01:00\" level=debug msg=\"b2cfde017504b180d1bc02a1d90725451b48efaaa3185eec4bc70a3ece61c582 (ai6hs0u).deleteSvcRecords(sa_postgres_replica, 10.0.0.18, <nil>, false) rmServiceBinding sid:lgczmoowxriiseuxksl7fibzw \"\r\nNov 12 22:06:19 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:19.539148875+01:00\" level=debug msg=\"rmServiceBinding from deleteServiceInfoFromCluster END for sa_postgres_replica b2cfde017504b180d1bc02a1d90725451b48efaaa3185eec4bc70a3ece61c582\"\r\nNov 12 22:06:19 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:19.539184113+01:00\" level=debug msg=\"deleteServiceInfoFromCluster from sbLeave END for sa_postgres_replica b2cfde017504b180d1bc02a1d90725451b48efaaa3185eec4bc70a3ece61c582\"\r\nNov 12 22:06:49 node-10-9-4-103 dockerd: time=\"2017-11-12T22:06:49.609746534+01:00\" level=debug msg=\"Releasing addresses for endpoint sa_postgres_replica.1.xi6oi1yy6kfp5nkqdfit3ypzd's interface on network sa\"\r\nNov 12 22:07:22 node-10-9-4-103 dockerd: time=\"2017-11-12T22:07:22.540628448+01:00\" level=debug msg=\"copying image data from 09635baa4d866d550c70c86ee1299b3abcfe93f0aa1f716040c0915711254437:/pgdata, to sa_postgres_replica\"\r\nNov 12 22:07:32 node-10-9-4-103 dockerd: time=\"2017-11-12T22:07:32.893196027+01:00\" level=debug msg=\"Creating service config in agent for t = &Task{ID:ho1w9rwm6792furoa7y8fzrwk,Meta:Meta{Version:Version{Index:596,},CreatedAt:2017-11-12T21:05:53.983038323Z,UpdatedAt:2017-11-12T21:05:54.137140678Z,},Spec:TaskSpec{Runtime:&TaskSpec_Container{Container:&ContainerSpec{Image:ramencloud/crunchy-postgres:centos7-9.6-1.5.1@sha256:073e420d5d8c41fb17487464adc5f1890a2997062c75f84ca81d5573449b86f7,Labels:map[string]string{com.docker.stack.namespace: sa,},Command:[],Args:[],Env:[<CAN'T_SHARE_THESE>],Dir:,User:,Mounts:[{VOLUME sa_postgres_replica /pgdata false nil Mount_VolumeOptions{NoCopy:false,Labels:map[string]string{com.docker.stack.namespace: sa,},DriverConfig:&Driver{Name:local,Options:map[string]string{},},} nil}],StopGracePeriod:(duration: nil google_protobuf1.Duration),PullOptions:nil,Groups:[],Secrets:[],TTY:false,Hostname:,DNSConfig:nil,Healthcheck:&HealthConfig{Test:[CMD-SHELL /usr/pgsql-9.6/bin/pg_isready --host localhost],Interval:10s,Timeout:5s,Retries:30,StartPeriod:0s,},Hosts:[],OpenStdin:false,ReadOnly:false,StopSignal:,Configs:[],Privileges:&Privileges{CredentialSpec:nil,SELinuxContext:nil,},},},Resources:&ResourceRequirements{Limits:nil,Reservations:nil,},Restart:nil,Placement:&Placement{Constraints:[node.labels.postgres.role == replica],Preferences:[],Platforms:[&Platform{Architecture:amd64,OS:linux,}],},LogDriver:nil,Networks:[&NetworkAttachmentConfig{Target:ai6hs0uuezm9hlqpi44aw0w22,Aliases:[postgres_replica],Addresses:[],DriverAttachmentOpts:map[string]string{},}],ForceUpdate:0,},ServiceID:lgczmoowxriiseuxksl7fibzw,Slot:1,NodeID:jvxg46j8bcwa8o5qa96psner9,Annotations:Annotations{Name:,Labels:map[string]string{},Indices:[],},ServiceAnnotations:Annotations{Name:sa_postgres_replica,Labels:map[string]string{com.docke\r\nNov 12 22:07:33 node-10-9-4-103 dockerd: time=\"2017-11-12T22:07:33.350005291+01:00\" level=debug msg=\"Assigning addresses for endpoint sa_postgres_replica.1.ho1w9rwm6792furoa7y8fzrwk's interface on network sa\"\r\nNov 12 22:07:33 node-10-9-4-103 dockerd: time=\"2017-11-12T22:07:33.352217431+01:00\" level=debug msg=\"Assigning addresses for endpoint sa_postgres_replica.1.ho1w9rwm6792furoa7y8fzrwk's interface on network sa\"\r\nNov 12 22:09:48 node-10-9-4-103 dockerd: time=\"2017-11-12T22:09:48.340170229+01:00\" level=debug msg=\"Name To resolve: sa_postgres.\"\r\nNov 12 22:09:48 node-10-9-4-103 dockerd: time=\"2017-11-12T22:09:48.340336962+01:00\" level=debug msg=\"Lookup name sa_postgres. present without IPv6 address\"\r\nNov 12 22:09:48 node-10-9-4-103 dockerd: time=\"2017-11-12T22:09:48.340458050+01:00\" level=debug msg=\"Name To resolve: sa_postgres.\"\r\nNov 12 22:09:48 node-10-9-4-103 dockerd: time=\"2017-11-12T22:09:48.340564002+01:00\" level=debug msg=\"Lookup for sa_postgres.: IP [10.0.0.16]\"\r\nNov 12 22:10:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:00.653118678+01:00\" level=debug msg=\"handleEpTableEvent DEL %!s(bool=false) R:52ef29d102a59bdec28785368a7d0d8a78f2b962a8a16d5bd7223466e87b55a8%!(EXTRA libnetwork.EndpointRecord={sa_postgres.1.hwoulyjm32wb3cngpk0dhk1h2 sa_postgres 5zdfyhotm7si0tlpvwwjjrys9 10.0.0.16 10.0.0.29 [] [postgres] [6f0304e02db9]})\"\r\nNov 12 22:10:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:00.653215576+01:00\" level=debug msg=\"rmServiceBinding from handleEpTableEvent START for sa_postgres 52ef29d102a59bdec28785368a7d0d8a78f2b962a8a16d5bd7223466e87b55a8 p:0xc420c92000 nid:ai6hs0uuezm9hlqpi44aw0w22 sKey:{5zdfyhotm7si0tlpvwwjjrys9 } deleteSvc:true\"\r\nNov 12 22:10:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:00.862800775+01:00\" level=debug msg=\"deleteEndpointNameResolution 52ef29d102a59bdec28785368a7d0d8a78f2b962a8a16d5bd7223466e87b55a8 sa_postgres rm_service:true suppress:false sAliases:[postgres] tAliases:[6f0304e02db9]\"\r\nNov 12 22:10:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:00.862854961+01:00\" level=debug msg=\"delContainerNameResolution 52ef29d102a59bdec28785368a7d0d8a78f2b962a8a16d5bd7223466e87b55a8 sa_postgres.1.hwoulyjm32wb3cngpk0dhk1h2\"\r\nNov 12 22:10:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:00.862881831+01:00\" level=debug msg=\"52ef29d102a59bdec28785368a7d0d8a78f2b962a8a16d5bd7223466e87b55a8 (ai6hs0u).deleteSvcRecords(sa_postgres.1.hwoulyjm32wb3cngpk0dhk1h2, 10.0.0.29, <nil>, true) rmServiceBinding sid:52ef29d102a59bdec28785368a7d0d8a78f2b962a8a16d5bd7223466e87b55a8 \"\r\nNov 12 22:10:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:00.862972641+01:00\" level=debug msg=\"52ef29d102a59bdec28785368a7d0d8a78f2b962a8a16d5bd7223466e87b55a8 (ai6hs0u).deleteSvcRecords(tasks.sa_postgres, 10.0.0.29, <nil>, false) rmServiceBinding sid:5zdfyhotm7si0tlpvwwjjrys9 \"\r\nNov 12 22:10:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:00.863039747+01:00\" level=debug msg=\"52ef29d102a59bdec28785368a7d0d8a78f2b962a8a16d5bd7223466e87b55a8 (ai6hs0u).deleteSvcRecords(sa_postgres, 10.0.0.16, <nil>, false) rmServiceBinding sid:5zdfyhotm7si0tlpvwwjjrys9 \"\r\nNov 12 22:10:00 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:00.863103500+01:00\" level=debug msg=\"rmServiceBinding from handleEpTableEvent END for sa_postgres 52ef29d102a59bdec28785368a7d0d8a78f2b962a8a16d5bd7223466e87b55a8\"\r\nNov 12 22:10:02 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:02.263681274+01:00\" level=debug msg=\"Name To resolve: sa_postgres.\"\r\nNov 12 22:10:02 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:02.263882687+01:00\" level=debug msg=\"Name To resolve: sa_postgres.\"\r\nNov 12 22:10:02 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:02.277630397+01:00\" level=debug msg=\"Query sa_postgres.[28] from 172.18.0.4:55359, forwarding to udp:10.9.4.4\"\r\nNov 12 22:10:02 node-10-9-4-103 dockerd: time=\"2017-11-12T22:10:02.277887855+01:00\" level=debug msg=\"Query sa_postgres.[1] from 172.18.0.4:34120, forwarding to udp:10.9.4.4\"\r\n...\r\nNov 13 21:52:50 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:50.349596635+01:00\" level=debug msg=\"Query sa_postgres.bdcluster.[1] from 172.18.0.4:39387, forwarding to udp:10.9.4.4\"\r\nNov 13 21:52:50 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:50.349946529+01:00\" level=debug msg=\"Query sa_postgres.bdcluster.[28] from 172.18.0.4:35307, forwarding to udp:10.9.4.4\"\r\nNov 13 21:52:51 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:51.452864532+01:00\" level=debug msg=\"handleEpTableEvent ADD %!s(bool=true) R:74d31fc4bbaeca6627dbd4b0037b34dc4635b037ff929b2c60c82c5c9780b536%!(EXTRA libnetwork.EndpointRecord={sa_postgres.1.b7u81fw6d3fykjy48p72zuec3 sa_postgres 5zdfyhotm7si0tlpvwwjjrys9 10.0.0.16 10.0.0.23 [] [postgres] [eaa9a1981266]})\"\r\nNov 13 21:52:51 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:51.452984154+01:00\" level=debug msg=\"addServiceBinding from handleEpTableEvent START for sa_postgres 74d31fc4bbaeca6627dbd4b0037b34dc4635b037ff929b2c60c82c5c9780b536 p:0xc4226c3480 nid:ai6hs0uuezm9hlqpi44aw0w22 skey:{5zdfyhotm7si0tlpvwwjjrys9 }\"\r\nNov 13 21:52:51 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:51.877031972+01:00\" level=debug msg=\"addEndpointNameResolution 74d31fc4bbaeca6627dbd4b0037b34dc4635b037ff929b2c60c82c5c9780b536 sa_postgres add_service:true sAliases:[postgres] tAliases:[eaa9a1981266]\"\r\nNov 13 21:52:51 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:51.877122380+01:00\" level=debug msg=\"addContainerNameResolution 74d31fc4bbaeca6627dbd4b0037b34dc4635b037ff929b2c60c82c5c9780b536 sa_postgres.1.b7u81fw6d3fykjy48p72zuec3\"\r\nNov 13 21:52:51 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:51.877173173+01:00\" level=debug msg=\"74d31fc4bbaeca6627dbd4b0037b34dc4635b037ff929b2c60c82c5c9780b536 (ai6hs0u).addSvcRecords(sa_postgres.1.b7u81fw6d3fykjy48p72zuec3, 10.0.0.23, <nil>, true) addServiceBinding sid:74d31fc4bbaeca6627dbd4b0037b34dc4635b037ff929b2c60c82c5c9780b536\"\r\nNov 13 21:52:51 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:51.877292066+01:00\" level=debug msg=\"74d31fc4bbaeca6627dbd4b0037b34dc4635b037ff929b2c60c82c5c9780b536 (ai6hs0u).addSvcRecords(tasks.sa_postgres, 10.0.0.23, <nil>, false) addServiceBinding sid:5zdfyhotm7si0tlpvwwjjrys9\"\r\nNov 13 21:52:51 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:51.877376445+01:00\" level=debug msg=\"74d31fc4bbaeca6627dbd4b0037b34dc4635b037ff929b2c60c82c5c9780b536 (ai6hs0u).addSvcRecords(sa_postgres, 10.0.0.16, <nil>, false) addServiceBinding sid:5zdfyhotm7si0tlpvwwjjrys9\"\r\nNov 13 21:52:51 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:51.877441389+01:00\" level=debug msg=\"addServiceBinding from handleEpTableEvent END for sa_postgres 74d31fc4bbaeca6627dbd4b0037b34dc4635b037ff929b2c60c82c5c9780b536\"\r\nNov 13 21:52:55 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:55.308085731+01:00\" level=debug msg=\"Name To resolve: sa_postgres.\"\r\nNov 13 21:52:55 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:55.308225796+01:00\" level=debug msg=\"Lookup for sa_postgres.: IP [10.0.0.16]\"\r\nNov 13 21:52:55 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:55.308086099+01:00\" level=debug msg=\"Name To resolve: sa_postgres.\"\r\nNov 13 21:52:55 node-10-9-4-103 dockerd: time=\"2017-11-13T21:52:55.308300136+01:00\" level=debug msg=\"Lookup name sa_postgres. present without IPv6 address\"\r\n```\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      17.06.2-ce\r\n API version:  1.30\r\n Go version:   go1.8.3\r\n Git commit:   cec0b72\r\n Built:        Tue Sep  5 19:59:06 2017\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      17.06.2-ce\r\n API version:  1.30 (minimum version 1.12)\r\n Go version:   go1.8.3\r\n Git commit:   cec0b72\r\n Built:        Tue Sep  5 20:00:25 2017\r\n OS/Arch:      linux/amd64\r\n Experimental: false\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 15\r\n Running: 6\r\n Paused: 0\r\n Stopped: 9\r\nImages: 14\r\nServer Version: 17.06.2-ce\r\nStorage Driver: devicemapper\r\n Pool Name: vg00-docker--pool\r\n Pool Blocksize: 524.3kB\r\n Base Device Size: 10.74GB\r\n Backing Filesystem: xfs\r\n Data file: \r\n Metadata file: \r\n Data Space Used: 6.452GB\r\n Data Space Total: 21.47GB\r\n Data Space Available: 15.02GB\r\n Metadata Space Used: 1.892MB\r\n Metadata Space Total: 100.7MB\r\n Metadata Space Available: 98.77MB\r\n Thin Pool Minimum Free Space: 2.147GB\r\n Udev Sync Supported: true\r\n Deferred Removal Enabled: true\r\n Deferred Deletion Enabled: true\r\n Deferred Deleted Device Count: 0\r\n Library Version: 1.02.107-RHEL7 (2015-10-14)\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins: \r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\n Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r\nSwarm: active\r\n NodeID: jvxg46j8bcwa8o5qa96psner9\r\n Is Manager: true\r\n ClusterID: et758znsr7ey9izjhlujntroz\r\n Managers: 4\r\n Nodes: 4\r\n Orchestration:\r\n  Task History Retention Limit: 5\r\n Raft:\r\n  Snapshot Interval: 10000\r\n  Number of Old Snapshots to Retain: 0\r\n  Heartbeat Tick: 1\r\n  Election Tick: 3\r\n Dispatcher:\r\n  Heartbeat Period: 5 seconds\r\n CA Configuration:\r\n  Expiry Duration: 3 months\r\n  Force Rotate: 0\r\n Root Rotation In Progress: false\r\n Node Address: 10.9.4.103\r\n Manager Addresses:\r\n  10.9.4.103:2377\r\n  10.9.4.117:2377\r\n  10.9.4.122:2377\r\n  10.9.4.212:2377\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170\r\nrunc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2\r\ninit version: 949e6fa\r\nSecurity Options:\r\n seccomp\r\n  Profile: default\r\nKernel Version: 3.10.0-327.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 4\r\nTotal Memory: 7.64GiB\r\nName: node-10-9-4-103\r\nID: GC43:XBDK:UUWT:QR3N:S7LB:IHDJ:TETW:2DW5:64UM:G7Q5:GRFJ:OXSO\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): true\r\n File Descriptors: 117\r\n Goroutines: 216\r\n System Time: 2017-11-15T12:58:17.016590262+01:00\r\n EventsListeners: 8\r\nRegistry: https://index.docker.io/v1/\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\nSwarm is setup on 4 manager nodes on VMs with CentOS 7.2 on KVM."},"comment":{"url":"https://api.github.com/repos/moby/moby/issues/comments/442114028","html_url":"https://github.com/moby/moby/issues/35508#issuecomment-442114028","issue_url":"https://api.github.com/repos/moby/moby/issues/35508","id":442114028,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MjExNDAyOA==","user":{"login":"kevinbrink","id":7232400,"node_id":"MDQ6VXNlcjcyMzI0MDA=","avatar_url":"https://avatars2.githubusercontent.com/u/7232400?v=4","gravatar_id":"","url":"https://api.github.com/users/kevinbrink","html_url":"https://github.com/kevinbrink","followers_url":"https://api.github.com/users/kevinbrink/followers","following_url":"https://api.github.com/users/kevinbrink/following{/other_user}","gists_url":"https://api.github.com/users/kevinbrink/gists{/gist_id}","starred_url":"https://api.github.com/users/kevinbrink/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kevinbrink/subscriptions","organizations_url":"https://api.github.com/users/kevinbrink/orgs","repos_url":"https://api.github.com/users/kevinbrink/repos","events_url":"https://api.github.com/users/kevinbrink/events{/privacy}","received_events_url":"https://api.github.com/users/kevinbrink/received_events","type":"User","site_admin":false},"created_at":"2018-11-27T16:03:38Z","updated_at":"2018-11-27T16:03:38Z","author_association":"NONE","body":"@thaJeztah : Thanks for the response.\r\n\r\nThe comparison I was drawing was between two instances of the same exact service. That service does not have any sort of health check, so I don't think that's a factor. \r\n\r\nOur infrastructure is such that updating docker can't happen as often as we'd like. We are preparing to update to `18.06.1`, but I don't have a ton of confidence this particular issue has been mitigated by that upgrade. "}}
{"actor":{"display_login":"kevinbrink"}}