{"action":"opened","issue":{"url":"https://api.github.com/repos/projectcalico/node/issues/9","repository_url":"https://api.github.com/repos/projectcalico/node","labels_url":"https://api.github.com/repos/projectcalico/node/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/projectcalico/node/issues/9/comments","events_url":"https://api.github.com/repos/projectcalico/node/issues/9/events","html_url":"https://github.com/projectcalico/node/issues/9","id":326546656,"number":9,"title":"allocate-ipip-addr can race","user":{"login":"ti-mo","id":3214460,"avatar_url":"https://avatars0.githubusercontent.com/u/3214460?v=4","gravatar_id":"","url":"https://api.github.com/users/ti-mo","html_url":"https://github.com/ti-mo","followers_url":"https://api.github.com/users/ti-mo/followers","following_url":"https://api.github.com/users/ti-mo/following{/other_user}","gists_url":"https://api.github.com/users/ti-mo/gists{/gist_id}","starred_url":"https://api.github.com/users/ti-mo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ti-mo/subscriptions","organizations_url":"https://api.github.com/users/ti-mo/orgs","repos_url":"https://api.github.com/users/ti-mo/repos","events_url":"https://api.github.com/users/ti-mo/events{/privacy}","received_events_url":"https://api.github.com/users/ti-mo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-25T14:28:23Z","updated_at":"2018-05-25T14:28:23Z","closed_at":null,"author_association":"NONE","body":"<!--- Provide a general summary of the issue in the Title above -->\r\n\r\nMentioned in https://github.com/projectcalico/libcalico-go/pull/785.\r\n\r\n## Expected Behavior\r\n<!--- If you're describing a bug, tell us what should happen -->\r\n<!--- If you're suggesting a change/improvement, tell us how it should work -->\r\n\r\nDeclaring an IP pool on an existing deployment should allocate a block to every cluster member.\r\n\r\n## Current Behavior\r\n<!--- If describing a bug, tell us what happens instead of the expected behavior -->\r\n<!--- If suggesting a change/improvement, explain the difference from current behavior -->\r\n```\r\n2018-04-27T08:49:12Z public-slave-14 confd[130]: ERROR \"\r\n2018-04-27 08:49:11.878 [INFO][277] client.go 202: Loading config from environment\r\n2018-04-27 08:49:11.895 [INFO][277] ipam.go 120: Auto-assign 1 ipv4, 0 ipv6 addrs for host 'public-slave-14'\r\n2018-04-27 08:49:11.903 [INFO][277] ipam.go 172: Ran out of existing affine blocks for host 'public-slave-14'\r\n2018-04-27 08:49:11.909 [INFO][277] ipam.go 195: Need to allocate 1 more addresses - allocate another block\r\n2018-04-27 08:49:11.932 [INFO][277] ipam_block_reader_writer.go 116: Claiming a new affine block for host 'public-slave-14'\r\n2018-04-27 08:49:11.944 [INFO][277] ipam_block_reader_writer.go 159: Host public-slave-14 claiming block affinity for 10.123.84.64/26\r\n2018-04-27 08:49:11.994 [WARNING][277] ipam_block_reader_writer.go 187: Problem claiming block affinity:%!!(MISSING)(EXTRA errors.ErrorResourceAlreadyExists=resource already exists: BlockKey(cidr=10.123.84.64/26))\r\n2018-04-27 08:49:12.060 [ERROR][277] ipam.go 204: Error claiming new block: 10.123.84.64/26 already claimed by %!!(MISSING)s(*string=0xc420283430)\r\n2018-04-27 08:49:12.060 [ERROR][277] ipam.go 135: Error assigning IPV4 addresses: 10.123.84.64/26 already claimed by %!!(MISSING)s(*string=0xc420283430)\r\n2018-04-27 08:49:12.060 [FATAL][277] allocate_ipip_addr.go 120: Unable to autoassign an address for IPIP error=10.123.84.64/26 already claimed by %!!(MISSING)s(*string=0xc420283430)\r\n\"\r\n2018-04-27T08:49:12Z public-slave-14 confd[130]: ERROR exit status 1\r\n```\r\n\r\nUnlike we initially assumed, `runsv` _does_ successfully restart confd when it crashes (obviously), but confd does not re-render a template when its `reload_cmd` fails. (and I don't think it should, either) The `ERROR exit status 1` is from the `allocate-ipip-addr` subprocess run by confd.\r\n\r\n## Possible Solution\r\n<!--- Not obligatory, but suggest a fix/reason for the bug, -->\r\n<!--- or ideas how to implement the addition or change -->\r\n\r\nOf course this is completely up to you, but to me it looks like a retry mechanism would be the most straightforward to implement. Locking is already in place, since one block clearly cannot be claimed by multiple hosts.\r\n\r\nCould be done on two levels, either\r\n- around the `clientv3.IPAM().AutoAssign()` at https://github.com/projectcalico/node/blob/master/pkg/allocateipip/allocate_ipip_addr.go#L162, or\r\n- inside `AutoAssign()` itself\r\n\r\n## Steps to Reproduce (for bugs)\r\n<!--- Provide a link to a live example, or an unambiguous set of steps to -->\r\n<!--- reproduce this bug. Include code to reproduce, if relevant -->\r\n\r\n1. Spin up a significant amount of `calico/node` instances connected to etcd\r\n2. Declare an ipPool (**with ipip!**)\r\n3. Check if every node has an address assigned to `tunl0` (this might go undetected in tests); alternatively, check `/var/log/confd/current` for error output\r\n4. Rinse and repeat if necessary, this happens about 80% of the time on at least one node in our larger clusters\r\n\r\nNote: I assume this could be tested in isolation by spawning a hundred `allocate-ipip-addr` simultaneously on a single box. This is not super cheap (money-wise) to replicate on a fleet of VMs.\r\n\r\n## Context\r\n<!--- How has this issue affected you? What are you trying to accomplish? -->\r\n<!--- Providing context helps us come up with a solution that is most useful in the real world -->\r\n\r\n## Your Environment\r\n<!--- Include as many relevant details about the environment you experienced the bug in -->\r\n\r\nSome details I did not mention in the original post:\r\n- Cluster size: about 40 nodes\r\n- Pool size: `/16`\r\n- Block size: `/26` (default?)\r\n- `calico/node` 2.6.7\r\n- etcd back-end\r\n\r\nThe ipPool has ipip enabled:\r\n```\r\nspec:\r\n  ipip:\r\n    enabled: true\r\n    mode: always\r\n  nat-outgoing: true\r\n```\r\n\r\n@caseydavenport @fasaxc \r\n\r\nThanks in advance for your time and for developing Calico!"}}
{"actor":{"display_login":"ti-mo"}}