{"action":"created","issue":{"url":"https://api.github.com/repos/Metaswitch/sprout/issues/1552","repository_url":"https://api.github.com/repos/Metaswitch/sprout","labels_url":"https://api.github.com/repos/Metaswitch/sprout/issues/1552/labels{/name}","comments_url":"https://api.github.com/repos/Metaswitch/sprout/issues/1552/comments","events_url":"https://api.github.com/repos/Metaswitch/sprout/issues/1552/events","html_url":"https://github.com/Metaswitch/sprout/issues/1552","id":180267588,"number":1552,"title":"memcached data not resynced on adding Sprout node","user":{"login":"sebrexmetaswitch","id":4903681,"avatar_url":"https://avatars.githubusercontent.com/u/4903681?v=3","gravatar_id":"","url":"https://api.github.com/users/sebrexmetaswitch","html_url":"https://github.com/sebrexmetaswitch","followers_url":"https://api.github.com/users/sebrexmetaswitch/followers","following_url":"https://api.github.com/users/sebrexmetaswitch/following{/other_user}","gists_url":"https://api.github.com/users/sebrexmetaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/sebrexmetaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebrexmetaswitch/subscriptions","organizations_url":"https://api.github.com/users/sebrexmetaswitch/orgs","repos_url":"https://api.github.com/users/sebrexmetaswitch/repos","events_url":"https://api.github.com/users/sebrexmetaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/sebrexmetaswitch/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Metaswitch/sprout/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/Metaswitch/sprout/labels/high-priority","name":"high-priority","color":"ff0000"}],"state":"open","locked":false,"assignee":{"login":"AME2","id":15612723,"avatar_url":"https://avatars.githubusercontent.com/u/15612723?v=3","gravatar_id":"","url":"https://api.github.com/users/AME2","html_url":"https://github.com/AME2","followers_url":"https://api.github.com/users/AME2/followers","following_url":"https://api.github.com/users/AME2/following{/other_user}","gists_url":"https://api.github.com/users/AME2/gists{/gist_id}","starred_url":"https://api.github.com/users/AME2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AME2/subscriptions","organizations_url":"https://api.github.com/users/AME2/orgs","repos_url":"https://api.github.com/users/AME2/repos","events_url":"https://api.github.com/users/AME2/events{/privacy}","received_events_url":"https://api.github.com/users/AME2/received_events","type":"User","site_admin":false},"assignees":[{"login":"AME2","id":15612723,"avatar_url":"https://avatars.githubusercontent.com/u/15612723?v=3","gravatar_id":"","url":"https://api.github.com/users/AME2","html_url":"https://github.com/AME2","followers_url":"https://api.github.com/users/AME2/followers","following_url":"https://api.github.com/users/AME2/following{/other_user}","gists_url":"https://api.github.com/users/AME2/gists{/gist_id}","starred_url":"https://api.github.com/users/AME2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AME2/subscriptions","organizations_url":"https://api.github.com/users/AME2/orgs","repos_url":"https://api.github.com/users/AME2/repos","events_url":"https://api.github.com/users/AME2/events{/privacy}","received_events_url":"https://api.github.com/users/AME2/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2016-09-30T10:07:12Z","updated_at":"2016-10-19T10:07:26Z","closed_at":null,"body":"<!--\r\nThis page is for reporting issues with Project Clearwater. If you have a question, rather than a bug report, the mailing list at clearwater@lists.projectclearwater.org is a better place for it.\r\n\r\nTo give us the best chance of fixing the problem, we've suggested some information to give - please follow these guidelines if possible.\r\n\r\nDon't forget that you can attach logs and screenshots to Github issues - this may help us debug a problem.\r\n-->\r\n\r\n#### Symptoms\r\nAdding a sprout node to an existing Sprout cluster by changing the cluster_settings file and reloading Astaire doesn't copy across memcached data to the new Sprout node.\r\nThe Astaire logs show:\r\n\r\n> 30-09-2016 09:03:26.542 UTC Debug signalhandler.h:161: Signal 1 raised\r\n> 30-09-2016 09:03:26.542 UTC Debug astaire.cpp:118: Reloading memcached config\r\n> \r\n> [...]\r\n> \r\n> 30-09-2016 09:03:26.543 UTC Debug memcachedstoreview.cpp:259: New view -\r\n> \r\n> [...]\r\n> \r\n> 30-09-2016 09:03:26.543 UTC Status memcached_backend.cpp:158: Finished preparing new view, so flag that workers should switch to it\r\n> 30-09-2016 09:03:26.543 UTC Debug astaire.cpp:161: View has been updated - resync required\r\n> 30-09-2016 09:03:26.544 UTC Debug astaire.cpp:818: Found tag - memcached is up-to-date\r\n> 30-09-2016 09:03:26.544 UTC Debug astaire.cpp:465: Start resync operation\r\n>\r\n> [...]\r\n> \r\n> 30-09-2016 09:03:26.544 UTC Info astaire.cpp:470: No resyncing required\r\n\r\n#### Impact\r\nIf the sprout node with the stored data goes down, data is lost.\r\n\r\n\r\n#### Release and environment \r\nCurrent dev branch\r\n\r\n\r\n#### Steps to reproduce\r\nStart with single Sprout node with registered subscribers.\r\nBring up a new Sprout node and attempted to add it to the memcached cluster manually by, on each:\r\n - disabling config manager\r\n - editing the cluster_settings file to have both nodes in the new_servers line (in addition to the existing node in the servers line)\r\n - reloading Sprout to use the new cluster settings: `sudo /usr/share/clearwater/bin/reload_memcached_users`\r\n - resycincing memcached:\r\n`sudo service astaire reload`\r\n`sudo service astaire wait-sync`\r\n\r\nThen kill memcached on node 1 and the subs can't make calls\r\n"},"comment":{"url":"https://api.github.com/repos/Metaswitch/sprout/issues/comments/254770277","html_url":"https://github.com/Metaswitch/sprout/issues/1552#issuecomment-254770277","issue_url":"https://api.github.com/repos/Metaswitch/sprout/issues/1552","id":254770277,"user":{"login":"AME2","id":15612723,"avatar_url":"https://avatars.githubusercontent.com/u/15612723?v=3","gravatar_id":"","url":"https://api.github.com/users/AME2","html_url":"https://github.com/AME2","followers_url":"https://api.github.com/users/AME2/followers","following_url":"https://api.github.com/users/AME2/following{/other_user}","gists_url":"https://api.github.com/users/AME2/gists{/gist_id}","starred_url":"https://api.github.com/users/AME2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AME2/subscriptions","organizations_url":"https://api.github.com/users/AME2/orgs","repos_url":"https://api.github.com/users/AME2/repos","events_url":"https://api.github.com/users/AME2/events{/privacy}","received_events_url":"https://api.github.com/users/AME2/received_events","type":"User","site_admin":false},"created_at":"2016-10-19T10:07:26Z","updated_at":"2016-10-19T10:07:26Z","body":"As it is the reproduction process described above does result in successful calls being made. However if the Sprout process is turned off on the second node as it rejoins the cluster then calls do fail, I am currently investigating why this is."}}
{"actor":{"display_login":"AME2"}}