{"action":"opened","issue":{"url":"https://api.github.com/repos/twitter/heron/issues/1556","repository_url":"https://api.github.com/repos/twitter/heron","labels_url":"https://api.github.com/repos/twitter/heron/issues/1556/labels{/name}","comments_url":"https://api.github.com/repos/twitter/heron/issues/1556/comments","events_url":"https://api.github.com/repos/twitter/heron/issues/1556/events","html_url":"https://github.com/twitter/heron/issues/1556","id":189196787,"number":1556,"title":"When setting TMaster location, check that it has changed","user":{"login":"billonahill","id":23758,"avatar_url":"https://avatars.githubusercontent.com/u/23758?v=3","gravatar_id":"","url":"https://api.github.com/users/billonahill","html_url":"https://github.com/billonahill","followers_url":"https://api.github.com/users/billonahill/followers","following_url":"https://api.github.com/users/billonahill/following{/other_user}","gists_url":"https://api.github.com/users/billonahill/gists{/gist_id}","starred_url":"https://api.github.com/users/billonahill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/billonahill/subscriptions","organizations_url":"https://api.github.com/users/billonahill/orgs","repos_url":"https://api.github.com/users/billonahill/repos","events_url":"https://api.github.com/users/billonahill/events{/privacy}","received_events_url":"https://api.github.com/users/billonahill/received_events","type":"User","site_admin":false},"labels":[{"id":265202765,"url":"https://api.github.com/repos/twitter/heron/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-11-14T19:05:35Z","updated_at":"2016-11-14T19:05:35Z","closed_at":null,"body":"The TMaster gets restarted during scaling. When it comes back up, it fails to start properly because it finds a pre-existing TMasterLocation in the state manager. As a result it retries for 30 seconds before setting itself as the TMaster and restarting itself. The next process starts up fine.\r\n\r\nThis polling and retrying unnecessarily slows down the scaling process, since the TMasterLocation has not changed. The TMaster should inspect the contents of the location against it's currently location and proceed if it's the same.\r\n\r\nThis is what the startup logs looks like.\r\n```\r\nLog file created at: 2016/11/14 18:44:30\r\nRunning on machine: smf1-cwr-04-sr1.prod.twitter.com\r\nLog line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg\r\nI1114 18:44:30.778712 63701 tmaster-main.cpp:63] Starting tmaster for topology summingbird_tsar_devel_interaction-counter_billg with topology id summingbird_tsar_devel_interaction-counter_billg881e6a58-403b-41c5-9046-5d01b2b295cc zkhostport szookeeper.smf1.twitter.com:2181 zkroot /storm/heron/states and nstmgrs 120\r\nI1114 18:44:30.778992 63701 yaml-file-reader.cpp:55] Config file ./heron-conf/metrics_sinks.yaml changed. reloading..\r\nI1114 18:44:30.783954 63701 zkclient.cpp:203] Creating zknode /storm/heron/states/tmasters/summingbird_tsar_devel_interaction-counter_billg\r\nI1114 18:44:30.783983 63701 metricsmgr-client.cpp:108] We have not yet registered to metricsmgr. Holding off sending TMasterLocation\r\nI1114 18:44:30.784034 63701 metricsmgr-client.cpp:49] Connected to metrics manager\r\nI1114 18:44:30.784837 63703 zkclient.cpp:307] ZKClient GlobalWatcher called with type SESSION_EVENT and state CONNECTED_STATE\r\nI1114 18:44:30.784849 63703 zkclient.cpp:318] Got a new session id: 385164178542423830\r\nE1114 18:44:30.786675 63701 heron-zkstatemgr.cpp:283] Setting TMaster Location failed because another zmaster exists\r\nI1114 18:44:30.786695 63701 tmaster.cpp:194] Topology Master node already exists. Maybe its because of our restart. Will try again\r\nI1114 18:44:30.786967 63701 metricsmgr-client.cpp:86] Successfully registered ourselves to the metricsmgr\r\nI1114 18:44:30.786974 63701 metricsmgr-client.cpp:92] Sending TMaster Location to metricsmgr\r\nI1114 18:44:31.787664 63701 zkclient.cpp:203] Creating zknode /storm/heron/states/tmasters/summingbird_tsar_devel_interaction-counter_billg\r\nE1114 18:44:31.789458 63701 heron-zkstatemgr.cpp:283] Setting TMaster Location failed because another zmaster exists\r\nI1114 18:44:31.789491 63701 tmaster.cpp:194] Topology Master node already exists. Maybe its because of our restart. Will try again\r\nI1114 18:44:32.789683 63701 zkclient.cpp:203] Creating zknode /storm/heron/states/tmasters/summingbird_tsar_devel_interaction-counter_billg\r\nE1114 18:44:32.791092 63701 heron-zkstatemgr.cpp:283] Setting TMaster Location failed because another zmaster exists\r\nI1114 18:44:32.791133 63701 tmaster.cpp:194] Topology Master node already exists. Maybe its because of our restart. Will try again\r\n.... [repeat for 30 seconds]\r\nI1114 18:44:59.846448 63701 zkclient.cpp:203] Creating zknode /storm/heron/states/tmasters/summingbird_tsar_devel_interaction-counter_billg\r\nE1114 18:44:59.848282 63701 heron-zkstatemgr.cpp:283] Setting TMaster Location failed because another zmaster exists\r\nE1114 18:44:59.848309 63701 tmaster.cpp:205] For topology summingbird_tsar_devel_interaction-counter_billg Error setting ourselves as TMaster. Errorcode is 2002\r\n```"}}
{"actor":{"display_login":"billonahill"}}