{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/862","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/862/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/862/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/862/events","html_url":"https://github.com/scylladb/scylla/issues/862","id":130390303,"number":862,"title":"\"stop-sigterm timed out\" while trying to stop a node with running repair on cluster","user":{"login":"noamha","id":2642373,"avatar_url":"https://avatars.githubusercontent.com/u/2642373?v=3","gravatar_id":"","url":"https://api.github.com/users/noamha","html_url":"https://github.com/noamha","followers_url":"https://api.github.com/users/noamha/followers","following_url":"https://api.github.com/users/noamha/following{/other_user}","gists_url":"https://api.github.com/users/noamha/gists{/gist_id}","starred_url":"https://api.github.com/users/noamha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noamha/subscriptions","organizations_url":"https://api.github.com/users/noamha/orgs","repos_url":"https://api.github.com/users/noamha/repos","events_url":"https://api.github.com/users/noamha/events{/privacy}","received_events_url":"https://api.github.com/users/noamha/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/high","name":"high","color":"d4c5f9"}],"state":"open","locked":false,"assignee":{"login":"gleb-cloudius","id":6554938,"avatar_url":"https://avatars.githubusercontent.com/u/6554938?v=3","gravatar_id":"","url":"https://api.github.com/users/gleb-cloudius","html_url":"https://github.com/gleb-cloudius","followers_url":"https://api.github.com/users/gleb-cloudius/followers","following_url":"https://api.github.com/users/gleb-cloudius/following{/other_user}","gists_url":"https://api.github.com/users/gleb-cloudius/gists{/gist_id}","starred_url":"https://api.github.com/users/gleb-cloudius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gleb-cloudius/subscriptions","organizations_url":"https://api.github.com/users/gleb-cloudius/orgs","repos_url":"https://api.github.com/users/gleb-cloudius/repos","events_url":"https://api.github.com/users/gleb-cloudius/events{/privacy}","received_events_url":"https://api.github.com/users/gleb-cloudius/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/1","html_url":"https://github.com/scylladb/scylla/milestones/GA","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/1/labels","id":1164307,"number":1,"title":"GA","description":null,"creator":{"login":"dorlaor","id":1735237,"avatar_url":"https://avatars.githubusercontent.com/u/1735237?v=3","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"open_issues":126,"closed_issues":164,"state":"open","created_at":"2015-06-15T06:37:52Z","updated_at":"2016-02-18T00:15:37Z","due_on":null,"closed_at":null},"comments":10,"created_at":"2016-02-01T15:24:53Z","updated_at":"2016-02-18T00:22:31Z","closed_at":null,"body":"Configuration: 3 x Rackspace OnMetal IO v1, Scylla 0.17, RF x 2\r\nInit Keyspace\r\n```\r\ncassandra-stress write n=1 cl=ALL -schema \"replication(factor=2)\" -mode native cql3 -node $SERVERS\r\n```\r\n\r\nWrote 150M keys using 5 stress servers:\r\n```\r\ntime cassandra-stress write cl=TWO n=30000000 -mode native cql3 -rate threads=700 -node $SERVERS -pop seq=1..30000000 -col 'size=FIXED(2048) n=FIXED(5)'\r\ntime cassandra-stress write cl=TWO n=30000000 -mode native cql3 -rate threads=700 -node $SERVERS -pop seq=30000000..60000000 -col 'size=FIXED(2048) n=FIXED(5)'\r\ntime cassandra-stress write cl=TWO n=30000000 -mode native cql3 -rate threads=700 -node $SERVERS -pop seq=60000000..90000000 -col 'size=FIXED(2048) n=FIXED(5)'\r\ntime cassandra-stress write cl=TWO n=30000000 -mode native cql3 -rate threads=700 -node $SERVERS -pop seq=90000000..120000000 -col 'size=FIXED(2048) n=FIXED(5)'\r\ntime cassandra-stress write cl=TWO n=30000000 -mode native cql3 -rate threads=700 -node $SERVERS -pop seq=120000000..150000000 -col 'size=FIXED(2048) n=FIXED(5)\r\n```\r\n\r\n- Deleted some sstables from node3\r\n- Started running repair on node3\r\n- tried to stop node1 with\r\n```\r\nservice scylla-server stop\r\n```\r\n\r\nGot the following errors on journalctl\r\n```\r\nFeb 01 15:11:59 noam-server-01.localdomain systemd[1]: Stopping Scylla Server...\r\nFeb 01 15:11:59 noam-server-01.localdomain scylla[37952]:  [shard 0] storage_service - Drain on shutdown: starts\r\nFeb 01 15:11:59 noam-server-01.localdomain scylla[37952]:  [shard 0] gossip - Announcing shutdown\r\nFeb 01 15:11:59 noam-server-01.localdomain scylla[37952]:  [shard 0] storage_service - Node 10.184.8.25 state jump to normal\r\nFeb 01 15:11:59 noam-server-01.localdomain scylla[37952]:  [shard 39] compaction - Compacting [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-439-Data.db:level=0, /var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-479-Data.db:level=0, /var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-519-Data.db:level=0, /var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-559-Data.db:level=0, ]\r\nFeb 01 15:11:59 noam-server-01.localdomain scylla[37952]:  [shard 39] compaction - Compacted 4 sstables to [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-599-Data.db:level=0, ]. 24892 bytes to 12181 (~48% of original) in 7ms = 1.48153MB/s. ~1024 total partitions merged to 1.\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] storage_service - Drain on shutdown: stop_gossiping done\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] storage_service - CQL server stopped\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] storage_service - Drain on shutdown: shutdown rpc and cql server done\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=10: rpc::closed_error (connection is closed)\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=10: rpc::closed_error (connection is closed)\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=10: rpc::closed_error (connection is closed)\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=10: rpc::closed_error (connection is closed)\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=10: rpc::closed_error (connection is closed)\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=10: rpc::closed_error (connection is closed)\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=10: rpc::closed_error (connection is closed)\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=10: rpc::closed_error (connection is closed)\r\nFeb 01 15:12:01 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=10: rpc::closed_error (connection is closed)\r\nFeb 01 15:12:31 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=9: OK\r\nFeb 01 15:12:31 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=9: OK\r\nFeb 01 15:12:31 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=9: OK\r\nFeb 01 15:12:31 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=9: OK\r\nFeb 01 15:12:31 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=9: OK\r\nFeb 01 15:12:31 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=9: OK\r\nFeb 01 15:12:31 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=9: OK\r\nFeb 01 15:12:31 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=9: OK\r\nFeb 01 15:12:31 noam-server-01.localdomain scylla[37952]:  [shard 0] messaging_service - Retry verb=16 to 10.184.8.248:0, retry=9: OK\r\nFeb 01 15:13:30 noam-server-01.localdomain systemd[1]: scylla-server.service stop-sigterm timed out. Killing.\r\nFeb 01 15:13:30 noam-server-01.localdomain systemd[1]: scylla-server.service: main process exited, code=killed, status=9/KILL\r\nFeb 01 15:13:30 noam-server-01.localdomain sudo[38062]:   scylla : TTY=unknown ; PWD=/var/lib/scylla ; USER=root ; COMMAND=/usr/lib/scylla/scylla_stop\r\nFeb 01 15:13:30 noam-server-01.localdomain systemd[1]: Stopped Scylla Server.\r\nFeb 01 15:13:30 noam-server-01.localdomain systemd[1]: Unit scylla-server.service entered failed state.\r\nFeb 01 15:13:30 noam-server-01.localdomain systemd[1]: scylla-server.service failed.\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/185475480","html_url":"https://github.com/scylladb/scylla/issues/862#issuecomment-185475480","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/862","id":185475480,"user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"created_at":"2016-02-18T00:22:31Z","updated_at":"2016-02-18T00:22:31Z","body":"@gleb-cloudius stopping new connection attempts - is that possible ?"}}