{"action":"created","issue":{"url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/591","repository_url":"https://api.github.com/repos/thelastpickle/cassandra-reaper","labels_url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/591/labels{/name}","comments_url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/591/comments","events_url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/591/events","html_url":"https://github.com/thelastpickle/cassandra-reaper/issues/591","id":386356541,"node_id":"MDU6SXNzdWUzODYzNTY1NDE=","number":591,"title":"Duplicate repairs getting scheduled.","user":{"login":"powellchristoph","id":1540688,"node_id":"MDQ6VXNlcjE1NDA2ODg=","avatar_url":"https://avatars1.githubusercontent.com/u/1540688?v=4","gravatar_id":"","url":"https://api.github.com/users/powellchristoph","html_url":"https://github.com/powellchristoph","followers_url":"https://api.github.com/users/powellchristoph/followers","following_url":"https://api.github.com/users/powellchristoph/following{/other_user}","gists_url":"https://api.github.com/users/powellchristoph/gists{/gist_id}","starred_url":"https://api.github.com/users/powellchristoph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/powellchristoph/subscriptions","organizations_url":"https://api.github.com/users/powellchristoph/orgs","repos_url":"https://api.github.com/users/powellchristoph/repos","events_url":"https://api.github.com/users/powellchristoph/events{/privacy}","received_events_url":"https://api.github.com/users/powellchristoph/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-11-30T21:28:04Z","updated_at":"2018-12-06T00:09:11Z","closed_at":"2018-12-06T00:09:12Z","author_association":"NONE","body":"Greetings everyone,\r\n\r\nThank you for the great work on this project. It is greatly appreciated.\r\n\r\nSomehow I seem to getting duplicate repairs scheduled and run for the same keyspace. This is obviously creating contention as both repairs block each other. Also, once they start, I am unable to remove either one of them. This is the second time that it has happened on two different keyspaces.\r\n\r\nI am running (2) reapers in a multi-region setup with replicated cassandra backend.\r\n\r\ncassandra-reaper version: 1.3.0\r\njava version \"1.8.0_181\"\r\nOS: Ubuntu 16.04.5 LTS\r\n\r\n```\r\ncurl -s localhost:8080/repair_schedule/ca773da0-b5f4-11e8-bbac-5fdbb7ebf105 | jq .\r\n{\r\n  \"id\": \"ca773da0-b5f4-11e8-bbac-5fdbb7ebf105\",\r\n  \"owner\": \"blackbook\",\r\n  \"state\": \"ACTIVE\",\r\n  \"intensity\": 0.8999999761581421,\r\n  \"cluster_name\": \"aux1\",\r\n  \"keyspace_name\": \"blackbook\",\r\n  \"column_families\": [],\r\n  \"incremental_repair\": false,\r\n  \"segment_count\": 0,\r\n  \"repair_parallelism\": \"DATACENTER_AWARE\",\r\n  \"scheduled_days_between\": 8,\r\n  \"nodes\": [],\r\n  \"datacenters\": [],\r\n  \"blacklisted_tables\": [],\r\n  \"segment_count_per_node\": 0,\r\n  \"repair_thread_count\": 1,\r\n  \"next_activation\": \"2018-12-08T18:50:00Z\",\r\n  \"creation_time\": \"2018-09-11T18:59:20Z\",\r\n  \"pause_time\": \"2018-11-21T16:57:51Z\"\r\n}\r\n```\r\n```\r\ncurl -s localhost:8080/repair_run | jq '.[] | select( .state | contains(\"RUNNING\")) | select( .owner | contains(\"blackbook\"))'\r\n{\r\n  \"cause\": \"scheduled run (schedule id ca773da0-b5f4-11e8-bbac-5fdbb7ebf105)\",\r\n  \"owner\": \"blackbook\",\r\n  \"id\": \"d191d350-f4d0-11e8-bd85-e111dadad660\",\r\n  \"state\": \"RUNNING\",\r\n  \"intensity\": 0.8999999761581421,\r\n  \"duration\": \"2 hours 24 minutes 49 seconds\",\r\n  \"cluster_name\": \"aux1\",\r\n  \"column_families\": [],\r\n  \"keyspace_name\": \"blackbook\",\r\n  \"incremental_repair\": false,\r\n  \"total_segments\": 10752,\r\n  \"repair_parallelism\": \"DATACENTER_AWARE\",\r\n  \"segments_repaired\": 219,\r\n  \"last_event\": \"Postponed due to affected hosts already doing repairs\",\r\n  \"nodes\": [],\r\n  \"datacenters\": [],\r\n  \"blacklisted_tables\": [],\r\n  \"repair_thread_count\": 1,\r\n  \"creation_time\": \"2018-11-30T18:50:34Z\",\r\n  \"start_time\": \"2018-11-30T18:50:34Z\",\r\n  \"end_time\": \"2018-11-30T18:52:34Z\",\r\n  \"pause_time\": \"2018-11-30T18:52:34Z\",\r\n  \"current_time\": \"2018-11-30T21:15:23Z\",\r\n  \"estimated_time_of_arrival\": \"2018-12-05T17:20:41Z\"\r\n}\r\n{\r\n  \"cause\": \"scheduled run (schedule id ca773da0-b5f4-11e8-bbac-5fdbb7ebf105)\",\r\n  \"owner\": \"blackbook\",\r\n  \"id\": \"d22dec40-f4d0-11e8-a415-b5e61892c678\",\r\n  \"state\": \"RUNNING\",\r\n  \"intensity\": 0.8999999761581421,\r\n  \"duration\": \"2 hours 24 minutes 48 seconds\",\r\n  \"cluster_name\": \"aux1\",\r\n  \"column_families\": [],\r\n  \"keyspace_name\": \"blackbook\",\r\n  \"incremental_repair\": false,\r\n  \"total_segments\": 10752,\r\n  \"repair_parallelism\": \"DATACENTER_AWARE\",\r\n  \"segments_repaired\": 202,\r\n  \"last_event\": \"Triggered repair of segment d22f733c-f4d0-11e8-a415-b5e61892c678 via host 34.216.185.148\",\r\n  \"nodes\": [],\r\n  \"datacenters\": [],\r\n  \"blacklisted_tables\": [],\r\n  \"repair_thread_count\": 1,\r\n  \"creation_time\": \"2018-11-30T18:50:35Z\",\r\n  \"start_time\": \"2018-11-30T18:50:35Z\",\r\n  \"end_time\": \"2018-11-30T18:50:35Z\",\r\n  \"pause_time\": \"2018-11-30T18:50:35Z\",\r\n  \"current_time\": \"2018-11-30T21:15:24Z\",\r\n  \"estimated_time_of_arrival\": \"2018-12-06T03:18:20Z\"\r\n}\r\n```\r\n```\r\nNov 30 21:16:13 cassandra-reaper-n01 cassandra-reaper[1496]: INFO   [2018-11-30 21:16:13,516] [dw-151148 - PUT /repair_run/d191d350-f4d0-11e8-bd85-e111dadad660/state/PAUSED] i.c.r.RepairRunResource - modify repair run state called with: id = d191d350-f4d0-11e8-bd85-e111dadad660, state = Optional[PAUSED]\r\nNov 30 21:16:13 cassandra-reaper-n01 cassandra-reaper[1496]: ERROR  [2018-11-30 21:16:13,757] [dw-151148 - PUT /repair_run/d191d350-f4d0-11e8-bd85-e111dadad660/state/PAUSED] i.c.r.RepairRunResource - repair unit already has run ca63dcb0-b5f4-11e8-bbac-5fdbb7ebf105 in RUNNING state\r\n\r\nNov 30 21:18:06 cassandra-reaper-n01 cassandra-reaper[1496]: INFO   [2018-11-30 21:18:06,717] [dw-151098 - PUT /repair_run/d191d350-f4d0-11e8-bd85-e111dadad660/state/ABORTED] i.c.r.RepairRunResource - modify repair run state called with: id = d191d350-f4d0-11e8-bd85-e111dadad660, state = Optional[ABORTED]\r\nNov 30 21:18:07 cassandra-reaper-n01 cassandra-reaper[1496]: ERROR  [2018-11-30 21:18:06,999] [dw-151098 - PUT /repair_run/d191d350-f4d0-11e8-bd85-e111dadad660/state/ABORTED] i.c.r.RepairRunResource - repair unit already has run ca63dcb0-b5f4-11e8-bbac-5fdbb7ebf105 in RUNNING state\r\n```\r\n```\r\nNov 30 21:16:43 cassandra-reaper-n01 cassandra-reaper[1496]: INFO   [2018-11-30 21:16:43,689] [dw-151121 - PUT /repair_run/d22dec40-f4d0-11e8-a415-b5e61892c678/state/PAUSED] i.c.r.RepairRunResource - modify repair run state called with: id = d22dec40-f4d0-11e8-a415-b5e61892c678, state = Optional[PAUSED]\r\nNov 30 21:16:44 cassandra-reaper-n01 cassandra-reaper[1496]: ERROR  [2018-11-30 21:16:44,016] [dw-151121 - PUT /repair_run/d22dec40-f4d0-11e8-a415-b5e61892c678/state/PAUSED] i.c.r.RepairRunResource - repair unit already has run ca63dcb0-b5f4-11e8-bbac-5fdbb7ebf105 in RUNNING state\r\n\r\nNov 30 21:17:58 cassandra-reaper-n01 cassandra-reaper[1496]: INFO   [2018-11-30 21:17:58,947] [dw-151162 - PUT /repair_run/d22dec40-f4d0-11e8-a415-b5e61892c678/state/ABORTED] i.c.r.RepairRunResource - modify repair run state called with: id = d22dec40-f4d0-11e8-a415-b5e61892c678, state = Optional[ABORTED]\r\nNov 30 21:17:59 cassandra-reaper-n01 cassandra-reaper[1496]: ERROR  [2018-11-30 21:17:59,225] [dw-151162 - PUT /repair_run/d22dec40-f4d0-11e8-a415-b5e61892c678/state/ABORTED] i.c.r.RepairRunResource - repair unit already has run ca63dcb0-b5f4-11e8-bbac-5fdbb7ebf105 in RUNNING state\r\n```\r\n\r\ncassandra-reaper.yaml from one region\r\n```\r\n# Cassandra Reaper Configuration \r\nsegmentCountPerNode: 16\r\nrepairParallelism: DATACENTER_AWARE\r\nrepairIntensity: 0.9\r\nscheduleDaysBetween: 1\r\nrepairRunThreadCount: 15\r\nhangingRepairTimeoutMins: 300\r\nstorageType: cassandra\r\nenableCrossOrigin: true\r\nincrementalRepair: false\r\nenableDynamicSeedList: true\r\nrepairManagerSchedulingIntervalSeconds: 10\r\nactivateQueryLogger: false\r\njmxConnectionTimeoutInSeconds: 5\r\nuseAddressTranslator: false\r\n\r\ndatacenterAvailability: EACH\r\n\r\ncassandra:\r\n  clusterName: \"Aux 1\"\r\n  contactPoints: [\"cassandra-n01\",\"cassandra-n02\",\"cassandra-n03\"]\r\n  keyspace: reaper_db\r\n  socketOptions:\r\n    readTimeoutMillis: 20000\r\n  queryOptions:\r\n    consistencyLevel: LOCAL_QUORUM\r\n    serialConsistencyLevel: SERIAL\r\n  loadBalancingPolicy:\r\n    type: tokenAware\r\n    shuffleReplicas: true\r\n    subPolicy:\r\n      type: dcAwareRoundRobin\r\n      localDC: us-east-1\r\n      usedHostsPerRemoteDC: 0\r\n      allowRemoteDCsForLocalConsistencyLevel: false\r\n```\r\nThe reaper_db keyspace is replicated appropriately.\r\n```\r\ncqlsh> desc keyspace reaper_db;\r\n\r\nCREATE KEYSPACE reaper_db WITH replication = {'class': 'NetworkTopologyStrategy', 'us-east-1': '3', 'us-west-2': '3'}  AND durable_writes = true;\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/comments/444699479","html_url":"https://github.com/thelastpickle/cassandra-reaper/issues/591#issuecomment-444699479","issue_url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/591","id":444699479,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NDY5OTQ3OQ==","user":{"login":"michaelsembwever","id":559444,"node_id":"MDQ6VXNlcjU1OTQ0NA==","avatar_url":"https://avatars2.githubusercontent.com/u/559444?v=4","gravatar_id":"","url":"https://api.github.com/users/michaelsembwever","html_url":"https://github.com/michaelsembwever","followers_url":"https://api.github.com/users/michaelsembwever/followers","following_url":"https://api.github.com/users/michaelsembwever/following{/other_user}","gists_url":"https://api.github.com/users/michaelsembwever/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelsembwever/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelsembwever/subscriptions","organizations_url":"https://api.github.com/users/michaelsembwever/orgs","repos_url":"https://api.github.com/users/michaelsembwever/repos","events_url":"https://api.github.com/users/michaelsembwever/events{/privacy}","received_events_url":"https://api.github.com/users/michaelsembwever/received_events","type":"User","site_admin":false},"created_at":"2018-12-06T00:09:11Z","updated_at":"2018-12-06T00:09:11Z","author_association":"MEMBER","body":"I have a fix in https://github.com/thelastpickle/cassandra-reaper/pull/593\r\nIt would be awesome if you could deploy and test it.\r\nOtherwise if it happens again those requested logs would be very useful."}}
{"actor":{"display_login":"michaelsembwever"}}