{"action":"created","issue":{"url":"https://api.github.com/repos/mesosphere/dcos-cassandra-service/issues/121","repository_url":"https://api.github.com/repos/mesosphere/dcos-cassandra-service","labels_url":"https://api.github.com/repos/mesosphere/dcos-cassandra-service/issues/121/labels{/name}","comments_url":"https://api.github.com/repos/mesosphere/dcos-cassandra-service/issues/121/comments","events_url":"https://api.github.com/repos/mesosphere/dcos-cassandra-service/issues/121/events","html_url":"https://github.com/mesosphere/dcos-cassandra-service/issues/121","id":153517112,"number":121,"title":"Node replacement failed","user":{"login":"brndnmtthws","id":3129093,"avatar_url":"https://avatars.githubusercontent.com/u/3129093?v=3","gravatar_id":"","url":"https://api.github.com/users/brndnmtthws","html_url":"https://github.com/brndnmtthws","followers_url":"https://api.github.com/users/brndnmtthws/followers","following_url":"https://api.github.com/users/brndnmtthws/following{/other_user}","gists_url":"https://api.github.com/users/brndnmtthws/gists{/gist_id}","starred_url":"https://api.github.com/users/brndnmtthws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brndnmtthws/subscriptions","organizations_url":"https://api.github.com/users/brndnmtthws/orgs","repos_url":"https://api.github.com/users/brndnmtthws/repos","events_url":"https://api.github.com/users/brndnmtthws/events{/privacy}","received_events_url":"https://api.github.com/users/brndnmtthws/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-06T19:01:19Z","updated_at":"2016-06-17T18:39:09Z","closed_at":null,"body":"I attempted to replace a node and got the following:\r\n\r\n```\r\ncat: /etc/ld.so.conf.d/*.conf: No such file or directory\r\nJava HotSpot(TM) 64-Bit Server VM warning: Cannot open file apache-cassandra-2.2.5/bin/../logs/gc.log due to No such file or directory\r\n\r\njava.lang.RuntimeException: A node required to move the data consistently is down (/10.0.1.122). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false\r\n\tat org.apache.cassandra.dht.RangeStreamer.getAllRangesWithStrictSourcesFor(RangeStreamer.java:264)\r\n\tat org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:147)\r\n\tat org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:82)\r\n\tat org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1230)\r\n\tat org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:924)\r\n\tat org.apache.cassandra.service.StorageService.initServer(StorageService.java:709)\r\n\tat org.apache.cassandra.service.StorageService.initServer(StorageService.java:585)\r\n\tat org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:300)\r\n\tat org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516)\r\n\tat org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625)\r\n```\r\n\r\nand:\r\n\r\n```\r\nCompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I\r\nCompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I\r\nCompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V\r\nCompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V\r\nCompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I\r\nCompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I\r\nCompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I\r\nCompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I\r\nCompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I\r\nCompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I\r\nCompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I\r\nCompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I\r\nINFO  18:59:37 Loading settings from file:/var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/apache-cassandra-2.2.5/conf/cassandra.yaml\r\nINFO  18:59:37 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=false; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=cassandra; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/volume/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=16; concurrent_reads=16; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/volume/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_user_defined_functions=false; endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.0.2.66; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_validity_in_ms=2000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=10.0.2.66; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/volume/saved_caches; seed_provider=[{class_name=com.mesosphere.dcos.cassandra.DcosSeedProvider, parameters=[{seeds_url=http://cassandra.marathon.mesos:9000/v1/seeds}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; windows_timer_interval=1; write_request_timeout_in_ms=2000]\r\nINFO  18:59:37 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap\r\nINFO  18:59:37 Global memtable on-heap threshold is enabled at 509MB\r\nINFO  18:59:37 Global memtable off-heap threshold is enabled at 509MB\r\nINFO  18:59:37 Unable to load cassandra-topology.properties; compatibility mode disabled\r\nWARN  18:59:37 Only 28482 MB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots\r\nINFO  18:59:37 Retrieved response {\"isSeed\":false,\"seeds\":[\"10.0.1.121\",\"10.0.1.123\"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds\r\nINFO  18:59:37 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]\r\nINFO  18:59:37 Hostname: ip-10-0-2-66.us-west-2.compute.internal\r\nINFO  18:59:37 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.8.0_74\r\nINFO  18:59:37 Heap size: 2136997888/2136997888\r\nINFO  18:59:37 Code Cache Non-heap memory: init = 2555904(2496K) used = 3151552(3077K) committed = 3211264(3136K) max = 251658240(245760K)\r\nINFO  18:59:37 Metaspace Non-heap memory: init = 0(0K) used = 13007080(12702K) committed = 13238272(12928K) max = -1(-1K)\r\nINFO  18:59:37 Compressed Class Space Non-heap memory: init = 0(0K) used = 1637264(1598K) committed = 1703936(1664K) max = 1073741824(1048576K)\r\nINFO  18:59:37 Par Eden Space Heap memory: init = 83886080(81920K) used = 62144072(60687K) committed = 83886080(81920K) max = 83886080(81920K)\r\nINFO  18:59:37 Par Survivor Space Heap memory: init = 10485760(10240K) used = 0(0K) committed = 10485760(10240K) max = 10485760(10240K)\r\nINFO  18:59:37 CMS Old Gen Heap memory: init = 2042626048(1994752K) used = 0(0K) committed = 2042626048(1994752K) max = 2042626048(1994752K)\r\nINFO  18:59:37 Classpath: apache-cassandra-2.2.5/bin/../conf:apache-cassandra-2.2.5/bin/../build/classes/main:apache-cassandra-2.2.5/bin/../build/classes/thrift:apache-cassandra-2.2.5/bin/../lib/ST4-4.0.8.jar:apache-cassandra-2.2.5/bin/../lib/airline-0.6.jar:apache-cassandra-2.2.5/bin/../lib/antlr-runtime-3.5.2.jar:apache-cassandra-2.2.5/bin/../lib/apache-cassandra-2.2.5.jar:apache-cassandra-2.2.5/bin/../lib/apache-cassandra-clientutil-2.2.5.jar:apache-cassandra-2.2.5/bin/../lib/apache-cassandra-thrift-2.2.5.jar:apache-cassandra-2.2.5/bin/../lib/cassandra-driver-core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:apache-cassandra-2.2.5/bin/../lib/commons-cli-1.1.jar:apache-cassandra-2.2.5/bin/../lib/commons-codec-1.2.jar:apache-cassandra-2.2.5/bin/../lib/commons-lang3-3.1.jar:apache-cassandra-2.2.5/bin/../lib/commons-math3-3.2.jar:apache-cassandra-2.2.5/bin/../lib/compress-lzf-0.8.4.jar:apache-cassandra-2.2.5/bin/../lib/concurrentlinkedhashmap-lru-1.4.jar:apache-cassandra-2.2.5/bin/../lib/crc32ex-0.1.1.jar:apache-cassandra-2.2.5/bin/../lib/disruptor-3.0.1.jar:apache-cassandra-2.2.5/bin/../lib/ecj-4.4.2.jar:apache-cassandra-2.2.5/bin/../lib/guava-16.0.jar:apache-cassandra-2.2.5/bin/../lib/high-scale-lib-1.0.6.jar:apache-cassandra-2.2.5/bin/../lib/jackson-core-asl-1.9.2.jar:apache-cassandra-2.2.5/bin/../lib/jackson-mapper-asl-1.9.2.jar:apache-cassandra-2.2.5/bin/../lib/jamm-0.3.0.jar:apache-cassandra-2.2.5/bin/../lib/javax.inject.jar:apache-cassandra-2.2.5/bin/../lib/jbcrypt-0.3m.jar:apache-cassandra-2.2.5/bin/../lib/jcl-over-slf4j-1.7.7.jar:apache-cassandra-2.2.5/bin/../lib/jna-4.0.0.jar:apache-cassandra-2.2.5/bin/../lib/joda-time-2.4.jar:apache-cassandra-2.2.5/bin/../lib/json-simple-1.1.jar:apache-cassandra-2.2.5/bin/../lib/libthrift-0.9.2.jar:apache-cassandra-2.2.5/bin/../lib/log4j-over-slf4j-1.7.7.jar:apache-cassandra-2.2.5/bin/../lib/logback-classic-1.1.3.jar:apache-cassandra-2.2.5/bin/../lib/logback-core-1.1.3.jar:apache-cassandra-2.2.5/bin/../lib/lz4-1.3.0.jar:apache-cassandra-2.2.5/bin/../lib/metrics-core-3.1.0.jar:apache-cassandra-2.2.5/bin/../lib/metrics-logback-3.1.0.jar:apache-cassandra-2.2.5/bin/../lib/metrics-statsd-common-4.1.2.jar:apache-cassandra-2.2.5/bin/../lib/metrics2-statsd-4.1.2.jar:apache-cassandra-2.2.5/bin/../lib/netty-all-4.0.23.Final.jar:apache-cassandra-2.2.5/bin/../lib/ohc-core-0.3.4.jar:apache-cassandra-2.2.5/bin/../lib/ohc-core-j8-0.3.4.jar:apache-cassandra-2.2.5/bin/../lib/reporter-config-base-3.0.0.jar:apache-cassandra-2.2.5/bin/../lib/reporter-config3-3.0.0.jar:apache-cassandra-2.2.5/bin/../lib/seedprovider-0.1.0.jar:apache-cassandra-2.2.5/bin/../lib/sigar-1.6.4.jar:apache-cassandra-2.2.5/bin/../lib/slf4j-api-1.7.7.jar:apache-cassandra-2.2.5/bin/../lib/snakeyaml-1.11.jar:apache-cassandra-2.2.5/bin/../lib/snappy-java-1.1.1.7.jar:apache-cassandra-2.2.5/bin/../lib/stream-2.5.2.jar:apache-cassandra-2.2.5/bin/../lib/super-csv-2.1.0.jar:apache-cassandra-2.2.5/bin/../lib/thrift-server-0.3.7.jar:apache-cassandra-2.2.5/bin/../lib/jsr223/*/*.jar:apache-cassandra-2.2.5/bin/../lib/jamm-0.3.0.jar\r\nINFO  18:59:37 JVM Arguments: [-ea, -javaagent:apache-cassandra-2.2.5/bin/../lib/jamm-0.3.0.jar, -XX:+CMSClassUnloadingEnabled, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -Xms2048M, -Xmx2048M, -Xmn100M, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseTLAB, -XX:+PerfDisableSharedMem, -XX:CompileCommandFile=apache-cassandra-2.2.5/bin/../conf/hotspot_compiler, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:CMSWaitDuration=10000, -XX:+UseCondCardMark, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -Xloggc:apache-cassandra-2.2.5/bin/../logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Djava.net.preferIPv4Stack=true, -Dcassandra.jmx.local.port=7199, -XX:+DisableExplicitGC, -Djava.library.path=apache-cassandra-2.2.5/bin/../lib/sigar-bin, -Dcassandra.metricsReporterConfigFile=metrics-reporter-config.yaml, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=apache-cassandra-2.2.5/bin/../logs, -Dcassandra.storagedir=apache-cassandra-2.2.5/bin/../data, -Dcassandra-foreground=yes]\r\nINFO  18:59:38 JNA mlockall successful\r\nWARN  18:59:38 jemalloc shared library could not be preloaded to speed up memory allocations\r\nWARN  18:59:38 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.\r\nINFO  18:59:38 Initializing SIGAR library\r\nINFO  18:59:38 Checked OS settings and found them configured for optimal performance.\r\nINFO  18:59:39 Initializing system.sstable_activity\r\nINFO  18:59:41 Initializing key cache with capacity of 100 MBs.\r\nINFO  18:59:41 Initializing row cache with capacity of 0 MBs\r\nINFO  18:59:41 Initializing counter cache with capacity of 50 MBs\r\nINFO  18:59:41 Scheduling counter cache save to every 7200 seconds (going to save all keys).\r\nINFO  18:59:41 Initializing system.hints\r\nINFO  18:59:41 Initializing system.compaction_history\r\nINFO  18:59:41 Initializing system.peers\r\nINFO  18:59:41 Initializing system.schema_columnfamilies\r\nINFO  18:59:41 Initializing system.schema_functions\r\nINFO  18:59:41 Initializing system.IndexInfo\r\nINFO  18:59:41 Initializing system.schema_columns\r\nINFO  18:59:41 Initializing system.schema_triggers\r\nINFO  18:59:41 Initializing system.local\r\nINFO  18:59:41 Initializing system.schema_usertypes\r\nINFO  18:59:41 Initializing system.batchlog\r\nINFO  18:59:41 Initializing system.available_ranges\r\nINFO  18:59:42 Initializing system.schema_aggregates\r\nINFO  18:59:42 Initializing system.paxos\r\nINFO  18:59:42 Initializing system.peer_events\r\nINFO  18:59:42 Initializing system.size_estimates\r\nINFO  18:59:42 Initializing system.compactions_in_progress\r\nINFO  18:59:42 Initializing system.schema_keyspaces\r\nINFO  18:59:42 Initializing system.range_xfers\r\nINFO  18:59:43 Initializing system_distributed.parent_repair_history\r\nINFO  18:59:43 Initializing system_distributed.repair_history\r\nINFO  18:59:43 Initializing system_auth.role_permissions\r\nINFO  18:59:43 Initializing system_auth.resource_role_permissons_index\r\nINFO  18:59:43 Initializing system_auth.roles\r\nINFO  18:59:43 Initializing system_auth.role_members\r\nINFO  18:59:43 Initializing system_traces.sessions\r\nINFO  18:59:43 Initializing system_traces.events\r\nINFO  18:59:43 Completed loading (104 ms; 7 keys) KeyCache cache\r\nINFO  18:59:43 Replaying /var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/volume/commitlog/CommitLog-5-1462561114318.log, /var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/volume/commitlog/CommitLog-5-1462561114319.log\r\nINFO  18:59:46 Log replay complete, 15 replayed mutations\r\nINFO  18:59:46 Cassandra version: 2.2.5\r\nINFO  18:59:46 Thrift API version: 20.1.0\r\nINFO  18:59:46 CQL supported versions: 3.3.1 (default: 3.3.1)\r\nINFO  18:59:47 Initializing index summary manager with a memory pool size of 101 MB and a resize interval of 60 minutes\r\nINFO  18:59:47 Loading persisted ring state\r\nINFO  18:59:47 Retrieved response {\"isSeed\":false,\"seeds\":[\"10.0.1.121\",\"10.0.1.123\"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds\r\nINFO  18:59:47 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]\r\nINFO  18:59:47 Starting Messaging Service on /10.0.2.66:7000 (eth0)\r\nINFO  18:59:47 Retrieved response {\"isSeed\":false,\"seeds\":[\"10.0.1.121\",\"10.0.1.123\"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds\r\nINFO  18:59:47 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]\r\nINFO  18:59:47 Handshaking version with /10.0.1.123\r\nINFO  18:59:47 Handshaking version with /10.0.1.121\r\nINFO  18:59:52 Node /10.0.1.121 has restarted, now UP\r\nINFO  18:59:52 Node /10.0.1.122 has restarted, now UP\r\nINFO  18:59:52 InetAddress /10.0.1.122 is now DOWN\r\nINFO  18:59:52 Node /10.0.1.123 has restarted, now UP\r\nINFO  18:59:53 Starting up server gossip\r\nINFO  18:59:54 Retrieved response {\"isSeed\":false,\"seeds\":[\"10.0.1.121\",\"10.0.1.123\"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds\r\nINFO  18:59:54 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]\r\nINFO  18:59:54 Retrieved response {\"isSeed\":false,\"seeds\":[\"10.0.1.121\",\"10.0.1.123\"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds\r\nINFO  18:59:54 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]\r\nINFO  18:59:54 Retrieved response {\"isSeed\":false,\"seeds\":[\"10.0.1.121\",\"10.0.1.123\"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds\r\nINFO  18:59:54 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]\r\nINFO  18:59:54 Retrieved response {\"isSeed\":false,\"seeds\":[\"10.0.1.121\",\"10.0.1.123\"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds\r\nINFO  18:59:54 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]\r\nWARN  18:59:54 Detected previous bootstrap failure; retrying\r\nINFO  18:59:54 JOINING: waiting for ring information\r\nINFO  18:59:54 JOINING: schema complete, ready to bootstrap\r\nINFO  18:59:54 JOINING: waiting for pending range calculation\r\nINFO  18:59:54 JOINING: calculation complete, ready to bootstrap\r\nINFO  18:59:54 JOINING: getting bootstrap token\r\nINFO  18:59:55 Handshaking version with /10.0.1.121\r\nINFO  18:59:55 Node /10.0.1.121 is now part of the cluster\r\nINFO  18:59:55 Node /10.0.1.121 state jump to NORMAL\r\nINFO  18:59:56 JOINING: sleeping 30000 ms for pending range setup\r\nWARN  18:59:56 Not marking nodes down due to local pause of 16496601209 > 5000000000\r\nINFO  18:59:56 Updating topology for /10.0.1.121\r\nINFO  18:59:56 Updating topology for /10.0.1.121\r\nINFO  18:59:56 Node /10.0.1.122 is now part of the cluster\r\nINFO  18:59:56 Node /10.0.1.122 state jump to shutdown\r\nINFO  18:59:56 Updating topology for /10.0.1.122\r\nINFO  18:59:56 Updating topology for /10.0.1.122\r\nINFO  18:59:56 InetAddress /10.0.1.122 is now DOWN\r\nINFO  18:59:56 Node /10.0.1.123 is now part of the cluster\r\nINFO  18:59:56 Node /10.0.1.123 state jump to NORMAL\r\nINFO  18:59:56 Updating topology for /10.0.1.123\r\nINFO  18:59:56 Updating topology for /10.0.1.123\r\nINFO  18:59:56 InetAddress /10.0.1.121 is now UP\r\nINFO  18:59:56 InetAddress /10.0.1.121 is now UP\r\nINFO  18:59:56 Handshaking version with /10.0.1.123\r\nINFO  18:59:56 InetAddress /10.0.1.123 is now UP\r\nINFO  19:00:26 JOINING: Starting to bootstrap...\r\nException (java.lang.RuntimeException) encountered during startup: A node required to move the data consistently is down (/10.0.1.122). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false\r\nERROR 19:00:26 Exception encountered during startup\r\njava.lang.RuntimeException: A node required to move the data consistently is down (/10.0.1.122). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false\r\n\tat org.apache.cassandra.dht.RangeStreamer.getAllRangesWithStrictSourcesFor(RangeStreamer.java:264) ~[apache-cassandra-2.2.5.jar:2.2.5]\r\n\tat org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:147) ~[apache-cassandra-2.2.5.jar:2.2.5]\r\n\tat org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:82) ~[apache-cassandra-2.2.5.jar:2.2.5]\r\n\tat org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1230) ~[apache-cassandra-2.2.5.jar:2.2.5]\r\n\tat org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:924) ~[apache-cassandra-2.2.5.jar:2.2.5]\r\n\tat org.apache.cassandra.service.StorageService.initServer(StorageService.java:709) ~[apache-cassandra-2.2.5.jar:2.2.5]\r\n\tat org.apache.cassandra.service.StorageService.initServer(StorageService.java:585) ~[apache-cassandra-2.2.5.jar:2.2.5]\r\n\tat org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:300) [apache-cassandra-2.2.5.jar:2.2.5]\r\n\tat org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [apache-cassandra-2.2.5.jar:2.2.5]\r\n\tat org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625) [apache-cassandra-2.2.5.jar:2.2.5]\r\nWARN  19:00:26 No local state or state is in silent shutdown, not announcing shutdown\r\n```"},"comment":{"url":"https://api.github.com/repos/mesosphere/dcos-cassandra-service/issues/comments/226848516","html_url":"https://github.com/mesosphere/dcos-cassandra-service/issues/121#issuecomment-226848516","issue_url":"https://api.github.com/repos/mesosphere/dcos-cassandra-service/issues/121","id":226848516,"user":{"login":"erikriverson","id":122208,"avatar_url":"https://avatars.githubusercontent.com/u/122208?v=3","gravatar_id":"","url":"https://api.github.com/users/erikriverson","html_url":"https://github.com/erikriverson","followers_url":"https://api.github.com/users/erikriverson/followers","following_url":"https://api.github.com/users/erikriverson/following{/other_user}","gists_url":"https://api.github.com/users/erikriverson/gists{/gist_id}","starred_url":"https://api.github.com/users/erikriverson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erikriverson/subscriptions","organizations_url":"https://api.github.com/users/erikriverson/orgs","repos_url":"https://api.github.com/users/erikriverson/repos","events_url":"https://api.github.com/users/erikriverson/events{/privacy}","received_events_url":"https://api.github.com/users/erikriverson/received_events","type":"User","site_admin":false},"created_at":"2016-06-17T18:39:09Z","updated_at":"2016-06-17T18:39:09Z","body":"@brndnmtthws Were you ever able to make progress with this? We are seeing same error message. "}}
{"actor":{"display_login":"erikriverson"}}