title: Issues migrating to Apache Cassandra 3.11.3 from DSE 5.0.9
body: <p>We are looking at migrating from DSE 5.0.9 to Apache Cassandra 3.11.3. We've gotten quite far and managed to fix various issues (including the EverywhereStrategy one) but are running into an issue with the system.local table.</p>
  
  <p>The migration/upgrade was done on just one server, so far. When we start Cassandra 3.11.3 on this one node we get an error when loading system.local:</p>
  
  <pre><code>INFO [main] 2018-12-07 10:56:12,963 ColumnFamilyStore.java:411 - Initializing system.local
  INFO [SSTableBatchOpen:1] 2018-12-07 10:56:12,993 BufferPool.java:230 - Global buffer pool is enabled, when pool is exhausted (max is 512.000MiB) it will allocate on heap
  ERROR [SSTableBatchOpen:1] 2018-12-07 10:56:13,013 DebuggableThreadPoolExecutor.java:239 - Error in ThreadPoolExecutor
  java.lang.RuntimeException: Unknown column server_id during deserialization
  at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:321) ~[apache-cassandra-3.11.3.jar:3.11.3]
  at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:522) ~[apache-cassandra-3.11.3.jar:3.11.3]
  at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:385) ~[apache-cassandra-3.11.3.jar:3.11.3]
  at org.apache.cassandra.io.sstable.format.SSTableReader$3.run(SSTableReader.java:570) ~[apache-cassandra-3.11.3.jar:3.11.3]
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_172]
  at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_172]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_172]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_172]
  at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3.jar:3.11.3]
  at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_172]
  </code></pre>
  
  <p>Looking at another Cassandra 3.11.3 cluster we have here, system_id doesn't exist in the table. It does, however, in the DSE 5.0.9 version of the table.
  Without being able to load system.local, we end up then getting the following warning:</p>
  
  <pre><code>WARN [main] 2018-12-06 10:43:57,241 SystemKeyspace.java:1087 - No host ID found, created a0bb8c11-2864-4d58-9c0c-59b97b16c48e (Note: This should happen exactly once per node).
  </code></pre>
  
  <p>(there is no host ID as system.local didn't load)
  which then causes for the following error:</p>
  
  <pre><code>ERROR [main] 2018-12-06 10:43:58,295 CassandraDaemon.java:708 - Exception encountered during startup
  java.lang.RuntimeException: A node with address dubdc1-oatjeeramp2dmcassandra-04/10.109.158.254 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
  at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:558) ~[apache-cassandra-3.11.3.jar:3.11.3]
  at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:804) ~[apache-cassandra-3.11.3.jar:3.11.3]
  at org.apache.cassandra.service.StorageService.initServer(StorageService.java:664) ~[apache-cassandra-3.11.3.jar:3.11.3]
  at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613) ~[apache-cassandra-3.11.3.jar:3.11.3]
  at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:379) [apache-cassandra-3.11.3.jar:3.11.3]
  at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) [apache-cassandra-3.11.3.jar:3.11.3]
  at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.3.jar:3.11.3]
  </code></pre>
  
  <p>At this point system.local has been overwritten and the new host ID value stored, and Cassandra has shutdown.</p>
  
  <p>Adding <code>-Dcassandra.replace_node=&lt;ip address&gt;</code> to cassandra-env.sh results in an error saying that the node has already been bootstrapped so can't be used. I know I can get around this by deleting all of the data, but I really don't want to have to do that.</p>
  
  <p>Recovering a backup of system.local will allow us to start up DSE again. Currently the node is back running DSE5.0.9</p>
  
  <p>Has anyone seen this issue before, and do you have any advice on how to resolve it?</p>