title: What is the problem of evolving a data schema of a serialization framework (like ProtoBuf, Thrift, etc.) by always adding new fields?
body: <p>I'm writing a simple serialization framework. My intention is <strong>not</strong> to compete with ProtoBuf, Thrift, Avro, etc., far from that. My goal is to learn.</p>
  <p>My question is related to evolving the schema of my data objects.</p>
  <p>Let's say I have two programs, A and B, that need to exchange data and I have a data object represented by the schema below:</p>
  <pre><code>public byte[] accountUser = new byte[8]; // first field
  </code></pre>
  <p>Great! Now I want to go ahead and add an accountId field to my data object schema:</p>
  <pre><code>public byte[] accountUser = new byte[8]; // first field
  public int accountId = -1; // second field just added (NEW FIELD)
  </code></pre>
  <h2>Scenario 1:</h2>
  <p>Program A has the new schema with accountId and Program B does not.</p>
  <p>Program A sends the data object to Program B.</p>
  <p>Program B will simply read the data up to accountUser and totally ignore accountId. It knows nothing about it and it wasn't updated to use the latest data object schema with the accountId.</p>
  <p>Everything works!</p>
  <h2>Scenario 2:</h2>
  <p>Program A has the old schema without accountId and Program B has the new schema with accountId.</p>
  <p>Program A sends the data object to Program B.</p>
  <p>Program B will read the data up to accountUser and proceed to try to read the new accountId. But there is nothing more to read in the data object received. No more data after accountUser. So Program B simply assumes the default null value of -1 for the accountId and move on with its life. I will most probably have logic to deal with a -1 accountId from legacy systems still operating with the old schema.</p>
  <p>Everything works!</p>
  <p><strong>So what is really the problem of this simple approach for schema evolution?</strong> It is not perfect I know, but can't it be successfully used? I just have to assume that I will never remove any field and that I will never mess with the order of the fields. I just keep adding more fields.</p>