title: Adding a field to differentiate parts of tables
body: <p>I have several gigabites of arducopter binary flight logs. Each log is a series of messages. </p>
  
  <pre><code>MessageType1: param1, param2, param3
  MessageType2: param3, param4, param5, param6
  ...
  </code></pre>
  
  <p>The logs are self describing in the sense that the first time a message appears in the log it tells what are the names of the params.</p>
  
  <pre><code>MessageType1: timestamp, a, b
  MessageType1: value 1, value 2, value 3
  MessageType2: timestamp, c, d, e
  MessageType1: value 4, value 5, value 6
  MessageType1: value 7, value 8, value 9
  MessageType2: value 10, value 11, value 12, value 13
  </code></pre>
  
  <p>I have written a python script that takes the logs apart and creates tables for each message type in a sqlite database where the message type is the table name and the parameter name is the column name. </p>
  
  <p>Table MessageType1</p>
  
  <pre><code>| Flight Index | Timestamp |   a   |    b    |
  |--------------|-----------|-------|---------|
  | ...          |           |       |         |
  | "Flight 1"   |       111 | 14725 | 10656.0 |
  | "Flight 1"   |       112 | 57643 | 10674.0 |
  | "Flight 1"   |       113 | 57157 | 13674.0 |
  | ...          |           |       |         |
  | "Flight 2"   |       111 | 56434 | 16543.7 |
  | "Flight 2"   |       112 | 56434 | 16543.7 |
  </code></pre>
  
  <p>Table MessageType2</p>
  
  <pre><code>| Flight Index | Timestamp |   c   |    d    |   e    |
  |--------------|-----------|-------|---------|--------|
  | ...          |           |       |         |        |
  | "Flight 1"   |       111 | 14725 | 10656.0 | 462642 |
  | "Flight 1"   |       112 | 57643 | 10674.0 | 426428 |
  | "Flight 1"   |       113 | 57157 | 13674.0 | 642035 |
  | ...          |           |       |         |        |
  | "Flight 2"   |       111 | 56434 | 16543.7 | 365454 |
  | "Flight 2"   |       112 | 56434 | 16543.7 | 754632 |
  | ...          |           |       |         |        |
  </code></pre>
  
  <p>For a single log this database is good enough but i would like to add several logs. Meaning messages of several logs of same type go into a single table.</p>
  
  <p>In this case I added a column "Flight Index" which is what I would like to have but: </p>
  
  <ul>
  <li>Each log processed should have a unique identifier</li>
  <li>The identifier should be minimal in size, as im dealing with tables that have possibly millions of rows.</li>
  </ul>
  
  <p>Im thinking of adding the flight index as an integer and just iterating the number when processing logs and if the database exists taking the last row of a table and using its index + 1. Is this optimal or is there a SQL native way of operating?</p>
  
  <p>Am i doing something wrong in general as I'm not experienced with SQL?</p>
  
  <p>EDIT: added a second table to show that messages dont have the same number of parameters and example messages.</p>