title: Storing large amounts of simulated time series data
body: <p>I am looking for a way to store large amounts of simulated time series data for later processing. A data point can be described by a mapping from String to a tuple of primitive data types (mostly floating point). Each data point has a time series and each time series has an id. I am looking for an efficient way to write/read this data to/from disk. The simulation roughly looks like this</p>
  
  <pre><code>Sample sample = new Sample();
  for (int i=0; i&lt;samplesize; i++) {
     State s = initialState();
     for (int t=0; t&lt;stages; t++) {
        Map&lt;String,Double&gt; data = s.getData();
        sample.add(i,t,data);
        s = s.nextState();
     }
  }
  </code></pre>
  
  <p>To implement the Sample class, I have so far tested the following</p>
  
  <ul>
  <li>Nested lists / key-value map: Quite straightforward, but keeping all data in memory and then serializing everything to disk does not scale well.</li>
  <li>MapDB: I have tested two variants, (a) using the time series id as key and a time series as value, (b) using a flat index of time series id and time index as key and a data point as value. I am a little worried about the performance of (b), but (a) worked reasonably well.</li>
  </ul>
  
  <p>I have also considered simply writing everything into a CSV. The upside of this is that virtually anybody could import the data later on. The downside is that searching for a particular time series or record with an arbitrary id would require reading the file line by line until the record is found.</p>