title: XML Schema for scientific instrumentation time-series logging
body: <h1>Q:</h1>
  
  <p>I know there is not one perfect answer to all of this nonsense; I am hoping for some experienced insight to narrow down the possible flavors, some general strategy to avoid conversion nightmares, and any ideas on reducing my data-storage footprint on the CPU/disk (large string operations are expensive and tedious). I am on restricted hardware, and somewhat new to XML <em>standards</em>. I can read and write it just fine (usually for website), never really as a dataset encapsulation.</p>
  
  <hr>
  
  <p>I have given this weeks of thought, and I am 92.3% sure that XML files are my ideal storage destination. I am logging various instrumentation readings/analysis, and holding it for months at a time. Although I do have concerns about my data-collection nodes having limited hardware resources (Excessive string operations can get slow, 512kB RAM, 3.2GB flash storage). </p>
  
  <p>I am trying to find a well formed ML with a minimal footprint that can handle <em>RAW</em> numerical datatypes. I do not need fully compliant files, BUT I am looking for a best-fit solution, so lets not deviate too far from proper form</p>
  
  <h2>Primary Data Model Factors</h2>
  
  <p><em>(and why I think XML is a better fit that Packed binary, FLAT TEXT, or even CSV)</em> </p>
  
  <ul>
  <li>Up to 8 different datapoints (different measurements, brands, and sensor types)</li>
  <li>various raw datatypes (REAL32, DINT, DWORD, BYTE, STRING(arbitrarily long)</li>
  <li>datasets need to be able to keep absolute timestamps within each file (I have a directory full of 100's of XML's that will eventually merge)</li>
  <li>datapoint configuration/quantity could change, so I need to be able to note alterations to the schema with minimal verbosity/confusion.</li>
  </ul>
  
  <h2>Performance Constraints/Considerations</h2>
  
  <ul>
  <li>I should normally only write out the XML from the embedded platform, so readability is not paramount, although if I do need to handle any kind of <em>inquiry</em>, tossing and parsing 3.0GB of text is not going to be fun even at its very cleanest.
  <ul>
  <li>I believe that intermittent DATE-TIME nodes will help me index such an inquiry</li>
  </ul></li>
  <li><em>Compressing</em> data excessively can actually become a problem at export time, because those become yet more calculations to <em>unzip</em> my laziness.</li>
  <li>Excessively verbose XML only gives me 111 days of storage. I would like to get that up to 180 days or longer. So I do need to <strong>condense</strong> text better.</li>
  <li>There are 3 potential targets once the data is offloaded. I don't want to run into conversion bottlenecks/mistakes by over-complicating.
  <ul>
  <li>Microsoft Excel (he doesn't have to understand it perfectly, but we don't want to spend hours manually importing non-compliant schema types/maps into a 2D grid.</li>
  <li>RRD Backend Server (I will be able to run any conversions needed, but hopefully I am already close to what RRD wants</li>
  <li>Some cute Javascript/Android tools. Although I expect these to perform custom datatype handling, well-formed XML will make retrieval and parsing simpler during development.</li>
  </ul></li>
  </ul>