 <p>Have you thought about bucketing by problem and stream processing the input data?</p> <p>A sample approach of this type would be:</p> <pre><code>for event in new_events: # new events to be written from the RESTful service for test in tests: # known tests you need to perform increment/decrement &lt;test.bucket&gt;.&lt;sensor&gt;.&lt;valueX&gt; # update the bucket as necessary # e.g. seconds_since_changed.sensor1.value3 += 1 # additionally, save each record to a standard database for long-term purposes # and to be able to rebuild/redo tests as necessary. </code></pre> <p>The ideal storage mechanism for such a transaction stream is one that lends itself well to low-latency writes/updates. NoSQL solutions work very well for this sort of work (I've personally used Redis for a similar type of analysis, but you're in no ways bound to that particular solution.)</p>