user26818: I have a large set of data and it is refreshed every 30 seconds or so. However, not all data is refreshed every time. It is possible that no new data for a particular calculation isn't received for even a few hours. For this reason I need to track what data is possibly outdated, but hasn't been updated (maybe due to technical malfunctions or a sensor is down). I was thinking about creating an additional (large) 2d array to do this, and increment some sort of counter. This way if a particular spot is at 1000 this means it is old. When new data is received it can be reset to 0. However, this seems like a slow, ineffective way to do this. How could I keep track of data more efficiently?

user4143: OK, so you have three possible states for your values:

No valid data (-1, or 0xFFFF in 16-bit 2's complement)
Valid data (0 to 1000, or 0x0000 to 0x03EB in 16-bit)
Valid but "old" data.

If you first check for invalid data before thinking about "old" data (invalid data is never considered "old") then you just have two states:

Valid data (0 to 1000, or 0x0000 to 0x03EB in 16-bit)
Valid but "old" data.

As you can see the highest 6 bits of the stored value aren't used at all except for storing -1 for invalid data (hence checking for that first). So you could utilize those 6 bits to store all sorts of things. In 6 bits you can store the values 0 to 63. That could be the number of seconds since the last time you got some data.
By masking and shifting you can separate out the two values:
val = data[x] & 0x3FF;
sec = data[x] >> 10;

Every second add one to the seconds count of each and every data entry that is less than (say) 60.
val = data[x] & 0x3FF;
sec = data[x] >> 10;
if (sec < 60) sec++;
data[x] = val | (sec << 10);

Now each entry in your array knows how old it is (up to 1 minute), and if it's older than a certain threshold (30 seconds? 45 seconds? you choose) then you consider it "old" and display it as such.
You can make it easier to program by using a union/struct combination:
typedef union {
    int16_t rawval;
    struct {
        unsigned value:10;
        unsigned seconds:6;
    } __attribute__((packed));
} data_t __attribute__((packed));

If your data array is now made up of that type instead of integers:
data_t data[500];

you can now access:

data[x].rawval as the raw 16 bit signed value (to look for -1).
data[x].value as the unsigned 10 bit value from 0 to 1000.
data[x].seconds as the unsigned 6 bit value from 0 to 63.

Updating the seconds then becomes as simple as:
if (data[x].seconds < 60) data[x].seconds++;

When you store a new value using
data[x].rawval = sensor.getValue();

the seconds is automatically overwritten with 0 which resets the count for you.
And both methods give you data ageing without using a single byte more memory.

26818: <p>I have a large set of data and it is refreshed every 30 seconds or so. However, not all data is refreshed every time. It is possible that no new data for a particular calculation isn't received for even a few hours. For this reason I need to track what data is possibly outdated, but hasn't been updated (maybe due to technical malfunctions or a sensor is down). I was thinking about creating an additional (large) 2d array to do this, and increment some sort of counter. This way if a particular spot is at <code>1000</code> this means it is old. When new data is received it can be reset to <code>0</code>. However, this seems like a slow, ineffective way to do this. How could I keep track of data more efficiently?</p>
 4143: <p>OK, so you have three possible states for your values:</p>

<ol>
<li>No valid data (-1, or 0xFFFF in 16-bit 2's complement)</li>
<li>Valid data (0 to 1000, or 0x0000 to 0x03EB in 16-bit)</li>
<li>Valid but "old" data.</li>
</ol>

<p>If you first check for invalid data before thinking about "old" data (invalid data is never considered "old") then you just have two states:</p>

<ol>
<li>Valid data (0 to 1000, or 0x0000 to 0x03EB in 16-bit)</li>
<li>Valid but "old" data.</li>
</ol>

<p>As you can see the highest 6 bits of the stored value aren't used at all except for storing -1 for invalid data (hence checking for that first). So you could utilize those 6 bits to store all sorts of things. In 6 bits you can store the values 0 to 63. That could be the number of seconds since the last time you got some data.</p>

<p>By masking and shifting you can separate out the two values:</p>

<pre><code>val = data[x] &amp; 0x3FF;
sec = data[x] &gt;&gt; 10;
</code></pre>

<p>Every second add one to the seconds count of each and every data entry that is less than (say) 60.</p>

<pre><code>val = data[x] &amp; 0x3FF;
sec = data[x] &gt;&gt; 10;
if (sec &lt; 60) sec++;
data[x] = val | (sec &lt;&lt; 10);
</code></pre>

<p>Now each entry in your array knows how old it is (up to 1 minute), and if it's older than a certain threshold (30 seconds? 45 seconds? you choose) then you consider it "old" and display it as such.</p>

<p>You can make it easier to program by using a union/struct combination:</p>

<pre><code>typedef union {
    int16_t rawval;
    struct {
        unsigned value:10;
        unsigned seconds:6;
    } __attribute__((packed));
} data_t __attribute__((packed));
</code></pre>

<p>If your data array is now made up of that type instead of integers:</p>

<pre><code>data_t data[500];
</code></pre>

<p>you can now access:</p>

<ul>
<li>data[x].rawval as the raw 16 bit signed value (to look for -1).</li>
<li>data[x].value as the unsigned 10 bit value from 0 to 1000.</li>
<li>data[x].seconds as the unsigned 6 bit value from 0 to 63.</li>
</ul>

<p>Updating the seconds then becomes as simple as:</p>

<pre><code>if (data[x].seconds &lt; 60) data[x].seconds++;
</code></pre>

<p>When you store a new value using</p>

<pre><code>data[x].rawval = sensor.getValue();
</code></pre>

<p>the seconds is automatically overwritten with 0 which resets the count for you.</p>

<p>And both methods give you data ageing without using a single byte more memory.</p>
