title: Timestamp Processing Brain Teaser
body: <p>I am processing 1Hz timestamps (variable 'timestamp_1hz') from a logger which doesn't log exactly at the same time every second (the difference varies from 0.984 to 1.094, but sometimes 0.5 or several seconds if the logger burps).  The 1Hz dataset is used to build a 10 minute averaged dataset, and each 10 minute interval must have 600 records.  Because the logger doesn't log exactly at the same time every second, the timestamp slowly drifts through the 1 second mark.  Issues come up when the timestamp cross the 0 mark, as well as the 0.5 mark.</p>
  
  <p>I have tried various ways to pre-process the timestamps.  The timestamps with around 1 second between them should be considered valid.  A few examples include:</p>
  
  <pre><code>% simple
  % this screws up around half second and full second values    
  rawseconds = raw_1hz_new(:,6)+(raw_1hz_new(:,7)./1000);
  rawsecondstest = rawseconds;
      rawsecondstest(:,1) = floor(rawseconds(:,1))+ rawseconds(1,1);
  
  % more complicated
  % this screws up if there is missing data, then the issue compounds because k+1 timestamp is dependent on k timestamp
  rawseconds = raw_1hz_new(:,6)+(raw_1hz_new(:,7)./1000);
   A = diff(rawseconds);
      numcheck = rawseconds(1,1);
      integ = floor(numcheck);
      fract = numcheck-integ;
      if fract&gt;0.5
          rawseconds(1,1) = rawseconds(1,1)-0.5;
      end
   for k=2:length(rawseconds)
          rawsecondstest(k,1) =  rawsecondstest(k-1,1)+round(A(k-1,1)); 
  end
  </code></pre>
  
  <p>I would like to pre-process the timestamps then compare it to a contiguous 1Hz timestamp using 'intersect' in order to find the missing, repeating, etc data such as this:</p>
  
  <pre><code> % pull out the time stamp (round to 1hz and convert to serial number)
  timestamp_1hz=round((datenum(raw_1hz_new(:,[1:6])))*86400)/86400;
  
  % calculate new start time and end time to find contig time
  starttime=min(timestamp_1hz);
  endtime=max(timestamp_1hz);
  
  % determine the contig time
  contigtime=round([floor(mean([starttime endtime])):1/86400:ceil(mean([starttime endtime]))-1/86400]'*86400)/86400;
  % find indices where logger time stamp matches real time and puts
  % the indices of a and b
  clear Ia Ib Ic Id
  [~,Ia,Ib]=intersect(timestamp_1hz,contigtime);
  % find indices where there is a value in real time that is not in
  % logger time
  [~,Ic] = setdiff(contigtime,timestamp_1hz);
  % finds the indices that are unique
  [~,Id] = unique(timestamp_1hz);
  </code></pre>
  
  <p>You can download 10 days of the raw_1hz_new timestamps <a href="https://docs.google.com/file/d/0B9EIozHd9086c1BWOXprbnJSNjg/edit?usp=sharing" rel="nofollow">here</a>.  Any help or tips would be much appreciated!</p>