title: How to simulate Single precision rounding with Doubles?
body: <p>i had a problem where i was trying to reconstruct the the formula used in an existing system, a fairly simple formula of one input and one output:</p>
  
  <pre><code>y = f(x)
  </code></pre>
  
  <p>After a lot of puzzling, <a href="http://sacredduty.net/2012/09/14/avoidance-diminishing-returns-in-mop-followup/" rel="nofollow noreferrer">we</a> managed to figure out the formula that fit our observed data points:</p>
  
  <p><img src="https://i.stack.imgur.com/Mt2R3.png" alt="enter image description here"></p>
  
  <p>And as you can see our theoretical model fit observed data very well:</p>
  
  <p><img src="https://i.stack.imgur.com/56kcQ.png" alt="enter image description here"></p>
  
  <p>Except when we plot residual errors (i.e. <code>y = f(x) - actualY</code>), we see some lines appear in the residuals:</p>
  
  <p><img src="https://i.stack.imgur.com/i4kpc.png" alt="enter image description here"></p>
  
  <p>It was obvious that these lines were the result of applying some intermediate rounding in our formula, but it was not obvious <em>where</em>. Eventually it was realized that the <strong>original</strong> system (the one we're trying to reverse engineer) is storing values in an intermediate <strong><code>Decimal</code></strong> data type:</p>
  
  <ul>
  <li>with <strong>8-bit precision</strong> of the fraction</li>
  <li>using the <strong>0.5 round-up</strong> rounding model:</li>
  </ul>
  
  <p>We could <em>simulate</em> this 8-bit precision in the fraction by:</p>
  
  <pre><code>multiply by 128 (i.e. 2^8)
  apply the round
  divide by 128 (i.e. 2^8)
  </code></pre>
  
  <p>Changing our equation above into:</p>
  
  <p><img src="https://i.stack.imgur.com/Dgr5e.png" alt="enter image description here"></p>
  
  <p>This reduces the residual errors <em>significantly</em>:</p>
  
  <p><img src="https://i.stack.imgur.com/eBTp0.png" alt="enter image description here"></p>
  
  <p>Now, all of that above has no relevance to my question except:</p>
  
  <ol>
  <li>To show that simulating the numerical representation in the computer can help the model</li>
  <li>To get people's attention with pretty pictures and colors </li>
  <li>Silence critics who <a href="https://stackoverflow.com/questions/11972829/how-to-call-use-net-rijndaelmanaged-from-native-com">would</a> <a href="https://stackoverflow.com/questions/12323303/how-to-inherit-from-control-rather-than-usercontrol">refuse</a> to <a href="https://stackoverflow.com/questions/2224220/win32-how-to-make-drop-shadow-honor-non-rectangular-layered-window">contribute</a> until i <a href="https://stackoverflow.com/questions/7111618/win32-how-to-validate-credentials-against-active-directory">explain</a> <em>why</em> i'm asking my question</li>
  </ol>
  
  <hr>
  
  <p>Now i want to simulate <strong><code>Single Precision</code></strong> floating point numbers, inside a programming language (and Excel) which use <strong><code>Double Precision</code></strong> floating point numbers. i want to do this because <em>i <strong>think</em></strong> it is what's needed.</p>
  
  <p>In the above example i <strong>thought</strong> the original system was using a <em><code>Decimal data type with fixed 8-bit fractional precision using 0.5 round-up rules</code></em>. i then had to find a way to simulate that computation model with <code>Double</code> math. Now i <em>think</em> the original system is using <code>Single</code> precision math, that i want to simulate using <code>Double</code>.</p>
  
  <blockquote>
    <p>How do i simulate single-precision rounding using doubles?</p>
  </blockquote>
  
  <p>In my current model, i once again have residuals that fall into the regular linear patterns - that are a tell-tale sign of rounding:</p>
  
  <p><img src="https://i.stack.imgur.com/51rm1.png" alt="enter image description here"></p>
  
  <p>The problem is that the error becomes larger, and only visible, as my input variables become larger. i realized this is likely caused by the fact that all floating point numbers are normalized into <a href="http://en.wikipedia.org/wiki/IEEE_floating_point" rel="nofollow noreferrer">IEEE 754</a> "scientific notation".</p>
  
  <p>And even if i'm wrong, i still want to try it.</p>
  
  <p>And even if i don't want to trying it, i'm still asking the question</p>
  
  <blockquote>
    <p>How do i simulate <code>Single</code> precision rounding using <code>Doubles</code>?</p>
  </blockquote>
  
  <hr>
  
  <p>It seems to me i could still apply the concept of <em>"rounding after 8 fractional bits"</em> (although 24 bits for <code>Single</code> precision floating point), as long as i can first <em>"normalize"</em> the value. e.g.</p>
  
  <pre><code>1234567898.76543
  </code></pre>
  
  <p>needs to be converted into (something similar to):</p>
  
  <pre><code>1.23456789876543 E-09
  </code></pre>
  
  <p>Then i could apply my "round to the 24th bit" (i.e. 2^24 = 16,777,216)</p>
  
  <pre><code>floor(1.23456789876543E-09 * 16777216 + 0.5) / 16777216;
  </code></pre>
  
  <p>The problem, then, is what combination of <code>sign</code>, <code>abs</code>, <code>ln</code>, <code>exp</code> (or other functions) can i possible apply so that i can "normalize" my value, round it to the n-th binary place, then "denormalize" it?</p>
  
  <p><strong>Note</strong>: i realize IEEE representation keeps a binary <code>1</code> as the most significant bit. i might not need to duplicate that behavior in order to get correct results. So it's not a deal-breaker, nor is it cause to suggest that the entire approach is a failure.</p>
  
  <h1>See also</h1>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/8130688/how-to-emulate-single-precision-float-operations-in-php">How to emulate single precision float operations in PHP?</a></li>
  <li><a href="http://www.mathworks.cn/matlabcentral/answers/35085-simulate-single-precision-arithmetic-in-matlab" rel="nofollow noreferrer">Simulate single precision arithmetic in Matlab?</a></li>
  <li><a href="http://en.wikipedia.org/wiki/IEEE_floating_point" rel="nofollow noreferrer">IEEE 754 Floating Point</a></li>
  </ul>