title: How can numpy be so much faster than my Fortran routine?
body: <p>I get a 512^3 array representing a Temperature distribution from a simulation (written in Fortran). The array is stored in a binary file that's about 1/2G in size. I need to know the minimum, maximum and mean of this array and as I will soon need to understand Fortran code anyway, I decided to give it a go and came up with the following very easy routine.</p>
  
  <pre><code>  integer gridsize,unit,j
    real mini,maxi
    double precision mean
  
    gridsize=512
    unit=40
    open(unit=unit,file='T.out',status='old',access='stream',&amp;
         form='unformatted',action='read')
    read(unit=unit) tmp
    mini=tmp
    maxi=tmp
    mean=tmp
    do j=2,gridsize**3
        read(unit=unit) tmp
        if(tmp&gt;maxi)then
            maxi=tmp
        elseif(tmp&lt;mini)then
            mini=tmp
        end if
        mean=mean+tmp
    end do
    mean=mean/gridsize**3
    close(unit=unit)
  </code></pre>
  
  <p>This takes about 25 seconds per file on the machine I use. That struck me as being rather long and so I went ahead and did the following in Python:</p>
  
  <pre><code>    import numpy
  
      mmap=numpy.memmap('T.out',dtype='float32',mode='r',offset=4,\
                                    shape=(512,512,512),order='F')
      mini=numpy.amin(mmap)
      maxi=numpy.amax(mmap)
      mean=numpy.mean(mmap)
  </code></pre>
  
  <p>Now, I expected this to be faster of course, but I was really blown away. It takes less than a second under identical conditions. The mean deviates from the one my Fortran routine finds (which I also ran with 128-bit floats, so I somehow trust it more) but only on the 7th significant digit or so.</p>
  
  <p>How can numpy be so fast? I mean you have to look at every entry of an array to find these values, right? Am I doing something very stupid in my Fortran routine for it to take so much longer?</p>
  
  <p><strong>EDIT:</strong></p>
  
  <p>To answer the questions in the comments:</p>
  
  <ul>
  <li>Yes, also I ran the Fortran routine with 32-bit and 64-bit floats but it had no impact on performance.</li>
  <li>I used <a href="https://gcc.gnu.org/onlinedocs/gfortran/ISO_005fFORTRAN_005fENV.html" rel="noreferrer"><code>iso_fortran_env</code></a> which provides 128-bit floats.</li>
  <li>Using 32-bit floats my mean is off quite a bit though, so precision is really an issue.</li>
  <li>I ran both routines on different files in different order, so the caching should have been fair in the comparison I guess ?</li>
  <li>I actually tried open MP, but to read from the file at different positions at the same time. Having read your comments and answers this sounds really stupid now and it made the routine take a lot longer as well. I might give it a try on the array operations but maybe that won't even be necessary.</li>
  <li>The files are actually 1/2G in size, that was a typo, Thanks.</li>
  <li>I will try the array implementation now.</li>
  </ul>
  
  <p><strong>EDIT 2:</strong></p>
  
  <p>I implemented what @Alexander Vogt and @casey suggested in their answers, and it is as fast as <code>numpy</code> but now I have a precision problem as @Luaan pointed out I might get. Using a 32-bit float array the mean computed by <code>sum</code> is 20% off. Doing</p>
  
  <pre><code>...
  real,allocatable :: tmp (:,:,:)
  double precision,allocatable :: tmp2(:,:,:)
  ...
  tmp2=tmp
  mean=sum(tmp2)/size(tmp)
  ...
  </code></pre>
  
  <p>Solves the issue but increases computing time (not by very much, but noticeably).
  Is there a better way to get around this issue? I couldn't find a way to read singles from the file directly to doubles.
  And how does <code>numpy</code> avoid this?</p>
  
  <p>Thanks for all the help so far.</p>