<p>If the result won't all fit into core memory, you can put it in a memory-mapped array so that the overflow will be written to your hard disk:</p>

<pre><code>shape = (QT.shape[2],)*2
result = np.memmap('result.dat', dtype=QT.dtype, mode='w+', shape=shape)
np.dot(QT.T, QT, out=result)
</code></pre>

<p>You may also want to take a look at <a href="http://arxiv.org/pdf/1007.5510.pdf" rel="nofollow">this algorithm</a> for performing out-of-core SVD on very large arrays.</p>
improving Numpy dot performance by removing arrays copy
<p><strong>Given a matrix QT:</strong></p>

<pre><code>% ipython
Python 2.7.3
In [3]: QT.dtype
Out[3]: dtype('float64')

In [4]: QT.__class__
Out[4]: numpy.ndarray

In [5]: QT.flags
Out[5]:
      C_CONTIGUOUS : True
      F_CONTIGUOUS : False
      OWNDATA : True
      WRITEABLE : True
      ALIGNED : True
      UPDATEIFCOPY : False
</code></pre>

<p><strong>I need the results of:</strong></p>

<pre><code>QT.T * QT
</code></pre>

<p><strong>Problem:</strong>
Whenever I try to compute these matrices multiplication, the memory overflows and the code stop running. This happen because of the matrix copy numpy is doing behind.</p>

<p><strong>Tried solutions:</strong></p>

<p>First:</p>

<pre><code>Q = numpy.array(QT.T, order='C')
numpy.dot(Q, QT)
</code></pre>

<p>Second:</p>

<pre><code>QT = numpy.array(QT, order='F')
Q = numpy.array(QT.T, order='F')
numpy.dot(Q, QT)
</code></pre>

<p>Third:</p>

<pre><code>QT = numpy.matrix(QT)
QT = QT.copy('F')
Q = numpy.matrix(QT.T)
Q = Q.copy('F')
Q.dot(QT)
</code></pre>

<p>However, none of them is solving.</p>

<p><strong>Question</strong></p>

<p>How can I operate QT.T * QT without having the memory to explode?</p>

<p><strong>References</strong></p>

<p><a href="http://numpy-discussion.10968.n7.nabble.com/inplace-matrix-multiplication-td21817.html" rel="nofollow noreferrer">http://numpy-discussion.10968.n7.nabble.com/inplace-matrix-multiplication-td21817.html</a></p>

<p><a href="https://stackoverflow.com/questions/9478791/is-there-an-enhanced-numpy-scipy-dot-method">Is there an &quot;enhanced&quot; numpy/scipy dot method?</a></p>

<p><a href="https://stackoverflow.com/questions/11856293/numpy-dot-product-very-slow-using-ints">Numpy dot product very slow using ints</a></p>

<p><a href="http://www.scipy.org/PerformanceTips" rel="nofollow noreferrer">http://www.scipy.org/PerformanceTips</a></p>