<p dir="auto">The <code class="notranslate">.expanding()</code> function is super cool. It's order of magnitude faster than<br>
the obvious <code class="notranslate">numpy</code> alternative:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy
import pandas
numpy.random.seed(123)
df = pandas.DataFrame((numpy.random.normal(0, 1, [10000, 2])), columns=['Value', 'Funny'])
%timeit method_2 = df.Value.expanding(0).apply(lambda x: x.sum())
%timeit method_1 = numpy.array([df.Value.iloc[range(j + 1)].sum() for j in range(df.shape[0])])"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span>
<span class="pl-k">import</span> <span class="pl-s1">pandas</span>
<span class="pl-s1">numpy</span>.<span class="pl-s1">random</span>.<span class="pl-en">seed</span>(<span class="pl-c1">123</span>)
<span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pandas</span>.<span class="pl-v">DataFrame</span>((<span class="pl-s1">numpy</span>.<span class="pl-s1">random</span>.<span class="pl-en">normal</span>(<span class="pl-c1">0</span>, <span class="pl-c1">1</span>, [<span class="pl-c1">10000</span>, <span class="pl-c1">2</span>])), <span class="pl-s1">columns</span><span class="pl-c1">=</span>[<span class="pl-s">'Value'</span>, <span class="pl-s">'Funny'</span>])
<span class="pl-c1">%</span><span class="pl-s1">timeit</span> <span class="pl-s1">method_2</span> <span class="pl-c1">=</span> <span class="pl-s1">df</span>.<span class="pl-v">Value</span>.<span class="pl-en">expanding</span>(<span class="pl-c1">0</span>).<span class="pl-en">apply</span>(<span class="pl-k">lambda</span> <span class="pl-s1">x</span>: <span class="pl-s1">x</span>.<span class="pl-en">sum</span>())
<span class="pl-c1">%</span><span class="pl-s1">timeit</span> <span class="pl-s1">method_1</span> <span class="pl-c1">=</span> <span class="pl-s1">numpy</span>.<span class="pl-en">array</span>([<span class="pl-s1">df</span>.<span class="pl-v">Value</span>.<span class="pl-s1">iloc</span>[<span class="pl-en">range</span>(<span class="pl-s1">j</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>)].<span class="pl-en">sum</span>() <span class="pl-k">for</span> <span class="pl-s1">j</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-s1">df</span>.<span class="pl-s1">shape</span>[<span class="pl-c1">0</span>])])</pre></div>
<p dir="auto">But it only works for univariate functions. Would it be possible to extend it to multiple columns while keeping the time efficiency? The idea would be to be able to compute functions involving several columns on expanding window of the data. Say something like:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def loc_fun(x, y):
    return numpy.linalg.det(numpy.cov([x, y]))

df.expanding(3).apply(lambda x: loc_fun(x.Value, x.Funny))"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">loc_fun</span>(<span class="pl-s1">x</span>, <span class="pl-s1">y</span>):
    <span class="pl-k">return</span> <span class="pl-s1">numpy</span>.<span class="pl-s1">linalg</span>.<span class="pl-en">det</span>(<span class="pl-s1">numpy</span>.<span class="pl-en">cov</span>([<span class="pl-s1">x</span>, <span class="pl-s1">y</span>]))

<span class="pl-s1">df</span>.<span class="pl-en">expanding</span>(<span class="pl-c1">3</span>).<span class="pl-en">apply</span>(<span class="pl-k">lambda</span> <span class="pl-s1">x</span>: <span class="pl-en">loc_fun</span>(<span class="pl-s1">x</span>.<span class="pl-v">Value</span>, <span class="pl-s1">x</span>.<span class="pl-v">Funny</span>))</pre></div>
<p dir="auto">faster than with <code class="notranslate">numpy</code> and list comprehension.</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">The result of say:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="numpy.array([loc_fun(df.Value.iloc[range(j + 1)],df.Funny.iloc[range(j + 1)]) for j in range(3, df.shape[0])])"><pre class="notranslate"><span class="pl-s1">numpy</span>.<span class="pl-en">array</span>([<span class="pl-en">loc_fun</span>(<span class="pl-s1">df</span>.<span class="pl-v">Value</span>.<span class="pl-s1">iloc</span>[<span class="pl-en">range</span>(<span class="pl-s1">j</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>)],<span class="pl-s1">df</span>.<span class="pl-v">Funny</span>.<span class="pl-s1">iloc</span>[<span class="pl-en">range</span>(<span class="pl-s1">j</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>)]) <span class="pl-k">for</span> <span class="pl-s1">j</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">3</span>, <span class="pl-s1">df</span>.<span class="pl-s1">shape</span>[<span class="pl-c1">0</span>])])</pre></div>
<p dir="auto">...but with the nicer speed;)</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-32-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: None
pip: 9.0.1
setuptools: 38.2.5
Cython: None
numpy: 1.13.3
scipy: None
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.6.5
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 0.7.3
lxml: None
bs4: None
html5lib: 1.0b10
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None"><pre class="notranslate"><code class="notranslate">INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-32-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: None
pip: 9.0.1
setuptools: 38.2.5
Cython: None
numpy: 1.13.3
scipy: None
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.6.5
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 0.7.3
lxml: None
bs4: None
html5lib: 1.0b10
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</code></pre></div>
<details>
<p dir="auto">[paste the output of <code class="notranslate">pd.show_versions()</code> here below this line]</p>
</details>
<p dir="auto">In <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="116942772" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/11603" data-hovercard-type="pull_request" data-hovercard-url="/pandas-dev/pandas/pull/11603/hovercard?comment_id=162113949&amp;comment_type=issue_comment" href="https://github.com/pandas-dev/pandas/pull/11603#issuecomment-162113949">#11603 (comment)</a> (the main PR implementing the deferred API for rolling / expanding / ewm), we discussed how to specify table-wise <code class="notranslate">apply</code>s. <code class="notranslate">Groupby.apply(f)</code> feeds the entire group (all columns) to <code class="notranslate">f</code>. For backwards-compatibility, <code class="notranslate">.rolling(n).apply(f)</code> needed to be column-wise.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="116942772" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/11603" data-hovercard-type="pull_request" data-hovercard-url="/pandas-dev/pandas/pull/11603/hovercard?comment_id=162116556&amp;comment_type=issue_comment" href="https://github.com/pandas-dev/pandas/pull/11603#issuecomment-162116556">#11603 (comment)</a> mentions a possible API like what I added for <code class="notranslate">.style</code></p>
<ul dir="auto">
<li><code class="notranslate">axis=0</code>: apply to each column independently</li>
<li><code class="notranslate">axis=1</code>: apply to each row independently</li>
<li><code class="notranslate">axis=None</code>: apply the supplied function to the entire table</li>
</ul>
<p dir="auto">So it'd be <code class="notranslate">df.rolling(n).apply(f, axis=None)</code>.<br>
Do people like the axis=0 / 1 / None idiom? Is it obvious enough?</p>
<p dir="auto">This is prompted by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/josef-pkt/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/josef-pkt">@josef-pkt</a>'s <a href="https://groups.google.com/forum/#!topic/pydata/FcAT8LBPmlg" rel="nofollow">post on the mailinglist</a>. Needing a rolling OLS.</p>
<p dir="auto">An example:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [2]: import numpy as np
   ...: import pandas as pd
   ...:
   ...: np.random.seed(0)
   ...: df = pd.DataFrame(np.random.randint(0, 10, size=(10, 2)), columns=[&quot;A&quot;, &quot;B&quot;])
   ...: df
   ...:
Out[2]:
   A  B
0  5  0
1  3  3
2  7  9
3  3  5
4  2  4
5  7  6
6  8  8
7  1  6
8  7  7
9  8  1"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">2</span>]: <span class="pl-s1">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
   ...: <span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
   ...:
   ...: <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">seed</span>(<span class="pl-c1">0</span>)
   ...: <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">0</span>, <span class="pl-c1">10</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span>(<span class="pl-c1">10</span>, <span class="pl-c1">2</span>)), <span class="pl-s1">columns</span><span class="pl-c1">=</span>[<span class="pl-s">"A"</span>, <span class="pl-s">"B"</span>])
   ...: <span class="pl-s1">df</span>
   ...:
<span class="pl-v">Out</span>[<span class="pl-c1">2</span>]:
   <span class="pl-v">A</span>  <span class="pl-v">B</span>
<span class="pl-c1">0</span>  <span class="pl-c1">5</span>  <span class="pl-c1">0</span>
<span class="pl-c1">1</span>  <span class="pl-c1">3</span>  <span class="pl-c1">3</span>
<span class="pl-c1">2</span>  <span class="pl-c1">7</span>  <span class="pl-c1">9</span>
<span class="pl-c1">3</span>  <span class="pl-c1">3</span>  <span class="pl-c1">5</span>
<span class="pl-c1">4</span>  <span class="pl-c1">2</span>  <span class="pl-c1">4</span>
<span class="pl-c1">5</span>  <span class="pl-c1">7</span>  <span class="pl-c1">6</span>
<span class="pl-c1">6</span>  <span class="pl-c1">8</span>  <span class="pl-c1">8</span>
<span class="pl-c1">7</span>  <span class="pl-c1">1</span>  <span class="pl-c1">6</span>
<span class="pl-c1">8</span>  <span class="pl-c1">7</span>  <span class="pl-c1">7</span>
<span class="pl-c1">9</span>  <span class="pl-c1">8</span>  <span class="pl-c1">1</span></pre></div>
<p dir="auto">For a concrete example, get the table-wise max (this is equivalent to <code class="notranslate">df.rolling(4).max().max(1)</code>)</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [10]: df.rolling(4).apply(np.max, axis=None)
Out[10]:
0    NaN
1    NaN
2    NaN
3    9.0
4    9.0
5    9.0
6    8.0
7    8.0
8    8.0
9    8.0
dtype: float64"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">10</span>]: <span class="pl-s1">df</span>.<span class="pl-en">rolling</span>(<span class="pl-c1">4</span>).<span class="pl-en">apply</span>(<span class="pl-s1">np</span>.<span class="pl-s1">max</span>, <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">None</span>)
<span class="pl-v">Out</span>[<span class="pl-c1">10</span>]:
<span class="pl-c1">0</span>    <span class="pl-v">NaN</span>
<span class="pl-c1">1</span>    <span class="pl-v">NaN</span>
<span class="pl-c1">2</span>    <span class="pl-v">NaN</span>
<span class="pl-c1">3</span>    <span class="pl-c1">9.0</span>
<span class="pl-c1">4</span>    <span class="pl-c1">9.0</span>
<span class="pl-c1">5</span>    <span class="pl-c1">9.0</span>
<span class="pl-c1">6</span>    <span class="pl-c1">8.0</span>
<span class="pl-c1">7</span>    <span class="pl-c1">8.0</span>
<span class="pl-c1">8</span>    <span class="pl-c1">8.0</span>
<span class="pl-c1">9</span>    <span class="pl-c1">8.0</span>
<span class="pl-s1">dtype</span>: <span class="pl-s1">float64</span></pre></div>
<p dir="auto">A real example is something like a rolling OLS:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import statsmodels.api as sm
f = lambda x: sm.OLS.from_formula('A ~ B', data=x).fit()  # wrong, but w/e

df.rolling(5).apply(f, axis=None)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">statsmodels</span>.<span class="pl-s1">api</span> <span class="pl-k">as</span> <span class="pl-s1">sm</span>
<span class="pl-s1">f</span> <span class="pl-c1">=</span> <span class="pl-k">lambda</span> <span class="pl-s1">x</span>: <span class="pl-s1">sm</span>.<span class="pl-v">OLS</span>.<span class="pl-en">from_formula</span>(<span class="pl-s">'A ~ B'</span>, <span class="pl-s1">data</span><span class="pl-c1">=</span><span class="pl-s1">x</span>).<span class="pl-en">fit</span>()  <span class="pl-c"># wrong, but w/e</span>

<span class="pl-s1">df</span>.<span class="pl-en">rolling</span>(<span class="pl-c1">5</span>).<span class="pl-en">apply</span>(<span class="pl-s1">f</span>, <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">None</span>)</pre></div>