<p dir="auto">Caching (memoizing) the date_parser function in read_csv might be an easy perf improvement. Seems it is not cached unless I am missing something?</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [55]: df = pd.DataFrame([datetime.datetime.today()] * 1000000)

In [56]: df.to_csv('j', index=False)

In [57]: !gzip j

In [58]: %time df = pd.read_csv('j.gz')
CPU times: user 703 ms, sys: 68.7 ms, total: 772 ms
Wall time: 774 ms

In [59]: d = {df['0'][0]: datetime.datetime.today()}

In [60]: %time s = df['0'].map(d)
CPU times: user 84.8 ms, sys: 14.8 ms, total: 99.6 ms
Wall time: 99.2 ms

In [61]: %time df = pd.read_csv('j.gz', parse_dates=['0'])
CPU times: user 1.49 s, sys: 88.7 ms, total: 1.58 s
Wall time: 1.58 s"><pre class="notranslate"><code class="notranslate">In [55]: df = pd.DataFrame([datetime.datetime.today()] * 1000000)

In [56]: df.to_csv('j', index=False)

In [57]: !gzip j

In [58]: %time df = pd.read_csv('j.gz')
CPU times: user 703 ms, sys: 68.7 ms, total: 772 ms
Wall time: 774 ms

In [59]: d = {df['0'][0]: datetime.datetime.today()}

In [60]: %time s = df['0'].map(d)
CPU times: user 84.8 ms, sys: 14.8 ms, total: 99.6 ms
Wall time: 99.2 ms

In [61]: %time df = pd.read_csv('j.gz', parse_dates=['0'])
CPU times: user 1.49 s, sys: 88.7 ms, total: 1.58 s
Wall time: 1.58 s
</code></pre></div>
<p dir="auto">I'll propose a</p>
<p dir="auto"><code class="notranslate">cache_datetime=False</code> keyword as an addition to <code class="notranslate">read_csv</code> and <code class="notranslate">pd.to_datetime</code></p>
<p dir="auto">this would use a lookup cache (a dict will probably work), to map datetime strings to Timestamp objects. For repeated dates this will lead to some dramatic speedups.</p>
<p dir="auto">Care must be taken if a <code class="notranslate">format</code> kw is provided (in <code class="notranslate">to_datetime</code> as the cache will have to be exposed). This would be an optional (and default <code class="notranslate">False</code>) as I think if you have unique dates this could modestly slow down things (but can be revisted if needed).</p>
<p dir="auto">This <em>might</em> need also want to accept a list of column names (like <code class="notranslate">parse_dates</code>) to enable per-column caching (e.g. you might want to apply to a column, but not the index of example).</p>
<p dir="auto">possibly we could overload <code class="notranslate">parse_dates='cache'</code> to mean this as well</p>
<p dir="auto">trivial example</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [1]: pd.DataFrame({'A' : ['20130101 00:00:00']*10000}).to_csv('test.csv',index=True)

In [14]: def parser(x):
   ....:         uniques = pd.Series(pd.unique(x))
   ....:         d = pd.to_datetime(uniques)
   ....:         d.index = uniques
   ....:         return Series(x).map(d).values
   ....: 
In [3]: df1 = pd.read_csv('test.csv',index_col=0, parse_dates=['A'])

In [4]: df2 = pd.read_csv('test.csv',index_col=0, parse_dates=['A'], date_parser=parser)

In [17]: %timeit pd.read_csv('test.csv',index_col=0, parse_dates=['A'])
1 loops, best of 3: 969 ms per loop

In [18]: %timeit pd.read_csv('test.csv',index_col=0, parse_dates=['A'], date_parser=parser)
100 loops, best of 3: 5.31 ms per loop

In [7]: df1.equals(df2)
Out[7]: True"><pre class="notranslate"><code class="notranslate">In [1]: pd.DataFrame({'A' : ['20130101 00:00:00']*10000}).to_csv('test.csv',index=True)

In [14]: def parser(x):
   ....:         uniques = pd.Series(pd.unique(x))
   ....:         d = pd.to_datetime(uniques)
   ....:         d.index = uniques
   ....:         return Series(x).map(d).values
   ....: 
In [3]: df1 = pd.read_csv('test.csv',index_col=0, parse_dates=['A'])

In [4]: df2 = pd.read_csv('test.csv',index_col=0, parse_dates=['A'], date_parser=parser)

In [17]: %timeit pd.read_csv('test.csv',index_col=0, parse_dates=['A'])
1 loops, best of 3: 969 ms per loop

In [18]: %timeit pd.read_csv('test.csv',index_col=0, parse_dates=['A'], date_parser=parser)
100 loops, best of 3: 5.31 ms per loop

In [7]: df1.equals(df2)
Out[7]: True
</code></pre></div>