<p dir="auto">Duplicate of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="157104029" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/13302" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/13302/hovercard" href="https://github.com/pandas-dev/pandas/issues/13302">#13302</a></p>
<hr>
<p dir="auto">When reading via read_fwf and passing both na_values and converters at the same time, na_value detection does not work:</p>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<p dir="auto">Assume a fwf-style text file like this:</p>
<blockquote>
<p dir="auto">A  B  C<br>
10 -1 10<br>
10 10 10</p>
</blockquote>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In: read_fwf(file_path, converters={'B':lambda x: int(x)/10.}, na_values={'B':-1})
Out:
&gt;     A    B   C
&gt; 0  10 -0.1  10
&gt; 1  10  1.0  10
"><pre class="notranslate"><span class="pl-v">In</span>: <span class="pl-en">read_fwf</span>(<span class="pl-s1">file_path</span>, <span class="pl-s1">converters</span><span class="pl-c1">=</span>{<span class="pl-s">'B'</span>:<span class="pl-k">lambda</span> <span class="pl-s1">x</span>: <span class="pl-en">int</span>(<span class="pl-s1">x</span>)<span class="pl-c1">/</span><span class="pl-c1">10.</span>}, <span class="pl-s1">na_values</span><span class="pl-c1">=</span>{<span class="pl-s">'B'</span>:<span class="pl-c1">-</span><span class="pl-c1">1</span>})
<span class="pl-v">Out</span>:
<span class="pl-c1">&gt;</span>     <span class="pl-v">A</span>    <span class="pl-v">B</span>   <span class="pl-v">C</span>
<span class="pl-c1">&gt;</span> <span class="pl-c1">0</span>  <span class="pl-c1">10</span> <span class="pl-c1">-</span><span class="pl-c1">0.1</span>  <span class="pl-c1">10</span>
<span class="pl-c1">&gt;</span> <span class="pl-c1">1</span>  <span class="pl-c1">10</span>  <span class="pl-c1">1.0</span>  <span class="pl-c1">10</span></pre></div>
<h4 dir="auto">Expected Output</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;     A    B   C
&gt; 0  10  NaN  10
&gt; 1  10  1.0  10"><pre class="notranslate"><code class="notranslate">&gt;     A    B   C
&gt; 0  10  NaN  10
&gt; 1  10  1.0  10
</code></pre></div>
<p dir="auto">It would be nice if this could be handled behind the scenes.<br>
In pandas.io.parsers._clean_na_values, float representations of the na_values are produced by _floatify_na_values. Maybe passing the converters to this function might be a solution?</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 23 Stepping 10, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
<p dir="auto">pandas: 0.19.2<br>
(...)</p>
</details>
<p dir="auto">On <code class="notranslate">master</code> (commit <a href="https://github.com/pydata/pandas/commit/40b4bb4bb2a7018ed08025c5c93cd1080a0b5f7f">40b4bb4</a>):</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; data = &quot;&quot;&quot;A
1
CAT
3&quot;&quot;&quot;
&gt;&gt;&gt; f = lambda x: x
&gt;&gt;&gt; read_csv(StringIO(data), na_values='CAT', converters={'A': f}, engine='c')
     A
0    1
1  CAT
2    3
&gt;&gt;&gt; read_csv(StringIO(data), na_values='CAT', converters={'A': f}, engine='python')
     A
0    1
1  NaN
2    3"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s">"""A</span>
<span class="pl-s">1</span>
<span class="pl-s">CAT</span>
<span class="pl-s">3"""</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">f</span> <span class="pl-c1">=</span> <span class="pl-k">lambda</span> <span class="pl-s1">x</span>: <span class="pl-s1">x</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">read_csv</span>(<span class="pl-v">StringIO</span>(<span class="pl-s1">data</span>), <span class="pl-s1">na_values</span><span class="pl-c1">=</span><span class="pl-s">'CAT'</span>, <span class="pl-s1">converters</span><span class="pl-c1">=</span>{<span class="pl-s">'A'</span>: <span class="pl-s1">f</span>}, <span class="pl-s1">engine</span><span class="pl-c1">=</span><span class="pl-s">'c'</span>)
     <span class="pl-v">A</span>
<span class="pl-c1">0</span>    <span class="pl-c1">1</span>
<span class="pl-c1">1</span>  <span class="pl-v">CAT</span>
<span class="pl-c1">2</span>    <span class="pl-c1">3</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">read_csv</span>(<span class="pl-v">StringIO</span>(<span class="pl-s1">data</span>), <span class="pl-s1">na_values</span><span class="pl-c1">=</span><span class="pl-s">'CAT'</span>, <span class="pl-s1">converters</span><span class="pl-c1">=</span>{<span class="pl-s">'A'</span>: <span class="pl-s1">f</span>}, <span class="pl-s1">engine</span><span class="pl-c1">=</span><span class="pl-s">'python'</span>)
     <span class="pl-v">A</span>
<span class="pl-c1">0</span>    <span class="pl-c1">1</span>
<span class="pl-c1">1</span>  <span class="pl-v">NaN</span>
<span class="pl-c1">2</span>    <span class="pl-c1">3</span></pre></div>
<p dir="auto">I expect both to give the same output, though I believe the Python output is more correct because it respects <code class="notranslate">na_values</code> unlike the C engine.   I thought the simple fix would be to remove the <code class="notranslate">continue</code> statement <a href="https://github.com/pydata/pandas/blob/master/pandas/parser.pyx#L1008">here</a>, but that causes test failures, so probably a more involved refactoring might be needed to align the order of converter application, <code class="notranslate">NaN</code> value conversion, and <code class="notranslate">dtype</code> conversion.</p>
<p dir="auto">IMO this should be added to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="142662289" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/12686" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/12686/hovercard" href="https://github.com/pandas-dev/pandas/issues/12686">#12686</a>, as this is a difference in behaviour between the two engines.</p>
<p dir="auto">xref <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="21030673" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/5232" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/5232/hovercard" href="https://github.com/pandas-dev/pandas/issues/5232">#5232</a></p>