title: What is a good heuristic to detect if a column in a pandas.DataFrame is categorical?
body: <p>I've been developing a tool that automatically preprocesses data in pandas.DataFrame format. During this preprocessing step, I want to treat continuous and categorical data differently. In particular, I want to be able to apply, e.g., a OneHotEncoder to <em>only</em> the categorical data.</p>
  
  <p>Now, let's assume that we're provided a pandas.DataFrame and have no other information about the data in the DataFrame. What is a good heuristic to use to determine whether a column in the pandas.DataFrame is categorical?</p>
  
  <p>My initial thoughts are:</p>
  
  <p>1) If there are strings in the column (e.g., the column data type is <code>object</code>), then the column very likely contains categorical data</p>
  
  <p>2) If some percentage of the values in the column is unique (e.g., >=20%), then the column very likely contains continuous data</p>
  
  <p>I've found <code>1)</code> to work fine, but <code>2)</code> hasn't panned out very well. I need better heuristics. How would you solve this problem?</p>
  
  <p><strong>Edit:</strong> Someone requested that I explain why <code>2)</code> didn't work well. There were some tests cases where we still had continuous values in a column but there weren't many unique values in the column. The heuristic in <code>2)</code> obviously failed in that case. There were also issues where we had a categorical column that had many, many unique values, e.g., passenger names in the Titanic data set. Same column type misclassification problem there.</p>