title: How to aggregate a boolean field with null values with pandas?
body: <p>I'm working with pandas for the first time and I'm having some issues with aggregation. I have a dataframe with three calculated fields added by an apply statement like this:</p>
  
  <pre><code>dataset['calculated_field'] = dataset.apply(
      lambda row: calculation_function(
          row['field1'],
          row['field2']
      ),
      axis = 1
  )
  </code></pre>
  
  <p>The calculated fields are booleans, but with a catch. They can contain null values.</p>
  
  <p>I am trying to find the average of the boolean columns, with each average ignoring the null fields for that column.</p>
  
  <p>I've tried something like this:</p>
  
  <pre><code>resultset = dataset.groupby(['grouping_field'])[['calculated_field','calculated_field_2','calculated_field_3']].mean()
  </code></pre>
  
  <p>The problem is that since the True/False/None boolean is an "object" type, pandas drops the columns entirely as a <a href="http://pandas.pydata.org/pandas-docs/stable/groupby.html#automatic-exclusion-of-nuisance-columns" rel="noreferrer">“nuisance” column.</a>.</p>
  
  <p>I can't convert the column to a bool, though, because it makes the null values "False"</p>
  
  <p>I also tried the long route and created 3 seperate dataframes for each aggregate, so I could drop the null values and convert to bool (which can be aggregated):</p>
  
  <pre><code>dataset_for_field1 = dataset.dropna(subset = ['calculated_field']).copy()
  dataset_for_field1['calculated_field'] = dataset_for_field1['calculated_field'].astype('bool')
  
  result_for_field1 = dataset_for_field1.groupby(['grouping_field'])['calculated_field'].mean()
  </code></pre>
  
  <p>This gives me the data I'm looking for, but in three separate dataframes.</p>
  
  <p>Is there a way to get one dataframe with the average of each column, ignoring nulls?</p>