title: How to fix "Cannot use null as map key!" error in Spark.SQL with Python 3 using Group_Map
body: <p>I am working with Spark.SQL and I am trying to make a pivot table with the MAP statement, in order to have the values of one column as different columns.</p>
  
  <p>I prepared a first table with this query:</p>
  
  <pre><code>spark.sql("""CREATE TABLE TABLE_01 STORED AS PARQUET AS
  select
  ROWS,
  COLUMNS,
  count(*) as NUM_ROWS
  from TABLE_00
  group by ROWS, COLUMNS
  order by ROWS, COLUMNS
  """)
  </code></pre>
  
  <p>The idea is convert this:</p>
  
  <pre><code>ROWS    COLUMNS NUM_ROWS
  VALUE1   COL2    2
  VALUE1   COL3    50
  VALUE2   COL1    20
  VALUE2   COL2    1
  VALUE2   COL3    30
  </code></pre>
  
  <p>Into this:</p>
  
  <pre><code>                COLUMNS
  
                COL1          |  COL2     |  COL3
  ROWS
        ------------------------------------------------------
  VALUE1  0 (The problem)     |    2      |   50
  
  VALUE2       20             |    1      |   30 
  </code></pre>
  
  <p>I had a query in Python2 which used to work, but now I have to change to Python3 and I going crazy. The query is:</p>
  
  <pre><code>spark.sql("""CREATE TABLE TABLE_02 STORED AS PARQUET AS
  select
  a.ROWS,
  coalesce(sum(a.group_map['COL1']),0) as COL1,
  coalesce(sum(a.group_map['COL2']),0) as COL2,
  coalesce(sum(a.group_map['COL3']),0) as COL3
  from 
      (
      select ROWS, map(COLUMNS,NUM_ROWS) as group_map
      from TABLE_01
      ) a
  group by a.ROWS
  """)
  </code></pre>
  
  <p>The output is: java.lang.RuntimeException: Cannot use null as map key!</p>
  
  <p>Any help? Why did it work with Python2?</p>
  
  <p>Thanks!</p>