title: Calculating Chi Square on Spark with sequence of pair of features
body: <p>Actually it is a few days that I'm stuck on this challenge and I wanted to know your ideas about it. I have a big dataframe in pyspark with around 150 features. Each row belongs to one person and the value of each cell is either 0 or 1. I am interested in knowing is there any correlation between these features or not. Because these features are Categorical, I chose Chi Square Independence test. </p>
  
  <p>For each pair I created the contingency table by using Crosstab and then convert the corsstab output to a dense matrix and calculate the p-value for each pair. For small scale it works, but for big data volume the code keeps running for long time and it is not efficient in my opinion. The problem is since I need the combination of all features it increases the calculation size exponentially.</p>
  
  <p>What approach do you suggest to do this job with better performance?</p>
  
  <pre><code>def calculate_chi(df, col1, col2):
     test = df.crosstab(col2, col1) 
     vals = test.rdd.map(lambda x:[x[1],x[2]]).flatMap(lambda y: y).collect()
     Matrix = Matrices.dense(2, 2, vals)
     return Statistics.chiSqTest(Matrix).pValue
  
  if __name__=="__main__":
     spark = SparkSession.builder.getOrCreate()
     df=spark.read.option("header",True).csv("s3a://*.csv")
     df = df.select(selected_cols)
     data = []
     cols =  combinations(df.columns,2)
     for c in cols:
        data.append((str(c), calculate_chi(df, c[0],c[1])))
     df_results = spark.createDataFrame(data, ["feature_pair","pvalue"])
     df_results.coalesce(1).write.format('csv').options(header='true').save("s3a://chiSqTest")
  </code></pre>