title: Is it possible to store custom class object in Spark Data Frame as a column value?
body: <p>I am working on duplicate documents detection problem using LSH algorithm. To handle large-scale data, we are using spark.</p>
  
  <p>I have around 300K documents with at least 100-200 words per document. On spark cluster, these are the steps we are performing on data frame.</p>
  
  <ol>
  <li>Run Spark ML pipeline for converting text into tokens.</li>
  </ol>
  
  <pre><code>
  pipeline = Pipeline().setStages([
          docAssembler,
          tokenizer,
          normalizer,
          stemmer,
          finisher,
          stopwordsRemover,
         # emptyRowsRemover
      ])
  model = pipeline.fit(spark_df)
  final_df = model.transform(spark_df)
  
  </code></pre>
  
  <ol start="2">
  <li>For each document, get  MinHash  value using datasketch(<a href="https://github.com/ekzhu/datasketch/" rel="nofollow noreferrer">https://github.com/ekzhu/datasketch/</a>) library and store it as a new column.</li>
  </ol>
  
  <pre><code>final_df_limit.rdd.map(lambda x: (CalculateMinHash(x),)).toDF()
  
  </code></pre>
  
  <p>2nd step is failing as spark is not allowing us to store custom type value as a column.  Value is an object of class MinHash.  </p>
  
  <p>Does anyone know how can i store Minhash objects in dataframes? </p>