title: praw - scalable way to put comments into spark df
body: <p>I am working with the reddit API using praw. I filtered all the comments of a subreddit and now want to put the data into a spark data frame.</p>
  <p>One way (which I tried) is to append everything to a list and then put that into a df. But this is not scalable. I also don't want to download/save it in an extra file.
  I now wondered if there is a scalable way to get the comments into a df?</p>
  <p>Thank you for your help in advance!</p>
  <p>Here is my code so far (but with the list implemented):</p>
  <pre><code>subreddit = reddit.subreddit('Refugees').new(limit = None)
  
  comments = []
  for submission in subreddit:
      #replace_more to solve error
      submission.comments.replace_more(limit = 0)
      for comment in submission.comments:
              #get comments
              comments.append(comment.body)
              if len(comment.replies) &gt; 0:
                  #dig deeper and get replies
                  for reply in comment.replies:
                      comments.append(reply.body)
  </code></pre>
  <pre><code>from pyspark.sql import SparkSession
  from pyspark.sql.types import ArrayType
  from pyspark.sql.types import StructField
  from pyspark.sql.types import StructType
  from pyspark.sql.types import StringType
  from pyspark.sql.types import IntegerType
  from pyspark.sql.functions import col,isnan, when, count
  from pyspark.sql.functions import initcap
  from pyspark.sql.functions import countDistinct
  
  
  import findspark
  
  findspark.init(&quot;/usr/local/spark/&quot;)
  
  spark = SparkSession.builder \
     .master(&quot;local[*]&quot;) \
     .appName(&quot;project&quot;) \
     .config(&quot;spark.executor.memory&quot;, &quot;1gb&quot;) \
     .getOrCreate()
  </code></pre>
  <pre><code>schema = StructType([
      StructField('Comments', StringType(), True),
  ])
  df = spark.createDataFrame([(c,) for c in comments], schema = schema)
  print(df.schema)
  df.show(70)
  </code></pre>