title: Is it possible to allow users to download the result of a pyspark dataframe in FastAPI or Flask
body: <p>I'm working on an API using FastAPI that users can make a request to in order for the following to happen:</p>
  
  <ol>
  <li>First, a get request will grab a file from Google Cloud Storage and load it into a pyspark DataFrame</li>
  <li>Then the application will perform some transformations on the DataFrame</li>
  <li>Finally, I want to write the DataFrame to the user's disk as a parquet file.</li>
  </ol>
  
  <p>I can't quite figure out how to deliver the file to the user in parquet format, for a few reasons:</p>
  
  <ul>
  <li><code>df.write.parquet('out/path.parquet')</code> writes the data into a directory at <code>out/path.parquet</code> which presents a challenge when I try to pass it to <code>starlette.responses.FileResponse</code></li>
  <li>Passing a single .parquet file that I know exists to <code>starlette.responses.FileResponse</code> just seems to print the binary to my console (as demonstrated in my code below)</li>
  <li>Writing the DataFrame to a BytesIO stream <a href="https://stackoverflow.com/a/52091707/7004653">like in pandas</a> seemed promising, but I can't quite figure out how to do that using any of DataFrame's methods or DataFrame.rdd's methods.</li>
  </ul>
  
  <p>Is this even possible in FastAPI? Is it possible in Flask using <a href="https://flask.palletsprojects.com/en/1.1.x/api/#flask.send_file" rel="nofollow noreferrer">send_file()</a>?</p>
  
  <p>Here's the code I have so far. Note that I've tried a few things like the commented code to no avail.</p>
  
  <pre><code>import tempfile
  
  from fastapi import APIRouter
  from pyspark.context import SparkContext
  from pyspark.sql.session import SparkSession
  from starlette.responses import FileResponse
  
  
  router = APIRouter()
  sc = SparkContext('local')
  spark = SparkSession(sc)
  
  df: spark.createDataFrame = spark.read.parquet('gs://my-bucket/sample-data/my.parquet')
  
  @router.get("/applications")
  def applications():
      df.write.parquet("temp.parquet", compression="snappy")
      return FileResponse("part-some-compressed-file.snappy.parquet")
      # with tempfile.TemporaryFile() as f:
      #     f.write(df.rdd.saveAsPickleFile("temp.parquet"))
      #     return FileResponse("test.parquet")
  </code></pre>
  
  <p>Thanks!</p>
  
  <p>Edit:
  I tried using the answers and info provided <a href="https://stackoverflow.com/questions/55873174/how-do-i-return-an-image-in-fastapi">here</a>, but I can't quite get it working.</p>