title: saving compressed json from spark
body: <p>from Spark RDDs, I want to stage and archive JSON data to AWS S3.  It only makes sense to compress it, and I have a process working using hadoop's <code>GzipCodec</code>, but there's things that make me nervous about this.</p>
  
  <p>When I look at the type signature of <code>org.apache.spark.rdd.RDD.saveAsTextFile</code> here:  </p>
  
  <p><a href="https://spark.apache.org/docs/2.3.0/api/scala/index.html#org.apache.spark.rdd.RDD" rel="noreferrer">https://spark.apache.org/docs/2.3.0/api/scala/index.html#org.apache.spark.rdd.RDD</a></p>
  
  <p>the type signature is:</p>
  
  <pre><code>def saveAsTextFile(path: String, codec: Class[_ &lt;: CompressionCodec]): Unit
  </code></pre>
  
  <p>but when I check the available compression codecs here: </p>
  
  <p><a href="https://spark.apache.org/docs/2.3.0/api/scala/index.html#org.apache.spark.io.CompressionCodec" rel="noreferrer">https://spark.apache.org/docs/2.3.0/api/scala/index.html#org.apache.spark.io.CompressionCodec</a></p>
  
  <p>the parent trait <code>CompressionCodec</code> and subtypes all say:</p>
  
  <blockquote>
    <p>The wire protocol for a codec is not guaranteed compatible across versions of Spark. This is intended for use as an internal compression utility within a single Spark application</p>
  </blockquote>
  
  <p>That's no good...  but it's fine, because gzip is probably easier to deal with across ecosystems anyway.</p>
  
  <p>The type signature says the codec must be a subtype of <code>CompressionCodec</code>...  but I tried the following to save as .gz, and it works fine, even though hadoop's GzipCodec is not <code>&lt;: CompressionCodec</code>.</p>
  
  <pre><code>import org.apache.hadoop.io.compress.GzipCodec
  rdd.saveAsTextFile(bucketName, classOf[GzipCodec])
  </code></pre>
  
  <p>my questions:</p>
  
  <ul>
  <li>this works, but are there any reasons to not do it this way...  or is there a better way?</li>
  <li>is this going to be robust across Spark versions (and elsewhere) unlike the built in compression codecs?</li>
  </ul>