title: Generic T as Spark Dataset[T] constructor
body: <p>In the following snippet, the <code>tryParquet</code> function tries to load a Dataset from a Parquet file if it exists. If not, it computes, persists and returns back the Dataset plan which was provided:</p>
  
  <pre><code>import scala.util.{Try, Success, Failure}
  import org.apache.spark.sql.SparkSession
  import org.apache.spark.sql.Dataset
  
  sealed trait CustomRow
  
  case class MyRow(
    id: Int,
    name: String
  ) extends CustomRow
  
  val ds: Dataset[MyRow] =
    Seq((1, "foo"),
        (2, "bar"),
        (3, "baz")).toDF("id", "name").as[MyRow]
  
  def tryParquet[T &lt;: CustomRow](session: SparkSession, path: String, target: Dataset[T]): Dataset[T] =
      Try(session.read.parquet(path)) match {
        case Success(df) =&gt; df.as[T] // &lt;---- compile error here
        case Failure(_)  =&gt; {
          target.write.parquet(path)
          target
        }
      }
  
  val readyDS: Dataset[MyRow] =
      tryParquet(spark, "/path/to/file.parq", ds)
  </code></pre>
  
  <p>However this produces a compile error on <code>df.as[T]</code>:</p>
  
  <blockquote>
    <p>Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._</p>
    
    <p>Support for serializing other types will be added in future releases.</p>
    
    <p>case Success(df) => df.as[T]</p>
  </blockquote>
  
  <p>One can circumvent this problem by making <code>tryParquet</code> cast <code>df</code> to return an untyped <code>DataFrame</code> and let caller cast to the desired constructor. However is there any solution in the case we want the type to be managed internally by the function?</p>