title: Load Dataset from Dynamically generated Case Class
body: <p><strong>What is Needed:</strong></p>
  <p>number of tables in source database are changing rapidly and thus I don't want to edit case classes so I dynamically generate them through SCALA code and put in package. But now not able to read it dynamically. If this works than I would parse &quot;com.example.datasources.fileSystemSource.schema.{}&quot; as object schema members in loop</p>
  <p><strong>What has already been Done:</strong></p>
  <p>I have some case classes dynamically generated from schema of database tables as below:</p>
  <pre><code>object schema{
  case class Users(name: String,
                   favorite_color: String,
                   favorite_numbers: Array[Int])
  
  case class UserData(registration_dttm: Timestamp,
                      id: Int,
                      first_name: String,
                      last_name: String,
                      email: String,
                      gender: String,
                      ip_address: String,
                      cc: String,
                      country: String,
                      birthdate: String,
                      salary: Double,
                      title: String,
                      comments: String)
  }
  </code></pre>
  <p>Then i have used them as dynamic type to read in Load[T] function in my Loader.scala as below:</p>
  <pre><code>import org.apache.spark.sql.{Dataset, Encoder, SparkSession}
  
  class Load[T &lt;: Product: Encoder](val tableName: String,
                                         val inputPath: String,
                                         val spark: SparkSession,
                                         val saveMode: String,
                                         val outputPath: String,
                                         val metadata: Boolean)
      extends Loader[T] {
  
    val fileSystemSourceInstance: FileSystem[T] =
      new FileSystem[T](inputPath, spark, saveMode, tableName)
  
    override def Load: Dataset[T] =
      fileSystemSourceInstance.provideData(metadata, outputPath).as[T]
  
  }
  </code></pre>
  <p>Now, by using reflect.api I am able to get TypeTag for my case classes.</p>
  <pre><code>def stringToTypeTag[A](name: String): TypeTag[A] = {
      val c = Class.forName(name)
      val mirror = runtimeMirror(c.getClassLoader)
      val sym = mirror.staticClass(name)
      val tpe = sym.selfType
      TypeTag(mirror, new api.TypeCreator {
        def apply[U &lt;: api.Universe with Singleton](m: api.Mirror[U]) =
  
          if (m eq mirror) tpe.asInstanceOf[U # Type]
          else throw new IllegalArgumentException(s&quot;Type tag defined in $mirror cannot be migrated to other mirrors.&quot;)
      })
    }
  </code></pre>
  <p>So if i print now my case class type tag I got:</p>
  <pre><code>val typetagDynamic = stringToTypeTag(&quot;com.example.datasources.fileSystemSource.schema.Users&quot;)
  println(typetags)
  TypeTag[com.example.datasources.fileSystemSource.schema.Users]
  </code></pre>
  <p><strong>Problem:</strong></p>
  <p>Need to read these TypeTag or Dynamically generated case classes, to encode my datasets as below:</p>
  <pre><code>new Load[typetagDynamic](tableName,inputPath,spark,
  saveMode,
  outputPath + tableName,
  metadata)(Encoders.product[typetagDynamic]).Load 
  </code></pre>
  <p>This is giving me error : <strong>Cannot resolve symbol typetagDynamic</strong></p>
  <p>if used like this:</p>
  <pre><code>new Load[typetagDynamic.type](tableName,inputPath,spark,
  saveMode,
  outputPath + tableName,
  metadata)(Encoders.product[typetagDynamic.type]).Load 
  </code></pre>
  <p>This is giving me error : <strong>type arguments [T] do not conform to method product's type parameter bounds [T &lt;: Product]</strong></p>