title: Spark dataframe udf No TypeTag available
body: <p>I am trying to extend <code>spark</code> <code>ml</code> pipeline model with a filter transformer, after </p>
  
  <pre><code>abstract class RuleFilter[IN, T &lt;: RuleFilter[IN, T]]
      extends RuleTransformer with HasInputCol  {
    // def filterFuntion: String
    /** @group setParam */
    def setInputCol(value: String): T = set(inputCol, value).asInstanceOf[T]
  
    protected def createFilterFunc: IN =&gt; Boolean
  
    override def transform(df: DataFrame): DataFrame = {
      transformSchema(df.schema, logging = true)
      val transformUDF = udf[Boolean, IN](this.createFilterFunc)
      df.filter(transformUDF(df($(inputCol))))
    }
  }
  </code></pre>
  
  <p>this code did not compile with an error:</p>
  
  <pre><code> No TypeTag available for IN
  [error]     val transformUDF = udf[Boolean, IN](this.createFilterFunc)
  </code></pre>
  
  <p>how do I let this work?</p>
  
  <p>I need it to work with some explicit defined type in  inherit class such as </p>
  
  <pre><code>class PriceFilter extends RuleFilter {
      def createFilterFunc(val: Double) = val &gt; 500
  }
  </code></pre>