title: spark dynamically create struct/json per group
body: <p>I have a spark dataframe like</p>
  
  <pre><code>+-----+---+---+---+------+
  |group|  a|  b|  c|config|
  +-----+---+---+---+------+
  |    a|  1|  2|  3|   [a]|
  |    b|  2|  3|  4|[a, b]|
  +-----+---+---+---+------+
  val df = Seq(("a", 1, 2, 3, Seq("a")),("b", 2, 3,4, Seq("a", "b"))).toDF("group", "a", "b","c", "config")
  </code></pre>
  
  <p>How can I add an additional column i.e. </p>
  
  <pre><code>df.withColumn("select_by_config", &lt;&lt;&gt;&gt;).show
  </code></pre>
  
  <p>as a struct or JSON which combines a number of columns (specified by <code>config</code>) in something similar to a hive named struct / spark struct / json column? Note, this struct is specific per group and not constant for the whole dataframe; it is specified in <code>config</code> column.</p>
  
  <p>I can imagine that a <code>df.map</code> could do the trick, but the serialization overhead does not seem to be efficient. How can this be achieved via SQL only expressions? Maybe as a Map-type column?</p>
  
  <h1>edit</h1>
  
  <p>a possible but really clumsy solution for 2.2 is:</p>
  
  <pre><code>val df = Seq((1,"a", 1, 2, 3, Seq("a")),(2, "b", 2, 3,4, Seq("a", "b"))).toDF("id", "group", "a", "b","c", "config")
    df.show
    import spark.implicits._
    final case class Foo(id:Int, c1:Int, specific:Map[String, Int])
    df.map(r =&gt; {
      val config = r.getAs[Seq[String]]("config")
      print(config)
      val others = config.map(elem =&gt; (elem, r.getAs[Int](elem))).toMap
      Foo(r.getAs[Int]("id"), r.getAs[Int]("c"), others)
    }).show
  </code></pre>
  
  <p>are there any better ways to solve the problem for 2.2?</p>