title: explode a row of spark dataset into several rows with added column using flatmap
body: <p>I have a DataFrame with the following schema :</p>
  
  <pre><code>root
   |-- journal: string (nullable = true)
   |-- topicDistribution: vector (nullable = true)
  </code></pre>
  
  <p>The <code>topicDistribution</code> field is a vector of doubles: [0.1, 0.2 0.15 ...]</p>
  
  <p>What I want is is to explode each row into several rows to obtain the following schema:</p>
  
  <pre><code>root
   |-- journal: string
   |-- topic-prob: double // this is the value from the vector
   |-- topic-id : integer // this is the index of the value from the vector
  </code></pre>
  
  <p>To clarify, I've created a case class:</p>
  
  <pre><code>case class JournalDis(journal: String, topic_id: Integer, prob: Double)
  </code></pre>
  
  <p>I've managed to achieve this using dataset.explode in a very awkward way:</p>
  
  <pre><code>val df1 = df.explode("topicDistribution", "topic") {
      topics: DenseVector =&gt; topics.toArray.zipWithIndex
  }.select("journal", "topic")
  val df2 = df1.withColumn("topic_id", df1("topic").getItem("_2")).withColumn("topic_prob", df1("topic").getItem("_1")).drop(df1("topic"))
  </code></pre>
  
  <p>But <strong>dataset.explode is deprecated</strong>. I wonder how to achieve this using <code>flatmap</code> method?</p>