title: Spark - Random Number Generation
body: <p>I have written a method that must consider a random number to simulate a Bernoulli distribution. I am using <code>random.nextDouble</code> to generate a number between 0 and 1 then making my decision based on that value given my probability parameter.</p>
  
  <p>My problem is that Spark is generating the same random numbers within each iteration of my for loop mapping function. I am using the <code>DataFrame</code> API. My code follows this format:</p>
  
  <pre><code>val myClass = new MyClass()
  val M = 3
  val myAppSeed = 91234
  val rand = new scala.util.Random(myAppSeed)
  
  for (m &lt;- 1 to M) {
    val newDF = sqlContext.createDataFrame(myDF
      .map{row =&gt; RowFactory
        .create(row.getString(0),
          myClass.myMethod(row.getString(2), rand.nextDouble())
      }, myDF.schema)
  }
  </code></pre>
  
  <p>Here is the class:</p>
  
  <pre><code>class myClass extends Serializable {
    val q = qProb
  
    def myMethod(s: String, rand: Double) = {
      if (rand &lt;= q) // do something
      else // do something else
    }
  }
  </code></pre>
  
  <p>I need a new random number every time <code>myMethod</code> is called. I also tried generating the number inside my method with <code>java.util.Random</code> (<code>scala.util.Random</code> v10 does not extend <code>Serializable</code>) like below, but I'm still getting the same numbers within each for loop </p>
  
  <pre><code>val r = new java.util.Random(s.hashCode.toLong)
  val rand = r.nextDouble()
  </code></pre>
  
  <p>I've done some research, and it seems this has do to with Sparks deterministic nature.</p>