title: Apache Spark timing forEach operation on JavaRDD
body: <p><strong>Question: Is this a valid way to test the time taken to build a RDD?</strong></p>
  
  <p>I am doing two things here. The basic approach is that we have M instances of what we call a DropEvaluation, and N DropResults. We need to compare each N DropResult to each of M DropEvaluations. Each N must be seen by each M, to give us M results in the end.</p>
  
  <p>If I don't use the .count() once the RDD is built, the driver continues on to the next line of code and says it look almost no time to build a RDD that takes 30 minutes to build.</p>
  
  <p><strong>I just want to make sure I am not missing something, like maybe the .count() taking a long time? I guess to time the .count() I'd have to modify Spark's source?</strong></p>
  
  <p>M = 1000 or 2000. N = 10^7.</p>
  
  <p>It's effectively a cartesian problem -- the accumulator was chosen because we need to write to each M in place. I would also be ugly to build the full cartesian RDD.</p>
  
  <p>We build a List of M Accumulators (can't do a List Accumulator in Java right?). Then we loop through each of N in a RDD with a foreach.</p>
  
  <p><strong>Clarifying the Question: The total time taken is measured correctly, I am asking if the .count() on the RDD forces Spark to wait until the RDD is finished before it can run a count. Is the .count() time significant?</strong></p>
  
  <p>Here's our code:</p>
  
  <pre><code>// assume standin exists and does it's thing correctly
  
  // this controls the final size of RDD, as we are not parallelizing something with an existing length
  List&lt;Integer&gt; rangeN = IntStream.rangeClosed(simsLeft - blockSize + 1, simsLeft).boxed().collect(Collectors.toList());
  
  // setup bogus array of size N for parallelize dataSetN to lead to dropResultsN       
  JavaRDD&lt;Integer&gt; dataSetN = context.parallelize(rangeN);
  
  // setup timing to create N
  long NCreationStartTime = System.nanoTime();
  
  // this maps each integer element of RDD dataSetN to a "geneDropped" chromosome simulation, we need N of these:
  JavaRDD&lt;TholdDropResult&gt; dropResultsN = dataSetN.map(s -&gt; standin.call(s)).persist(StorageLevel.MEMORY_ONLY());
  
  // **** this line makes the driver wait until the RDD is done, right?
  long dummyLength = dropResultsN.count();
  
  
  long NCreationNanoSeconds = System.nanoTime() - NCreationStartTime;
  double NCreationSeconds = (double)NCreationNanoSeconds / 1000000000.0;
  double NCreationMinutes = NCreationSeconds / 60.0;
  
  logger.error("{} test sims remaining", simsLeft);
  
  // now get the time for just the dropComparison (part of accumulable's add)
  long startDropCompareTime = System.nanoTime();
  
  // here we iterate through each accumulator in the list and compare all N elements of dropResultsN RDD to each M in turn, our .add() is a custom AccumulableParam
  for (Accumulable&lt;TholdDropTuple, TholdDropResult&gt; dropEvalAccum : accumList) {
      dropResultsN.foreach(new VoidFunction&lt;TholdDropResult&gt;() {
                      @Override
                      public void call(TholdDropResult dropResultFromN) throws Exception {
                              dropEvalAccum.add(dropResultFromN);
                      }
                  });
              }
  
      // all the dropComparisons for all N to all M for this blocksize are done, check the time...
     long dropCompareNanoSeconds = System.nanoTime() - startDropCompareTime;
     double dropCompareSeconds = (double)dropCompareNanoSeconds / 1000000000.0;
      double dropCompareMinutes = dropCompareSeconds / 60.0;
  
      // write lines to indicate timing section
      // log and write to file the time for the N-creation
  
      ...
  
  } // end for that goes through dropAccumList
  </code></pre>