title: How to check the number of partitions of a Spark DataFrame without incurring the cost of .rdd
body: <p>There are a number of questions about how to obtain the number of partitions  of a n <code>RDD</code> and or a <code>DataFrame</code> : the answers invariably are:</p>
  
  <pre><code> rdd.getNumPartitions
  </code></pre>
  
  <p>or</p>
  
  <pre><code> df.rdd.getNumPartitions
  </code></pre>
  
  <p>Unfortunately that is an <em>expensive</em> operation on a <code>DataFrame</code> because the</p>
  
  <pre><code> df.rdd
  </code></pre>
  
  <p>requires conversion from the <code>DataFrame</code> to an <code>rdd</code>.  This is on the order of the time it takes to run</p>
  
  <pre><code> df.count
  </code></pre>
  
  <p>I am writing logic that <em>optionally</em> <code>repartition</code>'s or <code>coalesce</code>'s a <code>DataFrame</code> - based on whether the <em>current</em> number of partitions were within a range of acceptable values or instead below or above them.  </p>
  
  <pre><code>  def repartition(inDf: DataFrame, minPartitions: Option[Int],
         maxPartitions: Option[Int]): DataFrame = {
      val inputPartitions= inDf.rdd.getNumPartitions  // EXPENSIVE!
      val outDf = minPartitions.flatMap{ minp =&gt;
        if (inputPartitions &lt; minp) {
          info(s"Repartition the input from $inputPartitions to $minp partitions..")
          Option(inDf.repartition(minp))
        } else {
          None
        }
      }.getOrElse( maxPartitions.map{ maxp =&gt;
        if (inputPartitions &gt; maxp) {
          info(s"Coalesce the input from $inputPartitions to $maxp partitions..")
          inDf.coalesce(maxp)
        } else inDf
      }.getOrElse(inDf))
      outDf
    }
  </code></pre>
  
  <p>But we can not afford to incur the cost of the <code>rdd.getNumPartitions</code> for <em>every</em> <code>DataFrame</code> in this manner.</p>
  
  <p>Is there not any way to obtain this information - e.g. from querying the online/temporary <code>catalog</code> for the <code>registered</code> table maybe?</p>
  
  <p><strong>Update</strong>  The Spark GUI showed the DataFrame.rdd operation as taking as long as the longest sql in the job. I will re-run the job and attach the screenshot in a bit here.</p>
  
  <p>The following is just a <em>testcase</em> : it is using a small fraction of the data size of that in production.  The longest <code>sql</code> is only five minutes - and this one is on its way to spending that amount of time <em>as well</em> (note that the <code>sql</code> is <em>not</em> helped out here: it also has to execute subsequently thus effectively doubling the cumulative execution time).</p>
  
  <p><a href="https://i.stack.imgur.com/iwYnm.png" rel="noreferrer"><img src="https://i.stack.imgur.com/iwYnm.png" alt="enter image description here"></a></p>
  
  <p>We can see that the <code>.rdd</code> operation at <code>DataFrameUtils</code> line 30 (shown in the snippet above) takes 5.1mins - and yet the <code>save</code> operation <em>still</em> took 5.2 mins later -i.e. we did <em>not</em> save any time by doing the <code>.rdd</code> in terms of the execution time of the subsequent <code>save</code>.</p>