title: How do I refer to a specific DataFrame in a spark pipeline?
body: <p>Suppose I have two spark DataFrames with the same features in Spark and I want to build a pipeline to cross validate them both. How can I refer to each table within a pipeline? I use <code>sparklyr</code> in R to do this, but I guess it should be the same with <code>pyspark</code>.</p>
  
  <p>First I can use the following code to build a linear regression and cross evaluate it using <code>ml_cross_validator()</code></p>
  
  <pre class="lang-r prettyprint-override"><code>suppressMessages(library(sparklyr))
  suppressMessages(library(tidyverse))
  sc &lt;- spark_connect(master = "local")
  
  copy_to(sc, mtcars, "mtcars")
  
  mtcars &lt;- tbl(sc, "mtcars")
  
  pipeline &lt;- ml_pipeline(sc) %&gt;%
      ft_r_formula(mpg ~ .) %&gt;%
      ml_linear_regression()
  
  grid &lt;- list(linear_regression = list(reg_param = 0))
  
  cv &lt;- ml_cross_validator(
      sc,
      estimator = pipeline, # use our pipeline to estimate the model
      estimator_param_maps = grid, # use the params in grid
      evaluator = ml_regression_evaluator(sc, metric_name = "rmse"),  # how to evaluate the CV
      num_folds = 2, # number of CV folds
      seed = 2018
  )
  
  cv_model &lt;- ml_fit(cv, mtcars)
  
  cv_model$avg_metrics_df
  #&gt;       rmse reg_param_1
  #&gt; 1 3.997882           0
  </code></pre>
  
  <p><sup>Created on 2019-09-13 by the <a href="https://reprex.tidyverse.org" rel="nofollow noreferrer">reprex package</a> (v0.3.0)</sup></p>
  
  <p>But if I add another table with the same features:</p>
  
  <pre><code>mtcars_sample &lt;- sdf_sample(mtcars, fraction = 0.8) %&gt;%
      sdf_register("mtcars_sample")
  </code></pre>
  
  <p>How can I refer to it within the pipeline?</p>
  
  <pre><code>─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
   setting  value                       
   version  R version 3.6.0 (2019-04-26)
   os       macOS Mojave 10.14.6        
   system   x86_64, darwin15.6.0        
   ui       RStudio                     
   language (EN)                        
   collate  en_US.UTF-8                 
   ctype    en_US.UTF-8                 
   tz       Europe/Stockholm            
   date     2019-09-13                  
  </code></pre>