title: Improve h2o DRF runtime on a multi-node cluster
body: <p>I am currently running <code>h2o</code>'s DRF algorithm an a 3-node EC2 cluster (the h2o server spans across all 3 nodes).
  My data set has 1m rows and 41 columns (40 predictors and 1 response).</p>
  
  <p>I use the <code>R</code> bindings to control the cluster and the RF call is as follows</p>
  
  <pre><code>model=h2o.randomForest(x=x,
                         y=y,
                         ignore_const_cols=TRUE,
                         training_frame=train_data,
                         seed=1234,
                         mtries=7,
                         ntrees=2000,
                         max_depth=15,
                         min_rows=50,
                         stopping_rounds=3,
                         stopping_metric="MSE",
                         stopping_tolerance=2e-5)
  </code></pre>
  
  <p>For the 3-node cluster (c4.8xlarge, enhanced networking turned on), this takes about 240sec; the CPU utilization is between 10-20%; RAM utilization is between 20-30%; network transfer is between 10-50MByte/sec (in and out). 300 trees are built until early stopping kicks in.</p>
  
  <p>On a <em>single-node</em> cluster, I can get the same results in about 80sec. So, instead of an expected 3-fold speed up, I get a 3-fold slow down for the 3-node cluster.</p>
  
  <p>I did some research and found a few resources that were reporting the same issue (not as extreme as mine though). See, for instance:
  <a href="https://groups.google.com/forum/#!topic/h2ostream/bnyhPyxftX8" rel="nofollow noreferrer">https://groups.google.com/forum/#!topic/h2ostream/bnyhPyxftX8</a></p>
  
  <p>Specifically, the author of <a href="http://datascience.la/benchmarking-random-forest-implementations/" rel="nofollow noreferrer">http://datascience.la/benchmarking-random-forest-implementations/</a> notes that</p>
  
  <blockquote>
    <p>While not the focus of this study, there are signs that running the
    distributed random forests implementations (e.g. H2O) on multiple
    nodes does not provide the speed benefit one would hope for (because
    of the high cost of shipping the histograms at each split over the
    network).</p>
  </blockquote>
  
  <p>Also <a href="https://www.slideshare.net/0xdata/rf-brighttalk" rel="nofollow noreferrer">https://www.slideshare.net/0xdata/rf-brighttalk</a> points at 2 different DRF implementations, where one has a larger network overhead.</p>
  
  <p>I think that I am running into the same problems as described in the links above.
  How can I improve <code>h2o</code>'s DRF performance on a multi-node cluster?
  Are there any settings that might improve runtime?
  Any help highly appreciated!</p>