title: PySpark fix/remove console progress bar
body: <p>As can be seen below, the Spark console output progress bar is messing up the outputs. Is there a configuration or flag that can be used to turn off the stage progress bar? Or better, how do I fix the console log so that the progress-bar disappears after the stages are finished? This may just be a bug for PySpark, but I'm not sure.</p>
  
  <pre><code>(CID, (v1 / n1, v2 / n2))
  [Stage 46:============================================&gt;           (19 + 4) / 24]('1', (0.020000000000000035, 4.805))
  ('5', (6.301249999999998, 0.125))
  ('10', (21.78000000000001, 3.125))
  ('7', (0.005000000000000009, 0.6049999999999996))
  
  (CID, sqrt(v1 / n1 + v2 / n2))
  ('1', 2.19658826364888)
  ('5', 2.5350049309616733)
  ('10', 4.990490957811667)
  ('7', 0.7810249675906652)
  
  (CID, (AD_MEAN, NCI_MEAN))
  ('7', (1.0, 5.5))
  ('5', (7.75, 5.3))
  ('10', (13.5, 6.0))
  ('1', (3.0, 5.0))
  
  (CID, (AD_MEAN - NCI_MEAN))
  ('7', -4.5)
  ('5', 2.45)
  ('1', -2.0)
  ('10', 7.5)
  
  (CID, (NUMER, DENOM))
  [Stage 100:===================================================&gt;   (30 + 2) / 32]('10', (7.5, 4.990490957811667))
  ('5', (2.45, 2.5350049309616733))
  ('7', (-4.5, 0.7810249675906652))
  ('1', (-2.0, 2.19658826364888))
  </code></pre>
  
  <p>It gets even worse sometimes (scroll to the right):</p>
  
  <pre><code>$ spark-submit main.py 
  17/04/28 11:36:23 WARN Utils: Your hostname, Pandora resolves to a loopback address: 127.0.1.1; using 146.95.36.193 instead (on interface wlp3s0)
  17/04/28 11:36:23 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
  17/04/28 11:36:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  [Stage 0:&gt;                                                          (0 + 2                                                                          [Stage 32:=============================&gt;                            (4 + 4[Stage 37:&gt;                                                         (0 + 0[Stage 35:=====&gt;           (4 + 2) / 12][Stage 37:&gt;                 (0 + 0[Stage 35:===========&gt;     (8 + 4) / 12][Stage 37:&gt;                 (0 + 0[Stage 37:=======&gt;                                                  (1 + 3[Stage 37:=============================&gt;                            (4 + 0[Stage 36:========&gt;       (13 + 4) / 24][Stage 37:=========&gt;        (4 + 0[Stage 36:==============&gt; (21 + 3) / 24][Stage 37:=========&gt;        (4 + 1[Stage 37:====================================&gt;                     (5 + 3[Stage 38:===================================&gt;                    (20 + 4)[Stage 38:====================================================&gt;   (30 + 2)                                                                          SORTED (t-value, CID)
  [(-5.761659596980321, '7'), (-0.9105029072119708, '1'), (0.9664675480810896, '5'), (1.5028581483070664, '10')]
  </code></pre>