title: Filtering and reassigning pyspark dataframe in a loop
body: <p>I'm trying to iterate through all the columns of a Pyspark data frame, calculating the IQR to filter the upper outliers, and reassigning the same dataframe. There are 200+ columns. This is working, but it goes slower and slower as it advances through the loop. I suspect the problem may be in the reassignment of the dataframe (dfBufferOutlier=dfBufferOutlier.filter(col(i)
  
  <pre class="lang-py prettyprint-override"><code>nameList=dfBuffer.schema.names[1:]    
  dfBufferOutlier=dfBuffer
  for i in nameList:
      cuenta=dfBufferOutlier.filter(col(i)&gt;0).count()
      if cuenta&gt;0:
          Q1, Q2, Q3 = dfBufferOutlier.filter(col(i)&gt;0).approxQuantile(col=i,probabilities=[0.25,0.5,0.75],relativeError=0.005)
          IQR=(Q3-Q1)
          top_limit=Q3+1.5*IQR
          dfBufferOutlier=dfBufferOutlier.filter(col(i)&lt;top_limit)
  </code></pre>
  
  <p>There sure must be another way to get this better, but can't figure out how.. any help please?</p>