title: Thread error: can't start new thread
body: <p>Here's a <code>MWE</code> of a much larger code I'm using. It performs a Monte Carlo integration over a KDE (<a href="http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html" rel="nofollow noreferrer">kernel density estimate</a>) for all values located below a certain threshold (the integration method was suggested over at this question: <a href="https://stackoverflow.com/questions/17822282/integrate-2d-kernel-density-estimate?rq=1">Integrate 2D kernel density estimate</a>) iteratively for a number of points in a list and returns a list made of these results.</p>
  
  <pre><code>import numpy as np
  from scipy import stats
  from multiprocessing import Pool
  import threading
  
  # Define KDE integration function.
  def kde_integration(m_list):
  
      # Put some of the values from the m_list into two new lists.
      m1, m2 = [], []
      for item in m_list:
          # x data.
          m1.append(item[0])
          # y data.
          m2.append(item[1])
  
      # Define limits.
      xmin, xmax = min(m1), max(m1)
      ymin, ymax = min(m2), max(m2)
  
      # Perform a kernel density estimate on the data:
      x, y = np.mgrid[xmin:xmax:100j, ymin:ymax:100j]
      values = np.vstack([m1, m2])
      kernel = stats.gaussian_kde(values)
  
      # This list will be returned at the end of this function.
      out_list = []
  
      # Iterate through all points in the list and calculate for each the integral
      # of the KDE for the domain of points located below the value of that point
      # in the KDE.
      for point in m_list:
  
          # Compute the point below which to integrate.
          iso = kernel((point[0], point[1]))
  
          # Sample KDE distribution
          sample = kernel.resample(size=1000)
  
          #Choose number of cores and split input array.
          cores = 4
          torun = np.array_split(sample, cores, axis=1)
  
          # Print number of active threads.
          print threading.active_count()
  
          #Calculate
          pool = Pool(processes=cores)
          results = pool.map(kernel, torun)
  
          #Reintegrate and calculate results
          insample_mp = np.concatenate(results) &lt; iso
  
          # Integrate for all values below iso.
          integral = insample_mp.sum() / float(insample_mp.shape[0])
  
          # Append integral value for this point to list that will return.
          out_list.append(integral)
  
      return out_list
  
  
  # Generate some random two-dimensional data:
  def measure(n):
      "Measurement model, return two coupled measurements."
      m1 = np.random.normal(size=n)
      m2 = np.random.normal(scale=0.5, size=n)
      return m1+m2, m1-m2
  
  # Create list to pass to KDE integral function.
  m_list = []
  for i in range(100):
      m1, m2 = measure(5)
      m_list.append(m1.tolist())
      m_list.append(m2.tolist())
  
  # Call KDE integration function.
  print 'Integral result: ', kde_integration(m_list)
  </code></pre>
  
  <p>The <code>multiprocessing</code> in the code was suggested over at this question <a href="https://stackoverflow.com/questions/18538790/speed-up-sampling-of-kernel-estimate">Speed up sampling of kernel estimate</a> to speed up the code (which it does up to ~3.4x).</p>
  
  <p>The code works ok until I try to pass to the KDE function a list of more than ~62-63 elements (ie: I set a value over 63 in the line <code>for i in range(100)</code>) If I do that I get the following error:</p>
  
  <pre><code>Traceback (most recent call last):
    File "~/gauss_kde_temp.py", line 78, in &lt;module&gt;
      print 'Integral result: ', kde_integration(m_list)
    File "~/gauss_kde_temp.py", line 48, in kde_integration
      pool = Pool(processes=cores)
    File "/usr/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool
      return Pool(processes, initializer, initargs, maxtasksperchild)
    File "/usr/lib/python2.7/multiprocessing/pool.py", line 144, in __init__
      self._worker_handler.start()
    File "/usr/lib/python2.7/threading.py", line 494, in start
      _start_new_thread(self.__bootstrap, ())
  thread.error: can't start new thread
  </code></pre>
  
  <p>usually (9 out of 10 times) around the active thread <code>374</code>. I'm <em>way</em> out of my league in terms of <code>python</code> coding here and I have no clue as to how I could fix this issue. Any help will be much appreciated.</p>
  
  <hr>
  
  <h2>Add</h2>
  
  <p>I tried adding a <code>while</code> loop to prevent the code from using too many threads. What I did was replacing the <code>print threading.active_count()</code> line by this bit of code:</p>
  
  <pre><code>    # Print number of active threads.
      exit_loop = True
      while exit_loop:
          if threading.active_count() &lt; 300:
              exit_loop = False
          else:
              # Pause for 10 seconds.
              time.sleep(10.)
              print 'waiting: ', threading.active_count()
  </code></pre>
  
  <p>The code halted (ie: got stuck inside the loop) when it reached <code>302</code> active threads. I waited for more than 10 minutes and the code never exited the loop and the number of active threads never dropped from <code>302</code>. Shouldn't the number of active threads diminish after a while?</p>