title: How to apply a custom function to pair wise elements of an array in parallel?
body: <p>I have an array and I want to compare each element with every other element and build a cross comparison table. It can be easily implemented by nested for loops, but it's computation time increases exponentially with input array size, so I want to implement a parallel processing approach reduce time consumption at larger sizes.</p>
  
  <p>I have an array, like <code>a = [1,2,3]</code> and I want to apply a custom function like:</p>
  
  <pre><code>def add_two_numbers(x,y):
       return x+y
  </code></pre>
  
  <p>A simple nested for loop implementation would be like:</p>
  
  <pre><code>array = [1,2,3]
  matrix = np.zeros([3,3])
  for i, one_element in enumerate(array):
      for j, other_element in enumerate(array):
          matrix[i][j] = add_two_numbers(one_element, other_element)
  </code></pre>
  
  <p>With the output being:</p>
  
  <pre><code>&gt;&gt;&gt; matrix
      1   2   3
  ______________
  1 | 2   3   4
  2 | 3   4   5
  3 | 4   5   6
  </code></pre>
  
  <p>What would be a good approach to apply parallel processing in python for large array sizes?<br>
  I used the process class in python multiprocessing library to create n processes for an n element array, but every process opens up a file at the backend and after 1024 parallel processes I get a "Too many open files" exception. And I have to make the matrix a global variable, so that every process updates a specific element.</p>
  
  <pre><code>import multiprocessing as mp
  
  def add_two_numbers_process(one_element, array, i):
      global matrix
      for j, other_element in enumerate(array):
          matrix[i][j] = add_two_numbers(one_element, other_element)
      return
  
  processes = []
  for i, one_element in enumerate(array):
      p = mp.Process(target=add_two_numbers_process, args=(one_element, array, i))
      processes.append(p)
      p.start()
  
  for process in processes:
      process.join()
  </code></pre>
  
  <p>I also used the Pool class, but that is taking 1000 times more time than the process class, which doesn't seem feasible.</p>
  
  <pre><code>import multiprocessing as mp
  
  def add_two_numbers_pool(one_element, array, i):
      row = [0 for x in range(len(array))]
      for j, other_element in enumerate(array):
          row[j] = add_two_numbers(one_element, other_element)
      return row
  
  pool = mp.Pool(mp.cpu_count())
  matrix = [pool.apply(add_two_numbers_pool, args=(one_element, array, i)) for i, one_element in enumerate(array)]
  pool.close()
  </code></pre>
  
  <p>I can't think of an approach using dask distributed. Could dask distributed prove helpful in this case?</p>