title: Unknown reason of dead multiprocessing in python-3.5
body: <p>I have a data frame <code>df</code> contains information of two-hundred-thousand items. And now I want to measure their pair-wise similarity and pick up the top <code>n</code> pairs. That means, I should calculate nearly tens of billions calculation. Given the huge computation cost, I choose to run it via multiprocessing. Here is the code I have written so far:</p>
  
  <pre><code>user_list = (list(range(1, 200001))
  def gen_pair():
      for u1 in reversed(user_list):
          for u2 in reversed(list(range(1, u1))):
              yield (u1, u2)
  
  def cal_sim(u_pair):
      u1, u2 = u_pair
      # do sth quite complex...
      sim = sim_f(df[u1], df[u2])
      if sim &lt; 0.5:
          return None
      else:
          return (u1, u2, sim)
  
  with multiprocessing.Pool(processes=3) as pool:
      vals = pool.map(cal_sim, gen_pair())
      for v in vals:
          if v is not None:
              with open('result.txt', 'a') as f:
                  f.write('{0}\t{1}\t{2}\n'.format(v[0], v[1], v[2]))
  </code></pre>
  
  <p>When I just take the first 1000 users, it works quite well. But when I take all of them, it is kind of dead and no single word in <code>result.txt</code>. But if I add the number of processes, it also be dead. I wonder what is the reason and how can I fix it? Thanks in advance.</p>
  
  <p>EDIT:</p>
  
  <p>Here is my code of <code>sim_f</code>:</p>
  
  <pre><code>def sim_f(t1, t2):
      def intersec_f(l1, l2):
          return set(l1)&amp;set(l2)
  
      def union_f(l1, l2):
          return set(l1)|set(l2)
  
      a_arr1, a_arr2 = t1[0], t1[1]
      b_arr1, b_arr2 = t2[0], t2[1]
      sim =  float(len(union_f(intersec_f(a_arr1, a_arr2), intersec_f(b_arr1, b_arr2))))\
          / float(len(union_f(union_f(a_arr1, a_arr2), union_f(b_arr1, b_arr2))))
      return sim
  </code></pre>