title: How do I take advantage of parallel processing in gcloud?
body: <p>Google Cloud offers a virtual machine instance with 96 cores. </p>
  
  <p>I thought that having those 96 cores you would be able to divide your program into 95 slices while leaving one other core to run the program and you could thus run the program 95 times faster. </p>
  
  <p>It's not working out that way however. </p>
  
  <p>I'm running a simple program in Python which simply counts to 20 million.  </p>
  
  <p>On my Mac book it takes 4.6 seconds to run this program in serial, and 1.2 seconds when I divide this process into 4 sections and run it in parallel. </p>
  
  <p>My Mac book has the following specs:</p>
  
  <p>version 10.14.5<br>
  processor 2.2 GHz Intel Core i7<br>
  Mid 2015<br>
  Memory 16GB 1600 MHz DDr3  </p>
  
  <p>The computer that gcloud offers basically is just a tad bit slower.  </p>
  
  <p>When I run the program in serial it takes the same amount of time.  When I divide the program into 95 division it actually takes more time: 2.6 seconds.  When I divide the program into 4, it takes 1.4 seconds to complete.  The computer has the following specs:</p>
  
  <p>n1-highmem-96 (96 vCPUs, 624 GB memory)  </p>
  
  <p>I need the high memory because I'm going to index a corpus of 14 billion words.  I'm going to save the locations of the 100,000 most frequent words.  </p>
  
  <p>When I then save those locations, I'm going to pickle them.  </p>
  
  <p>Pickling files takes up an enormous amount of time and will probably consume 90% of the time.  </p>
  
  <p>It is for this reason that I need to pickle the files as little as often.  So if I can put the locations into python objects for as long as possible then I will save myself a lot of time.</p>
  
  <p>Here is the python program I'm using just in case it helps:</p>
  
  <pre><code>import os, time
  
  p = print
  en = enumerate
  
  
  def count_2_million(**kwargs):
      p('hey')
      start = kwargs.get("start")
      stop = kwargs.get("stop")
      fork_num = kwargs.get('fork_num')
      for x in range(start, stop):
          pass
      b = time.time()
      c = round(b - kwargs['time'], 3)
      p(f'done fork {fork_num} in {c} seconds')
  
  
  def divide_range(divisions: int, total: int, idx: int, begin=0):
      sec = total // divisions
      start = (idx * sec) + begin
      if total % divisions != 0 and idx == divisions:
          stop = total
      else:
          stop = start + sec
      return start, stop
  
  
  def main_fork(func, total, **kwargs):
      forks = 4
      p(f'{forks} forks')
      kwargs['time'] = time.time()
      pids = []
      for i in range(forks):
          start1, stop1 = 0, 0
          if total != -1:
              start1, stop1 = divide_range(4, total, i)
  
          newpid = os.fork()
          pids.append(newpid)
          kwargs['start'] = start1
          kwargs['stop'] = stop1
          kwargs['fork_num'] = i
  
          if newpid == 0:
              p(f'fork num {i} {start1} {stop1}')
              child(func, **kwargs)
  
      return pids
  
  
  def child(func, **kwargs):
      func(**kwargs)
      os._exit(0)
  
  
  main_fork(count_2_million, 200_000_000, **{})
  </code></pre>