title: Split large json file and assign each part to a process
body: <p>I am trying to refine a large JSON data set ~ 14 GiB (12 Million lines) with multiple processes, so that i can be faster. I created two  <code>multiprocessing.Queue</code> instances, <code>in_q</code> and <code>out_q</code>. They would read the data in the <code>data-2021-09-29.jsonl</code> file (it's the data set), and output the line that contains the data that I am interested in and write it into another file <code>stokes_DE_file.jsonl</code> (This is the part where i am refining the data set, the output file is the refined version of the data set). I have 16 CPUs on my machine, let's say I want to split the file in 16 parts, the number of lines would be the variable LINES_PER_PROCESS. How can I assign to each process one part of the file ? Following is the code that I've written so far.
  Just starting with multiprocessing module in python. Here's my code:</p>
  <pre><code>import multiprocessing as mp
  import threading
  import json
  import reverse_geocoder as rg
  
  LINES_PER_PROCESS = 12137928/(mp.cpu_count()-1)
  
  def geocode_worker(in_q, out_q):
      while True:
          strike = in_q.get()
          if strike is None:
              out_q.put(None)
              return
          strike_location = (strike['lat'], strike['lon'])
          if rg.search(strike_location)[0]['cc'] == 'DE':
              out_q.put('{}\n'.format(strike))
  
  def file_write_worker(out_q, fileobj, worker_count):
      while worker_count:
          for msg in iter(out_q.get, None):
              
              if msg is None:
                  worker_count -= 1
              fileobj.write(msg)
  
  def get_germany_strokes(jsonl_file):
      
      worker_count = mp.cpu_count() - 1
      in_q, out_q = mp.Queue(), mp.Queue()
      processes = [mp.Process(target=geocode_worker, args=(in_q, out_q)) for _ in range(worker_count)]
      
      for p in processes:
          p.start()
          
      with open('strokes_DE_file.json', newline='') as strokes_DE_file:
          file_writer = threading.Thread(target=file_write_worker, args=(out_q, strokes_DE_file, worker_count))
          file_writer.start()
          
      with open(jsonl_file, newline='') as file:
          next(file)
          for line in file:
              strokes = json.loads(line)['strokes']
              for strike in strokes:
                  in_q.put(strike)
                      
  get_germany_strokes('data-2021-09-29.jsonl')
  </code></pre>