title: How to Multiprocessing pool with pymongo in python 2.7
body: <p>I'm working with Pymongo and Multiprocessing Pool to run 10 processes and get data from API and insert the output into mongodb.</p>
  
  <p>I think I've done something wrong the way I wrote my code, as python shows double connections open than usually it would; for example: If I run 10 processes Mongodb will output 20 or more connections been established and I will get the following warning on startup:</p>
  
  <blockquote>
    <p>UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: <a href="http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing" rel="nofollow noreferrer">http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing</a>></p>
  </blockquote>
  
  <p>Even tho I did input connect=False at the connector client in mongodb.
  here's an example code to understand how I use pymongo and requests API to send the requests in pool:</p>
  
  <pre><code># -*- coding: utf-8 -*-
  #!/usr/bin/python
  
  import json # to decode and encode json
  import requests # web POST and GET requests. 
  from pymongo import MongoClient # the mongo driver / connector
  from bson import ObjectId # to generate bson object for MongoDB
  from multiprocessing import Pool # for the multithreading
  
  # Create the mongoDB Database object, declare collections
  client = MongoClient('mongodb://192.168.0.1:27017,192.168.0.2:27017./?replicaSet=rs0', maxPoolSize=20, connect=False)
  index = client.database.index
  users = client.database.users
  
  def get_user(userid):
  
      params = {"userid":userid}
      r = requests.get("https://exampleapi.com/getUser",params=params)
      j = json.loads(r.content)
      return j
  
  def process(index_line):
  
      user = get_user(index_line["userid"])
      if(user):
          users.insert(user)
  
  def main():
  
      # limit to 100,000 lines of data each loop
      limited = 100
      # skip number of lines for the loop (getting updated)
      skipped = 0
      while True:
          # get cursor with data from index collection
          cursor = index.find({},no_cursor_timeout=True).skip(skipped).limit(limited)
          # prepare the pool with threads
          p = Pool(10)
          # start multiprocessing the pool with the dataset
          p.map(process, cursor)
          # after pool finished, kill it with fire
          p.close()
          p.terminate()
          p.join()
          # after finishing the 100k lines, go for another round, inifnite.
          skipped = skipped + limited
          print "[-] Skipping %s " % skipped
  
  if __name__ == '__main__':
      main()
  </code></pre>
  
  <p>Is there any issue in the algorithm of my code? is there any way to make it more efficient and to work better and have better control over my pool?</p>
  
  <p>I've been researching quite a while but couldn't find a way to do what I want to do in a better way, would love to get some help.</p>
  
  <p>Thank you.</p>