title: How do I fix KeyError while parsing instagram?
body: <p>Everyone,</p>
  
  <p>I have a small script parsing names on Instagram.
  Recently started having this error:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/jpegcoma/vk/posting_instagram.py", line 361, in &lt;module&gt;
      main()
    File "/home/jpegcoma/vk/posting_instagram.py", line 293, in main
      table_of_content = get_stuf_from_url(urls)
    File "/home/jpegcoma/vk/posting_instagram.py", line 64, in get_stuf_from_url
      if json.loads(shared_data)["entry_data"]["ProfilePage"][0]["graphql"]["user"]["is_private"] == False:
  KeyError: 'ProfilePage'
  </code></pre>
  
  <p>Currently it is running on some server. However, I tryed it on my laptop and script was working.</p>
  
  <p>Here is the code that does the thing:</p>
  
  <pre><code>import requests
  import json
  import os
  import random
  from time import sleep
  import time
  import re
  from io import BytesIO
  from PIL import Image
  from multiprocessing import Pool
  from multiprocessing.dummy import Pool as ThreadPool
  
  file_name = 'users_names.txt'
  
  def create_folder_photos():    
     if os.path.isdir(os.path.join(os.getcwd(), "photos")) == False:
         os.makedirs(os.path.join(os.getcwd(), "photos"))
     else:
         pass
  
  def make_list_of_users_to_scrap(file_name):
     '''Opens file with instagram user_names.
        Every name should be on a new line.
        Prepares full URL for parsing.
        Returns list URLs'''
  
     path = os.path.join(os.getcwd(), file_name)
     base_url = 'https://www.instagram.com/'
     users_url_dic = []
     with open(path, 'r') as file:
         for name in file:
             users_url_dic.append(base_url + name.rstrip() + '/')
     return users_url_dic
  
  def parsed_data(shared_data):
     '''Get to ["edges"] node in shared_data from instagram'''
     return json.loads(shared_data)['entry_data']['ProfilePage'][0]['graphql']['user']['edge_owner_to_timeline_media']["edges"]
  
  def get_stuf_from_url(urls):    
     # Open a request session
     with requests.session() as s:
         # Add some headers in case
         s.headers['user-agent'] = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'
  
         pool = ThreadPool(5)
         d={}
         # Go throught all the URLs on instagram
         responce = pool.map(requests.get, urls)
         pool.close()
         pool.join()
         for i in responce:
             c = i.text
  
             if 30000 &lt; len(c) &lt; 180000:
                 # Clean html, take only content of 'sharedData' part
                 shared_data = c.split('window._sharedData = ')[1].split(';&lt;/script&gt;')[0]
                 # Check is accaunt is private
                 if json.loads(shared_data)["entry_data"]["ProfilePage"][0]["graphql"]["user"]["is_private"] == False:
                     # Go throught all the nodes:
                     # If video - pass.
                     # If photo - take {"Id":"URL"}
                     for node in parsed_data(shared_data)[::]:
                         if node['node']['is_video'] == False:                            
                             d[node['node']['id']] = node['node']['display_url']
                         else:
                             continue
                 else:
                     continue
             else:
                 continue
     return d
  
  
  def check_for_new(new_data_from_request):
     '''Open 'before_log.txt with previous loggs {'id':'url'}
        Check if any new data is presented.
        Write 2 new files:
        "added.txt" - new photos with url from the last time
        "before_log.txt" - updated log with all the ids and urls
        returns dic with added {'id':'url'} photos'''
  
     # Open a before_log.txt or say that no such file is presented.
     if os.path.isfile(os.path.join(os.getcwd(), 'before_log.txt')):
         with open(os.path.join(os.getcwd(), 'before_log.txt'), mode='r', encoding='utf8') as f_file:
             before_log = json.load(f_file)
     else:
         print('Need to make "before_log.txt" file to use the script!!!')
  
     # Get new data from "def get_stuf_from_url(urls):"            
     after = new_data_from_request
  
     # Check if any new photos is avaliable
     added = {i:after[i] for i in after if not i in before_log}  
  
     # Add new {key:value} to before_log
     for key, value in after.items():
         if key not in before_log.keys():
             before_log[key] = value
  
     # Write added and before_log for future use    
     with open(os.path.join(os.getcwd(), 'added.txt'), mode='w', encoding='utf8') as add_file:
         add_file.write(json.dumps(added) + '\n')
     with open(os.path.join(os.getcwd(), 'before_log.txt'), mode='w', encoding='utf8') as out_file:
         out_file.write(json.dumps(before_log) + '\n')
  
     print('We got {} new photos.'.format(len(added)))          
     return added
  
  def createFilename(url, name, folder):
     result = re.split(r'.jpg', url)
     slashSplit = result[0].split('/')
     if name == None:
         name = slashSplit[-1]
     ext = "jpg"
     file = '{}{}.{}'.format(folder, name, ext)
     return file
  
  def getImageFast(url, name=None, folder= os.path.join(os.getcwd(), "photos/")):
     '''Download new photos from instagram
        Creates a photos folder'''
     print("Downloading photos.....")
     file = createFilename(url, name, folder)
     r = requests.get(url, stream=True)
     i = Image.open(BytesIO(r.content))
     i.save(file) 
  </code></pre>
  
  <p>I guess the problem is somewhere in here </p>
  
  <pre><code>if json.loads(shared_data)["entry_data"]["ProfilePage"][0]["graphql"]["user"]["is_private"] == False:
  </code></pre>
  
  <p>Some examples of parsed names on instagram:</p>
  
  <pre><code>_nail_ann_
  alena.nails.tallinn
  alyne_nails
  anna_nails_erbil
  aquarelle_nailstudio
  cantinhodalara_nails
  </code></pre>
  
  <p>In a shorter version it does work as intended:</p>
  
  <pre><code>urls = 'https://www.instagram.com/_linails_/'
  responce = requests.get(urls)
  response_text= responce.text
  shared_data = response_text.split('window._sharedData = ')[1].split(';&lt;/script&gt;')[0]
  #     print(shared_data)
  d={}
  f = json.loads(shared_data)['entry_data']['ProfilePage'][0]['graphql']['user']['edge_owner_to_timeline_media']["edges"]
  for node in f[::]:
      if node['node']['is_video'] == False:
          d[node['node']['id']] = node['node']['display_url']
      else:
          continue
  print (d)
  </code></pre>
  
  <p>After running it I'm getting all the URL and ids I need:</p>
  
  <pre><code>{
  '2073876006313498489': 'https://scontent-lax3-2.cdninstagram.com/vp/6e5c8c22e54aa0c853ee88db05dc79bf/5E1BBCA4/t51.2885-15/e35/65217639_634723610367271_4450988163128206846_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=107',
  '2024498169693824735': 'https://scontent-lax3-2.cdninstagram.com/vp/39188272c2305ed250ad466c7a715b91/5E2F4B15/t51.2885-15/e35/56352792_132736304460754_8293153588685230511_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=110',
  '2023266828574689831': 'https://scontent-lax3-2.cdninstagram.com/vp/f313d44c5bd398a8e6b3f04fb7dbb739/5E2BBB71/t51.2885-15/e35/56578225_1055286461334820_1507399846418163801_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=104',
  '2016110942668250132': 'https://scontent-lax3-2.cdninstagram.com/vp/349bbf6a920e440a4e71d5b2d149a61b/5E2BB7FE/t51.2885-15/e35/53745148_280247652888437_7055433742029015170_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=105',
  '2012783478764415885': 'https://scontent-lax3-2.cdninstagram.com/vp/72dfe2f67b6dc1ea75e2ddd832384475/5E1936CE/t51.2885-15/e35/54512001_2155869857812437_3429908829670998264_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=109',
  '2012464856204377926': 'https://scontent-lax3-2.cdninstagram.com/vp/5aefc3a4e047b08dc94366b0723f170d/5E32A5D3/t51.2885-15/e35/54513720_424627718315641_3423874379564248817_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=101',
  '2008135031155279090': 'https://scontent-lax3-2.cdninstagram.com/vp/09cc2e7631c115a0131bda6f597dde60/5E1B4C09/t51.2885-15/e35/53156526_1025783867629475_1693464480553968728_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=111',
  '2004990756607236359': 'https://scontent-lax3-2.cdninstagram.com/vp/5da04c640d70b52a3e3073667985f8e3/5E2A62EB/t51.2885-15/e35/54266355_225989821600275_560245954300705815_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=103',
  '2002388991416431681': 'https://scontent-lax3-2.cdninstagram.com/vp/77bb0bf9878ca2d175dbd51350c1ef03/5E37974D/t51.2885-15/e35/53217305_581829868953428_1147405223061346025_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=108',
  '2001312091952564411': 'https://scontent-lax3-2.cdninstagram.com/vp/64326e9675b389a7997ed86980cba7bc/5E30992A/t51.2885-15/e35/54513758_391705221628749_737855016941810571_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=109',
  '1999425996532762294': 'https://scontent-lax3-2.cdninstagram.com/vp/4c4a5ee2b0ad46d6e3eeb1a30c1e9130/5E1BC2CA/t51.2885-15/e35/52639028_2494445767266095_4453054116414455580_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=111',
  '1993652807341169347': 'https://scontent-lax3-2.cdninstagram.com/vp/d6d8ffef7fd23d1f12b14282d3bc9aca/5E17386F/t51.2885-15/e35/52024250_786523341734970_6491735451376989098_n.jpg?_nc_ht=scontent-lax3-2.cdninstagram.com&amp;_nc_cat=106'
  }
  </code></pre>