Q:

Scraping links from href on Sephora website

Hi So I am trying to scrap the links for all the products a specific page on Sephora.  My code only gives me the first 12 links while there are 48 products on the website.  I think this is because Sephora is a User-Interactive-website(Please correct me if I am wrong) so it doesn't load the rest.  But I do not know how to get the rest.  Please send some help!  Thank you!!!
Here is my code:
from bs4 import BeautifulSoup
import requests

url = "https://www.sephora.com/brand/estee-lauder/skincare"
r = requests.get(url)
data = r.text
soup = BeautifulSoup(data,'html.parser')

link_list = []
keyword = 'product'
for link in soup.findAll('a'):
    href = link.get('href')
    if keyword in href:
        link_list.append('https://www.sephora.com/' + href)
else:
    continue

A:

If you take a look at the source code, you will see their data stored as a json object. You can get the json object by this:
from bs4 import BeautifulSoup
import requests
import json

url = "https://www.sephora.com/brand/estee-lauder/skincare"
r = requests.get(url)
data = r.text
soup = BeautifulSoup(data,'html.parser')

data = json.loads(soup.find('script', id='linkJSON').text)
products = data[3]['props']['products']

prefix = "https://www.sephora.com"

url_links = [prefix+p["targetUrl"] for p in products]
print(url_links)

By investigating the json data, you can find where the links stored. To view the json data more clearly, I use this website: https://codebeautify.org/jsonviewer

