title: How to scrape full paper citation from Google Scholar search results (Python)?
body: <p>I am trying to web scrape some useful data on academic papers from Google Scholar.
  So far I've had no problem getting the Title, Year of publication, Citation count, and &quot;Cited by&quot; URL.</p>
  <p>I would like now to get the full citation that includes the full authors' list, journal, pages (if any) etc... (see snapshot below)
  <a href="https://i.stack.imgur.com/n0Mof.png" rel="nofollow noreferrer">Full APA citation appearing when clicking on the double quote (circled in red)</a></p>
  <p>I use ScraperAPI to handle proxies and CAPTCHAs (they offer 5000 requests for free).</p>
  <p>Below is the code I have (I'm aware it's very heavy and not optimal at all, but does the job for now):</p>
  <pre class="lang-py prettyprint-override"><code>import requests
  import numpy as np
  import pandas as pd
  import re
  from bs4 import BeautifulSoup
  
  APIKEY = &quot;????????????????????&quot;
  BASE_URL = f&quot;http://api.scraperapi.com?api_key={APIKEY}&amp;url=&quot;
  
  def scraper_api(query, n_pages):
      &quot;&quot;&quot;Uses scraperAPI to scrape Google Scholar for 
      papers' Title, Year, Citations, Cited By url returns a dataframe
      ---------------------------
      parameters:
      query: in the following format &quot;automation+container+terminal&quot;
      n_pages: number of pages to scrape
      ---------------------------
      returns:
      dataframe with the following columns: 
      &quot;Title&quot;: title of each papers
      &quot;Year&quot;: year of publication of each paper
      &quot;Citations&quot;: citations count
      &quot;cited_by_url&quot;: URL given by &quot;cited by&quot; button, reshaped to allow further
                      scraping
      ---------------------------&quot;&quot;&quot;
  
      pages = np.arange(0,(n_pages*10),10)
      papers = []
      for page in pages:
          print(f&quot;Scraping page {int(page/10) + 1}&quot;)
          webpage = f&quot;https://scholar.google.com/scholar?start={page}&amp;q={query}&amp;hl=fr&amp;as_sdt=0,5&quot;
          url = BASE_URL + webpage
          response = requests.get(url)
          soup = BeautifulSoup(response.content, &quot;html.parser&quot;)
          
          for paper in soup.find_all(&quot;div&quot;, class_=&quot;gs_ri&quot;):
              # get the title of each paper
              title = paper.find(&quot;h3&quot;, class_=&quot;gs_rt&quot;).find(&quot;a&quot;).text
              if title == None:
                  title = paper.find(&quot;h3&quot;, class_=&quot;gs_rt&quot;).find(&quot;span&quot;).text
              # get the year of publication of each paper
              txt_year = paper.find(&quot;div&quot;, class_=&quot;gs_a&quot;).text
              year = re.findall('[0-9]{4}', txt_year)
              if year:
                  year = list(map(int,year))[0]
              else:
                  year = 0
              # get number of citations for each paper
              txt_cite = paper.find(&quot;div&quot;, class_=&quot;gs_fl&quot;).find_all(&quot;a&quot;)[2].string
              if txt_cite:
                  citations = re.findall('[0-9]+', txt_cite)
                  if citations:
                      citations = list(map(int,citations))[0]
                  else:
                      citations = 0
              else:
                  citations = 0
              # get the &quot;cited_by&quot; url for later scraping of citing papers
              # had to extract the &quot;href&quot; tag and then reshuffle the url as not
              # following same pattern for pagination
              urls = paper.find(&quot;div&quot;, class_=&quot;gs_fl&quot;).find_all(href=True)
              if urls:
                  for url in urls:
                      if &quot;cites&quot; in url[&quot;href&quot;]:
                          cited_url = url[&quot;href&quot;]
                          index1 = cited_url.index(&quot;?&quot;)
                          url_slices = []
                          url_slices.append(cited_url[:index1+1])
                          url_slices.append(cited_url[index1+1:])
  
                          index_and = url_slices[1].index(&quot;&amp;&quot;)
                          url_slices.append(url_slices[1][:index_and+1])
                          url_slices.append(url_slices[1][index_and+1:])
                          url_slices.append(url_slices[3][:23])
                          del url_slices[1]
                          new_url = &quot;https://scholar.google.com.tw&quot;+url_slices[0]+&quot;start=00&amp;hl=en&amp;&quot;+url_slices[3]+url_slices[1]+&quot;scipsc=&quot;
              else:
                  new_url = &quot;no citations&quot;
              # appends everything in a list of dictionaries    
              papers.append({'title': title, 'year': year, 'citations': citations, 'cited_by_url': new_url})
      # converts the list of dict to a pandas df
      papers_df = pd.DataFrame(papers)
      return papers_df
  </code></pre>
  <p>I would like to retrieve the full APA citation but seems like it's not on the same HTML page and there are no <code>href</code> associated.</p>
  <p>If you have any lead that would help me a lot!! Thanks :)</p>