title: How can I attach the corresponding HTML ending to each piece of scraped text?
body: <p>In short I am making a program which scrapes specific citations from a list of URLs. I need the result to also have the MR number from the corresponding URL ending, added to each scraped citation.</p>
  <pre class="lang-py prettyprint-override"><code>from bs4 import BeautifulSoup
  import requests
  import re
  import pandas as pd
  import itertools
  
  #DEFINING TWO FUNCTIONS TO HELP FINDING ONLY THE WANTED CITATIONS:
  MATCH_ALL = r'.*'
  
  
  def like(string):
      &quot;&quot;&quot;
      Return a compiled regular expression that matches the given
      string with any prefix and postfix, e.g. if string = &quot;hello&quot;,
      the returned regex matches r&quot;.*hello.*&quot;
      &quot;&quot;&quot;
      string_ = string
      if not isinstance(string_, str):
          string_ = str(string_)
      regex = MATCH_ALL + re.escape(string_) + MATCH_ALL
      return re.compile(regex, flags=re.DOTALL)
  
  
  def find_by_text(soup, text, tag, **kwargs):
      &quot;&quot;&quot;
      Find the tag in soup that matches all provided kwargs, and contains the
      text.
  
      If no match is found, raise ValueError.
      &quot;&quot;&quot;
      elements = soup.find_all(tag, **kwargs)
      matches = []
      for element in elements:
          if element.find(text=like(text)):
              matches.append(element)
      if len(matches) == 0:
          raise ValueError(&quot;No matching citations were found&quot;)
      else:
          return matches
  
  #DEFINING URL LIST:
  
  base_URL = &quot;https://sis1.host.cs.st-andrews.ac.uk/GAP/&quot;
  mrn = [&quot;MR4044696&quot;, &quot;MR2900886&quot;, &quot;MR3169623&quot;, &quot;MR4180136&quot;]
  url_list = []
  
  for i in range(len(mrn)):
      url = (base_URL + mrn[i] + '.html')
      url_list.append(url)
      
  print(url_list)
  
  all_content = []
  all_matches = []
  
  #THIS IS THE LOOP WHICH ITERATES THROUGH AND GATHER THE RESULTS:
  
  for url in url_list:
      page = requests.get(url)
      soup = BeautifulSoup(page.content, 'html.parser')
      match = (find_by_text(soup, 'GAP', 'li'))
      all_matches.append(match)
  print(all_matches)
  
  OUTPUT
  
  &quot;[[&lt;li&gt;
    The GAP Group, GAP – groups, algorithms and programming, version 4.10, Available from http://www.gap-system.org, 2018. 
  
  &lt;/li&gt;], [&lt;li&gt;
    The GAP Group, &lt;span class=&quot;MathTeX&quot;&gt;$GAP$&lt;/span&gt;&lt;script type=&quot;math/tex&quot;&gt;GAP&lt;/script&gt; groups, algorithms, and programming, version 4.4.12 (2008), http://www.gap-system.org. 
  
  &lt;/li&gt;], [&lt;li&gt;
    Distler, A., Mitchell, J. D. (2011). &lt;span class=&quot;it&quot;&gt;Smallsemi - A Library of Small Semigroups.&lt;/span&gt; http://url.com, Oct A GAP 4 package [5], Version 0.6.4. 
  
  &lt;/li&gt;, &lt;li&gt;
    The GAP Group, (2008). (http://www.gap-system.org). &lt;span class=&quot;it&quot;&gt;GAP–Groups, Algorithms, and Programming, Version 4.4.12.&lt;/span&gt;
  &lt;/li&gt;], [&lt;li&gt;
    The GAP Group, 2019. GAP – Groups, Algorithms, and Programming, Version 4.10.1; https://www.gap-system.org. 
  
  &lt;/li&gt;]]&quot;
  
  </code></pre>
  <p>I need each result to have the corresponding MR number added to the beginning, for instance:</p>
  <pre><code>MR4044696, The GAP Group, GAP – groups, algorithms and programming, version 4.10, Available from http://www.gap-system.org, 2018. 
  
  MR2900886, Distler, A., Mitchell, J. D. (2011). &lt;span class=&quot;it&quot;&gt;Smallsemi - A Library of Small Semigroups.&lt;/span&gt; http://url.com, Oct A GAP 4 package [5], Version 0.6.4. 
  
  MR3169623, The GAP Group, (2008). (http://www.gap-system.org). &lt;span class=&quot;it&quot;&gt;GAP–Groups, Algorithms, and Programming, Version 4.4.12.&lt;/span&gt;
  </code></pre>
  <p>I think I need to add something like:</p>
  <pre><code>for i in range(len(url_list)):
      matching_mrn = mrn[i]
  </code></pre>
  <p>in the <code>find_by_text</code> function but wherever I add it everything breaks and it does not work well.
  For the love of god I cannot make this work, please help me out and a massive thank you !</p>