title: dataframe from 2 classes beautifulsoup
body: <p>I want to extract Arbeitsatmosphare rank and Stadt information based on review data from below website for ALL pages, so desired output should be as in example below</p>
  
  <pre><code>         Arbeitsatmosphare | Stadt
     1.      4.00            | Berlin     
     2.      5.00            | Frankfurt
     3.      3.00            | Munich
     4.      5.00            | Berlin
     5.      4.00            | Berlin
  </code></pre>
  
  <p>Below code extracts pro data from website from all pages and works fine. I tried to update it and to add 2 lists, Arbeitsatmosphare rank and Stadt in it and break cycle if Arbeitsatmosphare rank info is missing, but my code is not working. Can you help?</p>
  
  <pre><code>pro = []
  
  with requests.Session() as session:
      session.headers = {
          'x-requested-with': 'XMLHttpRequest'
      }
      page = 1
      while True:
          print(f"Processing page {page}..")
          url = f'https://www.kununu.com/de/volkswagen/kommentare/{page}'
          response = session.get(url)
          soup = BeautifulSoup(response.text, 'html.parser')
          new_comments = [
              pro.find_next_sibling('p').get_text()
              for pro in soup.find_all('h2', text='Pro')
          ]
          if not new_comments:
              print(f"No more comments. Page: {page}")
              break
          pro += new_comments
          print(pro)
          #print(len(pro))
          page += 1
  print(pro)
  </code></pre>
  
  <p>UPD
  Adding my code that is not working, however I think that there should be more simple solution</p>
  
  <pre><code>Arbeit = []
  Stadt=[]
  
  with requests.Session() as session:
      session.headers = {
          'x-requested-with': 'XMLHttpRequest'
      }
      page = 1
      while True:
          print(f"Processing page {page}..")
          url = f'https://www.kununu.com/de/volkswagen/kommentare/{page}'
          response = session.get(url)
          soup = BeautifulSoup(response.text, 'html.parser')
          new_comments1 = [
              Arbeit.find_next_sibling('span').get_text()
              for Arbeit in soup.find_all('span', text='Arbeitsatmosphäre')
          ]
          new_comments2 = [
              Stadt.find_next_sibling('div').get_text()
              for Stadt in soup.find_all('div', text='Stadt')
          ]
          if not new_comments1:
              print(f"No more comments. Page: {page}")
              break
          Arbeit += new_comments1
          Stadt += new_comments2
          print(Arbeit)
          print(Stadt)
          #print(len(pro))
          page += 1
  </code></pre>