title: Scrape a PHP variable with Beautiful Soup, urllib2, and Python
body: <p>I am attempting to use Beautiful Soup and Python to scrape a PHP value from a website. </p>
  
  <p>I also attempted to use the lxml library.</p>
  
  <p>Is there a way to do it this way? Or should I use something different? Thanks in advance.</p>
  
  <p><strong>Steps Followed</strong></p>
  
  <ol>
  <li>Find nested HTML tag desired. In this case "320"</li>
  <li>Parse HTML page.</li>
  <li>Search for the first "div" tag. </li>
  <li>Attempt to search all children of the "div" tag.</li>
  <li>Output the entire HTML page to a text file. </li>
  <li>Grep the desired "span" tag name.</li>
  <li>Notice that the value is a PHP variable.</li>
  </ol>
  
  <p><strong>xPath:</strong></p>
  
  <pre><code>//*[@id="monetary_offer_content"]/div[1]/div[2]/div/div[1]/h3/span
  </code></pre>
  
  <p><strong>CSS Selector</strong></p>
  
  <pre><code>monetary_offer_content &gt; div.monetary_offer &gt; div.offers.clear &gt; div &gt; div.clearfix &gt; h3 &gt; span
  </code></pre>
  
  <p><strong>HTML:</strong></p>
  
  <pre><code>&lt;span data-oldoffer=""&gt;320&lt;/span&gt;
  </code></pre>
  
  <p><strong>After outputting the entire HTML page to a .txt file and then grepping the element name</strong>:</p>
  
  <pre><code>| =&gt; cat text.txt | grep data-oldoffer
        &lt;h3&gt;$&lt;span data-oldoffer&gt;&lt;%= value['offer'] %&gt;&lt;/span&gt;&lt;/h3&gt;
        &lt;h3&gt;$&lt;span data-oldoffer&gt;&lt;%= value['offer'] %&gt;&lt;/span&gt;&lt;/h3&gt;
  </code></pre>
  
  <p><strong>Python Code</strong></p>
  
  <pre><code>from bs4 import BeautifulSoup
  import urllib2
  url="http://website_url.com"
  page=urllib2.urlopen(url)
  soup = BeautifulSoup(page.read(), "lxml")
  print(soup)
  </code></pre>