title: Export Web Scraped Table to Excel
body: <p>I am having trouble getting pandas to export some web scraped data in the format I want.</p>
  
  <p>I want to visit each URL in <code>URLs</code> and get the various elements from that page and put them into an Excel spreadsheet with the column names specified. I then want to visit the next URL in <code>URLs</code> and put this data on the next row of the Excel sheet so that I have an Excel sheet with 6 columns and three rows of data, one for each plant (each plant in on a separate URL).</p>
  
  <p>Currently I have an error saying <code>ValueError: Length mismatch: Expected axis has 18 elements, new values have 6 elements</code> as the new records are being placed horizontally next to each other rather than on a new row in Excel and Pandas isn't expecting that. </p>
  
  <p>Can someone help pls?
  Thanks</p>
  
  <pre><code>import csv
  import pandas as pd
  from pandas import ExcelWriter
  from pandas import ExcelFile
  import numpy as np
  from urllib2 import urlopen
  import bs4
  from bs4 import BeautifulSoup
  
  
  URLs = ["http://adbioresources.org/map/ajax-single/27881",
  "http://adbioresources.org/map/ajax-single/27967",
  "http://adbioresources.org/map/ajax-single/27880"]
  
  mylist = []
  
  for plant in URLs:
      soup = BeautifulSoup(urlopen(plant),'lxml')
  
      table = soup.find_all('td')
      for td in table:
          mylist.append(td.text)
  
      heading2 = soup.find_all('h2')
      for h2 in heading2:
          mylist.append(h2.text)
  
      para = soup.find_all('p')   
      for p in para:
          mylist.append(p.text)
  
  df = pd.DataFrame(mylist)
  transposed_df = df.T
  transposed_df.columns = 
  ['Status','Type','Capacity','Feedstock','Address1','Address2']
  writer = ExcelWriter('Pandas-Example.xlsx')
  transposed_df.to_excel(writer,'Sheet1',index=False)
  writer.save()
  </code></pre>