title: Scrapy, Crawling Reviews on Tripadvisor: extract more hotel and user information
body: <p>in need to extract more information from tripAdvisor</p>
  
  <p>my code:</p>
  
  <pre><code> item = TripadvisorItem()
  
  item['url'] = response.url.encode('ascii', errors='ignore') 
  
  item['state'] =  hxs.xpath('//*[@id="PAGE"]/div[2]/div[1]/ul/li[2]/a/span/text()').extract()[0].encode('ascii', errors='ignore')
  if(item['state']==[]):
      item['state']=hxs.xpath('//*[@id="HEADING_GROUP"]/div[2]/address/span/span/span[contains(@class,"region_title")][2]/text()').extract()
  
  item['city'] =  hxs.select('//*[@id="PAGE"]/div[2]/div[1]/ul/li[3]/a/span/text()').extract()
  if(item['city']==[]):
      item['city'] =hxs.xpath('//*[@id="HEADING_GROUP"]/div[2]/address/span/span/span[1]/span/text()').extract()
  if(item['city']==[]):
    item['city']=hxs.xpath('//*[@id="HEADING_GROUP"]/div[2]/address/span/span/span[3]/span/text()').extract()
  item['city']= item['city'][0].encode('ascii', errors='ignore')
  
  item['hotelName'] =  hxs.xpath('//*[@id="HEADING"]/span[2]/span/a/text()').extract()
  item['hotelName']=item['hotelName'][0].encode('ascii', errors='ignore')
  
  reviews = hxs.select('.//div[contains(@id, "review")]')
  </code></pre>
  
  <p>1. For every hotel in tripAdvisor, there is a id number for the hotel. like 80075 for this hotel: <a href="http://www.tripadvisor.com/Hotel_Review-g60763-d80075-Reviews-Amsterdam_Court_Hotel-New_York_City_New_York.html#REVIEWS" rel="nofollow">http://www.tripadvisor.com/Hotel_Review-g60763-d80075-Reviews-Amsterdam_Court_Hotel-New_York_City_New_York.html#REVIEWS</a></p>
  
  <p>how can i extract this id from the TA item?</p>
  
  <ol start="2">
  <li><p>More information i need for every hotel : shortDescription, stars, zipCode, country and coordinates(long, lat). Can i extract this things?</p></li>
  <li><p>i need to extract for every review the traveller type. how?
  my code for review:</p>
  
  <pre><code>for review in reviews:
  it = Review()
  
  it['state'] =  item['state']
  
  it['city'] =   item['city']
  
  it['hotelName'] = item['hotelName']
  
  it['date'] = review.xpath('.//div[1]/div[2]/div/div[2]/span[2]/@title').extract()
  if(it['date']==[]):
      it['date']=review.xpath('.//div[1]/div[2]/div/div[2]/span[2]/text()').extract()
  if(it['date']!=[]):
      it['date']=it['date'][0].encode('ascii', errors='ignore').replace("Reviewed","").strip()
  
  it['userName'] = review.xpath('.//div[contains(@class,"username mo")]/span/text()').extract()
  if (it['userName']!=[]):
          it['userName']=it['userName'][0].encode('ascii', errors='ignore')
  
  it['userLocation'] = ''.join(review.xpath('.//div[contains(@class,"location")]/text()').extract()).strip().encode('ascii', errors='ignore')
  
  it['reviewTitle'] = review.xpath('.//div[1]/div[2]/div[1]/div[contains(@class,"quote")]/text()').extract()
  if(it['reviewTitle']!=[]):
      it['reviewTitle']=it['reviewTitle'][0].encode('ascii', errors='ignore')
  else:
      it['reviewTitle'] = review.xpath('.//div[1]/div[2]/div/div[1]/a/span[contains(@class,"noQuotes")]/text()').extract()
      if(it['reviewTitle']!=[]):
          it['reviewTitle']=it['reviewTitle'][0].encode('ascii', errors='ignore')
  
  it['reviewContent'] = review.xpath('.//div[1]/div[2]/div[1]/div[3]/p/text()').extract()
  if(it['reviewContent']!=[]):
      it['reviewContent']=it['reviewContent'][0].encode('ascii', errors='ignore').strip()
  
  it['generalRating'] = review.xpath('.//div/div[2]/div/div[2]/span[1]/img/@alt').extract()
  if(it['generalRating']!=[]):
      it['generalRating'] =it['generalRating'][0].encode('ascii', errors='ignore').split()[0]
  </code></pre></li>
  </ol>
  
  <p>there is a good manual how to find these things? i lost myself with all the spans and the divs..</p>
  
  <p>thanks!</p>