Q:

Scrapy/Python Issue: [count] does not collect entire table

This is a spin off of a previous XPath thread (dude told me it's not XPath related).
So I am trying to scrape this web page: http://www.baseball-reference.com/teams/BOS/2013.shtml for Rank, Position, Name, Age, etc. 
Whenever I use: 
item['rank'] = stat.select('//table[@id="team_batting"]/tbody/tr/td[1]//text()')[count].extract()
item ['position'] = stat.select('//table[@id="team_batting"]/tbody/tr/td[2]//text()')[count].extract()

It only gives me 5 results:
[{"position": "C", "rank": "1"},
{"position": "1B", "rank": "2"},
{"position": "2B", "rank": "3"},
{"position": "SS", "rank": "4"},
{"position": "3B", "rank": "5"}]

If I remove [count], it gives me all of the ranks and positions, but not in the correct format, and it gives me 4 duplicate lines of it (I condensed this to fit in here,there are 44):
{"position": ["C", "1B", "2B", "SS", "3B", "LF"], "rank": ["1", "2", "3", "4", "5", "6", "7"]}

Here is my parser code is this will help:
def parse(self, response):
 hxs = HtmlXPathSelector(response)
 stats = hxs.select('//div[@class="table_container"]')
 count = 0
 items = []

 for stat in stats:

     item = MlbrefItem()

     items .append(item)
     count +=1

 return items

I am so confused. When I did this tutorial: http://davidwalsh.name/python-scrape [count] pulled all of the results. I also had no issue trying this on a different website. But this is the website I need.
Any help is greatly appreciated. Thanks
EDIT: Here is the actual code (ignore the comments): https://www.dropbox.com/s/rk29kotd09kioeg/mlbref_spyder.py 

A:

The code you have posted does not have the definition for the MlbrefItem class. Also, it appears from the posted code that you have incorrect indentation at the following line:
items.append(item)

You are appending an item object outside the for loop. This means you end up appending only the final instance of the type MlbrefItem. Please try the below code snippet for the for-loop and let us know if it works:
for stat in stats:

    item = MlbrefItem()

    items .append(item)
    count +=1

A big edit based on discussion with AngryWhopper
There are multiple aspects to consider here:
There are 5 divs on the page with style table_container The statement stats = hxs.select('//div[@class="table_container"]') selects all these 5 elements
The source code at the reference link provided is shown below. I have added comments to code beginning with my user name where you increment count (count += 1). Since this is within the for loop this happens only 5 times. Hence each of the 5 times, you end up selecting only a single element from the list of rows returned by the XPATH selectors for position and rank. This explains why you get only 5 pairs when you use the count variable within your code. 
stats = hxs.select('//div[@class="table_container"]')
        count = 0
        items = []

        for stat in stats:

            item = MlbrefItem()

            item['rank'] = stat.select('//table[@id="team_batting"]/tbody/tr/td[1]//text()').extract()
             item ['position'] = stat.select('//table[@id="team_batting"]/tbody/tr/td[2]//text()').extract()

            items .append(item)
            count +=1 #PrahaladDeshpande - This count variable is incremented only 5 times since it is within the for loop

        return items

Without the count variable, you get all rows - however they are not in the correct format. Well, here you will have to to do some work. I tried the some experiments on the scrapy terminal and the below code snippet should hopefully work for you
stats = hxs.select('//div[@class="table_container"]')
for stat in stats:
        rank_list = stat.select('//table[@id="team_batting"]/tbody/tr/td[1]//text()').extract()
        position_list = stat.select('//table[@id="team_batting"]/tbody/tr/td[2]//text()').extract()

         //TODO : verify that the length of the rank_list and position_list are the same. Else throw an error
         for i in range(0, len(rank_list)):
             item = MlbrefItem()
             item['rank'] = rank_list[i]
             item['position'] = position_list[i]
             items.append(item)

You can then return the items variable from your method.
Hope this points you towards a solution

