title: How to loop sklearn linear regression by values within a column - python
body: <p>I am relatively new to python and programming in general. </p>
  
  <p>I have a csv with three columns, an identifier, distance, and date. The data looks something like:</p>
  
  <pre><code>id, date, distance
  
  1, 1850, 150
  1, 1950, 200
  1, 1990, 250
  2, 1850, 130
  2, 1950, 180
  2, 1990, 210
  3, 1850, 200
  3, 1950, 220
  3, 1990, 250
  </code></pre>
  
  <p>etc...</p>
  
  <p>I'm trying to use sklearn.linear_model LinearRegression. To run a regression on each of these series. 
  So I need to basically separate the columns by id. I want a separate result for each group of id's (1, 2, 3, etc). </p>
  
  <p>I was able to do this easily with R by using:</p>
  
  <pre><code>reg1 &lt;- lmList(distance ~ date | id, mydata)
  </code></pre>
  
  <p>However, for several reasons, I need to do this with python.</p>
  
  <p>I've imported the csv using pandas.read_csv. Then I assumed I need to attribute each of these to a variable:</p>
  
  <pre><code>dataset = pandas.read_csv(location, sep = ",", header = 0)
  x = dataset['date']
  y = dataset['distance']
  z = dataset['id']
  </code></pre>
  
  <p>Then apparently to use them in LinearRegression() and model.fit(x,y) I needed to use</p>
  
  <pre><code>x2 = numpy.array(x).T and
  y2 = numpy.array(x).T
  </code></pre>
  
  <p>So now that that is all set up, what should I do to actually loop the regression through those id values?</p>
  
  <p>when I try:</p>
  
  <pre><code>for n in main_reg2.id:
      model1 = LinearRegression()
      model1.fit(xT1,yT)
      print(model1.coef_, model1.intercept_)
      break 
  </code></pre>
  
  <p>It just gives me one result for the whole dataset. And I'm not sure where to tell it to go through the id values.</p>
  
  <p>Any help would be greatly appreciated. Sorry for the mixing of jargon, I've had to dabble in 5 different programming and statistical languages over the past 2 years of grad school and am very confused sometimes. </p>
  
  <p>Perhaps I'm overthinking this, or maybe there is some way towards the beginning that I can separate the data? </p>
  
  <p>Also, how would I create a table from this that lists the coefficients and intercepts and whatnot. </p>
  
  <p>Thanks</p>
  
  <p>Updated code so far:
      with open(r'file_location','rb') as f:
          reader = csv.reader(f)
          headers = reader.next()
          for fields in reader:
              your_list = list(reader)</p>
  
  <pre><code>z = [_[0] for _ in your_list]
  x = [_[1] for _ in your_list]
  y = [_[2] for _ in your_list]
  
  id_max = max(z)
  new_list = [[]]
  for _ in range(id_max):
      new_list.append([])
  
  for index in range(len(z)):
      id = z[index]
      new_entry = [x[index],y[index]]
      print "Adding to index", id, new_entry
      new_list[id].append(new_entry)
  print new_list 
  </code></pre>
  
  <p>I used header = reader.next() code to get rid of the headers. When I was doing the max(z) before, the header was coming up as the max...</p>
  
  <p>Going through line by line:
  I get... </p>
  
  <pre><code>id_max = max(z)
  print(id_max)
  999
  </code></pre>
  
  <p>Which I already know is wrong since the max is 1576. </p>
  
  <p>But moving on.... after the append I get </p>
  
  <pre><code>Type Error: range() integer end argument expected, got str
  </code></pre>
  
  <p>Which I don't understand since z is a list of integers from 1 to 1576. </p>
  
  <p>Some examples of the indexes are:</p>
  
  <pre><code>z[1] = '10'
  y[1] = '2011'
  x[1] = '515.8938'
  </code></pre>