title: OLS Regression: Scikit vs. Statsmodels?
body: <p><em>Short version</em>: I was using the scikit LinearRegression on some data, but I'm used to p-values so put the data into the statsmodels OLS, and although the R^2 is about the same the variable coefficients are all different by large amounts. This concerns me since the most likely problem is that I've made an error somewhere and now I don't feel confident in either output (since likely I have made one model incorrectly but don't know which one).</p>
  
  <p><em>Longer version</em>: Because I don't know where the issue is, I don't know exactly which details to include, and including everything is probably too much. I am also not sure about including code or data. </p>
  
  <p>I am under the impression that scikit's LR and statsmodels OLS should both be doing OLS, and as far as I know OLS is OLS so the results should be the same.</p>
  
  <p>For scikit's LR, the results are (statistically) the same whether or not I set normalize=True or =False, which I find somewhat strange.</p>
  
  <p>For statsmodels OLS, I normalize the data using StandardScaler from sklearn. I add a column of ones so it includes an intercept (since scikit's output includes an intercept). More on that here: <a href="http://statsmodels.sourceforge.net/devel/examples/generated/example_ols.html" rel="noreferrer">http://statsmodels.sourceforge.net/devel/examples/generated/example_ols.html</a> (Adding this column did not change the variable coefficients to any notable degree and the intercept was very close to zero.) StandardScaler didn't like that my ints weren't floats, so I tried this: <a href="https://github.com/scikit-learn/scikit-learn/issues/1709" rel="noreferrer">https://github.com/scikit-learn/scikit-learn/issues/1709</a>
  That makes the warning go away but the results are exactly the same.</p>
  
  <p>Granted I'm using 5-folds cv for the sklearn approach (R^2 are consistent for both test and training data each time), and for statsmodels I just throw it all the data.</p>
  
  <p>R^2 is about 0.41 for both sklearn and statsmodels (this is good for social science). This could be a good sign or just a coincidence.</p>
  
  <p>The data is observations of avatars in WoW (from <a href="http://mmnet.iis.sinica.edu.tw/dl/wowah/" rel="noreferrer">http://mmnet.iis.sinica.edu.tw/dl/wowah/</a>) which I munged about to make it weekly with some different features. Originally this was a class project for a data science class.</p>
  
  <p>Independent variables include number of observations in a week (int), character level (int), if in a guild (Boolean), when seen (Booleans on weekday day, weekday eve, weekday late, and the same three for weekend), a dummy for character class (at the time for the data collection, there were only 8 classes in WoW, so there are 7 dummy vars and the original string categorical variable is dropped), and others.</p>
  
  <p>The dependent variable is how many levels each character gained during that week (int).</p>
  
  <p>Interestingly, some of the relative order within like variables is maintained across statsmodels and sklearn. So, rank order of "when seen" is the same although the loadings are very different, and rank order for the character class dummies is the same although again the loadings are very different.</p>
  
  <p>I think this question is similar to this one: <a href="https://stackoverflow.com/questions/11495051/difference-in-python-statsmodels-ols-and-rs-lm">Difference in Python statsmodels OLS and R&#39;s lm</a></p>
  
  <p>I am good enough at Python and stats to make a go of it, but then not good enough to figure something like this out. I tried reading the sklearn docs and the statsmodels docs, but if the answer was there staring me in the face I did not understand it. </p>
  
  <p>I would love to know:</p>
  
  <ol>
  <li>Which output might be accurate? (Granted they might both be if I missed a kwarg.)</li>
  <li>If I made a mistake, what is it and how to fix it?</li>
  <li>Could I have figured this out without asking here, and if so how?</li>
  </ol>
  
  <p>I know this question has some rather vague bits (no code, no data, no output), but I am thinking it is more about the general processes of the two packages. Sure, one seems to be more stats and one seems to be more machine learning, but they're both OLS so I don't understand why the outputs aren't the same.</p>
  
  <p>(I even tried some other OLS calls to triangulate, one gave a much lower R^2, one looped for five minutes and I killed it, and one crashed.)</p>
  
  <p>Thanks!</p>