title: Robustness issue of statsmodel Linear regression (ols) - Python
body: <p>I was testing some basic category regression using Stats model:
  I build up a deterministic model</p>
  
  <blockquote>
    <p>Y = X + Z</p>
  </blockquote>
  
  <p>where X can takes 3 values (a, b or c) and Z only 2 (d or e).
  At that stage the model is purely deterministic, I setup the weights for each variable as followed</p>
  
  <blockquote>
    <p>a's weight=1</p>
    
    <p>b's weight=2</p>
    
    <p>c's weight=3</p>
    
    <p>d's weight=1</p>
    
    <p>e's weight=2</p>
  </blockquote>
  
  <p>Therefore with 1(X=a) being 1 if X=a, 0 otherwise, the model is simply:</p>
  
  <blockquote>
    <p>Y = 1(X=a) + 2*1(X=b) + 3*1(X=c) + 1(Z=d) + 2*1(Z=e)</p>
  </blockquote>
  
  <p>Using the following code, to generate the different variables and run the regression</p>
  
  <pre><code>from statsmodels.formula.api import ols
  nbData = 1000
  rand1 = np.random.uniform(size=nbData)
  rand2 = np.random.uniform(size=nbData)
  a = 1 * (rand1 &lt;= (1.0/3.0))
  b = 1 * (((1.0/3.0)&lt; rand1) &amp; (rand1&lt; (4/5.0)))
  c = 1-b-a
  d = 1 * (rand2 &lt;= (3.0/5.0))
  e = 1-d
  weigths = [1,2,3,1,2]
  y = a+2*b+3*c+4*d+5*e
  df = pd.DataFrame({'y':y, 'a':a, 'b':b, 'c':c, 'd':d, 'e':e})
  
  mod = ols(formula='y ~ a + b + c + d + e - 1', data=df)
  res = mod.fit()
  print(res.summary())
  </code></pre>
  
  <p>I end up with the rights results (one has to look at the difference between coef rather than the coef themselfs)</p>
  
  <pre><code>                           OLS Regression Results                            
  ==============================================================================
  Dep. Variable:                      y   R-squared:                       1.000
  Model:                            OLS   Adj. R-squared:                  1.000
  Method:                 Least Squares   F-statistic:                 1.006e+30
  Date:                Wed, 16 Sep 2015   Prob (F-statistic):               0.00
  Time:                        03:05:40   Log-Likelihood:                 3156.8
  No. Observations:                 100   AIC:                            -6306.
  Df Residuals:                      96   BIC:                            -6295.
  Df Model:                           3                                         
  Covariance Type:            nonrobust                                         
  ==============================================================================
                   coef    std err          t      P&gt;|t|      [95.0% Conf. Int.]
  ------------------------------------------------------------------------------
  a              1.6000   7.47e-16   2.14e+15      0.000         1.600     1.600
  b              2.6000   6.11e-16   4.25e+15      0.000         2.600     2.600
  c              3.6000   9.61e-16   3.74e+15      0.000         3.600     3.600
  d              3.4000   5.21e-16   6.52e+15      0.000         3.400     3.400
  e              4.4000   6.85e-16   6.42e+15      0.000         4.400     4.400
  ==============================================================================
  Omnibus:                       11.299   Durbin-Watson:                   0.833
  Prob(Omnibus):                  0.004   Jarque-Bera (JB):                5.720
  Skew:                          -0.381   Prob(JB):                       0.0573
  Kurtosis:                       2.110   Cond. No.                     2.46e+15
  ==============================================================================
  
  Warnings:
  [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
  [2] The smallest eigenvalue is 1.67e-29. This might indicate that there are
  strong multicollinearity problems or that the design matrix is singular.
  </code></pre>
  
  <p>But when I increase the number of data point to (say) 600, the regression is producing really bad results. I have tried similar regression in Excel and in R and they are producing consistent results whatever the number of data points. Does anyone know if there is some restriction on statsmodel ols explaining such behaviour or am I missing something?</p>
  
  <pre><code>                            OLS Regression Results                            
  ==============================================================================
  Dep. Variable:                      y   R-squared:                       0.167
  Model:                            OLS   Adj. R-squared:                  0.161
  Method:                 Least Squares   F-statistic:                     29.83
  Date:                Wed, 16 Sep 2015   Prob (F-statistic):           1.23e-22
  Time:                        03:08:04   Log-Likelihood:                -701.02
  No. Observations:                 600   AIC:                             1412.
  Df Residuals:                     595   BIC:                             1434.
  Df Model:                           4                                         
  Covariance Type:            nonrobust                                         
  ==============================================================================
                   coef    std err          t      P&gt;|t|      [95.0% Conf. Int.]
  ------------------------------------------------------------------------------
  a              5.8070   1.15e+13   5.05e-13      1.000     -2.26e+13  2.26e+13
  b              6.4951   1.15e+13   5.65e-13      1.000     -2.26e+13  2.26e+13
  c              6.9033   1.15e+13   6.01e-13      1.000     -2.26e+13  2.26e+13
  d             -1.1927   1.15e+13  -1.04e-13      1.000     -2.26e+13  2.26e+13
  e             -0.1685   1.15e+13  -1.47e-14      1.000     -2.26e+13  2.26e+13
  ==============================================================================
  Omnibus:                       67.153   Durbin-Watson:                   0.328
  Prob(Omnibus):                  0.000   Jarque-Bera (JB):               70.964
  Skew:                           0.791   Prob(JB):                     3.89e-16
  Kurtosis:                       2.419   Cond. No.                     7.70e+14
  ==============================================================================
  
  Warnings:
  [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
  [2] The smallest eigenvalue is 9.25e-28. This might indicate that there are
  strong multicollinearity problems or that the design matrix is singular.
  </code></pre>