{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask-ml/issues/325","repository_url":"https://api.github.com/repos/dask/dask-ml","labels_url":"https://api.github.com/repos/dask/dask-ml/issues/325/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask-ml/issues/325/comments","events_url":"https://api.github.com/repos/dask/dask-ml/issues/325/events","html_url":"https://github.com/dask/dask-ml/issues/325","id":345839691,"node_id":"MDU6SXNzdWUzNDU4Mzk2OTE=","number":325,"title":"LinearRegression does not work with dask dataframe.values","user":{"login":"js3711","id":15849929,"node_id":"MDQ6VXNlcjE1ODQ5OTI5","avatar_url":"https://avatars3.githubusercontent.com/u/15849929?v=4","gravatar_id":"","url":"https://api.github.com/users/js3711","html_url":"https://github.com/js3711","followers_url":"https://api.github.com/users/js3711/followers","following_url":"https://api.github.com/users/js3711/following{/other_user}","gists_url":"https://api.github.com/users/js3711/gists{/gist_id}","starred_url":"https://api.github.com/users/js3711/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/js3711/subscriptions","organizations_url":"https://api.github.com/users/js3711/orgs","repos_url":"https://api.github.com/users/js3711/repos","events_url":"https://api.github.com/users/js3711/events{/privacy}","received_events_url":"https://api.github.com/users/js3711/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-07-30T16:41:02Z","updated_at":"2018-07-30T16:47:45Z","closed_at":null,"author_association":"NONE","body":"Hello,\r\n\r\nI am trying to fit a linear regression model from a dask dataframe because my data will not fit into local memory.\r\n\r\n```python\r\nimport sklearn.datasets as sk_datasets\r\nimport dask.dataframe as dd\r\n```\r\n\r\n```python\r\nX, y = sk_datasets.make_classification(n_samples=10000, n_informative=12, \r\n                                       n_redundant=18, n_features=30)\r\n```\r\n\r\n```python\r\ndf_x = dd.from_array(X)\r\ndf_y = dd.from_array(y)\r\n```\r\n\r\n```python\r\ndask_x = df_x.values\r\ndask_y = df_y.values\r\n```\r\n\r\n```python\r\nlr = LinearRegression(fit_intercept=True)\r\nlr.fit(dask_x, dask_y)\r\n```\r\n\r\nThis throws:\r\n```python\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask_glm/utils.py in add_intercept(X)\r\n    145 def add_intercept(X):\r\n    146     if np.isnan(np.sum(X.shape)):\r\n--> 147         raise NotImplementedError(\"Can not add intercept to array with \"\r\n    148                                   \"unknown chunk shape\")\r\n    149     j, k = X.chunks\r\n\r\nNotImplementedError: Can not add intercept to array with unknown chunk shape\r\n```\r\n\r\n```python\r\nlr = LinearRegression(fit_intercept=False)\r\nlr.fit(dask_x, dask_y)\r\n```\r\n\r\nThis throws:\r\n```python\r\n---------------------------------------------------------------------------\r\nAttributeError                            Traceback (most recent call last)\r\n<ipython-input-35-fedc331fe3fb> in <module>()\r\n      1 lr = LinearRegression(fit_intercept=False)\r\n      2 \r\n----> 3 lr.fit(X_values, y_values)\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask_ml/linear_model/glm.py in fit(self, X, y)\r\n    155         solver_kwargs = self._get_solver_kwargs()\r\n    156 \r\n--> 157         self._coef = algorithms._solvers[self.solver](X, y, **solver_kwargs)\r\n    158         if self.fit_intercept:\r\n    159             self.coef_ = self._coef[:-1]\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask_glm/utils.py in normalize_inputs(X, y, *args, **kwargs)\r\n     24             mean = mean if len(intercept_idx[0]) else np.zeros(mean.shape)\r\n     25             Xn = (X - mean) / std\r\n---> 26             out = algo(Xn, y, *args, **kwargs).copy()\r\n     27             i_adj = np.sum(out * mean / std)\r\n     28             out[intercept_idx] -= i_adj\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask_glm/algorithms.py in admm(X, y, regularizer, lamduh, rho, over_relax, max_iter, abstol, reltol, family, **kwargs)\r\n    262                                            fprime=fprime) for\r\n    263                      xx, yy, bb, uu in zip(XD, yD, betas, u)]\r\n--> 264         new_betas = np.array(da.compute(*new_betas))\r\n    265 \r\n    266         beta_hat = over_relax * new_betas + (1 - over_relax) * z\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask/base.py in compute(*args, **kwargs)\r\n    400     keys = [x.__dask_keys__() for x in collections]\r\n    401     postcomputes = [x.__dask_postcompute__() for x in collections]\r\n--> 402     results = schedule(dsk, keys, **kwargs)\r\n    403     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])\r\n    404 \r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask/threaded.py in get(dsk, result, cache, num_workers, **kwargs)\r\n     73     results = get_async(pool.apply_async, len(pool._pool), dsk, result,\r\n     74                         cache=cache, get_id=_thread_get_id,\r\n---> 75                         pack_exception=pack_exception, **kwargs)\r\n     76 \r\n     77     # Cleanup pools associated to dead threads\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask/local.py in get_async(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs)\r\n    503                         _execute_task(task, data)  # Re-execute locally\r\n    504                     else:\r\n--> 505                         raise_exception(exc, tb)\r\n    506                 res, worker_id = loads(res_info)\r\n    507                 state['cache'][key] = res\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask/compatibility.py in reraise(exc, tb)\r\n     67         if exc.__traceback__ is not tb:\r\n     68             raise exc.with_traceback(tb)\r\n---> 69         raise exc\r\n     70 \r\n     71 else:\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask/local.py in execute_task(key, task_info, dumps, loads, get_id, pack_exception)\r\n    272     try:\r\n    273         task, data = loads(task_info)\r\n--> 274         result = _execute_task(task, data)\r\n    275         id = get_id()\r\n    276         result = dumps((result, id))\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask/local.py in _execute_task(arg, cache, dsk)\r\n    253         func, args = arg[0], arg[1:]\r\n    254         args2 = [_execute_task(a, cache) for a in args]\r\n--> 255         return func(*args2)\r\n    256     elif not ishashable(arg):\r\n    257         return arg\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask/compatibility.py in apply(func, args, kwargs)\r\n     48     def apply(func, args, kwargs=None):\r\n     49         if kwargs:\r\n---> 50             return func(*args, **kwargs)\r\n     51         else:\r\n     52             return func(*args)\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask_glm/algorithms.py in local_update(X, y, beta, z, u, rho, f, fprime, solver)\r\n    297     beta, f, d = solver(f, beta, fprime=fprime, args=solver_args,\r\n    298                         maxiter=200,\r\n--> 299                         maxfun=250)\r\n    300 \r\n    301     return beta\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py in fmin_l_bfgs_b(func, x0, fprime, args, approx_grad, bounds, m, factr, pgtol, epsilon, iprint, maxfun, maxiter, disp, callback, maxls)\r\n    197 \r\n    198     res = _minimize_lbfgsb(fun, x0, args=args, jac=jac, bounds=bounds,\r\n--> 199                            **opts)\r\n    200     d = {'grad': res['jac'],\r\n    201          'task': res['message'],\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py in _minimize_lbfgsb(fun, x0, args, jac, bounds, disp, maxcor, ftol, gtol, eps, maxfun, maxiter, iprint, callback, maxls, **unknown_options)\r\n    333             # until the completion of the current minimization iteration.\r\n    334             # Overwrite f and g:\r\n--> 335             f, g = func_and_grad(x)\r\n    336         elif task_str.startswith(b'NEW_X'):\r\n    337             # new iteration\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py in func_and_grad(x)\r\n    283     else:\r\n    284         def func_and_grad(x):\r\n--> 285             f = fun(x, *args)\r\n    286             g = jac(x, *args)\r\n    287             return f, g\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/scipy/optimize/optimize.py in function_wrapper(*wrapper_args)\r\n    291     def function_wrapper(*wrapper_args):\r\n    292         ncalls[0] += 1\r\n--> 293         return function(*(wrapper_args + args))\r\n    294 \r\n    295     return ncalls, function_wrapper\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask_glm/algorithms.py in wrapped(beta, X, y, z, u, rho)\r\n    231         @functools.wraps(func)\r\n    232         def wrapped(beta, X, y, z, u, rho):\r\n--> 233             return func(beta, X, y) + (rho / 2) * np.dot(beta - z + u,\r\n    234                                                          beta - z + u)\r\n    235         return wrapped\r\n\r\n~/anaconda3/envs/correlation_exploration/lib/python3.6/site-packages/dask_glm/families.py in pointwise_loss(beta, X, y)\r\n     62     @staticmethod\r\n     63     def pointwise_loss(beta, X, y):\r\n---> 64         beta, y = beta.ravel(), y.ravel()\r\n     65         Xbeta = X.dot(beta)\r\n     66         return Normal.loglike(Xbeta, y)\r\n\r\nAttributeError: 'tuple' object has no attribute 'ravel'\r\n```"},"comment":{"url":"https://api.github.com/repos/dask/dask-ml/issues/comments/408931936","html_url":"https://github.com/dask/dask-ml/issues/325#issuecomment-408931936","issue_url":"https://api.github.com/repos/dask/dask-ml/issues/325","id":408931936,"node_id":"MDEyOklzc3VlQ29tbWVudDQwODkzMTkzNg==","user":{"login":"TomAugspurger","id":1312546,"node_id":"MDQ6VXNlcjEzMTI1NDY=","avatar_url":"https://avatars3.githubusercontent.com/u/1312546?v=4","gravatar_id":"","url":"https://api.github.com/users/TomAugspurger","html_url":"https://github.com/TomAugspurger","followers_url":"https://api.github.com/users/TomAugspurger/followers","following_url":"https://api.github.com/users/TomAugspurger/following{/other_user}","gists_url":"https://api.github.com/users/TomAugspurger/gists{/gist_id}","starred_url":"https://api.github.com/users/TomAugspurger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TomAugspurger/subscriptions","organizations_url":"https://api.github.com/users/TomAugspurger/orgs","repos_url":"https://api.github.com/users/TomAugspurger/repos","events_url":"https://api.github.com/users/TomAugspurger/events{/privacy}","received_events_url":"https://api.github.com/users/TomAugspurger/received_events","type":"User","site_admin":false},"created_at":"2018-07-30T16:47:45Z","updated_at":"2018-07-30T16:47:45Z","author_association":"MEMBER","body":"Just using `df_x` and `df_y` works, correct?\r\n\r\n---\r\n\r\nsome context: dask dataframe doesn't know its own length,so doing `df_x.values` results in a dask array with unknown length. We can't concatenate an array of ones to `X` in that case, since we don't know how long to make the ones.\r\n\r\nI plan to implement something like https://github.com/dask/dask/issues/3090 later today. There's the related https://github.com/dask/dask/issues/3293 issue."}}
{"actor":{"display_login":"TomAugspurger"}}