{"action":"created","issue":{"url":"https://api.github.com/repos/aigamedev/scikit-neuralnetwork/issues/181","repository_url":"https://api.github.com/repos/aigamedev/scikit-neuralnetwork","labels_url":"https://api.github.com/repos/aigamedev/scikit-neuralnetwork/issues/181/labels{/name}","comments_url":"https://api.github.com/repos/aigamedev/scikit-neuralnetwork/issues/181/comments","events_url":"https://api.github.com/repos/aigamedev/scikit-neuralnetwork/issues/181/events","html_url":"https://github.com/aigamedev/scikit-neuralnetwork/issues/181","id":135355373,"number":181,"title":"Support for memmap as input type","user":{"login":"iarroyof","id":6182245,"avatar_url":"https://avatars.githubusercontent.com/u/6182245?v=3","gravatar_id":"","url":"https://api.github.com/users/iarroyof","html_url":"https://github.com/iarroyof","followers_url":"https://api.github.com/users/iarroyof/followers","following_url":"https://api.github.com/users/iarroyof/following{/other_user}","gists_url":"https://api.github.com/users/iarroyof/gists{/gist_id}","starred_url":"https://api.github.com/users/iarroyof/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iarroyof/subscriptions","organizations_url":"https://api.github.com/users/iarroyof/orgs","repos_url":"https://api.github.com/users/iarroyof/repos","events_url":"https://api.github.com/users/iarroyof/events{/privacy}","received_events_url":"https://api.github.com/users/iarroyof/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/aigamedev/scikit-neuralnetwork/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2016-02-22T08:53:22Z","updated_at":"2016-02-22T16:19:04Z","closed_at":null,"body":"Hi, I have a problem while trying to use Regressor. Im using X_train=numpy.loadtxt() methods for loading dense data from file. This file contains a dense vector by each row. There are 525 rows with 300 dimensions each. I configured a randomizedSearch called rs. \r\n\r\n```\r\nX_train: (525, 300)\r\ny_train: (525,)\r\nX_test: (225, 300)\r\nTraceback (most recent call last):\r\n  File \"nn.py\", line 56, in <module>\r\n    rs.fit(X_train, y_train)\r\n  File \"/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.py\", line 996, in fit\r\n    return self._fit(X, y, sampled_params)\r\n  File \"/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.py\", line 553, in _fit\r\n    for parameters in parameter_iterable\r\n  File \"/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/parallel.py\", line 812, in __call__\r\n    self.retrieve()\r\n  File \"/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/parallel.py\", line 762, in retrieve\r\n    raise exception\r\nsklearn.externals.joblib.my_exceptions.JoblibAttributeError: JoblibAttributeError\r\n___________________________________________________________________________\r\nMultiprocessing exception:\r\n...........................................................................\r\n/almac/ignacio/nlp-pipeline/nn.py in <module>()\r\n     51     exit()\r\n     52 \r\n     53 crash = True\r\n     54 while(crash):\r\n     55     try:\r\n---> 56         rs.fit(X_train, y_train)\r\n     57         crash = False\r\n     58     except RuntimeError:\r\n     59         sys.stderr.write(\"----------- [Crashed by RunTimeERROR] --------------------- \\n\")\r\n     60         crash = True\r\n\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.py in fit(self=RandomizedSearchCV(cv=None, error_score='raise',...e, refit=True,\r\n          scoring=None, verbose=0), X=array([[ -7.19239362e+05,  -2.57225021e+05,  -5....758269e+00,   6.37787139e-02,  -6.47838211e-02]]), y=array([ 2.6 ,  2.6 ,  3.8 ,  4.2 ,  0.8 ,  1.4 ,... 4.2 ,  0.  ,  2.25,\r\n        2.6 ,  2.  ,  3.  ]))\r\n    991 \r\n    992         \"\"\"\r\n    993         sampled_params = ParameterSampler(self.param_distributions,\r\n    994                                           self.n_iter,\r\n    995                                           random_state=self.random_state)\r\n--> 996         return self._fit(X, y, sampled_params)\r\n        self._fit = <bound method RandomizedSearchCV._fit of Randomi..., refit=True,\r\n          scoring=None, verbose=0)>\r\n        X = array([[ -7.19239362e+05,  -2.57225021e+05,  -5....758269e+00,   6.37787139e-02,  -6.47838211e-02]])\r\n        y = array([ 2.6 ,  2.6 ,  3.8 ,  4.2 ,  0.8 ,  1.4 ,... 4.2 ,  0.  ,  2.25,\r\n        2.6 ,  2.  ,  3.  ])\r\n        sampled_params = <sklearn.grid_search.ParameterSampler object>\r\n    997 \r\n    998 \r\n    999 \r\n   1000 \r\n\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.py in _fit(self=RandomizedSearchCV(cv=None, error_score='raise',...e, refit=True,\r\n          scoring=None, verbose=0), X=array([[ -7.19239362e+05,  -2.57225021e+05,  -5....758269e+00,   6.37787139e-02,  -6.47838211e-02]]), y=array([ 2.6 ,  2.6 ,  3.8 ,  4.2 ,  0.8 ,  1.4 ,... 4.2 ,  0.  ,  2.25,\r\n        2.6 ,  2.  ,  3.  ]), parameter_iterable=<sklearn.grid_search.ParameterSampler object>)\r\n    548         )(\r\n    549             delayed(_fit_and_score)(clone(base_estimator), X, y, self.scorer_,\r\n    550                                     train, test, self.verbose, parameters,\r\n    551                                     self.fit_params, return_parameters=True,\r\n    552                                     error_score=self.error_score)\r\n--> 553                 for parameters in parameter_iterable\r\n        parameters = undefined\r\n        parameter_iterable = <sklearn.grid_search.ParameterSampler object>\r\n    554                 for train, test in cv)\r\n    555 \r\n    556         # Out is a list of triplet: score, estimator, n_test_samples\r\n    557         n_fits = len(out)\r\n\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/parallel.py in __call__(self=Parallel(n_jobs=16), iterable=<generator object <genexpr>>)\r\n    807             if pre_dispatch == \"all\" or n_jobs == 1:\r\n    808                 # The iterable was consumed all at once by the above for loop.\r\n    809                 # No need to wait for async callbacks to trigger to\r\n    810                 # consumption.\r\n    811                 self._iterating = False\r\n--> 812             self.retrieve()\r\n        self.retrieve = <bound method Parallel.retrieve of Parallel(n_jobs=16)>\r\n    813             # Make sure that we get a last message telling us we are done\r\n    814             elapsed_time = time.time() - self._start_time\r\n    815             self._print('Done %3i out of %3i | elapsed: %s finished',\r\n    816                         (len(self._output), len(self._output),\r\n\r\n---------------------------------------------------------------------------\r\nSub-process traceback:\r\n---------------------------------------------------------------------------\r\nAttributeError                                     Sun Feb 21 19:22:11 2016\r\nPID: 8977                                     Python 2.7.6: /usr/bin/python\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/parallel.pyc in __call__(self=<sklearn.externals.joblib.parallel.BatchedCalls object>)\r\n     67     def __init__(self, iterator_slice):\r\n     68         self.items = list(iterator_slice)\r\n     69         self._size = len(self.items)\r\n     70 \r\n     71     def __call__(self):\r\n---> 72         return [func(*args, **kwargs) for func, args, kwargs in self.items]\r\n     73 \r\n     74     def __len__(self):\r\n     75         return self._size\r\n     76 \r\n\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.pyc in _fit_and_score(estimator=Regressor(batch_size=1, callback=None, debug=Fal...e, warning=None, weight_decay=None, weights=None), X=memmap([[ -7.19239362e+05,  -2.57225021e+05,  -5...758269e+00,   6.37787139e-02,  -6.47838211e-02]]), y=array([ 2.6 ,  2.6 ,  3.8 ,  4.2 ,  0.8 ,  1.4 ,... 4.2 ,  0.  ,  2.25,\r\n        2.6 ,  2.  ,  3.  ]), scorer=<function _passthrough_scorer>, train=array([175, 176, 177, 178, 179, 180, 181, 182, 1...15, 516, 517, 518, 519, 520, 521, 522, 523, 524]), test=array([  0,   1,   2,   3,   4,   5,   6,   7,  ..., 167, 168,\r\n       169, 170, 171, 172, 173, 174]), verbose=0, parameters={'hidden0__type': 'Rectifier', 'hidden0__units': 5, 'hidden1__type': 'Rectifier', 'hidden1__units': 37, 'learning_momentum': 1.2204617397379565, 'learning_rate': 0.09480671449399372, 'learning_rule': 'rmsprop', 'output__type': 'Linear', 'regularize': None}, fit_params={}, return_train_score=False, return_parameters=True, error_score='raise')\r\n   1526 \r\n   1527     try:\r\n   1528         if y_train is None:\r\n   1529             estimator.fit(X_train, **fit_params)\r\n   1530         else:\r\n-> 1531             estimator.fit(X_train, y_train, **fit_params)\r\n   1532 \r\n   1533     except Exception as e:\r\n   1534         if error_score == 'raise':\r\n   1535             raise\r\n\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sknn/mlp.pyc in fit(self=Regressor(batch_size=1, callback=None, debug=Fal...e, warning=None, weight_decay=None, weights=None), X=memmap([[ -1.71586427e+07,   2.45456060e+06,   6...758269e+00,   6.37787139e-02,  -6.47838211e-02]]), y=array([ 4.4 ,  2.2 ,  3.4 ,  4.  ,  2.6 ,  4.2 ,... 4.8 ,  4.2 ,  0.  ,  2.25,  2.6 ,  2.  ,  3.  ]), w=None)\r\n    288         \"\"\"\r\n    289 \r\n    290         if self.valid_set is not None:\r\n    291             self.valid_set = self._reshape(*self.valid_set)\r\n    292 \r\n--> 293         return super(Regressor, self)._fit(X, y, w)\r\n    294 \r\n    295     def predict(self, X):\r\n    296         \"\"\"Calculate predictions for specified inputs.\r\n    297 \r\n\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sknn/mlp.pyc in _fit(self=Regressor(batch_size=1, callback=None, debug=Fal...e, warning=None, weight_decay=None, weights=None), X=memmap([[ -1.71586427e+07,   2.45456060e+06,   6...758269e+00,   6.37787139e-02,  -6.47838211e-02]]), y=array([[ 4.4 ],\r\n       [ 2.2 ],\r\n       [ 3.4 ],\r\n...\r\n       [ 2.6 ],\r\n       [ 2.  ],\r\n       [ 3.  ]]), w=None)\r\n    227         if self.verbose:\r\n    228             log.debug(\"\\nEpoch       Training Error       Validation Error       Time\"\r\n    229                       \"\\n------------------------------------------------------------\")\r\n    230 \r\n    231         try:\r\n--> 232             self._train(X, y, w)\r\n    233         except RuntimeError as e:\r\n    234             log.error(\"\\n{}{}{}\\n\\n{}\\n\".format(\r\n    235                 ansi.RED,\r\n    236                 \"A runtime exception was caught during training. This likely occurred due to\\n\"\r\n\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sknn/mlp.pyc in _train(self=Regressor(batch_size=1, callback=None, debug=Fal...e, warning=None, weight_decay=None, weights=None), X=memmap([[ -1.71586427e+07,   2.45456060e+06,   6...758269e+00,   6.37787139e-02,  -6.47838211e-02]]), y=array([[ 4.4 ],\r\n       [ 2.2 ],\r\n       [ 3.4 ],\r\n...\r\n       [ 2.6 ],\r\n       [ 2.  ],\r\n       [ 3.  ]]), w=None)\r\n    146         for i in itertools.count(1):\r\n    147             start_time = time.time()\r\n    148             self._do_callback('on_epoch_start', locals())\r\n    149 \r\n    150             is_best_train = False\r\n--> 151             avg_train_error = self._backend._train_impl(X, y, w)\r\n    152             if avg_train_error is not None:\r\n    153                 if math.isnan(avg_train_error):\r\n    154                     raise RuntimeError(\"Training diverged and returned NaN.\")\r\n    155                 \r\n\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sknn/backend/lasagne/mlp.pyc in _train_impl(self=<sknn.backend.lasagne.mlp.MultiLayerPerceptronBackend object>, X=memmap([[ -1.71586427e+07,   2.45456060e+06,   6...758269e+00,   6.37787139e-02,  -6.47838211e-02]]), y=array([[ 4.4 ],\r\n       [ 2.2 ],\r\n       [ 3.4 ],\r\n...\r\n       [ 2.6 ],\r\n       [ 2.  ],\r\n       [ 3.  ]]), w=None)\r\n    300 \r\n    301         self._print('\\r')\r\n    302         return loss / count\r\n    303 \r\n    304     def _train_impl(self, X, y, w=None):\r\n--> 305         return self._batch_impl(X, y, w, self.trainer, mode='train', output='.', shuffle=True)\r\n    306 \r\n    307     def _valid_impl(self, X, y, w=None):\r\n    308         return self._batch_impl(X, y, w, self.validator, mode='valid', output=' ', shuffle=False)\r\n    309 \r\n\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sknn/backend/lasagne/mlp.pyc in _batch_impl(self=<sknn.backend.lasagne.mlp.MultiLayerPerceptronBackend object>, X=memmap([[ -1.71586427e+07,   2.45456060e+06,   6...758269e+00,   6.37787139e-02,  -6.47838211e-02]]), y=array([[ 4.4 ],\r\n       [ 2.2 ],\r\n       [ 3.4 ],\r\n...\r\n       [ 2.6 ],\r\n       [ 2.  ],\r\n       [ 3.  ]]), w=None, processor=<theano.compile.function_module.Function object>, mode=u'train', output=u'.', shuffle=True)\r\n    281             sys.stdout.flush()\r\n    282 \r\n    283     def _batch_impl(self, X, y, w, processor, mode, output, shuffle):\r\n    284         progress, batches = 0, X.shape[0] / self.batch_size\r\n    285         loss, count = 0.0, 0\r\n--> 286         for Xb, yb, wb, _ in self._iterate_data(self.batch_size, X, y, w, shuffle):\r\n    287             self._do_callback('on_batch_start', locals())\r\n    288 \r\n    289             if mode == 'train':\r\n    290                 loss += processor(Xb, yb, wb if wb is not None else 1.0)\r\n\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sknn/backend/lasagne/mlp.pyc in _iterate_data(self=<sknn.backend.lasagne.mlp.MultiLayerPerceptronBackend object>, batch_size=1, X=memmap([[ -1.71586427e+07,   2.45456060e+06,   6...758269e+00,   6.37787139e-02,  -6.47838211e-02]]), y=array([[ 4.4 ],\r\n       [ 2.2 ],\r\n       [ 3.4 ],\r\n...\r\n       [ 2.6 ],\r\n       [ 2.  ],\r\n       [ 3.  ]]), w=None, shuffle=True)\r\n    270         if shuffle:\r\n    271             numpy.random.shuffle(indices)\r\n    272 \r\n    273         for index in range(0, total_size, batch_size):\r\n    274             excerpt = indices[index:index + batch_size]\r\n--> 275             Xb, yb, wb = cast(X, excerpt), cast(y, excerpt), cast(w, excerpt)\r\n    276             yield Xb, yb, wb, excerpt\r\n    277 \r\n    278     def _print(self, text):\r\n    279         if self.verbose:\r\n\r\n...........................................................................\r\n/usr/local/lib/python2.7/dist-packages/sknn/backend/lasagne/mlp.pyc in cast(array=memmap([[  6.39665702e+06,  -1.92380979e+05,   6...693613e+00,   5.43417984e+00,  -1.42057322e+00]]), indices=array([262]))\r\n    258             if array is None:\r\n    259                 return None\r\n    260 \r\n    261             array = array[indices]\r\n    262             if type(array) != numpy.ndarray:\r\n--> 263                 array = array.todense()\r\n    264             if array.dtype != theano.config.floatX:\r\n    265                 array = array.astype(theano.config.floatX)\r\n    266             return array\r\n    267 \r\nAttributeError: 'memmap' object has no attribute 'todense'\r\n```\r\n\r\nPlease help.."},"comment":{"url":"https://api.github.com/repos/aigamedev/scikit-neuralnetwork/issues/comments/187250488","html_url":"https://github.com/aigamedev/scikit-neuralnetwork/issues/181#issuecomment-187250488","issue_url":"https://api.github.com/repos/aigamedev/scikit-neuralnetwork/issues/181","id":187250488,"user":{"login":"alexjc","id":445208,"avatar_url":"https://avatars.githubusercontent.com/u/445208?v=3","gravatar_id":"","url":"https://api.github.com/users/alexjc","html_url":"https://github.com/alexjc","followers_url":"https://api.github.com/users/alexjc/followers","following_url":"https://api.github.com/users/alexjc/following{/other_user}","gists_url":"https://api.github.com/users/alexjc/gists{/gist_id}","starred_url":"https://api.github.com/users/alexjc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexjc/subscriptions","organizations_url":"https://api.github.com/users/alexjc/orgs","repos_url":"https://api.github.com/users/alexjc/repos","events_url":"https://api.github.com/users/alexjc/events{/privacy}","received_events_url":"https://api.github.com/users/alexjc/received_events","type":"User","site_admin":false},"created_at":"2016-02-22T16:19:04Z","updated_at":"2016-02-22T16:19:04Z","body":"I need to look at this in the context of Pandas DataFrames too... But I would make the following change for now:\r\n```\r\n262             if not isinstance(array, numpy.ndarray):\r\n263+                assert hasattr(array, 'todense'), \"Unknown data type; please convert to ndarray first.\"\r\n```\r\n\r\nDoes it work?"}}