{"action":"created","issue":{"url":"https://api.github.com/repos/blue-yonder/tsfresh/issues/418","repository_url":"https://api.github.com/repos/blue-yonder/tsfresh","labels_url":"https://api.github.com/repos/blue-yonder/tsfresh/issues/418/labels{/name}","comments_url":"https://api.github.com/repos/blue-yonder/tsfresh/issues/418/comments","events_url":"https://api.github.com/repos/blue-yonder/tsfresh/issues/418/events","html_url":"https://github.com/blue-yonder/tsfresh/issues/418","id":354086340,"node_id":"MDU6SXNzdWUzNTQwODYzNDA=","number":418,"title":"extract_features is failing with: \"OverflowError: value too large to convert to int\"","user":{"login":"yuval-nardi","id":34504826,"node_id":"MDQ6VXNlcjM0NTA0ODI2","avatar_url":"https://avatars0.githubusercontent.com/u/34504826?v=4","gravatar_id":"","url":"https://api.github.com/users/yuval-nardi","html_url":"https://github.com/yuval-nardi","followers_url":"https://api.github.com/users/yuval-nardi/followers","following_url":"https://api.github.com/users/yuval-nardi/following{/other_user}","gists_url":"https://api.github.com/users/yuval-nardi/gists{/gist_id}","starred_url":"https://api.github.com/users/yuval-nardi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yuval-nardi/subscriptions","organizations_url":"https://api.github.com/users/yuval-nardi/orgs","repos_url":"https://api.github.com/users/yuval-nardi/repos","events_url":"https://api.github.com/users/yuval-nardi/events{/privacy}","received_events_url":"https://api.github.com/users/yuval-nardi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":35,"created_at":"2018-08-26T11:14:17Z","updated_at":"2018-09-10T15:10:54Z","closed_at":null,"author_association":"NONE","body":"I am running extract_features on a very large matrix, having ~350 million rows and 6 features (as part of a complex data science pipeline). I am using a machine with 64 cores and 2TB memory, and utilizing all 64 cores. I am getting this error: \"Overflow Error: value too large to convert to int\".\r\nSome comments:\r\ni) When I split the matrix vertically into, say 3, chunks (each chunk having 2 features only), and run them sequentially, everything works fine. So it does not seem like I am having issues with \"problematic\" values in the matrix. \r\nii) It does not seem a memory related issue either (as alluded to in https://github.com/blue-yonder/tsfresh/issues/368) because I was babysitting the mentioned run that failed, and was checking memory usage regularly (using \"free -g\"). It never got above 400GB.\r\niii) I also tried running with LocalDaskDistributor and got the same error.\r\niv) All 6 features in the matrix are floats.\r\nv) pai_tsfresh below is a fork of tsfresh.\r\n\r\n1. Your operating system\r\nNo LSB modules are available.\r\nDistributor ID:\tUbuntu\r\nDescription:\tUbuntu 16.04.3 LTS\r\nRelease:\t16.04\r\nCodename:\txenial\r\n\r\n2. The version of *tsfresh* that you are using\r\nlatest\r\n\r\n4. A minimal code snippet which reproduces the problem/bug\r\nHere's the call in my code to extract_features:\r\nextracted_features_df = extract_features(rolled_design_matrix,\r\n                                                                      column_id='account_date_index',\r\n                                                                      column_sort='date',\r\n                                                                      default_fc_parameters=fc_parameters,\r\n                                                                      n_jobs=64)\r\nwhere fc_parameters is:\r\n{'abs_energy': None,\r\n 'autocorrelation': [{'lag': 1}],\r\n 'binned_entropy': [{'max_bins': 10}],\r\n 'c3': [{'lag': 1}],\r\n 'cid_ce': [{'normalize': True}],\r\n 'fft_aggregated': [{'aggtype': 'centroid'},\r\n  {'aggtype': 'variance'},\r\n  {'aggtype': 'skew'},\r\n  {'aggtype': 'kurtosis'}],\r\n 'fft_coefficient': [{'attr': 'real', 'coeff': 0}],\r\n 'sample_entropy': None,\r\n 'spkt_welch_density': [{'coeff': 2}],\r\n 'time_reversal_asymmetry_statistic': [{'lag': 1}]}\r\n\r\n5. Any reported errors or traceback\r\nHere's the traceback:\r\nTraceback (most recent call last):\r\n  File \"/home/yuval/pai/projects/ds-feature-engineering-service/feature_engineering_service/src/fe/stateless/time_series_features_enricher/time_series_features_enricher.py\", line 175, in do_enrich\r\n    distributor=local_dask_distributor)\r\n  File \"/home/yuval/pai/projects/pai-tsfresh/pai_tsfresh/feature_extraction/extraction.py\", line 152, in extract_features\r\n    distributor=distributor)\r\n  File \"/home/yuval/pai/projects/pai-tsfresh/pai_tsfresh/feature_extraction/extraction.py\", line 217, in _do_extraction\r\n    data_in_chunks = [x + (y,) for x, y in df.groupby([column_id, column_kind])[column_value]]\r\n  File \"/home/yuval/pai/projects/pai-tsfresh/pai_tsfresh/feature_extraction/extraction.py\", line 217, in <listcomp>\r\n    data_in_chunks = [x + (y,) for x, y in df.groupby([column_id, column_kind])[column_value]]\r\n  File \"/usr/local/lib/python3.6/dist-packages/pandas/core/groupby.py\", line 1922, in get_iterator\r\n    splitter = self._get_splitter(data, axis=axis)\r\n  File \"/usr/local/lib/python3.6/dist-packages/pandas/core/groupby.py\", line 1928, in _get_splitter\r\n    comp_ids, _, ngroups = self.group_info\r\n  File \"pandas/_libs/properties.pyx\", line 38, in pandas._libs.properties.cache_readonly.__get__\r\n  File \"/usr/local/lib/python3.6/dist-packages/pandas/core/groupby.py\", line 2040, in group_info\r\n    comp_ids, obs_group_ids = self._get_compressed_labels()\r\n  File \"/usr/local/lib/python3.6/dist-packages/pandas/core/groupby.py\", line 2056, in _get_compressed_labels\r\n    all_labels = [ping.labels for ping in self.groupings]\r\n  File \"/usr/local/lib/python3.6/dist-packages/pandas/core/groupby.py\", line 2056, in <listcomp>\r\n    all_labels = [ping.labels for ping in self.groupings]\r\n  File \"/usr/local/lib/python3.6/dist-packages/pandas/core/groupby.py\", line 2750, in labels\r\n    self._make_labels()\r\n  File \"/usr/local/lib/python3.6/dist-packages/pandas/core/groupby.py\", line 2767, in _make_labels\r\n    self.grouper, sort=self.sort)\r\n  File \"/usr/local/lib/python3.6/dist-packages/pandas/core/algorithms.py\", line 468, in factorize\r\n    table = hash_klass(size_hint or len(values))\r\n  File \"pandas/_libs/hashtable_class_helper.pxi\", line 1005, in pandas._libs.hashtable.StringHashTable.__init__\r\nOverflowError: value too large to convert to int"},"comment":{"url":"https://api.github.com/repos/blue-yonder/tsfresh/issues/comments/419948513","html_url":"https://github.com/blue-yonder/tsfresh/issues/418#issuecomment-419948513","issue_url":"https://api.github.com/repos/blue-yonder/tsfresh/issues/418","id":419948513,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTk0ODUxMw==","user":{"login":"yuval-nardi","id":34504826,"node_id":"MDQ6VXNlcjM0NTA0ODI2","avatar_url":"https://avatars0.githubusercontent.com/u/34504826?v=4","gravatar_id":"","url":"https://api.github.com/users/yuval-nardi","html_url":"https://github.com/yuval-nardi","followers_url":"https://api.github.com/users/yuval-nardi/followers","following_url":"https://api.github.com/users/yuval-nardi/following{/other_user}","gists_url":"https://api.github.com/users/yuval-nardi/gists{/gist_id}","starred_url":"https://api.github.com/users/yuval-nardi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yuval-nardi/subscriptions","organizations_url":"https://api.github.com/users/yuval-nardi/orgs","repos_url":"https://api.github.com/users/yuval-nardi/repos","events_url":"https://api.github.com/users/yuval-nardi/events{/privacy}","received_events_url":"https://api.github.com/users/yuval-nardi/received_events","type":"User","site_admin":false},"created_at":"2018-09-10T15:10:54Z","updated_at":"2018-09-10T15:10:54Z","author_association":"NONE","body":"@MaxBenChrist still failing, after upgrading to pandas 0.23.4.\r\n\r\n```\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.6.4.final.0\r\npython-bits: 64\r\nOS: Linux\r\nOS-release: 4.15.0-1018-gcp\r\nmachine: x86_64\r\nprocessor: x86_64\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: en_US.UTF-8\r\nLOCALE: en_US.UTF-8\r\n\r\npandas: 0.23.4\r\npytest: None\r\npip: 9.0.1\r\nsetuptools: 20.7.0\r\nCython: None\r\nnumpy: 1.15.0\r\nscipy: 1.1.0\r\npyarrow: None\r\nxarray: None\r\nIPython: 6.2.1\r\nsphinx: None\r\npatsy: 0.5.0\r\ndateutil: 2.6.1\r\npytz: 2018.3\r\nblosc: None\r\nbottleneck: None\r\ntables: None\r\nnumexpr: 2.6.4\r\nfeather: None\r\nmatplotlib: 2.1.2\r\nopenpyxl: None\r\nxlrd: None\r\nxlwt: None\r\nxlsxwriter: None\r\nlxml: None\r\nbs4: None\r\nhtml5lib: 0.9999999\r\nsqlalchemy: None\r\npymysql: None\r\npsycopg2: None\r\njinja2: 2.8\r\ns3fs: None\r\nfastparquet: None\r\npandas_gbq: None\r\npandas_datareader: None\r\n```"}}
{"actor":{"display_login":"yuval-nardi"}}