{"action":"created","issue":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/19493","repository_url":"https://api.github.com/repos/pandas-dev/pandas","labels_url":"https://api.github.com/repos/pandas-dev/pandas/issues/19493/labels{/name}","comments_url":"https://api.github.com/repos/pandas-dev/pandas/issues/19493/comments","events_url":"https://api.github.com/repos/pandas-dev/pandas/issues/19493/events","html_url":"https://github.com/pandas-dev/pandas/issues/19493","id":293554770,"number":19493,"title":"SegFault when using `to_parquet` with `flavor='spark'` when dataframe contains timestamps","user":{"login":"clembou","id":4275603,"avatar_url":"https://avatars3.githubusercontent.com/u/4275603?v=4","gravatar_id":"","url":"https://api.github.com/users/clembou","html_url":"https://github.com/clembou","followers_url":"https://api.github.com/users/clembou/followers","following_url":"https://api.github.com/users/clembou/following{/other_user}","gists_url":"https://api.github.com/users/clembou/gists{/gist_id}","starred_url":"https://api.github.com/users/clembou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clembou/subscriptions","organizations_url":"https://api.github.com/users/clembou/orgs","repos_url":"https://api.github.com/users/clembou/repos","events_url":"https://api.github.com/users/clembou/events{/privacy}","received_events_url":"https://api.github.com/users/clembou/received_events","type":"User","site_admin":false},"labels":[{"id":685114413,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/IO%20Parquet","name":"IO Parquet","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-02-01T14:41:31Z","updated_at":"2018-02-02T09:02:48Z","closed_at":null,"author_association":"NONE","body":"#### Code Sample, a copy-pastable example if possible\r\n\r\n```python\r\nimport pandas as pd\r\nimport pyarrow as pa\r\nimport pyarrow.parquet as pq\r\nfrom io import StringIO\r\n\r\ncontent = StringIO(\r\n\"\"\"report_time_of_generation\r\n2018-02-01 03:44:29\r\n2018-02-01 03:44:29\r\n2018-02-01 03:44:29\r\n2018-02-01 03:44:29\r\n2018-02-01 03:44:29\r\n\"\"\")\r\n\r\ndfs = pd.read_csv(content, parse_dates=['report_time_of_generation'])\r\n\r\nprint('data:')\r\nprint(dfs)\r\nprint('dtypes:')\r\nprint(dfs.dtypes)\r\n\r\nprint('writing the file using raw pyarrow works')\r\ntable = pa.Table.from_pandas(dfs)\r\npq.write_table(table, 'pyarrowviatable.parquet', flavor='spark')\r\n\r\nprint('using df.to_parquet segfaults')\r\ndfs.to_parquet('viapandas.parquet', flavor='spark')\r\n```\r\n\r\nresult:\r\n```\r\ndata:\r\n  report_time_of_generation\r\n0       2018-02-01 03:44:29\r\n1       2018-02-01 03:44:29\r\n2       2018-02-01 03:44:29\r\n3       2018-02-01 03:44:29\r\n4       2018-02-01 03:44:29\r\ndtypes:\r\nreport_time_of_generation    datetime64[ns]\r\ndtype: object\r\nwriting the file using raw pyarrow works\r\nusing df.to_parquet segfaults\r\n[1]    28905 segmentation fault  ./venv/bin/ipython\r\n```\r\n#### Problem description\r\n\r\n`dfs.to_parquet('viapandas.parquet', flavor='spark')`  gives me a segfault on macos and linux with pandas 0.22.0 and pyarrow v0.8.0 ( latest at the time of writing).\r\n\r\nI am filing the issue in the pandas repo since i am not hitting the issue when creating the file manually using pyarrow directly.\r\n\r\n#### Expected Output\r\n\r\n\r\n#### Output of ``pd.show_versions()``\r\n\r\n<details>\r\n\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.6.4.final.0\r\npython-bits: 64\r\nOS: Darwin\r\nOS-release: 17.3.0\r\nmachine: x86_64\r\nprocessor: i386\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: en_GB.UTF-8\r\nLOCALE: en_GB.UTF-8\r\n\r\npandas: 0.22.0\r\npytest: 3.4.0\r\npip: 9.0.1\r\nsetuptools: 38.4.0\r\nCython: None\r\nnumpy: 1.13.3\r\nscipy: None\r\npyarrow: 0.8.0\r\nxarray: None\r\nIPython: 6.2.1\r\nsphinx: None\r\npatsy: None\r\ndateutil: 2.6.1\r\npytz: 2017.3\r\nblosc: None\r\nbottleneck: None\r\ntables: None\r\nnumexpr: 2.6.4\r\nfeather: None\r\nmatplotlib: None\r\nopenpyxl: None\r\nxlrd: None\r\nxlwt: None\r\nxlsxwriter: None\r\nlxml: None\r\nbs4: 4.5.1\r\nhtml5lib: None\r\nsqlalchemy: None\r\npymysql: None\r\npsycopg2: 2.7.3.2 (dt dec pq3 ext lo64)\r\njinja2: 2.10\r\ns3fs: 0.1.3\r\nfastparquet: None\r\npandas_gbq: None\r\npandas_datareader: None\r\n\r\n</details>\r\n"},"comment":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/comments/362526800","html_url":"https://github.com/pandas-dev/pandas/issues/19493#issuecomment-362526800","issue_url":"https://api.github.com/repos/pandas-dev/pandas/issues/19493","id":362526800,"user":{"login":"jorisvandenbossche","id":1020496,"avatar_url":"https://avatars2.githubusercontent.com/u/1020496?v=4","gravatar_id":"","url":"https://api.github.com/users/jorisvandenbossche","html_url":"https://github.com/jorisvandenbossche","followers_url":"https://api.github.com/users/jorisvandenbossche/followers","following_url":"https://api.github.com/users/jorisvandenbossche/following{/other_user}","gists_url":"https://api.github.com/users/jorisvandenbossche/gists{/gist_id}","starred_url":"https://api.github.com/users/jorisvandenbossche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jorisvandenbossche/subscriptions","organizations_url":"https://api.github.com/users/jorisvandenbossche/orgs","repos_url":"https://api.github.com/users/jorisvandenbossche/repos","events_url":"https://api.github.com/users/jorisvandenbossche/events{/privacy}","received_events_url":"https://api.github.com/users/jorisvandenbossche/received_events","type":"User","site_admin":false},"created_at":"2018-02-02T09:02:48Z","updated_at":"2018-02-02T09:02:48Z","author_association":"OWNER","body":"@clembou when doing it manually with `pyarrow`, can you try passing `compression='snappy',\r\ncoerce_timestamps='ms'` to `write_table` ? Do you then get a segfault with pyarrow as well? \r\nThat seems the only difference with the internal pandas implementation and your code.\r\n"}}
{"actor":{"display_login":"jorisvandenbossche"}}