{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/3716","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/3716/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/3716/comments","events_url":"https://api.github.com/repos/dask/dask/issues/3716/events","html_url":"https://github.com/dask/dask/issues/3716","id":338278656,"node_id":"MDU6SXNzdWUzMzgyNzg2NTY=","number":3716,"title":"Set index and save to parquet dask 0.17.5 vs 0.18.1","user":{"login":"rpanai","id":21178372,"node_id":"MDQ6VXNlcjIxMTc4Mzcy","avatar_url":"https://avatars0.githubusercontent.com/u/21178372?v=4","gravatar_id":"","url":"https://api.github.com/users/rpanai","html_url":"https://github.com/rpanai","followers_url":"https://api.github.com/users/rpanai/followers","following_url":"https://api.github.com/users/rpanai/following{/other_user}","gists_url":"https://api.github.com/users/rpanai/gists{/gist_id}","starred_url":"https://api.github.com/users/rpanai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rpanai/subscriptions","organizations_url":"https://api.github.com/users/rpanai/orgs","repos_url":"https://api.github.com/users/rpanai/repos","events_url":"https://api.github.com/users/rpanai/events{/privacy}","received_events_url":"https://api.github.com/users/rpanai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-07-04T13:34:04Z","updated_at":"2018-07-04T14:11:07Z","closed_at":null,"author_association":"NONE","body":"I have several parquets file saved on `tmp_folder` I want to read, sort by a key and save the sorted file to parquets inside `folder`. My df has 59M rows and I'm running the following code on a machine with 16 cores 24GB of RAM on GCP. I would like to run this with the minimum amount as possible as then I'm going to process the single files one-by-one.\r\nThe following code runs in about 24 with `dask.__version__=='0.17.5'`\r\n```python\r\nfolder = \"test_dataset\"\r\ntmp_folder = \"_\".join([folder,\"tmp\"])\r\ndf = dd.read_parquet(tmp_folder)\r\nwith dask.set_options(get=dask.get):\r\n    nrows = len(df)\r\nnpartitions = int(nrows//2e6) # check\r\ndf = df.repartition(npartitions=npartitions)\r\nwith dask.set_options(get=dask.get):\r\n    df = df.set_index(\"key\")\r\nwith dask.set_options(get=dask.get):\r\n    df.to_parquet(folder, compression=\"snappy\") \r\n```\r\n\r\nBut if I try to run the same code in `dask.__version__=='0.18.1'` and changing `with dask.set_options(get=dask.get)` with `with dask.config.set(scheduler='single-threaded')` the process ends but I don't have any files in `folder`.\r\nI'm not sure if I'm getting confused with the right `scheduler` to use in this case or it's a bug."},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/402490694","html_url":"https://github.com/dask/dask/issues/3716#issuecomment-402490694","issue_url":"https://api.github.com/repos/dask/dask/issues/3716","id":402490694,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMjQ5MDY5NA==","user":{"login":"rpanai","id":21178372,"node_id":"MDQ6VXNlcjIxMTc4Mzcy","avatar_url":"https://avatars0.githubusercontent.com/u/21178372?v=4","gravatar_id":"","url":"https://api.github.com/users/rpanai","html_url":"https://github.com/rpanai","followers_url":"https://api.github.com/users/rpanai/followers","following_url":"https://api.github.com/users/rpanai/following{/other_user}","gists_url":"https://api.github.com/users/rpanai/gists{/gist_id}","starred_url":"https://api.github.com/users/rpanai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rpanai/subscriptions","organizations_url":"https://api.github.com/users/rpanai/orgs","repos_url":"https://api.github.com/users/rpanai/repos","events_url":"https://api.github.com/users/rpanai/events{/privacy}","received_events_url":"https://api.github.com/users/rpanai/received_events","type":"User","site_admin":false},"created_at":"2018-07-04T14:11:07Z","updated_at":"2018-07-04T14:11:07Z","author_association":"NONE","body":"@martindurant and @mrocklin i posted the [code](https://github.com/dask/dask/issues/3716#issuecomment-402488664) I'm running with the latest dask version and I'm using `dask.config.set(scheduler='single-threaded')`."}}
{"actor":{"display_login":"rpanai"}}