{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask-examples/issues/24","repository_url":"https://api.github.com/repos/dask/dask-examples","labels_url":"https://api.github.com/repos/dask/dask-examples/issues/24/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask-examples/issues/24/comments","events_url":"https://api.github.com/repos/dask/dask-examples/issues/24/events","html_url":"https://github.com/dask/dask-examples/pull/24","id":342392355,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAyMzAyNjQx","number":24,"title":"[WIP] Add Dataframe notebooks","user":{"login":"mrocklin","id":306380,"node_id":"MDQ6VXNlcjMwNjM4MA==","avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-18T15:59:07Z","updated_at":"2018-07-25T18:46:10Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/dask/dask-examples/pulls/24","html_url":"https://github.com/dask/dask-examples/pull/24","diff_url":"https://github.com/dask/dask-examples/pull/24.diff","patch_url":"https://github.com/dask/dask-examples/pull/24.patch"},"body":"This starts to add a series of notebooks for dask dataframe, similar to what we have now for machine learning.  \r\n\r\nI've also included a screencast in the first notebook as an experiment.  If we switch to JupyterLab we could also have this running on the side (if the users right clicks and moves the output to a new cell)\r\n\r\nBinder running here: https://mybinder.org/v2/gh/mrocklin/dask-examples/dataframes?filepath=dataframes%2F01-data-access.ipynb\r\n\r\nI'm happy to continue going here, but I'd like to get some feedback first if possible, especially from @AlbertDeFusco if he has time."},"comment":{"url":"https://api.github.com/repos/dask/dask-examples/issues/comments/407856542","html_url":"https://github.com/dask/dask-examples/pull/24#issuecomment-407856542","issue_url":"https://api.github.com/repos/dask/dask-examples/issues/24","id":407856542,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNzg1NjU0Mg==","user":{"login":"AlbertDeFusco","id":43654,"node_id":"MDQ6VXNlcjQzNjU0","avatar_url":"https://avatars0.githubusercontent.com/u/43654?v=4","gravatar_id":"","url":"https://api.github.com/users/AlbertDeFusco","html_url":"https://github.com/AlbertDeFusco","followers_url":"https://api.github.com/users/AlbertDeFusco/followers","following_url":"https://api.github.com/users/AlbertDeFusco/following{/other_user}","gists_url":"https://api.github.com/users/AlbertDeFusco/gists{/gist_id}","starred_url":"https://api.github.com/users/AlbertDeFusco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlbertDeFusco/subscriptions","organizations_url":"https://api.github.com/users/AlbertDeFusco/orgs","repos_url":"https://api.github.com/users/AlbertDeFusco/repos","events_url":"https://api.github.com/users/AlbertDeFusco/events{/privacy}","received_events_url":"https://api.github.com/users/AlbertDeFusco/received_events","type":"User","site_admin":false},"created_at":"2018-07-25T18:46:10Z","updated_at":"2018-07-25T18:46:10Z","author_association":"CONTRIBUTOR","body":"Here's a few things:\r\n\r\n## 01-data-access\r\n* perhaps mention that Dask can handle many files that in aggregate store more data than available memory\r\n* Dask can read globs of CSV files that have the same column names without loops or appends. The example shown here is that data is stored daily in separate CSV files.\r\n\r\n>Whenever we operate on our dataframe we read through all of our CSV data so that we don't fill up RAM. This is very efficient for memory use, but reading through all of the CSV files every time can be slow.\r\n\r\nI'd recommend\r\n\r\n>Whenever we operate on our dataframe only a small chunk of the data is read by each processor at a time. This means we won't fill up our RAM when operating on a large collection of data. Even though this is very memory efficient CSV files can be slow in general, which is also true for Pandas.\r\n\r\n\r\n## 02-groupby\r\n\r\nI hadn't seen `split_out` before. Would the best practice be that `split_out` should be just small enough so that a single partition can fit into memory?\r\n\r\nCustom aggregations be only be used with `.apply()`, not `.agg()`, like this function below? Perhaps *groupby-aggregation* could refer to the [common aggregation functions](https://pandas.pydata.org/pandas-docs/stable/groupby.html#aggregation)\r\n\r\n```python\r\ndef count_outliers(group):\r\n    z = (group - group.mean()) / group.std()\r\n    return (z.abs() >= 1.8).sum()\r\n\r\ndf.groupby('CAT')['VALUE'].apply(count_outliers, meta='int64')\r\n```"}}
{"actor":{"display_login":"AlbertDeFusco"}}