{"action":"created","comment":{"url":"https://api.github.com/repos/dask/dask/pulls/comments/74701865","id":74701865,"diff_hunk":"@@ -0,0 +1,152 @@\n+Dataframes from HDF5 files\n+=========================\n+\n+This section provides working examples of ``dask.dataframe`` methods to read HDF5 files. HDF5 is a unique technology suite that makes possible the management of extremely large and complex data collections. To learn more about HDF5, visit the `HDF Group Tutorial page <https://www.hdfgroup.org/HDF5/whatishdf5.html>`_.  For an overview of ``dask.dataframe``, its limitations, scope, and use, see the `DataFrame Overview section <http://dask.pydata.org/en/latest/dataframe-overview.html>`_.  \n+\n+  **Important Note** -- `h5py <http://docs.h5py.org/en/latest/index.html>`_ and `PyTables <http://www.pytables.org/index.html>`_ are the two premiere libraries to work with HDF5 data in Python.   ``dask.dataframe``, `like python pandas <http://pandas.pydata.org/pandas-docs/stable/io.html#io-hdf5>`_, uses the the `PyTables <http://www.pytables.org/FAQ.html#what-is-pytables>`_ library to interface with the HDF5 binary data format. HDF5 files made using the h5py project may not read into a dask DataFrame. \n+\n+  While there is `planned work to refactor PyTables to use h5py bindings to HDF5 and create 100% compatibility <https://groups.google.com/forum/#!msg/pytables-users/GBVkvGexSag/IwLjEtIfKIgJ>`_, there are differences in how the interfaces are implemented. `dask.dataframe` will have `nearly seamless compatibility with HDF5 files built in pandas <https://github.com/bokeh/bokeh/issues/1246>`_.  Here is more information on the differences between *h5py* and *PyTables* and work to improve compatibility:\n+\n+    -  `h5py explanation of differences from PyTables <http://docs.h5py.org/en/latest/faq.html#what-s-the-difference-between-h5py-and-pytables>`_\n+    -  `PyTables explanation of differences from h5py <http://www.pytables.org/FAQ.html#how-does-pytables-compare-with-the-h5py-project>`_\n+    -  `Compatibility Work: HackFest with h5py and PyTables teams <https://curtinic.github.io/python-and-hdf5-hackfest/>`_\n+    \n+    \n+**Core DataFrame Functions in this Tutorial:**\n+----------------------------------------------\n+\n+    *  Use ``dask.dataframe`` to:\n+        \n+        1.  Create dask DataFrame by loading a specific dataset (key) from a single HDF5 file\n+        2.  Read all datasets from a single HDF5 into a dask DataFrame \n+        3.  Create a dask DataFrame by loading multiple HDF5 files with different datasets (keys)","path":"docs/source/examples/dataframe-hdf5.rst","position":21,"original_position":22,"commit_id":"f7de75ddfd19401a91bb683448e12bc984bfc4d4","original_commit_id":"067f62c0364b013768b690f03df2574b19554ca4","user":{"login":"linwoodc3","id":2884914,"avatar_url":"https://avatars.githubusercontent.com/u/2884914?v=3","gravatar_id":"","url":"https://api.github.com/users/linwoodc3","html_url":"https://github.com/linwoodc3","followers_url":"https://api.github.com/users/linwoodc3/followers","following_url":"https://api.github.com/users/linwoodc3/following{/other_user}","gists_url":"https://api.github.com/users/linwoodc3/gists{/gist_id}","starred_url":"https://api.github.com/users/linwoodc3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linwoodc3/subscriptions","organizations_url":"https://api.github.com/users/linwoodc3/orgs","repos_url":"https://api.github.com/users/linwoodc3/repos","events_url":"https://api.github.com/users/linwoodc3/events{/privacy}","received_events_url":"https://api.github.com/users/linwoodc3/received_events","type":"User","site_admin":false},"body":"Just a lapse on my part.  Good catch.  Here's what it is now:\r\n *  Use ``dask.dataframe`` to:\r\n        \r\n        1.  Create dask DataFrame by loading a specific dataset (key) from a single HDF5 file\r\n        2.  Create dask DataFrame from a single HDF5 with multiple datasets (keys)  \r\n        3.  Create dask DataFrame by loading multiple HDF5 files with different datasets (keys)","created_at":"2016-08-14T13:40:52Z","updated_at":"2016-08-14T13:40:52Z","html_url":"https://github.com/dask/dask/pull/1467#discussion_r74701865","pull_request_url":"https://api.github.com/repos/dask/dask/pulls/1467","_links":{"self":{"href":"https://api.github.com/repos/dask/dask/pulls/comments/74701865"},"html":{"href":"https://github.com/dask/dask/pull/1467#discussion_r74701865"},"pull_request":{"href":"https://api.github.com/repos/dask/dask/pulls/1467"}}},"pull_request":{"url":"https://api.github.com/repos/dask/dask/pulls/1467","id":81216757,"html_url":"https://github.com/dask/dask/pull/1467","diff_url":"https://github.com/dask/dask/pull/1467.diff","patch_url":"https://github.com/dask/dask/pull/1467.patch","issue_url":"https://api.github.com/repos/dask/dask/issues/1467","number":1467,"state":"open","locked":false,"title":"DOC: Added standalone hdf5 examples page; linked from dataframe creat…","user":{"login":"linwoodc3","id":2884914,"avatar_url":"https://avatars.githubusercontent.com/u/2884914?v=3","gravatar_id":"","url":"https://api.github.com/users/linwoodc3","html_url":"https://github.com/linwoodc3","followers_url":"https://api.github.com/users/linwoodc3/followers","following_url":"https://api.github.com/users/linwoodc3/following{/other_user}","gists_url":"https://api.github.com/users/linwoodc3/gists{/gist_id}","starred_url":"https://api.github.com/users/linwoodc3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linwoodc3/subscriptions","organizations_url":"https://api.github.com/users/linwoodc3/orgs","repos_url":"https://api.github.com/users/linwoodc3/repos","events_url":"https://api.github.com/users/linwoodc3/events{/privacy}","received_events_url":"https://api.github.com/users/linwoodc3/received_events","type":"User","site_admin":false},"body":"Documentation update for HDF5.\r\n\r\nExplanations:\r\n*  Added standalone HDF5 examples page\r\n* Added link to HDF5 examples page from dataframe-examples.rst page\r\n* Added link to HDF5 examples page from dataframe-create.rst page\r\n\r\n","created_at":"2016-08-13T14:17:35Z","updated_at":"2016-08-14T13:40:52Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b7f7901a99ff833aa2dfd39c87ac960e2103481e","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/dask/dask/pulls/1467/commits","review_comments_url":"https://api.github.com/repos/dask/dask/pulls/1467/comments","review_comment_url":"https://api.github.com/repos/dask/dask/pulls/comments{/number}","comments_url":"https://api.github.com/repos/dask/dask/issues/1467/comments","statuses_url":"https://api.github.com/repos/dask/dask/statuses/f7de75ddfd19401a91bb683448e12bc984bfc4d4","head":{"label":"linwoodc3:hdf5Docs1","ref":"hdf5Docs1","sha":"f7de75ddfd19401a91bb683448e12bc984bfc4d4","user":{"login":"linwoodc3","id":2884914,"avatar_url":"https://avatars.githubusercontent.com/u/2884914?v=3","gravatar_id":"","url":"https://api.github.com/users/linwoodc3","html_url":"https://github.com/linwoodc3","followers_url":"https://api.github.com/users/linwoodc3/followers","following_url":"https://api.github.com/users/linwoodc3/following{/other_user}","gists_url":"https://api.github.com/users/linwoodc3/gists{/gist_id}","starred_url":"https://api.github.com/users/linwoodc3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linwoodc3/subscriptions","organizations_url":"https://api.github.com/users/linwoodc3/orgs","repos_url":"https://api.github.com/users/linwoodc3/repos","events_url":"https://api.github.com/users/linwoodc3/events{/privacy}","received_events_url":"https://api.github.com/users/linwoodc3/received_events","type":"User","site_admin":false},"repo":{"id":65620102,"name":"dask","full_name":"linwoodc3/dask","owner":{"login":"linwoodc3","id":2884914,"avatar_url":"https://avatars.githubusercontent.com/u/2884914?v=3","gravatar_id":"","url":"https://api.github.com/users/linwoodc3","html_url":"https://github.com/linwoodc3","followers_url":"https://api.github.com/users/linwoodc3/followers","following_url":"https://api.github.com/users/linwoodc3/following{/other_user}","gists_url":"https://api.github.com/users/linwoodc3/gists{/gist_id}","starred_url":"https://api.github.com/users/linwoodc3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linwoodc3/subscriptions","organizations_url":"https://api.github.com/users/linwoodc3/orgs","repos_url":"https://api.github.com/users/linwoodc3/repos","events_url":"https://api.github.com/users/linwoodc3/events{/privacy}","received_events_url":"https://api.github.com/users/linwoodc3/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/linwoodc3/dask","description":"Versatile parallel programming with task scheduling","fork":true,"url":"https://api.github.com/repos/linwoodc3/dask","forks_url":"https://api.github.com/repos/linwoodc3/dask/forks","keys_url":"https://api.github.com/repos/linwoodc3/dask/keys{/key_id}","collaborators_url":"https://api.github.com/repos/linwoodc3/dask/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/linwoodc3/dask/teams","hooks_url":"https://api.github.com/repos/linwoodc3/dask/hooks","issue_events_url":"https://api.github.com/repos/linwoodc3/dask/issues/events{/number}","events_url":"https://api.github.com/repos/linwoodc3/dask/events","assignees_url":"https://api.github.com/repos/linwoodc3/dask/assignees{/user}","branches_url":"https://api.github.com/repos/linwoodc3/dask/branches{/branch}","tags_url":"https://api.github.com/repos/linwoodc3/dask/tags","blobs_url":"https://api.github.com/repos/linwoodc3/dask/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/linwoodc3/dask/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/linwoodc3/dask/git/refs{/sha}","trees_url":"https://api.github.com/repos/linwoodc3/dask/git/trees{/sha}","statuses_url":"https://api.github.com/repos/linwoodc3/dask/statuses/{sha}","languages_url":"https://api.github.com/repos/linwoodc3/dask/languages","stargazers_url":"https://api.github.com/repos/linwoodc3/dask/stargazers","contributors_url":"https://api.github.com/repos/linwoodc3/dask/contributors","subscribers_url":"https://api.github.com/repos/linwoodc3/dask/subscribers","subscription_url":"https://api.github.com/repos/linwoodc3/dask/subscription","commits_url":"https://api.github.com/repos/linwoodc3/dask/commits{/sha}","git_commits_url":"https://api.github.com/repos/linwoodc3/dask/git/commits{/sha}","comments_url":"https://api.github.com/repos/linwoodc3/dask/comments{/number}","issue_comment_url":"https://api.github.com/repos/linwoodc3/dask/issues/comments{/number}","contents_url":"https://api.github.com/repos/linwoodc3/dask/contents/{+path}","compare_url":"https://api.github.com/repos/linwoodc3/dask/compare/{base}...{head}","merges_url":"https://api.github.com/repos/linwoodc3/dask/merges","archive_url":"https://api.github.com/repos/linwoodc3/dask/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/linwoodc3/dask/downloads","issues_url":"https://api.github.com/repos/linwoodc3/dask/issues{/number}","pulls_url":"https://api.github.com/repos/linwoodc3/dask/pulls{/number}","milestones_url":"https://api.github.com/repos/linwoodc3/dask/milestones{/number}","notifications_url":"https://api.github.com/repos/linwoodc3/dask/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/linwoodc3/dask/labels{/name}","releases_url":"https://api.github.com/repos/linwoodc3/dask/releases{/id}","deployments_url":"https://api.github.com/repos/linwoodc3/dask/deployments","created_at":"2016-08-13T14:05:04Z","updated_at":"2016-08-13T14:05:06Z","pushed_at":"2016-08-14T13:21:20Z","git_url":"git://github.com/linwoodc3/dask.git","ssh_url":"git@github.com:linwoodc3/dask.git","clone_url":"https://github.com/linwoodc3/dask.git","svn_url":"https://github.com/linwoodc3/dask","homepage":"http://dask.pydata.org","size":11265,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"dask:master","ref":"master","sha":"904f8e21ba7611e92c0f1abdf97f4cf1dae0feba","user":{"login":"dask","id":17131925,"avatar_url":"https://avatars.githubusercontent.com/u/17131925?v=3","gravatar_id":"","url":"https://api.github.com/users/dask","html_url":"https://github.com/dask","followers_url":"https://api.github.com/users/dask/followers","following_url":"https://api.github.com/users/dask/following{/other_user}","gists_url":"https://api.github.com/users/dask/gists{/gist_id}","starred_url":"https://api.github.com/users/dask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dask/subscriptions","organizations_url":"https://api.github.com/users/dask/orgs","repos_url":"https://api.github.com/users/dask/repos","events_url":"https://api.github.com/users/dask/events{/privacy}","received_events_url":"https://api.github.com/users/dask/received_events","type":"Organization","site_admin":false},"repo":{"id":28782747,"name":"dask","full_name":"dask/dask","owner":{"login":"dask","id":17131925,"avatar_url":"https://avatars.githubusercontent.com/u/17131925?v=3","gravatar_id":"","url":"https://api.github.com/users/dask","html_url":"https://github.com/dask","followers_url":"https://api.github.com/users/dask/followers","following_url":"https://api.github.com/users/dask/following{/other_user}","gists_url":"https://api.github.com/users/dask/gists{/gist_id}","starred_url":"https://api.github.com/users/dask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dask/subscriptions","organizations_url":"https://api.github.com/users/dask/orgs","repos_url":"https://api.github.com/users/dask/repos","events_url":"https://api.github.com/users/dask/events{/privacy}","received_events_url":"https://api.github.com/users/dask/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/dask/dask","description":"Versatile parallel programming with task scheduling","fork":false,"url":"https://api.github.com/repos/dask/dask","forks_url":"https://api.github.com/repos/dask/dask/forks","keys_url":"https://api.github.com/repos/dask/dask/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dask/dask/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dask/dask/teams","hooks_url":"https://api.github.com/repos/dask/dask/hooks","issue_events_url":"https://api.github.com/repos/dask/dask/issues/events{/number}","events_url":"https://api.github.com/repos/dask/dask/events","assignees_url":"https://api.github.com/repos/dask/dask/assignees{/user}","branches_url":"https://api.github.com/repos/dask/dask/branches{/branch}","tags_url":"https://api.github.com/repos/dask/dask/tags","blobs_url":"https://api.github.com/repos/dask/dask/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dask/dask/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dask/dask/git/refs{/sha}","trees_url":"https://api.github.com/repos/dask/dask/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dask/dask/statuses/{sha}","languages_url":"https://api.github.com/repos/dask/dask/languages","stargazers_url":"https://api.github.com/repos/dask/dask/stargazers","contributors_url":"https://api.github.com/repos/dask/dask/contributors","subscribers_url":"https://api.github.com/repos/dask/dask/subscribers","subscription_url":"https://api.github.com/repos/dask/dask/subscription","commits_url":"https://api.github.com/repos/dask/dask/commits{/sha}","git_commits_url":"https://api.github.com/repos/dask/dask/git/commits{/sha}","comments_url":"https://api.github.com/repos/dask/dask/comments{/number}","issue_comment_url":"https://api.github.com/repos/dask/dask/issues/comments{/number}","contents_url":"https://api.github.com/repos/dask/dask/contents/{+path}","compare_url":"https://api.github.com/repos/dask/dask/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dask/dask/merges","archive_url":"https://api.github.com/repos/dask/dask/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dask/dask/downloads","issues_url":"https://api.github.com/repos/dask/dask/issues{/number}","pulls_url":"https://api.github.com/repos/dask/dask/pulls{/number}","milestones_url":"https://api.github.com/repos/dask/dask/milestones{/number}","notifications_url":"https://api.github.com/repos/dask/dask/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dask/dask/labels{/name}","releases_url":"https://api.github.com/repos/dask/dask/releases{/id}","deployments_url":"https://api.github.com/repos/dask/dask/deployments","created_at":"2015-01-04T18:50:00Z","updated_at":"2016-08-13T22:18:03Z","pushed_at":"2016-08-14T13:21:21Z","git_url":"git://github.com/dask/dask.git","ssh_url":"git@github.com:dask/dask.git","clone_url":"https://github.com/dask/dask.git","svn_url":"https://github.com/dask/dask","homepage":"http://dask.pydata.org","size":12903,"stargazers_count":857,"watchers_count":857,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":149,"mirror_url":null,"open_issues_count":91,"forks":149,"open_issues":91,"watchers":857,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/dask/dask/pulls/1467"},"html":{"href":"https://github.com/dask/dask/pull/1467"},"issue":{"href":"https://api.github.com/repos/dask/dask/issues/1467"},"comments":{"href":"https://api.github.com/repos/dask/dask/issues/1467/comments"},"review_comments":{"href":"https://api.github.com/repos/dask/dask/pulls/1467/comments"},"review_comment":{"href":"https://api.github.com/repos/dask/dask/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/dask/dask/pulls/1467/commits"},"statuses":{"href":"https://api.github.com/repos/dask/dask/statuses/f7de75ddfd19401a91bb683448e12bc984bfc4d4"}}}}
{"actor":{"display_login":"linwoodc3"}}