{"action":"created","comment":{"url":"https://api.github.com/repos/dask/dask/pulls/comments/74692006","id":74692006,"diff_hunk":"@@ -0,0 +1,111 @@\n+Dataframes from HDF5 files\n+=========================\n+\n+This section provides working examples of ``dask.dataframe`` methods to read HDF5 files. HDF5 is a unique technology suite that makes possible the management of extremely large and complex data collections. To learn more about HDF5, visit the `HDF Group Tutorial page <https://www.hdfgroup.org/HDF5/whatishdf5.html>`_.  For an overview of ``dask.dataframe``, its limitations, scope, and use, see the `DataFrame Overview section <http://dask.pydata.org/en/latest/dataframe-overview.html>`_. \n+\n+  **Important Note** -- `h5py <http://docs.h5py.org/en/latest/index.html>`_ and `PyTables <http://www.pytables.org/index.html>`_ are the two premiere libraries to work with HDF5 data in Python.   ``dask.dataframe``, `like python pandas <http://pandas.pydata.org/pandas-docs/stable/io.html#io-hdf5>`_, uses the the `PyTables <http://www.pytables.org/FAQ.html#what-is-pytables>`_ library to interface with the HDF5 binary data format. HDF5 files made using the h5py project may not read into a dask DataFrame. \n+\n+  While there is `planned work to refactor PyTables to use h5py bindings to HDF5 and create 100% compatibility <https://groups.google.com/forum/#!msg/pytables-users/GBVkvGexSag/IwLjEtIfKIgJ>`_, there are differences in how the interfaces are implemented. `dask.dataframe` will have `nearly seamless compatibility with HDF5 files built in pandas <https://github.com/bokeh/bokeh/issues/1246>`_.  Here is more information on the differences between *h5py* and *PyTables* and work to improve compatibility:\n+\n+    -  `h5py explanation of differences from PyTables <http://docs.h5py.org/en/latest/faq.html#what-s-the-difference-between-h5py-and-pytables>`_\n+    -  `PyTables explanation of differences from h5py <http://www.pytables.org/FAQ.html#how-does-pytables-compare-with-the-h5py-project>`_\n+    -  `HackFest with h5py and PyTables teams <https://curtinic.github.io/python-and-hdf5-hackfest/>`_\n+\n+**Core DataFrame Functions in this Tutorial:**\n+----------------------------------------------\n+\n+    *  Use ``dask.dataframe`` to:\n+        \n+        1.  Create dask DataFrame by loading a specific dataset (key) from a single HDF5 file\n+        2.  Read all datasets from a single HDF5 into a dask DataFrame \n+        3.  Create a dask DataFrame by loading multiple HDF5 files with different datasets (keys)\n+\n+\n+**Generate Synthetic HDF5 files:**\n+--------------------------------------------\n+\n+These exercises require multiple HDF5 files with multiple dataset keys.  The `Python pandas library <http://pandas.pydata.org/pandas-docs/stable/>`_ has a convenient `I/O API to create an HDF5 file from a dataframe <http://pandas.pydata.org/pandas-docs/stable/io.html#io-hdf5>`_. Combining this method with the `Python numpy random integer generator <http://docs.scipy.org/doc/numpy/reference/generated/numpy.random.randint.html>`_ and several standard library modules provide all the tools necessary to create a synthetic dataset.\n+\n+.. code-block:: python\n+   :emphasize-lines: 40,45\n+   \n+    \n+    def create_hdf5s(iters):\n+           \n+       import random\n+       import pandas as pd\n+       import numpy as np\n+       import string \n+       import json\n+    \n+       # Intitalize counter to build specific number of hdf5 files\n+       count = 0\n+       setseed=1\n+       np.random.seed(seed=setseed)\n+       random.seed(setseed)","path":"docs/source/examples/dataframe-hdf5.rst","position":45,"original_position":45,"commit_id":"f2d9c646538868ece1f06eecb404bfacfea1d6c3","original_commit_id":"f2d9c646538868ece1f06eecb404bfacfea1d6c3","user":{"login":"linwoodc3","id":2884914,"avatar_url":"https://avatars.githubusercontent.com/u/2884914?v=3","gravatar_id":"","url":"https://api.github.com/users/linwoodc3","html_url":"https://github.com/linwoodc3","followers_url":"https://api.github.com/users/linwoodc3/followers","following_url":"https://api.github.com/users/linwoodc3/following{/other_user}","gists_url":"https://api.github.com/users/linwoodc3/gists{/gist_id}","starred_url":"https://api.github.com/users/linwoodc3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linwoodc3/subscriptions","organizations_url":"https://api.github.com/users/linwoodc3/orgs","repos_url":"https://api.github.com/users/linwoodc3/repos","events_url":"https://api.github.com/users/linwoodc3/events{/privacy}","received_events_url":"https://api.github.com/users/linwoodc3/received_events","type":"User","site_admin":false},"body":"Can drop; just wanted a way to create a data set so people could try this on their comp to get the hang of it.  Often folks are coming in brand new (like me) after a few googles so could benefit from synthetic data like the pandas documentation uses throughout.  Will change if it's wasteful.  From the Django documentation video, the concept was to consider the full spectrum of users (complete newbie looking for a quick win to your advanced guy).  There are examples in the dataframe-create.pst page that may be all the intermediate to advanced person needs.  For the hand-holders, the walkthrough with synthetic may keep the unnecessary traffic out of the gitter or issue or other communication methods.  I'm open to either way you want to go.  Putting in what would have helped me.  ","created_at":"2016-08-13T20:43:05Z","updated_at":"2016-08-13T20:43:05Z","html_url":"https://github.com/dask/dask/pull/1413#discussion_r74692006","pull_request_url":"https://api.github.com/repos/dask/dask/pulls/1413","_links":{"self":{"href":"https://api.github.com/repos/dask/dask/pulls/comments/74692006"},"html":{"href":"https://github.com/dask/dask/pull/1413#discussion_r74692006"},"pull_request":{"href":"https://api.github.com/repos/dask/dask/pulls/1413"}}},"pull_request":{"url":"https://api.github.com/repos/dask/dask/pulls/1413","id":78780002,"html_url":"https://github.com/dask/dask/pull/1413","diff_url":"https://github.com/dask/dask/pull/1413.diff","patch_url":"https://github.com/dask/dask/pull/1413.patch","issue_url":"https://api.github.com/repos/dask/dask/issues/1413","number":1413,"state":"open","locked":false,"title":"dask Documentation update: Added complete HDF5 examples,pyTables vs h5py section, and cross references related pages.","user":{"login":"linwoodc3","id":2884914,"avatar_url":"https://avatars.githubusercontent.com/u/2884914?v=3","gravatar_id":"","url":"https://api.github.com/users/linwoodc3","html_url":"https://github.com/linwoodc3","followers_url":"https://api.github.com/users/linwoodc3/followers","following_url":"https://api.github.com/users/linwoodc3/following{/other_user}","gists_url":"https://api.github.com/users/linwoodc3/gists{/gist_id}","starred_url":"https://api.github.com/users/linwoodc3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linwoodc3/subscriptions","organizations_url":"https://api.github.com/users/linwoodc3/orgs","repos_url":"https://api.github.com/users/linwoodc3/repos","events_url":"https://api.github.com/users/linwoodc3/events{/privacy}","received_events_url":"https://api.github.com/users/linwoodc3/received_events","type":"User","site_admin":false},"body":"Added more to the HDF5 documentation given the importance of the HDF5 method.  ","created_at":"2016-07-26T01:49:11Z","updated_at":"2016-08-13T20:43:05Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2d1aad362890a9efc15bb4368dc1a1da07229849","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/dask/dask/pulls/1413/commits","review_comments_url":"https://api.github.com/repos/dask/dask/pulls/1413/comments","review_comment_url":"https://api.github.com/repos/dask/dask/pulls/comments{/number}","comments_url":"https://api.github.com/repos/dask/dask/issues/1413/comments","statuses_url":"https://api.github.com/repos/dask/dask/statuses/f2d9c646538868ece1f06eecb404bfacfea1d6c3","head":{"label":"linwoodc3:new-hdf5-docs","ref":"new-hdf5-docs","sha":"f2d9c646538868ece1f06eecb404bfacfea1d6c3","user":{"login":"linwoodc3","id":2884914,"avatar_url":"https://avatars.githubusercontent.com/u/2884914?v=3","gravatar_id":"","url":"https://api.github.com/users/linwoodc3","html_url":"https://github.com/linwoodc3","followers_url":"https://api.github.com/users/linwoodc3/followers","following_url":"https://api.github.com/users/linwoodc3/following{/other_user}","gists_url":"https://api.github.com/users/linwoodc3/gists{/gist_id}","starred_url":"https://api.github.com/users/linwoodc3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linwoodc3/subscriptions","organizations_url":"https://api.github.com/users/linwoodc3/orgs","repos_url":"https://api.github.com/users/linwoodc3/repos","events_url":"https://api.github.com/users/linwoodc3/events{/privacy}","received_events_url":"https://api.github.com/users/linwoodc3/received_events","type":"User","site_admin":false},"repo":null},"base":{"label":"dask:master","ref":"master","sha":"bf376da278292d157d33564a52ef5e409e92bfc3","user":{"login":"dask","id":17131925,"avatar_url":"https://avatars.githubusercontent.com/u/17131925?v=3","gravatar_id":"","url":"https://api.github.com/users/dask","html_url":"https://github.com/dask","followers_url":"https://api.github.com/users/dask/followers","following_url":"https://api.github.com/users/dask/following{/other_user}","gists_url":"https://api.github.com/users/dask/gists{/gist_id}","starred_url":"https://api.github.com/users/dask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dask/subscriptions","organizations_url":"https://api.github.com/users/dask/orgs","repos_url":"https://api.github.com/users/dask/repos","events_url":"https://api.github.com/users/dask/events{/privacy}","received_events_url":"https://api.github.com/users/dask/received_events","type":"Organization","site_admin":false},"repo":{"id":28782747,"name":"dask","full_name":"dask/dask","owner":{"login":"dask","id":17131925,"avatar_url":"https://avatars.githubusercontent.com/u/17131925?v=3","gravatar_id":"","url":"https://api.github.com/users/dask","html_url":"https://github.com/dask","followers_url":"https://api.github.com/users/dask/followers","following_url":"https://api.github.com/users/dask/following{/other_user}","gists_url":"https://api.github.com/users/dask/gists{/gist_id}","starred_url":"https://api.github.com/users/dask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dask/subscriptions","organizations_url":"https://api.github.com/users/dask/orgs","repos_url":"https://api.github.com/users/dask/repos","events_url":"https://api.github.com/users/dask/events{/privacy}","received_events_url":"https://api.github.com/users/dask/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/dask/dask","description":"Versatile parallel programming with task scheduling","fork":false,"url":"https://api.github.com/repos/dask/dask","forks_url":"https://api.github.com/repos/dask/dask/forks","keys_url":"https://api.github.com/repos/dask/dask/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dask/dask/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dask/dask/teams","hooks_url":"https://api.github.com/repos/dask/dask/hooks","issue_events_url":"https://api.github.com/repos/dask/dask/issues/events{/number}","events_url":"https://api.github.com/repos/dask/dask/events","assignees_url":"https://api.github.com/repos/dask/dask/assignees{/user}","branches_url":"https://api.github.com/repos/dask/dask/branches{/branch}","tags_url":"https://api.github.com/repos/dask/dask/tags","blobs_url":"https://api.github.com/repos/dask/dask/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dask/dask/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dask/dask/git/refs{/sha}","trees_url":"https://api.github.com/repos/dask/dask/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dask/dask/statuses/{sha}","languages_url":"https://api.github.com/repos/dask/dask/languages","stargazers_url":"https://api.github.com/repos/dask/dask/stargazers","contributors_url":"https://api.github.com/repos/dask/dask/contributors","subscribers_url":"https://api.github.com/repos/dask/dask/subscribers","subscription_url":"https://api.github.com/repos/dask/dask/subscription","commits_url":"https://api.github.com/repos/dask/dask/commits{/sha}","git_commits_url":"https://api.github.com/repos/dask/dask/git/commits{/sha}","comments_url":"https://api.github.com/repos/dask/dask/comments{/number}","issue_comment_url":"https://api.github.com/repos/dask/dask/issues/comments{/number}","contents_url":"https://api.github.com/repos/dask/dask/contents/{+path}","compare_url":"https://api.github.com/repos/dask/dask/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dask/dask/merges","archive_url":"https://api.github.com/repos/dask/dask/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dask/dask/downloads","issues_url":"https://api.github.com/repos/dask/dask/issues{/number}","pulls_url":"https://api.github.com/repos/dask/dask/pulls{/number}","milestones_url":"https://api.github.com/repos/dask/dask/milestones{/number}","notifications_url":"https://api.github.com/repos/dask/dask/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dask/dask/labels{/name}","releases_url":"https://api.github.com/repos/dask/dask/releases{/id}","deployments_url":"https://api.github.com/repos/dask/dask/deployments","created_at":"2015-01-04T18:50:00Z","updated_at":"2016-08-13T14:07:27Z","pushed_at":"2016-08-13T14:17:35Z","git_url":"git://github.com/dask/dask.git","ssh_url":"git@github.com:dask/dask.git","clone_url":"https://github.com/dask/dask.git","svn_url":"https://github.com/dask/dask","homepage":"http://dask.pydata.org","size":12903,"stargazers_count":856,"watchers_count":856,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":149,"mirror_url":null,"open_issues_count":90,"forks":149,"open_issues":90,"watchers":856,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/dask/dask/pulls/1413"},"html":{"href":"https://github.com/dask/dask/pull/1413"},"issue":{"href":"https://api.github.com/repos/dask/dask/issues/1413"},"comments":{"href":"https://api.github.com/repos/dask/dask/issues/1413/comments"},"review_comments":{"href":"https://api.github.com/repos/dask/dask/pulls/1413/comments"},"review_comment":{"href":"https://api.github.com/repos/dask/dask/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/dask/dask/pulls/1413/commits"},"statuses":{"href":"https://api.github.com/repos/dask/dask/statuses/f2d9c646538868ece1f06eecb404bfacfea1d6c3"}}}}
{"actor":{"display_login":"linwoodc3"}}