{"action":"created","issue":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/14595","repository_url":"https://api.github.com/repos/pandas-dev/pandas","labels_url":"https://api.github.com/repos/pandas-dev/pandas/issues/14595/labels{/name}","comments_url":"https://api.github.com/repos/pandas-dev/pandas/issues/14595/comments","events_url":"https://api.github.com/repos/pandas-dev/pandas/issues/14595/events","html_url":"https://github.com/pandas-dev/pandas/issues/14595","id":187534013,"number":14595,"title":"Read_HDF with nrows=[small-number] takes Much Time and RAM","user":{"login":"MaxPowerWasTaken","id":18562882,"avatar_url":"https://avatars.githubusercontent.com/u/18562882?v=3","gravatar_id":"","url":"https://api.github.com/users/MaxPowerWasTaken","html_url":"https://github.com/MaxPowerWasTaken","followers_url":"https://api.github.com/users/MaxPowerWasTaken/followers","following_url":"https://api.github.com/users/MaxPowerWasTaken/following{/other_user}","gists_url":"https://api.github.com/users/MaxPowerWasTaken/gists{/gist_id}","starred_url":"https://api.github.com/users/MaxPowerWasTaken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MaxPowerWasTaken/subscriptions","organizations_url":"https://api.github.com/users/MaxPowerWasTaken/orgs","repos_url":"https://api.github.com/users/MaxPowerWasTaken/repos","events_url":"https://api.github.com/users/MaxPowerWasTaken/events{/privacy}","received_events_url":"https://api.github.com/users/MaxPowerWasTaken/received_events","type":"User","site_admin":false},"labels":[{"id":47229190,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/IO%20HDF5","name":"IO HDF5","color":"5319e7","default":false},{"id":34444536,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/Usage%20Question","name":"Usage Question","color":"0052cc","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/pandas-dev/pandas/milestones/33","html_url":"https://github.com/pandas-dev/pandas/milestones/No%20action","labels_url":"https://api.github.com/repos/pandas-dev/pandas/milestones/33/labels","id":997544,"number":33,"title":"No action","description":"A milestone for closed or open issues for which there is no action needed (eg not a bug, just a question / discussion, nothing to resolve, wontfix, etc).\r\n\r\n","creator":{"login":"jorisvandenbossche","id":1020496,"avatar_url":"https://avatars.githubusercontent.com/u/1020496?v=3","gravatar_id":"","url":"https://api.github.com/users/jorisvandenbossche","html_url":"https://github.com/jorisvandenbossche","followers_url":"https://api.github.com/users/jorisvandenbossche/followers","following_url":"https://api.github.com/users/jorisvandenbossche/following{/other_user}","gists_url":"https://api.github.com/users/jorisvandenbossche/gists{/gist_id}","starred_url":"https://api.github.com/users/jorisvandenbossche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jorisvandenbossche/subscriptions","organizations_url":"https://api.github.com/users/jorisvandenbossche/orgs","repos_url":"https://api.github.com/users/jorisvandenbossche/repos","events_url":"https://api.github.com/users/jorisvandenbossche/events{/privacy}","received_events_url":"https://api.github.com/users/jorisvandenbossche/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":491,"state":"open","created_at":"2015-02-26T19:29:05Z","updated_at":"2016-11-06T02:03:52Z","due_on":null,"closed_at":null},"comments":8,"created_at":"2016-11-06T00:51:58Z","updated_at":"2016-11-06T21:27:32Z","closed_at":"2016-11-06T02:02:07Z","body":"Using Python 3.5.2 and Pandas 0.18.1\r\n\r\nI find that sampling a large HDF5 file using the nrows parameter takes far more RAM and time than sampling the first n rows by pulling the first n indices using a where statement.\r\n\r\nMinimal Code Example:\r\n```\r\n\r\nstore = pd.HDFStore('data_path/hdf_store.h5')\r\nselect_indices = np.arange(1000)\r\n\r\ntime1 = time.time()\r\ntrain_date_hdf = store.select('table_name', where=select_indices)\r\ntime2=time.time()\r\nprint(time2-time1)\r\n\r\ntime1 = time.time()\r\ntrain_date_hdf2 = pd.read_hdf(store, 'table_name', nrows=1000)\r\ntime2=time.time()\r\nprint(time2-time1)\r\n\r\n```\r\n\r\nOutput is:\r\n\r\n> 0.10559678077697754\r\n> 84.43739724159241\r\n\r\nIn addition to the much longer time taken by using the nrows parameter, this small sample of a large table also spikes my RAM from 20% to 97% on my machine, which has 8GB of RAM. Whereas reading 1000 rows by selecting 1000 rows by index with a where clause doesn't even use 1% of my RAM. \r\n\r\nWould there be a downside to having nrows=n be implemented by pulling the first n indices in a where statement by default?"},"comment":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/comments/258712053","html_url":"https://github.com/pandas-dev/pandas/issues/14595#issuecomment-258712053","issue_url":"https://api.github.com/repos/pandas-dev/pandas/issues/14595","id":258712053,"user":{"login":"MaxPowerWasTaken","id":18562882,"avatar_url":"https://avatars.githubusercontent.com/u/18562882?v=3","gravatar_id":"","url":"https://api.github.com/users/MaxPowerWasTaken","html_url":"https://github.com/MaxPowerWasTaken","followers_url":"https://api.github.com/users/MaxPowerWasTaken/followers","following_url":"https://api.github.com/users/MaxPowerWasTaken/following{/other_user}","gists_url":"https://api.github.com/users/MaxPowerWasTaken/gists{/gist_id}","starred_url":"https://api.github.com/users/MaxPowerWasTaken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MaxPowerWasTaken/subscriptions","organizations_url":"https://api.github.com/users/MaxPowerWasTaken/orgs","repos_url":"https://api.github.com/users/MaxPowerWasTaken/repos","events_url":"https://api.github.com/users/MaxPowerWasTaken/events{/privacy}","received_events_url":"https://api.github.com/users/MaxPowerWasTaken/received_events","type":"User","site_admin":false},"created_at":"2016-11-06T21:27:32Z","updated_at":"2016-11-06T21:27:32Z","body":"Ok thanks, I understand now. Can I ask why you'd prefer they be named parameters rather than keyword parameters? And would you like the same for the other pd.read_ functions?"}}
{"actor":{"display_login":"MaxPowerWasTaken"}}