{"action":"created","issue":{"url":"https://api.github.com/repos/LTLA/beachmat/issues/10","repository_url":"https://api.github.com/repos/LTLA/beachmat","labels_url":"https://api.github.com/repos/LTLA/beachmat/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/LTLA/beachmat/issues/10/comments","events_url":"https://api.github.com/repos/LTLA/beachmat/issues/10/events","html_url":"https://github.com/LTLA/beachmat/issues/10","id":347738012,"node_id":"MDU6SXNzdWUzNDc3MzgwMTI=","number":10,"title":"Support for multi-column extraction","user":{"login":"LTLA","id":8166669,"node_id":"MDQ6VXNlcjgxNjY2Njk=","avatar_url":"https://avatars3.githubusercontent.com/u/8166669?v=4","gravatar_id":"","url":"https://api.github.com/users/LTLA","html_url":"https://github.com/LTLA","followers_url":"https://api.github.com/users/LTLA/followers","following_url":"https://api.github.com/users/LTLA/following{/other_user}","gists_url":"https://api.github.com/users/LTLA/gists{/gist_id}","starred_url":"https://api.github.com/users/LTLA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LTLA/subscriptions","organizations_url":"https://api.github.com/users/LTLA/orgs","repos_url":"https://api.github.com/users/LTLA/repos","events_url":"https://api.github.com/users/LTLA/events{/privacy}","received_events_url":"https://api.github.com/users/LTLA/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-08-05T22:48:14Z","updated_at":"2018-08-08T11:02:48Z","closed_at":null,"author_association":"OWNER","body":"The idea is to have a single function call that extracts multiple columns (or rows) at once. This could be more efficient for `HDF5Matrix` objects as _(hand-waving starts here)_ the HDF5 library would only need to make a single OS call to retrieve data from disk. It would also be a little bit more convenient for users.\r\n\r\nI'm torn about this. It's not a lot of work, but it's useful in only very limited circumstances (and even then I'm not sure). Specifically, it would only really be useful for `HDF5Matrix` objects, as multi-column requests would simply involve multiple calls to the single-column getters for the other object types. It would further only be useful for column-chunked files; we get multiple consecutive columns for free with rectangular chunks (via the chunk cache) and it would be madness to do random access with rectangular chunks.\r\n\r\nI'm curious to know how much of a difference there is between multiple `read` calls versus a single `read` call. But if performance for on-disk representations is critical, it seems more appropriate to directly call the HDF5 library, rather than relying on _beachmat_ to optimize for speed (it does so for memory).\r\n\r\nPulling in @drisso as he triggered this particular episode. I note that in the case of mini-batching k-means, it would probably be better to use the block-processing mechanism in R anyway, as he will probably need to realize chunks of a `DelayedArray` with non-trivial delayed operations.\r\n\r\nAlso pulling in @mikejiang, who probably has some thoughts about HDF5 file access speed."},"comment":{"url":"https://api.github.com/repos/LTLA/beachmat/issues/comments/411368601","html_url":"https://github.com/LTLA/beachmat/issues/10#issuecomment-411368601","issue_url":"https://api.github.com/repos/LTLA/beachmat/issues/10","id":411368601,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTM2ODYwMQ==","user":{"login":"LTLA","id":8166669,"node_id":"MDQ6VXNlcjgxNjY2Njk=","avatar_url":"https://avatars3.githubusercontent.com/u/8166669?v=4","gravatar_id":"","url":"https://api.github.com/users/LTLA","html_url":"https://github.com/LTLA","followers_url":"https://api.github.com/users/LTLA/followers","following_url":"https://api.github.com/users/LTLA/following{/other_user}","gists_url":"https://api.github.com/users/LTLA/gists{/gist_id}","starred_url":"https://api.github.com/users/LTLA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LTLA/subscriptions","organizations_url":"https://api.github.com/users/LTLA/orgs","repos_url":"https://api.github.com/users/LTLA/repos","events_url":"https://api.github.com/users/LTLA/events{/privacy}","received_events_url":"https://api.github.com/users/LTLA/received_events","type":"User","site_admin":false},"created_at":"2018-08-08T11:02:48Z","updated_at":"2018-08-08T11:02:48Z","author_association":"OWNER","body":"Uh. Well, this is embarrassing. Looks like it's the other way around - multiple `read` calls are actually 2-3 faster. This is definitely a result of the `H5File::read` call, based on timings after commenting it out. \r\n\r\nWe can make the single `read` go almost as fast as multiple `read`s if consecutive columns are extracted, _and_ the output dataspace is two-dimensional. However, it slows down for non-consecutive columns. My guess is that there's some copying inside the HDF5 library to get values in the right locations for output."}}
{"actor":{"display_login":"LTLA"}}