{"action":"created","issue":{"url":"https://api.github.com/repos/stan-dev/rstan/issues/536","repository_url":"https://api.github.com/repos/stan-dev/rstan","labels_url":"https://api.github.com/repos/stan-dev/rstan/issues/536/labels{/name}","comments_url":"https://api.github.com/repos/stan-dev/rstan/issues/536/comments","events_url":"https://api.github.com/repos/stan-dev/rstan/issues/536/events","html_url":"https://github.com/stan-dev/rstan/issues/536","id":327525190,"node_id":"MDU6SXNzdWUzMjc1MjUxOTA=","number":536,"title":"Store samples in matrix rather than dataframe","user":{"login":"aaronjg","id":933256,"node_id":"MDQ6VXNlcjkzMzI1Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/933256?v=4","gravatar_id":"","url":"https://api.github.com/users/aaronjg","html_url":"https://github.com/aaronjg","followers_url":"https://api.github.com/users/aaronjg/followers","following_url":"https://api.github.com/users/aaronjg/following{/other_user}","gists_url":"https://api.github.com/users/aaronjg/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronjg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronjg/subscriptions","organizations_url":"https://api.github.com/users/aaronjg/orgs","repos_url":"https://api.github.com/users/aaronjg/repos","events_url":"https://api.github.com/users/aaronjg/events{/privacy}","received_events_url":"https://api.github.com/users/aaronjg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-05-29T23:49:10Z","updated_at":"2018-05-30T18:50:43Z","closed_at":null,"author_association":"NONE","body":"Allocating very wide data.frames can take several hours. Though, data.frames take roughly the same amount of memory as matrices, they are much slower to allocate, create many more objects, and slow down garbage collection.\r\n\r\nIt would be nice to store the samples as matrices to make things faster. I am have a branch that has read_stan_csv create a matrix rather than a data.frame, while maintaining support for previous data.frame based stan.fit objects. \r\n\r\nFor a 10,000 parameter model, this is an additional 15% speed improvement in reading in the data  over https://github.com/stan-dev/rstan/pull/532, and I believe would offer even greater speed improvements for larger models.\r\n\r\nI'm not sure why things were initially created using data frames, but if using matrices makes sense, I can create a pull request.\r\n\r\nhttps://github.com/aaronjg/rstan/tree/use_matrices"},"comment":{"url":"https://api.github.com/repos/stan-dev/rstan/issues/comments/393275834","html_url":"https://github.com/stan-dev/rstan/issues/536#issuecomment-393275834","issue_url":"https://api.github.com/repos/stan-dev/rstan/issues/536","id":393275834,"node_id":"MDEyOklzc3VlQ29tbWVudDM5MzI3NTgzNA==","user":{"login":"aaronjg","id":933256,"node_id":"MDQ6VXNlcjkzMzI1Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/933256?v=4","gravatar_id":"","url":"https://api.github.com/users/aaronjg","html_url":"https://github.com/aaronjg","followers_url":"https://api.github.com/users/aaronjg/followers","following_url":"https://api.github.com/users/aaronjg/following{/other_user}","gists_url":"https://api.github.com/users/aaronjg/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronjg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronjg/subscriptions","organizations_url":"https://api.github.com/users/aaronjg/orgs","repos_url":"https://api.github.com/users/aaronjg/repos","events_url":"https://api.github.com/users/aaronjg/events{/privacy}","received_events_url":"https://api.github.com/users/aaronjg/received_events","type":"User","site_admin":false},"created_at":"2018-05-30T18:50:43Z","updated_at":"2018-05-30T18:50:43Z","author_association":"NONE","body":"@maverickg I wasn't suggesting the mmap as the default. However, it would be very nice to have that as optional functionality so that the entire parameter set doesn't need to be loaded into memory at once. I would set it up so the default would be a matrix, and the user could optionally write out to a mmap object, similar to how they can write out to a CSV file now. If the user wanted to serialize the data and move it to another computer they would be responsible for copying the extra files, just like they would need to copy multiple CSV files now. \r\n\r\nIf this sort of mmap interface wouldn't be used beyond me, I'd just leave it on my own fork.\r\n\r\nCurrently, I have it set up to use matrices when reading in from csv, with some changes in the C++ code to maintain compatibility with the current list of vectors generated from Rstan.\r\nhttps://github.com/aaronjg/rstan/tree/use_matrices\r\n\r\n@bgoodri Do you have any examples of models that cause segfaults from the nested std::vectors? I haven't run into this before and would like to use it as a test case."}}
{"actor":{"display_login":"aaronjg"}}