{"action":"created","issue":{"url":"https://api.github.com/repos/Mouse-Imaging-Centre/RMINC/issues/61","repository_url":"https://api.github.com/repos/Mouse-Imaging-Centre/RMINC","labels_url":"https://api.github.com/repos/Mouse-Imaging-Centre/RMINC/issues/61/labels{/name}","comments_url":"https://api.github.com/repos/Mouse-Imaging-Centre/RMINC/issues/61/comments","events_url":"https://api.github.com/repos/Mouse-Imaging-Centre/RMINC/issues/61/events","html_url":"https://github.com/Mouse-Imaging-Centre/RMINC/issues/61","id":146737906,"number":61,"title":"mincLm fails with too many open files","user":{"login":"bcdarwin","id":164148,"avatar_url":"https://avatars.githubusercontent.com/u/164148?v=3","gravatar_id":"","url":"https://api.github.com/users/bcdarwin","html_url":"https://github.com/bcdarwin","followers_url":"https://api.github.com/users/bcdarwin/followers","following_url":"https://api.github.com/users/bcdarwin/following{/other_user}","gists_url":"https://api.github.com/users/bcdarwin/gists{/gist_id}","starred_url":"https://api.github.com/users/bcdarwin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcdarwin/subscriptions","organizations_url":"https://api.github.com/users/bcdarwin/orgs","repos_url":"https://api.github.com/users/bcdarwin/repos","events_url":"https://api.github.com/users/bcdarwin/events{/privacy}","received_events_url":"https://api.github.com/users/bcdarwin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-04-07T20:49:00Z","updated_at":"2016-04-11T15:47:53Z","closed_at":null,"body":"Running mincLm on the Foster dataset causes a crash on the 1042nd (?!) file:\r\n\r\n```\r\n> abs_voxel_fits <- mincLm(abs_jacobians ~ sex, data=csv)\r\nMethod: lm\r\nNumber of volumes: 1055\r\n/projects/mice/share/arch/linux-3_2_0-36-generic-x86_64-eglibc-2_15/src/minc-toolkit/libminc/libsrc2/volume.c:1248 (from MINC2): Unable to open file '/hpf/largeprojects/MICe/bdarwin/2016-01-Foster//pipeline/build-model-jan27_processed/FOSTER_M1048_dc/stats-volumes/FOSTER_M1048_dc-final-nlin_with_additional_inverted_absolute_log_determinant_fwhm0.1.mnc'\r\nError in mincLm(abs_jacobians ~ sex, data = csv) : \r\n  Error opening input file: /hpf/largeprojects/MICe/bdarwin/2016-01-Foster//pipeline/build-model-jan27_processed/FOSTER_M1048_dc/stats-volumes/FOSTER_M1048_dc-final-nlin_with_additional_inverted_absolute_log_determinant_fwhm0.1.mnc.\r\n> \r\n```\r\n\r\nAt this point the R process is basically nonfunctional.  `lsof -p <pid>` shows it has 1023 regular files open...coincidentally, `ulimit -Sn` gives 1024 on this machine..."},"comment":{"url":"https://api.github.com/repos/Mouse-Imaging-Centre/RMINC/issues/comments/208410864","html_url":"https://github.com/Mouse-Imaging-Centre/RMINC/issues/61#issuecomment-208410864","issue_url":"https://api.github.com/repos/Mouse-Imaging-Centre/RMINC/issues/61","id":208410864,"user":{"login":"cfhammill","id":7467038,"avatar_url":"https://avatars.githubusercontent.com/u/7467038?v=3","gravatar_id":"","url":"https://api.github.com/users/cfhammill","html_url":"https://github.com/cfhammill","followers_url":"https://api.github.com/users/cfhammill/followers","following_url":"https://api.github.com/users/cfhammill/following{/other_user}","gists_url":"https://api.github.com/users/cfhammill/gists{/gist_id}","starred_url":"https://api.github.com/users/cfhammill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cfhammill/subscriptions","organizations_url":"https://api.github.com/users/cfhammill/orgs","repos_url":"https://api.github.com/users/cfhammill/repos","events_url":"https://api.github.com/users/cfhammill/events{/privacy}","received_events_url":"https://api.github.com/users/cfhammill/received_events","type":"User","site_admin":false},"created_at":"2016-04-11T15:47:53Z","updated_at":"2016-04-11T15:47:53Z","body":"A couple of ideas: \r\nMac can report ulimit -Sn, so we can call out and check if that minus the number of open files is greater than the number of minc files before running.\r\n\r\nWe can open and close the minc file descriptors repeatedly throughout the C  code, Ben did a back of the envelope calculation putting it at an extra minute or two for 1000 minc files with 500 slices on the dimension mincLm reads along. This could be used in combination with the earlier strategy, engaging slow mode if there are too many files.\r\n\r\nLastly we could provide an experiment abstraction for large collections of minc files. We could dump all of the voxel values into sqlite table along with necessary metadata. Since dplyr can lazily [manipulate databases](https://cran.r-project.org/web/packages/dplyr/vignettes/databases.html), many of our common operations can be reformulated as dplyr operations. The table would be filled iteratively, closing each minc file after writing to the db. We would then only need one file descriptor to access the database.\r\n\r\n "}}