{"action":"created","comment":{"url":"https://api.github.com/repos/fruux/sabre-dav/pulls/comments/79473068","id":79473068,"diff_hunk":"@@ -929,6 +933,25 @@ private function addPathNodesRecursively(&$propFindRequests, PropFind $propFind)\n      */\n     function getPropertiesForPath($path, $propertyNames = [], $depth = 0) {","path":"lib/DAV/Server.php","position":39,"original_position":39,"commit_id":"0b8d08334e6985100a15d58889b0aa42b6ae53c5","original_commit_id":"0b8d08334e6985100a15d58889b0aa42b6ae53c5","user":{"login":"staabm","id":120441,"avatar_url":"https://avatars.githubusercontent.com/u/120441?v=3","gravatar_id":"","url":"https://api.github.com/users/staabm","html_url":"https://github.com/staabm","followers_url":"https://api.github.com/users/staabm/followers","following_url":"https://api.github.com/users/staabm/following{/other_user}","gists_url":"https://api.github.com/users/staabm/gists{/gist_id}","starred_url":"https://api.github.com/users/staabm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/staabm/subscriptions","organizations_url":"https://api.github.com/users/staabm/orgs","repos_url":"https://api.github.com/users/staabm/repos","events_url":"https://api.github.com/users/staabm/events{/privacy}","received_events_url":"https://api.github.com/users/staabm/received_events","type":"User","site_admin":false},"body":"Should we deprecate this method and mention the new Iterator based method call the successor? Or do you think there are still good usecases in the future in which a iterator is not sufficient?","created_at":"2016-09-19T19:45:42Z","updated_at":"2016-09-19T19:45:50Z","html_url":"https://github.com/fruux/sabre-dav/pull/886#discussion_r79473068","pull_request_url":"https://api.github.com/repos/fruux/sabre-dav/pulls/886","_links":{"self":{"href":"https://api.github.com/repos/fruux/sabre-dav/pulls/comments/79473068"},"html":{"href":"https://github.com/fruux/sabre-dav/pull/886#discussion_r79473068"},"pull_request":{"href":"https://api.github.com/repos/fruux/sabre-dav/pulls/886"}}},"pull_request":{"url":"https://api.github.com/repos/fruux/sabre-dav/pulls/886","id":85405284,"html_url":"https://github.com/fruux/sabre-dav/pull/886","diff_url":"https://github.com/fruux/sabre-dav/pull/886.diff","patch_url":"https://github.com/fruux/sabre-dav/pull/886.patch","issue_url":"https://api.github.com/repos/fruux/sabre-dav/issues/886","number":886,"state":"open","locked":false,"title":"Optimize memory usage","user":{"login":"petrkotek","id":5679032,"avatar_url":"https://avatars.githubusercontent.com/u/5679032?v=3","gravatar_id":"","url":"https://api.github.com/users/petrkotek","html_url":"https://github.com/petrkotek","followers_url":"https://api.github.com/users/petrkotek/followers","following_url":"https://api.github.com/users/petrkotek/following{/other_user}","gists_url":"https://api.github.com/users/petrkotek/gists{/gist_id}","starred_url":"https://api.github.com/users/petrkotek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petrkotek/subscriptions","organizations_url":"https://api.github.com/users/petrkotek/orgs","repos_url":"https://api.github.com/users/petrkotek/repos","events_url":"https://api.github.com/users/petrkotek/events{/privacy}","received_events_url":"https://api.github.com/users/petrkotek/received_events","type":"User","site_admin":false},"body":"Hi!\r\n\r\nOur company is currently running WebDAV server in production - which is powered by `sabre/dav`.\r\n\r\nIn general it works for us, however we need to be able to list directories with tens of thousands of files while having strict memory limits.\r\n\r\nCurrent upstream implementation prepares all the results (\"properties\") into an array and then generates XML response, so memory requirements grow linearly with count of files in a directory, which doesn't really scale well. While 1000 files in a directory uses 17.3MB (peak memory usage) when listing, 50000 files uses 338 MB.\r\n\r\nI wanted to open a discussion about this problem to discuss possible solutions.\r\n\r\nIn this PR, I implemented it ~~very simple proof of concept~~ using [Generators](http://php.net/manual/en/language.generators.php), which **reduces peak memory usage from aforementioned 338MB to 46.6MB when listing a directory with 50000 files**.\r\n\r\nI was wondering if somebody else has come across this issue and if the repo owners or community is interested in memory optimization.\r\n\r\nAny feedback appreciated.\r\n\r\nNotes:\r\n- ~~this PR is proof of concept to demonstrate how memory usage could be reduced~~\r\n- ~tests will fail, because `Server::getPropertiesForPath()` now returns a `Generator` rather than an array; this functionality could be non-default - i.e. flip some switch (e.g. by a call of new method `Server::useGeneratorsForPropFind(true)`, which would make it non-breaking change)~~\r\n- ~~proposed implementation optimizes only listings with depth 1~~","created_at":"2016-09-15T13:37:35Z","updated_at":"2016-09-19T19:45:50Z","closed_at":null,"merged_at":null,"merge_commit_sha":"be28feaa7afa78d369104bc3d6c9d70e940f889d","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/fruux/sabre-dav/pulls/886/commits","review_comments_url":"https://api.github.com/repos/fruux/sabre-dav/pulls/886/comments","review_comment_url":"https://api.github.com/repos/fruux/sabre-dav/pulls/comments{/number}","comments_url":"https://api.github.com/repos/fruux/sabre-dav/issues/886/comments","statuses_url":"https://api.github.com/repos/fruux/sabre-dav/statuses/0b8d08334e6985100a15d58889b0aa42b6ae53c5","head":{"label":"petrkotek:memory-optimizations","ref":"memory-optimizations","sha":"0b8d08334e6985100a15d58889b0aa42b6ae53c5","user":{"login":"petrkotek","id":5679032,"avatar_url":"https://avatars.githubusercontent.com/u/5679032?v=3","gravatar_id":"","url":"https://api.github.com/users/petrkotek","html_url":"https://github.com/petrkotek","followers_url":"https://api.github.com/users/petrkotek/followers","following_url":"https://api.github.com/users/petrkotek/following{/other_user}","gists_url":"https://api.github.com/users/petrkotek/gists{/gist_id}","starred_url":"https://api.github.com/users/petrkotek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petrkotek/subscriptions","organizations_url":"https://api.github.com/users/petrkotek/orgs","repos_url":"https://api.github.com/users/petrkotek/repos","events_url":"https://api.github.com/users/petrkotek/events{/privacy}","received_events_url":"https://api.github.com/users/petrkotek/received_events","type":"User","site_admin":false},"repo":{"id":20008668,"name":"sabre-dav","full_name":"petrkotek/sabre-dav","owner":{"login":"petrkotek","id":5679032,"avatar_url":"https://avatars.githubusercontent.com/u/5679032?v=3","gravatar_id":"","url":"https://api.github.com/users/petrkotek","html_url":"https://github.com/petrkotek","followers_url":"https://api.github.com/users/petrkotek/followers","following_url":"https://api.github.com/users/petrkotek/following{/other_user}","gists_url":"https://api.github.com/users/petrkotek/gists{/gist_id}","starred_url":"https://api.github.com/users/petrkotek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petrkotek/subscriptions","organizations_url":"https://api.github.com/users/petrkotek/orgs","repos_url":"https://api.github.com/users/petrkotek/repos","events_url":"https://api.github.com/users/petrkotek/events{/privacy}","received_events_url":"https://api.github.com/users/petrkotek/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/petrkotek/sabre-dav","description":"SabreDAV is a WebDAV framework for PHP","fork":true,"url":"https://api.github.com/repos/petrkotek/sabre-dav","forks_url":"https://api.github.com/repos/petrkotek/sabre-dav/forks","keys_url":"https://api.github.com/repos/petrkotek/sabre-dav/keys{/key_id}","collaborators_url":"https://api.github.com/repos/petrkotek/sabre-dav/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/petrkotek/sabre-dav/teams","hooks_url":"https://api.github.com/repos/petrkotek/sabre-dav/hooks","issue_events_url":"https://api.github.com/repos/petrkotek/sabre-dav/issues/events{/number}","events_url":"https://api.github.com/repos/petrkotek/sabre-dav/events","assignees_url":"https://api.github.com/repos/petrkotek/sabre-dav/assignees{/user}","branches_url":"https://api.github.com/repos/petrkotek/sabre-dav/branches{/branch}","tags_url":"https://api.github.com/repos/petrkotek/sabre-dav/tags","blobs_url":"https://api.github.com/repos/petrkotek/sabre-dav/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/petrkotek/sabre-dav/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/petrkotek/sabre-dav/git/refs{/sha}","trees_url":"https://api.github.com/repos/petrkotek/sabre-dav/git/trees{/sha}","statuses_url":"https://api.github.com/repos/petrkotek/sabre-dav/statuses/{sha}","languages_url":"https://api.github.com/repos/petrkotek/sabre-dav/languages","stargazers_url":"https://api.github.com/repos/petrkotek/sabre-dav/stargazers","contributors_url":"https://api.github.com/repos/petrkotek/sabre-dav/contributors","subscribers_url":"https://api.github.com/repos/petrkotek/sabre-dav/subscribers","subscription_url":"https://api.github.com/repos/petrkotek/sabre-dav/subscription","commits_url":"https://api.github.com/repos/petrkotek/sabre-dav/commits{/sha}","git_commits_url":"https://api.github.com/repos/petrkotek/sabre-dav/git/commits{/sha}","comments_url":"https://api.github.com/repos/petrkotek/sabre-dav/comments{/number}","issue_comment_url":"https://api.github.com/repos/petrkotek/sabre-dav/issues/comments{/number}","contents_url":"https://api.github.com/repos/petrkotek/sabre-dav/contents/{+path}","compare_url":"https://api.github.com/repos/petrkotek/sabre-dav/compare/{base}...{head}","merges_url":"https://api.github.com/repos/petrkotek/sabre-dav/merges","archive_url":"https://api.github.com/repos/petrkotek/sabre-dav/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/petrkotek/sabre-dav/downloads","issues_url":"https://api.github.com/repos/petrkotek/sabre-dav/issues{/number}","pulls_url":"https://api.github.com/repos/petrkotek/sabre-dav/pulls{/number}","milestones_url":"https://api.github.com/repos/petrkotek/sabre-dav/milestones{/number}","notifications_url":"https://api.github.com/repos/petrkotek/sabre-dav/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/petrkotek/sabre-dav/labels{/name}","releases_url":"https://api.github.com/repos/petrkotek/sabre-dav/releases{/id}","deployments_url":"https://api.github.com/repos/petrkotek/sabre-dav/deployments","created_at":"2014-05-21T05:13:45Z","updated_at":"2014-06-20T06:41:59Z","pushed_at":"2016-09-19T19:34:58Z","git_url":"git://github.com/petrkotek/sabre-dav.git","ssh_url":"git@github.com:petrkotek/sabre-dav.git","clone_url":"https://github.com/petrkotek/sabre-dav.git","svn_url":"https://github.com/petrkotek/sabre-dav","homepage":"http://code.google.com/p/sabredav/","size":13277,"stargazers_count":0,"watchers_count":0,"language":"PHP","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"fruux:master","ref":"master","sha":"5380ab585709205220f88ce731a24cac1c7e7a29","user":{"login":"fruux","id":1384629,"avatar_url":"https://avatars.githubusercontent.com/u/1384629?v=3","gravatar_id":"","url":"https://api.github.com/users/fruux","html_url":"https://github.com/fruux","followers_url":"https://api.github.com/users/fruux/followers","following_url":"https://api.github.com/users/fruux/following{/other_user}","gists_url":"https://api.github.com/users/fruux/gists{/gist_id}","starred_url":"https://api.github.com/users/fruux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fruux/subscriptions","organizations_url":"https://api.github.com/users/fruux/orgs","repos_url":"https://api.github.com/users/fruux/repos","events_url":"https://api.github.com/users/fruux/events{/privacy}","received_events_url":"https://api.github.com/users/fruux/received_events","type":"Organization","site_admin":false},"repo":{"id":2536145,"name":"sabre-dav","full_name":"fruux/sabre-dav","owner":{"login":"fruux","id":1384629,"avatar_url":"https://avatars.githubusercontent.com/u/1384629?v=3","gravatar_id":"","url":"https://api.github.com/users/fruux","html_url":"https://github.com/fruux","followers_url":"https://api.github.com/users/fruux/followers","following_url":"https://api.github.com/users/fruux/following{/other_user}","gists_url":"https://api.github.com/users/fruux/gists{/gist_id}","starred_url":"https://api.github.com/users/fruux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fruux/subscriptions","organizations_url":"https://api.github.com/users/fruux/orgs","repos_url":"https://api.github.com/users/fruux/repos","events_url":"https://api.github.com/users/fruux/events{/privacy}","received_events_url":"https://api.github.com/users/fruux/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/fruux/sabre-dav","description":"sabre/dav is a CalDAV, CardDAV and WebDAV framework for PHP","fork":false,"url":"https://api.github.com/repos/fruux/sabre-dav","forks_url":"https://api.github.com/repos/fruux/sabre-dav/forks","keys_url":"https://api.github.com/repos/fruux/sabre-dav/keys{/key_id}","collaborators_url":"https://api.github.com/repos/fruux/sabre-dav/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/fruux/sabre-dav/teams","hooks_url":"https://api.github.com/repos/fruux/sabre-dav/hooks","issue_events_url":"https://api.github.com/repos/fruux/sabre-dav/issues/events{/number}","events_url":"https://api.github.com/repos/fruux/sabre-dav/events","assignees_url":"https://api.github.com/repos/fruux/sabre-dav/assignees{/user}","branches_url":"https://api.github.com/repos/fruux/sabre-dav/branches{/branch}","tags_url":"https://api.github.com/repos/fruux/sabre-dav/tags","blobs_url":"https://api.github.com/repos/fruux/sabre-dav/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/fruux/sabre-dav/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/fruux/sabre-dav/git/refs{/sha}","trees_url":"https://api.github.com/repos/fruux/sabre-dav/git/trees{/sha}","statuses_url":"https://api.github.com/repos/fruux/sabre-dav/statuses/{sha}","languages_url":"https://api.github.com/repos/fruux/sabre-dav/languages","stargazers_url":"https://api.github.com/repos/fruux/sabre-dav/stargazers","contributors_url":"https://api.github.com/repos/fruux/sabre-dav/contributors","subscribers_url":"https://api.github.com/repos/fruux/sabre-dav/subscribers","subscription_url":"https://api.github.com/repos/fruux/sabre-dav/subscription","commits_url":"https://api.github.com/repos/fruux/sabre-dav/commits{/sha}","git_commits_url":"https://api.github.com/repos/fruux/sabre-dav/git/commits{/sha}","comments_url":"https://api.github.com/repos/fruux/sabre-dav/comments{/number}","issue_comment_url":"https://api.github.com/repos/fruux/sabre-dav/issues/comments{/number}","contents_url":"https://api.github.com/repos/fruux/sabre-dav/contents/{+path}","compare_url":"https://api.github.com/repos/fruux/sabre-dav/compare/{base}...{head}","merges_url":"https://api.github.com/repos/fruux/sabre-dav/merges","archive_url":"https://api.github.com/repos/fruux/sabre-dav/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/fruux/sabre-dav/downloads","issues_url":"https://api.github.com/repos/fruux/sabre-dav/issues{/number}","pulls_url":"https://api.github.com/repos/fruux/sabre-dav/pulls{/number}","milestones_url":"https://api.github.com/repos/fruux/sabre-dav/milestones{/number}","notifications_url":"https://api.github.com/repos/fruux/sabre-dav/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/fruux/sabre-dav/labels{/name}","releases_url":"https://api.github.com/repos/fruux/sabre-dav/releases{/id}","deployments_url":"https://api.github.com/repos/fruux/sabre-dav/deployments","created_at":"2011-10-08T00:38:42Z","updated_at":"2016-09-19T05:26:10Z","pushed_at":"2016-09-19T19:34:58Z","git_url":"git://github.com/fruux/sabre-dav.git","ssh_url":"git@github.com:fruux/sabre-dav.git","clone_url":"https://github.com/fruux/sabre-dav.git","svn_url":"https://github.com/fruux/sabre-dav","homepage":"http://sabre.io","size":22233,"stargazers_count":680,"watchers_count":680,"language":"PHP","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":189,"mirror_url":null,"open_issues_count":43,"forks":189,"open_issues":43,"watchers":680,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/fruux/sabre-dav/pulls/886"},"html":{"href":"https://github.com/fruux/sabre-dav/pull/886"},"issue":{"href":"https://api.github.com/repos/fruux/sabre-dav/issues/886"},"comments":{"href":"https://api.github.com/repos/fruux/sabre-dav/issues/886/comments"},"review_comments":{"href":"https://api.github.com/repos/fruux/sabre-dav/pulls/886/comments"},"review_comment":{"href":"https://api.github.com/repos/fruux/sabre-dav/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/fruux/sabre-dav/pulls/886/commits"},"statuses":{"href":"https://api.github.com/repos/fruux/sabre-dav/statuses/0b8d08334e6985100a15d58889b0aa42b6ae53c5"}}}}
{"actor":{"display_login":"staabm"}}