{"action":"created","comment":{"url":"https://api.github.com/repos/openzfs/openzfs/pulls/comments/93293058","pull_request_review_id":13817530,"id":93293058,"diff_hunk":"@@ -739,9 +758,29 @@ dmu_free_long_range_impl(objset_t *os, dnode_t *dn, uint64_t offset,\n \t\tASSERT3U(chunk_begin, >=, offset);\n \t\tASSERT3U(chunk_begin, <=, chunk_end);\n \n-\t\tdmu_tx_t *tx = dmu_tx_create(os);\n-\t\tdmu_tx_hold_free(tx, dn->dn_object,\n-\t\t    chunk_begin, chunk_end - chunk_begin);\n+\t\tchunk_len = chunk_end - chunk_begin;\n+\n+\t\tmutex_enter(&dp->dp_lock);\n+\t\tfor (int t = 0; t < TXG_SIZE; t++) {\n+\t\t\tlong_free_dirty_all_txgs +=\n+\t\t\t    dp->dp_long_free_dirty_pertxg[t];\n+\t\t}\n+\t\tmutex_exit(&dp->dp_lock);\n+\t\tfree_dirty_pct = 100 * long_free_dirty_all_txgs /","path":"usr/src/uts/common/fs/zfs/dmu.c","position":58,"original_position":58,"commit_id":"5328a1faacc34065a43809b891964e06068d48ea","original_commit_id":"5328a1faacc34065a43809b891964e06068d48ea","user":{"login":"pzakha","id":2942705,"avatar_url":"https://avatars.githubusercontent.com/u/2942705?v=3","gravatar_id":"","url":"https://api.github.com/users/pzakha","html_url":"https://github.com/pzakha","followers_url":"https://api.github.com/users/pzakha/followers","following_url":"https://api.github.com/users/pzakha/following{/other_user}","gists_url":"https://api.github.com/users/pzakha/gists{/gist_id}","starred_url":"https://api.github.com/users/pzakha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pzakha/subscriptions","organizations_url":"https://api.github.com/users/pzakha/orgs","repos_url":"https://api.github.com/users/pzakha/repos","events_url":"https://api.github.com/users/pzakha/events{/privacy}","received_events_url":"https://api.github.com/users/pzakha/received_events","type":"User","site_admin":false},"body":"I think it would be better to calculate `dirty_frees_threshold` in bytes rather than as a percentage. This way we do not need to do any divisions in the loop and can compare `long_free_dirty_all_txgs` directly with `dirty_frees_threshold`.","created_at":"2016-12-20T18:10:53Z","updated_at":"2016-12-20T18:15:49Z","html_url":"https://github.com/openzfs/openzfs/pull/214#discussion_r93293058","pull_request_url":"https://api.github.com/repos/openzfs/openzfs/pulls/214","_links":{"self":{"href":"https://api.github.com/repos/openzfs/openzfs/pulls/comments/93293058"},"html":{"href":"https://github.com/openzfs/openzfs/pull/214#discussion_r93293058"},"pull_request":{"href":"https://api.github.com/repos/openzfs/openzfs/pulls/214"}}},"pull_request":{"url":"https://api.github.com/repos/openzfs/openzfs/pulls/214","id":90884453,"html_url":"https://github.com/openzfs/openzfs/pull/214","diff_url":"https://github.com/openzfs/openzfs/pull/214.diff","patch_url":"https://github.com/openzfs/openzfs/pull/214.patch","issue_url":"https://api.github.com/repos/openzfs/openzfs/issues/214","number":214,"state":"open","locked":false,"title":"6569 large file delete can starve out write ops","user":{"login":"alek-p","id":2172305,"avatar_url":"https://avatars.githubusercontent.com/u/2172305?v=3","gravatar_id":"","url":"https://api.github.com/users/alek-p","html_url":"https://github.com/alek-p","followers_url":"https://api.github.com/users/alek-p/followers","following_url":"https://api.github.com/users/alek-p/following{/other_user}","gists_url":"https://api.github.com/users/alek-p/gists{/gist_id}","starred_url":"https://api.github.com/users/alek-p/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alek-p/subscriptions","organizations_url":"https://api.github.com/users/alek-p/orgs","repos_url":"https://api.github.com/users/alek-p/repos","events_url":"https://api.github.com/users/alek-p/events{/privacy}","received_events_url":"https://api.github.com/users/alek-p/received_events","type":"User","site_admin":false},"body":"The core issue I've found is that there is no throttle for how many deletes get assigned to one TXG. As a results when deleting large file(s) we end up filling consecutive TXGs with deletes/frees, then write throttling other (more important) write ops.\n\nThere is an easy test case for this problem. Try deleting several large files (at least 1/2 TB) while you do write ops on the same pool. What we've seen is performance of these write ops (let's call it sideload I/O) would drop to zero.\n\nMore specifically the problem is that dmu_free_long_range_impl() can/will fill up all of the dirty data in the pool \"instantly\", before many of the sideload ops can get in. So sideload performance will be impacted until all the files are freed.\nThe solution we have tested at Nexenta (with positive results) creates a relatively simple throttle for how many \"free\" ops we let into one TXG.\n","created_at":"2016-10-25T20:24:25Z","updated_at":"2016-12-20T18:15:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7040dd1807d2aecd158cd3e214a1caf702c4b631","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/openzfs/openzfs/pulls/214/commits","review_comments_url":"https://api.github.com/repos/openzfs/openzfs/pulls/214/comments","review_comment_url":"https://api.github.com/repos/openzfs/openzfs/pulls/comments{/number}","comments_url":"https://api.github.com/repos/openzfs/openzfs/issues/214/comments","statuses_url":"https://api.github.com/repos/openzfs/openzfs/statuses/5328a1faacc34065a43809b891964e06068d48ea","head":{"label":"alek-p:6569","ref":"6569","sha":"5328a1faacc34065a43809b891964e06068d48ea","user":{"login":"alek-p","id":2172305,"avatar_url":"https://avatars.githubusercontent.com/u/2172305?v=3","gravatar_id":"","url":"https://api.github.com/users/alek-p","html_url":"https://github.com/alek-p","followers_url":"https://api.github.com/users/alek-p/followers","following_url":"https://api.github.com/users/alek-p/following{/other_user}","gists_url":"https://api.github.com/users/alek-p/gists{/gist_id}","starred_url":"https://api.github.com/users/alek-p/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alek-p/subscriptions","organizations_url":"https://api.github.com/users/alek-p/orgs","repos_url":"https://api.github.com/users/alek-p/repos","events_url":"https://api.github.com/users/alek-p/events{/privacy}","received_events_url":"https://api.github.com/users/alek-p/received_events","type":"User","site_admin":false},"repo":{"id":71934835,"name":"openzfs","full_name":"alek-p/openzfs","owner":{"login":"alek-p","id":2172305,"avatar_url":"https://avatars.githubusercontent.com/u/2172305?v=3","gravatar_id":"","url":"https://api.github.com/users/alek-p","html_url":"https://github.com/alek-p","followers_url":"https://api.github.com/users/alek-p/followers","following_url":"https://api.github.com/users/alek-p/following{/other_user}","gists_url":"https://api.github.com/users/alek-p/gists{/gist_id}","starred_url":"https://api.github.com/users/alek-p/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alek-p/subscriptions","organizations_url":"https://api.github.com/users/alek-p/orgs","repos_url":"https://api.github.com/users/alek-p/repos","events_url":"https://api.github.com/users/alek-p/events{/privacy}","received_events_url":"https://api.github.com/users/alek-p/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/alek-p/openzfs","description":"OpenZFS is a storage platform that encompasses the functionality of traditional filesystems and volume managers, delivering enterprise reliability, modern functionality, and consistent performance in an easy to administer package on several operating system platforms.","fork":true,"url":"https://api.github.com/repos/alek-p/openzfs","forks_url":"https://api.github.com/repos/alek-p/openzfs/forks","keys_url":"https://api.github.com/repos/alek-p/openzfs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alek-p/openzfs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alek-p/openzfs/teams","hooks_url":"https://api.github.com/repos/alek-p/openzfs/hooks","issue_events_url":"https://api.github.com/repos/alek-p/openzfs/issues/events{/number}","events_url":"https://api.github.com/repos/alek-p/openzfs/events","assignees_url":"https://api.github.com/repos/alek-p/openzfs/assignees{/user}","branches_url":"https://api.github.com/repos/alek-p/openzfs/branches{/branch}","tags_url":"https://api.github.com/repos/alek-p/openzfs/tags","blobs_url":"https://api.github.com/repos/alek-p/openzfs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alek-p/openzfs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alek-p/openzfs/git/refs{/sha}","trees_url":"https://api.github.com/repos/alek-p/openzfs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alek-p/openzfs/statuses/{sha}","languages_url":"https://api.github.com/repos/alek-p/openzfs/languages","stargazers_url":"https://api.github.com/repos/alek-p/openzfs/stargazers","contributors_url":"https://api.github.com/repos/alek-p/openzfs/contributors","subscribers_url":"https://api.github.com/repos/alek-p/openzfs/subscribers","subscription_url":"https://api.github.com/repos/alek-p/openzfs/subscription","commits_url":"https://api.github.com/repos/alek-p/openzfs/commits{/sha}","git_commits_url":"https://api.github.com/repos/alek-p/openzfs/git/commits{/sha}","comments_url":"https://api.github.com/repos/alek-p/openzfs/comments{/number}","issue_comment_url":"https://api.github.com/repos/alek-p/openzfs/issues/comments{/number}","contents_url":"https://api.github.com/repos/alek-p/openzfs/contents/{+path}","compare_url":"https://api.github.com/repos/alek-p/openzfs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alek-p/openzfs/merges","archive_url":"https://api.github.com/repos/alek-p/openzfs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alek-p/openzfs/downloads","issues_url":"https://api.github.com/repos/alek-p/openzfs/issues{/number}","pulls_url":"https://api.github.com/repos/alek-p/openzfs/pulls{/number}","milestones_url":"https://api.github.com/repos/alek-p/openzfs/milestones{/number}","notifications_url":"https://api.github.com/repos/alek-p/openzfs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alek-p/openzfs/labels{/name}","releases_url":"https://api.github.com/repos/alek-p/openzfs/releases{/id}","deployments_url":"https://api.github.com/repos/alek-p/openzfs/deployments","created_at":"2016-10-25T20:10:57Z","updated_at":"2016-10-25T20:14:01Z","pushed_at":"2016-12-01T23:36:11Z","git_url":"git://github.com/alek-p/openzfs.git","ssh_url":"git@github.com:alek-p/openzfs.git","clone_url":"https://github.com/alek-p/openzfs.git","svn_url":"https://github.com/alek-p/openzfs","homepage":"http://open-zfs.org","size":262270,"stargazers_count":0,"watchers_count":0,"language":"C","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"openzfs:master","ref":"master","sha":"3105d95625b6e58e9d6ad7220b9ea056cd08defc","user":{"login":"openzfs","id":4510897,"avatar_url":"https://avatars.githubusercontent.com/u/4510897?v=3","gravatar_id":"","url":"https://api.github.com/users/openzfs","html_url":"https://github.com/openzfs","followers_url":"https://api.github.com/users/openzfs/followers","following_url":"https://api.github.com/users/openzfs/following{/other_user}","gists_url":"https://api.github.com/users/openzfs/gists{/gist_id}","starred_url":"https://api.github.com/users/openzfs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/openzfs/subscriptions","organizations_url":"https://api.github.com/users/openzfs/orgs","repos_url":"https://api.github.com/users/openzfs/repos","events_url":"https://api.github.com/users/openzfs/events{/privacy}","received_events_url":"https://api.github.com/users/openzfs/received_events","type":"Organization","site_admin":false},"repo":{"id":44220580,"name":"openzfs","full_name":"openzfs/openzfs","owner":{"login":"openzfs","id":4510897,"avatar_url":"https://avatars.githubusercontent.com/u/4510897?v=3","gravatar_id":"","url":"https://api.github.com/users/openzfs","html_url":"https://github.com/openzfs","followers_url":"https://api.github.com/users/openzfs/followers","following_url":"https://api.github.com/users/openzfs/following{/other_user}","gists_url":"https://api.github.com/users/openzfs/gists{/gist_id}","starred_url":"https://api.github.com/users/openzfs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/openzfs/subscriptions","organizations_url":"https://api.github.com/users/openzfs/orgs","repos_url":"https://api.github.com/users/openzfs/repos","events_url":"https://api.github.com/users/openzfs/events{/privacy}","received_events_url":"https://api.github.com/users/openzfs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/openzfs/openzfs","description":"OpenZFS is a storage platform that encompasses the functionality of traditional filesystems and volume managers, delivering enterprise reliability, modern functionality, and consistent performance in an easy to administer package on several operating system platforms.","fork":true,"url":"https://api.github.com/repos/openzfs/openzfs","forks_url":"https://api.github.com/repos/openzfs/openzfs/forks","keys_url":"https://api.github.com/repos/openzfs/openzfs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/openzfs/openzfs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/openzfs/openzfs/teams","hooks_url":"https://api.github.com/repos/openzfs/openzfs/hooks","issue_events_url":"https://api.github.com/repos/openzfs/openzfs/issues/events{/number}","events_url":"https://api.github.com/repos/openzfs/openzfs/events","assignees_url":"https://api.github.com/repos/openzfs/openzfs/assignees{/user}","branches_url":"https://api.github.com/repos/openzfs/openzfs/branches{/branch}","tags_url":"https://api.github.com/repos/openzfs/openzfs/tags","blobs_url":"https://api.github.com/repos/openzfs/openzfs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/openzfs/openzfs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/openzfs/openzfs/git/refs{/sha}","trees_url":"https://api.github.com/repos/openzfs/openzfs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/openzfs/openzfs/statuses/{sha}","languages_url":"https://api.github.com/repos/openzfs/openzfs/languages","stargazers_url":"https://api.github.com/repos/openzfs/openzfs/stargazers","contributors_url":"https://api.github.com/repos/openzfs/openzfs/contributors","subscribers_url":"https://api.github.com/repos/openzfs/openzfs/subscribers","subscription_url":"https://api.github.com/repos/openzfs/openzfs/subscription","commits_url":"https://api.github.com/repos/openzfs/openzfs/commits{/sha}","git_commits_url":"https://api.github.com/repos/openzfs/openzfs/git/commits{/sha}","comments_url":"https://api.github.com/repos/openzfs/openzfs/comments{/number}","issue_comment_url":"https://api.github.com/repos/openzfs/openzfs/issues/comments{/number}","contents_url":"https://api.github.com/repos/openzfs/openzfs/contents/{+path}","compare_url":"https://api.github.com/repos/openzfs/openzfs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/openzfs/openzfs/merges","archive_url":"https://api.github.com/repos/openzfs/openzfs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/openzfs/openzfs/downloads","issues_url":"https://api.github.com/repos/openzfs/openzfs/issues{/number}","pulls_url":"https://api.github.com/repos/openzfs/openzfs/pulls{/number}","milestones_url":"https://api.github.com/repos/openzfs/openzfs/milestones{/number}","notifications_url":"https://api.github.com/repos/openzfs/openzfs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/openzfs/openzfs/labels{/name}","releases_url":"https://api.github.com/repos/openzfs/openzfs/releases{/id}","deployments_url":"https://api.github.com/repos/openzfs/openzfs/deployments","created_at":"2015-10-14T03:03:50Z","updated_at":"2016-12-08T17:57:31Z","pushed_at":"2016-12-20T17:43:13Z","git_url":"git://github.com/openzfs/openzfs.git","ssh_url":"git@github.com:openzfs/openzfs.git","clone_url":"https://github.com/openzfs/openzfs.git","svn_url":"https://github.com/openzfs/openzfs","homepage":"http://open-zfs.org","size":267098,"stargazers_count":106,"watchers_count":106,"language":"C","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":37,"mirror_url":null,"open_issues_count":40,"forks":37,"open_issues":40,"watchers":106,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/openzfs/openzfs/pulls/214"},"html":{"href":"https://github.com/openzfs/openzfs/pull/214"},"issue":{"href":"https://api.github.com/repos/openzfs/openzfs/issues/214"},"comments":{"href":"https://api.github.com/repos/openzfs/openzfs/issues/214/comments"},"review_comments":{"href":"https://api.github.com/repos/openzfs/openzfs/pulls/214/comments"},"review_comment":{"href":"https://api.github.com/repos/openzfs/openzfs/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/openzfs/openzfs/pulls/214/commits"},"statuses":{"href":"https://api.github.com/repos/openzfs/openzfs/statuses/5328a1faacc34065a43809b891964e06068d48ea"}}}}
{"actor":{"display_login":"pzakha"}}