{"action":"created","comment":{"url":"https://api.github.com/repos/prestodb/presto/pulls/comments/68318442","id":68318442,"diff_hunk":"@@ -191,6 +188,194 @@ This table can then be queried in Presto::\n \n     SELECT * FROM hive.web.page_view;\n \n+\n+.. _tuning-pref-hive:\n+\n+Tuning\n+-------\n+\n+The following configuration properties may have an impact on connector performance:\n+\n+``hive.assume-canonical-partition-keys``\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+ * **Type:** ``Boolean``\n+ * **Default value:** ``false``\n+ * **Description:** Disable optimized metastore partition fetching for non-string partition keys. Setting this property allows to avoid ignoring data with non-canonical partition values.\n+\n+\n+``hive.domain-compaction-threshold``\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+ * **Type:** ``Integer`` (at least ``1``)\n+ * **Default value:** ``100``\n+ * **Description:** Maximum number of ranges allowed in a tuple domain without compacting it. Higher value will cause more data fragmentation but allows to use row skipping feature when reading ORC data. Setting this value higher may have large impact on ``IN`` and ``OR`` clauses performance in scenarios making use of row skipping.\n+\n+\n+.. _force-local-scheduling:\n+\n+``hive.force-local-scheduling``\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+ * **Type:** ``Boolean``\n+ * **Default value:** ``false``\n+ * **Description:** Force splits to be scheduled on the same node (ignoring normal node selection procedures) as the Hadoop DataNode process serving the split data. This is useful for installations where Presto is collocated with every DataNode and may increase queries time significantly. The drawback may be that if some data are accessed more often, the utilization of some nodes may be low even if the whole system is heavy loaded. See also :ref:`node-scheduler.network-topology<node-scheduler-network-topology>` if less strict constrain is preferred - especially if some nodes are overloaded and other are not fully utilized.\n+\n+\n+``hive.max-initial-split-size``\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+ * **Type:** ``String`` (data size)\n+ * **Default value:** ``hive.max-split-size`` / ``2`` (``32 MB``)\n+ * **Description:** This property describes max size of each of initially created splits for a single query. The logic of initial splits is described in ``hive.max-initial-splits`` property. Changing this value changes what is considered small query. Higher value causes smaller parallelism for small queries. Lower value increases concurrency for them. This is max size, as the real size may be lower when end of blocks in single DataNode is reached.\n+\n+\n+``hive.max-initial-splits``\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+ * **Type:** ``Integer``\n+ * **Default value:** ``200``\n+ * **Description:** This property describes how many splits may be initially created for a single query. The initial splits are created to allow better concurrency for small queries. Hive connector will create first ``hive.max-initial-splits`` splits with size of ``hive.max-initial-split-size`` instead of ``hive.max-split-size``. Having this value higher will force more splits to have smaller size effectively increasing definition of what is considered small query in database.\n+\n+\n+``hive.max-outstanding-splits``\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+ * **Type:** ``Integer`` (at least ``1``)\n+ * **Default value:** ``1000``\n+ * **Description:** Limit of number of splits waiting to be served by split source. After reaching this limit writers will stop writing new splits to split source until some of them are used by workers. Higher value will increase memory usage, but will allow to concentrate all IO at one time which may be much faster and increase resources utilization.\n+\n+\n+``hive.max-partitions-per-writers``\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+ * **Type:** ``Integer`` (at least ``1``)\n+ * **Default value:** ``100``\n+ * **Description:** Maximum number of partitions per writer. If higher number of partitions per writer will be required to complete query, the query will fail. By manipulating this value one may change how large queries are meant to be dropped from DB which may help with error detection.\n+\n+\n+``hive.max-split-iterator-threads``\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+ * **Type:** ``Integer`` (at least ``1``)\n+ * **Default value:** ``1000``\n+ * **Description:** This property describes how many threads may be used to iterate through splits when loading them to the worker nodes. Higher value may increase parallelism, but high concurrency may cause time being wasted on context switching.","path":"presto-docs/src/main/sphinx/connector/hive.rst","position":87,"original_position":87,"commit_id":"27cda7c874af77bc9f214cfd27a54acc11b4beae","original_commit_id":"27cda7c874af77bc9f214cfd27a54acc11b4beae","user":{"login":"rschlussel","id":10963789,"avatar_url":"https://avatars.githubusercontent.com/u/10963789?v=3","gravatar_id":"","url":"https://api.github.com/users/rschlussel","html_url":"https://github.com/rschlussel","followers_url":"https://api.github.com/users/rschlussel/followers","following_url":"https://api.github.com/users/rschlussel/following{/other_user}","gists_url":"https://api.github.com/users/rschlussel/gists{/gist_id}","starred_url":"https://api.github.com/users/rschlussel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rschlussel/subscriptions","organizations_url":"https://api.github.com/users/rschlussel/orgs","repos_url":"https://api.github.com/users/rschlussel/repos","events_url":"https://api.github.com/users/rschlussel/events{/privacy}","received_events_url":"https://api.github.com/users/rschlussel/received_events","type":"User","site_admin":false},"body":"... A higher value may increase parallelism, but increased concurrency may cause too much time to be spent on context switching.","created_at":"2016-06-23T21:28:01Z","updated_at":"2016-06-23T21:28:01Z","html_url":"https://github.com/prestodb/presto/pull/4987#discussion_r68318442","pull_request_url":"https://api.github.com/repos/prestodb/presto/pulls/4987","_links":{"self":{"href":"https://api.github.com/repos/prestodb/presto/pulls/comments/68318442"},"html":{"href":"https://github.com/prestodb/presto/pull/4987#discussion_r68318442"},"pull_request":{"href":"https://api.github.com/repos/prestodb/presto/pulls/4987"}}},"pull_request":{"url":"https://api.github.com/repos/prestodb/presto/pulls/4987","id":66011713,"html_url":"https://github.com/prestodb/presto/pull/4987","diff_url":"https://github.com/prestodb/presto/pull/4987.diff","patch_url":"https://github.com/prestodb/presto/pull/4987.patch","issue_url":"https://api.github.com/repos/prestodb/presto/issues/4987","number":4987,"state":"open","locked":false,"title":"Add basic tuning information using properties to presto-doc.","user":{"login":"fiedukow","id":297223,"avatar_url":"https://avatars.githubusercontent.com/u/297223?v=3","gravatar_id":"","url":"https://api.github.com/users/fiedukow","html_url":"https://github.com/fiedukow","followers_url":"https://api.github.com/users/fiedukow/followers","following_url":"https://api.github.com/users/fiedukow/following{/other_user}","gists_url":"https://api.github.com/users/fiedukow/gists{/gist_id}","starred_url":"https://api.github.com/users/fiedukow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fiedukow/subscriptions","organizations_url":"https://api.github.com/users/fiedukow/orgs","repos_url":"https://api.github.com/users/fiedukow/repos","events_url":"https://api.github.com/users/fiedukow/events{/privacy}","received_events_url":"https://api.github.com/users/fiedukow/received_events","type":"User","site_admin":false},"body":"This change in documentation covers basic tuning using presto properties,\r\nsession properties and hive connector properties.\r\n\r\nThis is meant as the base for further development and basic information\r\nfor users about properties existence and usage.","created_at":"2016-04-11T15:19:25Z","updated_at":"2016-06-23T21:28:01Z","closed_at":null,"merged_at":null,"merge_commit_sha":"879b2c39dfd6c60e1d69aaa5d429222961d40fee","assignee":{"login":"fiedukow","id":297223,"avatar_url":"https://avatars.githubusercontent.com/u/297223?v=3","gravatar_id":"","url":"https://api.github.com/users/fiedukow","html_url":"https://github.com/fiedukow","followers_url":"https://api.github.com/users/fiedukow/followers","following_url":"https://api.github.com/users/fiedukow/following{/other_user}","gists_url":"https://api.github.com/users/fiedukow/gists{/gist_id}","starred_url":"https://api.github.com/users/fiedukow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fiedukow/subscriptions","organizations_url":"https://api.github.com/users/fiedukow/orgs","repos_url":"https://api.github.com/users/fiedukow/repos","events_url":"https://api.github.com/users/fiedukow/events{/privacy}","received_events_url":"https://api.github.com/users/fiedukow/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/prestodb/presto/pulls/4987/commits","review_comments_url":"https://api.github.com/repos/prestodb/presto/pulls/4987/comments","review_comment_url":"https://api.github.com/repos/prestodb/presto/pulls/comments{/number}","comments_url":"https://api.github.com/repos/prestodb/presto/issues/4987/comments","statuses_url":"https://api.github.com/repos/prestodb/presto/statuses/27cda7c874af77bc9f214cfd27a54acc11b4beae","head":{"label":"Teradata:tuning_docummentation","ref":"tuning_docummentation","sha":"27cda7c874af77bc9f214cfd27a54acc11b4beae","user":{"login":"Teradata","id":1615979,"avatar_url":"https://avatars.githubusercontent.com/u/1615979?v=3","gravatar_id":"","url":"https://api.github.com/users/Teradata","html_url":"https://github.com/Teradata","followers_url":"https://api.github.com/users/Teradata/followers","following_url":"https://api.github.com/users/Teradata/following{/other_user}","gists_url":"https://api.github.com/users/Teradata/gists{/gist_id}","starred_url":"https://api.github.com/users/Teradata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Teradata/subscriptions","organizations_url":"https://api.github.com/users/Teradata/orgs","repos_url":"https://api.github.com/users/Teradata/repos","events_url":"https://api.github.com/users/Teradata/events{/privacy}","received_events_url":"https://api.github.com/users/Teradata/received_events","type":"Organization","site_admin":false},"repo":{"id":36903368,"name":"presto","full_name":"Teradata/presto","owner":{"login":"Teradata","id":1615979,"avatar_url":"https://avatars.githubusercontent.com/u/1615979?v=3","gravatar_id":"","url":"https://api.github.com/users/Teradata","html_url":"https://github.com/Teradata","followers_url":"https://api.github.com/users/Teradata/followers","following_url":"https://api.github.com/users/Teradata/following{/other_user}","gists_url":"https://api.github.com/users/Teradata/gists{/gist_id}","starred_url":"https://api.github.com/users/Teradata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Teradata/subscriptions","organizations_url":"https://api.github.com/users/Teradata/orgs","repos_url":"https://api.github.com/users/Teradata/repos","events_url":"https://api.github.com/users/Teradata/events{/privacy}","received_events_url":"https://api.github.com/users/Teradata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Teradata/presto","description":"Teradata's Certified Version of Presto -- A Distributed SQL Query Engine for Big Data","fork":true,"url":"https://api.github.com/repos/Teradata/presto","forks_url":"https://api.github.com/repos/Teradata/presto/forks","keys_url":"https://api.github.com/repos/Teradata/presto/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Teradata/presto/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Teradata/presto/teams","hooks_url":"https://api.github.com/repos/Teradata/presto/hooks","issue_events_url":"https://api.github.com/repos/Teradata/presto/issues/events{/number}","events_url":"https://api.github.com/repos/Teradata/presto/events","assignees_url":"https://api.github.com/repos/Teradata/presto/assignees{/user}","branches_url":"https://api.github.com/repos/Teradata/presto/branches{/branch}","tags_url":"https://api.github.com/repos/Teradata/presto/tags","blobs_url":"https://api.github.com/repos/Teradata/presto/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Teradata/presto/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Teradata/presto/git/refs{/sha}","trees_url":"https://api.github.com/repos/Teradata/presto/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Teradata/presto/statuses/{sha}","languages_url":"https://api.github.com/repos/Teradata/presto/languages","stargazers_url":"https://api.github.com/repos/Teradata/presto/stargazers","contributors_url":"https://api.github.com/repos/Teradata/presto/contributors","subscribers_url":"https://api.github.com/repos/Teradata/presto/subscribers","subscription_url":"https://api.github.com/repos/Teradata/presto/subscription","commits_url":"https://api.github.com/repos/Teradata/presto/commits{/sha}","git_commits_url":"https://api.github.com/repos/Teradata/presto/git/commits{/sha}","comments_url":"https://api.github.com/repos/Teradata/presto/comments{/number}","issue_comment_url":"https://api.github.com/repos/Teradata/presto/issues/comments{/number}","contents_url":"https://api.github.com/repos/Teradata/presto/contents/{+path}","compare_url":"https://api.github.com/repos/Teradata/presto/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Teradata/presto/merges","archive_url":"https://api.github.com/repos/Teradata/presto/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Teradata/presto/downloads","issues_url":"https://api.github.com/repos/Teradata/presto/issues{/number}","pulls_url":"https://api.github.com/repos/Teradata/presto/pulls{/number}","milestones_url":"https://api.github.com/repos/Teradata/presto/milestones{/number}","notifications_url":"https://api.github.com/repos/Teradata/presto/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Teradata/presto/labels{/name}","releases_url":"https://api.github.com/repos/Teradata/presto/releases{/id}","deployments_url":"https://api.github.com/repos/Teradata/presto/deployments","created_at":"2015-06-05T00:38:37Z","updated_at":"2016-06-14T07:29:49Z","pushed_at":"2016-06-23T20:46:12Z","git_url":"git://github.com/Teradata/presto.git","ssh_url":"git@github.com:Teradata/presto.git","clone_url":"https://github.com/Teradata/presto.git","svn_url":"https://github.com/Teradata/presto","homepage":"http://www.teradata.com/presto","size":143901,"stargazers_count":51,"watchers_count":51,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":8,"mirror_url":null,"open_issues_count":13,"forks":8,"open_issues":13,"watchers":51,"default_branch":"master"}},"base":{"label":"prestodb:master","ref":"master","sha":"0f96a7e18f1e8eb654a55bdf15da4b8a1638d746","user":{"login":"prestodb","id":6882181,"avatar_url":"https://avatars.githubusercontent.com/u/6882181?v=3","gravatar_id":"","url":"https://api.github.com/users/prestodb","html_url":"https://github.com/prestodb","followers_url":"https://api.github.com/users/prestodb/followers","following_url":"https://api.github.com/users/prestodb/following{/other_user}","gists_url":"https://api.github.com/users/prestodb/gists{/gist_id}","starred_url":"https://api.github.com/users/prestodb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prestodb/subscriptions","organizations_url":"https://api.github.com/users/prestodb/orgs","repos_url":"https://api.github.com/users/prestodb/repos","events_url":"https://api.github.com/users/prestodb/events{/privacy}","received_events_url":"https://api.github.com/users/prestodb/received_events","type":"Organization","site_admin":false},"repo":{"id":5349565,"name":"presto","full_name":"prestodb/presto","owner":{"login":"prestodb","id":6882181,"avatar_url":"https://avatars.githubusercontent.com/u/6882181?v=3","gravatar_id":"","url":"https://api.github.com/users/prestodb","html_url":"https://github.com/prestodb","followers_url":"https://api.github.com/users/prestodb/followers","following_url":"https://api.github.com/users/prestodb/following{/other_user}","gists_url":"https://api.github.com/users/prestodb/gists{/gist_id}","starred_url":"https://api.github.com/users/prestodb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prestodb/subscriptions","organizations_url":"https://api.github.com/users/prestodb/orgs","repos_url":"https://api.github.com/users/prestodb/repos","events_url":"https://api.github.com/users/prestodb/events{/privacy}","received_events_url":"https://api.github.com/users/prestodb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/prestodb/presto","description":"Distributed SQL query engine for big data","fork":false,"url":"https://api.github.com/repos/prestodb/presto","forks_url":"https://api.github.com/repos/prestodb/presto/forks","keys_url":"https://api.github.com/repos/prestodb/presto/keys{/key_id}","collaborators_url":"https://api.github.com/repos/prestodb/presto/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/prestodb/presto/teams","hooks_url":"https://api.github.com/repos/prestodb/presto/hooks","issue_events_url":"https://api.github.com/repos/prestodb/presto/issues/events{/number}","events_url":"https://api.github.com/repos/prestodb/presto/events","assignees_url":"https://api.github.com/repos/prestodb/presto/assignees{/user}","branches_url":"https://api.github.com/repos/prestodb/presto/branches{/branch}","tags_url":"https://api.github.com/repos/prestodb/presto/tags","blobs_url":"https://api.github.com/repos/prestodb/presto/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/prestodb/presto/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/prestodb/presto/git/refs{/sha}","trees_url":"https://api.github.com/repos/prestodb/presto/git/trees{/sha}","statuses_url":"https://api.github.com/repos/prestodb/presto/statuses/{sha}","languages_url":"https://api.github.com/repos/prestodb/presto/languages","stargazers_url":"https://api.github.com/repos/prestodb/presto/stargazers","contributors_url":"https://api.github.com/repos/prestodb/presto/contributors","subscribers_url":"https://api.github.com/repos/prestodb/presto/subscribers","subscription_url":"https://api.github.com/repos/prestodb/presto/subscription","commits_url":"https://api.github.com/repos/prestodb/presto/commits{/sha}","git_commits_url":"https://api.github.com/repos/prestodb/presto/git/commits{/sha}","comments_url":"https://api.github.com/repos/prestodb/presto/comments{/number}","issue_comment_url":"https://api.github.com/repos/prestodb/presto/issues/comments{/number}","contents_url":"https://api.github.com/repos/prestodb/presto/contents/{+path}","compare_url":"https://api.github.com/repos/prestodb/presto/compare/{base}...{head}","merges_url":"https://api.github.com/repos/prestodb/presto/merges","archive_url":"https://api.github.com/repos/prestodb/presto/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/prestodb/presto/downloads","issues_url":"https://api.github.com/repos/prestodb/presto/issues{/number}","pulls_url":"https://api.github.com/repos/prestodb/presto/pulls{/number}","milestones_url":"https://api.github.com/repos/prestodb/presto/milestones{/number}","notifications_url":"https://api.github.com/repos/prestodb/presto/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/prestodb/presto/labels{/name}","releases_url":"https://api.github.com/repos/prestodb/presto/releases{/id}","deployments_url":"https://api.github.com/repos/prestodb/presto/deployments","created_at":"2012-08-09T01:03:37Z","updated_at":"2016-06-23T10:31:34Z","pushed_at":"2016-06-23T20:29:16Z","git_url":"git://github.com/prestodb/presto.git","ssh_url":"git@github.com:prestodb/presto.git","clone_url":"https://github.com/prestodb/presto.git","svn_url":"https://github.com/prestodb/presto","homepage":"https://prestodb.io","size":155621,"stargazers_count":4847,"watchers_count":4847,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1450,"mirror_url":null,"open_issues_count":553,"forks":1450,"open_issues":553,"watchers":4847,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/prestodb/presto/pulls/4987"},"html":{"href":"https://github.com/prestodb/presto/pull/4987"},"issue":{"href":"https://api.github.com/repos/prestodb/presto/issues/4987"},"comments":{"href":"https://api.github.com/repos/prestodb/presto/issues/4987/comments"},"review_comments":{"href":"https://api.github.com/repos/prestodb/presto/pulls/4987/comments"},"review_comment":{"href":"https://api.github.com/repos/prestodb/presto/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/prestodb/presto/pulls/4987/commits"},"statuses":{"href":"https://api.github.com/repos/prestodb/presto/statuses/27cda7c874af77bc9f214cfd27a54acc11b4beae"}}}}
{"actor":{"display_login":"rschlussel"}}