{"action":"created","comment":{"url":"https://api.github.com/repos/apache/nifi/pulls/comments/166294484","pull_request_review_id":94321390,"id":166294484,"diff_hunk":"@@ -249,34 +260,56 @@ public void setup(final ProcessContext context, boolean shouldCleanCache, FlowFi\n                 return;\n             }\n \n-            // Try to fill the columnTypeMap with the types of the desired max-value columns\n-            final DBCPService dbcpService = context.getProperty(DBCP_SERVICE).asControllerService(DBCPService.class);\n-            final String tableName = context.getProperty(TABLE_NAME).evaluateAttributeExpressions(flowFile).getValue();\n+        // Try to fill the columnTypeMap with the types of the desired max-value columns\n+        final DBCPService dbcpService = context.getProperty(DBCP_SERVICE).asControllerService(DBCPService.class);\n+        final String tableName = context.getProperty(TABLE_NAME).evaluateAttributeExpressions(flowFile).getValue();\n+        final String sqlQuery = context.getProperty(SQL_QUERY).evaluateAttributeExpressions().getValue();\n \n             final DatabaseAdapter dbAdapter = dbAdapters.get(context.getProperty(DB_TYPE).getValue());\n             try (final Connection con = dbcpService.getConnection();\n                  final Statement st = con.createStatement()) {\n \n-                // Try a query that returns no rows, for the purposes of getting metadata about the columns. It is possible\n-                // to use DatabaseMetaData.getColumns(), but not all drivers support this, notably the schema-on-read\n-                // approach as in Apache Drill\n-                String query = dbAdapter.getSelectStatement(tableName, maxValueColumnNames, \"1 = 0\", null, null, null);\n-                ResultSet resultSet = st.executeQuery(query);\n-                ResultSetMetaData resultSetMetaData = resultSet.getMetaData();\n-                int numCols = resultSetMetaData.getColumnCount();\n-                if (numCols > 0) {\n-                    if (shouldCleanCache) {\n-                        columnTypeMap.clear();\n-                    }\n-                    for (int i = 1; i <= numCols; i++) {\n-                        String colName = resultSetMetaData.getColumnName(i).toLowerCase();\n-                        String colKey = getStateKey(tableName, colName);\n-                        int colType = resultSetMetaData.getColumnType(i);\n-                        columnTypeMap.putIfAbsent(colKey, colType);\n+            // Try a query that returns no rows, for the purposes of getting metadata about the columns. It is possible\n+            // to use DatabaseMetaData.getColumns(), but not all drivers support this, notably the schema-on-read\n+            // approach as in Apache Drill\n+            String query;\n+\n+            if(StringUtils.isEmpty(sqlQuery)) {\n+                query = dbAdapter.getSelectStatement(tableName, maxValueColumnNames, \"1 = 0\", null, null, null);\n+            } else {\n+                StringBuilder sbQuery = getWrappedQuery(sqlQuery, tableName);","path":"nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractDatabaseFetchProcessor.java","position":67,"original_position":67,"commit_id":"447fdc7a711a7918e71fa86b8ec981bfb710ad34","original_commit_id":"447fdc7a711a7918e71fa86b8ec981bfb710ad34","user":{"login":"ijokarumawak","id":1107620,"avatar_url":"https://avatars1.githubusercontent.com/u/1107620?v=4","gravatar_id":"","url":"https://api.github.com/users/ijokarumawak","html_url":"https://github.com/ijokarumawak","followers_url":"https://api.github.com/users/ijokarumawak/followers","following_url":"https://api.github.com/users/ijokarumawak/following{/other_user}","gists_url":"https://api.github.com/users/ijokarumawak/gists{/gist_id}","starred_url":"https://api.github.com/users/ijokarumawak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ijokarumawak/subscriptions","organizations_url":"https://api.github.com/users/ijokarumawak/orgs","repos_url":"https://api.github.com/users/ijokarumawak/repos","events_url":"https://api.github.com/users/ijokarumawak/events{/privacy}","received_events_url":"https://api.github.com/users/ijokarumawak/received_events","type":"User","site_admin":false},"body":"Since `getWrappedQuery` fetches all available columns in the sub-query by '*', subsequent loop stores all column values into the managed state. We should avoid storing unnecessary values into the state by either specifying the maxValueColumns to this query, or do some filtering in the loop below.","created_at":"2018-02-06T13:21:51Z","updated_at":"2018-02-06T13:30:57Z","html_url":"https://github.com/apache/nifi/pull/2162#discussion_r166294484","pull_request_url":"https://api.github.com/repos/apache/nifi/pulls/2162","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/apache/nifi/pulls/comments/166294484"},"html":{"href":"https://github.com/apache/nifi/pull/2162#discussion_r166294484"},"pull_request":{"href":"https://api.github.com/repos/apache/nifi/pulls/2162"}}},"pull_request":{"url":"https://api.github.com/repos/apache/nifi/pulls/2162","id":141762856,"html_url":"https://github.com/apache/nifi/pull/2162","diff_url":"https://github.com/apache/nifi/pull/2162.diff","patch_url":"https://github.com/apache/nifi/pull/2162.patch","issue_url":"https://api.github.com/repos/apache/nifi/issues/2162","number":2162,"state":"open","locked":false,"title":"NIFI-1706 Extend QueryDatabaseTable to support arbitrary queries","user":{"login":"patricker","id":1015990,"avatar_url":"https://avatars3.githubusercontent.com/u/1015990?v=4","gravatar_id":"","url":"https://api.github.com/users/patricker","html_url":"https://github.com/patricker","followers_url":"https://api.github.com/users/patricker/followers","following_url":"https://api.github.com/users/patricker/following{/other_user}","gists_url":"https://api.github.com/users/patricker/gists{/gist_id}","starred_url":"https://api.github.com/users/patricker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/patricker/subscriptions","organizations_url":"https://api.github.com/users/patricker/orgs","repos_url":"https://api.github.com/users/patricker/repos","events_url":"https://api.github.com/users/patricker/events{/privacy}","received_events_url":"https://api.github.com/users/patricker/received_events","type":"User","site_admin":false},"body":"\r\n\r\n### For all changes:\r\n- [x] Is there a JIRA ticket associated with this PR? Is it referenced \r\n     in the commit message?\r\n\r\n- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen \"-\" character.\r\n\r\n- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?\r\n\r\n- [x] Is your initial contribution a single, squashed commit?\r\n\r\n### For code changes:\r\n- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?\r\n- [x] Have you written or updated unit tests to verify your changes?\r\n- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? \r\n- [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?\r\n\r\n### For documentation related changes:\r\n- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?\r\n\r\n### Note:\r\nPlease ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.\r\n","created_at":"2017-09-19T05:52:30Z","updated_at":"2018-02-06T13:30:57Z","closed_at":null,"merged_at":null,"merge_commit_sha":"44a14d7be1e36cf8264fe86c0df373c3c2ce6a43","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/nifi/pulls/2162/commits","review_comments_url":"https://api.github.com/repos/apache/nifi/pulls/2162/comments","review_comment_url":"https://api.github.com/repos/apache/nifi/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/nifi/issues/2162/comments","statuses_url":"https://api.github.com/repos/apache/nifi/statuses/447fdc7a711a7918e71fa86b8ec981bfb710ad34","head":{"label":"patricker:NIFI-1706","ref":"NIFI-1706","sha":"447fdc7a711a7918e71fa86b8ec981bfb710ad34","user":{"login":"patricker","id":1015990,"avatar_url":"https://avatars3.githubusercontent.com/u/1015990?v=4","gravatar_id":"","url":"https://api.github.com/users/patricker","html_url":"https://github.com/patricker","followers_url":"https://api.github.com/users/patricker/followers","following_url":"https://api.github.com/users/patricker/following{/other_user}","gists_url":"https://api.github.com/users/patricker/gists{/gist_id}","starred_url":"https://api.github.com/users/patricker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/patricker/subscriptions","organizations_url":"https://api.github.com/users/patricker/orgs","repos_url":"https://api.github.com/users/patricker/repos","events_url":"https://api.github.com/users/patricker/events{/privacy}","received_events_url":"https://api.github.com/users/patricker/received_events","type":"User","site_admin":false},"repo":{"id":64414654,"name":"nifi","full_name":"patricker/nifi","owner":{"login":"patricker","id":1015990,"avatar_url":"https://avatars3.githubusercontent.com/u/1015990?v=4","gravatar_id":"","url":"https://api.github.com/users/patricker","html_url":"https://github.com/patricker","followers_url":"https://api.github.com/users/patricker/followers","following_url":"https://api.github.com/users/patricker/following{/other_user}","gists_url":"https://api.github.com/users/patricker/gists{/gist_id}","starred_url":"https://api.github.com/users/patricker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/patricker/subscriptions","organizations_url":"https://api.github.com/users/patricker/orgs","repos_url":"https://api.github.com/users/patricker/repos","events_url":"https://api.github.com/users/patricker/events{/privacy}","received_events_url":"https://api.github.com/users/patricker/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/patricker/nifi","description":"Mirror of Apache NiFi","fork":true,"url":"https://api.github.com/repos/patricker/nifi","forks_url":"https://api.github.com/repos/patricker/nifi/forks","keys_url":"https://api.github.com/repos/patricker/nifi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/patricker/nifi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/patricker/nifi/teams","hooks_url":"https://api.github.com/repos/patricker/nifi/hooks","issue_events_url":"https://api.github.com/repos/patricker/nifi/issues/events{/number}","events_url":"https://api.github.com/repos/patricker/nifi/events","assignees_url":"https://api.github.com/repos/patricker/nifi/assignees{/user}","branches_url":"https://api.github.com/repos/patricker/nifi/branches{/branch}","tags_url":"https://api.github.com/repos/patricker/nifi/tags","blobs_url":"https://api.github.com/repos/patricker/nifi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/patricker/nifi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/patricker/nifi/git/refs{/sha}","trees_url":"https://api.github.com/repos/patricker/nifi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/patricker/nifi/statuses/{sha}","languages_url":"https://api.github.com/repos/patricker/nifi/languages","stargazers_url":"https://api.github.com/repos/patricker/nifi/stargazers","contributors_url":"https://api.github.com/repos/patricker/nifi/contributors","subscribers_url":"https://api.github.com/repos/patricker/nifi/subscribers","subscription_url":"https://api.github.com/repos/patricker/nifi/subscription","commits_url":"https://api.github.com/repos/patricker/nifi/commits{/sha}","git_commits_url":"https://api.github.com/repos/patricker/nifi/git/commits{/sha}","comments_url":"https://api.github.com/repos/patricker/nifi/comments{/number}","issue_comment_url":"https://api.github.com/repos/patricker/nifi/issues/comments{/number}","contents_url":"https://api.github.com/repos/patricker/nifi/contents/{+path}","compare_url":"https://api.github.com/repos/patricker/nifi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/patricker/nifi/merges","archive_url":"https://api.github.com/repos/patricker/nifi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/patricker/nifi/downloads","issues_url":"https://api.github.com/repos/patricker/nifi/issues{/number}","pulls_url":"https://api.github.com/repos/patricker/nifi/pulls{/number}","milestones_url":"https://api.github.com/repos/patricker/nifi/milestones{/number}","notifications_url":"https://api.github.com/repos/patricker/nifi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/patricker/nifi/labels{/name}","releases_url":"https://api.github.com/repos/patricker/nifi/releases{/id}","deployments_url":"https://api.github.com/repos/patricker/nifi/deployments","created_at":"2016-07-28T17:23:02Z","updated_at":"2016-07-28T17:23:16Z","pushed_at":"2018-02-06T03:48:02Z","git_url":"git://github.com/patricker/nifi.git","ssh_url":"git@github.com:patricker/nifi.git","clone_url":"https://github.com/patricker/nifi.git","svn_url":"https://github.com/patricker/nifi","homepage":"","size":112579,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"3ca7c3e7a19c7f496a214d9b93ace43b1507c9ec","user":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":27911088,"name":"nifi","full_name":"apache/nifi","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/nifi","description":"Mirror of Apache NiFi","fork":false,"url":"https://api.github.com/repos/apache/nifi","forks_url":"https://api.github.com/repos/apache/nifi/forks","keys_url":"https://api.github.com/repos/apache/nifi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/nifi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/nifi/teams","hooks_url":"https://api.github.com/repos/apache/nifi/hooks","issue_events_url":"https://api.github.com/repos/apache/nifi/issues/events{/number}","events_url":"https://api.github.com/repos/apache/nifi/events","assignees_url":"https://api.github.com/repos/apache/nifi/assignees{/user}","branches_url":"https://api.github.com/repos/apache/nifi/branches{/branch}","tags_url":"https://api.github.com/repos/apache/nifi/tags","blobs_url":"https://api.github.com/repos/apache/nifi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/nifi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/nifi/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/nifi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/nifi/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/nifi/languages","stargazers_url":"https://api.github.com/repos/apache/nifi/stargazers","contributors_url":"https://api.github.com/repos/apache/nifi/contributors","subscribers_url":"https://api.github.com/repos/apache/nifi/subscribers","subscription_url":"https://api.github.com/repos/apache/nifi/subscription","commits_url":"https://api.github.com/repos/apache/nifi/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/nifi/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/nifi/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/nifi/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/nifi/contents/{+path}","compare_url":"https://api.github.com/repos/apache/nifi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/nifi/merges","archive_url":"https://api.github.com/repos/apache/nifi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/nifi/downloads","issues_url":"https://api.github.com/repos/apache/nifi/issues{/number}","pulls_url":"https://api.github.com/repos/apache/nifi/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/nifi/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/nifi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/nifi/labels{/name}","releases_url":"https://api.github.com/repos/apache/nifi/releases{/id}","deployments_url":"https://api.github.com/repos/apache/nifi/deployments","created_at":"2014-12-12T08:00:05Z","updated_at":"2018-02-06T13:19:48Z","pushed_at":"2018-02-06T11:10:16Z","git_url":"git://github.com/apache/nifi.git","ssh_url":"git@github.com:apache/nifi.git","clone_url":"https://github.com/apache/nifi.git","svn_url":"https://github.com/apache/nifi","homepage":"","size":113717,"stargazers_count":796,"watchers_count":796,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":767,"mirror_url":null,"archived":false,"open_issues_count":117,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":767,"open_issues":117,"watchers":796,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/nifi/pulls/2162"},"html":{"href":"https://github.com/apache/nifi/pull/2162"},"issue":{"href":"https://api.github.com/repos/apache/nifi/issues/2162"},"comments":{"href":"https://api.github.com/repos/apache/nifi/issues/2162/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/nifi/pulls/2162/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/nifi/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/nifi/pulls/2162/commits"},"statuses":{"href":"https://api.github.com/repos/apache/nifi/statuses/447fdc7a711a7918e71fa86b8ec981bfb710ad34"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"ijokarumawak"}}