{"action":"created","comment":{"url":"https://api.github.com/repos/treasure-data/digdag/pulls/comments/67247908","id":67247908,"diff_hunk":"@@ -0,0 +1,132 @@\n+package io.digdag.standards.operator.td;\n+\n+import com.google.inject.Inject;\n+import com.treasuredata.client.TDClientHttpNotFoundException;\n+import com.treasuredata.client.model.TDJobRequest;\n+import com.treasuredata.client.model.TDJobRequestBuilder;\n+import io.digdag.client.config.Config;\n+import io.digdag.client.config.ConfigElement;\n+import io.digdag.client.config.ConfigException;\n+import io.digdag.spi.Operator;\n+import io.digdag.spi.OperatorFactory;\n+import io.digdag.spi.TaskExecutionException;\n+import io.digdag.spi.TaskRequest;\n+import io.digdag.spi.TaskResult;\n+import io.digdag.spi.TemplateEngine;\n+import io.digdag.util.BaseOperator;\n+import org.slf4j.Logger;\n+import org.slf4j.LoggerFactory;\n+\n+import java.nio.file.Path;\n+\n+import static io.digdag.standards.operator.td.TdOperatorFactory.joinJob;\n+import static java.nio.charset.StandardCharsets.UTF_8;\n+\n+public class TdWaitOperatorFactory\n+        extends AbstractWaitOperatorFactory\n+        implements OperatorFactory\n+{\n+    private static Logger logger = LoggerFactory.getLogger(TdWaitOperatorFactory.class);\n+\n+    private final TemplateEngine templateEngine;\n+\n+    @Inject\n+    public TdWaitOperatorFactory(TemplateEngine templateEngine, Config systemConfig)\n+    {\n+        super(systemConfig);\n+        this.templateEngine = templateEngine;\n+    }\n+\n+    public String getType()\n+    {\n+        return \"td_wait\";\n+    }\n+\n+    @Override\n+    public Operator newTaskExecutor(Path workspacePath, TaskRequest request)\n+    {\n+        return new TdWaitOperator(workspacePath, request);\n+    }\n+\n+    private class TdWaitOperator\n+            extends BaseOperator\n+    {\n+        private TdWaitOperator(Path workspacePath, TaskRequest request)\n+        {\n+            super(workspacePath, request);\n+        }\n+\n+        @Override\n+        public TaskResult runTask()\n+        {\n+            Config params = request.getConfig().mergeDefault(\n+                    request.getConfig().getNestedOrGetEmpty(\"td\"));\n+\n+            int pollInterval = getPollInterval(params);\n+\n+            String query = templateEngine.templateCommand(workspacePath, params, \"query\", UTF_8);\n+\n+            int defaultRows = query.trim().isEmpty() ? 0 : 1;\n+            int rows = params.get(\"rows\", int.class, defaultRows);\n+\n+            boolean done = runPollQuery(request, params, query, rows);\n+\n+            if (done) {\n+                return TaskResult.empty(request);\n+            }\n+            else {\n+                throw TaskExecutionException.ofNextPolling(pollInterval, ConfigElement.copyOf(request.getLastStateParams()));\n+            }\n+        }\n+    }\n+\n+    private boolean runPollQuery(TaskRequest request, Config params, String query, int rows)\n+    {\n+        String engine = params.get(\"engine\", String.class, \"presto\");\n+        if (!engine.equals(\"presto\") && !engine.equals(\"hive\")) {\n+            throw new ConfigException(\"Unknown 'engine:' option (available options are: hive and presto): \" + engine);\n+        }\n+\n+        try (TDOperator op = TDOperator.fromConfig(params)) {\n+\n+            int priority = params.get(\"priority\", int.class, 0);  // TODO this should accept string (VERY_LOW, LOW, NORMAL, HIGH VERY_HIGH)\n+            int jobRetry = params.get(\"job_retry\", int.class, 0);\n+\n+            TDJobRequest req = new TDJobRequestBuilder()\n+                    .setType(engine)\n+                    .setDatabase(op.getDatabase())\n+                    .setQuery(query)\n+                    .setRetryLimit(jobRetry)\n+                    .setPriority(priority)\n+                    .setScheduledTime(request.getSessionTime().getEpochSecond())\n+                    .createTDJobRequest();\n+\n+            TDJobOperator job = op.submitNewJob(req);\n+            logger.info(\"Started {} job id={}:\\n{}\", engine, job.getJobId(), query);\n+\n+            joinJob(job);\n+\n+            // TODO: wrap query in a query using count and limit instead of storing and downloading the actual full query result?","path":"digdag-standards/src/main/java/io/digdag/standards/operator/td/TdWaitOperatorFactory.java","position":109,"original_position":109,"commit_id":"09db8d815cf4e4b020dd9ee3c1eb3965ee54b2c6","original_commit_id":"09db8d815cf4e4b020dd9ee3c1eb3965ee54b2c6","user":{"login":"frsyuki","id":40720,"avatar_url":"https://avatars.githubusercontent.com/u/40720?v=3","gravatar_id":"","url":"https://api.github.com/users/frsyuki","html_url":"https://github.com/frsyuki","followers_url":"https://api.github.com/users/frsyuki/followers","following_url":"https://api.github.com/users/frsyuki/following{/other_user}","gists_url":"https://api.github.com/users/frsyuki/gists{/gist_id}","starred_url":"https://api.github.com/users/frsyuki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frsyuki/subscriptions","organizations_url":"https://api.github.com/users/frsyuki/orgs","repos_url":"https://api.github.com/users/frsyuki/repos","events_url":"https://api.github.com/users/frsyuki/events{/privacy}","received_events_url":"https://api.github.com/users/frsyuki/received_events","type":"User","site_admin":false},"body":"I think there are 2 ideas.\r\n\r\n(A) td_wait operator always checks that the query returns at least 1 row and the first column of the row is not false or NULL. And users use td_wait as following:\r\n* if you want to check existence of some rows (at least 1 row), use `select true from tbl where [CONDITIONS] limit 1`.\r\n* if you want to check number of rows (at least N rows), use `select count(*) > 100 from tbl where [CONDITIONS]`.\r\n\r\n(B) td_wait operator requires `rows: N` option (default: 1). Users will set a query that returns some rows like: `select * where [CONDITIONS]`.\r\n* if N is 1, td_wait operator wraps the query in `select true from (ORIGINAL_QUERY) sub limit 1`\r\n* if N is larger than 1, td_wait operator wraps the query in `select count(*) >= N from (select 1 from (ORIGINAL_QUERY) raw limit N) sub`\r\n\r\n(A) is more explicit. (B) is more implicit and has more opportunities to optimize automatically.\r\nI was assuming (A) and this implementation is toward (B).\r\n","created_at":"2016-06-15T21:21:02Z","updated_at":"2016-06-15T21:21:02Z","html_url":"https://github.com/treasure-data/digdag/pull/116#discussion_r67247908","pull_request_url":"https://api.github.com/repos/treasure-data/digdag/pulls/116","_links":{"self":{"href":"https://api.github.com/repos/treasure-data/digdag/pulls/comments/67247908"},"html":{"href":"https://github.com/treasure-data/digdag/pull/116#discussion_r67247908"},"pull_request":{"href":"https://api.github.com/repos/treasure-data/digdag/pulls/116"}}},"pull_request":{"url":"https://api.github.com/repos/treasure-data/digdag/pulls/116","id":73698448,"html_url":"https://github.com/treasure-data/digdag/pull/116","diff_url":"https://github.com/treasure-data/digdag/pull/116.diff","patch_url":"https://github.com/treasure-data/digdag/pull/116.patch","issue_url":"https://api.github.com/repos/treasure-data/digdag/issues/116","number":116,"state":"open","locked":false,"title":"td_wait & td_wait_table operators","user":{"login":"danielnorberg","id":393289,"avatar_url":"https://avatars.githubusercontent.com/u/393289?v=3","gravatar_id":"","url":"https://api.github.com/users/danielnorberg","html_url":"https://github.com/danielnorberg","followers_url":"https://api.github.com/users/danielnorberg/followers","following_url":"https://api.github.com/users/danielnorberg/following{/other_user}","gists_url":"https://api.github.com/users/danielnorberg/gists{/gist_id}","starred_url":"https://api.github.com/users/danielnorberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielnorberg/subscriptions","organizations_url":"https://api.github.com/users/danielnorberg/orgs","repos_url":"https://api.github.com/users/danielnorberg/repos","events_url":"https://api.github.com/users/danielnorberg/events{/privacy}","received_events_url":"https://api.github.com/users/danielnorberg/received_events","type":"User","site_admin":false},"body":"","created_at":"2016-06-14T08:25:26Z","updated_at":"2016-06-15T21:21:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ec0883c6eba740acd55e7610f47699463fd5e60f","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/treasure-data/digdag/pulls/116/commits","review_comments_url":"https://api.github.com/repos/treasure-data/digdag/pulls/116/comments","review_comment_url":"https://api.github.com/repos/treasure-data/digdag/pulls/comments{/number}","comments_url":"https://api.github.com/repos/treasure-data/digdag/issues/116/comments","statuses_url":"https://api.github.com/repos/treasure-data/digdag/statuses/09db8d815cf4e4b020dd9ee3c1eb3965ee54b2c6","head":{"label":"treasure-data:dano/td_wait","ref":"dano/td_wait","sha":"09db8d815cf4e4b020dd9ee3c1eb3965ee54b2c6","user":{"login":"treasure-data","id":747746,"avatar_url":"https://avatars.githubusercontent.com/u/747746?v=3","gravatar_id":"","url":"https://api.github.com/users/treasure-data","html_url":"https://github.com/treasure-data","followers_url":"https://api.github.com/users/treasure-data/followers","following_url":"https://api.github.com/users/treasure-data/following{/other_user}","gists_url":"https://api.github.com/users/treasure-data/gists{/gist_id}","starred_url":"https://api.github.com/users/treasure-data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/treasure-data/subscriptions","organizations_url":"https://api.github.com/users/treasure-data/orgs","repos_url":"https://api.github.com/users/treasure-data/repos","events_url":"https://api.github.com/users/treasure-data/events{/privacy}","received_events_url":"https://api.github.com/users/treasure-data/received_events","type":"Organization","site_admin":false},"repo":{"id":42751387,"name":"digdag","full_name":"treasure-data/digdag","owner":{"login":"treasure-data","id":747746,"avatar_url":"https://avatars.githubusercontent.com/u/747746?v=3","gravatar_id":"","url":"https://api.github.com/users/treasure-data","html_url":"https://github.com/treasure-data","followers_url":"https://api.github.com/users/treasure-data/followers","following_url":"https://api.github.com/users/treasure-data/following{/other_user}","gists_url":"https://api.github.com/users/treasure-data/gists{/gist_id}","starred_url":"https://api.github.com/users/treasure-data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/treasure-data/subscriptions","organizations_url":"https://api.github.com/users/treasure-data/orgs","repos_url":"https://api.github.com/users/treasure-data/repos","events_url":"https://api.github.com/users/treasure-data/events{/privacy}","received_events_url":"https://api.github.com/users/treasure-data/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/treasure-data/digdag","description":"Workload Automation System","fork":false,"url":"https://api.github.com/repos/treasure-data/digdag","forks_url":"https://api.github.com/repos/treasure-data/digdag/forks","keys_url":"https://api.github.com/repos/treasure-data/digdag/keys{/key_id}","collaborators_url":"https://api.github.com/repos/treasure-data/digdag/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/treasure-data/digdag/teams","hooks_url":"https://api.github.com/repos/treasure-data/digdag/hooks","issue_events_url":"https://api.github.com/repos/treasure-data/digdag/issues/events{/number}","events_url":"https://api.github.com/repos/treasure-data/digdag/events","assignees_url":"https://api.github.com/repos/treasure-data/digdag/assignees{/user}","branches_url":"https://api.github.com/repos/treasure-data/digdag/branches{/branch}","tags_url":"https://api.github.com/repos/treasure-data/digdag/tags","blobs_url":"https://api.github.com/repos/treasure-data/digdag/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/treasure-data/digdag/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/treasure-data/digdag/git/refs{/sha}","trees_url":"https://api.github.com/repos/treasure-data/digdag/git/trees{/sha}","statuses_url":"https://api.github.com/repos/treasure-data/digdag/statuses/{sha}","languages_url":"https://api.github.com/repos/treasure-data/digdag/languages","stargazers_url":"https://api.github.com/repos/treasure-data/digdag/stargazers","contributors_url":"https://api.github.com/repos/treasure-data/digdag/contributors","subscribers_url":"https://api.github.com/repos/treasure-data/digdag/subscribers","subscription_url":"https://api.github.com/repos/treasure-data/digdag/subscription","commits_url":"https://api.github.com/repos/treasure-data/digdag/commits{/sha}","git_commits_url":"https://api.github.com/repos/treasure-data/digdag/git/commits{/sha}","comments_url":"https://api.github.com/repos/treasure-data/digdag/comments{/number}","issue_comment_url":"https://api.github.com/repos/treasure-data/digdag/issues/comments{/number}","contents_url":"https://api.github.com/repos/treasure-data/digdag/contents/{+path}","compare_url":"https://api.github.com/repos/treasure-data/digdag/compare/{base}...{head}","merges_url":"https://api.github.com/repos/treasure-data/digdag/merges","archive_url":"https://api.github.com/repos/treasure-data/digdag/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/treasure-data/digdag/downloads","issues_url":"https://api.github.com/repos/treasure-data/digdag/issues{/number}","pulls_url":"https://api.github.com/repos/treasure-data/digdag/pulls{/number}","milestones_url":"https://api.github.com/repos/treasure-data/digdag/milestones{/number}","notifications_url":"https://api.github.com/repos/treasure-data/digdag/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/treasure-data/digdag/labels{/name}","releases_url":"https://api.github.com/repos/treasure-data/digdag/releases{/id}","deployments_url":"https://api.github.com/repos/treasure-data/digdag/deployments","created_at":"2015-09-18T23:18:08Z","updated_at":"2016-06-15T20:22:15Z","pushed_at":"2016-06-15T21:07:52Z","git_url":"git://github.com/treasure-data/digdag.git","ssh_url":"git@github.com:treasure-data/digdag.git","clone_url":"https://github.com/treasure-data/digdag.git","svn_url":"https://github.com/treasure-data/digdag","homepage":"http://www.digdag.io/","size":2593,"stargazers_count":73,"watchers_count":73,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":7,"mirror_url":null,"open_issues_count":26,"forks":7,"open_issues":26,"watchers":73,"default_branch":"master"}},"base":{"label":"treasure-data:master","ref":"master","sha":"bf0ce8031c6e1511b54f3eaef651cb84afa855f4","user":{"login":"treasure-data","id":747746,"avatar_url":"https://avatars.githubusercontent.com/u/747746?v=3","gravatar_id":"","url":"https://api.github.com/users/treasure-data","html_url":"https://github.com/treasure-data","followers_url":"https://api.github.com/users/treasure-data/followers","following_url":"https://api.github.com/users/treasure-data/following{/other_user}","gists_url":"https://api.github.com/users/treasure-data/gists{/gist_id}","starred_url":"https://api.github.com/users/treasure-data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/treasure-data/subscriptions","organizations_url":"https://api.github.com/users/treasure-data/orgs","repos_url":"https://api.github.com/users/treasure-data/repos","events_url":"https://api.github.com/users/treasure-data/events{/privacy}","received_events_url":"https://api.github.com/users/treasure-data/received_events","type":"Organization","site_admin":false},"repo":{"id":42751387,"name":"digdag","full_name":"treasure-data/digdag","owner":{"login":"treasure-data","id":747746,"avatar_url":"https://avatars.githubusercontent.com/u/747746?v=3","gravatar_id":"","url":"https://api.github.com/users/treasure-data","html_url":"https://github.com/treasure-data","followers_url":"https://api.github.com/users/treasure-data/followers","following_url":"https://api.github.com/users/treasure-data/following{/other_user}","gists_url":"https://api.github.com/users/treasure-data/gists{/gist_id}","starred_url":"https://api.github.com/users/treasure-data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/treasure-data/subscriptions","organizations_url":"https://api.github.com/users/treasure-data/orgs","repos_url":"https://api.github.com/users/treasure-data/repos","events_url":"https://api.github.com/users/treasure-data/events{/privacy}","received_events_url":"https://api.github.com/users/treasure-data/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/treasure-data/digdag","description":"Workload Automation System","fork":false,"url":"https://api.github.com/repos/treasure-data/digdag","forks_url":"https://api.github.com/repos/treasure-data/digdag/forks","keys_url":"https://api.github.com/repos/treasure-data/digdag/keys{/key_id}","collaborators_url":"https://api.github.com/repos/treasure-data/digdag/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/treasure-data/digdag/teams","hooks_url":"https://api.github.com/repos/treasure-data/digdag/hooks","issue_events_url":"https://api.github.com/repos/treasure-data/digdag/issues/events{/number}","events_url":"https://api.github.com/repos/treasure-data/digdag/events","assignees_url":"https://api.github.com/repos/treasure-data/digdag/assignees{/user}","branches_url":"https://api.github.com/repos/treasure-data/digdag/branches{/branch}","tags_url":"https://api.github.com/repos/treasure-data/digdag/tags","blobs_url":"https://api.github.com/repos/treasure-data/digdag/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/treasure-data/digdag/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/treasure-data/digdag/git/refs{/sha}","trees_url":"https://api.github.com/repos/treasure-data/digdag/git/trees{/sha}","statuses_url":"https://api.github.com/repos/treasure-data/digdag/statuses/{sha}","languages_url":"https://api.github.com/repos/treasure-data/digdag/languages","stargazers_url":"https://api.github.com/repos/treasure-data/digdag/stargazers","contributors_url":"https://api.github.com/repos/treasure-data/digdag/contributors","subscribers_url":"https://api.github.com/repos/treasure-data/digdag/subscribers","subscription_url":"https://api.github.com/repos/treasure-data/digdag/subscription","commits_url":"https://api.github.com/repos/treasure-data/digdag/commits{/sha}","git_commits_url":"https://api.github.com/repos/treasure-data/digdag/git/commits{/sha}","comments_url":"https://api.github.com/repos/treasure-data/digdag/comments{/number}","issue_comment_url":"https://api.github.com/repos/treasure-data/digdag/issues/comments{/number}","contents_url":"https://api.github.com/repos/treasure-data/digdag/contents/{+path}","compare_url":"https://api.github.com/repos/treasure-data/digdag/compare/{base}...{head}","merges_url":"https://api.github.com/repos/treasure-data/digdag/merges","archive_url":"https://api.github.com/repos/treasure-data/digdag/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/treasure-data/digdag/downloads","issues_url":"https://api.github.com/repos/treasure-data/digdag/issues{/number}","pulls_url":"https://api.github.com/repos/treasure-data/digdag/pulls{/number}","milestones_url":"https://api.github.com/repos/treasure-data/digdag/milestones{/number}","notifications_url":"https://api.github.com/repos/treasure-data/digdag/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/treasure-data/digdag/labels{/name}","releases_url":"https://api.github.com/repos/treasure-data/digdag/releases{/id}","deployments_url":"https://api.github.com/repos/treasure-data/digdag/deployments","created_at":"2015-09-18T23:18:08Z","updated_at":"2016-06-15T20:22:15Z","pushed_at":"2016-06-15T21:07:52Z","git_url":"git://github.com/treasure-data/digdag.git","ssh_url":"git@github.com:treasure-data/digdag.git","clone_url":"https://github.com/treasure-data/digdag.git","svn_url":"https://github.com/treasure-data/digdag","homepage":"http://www.digdag.io/","size":2593,"stargazers_count":73,"watchers_count":73,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":7,"mirror_url":null,"open_issues_count":26,"forks":7,"open_issues":26,"watchers":73,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/treasure-data/digdag/pulls/116"},"html":{"href":"https://github.com/treasure-data/digdag/pull/116"},"issue":{"href":"https://api.github.com/repos/treasure-data/digdag/issues/116"},"comments":{"href":"https://api.github.com/repos/treasure-data/digdag/issues/116/comments"},"review_comments":{"href":"https://api.github.com/repos/treasure-data/digdag/pulls/116/comments"},"review_comment":{"href":"https://api.github.com/repos/treasure-data/digdag/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/treasure-data/digdag/pulls/116/commits"},"statuses":{"href":"https://api.github.com/repos/treasure-data/digdag/statuses/09db8d815cf4e4b020dd9ee3c1eb3965ee54b2c6"}}}}
{"actor":{"display_login":"frsyuki"}}