{"action":"created","comment":{"url":"https://api.github.com/repos/apache/nifi/pulls/comments/78652271","id":78652271,"diff_hunk":"@@ -117,31 +173,78 @@\n             .allowableValues(LOCATION_LOCAL, LOCATION_REMOTE)\n             .defaultValue(LOCATION_LOCAL.getValue())\n             .build();\n+\n     static final PropertyDescriptor START_POSITION = new PropertyDescriptor.Builder()\n             .name(\"Initial Start Position\")\n-            .description(\"When the Processor first begins to tail data, this property specifies where the Processor should begin reading data. Once data has been ingested from the file, \"\n+            .description(\"When the Processor first begins to tail data, this property specifies where the Processor should begin reading data. Once data has been ingested from a file, \"\n                     + \"the Processor will continue from the last point from which it has received data.\")\n             .allowableValues(START_BEGINNING_OF_TIME, START_CURRENT_FILE, START_CURRENT_TIME)\n             .defaultValue(START_CURRENT_FILE.getValue())\n             .required(true)\n             .build();\n \n+    static final PropertyDescriptor RECURSIVE = new PropertyDescriptor.Builder()\n+            .name(\"tailfile-recursive-lookup\")\n+            .displayName(\"Recursive lookup\")\n+            .description(\"When using Multiple files mode, this property defines if files must be listed recursively or not\"\n+                    + \" in the base directory.\")\n+            .allowableValues(\"true\", \"false\")\n+            .defaultValue(\"true\")\n+            .required(true)\n+            .build();\n+\n+    static final PropertyDescriptor ROLLING_STRATEGY = new PropertyDescriptor.Builder()\n+            .name(\"tailfile-rolling-strategy\")\n+            .displayName(\"Rolling Strategy\")\n+            .description(\"Specifies if the files to tail have a fixed name or not.\")\n+            .required(true)\n+            .allowableValues(FIXED_NAME, CHANGING_NAME)\n+            .defaultValue(FIXED_NAME.getValue())\n+            .build();\n+\n+    static final PropertyDescriptor LOOKUP_FREQUENCY = new PropertyDescriptor.Builder()\n+            .name(\"tailfile-lookup-frequency\")\n+            .displayName(\"Lookup frequency\")\n+            .description(\"Only used in Multiple files mode and Changing name rolling strategy, it specifies the minimum \"\n+                    + \"duration the processor will wait before listing again the files to tail.\")\n+            .required(false)\n+            .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)\n+            .defaultValue(\"10 minutes\")","path":"nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java","position":null,"original_position":186,"commit_id":"c46ac2665a119187f4b7731787cd2f5475a40a34","original_commit_id":"773cd7f0b0142402fd571330a3b1628db538616c","user":{"login":"pvillard31","id":11541012,"avatar_url":"https://avatars.githubusercontent.com/u/11541012?v=3","gravatar_id":"","url":"https://api.github.com/users/pvillard31","html_url":"https://github.com/pvillard31","followers_url":"https://api.github.com/users/pvillard31/followers","following_url":"https://api.github.com/users/pvillard31/following{/other_user}","gists_url":"https://api.github.com/users/pvillard31/gists{/gist_id}","starred_url":"https://api.github.com/users/pvillard31/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pvillard31/subscriptions","organizations_url":"https://api.github.com/users/pvillard31/orgs","repos_url":"https://api.github.com/users/pvillard31/repos","events_url":"https://api.github.com/users/pvillard31/events{/privacy}","received_events_url":"https://api.github.com/users/pvillard31/received_events","type":"User","site_admin":false},"body":"In my mind the lookup frequency would depend on the rolling configuration of the log files. Let's say there is a daily rollover, then the lookup frequency could be something like one hour (in fact the maximum duration the user is ready to wait before receiving log messages from a fresh new file). On the other side, the maximum age duration is the time we need to wait after the last received message in a log file before considering there won't be any more message in this file and that we can \"forget\" about its state (and if it happens that we do receive new messages, then we will duplicate data). With a daily rollover, it could be 24h.\r\n\r\nIf we set the lookup frequency to infinity, it means that only the files that exist when the processor is started will be considered. I don't know if it's really what we want for the user to be the default, don't you think?\r\n\r\nOn the other hand, I could agree that for the maximum age we set it to infinity by default to ensure there is no data duplication unless the user is overriding the property. But in any case, how would you set a default value to infinity in the property descriptor configuration? ``.defaultValue()`` is expecting a String, so I'm not sure to see how you would use Max Long in combination with the time period validator. Otherwise, I could use the standard '0' to represent this case. Let me know what you think.","created_at":"2016-09-13T21:56:45Z","updated_at":"2016-09-13T21:56:45Z","html_url":"https://github.com/apache/nifi/pull/980#discussion_r78652271","pull_request_url":"https://api.github.com/repos/apache/nifi/pulls/980","_links":{"self":{"href":"https://api.github.com/repos/apache/nifi/pulls/comments/78652271"},"html":{"href":"https://github.com/apache/nifi/pull/980#discussion_r78652271"},"pull_request":{"href":"https://api.github.com/repos/apache/nifi/pulls/980"}}},"pull_request":{"url":"https://api.github.com/repos/apache/nifi/pulls/980","id":83536610,"html_url":"https://github.com/apache/nifi/pull/980","diff_url":"https://github.com/apache/nifi/pull/980.diff","patch_url":"https://github.com/apache/nifi/pull/980.patch","issue_url":"https://api.github.com/repos/apache/nifi/issues/980","number":980,"state":"open","locked":false,"title":"NIFI-1170 - Improved TailFile processor to support multiple files tailing","user":{"login":"pvillard31","id":11541012,"avatar_url":"https://avatars.githubusercontent.com/u/11541012?v=3","gravatar_id":"","url":"https://api.github.com/users/pvillard31","html_url":"https://github.com/pvillard31","followers_url":"https://api.github.com/users/pvillard31/followers","following_url":"https://api.github.com/users/pvillard31/following{/other_user}","gists_url":"https://api.github.com/users/pvillard31/gists{/gist_id}","starred_url":"https://api.github.com/users/pvillard31/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pvillard31/subscriptions","organizations_url":"https://api.github.com/users/pvillard31/orgs","repos_url":"https://api.github.com/users/pvillard31/repos","events_url":"https://api.github.com/users/pvillard31/events{/privacy}","received_events_url":"https://api.github.com/users/pvillard31/received_events","type":"User","site_admin":false},"body":"Modifications should not break backward compatibility.\n\nTwo modes: single file (default) or multi files.\nIn case of multi files mode, a base directory property is required.\nThis will be the base directory to recursively look for the files to\ntail. Since this lookup can be consuming, I assumed that files to tail\nmust exist when starting the processor.\n\nThe rolling file pattern now accepts the ‘${filename}’ keyword to reference\nthe original filename (without the extension). This is useful when\ndefining the rolling file pattern for multiple files in the same\ndirectory. It obviously supposes that all the files to tail have a\nsimilar rolling pattern. Besides the rolling file must be plain text\n(no compression supported so far, this is wip).\n\nThis PR includes the modifications proposed in PR #490.","created_at":"2016-08-31T21:53:32Z","updated_at":"2016-09-13T21:56:45Z","closed_at":null,"merged_at":null,"merge_commit_sha":"5aca8f71b886f12e313caa1b8b34bc6a901c87c8","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/nifi/pulls/980/commits","review_comments_url":"https://api.github.com/repos/apache/nifi/pulls/980/comments","review_comment_url":"https://api.github.com/repos/apache/nifi/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/nifi/issues/980/comments","statuses_url":"https://api.github.com/repos/apache/nifi/statuses/c46ac2665a119187f4b7731787cd2f5475a40a34","head":{"label":"pvillard31:tailfile-regex","ref":"tailfile-regex","sha":"c46ac2665a119187f4b7731787cd2f5475a40a34","user":{"login":"pvillard31","id":11541012,"avatar_url":"https://avatars.githubusercontent.com/u/11541012?v=3","gravatar_id":"","url":"https://api.github.com/users/pvillard31","html_url":"https://github.com/pvillard31","followers_url":"https://api.github.com/users/pvillard31/followers","following_url":"https://api.github.com/users/pvillard31/following{/other_user}","gists_url":"https://api.github.com/users/pvillard31/gists{/gist_id}","starred_url":"https://api.github.com/users/pvillard31/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pvillard31/subscriptions","organizations_url":"https://api.github.com/users/pvillard31/orgs","repos_url":"https://api.github.com/users/pvillard31/repos","events_url":"https://api.github.com/users/pvillard31/events{/privacy}","received_events_url":"https://api.github.com/users/pvillard31/received_events","type":"User","site_admin":false},"repo":{"id":51175123,"name":"nifi","full_name":"pvillard31/nifi","owner":{"login":"pvillard31","id":11541012,"avatar_url":"https://avatars.githubusercontent.com/u/11541012?v=3","gravatar_id":"","url":"https://api.github.com/users/pvillard31","html_url":"https://github.com/pvillard31","followers_url":"https://api.github.com/users/pvillard31/followers","following_url":"https://api.github.com/users/pvillard31/following{/other_user}","gists_url":"https://api.github.com/users/pvillard31/gists{/gist_id}","starred_url":"https://api.github.com/users/pvillard31/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pvillard31/subscriptions","organizations_url":"https://api.github.com/users/pvillard31/orgs","repos_url":"https://api.github.com/users/pvillard31/repos","events_url":"https://api.github.com/users/pvillard31/events{/privacy}","received_events_url":"https://api.github.com/users/pvillard31/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/pvillard31/nifi","description":"Mirror of Apache NiFi","fork":true,"url":"https://api.github.com/repos/pvillard31/nifi","forks_url":"https://api.github.com/repos/pvillard31/nifi/forks","keys_url":"https://api.github.com/repos/pvillard31/nifi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pvillard31/nifi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pvillard31/nifi/teams","hooks_url":"https://api.github.com/repos/pvillard31/nifi/hooks","issue_events_url":"https://api.github.com/repos/pvillard31/nifi/issues/events{/number}","events_url":"https://api.github.com/repos/pvillard31/nifi/events","assignees_url":"https://api.github.com/repos/pvillard31/nifi/assignees{/user}","branches_url":"https://api.github.com/repos/pvillard31/nifi/branches{/branch}","tags_url":"https://api.github.com/repos/pvillard31/nifi/tags","blobs_url":"https://api.github.com/repos/pvillard31/nifi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pvillard31/nifi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pvillard31/nifi/git/refs{/sha}","trees_url":"https://api.github.com/repos/pvillard31/nifi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pvillard31/nifi/statuses/{sha}","languages_url":"https://api.github.com/repos/pvillard31/nifi/languages","stargazers_url":"https://api.github.com/repos/pvillard31/nifi/stargazers","contributors_url":"https://api.github.com/repos/pvillard31/nifi/contributors","subscribers_url":"https://api.github.com/repos/pvillard31/nifi/subscribers","subscription_url":"https://api.github.com/repos/pvillard31/nifi/subscription","commits_url":"https://api.github.com/repos/pvillard31/nifi/commits{/sha}","git_commits_url":"https://api.github.com/repos/pvillard31/nifi/git/commits{/sha}","comments_url":"https://api.github.com/repos/pvillard31/nifi/comments{/number}","issue_comment_url":"https://api.github.com/repos/pvillard31/nifi/issues/comments{/number}","contents_url":"https://api.github.com/repos/pvillard31/nifi/contents/{+path}","compare_url":"https://api.github.com/repos/pvillard31/nifi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pvillard31/nifi/merges","archive_url":"https://api.github.com/repos/pvillard31/nifi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pvillard31/nifi/downloads","issues_url":"https://api.github.com/repos/pvillard31/nifi/issues{/number}","pulls_url":"https://api.github.com/repos/pvillard31/nifi/pulls{/number}","milestones_url":"https://api.github.com/repos/pvillard31/nifi/milestones{/number}","notifications_url":"https://api.github.com/repos/pvillard31/nifi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pvillard31/nifi/labels{/name}","releases_url":"https://api.github.com/repos/pvillard31/nifi/releases{/id}","deployments_url":"https://api.github.com/repos/pvillard31/nifi/deployments","created_at":"2016-02-05T21:27:44Z","updated_at":"2016-02-05T21:27:54Z","pushed_at":"2016-09-12T19:32:05Z","git_url":"git://github.com/pvillard31/nifi.git","ssh_url":"git@github.com:pvillard31/nifi.git","clone_url":"https://github.com/pvillard31/nifi.git","svn_url":"https://github.com/pvillard31/nifi","homepage":"","size":63774,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"96a766464c2b2402e73ef77ef91222069ca54ab1","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":27911088,"name":"nifi","full_name":"apache/nifi","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/nifi","description":"Mirror of Apache NiFi","fork":false,"url":"https://api.github.com/repos/apache/nifi","forks_url":"https://api.github.com/repos/apache/nifi/forks","keys_url":"https://api.github.com/repos/apache/nifi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/nifi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/nifi/teams","hooks_url":"https://api.github.com/repos/apache/nifi/hooks","issue_events_url":"https://api.github.com/repos/apache/nifi/issues/events{/number}","events_url":"https://api.github.com/repos/apache/nifi/events","assignees_url":"https://api.github.com/repos/apache/nifi/assignees{/user}","branches_url":"https://api.github.com/repos/apache/nifi/branches{/branch}","tags_url":"https://api.github.com/repos/apache/nifi/tags","blobs_url":"https://api.github.com/repos/apache/nifi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/nifi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/nifi/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/nifi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/nifi/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/nifi/languages","stargazers_url":"https://api.github.com/repos/apache/nifi/stargazers","contributors_url":"https://api.github.com/repos/apache/nifi/contributors","subscribers_url":"https://api.github.com/repos/apache/nifi/subscribers","subscription_url":"https://api.github.com/repos/apache/nifi/subscription","commits_url":"https://api.github.com/repos/apache/nifi/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/nifi/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/nifi/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/nifi/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/nifi/contents/{+path}","compare_url":"https://api.github.com/repos/apache/nifi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/nifi/merges","archive_url":"https://api.github.com/repos/apache/nifi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/nifi/downloads","issues_url":"https://api.github.com/repos/apache/nifi/issues{/number}","pulls_url":"https://api.github.com/repos/apache/nifi/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/nifi/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/nifi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/nifi/labels{/name}","releases_url":"https://api.github.com/repos/apache/nifi/releases{/id}","deployments_url":"https://api.github.com/repos/apache/nifi/deployments","created_at":"2014-12-12T08:00:05Z","updated_at":"2016-09-10T12:48:12Z","pushed_at":"2016-09-13T21:52:45Z","git_url":"git://github.com/apache/nifi.git","ssh_url":"git@github.com:apache/nifi.git","clone_url":"https://github.com/apache/nifi.git","svn_url":"https://github.com/apache/nifi","homepage":"","size":90857,"stargazers_count":354,"watchers_count":354,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":316,"mirror_url":"git://git.apache.org/nifi.git","open_issues_count":73,"forks":316,"open_issues":73,"watchers":354,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/nifi/pulls/980"},"html":{"href":"https://github.com/apache/nifi/pull/980"},"issue":{"href":"https://api.github.com/repos/apache/nifi/issues/980"},"comments":{"href":"https://api.github.com/repos/apache/nifi/issues/980/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/nifi/pulls/980/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/nifi/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/nifi/pulls/980/commits"},"statuses":{"href":"https://api.github.com/repos/apache/nifi/statuses/c46ac2665a119187f4b7731787cd2f5475a40a34"}}}}
{"actor":{"display_login":"pvillard31"}}