{"action":"created","comment":{"url":"https://api.github.com/repos/apache/incubator-metron/pulls/comments/93142265","pull_request_review_id":13664332,"id":93142265,"diff_hunk":"@@ -18,21 +18,44 @@\n #\n #\n \n-##### Storm #####\n+##### Profiler #####\n \n-profiler.workers=1\n-profiler.executors=0\n+# topic from which the profiler consumes messages\n profiler.input.topic=indexing\n+\n+# how frequently the profiler summarizes data - aka duration of each profile period\n profiler.period.duration=15\n profiler.period.duration.units=MINUTES\n+\n+# lifespan of a profile - must be greater than twice the profile period\n+# intermediate state may be lost if no messages received within this time span\n profiler.ttl=30\n profiler.ttl.units=MINUTES\n-profiler.hbase.salt.divisor=1000\n+\n+# event time processing - the name of the field containing the event timestamp\n+profiler.event.timestamp.field=timestamp\n+\n+# event time processing - how long time lags behind the last seen timestamp\n+profiler.event.time.lag=1\n+profiler.event.time.lag.units=MINUTES\n+\n+# where profiles are written to in hbase\n profiler.hbase.table=profiler\n profiler.hbase.column.family=P\n+\n+# how profiles are written to hbase\n+profiler.hbase.salt.divisor=1000","path":"metron-analytics/metron-profiler/src/main/config/profiler.properties","position":34,"original_position":34,"commit_id":"47fc0afb6d6b3a8d9661ae1d356fb0efa91e1756","original_commit_id":"47fc0afb6d6b3a8d9661ae1d356fb0efa91e1756","user":{"login":"cestella","id":540359,"avatar_url":"https://avatars.githubusercontent.com/u/540359?v=3","gravatar_id":"","url":"https://api.github.com/users/cestella","html_url":"https://github.com/cestella","followers_url":"https://api.github.com/users/cestella/followers","following_url":"https://api.github.com/users/cestella/following{/other_user}","gists_url":"https://api.github.com/users/cestella/gists{/gist_id}","starred_url":"https://api.github.com/users/cestella/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cestella/subscriptions","organizations_url":"https://api.github.com/users/cestella/orgs","repos_url":"https://api.github.com/users/cestella/repos","events_url":"https://api.github.com/users/cestella/events{/privacy}","received_events_url":"https://api.github.com/users/cestella/received_events","type":"User","site_admin":false},"body":"Perhaps a stupid question, but wouldn't we want the salt divisor in zookeeper so the reader and the writer are both sync'd?  Profiles written with salt divisors of `x` can only be read by clients that use salt divisors of `x`, right?","created_at":"2016-12-19T23:24:30Z","updated_at":"2016-12-19T23:24:30Z","html_url":"https://github.com/apache/incubator-metron/pull/395#discussion_r93142265","pull_request_url":"https://api.github.com/repos/apache/incubator-metron/pulls/395","_links":{"self":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/comments/93142265"},"html":{"href":"https://github.com/apache/incubator-metron/pull/395#discussion_r93142265"},"pull_request":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/395"}}},"pull_request":{"url":"https://api.github.com/repos/apache/incubator-metron/pulls/395","id":98028098,"html_url":"https://github.com/apache/incubator-metron/pull/395","diff_url":"https://github.com/apache/incubator-metron/pull/395.diff","patch_url":"https://github.com/apache/incubator-metron/pull/395.patch","issue_url":"https://api.github.com/repos/apache/incubator-metron/issues/395","number":395,"state":"open","locked":false,"title":"METRON-590 Enable Use of Event Time in Profiler","user":{"login":"nickwallen","id":2475409,"avatar_url":"https://avatars.githubusercontent.com/u/2475409?v=3","gravatar_id":"","url":"https://api.github.com/users/nickwallen","html_url":"https://github.com/nickwallen","followers_url":"https://api.github.com/users/nickwallen/followers","following_url":"https://api.github.com/users/nickwallen/following{/other_user}","gists_url":"https://api.github.com/users/nickwallen/gists{/gist_id}","starred_url":"https://api.github.com/users/nickwallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nickwallen/subscriptions","organizations_url":"https://api.github.com/users/nickwallen/orgs","repos_url":"https://api.github.com/users/nickwallen/repos","events_url":"https://api.github.com/users/nickwallen/events{/privacy}","received_events_url":"https://api.github.com/users/nickwallen/received_events","type":"User","site_admin":false},"body":"## [METRON-590](https://issues.apache.org/jira/browse/METRON-590)\r\n\r\n### Usage\r\n\r\n#### Creating Profiles with Live Data\r\n\r\nThe most common Profiler use case is creating profiles on live, streaming data. In this case the processing time and event time will normally remain close, but could differ under abnormal conditions.  Event time processing has some significant advantages in this scenario. \r\n\r\n* Profiles are not skewed by high demand that might delay the Profiler's processing.\r\n* Allows the Profiler to take planned/unplanned outages and pick up where it left off\r\n* Produces more accurate behavioral profiles when there is a time difference between when a behavior occurs and when the telemetry produced to tell us about that behavior is received. For example, think of a sensor that collects data in batches or mini-batches where we get data on regular intervals; every 10 minutes, hourly, etc.\r\n\r\n#### Creating Profiles with Replayed Data\r\n\r\nThe other use case that this positions us for is creating profiles from replayed or reprocessed archival data. \r\n\r\n * For example, I am creating a model based on a new feature that the Profiler is generating for me. When I move that model into Production, I need a historical view of that feature, to train my model. I can replay archived telemetry through the Profiler generating that history of my new feature. This allows the model to be effective in Production on day 1.  \r\n\r\nIt is important to note that this PR doesn't actually deliver all we need to handle replaying data. This just provides one critical component. I don't want to give anyone the impression that this PR allows us to replay data at this point in time.  [METRON-594](https://issues.apache.org/jira/browse/METRON-594) is tracking the greater effort of replaying telemetry data through the Profiler.\r\n\r\n### Changes\r\n\r\n* Added event time processing support to the Profiler.  Previously the Profiler only supported processing time aka wall clock time processing.  Event time processing is advantageous as it is not susceptible to skew caused by heavy processing load, allows the reprocessing/replay of archived telemetry data, and under certain circumstances can produce a more accurate profile of entity behavior.\r\n\r\n* By default, the Profiler will use event time processing.  The Flux topology definition file must be edited to switch the Profiler to wall clock or processing time.\r\n\r\n* The Profiler is now leveraging Storm's windowing functionality introduced in Storm 1.x.  This provides the core engine for event time processing.  This also provides a means for the use of different window types, like sliding windows, in the Profiler.  This is currently not exposed to users of the Profiler as the Flux topology definition file must be edited to use different window types.\r\n\r\n* Enhanced the Profiler integration tests which was enabled by the use of event time processing.  The integration tests now generate 24 hours of telemetry data at roughly 3 messages per minute, and then flush profile values every 15 minutes.  The entire stream of values generated by the Profiler is then validated for correctness.\r\n\r\n*  Created a `ConfigurationManager` that can be used to read the latest configuration changes in a remote data store like Zookeeper.  The default implementation, `ZkConfigurationManager` replicates the functionality that is embedded in the `ConfiguredBolt` base class.  The Profiler bolts can no longer subclass `ConfiguredBolt` as it subclasses Storm's `BaseRichBolt` which will not work for the Profiler bolts.\r\n\r\n* The usability of the Profiler was enhanced to better support active profiles that are subsequently edited by the user. Changes should be handled seamlessly by the Profiler.  This is especially helpful when a mistake is made when creating a profile, which then needs to be fixed and updated.  The Profiler was also made more resilient to failures specific to a single Profile or Tuple.  Individual failures should not impact other Profiles or Tuples.\r\n\r\n### Testing\r\n\r\nTested on a multi-node AWS cluster and the Quick Dev environment. Created, edited, and deleted multiple profile definitions as the Profiler was running and responding to the changes.  \r\n\r\nHere are manual steps that can be followed to test this PR.  At a high-level, we setup a Metron environment and run it for a period of time to create an 'enrichments' topic containing test data.  We then wait a period of time so that the data becomes stale and the timestamps attached to that data have passed.  We then start the Profiler, but tell it to consume from the `START` of the topic so that it reads all of the stale data.  We then query the Profiler to ensure that it is using the 'old' event timestamp.\r\n\r\n* Setup the Environment\r\n    * Start with a clean Metron environment.  \r\n    * Stop all sensors and topologies.\r\n    * Start consuming data from one sensor, like Snort, by starting Pcap Replay, Snort, and the Snort parser topology.  \r\n    * Ensure that data is landing in the 'enrichments' topic.\r\n    * Allow the environment to accumulate data in the 'enrichments' topic for 15 minutes.\r\n    * Stop all topologies and wait for an additional 15 - 60 minutes.\r\n* Setup the Profiler\r\n    * Create the Profiler table in HBase.\r\n        * `create 'profiler', 'P'`\r\n    * Configure the Profiler topology by editing `config/profiler.properties`.\r\n        * `profiler.period.duration=1`  \r\n        * `profiler.input.topic=enrichments`\r\n        * `kafka.start=START`\r\n    * Configure the Profiler Client by adding the following to the global properties.\r\n       * `\"profiler.client.period.duration\": \"1\",`\r\n       * `\"profiler.client.period.duration.units\": \"MINUTES\"`\r\n    * Define a profile that simply counts messages.\r\n    ```\r\n       {\r\n         \"profiles\" : [\r\n            {\r\n              \"profile\": \"counter\",\r\n              \"foreach\": \"'counter'\",\r\n              \"update\": { \"s\" : \"STATS_ADD(s, 1)\" },\r\n              \"result\": \"s\"\r\n            }\r\n         ]\r\n       }\r\n    ```\r\n    * Start the Profiler topology.\r\n* Validate\r\n    * Query the Profiler for data with a current timestamp.  There should be none.\r\n    ```\r\n    PROFILE_GET_FROM('counter', 'counter', MILLIS(5, 'MINUTES'))\r\n    ```\r\n    * Query the Profiler for data from 15 - 60 minutes ago, depending on how long you waited.  Data should be returned.\r\n    ```\r\n    PROFILE_GET_FROM('counter', 'counter', MILLIS(5, 'MINUTES'), NOW() - MILLIS(60, 'MINUTES')\r\n    ```\r\n\r\n","created_at":"2016-12-14T20:20:40Z","updated_at":"2016-12-19T23:24:30Z","closed_at":null,"merged_at":null,"merge_commit_sha":"3448608a11be3452fdbe65bc3e0a45f87b72dfd3","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/incubator-metron/pulls/395/commits","review_comments_url":"https://api.github.com/repos/apache/incubator-metron/pulls/395/comments","review_comment_url":"https://api.github.com/repos/apache/incubator-metron/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/incubator-metron/issues/395/comments","statuses_url":"https://api.github.com/repos/apache/incubator-metron/statuses/47fc0afb6d6b3a8d9661ae1d356fb0efa91e1756","head":{"label":"nickwallen:METRON-590","ref":"METRON-590","sha":"47fc0afb6d6b3a8d9661ae1d356fb0efa91e1756","user":{"login":"nickwallen","id":2475409,"avatar_url":"https://avatars.githubusercontent.com/u/2475409?v=3","gravatar_id":"","url":"https://api.github.com/users/nickwallen","html_url":"https://github.com/nickwallen","followers_url":"https://api.github.com/users/nickwallen/followers","following_url":"https://api.github.com/users/nickwallen/following{/other_user}","gists_url":"https://api.github.com/users/nickwallen/gists{/gist_id}","starred_url":"https://api.github.com/users/nickwallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nickwallen/subscriptions","organizations_url":"https://api.github.com/users/nickwallen/orgs","repos_url":"https://api.github.com/users/nickwallen/repos","events_url":"https://api.github.com/users/nickwallen/events{/privacy}","received_events_url":"https://api.github.com/users/nickwallen/received_events","type":"User","site_admin":false},"repo":{"id":51026795,"name":"incubator-metron","full_name":"nickwallen/incubator-metron","owner":{"login":"nickwallen","id":2475409,"avatar_url":"https://avatars.githubusercontent.com/u/2475409?v=3","gravatar_id":"","url":"https://api.github.com/users/nickwallen","html_url":"https://github.com/nickwallen","followers_url":"https://api.github.com/users/nickwallen/followers","following_url":"https://api.github.com/users/nickwallen/following{/other_user}","gists_url":"https://api.github.com/users/nickwallen/gists{/gist_id}","starred_url":"https://api.github.com/users/nickwallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nickwallen/subscriptions","organizations_url":"https://api.github.com/users/nickwallen/orgs","repos_url":"https://api.github.com/users/nickwallen/repos","events_url":"https://api.github.com/users/nickwallen/events{/privacy}","received_events_url":"https://api.github.com/users/nickwallen/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/nickwallen/incubator-metron","description":"Mirror of Apache Metron (Incubating)","fork":true,"url":"https://api.github.com/repos/nickwallen/incubator-metron","forks_url":"https://api.github.com/repos/nickwallen/incubator-metron/forks","keys_url":"https://api.github.com/repos/nickwallen/incubator-metron/keys{/key_id}","collaborators_url":"https://api.github.com/repos/nickwallen/incubator-metron/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/nickwallen/incubator-metron/teams","hooks_url":"https://api.github.com/repos/nickwallen/incubator-metron/hooks","issue_events_url":"https://api.github.com/repos/nickwallen/incubator-metron/issues/events{/number}","events_url":"https://api.github.com/repos/nickwallen/incubator-metron/events","assignees_url":"https://api.github.com/repos/nickwallen/incubator-metron/assignees{/user}","branches_url":"https://api.github.com/repos/nickwallen/incubator-metron/branches{/branch}","tags_url":"https://api.github.com/repos/nickwallen/incubator-metron/tags","blobs_url":"https://api.github.com/repos/nickwallen/incubator-metron/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/nickwallen/incubator-metron/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/nickwallen/incubator-metron/git/refs{/sha}","trees_url":"https://api.github.com/repos/nickwallen/incubator-metron/git/trees{/sha}","statuses_url":"https://api.github.com/repos/nickwallen/incubator-metron/statuses/{sha}","languages_url":"https://api.github.com/repos/nickwallen/incubator-metron/languages","stargazers_url":"https://api.github.com/repos/nickwallen/incubator-metron/stargazers","contributors_url":"https://api.github.com/repos/nickwallen/incubator-metron/contributors","subscribers_url":"https://api.github.com/repos/nickwallen/incubator-metron/subscribers","subscription_url":"https://api.github.com/repos/nickwallen/incubator-metron/subscription","commits_url":"https://api.github.com/repos/nickwallen/incubator-metron/commits{/sha}","git_commits_url":"https://api.github.com/repos/nickwallen/incubator-metron/git/commits{/sha}","comments_url":"https://api.github.com/repos/nickwallen/incubator-metron/comments{/number}","issue_comment_url":"https://api.github.com/repos/nickwallen/incubator-metron/issues/comments{/number}","contents_url":"https://api.github.com/repos/nickwallen/incubator-metron/contents/{+path}","compare_url":"https://api.github.com/repos/nickwallen/incubator-metron/compare/{base}...{head}","merges_url":"https://api.github.com/repos/nickwallen/incubator-metron/merges","archive_url":"https://api.github.com/repos/nickwallen/incubator-metron/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/nickwallen/incubator-metron/downloads","issues_url":"https://api.github.com/repos/nickwallen/incubator-metron/issues{/number}","pulls_url":"https://api.github.com/repos/nickwallen/incubator-metron/pulls{/number}","milestones_url":"https://api.github.com/repos/nickwallen/incubator-metron/milestones{/number}","notifications_url":"https://api.github.com/repos/nickwallen/incubator-metron/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/nickwallen/incubator-metron/labels{/name}","releases_url":"https://api.github.com/repos/nickwallen/incubator-metron/releases{/id}","deployments_url":"https://api.github.com/repos/nickwallen/incubator-metron/deployments","created_at":"2016-02-03T20:03:41Z","updated_at":"2016-10-31T16:48:05Z","pushed_at":"2016-12-19T18:42:30Z","git_url":"git://github.com/nickwallen/incubator-metron.git","ssh_url":"git@github.com:nickwallen/incubator-metron.git","clone_url":"https://github.com/nickwallen/incubator-metron.git","svn_url":"https://github.com/nickwallen/incubator-metron","homepage":null,"size":47416,"stargazers_count":2,"watchers_count":2,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":2,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"ec41e732170ebf80c00a8a9013a390f7ffa48899","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":47607561,"name":"incubator-metron","full_name":"apache/incubator-metron","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/incubator-metron","description":"Mirror of Apache Metron (Incubating)","fork":false,"url":"https://api.github.com/repos/apache/incubator-metron","forks_url":"https://api.github.com/repos/apache/incubator-metron/forks","keys_url":"https://api.github.com/repos/apache/incubator-metron/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/incubator-metron/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/incubator-metron/teams","hooks_url":"https://api.github.com/repos/apache/incubator-metron/hooks","issue_events_url":"https://api.github.com/repos/apache/incubator-metron/issues/events{/number}","events_url":"https://api.github.com/repos/apache/incubator-metron/events","assignees_url":"https://api.github.com/repos/apache/incubator-metron/assignees{/user}","branches_url":"https://api.github.com/repos/apache/incubator-metron/branches{/branch}","tags_url":"https://api.github.com/repos/apache/incubator-metron/tags","blobs_url":"https://api.github.com/repos/apache/incubator-metron/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/incubator-metron/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/incubator-metron/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/incubator-metron/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/incubator-metron/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/incubator-metron/languages","stargazers_url":"https://api.github.com/repos/apache/incubator-metron/stargazers","contributors_url":"https://api.github.com/repos/apache/incubator-metron/contributors","subscribers_url":"https://api.github.com/repos/apache/incubator-metron/subscribers","subscription_url":"https://api.github.com/repos/apache/incubator-metron/subscription","commits_url":"https://api.github.com/repos/apache/incubator-metron/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/incubator-metron/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/incubator-metron/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/incubator-metron/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/incubator-metron/contents/{+path}","compare_url":"https://api.github.com/repos/apache/incubator-metron/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/incubator-metron/merges","archive_url":"https://api.github.com/repos/apache/incubator-metron/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/incubator-metron/downloads","issues_url":"https://api.github.com/repos/apache/incubator-metron/issues{/number}","pulls_url":"https://api.github.com/repos/apache/incubator-metron/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/incubator-metron/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/incubator-metron/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/incubator-metron/labels{/name}","releases_url":"https://api.github.com/repos/apache/incubator-metron/releases{/id}","deployments_url":"https://api.github.com/repos/apache/incubator-metron/deployments","created_at":"2015-12-08T08:00:05Z","updated_at":"2016-12-19T09:49:08Z","pushed_at":"2016-12-19T21:05:33Z","git_url":"git://github.com/apache/incubator-metron.git","ssh_url":"git@github.com:apache/incubator-metron.git","clone_url":"https://github.com/apache/incubator-metron.git","svn_url":"https://github.com/apache/incubator-metron","homepage":null,"size":47360,"stargazers_count":146,"watchers_count":146,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":152,"mirror_url":"git://git.apache.org/incubator-metron.git","open_issues_count":14,"forks":152,"open_issues":14,"watchers":146,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/395"},"html":{"href":"https://github.com/apache/incubator-metron/pull/395"},"issue":{"href":"https://api.github.com/repos/apache/incubator-metron/issues/395"},"comments":{"href":"https://api.github.com/repos/apache/incubator-metron/issues/395/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/395/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/395/commits"},"statuses":{"href":"https://api.github.com/repos/apache/incubator-metron/statuses/47fc0afb6d6b3a8d9661ae1d356fb0efa91e1756"}}}}
{"actor":{"display_login":"cestella"}}