{"action":"created","comment":{"url":"https://api.github.com/repos/DataTorrent/docs/pulls/comments/61514614","id":61514614,"diff_hunk":"@@ -0,0 +1,170 @@\n+App Data Tracker\n+================\n+\n+# Introduction\n+App Data Tracker, called ADT from now on for conciseness, is a system Apex application that utilizes App Data Framework for collecting and aggregating stats and auto-metrics of user Apex applications.\n+\n+**Stats** are pre-defined static measurements that tracks the health of an application. Per Apex Application there are 2 types of stats:\n+* Application stats - these include stats at the application level. For example - `plannedContainers`, `numOperators`, `failedContainers`, `totalTuplesProcessed`, etc.\n+* Operator stats - these include stats of each operator in an application. Example of per operator stats are - `totalTuplesProcessed`, `totalTuplesEmitted`, `cpuPercentageMA`, etc.\n+\n+**AutoMetrics** are custom measurements that an operator developer can define for their operator. When an operator having auto-metrics is integrated to an application, then these auto-metrics become eligible\n+to be collected by ADT for aggregation. AutoMetrics are explained in detail [here](http://docs.datatorrent.com/autometrics/).\n+\n+# Interaction between ADT and user Apex Applications\n+\n+All the user applications which are launched after ADT is enabled, push stats and metrics to the Gateway which serves as a message bus. ADT acts as a\n+subscriber to this message bus. The diagram below depicts the message flow between user Apex applications, gateway and ADT.\n+\n+<a name=\"gateway-adt\"/>![Message flow between user applications, gateway and ADT](images/adt/adt.png)\n+\n+# Launching ADT\n+\n+ADT can be automatically launched by the gateway if the dt-site.xml that is used by the gateway has the following properties:\n+\n+```xml\n+<property>\n+  <name>dt.appDataTracker.enable</name>\n+  <value>true</value>\n+</property>\n+<property>\n+  <name>dt.appDataTracker.transport</name>\n+  <value>com.datatorrent.common.metric.AutoMetricBuiltInTransport:AppDataTrackerFeed</value>\n+</property>\n+<property>\n+  <name>dt.attr.METRICS_TRANSPORT</name>\n+  <value>com.datatorrent.common.metric.AutoMetricBuiltInTransport:AppDataTrackerFeed</value>\n+</property>\n+```\n+\n+ **Note**: The App Data Tracker will be shown running in dtManage as a “system app”.  It will show up if the “show system apps” button is pressed.\n+\n+# ADT Topology\n+\n+![ADT Topology](images/adt/adt_topology.png)\n+\n+The input operator in ADT is a web-socket input operator which listens to the topic `AppDataTrackerFeed`. This is the topic to which the\n+application masters of the user applications are publishing to.\n+\n+Once the messages enter ADT, they are processed for aggregation and stored in HDHT.\n+\n+# Aggregations\n+\n+ADT aggregates only **Metrics** by time. **Stats** - application or operator level represent cumulative values and therefore are not time-aggregated.\n+\n+By default, the time buckets App Data Tracker aggregates upon are one minute, one hour and one day. It can be overridden by changing the operator attribute `METRICS_DIMENSIONS_SCHEME`.\n+\n+Also by default, the app data tracker performs all these aggregations : SUM, MIN, MAX, AVG, COUNT, FIRST, LAST on all number metrics.  You can also override by changing the same operator attribute `METRICS_DIMENSIONS_SCHEME`, provided the custom aggregator is known to the App Data Tracker.  (See next section)\n+\n+## Custom Aggregator in App Data Tracker\n+Custom aggregators allow you to do your own custom computation on statistics generated by any of your applications. In order to implement a Custom aggregator you have to do two things:\n+\n+1. Combining new inputs with the current aggregation\n+2. Combining two aggregations together into one aggregation\n+\n+Let’s consider the case where we want to perform the following rolling average:\n+\n+Y_n = ½ * X_n + ½ * X_n-1 + ¼ * X_n-2 + ⅛ * X_n-3 +...\n+\n+This aggregation could be performed by the following Custom Aggregator:\n+\n+```java\n+@Name(\"IIRAVG\")\n+public class AggregatorIIRAVG extends AbstractIncrementalAggregator\n+{\n+  ...\n+\n+  private void aggregateHelper(DimensionsEvent dest, DimensionsEvent src)\n+  {\n+    double[] destVals = dest.getAggregates().getFieldsDouble();\n+    double[] srcVals = src.getAggregates().getFieldsDouble();\n+\n+    for (int index = 0; index < destLongs.length; index++) {\n+      destVals[index] = .5 * destVals[index] + .5 * srcVals[index];\n+    }\n+  }\n+\n+  @Override\n+  public void aggregate(Aggregate dest, InputEvent src)\n+  {\n+    //Aggregate a current aggregation with a new input\n+    aggregateHelper(dest, src);\n+  }\n+\n+  @Override\n+  public void aggregate(Aggregate destAgg, Aggregate srcAgg)\n+  {\n+    //Combine two existing aggregations together\n+    aggregateHelper(destAgg, srcAgg);\n+  }\n+}\n+```\n+\n+## Discovery of Custom Aggregators\n+AppDataTracker searches for custom aggregator jars under the following directories statically before launching:\n+\n+1. {dt\\_installation\\_dir}/plugin/aggregators\n+2. {user\\_home\\_dir}/.dt/plugin/aggregators\n+\n+It uses reflection to find all the classes that extend from `IncrementalAggregator` and `OTFAggregator` in these jars and registers them with the name provided by `@Name` annotation (or class name when `@Name` is absent).\n+\n+## Using `METRICS_DIMENSIONS_SCHEME`\n+\n+Here is a sample code snippet on how you can make use of `METRICS_DIMENSIONS_SCHEME` to set your own time buckets and your own set of aggregators for certain `AutoMetric`s performed by the App Data Tracker in your application.\n+\n+```java\n+  @Override\n+  public void populateDAG(DAG dag, Configuration configuration)\n+  {\n+    ...\n+    LineReceiver lineReceiver = dag.addOperator(\"LineReceiver\", new LineReceiver());\n+    ...\n+    AutoMetric.DimensionsScheme dimensionsScheme = new AutoMetric.DimensionsScheme()\n+    {\n+      String[] timeBuckets = new String[] { \"1s\", \"1m\", \"1h\" };\n+      String[] lengthAggregators = new String[] { \"IIRAVG\", \"SUM\" };\n+      String[] countAggregators = new String[] { \"SUM\" };\n+\n+      /* Setting the aggregation time bucket to be one second, one minute and one hour */\n+      @Override\n+      public String[] getTimeBuckets()\n+      {\n+        return timeBuckets;\n+      }\n+\n+      @Override\n+      public String[] getDimensionAggregationsFor(String logicalMetricName)\n+      {\n+        if (\"length\".equals(logicalMetricName)) {\n+          return lengthAggregators;\n+        } else if (\"count\".equals(logicalMetricName)) {\n+          return countAggregators;\n+        } else {\n+          return null; // use default\n+        }\n+      }\n+    };\n+\n+    dag.setAttribute(lineReceiver, OperatorContext.METRICS_DIMENSIONS_SCHEME, dimensionsScheme);\n+    ...\n+  }\n+```\n+\n+\n+# Dashboards\n+With App Data Tracker enabled, you can visualize the AutoMetrics and system metrics in the Dashboards within dtManage.   Refer back to the [diagram](#gateway_adt), dtGateway relays queries and query results to and from the App Data Tracker.  In this way, dtManage sends queries and receives results from the App Data Tracker via dtGateway and uses the results to let the user visualize the data.","path":"docs/app_data_tracker.md","position":155,"original_position":155,"commit_id":"aeffb7f7554f80b55ae97797ac9469da67237b56","original_commit_id":"aeffb7f7554f80b55ae97797ac9469da67237b56","user":{"login":"davidyan74","id":4419148,"avatar_url":"https://avatars.githubusercontent.com/u/4419148?v=3","gravatar_id":"","url":"https://api.github.com/users/davidyan74","html_url":"https://github.com/davidyan74","followers_url":"https://api.github.com/users/davidyan74/followers","following_url":"https://api.github.com/users/davidyan74/following{/other_user}","gists_url":"https://api.github.com/users/davidyan74/gists{/gist_id}","starred_url":"https://api.github.com/users/davidyan74/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidyan74/subscriptions","organizations_url":"https://api.github.com/users/davidyan74/orgs","repos_url":"https://api.github.com/users/davidyan74/repos","events_url":"https://api.github.com/users/davidyan74/events{/privacy}","received_events_url":"https://api.github.com/users/davidyan74/received_events","type":"User","site_admin":false},"body":"AppDataTracker -> ADT","created_at":"2016-04-28T22:44:17Z","updated_at":"2016-04-28T22:44:17Z","html_url":"https://github.com/DataTorrent/docs/pull/63#discussion_r61514614","pull_request_url":"https://api.github.com/repos/DataTorrent/docs/pulls/63","_links":{"self":{"href":"https://api.github.com/repos/DataTorrent/docs/pulls/comments/61514614"},"html":{"href":"https://github.com/DataTorrent/docs/pull/63#discussion_r61514614"},"pull_request":{"href":"https://api.github.com/repos/DataTorrent/docs/pulls/63"}}},"pull_request":{"url":"https://api.github.com/repos/DataTorrent/docs/pulls/63","id":68299219,"html_url":"https://github.com/DataTorrent/docs/pull/63","diff_url":"https://github.com/DataTorrent/docs/pull/63.diff","patch_url":"https://github.com/DataTorrent/docs/pull/63.patch","issue_url":"https://api.github.com/repos/DataTorrent/docs/issues/63","number":63,"state":"open","locked":false,"title":"SPOI-6565 add app data tracker documentation","user":{"login":"chandnisingh","id":4690923,"avatar_url":"https://avatars.githubusercontent.com/u/4690923?v=3","gravatar_id":"","url":"https://api.github.com/users/chandnisingh","html_url":"https://github.com/chandnisingh","followers_url":"https://api.github.com/users/chandnisingh/followers","following_url":"https://api.github.com/users/chandnisingh/following{/other_user}","gists_url":"https://api.github.com/users/chandnisingh/gists{/gist_id}","starred_url":"https://api.github.com/users/chandnisingh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chandnisingh/subscriptions","organizations_url":"https://api.github.com/users/chandnisingh/orgs","repos_url":"https://api.github.com/users/chandnisingh/repos","events_url":"https://api.github.com/users/chandnisingh/events{/privacy}","received_events_url":"https://api.github.com/users/chandnisingh/received_events","type":"User","site_admin":false},"body":"@davidyan74 @sashadt \r\nPlease review","created_at":"2016-04-28T22:13:31Z","updated_at":"2016-04-28T22:44:17Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a2f75ffc725020422f8f39330201ed886d041306","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/DataTorrent/docs/pulls/63/commits","review_comments_url":"https://api.github.com/repos/DataTorrent/docs/pulls/63/comments","review_comment_url":"https://api.github.com/repos/DataTorrent/docs/pulls/comments{/number}","comments_url":"https://api.github.com/repos/DataTorrent/docs/issues/63/comments","statuses_url":"https://api.github.com/repos/DataTorrent/docs/statuses/aeffb7f7554f80b55ae97797ac9469da67237b56","head":{"label":"chandnisingh:SPOI-6565","ref":"SPOI-6565","sha":"aeffb7f7554f80b55ae97797ac9469da67237b56","user":{"login":"chandnisingh","id":4690923,"avatar_url":"https://avatars.githubusercontent.com/u/4690923?v=3","gravatar_id":"","url":"https://api.github.com/users/chandnisingh","html_url":"https://github.com/chandnisingh","followers_url":"https://api.github.com/users/chandnisingh/followers","following_url":"https://api.github.com/users/chandnisingh/following{/other_user}","gists_url":"https://api.github.com/users/chandnisingh/gists{/gist_id}","starred_url":"https://api.github.com/users/chandnisingh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chandnisingh/subscriptions","organizations_url":"https://api.github.com/users/chandnisingh/orgs","repos_url":"https://api.github.com/users/chandnisingh/repos","events_url":"https://api.github.com/users/chandnisingh/events{/privacy}","received_events_url":"https://api.github.com/users/chandnisingh/received_events","type":"User","site_admin":false},"repo":{"id":44461758,"name":"docs","full_name":"chandnisingh/docs","owner":{"login":"chandnisingh","id":4690923,"avatar_url":"https://avatars.githubusercontent.com/u/4690923?v=3","gravatar_id":"","url":"https://api.github.com/users/chandnisingh","html_url":"https://github.com/chandnisingh","followers_url":"https://api.github.com/users/chandnisingh/followers","following_url":"https://api.github.com/users/chandnisingh/following{/other_user}","gists_url":"https://api.github.com/users/chandnisingh/gists{/gist_id}","starred_url":"https://api.github.com/users/chandnisingh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chandnisingh/subscriptions","organizations_url":"https://api.github.com/users/chandnisingh/orgs","repos_url":"https://api.github.com/users/chandnisingh/repos","events_url":"https://api.github.com/users/chandnisingh/events{/privacy}","received_events_url":"https://api.github.com/users/chandnisingh/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/chandnisingh/docs","description":"DataTorrent Product Documentation","fork":true,"url":"https://api.github.com/repos/chandnisingh/docs","forks_url":"https://api.github.com/repos/chandnisingh/docs/forks","keys_url":"https://api.github.com/repos/chandnisingh/docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/chandnisingh/docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/chandnisingh/docs/teams","hooks_url":"https://api.github.com/repos/chandnisingh/docs/hooks","issue_events_url":"https://api.github.com/repos/chandnisingh/docs/issues/events{/number}","events_url":"https://api.github.com/repos/chandnisingh/docs/events","assignees_url":"https://api.github.com/repos/chandnisingh/docs/assignees{/user}","branches_url":"https://api.github.com/repos/chandnisingh/docs/branches{/branch}","tags_url":"https://api.github.com/repos/chandnisingh/docs/tags","blobs_url":"https://api.github.com/repos/chandnisingh/docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/chandnisingh/docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/chandnisingh/docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/chandnisingh/docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/chandnisingh/docs/statuses/{sha}","languages_url":"https://api.github.com/repos/chandnisingh/docs/languages","stargazers_url":"https://api.github.com/repos/chandnisingh/docs/stargazers","contributors_url":"https://api.github.com/repos/chandnisingh/docs/contributors","subscribers_url":"https://api.github.com/repos/chandnisingh/docs/subscribers","subscription_url":"https://api.github.com/repos/chandnisingh/docs/subscription","commits_url":"https://api.github.com/repos/chandnisingh/docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/chandnisingh/docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/chandnisingh/docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/chandnisingh/docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/chandnisingh/docs/contents/{+path}","compare_url":"https://api.github.com/repos/chandnisingh/docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/chandnisingh/docs/merges","archive_url":"https://api.github.com/repos/chandnisingh/docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/chandnisingh/docs/downloads","issues_url":"https://api.github.com/repos/chandnisingh/docs/issues{/number}","pulls_url":"https://api.github.com/repos/chandnisingh/docs/pulls{/number}","milestones_url":"https://api.github.com/repos/chandnisingh/docs/milestones{/number}","notifications_url":"https://api.github.com/repos/chandnisingh/docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/chandnisingh/docs/labels{/name}","releases_url":"https://api.github.com/repos/chandnisingh/docs/releases{/id}","deployments_url":"https://api.github.com/repos/chandnisingh/docs/deployments","created_at":"2015-10-18T01:48:58Z","updated_at":"2015-09-23T23:13:45Z","pushed_at":"2016-04-28T22:12:12Z","git_url":"git://github.com/chandnisingh/docs.git","ssh_url":"git@github.com:chandnisingh/docs.git","clone_url":"https://github.com/chandnisingh/docs.git","svn_url":"https://github.com/chandnisingh/docs","homepage":null,"size":38070,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"DataTorrent:master","ref":"master","sha":"386a04cce83e3cc00b427665e1b4d32612ecb193","user":{"login":"DataTorrent","id":4400063,"avatar_url":"https://avatars.githubusercontent.com/u/4400063?v=3","gravatar_id":"","url":"https://api.github.com/users/DataTorrent","html_url":"https://github.com/DataTorrent","followers_url":"https://api.github.com/users/DataTorrent/followers","following_url":"https://api.github.com/users/DataTorrent/following{/other_user}","gists_url":"https://api.github.com/users/DataTorrent/gists{/gist_id}","starred_url":"https://api.github.com/users/DataTorrent/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataTorrent/subscriptions","organizations_url":"https://api.github.com/users/DataTorrent/orgs","repos_url":"https://api.github.com/users/DataTorrent/repos","events_url":"https://api.github.com/users/DataTorrent/events{/privacy}","received_events_url":"https://api.github.com/users/DataTorrent/received_events","type":"Organization","site_admin":false},"repo":{"id":43032394,"name":"docs","full_name":"DataTorrent/docs","owner":{"login":"DataTorrent","id":4400063,"avatar_url":"https://avatars.githubusercontent.com/u/4400063?v=3","gravatar_id":"","url":"https://api.github.com/users/DataTorrent","html_url":"https://github.com/DataTorrent","followers_url":"https://api.github.com/users/DataTorrent/followers","following_url":"https://api.github.com/users/DataTorrent/following{/other_user}","gists_url":"https://api.github.com/users/DataTorrent/gists{/gist_id}","starred_url":"https://api.github.com/users/DataTorrent/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataTorrent/subscriptions","organizations_url":"https://api.github.com/users/DataTorrent/orgs","repos_url":"https://api.github.com/users/DataTorrent/repos","events_url":"https://api.github.com/users/DataTorrent/events{/privacy}","received_events_url":"https://api.github.com/users/DataTorrent/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/DataTorrent/docs","description":"DataTorrent Product Documentation","fork":false,"url":"https://api.github.com/repos/DataTorrent/docs","forks_url":"https://api.github.com/repos/DataTorrent/docs/forks","keys_url":"https://api.github.com/repos/DataTorrent/docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/DataTorrent/docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/DataTorrent/docs/teams","hooks_url":"https://api.github.com/repos/DataTorrent/docs/hooks","issue_events_url":"https://api.github.com/repos/DataTorrent/docs/issues/events{/number}","events_url":"https://api.github.com/repos/DataTorrent/docs/events","assignees_url":"https://api.github.com/repos/DataTorrent/docs/assignees{/user}","branches_url":"https://api.github.com/repos/DataTorrent/docs/branches{/branch}","tags_url":"https://api.github.com/repos/DataTorrent/docs/tags","blobs_url":"https://api.github.com/repos/DataTorrent/docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/DataTorrent/docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/DataTorrent/docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/DataTorrent/docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/DataTorrent/docs/statuses/{sha}","languages_url":"https://api.github.com/repos/DataTorrent/docs/languages","stargazers_url":"https://api.github.com/repos/DataTorrent/docs/stargazers","contributors_url":"https://api.github.com/repos/DataTorrent/docs/contributors","subscribers_url":"https://api.github.com/repos/DataTorrent/docs/subscribers","subscription_url":"https://api.github.com/repos/DataTorrent/docs/subscription","commits_url":"https://api.github.com/repos/DataTorrent/docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/DataTorrent/docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/DataTorrent/docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/DataTorrent/docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/DataTorrent/docs/contents/{+path}","compare_url":"https://api.github.com/repos/DataTorrent/docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/DataTorrent/docs/merges","archive_url":"https://api.github.com/repos/DataTorrent/docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/DataTorrent/docs/downloads","issues_url":"https://api.github.com/repos/DataTorrent/docs/issues{/number}","pulls_url":"https://api.github.com/repos/DataTorrent/docs/pulls{/number}","milestones_url":"https://api.github.com/repos/DataTorrent/docs/milestones{/number}","notifications_url":"https://api.github.com/repos/DataTorrent/docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/DataTorrent/docs/labels{/name}","releases_url":"https://api.github.com/repos/DataTorrent/docs/releases{/id}","deployments_url":"https://api.github.com/repos/DataTorrent/docs/deployments","created_at":"2015-09-23T23:13:45Z","updated_at":"2016-03-12T06:36:12Z","pushed_at":"2016-04-28T22:13:31Z","git_url":"git://github.com/DataTorrent/docs.git","ssh_url":"git@github.com:DataTorrent/docs.git","clone_url":"https://github.com/DataTorrent/docs.git","svn_url":"https://github.com/DataTorrent/docs","homepage":null,"size":46899,"stargazers_count":3,"watchers_count":3,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":15,"mirror_url":null,"open_issues_count":4,"forks":15,"open_issues":4,"watchers":3,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/DataTorrent/docs/pulls/63"},"html":{"href":"https://github.com/DataTorrent/docs/pull/63"},"issue":{"href":"https://api.github.com/repos/DataTorrent/docs/issues/63"},"comments":{"href":"https://api.github.com/repos/DataTorrent/docs/issues/63/comments"},"review_comments":{"href":"https://api.github.com/repos/DataTorrent/docs/pulls/63/comments"},"review_comment":{"href":"https://api.github.com/repos/DataTorrent/docs/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/DataTorrent/docs/pulls/63/commits"},"statuses":{"href":"https://api.github.com/repos/DataTorrent/docs/statuses/aeffb7f7554f80b55ae97797ac9469da67237b56"}}}}