{"action":"created","issue":{"url":"https://api.github.com/repos/dotnet/orleans/issues/992","repository_url":"https://api.github.com/repos/dotnet/orleans","labels_url":"https://api.github.com/repos/dotnet/orleans/issues/992/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/orleans/issues/992/comments","events_url":"https://api.github.com/repos/dotnet/orleans/issues/992/events","html_url":"https://github.com/dotnet/orleans/issues/992","id":115207900,"number":992,"title":"Orleans Telemetry APIs","user":{"login":"galvesribeiro","id":4714040,"avatar_url":"https://avatars.githubusercontent.com/u/4714040?v=3","gravatar_id":"","url":"https://api.github.com/users/galvesribeiro","html_url":"https://github.com/galvesribeiro","followers_url":"https://api.github.com/users/galvesribeiro/followers","following_url":"https://api.github.com/users/galvesribeiro/following{/other_user}","gists_url":"https://api.github.com/users/galvesribeiro/gists{/gist_id}","starred_url":"https://api.github.com/users/galvesribeiro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/galvesribeiro/subscriptions","organizations_url":"https://api.github.com/users/galvesribeiro/orgs","repos_url":"https://api.github.com/users/galvesribeiro/repos","events_url":"https://api.github.com/users/galvesribeiro/events{/privacy}","received_events_url":"https://api.github.com/users/galvesribeiro/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dotnet/orleans/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":53,"created_at":"2015-11-05T04:36:44Z","updated_at":"2016-02-23T01:59:46Z","closed_at":null,"body":"\r\n## Problem ##\r\n\r\nToday is hard to measure performance live on Orleans due to the lack of metrics and KPIs being extracted at runtime from Orleans runtime, and neither from Grain code.\r\n\r\nWe have today [ILogConsumer](https://github.com/dotnet/orleans/blob/master/src/Orleans/Logging/ILogConsumer.cs  \"ILogConsumer\") which is pretty handy when we talk about Logging and tracing but not when we talk about KPIs.\r\n\r\nThe idea here is to create a set of APIs that will be used to extract in real-time information from Grains, Requests, Silos, Cluster, Streams, and many other Orleans Runtime components in order to push it to a [APM](https://en.wikipedia.org/wiki/Application_performance_management \"APM\") external tool, or any kind of persistent storage for further processing or real-time display pushed data on a dashboard like [PowerBI](http://powerbi.com \"PowerBi\").\r\n\r\n## Proposed Solution ##\r\n\r\n**Telemetry Provider**\r\n\r\nBased on common APM frameworks, provide a plugable abstraction for a APM provider such as this interface:\r\n\r\n    public interface ITelemetryProvider\r\n\t{\r\n\t\tTask TrackDependency(string dependencyName, string commandName, DateTimeOffset startTime, TimeSpan duration, bool success);\r\n\t\tTask TrackDependency(DependencyTelemetry telemetry);\r\n\t\tTask TrackEvent(EventTelemetry telemetry);\r\n\t\tTask TrackEvent(string eventName,IDictionary<string, string> properties = null, IDictionary<string, double> metrics = null);\r\n\t\tTask TrackException(ExceptionTelemetry telemetry);\r\n\t\tTask TrackException(Exception exception, IDictionary<string, string> properties = null, IDictionary<string, double> metrics = null);\r\n\t\tTask TrackMetric(MetricTelemetry telemetry);\r\n\t\tTask TrackMetric(string name, double value, IDictionary<string, string> properties = null);\r\n\t\tTask TrackRequest(RequestTelemetry request);\r\n\t\tTask TrackRequest(string name, DateTimeOffset startTime, TimeSpan duration, string responseCode, bool success);\r\n\t\tTask TrackTrace(TraceTelemetry telemetry);\r\n\t\tTask TrackTrace(string message);\r\n\t\tTask TrackTrace(string message, SeverityLevel severityLevel);\r\n\t\tTask TrackTrace(string message, SeverityLevel severityLevel, IDictionary<string, string> properties);\r\n\t\tTask TrackTrace(string message, IDictionary<string, string> properties);\r\n\r\n\t}\r\n\r\nThis interface covers most of the reportable telemetry types for most of the APMs tools I've seen in the marked. A provider can be an APM tool/product where Orleans should post the telemetry data like for example [NewRelic](http://newrelic.com \"NewRelic\"), Azure Application Insights, App Dynamics, DataDogHQ, etc., platform performance counters (on Windows regular performance counters, on Linux its own logd etc), a live even ingestion system like Azure EventHubs, or just a cold storage for metrics like a SQL Database, DocumentDb, Azure Table storage, whatever. The idea with the provider is to receive the telemetry incoming from Orleans clusters and store/process the way it usually does it.\r\n\r\nThe intention with the provider is to **report** telemetry data and not to analyze and neither parse telemetry. It behaves just as a proxy from the data coming from Orleans and the underlying APM Tool/storage APIs.\r\n\r\n**Provider registration/configuration**\r\n\r\nThe providers should be registered using reflection and loaded at Silo startup. The configuration should be made by regular Orleans XML config, or by Orleans configuration objects in order to keep the config experience seamless with the other system components like Storage Providers for example. There can be multiple `ITelemetryProvider` registrations so all should post the telemetry to its underlying implementations in parallel e.g., you want the data to go to Event Viewer, Performance Counter and a SQL Database for different kinds of analysis. \r\n\r\n**Telemetry Consumer - System Grain (the \"Consumer\")**\r\n\r\nThis grain behave as a consumer from a internal stream where all the telemetry reported by Orleans is pushed from the producer (more on that later) and will forward the telemetry request to all the registered `ITelemetryProvider` providers in order push the telemetry forward to the underlying APM Tool/storage.\r\n\r\n**Telemetry Manager - Proxy (the \"Producer\")**\r\n\r\nOne concrete **internal** class that behaves as a proxy for the telemetry providers. This class should implement the `ITelemetryProvider` interface in order to expose the same methods to be consumed by Orleans Runtime components so key measurable/critical points on Orleans infrastructure can be measured as KPIs.\r\n\r\nThis Proxy is responsible for receive the Telemetry and push this request to the stream in a **Fire & Forget** async model so there is no wait time during the regular process of a Grain method call (for example), since it is not a critical component for the actual request and should not impact its behavior. This class should be accessible internally by all Orleans infrastructure and from the `Grain`-based classes as well (more on that later).\r\n\r\n**Custom metrics? Oh yeah!**\r\n\r\nMost of the APM tools besides regular/common performance counters like requests p/ second, CPU, Memory, I/O, etc., also gives its users the ability to push their own technical and business metrics for further analysis, we should provide that ability for Orleans developers as well! \r\n\r\nThe implementation of the *Telemetry Manager* is internal to Orleans Runtime however, `ITelemetryProvider` is publicly available for Grain developers from the `Grain` class. This allow them to push custom telemetry just the same way as we do internally within Orleans framework boundaries. \r\n\r\n    public class MyGrain : Grain, ISomeGrain \r\n    {\r\n      protected override Task<object> Invoke(InvokeMethodRequest request, IGrainMethodInvoker invoker)\r\n      {\r\n\t\t\tbase.TelemetryManager.TrackTrace(\"Hey just pushed a trace!\");\r\n\t\t\t\r\n\t\t\t//Terrible sample on how measure request time but you guys got the idea :P\r\n\t\t\tvar now = DateTime.UtcNow;\r\n\t\t    await base.Invoke(request, invoker);\r\n\t\t\tvar elapsedTime = DateTime.UtcNow.Subtract(now).TotalMiliseconds;\r\n\t\t\tbase.TelemetryManager.TrackMetric(\"elapsedTime\", elapsedTime);\r\n\t\t\treturn Task.Done;\r\n      }\r\n    \r\n      public Task MyGrainMethod()\r\n\t  {\r\n\t\t\tbase.TelemetryManager.TrackEvent(\"MethodCalled\");\r\n\t  }\r\n    }\r\n\r\n\r\nPlease add feedback on it. I would like to move forward with this issue since I saw many questions on gitter regarding telemetry, application insights and this is personally, one of my main pain points on Orleans since besides my business/custom KPIs, I have no vision on what is going on with my cluster. This could even enhance awesome ideas like [SiloMon](http://www.silomon.net \"SiloMon\") from @richorama and so many other monitoring tools that would be nice to tap on Orleans like *Microsoft System Center Operations Manager* and so on.\r\n\r\nThanks"},"comment":{"url":"https://api.github.com/repos/dotnet/orleans/issues/comments/187478840","html_url":"https://github.com/dotnet/orleans/issues/992#issuecomment-187478840","issue_url":"https://api.github.com/repos/dotnet/orleans/issues/992","id":187478840,"user":{"login":"sergeybykov","id":8248806,"avatar_url":"https://avatars.githubusercontent.com/u/8248806?v=3","gravatar_id":"","url":"https://api.github.com/users/sergeybykov","html_url":"https://github.com/sergeybykov","followers_url":"https://api.github.com/users/sergeybykov/followers","following_url":"https://api.github.com/users/sergeybykov/following{/other_user}","gists_url":"https://api.github.com/users/sergeybykov/gists{/gist_id}","starred_url":"https://api.github.com/users/sergeybykov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sergeybykov/subscriptions","organizations_url":"https://api.github.com/users/sergeybykov/orgs","repos_url":"https://api.github.com/users/sergeybykov/repos","events_url":"https://api.github.com/users/sergeybykov/events{/privacy}","received_events_url":"https://api.github.com/users/sergeybykov/received_events","type":"User","site_admin":false},"created_at":"2016-02-23T01:59:46Z","updated_at":"2016-02-23T01:59:46Z","body":"Good.\r\n\r\nIf you already instrumented their app with AD, how much work do you think will it be to create an AD telemetry consumer?"}}