{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/ApplicationInsights-Home/issues/285","repository_url":"https://api.github.com/repos/Microsoft/ApplicationInsights-Home","labels_url":"https://api.github.com/repos/Microsoft/ApplicationInsights-Home/issues/285/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/ApplicationInsights-Home/issues/285/comments","events_url":"https://api.github.com/repos/Microsoft/ApplicationInsights-Home/issues/285/events","html_url":"https://github.com/Microsoft/ApplicationInsights-Home/issues/285","id":378538395,"node_id":"MDU6SXNzdWUzNzg1MzgzOTU=","number":285,"title":"Snapshot Collector not cleaning up suspended processes","user":{"login":"ChrisDurante","id":34775205,"node_id":"MDQ6VXNlcjM0Nzc1MjA1","avatar_url":"https://avatars3.githubusercontent.com/u/34775205?v=4","gravatar_id":"","url":"https://api.github.com/users/ChrisDurante","html_url":"https://github.com/ChrisDurante","followers_url":"https://api.github.com/users/ChrisDurante/followers","following_url":"https://api.github.com/users/ChrisDurante/following{/other_user}","gists_url":"https://api.github.com/users/ChrisDurante/gists{/gist_id}","starred_url":"https://api.github.com/users/ChrisDurante/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ChrisDurante/subscriptions","organizations_url":"https://api.github.com/users/ChrisDurante/orgs","repos_url":"https://api.github.com/users/ChrisDurante/repos","events_url":"https://api.github.com/users/ChrisDurante/events{/privacy}","received_events_url":"https://api.github.com/users/ChrisDurante/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-11-08T01:19:25Z","updated_at":"2018-11-10T17:30:36Z","closed_at":null,"author_association":"NONE","body":"We have a console application that is hosted as a Windows Service using TopShelf.  It has many dependencies, here are the basic ones:\r\n- .NET 4.6.2\r\n- Microsoft.ApplicationInsights.Log4NetAppender 2.8.1\r\n- Microsoft.ApplicationInsights.SnapshotCollector 1.3.1\r\n- Microsoft.ApplicationInsights.WindowsServer 2.8.1\r\n- Topshelf 4.1.0\r\n\r\nOur ApplicationInsights configuration includes the `SnapshotCollecter` with default settings.\r\n```xml\r\n<?xml version=\"1.0\" encoding=\"utf-8\"?>\r\n<ApplicationInsights xmlns=\"http://schemas.microsoft.com/ApplicationInsights/2013/Settings\">\r\n  <TelemetryModules>\r\n    <Add Type=\"Microsoft.ApplicationInsights.DependencyCollector.DependencyTrackingTelemetryModule, Microsoft.AI.DependencyCollector\" />\r\n    <Add Type=\"Microsoft.ApplicationInsights.WindowsServer.AppServicesHeartbeatTelemetryModule, Microsoft.AI.WindowsServer\"/>\r\n    <Add Type=\"Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsTelemetryModule, Microsoft.ApplicationInsights\" />\r\n    <Add Type=\"Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse.QuickPulseTelemetryModule, Microsoft.AI.PerfCounterCollector\"/>\r\n    <Add Type=\"Microsoft.ApplicationInsights.WindowsServer.UnhandledExceptionTelemetryModule, Microsoft.AI.WindowsServer\"/>\r\n    <Add Type=\"Microsoft.ApplicationInsights.WindowsServer.UnobservedExceptionTelemetryModule, Microsoft.AI.WindowsServer\" />\r\n  </TelemetryModules>\r\n  <TelemetryInitializers>\r\n    <Add Type=\"Microsoft.ApplicationInsights.DependencyCollector.HttpDependenciesParsingTelemetryInitializer, Microsoft.AI.DependencyCollector\"/>\r\n    <Add Type=\"Microsoft.ApplicationInsights.WindowsServer.DomainNameRoleInstanceTelemetryInitializer, Microsoft.AI.WindowsServer\" />\r\n    <Add Type=\"Microsoft.ApplicationInsights.WindowsServer.BuildInfoConfigComponentVersionTelemetryInitializer, Microsoft.AI.WindowsServer\" />\r\n    <Add Type=\"Microsoft.ApplicationInsights.WindowsServer.DeviceTelemetryInitializer, Microsoft.AI.WindowsServer\" />\r\n    <Add Type=\"Microsoft.ApplicationInsights.WindowsServer.AzureRoleEnvironmentTelemetryInitializer, Microsoft.AI.WindowsServer\"/>\r\n  </TelemetryInitializers>\r\n  <ApplicationIdProvider Type=\"Microsoft.ApplicationInsights.Extensibility.Implementation.ApplicationId.ApplicationInsightsApplicationIdProvider, Microsoft.ApplicationInsights\"/>\r\n  <TelemetryProcessors>\r\n    <Add Type=\"Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse.QuickPulseTelemetryProcessor, Microsoft.AI.PerfCounterCollector\"/>\r\n    <Add Type=\"Microsoft.ApplicationInsights.Extensibility.AutocollectedMetricsExtractor, Microsoft.ApplicationInsights\"/>\r\n    <Add Type=\"Microsoft.ApplicationInsights.SnapshotCollector.SnapshotCollectorTelemetryProcessor, Microsoft.ApplicationInsights.SnapshotCollector\">\r\n      <!-- The default is true, but you can disable Snapshot Debugging by setting it to false -->\r\n      <IsEnabled>true</IsEnabled>\r\n      <!-- Snapshot Debugging is usually disabled in developer mode, but you can enable it by setting this to true. -->\r\n      <!-- DeveloperMode is a property on the active TelemetryChannel. -->\r\n      <IsEnabledInDeveloperMode>false</IsEnabledInDeveloperMode>\r\n      <!-- How many times we need to see an exception before we ask for snapshots. -->\r\n      <ThresholdForSnapshotting>1</ThresholdForSnapshotting>\r\n      <!-- The maximum number of examples we create for a single problem. -->\r\n      <MaximumSnapshotsRequired>3</MaximumSnapshotsRequired>\r\n      <!-- The maximum number of problems that we can be tracking at any time. -->\r\n      <MaximumCollectionPlanSize>50</MaximumCollectionPlanSize>\r\n      <!-- How often we reconnect to the stamp. The default value is 15 minutes.-->\r\n      <ReconnectInterval>00:15:00</ReconnectInterval>\r\n      <!-- How often to reset problem counters. -->\r\n      <ProblemCounterResetInterval>1.00:00:00</ProblemCounterResetInterval>\r\n      <!-- The maximum number of snapshots allowed in ten minutes.The default value is 1. -->\r\n      <SnapshotsPerTenMinutesLimit>1</SnapshotsPerTenMinutesLimit>\r\n      <!-- The maximum number of snapshots allowed per day. -->\r\n      <SnapshotsPerDayLimit>30</SnapshotsPerDayLimit>\r\n      <!-- Whether or not to collect snapshot in low IO priority thread. The default value is true. -->\r\n      <SnapshotInLowPriorityThread>true</SnapshotInLowPriorityThread>\r\n      <!-- Agree to send anonymous data to Microsoft to make this product better. -->\r\n      <ProvideAnonymousTelemetry>true</ProvideAnonymousTelemetry>\r\n      <!-- The limit on the number of failed requests to request snapshots before the telemetry processor is disabled. -->\r\n      <FailedRequestLimit>3</FailedRequestLimit>\r\n    </Add>\r\n  </TelemetryProcessors>\r\n  <TelemetryChannel Type=\"Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.ServerTelemetryChannel, Microsoft.AI.ServerTelemetryChannel\"/>\r\n</ApplicationInsights>\r\n```\r\n\r\nLeft to run and track exceptions like expected, we end up with multiple `Suspended` processes that appear to not be cleaned up after snapshot collection.\r\n![image](https://user-images.githubusercontent.com/34775205/48171025-e6d97d00-e2ae-11e8-8434-3e5ee8af03a6.png)\r\n\r\nAfter some time, this leads to Memory Exhaustion on the machine.  The server is Windows 2012R2\r\n\r\nFrom the event viewer, maybe unrelated, but noticed that SnapshotUploader64.exe had multiple crashes:\r\n```xml\r\n<Event xmlns=\"http://schemas.microsoft.com/win/2004/08/events/event\">\r\n\t<System>\r\n\t\t<Provider Name=\".NET Runtime\" /> \r\n\t\t<EventID Qualifiers=\"0\">1026</EventID> \r\n\t\t<Level>2</Level> \r\n\t\t<Task>0</Task> \r\n\t\t<Keywords>0x80000000000000</Keywords> \r\n\t\t<TimeCreated SystemTime=\"2018-11-07T14:45:14.000000000Z\" /> \r\n\t\t<EventRecordID>1318885</EventRecordID> \r\n\t\t<Channel>Application</Channel> \r\n\t\t<Computer>ServerX</Computer> \r\n\t\t<Security /> \r\n\t</System>\r\n\t <EventData>\r\n\t\t<Data>\r\nApplication: SnapshotUploader64.exe Framework Version: v4.0.30319 \r\nDescription: The process was terminated due to an unhandled exception. \r\nException Info: System.IO.IOException at System.IO.__Error.WinIOError(Int32, System.String)\r\n at System.IO.FileStream.WriteCore(Byte[], Int32, Int32) at System.IO.FileStream.FlushWrite(Boolean)\r\n at System.IO.FileStream.Flush(Boolean) at System.IO.StreamWriter.Flush(Boolean, Boolean)\r\n at System.Diagnostics.TextWriterTraceListener.Flush()\r\n at System.Diagnostics.TraceInternal.TraceEvent(System.Diagnostics.TraceEventType, Int32, System.String, System.Object[])\r\n at Microsoft.ApplicationInsights.SnapshotCollector.Uploader.TraceListener.Setup(System.String)\r\n at Microsoft.ApplicationInsights.SnapshotCollector.Uploader.Program.Run(System.String[])\r\n at Microsoft.ApplicationInsights.SnapshotCollector.Uploader.Program.Main(System.String[])\r\n at Microsoft.ApplicationInsights.SnapshotCollector.Uploader.Program64.Main(System.String[])\r\n\t\t</Data> \r\n\t</EventData>\r\n</Event>\r\n<Event xmlns=\"http://schemas.microsoft.com/win/2004/08/events/event\">\r\n\t<System>\r\n\t\t<Provider Name=\"Application Error\" /> \r\n\t\t<EventID Qualifiers=\"0\">1000</EventID> \r\n\t\t<Level>2</Level> \r\n\t\t<Task>100</Task> \r\n\t\t<Keywords>0x80000000000000</Keywords> \r\n\t\t<TimeCreated SystemTime=\"2018-11-07T14:45:14.000000000Z\" /> \r\n\t\t<EventRecordID>1318886</EventRecordID> \r\n\t\t<Channel>Application</Channel> \r\n\t\t<Computer>ServerX</Computer> \r\n\t\t<Security /> \r\n\t</System>\r\n\t<EventData>\r\n\t\t<Data>SnapshotUploader64.exe</Data> \r\n\t\t<Data>2.4.60618.2</Data> \r\n\t\t<Data>af9ba02c</Data> \r\n\t\t<Data>KERNELBASE.dll</Data> \r\n\t\t<Data>6.3.9600.18938</Data> \r\n\t\t<Data>5a7ddf0a</Data> \r\n\t\t<Data>e0434352</Data> \r\n\t\t<Data>0000000000008eac</Data> \r\n\t\t<Data>1ea0</Data> \r\n\t\t<Data>01d476a87e5103f5</Data> \r\n\t\t<Data>C:\\Users\\userX\\AppData\\Local\\SnapshotUploader\\SnapshotUploader64.exe</Data> \r\n\t\t<Data>C:\\Windows\\system32\\KERNELBASE.dll</Data> \r\n\t\t<Data>bc2152d5-e29b-11e8-8108-0050568c1491</Data> \r\n\t\t<Data /> \r\n\t\t<Data /> \r\n\t</EventData>\r\n</Event>\r\n```\r\nPlease let me know if you need more specific details to help."},"comment":{"url":"https://api.github.com/repos/Microsoft/ApplicationInsights-Home/issues/comments/437600991","html_url":"https://github.com/Microsoft/ApplicationInsights-Home/issues/285#issuecomment-437600991","issue_url":"https://api.github.com/repos/Microsoft/ApplicationInsights-Home/issues/285","id":437600991,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzYwMDk5MQ==","user":{"login":"ChrisDurante","id":34775205,"node_id":"MDQ6VXNlcjM0Nzc1MjA1","avatar_url":"https://avatars3.githubusercontent.com/u/34775205?v=4","gravatar_id":"","url":"https://api.github.com/users/ChrisDurante","html_url":"https://github.com/ChrisDurante","followers_url":"https://api.github.com/users/ChrisDurante/followers","following_url":"https://api.github.com/users/ChrisDurante/following{/other_user}","gists_url":"https://api.github.com/users/ChrisDurante/gists{/gist_id}","starred_url":"https://api.github.com/users/ChrisDurante/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ChrisDurante/subscriptions","organizations_url":"https://api.github.com/users/ChrisDurante/orgs","repos_url":"https://api.github.com/users/ChrisDurante/repos","events_url":"https://api.github.com/users/ChrisDurante/events{/privacy}","received_events_url":"https://api.github.com/users/ChrisDurante/received_events","type":"User","site_admin":false},"created_at":"2018-11-10T17:30:36Z","updated_at":"2018-11-10T17:30:36Z","author_association":"NONE","body":"Hi @clguimanMSFT, I checked for the application user's temp folder and it exists and is writable for the user. \r\nThat did lead me to the SnapshotUploader log file in `%TEMP%`, which has an error at the end that coincides with a still existing suspended process:\r\n``` SnapshotUploader64.exe Information: 0 : Scanning C:\\ApplicationX\\Int\\Process for local PDBs.\r\n    DateTime=2018-11-10T17:15:46.2147265Z\r\nSnapshotUploader64.exe Information: 0 : Local PDB scan complete. Found 12 PDB(s).\r\n    DateTime=2018-11-10T17:15:46.2303458Z\r\nSnapshotUploader64.exe Information: 0 : Deleted PDB scan marker : C:\\Users\\UserX\\AppData\\Local\\Temp\\Dumps64\\c8cf821b1adf4eafa4fae069e41d9a84\\11592.pdbscan\r\n    DateTime=2018-11-10T17:15:46.2303458Z\r\nSnapshotUploader64.exe Error: 0 : Access is denied in OpenProcess\r\n    DateTime=2018-11-10T17:18:34.0275535Z\r\n```\r\nAre there explicit OS level perms that have to be granted to the application user, or is something else possibly misconfigured?"}}
{"actor":{"display_login":"ChrisDurante"}}