{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-profiler-nodejs/issues/199","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-profiler-nodejs","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-profiler-nodejs/issues/199/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-profiler-nodejs/issues/199/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-profiler-nodejs/issues/199/events","html_url":"https://github.com/GoogleCloudPlatform/cloud-profiler-nodejs/issues/199","id":323308124,"node_id":"MDU6SXNzdWUzMjMzMDgxMjQ=","number":199,"title":"Heap profiler occasionally hangs for long periods of time on Node10 due to a GC bug in V8","user":{"login":"nolanmar511","id":14046562,"node_id":"MDQ6VXNlcjE0MDQ2NTYy","avatar_url":"https://avatars0.githubusercontent.com/u/14046562?v=4","gravatar_id":"","url":"https://api.github.com/users/nolanmar511","html_url":"https://github.com/nolanmar511","followers_url":"https://api.github.com/users/nolanmar511/followers","following_url":"https://api.github.com/users/nolanmar511/following{/other_user}","gists_url":"https://api.github.com/users/nolanmar511/gists{/gist_id}","starred_url":"https://api.github.com/users/nolanmar511/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nolanmar511/subscriptions","organizations_url":"https://api.github.com/users/nolanmar511/orgs","repos_url":"https://api.github.com/users/nolanmar511/repos","events_url":"https://api.github.com/users/nolanmar511/events{/privacy}","received_events_url":"https://api.github.com/users/nolanmar511/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-profiler-nodejs/milestones/2","html_url":"https://github.com/GoogleCloudPlatform/cloud-profiler-nodejs/milestone/2","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-profiler-nodejs/milestones/2/labels","id":3167137,"node_id":"MDk6TWlsZXN0b25lMzE2NzEzNw==","number":2,"title":"Agent Beta","description":"","creator":{"login":"aalexand","id":1519040,"node_id":"MDQ6VXNlcjE1MTkwNDA=","avatar_url":"https://avatars1.githubusercontent.com/u/1519040?v=4","gravatar_id":"","url":"https://api.github.com/users/aalexand","html_url":"https://github.com/aalexand","followers_url":"https://api.github.com/users/aalexand/followers","following_url":"https://api.github.com/users/aalexand/following{/other_user}","gists_url":"https://api.github.com/users/aalexand/gists{/gist_id}","starred_url":"https://api.github.com/users/aalexand/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aalexand/subscriptions","organizations_url":"https://api.github.com/users/aalexand/orgs","repos_url":"https://api.github.com/users/aalexand/repos","events_url":"https://api.github.com/users/aalexand/events{/privacy}","received_events_url":"https://api.github.com/users/aalexand/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":11,"state":"open","created_at":"2018-03-06T20:49:46Z","updated_at":"2018-06-20T17:03:37Z","due_on":null,"closed_at":null},"comments":45,"created_at":"2018-05-15T17:14:22Z","updated_at":"2018-06-20T17:03:37Z","closed_at":"2018-06-20T17:03:37Z","author_association":"COLLABORATOR","body":"On Node10 e2e tests, we've seen some failures which seem to be due to long pauses when collecting profiles in Node10.\r\n\r\nThis happens consistently. Below I show some of the pauses in logs seen on the continuous tests.\r\n\r\n\r\nToday's 7am test (node10) had a ~7 minute gap between starting profile creating and getting created profile:\r\n```\r\n...\r\nTue May 15 14:08:45 UTC 2018: DEBUG:@google-cloud/profiler: Attempting to create profile.\r\nTue May 15 14:16:13 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.\r\n...\r\n```\r\n\r\nToday's 5am  test (node10) had a ~7 minute gap between getting a profile after creation and collecting the profile, and another equally long gap between starting profile creation and getting the created profile:\r\n```\r\n...\r\nTue May 15 12:08:51 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.\r\nTue May 15 12:15:11 UTC 2018: DEBUG:@google-cloud/profiler: Successfully collected profile WALL.\r\n...\r\nTue May 15 12:16:52 UTC 2018: DEBUG:@google-cloud/profiler: Attempting to create profile.\r\nTue May 15 12:22:45 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.\r\n...\r\n```\r\n\r\nThe 3am run had a ~11 minute gap between starting profile creation and getting created profile:\r\n```\r\n...\r\nTue May 15 10:08:46 UTC 2018: DEBUG:@google-cloud/profiler: Attempting to create profile.\r\nTue May 15 10:19:48 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.\r\n...\r\n```\r\n\r\nThe 1am run had a ~15 minute gap between starting profile creation and getting created profile:\r\n```\r\n...\r\nTue May 15 08:08:59 UTC 2018: DEBUG:@google-cloud/profiler: Attempting to create profile.\r\nTue May 15 08:23:18 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.\r\n...\r\n```\r\nThe 11am run had a ~10 minute gap between getting created profile and collecting the profile:\r\n```\r\n...\r\nTue May 15 06:08:45 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.\r\nTue May 15 06:19:24 UTC 2018: DEBUG:@google-cloud/profiler: Successfully collected profile WALL.\r\n...\r\n```\r\n\r\nI'm inclined to believe that the long pauses occur almost randomly with respect to what the profiler is doing (but show up most frequently between log statements that typically have more time in between them).\r\n\r\nWill work on reproducing locally. If that doesn't work, then I'll try reproducing on a GCE instance with minimal resources. Then I want to see if this is more associated with the heap or time profiler.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-profiler-nodejs/issues/comments/398824443","html_url":"https://github.com/GoogleCloudPlatform/cloud-profiler-nodejs/issues/199#issuecomment-398824443","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/cloud-profiler-nodejs/issues/199","id":398824443,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODgyNDQ0Mw==","user":{"login":"nolanmar511","id":14046562,"node_id":"MDQ6VXNlcjE0MDQ2NTYy","avatar_url":"https://avatars0.githubusercontent.com/u/14046562?v=4","gravatar_id":"","url":"https://api.github.com/users/nolanmar511","html_url":"https://github.com/nolanmar511","followers_url":"https://api.github.com/users/nolanmar511/followers","following_url":"https://api.github.com/users/nolanmar511/following{/other_user}","gists_url":"https://api.github.com/users/nolanmar511/gists{/gist_id}","starred_url":"https://api.github.com/users/nolanmar511/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nolanmar511/subscriptions","organizations_url":"https://api.github.com/users/nolanmar511/orgs","repos_url":"https://api.github.com/users/nolanmar511/repos","events_url":"https://api.github.com/users/nolanmar511/events{/privacy}","received_events_url":"https://api.github.com/users/nolanmar511/received_events","type":"User","site_admin":false},"created_at":"2018-06-20T17:03:36Z","updated_at":"2018-06-20T17:03:36Z","author_association":"COLLABORATOR","body":"Confirmed by looking at integration tests that we are not seeing this in 10.4.1."}}
{"actor":{"display_login":"nolanmar511"}}