{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5504","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5504/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5504/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5504/events","html_url":"https://github.com/cockroachdb/cockroach/issues/5504","id":142990965,"number":5504,"title":"server: make CGO profiles accessible","user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/1","html_url":"https://github.com/cockroachdb/cockroach/milestones/Beta","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/1/labels","id":778698,"number":1,"title":"Beta","description":"","creator":{"login":"andybons","id":365204,"avatar_url":"https://avatars.githubusercontent.com/u/365204?v=3","gravatar_id":"","url":"https://api.github.com/users/andybons","html_url":"https://github.com/andybons","followers_url":"https://api.github.com/users/andybons/followers","following_url":"https://api.github.com/users/andybons/following{/other_user}","gists_url":"https://api.github.com/users/andybons/gists{/gist_id}","starred_url":"https://api.github.com/users/andybons/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andybons/subscriptions","organizations_url":"https://api.github.com/users/andybons/orgs","repos_url":"https://api.github.com/users/andybons/repos","events_url":"https://api.github.com/users/andybons/events{/privacy}","received_events_url":"https://api.github.com/users/andybons/received_events","type":"User","site_admin":false},"open_issues":89,"closed_issues":514,"state":"open","created_at":"2014-09-04T16:42:18Z","updated_at":"2016-03-23T17:54:57Z","due_on":null,"closed_at":null},"comments":10,"created_at":"2016-03-23T15:23:03Z","updated_at":"2016-03-23T18:27:48Z","closed_at":null,"body":"OOM issues have moved into CGO territory.\r\n\r\nRestarting a node with @cuongdo's change to dump mallinfo shows the following after a couple of minutes (no external activity, but the cluster does have quite a bit of data on it):\r\n```\r\nI160323 15:21:41.183328 server/status/runtime.go:144  runtime stats: 115 goroutines, 12.04MiB active, 84897.97cgo/sec, 0\r\n.31/0.03 %(u/s)time, 0.00 %gc (47x)\r\nI160323 15:21:41.183848 server/status/linuxstats.go:56  mallinfo stats: ordblks=1.6 KiB, smblks=176 B, hblks=0 B, hblkhd\r\n=0 B, usmblks=0 B, fsmblks=10 KiB, uordblks=980 MiB, fordblks=109 MiB, keepcost=10 KiB\r\n```\r\n\r\nThis means 12MB active in Go's heap, and 980MB allocated in the main C arena.\r\nThe standard heap profile shows nothing, so we're left mostly blind. We need to add the ability to grab and dump C profiles."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/200482731","html_url":"https://github.com/cockroachdb/cockroach/issues/5504#issuecomment-200482731","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5504","id":200482731,"user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"created_at":"2016-03-23T18:27:48Z","updated_at":"2016-03-23T18:27:48Z","body":"uh. that doesn't seem right:\r\n```\r\nI160323 18:27:21.112587 server/status/linuxstats.go:56  mallinfo stats: ordblks=6.8 KiB, smblks=166 B, hblks=0 B, hblkhd=0 B, usmblks=0 B, fsmblks=9.0 KiB, uordblks=16 EiB, fordblks=112 MiB, keepcost=9.0 KiB\r\n```\r\n\r\nI don't think we're using 16 exabytes though, most likely a uint64 overflow/cast problem somewhere."}}