{"action":"created","issue":{"url":"https://api.github.com/repos/miekg/dns/issues/786","repository_url":"https://api.github.com/repos/miekg/dns","labels_url":"https://api.github.com/repos/miekg/dns/issues/786/labels{/name}","comments_url":"https://api.github.com/repos/miekg/dns/issues/786/comments","events_url":"https://api.github.com/repos/miekg/dns/issues/786/events","html_url":"https://github.com/miekg/dns/issues/786","id":369304694,"node_id":"MDU6SXNzdWUzNjkzMDQ2OTQ=","number":786,"title":"Memory Leak in github.com/miekg/dns.parseZone","user":{"login":"w8rbt","id":9010108,"node_id":"MDQ6VXNlcjkwMTAxMDg=","avatar_url":"https://avatars0.githubusercontent.com/u/9010108?v=4","gravatar_id":"","url":"https://api.github.com/users/w8rbt","html_url":"https://github.com/w8rbt","followers_url":"https://api.github.com/users/w8rbt/followers","following_url":"https://api.github.com/users/w8rbt/following{/other_user}","gists_url":"https://api.github.com/users/w8rbt/gists{/gist_id}","starred_url":"https://api.github.com/users/w8rbt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/w8rbt/subscriptions","organizations_url":"https://api.github.com/users/w8rbt/orgs","repos_url":"https://api.github.com/users/w8rbt/repos","events_url":"https://api.github.com/users/w8rbt/events{/privacy}","received_events_url":"https://api.github.com/users/w8rbt/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-10-11T21:05:07Z","updated_at":"2018-10-12T15:40:00Z","closed_at":null,"author_association":"NONE","body":"I'm using coredns 1.2.2 with the built-in file plugin (https://github.com/coredns/coredns/tree/master/plugin/file) to serve a few RFC 1035 master zone files. coredns uses github.com/miekg/dns. I'm also using the more resent version of Go (1.11.1). \r\n\r\nIn this configuration, coredns memory usage grows steadily and crashes after about 18 hours with an OOM error with a server that contains 2GB of memory. Here's a snippet from the coredns log file showing the crash:\r\n\r\n2018/10/10 13:56:17 [INFO] CoreDNS-1.2.2\r\n2018/10/10 13:56:17 [INFO] linux/amd64, go1.11.1, 8a9c6174\r\n...\r\n 1360 fatal error: runtime: out of memory\r\n 1361 \r\n 1362 runtime stack:\r\n 1363 runtime.throw(0x192f945, 0x16)\r\n 1364     /home/ubuntu/go1.11/src/runtime/panic.go:608 +0x72\r\n 1365 runtime.sysMap(0xc064000000, 0x4000000, 0x29ef978)\r\n 1366     /home/ubuntu/go1.11/src/runtime/mem_linux.go:156 +0xc7\r\n 1367 runtime.(*mheap).sysAlloc(0x29d59a0, 0x4000000, 0x29d59b8, 0x7fd3ecb43a28)\r\n 1368     /home/ubuntu/go1.11/src/runtime/malloc.go:619 +0x1c7\r\n 1369 runtime.(*mheap).grow(0x29d59a0, 0x1, 0x0)\r\n 1370     /home/ubuntu/go1.11/src/runtime/mheap.go:920 +0x42\r\n 1371 runtime.(*mheap).allocSpanLocked(0x29d59a0, 0x1, 0x29ef988, 0x0)\r\n 1372     /home/ubuntu/go1.11/src/runtime/mheap.go:848 +0x337\r\n 1373 runtime.(*mheap).alloc_m(0x29d59a0, 0x1, 0x430008, 0xc000044500)\r\n 1374     /home/ubuntu/go1.11/src/runtime/mheap.go:692 +0x119\r\n 1375 runtime.(*mheap).alloc.func1()\r\n 1376     /home/ubuntu/go1.11/src/runtime/mheap.go:759 +0x4c\r\n 1377 runtime.(*mheap).alloc(0x29d59a0, 0x1, 0xc000010008, 0x432339)\r\n 1378     /home/ubuntu/go1.11/src/runtime/mheap.go:758 +0x8a\r\n 1379 runtime.(*mcentral).grow(0x29d6ed8, 0x0)\r\n 1380     /home/ubuntu/go1.11/src/runtime/mcentral.go:232 +0x94\r\n 1381 runtime.(*mcentral).cacheSpan(0x29d6ed8, 0xc063aef0d8)\r\n 1382     /home/ubuntu/go1.11/src/runtime/mcentral.go:106 +0x2f8\r\n 1383 runtime.(*mcache).refill(0x7fd3f5229000, 0xc0000a7b08)\r\n 1384     /home/ubuntu/go1.11/src/runtime/mcache.go:122 +0x95\r\n 1385 runtime.(*mcache).nextFree.func1()\r\n 1386     /home/ubuntu/go1.11/src/runtime/malloc.go:749 +0x32\r\n 1387 runtime.systemstack(0xc00008cd00)\r\n 1388     /home/ubuntu/go1.11/src/runtime/asm_amd64.s:351 +0x66\r\n 1389 runtime.mstart()\r\n 1390     /home/ubuntu/go1.11/src/runtime/proc.go:1229\r\n 1392 goroutine 4058 [running]:\r\n 1393 runtime.systemstack_switch()\r\n 1394     /home/ubuntu/go1.11/src/runtime/asm_amd64.s:311 fp=0xc0620b7508 sp=0xc0620b7500 pc=0x45a340\r\n 1395 runtime.(*mcache).nextFree(0x7fd3f5229000, 0x8, 0xc06370d3c0, 0x0, 0xc000146a80)\r\n 1396     /home/ubuntu/go1.11/src/runtime/malloc.go:748 +0xb6 fp=0xc0620b7560 sp=0xc0620b7508 pc=0x40b7b6\r\n 1397 runtime.mallocgc(0x30, 0x176dfc0, 0x1, 0xc06370d3c0)\r\n 1398     /home/ubuntu/go1.11/src/runtime/malloc.go:903 +0x793 fp=0xc0620b7600 sp=0xc0620b7560 pc=0x40c103\r\n 1399 runtime.newobject(0x176dfc0, 0x446d56)\r\n 1400     /home/ubuntu/go1.11/src/runtime/malloc.go:1032 +0x38 fp=0xc0620b7630 sp=0xc0620b7600 pc=0x40c4e8\r\n 1401 github.com/miekg/dns.setCNAME(0xc063ff3770, 0x23, 0x3c00010005, 0x0, 0xc063aef080, 0xc0000a1d86, 0x12, 0xc000092b40, 0x22, 0x23, ...)\r\n 1402     /home/ubuntu/go/src/github.com/miekg/dns/scan_rr.go:515 +0x3f fp=0xc0620b7730 sp=0xc0620b7630 pc=0x71d01f\r\n 1403 github.com/miekg/dns.setRR(0xc063ff3770, 0x23, 0x3c00010005, 0x0, 0xc063aef080, 0xc0000a1d86, 0x12, 0xc000092b40, 0x22, 0x1b68b80, ...)\r\n 1404     /home/ubuntu/go/src/github.com/miekg/dns/scan_rr.go:29 +0xdd fp=0xc0620b78c0 sp=0xc0620b7730 pc=0x71731d\r\n 1405 github.com/miekg/dns.parseZone(0x1b44740, 0xc00000cea0, 0xc0000a1d86, 0x12, 0xc000092b40, 0x22, 0xc014998348, 0xc063256180, 0x0)\r\n 1406     /home/ubuntu/go/src/github.com/miekg/dns/scan.go:469 +0x1fdc fp=0xc0620b7f98 sp=0xc0620b78c0 pc=0x71292c\r\n 1407 runtime.goexit()\r\n 1408     /home/ubuntu/go1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0620b7fa0 sp=0xc0620b7f98 pc=0x45c2a1\r\n 1409 created by github.com/miekg/dns.parseZoneHelper\r\n 1410     /home/ubuntu/go/src/github.com/miekg/dns/scan.go:162 +0xc3\r\n\r\nI enabled profiling in the coredns config file and have been watching the heap. Here are two examples showing this issue. Notice that  github.com/miekg/dns.parseZone grows continually by about 100MB each hour:\r\n\r\nFile: coredns\r\nType: inuse_space\r\nTime: Oct 11, 2018 at 3:43pm (EDT)\r\nEntering interactive mode (type \"help\" for commands, \"o\" for options)\r\n(pprof) top5\r\nShowing nodes accounting for 385.64MB, 95.53% of 403.68MB total\r\nDropped 11 nodes (cum <= 2.02MB)\r\nShowing top 5 nodes out of 23\r\n      flat  flat%   sum%        cum   cum%\r\n     106MB 26.26% 26.26%   303.02MB 75.06%  github.com/miekg/dns.parseZone\r\n   98.50MB 24.40% 50.66%    98.50MB 24.40%  github.com/miekg/dns.appendOrigin (inline)\r\n      98MB 24.28% 74.94%       98MB 24.28%  github.com/miekg/dns.setCNAME\r\n      68MB 16.85% 91.78%       68MB 16.85%  github.com/miekg/dns.zlexer\r\n   15.12MB  3.75% 95.53%    15.12MB  3.75%  github.com/miekg/dns.parseZoneHelper\r\n\r\nFile: coredns\r\nType: inuse_space\r\nTime: Oct 11, 2018 at 4:58pm (EDT)\r\nEntering interactive mode (type \"help\" for commands, \"o\" for options)\r\n(pprof) top5\r\nShowing nodes accounting for 512.96MB, 96.51% of 531.51MB total\r\nDropped 11 nodes (cum <= 2.66MB)\r\nShowing top 5 nodes out of 23\r\n      flat  flat%   sum%        cum   cum%\r\n  138.01MB 25.96% 25.96%   138.01MB 25.96%  github.com/miekg/dns.appendOrigin (inline)\r\n  137.51MB 25.87% 51.84%   404.02MB 76.01%  github.com/miekg/dns.parseZone\r\n  127.51MB 23.99% 75.83%   127.51MB 23.99%  github.com/miekg/dns.setCNAME\r\n   90.50MB 17.03% 92.85%    90.50MB 17.03%  github.com/miekg/dns.zlexer\r\n\r\nHere is the coredns config file I am using: \r\n[coredns.conf.txt](https://github.com/miekg/dns/files/2470687/coredns.conf.txt)\r\n\r\nThe zone files being served have about 100,000 records.\r\n\r\nThanks,\r\n\r\nBrad\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/miekg/dns/issues/comments/429368908","html_url":"https://github.com/miekg/dns/issues/786#issuecomment-429368908","issue_url":"https://api.github.com/repos/miekg/dns/issues/786","id":429368908,"node_id":"MDEyOklzc3VlQ29tbWVudDQyOTM2ODkwOA==","user":{"login":"w8rbt","id":9010108,"node_id":"MDQ6VXNlcjkwMTAxMDg=","avatar_url":"https://avatars0.githubusercontent.com/u/9010108?v=4","gravatar_id":"","url":"https://api.github.com/users/w8rbt","html_url":"https://github.com/w8rbt","followers_url":"https://api.github.com/users/w8rbt/followers","following_url":"https://api.github.com/users/w8rbt/following{/other_user}","gists_url":"https://api.github.com/users/w8rbt/gists{/gist_id}","starred_url":"https://api.github.com/users/w8rbt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/w8rbt/subscriptions","organizations_url":"https://api.github.com/users/w8rbt/orgs","repos_url":"https://api.github.com/users/w8rbt/repos","events_url":"https://api.github.com/users/w8rbt/events{/privacy}","received_events_url":"https://api.github.com/users/w8rbt/received_events","type":"User","site_admin":false},"created_at":"2018-10-12T15:40:00Z","updated_at":"2018-10-12T15:40:00Z","author_association":"NONE","body":"OK, thanks again @tmthrgd, I've rebuilt with the change. Will see how it does over the weekend. Appreciate your help."}}
{"actor":{"display_login":"w8rbt"}}