{"action":"created","issue":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/300","repository_url":"https://api.github.com/repos/Norconex/collector-http","labels_url":"https://api.github.com/repos/Norconex/collector-http/issues/300/labels{/name}","comments_url":"https://api.github.com/repos/Norconex/collector-http/issues/300/comments","events_url":"https://api.github.com/repos/Norconex/collector-http/issues/300/events","html_url":"https://github.com/Norconex/collector-http/issues/300","id":176274988,"number":300,"title":"OutOfMemoryError on fetch stage","user":{"login":"nemojjh","id":19145034,"avatar_url":"https://avatars.githubusercontent.com/u/19145034?v=3","gravatar_id":"","url":"https://api.github.com/users/nemojjh","html_url":"https://github.com/nemojjh","followers_url":"https://api.github.com/users/nemojjh/followers","following_url":"https://api.github.com/users/nemojjh/following{/other_user}","gists_url":"https://api.github.com/users/nemojjh/gists{/gist_id}","starred_url":"https://api.github.com/users/nemojjh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nemojjh/subscriptions","organizations_url":"https://api.github.com/users/nemojjh/orgs","repos_url":"https://api.github.com/users/nemojjh/repos","events_url":"https://api.github.com/users/nemojjh/events{/privacy}","received_events_url":"https://api.github.com/users/nemojjh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-12T00:38:46Z","updated_at":"2016-09-20T01:04:40Z","closed_at":null,"body":"```\r\njava.lang.OutOfMemoryError: Java heap space\r\n        at com.norconex.commons.lang.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:357)\r\n        at com.norconex.commons.lang.io.CachedInputStream.cacheToFile(CachedInputStream.java:527)\r\n        at com.norconex.commons.lang.io.CachedInputStream.realRead(CachedInputStream.java:325)\r\n        at com.norconex.commons.lang.io.CachedInputStream.read(CachedInputStream.java:301)\r\n        at java.io.InputStream.read(InputStream.java:101)\r\n        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2146)\r\n        at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102)\r\n        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123)\r\n        at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078)\r\n        at com.norconex.collector.http.fetch.impl.GenericDocumentFetcher.fetchDocument(GenericDocumentFetcher.java:138)\r\n        at com.norconex.collector.http.pipeline.importer.DocumentFetcherStage.executeStage(DocumentFetcherStage.java:50)\r\n        at com.norconex.collector.http.pipeline.importer.AbstractImporterStage.execute(AbstractImporterStage.java:31)\r\n        at com.norconex.collector.http.pipeline.importer.AbstractImporterStage.execute(AbstractImporterStage.java:24)\r\n        at com.norconex.commons.lang.pipeline.Pipeline.execute(Pipeline.java:90)\r\n        at com.norconex.collector.http.crawler.HttpCrawler.executeImporterPipeline(HttpCrawler.java:300)\r\n        at com.norconex.collector.core.crawler.AbstractCrawler.processNextQueuedCrawlData(AbstractCrawler.java:488)\r\n        at com.norconex.collector.core.crawler.AbstractCrawler.processNextReference(AbstractCrawler.java:378)\r\n        at com.norconex.collector.core.crawler.AbstractCrawler$ProcessReferencesRunnable.run(AbstractCrawler.java:736)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n```\r\n\r\nI wrote code that runs crawlers periodically.\r\n\r\nThe code runs new 50 Crawler Threads(50 Web Site)on every hour.\r\n\r\nBut It occurs this error when It had run for 3~4 days.\r\n\r\nThank you.\r\n"},"comment":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/comments/248173285","html_url":"https://github.com/Norconex/collector-http/issues/300#issuecomment-248173285","issue_url":"https://api.github.com/repos/Norconex/collector-http/issues/300","id":248173285,"user":{"login":"nemojjh","id":19145034,"avatar_url":"https://avatars.githubusercontent.com/u/19145034?v=3","gravatar_id":"","url":"https://api.github.com/users/nemojjh","html_url":"https://github.com/nemojjh","followers_url":"https://api.github.com/users/nemojjh/followers","following_url":"https://api.github.com/users/nemojjh/following{/other_user}","gists_url":"https://api.github.com/users/nemojjh/gists{/gist_id}","starred_url":"https://api.github.com/users/nemojjh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nemojjh/subscriptions","organizations_url":"https://api.github.com/users/nemojjh/orgs","repos_url":"https://api.github.com/users/nemojjh/repos","events_url":"https://api.github.com/users/nemojjh/events{/privacy}","received_events_url":"https://api.github.com/users/nemojjh/received_events","type":"User","site_admin":false},"created_at":"2016-09-20T01:04:40Z","updated_at":"2016-09-20T01:04:40Z","body":"I tested it for 7 days.\r\n\r\nAnd.. Seems that found a solution. :)\r\n\r\n--Env--\r\nCrawler Thread Count : 25 Instance * 5 Thread\r\nDelay : 100ms\r\nOS : Debian 7\r\nJava : java7, -Xmx10G\r\n\r\nSwiping Code after 1 cycle finished:\r\n```\r\nMBeanServer mbs = ManagementFactory.getPlatformMBeanServer();\r\nmbs.unregisterMBean(new ObjectName(\"com.norconex.collector.http.crawler:type=\" +id));\r\n```\r\n\r\nUnregistered crawler descriptor from MBeanServer for gc(maybe?).\r\n\r\nThen now it doesn't occur OOM.\r\n\r\nI'm not sure it is a proper solution but it works on my case.\r\n\r\nThank you :)"}}
{"actor":{"display_login":"nemojjh"}}