{"action":"opened","issue":{"url":"https://api.github.com/repos/ViDA-NYU/ache/issues/62","repository_url":"https://api.github.com/repos/ViDA-NYU/ache","labels_url":"https://api.github.com/repos/ViDA-NYU/ache/issues/62/labels{/name}","comments_url":"https://api.github.com/repos/ViDA-NYU/ache/issues/62/comments","events_url":"https://api.github.com/repos/ViDA-NYU/ache/issues/62/events","html_url":"https://github.com/ViDA-NYU/ache/issues/62","id":194691123,"number":62,"title":"OutOfMemoryError problems during large crawls","user":{"login":"aecio","id":150570,"avatar_url":"https://avatars.githubusercontent.com/u/150570?v=3","gravatar_id":"","url":"https://api.github.com/users/aecio","html_url":"https://github.com/aecio","followers_url":"https://api.github.com/users/aecio/followers","following_url":"https://api.github.com/users/aecio/following{/other_user}","gists_url":"https://api.github.com/users/aecio/gists{/gist_id}","starred_url":"https://api.github.com/users/aecio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aecio/subscriptions","organizations_url":"https://api.github.com/users/aecio/orgs","repos_url":"https://api.github.com/users/aecio/repos","events_url":"https://api.github.com/users/aecio/events{/privacy}","received_events_url":"https://api.github.com/users/aecio/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-12-09T20:37:31Z","updated_at":"2016-12-09T20:37:31Z","closed_at":null,"body":"ACHE is stopping due to different types of `OutOfMemoryError`s being thrown during large crawls.\r\n\r\nExamples:\r\n```\r\njava.lang.OutOfMemoryError: Java heap space\r\n        at java.util.Arrays.copyOfRange(Arrays.java:3664)\r\n        at java.lang.String.<init>(String.java:201)\r\n        at java.lang.String.toLowerCase(String.java:2635)\r\n        at java.lang.String.toLowerCase(String.java:2658)\r\n        at focusedCrawler.util.parser.PaginaURL.separadorTextoCodigo(PaginaURL.java:410)\r\n        at focusedCrawler.util.parser.PaginaURL.<init>(PaginaURL.java:128)\r\n        at focusedCrawler.util.parser.PaginaURL.<init>(PaginaURL.java:110)\r\n        at focusedCrawler.util.vsm.VSMVector.<init>(VSMVector.java:167)\r\n        at focusedCrawler.target.classifier.WekaTargetClassifier.getValues(WekaTargetClassifier.java:104)\r\n        at focusedCrawler.target.classifier.WekaTargetClassifier.distributionForInstance(WekaTargetClassifier.java:90)\r\n        at focusedCrawler.target.classifier.WekaTargetClassifier.classify(WekaTargetClassifier.java:74)\r\n        at focusedCrawler.target.classifier.KeepLinkRelevanceTargetClassifier.classify(KeepLinkRelevanceTargetClassifier.java:22)\r\n        at focusedCrawler.target.TargetStorage.insert(TargetStorage.java:80)\r\n        at focusedCrawler.crawler.async.FetchedResultHandler.processData(FetchedResultHandler.java:55)\r\n        at focusedCrawler.crawler.async.FetchedResultHandler.completed(FetchedResultHandler.java:29)\r\n        at focusedCrawler.crawler.async.HttpDownloader$FetchFinishedHandler.doHandle(HttpDownloader.java:330)\r\n        at focusedCrawler.crawler.async.HttpDownloader$FetchFinishedHandler.run(HttpDownloader.java:313)\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\r\n        at java.util.concurrent.FutureTask.run(FutureTask.java:266)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\n```\r\njava.lang.OutOfMemoryError: GC overhead limit exceeded\r\n        at java.lang.String.substring(String.java:1957)\r\n        at java.lang.String.split(String.java:2341)\r\n        at java.lang.String.split(String.java:2410)\r\n        at focusedCrawler.link.BipartiteGraphRepository.getLNs(BipartiteGraphRepository.java:104)\r\n        at focusedCrawler.link.OnlineLearning.forwardClassifier(OnlineLearning.java:233)\r\n        at focusedCrawler.link.OnlineLearning.execute(OnlineLearning.java:64)\r\n        at focusedCrawler.link.LinkStorage.insert(LinkStorage.java:183)\r\n        at focusedCrawler.link.LinkStorage.insert(LinkStorage.java:124)\r\n        at focusedCrawler.target.TargetStorage.insert(TargetStorage.java:98)\r\n        at focusedCrawler.crawler.async.FetchedResultHandler.processData(FetchedResultHandler.java:55)\r\n        at focusedCrawler.crawler.async.FetchedResultHandler.completed(FetchedResultHandler.java:29)\r\n        at focusedCrawler.crawler.async.HttpDownloader$FetchFinishedHandler.doHandle(HttpDownloader.java:330)\r\n        at focusedCrawler.crawler.async.HttpDownloader$FetchFinishedHandler.run(HttpDownloader.java:313)\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\r\n        at java.util.concurrent.FutureTask.run(FutureTask.java:266)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\n```\r\njava.lang.OutOfMemoryError: GC overhead limit exceeded\r\n        at com.esotericsoftware.kryo.serializers.DefaultSerializers$URLSerializer.read(DefaultSerializers.java:869)\r\n        at com.esotericsoftware.kryo.serializers.DefaultSerializers$URLSerializer.read(DefaultSerializers.java:859)\r\n        at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:782)\r\n        at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)\r\n        at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:540)\r\n        at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:709)\r\n        at focusedCrawler.util.persistence.rocksdb.RocksDBHashtable.unserializeObject(RocksDBHashtable.java:79)\r\n        at focusedCrawler.util.persistence.rocksdb.RocksDBHashtable.access$100(RocksDBHashtable.java:21)\r\n        at focusedCrawler.util.persistence.rocksdb.RocksDBHashtable$RocksDBIterator.next(RocksDBHashtable.java:172)\r\n        at focusedCrawler.util.persistence.rocksdb.RocksDBHashtable$RocksDBIterator.next(RocksDBHashtable.java:125)\r\n        at focusedCrawler.link.frontier.FrontierManager.loadQueue(FrontierManager.java:138)\r\n        at focusedCrawler.link.frontier.FrontierManager.nextURL(FrontierManager.java:231)\r\n        at focusedCrawler.link.LinkStorage.select(LinkStorage.java:213)\r\n        at focusedCrawler.crawler.async.AsyncCrawler.run(AsyncCrawler.java:60)\r\n        at focusedCrawler.Main.startCrawl(Main.java:310)\r\n        at focusedCrawler.Main.main(Main.java:113)\r\n\r\n```"}}
{"actor":{"display_login":"aecio"}}