{"action":"created","issue":{"url":"https://api.github.com/repos/INL/BlackLab/issues/14","repository_url":"https://api.github.com/repos/INL/BlackLab","labels_url":"https://api.github.com/repos/INL/BlackLab/issues/14/labels{/name}","comments_url":"https://api.github.com/repos/INL/BlackLab/issues/14/comments","events_url":"https://api.github.com/repos/INL/BlackLab/issues/14/events","html_url":"https://github.com/INL/BlackLab/issues/14","id":135701060,"number":14,"title":"Two Performance Issues During the Retrieval Time","user":{"login":"languagerecipes","id":8106003,"avatar_url":"https://avatars.githubusercontent.com/u/8106003?v=3","gravatar_id":"","url":"https://api.github.com/users/languagerecipes","html_url":"https://github.com/languagerecipes","followers_url":"https://api.github.com/users/languagerecipes/followers","following_url":"https://api.github.com/users/languagerecipes/following{/other_user}","gists_url":"https://api.github.com/users/languagerecipes/gists{/gist_id}","starred_url":"https://api.github.com/users/languagerecipes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/languagerecipes/subscriptions","organizations_url":"https://api.github.com/users/languagerecipes/orgs","repos_url":"https://api.github.com/users/languagerecipes/repos","events_url":"https://api.github.com/users/languagerecipes/events{/privacy}","received_events_url":"https://api.github.com/users/languagerecipes/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2016-02-23T10:45:41Z","updated_at":"2016-02-24T09:00:20Z","closed_at":null,"body":"Hello,\r\n\r\nI use the following code for fetching concordances from a relatively large index (120G):\r\n\r\n\r\n```\r\nHits hits = searcher.find(some_query);\r\n\r\nif (hits.doneFetchingHits()) {\r\n            System.out.println(\"#hits: \" + hits.countSoFarHitsCounted());\r\n            System.out.println(\"#doc: \" + hits.countSoFarDocsCounted());\r\n}\r\n\r\n```\r\nFor example, for a query I see this output:\r\n\r\n```\r\n#hits: 49050\r\n#doc: 39587\r\n\r\n```\r\nI can this result fast (in a few seconds).\r\n\r\nThen, I use the following code to fetch the results and do some process (e.g., a simple file print):\r\n\r\n```\r\n\r\nfor (Hit hit : hits) {\r\nKwic kwic = hits.getKwic(hit, CONTEXT_WINDOW_SIZE);\r\ndoSomeProcessOnKwic(kwic.getFullXml())\r\n}\r\n\r\n```\r\n\r\nFor a small `CONTEXT_WINDOW_SIZE`, i.e.,` CONTEXT_WINDOW_SIZE` <=3, the IO performs is ok, i.e., the time spent for retrieving the first kwic is not noticeable. In fact, the first few thousands  `hit` can be retrieved fast. However, (i.e., **ISSUE 1**) this time grows when retrieving more `hit` (e.g., ranked at 6k and above).\r\n\r\nIn addition, (**ISSUE 2**), for `CONTEXT_WINDOW_SIZE`>3, the performance drops drastically and the retrieval time grows rather exponential that makes the indexing almost usefulness: that is to say, in the same amount of time (or less), the original vertical text can be parsed to find the required data.\r\n\r\nMy first intuition was that this is due to some memory problem, but these issues persist even if run the code with a heap size such as 50GB. \r\n\r\nCould you please tell me what do I do wrong? \r\n\r\nThanks!\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/INL/BlackLab/issues/comments/188147146","html_url":"https://github.com/INL/BlackLab/issues/14#issuecomment-188147146","issue_url":"https://api.github.com/repos/INL/BlackLab/issues/14","id":188147146,"user":{"login":"languagerecipes","id":8106003,"avatar_url":"https://avatars.githubusercontent.com/u/8106003?v=3","gravatar_id":"","url":"https://api.github.com/users/languagerecipes","html_url":"https://github.com/languagerecipes","followers_url":"https://api.github.com/users/languagerecipes/followers","following_url":"https://api.github.com/users/languagerecipes/following{/other_user}","gists_url":"https://api.github.com/users/languagerecipes/gists{/gist_id}","starred_url":"https://api.github.com/users/languagerecipes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/languagerecipes/subscriptions","organizations_url":"https://api.github.com/users/languagerecipes/orgs","repos_url":"https://api.github.com/users/languagerecipes/repos","events_url":"https://api.github.com/users/languagerecipes/events{/privacy}","received_events_url":"https://api.github.com/users/languagerecipes/received_events","type":"User","site_admin":false},"created_at":"2016-02-24T09:00:20Z","updated_at":"2016-02-24T09:00:20Z","body":"The latest solution I cam up with (in order to be able to run the code with not so big mem is):\r\n\r\n  for (int i = 0; i < hits.size(); i++) {\r\n            Hit h = hits.get(i);\r\n            DocContentsFromForwardIndex contentFromForwardIndex = \r\n                    searcher.getContentFromForwardIndex(h.doc, (h.start-5), h.end +5);\r\n}\r\n\r\nWhat do you think?\r\n"}}