{"action":"created","issue":{"url":"https://api.github.com/repos/yacy/yacy_search_server/issues/91","repository_url":"https://api.github.com/repos/yacy/yacy_search_server","labels_url":"https://api.github.com/repos/yacy/yacy_search_server/issues/91/labels{/name}","comments_url":"https://api.github.com/repos/yacy/yacy_search_server/issues/91/comments","events_url":"https://api.github.com/repos/yacy/yacy_search_server/issues/91/events","html_url":"https://github.com/yacy/yacy_search_server/pull/91","id":190240820,"number":91,"title":"(WIP) Tradeoff between sorting accuracy and display latency","user":{"login":"JeremyRand","id":244188,"avatar_url":"https://avatars.githubusercontent.com/u/244188?v=3","gravatar_id":"","url":"https://api.github.com/users/JeremyRand","html_url":"https://github.com/JeremyRand","followers_url":"https://api.github.com/users/JeremyRand/followers","following_url":"https://api.github.com/users/JeremyRand/following{/other_user}","gists_url":"https://api.github.com/users/JeremyRand/gists{/gist_id}","starred_url":"https://api.github.com/users/JeremyRand/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JeremyRand/subscriptions","organizations_url":"https://api.github.com/users/JeremyRand/orgs","repos_url":"https://api.github.com/users/JeremyRand/repos","events_url":"https://api.github.com/users/JeremyRand/events{/privacy}","received_events_url":"https://api.github.com/users/JeremyRand/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2016-11-18T05:45:05Z","updated_at":"2016-12-07T06:27:59Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/yacy/yacy_search_server/pulls/91","html_url":"https://github.com/yacy/yacy_search_server/pull/91","diff_url":"https://github.com/yacy/yacy_search_server/pull/91.diff","patch_url":"https://github.com/yacy/yacy_search_server/pull/91.patch"},"body":"(Don't merge this PR, it's an experiment.)\r\n\r\nAt the moment, from what I can tell, YaCy waits a very short period of time, then drains exactly 1 result from the Solr and RWI queues into the output queue, and then checks whether the output queue has at least 1 result in it that hasn't yet been displayed to the user.  Naturally, this is true 100% of the time.  When YaCy sees this, it writes the best result in its output queue (i.e. either the single Solr result or the single RWI result) to the user.  It then repeats.  (See the section of code modified by this PR to see what code I'm talking about.)\r\n\r\nThe effect is that there is almost no correlation between the ordering of the results displayed to the user and the ranking calculated by YaCy; the dominant factor is which results got returned first by a peer.  Kind of useless as a sorting mechanism.\r\n\r\nThe easy hack to fix this is to add a significant delay right before the above algorithm runs, which gives the Solr and RWI queues enough time to receive a lot of results before they start being drained into the output queue.  The example code in this PR is a hacky way of doing this.\r\n\r\nHowever, the problem is that it adds a long delay before the end user sees any results at all, and there is a tradeoff between the delay and the sorting quality.\r\n\r\nMy suggestions for a better way to implement a fix than this PR:\r\n\r\n1. For API-based use cases that retrieve a single list, e.g. the XML and JSON output formats, provide a way for the user to choose how long to wait for results before the queues start being drained (probably an HTTP GET parameter is reasonable for this).  My guess is that 5 to 10 seconds is going to be enough time for an API user on a fast residential Internet connection (or at least, the first 100 results are likely to be properly sorted in such cases).  Users on Tor are likely to need more time.\r\n2. For human-user use cases where Javascript is available, load each result via AJAX and use Javascript to insert it in the correct order.  Don't stop receiving results over AJAX when the result limit is hit; keep receiving them for approximately 1 minute.  Hide the least relevant results via Javascript once more results have been received over AJAX than the result limit.\r\n3. For API-based use cases where the user is willing to call the API multiple times, re-sort the result list each time the API is called, so that the user gets more relevant results after waiting but they get some possibly-relevant results quickly.  Probably one way to do this with minimal code changes is to create a new queue, drain the entire output queue into the new queue, and then replace the output queue with the new queue.\r\n\r\nIf these suggestions are okay with you, I'm willing to try my hand at implementing them.  Or, if you have a better suggestion for how this should be handled, please feel free to let me know."},"comment":{"url":"https://api.github.com/repos/yacy/yacy_search_server/issues/comments/265366906","html_url":"https://github.com/yacy/yacy_search_server/pull/91#issuecomment-265366906","issue_url":"https://api.github.com/repos/yacy/yacy_search_server/issues/91","id":265366906,"user":{"login":"oyvinds","id":993049,"avatar_url":"https://avatars.githubusercontent.com/u/993049?v=3","gravatar_id":"","url":"https://api.github.com/users/oyvinds","html_url":"https://github.com/oyvinds","followers_url":"https://api.github.com/users/oyvinds/followers","following_url":"https://api.github.com/users/oyvinds/following{/other_user}","gists_url":"https://api.github.com/users/oyvinds/gists{/gist_id}","starred_url":"https://api.github.com/users/oyvinds/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oyvinds/subscriptions","organizations_url":"https://api.github.com/users/oyvinds/orgs","repos_url":"https://api.github.com/users/oyvinds/repos","events_url":"https://api.github.com/users/oyvinds/events{/privacy}","received_events_url":"https://api.github.com/users/oyvinds/received_events","type":"User","site_admin":false},"created_at":"2016-12-07T06:27:59Z","updated_at":"2016-12-07T06:27:59Z","body":"DuckDuckGo's normal search result page is https://duckduckgo.com/?q=javascript&ia=web with JavaScript everything like that. It works fine on desktops.\r\n\r\nFor mobile devices with a weak CPU or a small screen and old computers there is  https://duckduckgo.com/lite/?q=mobilefriendly which has no JavaScript and basically just the search results.\r\n\r\nI think it would be easier to get everyone on board with heavy client-side JavaScript if YaCy had a similar solution with two different pages. Like http://yoona.everdot.org/yacysearch.html?query=test for desktop search and http://yoona.everdot.org/litesearch.html?query=test or something. If not /litesearch.html then perhaps /lite/yacysearch.html but you get the idea.\r\n\r\nAs for waiting 20 seconds before getting search results, such a solution would have to be configurable. Some would want 4 seconds and others 20. I am just saying that I would rather wait and have the results sorted than have results presented in a seemingly random order. YaCy is so hard to install and setup anyway that everyone who goes through all that would be able to configure such a setting.\r\n\r\n Jeremy, thank you for your hard work on this. "}}
{"actor":{"display_login":"oyvinds"}}