{"action":"created","issue":{"url":"https://api.github.com/repos/LMFDB/lmfdb/issues/2423","repository_url":"https://api.github.com/repos/LMFDB/lmfdb","labels_url":"https://api.github.com/repos/LMFDB/lmfdb/issues/2423/labels{/name}","comments_url":"https://api.github.com/repos/LMFDB/lmfdb/issues/2423/comments","events_url":"https://api.github.com/repos/LMFDB/lmfdb/issues/2423/events","html_url":"https://github.com/LMFDB/lmfdb/pull/2423","id":316597963,"number":2423,"title":"Timeout Decorator for search queries","user":{"login":"edgarcosta","id":322822,"avatar_url":"https://avatars3.githubusercontent.com/u/322822?v=4","gravatar_id":"","url":"https://api.github.com/users/edgarcosta","html_url":"https://github.com/edgarcosta","followers_url":"https://api.github.com/users/edgarcosta/followers","following_url":"https://api.github.com/users/edgarcosta/following{/other_user}","gists_url":"https://api.github.com/users/edgarcosta/gists{/gist_id}","starred_url":"https://api.github.com/users/edgarcosta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edgarcosta/subscriptions","organizations_url":"https://api.github.com/users/edgarcosta/orgs","repos_url":"https://api.github.com/users/edgarcosta/repos","events_url":"https://api.github.com/users/edgarcosta/events{/privacy}","received_events_url":"https://api.github.com/users/edgarcosta/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-04-22T17:38:25Z","updated_at":"2018-04-25T02:17:00Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/LMFDB/lmfdb/pulls/2423","html_url":"https://github.com/LMFDB/lmfdb/pull/2423","diff_url":"https://github.com/LMFDB/lmfdb/pull/2423.diff","patch_url":"https://github.com/LMFDB/lmfdb/pull/2423.patch"},"body":"Hi,\r\n\r\nThis is related to #2422.\r\n\r\nWith the goal of being able to test if searches are performed in a reasonable amount of time and give a better feedback to the user whenever the query takes too long, I wrote a small script that decorates our db queries with `.max_time_ms(timeout)` and should replace the following two lines\r\n```\r\nnres = res.count()\r\nres = res.skip(start).limit(count)\r\n```\r\npresent in almost every search handling code.\r\nI have only replaced these two lines for numberfieds, but should be easy to do the same for rest of the search fields.\r\n\r\ntry it out with: \r\nhttp://localhost:37777/NumberField/?start=0&ur_primes=2%2C3%2C5%2C7&ram_quantifier=contained&count=20\r\n\r\nIn this case, the query times out while trying to do the count:\r\n```\r\n2018-04-22T17:27:03.625+0000 I COMMAND  [conn4343] command numberfields.fields command: count { count: \"fields\", query: { ramps: { $nin: [ \"2\", \"3\", \"5\", \"7\" ] } }, maxTimeMS: 60000 } planSummary: IXSCAN { ramps: 1, degree: 1 } keyUpdates:0 writeConflicts:0 exception: operation exceeded time limit code:50 numYields:2373 reslen:89 locks:{ Global: { acquireCount: { r: 4748 } }, Database: { acquireCount: { r: 2374 } }, Collection: { acquireCount: { r: 2374 } } } protocol:op_query 60662ms\r\n```\r\n\r\n\r\n\r\n\r\nThis isn't perfect.  But I think it is much more helpful to see:\r\n```\r\nError: The search query took longer than expected! Please help us improve by reporting this error here.\r\n```\r\nthan Google's `Error: Server Error`.\r\nAlso, if the server is overloaded, the timeout will not help us, as it only limits the CPU cumulative time.\r\n\r\nLet me know what you think."},"comment":{"url":"https://api.github.com/repos/LMFDB/lmfdb/issues/comments/384139868","html_url":"https://github.com/LMFDB/lmfdb/pull/2423#issuecomment-384139868","issue_url":"https://api.github.com/repos/LMFDB/lmfdb/issues/2423","id":384139868,"user":{"login":"edgarcosta","id":322822,"avatar_url":"https://avatars3.githubusercontent.com/u/322822?v=4","gravatar_id":"","url":"https://api.github.com/users/edgarcosta","html_url":"https://github.com/edgarcosta","followers_url":"https://api.github.com/users/edgarcosta/followers","following_url":"https://api.github.com/users/edgarcosta/following{/other_user}","gists_url":"https://api.github.com/users/edgarcosta/gists{/gist_id}","starred_url":"https://api.github.com/users/edgarcosta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edgarcosta/subscriptions","organizations_url":"https://api.github.com/users/edgarcosta/orgs","repos_url":"https://api.github.com/users/edgarcosta/repos","events_url":"https://api.github.com/users/edgarcosta/events{/privacy}","received_events_url":"https://api.github.com/users/edgarcosta/received_events","type":"User","site_admin":false},"created_at":"2018-04-25T02:17:00Z","updated_at":"2018-04-25T02:17:00Z","author_association":"MEMBER","body":"The sad thing is that the wrapper cannot really handle the timeout exception, and must pass an exception outside, as the outside function `whatever_search` must still return something.\r\n\r\nI will try to do the other pages over the next couple of days."}}
{"actor":{"display_login":"edgarcosta"}}