{"action":"created","issue":{"url":"https://api.github.com/repos/flarum/core/issues/1042","repository_url":"https://api.github.com/repos/flarum/core","labels_url":"https://api.github.com/repos/flarum/core/issues/1042/labels{/name}","comments_url":"https://api.github.com/repos/flarum/core/issues/1042/comments","events_url":"https://api.github.com/repos/flarum/core/issues/1042/events","html_url":"https://github.com/flarum/core/issues/1042","id":178991214,"number":1042,"title":"Search stops working properly on large forums","user":{"login":"marcozambi","id":1640859,"avatar_url":"https://avatars.githubusercontent.com/u/1640859?v=3","gravatar_id":"","url":"https://api.github.com/users/marcozambi","html_url":"https://github.com/marcozambi","followers_url":"https://api.github.com/users/marcozambi/followers","following_url":"https://api.github.com/users/marcozambi/following{/other_user}","gists_url":"https://api.github.com/users/marcozambi/gists{/gist_id}","starred_url":"https://api.github.com/users/marcozambi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcozambi/subscriptions","organizations_url":"https://api.github.com/users/marcozambi/orgs","repos_url":"https://api.github.com/users/marcozambi/repos","events_url":"https://api.github.com/users/marcozambi/events{/privacy}","received_events_url":"https://api.github.com/users/marcozambi/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/flarum/core/labels/Bug","name":"Bug","color":"eb6420"},{"url":"https://api.github.com/repos/flarum/core/labels/Performance","name":"Performance","color":"eeeeee"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-09-23T23:04:49Z","updated_at":"2016-10-08T07:57:54Z","closed_at":null,"body":"After having imported about 18000 discussions and 256000 posts from my existing SMF2 forum (I wrote a specific tool for that) I noticed that all the searches I was launching were generating several server Error 500. Flarum became unresponsive then for several minutes, despite many tries to simply refresh the home page.\r\nInitially I started small import tests, taking in only few hundred then thousands of posts, and every was working very well.\r\nWhen the number of imported posts goes above 60000 the search functionality starts to slow down considerably. At 100000 imported posts, every search using a single term reach a good end only after at least 10 to 20 seconds of waiting, while *99%* search with 2 or more words fails with an Error 500.\r\n\r\nI ran numerous tests trying to understand whether it could merely be a configuration issue on my server's side, but after having performed a flarum installation and a subsequent data import on a second sever (double the \"power\" of the first one, 8 GB of RAM, quad core, OS ubuntu 16.04, 60 gb ssd disk) and having run in the same exact problems, I am now convinced that there are some too-greedy or inefficient queries which are literally killing the server.\r\n\r\nI have documented in detail my experiments here [on Flarum Community](https://discuss.flarum.org/d/3758-flarum-search-functionality-error-500/13)\r\n\r\nAn example of the query creating the problem is visible here [flarum_error_500.txt](https://dl.dropboxusercontent.com/u/25636589/flarum_error_500.txt)\r\n\r\nI would be more than happy to help either doing more tests or receiving a few tips to understand the mechanisms and read the source code for the search functionality in Flarum."},"comment":{"url":"https://api.github.com/repos/flarum/core/issues/comments/252410295","html_url":"https://github.com/flarum/core/issues/1042#issuecomment-252410295","issue_url":"https://api.github.com/repos/flarum/core/issues/1042","id":252410295,"user":{"login":"marcozambi","id":1640859,"avatar_url":"https://avatars.githubusercontent.com/u/1640859?v=3","gravatar_id":"","url":"https://api.github.com/users/marcozambi","html_url":"https://github.com/marcozambi","followers_url":"https://api.github.com/users/marcozambi/followers","following_url":"https://api.github.com/users/marcozambi/following{/other_user}","gists_url":"https://api.github.com/users/marcozambi/gists{/gist_id}","starred_url":"https://api.github.com/users/marcozambi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcozambi/subscriptions","organizations_url":"https://api.github.com/users/marcozambi/orgs","repos_url":"https://api.github.com/users/marcozambi/repos","events_url":"https://api.github.com/users/marcozambi/events{/privacy}","received_events_url":"https://api.github.com/users/marcozambi/received_events","type":"User","site_admin":false},"created_at":"2016-10-08T07:57:54Z","updated_at":"2016-10-08T07:57:54Z","body":"Thanks Franz, actually you're perfectly right. The added OFFSET solution is perfect because hardcoding an arbitrary maxmimum number of results excludes a large number of results in large forums with hundreds of thousands of discussions.\r\nA \"buffered\" solution, which returns a relatively large number of results with the first query, then followed by more OFFSET-ed queries if necessary could be considered. This way further queries (with OFFSET serr) are triggered only if the user has gone through the whole set of initial returned entries without finding what he/she was looking for.\r\n"}}
{"actor":{"display_login":"marcozambi"}}