{"action":"created","comment":{"url":"https://api.github.com/repos/mozilla/addons-server/pulls/comments/199550970","pull_request_review_id":133690546,"id":199550970,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5OTU1MDk3MA==","diff_hunk":"@@ -0,0 +1,92 @@\n+.. _search:\n+\n+============================\n+How does search on AMO work?\n+============================\n+\n+.. note::\n+\n+  This is documenting our current state of how search is implemented in addons-server.\n+  We will be using this to plan future improvements so please note that we are\n+  aware that the process written below is not perfect and has bugs here and there.\n+\n+  Please see https://github.com/orgs/mozilla/projects/17#card-10287357 for more planning.\n+\n+\n+General structure\n+=================\n+\n+Our search contains these data types:\n+\n+ * addons (`Add-ons indexer / serializer <https://github.com/mozilla/addons-server/blob/master/src/olympia/addons/indexers.py#L22-L379>`_)\n+\n+  - versions (`Indexer / Serializer <https://github.com/mozilla/addons-server/blob/master/src/olympia/addons/indexers.py#L215-L237>`_)\n+\n+   + files\n+   + compatibility information\n+\n+  - authors\n+  - previews (image links)\n+  - translations (`Translations mapping generation <https://github.com/mozilla/addons-server/blob/master/src/olympia/amo/indexers.py#L40-L136>`_)\n+  - … various other add-on related properties\n+\n+ * add-on compatibility reports\n+\n+Our search can be reached either via the API through :ref:`/addons/search/ <addon-search>` or :ref:`/addons/autocomplete/ <addon-autocomplete>` which are used by our addons-frontend as well as via our legacy pages (used much less).\n+\n+Both use similar filtering and scoring code. For legacy reasons they're not identical though, we should try to focus on our API-based search though since the legacy search will be removed once support for Thunderbird and Seamonkey will be moved to a new platform.\n+\n+The legacy search uses ElasticSearch to query the data and then requests the actual model objects from the database. The newer API-based search only hits ElasticSearch and uses data directly stored from ES which is much more efficient.\n+\n+\n+Flow of a search query through AMO\n+==================================\n+\n+Let's assume we search on addons-frontend (not legacy) the search query hits the API and get's handled by `AddonSearchView` which directly queries ElasticSearch and doesn't involve the database at all.\n+\n+There are a few filters that are described in the :ref:`/addons/search/ docs <addon-search>` but most of them are not very relevant for raw search queries. Examplex are filters by guid, platform, category or add-on type.\n+\n+Much more relevant for raw add-on searches (And this is primarily used when you use the search on the frontend) is `SearchQueryFilter`.\n+\n+It composes various rules to define a more or less usable ranking:\n+\n+\n+Primary rules\n+  These are the ones using the strongest boosts, so they are only applied\n+  to a specific set of fields like the name, the slug and authors.\n+\n+  Applied rules (merged via *should*):\n+\n+  1. Prefer `term` matches on `name.raw` (boost=100) - our attempt to implement exact matches\n+  2. Prefer phrase matches that allows swapped terms (boost=4, slop=1)\n+  3. If a query is < 20 characters long, try to do a fuzzy match on the search query (boost=2, prefix_length=4, fuzziness=AUTO)\n+  4. Then text matches, using the standard text analyzer (boost=3, analyzer=standard, operator=and)\n+  5. Then text matches, using a language specific analyzer (boost=2.5)\n+  6. Then look for the query as a prefix(boost=1.5)\n+  7. If we have a matching analyzer, add a query to `name_l10n_{LANG}` (boost=2.5, operator=and)\n+\n+  Rules 4, 5 and 6 are added for `name` and `listed_authors.name`.\n+\n+Secondary rules\n+  These are the ones using the weakest boosts, they are applied to fields\n+  containing more text like description, summary and tags.\n+\n+  Applied rules:\n+\n+  1. Look for phrase matches inside the summary (boost=0.8)\n+  2. Look for phrase matches inside the summary using language specific\n+     analyzer (boost=0.6)\n+  3. Look for phrase matches inside the description (boost=0.3).\n+  4. Look for phrase matches inside the description using language\n+     specific analyzer (boost=0.6).\n+  5. Look for matches inside tags (boost=0.1).\n+  6. Append a separate 'match' query for every word to boost tag matches (boost=0.1)\n+\n+\n+General query flow:","path":"docs/topics/development/search.rst","position":86,"original_position":86,"commit_id":"cd2a744dde9f3dc503f653668b545e531dde125b","original_commit_id":"cd2a744dde9f3dc503f653668b545e531dde125b","user":{"login":"diox","id":187006,"node_id":"MDQ6VXNlcjE4NzAwNg==","avatar_url":"https://avatars0.githubusercontent.com/u/187006?v=4","gravatar_id":"","url":"https://api.github.com/users/diox","html_url":"https://github.com/diox","followers_url":"https://api.github.com/users/diox/followers","following_url":"https://api.github.com/users/diox/following{/other_user}","gists_url":"https://api.github.com/users/diox/gists{/gist_id}","starred_url":"https://api.github.com/users/diox/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diox/subscriptions","organizations_url":"https://api.github.com/users/diox/orgs","repos_url":"https://api.github.com/users/diox/repos","events_url":"https://api.github.com/users/diox/events{/privacy}","received_events_url":"https://api.github.com/users/diox/received_events","type":"User","site_admin":false},"body":"And another subheader.","created_at":"2018-07-02T16:20:47Z","updated_at":"2018-07-02T16:21:54Z","html_url":"https://github.com/mozilla/addons-server/pull/8713#discussion_r199550970","pull_request_url":"https://api.github.com/repos/mozilla/addons-server/pulls/8713","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/mozilla/addons-server/pulls/comments/199550970"},"html":{"href":"https://github.com/mozilla/addons-server/pull/8713#discussion_r199550970"},"pull_request":{"href":"https://api.github.com/repos/mozilla/addons-server/pulls/8713"}}},"pull_request":{"url":"https://api.github.com/repos/mozilla/addons-server/pulls/8713","id":198639928,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk4NjM5OTI4","html_url":"https://github.com/mozilla/addons-server/pull/8713","diff_url":"https://github.com/mozilla/addons-server/pull/8713.diff","patch_url":"https://github.com/mozilla/addons-server/pull/8713.patch","issue_url":"https://api.github.com/repos/mozilla/addons-server/issues/8713","number":8713,"state":"open","locked":false,"title":"Document how our search works.","user":{"login":"EnTeQuAk","id":139033,"node_id":"MDQ6VXNlcjEzOTAzMw==","avatar_url":"https://avatars1.githubusercontent.com/u/139033?v=4","gravatar_id":"","url":"https://api.github.com/users/EnTeQuAk","html_url":"https://github.com/EnTeQuAk","followers_url":"https://api.github.com/users/EnTeQuAk/followers","following_url":"https://api.github.com/users/EnTeQuAk/following{/other_user}","gists_url":"https://api.github.com/users/EnTeQuAk/gists{/gist_id}","starred_url":"https://api.github.com/users/EnTeQuAk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EnTeQuAk/subscriptions","organizations_url":"https://api.github.com/users/EnTeQuAk/orgs","repos_url":"https://api.github.com/users/EnTeQuAk/repos","events_url":"https://api.github.com/users/EnTeQuAk/events{/privacy}","received_events_url":"https://api.github.com/users/EnTeQuAk/received_events","type":"User","site_admin":false},"body":"Fixes #8090\r\n\r\nThis is simply stating the facts about how search currently works. This doesn't mention any improvement ideas in the docs directly.\r\n\r\nIn addition to that, personal improvement ideas:\r\n\r\n* Remove `listed_authors.name` from the scoring, make search against authors only possible via separate field queries, e.g `Search addons author:Chris` or so. This may simplify scoring against the add-on name a lot\r\n* https://github.com/mozilla/addons-server/issues/6815 (Remove prefix boosting to simplify scoring)\r\n* Maybe just use https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html instead of all our custom rules here and there?\r\n* Remove negative boosts https://github.com/mozilla/addons-server/issues/6838","created_at":"2018-07-02T10:42:52Z","updated_at":"2018-07-02T16:21:54Z","closed_at":null,"merged_at":null,"merge_commit_sha":"5aa7f839d0fc9a8bb227594b0f826479392ab472","assignee":null,"assignees":[],"requested_reviewers":[{"login":"muffinresearch","id":1514,"node_id":"MDQ6VXNlcjE1MTQ=","avatar_url":"https://avatars1.githubusercontent.com/u/1514?v=4","gravatar_id":"","url":"https://api.github.com/users/muffinresearch","html_url":"https://github.com/muffinresearch","followers_url":"https://api.github.com/users/muffinresearch/followers","following_url":"https://api.github.com/users/muffinresearch/following{/other_user}","gists_url":"https://api.github.com/users/muffinresearch/gists{/gist_id}","starred_url":"https://api.github.com/users/muffinresearch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/muffinresearch/subscriptions","organizations_url":"https://api.github.com/users/muffinresearch/orgs","repos_url":"https://api.github.com/users/muffinresearch/repos","events_url":"https://api.github.com/users/muffinresearch/events{/privacy}","received_events_url":"https://api.github.com/users/muffinresearch/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/mozilla/addons-server/pulls/8713/commits","review_comments_url":"https://api.github.com/repos/mozilla/addons-server/pulls/8713/comments","review_comment_url":"https://api.github.com/repos/mozilla/addons-server/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mozilla/addons-server/issues/8713/comments","statuses_url":"https://api.github.com/repos/mozilla/addons-server/statuses/cd2a744dde9f3dc503f653668b545e531dde125b","head":{"label":"mozilla:8090-search-docs","ref":"8090-search-docs","sha":"cd2a744dde9f3dc503f653668b545e531dde125b","user":{"login":"mozilla","id":131524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzMTUyNA==","avatar_url":"https://avatars2.githubusercontent.com/u/131524?v=4","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"repo":{"id":16416867,"node_id":"MDEwOlJlcG9zaXRvcnkxNjQxNjg2Nw==","name":"addons-server","full_name":"mozilla/addons-server","owner":{"login":"mozilla","id":131524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzMTUyNA==","avatar_url":"https://avatars2.githubusercontent.com/u/131524?v=4","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mozilla/addons-server","description":"🕶 addons.mozilla.org Django app and API 🎉","fork":false,"url":"https://api.github.com/repos/mozilla/addons-server","forks_url":"https://api.github.com/repos/mozilla/addons-server/forks","keys_url":"https://api.github.com/repos/mozilla/addons-server/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mozilla/addons-server/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mozilla/addons-server/teams","hooks_url":"https://api.github.com/repos/mozilla/addons-server/hooks","issue_events_url":"https://api.github.com/repos/mozilla/addons-server/issues/events{/number}","events_url":"https://api.github.com/repos/mozilla/addons-server/events","assignees_url":"https://api.github.com/repos/mozilla/addons-server/assignees{/user}","branches_url":"https://api.github.com/repos/mozilla/addons-server/branches{/branch}","tags_url":"https://api.github.com/repos/mozilla/addons-server/tags","blobs_url":"https://api.github.com/repos/mozilla/addons-server/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mozilla/addons-server/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mozilla/addons-server/git/refs{/sha}","trees_url":"https://api.github.com/repos/mozilla/addons-server/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mozilla/addons-server/statuses/{sha}","languages_url":"https://api.github.com/repos/mozilla/addons-server/languages","stargazers_url":"https://api.github.com/repos/mozilla/addons-server/stargazers","contributors_url":"https://api.github.com/repos/mozilla/addons-server/contributors","subscribers_url":"https://api.github.com/repos/mozilla/addons-server/subscribers","subscription_url":"https://api.github.com/repos/mozilla/addons-server/subscription","commits_url":"https://api.github.com/repos/mozilla/addons-server/commits{/sha}","git_commits_url":"https://api.github.com/repos/mozilla/addons-server/git/commits{/sha}","comments_url":"https://api.github.com/repos/mozilla/addons-server/comments{/number}","issue_comment_url":"https://api.github.com/repos/mozilla/addons-server/issues/comments{/number}","contents_url":"https://api.github.com/repos/mozilla/addons-server/contents/{+path}","compare_url":"https://api.github.com/repos/mozilla/addons-server/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mozilla/addons-server/merges","archive_url":"https://api.github.com/repos/mozilla/addons-server/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mozilla/addons-server/downloads","issues_url":"https://api.github.com/repos/mozilla/addons-server/issues{/number}","pulls_url":"https://api.github.com/repos/mozilla/addons-server/pulls{/number}","milestones_url":"https://api.github.com/repos/mozilla/addons-server/milestones{/number}","notifications_url":"https://api.github.com/repos/mozilla/addons-server/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mozilla/addons-server/labels{/name}","releases_url":"https://api.github.com/repos/mozilla/addons-server/releases{/id}","deployments_url":"https://api.github.com/repos/mozilla/addons-server/deployments","created_at":"2014-01-31T18:44:15Z","updated_at":"2018-07-02T16:13:38Z","pushed_at":"2018-07-02T16:19:26Z","git_url":"git://github.com/mozilla/addons-server.git","ssh_url":"git@github.com:mozilla/addons-server.git","clone_url":"https://github.com/mozilla/addons-server.git","svn_url":"https://github.com/mozilla/addons-server","homepage":"https://addons.mozilla.org/","size":894595,"stargazers_count":318,"watchers_count":318,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":299,"mirror_url":null,"archived":false,"open_issues_count":369,"license":{"key":"bsd-3-clause","name":"BSD 3-Clause \"New\" or \"Revised\" License","spdx_id":"BSD-3-Clause","url":"https://api.github.com/licenses/bsd-3-clause","node_id":"MDc6TGljZW5zZTU="},"forks":299,"open_issues":369,"watchers":318,"default_branch":"master"}},"base":{"label":"mozilla:master","ref":"master","sha":"762142ee5a169e15bb37a325f1befa4c6ded57f1","user":{"login":"mozilla","id":131524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzMTUyNA==","avatar_url":"https://avatars2.githubusercontent.com/u/131524?v=4","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"repo":{"id":16416867,"node_id":"MDEwOlJlcG9zaXRvcnkxNjQxNjg2Nw==","name":"addons-server","full_name":"mozilla/addons-server","owner":{"login":"mozilla","id":131524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzMTUyNA==","avatar_url":"https://avatars2.githubusercontent.com/u/131524?v=4","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mozilla/addons-server","description":"🕶 addons.mozilla.org Django app and API 🎉","fork":false,"url":"https://api.github.com/repos/mozilla/addons-server","forks_url":"https://api.github.com/repos/mozilla/addons-server/forks","keys_url":"https://api.github.com/repos/mozilla/addons-server/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mozilla/addons-server/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mozilla/addons-server/teams","hooks_url":"https://api.github.com/repos/mozilla/addons-server/hooks","issue_events_url":"https://api.github.com/repos/mozilla/addons-server/issues/events{/number}","events_url":"https://api.github.com/repos/mozilla/addons-server/events","assignees_url":"https://api.github.com/repos/mozilla/addons-server/assignees{/user}","branches_url":"https://api.github.com/repos/mozilla/addons-server/branches{/branch}","tags_url":"https://api.github.com/repos/mozilla/addons-server/tags","blobs_url":"https://api.github.com/repos/mozilla/addons-server/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mozilla/addons-server/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mozilla/addons-server/git/refs{/sha}","trees_url":"https://api.github.com/repos/mozilla/addons-server/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mozilla/addons-server/statuses/{sha}","languages_url":"https://api.github.com/repos/mozilla/addons-server/languages","stargazers_url":"https://api.github.com/repos/mozilla/addons-server/stargazers","contributors_url":"https://api.github.com/repos/mozilla/addons-server/contributors","subscribers_url":"https://api.github.com/repos/mozilla/addons-server/subscribers","subscription_url":"https://api.github.com/repos/mozilla/addons-server/subscription","commits_url":"https://api.github.com/repos/mozilla/addons-server/commits{/sha}","git_commits_url":"https://api.github.com/repos/mozilla/addons-server/git/commits{/sha}","comments_url":"https://api.github.com/repos/mozilla/addons-server/comments{/number}","issue_comment_url":"https://api.github.com/repos/mozilla/addons-server/issues/comments{/number}","contents_url":"https://api.github.com/repos/mozilla/addons-server/contents/{+path}","compare_url":"https://api.github.com/repos/mozilla/addons-server/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mozilla/addons-server/merges","archive_url":"https://api.github.com/repos/mozilla/addons-server/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mozilla/addons-server/downloads","issues_url":"https://api.github.com/repos/mozilla/addons-server/issues{/number}","pulls_url":"https://api.github.com/repos/mozilla/addons-server/pulls{/number}","milestones_url":"https://api.github.com/repos/mozilla/addons-server/milestones{/number}","notifications_url":"https://api.github.com/repos/mozilla/addons-server/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mozilla/addons-server/labels{/name}","releases_url":"https://api.github.com/repos/mozilla/addons-server/releases{/id}","deployments_url":"https://api.github.com/repos/mozilla/addons-server/deployments","created_at":"2014-01-31T18:44:15Z","updated_at":"2018-07-02T16:13:38Z","pushed_at":"2018-07-02T16:19:26Z","git_url":"git://github.com/mozilla/addons-server.git","ssh_url":"git@github.com:mozilla/addons-server.git","clone_url":"https://github.com/mozilla/addons-server.git","svn_url":"https://github.com/mozilla/addons-server","homepage":"https://addons.mozilla.org/","size":894595,"stargazers_count":318,"watchers_count":318,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":299,"mirror_url":null,"archived":false,"open_issues_count":369,"license":{"key":"bsd-3-clause","name":"BSD 3-Clause \"New\" or \"Revised\" License","spdx_id":"BSD-3-Clause","url":"https://api.github.com/licenses/bsd-3-clause","node_id":"MDc6TGljZW5zZTU="},"forks":299,"open_issues":369,"watchers":318,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mozilla/addons-server/pulls/8713"},"html":{"href":"https://github.com/mozilla/addons-server/pull/8713"},"issue":{"href":"https://api.github.com/repos/mozilla/addons-server/issues/8713"},"comments":{"href":"https://api.github.com/repos/mozilla/addons-server/issues/8713/comments"},"review_comments":{"href":"https://api.github.com/repos/mozilla/addons-server/pulls/8713/comments"},"review_comment":{"href":"https://api.github.com/repos/mozilla/addons-server/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mozilla/addons-server/pulls/8713/commits"},"statuses":{"href":"https://api.github.com/repos/mozilla/addons-server/statuses/cd2a744dde9f3dc503f653668b545e531dde125b"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"diox"}}