{"action":"created","issue":{"url":"https://api.github.com/repos/joomla/joomla-cms/issues/20391","repository_url":"https://api.github.com/repos/joomla/joomla-cms","labels_url":"https://api.github.com/repos/joomla/joomla-cms/issues/20391/labels{/name}","comments_url":"https://api.github.com/repos/joomla/joomla-cms/issues/20391/comments","events_url":"https://api.github.com/repos/joomla/joomla-cms/issues/20391/events","html_url":"https://github.com/joomla/joomla-cms/pull/20391","id":322735007,"number":20391,"title":"[4.0] Using language specific tokeniser and stemmer for com_finder","user":{"login":"Hackwar","id":313866,"avatar_url":"https://avatars2.githubusercontent.com/u/313866?v=4","gravatar_id":"","url":"https://api.github.com/users/Hackwar","html_url":"https://github.com/Hackwar","followers_url":"https://api.github.com/users/Hackwar/followers","following_url":"https://api.github.com/users/Hackwar/following{/other_user}","gists_url":"https://api.github.com/users/Hackwar/gists{/gist_id}","starred_url":"https://api.github.com/users/Hackwar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Hackwar/subscriptions","organizations_url":"https://api.github.com/users/Hackwar/orgs","repos_url":"https://api.github.com/users/Hackwar/repos","events_url":"https://api.github.com/users/Hackwar/events{/privacy}","received_events_url":"https://api.github.com/users/Hackwar/received_events","type":"User","site_admin":false},"labels":[{"id":198658453,"url":"https://api.github.com/repos/joomla/joomla-cms/labels/Language%20Change","name":"Language Change","color":"eb6420","default":false},{"id":444033883,"url":"https://api.github.com/repos/joomla/joomla-cms/labels/PR-4.0-dev","name":"PR-4.0-dev","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":21,"created_at":"2018-05-14T09:20:10Z","updated_at":"2018-05-21T09:13:32Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/joomla/joomla-cms/pulls/20391","html_url":"https://github.com/joomla/joomla-cms/pull/20391","diff_url":"https://github.com/joomla/joomla-cms/pull/20391.diff","patch_url":"https://github.com/joomla/joomla-cms/pull/20391.patch"},"body":"## Tokenisation\r\nIn order to build the index, com_finder has to tokenise the input from a string into single terms. Right now com_finder here basically only really supports latin based languages and chinese (by a hack). Languages that do not seperate words by white space are not supported. This PR introduces a FinderIndexerLanguage class, which tokenises strings and which can be overridden by language specific tokenisers, so that all languages can be supported. The different tokenisation scripts would have to be supported by the translation teams.\r\n\r\n## Stemming\r\ncom_finder also supports stemming words in order to improve the search results. Stemming reduces a term to its base and thus allows for searching both singular and plural variants of a word with the same search term or different timecases of a word. Unfortunately com_finder up till now only had 3 stemmers, which supported only a small range of languages. Again, this has been outsourced into a file in the language pack.\r\n\r\n## Providing language specific tokeniser and stemmer\r\nTranslation teams would have to ship a {language}.finder.php file which extends from FinderIndexerLanguage. See the en-GB language pack for an example of adding your own stemmer.\r\n\r\n## How to test\r\nUnfortunately this is a pretty opaque change as long as there is only the en-GB implementation of the language support file. Simply said, you should be able to take a current 4.0-dev, test indexing and searching, apply these changes and do the same tests, seeing that you got the same results."},"comment":{"url":"https://api.github.com/repos/joomla/joomla-cms/issues/comments/390599043","html_url":"https://github.com/joomla/joomla-cms/pull/20391#issuecomment-390599043","issue_url":"https://api.github.com/repos/joomla/joomla-cms/issues/20391","id":390599043,"user":{"login":"ggppdk","id":5092940,"avatar_url":"https://avatars2.githubusercontent.com/u/5092940?v=4","gravatar_id":"","url":"https://api.github.com/users/ggppdk","html_url":"https://github.com/ggppdk","followers_url":"https://api.github.com/users/ggppdk/followers","following_url":"https://api.github.com/users/ggppdk/following{/other_user}","gists_url":"https://api.github.com/users/ggppdk/gists{/gist_id}","starred_url":"https://api.github.com/users/ggppdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ggppdk/subscriptions","organizations_url":"https://api.github.com/users/ggppdk/orgs","repos_url":"https://api.github.com/users/ggppdk/repos","events_url":"https://api.github.com/users/ggppdk/events{/privacy}","received_events_url":"https://api.github.com/users/ggppdk/received_events","type":"User","site_admin":false},"created_at":"2018-05-21T09:13:32Z","updated_at":"2018-05-21T09:13:32Z","author_association":"CONTRIBUTOR","body":"Why not to use stemmer in monolanguage site ? can someone explain why not to use ?\r\n\r\nIt is sane to assume that the language of article is such website is \r\n```php\r\n$lang = JComponentHelper::getParams('com_languages')->get('site', 'en-GB');\r\n```\r\n\r\nI mean why would anyone have a different UI language than the language of articles in a monolanguage site ?"}}
{"actor":{"display_login":"ggppdk"}}