{"action":"created","issue":{"url":"https://api.github.com/repos/joomla/joomla-cms/issues/20391","repository_url":"https://api.github.com/repos/joomla/joomla-cms","labels_url":"https://api.github.com/repos/joomla/joomla-cms/issues/20391/labels{/name}","comments_url":"https://api.github.com/repos/joomla/joomla-cms/issues/20391/comments","events_url":"https://api.github.com/repos/joomla/joomla-cms/issues/20391/events","html_url":"https://github.com/joomla/joomla-cms/pull/20391","id":322735007,"node_id":"MDExOlB1bGxSZXF1ZXN0MTg3NzY1Mzg0","number":20391,"title":"[4.0] Using language specific tokeniser and stemmer for com_finder","user":{"login":"Hackwar","id":313866,"node_id":"MDQ6VXNlcjMxMzg2Ng==","avatar_url":"https://avatars2.githubusercontent.com/u/313866?v=4","gravatar_id":"","url":"https://api.github.com/users/Hackwar","html_url":"https://github.com/Hackwar","followers_url":"https://api.github.com/users/Hackwar/followers","following_url":"https://api.github.com/users/Hackwar/following{/other_user}","gists_url":"https://api.github.com/users/Hackwar/gists{/gist_id}","starred_url":"https://api.github.com/users/Hackwar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Hackwar/subscriptions","organizations_url":"https://api.github.com/users/Hackwar/orgs","repos_url":"https://api.github.com/users/Hackwar/repos","events_url":"https://api.github.com/users/Hackwar/events{/privacy}","received_events_url":"https://api.github.com/users/Hackwar/received_events","type":"User","site_admin":false},"labels":[{"id":307745144,"node_id":"MDU6TGFiZWwzMDc3NDUxNDQ=","url":"https://api.github.com/repos/joomla/joomla-cms/labels/Composer%20Dependency%20Changed","name":"Composer Dependency Changed","color":"207de5","default":false},{"id":444033883,"node_id":"MDU6TGFiZWw0NDQwMzM4ODM=","url":"https://api.github.com/repos/joomla/joomla-cms/labels/PR-4.0-dev","name":"PR-4.0-dev","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":113,"created_at":"2018-05-14T09:20:10Z","updated_at":"2018-06-13T20:31:41Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/joomla/joomla-cms/pulls/20391","html_url":"https://github.com/joomla/joomla-cms/pull/20391","diff_url":"https://github.com/joomla/joomla-cms/pull/20391.diff","patch_url":"https://github.com/joomla/joomla-cms/pull/20391.patch"},"body":"## Tokenisation\r\nIn order to build the index, com_finder has to tokenise the input from a string into single terms. Right now com_finder here basically only really supports latin based languages and chinese (by a hack). Languages that do not seperate words by white space are not supported. This PR introduces a FinderIndexerLanguage class, which tokenises strings and which can be overridden by language specific tokenisers, so that all languages can be supported. The different tokenisation scripts would have to be supported by the translation teams.\r\n\r\n## Stemming\r\ncom_finder also supports stemming words in order to improve the search results. Stemming reduces a term to its base and thus allows for searching both singular and plural variants of a word with the same search term or different timecases of a word. Unfortunately com_finder up till now only had 3 stemmers, which supported only a small range of languages. Again, this has been outsourced into a file in the language pack.\r\n\r\n## Providing language specific tokeniser and stemmer\r\nTranslation teams would have to ship a {language}.finder.php file which extends from FinderIndexerLanguage. See the en-GB language pack for an example of adding your own stemmer.\r\n\r\n## How to test\r\nUnfortunately this is a pretty opaque change as long as there is only the en-GB implementation of the language support file. Simply said, you should be able to take a current 4.0-dev, test indexing and searching, apply these changes and do the same tests, seeing that you got the same results."},"comment":{"url":"https://api.github.com/repos/joomla/joomla-cms/issues/comments/397075866","html_url":"https://github.com/joomla/joomla-cms/pull/20391#issuecomment-397075866","issue_url":"https://api.github.com/repos/joomla/joomla-cms/issues/20391","id":397075866,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NzA3NTg2Ng==","user":{"login":"Hackwar","id":313866,"node_id":"MDQ6VXNlcjMxMzg2Ng==","avatar_url":"https://avatars2.githubusercontent.com/u/313866?v=4","gravatar_id":"","url":"https://api.github.com/users/Hackwar","html_url":"https://github.com/Hackwar","followers_url":"https://api.github.com/users/Hackwar/followers","following_url":"https://api.github.com/users/Hackwar/following{/other_user}","gists_url":"https://api.github.com/users/Hackwar/gists{/gist_id}","starred_url":"https://api.github.com/users/Hackwar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Hackwar/subscriptions","organizations_url":"https://api.github.com/users/Hackwar/orgs","repos_url":"https://api.github.com/users/Hackwar/repos","events_url":"https://api.github.com/users/Hackwar/events{/privacy}","received_events_url":"https://api.github.com/users/Hackwar/received_events","type":"User","site_admin":false},"created_at":"2018-06-13T20:31:41Z","updated_at":"2018-06-13T20:31:41Z","author_association":"MEMBER","body":"You are indeed my favorite developer, since you get stuff done. :wink: Now we just need to get the code to pass the codestyle and I need to make a bunch of additional changes. After some considerations, I've decided to revert some changes and to work with the result of ```Locale::getPrimaryLanguage()``` for the stemmer, tokeniser and the value stored for the token in the DB. That makes stuff a bit easier for everybody..."}}
{"actor":{"display_login":"Hackwar"}}