{"action":"created","comment":{"url":"https://api.github.com/repos/joomla/joomla-cms/pulls/comments/224981063","pull_request_review_id":164486405,"id":224981063,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIyNDk4MTA2Mw==","diff_hunk":"@@ -327,9 +327,9 @@ public static function isCommon($token, $lang)\n \t\t$langCode = $lang;\n \n \t\t// If language requested is wildcard, use the default language.\n-\t\tif ($lang == '*')\n+\t\tif ($default === null && $lang === '*')\n \t\t{\n-\t\t\t$default = $default === null ? substr(self::getDefaultLanguage(), 0, 2) : $default;\n+\t\t\t$default = substr(self::getDefaultLanguage(), 0, 2);","path":"administrator/components/com_finder/helpers/indexer/helper.php","position":8,"original_position":8,"commit_id":"8d9a9af439df561ee62c890042d52f12d71544d3","original_commit_id":"8d9a9af439df561ee62c890042d52f12d71544d3","user":{"login":"infograf768","id":869724,"node_id":"MDQ6VXNlcjg2OTcyNA==","avatar_url":"https://avatars3.githubusercontent.com/u/869724?v=4","gravatar_id":"","url":"https://api.github.com/users/infograf768","html_url":"https://github.com/infograf768","followers_url":"https://api.github.com/users/infograf768/followers","following_url":"https://api.github.com/users/infograf768/following{/other_user}","gists_url":"https://api.github.com/users/infograf768/gists{/gist_id}","starred_url":"https://api.github.com/users/infograf768/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/infograf768/subscriptions","organizations_url":"https://api.github.com/users/infograf768/orgs","repos_url":"https://api.github.com/users/infograf768/repos","events_url":"https://api.github.com/users/infograf768/events{/privacy}","received_events_url":"https://api.github.com/users/infograf768/received_events","type":"User","site_admin":false},"body":"Just remarked that this code (already present before this PR) will not work when language has a langcode of type `xxx-XX`. (We have a few in J).\r\nWould we ever get common terms for these languages? No idea.\r\nLang concerned are ckb-IQ, prs-AF, srp-ME as of 3.9.0\r\n\r\nThe code to modify this would be something like\r\n```\r\n\t\t\t\t\t\t$tag = strlen(self::getDefaultLanguage());\r\n\r\n\t\t\t\t\t\tif ($tag == 5)\r\n\t\t\t\t\t\t{\r\n\t\t\t\t\t\t\t$default = substr(self::getDefaultLanguage(), 0, 2);\r\n\t\t\t\t\t\t}\r\n\t\t\t\t\t\telseif ($tag == 6)\r\n\t\t\t\t\t\t{\r\n\t\t\t\t\t\t\t$default  = substr(self::getDefaultLanguage(), 0, 3);\r\n\t\t\t\t\t\t}\r\n```\r\n","created_at":"2018-10-14T06:15:17Z","updated_at":"2018-10-14T06:15:17Z","html_url":"https://github.com/joomla/joomla-cms/pull/22599#discussion_r224981063","pull_request_url":"https://api.github.com/repos/joomla/joomla-cms/pulls/22599","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/joomla/joomla-cms/pulls/comments/224981063"},"html":{"href":"https://github.com/joomla/joomla-cms/pull/22599#discussion_r224981063"},"pull_request":{"href":"https://api.github.com/repos/joomla/joomla-cms/pulls/22599"}}},"pull_request":{"url":"https://api.github.com/repos/joomla/joomla-cms/pulls/22599","id":222681936,"node_id":"MDExOlB1bGxSZXF1ZXN0MjIyNjgxOTM2","html_url":"https://github.com/joomla/joomla-cms/pull/22599","diff_url":"https://github.com/joomla/joomla-cms/pull/22599.diff","patch_url":"https://github.com/joomla/joomla-cms/pull/22599.patch","issue_url":"https://api.github.com/repos/joomla/joomla-cms/issues/22599","number":22599,"state":"open","locked":false,"title":"Indexer fix for large articles","user":{"login":"frankmayer","id":825497,"node_id":"MDQ6VXNlcjgyNTQ5Nw==","avatar_url":"https://avatars2.githubusercontent.com/u/825497?v=4","gravatar_id":"","url":"https://api.github.com/users/frankmayer","html_url":"https://github.com/frankmayer","followers_url":"https://api.github.com/users/frankmayer/followers","following_url":"https://api.github.com/users/frankmayer/following{/other_user}","gists_url":"https://api.github.com/users/frankmayer/gists{/gist_id}","starred_url":"https://api.github.com/users/frankmayer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frankmayer/subscriptions","organizations_url":"https://api.github.com/users/frankmayer/orgs","repos_url":"https://api.github.com/users/frankmayer/repos","events_url":"https://api.github.com/users/frankmayer/events{/privacy}","received_events_url":"https://api.github.com/users/frankmayer/received_events","type":"User","site_admin":false},"body":"Pull Request for Issue #22098 .\r\n\r\n### Summary of Changes\r\nThis PR re-adds and optimizes chunking (for the VALUES clause) to the smart search indexer, allowing for larget articles to be saved correctly, and their terms to be correctly added to the index, again.\r\nIt also optimizes a condition check in the helper.\r\n\r\nWith a standard configuration of a PHP memoy limit at 128MB and the default Mysql HEAP limits for MEMORY tables these instructions should provide solid results.\r\n\r\n\r\n### Testing Instructions\r\n\r\n#### Introduction:\r\n\r\nThere are three scenarios. Each one involves saving a predefined article and getting an expected result.\r\nThe first scenario uses a text of ~32000 words. This one should be saved successfully. \r\nIn the second scenario, we use a text of ~45000 words. This one will fail to save because of surpassing the finder_token table limits. \r\nFinally, in the third scenario we use a text of ~65000 words. This should cause a php fatal error, stating that the memory has been exhausted.\r\n\r\n\r\n#### Preparation: \r\n\r\n- **Error reporting** should be set to **development mode**\r\n- Enable the \"Content - Smart Search\" plugin (and the other Smart Search related) \r\n- Open a tab for a new article\r\n- Open a tab for components/Smart Search\r\n\r\n#### Scenarios:\r\n\r\n1) Article with near maximum possible text\r\n a) On the \"Smart Search\" tab, click \"Clear Index\"\r\n b) Copy and paste the complete text (select all) from this [gist](https://gist.githubusercontent.com/frankmayer/f0864c40597a7f71d33da63ca18b7599/raw/35c992afa59fde43298c41e3a5889b4becab939a/lorem%2520x5) (213,4KiB unformatted text) into the article in the other tab\r\n c) Give it a title and \"save\" the article\r\n d) Expected result:  **Message** Article saved.\r\n e) On the \"Smart Search\" tab, click \"Statistics\" and check that it says \"The indexed content on this site includes 11,995 terms across 1 links...\" \r\n\r\n2) Article which exceeds the maximum entries for the `finder_token` table\r\n a) On the Smart Search tab, click \"Clear Index\"\r\n b) Copy and paste the complete text (select all) from this [gist](https://gist.githubusercontent.com/frankmayer/f0864c40597a7f71d33da63ca18b7599/raw/35c992afa59fde43298c41e3a5889b4becab939a/lorem%2520x7) (298,8KiB unformatted text) into the article in the other tab\r\n c) Save the article\r\n d) Expected result:  **Error** Save failed with the following error: The table '#__finder_tokens' is full\r\n e) On the \"Smart Search\" tab, click \"Statistics\" and check that it says \"The indexed content on this site includes 0 terms across 1 links...\"\r\n**Remarks** \r\nIf you get a message that the acrticle was saved, instead of the error, you probably have higher heap settings for the MEMORY tables in mysql. Still this result would count as expected.\r\nIf you get a php error showing that the memory is exhausted, you are probably running php with a limit of less than 128MB. If that happens, please copy/paste the error message into a comment of this issue.\r\n\r\n3) Article which results in a fatal php error due to memory exhaustion\r\n a) On the Smart Search tab, click \"Clear Index\"\r\n b) Copy and paste the complete text (select all) from this [gist](https://gist.githubusercontent.com/frankmayer/f0864c40597a7f71d33da63ca18b7599/raw/35c992afa59fde43298c41e3a5889b4becab939a/lorem%2520x10) into the article in the other tab\r\n c) Save the article\r\n d) Expected result: a php error showing that the memory is exhausted\r\n e) On the \"Smart Search\" tab, click \"Statistics\" and check that it says \"The indexed content on this site includes 0 terms across 1 links...\"\r\n**Remarks** \r\nIf you get a message that the acrticle was saved, instead of the error, you probably have higher heap settings for the MEMORY tables in mysql. Still this result would count as expected.\r\nIf you get a php error showing that the memory is exhausted, you are probably running php with a limit of less than 128MB. If that happens, please copy/paste the error message into a comment of this issue.\r\n\r\n\r\n### Documentation Changes Required\r\nnone\r\n","created_at":"2018-10-13T23:08:50Z","updated_at":"2018-10-14T06:15:17Z","closed_at":null,"merged_at":null,"merge_commit_sha":"5ad3f61db282c4f361c4d67f9fe3d8c7cbc093b0","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":60721348,"node_id":"MDU6TGFiZWw2MDcyMTM0OA==","url":"https://api.github.com/repos/joomla/joomla-cms/labels/PR-staging","name":"PR-staging","color":"c7def8","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/joomla/joomla-cms/pulls/22599/commits","review_comments_url":"https://api.github.com/repos/joomla/joomla-cms/pulls/22599/comments","review_comment_url":"https://api.github.com/repos/joomla/joomla-cms/pulls/comments{/number}","comments_url":"https://api.github.com/repos/joomla/joomla-cms/issues/22599/comments","statuses_url":"https://api.github.com/repos/joomla/joomla-cms/statuses/8d9a9af439df561ee62c890042d52f12d71544d3","head":{"label":"frankmayer:indexer-fix","ref":"indexer-fix","sha":"8d9a9af439df561ee62c890042d52f12d71544d3","user":{"login":"frankmayer","id":825497,"node_id":"MDQ6VXNlcjgyNTQ5Nw==","avatar_url":"https://avatars2.githubusercontent.com/u/825497?v=4","gravatar_id":"","url":"https://api.github.com/users/frankmayer","html_url":"https://github.com/frankmayer","followers_url":"https://api.github.com/users/frankmayer/followers","following_url":"https://api.github.com/users/frankmayer/following{/other_user}","gists_url":"https://api.github.com/users/frankmayer/gists{/gist_id}","starred_url":"https://api.github.com/users/frankmayer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frankmayer/subscriptions","organizations_url":"https://api.github.com/users/frankmayer/orgs","repos_url":"https://api.github.com/users/frankmayer/repos","events_url":"https://api.github.com/users/frankmayer/events{/privacy}","received_events_url":"https://api.github.com/users/frankmayer/received_events","type":"User","site_admin":false},"repo":{"id":69169032,"node_id":"MDEwOlJlcG9zaXRvcnk2OTE2OTAzMg==","name":"joomla-cms","full_name":"frankmayer/joomla-cms","private":false,"owner":{"login":"frankmayer","id":825497,"node_id":"MDQ6VXNlcjgyNTQ5Nw==","avatar_url":"https://avatars2.githubusercontent.com/u/825497?v=4","gravatar_id":"","url":"https://api.github.com/users/frankmayer","html_url":"https://github.com/frankmayer","followers_url":"https://api.github.com/users/frankmayer/followers","following_url":"https://api.github.com/users/frankmayer/following{/other_user}","gists_url":"https://api.github.com/users/frankmayer/gists{/gist_id}","starred_url":"https://api.github.com/users/frankmayer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frankmayer/subscriptions","organizations_url":"https://api.github.com/users/frankmayer/orgs","repos_url":"https://api.github.com/users/frankmayer/repos","events_url":"https://api.github.com/users/frankmayer/events{/privacy}","received_events_url":"https://api.github.com/users/frankmayer/received_events","type":"User","site_admin":false},"html_url":"https://github.com/frankmayer/joomla-cms","description":"Home of the Joomla! Content Management System","fork":true,"url":"https://api.github.com/repos/frankmayer/joomla-cms","forks_url":"https://api.github.com/repos/frankmayer/joomla-cms/forks","keys_url":"https://api.github.com/repos/frankmayer/joomla-cms/keys{/key_id}","collaborators_url":"https://api.github.com/repos/frankmayer/joomla-cms/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/frankmayer/joomla-cms/teams","hooks_url":"https://api.github.com/repos/frankmayer/joomla-cms/hooks","issue_events_url":"https://api.github.com/repos/frankmayer/joomla-cms/issues/events{/number}","events_url":"https://api.github.com/repos/frankmayer/joomla-cms/events","assignees_url":"https://api.github.com/repos/frankmayer/joomla-cms/assignees{/user}","branches_url":"https://api.github.com/repos/frankmayer/joomla-cms/branches{/branch}","tags_url":"https://api.github.com/repos/frankmayer/joomla-cms/tags","blobs_url":"https://api.github.com/repos/frankmayer/joomla-cms/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/frankmayer/joomla-cms/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/frankmayer/joomla-cms/git/refs{/sha}","trees_url":"https://api.github.com/repos/frankmayer/joomla-cms/git/trees{/sha}","statuses_url":"https://api.github.com/repos/frankmayer/joomla-cms/statuses/{sha}","languages_url":"https://api.github.com/repos/frankmayer/joomla-cms/languages","stargazers_url":"https://api.github.com/repos/frankmayer/joomla-cms/stargazers","contributors_url":"https://api.github.com/repos/frankmayer/joomla-cms/contributors","subscribers_url":"https://api.github.com/repos/frankmayer/joomla-cms/subscribers","subscription_url":"https://api.github.com/repos/frankmayer/joomla-cms/subscription","commits_url":"https://api.github.com/repos/frankmayer/joomla-cms/commits{/sha}","git_commits_url":"https://api.github.com/repos/frankmayer/joomla-cms/git/commits{/sha}","comments_url":"https://api.github.com/repos/frankmayer/joomla-cms/comments{/number}","issue_comment_url":"https://api.github.com/repos/frankmayer/joomla-cms/issues/comments{/number}","contents_url":"https://api.github.com/repos/frankmayer/joomla-cms/contents/{+path}","compare_url":"https://api.github.com/repos/frankmayer/joomla-cms/compare/{base}...{head}","merges_url":"https://api.github.com/repos/frankmayer/joomla-cms/merges","archive_url":"https://api.github.com/repos/frankmayer/joomla-cms/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/frankmayer/joomla-cms/downloads","issues_url":"https://api.github.com/repos/frankmayer/joomla-cms/issues{/number}","pulls_url":"https://api.github.com/repos/frankmayer/joomla-cms/pulls{/number}","milestones_url":"https://api.github.com/repos/frankmayer/joomla-cms/milestones{/number}","notifications_url":"https://api.github.com/repos/frankmayer/joomla-cms/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/frankmayer/joomla-cms/labels{/name}","releases_url":"https://api.github.com/repos/frankmayer/joomla-cms/releases{/id}","deployments_url":"https://api.github.com/repos/frankmayer/joomla-cms/deployments","created_at":"2016-09-25T14:24:41Z","updated_at":"2016-09-25T14:24:55Z","pushed_at":"2018-10-13T22:06:54Z","git_url":"git://github.com/frankmayer/joomla-cms.git","ssh_url":"git@github.com:frankmayer/joomla-cms.git","clone_url":"https://github.com/frankmayer/joomla-cms.git","svn_url":"https://github.com/frankmayer/joomla-cms","homepage":"https://www.joomla.org","size":242187,"stargazers_count":0,"watchers_count":0,"language":"PHP","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"gpl-2.0","name":"GNU General Public License v2.0","spdx_id":"GPL-2.0","url":"https://api.github.com/licenses/gpl-2.0","node_id":"MDc6TGljZW5zZTg="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"staging"}},"base":{"label":"joomla:staging","ref":"staging","sha":"32952ffe17703f510bcb62eb0fa1d2fa9d56d824","user":{"login":"joomla","id":751633,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc1MTYzMw==","avatar_url":"https://avatars3.githubusercontent.com/u/751633?v=4","gravatar_id":"","url":"https://api.github.com/users/joomla","html_url":"https://github.com/joomla","followers_url":"https://api.github.com/users/joomla/followers","following_url":"https://api.github.com/users/joomla/following{/other_user}","gists_url":"https://api.github.com/users/joomla/gists{/gist_id}","starred_url":"https://api.github.com/users/joomla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joomla/subscriptions","organizations_url":"https://api.github.com/users/joomla/orgs","repos_url":"https://api.github.com/users/joomla/repos","events_url":"https://api.github.com/users/joomla/events{/privacy}","received_events_url":"https://api.github.com/users/joomla/received_events","type":"Organization","site_admin":false},"repo":{"id":2464908,"node_id":"MDEwOlJlcG9zaXRvcnkyNDY0OTA4","name":"joomla-cms","full_name":"joomla/joomla-cms","private":false,"owner":{"login":"joomla","id":751633,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc1MTYzMw==","avatar_url":"https://avatars3.githubusercontent.com/u/751633?v=4","gravatar_id":"","url":"https://api.github.com/users/joomla","html_url":"https://github.com/joomla","followers_url":"https://api.github.com/users/joomla/followers","following_url":"https://api.github.com/users/joomla/following{/other_user}","gists_url":"https://api.github.com/users/joomla/gists{/gist_id}","starred_url":"https://api.github.com/users/joomla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joomla/subscriptions","organizations_url":"https://api.github.com/users/joomla/orgs","repos_url":"https://api.github.com/users/joomla/repos","events_url":"https://api.github.com/users/joomla/events{/privacy}","received_events_url":"https://api.github.com/users/joomla/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/joomla/joomla-cms","description":"Home of the Joomla! Content Management System","fork":false,"url":"https://api.github.com/repos/joomla/joomla-cms","forks_url":"https://api.github.com/repos/joomla/joomla-cms/forks","keys_url":"https://api.github.com/repos/joomla/joomla-cms/keys{/key_id}","collaborators_url":"https://api.github.com/repos/joomla/joomla-cms/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/joomla/joomla-cms/teams","hooks_url":"https://api.github.com/repos/joomla/joomla-cms/hooks","issue_events_url":"https://api.github.com/repos/joomla/joomla-cms/issues/events{/number}","events_url":"https://api.github.com/repos/joomla/joomla-cms/events","assignees_url":"https://api.github.com/repos/joomla/joomla-cms/assignees{/user}","branches_url":"https://api.github.com/repos/joomla/joomla-cms/branches{/branch}","tags_url":"https://api.github.com/repos/joomla/joomla-cms/tags","blobs_url":"https://api.github.com/repos/joomla/joomla-cms/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/joomla/joomla-cms/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/joomla/joomla-cms/git/refs{/sha}","trees_url":"https://api.github.com/repos/joomla/joomla-cms/git/trees{/sha}","statuses_url":"https://api.github.com/repos/joomla/joomla-cms/statuses/{sha}","languages_url":"https://api.github.com/repos/joomla/joomla-cms/languages","stargazers_url":"https://api.github.com/repos/joomla/joomla-cms/stargazers","contributors_url":"https://api.github.com/repos/joomla/joomla-cms/contributors","subscribers_url":"https://api.github.com/repos/joomla/joomla-cms/subscribers","subscription_url":"https://api.github.com/repos/joomla/joomla-cms/subscription","commits_url":"https://api.github.com/repos/joomla/joomla-cms/commits{/sha}","git_commits_url":"https://api.github.com/repos/joomla/joomla-cms/git/commits{/sha}","comments_url":"https://api.github.com/repos/joomla/joomla-cms/comments{/number}","issue_comment_url":"https://api.github.com/repos/joomla/joomla-cms/issues/comments{/number}","contents_url":"https://api.github.com/repos/joomla/joomla-cms/contents/{+path}","compare_url":"https://api.github.com/repos/joomla/joomla-cms/compare/{base}...{head}","merges_url":"https://api.github.com/repos/joomla/joomla-cms/merges","archive_url":"https://api.github.com/repos/joomla/joomla-cms/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/joomla/joomla-cms/downloads","issues_url":"https://api.github.com/repos/joomla/joomla-cms/issues{/number}","pulls_url":"https://api.github.com/repos/joomla/joomla-cms/pulls{/number}","milestones_url":"https://api.github.com/repos/joomla/joomla-cms/milestones{/number}","notifications_url":"https://api.github.com/repos/joomla/joomla-cms/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/joomla/joomla-cms/labels{/name}","releases_url":"https://api.github.com/repos/joomla/joomla-cms/releases{/id}","deployments_url":"https://api.github.com/repos/joomla/joomla-cms/deployments","created_at":"2011-09-27T02:07:38Z","updated_at":"2018-10-13T23:04:43Z","pushed_at":"2018-10-13T23:08:51Z","git_url":"git://github.com/joomla/joomla-cms.git","ssh_url":"git@github.com:joomla/joomla-cms.git","clone_url":"https://github.com/joomla/joomla-cms.git","svn_url":"https://github.com/joomla/joomla-cms","homepage":"https://www.joomla.org","size":244738,"stargazers_count":2944,"watchers_count":2944,"language":"PHP","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2770,"mirror_url":null,"archived":false,"open_issues_count":816,"license":{"key":"gpl-2.0","name":"GNU General Public License v2.0","spdx_id":"GPL-2.0","url":"https://api.github.com/licenses/gpl-2.0","node_id":"MDc6TGljZW5zZTg="},"forks":2770,"open_issues":816,"watchers":2944,"default_branch":"staging"}},"_links":{"self":{"href":"https://api.github.com/repos/joomla/joomla-cms/pulls/22599"},"html":{"href":"https://github.com/joomla/joomla-cms/pull/22599"},"issue":{"href":"https://api.github.com/repos/joomla/joomla-cms/issues/22599"},"comments":{"href":"https://api.github.com/repos/joomla/joomla-cms/issues/22599/comments"},"review_comments":{"href":"https://api.github.com/repos/joomla/joomla-cms/pulls/22599/comments"},"review_comment":{"href":"https://api.github.com/repos/joomla/joomla-cms/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/joomla/joomla-cms/pulls/22599/commits"},"statuses":{"href":"https://api.github.com/repos/joomla/joomla-cms/statuses/8d9a9af439df561ee62c890042d52f12d71544d3"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"infograf768"}}