[14:16:31] I'm about to repool db1029 with 10% of the load [14:16:39] +1 [15:26:59] https://mariadb.com/kb/en/mariadb/performance-schema-system-variables/#performance_schema_digests_size [15:27:24] Range: -1 to 200... default value (mariadb <= 10.0.11) 1000.... LOL [15:47:13] I spent some time tuning that some time ago, didn't touch them since then [15:47:42] it was taking by default like 10GB of memory due to max_connections [15:47:59] for this specific config we have -1 on my.cnf that is autosizing [15:48:05] and the default since 10.0.12 [15:54:22] if you agree I would like to change live max_allowed_packet, innodb_buffer_pool_dump_at_shutdown and innodb_buffer_pool_load_at_startup to have the same values of my.cnf (33554432, 1, 1) [15:54:50] sure [15:55:38] db1033 is almost ready to come back [15:56:38] great [15:57:28] didn't you want to keep db1079 name spare and start from db1080? [15:57:34] for the new hosts [16:00:54] I did, but I do not want to bother chris just for that [16:01:21] it is not like we will run out of numbers [16:01:23] ugh..wish you would've told me I just did all the dns [16:01:33] ok, saw the CR on -operations [16:01:57] Actually, I did https://phabricator.wikimedia.org/T131368#2269247 [16:02:14] yep you did....sorry missed that [16:02:25] do you want me to fix it? [16:02:28] no [16:02:44] we just need to adjust the names on robs accounting [16:03:28] ok..sorry it was on teh procurement ticket I don't look at those for instructions.. [16:03:35] no problem [16:03:49] if that would have caused issues, I would have commented it before [16:04:32] cool, thanks! [16:05:01] yeah I am also seeing racking instructions [16:05:28] I have 4 in each row..including D [16:07:11] cmjohnson1: yes saw your physical disposition, I've already made a plan for which one to pick for each shard here: https://phabricator.wikimedia.org/T111992#2308149 [16:07:45] cool...sorry for the miscommunication [16:08:17] no need to say sorry, as I said, if it was an issue, I would have pinged you [16:08:34] and this just thanks to the details you had in your ticket, I was able to make it even before we know the names :) [16:18:50] volans: 1 small change to that racking order...I found out I am short 1 power outlet still in B1 so instead of asking for another server to decom I can move 2 to B3 [16:20:31] cmjohnson1: for me B3 is even better, quite a few shards have already servers on B1 [16:20:48] cool...then db1085 and db1086 [16:21:00] go to b3...i will update task [16:22:07] if you could add there the mapping rack-name there for those it will save me to lookup all of them on racktables :-P [16:22:14] if you have it at hand [16:33:48] jynus: could you also increase a bit the weight of db1029 please? Probably like 4 [16:33:48] volans, I transfered the rows of the old table which is 23.7 GiB in size to the three new tables which totals 9.7 GiB in size. [16:34:12] So now it's using 2.5 times less space. [16:35:22] that's good, most of the saved space was in the indexes [16:36:05] I have some doubts about x1, it used to be 20 / 10 [16:36:16] but I do not know if it was before or after failover [16:36:20] let me check the history [16:37:13] volans, now to test the most important query [16:37:15] SELECT pageid, externallinks_global.url_id, externallinks_global.paywall_id, url, archive_url, has_archive, live_state, unix_timestamp(last_deadCheck) AS last_deadCheck, archivable, archived, archive_failure, unix_timestamp(access_time) AS access_time, unix_timestamp(archive_time) AS archive_time, paywall_status, reviewed, notified [16:37:15] FROM externallinks_".WIKIPEDIA." [16:37:15] LEFT JOIN externallinks_global ON externallinks_global.url_id = externallinks_".WIKIPEDIA.".url_id [16:37:15] LEFT JOIN externallinks_paywall ON externallinks_global.paywall_id = externallinks_paywall.paywall_id [16:37:18] WHERE `pageid` = '{$this->commObject->pageid}'; [16:38:11] Cyberpower678, plase use a paste for that: https://phabricator.wikimedia.org/paste/edit/form/14/ [16:38:56] jynus, I generally use a paste if it's more than 5 lines [16:40:43] Wow. Fast. 0.063 seconds to execute and deliver a query. :D [17:06:38] volans, I've checked and the usual load was 10-master/20-slave for x1 [17:06:59] it make sense [17:07:14] I will deploy now https://gerrit.wikimedia.org/r/#/c/289686/ [17:07:53] ok, 13GB of RAM used on a 64GB host... it will help to fill the buffer pool [17:53:39] jynus: lag on db2037 started when the error log started to log: [17:53:48] InnoDB: Space id 24 trying to reserve 3 extents actually reserved 256 reserve 1144 free 977 size 7313408 rounds 51 total_reserved 13056 [17:54:57] https://github.com/MariaDB/server/blob/10.0/storage/xtradb/fsp/fsp0fsp.cc#L2830 [17:55:05] which version? [17:55:32] 10.0.22-1 [17:55:36] ours [17:55:52] my link is for the HEAD of 10.0 [17:56:08] yes was asking for ours [17:56:42] I think I've never found contention on extending data- my bet is on disk issues [17:57:19] this should be the right one [17:57:19] https://github.com/MariaDB/server/blob/d775ecdd010daad4dc6147fba58acd006bf2c60c/storage/xtradb/fsp/fsp0fsp.cc#L2812 [17:59:42] I mean, we had way more intense writes before and didn't find this, this is coming from replication, single thread! [18:04:25] bah, not worth it [18:04:50] what, investigating? :)