{"action":"created","issue":{"url":"https://api.github.com/repos/Qubes-Community/Contents/issues/26","repository_url":"https://api.github.com/repos/Qubes-Community/Contents","labels_url":"https://api.github.com/repos/Qubes-Community/Contents/issues/26/labels{/name}","comments_url":"https://api.github.com/repos/Qubes-Community/Contents/issues/26/comments","events_url":"https://api.github.com/repos/Qubes-Community/Contents/issues/26/events","html_url":"https://github.com/Qubes-Community/Contents/issues/26","id":320336041,"number":26,"title":"Qubes Kernel - Optimization & Debugging (Ongoing)","user":{"login":"Aekez","id":33869013,"avatar_url":"https://avatars1.githubusercontent.com/u/33869013?v=4","gravatar_id":"","url":"https://api.github.com/users/Aekez","html_url":"https://github.com/Aekez","followers_url":"https://api.github.com/users/Aekez/followers","following_url":"https://api.github.com/users/Aekez/following{/other_user}","gists_url":"https://api.github.com/users/Aekez/gists{/gist_id}","starred_url":"https://api.github.com/users/Aekez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Aekez/subscriptions","organizations_url":"https://api.github.com/users/Aekez/orgs","repos_url":"https://api.github.com/users/Aekez/repos","events_url":"https://api.github.com/users/Aekez/events{/privacy}","received_events_url":"https://api.github.com/users/Aekez/received_events","type":"User","site_admin":false},"labels":[{"id":922884405,"url":"https://api.github.com/repos/Qubes-Community/Contents/labels/Discussions%20&%20Help%20requests","name":"Discussions & Help requests","color":"b8b9f2","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-04T16:18:29Z","updated_at":"2018-05-10T18:48:04Z","closed_at":null,"author_association":"OWNER","body":"## Topic is split into four different categories of discussion themes ##\r\n_Currently it should be sufficient to keep a single issue for all things Qubes kernel. Should the discussions start to overlap, or it is desired to keep separate issues for the sake of future search-ability, then this issue may be closed and replaced._\r\n- First is **kernel optimization**, which means either improving performance, or including/exclude certain kernel features, towards the end of boosting system performance and/or system capabilities. \r\n- Second is **kernel debugging**, an open invitation for questions and help requests. It is highly desired if you can post your kernel config and error logs. Furthermore, sharing solutions is also appreciated.\r\n- Third is **kernel security**, while the Qubes kernel is already pre-set, it is a good idea to try be aware of which settings should not be changed, and thereby weaken any pre-set security configurations.\r\n- Fourth is **kernel learning**, essentially if you want to learn and improve on above topics, hanging out in this issue thread from time to time may be worth the effort. \r\n\r\n## Tools & know-how to gather relevant logs ##\r\n_This section is currently not written. Feel free to edit post and add, or post suggestion below so that others can modify the post to include relevant logs, tools, and how to find/use them. Credit is due and included for whom help improve this post. If you do not want to be credited, please explicitly say so._\r\n<br />\r\n\r\n\r\n\r\n## Kernel highlights ##\r\n_Below are key terms, please use the search engine to find more information, or ask in the discussion/comment field down below._\r\n\r\n_Note that kernel patches and improvements may have been patched by the Qubes developer team, therefore kernel version is not exact. Late kernel changes may also affect list reliability. Below kernel version highlights either confirmed or likely to be included, which mean they are officially integrated into the kernel._\r\n\r\n\r\n- kernel 4.17.\r\n  - Improved Power Saving Management.\r\n  - Discrete AMD/Radeon APU/GPU support (easy graphic support).\r\n  - Improved APU Ryzen mobile & desktop support.\r\n- kernel 4.16.\r\n  - KVM hypervisor - AMD Secure Encrypted Virtualization (SEV).\r\n  - IOMMU improvements (AMD & Intel).\r\n    - AMD: Improved error reporting, with special focus on the previous uncategorized errors.\r\n    - Intel: Boosted limitations on virtual address space limit.\r\n    - 5-level page-tables.\r\n  - x86_64 Meltdown and Spectre.\r\n  - ARM-64 Meltdown and Spectre.\r\n  - Restricting access /dev/mem via CONFIG_STRICT_DEVMEM.\r\n  - Improved APU Ryzen support (graphic support).\r\n  - POWER Memory key protection support (Open Source CPU / hardware).\r\n  - Improvements to EXT4, BTRFS Raid 5/6 (but not finished yet), XFS, NFS, F2FS, etc.\r\n  - AMD PCI improvements: Hot-plug speed / reliability for (graphics /thunderbolt), Latency reporting/etc. \r\n  - Significant improvement for various smaller system I/O speeds.\r\n- kernel 4.15.\r\n  - x86 Meltdown and Spectre.\r\n  - AMDGPU for Ryzen APU (Minimum kernel for Ryzen APU graphic support).\r\n- kernel 4.14.\r\n  - Linux-firmware blobs no longer included, the Linux-firmware package is instead required.\r\n  - Xen/KVM improvements, speed, and further feature hardware support.\r\n  - Improvements to EFI booting.\r\n  - Btrfs Zstd compression support.\r\n  - Improvements and fixes to EXT4, BTRFS, ZFS, BFQ, F2FS, Zstd, CFQ, etc. \r\n  - AMDGPU Vega improvements.\r\n  - Nouveau improvements.\r\n  - Intel graphics improvements.\r\n  - AMD Secure Memory Encryption (May currently cause instability, please look it up).\r\n  - New driver and improvements: Sound, Wifi, Media, among others. \r\n- kernel 4.9.\r\n  - Vmapped Stacks (Kernel stack overflow protection, x86_64).\r\n  -  Memory Protection Keys (MPK) support (Intel).\r\n- Kernel 4.4.\r\n\r\nPlease checkout https://www.phoronix.com/. This is not an endorsement, paid or similar, but is common courtesy because the website is used a lot to help keep track of highlights, given news and highlights related to the kernel is generally blogged here and discussed. So returning the favor is only normal. There are no relations or communications between parties. \r\n\r\n_Eventually this list will grow too big to be managed here properly. At this time a new issue, which will be locked and closed, is scheduled to be made to keep all older kernel highlights, with a link reference from here._\r\n<br />\r\n\r\n\r\n\r\n## Sources, relevant literature, & existing instructions ##\r\n_More may be added in the future. If you know, or find, good sources, literature or instructions that may provide value for this issue's purpose, then it'd be very appreciated if you share it here._\r\n\r\n**General insight**\r\n- Getting started: Introduction \r\nhttps://en.wikipedia.org/wiki/Linux_kernel\r\n- Quick Practical Introduction - Is not adjusted to Qubes, do not use blindly!\r\nhttps://kernelnewbies.org/KernelBuild\r\n- Fedora Wiki - Is not adjusted to Qubes, do not use blindly!\r\nhttps://fedoraproject.org/wiki/Building_a_custom_kernel\r\n- Debian FAQ - Is not adjusted to Qubes, do not use blindly!\r\nhttps://www.debian.org/doc/manuals/debian-faq/index.en.html#contents\r\n- Debian official Manual - Is not adjusted to Qubes, do not use blindly!\r\nhttps://www.debian.org/doc/manuals/debian-reference/ch09.en.html\r\n- Debian kernel Handbook - Is not adjusted to Qubes, do not use blindly!\r\nhttps://kernel-handbook.debian.net/\r\n<br />\r\n\r\n**Qubes specific**\r\n- Current Qubes Linux Kernel Source (Master Branch):\r\nhttps://github.com/QubesOS/qubes-linux-kernel/tree/master\r\n- Current Qubes Linux Kernel Source (All Branches):\r\nhttps://github.com/QubesOS/qubes-linux-kernel/branches\r\n- Qubes kernel build instructions (may be outdated): \r\nhttps://groups.google.com/d/topic/qubes-users/yBeUJPwKwHM/discussion \r\n<br />\r\n\r\n**News & Highlights**\r\n- Frequent kernel mews & highlights\r\n  - https://www.phoronix.com/scan.php?page=category&item=Software\r\n  - https://www.linux.com/news/category/linux-kernel\r\n- Less frequent kernel news & highlights\r\n  - https://www.qubes-os.org/news/\r\n  - https://www.kernel.org/category/site-news.html\r\n  - other relevant sources to help keeping informed, feel free to add or suggest!\r\n<br />\r\n\r\n**Benchmarks & performance**\r\n- VM Performance Showing Mixed Impact With Linux 4.15 KPTI Patches\r\nhttps://www.phoronix.com/scan.php?page=article&item=linux-kpti-kvm&num=1\r\n- Linux 4.0 To Linux 4.15 Kernel Benchmarks\r\nhttps://www.phoronix.com/scan.php?page=article&item=linux-40-415&num=1\r\n- Linux 4.14 File-System Benchmarks: Btrfs, EXT4, F2FS, XFS\r\nhttps://www.phoronix.com/scan.php?page=article&item=linux414-fs-compare&num=1\r\n- Feel free to add or suggest others.\r\n<br />\r\n\r\n## Remember! ##\r\nJust like everything else in Qubes Community Collaboration, this is a volunteer issue. That means, you cannot expect people to help you with your questions or expect anyone to reply. It is recommended to provide as much information, logs, etc. as you can, however, also try to provide relevant information and logs. If you're in doubt, feel free to ask.\r\n<br />\r\n\r\n## Contributing to this first post ##\r\nCredit is due and included for whom help improve this post. If you do not want to be credited, please explicitly say so. Feel free to edit post and add, or post suggestion below so that others can modify the post to include anything to which may be of value to this issues topics. Keep in mind that not everything will be edited into the first post, but only content which may be of value to the general reader, or smaller specific Qubes user groups.\r\n<br />\r\n"},"comment":{"url":"https://api.github.com/repos/Qubes-Community/Contents/issues/comments/388148519","html_url":"https://github.com/Qubes-Community/Contents/issues/26#issuecomment-388148519","issue_url":"https://api.github.com/repos/Qubes-Community/Contents/issues/26","id":388148519,"user":{"login":"Aekez","id":33869013,"avatar_url":"https://avatars1.githubusercontent.com/u/33869013?v=4","gravatar_id":"","url":"https://api.github.com/users/Aekez","html_url":"https://github.com/Aekez","followers_url":"https://api.github.com/users/Aekez/followers","following_url":"https://api.github.com/users/Aekez/following{/other_user}","gists_url":"https://api.github.com/users/Aekez/gists{/gist_id}","starred_url":"https://api.github.com/users/Aekez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Aekez/subscriptions","organizations_url":"https://api.github.com/users/Aekez/orgs","repos_url":"https://api.github.com/users/Aekez/repos","events_url":"https://api.github.com/users/Aekez/events{/privacy}","received_events_url":"https://api.github.com/users/Aekez/received_events","type":"User","site_admin":false},"created_at":"2018-05-10T18:48:04Z","updated_at":"2018-05-10T18:48:04Z","author_association":"OWNER","body":"In order to kick off this thread and get it rolling, here is an issue I encountered which I'm having trouble debugged and solving. Hopefully discussing this type of issue can help others in the future too.\r\n\r\n**The uncertainties around the issue**\r\nThis laptoop `HP Envy x360 15 (Ryzen 5 2500U, Radeon Vega 8)` requires minimum kernel 4.15, due to the vega 8 graphics integrated into the kernel 5 2500U. Without kernel 4.15+, there will be no graphics support, which also results in xorg failing as a result. I've compiled and build the the current only 4.15+ Qubes kernel, `4.16.2.-1.pvops.qubes.x86_64` and the kernel has been successfully tested on other Qubes OS installations. However, the moment this kernel is used on this laptop, it fails. It seems that it is either due to the NVMe drive, or more likely, the graphic card. There are some ACPI errors, but to my knowledge it shouldn't prevent it from booting when doing acpi=off, there seems to be something else wrong, and in general other Qubes (Linux) Ryzen systems have a lot of acpi errors as of late, but none of the ones I've seen working despite the acpi errors are with integrated graphics either, so it may be too early to ignore the acpi errors, but nonetheless, it seems to be something else causing the kernel panic issue. There is also the question if this is related to the NVMe driver or the graphics card. I cannot easily test this without opening and void the warranty by inserting an SSD instead of the NVMe, and the NVMe seems to annoyingly conflict with Qubes installed on USB. For now though, my prime suspect is a wrong config setting in the power/memory management in the kernel config of the compiled kernel.\r\n\r\n\r\n**Different testing results with other OS's and kernels**\r\n- Fedora 27 works reasonably perfectly on the x360 laptop.\r\n- Fedora 28 works reasonably perfectly on the x360 laptop.\r\n- Ubuntu 18 works reasonably perfectly on the x360 laptop.\r\n- Windows 10 works reasonably perfectly on the x360 laptop.\r\n- Qubes 4 with my compiled kernel 4.16.2.-1.pvops.qubes.x86_64 works on older HP laptop.\r\n- Qubes 4 with my compiled kernel 4.16.2.-1.pvops.qubes.x86_64 does **not work** on my new HP laptop.\r\n- x360 seems to work without kernel panic on any included kernels from either install, or updated from qubes-update. However none of the default Qubes kernels are new enough to be 4.15 to include the graphic support.\r\n\r\n**The major difference between default kernel and the compiled kernel**\r\nUsing Qubes kernels build by the Qubes OS staff, there are no kernel panic, and only the natural no graphic when using kernels below 4.15. When compiling and upgrading to kernel 4.16.2.-1.pvops.qubes.x86_64, the graphics seem to work (the amdgpu driver loads, no xorg errors, etc.), however a new problem appears, which looks like memory/power-management issues related to the graphic driver.\r\n\r\nWhat it looks like to be, is a mixed problem of memory and power management, strictly related to the GPU, and my guess it is probably a kernel configuration issue, but I'm not sure which one to adjust, the search engine gave no usable leads to go on either. I'm not fully sure if this is the case, but it is my one and only lead for now, and it does seem likely too. Furthermore, using acpi=off as a base kernel setting in grub config during boot, and changing the graphic driver settings, seems to change the error quoted below in some way or another, however it more looks like the length of the error gets shortened, it doesn't actually change. I'll transcribe more errors if it is needed to debug, for now I included the default one below with just acpi=off.\r\n\r\n**There is a big clue - The current prime suspect**\r\nThis laptop is successfully tested to work on various of the latest different major Linux distributions. The failure on Qubes does not appear to be related to IOMMU or other Qubes specific hardware requirements, given its showing a more general Linux hardware support issue. The older kernels seems to work just fine (except for the lack of graphics of course). What is failing, is the customized compiled kernel from the current latest Qubes kernel https://github.com/QubesOS/qubes-linux-kernel/tree/master\r\nIt seems to me, that it almost certainly must be a configuration done wrong for this hardware, somewhere in https://github.com/QubesOS/qubes-linux-kernel/blob/master/config-base but so far, I have no idea which setting this may be.\r\n\r\n**The kernel panic on kernel with the kernel 4.16.2.-1.pvops.qubes.x86_64**\r\nThe only useful information I could extract from the generic kernel panic is the lines on-screen before the kernel panic and shortly thereafter forced reboot. This is with acpi=off, so the acpi errors are not included.\r\n\r\nThe (( )) messages below are my own inserted notes, and (~) represents memory/code address code-lines (bunch of numbers and letters), I'll edit and include these if they are needed to debug.\r\n\r\n```\r\n((Starting from the usual green (OK) messages during Qubes boot))\r\n- Starting The Xen xenstore... (OK)\r\n- Starting Qubes memory management... (OK)\r\n- Starting Daemon for power management... (OK)\r\n- Started D-Bus System Message Bus. (OK)\r\n- general protection fault: 0000 [s1] SMP MOPTI\r\n- Modules linked in: btusb btrtl btbcm btintel, and the list goes on for several lines of various different modules. \r\n((Oddly it posts a second output (two outputs with a list of modules) without any explanation as to why, but some of  the modules in the 2nd output could be part of the reason it fails, so I'm suspicious. For example it lists sensors (previous sensor errors, probably not important though), amdgpu (the graphic card driver), gpu_sched, drm_kms_helper, drm, nvme, mmc_core, and some other less suspecting modules, like wireless (which might not work out of the box anyway, since its new hardware, but I cant tell without actually booting it up first. Then it lists tpm, input_polldev, etc. Most of these I suspect may be failing in one way or another, and many of them are directly linked to the graphic card (like gpu_sched, drm_kms_helper, drm, mmc_core, etc. and I've also heard of some kernels not handling nvme drives very well either). Plus wireless cards are known to not work out-of-the-box when new models. So most of these modules seems to be failing modules, although it doesn't explicit say so. Then the next line goes;)) \r\n- CPU: 0 PID: 8 Comm: swapper/0 Tainted: G       WC      4.16.2.-1.pvops.qubes.x86_64 #1 \r\n- Hardware name: HP HP ENVY x360 Convertible\r\n- RIP: e838:switch_mm:irqs_off+8xe2/8x510\r\n((A whole bunch of code addresses specified by letters and numbers that I don't believe is needed, referenced as (~), however I'll write them down if they can be useful to debug.))\r\n- RSP: (e02b:ffffc900017c7940) EFLAGS: (~) ORIG_RAX: (~)\r\n- RAX: (~) RBX: (~) RCX: (~)\r\n- RDX: (~) RSI: (~) RDI: (~)\r\n- RBP: (~) R08: (~) R09: (~)\r\n- R10: (~) R11: (~) R12: (~)\r\n- R13: (~) R14: (~) R15: (~)\r\n- FS: (~) GS: (~) knIGS: (~)\r\n- CS: (~) ES: (~) CR0: (~)\r\n- CR2: (~) CR3: (~) CR4: (~)\r\n- Call Trace:\r\n__shcedule+0x39d/0x880\r\n? hrtimer_start_range_ns+0x199/0x2c0\r\nschedule+0x32/0x80\r\nschedule_hrtimeout_range_clock+8xb9/0x1a0\r\n? __hrtimer_init+0xb0/0xb0\r\npoll_schdule_timeout+0x41/0x70\r\ndo_sys_poll+0x514/0x5d0\r\n? _raw_spin_unlock_irqrestore+0x16/0x20\r\n? ep_poll_callback+0x11d/0x2e0\r\n? __alloc_skb+0x96/0x1c0\r\n? __wake_up_common+0x96/0x100\r\n? _raw_spin_unlock_irqrestore+0x16/0x20\r\n? __wake_up_common_lock+0x09/0xc0\r\n? cond_resched+0x16/0x40\r\n? mutex_lock+0xe/0x30\r\n? unix_stream_read_generic+0x225/0x0e0\r\n? compact_poll_select_copy_remaining+0x140/0x140\r\n? import_iovoc+0x37/0xd0\r\n? unix_stream_recvmsg+0x53/0x230\r\n? unix_state_double_unlock+0x40/0x40\r\n? ___sys_recvmsg+0x18b/0x230\r\n? seccomp_run_filters+0x58/0xb0\r\n? xen_set_pte_at+0x8d/0x520\r\n? __seccomp_filter+0x59/0x170\r\n? ktime_get_ts64+0x58/0x520\r\n? sys_ppoll+0x162/0x180\r\n- SyS_ppoll+0x162/0x180\r\n- do_syscall_64+0x74/0x180\r\n- entry_SYSCALL_64_after_hwframe+0x3d/0xa2\r\n- RIP: 00xx:0x776c0fa3dfad\r\n((Below different numbers to the above numbers and letters))\r\n- RSP: (002b:00007ffe225f0f80) EFLAGS: (~) ORIG_RAX: (~)\r\n- RAX: (~) RBX: (~) RCX: (~)\r\n- RDX: (~) RSI: (~) RDI: (~)\r\n- RBP: (~) R08: (~) R09: (~)\r\n- R10: (~) R11: (~) R12: (~)\r\n- R13: (~) R14: (~) R15: (~)\r\n- Code: 85 c0 74 29 48 3b 90 e8 +2 ++ ++ 74 20 48 8b 80 70 03 00 00 83 e0 03 83 83f8 01 74 11b9 49 00 00 00 b8 01 00 00 00 ba 00 00 00 00 <0f> 30 48 89 e0 48 8b 53 50 48 c1 e8 27 25 ff 01 00 00 48 8d 3c\r\n- RIP: switch_mm_irqs_off+0xe2/0x510 RSP: ffffc900017c7940\r\n---[ end trace b98f9ba97a6ed38 ]---\r\n- Kernel panic - not syncing: Fatal exception \r\n- Kernel Offset: disabled \r\n```\r\n\r\nScreen stays like a freeze for a second or two, and then goes into a reboot. Repeat in an endless circle.\r\n\r\n**About the hardware**\r\nThis is a **HP Envy x360 15 (Ryzen 5 2500U, Radeon Vega 8)** laptop.\r\nWhat I in particular like about it (in regard to Qubes) is the powerful CPU, RAM and NVMe/SSD upgrade-ability, and it also has two USB controllers. Of course it has closed firmware blobs though, but hopefully we'll see POWER laptops one day in the future. A big minus Qubes or Linux on general on this particular hardware is that it requires Windows to upgrade the BIOS, but it can be quickly worked around by re-installing (still sucks though). Actually it looks like to be like that for other HP models too. Anyway, the BIOS being windows upgrade only, is the only minus of the laptop I found so far (taking aside the kernel issue, but it doesn't look like it's caused by HP, after all, most other Linux distributions work just fine on this machine, it's only Qubes failing).\r\n\r\n\r\n**Any thoughts on this issue?**\r\nI'd appreciate even the smallest of smallest clues or leads. I've been trying to debug this machine for a while now, and anything new could be really useful."}}
{"actor":{"display_login":"Aekez"}}