Latest Posts

Fedora 31 mini-review on the Blackbird and Talos II

As promised here's my periodic mini-review after upgrading both our Blackbird and Talos II systems to Fedora 31, the most current release, typed up in Firefox 70 running on Fedora 31 on my T2. Even though there are many of you who don't run Fedora on OpenPOWER, these reviews are still relevant because Red Hat does a lot of the work on the components you do use, and problems are likely to turn up here first. Much to my disappointment, one late breaking note is that 128-bit long double still isn't in Fedora ppc64le, and didn't make 31 either. I can't tell from the bug or the wiki page why the deadline keeps slipping.

I did the upgrade first on my home theater GPU-less 4-core Blackbird because it was already bitten by the librsvg2 issue and early reports indicate the updated LLVM-rustc pair in F31 fixed it. The steps are the same as I used for F29 and F30 except for changing the parameter to --releasever=31 (duh). A quick check demonstrated updating librsvg2 to the latest available for F30 didn't solve the problem, so I went on to downloading the packages for F31.

When I rebooted into the F31 installer, however, the projector freaked out and went into an endless loop of trying and failing to sync to the display. I don't know if it was unhappy with the video mode the installer set, but even the A/V receiver wouldn't pass through the HDMI video (the T2 did something similar which I'll note in a moment). I eventually had to pull up a second VTY and then and only then would the projector display anything. I then logged in as root and monitored the messages from dnf with periodic dnf system-upgrade log --number=-1 | tail -10 until the machine rebooted on its own.

Fortunately, F31 came right back up. I've done only minimal customization on the Blackbird, so pretty much everything transferred over unchanged, and no packages had to be dropped to do the installation. F31 comes with GNOME 3.34, which is alleged to have performance improvements, and actually I was very pleasantly impressed as you can see from the screen shots:

Video playback on this GPU-less Blackbird was a lot better in this release; in fact, Firefox 70 didn't drop any frames or audio at all (though I'm sure the rapidly improving VMX/VSX support has something to do with it ;). Although VLC on the unaccelerated Blackbird is still not perfect and playback was not completely smooth, pixel pushing was much improved in both DVD and Blu-ray playback and there were fewer dropouts with the TOSLINK surround sound (mplayer of course still played everything just fine). Unfortunately, I think the improvements are strictly in Mutter and GNOME itself, not llvmpipe, because Xonotic was still only ekeing out a bare 5fps at 1920x1080 as in F30.

As advertised, librsvg2 was working again. If you have exclude=librsvg2 in /etc/dnf/dnf.conf, you should remove it before you do the update. Every GNOME release has some vanity changes for no good reason and the new icons and minor UI tweaks seemed largely unnecessary but they weren't objectionable. On the apps side, GNOME Videos doesn't seem to grok the length of my AIFF music files correctly, though it does play them (MP3 was fine). GNOME Web was also working again after a long hiatus but it seemed to have minor glitches, and since there are people who use POWER9 now who are actually helping to maintain Firefox, you should just use Firefox.

Since I didn't find any obvious major regressions in my normal usage, the next step was to update the Talos II. The T2 does not have the WX7100 firmware in the BMC PNOR, so I expect to run the installer "blind," but interestingly my LCD would not sync to the display either just like the projector wouldn't. The LCD synced fine when I popped open a VTY, just as with the Blackbird, so I'm thinking there's something up with the installer's video mode. Otherwise, the install proceeded unattended and rebooted uneventfully.

As my daily driver the T2 is rather more customized than the Blackbird. It's pretty much a given that I'll lose some of my GNOME extensions in the upgrade or the custom "classic" OS X-like theme I use will have some odd breaking edge case, and that happened here as usual. In this case Dash to Dock was the casualty and the GNOME Extensions Manager refused to update it, requiring me to manually install it. Tweaks and Settings still have visual issues with my theme, but didn't seem worse, just annoying. The only thing installed that didn't transfer over were my custom Perl libraries which got eaten and needed to be reinstalled. I know, I know, I'm the last person on Earth who still likes Perl apparently.

On the T2 with its WX7100 workstation card, the graphical performance improvements were not as notable as with the Blackbird, but some things seemed better, and a few 3D games that chugged a bit on F30 seemed faster on F31. I'd still say performance was a net win, just smaller.

Both systems use X.Org, but I do try to at least test Wayland. My T2 is configured to come up in a text boot so that I have a console to fall back on, an artifact of originally being Fedora Server and converted to Workstation. I was able to start GNOME in Wayland from the command line with XDG_SESSION_TYPE=wayland exec dbus-run-session gnome-session (instead of startx). I would call it incrementally improved from before. Some apps (mostly games) still don't start, and some games that do start have odd aspect ratios, but more at least work. The issue with some apps, particularly XWayland ones, not obeying GNOME theming seems to be fixed, and while it didn't feel quite as snappy as X.Org it was still better than previous releases. However, my custom appmodmap tool for dynamically remapping the keyboard only works with things that run in XWayland, because it watches X events to know which window is up, and the GNOME Wayland compositor currently has no plans to offer this information. So back to X.Org.

However, the situation was even worse with the Blackbird. Since overall graphical performance seemed better I decided to push my luck and see how it worked in Wayland (which previously ran like treacle on Thorazine in January north of the Arctic Circle), but as soon as I switched to Wayland and rebooted, this time the Blackbird would not come up in a graphical boot at all. On several test boots after the kernel messages it immediately went to a grey screen with the mouse pointer and then froze hard, requiring me to power cycle it because I couldn't open any VTYs or get to the OS. Since this is a workstation installation rather than a server installation converted into workstation, I had to boot the Fedora rescue installer to fix /etc/gdm/custom.conf because I couldn't get the machine to come up otherwise. If you are installing F31 from scratch, you may want to make sure that WaylandEnable=false is uncommented before you try your installation out.

Overall Fedora 31 is both a good release and a bad omen. Performance (at least in GNOME under X.Org) is overall much improved, especially if you don't have a GPU, but it's still obviously better even if you do. Some bugs were fixed and packages installed uneventfully. There were the regular growing pains in GNOME, but I didn't lose anything irreplaceable, and other than the usual bumps one experiences with custom themes and extensions pretty much everything just worked.

But Wayland on ppc64le continues to be worrisome. I must concede that at least on the T2+WX7100 things has improved since F30, and since I freely admit I'm a Wayland sceptic those of you who are heavily invested in it probably don't care about my opinion. But overall it's still a step backwards because there are still things that won't run in it, a big part of my own personal workflow may never work with it, and on the GPU-less Blackbird beforehand I couldn't use it and now I can't even start the machine in it. Meanwhile, Red Hat's made some very public signals that Wayland is the future and X.Org will be going away. In their rush to do so not much attention is being paid to people using 2D framebuffers with Wayland, and this is a real problem because no currently available GPU is libre and not supporting the built-in BMC in every shipping Raptor system is a waste (not to mention requiring people to incur additional expense just to get something to work that was "already working"). If you want a truly blob-free system, right now you just plain can't use Wayland, and it doesn't seem like they care.

DD2.3 POWER9 steppings now available


If you have a T2 family system, you can order these today, and the SKUs are reported as in-stock. They are drop-in replacements for all T2s and Blackbirds and because their TDPs are the same can use the same heat sinks and HSFs. Systems shipping now may still have DD2.2 chips in them, though Raptor says you can get a DD2.3 for a slight upcharge.

Talos II and Talos II Lite officially FSF Respects Your Freedom products


The designation recognizes that the T2 and T2 family have full system schematics and source code available for the entire firmware stack from the BMC up, and no keys are needed to update or replace any firmware component unless you require your own. (The same applies to the Blackbird, too, of course; presumably its own FSF RYF certification is soon to follow.) Naturally the designation presupposes you are using a free distribution, as the FSF defines it.

The T2 family joins a relatively small number of complete systems that have RYF endorsements and given those systems' loadouts is easily the most powerful, at least of this writing. Not only is this a nice win for Raptor, who have made libre computing a cornerstone of their company, but it's also a great validation for OpenPOWER. A designation like this from the FSF, who stakes their entire reputation on libre computing, is no small matter no matter how you slice it. Congratulations!

FreeBSD 12.1 available


The installation directions for the Blackbird should work as is for the Talos II. However, if you've already got the ISO (not the .img) dd'ed to a USB stick, it seems to me that it should "just work" in Petitboot without all the goofing around at the BMC prompt (if you don't, though, then these instructions will allow you to bring the machine "up from nothing").

If you are already running FreeBSD, unfortunately it does not seem that the PowerPC port of FreeBSD supports freebsd-update(8) yet, though I imagine this is planned. FreeBSD 13-CURRENT boots and runs fine on the Raptor family as well, but no clear word on when that will reach release yet.

Fedora 31 available




Firefox 70 on POWER








Is the warrant canary still warranted?





(*not sponsored or endorsed by Allstate)

Ubuntu 19.10 available


librsvg2 issue on ppc64le







CentOS 8 and CentOS Stream (The Freshmaker)

Yeah, okay, we've had a lot to say about Red Hat derivatives lately. On the heels of CentOS 7's latest service release now comes CentOS 8 in a new minty flavour CentOS Stream, "a midstream distribution that provides a cleared-path for participation in creating the next version of RHEL," rebranding the "classic" CentOS build from RHEL as CentOS Linux. Mentally translating, the intention appears to be as a staging area for updates from Fedora mainline to trickle into minor releases of RHEL (and thence to mainline CentOS), using CentOS Stream to more gradually introduce updates and incorporate user feedback in a rolling release fashion rather than the typical all-at-once version churn that previously resulted. You know, like chewy candy mints that make things fresher the moment you pop one in your mouth.

That said, mainline Fedora is plenty stable for (my) daily use on POWER9 and elsewhere (we're not talking bleeding-edge saddle sore Rawhide, kids), and Fedora will still be the ultimate upstream, so while I think this will help CentOS developers dogfood changes more gradually I'm having difficulty envisioning the small slice of conservative-but-not-that-conservative users this will appeal to as a daily driver. More likely people will simply regard it as the "public beta" channel for CentOS and RHEL, and I think that will be the actual role it serves regardless of the frilly language.

The CentOS Download site is not currently showing Power ISA (or other AltArch) builds for either CentOS 8 or CentOS Stream yet, but I expect these to emerge soon. It will be interesting to see if big-endian ppc64 is still supported when they do, but there should be POWER9 and "generic" little-endian builds at minimum.

Low-level change to Firefox 70 and ESR coming

If you are using Firefox on 64-bit Power, you'll want to know about bug 1576303 which will be landing soon on the beta and ESR68 trees to be incorporated into 70 and the next ESR respectively. This fixes a long-standing issue with intermittent and difficult to trace crashes (thanks to Ted Campbell at Mozilla for figuring out the root cause and Dan Horák for providing the hardware access) due to what in retrospect was a blatant violation of the ELF ABI in xpconnect, which glues JavaScript to native XPCOM. This needed several dodgy workarounds until we found the actual culprit.

The patch is well tested on multiple little-endian systems including this Talos II, but because it's an issue with register allocation in function calls the issue also theoretically affects big-endian Power even though we haven't seen any reports. I'm pretty sure the code I wrote will work for big-endian but none of my big-endian Power systems run mainline Firefox (and TenFourFox even on the G5 is 32-bit, where the problem isn't present). If you're using a big-endian system, you may want to pull a current release and make sure there is no regression in the browser with the changes; if there is and you can bisect to it, post in the bug so we can do a follow-up fix. On the other hand, if you're building from an old ESR such as 52 (the last non-Rust-required one), you may want to backport this fix because the problem has been there pretty much since it was first written.

Stuff like this actually proves Linus Torvalds' point that "as long as everybody does cross-development, the platform won't be all that stable." Linus was talking about ARM-based servers being undercut by a dearth of ARM-based PCs, but the point is also true here: 64-bit Power may do well in the data center but it was rarely used for workstations other than the Power Mac G5 and the small number of non-Apple PowerPC 970 towers, meaning this bug went undiscovered until people like us finally started dogfooding Power-based desktops again. (For that matter, the official PowerPC Mac OS X builds of Firefox were also always 32-bit, even on the G5, so no one would have noticed it there.) There's just no substitute for improving the quality and quantity of software for Power ISA like having one under your desk, and as the number of machines increases I expect we'll get more of these ugly corner bugs ironed out in other packages too.