Void Linux: we heard you like little endian
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kolesa, Daniel
Void's POWER architecture port has been progressing steadily since the last OpenPOWER Summit EU talk in 2019. Recently we introduced a completely new 32-bit little endian port, which will be a big part of this talk's focus, and is a first among Linux distributions. I will not stay there though - we have more to cover, including stuff like Chromium and Electron applications in repos, faster POWER crypto in LibreSSL, reworked crosstoolchains, stable support for newest AMD GPUs, and our big endian variants are also receiving attention, including properly clearing up the 64-bit ABI situation. Void Linux is an independent, rolling release, general purpose Linux distribution (leaning towards desktop/workstation focus) originally created in 2008 on the x86_64 architecture as a testbed for its own XBPS package manager. Over time, it has received a variety of ports, including 32-bit and 64-bit ARM, MIPS, and eventually POWER and PowerPC. It has variants for 64 and 32 bits, little and big endian, and glibc and musl C standard libraries. Most recently it has received an experimental 32-bit little endian port, which is a first among Linux distributions. I will be focusing partially on the new port, and partially on other news in the distribution since my last talk I gave at OpenPOWER Summit EU 2019. I will explain our goals with the new port, as well as our plans. Additionally, the future of the distribution will also be covered, as well as currently remaining issues and blockers that prevent us from achieving that.
 you Hello everyone and welcome to my presentation on Void Linux. I'll be talking about the progress of Void Linux on the power architecture in the last year and a half or so. But first who am I? I'm a programmer from the Czech Republic and I've been involved in open source since 2007 or so. I have contributed to a fair amount of projects over time. One big one is the Enlightenment Foundation libraries which is the set of libraries intended for the Enlightenment desktop environment. I've also messed around with game development, programming, language compilers and so on. But these days I'm the primary maintainer for power architecture, port in Void and a few other things in Void. So what is Void? Of course I gave a talk previously at the OpenPower Summit 2019 in Lyon, France. But there are probably many of you who haven't seen the talk so I'll describe it in short. It's an independent Linux distribution with custom package manager. It was originally created as a test bet for this package manager. It has a focus on simplicity but it's not like these minimalist distributions. It's more like tries to strike a balance between the two things. We have a fair amount of packages and we have a similar functionality to other big distributions. In order to stay simple, it's rolling release so we don't have to deal with release management. We mostly ship stable software though. It doesn't really ship lots of bleeding edge stuff. We make sure that everything stays stable. It's binary based but there's a system called XPSSRC which is similar to BSD ports like system which you can use to build your software packages from source. So basically every package which is present in the repo is buildable from source using XPSSRC. It's sort of one or three kind of thing with all the package recipes. Of course besides power which is supported at this point. It's on x86 where it started. It's on ARM and I-64 of course and we also have build profiles for MIPS but we do not ship any packages for MIPS. The intention is so there is a low barrier of entry, very open and inclusive and informal community. So if you want to contribute, it's easy for you to get started. So how is voice on power different from other distributions? Well for one we have a particular focus on desktops and workstations, not servers. Most distros are focused on servers. We feel like there's enough of that stuff and voice on power was originally created to provide the desktop operating system for me. So sort of selfish reason. We support both little and big end-in. With little end-in having the standard requirements for power rate or newer CPUs, big end-in runs on all the old Mac hardware and so on. What's different from other big end-in distributions? We use the modern ABI on G-Lip-C as well which is something other distros don't know. It's sort of an obvious decision for us because it was a fresh port and we didn't want to deal with 30 years old ABI so we just went with the modern one immediately. Of course there's also the page size in the kernel. We use 4k pages by default. Most distributions use 64k pages by default. 4k pages is what's also used by x86 and most other architectures so it's more compatible with different software and hardware. We also have wider desktop software support. This includes things like the Chromium browser and electron applications and so on. Now let's go over the quick history of Void because while I describe this in detail at OpenPower Summit, many of you are probably not in sync with this. I started the project in late 2018. This was because I bought Talos 2 Lite to use as my primary workstation. I wanted a desktop operating system for this and I used Ubuntu for a bit and I was not very happy with it. So I just did initial port and it was done in like three days and I had a graphical environment ready. Over the coming months I kept expanding the package coverage and eventually during 2019 we got to complete coverage on Lysolundian. There was also big-end-in ports from the start for 64-bit muscle Lysolundian and for 32-bit G-Lypsy and muscle Lysolundian. There was not a 64-bit G-Lypsy big-end-in port because of the ABI situation we weren't aware at this point that it actually just works. So this port came later in April. During late 2019 there was complete coverage on all repos and I gave my OpenPower Summit EU talk and in November I got a comment bit in upstream repos. So from that point all changes go directly into Void without actually lingering in my custom repo for too long. But this talk is not about this, it's about what happens since that point. So this screenshot it's visuals to the code, an application which lots of people have been requesting running on Void with PPC64 early and muscle. As you can see in the corner it's all printed there and it works fine. But first Java Bootstrap. That's one big thing we actually did and it was kind of a pain. We have OpenJDK8 and 11 in repos and since you need OpenJDK to build OpenJDK and you need a version which is one version older we had to eventually reintroduce GCC6 which shipped GCJ which is GNU Java compiler now deprecated but in GCC6 it was still present. We used this to Bootstrap OpenJDK7 which is a Bootstrap only package it's not meant to be used by people. JDK7 is then used to Bootstrap8 and then we have Bootstrap packages for 9 and 10 so we can eventually Bootstrap 11. This is a fairly fast process on PPC64 and early because there's a JIT in there and it takes maybe 10 minutes or so to Bootstrap one OpenJDK. On 32-bit PPC this is very slow because there's no JIT support there's only the interpreter and it instead of 10 minutes it takes like four hours so that's kind of pain but oh well. Go. Go is the language which is fairly widely used for specific kinds of software and we have a bit of a problem there because Google who maintain the tool chain have only support for PPC64 LE. Well technically speaking they do have BE also but their BE part uses incorrect ABI or well do ABI we do not use and it also needs power rate which would be a problem by itself because we make sure that all software in the repo runs on all hardware we actually support. There's a GCC Go but GCC Go is unreliable. For some reason it hangs while we do know the reason it there's some bug in variable tracking code which results in GCC Go entering infinite loop when you try to actually compile Go with it. So we can add this to Bootstrap we use official binaries to Bootstrap these official binaries are provided for every architecture they are not provided for muscle lip see but for muscle lip see we deal with it by using G-compact. Since the Go compiler is written in itself it doesn't actually use a lot of lip see at all so G-compact actually can run it reliably and we can bootstrap with the official binaries pretty to a base support of course is missing completely we don't have it. Askel. Askel was also fun under some definitions of fun we use a self-built binary snapshots to bootstrap the compiler this includes muscle up to this point because recently I figured out how to cross compile a partial back binary distribution of GCC this can be unused to compile the actual distribution on a native environment. Bigendin is missing for now because of some bugs in the compiler we will try to introduce it for 8.10 for now the patches for future Bigendin introduction are actually ready. Other languages we since recently had greater support for common lisp as PCL is finally in the repos but for now only on little and in only to test the Bigendin much better. It's bootstrapped with ECL which is a portable common lisp compiler which works on every platform. We are also preparing an update to C-Lisp which hasn't seen a release since 2010 but there's an active version control so we'll make a snapshot from there. There's also things like D and Zigg and some others which are currently still missing they don't have anything depending on them so we haven't paid much attention to these yet. Libre.SSL performance. Void uses Libre.SSL instead of Open.SSL at this point. Libre.SSL is the fork of Open.SSL 1.0 made by OpenBSD project. It was with the intention to have greater security. The problem is it only has assembly files on x86 and some 32-bit ARM. These results in performance which is much inferior to Open.SSL and of course it also results in lack of hardware crypto support on PPC64 early and on R64. So I made a stopgap solution by importing those assembly bits from Open.SSL upstream and this is now present on my GitHub and it's shipping in Void. It has up to 20 times performance increase in things where hardware crypto can be used. This is especially visible for example in Open.SSL if you use SSHFS you can use this to increase your throughput from 18 max per second or so to like 500 max per second so that's a big difference. Chromium upstream does not support PPC64 early but there's downstream patches available since Chromium 84 we've been shipping these downstream patches. It works quite well. It's currently the only browser engine which has JIT support in the distro and in any distro and it works on muscle as well. It's a fairly large patch and Google has been rejecting it to most of it for a while now so there's a chance we'll have to ship it downstream forever but that's life I guess. Related to this we now have support for Electron which is a framework for applications so you can write applications in HTML5 and so on. It uses Chromium, it's basically a version of Chromium. I don't use it but people have been requesting different applications like Visual Studio Code and Element. These do work. They work because we have a system-wide Electron. There are some funny build system workarounds for these applications because Electron build system from Node.js tries to download the binary snapshot of Electron and it doesn't know about the architecture so it fails and what we do is just enforce the x8664 architecture, make it present, it's x86, it will download its binary snapshot which obviously doesn't work but in the end it will be using the system-wide one anyway and it's pure JS anyway so it doesn't make any difference. It's obviously only on LE just like Chromium itself. I tried patching things for a big ending before and it doubled the patch and it never worked quite right so I eventually gave up. It was too much of a pain to do downstream muscle. That's work. We had muscle support previously so I just had to add a small patch adding the PowerPC muscle specific things. MD GPUs. There's this problem called MD GPU DC or display core. It's in the kernel and it has a very haphazard usage of hardware floating point which results in for one it's not portable. At some point Raptor computing systems added some code to make it work with these new Navi cards but this was never back ported to older kernels so we back ported ourselves down to 5.4. Since 5.10 there's apparently a dependency on page size which is kind of bad but it still works for us because we have 4k pages. Other distros have issues with this apparently or at least that's what I've been told. They have 64k pages and AMD doesn't work with this. There was also a big rework of cross tool chains recently. There was this issue that we had many different cross tool chains. The list of targets grew a lot recently. We've needed to deal with this because the cross tool chain build recipes were duplicating lots of code so I introduced a common what we call build style for cross tool chains. Most of the code is in the build style and now each cross tool chain is only 50 or so lines defining the actual tool chain specific things. As some extra goodies which I introduced during cleaning it up. We now have a G-Lipseed cross tool chains on muscle hosts which was not possible before so we have much expanded support for this now. The configure arguments and so on are unified for all of the staff now so that's better also and of course no more dirty master there's previously the master there is a sort of a built container used by Xvpssrc. Previously the intermediate built artifacts because you need to build initial GCC without lipseed first. You need to build lipseed and you need to build the final GCC. The intermediate built artifacts were messing up the container. We now deal with this by just restricting everything to temporary directories. In first status this is a screenshot from our build server. It's currently still a Talos 2 Lite with 18 cores and 128 gigs of RAM and it runs in my bedroom it's actually next to me right now. This is not ideal of course but I don't have a better place to put it. Big and in built run in a VM thanks to KVM you can run cross-end in VMs and there's no performance decrease. The little ending builds run on bare metal and the primary actual package mirror where people actually download stuff it's on a separate server in Chicago in the USA. The mirror has 10g network and two terabytes of storage so we cannot really compete with this. I do have a public IP here but I don't think anybody would want to download stuff from me. There's several other mirrors provided by the community. We could certainly use faster build hardware and we could certainly use hardware which is not in my bedroom. The primary mirror after we have some better built machine which is actually co-located in a real server place we should migrate the primary mirror to that machine so it can be available immediately after built so we don't have to upload batches to actual server on the other side of the world. We also need to do more built automation this is work in progress I'm working on some software to actually improve the infrastructure. Now the main thing 32 bits a little end in. This is a screenshot from from a root userant which is 32 bit little end in you can see it's running. So first why why not? It officially does not exist but only officially. The thing is so you can run 32 bit binaries on a 64 bit power because 64 bit is just an extension of the original power PC and you can certainly run natively 32 bit binaries on 64 bit CPUs. This is on big end in though. On little end in this is actually the same because because that makes sense. In practice this therefore just works. The bring up process was fairly basic made easy by XPSSRC it was similar to the other architectures we have. But there is no native support for 32 bit little end in Linux kernel itself so we cannot actually boot this on real hardware. It does work as a root on 64 bit kernel for the compatibility stuff but not out of box so there are some fixes needed for the kernel. So first we have to have to fix 32 bit compatibility in the kernel. The first issue was that signal handlers were clearing the little end in bit in the machine state register. This was fixed for a 3-bill patch. Also 32 bit VDSO were disabled by default. This is just a build system change. Some VDSO's are broken on little end in. We had to disable these coincidentally we found that the power PC 601 had these same VDSO's broken. We are not a worry of any relationship between this but we just had to extend to the disablement. System calls with 64 bit arguments were also an issue because the high 32 bits and the low 32 bits were passed in the wrong order. There's some wrappers which do the passing so we had to fix the wrappers. All of these are very trivial patches and things do run without them but they also crash at some random points usually like at the obvious ones like when entering a signal handler or when exiting. After patching there's no more known issues at least we are not aware of any known issues. Fixes for this are being upstreamed into the kernel except the VDSO fixes which will be fixed by just moving the VDSO codes to see. This is a work in progress by other people in in current mainline so at one point it will just get fixed. There will be or there are backports to all void kernels which do run on little and then that is 4.19 and higher. Now for bootstrapping for the 2-bit little and then well first we had to create built-in cross profiles for Xbpssrc. These profiles define the target triplet and some other specifics of the compiler. Then we had to create cross tool chains for 32-bit elite targets. That was fairly easy we just had to copy the beginning and once and adjust the triplet and so on. Some minor patching was needed in G-Lip C and in Muscle. In G-Lip C just some warning fixes. In Muscle we had to adjust the dynamic linker name so it matches GCC properly. Adjustments around Xbpssrc were needed obviously just to look where architecture specific code is and usually just trivial adjustments needed. After this we could compile or cross compile a base change route. This is basically a minimal set of packages needed to assemble the container in which everything is built. Any errors had to be fixed along the way and there were relatively few and very trivial ones. After that we had to a basic set of packages which we could use to binary bootstrap a built container. Then we could rebuild every package of the build container natively without cross compilation. Then we could build other base math packages like base void strap and so on which add more software to the route and after that more software in general can be built like all sorts of different stuff in the repo. Of course when errors come up fix these. Usually there's not a lot of them. We did run into some issues with like a VSX code and say image decoder assuming that little ending means power 8. For now we compile for power 8 so this is not a problem for now but eventually these things will have to be patched. At this point I could probably function on a 32-bit user run if I compiled enough of the user runs. But there are some buts. There's no official support on G-LipTee that means the dynamic linker name is shared with big end-in. The simple versions are also shared with big end-in. This is not good for having an official port because every every architecture has to have a unique dynamic linker and fresh single versions also. So I'm going to propose an official port later and once it's actually upstream I will rebuild the whole experimental repo from scratch. The ABI is the same as big end-in and this is fine probably. The ABI is well defined for both end-ins and it must pretty much just works as it is. Even the dynamic linker name is stable and since muscle doesn't do a simple versioning there's nothing to worry about. Of course we did because of the we could but another big thing was the easier emulation of 32-bit x86 binaries. There's a project called Box86 I'll get to it later which we will be previewing. Of course eventually it could be possible to actually port Linux to G3s and G4s since the CPUs do support little end-in mode it would just need to actually work. Or we could take the easier path which I was made aware of and just use Be Linux kernel and then use the switch end-in call and the swizzle this is called data for user run. Apparently this should work I'm not sure how it would work yet but I was told it would work. For now it's strictly Power 8 Plus though so we only run as compatibility on our Power 9 workstations and so on. There are some obstacles for one LLVM support which is needed for MESA and some other things. Fortunately this was recently upstreamed by the free BSD folks who needed it for the loader. We back ported the patches to Voigt. It just came in handy because I was just about to start working on porting LLVM when I found that there are patches available already so this saved a lot of work for me. We also needed for Rust support which is currently blocking a fair amount of user run mainly because of things like clip-rs SVG which is depended on by GTK and lots of other things. There are some work in progress patches available which I wrote for Rust but it doesn't work at cargo crashes and so on. We need to figure all of this out. Proper G-Lift supports needs to be made and we could also use the opportunity to improve the ABI perhaps but we are not yet sure about this. One thing we definitely do want to do is use 64-bit long double type. This will allow us to ditch the weird IBM 128-bit floating point type which is basically composed of two standard 64-bit ones. G-Lip-C is moving on from this to real 128-bit floating point 464-bit but we cannot use this on 32-bit because we do not depend on VSX being present and these new floats require vector registers to pass them around. We all need to choose a unique dynamic linker name and fresh symbol versions for the new port. I'm kind of thinking we could use this to fix up our 32-bit big-end imports as well. I mean we could at least ditch IBM long double as well but I'm not yet sure on what the best technical approach to this would be to ideally preserve compatibility with all your stuff and ideally also make it upstream mobile. As for results there are testing repos now available and it can fully compile itself. A portion of the repo is packaged by portion I mean some 8% or so. This might not seem like much but lots of core things are already available and to lots of most other things could already built. We had just haven't built them yet. There's core user ones, there's development tools like the compilers, LVM, there's Mesa, there's SDL and of course there's initial port of Box86. What is Box86? Well Box86 is Linux X86 and later developed for ARM devices. It has nothing to do with PowerPC but we wanted to use this to emulate some old games which only had binary versions on our power machines. The problem is this needs a 32-bit little end-in host and you can probably see what I'm getting at. This is why we needed a 32-bit little end-in port. So there's now an initial PowerPC port done by our community. It can run GLX gears and I also got it to run Unreal Tournament. The old one from 1999 it works but it's fairly slow. Sometimes it's playable, sometimes it's not playable. There's no dynamic recompiler for PowerPC yet in Box86 so it's purely interpreted. Of course this is still faster than the likes of QMU because unlike QMU it actually runs lots of code natively. How Box86 works is that it wraps native system libraries for different functionality things like SDL and image libraries and audio libraries and OpenGL and so on which means basically most things which are provided by the system do not run emulated. They use the native versions and all the emulated code is the actual game only. Now for the future. This is the screenshot from our statistics page. You can see there's the coverage for at least very high for be it slightly lower and for 32-bit little end-in it's fairly small for now. One thing is infrastructure. I would like to get things made official at one point. Mainly the PPC64LE port should get an official airport one point. It should be easy because that port is stable and there's nothing much to change in it. As for BE this has some issues. BE because of the API situation which we need to clarify with upstream first. 32-bit has some issues also. There's also the issue that some server software on official Void repos is written and goes so it's not portable to these. This can be worked around differently but yeah. The LV2 BE-ABI we will need official support in G-Lipsey for this and right now things mostly work by accident. That means the dynamic linker name and symbol versions are shared with the 64-bit LE port. An upstream did say they wouldn't be against this but we would need to formalize the API first. IBM thinks VSX requirement is a part of the API. We do not entirely agree but we are still going to solve this so we are going to formalize LV2 legacy compatible API. This legacy compatible API will be just an extension of the current API. It will remove the requirements for VSX and VMX and it will also switch to 64-bit long doubles to the HD 128-bit IBM floating point. It will come during this year. It will be shared by G-Lipsey. It will be shared by Muscle and FreeBSD. Muscle and FreeBSD already implemented in the way I intended to specify so basically there's existing compliant implementations before the spec specification exists which is kind of funny but oh well. There will be other things. We will have to upstream as many patches as possible. We will want to write a new installer for our live media and also work on an enablement of new software and as well as working with upstream of different software projects which do not have power supported to add support for power. Also work has been done in our year and a half and there's still plenty more to be done. If you want to talk with us or work with us you can join our IRC channel on a free note and if you want to see some detailed statistics and so on you can visit our website. For now that's the end of it and I'll well I will show you some cool things but thanks for listening and there will be a live Q&A session after this. Okay so well added. I will show you some cool stuff I guess since we still have some time. Here I have a route of 32-bit little end-in Void Linux port. It's fairly minimal and doesn't have a lot of software in it but it does have Box86 in it. Obviously as you can see it has a build of Cmake and it has a build of dependencies for Box86 so we can try compiling it and since this is slow we can try compiling it faster. And it's linking I believe. Well maybe not yet. This will take a bit. Well for now we have some other stuff. We have a build of GLX peers which we are now going to run with this. It spawned on a different screen so let's move it just over here. As you can see it's running at native refresh rate of the screen and admin 60 FPS but it can do the 60 FPS. Obviously it's mostly open GL application and very simple so even emulated it will run at decent frame rate. But I have a cooler frame and that's Unreal Tournament. This will take a bit to start up because there's a screen recording and so on running so yeah. And here it is. You can see this is choppy. It's usually faster but yeah there's other stuff loading the CPU right now. You can see at some points it's also fairly smooth. It depends on how far into into the screen you can see. Anyways here's the menu. We can start a practice session. I believe this is also an unoptimized build of box86. So this will also run faster if we try to... You see now it's like smooth if there's no distance visibility but once you can actually see far it just blows up. But yeah it does work and I think that's pretty good enough for now. So I think that's the end and there will be a live Q&A session after this so feel free to stop by and ask some questions. Okay we're going to go live in a few seconds. Hi Daniel. So you're going to answer some of the questions or should I read them out? So... There were some questions about how the builds were done. I don't know if you could elaborate on that level. I think you're still muted. Hold on now you should be able to hear me. Yes now it's better. Okay good. So let's start with further questions I guess. Yes. So what exactly were you asking regarding the builds? So there were some questions about you. You mentioned that you have a scratchpad SSD going through VIRT.io. Oh yeah like this. Yeah basically we were... The first thing we did was just set up a big end-in VM and to use a QCao2 disk to like store the operating system and then in order to like share the repo directory with the VM so it can actually write packages into repo we just exposed it over VIRT.io.9p. Of course this has problems because the throughput of the QCao2 storage is not as great as just a native SSD and it was actually slowing down the builds. So we got around to this by just buying a new SSD separately and just splitting into half and exposing a one-half or one partition as a block device directly to the VM and that way it can basically run as fast as native. It does run as fast as native so far it works very well. Okay then there were some questions around Embro-la which is broken. I don't know if you could comment on that. I don't see that here actually. Okay so Embro-la is broken on little end-in and apparently it worked on the big end-in. Okay that's interesting we've never run into this so I haven't really looked into it so I don't think I can answer this but if you can point it to me in chat so I can take a look afterwards. Yeah I haven't actually also had that error or that issue on on our builds so that's why I was just asking maybe you had that experience but we will see if T-platin can can actually comment in chat more on that. Okay I see some questions in the chat so I'll reply to these directly. I see with Box86 I could be able to run Unreal Engine for x86 on power. No because it's mostly suited for old games. If you run a new stuff on this it will run very slow. I tried to run the well-known Linux game Xanatic on Box86 just as an experiment to see like how slow it would be and it's not a particularly demanding game it works on my testing G5 but in the Box86 emulator it runs at like well every five seconds it updates the frame. The menu runs fairly fast but once you get in game it slows to crawl. I see a second question what package manager is void most similar to? Well it's kind of its own thing really like it uses several binaries xbps install xbps remove xbps query and so on and then it uses like regular positional regular optional arguments to pass to that so like to upgrade your system you run xbps-install then you pass capital S and lowercase u once is to sync the repo indexes once is to upgrade if you want to install a package you just run xbps install your package name so it's not particular similar to anything else it's like a hybrid between most things. As for Qt5 packages cross-compile to h64 well Qt5 packages can be cross-compiled for h64 I don't think there's any issues with this because we do cross-compile and for h64 and void I mean our ARM ports are fully cross-compiled and Qt5 is definitely present there what's problematic is Qt5 web engine which can be cross-compiled but the problem is it needs to match the word size of the host and target so if you want to cross-compile for 64 bit target you need to cross-compile from a 64 bit host if you want to cross-compile for 32 bit you also need a 32 bit host this is a bit of a problem for us because we cannot cross-compile Qt5 web engine for 32 bit ARM because we cross-compile from 64 bit host we could deal with this by using a 32 bit build container but this is not it and what's the performance of a 32 bit kernel user run to running on power 8 or power 9 versus 64 bit one well I haven't really done any in-depth for benchmarking but it seems to be fairly close I haven't noticed any actual difference in things like compilation times things mostly compile about as fast on a 32 bit CZ run and on you
 you  Hello everyone and welcome to my presentation on Void Linux. I'll be  talking about the progress of Void Linux on the power architecture in the last  year and a half or so. But first who am I? I'm a programmer from the Czech  Republic and I've been involved in open source since 2007 or so. I have  contributed to a fair amount of projects over time. One big one is the  Enlightenment Foundation libraries which is the set of libraries intended for  the Enlightenment desktop environment. I've also messed around with game  development, programming, language compilers and so on. But these days I'm  the primary maintainer for power architecture, port in Void and a few  other things in Void. So what is Void? Of course I gave a talk previously at  the OpenPower Summit 2019 in Lyon, France. But there are probably many of you who  haven't seen the talk so I'll describe it in short. It's an independent Linux  distribution with custom package manager. It was originally created as a test  bet for this package manager. It has a focus on simplicity but it's not like  these minimalist distributions. It's more like tries to strike a balance  between the two things. We have a fair amount of packages and we have a similar  functionality to other big distributions. In order to stay simple, it's rolling  release so we don't have to deal with release management. We mostly ship stable  software though. It doesn't really ship lots of bleeding edge stuff. We make sure  that everything stays stable. It's binary based but there's a system called  XPSSRC which is similar to BSD ports like system which you can use to build  your software packages from source. So basically every package which is present  in the repo is buildable from source using XPSSRC. It's sort of one or three  kind of thing with all the package recipes. Of course besides power which is  supported at this point. It's on x86 where it started. It's on ARM and I-64 of  course and we also have build profiles for MIPS but we do not ship any  packages for MIPS. The intention is so there is a low barrier of entry, very  open and inclusive and informal community. So if you want to contribute, it's easy  for you to get started. So how is voice on power different from other distributions?  Well for one we have a particular focus on desktops and workstations, not servers.  Most distros are focused on servers. We feel like there's enough of that  stuff and voice on power was originally created to provide the desktop operating  system for me. So sort of selfish reason. We support both little and big  end-in. With little end-in having the standard requirements for power rate or  newer CPUs, big end-in runs on all the old Mac hardware and so on. What's  different from other big end-in distributions? We use the modern ABI on  G-Lip-C as well which is something other distros don't know. It's sort of an  obvious decision for us because it was a fresh port and we didn't want to deal  with 30 years old ABI so we just went with the modern one immediately. Of  course there's also the page size in the kernel. We use 4k pages by default.  Most distributions use 64k pages by default. 4k pages is what's also used by  x86 and most other architectures so it's more compatible with different  software and hardware. We also have wider desktop software support. This includes  things like the Chromium browser and electron applications and so on. Now let's  go over the quick history of Void because while I describe this in detail  at OpenPower Summit, many of you are probably not in sync with this. I started  the project in late 2018. This was because I bought Talos 2 Lite to use as  my primary workstation. I wanted a desktop operating system for this and I  used Ubuntu for a bit and I was not very happy with it. So I just  did initial port and it was done in like three days and I had a graphical  environment ready. Over the coming months I kept expanding the package  coverage and eventually during 2019 we got to complete coverage on Lysolundian.  There was also big-end-in ports from the start for 64-bit muscle Lysolundian and  for 32-bit G-Lypsy and muscle Lysolundian. There was not a 64-bit G-Lypsy  big-end-in port because of the ABI situation we weren't aware at this point  that it actually just works. So this port came later in April. During late 2019  there was complete coverage on all repos and I gave my OpenPower Summit  EU talk and in November I got a comment bit in upstream repos. So from that point  all changes go directly into Void without actually lingering in my custom  repo for too long. But this talk is not about this, it's about what happens since  that point. So this screenshot it's visuals to the code, an application which  lots of people have been requesting running on Void with PPC64 early and muscle.  As you can see in the corner it's all printed there and it works fine. But  first Java Bootstrap. That's one big thing we actually did and it was kind of a  pain. We have OpenJDK8 and 11 in repos and since you need OpenJDK to build  OpenJDK and you need a version which is one version older we had to  eventually reintroduce GCC6 which shipped GCJ which is GNU Java compiler now  deprecated but in GCC6 it was still present. We used this to Bootstrap OpenJDK7  which is a Bootstrap only package it's not meant to be used by people. JDK7 is  then used to Bootstrap8 and then we have Bootstrap packages for 9 and 10 so we  can eventually Bootstrap 11. This is a fairly fast process on PPC64 and early  because there's a JIT in there and it takes maybe 10 minutes or so to Bootstrap  one OpenJDK. On 32-bit PPC this is very slow because there's no JIT  support there's only the interpreter and it instead of 10 minutes it takes like  four hours so that's kind of pain but oh well. Go. Go is the language which is  fairly widely used for specific kinds of software and we have a bit of a problem  there because Google who maintain the tool chain have only support for PPC64  LE. Well technically speaking they do have BE also but their BE part uses  incorrect ABI or well do ABI we do not use and it also needs power rate which  would be a problem by itself because we make sure that all software in the  repo runs on all hardware we actually support. There's a GCC Go but GCC Go is  unreliable. For some reason it hangs while we do know the reason it there's  some bug in variable tracking code which results in GCC Go entering infinite  loop when you try to actually compile Go with it. So we can add this to  Bootstrap we use official binaries to Bootstrap these official binaries are  provided for every architecture they are not provided for muscle lip see but for  muscle lip see we deal with it by using G-compact. Since the Go compiler is  written in itself it doesn't actually use a lot of lip see at all so G-compact  actually can run it reliably and we can bootstrap with the official binaries  pretty to a base support of course is missing completely we don't have it.  Askel. Askel was also fun under some definitions of fun we use a self-built  binary snapshots to bootstrap the compiler this includes muscle up to this  point because recently I figured out how to cross compile a partial back  binary distribution of GCC this can be unused to compile the actual  distribution on a native environment. Bigendin is missing for now because of  some bugs in the compiler we will try to introduce it for 8.10 for now the  patches for future Bigendin introduction are actually ready. Other languages we  since recently had greater support for common lisp as PCL is finally in the  repos but for now only on little and in only to test the Bigendin much better.  It's bootstrapped with ECL which is a portable common lisp compiler which  works on every platform. We are also preparing an update to C-Lisp which  hasn't seen a release since 2010 but there's an active version control so we'll  make a snapshot from there. There's also things like D and Zigg and some others  which are currently still missing they don't have anything depending on them so  we haven't paid much attention to these yet. Libre.SSL performance.  Void uses Libre.SSL instead of Open.SSL at this point. Libre.SSL is the fork of  Open.SSL 1.0 made by OpenBSD project. It was with the intention to have  greater security. The problem is it only has assembly files on x86 and some  32-bit ARM. These results in performance which is much inferior to Open.SSL  and of course it also results in lack of hardware crypto support on PPC64  early and on R64. So I made a stopgap solution by importing those  assembly bits from Open.SSL upstream and this is now present on my GitHub and  it's shipping in Void. It has up to 20 times performance increase in things  where hardware crypto can be used. This is especially visible for example in  Open.SSL if you use SSHFS you can use this to increase your throughput from  18 max per second or so to like 500 max per second so that's a big difference.  Chromium upstream does not support PPC64 early but there's downstream patches  available since Chromium 84 we've been shipping these downstream patches. It  works quite well. It's currently the only browser engine which has JIT support  in the distro and in any distro and it works on muscle as well. It's a fairly  large patch and Google has been rejecting it to most of it for a while  now so there's a chance we'll have to ship it downstream forever but that's life  I guess. Related to this we now have support for Electron which is a framework  for applications so you can write applications in HTML5 and so on.  It uses Chromium, it's basically a version of Chromium. I don't use it but  people have been requesting different applications like Visual Studio Code and  Element. These do work. They work because we have a system-wide Electron.  There are some funny build system workarounds for these applications  because Electron build system from Node.js tries to download the binary  snapshot of Electron and it doesn't know about the architecture so it fails and  what we do is just enforce the x8664 architecture, make it present, it's x86,  it will download its binary snapshot which obviously doesn't work but in the  end it will be using the system-wide one anyway and it's pure JS anyway so it  doesn't make any difference. It's obviously only on LE just like Chromium  itself. I tried patching things for a big ending before and it doubled the patch  and it never worked quite right so I eventually gave up. It was too much of a  pain to do downstream muscle. That's work. We had muscle support previously so I  just had to add a small patch adding the PowerPC muscle specific things.  MD GPUs. There's this problem called MD GPU DC or display core. It's in the  kernel and it has a very haphazard usage of hardware floating point which  results in for one it's not portable. At some point Raptor computing systems  added some code to make it work with these new Navi cards but this was never  back ported to older kernels so we back ported ourselves down to 5.4.  Since 5.10 there's apparently a dependency on page size which is kind of  bad but it still works for us because we have 4k pages. Other distros have issues  with this apparently or at least that's what I've been told. They have 64k  pages and AMD doesn't work with this. There was also a big rework of cross  tool chains recently. There was this issue that we had many different cross  tool chains. The list of targets grew a lot recently. We've needed to deal with  this because the cross tool chain build recipes were duplicating lots of code so  I introduced a common what we call build style for cross tool chains. Most of the  code is in the build style and now each cross tool chain is only 50 or so lines  defining the actual tool chain specific things. As some extra goodies which I  introduced during cleaning it up. We now have a G-Lipseed cross tool chains on  muscle hosts which was not possible before so we have much expanded support  for this now. The configure arguments and so on are unified for all of the  staff now so that's better also and of course no more dirty master there's  previously the master there is a sort of a built container used by Xvpssrc.  Previously the intermediate built artifacts because you need to build  initial GCC without lipseed first. You need to build lipseed and you need to  build the final GCC. The intermediate built artifacts were messing up the  container. We now deal with this by just restricting everything to temporary  directories. In first status this is a screenshot from our build server. It's  currently still a Talos 2 Lite with 18 cores and 128 gigs of RAM and it runs in  my bedroom it's actually next to me right now. This is not ideal of course but I  don't have a better place to put it. Big and in built run in a VM thanks to KVM  you can run cross-end in VMs and there's no performance decrease. The  little ending builds run on bare metal and the primary actual package mirror  where people actually download stuff it's on a separate server in Chicago in  the USA. The mirror has 10g network and two terabytes of storage so we cannot  really compete with this. I do have a public IP here but I don't think anybody  would want to download stuff from me. There's several other mirrors provided by  the community. We could certainly use faster build hardware and we could  certainly use hardware which is not in my bedroom. The primary mirror after we  have some better built machine which is actually co-located in a real server  place we should migrate the primary mirror to that machine so it can be  available immediately after built so we don't have to upload batches to actual  server on the other side of the world. We also need to do more built automation  this is work in progress I'm working on some software to actually improve the  infrastructure. Now the main thing 32 bits a little end in. This is a  screenshot from from a root userant which is 32 bit little end in you can  see it's running. So first why why not? It officially does not exist but only  officially. The thing is so you can run 32 bit binaries on a 64 bit power because  64 bit is just an extension of the original power PC and you can certainly  run natively 32 bit binaries on 64 bit CPUs. This is on big end in though. On  little end in this is actually the same because because that makes sense. In  practice this therefore just works. The bring up process was fairly basic made  easy by XPSSRC it was similar to the other architectures we have. But there is  no native support for 32 bit little end in Linux kernel itself so we cannot  actually boot this on real hardware. It does work as a root on 64 bit kernel  for the compatibility stuff but not out of box so there are some fixes needed for  the kernel. So first we have to have to fix 32 bit compatibility in the kernel.  The first issue was that signal handlers were clearing the little end in bit in  the machine state register. This was fixed for a 3-bill patch. Also 32 bit VDSO  were disabled by default. This is just a build system change. Some VDSO's are  broken on little end in. We had to disable these coincidentally we found  that the power PC 601 had these same VDSO's broken. We are not a  worry of any relationship between this but we just had to extend to the  disablement. System calls with 64 bit arguments were also an issue because the  high 32 bits and the low 32 bits were passed in the wrong order. There's some  wrappers which do the passing so we had to fix the wrappers. All of these are  very trivial patches and things do run without them but they also crash at some  random points usually like at the obvious ones like when entering a signal  handler or when exiting. After patching there's no more known issues at least we  are not aware of any known issues. Fixes for this are being upstreamed into  the kernel except the VDSO fixes which will be fixed by just moving the VDSO  codes to see. This is a work in progress by other people in in current mainline so  at one point it will just get fixed. There will be or there are backports to  all void kernels which do run on little and then that is 4.19 and higher.  Now for bootstrapping for the 2-bit little and then well first we had to  create built-in cross profiles for Xbpssrc. These profiles define the  target triplet and some other specifics of the compiler. Then we had to create  cross tool chains for 32-bit elite targets. That was fairly easy we just had to  copy the beginning and once and adjust the triplet and so on. Some minor  patching was needed in G-Lip C and in Muscle. In G-Lip C just some warning  fixes. In Muscle we had to adjust the dynamic linker name so it matches GCC  properly. Adjustments around Xbpssrc were needed obviously just to look where  architecture specific code is and usually just trivial adjustments needed.  After this we could compile or cross compile a base change route. This is  basically a minimal set of packages needed to assemble the container in which  everything is built. Any errors had to be fixed along the way and there were  relatively few and very trivial ones. After that we had to a basic set of  packages which we could use to binary bootstrap a built container. Then we  could rebuild every package of the build container natively without cross  compilation. Then we could build other base math packages like base void  strap and so on which add more software to the route and after that more  software in general can be built like all sorts of different stuff in the  repo. Of course when errors come up fix these. Usually there's not a lot of them.  We did run into some issues with like a VSX code and say image decoder assuming  that little ending means power 8. For now we compile for power 8 so this is not  a problem for now but eventually these things will have to be patched. At this  point I could probably function on a 32-bit user run if I compiled enough  of the user runs. But there are some buts. There's no official support on  G-LipTee that means the dynamic linker name is shared with big end-in.  The simple versions are also shared with big end-in. This is not good for having  an official port because every every architecture has to have a unique dynamic  linker and fresh single versions also. So I'm going to propose an official port  later and once it's actually upstream I will rebuild the whole experimental  repo from scratch. The ABI is the same as big end-in and this is fine probably.  The ABI is well defined for both end-ins and it must pretty much just works as  it is. Even the dynamic linker name is stable and since muscle doesn't do a  simple versioning there's nothing to worry about. Of course we did because of  the we could but another big thing was the easier emulation of 32-bit x86  binaries. There's a project called Box86 I'll get to it later which we will be  previewing. Of course eventually it could be possible to actually port Linux to  G3s and G4s since the CPUs do support little end-in mode it would just need  to actually work. Or we could take the easier path which I was made aware of  and just use Be Linux kernel and then use the switch end-in call and the  swizzle this is called data for user run. Apparently this should work I'm not  sure how it would work yet but I was told it would work. For now it's strictly  Power 8 Plus though so we only run as compatibility on our Power 9  workstations and so on. There are some obstacles for one LLVM support which is  needed for MESA and some other things. Fortunately this was recently upstreamed  by the free BSD folks who needed it for the loader. We back ported the patches  to Voigt. It just came in handy because I was just about to start working on  porting LLVM when I found that there are patches available already so this  saved a lot of work for me. We also needed for Rust support which is currently  blocking a fair amount of user run mainly because of things like clip-rs  SVG which is depended on by GTK and lots of other things. There are some work in  progress patches available which I wrote for Rust but it doesn't work at cargo  crashes and so on. We need to figure all of this out. Proper G-Lift supports needs  to be made and we could also use the opportunity to improve the ABI perhaps  but we are not yet sure about this. One thing we definitely do want to do is  use 64-bit long double type. This will allow us to ditch the weird IBM 128-bit  floating point type which is basically composed of two standard 64-bit ones.  G-Lip-C is moving on from this to real 128-bit floating point  464-bit but we cannot use this on 32-bit because we do not depend on VSX  being present and these new floats require vector registers to pass them  around. We all need to choose a unique dynamic linker name and fresh symbol  versions for the new port. I'm kind of thinking we could use this to fix up  our 32-bit big-end imports as well. I mean we could at least ditch IBM long  double as well but I'm not yet sure on what the best technical approach to this  would be to ideally preserve compatibility with all your stuff and  ideally also make it upstream mobile. As for results there are testing  repos now available and it can fully compile itself. A portion of the repo  is packaged by portion I mean some 8% or so. This might not seem like much but  lots of core things are already available and to lots of most other  things could already built. We had just haven't built them yet. There's core  user ones, there's development tools like the compilers, LVM, there's Mesa, there's  SDL and of course there's initial port of Box86. What is Box86? Well Box86 is  Linux X86 and later developed for ARM devices. It has nothing to do with  PowerPC but we wanted to use this to emulate some old games which only had  binary versions on our power machines. The problem is this needs a 32-bit  little end-in host and you can probably see what I'm getting at. This is why we  needed a 32-bit little end-in port. So there's now an initial PowerPC port  done by our community. It can run GLX gears and I also got it to run Unreal  Tournament. The old one from 1999 it works but it's fairly slow. Sometimes  it's playable, sometimes it's not playable. There's no dynamic recompiler for  PowerPC yet in Box86 so it's purely interpreted. Of course this is still  faster than the likes of QMU because unlike QMU it actually runs lots of  code natively. How Box86 works is that it wraps native system libraries for  different functionality things like SDL and image libraries and audio libraries  and OpenGL and so on which means basically most things which are provided  by the system do not run emulated. They use the native versions and all  the emulated code is the actual game only. Now for the future. This is the  screenshot from our statistics page. You can see there's the coverage for at least  very high for be it slightly lower and for 32-bit little end-in it's fairly  small for now. One thing is infrastructure. I would like to get things  made official at one point. Mainly the PPC64LE port should get an official  airport one point. It should be easy because that port is stable and there's  nothing much to change in it. As for BE this has some issues. BE because of the  API situation which we need to clarify with upstream first. 32-bit has some  issues also. There's also the issue that some server software on official Void  repos is written and goes so it's not portable to these. This can be worked  around differently but yeah. The LV2 BE-ABI we will need official support in  G-Lipsey for this and right now things mostly work by accident. That means the  dynamic linker name and symbol versions are shared with the 64-bit LE port. An  upstream did say they wouldn't be against this but we would need to  formalize the API first. IBM thinks VSX requirement is a part of the API. We do  not entirely agree but we are still going to solve this so we are going to  formalize LV2 legacy compatible API. This legacy compatible API will be just an  extension of the current API. It will remove the requirements for VSX and  VMX and it will also switch to 64-bit long doubles to the HD 128-bit IBM  floating point. It will come during this year. It will be shared by  G-Lipsey. It will be shared by Muscle and FreeBSD. Muscle and FreeBSD already  implemented in the way I intended to specify so basically there's existing  compliant implementations before the spec specification exists which is kind of  funny but oh well. There will be other things. We will have to upstream as many  patches as possible. We will want to write a new installer for our live media  and also work on an enablement of new software and as well as working with  upstream of different software projects which do not have power supported to add  support for power. Also work has been done in our year and a half and there's still  plenty more to be done. If you want to talk with us or work with us you can join  our IRC channel on a free note and if you want to see some detailed statistics and  so on you can visit our website. For now that's the end of it and I'll  well I will show you some cool things but thanks for listening and there will be  a live Q&A session after this.  Okay so well added. I will show you some cool stuff I guess since we still have  some time. Here I have a route of 32-bit little end-in Void Linux port.  It's fairly minimal and doesn't have a lot of software in it but it does have  Box86 in it. Obviously as you can see it has a build of Cmake and it has a build  of dependencies for Box86 so we can try compiling it and since this is slow we  can try compiling it faster. And it's linking I believe. Well maybe not yet.  This will take a bit.  Well for now we have some other stuff. We have a build of GLX peers which we are  now going to run with this. It spawned on a different screen so let's move it just  over here. As you can see it's running at native refresh rate of the screen and  admin 60 FPS but it can do the 60 FPS. Obviously it's mostly open GL application  and very simple so even emulated it will run at decent frame rate. But I have a  cooler frame and that's Unreal Tournament. This will take a bit to start up  because there's a screen recording and so on running so yeah. And here it is.  You can see this is choppy. It's usually faster but yeah there's other stuff  loading the CPU right now. You can see at some points it's also fairly smooth. It  depends on how far into into the screen you can see.  Anyways here's the menu. We can start a practice session. I believe this is also  an unoptimized build of box86. So this will also run faster if we try to...  You see now it's like smooth if there's no distance visibility but  once you can actually see far it just blows up. But yeah it does work and I  think that's pretty good enough for now. So I think that's the end and there will  be a live Q&A session after this so feel free to stop by and ask some questions.  Okay we're going to go live in a few seconds. Hi Daniel.  So you're going to answer some of the questions or should I read them out?  So...  There were some questions about how the builds were done. I don't know if you  could elaborate on that level. I think you're still muted. Hold on now you  should be able to hear me. Yes now it's better. Okay good. So let's start with  further questions I guess. Yes. So what exactly were you asking regarding the  builds? So there were some questions about you. You mentioned that you have a  scratchpad SSD going through VIRT.io. Oh yeah like this. Yeah basically we were...  The first thing we did was just set up a big end-in VM and to use a QCao2  disk to like store the operating system and then in order to like share the  repo directory with the VM so it can actually write packages into repo we  just exposed it over VIRT.io.9p. Of course this has problems because the  throughput of the QCao2 storage is not as great as just a native SSD and  it was actually slowing down the builds. So we got around to this by just buying  a new SSD separately and just splitting into half and exposing a one-half or  one partition as a block device directly to the VM and that way it  can basically run as fast as native. It does run as fast as native so far it  works very well. Okay then there were some questions around  Embro-la which is broken. I don't know if you could comment on that. I don't see that here actually. Okay so Embro-la is broken on little  end-in and apparently it worked on the big end-in. Okay that's interesting we've  never run into this so I haven't really looked into it so I don't think I can  answer this but if you can point it to me in chat so I can take a  look afterwards. Yeah I haven't actually also had that error or that issue on  on our builds so that's why I was just asking maybe you had that experience but  we will see if T-platin can can actually comment in chat more on that. Okay I  see some questions in the chat so I'll reply to these directly. I see with  Box86 I could be able to run Unreal Engine for x86 on power. No because it's  mostly suited for old games. If you run a new stuff on this it will run very  slow. I tried to run the well-known Linux game Xanatic on Box86 just as an  experiment to see like how slow it would be and it's not a particularly  demanding game it works on my testing G5 but in the Box86 emulator it runs at  like well every five seconds it updates the frame. The menu runs fairly fast but  once you get in game it slows to crawl. I see a second question what package  manager is void most similar to? Well it's kind of its own thing really like it  uses several binaries xbps install xbps remove xbps query and so on and then it  uses like regular positional regular optional arguments to pass to that so like  to upgrade your system you run xbps-install then you pass capital S  and lowercase u once is to sync the repo indexes once is to upgrade if you  want to install a package you just run xbps install your package name so it's  not particular similar to anything else it's like a hybrid between most things.  As for Qt5 packages cross-compile to h64 well Qt5 packages can be cross-compiled  for h64 I don't think there's any issues with this because we do cross-compile  and for h64 and void I mean our ARM ports are fully cross-compiled and Qt5  is definitely present there what's problematic is Qt5 web engine which can  be cross-compiled but the problem is it needs to match the word size of the  host and target so if you want to cross-compile for 64 bit target you need  to cross-compile from a 64 bit host if you want to cross-compile for 32 bit you  also need a 32 bit host this is a bit of a problem for us because we cannot  cross-compile Qt5 web engine for 32 bit ARM because we cross-compile  from 64 bit host we could deal with this by using a 32 bit build container but  this is not it and what's the performance of a 32 bit kernel user  run to running on power 8 or power 9 versus 64 bit one well I haven't really  done any in-depth for benchmarking but it seems to be fairly close I haven't  noticed any actual difference in things like compilation times things mostly  compile about as fast on a 32 bit CZ run and on  you
voidlinuxlittleendian_0_167.png voidlinuxlittleendian_168_935.png voidlinuxlittleendian_936_983.png voidlinuxlittleendian_984_1331.png voidlinuxlittleendian_1332_2207.png voidlinuxlittleendian_2208_5027.png voidlinuxlittleendian_5028_7391.png voidlinuxlittleendian_7392_9815.png voidlinuxlittleendian_9816_10415.png voidlinuxlittleendian_10416_12239.png voidlinuxlittleendian_12240_14363.png voidlinuxlittleendian_14364_15371.png voidlinuxlittleendian_15372_17627.png voidlinuxlittleendian_17628_18299.png voidlinuxlittleendian_18300_19223.png voidlinuxlittleendian_19224_21407.png voidlinuxlittleendian_21408_22787.png voidlinuxlittleendian_22788_25091.png voidlinuxlittleendian_25092_25259.png voidlinuxlittleendian_25260_26387.png voidlinuxlittleendian_26388_27383.png voidlinuxlittleendian_27384_27743.png voidlinuxlittleendian_27744_28811.png voidlinuxlittleendian_28812_29399.png voidlinuxlittleendian_29400_31835.png voidlinuxlittleendian_31836_33215.png voidlinuxlittleendian_33216_34055.png voidlinuxlittleendian_34056_35183.png voidlinuxlittleendian_35184_36419.png voidlinuxlittleendian_36420_37859.png voidlinuxlittleendian_37860_39143.png voidlinuxlittleendian_39144_41039.png voidlinuxlittleendian_41040_41951.png voidlinuxlittleendian_41952_44903.png voidlinuxlittleendian_44904_46163.png voidlinuxlittleendian_46164_46211.png voidlinuxlittleendian_46212_46343.png voidlinuxlittleendian_46344_48107.png voidlinuxlittleendian_48108_49007.png voidlinuxlittleendian_49008_49199.png voidlinuxlittleendian_49200_49247.png voidlinuxlittleendian_49248_49343.png voidlinuxlittleendian_49344_52931.png voidlinuxlittleendian_52932_53003.png voidlinuxlittleendian_53004_53051.png voidlinuxlittleendian_53052_53111.png voidlinuxlittleendian_53112_53183.png voidlinuxlittleendian_53184_53231.png voidlinuxlittleendian_53232_53303.png voidlinuxlittleendian_53304_53375.png voidlinuxlittleendian_53376_53423.png voidlinuxlittleendian_53424_53495.png voidlinuxlittleendian_53496_53903.png voidlinuxlittleendian_53904_54155.png voidlinuxlittleendian_54156_54227.png voidlinuxlittleendian_54228_54275.png voidlinuxlittleendian_54276_54467.png voidlinuxlittleendian_54468_54563.png voidlinuxlittleendian_54564_54611.png voidlinuxlittleendian_54612_54659.png voidlinuxlittleendian_54660_54971.png voidlinuxlittleendian_54972_55175.png voidlinuxlittleendian_55176_55451.png voidlinuxlittleendian_55452_57263.png voidlinuxlittleendian_57264_66683.png voidlinuxlittleendian_66684_67631.png voidlinuxlittleendian_67632_67754.png