A real life Newcomb's Paraodox


The paradox:



What do you do?











Fixing upgrades -- a database recording ease-of-upgrade

I’ve been writing recently about the linux upgrade nightmares that continue to trouble the world. The next in my series of ideas is a suggestion that we try to measure how well upgrades go, and make a database of results available.

Millions of people are upgrading packages every day. And it usually goes smoothly. However, when it doesn’t, it would be nice if that were recorded and shared. Over time, one could develop an idea of which upgrades are safer than others. Thus, when it’s time to upgrade many packages, the system could know which ones always go well, and which ones might deserve a warning, or should only be done if you don’t have something critical coming up that day.

We already know some of these. Major packages like Apache are often a chore, though they’ve done a lot more by using a philosophy of configuration files I heartily approve of — dividing up configuration to put config by different people in different files.

Some detection is automated. For example, the package tools detect if a configuration file is being upgraded after it’s been changed and offer the user a chance to keep the new one, their old one, or hand-mix them. What choice the user makes could be noted to measure how well the upgrades go. Frankly, any upgrade that even presents the user with questions should get some minor points against it, but if a user has to do a hand merge it should get lots of negative points.

Upgrades that got no complaint should be recorded, and upgrades that get an explicit positive comment (ie. the user actively says it went great) should also be noted. Of course, any time a user does an explicit negative comment that’s the most useful info of all. Users should be able to browse a nice GUI of all their recent upgrades — even months later — and make notes on how well things are going. If you discover something broken, it should be easy to make the report.

Then, when it comes time to do a big upgrade, such as a distribution upgrade, certain of the upgrades can be branded as very, very safe, and others as more risky. In fact, users could elect to just do only the safe ones. Or they could even elect to automatically do safe upgrades, particularly if there are lots of safety reports on their exact conditions (former and current version, dependencies in place.) Automatic upgrading is normally a risky thing, it can generate the risk of a problem accidentally spreading like wildfire, but once you have lots of reports about how safe it is, you can make it more and more automatic.

Thus the process might start with upgrading the 80% of packages that are safe, and then the 15% that are mostly safe. Then allocate some time and get ready for the ones that probably will involve some risk or work. Of course, if everything depends on a risky change (such as a new libc) you can’t get that order, but you can still improve things.

There is a risk of people gaming the database, though in non-commercial environments that is hopefully small. It may be necessary to have reporters use IDs that get reputations. For privacy reasons, however, you want to anonymize data after verifying it.

Towards a Zero User Interface backup system

I’ve spoken before about ZUI (Zero User Interface) and how often it’s the right interface.

One important system that often has too complex a UI is backup. Because of that, backups often don’t get done. In particular offsite backups, which are the only way to deal with fire and similar catastrophe.

Here’s a rough design for a ZUI offsite backup. The only UI at a basic level is just installing and enabling it — and choosing a good password (that’s not quite zero UI but it’s pretty limited.)

Once enabled, the backup system will query a central server to start looking for backup buddies. It will be particularly interested in buddies on your same LAN (though it will not consider them offsite.) It will also look for buddies on the same ISP or otherwise close by, network-topology wise. For potential buddies, it will introduce the two of you and let you do bandwidth tests to measure your bandwidth.

At night, the tool would wait for your machine and network to go quiet, and likewise the buddy’s machines. It would then do incremental backups over the network. These would be encrypted with secure keys. Those secure keys would in turn be stored on your own machine (in the clear) and on a central server (encrypted by your password.)

The backup would be clever. It would identify files on your system which are common around the network — ie. files of the OS and installed software packages — and know it doesn’t have to back them up directly, it just has to record their presence and the fact that they exist in many places. It only has to transfer your own created files.

Your backups are sent to two or more different buddies each, compressed. Regular checks are done to see if the buddy is still around. If a buddy leaves the net, it quickly will find other buddies to store data on. Alas, some files, like video, images and music are already compressed, so this means twice as much storage is needed for backup as the files took — though only for your own generated files. So you do have to have a very big disk 3 times bigger than you need, because you must store data for the buddies just as they are storing for you. But disk is getting very cheap.

(Another alternative is RAID-5 style. In RAID-5 style, you distribute each file to 3 or more buddies, except in the RAID-5 parity system, so that any one buddy can vanish and you can still recover the file. This means you may be able to get away with much less excess disk space. There are also redundant storage algorithms that let you tolerate the loss of 2 or even 3 of a larger pool of storers, at a much more modest cost than using double the space.)

All this is, as noted, automatic. You don’t have to do anything to make it happen, and if it’s good at spotting quiet times on the system and network, you don’t even notice it’s happening, except a lot more of your disk is used up storing data for others.

It is the automated nature that is so important. There have been other proposals along these lines, such as MNET and some commercial network backup apps, but never an app you just install, do quick setup and then forget about until you need to restore a file. Only such an app will truly get used and work for the user.

Restore of individual files (if your system is still alive) is easy. You have the keys on file, and can pull your file from the buddies and decrypt it with the keys.

Loss of a local disk is more work, but if you have multiple computers in the household, the keys could be stored on other computers on the same LAN (alas this does require UI to approve this) and then you can go to another computer to get the keys to rebuild the lost disk. Indeed, using local computers as buddies is a good idea due to speed, but they don’t provide offsite backup. It would make sense for the system, at the cost of more disk space, to do both same-LAN backup and offsite. Same-LAN for hardware failures, offsite for building-burns-down failures.

In the event of a building-burns-down failure, you would have to go to the central server, and decrypt your keys with that password. Then you can get your keys and find your buddies and restore your files. Restore would not be ZUI, because we need no motiviation to do restore. It is doing regular backups we lack motivation for.

Of course, many people have huge files on disk. This is particularly true if you do things like record video with MythTV or make giant photographs, as I do. This may be too large for backup over the internet.

In this case, the right thing to do is to backup the smaller files first, and have some UI. This UI would warn the user about this, and suggest options. One option is to not back up things like recorded video. Another is to rely only on local backup if it’s available. Finally, the system should offer a manual backup of the large files, where you connect a removable disk (USB disk for example) and transfer the largest files to it. It is up to you to take that offsite on a regular basis if you can.

However, while this has a UI and physical tasks to do, if you don’t do it it’s not the end of the world. Indeed, your large files may get backed up, slowly, if there’s enough bandwidth.

Something isn't CLEAR about airport line-jumping program







Generic internet appliances

Normally I’m a general-purpose computing guy. I like that the computer that runs my TV with MythTV is a general purpose computer that does far more than a Tivo ever would. My main computer is normally on and ready for me to do a thousand things.

But there is value in specialty internet appliances, especially ones that can be very low power and small. But it doesn’t make sense to have a ton of those either.

I propose a generic internet appliance box. It would be based on the same small single-board computers which run linux that you find in the typical home router and many other small network appliances. It would ideally be so useful that it would be sold in vast quantities, either in its generic form or with minor repurposings.

Here’s what would be in level 1 of the box:

  • A small, single-board linux computer with low power processor such as the ARM
  • Similar RAM and flash to today’s small boxes, enough to run a modest linux.
  • WiFi radio, usually to be a client — but presumably adaptable to make access points (in which case you need ethernet ports, so perhaps not.)
  • USB port
  • Infrared port for remote control or IR keyboard (optionally a USB add-on)

Optional features would include:

  • Audio output with low-fi speaker
  • Small LCD panel
  • DVI output for flat panel display
  • 3 or 4 buttons arranged next to the LCD panel

The USB port on the basic unit provides a handy way to configure the box. On a full PC, write a thumb-drive with the needed configuration (in particular WiFi encryption keys) and then move the thumb drive to the unit. Thumb drives can also provide a complete filesystem, software or can contain photo slide shows in the version with the video output. Thumb drives could in fact contain entire applications, so you insert one and it copies the app to the box’s flash to give it a personality.

Here are some useful applications:

  • In many towns, you can see when a bus or train will arrive at your stop over the internet. Program the appliance with your stop and how long it takes to walk there after a warning. Press a button when you want to leave, and the box announces over the speaker a countdown of when to go to meet the transit perfectly.
  • Email notifier
  • MP3 output to stereo or digital speakers
  • File server (USB connect to external drives — may require full ethernet.)
  • VOIP phone system speakerphone/ringer/announcer
  • Printer server for USB printers
  • Household controller interface (X10, thermostat control, etc.)

Slap on the back of cheap flat panel display mounted on the wall, connected with video cable. Now offer a vast array of applications such as:

  • Slide show
  • Security video (low-res unless there is an mpeg decoder in the box.)
  • Weather/News/Traffic updates
  • With an infrared keyboard, be a complete terminal to other computer apps and a minimal web browser.

There are many more applications people can dream up. The idea is that one cheap box can do all these things, and since it could be made in serious quantities, it could end up cheaper than the slightly more specialized boxes, which themselves retail for well under $50 today. Indeed today’s USB printer servers turn out to be pretty close to this box.

The goal is to get these out and let people dream up the applications.






Let the world search for the lost









Avoid thermal printers for long-term uses





The linux package upgrade nightmare, part 1

















Flying Cars -- Airport Carshare system





Read on for more thinking on it…  read more »

Darfur movie, with white actors


Which is to make a movie to tell the story, a true movie that is, hopefully a moving as a Schinder’s List or the Pianist. Put the story in front of the first world audience.

And, I suggest with a sad dose of cynicism, do it with whitebread american actors. Not that African actors can’t do a great job and make a moving film like Hotel Rwanda. I just have a feeling that first world audiences would be more affected if they saw it happening to people like them, rather than people who live in a tiny poor muslim villages in a remote desert. The skin colour is only part of what seems to have distanced this story to the point that little is being done. We may have to never again believe that people will keep the vow of never again.

So change the setting a bit and the people, but keep the story and the atrocities, and perhaps it can have the same effect that seeing a Schindler’s list can have on white euro descended Jews and non-Jews. And the Hollywood folks would be doing exactly what they are best at.

Panorama in ad, and more on automatic reset.

I’m pleased to see that more of my photography is getting licenced for ads and web sites these days. I like the job that this PDA ad does with my 360 degree view of Shanghai People’s Square. Of course I can’t read the text very well.

By the way, I learned the hard way how valuable the feature I proposed earlier for digital cameras — where they would notice if they’ve been set in an unusual state after a long gap between sessions — while on my trip this month to Edmonton, and one of my favourite spots on the planet — the rocky mountains in Banff and Jasper. Just before the trip I had put the camera into the “small” image size mode because I was shooting some stuff for eBay, and you really don’t need 8 megapixel shots for that. Alas, I left it there, and this is one of those mode switches which is not at all obvious. You won’t notice it unless you pay careful attention to the tiny “s” on the LCD panel, or if you download the photos. Alas, on my 4gb card I can go a long way without downloading, so a full days shots, including a lovely snow dusted Lake Louise were shot in small size, high compression.

The other way you would spot this is the camera shows you how many shots you have left. My 4gb card shows 999 when it starts even in large mode. But after shooting for a short while it eventually starts counting down. I only noticed I was in small mode when the 999 didn’t start counting down with hundreds of shots.

So this is definitely a case where the camera should notice it’s been days since I shot, and warn me I’m shooting with this unusual setting. I will still get quite serviceable web photos from that day, but not the wall sized prints I love.

Voter turnout in contested races is the real statistic


However the statistic I would like to see is “Voter turnout in districts where there is an important, hotly contested race.” This is the number we might want to monitor from year to year.

Virginia, it turns out, which had the Webb-Allen “Macaca” race, had the highest voter turnout in its history. You wouldn’t think that after hearing about the low turnout of a typical mid-term. Of course it will also go down as the first time a major U.S. politician was taken down due to blogs, the web and YouTube. Since it was so close, almost any factor can be given credit for Allen’s loss.

It is not surprising that when there is no contested race, that turnout is low. The U.S. for various bizarre reasons, has most incumbents always safe in their seats. This switch of 30 or so seats in the house and 6 in the senate is considered a major upheaval, nigh a revolution, by Americans. With seats so safe, there is no suprise there is little incentive in voting. U.S. ballots are very complex compared to many countries, and there are often long voting lines, and you don’t get official time off to vote.

Contrast that to Canada, where a public upset with the Conservative party’s introduction of the visible Goods and Services Tax (a 7% VAT) took the party from having a majority of parliament to having TWO seats. 2, as in 1 plus 1. There’s no such safety zone for incumbents, no cry for term limits in much of the rest of the world. There, if the public gets upset it throws the bums out, or drops them back to a minority position due to the fact that there are more than 2 parties.

I hope one of the major statistical agencies starts tracking voter turnout modulated by how motivated the voters are in particular districts. Of course voter turnout is the final metric of how motivated they were, but there are other, earlier indicators in most cases.

Cute pun idea -- real crib sheets


The strange story of eBay Playstation 3 auctions















Better forms of differential pricing that don't punish flexibility so much















Can the big web sites save the political system















Sponsored conference bags with logos on the inside





Call my car











Beware the Weather Warn








Syndicate content