Articles / Security Issues of Auto-upg...

Security Issues of Auto-upgrades


apt-get install trojan

Users of certain other operating systems upgrade the software on their machines every few years (95, 98, 2000...). When you deal with software that moves at Open Source speeds and have a powerful package manager at your disposal, you can get in the habit of updating your system every morning while you sip the first cup of coffee. It's certainly convenient to be able to say "Grab anything new and install it for me", but do you know what procedures are place to ensure that you get what you were expecting and not an unwelcome surprise?

Today, I offer the results of an email discussion about package management security issues that I had with Jason Gunthorpe of Debian and Jeff Johnson of Red Hat.

Jeff Covey: The popularity of apt and rpm has led to a large number of users relying on automatic upgrades through their package management systems. Old timers who insist on compiling everything from source can be understandably concerned about the process of downloading a binary and installing it with minimal admin intervention. The convenience is bought at the price of trust in the system. How would you answer the following questions? Do you agree or disagree that the concerns they express are valid? If they are valid and are not currently addressed, do you have any ideas about how the problems could be fixed?
Jason Gunthorpe (Debian): It depends who is expressing them :> If the 'old timers' use source packages with signature files and check those signatures, then they are pretty safe. If they audit all the source code they hand compile, then they are pretty safe. But if they just download random .tars and compile them without even looking, they are no better off than we are, possibly worse off.
Jeff Covey: What facilities does your package manager (or a third party add-on such as autorpm) provide for automatic upgrading of installed packages?
Jason Gunthorpe (Debian): APT! :>
Jeff Johnson (Red Hat): rpm (and all package managers based on rpmlib) depend on ordering based on epoch:version-release comparison in order to identify the "newer" of two instances of a package with the same name.

[Jeff Covey: What I meant was: Does Red Hat provide the ability for a user to issue a command that says, "Go get any new versions of the software I have installed, and install them for me" as Debian does with APT, or can this only be done with third-party tools such as autorpm?

Jeff Johnson (Red Hat): RPM has some rudimentary support for FTP and HTTP transfers, but does not try to download anything other than what was requested. Neither does dpkg, which is a closer analogue to rpm than APT.

Closer to APT in functionality is the Red Hat up2date package, which does dependency resolution using HTTP POSTs and is able to augment an update request with other packages that will be needed to complete an upgrade.]

Jeff Covey: Who controls the package archives from which new packages are downloaded? If it's possible for third party archives to be used, does your package manager warn the user that packages are being downloaded from somewhere other than the official source?
Jason Gunthorpe (Debian): The user has choice here. Our default setup uses our top level 'official' mirrors. If third party archives are used, they would have to be manually configured by the user. No special warnings are given for these sources.
Jeff Johnson (Red Hat): Um, not applicable, as Red Hat packages are often the "official source".

[Jeff Covey: Red Hat only provides a limited subset of the software available in the RPM format. On , users will find all of these archives in addition to Red Hat versions and updates:

RedHat Contrib Net
Gnome Desktop Environment
Libc6 Contribs
Libc5 RedHat Contribs
Arch Independent RedHat Contribs
No Sources RedHat Contribs
Conectiva Linux
HelixCode Gnome distribs
Mandrake Cooker
XBF X-Windows servers
Yellow Dog PPC
Caldera Contribs
Archives for blind users
Stuff on Rufus.W3.Org
Solaris packages on Real-Time.Com
RPMs for Cygwin32/Windows

and many of these have multiple subdivisions.

Jeff Johnson (Red Hat): Um, almost all, if not all, binary software distributed by Red Hat (I do not speak about Cygnus, yet) is in rpm package format with signatures. Perhaps by "Red Hat" you mean "Red Hat Distribution"? Or do you mean that has not just Red Hat software?

Jeff Covey: It seems you're reading what you want to see instead of what I wrote. I said that Red Hat's RPMs make up only a subset of the software which is available in RPM packages, and that people will therefore be, of necessity, downloading RPMs from sources other than you, possibly overwriting your own packages, and I was asking whether issues of security related to this are taken into account in RPM's design.

Jeff Johnson (Red Hat): I was confused by your rpmfind output, as you choose to, for example, include Red Hat Power Tools as a separate entity even though those are packages produced by Red Hat. Furthermore, many (but not all) of the rpmfind categories (e.g. Mandrake, LinuxPPC come to mind) that you chose to distinguish are closely related to Red Hat packages -- in fact, often *are* Red Hat source packages except for signatures or lack thereof -- that have been rebuilt and/or redistributed for other architectures and other purposes. I can speak meaningfully only about packages produced by Red Hat, not derivative works.

Addressing issues of heterogeneous mixtures of "You pick 'em" package installation is a difficult problem that will require administrative superstructure and standards development in order to be solved meaningfully, and none of that process is complete (although it has been started).

Jeff Covey: The question I was asking was: Since you obviously can't verify the thousands of RPM files packaged by all of these distributions and individuals, does rpm provide a warning like "This package has not been prepared by Red Hat. While it's probably fine, we cannot confirm that it will work with your system. Continue installation? [Y/n]"?

Jeff Johnson (Red Hat): The goal of rpm (and tools that use rpmlib, including the Red Hat installer), by design, is to not prevent unattended installs and automatic updates by blocking on user interaction. Please note that in the example above there is little information ("Not packaged by Red Hat") that is helpful in or pertinent to answering the question "Continue installation? [Y/n]". Therefore, rpm (and the Red Hat installer) do not ask these questions during package install.

This doesn't mean that better policy (e.g. "Install only packages from Red Hat") tools aren't needed or useful, just that rpm (and the Red Hat installer) is not where the implementation should be.

Again, the Red Hat up2date agent currently implements certain install policies (but not the example above) like "Don't permit /bin/sh to be replaced" or "Don't upgrade the kernel package".

Jason Gunthorpe (Debian): One thing I hear often about .debs is that [since] we basically are the only provider (particularly of the base system), all .debs 'work' with your system.]

Jeff Covey: Does your package manager support digital signatures that can confirm that the package is from the packager it claims to be from and has not been tampered with?
Jason Gunthorpe (Debian): No. This is a very tricky topic given Debian's distributed nature.
Jeff Johnson (Red Hat): rpm supports header/payload signatures using md5 as well as all algorithms implemented in either pgp/pgp5/gpg (e.g. RSA, DSS, Diffie-Hellman, ...).
Jeff Covey: Are there procedures in place to check for trojans/virii/etc. in the original source package?
Jason Gunthorpe (Debian): Depending on which maintainer you talk to, yes or no. Some packages are inspected carefully, some are not.
Jeff Johnson (Red Hat): Checking for trojans/virii in sources is outside rpm's abilities and is solely the responsibility of the packager.
Jeff Covey: Are there procedures in place to check for trojans/virii/etc. in the package itself (for example, in the scripts used to install the package)?
Jason Gunthorpe (Debian): I don't think we have an official program for this. People do look occasionally, I'm sure.
Jeff Johnson (Red Hat): Signed rpm packages cannot be altered without being able to detect the alteration. The scripts are part of the header, which is signed, and so cannot be altered without being able to detect the change.
Jeff Covey: I'm not asking about them being altered after the fact; I'm just confirming that a procedure is in place to double-check the official signed packages to confirm that, for example, a disgruntled employee on his last day of work doesn't add "/bin/rm -rf /" to the preinstall script of the binutils package and place it in the errata FTP space.

(Debian folks: This is even more of a question for you, since you're accepting packages from people from all over, who may only have their reputations, not their jobs and the threat of prosecution, hanging over them and keeping them in line.)

Jeff Johnson (Red Hat): Part of Red Hat QA involves repeated installs of packages before and after signing that would easily detect the example you have given. Red Hat also does not release unsigned packages as errata, and there is sufficient process in place that no single employee, disgruntled or otherwise, is able to put an errata on the FTP site.

There are, of course, time bombs, viruses, and many other forms of damage more sophisticated than your example, and more generally, Red Hat, like many distributions, relies on the scrutiny of the community at large to detect and correct problems promptly. Enhancing the ability of the community to detect and correct problems before damage becomes widespread is the single best approach to insuring the safety (and quality) of packages that I can think of.

Jason Gunthorpe (Debian): We have no official auditing of packages, but before we make a stable release, the packages are put through a lot of testing and investigation; it would be hard for a simple attack to get through. Smart Devilish attacks I think could pass into stable undetected if one of our maintainers decided to make one.

People do monitor the upload list to make sure that the 'right people' are uploading the 'right packages', which tends to defuse the worst things (like libc6 trojans, etc.).

Actually, we go through a fairly intensive ID process before we accept a package from anyone. If someone does decide to do something nasty, we will know exactly who it was, and depending on local laws, they may face prosecution. Look at; it has some information about this process.

Jeff Covey: If someone were to sneak a trojan into a package, it could spread to thousands of machines overnight as admins performed automated upgrades on their systems. If this were to happen, would it be possible for you to prepare a package that would fix the problem on the next dist-upgrade (not everyone reads security bulletins, so not everyone will be aware that she's been compromised)?
Jason Gunthorpe (Debian): Yes, barring the point below.
Jeff Johnson (Red Hat): Um, yes, as Red Hat releases security errata as quickly as possible, and these updates are copied to mirror sites and up2date servers as part of the process of releasing an errata.
Jeff Covey: The answer to the previous question is naturally somewhat dependent on the nature of the trojan. As a worst case scenario: Is it possible for someone to insert a trojan into your upgrade stream which would disable your package upgrade system on the client side, making it impossible for you to distribute a fix through the normal method?
Jason Gunthorpe (Debian): Yeap. The packages install with root privilege; a well-written trojan can do anything.
Jeff Johnson (Red Hat): Signed rpm packages cannot be used as a vector for trojan horses (assuming the installer checks the signature).
Jeff Covey: Let's say Joe Packager uploads a new package of sendmail to a contrib directory. Jane User runs her automatic update script. It sees that she has sendmail installed, spots Joe's package, and offers to install it for her. Jane checks the signature. Yes, it's from Joe and has not been tampered with, so she installs it.

A couple of days later, someone notices that sendmail has been altered in this package to silently send copies of all mail to Joe and all his friends. You put out warnings about it and distribute a package with a version number higher than Joe's, so those people (like Jane) who don't bother to read security lists will at least get the fix when they run their update scripts.

Unfortunately, Joe's package also did something else: It replaced /bin/rpm with a version that will not install any version of sendmail or RPM other than Joe's. It will pretend to install your replacement, but won't actually do it, so Jane will never know that her system's been compromised.

[Jason Gunthorpe (Debian): Unless you sandbox the install scripts, this is impossible to prevent :<]

You might say this really isn't your problem, because if people want to be safe, they shouldn't install any packages that don't come from you, but it isn't reasonable to expect that, since there's a lot of software people want that Red Hat doesn't package (otherwise, Mandrake, etc. wouldn't exist). People *are* going to be getting their RPMs from other places.

[Jeff Johnson (Red Hat): Um, I question whether the example above illustrates anything but

  • "Joe is a dangerous moron"
  • "Jane is too trusting"
  • "Thank God someone noticed"
  • "People are going to do whatever they want"
so a specifically informed reply is difficult.]

Would you consider either of these valid solutions to the problem?:

  1. Informing users when they're about to install a package that didn't come from you.

    [Jeff Johnson (Red Hat): Presenting repeated yes/no questions to users usually leads to simple carriage return answers to accept the default. That isn't exactly "informing users" in anything other than a (possible) legal sense.

    Not valid.]

  2. Making certain core files untouchable by non-Red Hat packages, or at least providing much stronger warnings like "WARNING! This package, which is not an official Red Hat package, wishes to overwrite /bin/sh, which is a protected Red Hat file. This could have dangerous consequences. Proceed at your own risk, and run rpm again with --force if you really want to install this package."

    [Jeff Johnson (Red Hat): Attempting to make files unremovable makes the package manager (rather than the end user) apparently responsible for the end user's safety, and, like shouting "WOLF! ..." too often or mistakenly, lulls the end user into a false sense of security. Adding "Proceed at your own risk" would keep the lawyers happy I'm sure, but would do little to protect the end user. Instrumenting overrides like "--force" is necessary for many reasons, but the functionality is more often abused than used correctly.

    Not valid.]

Do you have any other ideas about what could be done?

[Jeff Johnson (Red Hat): Yes, but judging from the types of examples and questions you are asking, I don't believe that this is the correct forum to present other ideas.]

Jeff Covey: If the answer to the previous question [whether a trojan could disable the update stream] is "yes", do you think it would be beneficial to establish a class of protected packages which can only be upgraded with packages that come signed by you?
Jason Gunthorpe (Debian): This would not really help prevent the attack above; you can always use some trivial package to modify the files of an important one.
Jeff Johnson (Red Hat): Implementing better install policies based on explicitly verifying signatures is in everyone's interest.
Jason Gunthorpe (Debian): I think given what Jeff [Johnson of Red Hat] said, I'd just like to mention basically the key point in the list thread I mentioned. [See References.]

Red Hat has a single (hopefully) well-secured signing key that can only be used for packages that they produce in house. This is feasible for them because their development is concentrated in one physical location, and they don't have the constantly-changing archive like we do

Debian has a similar key kept by the Security Team, but it is infeasible to sign every official package that is created with this key, particularly since there are hundreds of new packages produced every single day. (Though we have been talking about signing full releases with this key.)

So, in the Debian situation, the next logical option is to use the maintainer's personal key for signatures. This brings up the really interesting question of 'who should sign a package'. With some 500 signing keys, the security of the whole scheme is in question. It is entirely possible to trojan an important package like libc using the most weakly secured key out of the 500.

This same problem can be applied to upstream source packages, too. If someone downloads a package, he can check the signature, but he also has to *check the key*.

The main point here is that just slapping a signature on packages does not necessarily make them as safe as the cryptosystem being used, or any safer than not having a signature.


Jeff Covey received his degree in classical guitar performance but spent so much time with his computer that he fell in with a bad crowd and ended up working for He currently works on freshmeat and runs a computer lab for the kids in his neighborhood in his spare time.

T-Shirts and Fame!


RSS Recent comments

13 May 2000 03:04 clausen

Debian keys
Would it be possible to have a master key stored on some
well secured machine that maintainers have access to?

So you could do something like:

ssh maintainers-account@si...
$ debsign PACKAGE

debsign being setuid, so it can read /etc/debian.private-key

Perhaps a database could be kept, saying which packages each
maintainer is allowed to sign.

13 May 2000 03:08 dirac

Great Discussion!
Hey, that was an excellent discussion about the dangers of trusting automatic binary updates/installs. Given the way that the Linux software distribution method is in effect at this time, I wonder about things like this - why don't we have trojan horses and virii to the extant that we possibly could? In truth, I believe it has to do with the way that the Linux system is designed and the way that software is distributed. In fact, I have noticed that in most cases, people who code Linux software are rather outspoken about their work, making web-pages and such to describe it - this makes them more than just faceless coders. The community, and the ethic of Linux itself, seem to be the main reasons why we do not have the kind of problems that Windows has.

However, if Linux did have these kind of problems, I do not doubt for a second that these "automagic" install methods would be targetted. It is wonderful that we have the trust that we have here - but maybe all the distros should start looking into some security, for peace of mind, of course. :)

13 May 2000 08:44 bnej

Why do you need to upgrade everything that often?
It seems to me that updating a system every day (or even every week) is a good way to ensure you have an unstable system you can never get work done on. People who are using their systems for real work probably have better things to do than starting automatic upgrades that never change that much anyway. I can think of better uses for bandwidth too.

Compared to doing a full system upgrade to get the latest stuff, isn't it better, faster and easier to simply get the software you need, compile it in a spare user account, use make -n install to ensure it doesn't do anything too rude, then install?

That said, things like dpkg --listfiles should help somewhat here. See what's going to happen before you install a package if you don't trust it. Some kind of option to show the actions which would be performed would also be good (like make -n). This does not help if you are doing huge automated upgrades, but it would help if you are installing third party packages on a system you need to know is secure.

13 May 2000 09:50 andylongton

Assume the packages are hostile...
It is an axiom that lack of physical security equals no security...but that's basically what we're up against these days.

With pervasive networking, we can't assume that there's any difference between having our hands on the only key to a locked room and leaving the door unlocked. Sure, we would protect the cabling and the ability of someone to either walk off with a box or some other equipment...but the data can be compromosed in either case. That it can be more easily compromised if the room isn't locked is no longer a given.

If you're using any kind of automatic package update system, you have to assume that the packages could do some uninteneded dammage. This could be because of a trojan/virus/mole/..., unintended configuration issues, or just bad/incompatible code.

The results could be very similar, though admitedly an intentionally hostile package would likely do the most dammage since it could compromise the security of the system.

After reading the LIDS mailing list -- -- and considering the possibilities for intrusion, I've come to a few conclusions on how to deal with hostile packages and users or even administrators who want -- out of ignorance or intent -- to change the system in hazzardous ways.

Keep in mind that I'm recommending these types of changes for all systems, not just the firewall or some sensitive equipment stored in the server rooms. Additionally, not all the tools needed to perform these tasks are robust enough at this time, but many are comming along.

Allow no modules to load after a specific point in init. Establish networking and login support only after this point.
Trust nobody, lock it all down; Strip most of the capabilities from all user accounts that don't explicitly require them, including root. This includes but is not limited to running specific programs, modifying specific directories/files, and using symlinks and other handy features normally available to daemons/wheel/root accounts.
Require switching to single user mode to perform changes in the above two. (Everyone, go ahead and shout about is an issue!)
Require all daemons to both be installed and run in a seperate environment similar to FreeBSD 4.0's jail() or -- potentially -- User Mode Linux when it becomes robust enough for the task. The idea is to allow an errant/hostile program to only do dammage in it's own area...and then monitor it to see if it does have problems. Killing it or reinitializing it would be fairly easy, causing minimal down time for most services.
Provide a mechanism -- such as a journaling file system or aggressive backups -- that can back out specific changes, undoing most/all of the dammage caused when it occurs.

Now, I realize that this is a colossal pain in the ass to administer, but steps like these would eliminate the need to vainly try and repair dammage to files, or to resort to backups from yesterday. The procedures/policies and scripts to make this practical or even easy still have to be created...and those aren't trivial either.

I doubt that any current OS will lack these types of features in 10 years.

13 May 2000 18:30 vdhoeven

security of Linux distributions
Just another question:

I do not upgrade my system every day and basically reinstall a new linux system each time a buy a new computer together with a few packages I download from the web.

1. What is actually the risk that a linux distribution on CD is infected?
2. If some packages are infected, what is the risk that some deamons, which are run in the background by default, are infected? If this risk is minimal, then bad surprises will not be likely if you only use a restricted number of programs in which you ``trust''.

13 May 2000 21:36 jtkirk

Apt and conflicting packages/files

This portion confuses me:


As anyone familiar with apt would know that when Jane attempted to install this bogus package with dpkg or apt they would complain that /bin/rpm is already owned by the package 'rpm' ( if installed ). This hopefully will seem suspicous enough to jane she won't force it to install.

On a second note, I can't wait for apt to contain a list of developers gnupg keys and automagically check while downloading packages.

13 May 2000 21:55 jtkirk

Re: Assume the packages are hostile... ( above )

You make some good, if not paraniod points


Note: dpkg --listfiles is only for already installed packages, you should use dpkg --contents PACKAGENAME.deb instead if the package is not already installed

The real problem will relying on this is that both RPMs and debs have install scripts which you cannot examine will out opening the package ( which is no big deal, but far less convenient ). If I really wanted to write a trojan I'd put the file corruption/replacement stuff in the script and then after the package is installed I'd replace the script with a blank file or innocous script, covering my tracks. This would also prevent RPM/dpkg/apt from complaining that I was relpacing a file owned by another package.

14 May 2000 01:17 bdale

As someone who has worked on Debian for a long time, I find Jeff's implication that Debian developers might act less responsibly because they "may only have their reputations, not their jobs" on the line... intriguing.

I don't think I'm alone in saying that I care an awful lot more about my reputation than I do about my job. In fact, my experience is that people who do things because they are passionate about them are far more likely to "get it right" than people working for a paycheck.

14 May 2000 01:27 bdale

Inspecting .deb files with ar
The .deb format had as an explicit design criteria that it be possible to unpack and inspect a .deb without needing to trust any binary tools not already on your system. Back then, very few systems ran either Debian or Redhat, so even the package mangement tool started out as something you might want to inspect before installation.

The next time you're starved for amusement (or worried that there might be a Trojan in your .deb), read the man page for 'ar' and play around unpacking a .deb with it to inspect the contents...

14 May 2000 01:58 jtkirk

Re: Inspecting .deb files with ar


The command is ar -x PACKAGENAME

You could also try dpkg -x PACKAGENAME OUTPUTDIR which is likely to work even when the deb format changes, as I assume at sometime in the future it will. Not using bzip2 for these packages is silly, it would cut down package size and bandwidth requirements at the cost of a few extra cycles on local machines. Anyone know what compression/archiver redhat uses? I seem to recall using cpio and gzip (?) maybe.

14 May 2000 21:43 mattdm

A couple of responses to other comments.
First, why upgrade so often? Simple: to keep up to date with security issues. Not that there's necessarily one of these every week, let alone every night, but when one does come out, it's nice to have the fix applied as soon as possible.

Second, how could /bin/rpm be replaced by a different package? Simple. It's not done via the normal install-a-file mechanism of package manager -- instead, it's copied into place by a script.

14 May 2000 22:33 waldoj

I just want to point out that there's an auto-updater that wasn't mentioned in the discussion: yup, put out by Yellow Dog Linux. While it doesn't help with any of the problems discussed here, I feel I should at least point out that it exists. :)

15 May 2000 19:53 jmitchel

Yet another automated distribution system
AT&amp;T/Bell Labs/Lucent has had an internal automatic distribution system for some time. A variation on the current version was detailed in a September 1998 Dr. Dobbs article titled &quot;NSBD and Automatic Software Distribution&quot;, or

The solution to the problem of signed packages in a distributed development environment is for the central administrators to sign the public keys of the developers, and distributing them to the users, along with a signed list of packages that the developer has the right to distribute. This does not mean the package may not contain malware, but it does certify that the person who built the package had a right to do so.

More significantly here is to not do anything as root. This may render it useless for updating core parts of the environment, but a bit of creativity could, I suspect, at least nearly surmount this. The only program which must run in the context of this user is the automatic update program, which fetches modified files.

No untrusted program is ever run here in a context (other than by a careless root user) where it can possibly modify any system-wider resources. Further all files can be audited against a signed database of checksums, which highlights any discrepancies. This does remove the posibility of the usual pre and post install scripts, but in most circumstances this can be surmounted.

NSBD dosen't solve the problem that RedHat and Debian are trying to solve, updating 100% of the system without user intervention. It does solve 90% though, updating all non-suid/non-sgid programs without intervention, and requiring user intervention for the others. By lowering it sights slightly, it is able to acomplish this without the breathtaking security implications that rpm and dpkg have.

Further information is available at


Project Spotlight

Opencart Product Video

An Opencart extension that adds images and videos.


Project Spotlight

