
[ CC to Kurt and Steven on bsdi list.]

Guys, I just replied to this email on the BSDi email list.  The issue is
that someone found that some(most?) IDE drives have write cache enabled,
though the drives do not preserve the write cache data on power failure.

I am surprised we have not heard of this failure before because I know
most vendors who ship PostgreSQL test our crash recovery thoroughly. 
Are they testing only using SCSI drives?

Below you will read that my Seagate SCSI drive has write cache disabled,
but another guy has a Seagate IDE drive that has it enabled, though it
loses data on power failure.

Scarey!

Does anyone have any more detailed information on this?

---------------------------------------------------------------------------

Bruce Momjian wrote:
> 
> I can tell you that if this was posted on a PostgreSQL mailing list, we
> would be freaking out!
> 
> Having the data permanently on the disk is important for soft updates,
> but it is critical for databases.  We go to great lengths to make sure
> the PostgreSQL write-ahead log is fsync'ed to the disk.  When we report
> a transaction as committed, we expect to have that transaction reliably
> recorded even if you pull the plug on the computer right after the
> transaction completes.  With the write cache enabled and not preserved
> on power failure, re-powering a system could leave a transaction
> partially completed, e.g. if you move $500 from one account to another,
> the money might show as removed from the original account but not appear
> in the new account.  This, of course, is a disaster for a database that
> must be 100% reliable.
> 
> I have heard of drives that either have battery backup for their cache,
> or have enough battery to guarantee the data gets written to the
> platters, or uses the rotational energy of the spinning platters to
> write the data to the platters.  I assumed that drive manufacturers
> where honest in enabling write cache by default only on drives that
> support a guaranteed safe cache.
> 
> On my SCSI SEAGATE ST336607LW, I see:
> 	
> 	$ scsicmd -c msel -p all -f /dev/rsd0h
> 	...
> 	disk caching page [pcode=0x08] (dcp):
> 	
> 	parameters saveable [sense] (ps):                                     0
> 	mode page code (mpcode):                                              0
> 	mode page length (mpl):                                               0
> 	write cache enable (wce):                                             0
> 
> so it seems SCSI drives are being honest.  It is a shame to learn IDE
> drives are not, and particularly when you can't control it using
> something like 'scsicmd'.
> 
> I know that most of the vendors who ship PostgreSQL repeatedly test our
> crash recovery capabilities, and I have not heard of anyone reporting it
> didn't work, so I assume they are testing on SCSI or honest IDE drives.
> 
> Seeing that Kurt is testing with a Seagate IDE drive, and Seagate is
> honest in their SCSI setting of disabled by default of the write-ahead
> cache, he might be right that IDE by default enables the cache because
> they assume it is for a non-critical desktop machine.  However, I know
> of many who use IDE drives for mission-critical servers, particularly in
> RAID configurations, so this is a serious concern.
> 
> Because this was posted to the public BSDI list, I will CC it over to
> the PostgreSQL hackers list for comment, and keep Kurt in the CC.
> 
> ---------------------------------------------------------------------------
> 
> Kurt J. Lidl wrote:
> > On Tue, Jul 29, 2003 at 01:31:09PM -0700, Steven M. Schultz wrote:
> > > > From: Bruce Momjian <pgman@candle.pha.pa.us>
> > > > 
> > > > The big question is whether that write cache is preserved if power is
> > > > removed from the drive.  If it isn't, I don't think enabling it is a
> > > > good idea.
> > > 
> > > 	Usually if a drive loses power that means the system has also
> > > 	lost power.   The 4-8MB of data in the drive's cache would be lost
> > > 	along with the 100 to 200MB in the system's buffer cache.
> > > 
> > > 	I've not had a problem with enabling the write cache in the drive but
> > > 	if it's a concern then using CTQ instead is equally effective and
> > > 	all one worries about is losing the system buffer cache on a power
> > > 	fail.
> > 
> > Well, I cannot imagine that the difference between write-caches is
> > significant between IDE and SCSI drives in this context.  On modern
> > IDE drives, the write cache is almost always on.  With softupdates
> > enabled, any power loss (or even a hard reset of the machine) will
> > curdle the filesystem repeatably.  Under 5.0 this is especially true,
> > in my experience.  I cannot imagine running with the write cache
> > enabled on machines where I care about getting the data back off the
> > drive reliably...  Let me rephrase that -- where I care about the
> > machine rebooting with requireing a manual fsck of the filesystem.
> > 
> > This problem is so severe that Chris Ross and I worked on a patch
> > that allows you to turn off the write cache on a drive by drive
> > basis.  Now my 5.0 machine(s) are much, much, much more likely to
> > survive a reset or sudden poweroff and come back up without manual
> > intervention.
> > 
> > I'll echo Henry's statement -- the lost of the buffer cache is somewhat
> > problematic, but the drive's lying about what's on disk and what is really
> > just in the write cache is catastrophic when softupdates are turn on.
> > 
> > I've attached my bug report and fix for the 5.0 code.  The same change
> > probably would "just work" under 4.3 too.  Too bad this didn't make it
> > into the stock 5.0 source tree.  Oh well.
> > 
> > -Kurt
> > 
> > ----- snip, snip -----
> > Date: Tue, 10 Dec 2002 18:54:25 -0500
> > From: "Kurt J. Lidl" <lidl@pix.net>
> > To: tsa-beta-problems@windriver.com
> > Cc: cross@distal.com, "Kurt J. Lidl" <lidl@pix.net>
> > Subject: turning off write cache on IDE drives
> > Message-ID: <20021210185425.A9454@pix.net>
> > 
> > I've noticed a really horrible interaction with the softupdate
> > code in the 5.0 beta and 5.0 final when the disk drive has
> > a write-cache enabled.
> > 
> > I'd call this a RFE, but since I'm also sending the enhancement at
> > the bottom of the message, it's not really a "request for enhancement",
> > so much as a "here's the code for an enhancement".  So, CFAE
> > follows...
> > 
> > This isn't really a problem with softupdates, but rather a problem in
> > that BSD/OS doesn't have a way of turning off the write-cache on
> > IDE disk drives.
> > 
> > My test machine has a Seagate ST340180A drive in it, which is a
> > pretty run of the mill, 40GB disk drive.  Naturally, it comes
> > with both the read cache and write cache enabled, and there isn't
> > a physical jumper on the drive to turn either cache off.
> > 
> > To demonstrate a catastrophic filesystem failure, boot a machine
> > with with softupdates on /usr, that also has a drive write-cache.
> > Run "find /usr" -- this will dirty up a bunch of inodes with modified
> > access times for the files.  Power off the machine to simulate a
> > sudden loss of power in the field.
> > 
> > Power the machine on.  Notice the almost 100% occurance of the error
> > "UNEXPECTED SOFT UPDATE INCONSISTANCY; RUN fsck MANUALLY".  And notice
> > that part of your /usr is now gone.  Repeat as often as you like,
> > eventually you'll run out of files in /usr to have unlinked :-)
> > 
> > The solution to this rather horrific interaction between softupdates
> > and the write cache in the drive is to turn off the write cache.
> > This seems to solve the problem.
> > 
> > Here's the patch that Chris Ross and I worked up to turn off the
> > write cache on a drive.  With this patch, the user can set a line
> > like:
> > 
> > 	-parm wd0 disable_wrcache=yes
> > 
> > In their /etc/boot.default and have a much more robust system in
> > the face of sudden power loss (when using a modern IDE drive). 
> > 
> > It might be more clever to check for the write cache before attempting
> > to turn it off, but the following patch is tested and works.
> > 
> > -Kurt
> > 
> > --- wd.c	2002/12/10 22:18:47	1.1
> > +++ wd.c	2002/12/10 23:16:38	1.2
> > @@ -133,6 +133,7 @@
> >  #define PARM4_WD_ISMAPPED	1
> >  #define PARM4_WD_USEDMA		2
> >  #define PARM4_WD_USELBA		3
> > +#define PARM4_WD_DISWRCACHE	4
> >  
> >  /*
> >   * Drive states.  Used for open and format operations.
> > @@ -509,6 +510,21 @@
> >  		(*prfunc)("%s%d*%d",
> >  		    sep, (wp->wdp_fixedcyl + wp->wdp_removcyl) *
> >  		    wp->wdp_heads * wp->wdp_sectors, WD_SECSIZE); 
> > +
> > +	if ((mp = getparamfor(sc->wd_dk.dk_dev.dv_xname,
> > +	    PARM4_WD_DISWRCACHE)) && (*mp != 0)) {
> > +		(*prfunc)(": write cache disabled");
> > +		/* Set the "disable write cache" feature */
> > +		outb(sc->wd_iobase + wd_feature, 0x82);
> > +		/* Run the command, wait for it, and eat the interrupt */
> > +		outb(sc->wd_iobase + wd_command, WDCC_SETFEATURES);
> > +		if (wd_wait_nbusy(sc->wd_iobase)) {
> > +			printf("%s: Timeout disabling drive write cache\n",
> > +		    	sc->wd_dk.dk_dev.dv_xname);
> > +		}
> > +		DELAY(1000);	/* Sometimes BUSY clears before interrupt */
> > +		inb(sc->wd_iobase + wd_status);
> > +	}
> >  out:
> >  	printf("\n");
> >  }
> > --- wd.4	2002/12/10 22:18:47	1.1
> > +++ wd.4	2002/12/10 23:32:34	1.2
> > @@ -85,6 +85,12 @@
> >  .Do no Dc ,
> >  Logical-Block Addressing is disabled, and Cylinder/Head/Sector addressing
> >  is used instead.
> > +.It disable_wrcache
> > +If this parameter is specified as
> > +.Do yes Dc ,
> > +the drive will be sent a SET_FEATURE command which will attempt to turn
> > +off the write cache on the drive.  With the write cache enabled, a
> > +sudden power loss can cause catastrophic filesystem failure.
> >  .El
> >  .Sh FILES
> >  .Bl -tag -width /dev/rwd[0-7][a-h] -compact
> > --- /usr/src/etc/etc.i386/boot.define	Tue Jun 26 17:13:01 2001
> > +++ /etc/boot.define	Mon Dec  9 19:24:44 2002
> > @@ -294,6 +294,9 @@
> >  wd	use_lba		3
> >  	include		generic yesno
> >  ;
> > +wd	disable_wrcache	4
> > +	include		generic yesno
> > +;
> >  wdpi	use_dma		2
> >  	include		generic yesno
> >  ;
> > 
> > ----- snip, snip -----
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: bsdi-users-unsubscribe@mailinglists.org
> > For additional commands, e-mail: bsdi-users-help@mailinglists.org
> > 
> > 
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 30 Jul 2003 00:38:54 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Upgrading my BSDI box, again
On Wed, Jul 30, 2003 at 12:38:54AM -0400, Bruce Momjian wrote:
> 
> [ CC to Kurt and Steven on bsdi list.]
> 
> Guys, I just replied to this email on the BSDi email list.  The issue is
> that someone found that some(most?) IDE drives have write cache enabled,
> though the drives do not preserve the write cache data on power failure.
> 
> I am surprised we have not heard of this failure before because I know
> most vendors who ship PostgreSQL test our crash recovery thoroughly. 
> Are they testing only using SCSI drives?
> 
> Below you will read that my Seagate SCSI drive has write cache disabled,
> but another guy has a Seagate IDE drive that has it enabled, though it
> loses data on power failure.
> 
> Scarey!
> 
> Does anyone have any more detailed information on this?

Not really, just that I had data corruption on SCSI disks which had write
cache enabled and tagged queueing. Disabling write cache has allowed them
to function flawlessly since. The moral seems to be: write cache gives
pretty performance figures, but you really want it swiched off.

Cheers,

Patrick

Wed, 30 Jul 2003 14:50:11 +0100
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: Upgrading my BSDI box, again
On Wed, Jul 30, 2003 at 12:38:54AM -0400, Bruce Momjian wrote:
> Guys, I just replied to this email on the BSDi email list.  The issue is
> that someone found that some(most?) IDE drives have write cache enabled,
> though the drives do not preserve the write cache data on power failure.

It's yet worse than this.  Some IDE drives will respond to the
command to turn off write cacheing, but they _don't actually turn it
off_.  In other words, you think you've done the right thing, and you
haven't anyway.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Wed, 30 Jul 2003 12:42:46 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: Upgrading my BSDI box, again
Andrew Sullivan wrote:
> On Wed, Jul 30, 2003 at 12:38:54AM -0400, Bruce Momjian wrote:
> > Guys, I just replied to this email on the BSDi email list.  The issue is
> > that someone found that some(most?) IDE drives have write cache enabled,
> > though the drives do not preserve the write cache data on power failure.
> 
> It's yet worse than this.  Some IDE drives will respond to the
> command to turn off write cacheing, but they _don't actually turn it
> off_.  In other words, you think you've done the right thing, and you
> haven't anyway.

Is there an interface to control this with IDE drives? I know there is
scsicmd for SCSI, but I didn't think IDE had any similar interface to
control the drive.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 30 Jul 2003 15:39:53 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Upgrading my BSDI box, again
Something similar to Linux's hdparm ?

http://www.google.com/search?q=hdparm%20bsd

-*- Bruce Momjian <pgman@candle.pha.pa.us> [ 2003-07-30 19:41 ]:
> Andrew Sullivan wrote:
> > On Wed, Jul 30, 2003 at 12:38:54AM -0400, Bruce Momjian wrote:
> > > Guys, I just replied to this email on the BSDi email list.  The issue is
> > > that someone found that some(most?) IDE drives have write cache enabled,
> > > though the drives do not preserve the write cache data on power failure.
> > 
> > It's yet worse than this.  Some IDE drives will respond to the
> > command to turn off write cacheing, but they _don't actually turn it
> > off_.  In other words, you think you've done the right thing, and you
> > haven't anyway.
> 
> Is there an interface to control this with IDE drives? I know there is
> scsicmd for SCSI, but I didn't think IDE had any similar interface to
> control the drive.
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
>                http://www.postgresql.org/docs/faqs/FAQ.html

-- 
Kveï¿½ja,
Tolli
tolli@tol.li

Wed, 30 Jul 2003 19:46:22 +0000
=?iso-8859-1?Q?=DE=F3rhallur_H=E1lfd=E1narson?= <tolli@tol.li>
Re: Upgrading my BSDI box, again
On Wednesday 30 July 2003 15:39, Bruce Momjian wrote:
> Andrew Sullivan wrote:
> > It's yet worse than this.  Some IDE drives will respond to the
> > command to turn off write cacheing, but they _don't actually turn it
> > off_.  In other words, you think you've done the right thing, and you
> > haven't anyway.

> Is there an interface to control this with IDE drives? I know there is
> scsicmd for SCSI, but I didn't think IDE had any similar interface to
> control the drive.

What Andrew is saying is that on some IDE drives it doesn't matter what the OS 
tells the drive to do.  According to the Linux hdparm man page, the desired 
flag is '-W':

       -W     Disable/enable the IDE drive’s  write-caching  feature  (default
              state is undeterminable; manufacturer/model specific).

-- 
Lamar Owen
Director of Information Technology
Pisgah Astronomical Research Institute

Wed, 30 Jul 2003 16:18:39 -0400
Lamar Owen <lowen@pari.edu>
Re: Upgrading my BSDI box, again
On Wed, Jul 30, 2003 at 04:18:39PM -0400, Lamar Owen wrote:
> 
> What Andrew is saying is that on some IDE drives it doesn't matter
> what the OS tells the drive to do.  According to the Linux hdparm

Right.  In other words, you can't really trust IDE drives, until the
manufacturers start guaranteeing that their drives will respond to
the commands you send.  I suspect that some manufacturers are better
than others, but I haven't been able to find any reliable data about
this.  The failure to obey commands to turn off write caching is
something I have both seen reported by others and experienced myself,
so I am sure that it happens.  But who does it (or doesn't do it),
I've no idea.  The case I saw was, I believe, a Maxtor drive, and it
happened about 2 years ago, IIRC.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Thu, 31 Jul 2003 10:51:52 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: Upgrading my BSDI box, again