> I have just sent to the pgsql-patches list a rather large set of

Please send it to me directly - pgsql-patches' archieve is dated by Feb -:(

> proposed diffs for the WAL code.  These changes:
> 
> * Store two past checkpoint locations, not just one, in pg_control.
>   On startup, we fall back to the older checkpoint if the newer one
>   is unreadable.  Also, a physical copy of the newest checkpoint record

And what to do if older one is unreadable too?
(Isn't it like using 2 x CRC32 instead of CRC64 ? -:))
And what to do if pg_control was lost? (We already discussed that we
should read all logs from newest to oldest ones to find checkpoint).
And why to keep old log files with older checkpoint?

>   is kept in pg_control for possible use in disaster recovery (ie,
>   complete loss of pg_xlog).  Also add a version number for pg_control

Mmmm, how recovery is possible if log was lost? All what could be done
with DB in the event of corrupted/lost log is dumping data from tables
*asis*, without any guarantee about consistency. How checkpoint' content
could be useful?

I feel that the fact that

WAL can't help in the event of disk errors

is often overlooked.

>   itself.  Remove archdir from pg_control; it ought to be a GUC
>   parameter, not a special case (not that it's implemented yet anyway).

I would discuss WAL based BAR management before deciding how to
store/assign archdir. On the other hand it's easy to add archdir
to pg_control later -:)

> * Change CRC scheme to a true 64-bit CRC, not a pair of 32-bit CRCs
>   on alternate bytes.

Great if you've found reliable CRC64 impl!

> * Fix XLOG record length handling so that it will work at BLCKSZ = 32k.

Case I've overlooked -:(
(Though, I always considered BLCKSZ > 8K as temp hack -:))

> * Change XID allocation to work more like OID allocation, so that we
>   can flush XID alloc info to the log before there is any chance an XID
>   will appear in heap files.

I didn't read you postings about this yet.

> * Add documentation and clean up some coding infelicities; move file
>   format declarations out to include files where planned contrib
>   utilities can get at them.

Thanks for that!
 
> Before committing this stuff, I intend to prepare a contrib utility that
> can be used to reset pg_control and pg_xlog.  This is mainly for
> disaster recovery purposes, but as a side benefit it will allow people

Once again, I would call this "disaster *dump* purposes" -:)
After such operation DB shouldn't be used for anything but dump!

> to update 7.1beta installations to this new code without doing initdb.
> I need to update contrib/pg_controldata, too.

Vadim



Wed, 7 Mar 2001 01:07:53 -0800
"Vadim Mikheev" <vmikheev@sectorbase.com>
Re: Proposed WAL changes
"Vadim Mikheev" <vmikheev@sectorbase.com> writes:
>> I have just sent to the pgsql-patches list a rather large set of
> Please send it to me directly - pgsql-patches' archieve is dated by Feb -:(

Done under separate cover.

>> proposed diffs for the WAL code.  These changes:
>> 
>> * Store two past checkpoint locations, not just one, in pg_control.
>> On startup, we fall back to the older checkpoint if the newer one
>> is unreadable.  Also, a physical copy of the newest checkpoint record

> And what to do if older one is unreadable too?
> (Isn't it like using 2 x CRC32 instead of CRC64 ? -:))

Then you lose --- but two checkpoints gives you twice the chance of
recovery (probably more, actually, since it's much more likely that
the previous checkpoint will have reached disk safely).

> And what to do if pg_control was lost? (We already discussed that we
> should read all logs from newest to oldest ones to find checkpoint).

If you have valid WAL files and broken pg_control, then reading the WAL
files is a way to recover.  If you have valid pg_control and broken WAL
files, you have a big problem, but using pg_control to generate a new
empty WAL will at least let you get at your heap files.

> And why to keep old log files with older checkpoint?

Not much point in remembering the older checkpoint location if the
associated WAL file is removed...

> Mmmm, how recovery is possible if log was lost? All what could be done
> with DB in the event of corrupted/lost log is dumping data from tables
> *asis*, without any guarantee about consistency.

Exactly.  That is still better than not being able to dump the data at
all.

>> * Change XID allocation to work more like OID allocation, so that we
>> can flush XID alloc info to the log before there is any chance an XID
>> will appear in heap files.

> I didn't read you postings about this yet.

See later discussion --- Andreas convinced me that flushing NEXTXID
records to disk isn't really needed after all.  (I didn't take the flush
out of my patch yet, but will do so.)  I still want to leave the NEXTXID
records in there, though, because I think that XID and OID assignment
ought to work as nearly alike as possible.

>> Before committing this stuff, I intend to prepare a contrib utility that
>> can be used to reset pg_control and pg_xlog.  This is mainly for
>> disaster recovery purposes, but as a side benefit it will allow people

> Once again, I would call this "disaster *dump* purposes" -:)
> After such operation DB shouldn't be used for anything but dump!

Fair enough.  But we need it.

			regards, tom lane

Wed, 07 Mar 2001 11:09:25 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Proposed WAL changes 
On Wed, 7 Mar 2001, Vadim Mikheev wrote:

> > I have just sent to the pgsql-patches list a rather large set of
>
> Please send it to me directly - pgsql-patches' archieve is dated by Feb -:(

Huh?

http://www.postgresql.org/mhonarc/pgsql-patches/2001-03/index.html



Wed, 7 Mar 2001 12:57:02 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Proposed WAL changes
"Vadim Mikheev" <vmikheev@sectorbase.com> writes:

> I feel that the fact that
> 
> WAL can't help in the event of disk errors
> 
> is often overlooked.

This is true in general.  But, nevertheless, WAL can be written to
protect against predictable disk errors, when possible.  Failing to
write a couple of disk blocks when the system crashes is a reasonably
predictable disk error.  WAL should ideally be written to work
correctly in that situation.

Ian

---------------------------(end of broadcast)---------------------------
TIP 102: An atom-blaster is a good weapon, but it can point both ways.
		-- Isaac Asimov

07 Mar 2001 09:37:49 -0800
Ian Lance Taylor <ian@airs.com>
Re: Proposed WAL changes
On Wed, Mar 07, 2001 at 11:09:25AM -0500, Tom Lane wrote:
> "Vadim Mikheev" <vmikheev@sectorbase.com> writes:
> >> * Store two past checkpoint locations, not just one, in pg_control.
> >> On startup, we fall back to the older checkpoint if the newer one
> >> is unreadable.  Also, a physical copy of the newest checkpoint record
> 
> > And what to do if older one is unreadable too?
> > (Isn't it like using 2 x CRC32 instead of CRC64 ? -:))
> 
> Then you lose --- but two checkpoints gives you twice the chance of
> recovery (probably more, actually, since it's much more likely that
> the previous checkpoint will have reached disk safely).

Actually far more: if the checkpoints are minutes apart, even the 
worst disk drive will certainly have flushed any blocks written for 
the earlier checkpoint.

--
Nathan Myers
ncm@zembu.com

Wed, 7 Mar 2001 13:10:52 -0800
ncm@zembu.com (Nathan Myers)
Re: Proposed WAL changes