> > Should we try to read log up to the *physical end* - ie end of last
> > log file - regardless invalid CRC-s/zero pages with attempt to
> > re-apply interim valid records? (Or do we already do this?)
> 
> That doesn't seem like a good idea --- once we fail to read an XLOG
> record, it's probably best to stop there rather than continue on.
> I think we want to try for a consistent recovery to a past point in
> time (ie, wherever the xlog gap is) not a partial recovery to a later
> time.

No way for consistent recovery if there is gap in log due to
disk write re-ordering anyway (and we can't know what was
the reason of the gap). I thought that you wanted apply as much of log
as we can. If you don't then I missed your point in first message:

> xlog.c's ReadRecord formerly contained code that would zero 
> out the rest of the log segment (and delete the next log segment,
> if any) upon detecting a missing or corrupted xlog record.
> I removed that code because I considered it horribly dangerous
> where it was. If there is anything wrong with either the xlog or
> pg_control's pointers to it, that code was quite capable of wiping
> out all hope of recovery *and* all evidence of what went wrong.
  ^^^^^^^^^^^^^^^^^^^^^^^^

So, if we are not going to re-apply as much valid records as
we can read from log then zeroing is no more dangerous than
SUI in headers. But I totaly agreed that SUI is much better.

> > Anyway I like idea of StartUpID in page headers - this will help
> > if some log files disappeared. Should we add CRC to page header?
> 
> That seems like overkill. I was hoping to keep the page 
> header overhead at eight bytes. We could do that either by storing just
> the two LSBs of StartUpID (and doing the sequence checking mod 64k) or
> by reducing the magic number to two bytes so there's room for four bytes
of
> StartUpID.  I think I like the first alternative better --- comments?

I don't think a few additional bytes in header is a problem.
BTW, why not use CRC32 in header instead of magic?
Or just StartUpID instead of magic if you don't want to calculate
CRC for header - xlp_magic doesn't seem to be more useful than SUI.

> > Hm, maybe XLogFileInit should initialize files with StartUpID & CRC
> > in pages? We would avoid reading empty files.
> 
> We already stop when we hit a zeroed page (because it's not got the
> right magic number).  That seems sufficient.

What if the next page after zeroed one is correct (due to write
re-ordering)?
(But I take back SUI+CRC in XLogFileInit - useless -:))

Vadim

Mon, 12 Mar 2001 21:38:07 -0800
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
RE: xlog loose ends, continued 
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes:
>> That doesn't seem like a good idea --- once we fail to read an XLOG
>> record, it's probably best to stop there rather than continue on.
>> I think we want to try for a consistent recovery to a past point in
>> time (ie, wherever the xlog gap is) not a partial recovery to a later
>> time.

> No way for consistent recovery if there is gap in log due to
> disk write re-ordering anyway (and we can't know what was
> the reason of the gap). I thought that you wanted apply as much of log
> as we can. If you don't then I missed your point in first message:

>> xlog.c's ReadRecord formerly contained code that would zero 
>> out the rest of the log segment (and delete the next log segment,
>> if any) upon detecting a missing or corrupted xlog record.
>> I removed that code because I considered it horribly dangerous
>> where it was. If there is anything wrong with either the xlog or
>> pg_control's pointers to it, that code was quite capable of wiping
>> out all hope of recovery *and* all evidence of what went wrong.
>   ^^^^^^^^^^^^^^^^^^^^^^^^

What I was thinking about in that last paragraph was manual analysis and
recovery.  I don't think it's a good idea for automatic system startup
to skip over gaps in the log.

> So, if we are not going to re-apply as much valid records as
> we can read from log then zeroing is no more dangerous than
> SUI in headers. But I totaly agreed that SUI is much better.

Okay, I will change the page headers to include SUI (or the low-order
bits of it anyway), and make ReadRecord stop if it notices a backwards
jump in SUI.

> I don't think a few additional bytes in header is a problem.
> BTW, why not use CRC32 in header instead of magic?

There is so little changeable information in a page header that a CRC
wouldn't be much more than an eight-byte magic number.  And we don't
need eight bytes worth of magic number (even four is more than enough,
really).  So I'd just as soon keep the headers simple and small.

			regards, tom lane

Tue, 13 Mar 2001 12:05:42 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: xlog loose ends, continued 