> > If there is no pg_control or it's corrupted or points to
> > unexistent/corrupted checkpoint record then scan logs from
> > newest to oldest one till we find last valid checkpoint record
> > or oldest valid log record and than redo from there.
> 
> And how well will that approach work if the last checkpoint record
> got written near the start of a log segment file, and then the
> checkpointer discarded all your prior log segments because "you don't
> need those anymore"?  If the checkpoint record gets corrupted,
> you have no readable log at all.

The question - why should we have it? It is assumed that data files
are flushed before checkpoint appears in log. If this assumtion
is wrong due to *bogus* fsync/disk/whatever why should we increase
disk space requirements which will affect *good* systems too?
What will we buy with extra logs? Just some data we can't
guarantee consistency anyway?
It seems that you want guarantee more than me, Tom -:)

BTW, in some my tests size of on-line logs was ~ 200Mb with default
checkpoint interval. So, it's worth to care about on-line logs size.

> > As I explained in short already: with UNDO we'll be able to reuse
> > XIDs after restart - ie there will be no point to have NEXTXID
> > records at all. And there is no point to add it now.
> > Does it fix anything? Isn't "fixing" all what we must do in beta?
> 
> If you really insist, I'll take it out again, but I'm unconvinced
> that that's necessary.

Please convince me that NEXTXID is necessary.
Why add anything that is not useful?

Vadim

Wed, 7 Mar 2001 19:27:13 -0800 
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
RE: Proposed WAL changes 
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes:
>> And how well will that approach work if the last checkpoint record
>> got written near the start of a log segment file, and then the
>> checkpointer discarded all your prior log segments because "you don't
>> need those anymore"?  If the checkpoint record gets corrupted,
>> you have no readable log at all.

> The question - why should we have it? It is assumed that data files
> are flushed before checkpoint appears in log. If this assumtion
> is wrong due to *bogus* fsync/disk/whatever why should we increase
> disk space requirements which will affect *good* systems too?
> What will we buy with extra logs? Just some data we can't
> guarantee consistency anyway?
> It seems that you want guarantee more than me, Tom -:)

No, but I want a system that's not brittle.  You seem to be content to
design a system that is reliable as long as the WAL log is OK but loses
the entire database unrecoverably as soon as one bit goes bad in the
log.  I'd like a slightly softer failure mode.  WAL logs *will* go bad
(even without system crashes; what of unrecoverable disk read errors?)
and we ought to be able to deal with that with some degree of grace.
Yes, we lost our guarantee of consistency.  That doesn't mean we should
not do the best we can with what we've got left.

> BTW, in some my tests size of on-line logs was ~ 200Mb with default
> checkpoint interval. So, it's worth to care about on-line logs size.

Okay, but to me that suggests we need a smarter log management strategy,
not a management strategy that throws away data we might wish we still
had (for manual analysis if nothing else).  Perhaps the checkpoint
creation rule should be "every M seconds *or* every N megabytes of log,
whichever comes first".  It'd be fairly easy to signal the postmaster to
start up a new checkpoint process when XLogWrite rolls over to a new log
segment, if the last checkpoint was further back than some number of
segments.  Comments?

> Please convince me that NEXTXID is necessary.
> Why add anything that is not useful?

I'm not convinced that it's not necessary.  In particular, consider the
case where we are trying to recover from a crash using an on-line
checkpoint as our last readable WAL entry.  In the pre-NEXTXID code,
this checkpoint would contain the current XID counter and an
advanced-beyond-current OID counter.  I think both of those numbers
should be advanced beyond current, so that there's some safety margin
against reusing XIDs/OIDs that were allocated by now-lost XLOG entries.
The OID code is doing this right, but the XID code wasn't.

Again, it's a question of brittleness.  Yes, as long as everything
operates as designed and the WAL log never drops a bit, we don't need
it.  But I want a safety margin for when things aren't perfect.

			regards, tom lane

Thu, 08 Mar 2001 12:03:33 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Proposed WAL changes 