> > It seems that you want guarantee more than me, Tom -:)
> 
> No, but I want a system that's not brittle. You seem to be content to
> design a system that is reliable as long as the WAL log is OK 
> but loses the entire database unrecoverably as soon as one bit goes bad
> in the log.

I don't see how absence of old checkpoint forces losing entire database.
You probably will get better consistency by re-applying modifications
which supposed to be in data files already but it seems questionable
to me.

> > BTW, in some my tests size of on-line logs was ~ 200Mb with default
> > checkpoint interval. So, it's worth to care about on-line logs size.
> 
> Okay, but to me that suggests we need a smarter log management strategy,
> not a management strategy that throws away data we might wish we still
> had (for manual analysis if nothing else).

This is what should be covered by archiving log files. Unimplemented -:(

> Perhaps the checkpoint creation rule should be "every M seconds *or*
> every N megabytes of log, whichever comes first".

I like this! Regardless usability of keeping older checkpoint (especially
in future, with log archiving) your rule is worth in any case.
(Nevertheless, keeping two checkpoints still increases disk requirements -:)
But seems I have to waive my objection if I didn't convince you - it's
really simplest way to get WAL restart-ability and I personally have no
ability to implement log scanning now).

> > Please convince me that NEXTXID is necessary.
> > Why add anything that is not useful?
> 
> I'm not convinced that it's not necessary. In particular, 
> consider the case where we are trying to recover from a crash using
> an on-line checkpoint as our last readable WAL entry. In the pre-NEXTXID
> code, this checkpoint would contain the current XID counter and an
> advanced-beyond-current OID counter. I think both of those numbers should
> be advanced beyond current, so that there's some safety margin against
> reusing XIDs/OIDs that were allocated by now-lost XLOG entries.
> The OID code is doing this right, but the XID code wasn't.
> 
> Again, it's a question of brittleness.  Yes, as long as everything
> operates as designed and the WAL log never drops a bit, we don't need
> it.  But I want a safety margin for when things aren't perfect.

Once again - my point is that in the event of lost log record one shouldn't
try to use existent database but just dump it, etc - ie no reason to keep
info about allocated XIDs. But keeping NEXTXID costs nothing, at least -:)

Vadim

Thu, 8 Mar 2001 11:00:47 -0800 
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
RE: Proposed WAL changes 
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes:
>> No, but I want a system that's not brittle. You seem to be content to
>> design a system that is reliable as long as the WAL log is OK 
>> but loses the entire database unrecoverably as soon as one bit goes bad
>> in the log.

> I don't see how absence of old checkpoint forces losing entire database.

As the code stood last week that's what would happen, because the system
would not restart unless pg_control pointed to a valid checkpoint
record.  I addressed that in a way that seemed good to me.

Now, from what you've said in this conversation you would rather have
the system scan XLOG to decide where to replay from if it cannot read
the last checkpoint record.  That would be okay with me, but even with
that approach I do not think it's safe to truncate the log to nothing
as soon as we've written a checkpoint record.  I want to see a reasonable
amount of log data there at all times.  I don't insist that "reasonable
amount" necessarily means "back to the prior checkpoint" --- but that's
a simple and easy-to-implement interpretation.

> You probably will get better consistency by re-applying modifications
> which supposed to be in data files already but it seems questionable
> to me.

It's not a guarantee, no, but it gives you a better probability of
recovering recent changes when things are hosed.

BTW, can we really trust checkpoint to mean that all data file changes
are down on disk?  I see that the actual implementation of checkpoint is

	write out all dirty shmem buffers;
	sync();
	if (IsUnderPostmaster)
		sleep(2);
	sync();
	write checkpoint record to XLOG;
	fsync XLOG;

Now HP's man page for sync() says

     The writing, although scheduled, is not necessarily complete upon
     return from sync.

I can assure you that 2 seconds is nowhere near enough to ensure that a
sync is complete on my workstation... and I doubt that "scheduled" means
"guaranteed to complete before any subsequently-requested I/O is done".
I think it's entirely possible that the checkpoint record will hit the
disk before the last heap buffer does.

Therefore, even without considering disk drive write reordering, I do
not believe that a checkpoint guarantees very much, and so I think it's
pretty foolish to delete the preceding XLOG data immediately afterwards.


>> Perhaps the checkpoint creation rule should be "every M seconds *or*
>> every N megabytes of log, whichever comes first".

> I like this! Regardless usability of keeping older checkpoint (especially
> in future, with log archiving) your rule is worth in any case.

Okay, I'll see if I can do something with this idea.

Other than what we've discussed, do you have any comments/objections to
my proposed patch?  I've been holding off committing it so that you have
time to review it...

			regards, tom lane

Thu, 08 Mar 2001 14:30:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Proposed WAL changes 