Hi all,

While testing vacuum,I noticed that dirty buffers aren't
necessarily written to disk in case of abort.
In addtion they have no guarantee to be flushed before
the shutdown of postmaster. The recent bufmgr changes
seems to have increased the possibility much more than
before.
Certainly it's not bad unless there are indexes. However
if heap data wasn't flushed while corresponding indices
are written to disk,the indices would point to non-existence
heap block. It would be the cause of inconsistency after the
restart of postmaster. Shouldn't there be a mechanism to
flush dirty buffers at(or before) the shutdown of postmaster ?

Comments ?

Regards. 

Hiroshi Inoue
Inoue@tpf.co.jp

Tue, 9 May 2000 23:34:17 +0900
"Hiroshi Inoue" <Inoue@tpf.co.jp>
Shouldn't flush dirty buffers at shutdown ?
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> if heap data wasn't flushed while corresponding indices
> are written to disk,the indices would point to non-existence
> heap block. It would be the cause of inconsistency after the
> restart of postmaster. Shouldn't there be a mechanism to
> flush dirty buffers at(or before) the shutdown of postmaster ?

Hmm, good point, but that doesn't seem like the right answer.
Suppose the system crashes before we are able to flush the
dirty buffers?  I think you have identified a problem that needs
a more general solution: we need to be robust in the case that
an index entry is on disk that points to a tuple that never made
it to disk.  We can legitimately assume that the tuple is uncommitted
and ignore the index entry --- we just have to not fail ;-)

Not sure how to do it offhand; maybe some additional checking in the
index fetch code will do?

			regards, tom lane

Tue, 09 May 2000 10:49:31 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Shouldn't flush dirty buffers at shutdown ? 
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sent: Tuesday, May 09, 2000 11:50 PM
> 
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > if heap data wasn't flushed while corresponding indices
> > are written to disk,the indices would point to non-existence
> > heap block. It would be the cause of inconsistency after the
> > restart of postmaster. Shouldn't there be a mechanism to
> > flush dirty buffers at(or before) the shutdown of postmaster ?
> 
> Hmm, good point, but that doesn't seem like the right answer.
> Suppose the system crashes before we are able to flush the
> dirty buffers?

You are right but we could hardly expect the completeness
of indexes in case of system crash because indexes are out
of transactional control currently. What surprized me was that
such problems could easily occur even in case of graceful
shutdown of postmaster.
 
> I think you have identified a problem that needs
> a more general solution: we need to be robust in the case that
> an index entry is on disk that points to a tuple that never made
> it to disk.

Probably an index entry that points to a non-existent heap
block doesn't cause a problem immediately because heap_
fetch() ignores UNUSED heap blocks.
However it will not be long before the heap block is filled with
another tuple. heap_insert/update() could hardly check that
the inserting heap block is already pointed from some index
entry.

There could be another case which I came across while
testing vacuum abort. i.e,index_delete() wasn't flushed while
the corresponding heap block is cleaned(set to UNUSED) 
and flushed. 

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp 

Wed, 10 May 2000 02:14:43 +0900
"Hiroshi Inoue" <Inoue@tpf.co.jp>
RE: Shouldn't flush dirty buffers at shutdown ? 
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
>> Hmm, good point, but that doesn't seem like the right answer.
>> Suppose the system crashes before we are able to flush the
>> dirty buffers?

> You are right but we could hardly expect the completeness
> of indexes in case of system crash because indexes are out
> of transactional control currently.

What?  We do flush the index blocks before committing, just the
same as heap blocks, so I do not see the risk there.  If we have
committed a tuple then its index entries should be on disk too.
The risk I think you have shown is that there may be extra index
entries that correspond to no tuple because we wrote the index
blocks first and then crashed before writing the tuple.

> Probably an index entry that points to a non-existent heap
> block doesn't cause a problem immediately because heap_
> fetch() ignores UNUSED heap blocks.
> However it will not be long before the heap block is filled with
> another tuple. heap_insert/update() could hardly check that
> the inserting heap block is already pointed from some index
> entry.

Yes.  We need some way of cross-checking that an index entry actually
does match the tuple it thinks it's pointing at.  The first thought that
comes to mind is that the index tuple should contain the OID of the
tuple it is for, and then we can check that against the pointed-to
tuple.  If they don't match, the index entry is bogus and can be
discarded at the next VACUUM.

			regards, tom lane

Tue, 09 May 2000 13:20:07 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Shouldn't flush dirty buffers at shutdown ? 