In the BufferDesc struct, there seem to be two ways to mark a buffer
page as dirty: setting the BM_DIRTY bit mask in the 'flags' field of the
struct, and setting the 'cntxDirty' field to true. What is the
difference between these two indications of a page's dirtiness?

Or, more to the point, is there a reason we have two ways to do what
looks like the same thing?

BTW, I'd like to remove the behavior that LockBuffer(buf, EXCLUSIVE)
automatically marks the page as dirty. Since there are some situations
in which we acquire an exclusive buffer lock but don't actually end up
modifying the page, this results in dirtying more buffers than are
necessary. I think it's also good practise for code that modifies a
buffer to explicitly mark it as dirty, rather than depending upon
LockBuffer() to do it. Does this sound reasonable, provided I find and
catch all the places that depend upon this behavior? (i.e. and change
them to explicitly mark the buffer as dirty)

-Neil



Tue, 04 Nov 2003 00:36:56 -0500
Neil Conway <neilc@samurai.com>
bufmgr code question
Neil Conway wrote:
> In the BufferDesc struct, there seem to be two ways to mark a buffer
> page as dirty: setting the BM_DIRTY bit mask in the 'flags' field of the
> struct, and setting the 'cntxDirty' field to true. What is the
> difference between these two indications of a page's dirtiness?

I don't see any either ... could that be some artefact?

> 
> Or, more to the point, is there a reason we have two ways to do what
> looks like the same thing?
> 
> BTW, I'd like to remove the behavior that LockBuffer(buf, EXCLUSIVE)
> automatically marks the page as dirty. Since there are some situations
> in which we acquire an exclusive buffer lock but don't actually end up
> modifying the page, this results in dirtying more buffers than are
> necessary. I think it's also good practise for code that modifies a
> buffer to explicitly mark it as dirty, rather than depending upon
> LockBuffer() to do it. Does this sound reasonable, provided I find and
> catch all the places that depend upon this behavior? (i.e. and change
> them to explicitly mark the buffer as dirty)

Does make sense to me. "Provided you find and catch all" sounds like an 
"as early as possible in the 7.5 devel cycle" change to me, no?

It also looks to me that we are actually modifying the same sourcecode. 
We should try to avoid conflicting patches.


On 10/31 you wrote:
> I'd rather see us implement a buffer replacement policy that considers
> both frequency + recency (unlike LRU, which considers only recency).
> Ideally, that would work "automagically". I'm hoping to get a chance to
> implement ARC[1] during the 7.5 cycle.

I just posted another fix to my experimental ARC implementation to 
hackers. From the above I assume you're familiar with the algorithm. 
Could you please take a look at the "v2" diff and tell me if there's 
something fundamentally wrong?


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Tue, 04 Nov 2003 09:35:50 -0500
Jan Wieck <JanWieck@Yahoo.com>
Re: bufmgr code question
Neil Conway <neilc@samurai.com> writes:
> In the BufferDesc struct, there seem to be two ways to mark a buffer
> page as dirty: setting the BM_DIRTY bit mask in the 'flags' field of the
> struct, and setting the 'cntxDirty' field to true. What is the
> difference between these two indications of a page's dirtiness?
> Or, more to the point, is there a reason we have two ways to do what
> looks like the same thing?

I believe the reason for this is that you are allowed to set cntxDirty
to TRUE while holding (only) the context lock on the buffer, while
messing with the buffer flags word requires holding (only) the
BufMgrLock.  Getting rid of cntxDirty would mean additional grabbings of
the BufMgrLock when we want to mark buffers dirty.

The real solution to this is probably to rethink the rules for locking
in the buffer manager.  I've thought for some time that the BufMgrLock is
a system-wide bottleneck; if we could replace it by finer-grain locks,
and in particular use the per-buffer locks for operations affecting just
the state of a single buffer, we'd be ahead of the game.

> BTW, I'd like to remove the behavior that LockBuffer(buf, EXCLUSIVE)
> automatically marks the page as dirty.

Yeah.  This has been discussed before, see the archives:
http://archives.postgresql.org/pgsql-hackers/2002-11/msg00488.php
http://archives.postgresql.org/pgsql-hackers/2002-11/msg00679.php
http://archives.postgresql.org/pgsql-hackers/2002-11/msg00512.php

			regards, tom lane

Tue, 04 Nov 2003 12:07:34 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: bufmgr code question 

Is there a TODO here?

---------------------------------------------------------------------------

Tom Lane wrote:
> Neil Conway <neilc@samurai.com> writes:
> > In the BufferDesc struct, there seem to be two ways to mark a buffer
> > page as dirty: setting the BM_DIRTY bit mask in the 'flags' field of the
> > struct, and setting the 'cntxDirty' field to true. What is the
> > difference between these two indications of a page's dirtiness?
> > Or, more to the point, is there a reason we have two ways to do what
> > looks like the same thing?
> 
> I believe the reason for this is that you are allowed to set cntxDirty
> to TRUE while holding (only) the context lock on the buffer, while
> messing with the buffer flags word requires holding (only) the
> BufMgrLock.  Getting rid of cntxDirty would mean additional grabbings of
> the BufMgrLock when we want to mark buffers dirty.
> 
> The real solution to this is probably to rethink the rules for locking
> in the buffer manager.  I've thought for some time that the BufMgrLock is
> a system-wide bottleneck; if we could replace it by finer-grain locks,
> and in particular use the per-buffer locks for operations affecting just
> the state of a single buffer, we'd be ahead of the game.
> 
> > BTW, I'd like to remove the behavior that LockBuffer(buf, EXCLUSIVE)
> > automatically marks the page as dirty.
> 
> Yeah.  This has been discussed before, see the archives:
> http://archives.postgresql.org/pgsql-hackers/2002-11/msg00488.php
> http://archives.postgresql.org/pgsql-hackers/2002-11/msg00679.php
> http://archives.postgresql.org/pgsql-hackers/2002-11/msg00512.php
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 8: explain analyze is your friend
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 11 Nov 2003 22:03:08 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: bufmgr code question
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Is there a TODO here?

You needn't add one: I hope to tackle this during the 7.5 dev cycle.

-Neil


Sat, 15 Nov 2003 19:58:43 -0500
Neil Conway <neilc@samurai.com>
Re: bufmgr code question