I just finished running the parallel regress tests with inval.c rigged
to flush the relcache and syscache at every available opportunity,
that is anytime we could recognize a shared-cache-inval message from
another backend (see diff below).  This setup gives a whole new universe
of meaning to the word "slow" --- it took *three full days* to run the
standard "make check" procedure, including eighteen hours just to do the
"vacuum template1" part of initdb.  I kid you not.  But it worked.
Looks like the unexpected-cache-entry-drop class of problems are indeed
gone.

BTW, the reason the diff is rigged not to allow recursive cache flush
is not that it wouldn't work, it's that I didn't expect to live long
enough to finish such a test.  I didn't originally have that restriction
in there (and indeed found a bug that way: relcache flush could go into
an infinite loop if hit with another SI inval before it'd finished the
initial flush).  After fixing that bug, initdb was making steady
progress, but not at a rate that I wanted to wait out...

			regards, tom lane

*** src/backend/utils/cache/inval.c.orig	Wed Nov 15 23:57:44 2000
--- src/backend/utils/cache/inval.c	Mon Jan  1 17:27:53 2001
***************
*** 643,649 ****
--- 643,661 ----
  	elog(DEBUG, "DiscardInvalid called");
  #endif	 /* defined(INVALIDDEBUG) */
  
+ #if 1
+ 	/* DEBUG CHECK ONLY ... force cache reset at any opportunity */
+ 	static bool inReset = false;
+ 
+ 	if (! IsBootstrapProcessingMode() && !inReset)
+ 	{
+ 		inReset = true;
+ 		ResetSystemCaches();
+ 		inReset  = false;
+ 	}
+ #else
  	InvalidateSharedInvalid(CacheIdInvalidate, ResetSystemCaches);
+ #endif
  }
  
  /*

Fri, 05 Jan 2001 01:18:47 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Well, we seem to be proof against cache-inval problems now
Tom Lane wrote:
> 
> I just finished running the parallel regress tests with inval.c rigged
> to flush the relcache and syscache at every available opportunity,
> that is anytime we could recognize a shared-cache-inval message from
> another backend (see diff below).  This setup gives a whole new universe
> of meaning to the word "slow" --- it took *three full days* to run the
> standard "make check" procedure, including eighteen hours just to do the
> "vacuum template1" part of initdb.  I kid you not.  But it worked.
> Looks like the unexpected-cache-entry-drop class of problems are indeed
> gone.
> 

Great.
Thanks.

Hiroshi Inoue

Fri, 05 Jan 2001 16:55:23 +0900
Hiroshi Inoue <Inoue@tpf.co.jp>
Re: Well, we seem to be proof against cache-inval problems now
On Fri, 5 Jan 2001, Tom Lane wrote:

> I just finished running the parallel regress tests with inval.c rigged
> to flush the relcache and syscache at every available opportunity,
> that is anytime we could recognize a shared-cache-inval message from
> another backend (see diff below).  This setup gives a whole new universe
> of meaning to the word "slow" --- it took *three full days* to run the
> standard "make check" procedure, including eighteen hours just to do the
> "vacuum template1" part of initdb.  I kid you not.  But it worked.
> Looks like the unexpected-cache-entry-drop class of problems are indeed
> gone.
Tom, I'm not sure how (or whether) this relates to "alter table" happening
when someone else is doing a SELECT from table. Are you saying that it
should work without any locking or I'm completely off base?

-alex




Fri, 5 Jan 2001 14:13:26 -0500 (EST)
Alex Pilosov <alex@pilosoft.com>
Re: Well, we seem to be proof against cache-inval problems
 now
Alex Pilosov <alex@pilosoft.com> writes:
> Tom, I'm not sure how (or whether) this relates to "alter table" happening
> when someone else is doing a SELECT from table.

The ALTER will wait for the SELECT to finish.  That's not related to the
internal cache problem that I was worried about.

			regards, tom lane

Fri, 05 Jan 2001 14:14:29 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Well, we seem to be proof against cache-inval problems now 
Can this now be marked as done?

* Modification  of  pg_class  can  happen while table in use by  another
backend.  Might  lead  to  MVCC  inside  of  syscache

> I just finished running the parallel regress tests with inval.c rigged
> to flush the relcache and syscache at every available opportunity,
> that is anytime we could recognize a shared-cache-inval message from
> another backend (see diff below).  This setup gives a whole new universe
> of meaning to the word "slow" --- it took *three full days* to run the
> standard "make check" procedure, including eighteen hours just to do the
> "vacuum template1" part of initdb.  I kid you not.  But it worked.
> Looks like the unexpected-cache-entry-drop class of problems are indeed
> gone.
> 
> BTW, the reason the diff is rigged not to allow recursive cache flush
> is not that it wouldn't work, it's that I didn't expect to live long
> enough to finish such a test.  I didn't originally have that restriction
> in there (and indeed found a bug that way: relcache flush could go into
> an infinite loop if hit with another SI inval before it'd finished the
> initial flush).  After fixing that bug, initdb was making steady
> progress, but not at a rate that I wanted to wait out...
> 
> 			regards, tom lane
> 
> *** src/backend/utils/cache/inval.c.orig	Wed Nov 15 23:57:44 2000
> --- src/backend/utils/cache/inval.c	Mon Jan  1 17:27:53 2001
> ***************
> *** 643,649 ****
> --- 643,661 ----
>   	elog(DEBUG, "DiscardInvalid called");
>   #endif	 /* defined(INVALIDDEBUG) */
>   
> + #if 1
> + 	/* DEBUG CHECK ONLY ... force cache reset at any opportunity */
> + 	static bool inReset = false;
> + 
> + 	if (! IsBootstrapProcessingMode() && !inReset)
> + 	{
> + 		inReset = true;
> + 		ResetSystemCaches();
> + 		inReset  = false;
> + 	}
> + #else
>   	InvalidateSharedInvalid(CacheIdInvalidate, ResetSystemCaches);
> + #endif
>   }
>   
>   /*
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 9 Jan 2001 08:50:08 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Well, we seem to be proof against cache-inval problems
 now
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Can this now be marked as done?
> * Modification  of  pg_class  can  happen while table in use by  another
> backend.  Might  lead  to  MVCC  inside  of  syscache

I'm not sure.  Do you have any record of what the concern was, in
detail?  I don't understand what the TODO item is trying to say.

			regards, tom lane

Tue, 09 Jan 2001 13:00:55 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Well, we seem to be proof against cache-inval problems now 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Can this now be marked as done?
> > * Modification  of  pg_class  can  happen while table in use by  another
> > backend.  Might  lead  to  MVCC  inside  of  syscache
> 
> I'm not sure.  Do you have any record of what the concern was, in
> detail?  I don't understand what the TODO item is trying to say.

I assumed it was the problem of table lookups with no locking.  No idea
what the MVCC mention is about.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 9 Jan 2001 13:02:02 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Well, we seem to be proof against cache-inval problems
 now
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Bruce Momjian <pgman@candle.pha.pa.us> writes:
>>>> Can this now be marked as done?
>>>> * Modification  of  pg_class  can  happen while table in use by  another
>>>> backend.  Might  lead  to  MVCC  inside  of  syscache
>> 
>> I'm not sure.  Do you have any record of what the concern was, in
>> detail?  I don't understand what the TODO item is trying to say.

> I assumed it was the problem of table lookups with no locking.  No idea
> what the MVCC mention is about.

I checked the CVS archives and found that you added that TODO item on
4-Feb-2000.  I could not, however, find any relevant discussion in the
pghackers archives in the first few days of February.  Do you have
anything archived that might help narrow it down?

			regards, tom lane

Tue, 09 Jan 2001 19:37:46 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Well, we seem to be proof against cache-inval problems now 
No?  :-)

> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >>>> Can this now be marked as done?
> >>>> * Modification  of  pg_class  can  happen while table in use by  another
> >>>> backend.  Might  lead  to  MVCC  inside  of  syscache
> >> 
> >> I'm not sure.  Do you have any record of what the concern was, in
> >> detail?  I don't understand what the TODO item is trying to say.
> 
> > I assumed it was the problem of table lookups with no locking.  No idea
> > what the MVCC mention is about.
> 
> I checked the CVS archives and found that you added that TODO item on
> 4-Feb-2000.  I could not, however, find any relevant discussion in the
> pghackers archives in the first few days of February.  Do you have
> anything archived that might help narrow it down?
> 
> 			regards, tom lane
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 9 Jan 2001 21:32:00 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Well, we seem to be proof against cache-inval problems
 now
I barely understand the items sometimes.

> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >>>> Can this now be marked as done?
> >>>> * Modification  of  pg_class  can  happen while table in use by  another
> >>>> backend.  Might  lead  to  MVCC  inside  of  syscache
> >> 
> >> I'm not sure.  Do you have any record of what the concern was, in
> >> detail?  I don't understand what the TODO item is trying to say.
> 
> > I assumed it was the problem of table lookups with no locking.  No idea
> > what the MVCC mention is about.
> 
> I checked the CVS archives and found that you added that TODO item on
> 4-Feb-2000.  I could not, however, find any relevant discussion in the
> pghackers archives in the first few days of February.  Do you have
> anything archived that might help narrow it down?
> 
> 			regards, tom lane
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 9 Jan 2001 21:32:18 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Well, we seem to be proof against cache-inval problems
 now