
I am just creating this now...this is to be considered the first *releaes*
candidate for v6.5 ...

I expect there to be *no more* commits to the source tree until after a
formal release is done...

Baring any bugs, this will become the formal release as of 15:30EST on
Wednesday, giving two days for ppl to run her through and make sure
nothing obvious is outstanding.

Consider the cvs repository *frozen* until I announce otherwise, as I will
once more tag things so that we have a -CURRENT vs -STABLE source tree to
work with.

There are to be *no* code changes between now and then, except by having
such changes reviewed by one other committer...about the only thing I'd
expect to see is something port specific...

At this time, I hope everyone will take a second to download the above
.tar.gz file and run things through to make sure there aren't any
outstanding problems ...

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Mon, 7 Jun 1999 00:40:13 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
postgresql-v6.5beta2.tar.gz ...
The Hermit Hacker wrote:
> 
> I am just creating this now...this is to be considered the first *releaes*
> candidate for v6.5 ...
> 
> I expect there to be *no more* commits to the source tree until after a
> formal release is done...

Ah. I wanted to implement proper handling of case when tuple in 
update-chain is updated by "too old" xaction (part of chain should
be deleted) but ok - it's not bug, just unimplemented feature.

(BTW, Mark, I have to update lock.sgml)

Hiroshi, please run your test. You'll get NOTICEs instead of ERRORs.
Unfortunately, I can't get your test finished locally - I always
get core and gdb says too little:

#0  0x2823452f in strcpy ()                                                    
(gdb) bt                                                                       
#0  0x2823452f in strcpy ()                                                    
#1  0xefbf8944 in ?? ()                                                        
#2  0x812d19d in hash_search (hashp=0x1c518bfc,                                
    keyPtr=0x8901428d <Address 0x8901428d out of bounds>, action=1233853505,   
    foundPtr=0x4d89410c <Address 0x4d89410c out of bounds>) at dynahash.c:651  
#3  0x4d8b0000 in ?? ()                                                        
Cannot access memory at address 0x3c7.                                         
(gdb) f 2                                                                      
#2  0x812cfbd in hash_search (hashp=0x1c518bfc,                                
    keyPtr=0x8901428d <Address 0x8901428d out of bounds>, action=1233853505,   
    foundPtr=0x4d89410c <Address 0x4d89410c out of bounds>) at dynahash.c:651  
651             memmove(destAddr, keyPtr, hctl->keysize);                      
                ^^^^^^^
??? memmove shouldn't call strcpy. Something is broken,
but maybe due to FreeBSD 3.0 on my box... I'll run test
on 2.2.6 soon.

Vadim

Mon, 07 Jun 1999 12:11:48 +0800
Vadim Mikheev <vadim@krs.ru>
Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...
On Mon, 7 Jun 1999, Vadim Mikheev wrote:

> The Hermit Hacker wrote:
> > 
> > I am just creating this now...this is to be considered the first *releaes*
> > candidate for v6.5 ...
> > 
> > I expect there to be *no more* commits to the source tree until after a
> > formal release is done...
> 
> Ah. I wanted to implement proper handling of case when tuple in 
> update-chain is updated by "too old" xaction (part of chain should
> be deleted) but ok - it's not bug, just unimplemented feature.

Can that wait until *after* 6.5 is released, for inclusion on v6.5.1? 

> (BTW, Mark, I have to update lock.sgml)

I have no major qualms about docs, if thomas doesn't ...

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Mon, 7 Jun 1999 01:27:59 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...
The Hermit Hacker wrote:
> 
> On Mon, 7 Jun 1999, Vadim Mikheev wrote:
> 
> > The Hermit Hacker wrote:
> > >
> > > I am just creating this now...this is to be considered the first *releaes*
> > > candidate for v6.5 ...
> > >
> > > I expect there to be *no more* commits to the source tree until after a
> > > formal release is done...
> >
> > Ah. I wanted to implement proper handling of case when tuple in
> > update-chain is updated by "too old" xaction (part of chain should
> > be deleted) but ok - it's not bug, just unimplemented feature.
> 
> Can that wait until *after* 6.5 is released, for inclusion on v6.5.1?

Yes. And if I'll not implement it in 6.5.1 then I'll not have time
for this after -:)

Vadim

Mon, 07 Jun 1999 12:31:07 +0800
Vadim Mikheev <vadim@krs.ru>
Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...
> -----Original Message-----
> From: owner-pgsql-hackers@postgreSQL.org
> [mailto:owner-pgsql-hackers@postgreSQL.org]On Behalf Of Vadim Mikheev
> Sent: Monday, June 07, 1999 1:12 PM
> To: The Hermit Hacker
> Cc: pgsql-hackers@postgreSQL.org
> Subject: Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...
> 
> 
> The Hermit Hacker wrote:
> > 
> > I am just creating this now...this is to be considered the 
> first *releaes*
> > candidate for v6.5 ...
> > 
> > I expect there to be *no more* commits to the source tree until after a
> > formal release is done...
> 
> Ah. I wanted to implement proper handling of case when tuple in 
> update-chain is updated by "too old" xaction (part of chain should
> be deleted) but ok - it's not bug, just unimplemented feature.
> 
> (BTW, Mark, I have to update lock.sgml)
> 
> Hiroshi, please run your test. You'll get NOTICEs instead of ERRORs.

I have run my test and got the following NOTICEs.

NOTICE:  Child itemid in update-chain marked as unused - can't 
continue vc_rpfheap

Thanks.

Hiroshi Inoue
Inoue@tpf.co.jp

Mon, 7 Jun 1999 15:17:49 +0900
"Hiroshi Inoue" <Inoue@tpf.co.jp>
RE: [HACKERS] postgresql-v6.5beta2.tar.gz ...
Hiroshi Inoue wrote:
> 
> >
> > Hiroshi, please run your test. You'll get NOTICEs instead of ERRORs.
> 
> I have run my test and got the following NOTICEs.
> 
> NOTICE:  Child itemid in update-chain marked as unused - can't
> continue vc_rpfheap

I'm still getting troubles when running your test due to

1. bug in cache invalidation code: when we invalidate relcache
   we forget to free MdfdVec in md.c!

   Vacuum invalidates a relation tuple in pg_class and concurrent
   xactions invalidate corresponding relcache entry, but don't
   free MdfdVec and so allocate new one for the same relation
   more and more. Each MdfdVed requires own fd.c:Vfd entry -> below

2. fd.c:pg_nofile()->sysconf(_SC_OPEN_MAX) returns in FreeBSD 
   near total number of files that can be opened in system
   (by _all_ users/procs). With total number of opened files
   ~ 2000 I can run your test with 10-20 simultaneous
   xactions for very short time, -:)

   Should we limit fd.c:no_files to ~ 256?
   This is port-specific, of course...

Vadim
P.S. Hiroshi test I run:

10-20 psql < file with 10000 xactions:

BEGIN;
UPDATE single row in table;
END;

A script run "vacuum table" each rand(10) + 1 sec.

Mon, 07 Jun 1999 18:49:16 +0800
Vadim Mikheev <vadim@krs.ru>
Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...
> > Ah. I wanted to implement proper handling of case when tuple in 
> > update-chain is updated by "too old" xaction (part of chain should
> > be deleted) but ok - it's not bug, just unimplemented feature.
> 
> Can that wait until *after* 6.5 is released, for inclusion on v6.5.1? 
> 
> > (BTW, Mark, I have to update lock.sgml)
> 
> I have no major qualms about docs, if thomas doesn't ...

No.  If you want it, do it now, not in 6.5.1.  Adding features in
minor releases is worse than adding stuff now.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 7 Jun 1999 09:18:47 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...
On Mon, 7 Jun 1999, Bruce Momjian wrote:

> > > Ah. I wanted to implement proper handling of case when tuple in 
> > > update-chain is updated by "too old" xaction (part of chain should
> > > be deleted) but ok - it's not bug, just unimplemented feature.
> > 
> > Can that wait until *after* 6.5 is released, for inclusion on v6.5.1? 
> > 
> > > (BTW, Mark, I have to update lock.sgml)
> > 
> > I have no major qualms about docs, if thomas doesn't ...
> 
> No.  If you want it, do it now, not in 6.5.1.  Adding features in
> minor releases is worse than adding stuff now.

With Vadim himself uncertain as to the effect of doing the patch, then it
should bve left for 6.6 then...

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Mon, 7 Jun 1999 10:40:55 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...
> -----Original Message-----
> From: root@sunpine.krs.ru [mailto:root@sunpine.krs.ru]On Behalf Of Vadim
> Mikheev
> Sent: Monday, June 07, 1999 7:49 PM
> To: Hiroshi Inoue
> Cc: The Hermit Hacker; pgsql-hackers@postgreSQL.org
> Subject: Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...
> 
> 
> Hiroshi Inoue wrote:
> > 
> > >
> > > Hiroshi, please run your test. You'll get NOTICEs instead of ERRORs.
> > 
> > I have run my test and got the following NOTICEs.
> > 
> > NOTICE:  Child itemid in update-chain marked as unused - can't
> > continue vc_rpfheap
> 
> I'm still getting troubles when running your test due to
> 
> 1. bug in cache invalidation code: when we invalidate relcache
>    we forget to free MdfdVec in md.c!
> 
>    Vacuum invalidates a relation tuple in pg_class and concurrent
>    xactions invalidate corresponding relcache entry, but don't
>    free MdfdVec and so allocate new one for the same relation

I have noticed the same thing   But at that point my concern was 
truncation of segmented relations. AFAIC concurrent xactions 
invalidate relcache entry when StartTransaction() or Command-
CounterIncrement() is called.  Unfortunately vacuum couldn't stop 
other sessions from executing StartTransaction() nor Command-
CounterIncrement().

>    more and more. Each MdfdVed requires own fd.c:Vfd entry -> below
> 
> 2. fd.c:pg_nofile()->sysconf(_SC_OPEN_MAX) returns in FreeBSD 
>    near total number of files that can be opened in system
>    (by _all_ users/procs). With total number of opened files
>    ~ 2000 I can run your test with 10-20 simultaneous
>    xactions for very short time, -:)
>

I have remembered another thing.
Someone has already reported to ML in Japan and I was able to 
find the cause easily thanks to him.

vacuum calls open() directly to create pg_vlock file.
Sometimes I was not able to open the file because of the lack of 
file descriptors().
FileNameOpenFile()/FileClose() etc should be called instead of 
open()/close() etc ?

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp

Tue, 8 Jun 1999 10:08:57 +0900
"Hiroshi Inoue" <Inoue@tpf.co.jp>
RE: [HACKERS] postgresql-v6.5beta2.tar.gz ...
>vacuum calls open() directly to create pg_vlock file.
>Sometimes I was not able to open the file because of the lack of 
>file descriptors().
>FileNameOpenFile()/FileClose() etc should be called instead of 
>open()/close() etc ?

Thanks Hiroshi! I forgot to forward his message regarding pg_vlock
issue. Also note that there are some codes that directly open files in
createdb/destroydb stuffs...
--
Tatsuo Ishii


Tue, 08 Jun 1999 10:17:34 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] postgresql-v6.5beta2.tar.gz ... 
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> vacuum calls open() directly to create pg_vlock file.
> Sometimes I was not able to open the file because of the lack of 
> file descriptors().
> FileNameOpenFile()/FileClose() etc should be called instead of 
> open()/close() etc ?

That is a good point, but actually I think it should not be a problem.
vacuum.c just does open() and immediately close() on pg_vlock; it
doesn't hold the file descriptor open.

When I fixed psort.c a few weeks ago, I looked through all the other
direct calls of open() and fopen() in the backend.  There are still half
a dozen or so, but none of them open more than one file or will hold the
file descriptor for longer than the execution of the function they're
in.  So I felt it was OK to leave them alone.

The reason it's OK is that fd.c doesn't use up all the available
file descriptors --- it tries to leave ten or so unused.  That's
necessary to ensure that library functions like dlopen() will work,
because they don't know anything about using fd.c's routines.

So, the occasional short-term file opening in vacuum.c and similar
places should not matter.  If those do fail for lack of FDs, then the
*real* problem is that fd.c is not estimating correctly how many file
descriptors it can safely use; that's what we need to fix.

But what I want to know right now is whether this behavior has been
seen with code from the last week or two.  Maybe the report is just
a side-effect of the FD leaks that used to exist in several places...

			regards, tom lane

Mon, 07 Jun 1999 22:34:20 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...
>That is a good point, but actually I think it should not be a problem.
>vacuum.c just does open() and immediately close() on pg_vlock; it
>doesn't hold the file descriptor open.
>
>When I fixed psort.c a few weeks ago, I looked through all the other
>direct calls of open() and fopen() in the backend.  There are still half
>a dozen or so, but none of them open more than one file or will hold the
>file descriptor for longer than the execution of the function they're
>in.  So I felt it was OK to leave them alone.
>
>The reason it's OK is that fd.c doesn't use up all the available
>file descriptors --- it tries to leave ten or so unused.  That's
>necessary to ensure that library functions like dlopen() will work,
>because they don't know anything about using fd.c's routines.
>
>So, the occasional short-term file opening in vacuum.c and similar
>places should not matter.  If those do fail for lack of FDs, then the
>*real* problem is that fd.c is not estimating correctly how many file
>descriptors it can safely use; that's what we need to fix.
>
>But what I want to know right now is whether this behavior has been
>seen with code from the last week or two.  Maybe the report is just
>a side-effect of the FD leaks that used to exist in several places...

If I correctly remember, the report was regarding 6.4.2. I will check
if it happens with current, and report back soon.
---
Tatsuo Ishii


Tue, 08 Jun 1999 11:42:09 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] postgresql-v6.5beta2.tar.gz ... 
> -----Original Message-----
> From: owner-pgsql-hackers@postgreSQL.org
> [mailto:owner-pgsql-hackers@postgreSQL.org]On Behalf Of Tom Lane
> Sent: Tuesday, June 08, 1999 11:34 AM
> To: pgsql-hackers@postgreSQL.org
> Subject: Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...
> 
> 
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > vacuum calls open() directly to create pg_vlock file.
> > Sometimes I was not able to open the file because of the lack of 
> > file descriptors().
> > FileNameOpenFile()/FileClose() etc should be called instead of 
> > open()/close() etc ?
> 
> That is a good point, but actually I think it should not be a problem.
> vacuum.c just does open() and immediately close() on pg_vlock; it
> doesn't hold the file descriptor open.
> 
> When I fixed psort.c a few weeks ago, I looked through all the other
> direct calls of open() and fopen() in the backend.  There are still half
> a dozen or so, but none of them open more than one file or will hold the
> file descriptor for longer than the execution of the function they're
> in.  So I felt it was OK to leave them alone.
> 
> The reason it's OK is that fd.c doesn't use up all the available
> file descriptors --- it tries to leave ten or so unused.  That's
> necessary to ensure that library functions like dlopen() will work,
> because they don't know anything about using fd.c's routines.
> 
> So, the occasional short-term file opening in vacuum.c and similar
> places should not matter.  If those do fail for lack of FDs, then the
> *real* problem is that fd.c is not estimating correctly how many file
> descriptors it can safely use; that's what we need to fix.
> 
> But what I want to know right now is whether this behavior has been
> seen with code from the last week or two.  Maybe the report is just
> a side-effect of the FD leaks that used to exist in several places...
>

I don't remember correctly how the phenomenon occured,sorry.
I couldn't reproduce it now.

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp

Tue, 8 Jun 1999 18:45:23 +0900
"Hiroshi Inoue" <Inoue@tpf.co.jp>
RE: [HACKERS] postgresql-v6.5beta2.tar.gz ...
>>That is a good point, but actually I think it should not be a problem.
>>vacuum.c just does open() and immediately close() on pg_vlock; it
>>doesn't hold the file descriptor open.
>>
>>When I fixed psort.c a few weeks ago, I looked through all the other
>>direct calls of open() and fopen() in the backend.  There are still half
>>a dozen or so, but none of them open more than one file or will hold the
>>file descriptor for longer than the execution of the function they're
>>in.  So I felt it was OK to leave them alone.
>>
>>The reason it's OK is that fd.c doesn't use up all the available
>>file descriptors --- it tries to leave ten or so unused.  That's
>>necessary to ensure that library functions like dlopen() will work,
>>because they don't know anything about using fd.c's routines.
>>
>>So, the occasional short-term file opening in vacuum.c and similar
>>places should not matter.  If those do fail for lack of FDs, then the
>>*real* problem is that fd.c is not estimating correctly how many file
>>descriptors it can safely use; that's what we need to fix.
>>
>>But what I want to know right now is whether this behavior has been
>>seen with code from the last week or two.  Maybe the report is just
>>a side-effect of the FD leaks that used to exist in several places...
>
>If I correctly remember, the report was regarding 6.4.2. I will check
>if it happens with current, and report back soon.

Done. With 32/64 users and 100 transactions each, I see no problem so
far.
--
Tatsuo Ishii



Tue, 08 Jun 1999 19:12:26 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] postgresql-v6.5beta2.tar.gz ... 