
Okay, this is with code of ~May 4th ... a 'psql' connection to the
database hangs solid.

errout is dated:

pgsql% !ls
ls -lt
total 13324
-rw-------   1 pgsql  pgsql  4842715 May  7 10:57 errout.5432

and the last few lines contain:

ERROR:  parser: parse error at or near "vpti"
pq_recvbuf: unexpected EOF on client connection
pq_flush: send() failed: Broken pipe
pq_recvbuf: recv() failed: Connection reset by peer
pq_recvbuf: unexpected EOF on client connection
pq_recvbuf: unexpected EOF on client connection
pq_flush: send() failed: Broken pipe
pq_recvbuf: recv() failed: Connection reset by peer

But, of course, no date/time ...

ps shows:

USER    PID %CPU %MEM   VSZ  RSS  TT  STAT STARTED      TIME COMMAND
pgsql 33515  0.0  0.0     0    0  ??  Z     4:45PM   0:00.00  (postgres)
pgsql 33516  0.0  0.0     0    0  ??  Z     4:45PM   0:00.00  (postgres)
pgsql 93757  0.0  0.2  1456 1088  p0  S    Wed03PM   0:01.11 -su (tcsh)
pgsql  7100  0.0  0.5 38692 2616  ??  Is   Fri12AM   8:43.44 /pgsql/bin/postmas
pgsql 33667  0.0  0.0   396  224  p0  R+    7:35PM   0:00.00 ps ux

and postmaster is started with:

pgsql% cat pgstart
#!/bin/tcsh
setenv PORT 5432
setenv POSTMASTER /pgsql/bin/postmaster
unlimit
${POSTMASTER} -B 4096 -N 128 -S -o "-F -o /pgsql/errout.${PORT} -S 32768" \
        -i -p ${PORT} -D/pgsql/data

The machine is a Dual PIII with 512Meg of RAM, running FreeBSD 4.0-STABLE
from April 22nd ...

pgsql% truss -p 7100

Shows zilch ...

Since this is a production server, I can't just leave it there hung like
that, but if someone wants to give some instructions on what to do the
next time this happens, please feel free to do so, and I'll add that to my
list ... maybe run a gdb command on it, since truss doesn't appear to
help?

At this time, I consider this to be a show-stopper on the release ... this
is what happened the last time when the result appeared to be the index
corruption ... this time, I've checked a VACUUM after re-starting and it
doesn't appear to be a problem, but they might not have been related, just
a fluke ...

Marc G. Fournier                   ICQ#7615664              IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Sun, 7 May 2000 20:44:08 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
It happened again: Server hung up solid 
The Hermit Hacker <scrappy@hub.org> writes:
> Okay, this is with code of ~May 4th ... a 'psql' connection to the
> database hangs solid.

Do you mean you can't make a connection at all?  Is there any indication
that the postmaster is lighting off a backend for you?  Since you show
a couple of zombie backends hanging around, it would seem like a good
bet that the postmaster itself is wedged and not responding to events,
but I'm not sure.

> errout is dated:

> pgsql% !ls
> ls -lt
> total 13324
> -rw-------   1 pgsql  pgsql  4842715 May  7 10:57 errout.5432

> and the last few lines contain:

> ERROR:  parser: parse error at or near "vpti"
> pq_recvbuf: unexpected EOF on client connection
> pq_flush: send() failed: Broken pipe
> pq_recvbuf: recv() failed: Connection reset by peer
> pq_recvbuf: unexpected EOF on client connection
> pq_recvbuf: unexpected EOF on client connection
> pq_flush: send() failed: Broken pipe
> pq_recvbuf: recv() failed: Connection reset by peer

> But, of course, no date/time ...

Given that the file mod time is considerably before the hang (right?)
the messages in it are probably unrelated.  It does seem odd that you
have so many clients disconnecting ungracefully; what client apps are
you running?

> Since this is a production server, I can't just leave it there hung like
> that, but if someone wants to give some instructions on what to do the
> next time this happens, please feel free to do so, and I'll add that to my
> list ... maybe run a gdb command on it, since truss doesn't appear to
> help?

Try killing the postmaster itself in such a way as to produce a coredump
(kill -ABORT ought to do) and get a backtrace from that.  It might also
be worth running the postmaster with connection tracing turned on (I
forget the incantation for that, but it should be in TFM).

> At this time, I consider this to be a show-stopper on the release ... this
> is what happened the last time when the result appeared to be the index
> corruption

If the postmaster is hanging then it's almost certainly unrelated to
index corruption...

			regards, tom lane

Sun, 07 May 2000 20:35:33 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: It happened again: Server hung up solid 
On Sun, 7 May 2000, Tom Lane wrote:

> The Hermit Hacker <scrappy@hub.org> writes:
> > Okay, this is with code of ~May 4th ... a 'psql' connection to the
> > database hangs solid.
> 
> Do you mean you can't make a connection at all?  Is there any indication
> that the postmaster is lighting off a backend for you?  Since you show
> a couple of zombie backends hanging around, it would seem like a good
> bet that the postmaster itself is wedged and not responding to events,
> but I'm not sure.

This appears to be the case, but next time it happens I will make
double-sure of that ... considering that it was ~7pm at night when I
tried, my initial guess is that nothing is going through postmaster at the
time of hte hang ...

> 
> Given that the file mod time is considerably before the hang (right?)
> the messages in it are probably unrelated.  It does seem odd that you
> have so many clients disconnecting ungracefully; what client apps are
> you running?

alot of dbi stuff, the search engine for udmsearch, some php ... the
server is currently serving ~12 databases for various clients ...

> Try killing the postmaster itself in such a way as to produce a coredump
> (kill -ABORT ought to do) and get a backtrace from that.  It might also
> be worth running the postmaster with connection tracing turned on (I
> forget the incantation for that, but it should be in TFM).

Will look at that one ...



Sun, 7 May 2000 21:48:28 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: It happened again: Server hung up solid 


Okay, just happened again ... no postgres backend is being started:

USER    PID %CPU %MEM   VSZ  RSS  TT  STAT STARTED      TIME COMMAND
pgsql 34611  0.0  0.0     0    0  ??  Z     8:43PM   0:00.00  (postgres)
pgsql 93757  0.0  0.2  1456 1104  p0  S    Wed03PM   0:01.16 -su (tcsh)
pgsql 33683  0.0  0.6 38356 3024  ??  Is    7:38PM   0:03.54 /pgsql/bin/postmaster -B 4096 -N 128 -S -o -F -o /pgsql/errout.5432
pgsql 34677  0.0  0.2  1408 1048  p2  S     8:50PM   0:00.07 -su (tcsh)
pgsql 34685  0.0  0.2  1652 1032  p0  S+    8:51PM   0:00.01 psql udmsearch
pgsql 34687  0.0  0.0   400  232  p2  R+    8:51PM   0:00.00 ps ux

Going to look at the connection tracing option now and see what I can come
up with ...


On Sun, 7 May 2000, Tom Lane wrote:

> The Hermit Hacker <scrappy@hub.org> writes:
> > Okay, this is with code of ~May 4th ... a 'psql' connection to the
> > database hangs solid.
> 
> Do you mean you can't make a connection at all?  Is there any indication
> that the postmaster is lighting off a backend for you?  Since you show
> a couple of zombie backends hanging around, it would seem like a good
> bet that the postmaster itself is wedged and not responding to events,
> but I'm not sure.
> 
> > errout is dated:
> 
> > pgsql% !ls
> > ls -lt
> > total 13324
> > -rw-------   1 pgsql  pgsql  4842715 May  7 10:57 errout.5432
> 
> > and the last few lines contain:
> 
> > ERROR:  parser: parse error at or near "vpti"
> > pq_recvbuf: unexpected EOF on client connection
> > pq_flush: send() failed: Broken pipe
> > pq_recvbuf: recv() failed: Connection reset by peer
> > pq_recvbuf: unexpected EOF on client connection
> > pq_recvbuf: unexpected EOF on client connection
> > pq_flush: send() failed: Broken pipe
> > pq_recvbuf: recv() failed: Connection reset by peer
> 
> > But, of course, no date/time ...
> 
> Given that the file mod time is considerably before the hang (right?)
> the messages in it are probably unrelated.  It does seem odd that you
> have so many clients disconnecting ungracefully; what client apps are
> you running?
> 
> > Since this is a production server, I can't just leave it there hung like
> > that, but if someone wants to give some instructions on what to do the
> > next time this happens, please feel free to do so, and I'll add that to my
> > list ... maybe run a gdb command on it, since truss doesn't appear to
> > help?
> 
> Try killing the postmaster itself in such a way as to produce a coredump
> (kill -ABORT ought to do) and get a backtrace from that.  It might also
> be worth running the postmaster with connection tracing turned on (I
> forget the incantation for that, but it should be in TFM).
> 
> > At this time, I consider this to be a show-stopper on the release ... this
> > is what happened the last time when the result appeared to be the index
> > corruption
> 
> If the postmaster is hanging then it's almost certainly unrelated to
> index corruption...
> 
> 			regards, tom lane
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Sun, 7 May 2000 21:52:41 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: It happened again: Server hung up solid 


kill -ABRT does nothing:

pgsql% kill -ABRT 33683
pgsql% !ps
ps ux
USER    PID %CPU %MEM   VSZ  RSS  TT  STAT STARTED      TIME COMMAND
pgsql 34611  0.0  0.0     0    0  ??  Z     8:43PM   0:00.00  (postgres)
pgsql 93757  0.0  0.2  1456 1104  p0  S    Wed03PM   0:01.17 -su (tcsh)
pgsql 33683  0.0  0.6 38356 3024  ??  Is    7:38PM   0:03.54 /pgsql/bin/postmas
pgsql 34677  0.0  0.2  1408 1048  p2  S+    8:50PM   0:00.08 -su (tcsh)
pgsql 34696  0.0  0.0   396  232  p0  R+    8:56PM   0:00.00 ps ux
pgsql% !ps
ps ux
USER    PID %CPU %MEM   VSZ  RSS  TT  STAT STARTED      TIME COMMAND
pgsql 34611  0.0  0.0     0    0  ??  Z     8:43PM   0:00.00  (postgres)
pgsql 93757  0.0  0.2  1456 1104  p0  S    Wed03PM   0:01.17 -su (tcsh)
pgsql 33683  0.0  0.6 38356 3024  ??  Is    7:38PM   0:03.54 /pgsql/bin/postmas
pgsql 34677  0.0  0.2  1408 1048  p2  S+    8:50PM   0:00.08 -su (tcsh)
pgsql 34697  0.0  0.0   396  232  p0  R+    8:56PM   0:00.00 ps ux


On Sun, 7 May 2000, The Hermit Hacker wrote:

> 
> 
> Okay, just happened again ... no postgres backend is being started:
> 
> USER    PID %CPU %MEM   VSZ  RSS  TT  STAT STARTED      TIME COMMAND
> pgsql 34611  0.0  0.0     0    0  ??  Z     8:43PM   0:00.00  (postgres)
> pgsql 93757  0.0  0.2  1456 1104  p0  S    Wed03PM   0:01.16 -su (tcsh)
> pgsql 33683  0.0  0.6 38356 3024  ??  Is    7:38PM   0:03.54 /pgsql/bin/postmaster -B 4096 -N 128 -S -o -F -o /pgsql/errout.5432
> pgsql 34677  0.0  0.2  1408 1048  p2  S     8:50PM   0:00.07 -su (tcsh)
> pgsql 34685  0.0  0.2  1652 1032  p0  S+    8:51PM   0:00.01 psql udmsearch
> pgsql 34687  0.0  0.0   400  232  p2  R+    8:51PM   0:00.00 ps ux
> 
> Going to look at the connection tracing option now and see what I can come
> up with ...
> 
> 
> On Sun, 7 May 2000, Tom Lane wrote:
> 
> > The Hermit Hacker <scrappy@hub.org> writes:
> > > Okay, this is with code of ~May 4th ... a 'psql' connection to the
> > > database hangs solid.
> > 
> > Do you mean you can't make a connection at all?  Is there any indication
> > that the postmaster is lighting off a backend for you?  Since you show
> > a couple of zombie backends hanging around, it would seem like a good
> > bet that the postmaster itself is wedged and not responding to events,
> > but I'm not sure.
> > 
> > > errout is dated:
> > 
> > > pgsql% !ls
> > > ls -lt
> > > total 13324
> > > -rw-------   1 pgsql  pgsql  4842715 May  7 10:57 errout.5432
> > 
> > > and the last few lines contain:
> > 
> > > ERROR:  parser: parse error at or near "vpti"
> > > pq_recvbuf: unexpected EOF on client connection
> > > pq_flush: send() failed: Broken pipe
> > > pq_recvbuf: recv() failed: Connection reset by peer
> > > pq_recvbuf: unexpected EOF on client connection
> > > pq_recvbuf: unexpected EOF on client connection
> > > pq_flush: send() failed: Broken pipe
> > > pq_recvbuf: recv() failed: Connection reset by peer
> > 
> > > But, of course, no date/time ...
> > 
> > Given that the file mod time is considerably before the hang (right?)
> > the messages in it are probably unrelated.  It does seem odd that you
> > have so many clients disconnecting ungracefully; what client apps are
> > you running?
> > 
> > > Since this is a production server, I can't just leave it there hung like
> > > that, but if someone wants to give some instructions on what to do the
> > > next time this happens, please feel free to do so, and I'll add that to my
> > > list ... maybe run a gdb command on it, since truss doesn't appear to
> > > help?
> > 
> > Try killing the postmaster itself in such a way as to produce a coredump
> > (kill -ABORT ought to do) and get a backtrace from that.  It might also
> > be worth running the postmaster with connection tracing turned on (I
> > forget the incantation for that, but it should be in TFM).
> > 
> > > At this time, I consider this to be a show-stopper on the release ... this
> > > is what happened the last time when the result appeared to be the index
> > > corruption
> > 
> > If the postmaster is hanging then it's almost certainly unrelated to
> > index corruption...
> > 
> > 			regards, tom lane
> > 
> 
> Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
> Systems Administrator @ hub.org 
> primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Sun, 7 May 2000 21:57:20 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: It happened again: Server hung up solid 
On Sun, 7 May 2000, The Hermit Hacker wrote:

> 
> 
> Okay, just happened again ... no postgres backend is being started:

I don't know how close in time it was, but I just hit reload on that
query that was sent to webmaster.

Vince.



> 
> USER    PID %CPU %MEM   VSZ  RSS  TT  STAT STARTED      TIME COMMAND
> pgsql 34611  0.0  0.0     0    0  ??  Z     8:43PM   0:00.00  (postgres)
> pgsql 93757  0.0  0.2  1456 1104  p0  S    Wed03PM   0:01.16 -su (tcsh)
> pgsql 33683  0.0  0.6 38356 3024  ??  Is    7:38PM   0:03.54 /pgsql/bin/postmaster -B 4096 -N 128 -S -o -F -o /pgsql/errout.5432
> pgsql 34677  0.0  0.2  1408 1048  p2  S     8:50PM   0:00.07 -su (tcsh)
> pgsql 34685  0.0  0.2  1652 1032  p0  S+    8:51PM   0:00.01 psql udmsearch
> pgsql 34687  0.0  0.0   400  232  p2  R+    8:51PM   0:00.00 ps ux
> 
> Going to look at the connection tracing option now and see what I can come
> up with ...
> 
> 
> On Sun, 7 May 2000, Tom Lane wrote:
> 
> > The Hermit Hacker <scrappy@hub.org> writes:
> > > Okay, this is with code of ~May 4th ... a 'psql' connection to the
> > > database hangs solid.
> > 
> > Do you mean you can't make a connection at all?  Is there any indication
> > that the postmaster is lighting off a backend for you?  Since you show
> > a couple of zombie backends hanging around, it would seem like a good
> > bet that the postmaster itself is wedged and not responding to events,
> > but I'm not sure.
> > 
> > > errout is dated:
> > 
> > > pgsql% !ls
> > > ls -lt
> > > total 13324
> > > -rw-------   1 pgsql  pgsql  4842715 May  7 10:57 errout.5432
> > 
> > > and the last few lines contain:
> > 
> > > ERROR:  parser: parse error at or near "vpti"
> > > pq_recvbuf: unexpected EOF on client connection
> > > pq_flush: send() failed: Broken pipe
> > > pq_recvbuf: recv() failed: Connection reset by peer
> > > pq_recvbuf: unexpected EOF on client connection
> > > pq_recvbuf: unexpected EOF on client connection
> > > pq_flush: send() failed: Broken pipe
> > > pq_recvbuf: recv() failed: Connection reset by peer
> > 
> > > But, of course, no date/time ...
> > 
> > Given that the file mod time is considerably before the hang (right?)
> > the messages in it are probably unrelated.  It does seem odd that you
> > have so many clients disconnecting ungracefully; what client apps are
> > you running?
> > 
> > > Since this is a production server, I can't just leave it there hung like
> > > that, but if someone wants to give some instructions on what to do the
> > > next time this happens, please feel free to do so, and I'll add that to my
> > > list ... maybe run a gdb command on it, since truss doesn't appear to
> > > help?
> > 
> > Try killing the postmaster itself in such a way as to produce a coredump
> > (kill -ABORT ought to do) and get a backtrace from that.  It might also
> > be worth running the postmaster with connection tracing turned on (I
> > forget the incantation for that, but it should be in TFM).
> > 
> > > At this time, I consider this to be a show-stopper on the release ... this
> > > is what happened the last time when the result appeared to be the index
> > > corruption
> > 
> > If the postmaster is hanging then it's almost certainly unrelated to
> > index corruption...
> > 
> > 			regards, tom lane
> > 
> 
> Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
> Systems Administrator @ hub.org 
> primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 
> 
> 

-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sun, 7 May 2000 20:57:42 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: It happened again: Server hung up solid 

With -d set to 1 (connection tracing), all I see when I connect, in the
log files, is:

FindExec: found "/pgsql/bin/postgres" using argv[0]
FindExec: found "/pgsql/bin/postgres" using argv[0]

doesn't tell me to what I'm connecting through ...


On Sun, 7 May 2000, The Hermit Hacker wrote:

> 
> Okay, this is with code of ~May 4th ... a 'psql' connection to the
> database hangs solid.
> 
> errout is dated:
> 
> pgsql% !ls
> ls -lt
> total 13324
> -rw-------   1 pgsql  pgsql  4842715 May  7 10:57 errout.5432
> 
> and the last few lines contain:
> 
> ERROR:  parser: parse error at or near "vpti"
> pq_recvbuf: unexpected EOF on client connection
> pq_flush: send() failed: Broken pipe
> pq_recvbuf: recv() failed: Connection reset by peer
> pq_recvbuf: unexpected EOF on client connection
> pq_recvbuf: unexpected EOF on client connection
> pq_flush: send() failed: Broken pipe
> pq_recvbuf: recv() failed: Connection reset by peer
> 
> But, of course, no date/time ...
> 
> ps shows:
> 
> USER    PID %CPU %MEM   VSZ  RSS  TT  STAT STARTED      TIME COMMAND
> pgsql 33515  0.0  0.0     0    0  ??  Z     4:45PM   0:00.00  (postgres)
> pgsql 33516  0.0  0.0     0    0  ??  Z     4:45PM   0:00.00  (postgres)
> pgsql 93757  0.0  0.2  1456 1088  p0  S    Wed03PM   0:01.11 -su (tcsh)
> pgsql  7100  0.0  0.5 38692 2616  ??  Is   Fri12AM   8:43.44 /pgsql/bin/postmas
> pgsql 33667  0.0  0.0   396  224  p0  R+    7:35PM   0:00.00 ps ux
> 
> and postmaster is started with:
> 
> pgsql% cat pgstart
> #!/bin/tcsh
> setenv PORT 5432
> setenv POSTMASTER /pgsql/bin/postmaster
> unlimit
> ${POSTMASTER} -B 4096 -N 128 -S -o "-F -o /pgsql/errout.${PORT} -S 32768" \
>         -i -p ${PORT} -D/pgsql/data
> 
> The machine is a Dual PIII with 512Meg of RAM, running FreeBSD 4.0-STABLE
> from April 22nd ...
> 
> pgsql% truss -p 7100
> 
> Shows zilch ...
> 
> Since this is a production server, I can't just leave it there hung like
> that, but if someone wants to give some instructions on what to do the
> next time this happens, please feel free to do so, and I'll add that to my
> list ... maybe run a gdb command on it, since truss doesn't appear to
> help?
> 
> At this time, I consider this to be a show-stopper on the release ... this
> is what happened the last time when the result appeared to be the index
> corruption ... this time, I've checked a VACUUM after re-starting and it
> doesn't appear to be a problem, but they might not have been related, just
> a fluke ...
> 
> Marc G. Fournier                   ICQ#7615664              IRC Nick: Scrappy
> Systems Administrator @ hub.org 
> primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Sun, 7 May 2000 21:59:26 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: It happened again: Server hung up solid 
The Hermit Hacker <scrappy@hub.org> writes:
> kill -ABRT does nothing:

Oh?  Must be hung up in a kernel call then.  That will probably mean
that you can't attach to the stuck process with gdb either (though
it'd be worth trying, since a backtrace would be mighty useful if
you could get it).

My next thought is to truss the postmaster process before it hangs
up, with hopes of finding out what kernel call is hanging.

Also, you might try netstat to see if you can see any freshly-opened
incoming connections when it happens.  Also, "lsof -p" or local
equivalent on the stuck postmaster.

			regards, tom lane

Sun, 07 May 2000 22:13:52 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: It happened again: Server hung up solid 