I've just noticed that libpq doesn't cope very gracefully if the backend
exits when not in the middle of a query (ie, because the postmaster told
it to quit after some other BE crashed).  The behavior in psql, for
example, is that the next time you issue a query, psql just exits
without printing anything at all.  This is Not Friendly, especially
considering that the BE sent a nice little notice message before it quit.

The main problem is that if the next thing you do is to send a new query,
send() sees that the connection has been closed and generates a SIGPIPE
signal.  By default that terminates the frontend process.

We could cure this by having libpq disable SIGPIPE, but we would have
to disable it before each send() and re-enable afterwards to avoid
affecting the behavior of the rest of the frontend application.
Two additional kernel calls per query sounds like a lot of overhead.
(We do actually do this when trying to close the connection, but not
during normal queries.)

Perhaps a better answer is to have PQsendQuery check for fresh input
from the backend before trying to send the query.  This would have two
side effects:
  1. If a NOTICE message has arrived, we could print it.
  2. If EOF is detected, we will reset the connection state to
     CONNECTION_BAD, which PQsendQuery can use to avoid trying to send.

The minimum cost to do this is one kernel call (a select(), which
unfortunately is probably a fairly expensive call) in the normal
case where no new input has arrived.  Another objection is that it's
not 100% bulletproof --- if the backend closes the connection in the
window between select() and send() then you can still get SIGPIPE'd.
The odds of this seem pretty small however.

I'm inclined to go with answer #2, because it seems to have less
of a performance impact, and it will ensure that the backend's polite
"The Postmaster has informed me that some other backend died abnormally
and possibly corrupted shared memory." message gets displayed.  With
approach #1 we'd still have to go through some pushups to get the
notice to come out.

Does anyone have an objection, or a better idea?

			regards, tom lane

Tue, 28 Jul 1998 13:23:35 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Coping with backend crash in libpq
On Tue, Jul 28, 1998 at 01:23:35PM -0400, Tom Lane wrote:
> I've just noticed that libpq doesn't cope very gracefully if the backend
> exits when not in the middle of a query (ie, because the postmaster told
> it to quit after some other BE crashed).  The behavior in psql, for
> example, is that the next time you issue a query, psql just exits
> without printing anything at all.  This is Not Friendly, especially
> considering that the BE sent a nice little notice message before it quit.
> 
> The main problem is that if the next thing you do is to send a new query,
> send() sees that the connection has been closed and generates a SIGPIPE
> signal.  By default that terminates the frontend process.
> 
> We could cure this by having libpq disable SIGPIPE, but we would have
> to disable it before each send() and re-enable afterwards to avoid
> affecting the behavior of the rest of the frontend application.
> Two additional kernel calls per query sounds like a lot of overhead.
> (We do actually do this when trying to close the connection, but not
> during normal queries.)
> 
> Perhaps a better answer is to have PQsendQuery check for fresh input
> from the backend before trying to send the query.  This would have two
> side effects:
>   1. If a NOTICE message has arrived, we could print it.
>   2. If EOF is detected, we will reset the connection state to
>      CONNECTION_BAD, which PQsendQuery can use to avoid trying to send.
> 
> The minimum cost to do this is one kernel call (a select(), which
> unfortunately is probably a fairly expensive call) in the normal
> case where no new input has arrived.  Another objection is that it's
> not 100% bulletproof --- if the backend closes the connection in the
> window between select() and send() then you can still get SIGPIPE'd.
> The odds of this seem pretty small however.
> 
> I'm inclined to go with answer #2, because it seems to have less
> of a performance impact, and it will ensure that the backend's polite
> "The Postmaster has informed me that some other backend died abnormally
> and possibly corrupted shared memory." message gets displayed.  With
> approach #1 we'd still have to go through some pushups to get the
> notice to come out.
> 
> Does anyone have an objection, or a better idea?
> 
> 			regards, tom lane
> 

Not really.

I've noticed this kind of problem where the backend will fault in some way, 
and after it does so, the library gets "confused".

We have a couple of processes here that are NEVER supposed to exit.  They
open a connection for each transaction, and close it at the end.  If
something happens to the backend where it dies abnormally, these processes
will sometimes get into an odd state in the libpq library where all new
connection attempts fail immediately.

I've yet to find a foolproof coding way around this particular problem.

--
-- 
Karl Denninger (karl@MCS.Net)| MCSNet - Serving Chicagoland and Wisconsin
http://www.mcs.net/          | T1's from $600 monthly / All Lines K56Flex/DOV
			     | NEW! Corporate ISDN Prices dropped by up to 50%!
Voice: [+1 312 803-MCS1 x219]| EXCLUSIVE NEW FEATURE ON ALL PERSONAL ACCOUNTS
Fax:   [+1 312 803-4929]     | *SPAMBLOCK* Technology now included at no cost

Tue, 28 Jul 1998 12:44:59 -0500
Karl Denninger  <karl@mcs.net>
Re: [INTERFACES] Coping with backend crash in libpq
> I've just noticed that libpq doesn't cope very gracefully if the backend
> exits when not in the middle of a query (ie, because the postmaster told
> it to quit after some other BE crashed).  The behavior in psql, for
> example, is that the next time you issue a query, psql just exits
> without printing anything at all.  This is Not Friendly, especially
> considering that the BE sent a nice little notice message before it quit.

I say, install the signal handler for SIGPIPE on connection startup, but
when you install it, it returns the previous defined action.  If we find
there was a previous defined action, we can re-install theirs, and let
it handle the sigpipe.  If an application later defines it's own
sigpipe, over-riding ours, then they get no error message.

However, I see psql setting the SIGPIPE handler all over the place, so I
don't think that will work there.  How about SIGURG?  Oops, not portable
for unix domain sockets.  Can we send a signal to the process, telling
it the backend has exited.  We have that information now, so why not use
it.  Define a signal handler for SIGURG or SIGUSR1, and have that print
out a message.  If the app redefines that, it will get confused when we
send the signal from the postmaster.  Oops, we can't send signals to the
client because they may be owned by other users.

I am stumped.  Let me think about it.



-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

Wed, 29 Jul 1998 00:59:32 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Coping with backend crash in libpq
> > I've just noticed that libpq doesn't cope very gracefully if the backend
> > exits when not in the middle of a query (ie, because the postmaster told
> > it to quit after some other BE crashed).  The behavior in psql, for
> > example, is that the next time you issue a query, psql just exits
> > without printing anything at all.  This is Not Friendly, especially
> > considering that the BE sent a nice little notice message before it quit.
> 
> I say, install the signal handler for SIGPIPE on connection startup, but
> when you install it, it returns the previous defined action.  If we find
> there was a previous defined action, we can re-install theirs, and let
> it handle the sigpipe.  If an application later defines it's own
> sigpipe, over-riding ours, then they get no error message.
> 
> However, I see psql setting the SIGPIPE handler all over the place, so I
> don't think that will work there.  How about SIGURG?  Oops, not portable
> for unix domain sockets.  Can we send a signal to the process, telling
> it the backend has exited.  We have that information now, so why not use
> it.  Define a signal handler for SIGURG or SIGUSR1, and have that print
> out a message.  If the app redefines that, it will get confused when we
> send the signal from the postmaster.  Oops, we can't send signals to the
> client because they may be owned by other users.
> 
> I am stumped.  Let me think about it.

Hmmm, perhaps fix psql so that it uses SIGPIPE more sensibly. SIGPIPE really
is the right signal to catch here. 

-dg

David Gould            dg@illustra.com           510.628.3783 or 510.305.9468 
Informix Software  (No, really)         300 Lakeside Drive  Oakland, CA 94612
 - If simplicity worked, the world would be overrun with insects. -

Tue, 28 Jul 1998 22:16:02 -0700 (PDT)
dg@illustra.com (David Gould)
Re: [HACKERS] Coping with backend crash in libpq
>> I say, install the signal handler for SIGPIPE on connection startup, but
>> when you install it, it returns the previous defined action.  If we find
>> there was a previous defined action, we can re-install theirs, and let
>> it handle the sigpipe.  If an application later defines it's own
>> sigpipe, over-riding ours, then they get no error message.

This makes our correct functioning dependent on the application's
SIGPIPE handler, which doesn't strike me as a good solution.
Another problem is that if we leave a SIGPIPE handler in place,
it will get called for SIGPIPEs on *other* pipes that the surrounding
application may have open, and we have no way to know what the right
response is.  (AFAIK a SIGPIPE handler can't even portably tell which
connection has SIGPIPEd.)

>> Can we send a signal to the process, telling
>> it the backend has exited.

No.  The client isn't necessarily even on the same machine as the
postmaster/backend.  Even if it were, I don't think we can take over
a signal code that the frontend application might be using for something
else.

> Hmmm, perhaps fix psql so that it uses SIGPIPE more sensibly. SIGPIPE really
> is the right signal to catch here. 

Well, psql is also using SIGPIPE sensibly: it's trying to prevent a
hangup when sending data down a pipe to a subprocess that might
terminate early.  The real problem here is that SIGPIPE is designed
wrong.  It ought to be possible to enable/disable SIGPIPE on a per-
file-handle basis ... but AFAIK that's not possible, and it's certainly
not portable even if some Unixes support it.


I'm still in favor of the check-for-input-just-before-send solution.
That does leave a small window where we can fail, but really the failure
should be pretty improbable: you have to assume that some other backend
coredumps while yours is idle, and in a window of microseconds right
before you are going to send a new command to your backend.  I think the
mess-with-catching-SIGPIPE approach is actually more likely to have
problems in practice.  It could interfere with normal functioning of
the frontend app, whereas any possible failure of the other way requires
a previous failure in some backend.  Production backends shouldn't
coredump too darn often, one hopes.

			regards, tom lane

Wed, 29 Jul 1998 11:02:26 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Coping with backend crash in libpq 