If we did this the straightforward way (exchange authentication packets
after fork()) then rogue clients could connect, start a backend, twiddle
thumbs, never finish the authentication exchange, meanwhile having filled
up the limit on the number of connections.  Somehow the backends would
have to report back to the postmaster that the authentication passed.
But then an attacker could easily fill up the system's process table with
this approach.  If you in turn put a cap on that to save your system at
large, you're back to having DoS'ed your database server.

Then you would have to put a timeout on the completion of the
authentication sequence.  This would be a fairly tricky thing to configure
given the various choices of ways to authenticate, including interactive
ones.

ISTM that there is some merit in having authentication happen *before*
doing much else, especially allocating per-connection resources.

Comments?

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Thu, 14 Jun 2001 18:31:54 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Doing authentication in backend
Peter Eisentraut <peter_e@gmx.net> writes:
> If we did this the straightforward way (exchange authentication packets
> after fork()) then rogue clients could connect, start a backend, twiddle
> thumbs, never finish the authentication exchange, meanwhile having filled
> up the limit on the number of connections.

True, but don't fool yourself that a similar DOS attack is not possible
now.  The resource limit that an attacker can hit now is the maximum
number of open file descriptors for the single postmaster process, which
may be quite a lot lower than the maximum number of process table
entries, depending on how your system is configured.

Also note that we could easily fix things so that the max-number-of-
backends limit is not checked until we have passed the authentication
procedure.  A PM child that's still busy authenticating doesn't have
to count.


> ISTM that there is some merit in having authentication happen *before*
> doing much else, especially allocating per-connection resources.

Sure, which is why the postmaster is written the way it is.  But you
have to be willing to code the postmaster in a way that prevents it from
blocking on behalf of one client.  We don't have that now for IDENT,
we are about to not have it for PAM, and I don't see a lot of enthusiasm
out there for adhering to those coding rules with the rigidity needed to
realize the theoretical benefit.

Another problem with the present setup is total cost of servicing each
connection request.  We've seen several complaints about connection-
refused problems under heavy load, occurring because the single
postmaster process simply can't service the requests quickly enough to
keep its accept() queue from overflowing.

Forking the postmaster (without an exec) is a relatively cheap
operation, since the PM has only a small amount of writable data and
very few open files.  I believe forking before authenticating would
improve the accept-queue-overflow problem by reducing the amount of
work done before the PM can accept() another connection request.

Moreover, if we went over to fork-before-authenticate, we could rip out
all the poor man's multitasking code that's in the postmaster.  That
would make the PM simpler, more understandable, and ultimately more
reliable.

So on the whole I think changing would be a win.

			regards, tom lane

Thu, 14 Jun 2001 13:42:26 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Doing authentication in backend 
Tom Lane writes:

> Also note that we could easily fix things so that the max-number-of-
> backends limit is not checked until we have passed the authentication
> procedure.  A PM child that's still busy authenticating doesn't have
> to count.

How does the postmaster know?  And if the postmaster does get to know,
what does it do with children it has "accidentally" allowed in excess?

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Thu, 14 Jun 2001 22:34:08 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Doing authentication in backend 
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> Also note that we could easily fix things so that the max-number-of-
>> backends limit is not checked until we have passed the authentication
>> procedure.  A PM child that's still busy authenticating doesn't have
>> to count.

> How does the postmaster know?  And if the postmaster does get to know,
> what does it do with children it has "accidentally" allowed in excess?

The postmaster doesn't have to know, nor should it be in the business of
enforcing the MaxBackends limit.  It should just spawn off children
(though maybe we should put a limit on total children, somewhat higher
than MaxBackends, as a crude form of preventing runaway resource usage
under a DOS attack).

A spawned child will first proceed with the authorization cycle; if it
fails, it just exits.  If it succeeds, it will then try to become a
backend.  When it tries to insert an entry into the PROC array, if
there's not an available slot then you lose (send "too many clients"
failure message to client, and exit).  We might have to reorganize the
code a little bit so that this exit happens cleanly before anything else
is done to shared memory, but it's surely doable.

The PM itself has no real need to distinguish children that are active
backends from those that are still doing authentication, AFAICS.

			regards, tom lane

Thu, 14 Jun 2001 16:49:35 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Doing authentication in backend 
On Thu, Jun 14, 2001 at 01:42:26PM -0400, Tom Lane wrote:
> Also note that we could easily fix things so that the max-number-of-
> backends limit is not checked until we have passed the authentication
> procedure.  A PM child that's still busy authenticating doesn't have
> to count.

And impose a very short timeout on authentication.

> Another problem with the present setup is total cost of servicing each
> connection request.  We've seen several complaints about connection-
> refused problems under heavy load, occurring because the single
> postmaster process simply can't service the requests quickly enough to
> keep its accept() queue from overflowing.

This last could also be addressed (along with Solaris's Unix Sockets 
problem!) by changing the second argument to listen(2) from the current 
SOMAXCONN -- which is 5 in Solaris 2.7 -- to 127.  See the six-page
discussion in Stevens UNPv1 beginning at page 93.

This is not to say we shouldn't fork before authentication, for
the above and other reasons, but the fix to listen(2)'s argument 
should happen anyway.

Nathan Myers
ncm@zembu.com

Fri, 15 Jun 2001 16:51:36 -0700
ncm@zembu.com (Nathan Myers)
Re: Doing authentication in backend
ncm@zembu.com (Nathan Myers) writes:
> On Thu, Jun 14, 2001 at 01:42:26PM -0400, Tom Lane wrote:
>> Also note that we could easily fix things so that the max-number-of-
>> backends limit is not checked until we have passed the authentication
>> procedure.  A PM child that's still busy authenticating doesn't have
>> to count.

> And impose a very short timeout on authentication.

Yes.  There's no time limit at present, but it will be easy to add one
after we change to fork-before-authenticate (since each PM child can
have its own itimer).

> This last could also be addressed (along with Solaris's Unix Sockets 
> problem!) by changing the second argument to listen(2) from the current 
> SOMAXCONN -- which is 5 in Solaris 2.7 -- to 127.  See the six-page
> discussion in Stevens UNPv1 beginning at page 93.

Unfortunately I only have Stevens' first edition, and it doesn't seem
to have any such advice in it.  Why is it a good idea to ignore the
platform's specification of SOMAXCONN?  Seems like on non-broken
platforms, that would do more harm than good.

			regards, tom lane

Sat, 16 Jun 2001 13:02:15 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Doing authentication in backend 
On Sat, Jun 16, 2001 at 01:02:15PM -0400, Tom Lane wrote:
> ncm@zembu.com (Nathan Myers) writes:
> > On Thu, Jun 14, 2001 at 01:42:26PM -0400, Tom Lane wrote:
> > This last could also be addressed (along with Solaris's Unix Sockets 
> > problem!) by changing the second argument to listen(2) from the current 
> > SOMAXCONN -- which is 5 in Solaris 2.7 -- to 127.  See the six-page
> > discussion in Stevens UNPv1 beginning at page 93.
> 
> Unfortunately I only have Stevens' first edition, and it doesn't seem
> to have any such advice in it.  Why is it a good idea to ignore the
> platform's specification of SOMAXCONN?  Seems like on non-broken
> platforms, that would do more harm than good.

The second edition is easily worth the price, for any number of reasons.

Do you want me to type in all six pages?  (I'll fax you a copy if you
ask.)  He includes graphs of sample daemon transaction rates for 
different settings of that argument, from SOMAXCONN on up, as well 
as analyses of what is going on, including diagrams.

The short description is that half-completed connections occupy 
a sort of foyer or vestibule (my terms).  The second argument to 
listen(2), usually called "backlog", is defined vaguely in Posix 
and therefore has various meanings on different systems, but is 
supposed to limit how many half-open connections are allowed to 
wait there.  

I don't have the book at home, but IIRC, Solarix's interpretation 
is unusually strict, which causes it to reject connections much 
more aggressively for a given value.  Independently of that, the 
low value that was originally suggested just turned out to be a 
bad guess.  The mistake got worse as longer, fatter pipes got 
deployed and protocols that did more opens got popular.

Posix says that a backlog value more than the platform's maximum
gets folded, so there is no danger in exceeding it.  On Solaris 2.7, 
SOMAXCONN is still 5 in the headers, but larger values are both legal 
and effective.  A parameter might reasonably be added to the config 
file to crank the backlog value down on installations where kernel
memory is scarce and other daemons must compete with PG for connection 
resources.

I have seen reports that it is this strict interpretation, and the
too-low backlog passed to listen(2), that accounts for reports of
Unix sockets working poorly on Solaris.  (Certainly it is hard to 
believe that Sun could not make their Unix sockets work right!)

Nathan Myers
ncm@zembu.com

Sun, 17 Jun 2001 02:58:09 -0700
ncm@zembu.com (Nathan Myers)
Re: Doing authentication in backend