I have spent several days now puzzling over the corrupted WAL logfile
that Scott Parish was kind enough to send me from a 7.1beta4 crash.
It looks a lot like two different series of transactions were getting
written into the same logfile.  I'd been digging like mad in the WAL
code to try to explain this as a buffer-management logic error, but
after a fresh exchange of info it turns out that I was barking up the
wrong tree.  There *were* two different series of transactions.
Specifically, here's what happened:

1. Scott (or actually his associate) shut down and restarted the
postmaster using the /etc/rc.d/init.d/pgsql script that ships with
our RPMs.  That script shuts down the old postmaster with
	killproc postmaster
It turns out that at least on Scott's machine (RedHat 6.1), the default
kill level for the killproc function is kill -9.  (This is clearly a bad
bug in the init script, but I digress.)

2. So, the old postmaster was killed with kill -9, but its child
backends were still running.  The new postmaster will start up
successfully because it'll think the old postmaster crashed, and
so it will go through the usual recovery procedure.

3. Now we have two sets of backends running in different shmem blocks
(7.0 might have choked on that part, but 7.1 doesn't care) and running
different sets of transactions.  But they're writing to the same WAL
log.  Result: guaranteed corruption of the log.

It actually took two iterations of this to expose the bug: the third
attempted postmaster start went looking for the checkpoint record last
written by the second one, which meanwhile had got overwritten by
activity of the first backend set.


Now, killing the postmaster -9 and not cleaning up the backends has
always been a good way to shoot yourself in the foot, but up to now the
worst thing that was likely to happen to you was isolated corruption in
specific tables.  In the brave new world of WAL the stakes are higher,
because the system will refuse to start up if it finds a corrupted
checkpoint record.  Clueless admins who resort to kill -9 as a routine
admin tool *will* lose their databases.  Moreover, the init scripts
that are running around now are dangerous weapons if used with 7.1.

I think we need a stronger interlock to prevent this scenario, but I'm
unsure what it should be.  Ideas?

			regards, tom lane

Mon, 05 Mar 2001 17:30:26 -0500
Tom Lane <tgl@sss.pgh.pa.us>
How to shoot yourself in the foot: kill -9 postmaster
At 3/5/2001 04:30 PM, you wrote:
>Now, killing the postmaster -9 and not cleaning up the backends has
>always been a good way to shoot yourself in the foot, but up to now the
>worst thing that was likely to happen to you was isolated corruption in
>specific tables.  In the brave new world of WAL the stakes are higher,
>because the system will refuse to start up if it finds a corrupted
>checkpoint record.  Clueless admins who resort to kill -9 as a routine
>admin tool *will* lose their databases.  Moreover, the init scripts
>that are running around now are dangerous weapons if used with 7.1.
>
>I think we need a stronger interlock to prevent this scenario, but I'm
>unsure what it should be.  Ideas?

Is there anyway to see if the other processes (child) have a lock on the 
log file?

On a lot of systems, when a daemon starts, will record the PID in a file so 
it/'the admin' can do a 'shutdown' script with the PID listed.
Can child processes list themselves like child.PID in a configurable 
directory, and have the starting process look for all of these and shut the 
"orphaned" child processes down?

Just thoughts...

Thomas


Mon, 05 Mar 2001 17:19:34 -0600
Thomas Swan <tswan-lst@ics.olemiss.edu>
Re: How to shoot yourself in the foot: kill -9 postmaster
* Tom Lane <tgl@sss.pgh.pa.us> [010305 14:51] wrote:
> 
> I think we need a stronger interlock to prevent this scenario, but I'm
> unsure what it should be.  Ideas?

Re having multiple postmasters active by accident.

The sysV IPC stuff has some hooks in it that may help you.

One idea is to check the 'struct shmid_ds' feild 'shm_nattch',
basically at startup if it's not 1 (or 0) then you have more than
one postgresql instance messing with it and it should not proceed.

I'd also suggest looking into using sysV semaphores and the semundo
stuff, afaik it can be used to track the number of consumers of
a reasource.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Mon, 5 Mar 2001 15:47:51 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
Tom Lane wrote:
> checkpoint record.  Clueless admins who resort to kill -9 as a routine
> admin tool *will* lose their databases.  Moreover, the init scripts
> that are running around now are dangerous weapons if used with 7.1.

Thanks for the headsup, Tom.  Time to nix killproc and do something
cleaner -- compatible, but cleaner.  I'll have to research what the
defaults are for later RH's -- but, as 6.1 is one of my target platforms
at this time, I have to fix that issue for sure.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 20:46:41 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
Lamar Owen <lamar.owen@wgcr.org> writes:
> Thanks for the headsup, Tom.  Time to nix killproc and do something
> cleaner -- compatible, but cleaner.

As far as I could tell from the 6.1 scripts, it would work to do

	killproc postmaster -TERM

The problem is just that killproc has an overenthusiastic default...

			regards, tom lane

Mon, 05 Mar 2001 20:49:01 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
killproc should send a kill -15 to the process, wait a few seconds for
it to exit.  If it does not, try kill -1, and if that doesn't kill it,
then kill -9.

> Tom Lane wrote:
> > checkpoint record.  Clueless admins who resort to kill -9 as a routine
> > admin tool *will* lose their databases.  Moreover, the init scripts
> > that are running around now are dangerous weapons if used with 7.1.
> 
> Thanks for the headsup, Tom.  Time to nix killproc and do something
> cleaner -- compatible, but cleaner.  I'll have to research what the
> defaults are for later RH's -- but, as 6.1 is one of my target platforms
> at this time, I have to fix that issue for sure.
> --
> Lamar Owen
> WGCR Internet Radio
> 1 Peter 4:11
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 5 Mar 2001 20:52:13 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > Thanks for the headsup, Tom.  Time to nix killproc and do something
> > cleaner -- compatible, but cleaner.
> 
> As far as I could tell from the 6.1 scripts, it would work to do
> 
> 	killproc postmaster -TERM
> 

Yes, amazing it has a -9 default.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 5 Mar 2001 20:52:40 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> killproc should send a kill -15 to the process, wait a few seconds for
> it to exit.  If it does not, try kill -1, and if that doesn't kill it,
> then kill -9.

Tell it to the Linux people ... this is their boot-script code we're
talking about.

			regards, tom lane

Mon, 05 Mar 2001 20:55:41 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Tom Lane wrote:
> 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > killproc should send a kill -15 to the process, wait a few seconds for
> > it to exit.  If it does not, try kill -1, and if that doesn't kill it,
> > then kill -9.
> 
> Tell it to the Linux people ... this is their boot-script code we're
> talking about.

RedHat, in particular.  I can't vouch for any others.

On my RH 6.2 box, with initscripts-5.00-1 loaded, here's what killproc
does if no killlevel is set (even though a default $killlevel is set to
-9, it's not used in this code):
($pid is the pid of the proc to kill, $base is the name of the proc,
etc)

   if [ "$notset" = "1" ] ; then
      if ps h $pid>/dev/null 2>&1; then
         # TERM first, then KILL if not dead
         kill -TERM $pid
         usleep 100000
         if ps h $pid >/dev/null 2>&1 ; then
            sleep 1
            if ps h $pid >/dev/null 2>&1 ; then
               sleep 3
               if ps h $pid >/dev/null 2>&1 ; then
                  kill -KILL $pid
               fi
            fi
         fi
      fi
      ps h $pid >/dev/null 2>&1
      RC=$?
      [ $RC -eq 0 ] && failure "$base shutdown" || success "$base
shutdown"
      RC=$((! $RC))
      # use specified level only
  else
      if ps h $pid >/dev/null 2>&1; then
         kill $killlevel $pid
         RC=$?
         [ $RC -eq 0 ] && success "$base $killlevel" || failure "$base
$killlevel"
      fi
  fi


Is 6.1 this different from 6.2?  This code on the surface seems
reasonable to me -- am I missing something?  The 6.2 code (found in
/etc/rc.d/init.d/functions, for those who might not know where to find
killproc) sets a default killlevel but never uses it -- ignorant but not
stupid.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 21:11:38 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
>    if [ "$notset" = "1" ] ; then
>       if ps h $pid>/dev/null 2>&1; then
>          # TERM first, then KILL if not dead
>          kill -TERM $pid
>          usleep 100000
>          if ps h $pid >/dev/null 2>&1 ; then
>             sleep 1
>             if ps h $pid >/dev/null 2>&1 ; then
>                sleep 3
>                if ps h $pid >/dev/null 2>&1 ; then
>                   kill -KILL $pid
>                fi
>             fi
>          fi
>       fi

Yes, this seems like the proper way to do it.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 5 Mar 2001 21:14:16 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster
On Mon, Mar 05, 2001 at 08:55:41PM -0500, Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > killproc should send a kill -15 to the process, wait a few seconds for
> > it to exit.  If it does not, try kill -1, and if that doesn't kill it,
> > then kill -9.
> 
> Tell it to the Linux people ... this is their boot-script code we're
> talking about.

Not to be a zealot, but this isn't _Linux_ boot-script code, it's
_Red Hat_ boot-script code.  Red Hat would like for us all to confuse
the two, but they jes' ain't the same.  (As a rule of thumb, where it
works right, credit Linux; where it doesn't, blame Red Hat. :-)

Nathan Myers
ncm@zembu.com

Mon, 5 Mar 2001 18:19:25 -0800
ncm@zembu.com (Nathan Myers)
Re: How to shoot yourself in the foot: kill -9 postmaster
Tom Lane wrote:
> 
> Now, killing the postmaster -9 and not cleaning up the backends has
> always been a good way to shoot yourself in the foot, but up to now the
> worst thing that was likely to happen to you was isolated corruption in
> specific tables.  In the brave new world of WAL the stakes are higher,
> because the system will refuse to start up if it finds a corrupted
> checkpoint record.  Clueless admins who resort to kill -9 as a routine
> admin tool *will* lose their databases.  Moreover, the init scripts
> that are running around now are dangerous weapons if used with 7.1.
> 
> I think we need a stronger interlock to prevent this scenario, but I'm
> unsure what it should be.  Ideas?
> 

Seems the simplest way is to inhibit starting postmaster
if the pid file exists.
Another way is to use flock() if flock() is available.
We could flock() the pid file so that another postmaster
could detect the lock of the file.

Regards,
Hiroshi Inoue

Tue, 06 Mar 2001 11:19:33 +0900
Hiroshi Inoue <Inoue@tpf.co.jp>
Re: How to shoot yourself in the foot: kill -9 postmaster
Bruce Momjian wrote:
> >          # TERM first, then KILL if not dead
> Yes, this seems like the proper way to do it.

Now to verify that 6.1 is the same....or different.... Hmmmm.... The
mirrors of ftp.redhat.com (and, in fact, RedHat.com itself) no longer
have the updates or the original for 6.1's initscripts-4.70 package. 
Can a RedHat 6.1 user (using as close as possible to 6.1's release
initscripts package) send me a copy of /etc/rc.d/init.d/functions, or
verify how that initscripts package defines killproc?  I cannot at this
moment locate my RH 6.1 SRPMS CD.  Found my RH _4_.1 CD, but that's just
a _little_ old :-).
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 21:23:55 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
Hiroshi Inoue <Inoue@tpf.co.jp> writes:
> Tom Lane wrote:
>> I think we need a stronger interlock to prevent this scenario, but I'm
>> unsure what it should be.  Ideas?

> Seems the simplest way is to inhibit starting postmaster
> if the pid file exists.

Then we're unable to recover from a crash without manual intervention.

The tricky part of this is not to give up the ability to restart when
there *has* been a crash.

> Another way is to use flock() if flock() is available.
> We could flock() the pid file so that another postmaster
> could detect the lock of the file.

This would only work if every backend is holding flock on the file,
which would mean they'd all have to keep it open all the time.  Kind
of annoying to use up that many file descriptors on it.  Might be the
best answer though; I haven't thought of anything I like better...

			regards, tom lane

Mon, 05 Mar 2001 21:28:20 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Nathan Myers wrote:
> Not to be a zealot, but this isn't _Linux_ boot-script code, it's
> _Red Hat_ boot-script code.  Red Hat would like for us all to confuse
> the two, but they jes' ain't the same.  (As a rule of thumb, where it
> works right, credit Linux; where it doesn't, blame Red Hat. :-)

So we're going to credit Linux for PostgreSQL being shipped as part of
the RedHat distribution since RH 5.0, then? :-0
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 21:33:19 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
Lamar Owen <lamar.owen@wgcr.org> writes:
> Is 6.1 this different from 6.2?

Scott sent me a copy of /etc/init.d/functions from his box, and it has
largely the same behavior (I hadn't read the whole code to notice that
it doesn't use the default killlevel...).  What's actually happening
here is that the init script sends SIGTERM, and then SIGKILL four
seconds later if the postmaster hasn't shut down yet.  Unfortunately,
unless your clients are very short-lived four seconds isn't going to
be enough for a "polite" shutdown.  (It's pretty marginal even for
an impolite one, since a checkpoint will take at least a couple of
seconds.)

However, with an explicit kill level that doesn't happen: you get one
signal of the specified value, no more.  Possibly it would be better for
the init script to send SIGINT (forcibly disconnect clients) instead of
SIGTERM, however.  So I'm now leaning to "killproc postmaster -INT".

			regards, tom lane

Mon, 05 Mar 2001 21:36:28 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Tom Lane wrote:
> The tricky part of this is not to give up the ability to restart when
> there *has* been a crash.

But kill -9 effectively _is_ an admin-initiated crash.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 21:36:56 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
Lamar Owen <lamar.owen@wgcr.org> writes:
> Tom Lane wrote:
>> The tricky part of this is not to give up the ability to restart when
>> there *has* been a crash.

> But kill -9 effectively _is_ an admin-initiated crash.

Yeah, but only a partial crash.  If the admin finishes the job by
killing the backends too, we're fine.  Postmaster down, backends alive
is not a scenario we're currently prepared for.  We need a way to plug
that gap.

			regards, tom lane

Mon, 05 Mar 2001 21:40:41 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Tom Lane wrote:
> However, with an explicit kill level that doesn't happen: you get one
> signal of the specified value, no more.  Possibly it would be better for
> the init script to send SIGINT (forcibly disconnect clients) instead of
> SIGTERM, however.  So I'm now leaning to "killproc postmaster -INT".

Ok, since I can't seem to count on killproc's exact behavior, istm that
I can:
killproc postmaster -INT
wait some number of seconds
if postmaster still up
   killproc postmaster -TERM
wait some number of seconds
if postmaster STILL up
   killproc postmaster  #and let the grim reaper do its dirty work.

After all, the system shutdown is relying on this script to properly and
thoroughly shut things down, or it WILL do the 'kill -9
pid-of-postmaster' for you.

Now, what's a good delay here?  Or is there a better metric that a
simple delay?  After all, I want to avoid the kill -9 unless we have an
emergency hard lock situation -- what's a good indicator of the backend
fleet of processes actually _doing_ something?  Or should I key on an
indicator of processor speed (Linux does provide a nice bogus metric
known as BogoMIPS for such a purpose)?  The last thing I want to do is
wait too long on some platforms and not long enough on others.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 21:45:08 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
Lamar Owen <lamar.owen@wgcr.org> writes:
> The last thing I want to do is
> wait too long on some platforms and not long enough on others.

The difficulty is to know how long the final checkpoint will take.
This depends on (at least) your hard disk speed and the number of
dirty buffers, so I think you're going to have some difficulty
estimating it with any reliability.  BogoMIPS won't help, for sure.

However, if you do SIGINT and then wait a few seconds, you can be fairly
sure that all the extant backends are dead (if not frozen up...) and
that the checkpoint is in progress.  That may be about the best you can
do.

I do not agree that this script should take it on itself to kill -9 the
postmaster.  Please note that the reason we're having this discussion at
all is that the init script may be used for purposes other than system
shutdown.  So the argument that "it's going to happen anyway" is wrong.

			regards, tom lane

Mon, 05 Mar 2001 21:53:05 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Tom Lane wrote:
> Yeah, but only a partial crash.  If the admin finishes the job by
> killing the backends too, we're fine.  Postmaster down, backends alive
> is not a scenario we're currently prepared for.  We need a way to plug
> that gap.

Postmaster can easily enough find out if zombie backends are 'out there'
during startup, right?  What can postmaster _do_ about it, though?  It
won't necessarily be able to kill them -- but it also can't control
them.  If it _can_ kill them, should it try?

After all, if those zombies are out there on this PGDATA there's going
to be big trouble if we even try to start.  If we can't kill the zombies
(that might still be doing something useful with their clients) from our
starting postmaster, how can we possibly start up underneath running
backends?

Should the backend look for the presence of its parent postmaster
periodically and gracefully come down if postmaster goes away without
the proper handshake?  A watchdog semaphore (or shared memory flag) that
the backend resets and then checks periodically for it being set by its
parent postmaster?

Should a set of backends detect a new postmaster coming up and try to
'sync up' with that postmaster, like the baroque GEMM handshake dance
performed by 386 memory managers when Windows needs to start its own
VMM?

Or should we spend that much time protecting Barney Fife's from their
own single bullet? :-)

Just a nor-easter of a brainstorm....
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 21:55:11 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
> Ok, since I can't seem to count on killproc's exact behavior, istm that
> I can:
> killproc postmaster -INT
> wait some number of seconds
> if postmaster still up
>    killproc postmaster -TERM
> wait some number of seconds
> if postmaster STILL up
>    killproc postmaster  #and let the grim reaper do its dirty work.
> 
> After all, the system shutdown is relying on this script to properly and
> thoroughly shut things down, or it WILL do the 'kill -9
> pid-of-postmaster' for you.
> 
> Now, what's a good delay here?  Or is there a better metric that a
> simple delay?  After all, I want to avoid the kill -9 unless we have an
> emergency hard lock situation -- what's a good indicator of the backend
> fleet of processes actually _doing_ something?  Or should I key on an
> indicator of processor speed (Linux does provide a nice bogus metric
> known as BogoMIPS for such a purpose)?  The last thing I want to do is
> wait too long on some platforms and not long enough on others.

In remembering how other databases handle it, I think you should use
pg_ctl to shut it down.  You need to enable wait mode, not sure if that
is the default or not.  That will wait for it to shut down before
continuing.  I realize a hung shutdown would stop the kernel from
shutting down.  You could put a sleep 100 in there and call a trap on a
timeout.

Here is some shell code:

	TIME=60	
	pg_ctl -w stop &
	BG="$!"; export BG

	(sleep "$TIME"; kill "$BG" ) &
	BG2="$!"; export BG2

	wait "$BG"
	if ! kill -0 "$BG2"
	else    kill "$BG2"
	fi


This will try a pg_ctl shutdown for 60 seconds, then kill pg_ctl.  You
would then need a kill of you own.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 5 Mar 2001 21:57:56 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster
Tom Lane wrote:
> Please note that the reason we're having this discussion at
> all is that the init script may be used for purposes other than system
> shutdown.  So the argument that "it's going to happen anyway" is wrong.

Believe it or not, you just disproved your own statement that the
initscript should not take it upon itself to issue the kill -9.  So,
what if I issue '/etc/rc.d/init.d/postgresql restart' -- and backends
don't go away during the 'stop' phase, while postmaster may actually
have died?  Or is it even possible for postmaster to drop out with a
running backend out there?

No, more is needed.  But I think a careful reap through the running
backends to kill those that need killing if postmaster won't go down
might be prudent.  Currently it is not possible to run multiple
postmasters with the RPM install (I am working on that little problem,
but it won't be for 7.1's RPMset yet), so all backends that are running
on the RPM PGDATA location (which I am looking at making configurable as
well) will belong to the one postmaster.  Of course, that would be an
absolute last resort.

Oh well -- the real solution is elsewhere, anyway.  I just have to make
sure it is not data-corruption broken.  And, if leaving the -9 out
completely is the only solution, then, well, it's the only solution.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 22:03:49 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
Lamar Owen <lamar.owen@wgcr.org> writes:
> Tom Lane wrote:
>> Postmaster down, backends alive is not a scenario we're currently
>> prepared for.  We need a way to plug that gap.

> Postmaster can easily enough find out if zombie backends are 'out there'
> during startup, right?

If you think it's easy enough, enlighten the rest of us ;-).  Be sure
your solution only finds leftover backends from the previous instance of
the same postmaster, else it will prevent running multiple postmasters
on one system.

> What can postmaster _do_ about it, though?  It
> won't necessarily be able to kill them -- but it also can't control
> them.  If it _can_ kill them, should it try?

I think refusal to start is sufficient.  They should go away by
themselves as their clients disconnect, and forcing the issue doesn't
seem like it will improve matters.  The admin can kill them (hopefully
with just a SIGTERM ;-)) if he wants to move things along ... but I'd
not like to see a newly-starting postmaster do that automatically.

> Should the backend look for the presence of its parent postmaster
> periodically and gracefully come down if postmaster goes away without
> the proper handshake?

Unless we checked just before every disk write, this wouldn't represent
a safe failure mode.  The onus has to be on the newly-starting
postmaster, I think, not on the old backends.

> Should a set of backends detect a new postmaster coming up and try to
> 'sync up' with that postmaster,

Nice try ;-).  How will you persuade the kernel that these processes are
now children of the new postmaster?

			regards, tom lane

Mon, 05 Mar 2001 22:04:47 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Bruce Momjian wrote:
> This will try a pg_ctl shutdown for 60 seconds, then kill pg_ctl.  You
> would then need a kill of you own.

I missed something somehwere: wasn't the consensus a few weeks ago that
pg_ctl shouldn't be used for a system initscript?  Or did I black out
that day? :-)  I certainly have no problem using pg_ctl for this purpose
-- as I have been using pg_ctl to start postmaster all along (then why
am I not using it to stop -- don't answer that :-))......
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 22:06:41 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
> Bruce Momjian wrote:
> > This will try a pg_ctl shutdown for 60 seconds, then kill pg_ctl.  You
> > would then need a kill of you own.
> 
> I missed something somehwere: wasn't the consensus a few weeks ago that
> pg_ctl shouldn't be used for a system initscript?  Or did I black out
> that day? :-)  I certainly have no problem using pg_ctl for this purpose
> -- as I have been using pg_ctl to start postmaster all along (then why
> am I not using it to stop -- don't answer that :-))......

I don't remember that discussion.  My guess was that you didn't want
pg_ctl to hang forever.  My script handles that, I think.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 5 Mar 2001 22:08:26 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster
Lamar Owen <lamar.owen@wgcr.org> writes:
> Tom Lane wrote:
>> Please note that the reason we're having this discussion at
>> all is that the init script may be used for purposes other than system
>> shutdown.  So the argument that "it's going to happen anyway" is wrong.

> Believe it or not, you just disproved your own statement that the
> initscript should not take it upon itself to issue the kill -9.

How?

> So, what if I issue '/etc/rc.d/init.d/postgresql restart' -- and
> backends don't go away during the 'stop' phase, while postmaster may
> actually have died?  Or is it even possible for postmaster to drop out
> with a running backend out there?

The postmaster will certainly not do so voluntarily.  If you kill -9 it,
of course, that's the situation you're left with ... but your reasoning
seems circular to me.  "I should kill -9 the postmaster to prevent the
situation where I've kill -9'd the postmaster."

			regards, tom lane

Mon, 05 Mar 2001 22:10:18 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Lamar Owen <lamar.owen@wgcr.org> writes:
> I missed something somehwere: wasn't the consensus a few weeks ago that
> pg_ctl shouldn't be used for a system initscript?

I thought there was some concern about whether pg_ctl is really "ready
for prime time".  But I don't recall the details either.

			regards, tom lane

Mon, 05 Mar 2001 22:11:56 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Tom Lane wrote:
> Lamar Owen wrote:
> > Postmaster can easily enough find out if zombie backends are 'out there'
> > during startup, right?
 
> If you think it's easy enough, enlighten the rest of us ;-).

If postgres reported PGDATA on the command line it would be easy enough.

> > What can postmaster _do_ about it, though?  It
> > won't necessarily be able to kill them -- but it also can't control
> > them.  If it _can_ kill them, should it try?
 
> I think refusal to start is sufficient.  They should go away by
> themselves as their clients disconnect, and forcing the issue doesn't

???? I have misunderstood your previous statement about not wanting to
force a manual crash recovery, then.

> > Should a set of backends detect a new postmaster coming up and try to
> > 'sync up' with that postmaster,
 
> Nice try ;-).  How will you persuade the kernel that these processes are
> now children of the new postmaster?

Yeah, that's the kicker.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 22:12:00 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
Lamar Owen <lamar.owen@wgcr.org> writes:
> Tom Lane wrote:
>> If you think it's easy enough, enlighten the rest of us ;-).

> If postgres reported PGDATA on the command line it would be easy enough.

In ps status you mean?  I don't think we are prepared to require ps
status functionality to let the system start up... we'd lose a number
of supported platforms that way.


>> I think refusal to start is sufficient.  They should go away by
>> themselves as their clients disconnect, and forcing the issue doesn't

> ???? I have misunderstood your previous statement about not wanting to
> force a manual crash recovery, then.

In the case of an actual crash and restart, postgres should come back up
without help.  However, the situation here is not a crash, it is
incomplete admin intervention.  I don't think that expecting the admin
to complete his intervention is the same thing as manual crash recovery.
I especially don't think that we should second-guess what the admin
wants us to do by auto-killing backends that are still serving clients.

			regards, tom lane

Mon, 05 Mar 2001 22:17:36 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Tom Lane wrote:
> of course, that's the situation you're left with ... but your reasoning
> seems circular to me.  "I should kill -9 the postmaster to prevent the
> situation where I've kill -9'd the postmaster."

Ok, while the script can certainly be used from the command line, its
primary purpose is system shutdown.

And, I am thinking kindof circituitously at this point -- I only now
realize just how circituitously.  If I keep slapping my forehead like
this, I'm going to be bald in a few years....

I don't want to reap the postmaster off -- I want to reap off the
backends associated with that particular postmaster, allowing that
postmaster to die on its own.  Duh.  Doing this in a safe manner is not
going to be easy, given that the PGDATA is not on the command line to
the backend as echoed by ps.  Although I could key on PPID for the
backends....  I'll have to experiment.  But not tonight -- last week was
more taxing than I thought. :-(.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 22:24:11 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > Tom Lane wrote:
> >> If you think it's easy enough, enlighten the rest of us ;-).
> > If postgres reported PGDATA on the command line it would be easy enough.
 
> In ps status you mean?  I don't think we are prepared to require ps
> status functionality to let the system start up... we'd lose a number
> of supported platforms that way.

That is one downside.  A major downside.  Again, alot of work to protect
the Barney Fife's out there.

> In the case of an actual crash and restart, postgres should come back up
> without help.  However, the situation here is not a crash, it is
> incomplete admin intervention.  I don't think that expecting the admin

Is it a correct assumption that this is the only time postmaster might
drop out?

But, thanks for the clarification, as I had misunderstood what you
meant.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 22:27:23 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
Lamar Owen <lamar.owen@wgcr.org> writes:
> I don't want to reap the postmaster off -- I want to reap off the
> backends associated with that particular postmaster, allowing that
> postmaster to die on its own.  Duh.  Doing this in a safe manner is not
> going to be easy, given that the PGDATA is not on the command line to
> the backend as echoed by ps.  Although I could key on PPID for the
> backends....  I'll have to experiment.

PPID should work fine, actually.  Keep in mind though that SIGINT'ing
the postmaster will already have sent a terminate signal to its children
(barring postmaster breakage), and that if you wait around for awhile
and then kill off remaining children, you may well accomplish nothing
except to kill off the checkpoint process :-(

			regards, tom lane

Mon, 05 Mar 2001 22:31:05 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Lamar Owen <lamar.owen@wgcr.org> writes:
> Is it a correct assumption that this is the only time postmaster might
> drop out?

Well, there's always the possibility of a bug leading to postmaster
coredump.  Historically those have been pretty rare though.

In any case, I'm not sure that the init script is the place to be
solving these problems.  We do need some internal mechanism to protect
against a crashed or kill -9'd postmaster.

			regards, tom lane

Mon, 05 Mar 2001 22:33:49 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Tom Lane wrote:
> Well, there's always the possibility of a bug leading to postmaster
> coredump.  Historically those have been pretty rare though.

I have never personally seen one, since 6.1.1.
 
> In any case, I'm not sure that the init script is the place to be
> solving these problems.

Well, I do kindof have the responsibility to allow the system to shut
down..... I'll have to double check -- there may be a timeout mechanism
in the RedHat init to reap off shutdown scripts -- but I haven't yet
found it.  Better to gracefully yank the plugs than have the grim reaper
yank them in the wrong order for you, in any case.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Mar 2001 22:44:36 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
* Tom Lane <tgl@sss.pgh.pa.us> [010305 19:13] wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > Tom Lane wrote:
> >> Postmaster down, backends alive is not a scenario we're currently
> >> prepared for.  We need a way to plug that gap.
> 
> > Postmaster can easily enough find out if zombie backends are 'out there'
> > during startup, right?
> 
> If you think it's easy enough, enlighten the rest of us ;-).  Be sure
> your solution only finds leftover backends from the previous instance of
> the same postmaster, else it will prevent running multiple postmasters
> on one system.

I'm sure some sort of encoding of the PGDATA directory along with
the pids stored in the shm segment...

> > What can postmaster _do_ about it, though?  It
> > won't necessarily be able to kill them -- but it also can't control
> > them.  If it _can_ kill them, should it try?
> 
> I think refusal to start is sufficient.  They should go away by
> themselves as their clients disconnect, and forcing the issue doesn't
> seem like it will improve matters.  The admin can kill them (hopefully
> with just a SIGTERM ;-)) if he wants to move things along ... but I'd
> not like to see a newly-starting postmaster do that automatically.

I agree, shooting down processes incorrectly should be left up to
vendors braindead scripts. :)

> > Should the backend look for the presence of its parent postmaster
> > periodically and gracefully come down if postmaster goes away without
> > the proper handshake?
> 
> Unless we checked just before every disk write, this wouldn't represent
> a safe failure mode.  The onus has to be on the newly-starting
> postmaster, I think, not on the old backends.
> 
> > Should a set of backends detect a new postmaster coming up and try to
> > 'sync up' with that postmaster,
> 
> Nice try ;-).  How will you persuade the kernel that these processes are
> now children of the new postmaster?

Oh, easy, use ptrace. :)

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Mon, 5 Mar 2001 21:43:13 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
> I especially don't think that we should second-guess what the admin
> wants us to do by auto-killing backends that are still serving
> clients.

  Sure. But it would be nice anyway if pg_ctl could do this with a
specific command line switch. 

-- 
<< Tout n'y est pas parfait, mais on y honore certainement les jardiniers >>

			Dominique Quatravaux <dom@kilimandjaro.dyndns.org>

Tue, 6 Mar 2001 13:30:06 +0100
dom@idealx.com
Re: How to shoot yourself in the foot: kill -9 postmaster
> Bruce Momjian writes:
> 
> > This will try a pg_ctl shutdown for 60 seconds, then kill pg_ctl.  You
> > would then need a kill of you own.
> 
> pg_ctl automatically times out after 60 seconds.

Oh, yea, that's right, I saw that in the documenation.  Forget my
script.  Just run pg_ctl first, then kill the postmaster if it is still
there.  Much safer than doing kill and checking because pg_ctl knows
when the system cleanly shuts down and exits.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 6 Mar 2001 11:54:49 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster
Bruce Momjian writes:

> This will try a pg_ctl shutdown for 60 seconds, then kill pg_ctl.  You
> would then need a kill of you own.

pg_ctl automatically times out after 60 seconds.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Tue, 6 Mar 2001 17:57:48 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
Lamar Owen writes:

> I missed something somehwere: wasn't the consensus a few weeks ago that
> pg_ctl shouldn't be used for a system initscript?

The consensus(?) was that there was some work to do in pg_ctl before it
was robust enough to be used (for anything).  That work has been done.
An example Linux init.d script is at contrib/start-scripts/linux.  The
only fault in that script that I can see is that it has no recipe for the
case when the postmaster does not come down after 60 seconds.  But this is
really no problem for the issue at hand because if you do a normal
runlevel switch then the postmaster will simply keep running, and during a
system shutdown all the backends are going to die anyway.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Tue, 6 Mar 2001 18:04:44 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
Peter Eisentraut wrote:
> 
> Lamar Owen writes:
> 
> > I missed something somehwere: wasn't the consensus a few weeks ago that
> > pg_ctl shouldn't be used for a system initscript?
> 
> The consensus(?) was that there was some work to do in pg_ctl before it
> was robust enough to be used (for anything).  That work has been done.

That was the detail I missed.

> case when the postmaster does not come down after 60 seconds.  But this is
> really no problem for the issue at hand because if you do a normal
> runlevel switch then the postmaster will simply keep running, and during a
> system shutdown all the backends are going to die anyway.

Only if each and every shutdown script succeeds in its task.  And I have
to make sure that the RPM's shipping script successfully pulls down the
system in an orderly fashion -- of course, I don't have to worry about
the case where a postmaster is going to be started back up if we are in
system shutdown -- but, as Tom also stated, I can't assume I'm in the
system's death throes when called with the stop parameter.

And it _is_ possible for an admin to set up the runlevels such that a
level is set aside where even networking isn't running (actually, that
level already exists, and is called 'single user mode') -- or a run
level for website maintenance where networking is still up, but the
webserver and postgresql (and other associated) processes are to be shut
down.  I personally use this -- I have set up runlevel 4 as a 'remote
single user mode' of sorts where I still have sshd running (and the
networking stack, obviously), but AOLserver, postgresql, and RealServer
are shut down.  I then switch runlevels back to 3 to return to normal. 
Much easier than manually stopping and restarting (in the correct order,
as AOLserver is not a happy camper if postmaster drops out from
underneath it) all the necessary pieces.

So I can't assume anything.  The default RPM installation used to
automatically configure runlevels 3, 4, and 5 (not any more), but my
script can't assume that the system is actually in that state by any
means.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 06 Mar 2001 12:55:54 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
Alfred Perlstein <bright@wintelcom.net> writes:
> I'm sure some sort of encoding of the PGDATA directory along with
> the pids stored in the shm segment...

I thought about this too, but it strikes me as not very trustworthy.
The problem is that there's no guarantee that the new postmaster will
even notice the old shmem segment: it might select a different shmem
key.  (The 7.1 coding of shmem key selection makes this more likely
than it used to be, but even under 7.0, it will certainly fail to work
if I choose to start the new postmaster using a different port number
than the old one had.  The shmem key is driven primarily by port number
not data directory ...)

The interlock has to be tightly tied to the PGDATA directory, because
what we're trying to protect is the files in and under that directory.
It seems that something based on file(s) in that directory is the way
to go.

The best idea I've seen so far is Hiroshi's idea of having all the
backends hold fcntl locks on the same file (probably postmaster.pid
would do fine).  Then the new postmaster can test whether any backends
are still alive by trying to lock the old postmaster.pid file.
Unfortunately, I read in the fcntl man page:

    Locks are not inherited by a child process in a fork(2) system call.

This makes the idea much less attractive than I originally thought:
a new backend would not automatically inherit a lock on the
postmaster.pid file from the postmaster, but would have to open/lock it
for itself.  That means there's a window where the new backend exists
but would be invisible to a hypothetical new postmaster.

We could work around this with the following, very ugly protocol:

1. Postmaster normally maintains fcntl read lock on its postmaster.pid
file.  Each spawned backend immediately opens and read-locks
postmaster.pid, too, and holds that file open until it dies.  (Thus
wasting a kernel FD per backend, which is one of the less attractive
things about this.)  If the backend is unable to obtain read lock on
postmaster.pid, then it complains and dies.  We must use read locks
here so that all these processes can hold them separately.

2. If a newly started postmaster sees a pre-existing postmaster.pid
file, it tries to obtain a *write* lock on that file.  If it fails,
conclude that an old postmaster or backend is still alive; complain
and quit.  If it succeeds, sit for say 1 second before deleting the file
and creating a new one.  (The delay here is to allow any just-started
old backends to fail to acquire read lock and quit.  A possible
objection is that we have no way to guarantee 1 second is enough, though
it ought to be plenty if the lock acquisition is just after the fork.)

One thing that worries me a little bit is that this means an fcntl
read-lock request will exist inside the kernel for each active backend.
Does anyone know of any performance problems or hard kernel limits we
might run into with large numbers of backends (lots and lots of fcntl
locks)?  At least the locks are on a file that we don't actually touch
in the normal course of business.

A small savings is that the backends don't actually need to open new FDs
for the postmaster.pid file; they can use the one they inherit from the
postmaster, even though they do need to lock it again.  I'm not sure how
much that saves inside the kernel, but at least something.

There are also the usual set of concerns about portability of flock,
though this time we're locking a plain file and not a socket, so it
shouldn't be as much trouble as it was before.

Comments?  Does anyone see a better way to do it?

			regards, tom lane

Tue, 06 Mar 2001 13:10:47 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Lamar Owen writes:

> > case when the postmaster does not come down after 60 seconds.  But this is
> > really no problem for the issue at hand because if you do a normal
> > runlevel switch then the postmaster will simply keep running, and during a
> > system shutdown all the backends are going to die anyway.
>
> Only if each and every shutdown script succeeds in its task.  And I have
> to make sure that the RPM's shipping script successfully pulls down the
> system in an orderly fashion -- of course, I don't have to worry about
> the case where a postmaster is going to be started back up if we are in
> system shutdown -- but, as Tom also stated, I can't assume I'm in the
> system's death throes when called with the stop parameter.

Well, if you have something clever you want to do if the postmaster
doesn't come down after an orderly shutdown then please share it.  The
current alternatives are 'leave running' or 'kill -9'.  I know I'd prefer
the former.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Tue, 6 Mar 2001 19:20:40 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
* Tom Lane <tgl@sss.pgh.pa.us> [010306 10:10] wrote:
> Alfred Perlstein <bright@wintelcom.net> writes:
> > I'm sure some sort of encoding of the PGDATA directory along with
> > the pids stored in the shm segment...
> 
> I thought about this too, but it strikes me as not very trustworthy.
> The problem is that there's no guarantee that the new postmaster will
> even notice the old shmem segment: it might select a different shmem
> key.  (The 7.1 coding of shmem key selection makes this more likely
> than it used to be, but even under 7.0, it will certainly fail to work
> if I choose to start the new postmaster using a different port number
> than the old one had.  The shmem key is driven primarily by port number
> not data directory ...)

This seems like a mistake.  

I'm suprised you guys aren't just using some form of the FreeBSD
ftok() algorithm for this:

FTOK(3)                FreeBSD Library Functions Manual                FTOK(3)

...

     The ftok() function attempts to create a unique key suitable for use with
     the msgget(3), semget(2) and shmget(2) functions given the path of an ex-
     isting file and a user-selectable id.

     The specified path must specify an existing file that is accessible to
     the calling process or the call will fail.  Also, note that links to
     files will return the same key, given the same id.

BUGS
     The returned key is computed based on the device minor number and inode
     of the specified path in combination with the lower 8 bits of the given
     id.  Thus it is quite possible for the routine to return duplicate keys.

The "BUGS" seems to be exactly what you guys are looking for, a somewhat
reliable method of obtaining a system id.  If that sounds evil, read 
below for an alternate suggestion.

> The interlock has to be tightly tied to the PGDATA directory, because
> what we're trying to protect is the files in and under that directory.
> It seems that something based on file(s) in that directory is the way
> to go.
> 
> The best idea I've seen so far is Hiroshi's idea of having all the
> backends hold fcntl locks on the same file (probably postmaster.pid
> would do fine).  Then the new postmaster can test whether any backends
> are still alive by trying to lock the old postmaster.pid file.
> Unfortunately, I read in the fcntl man page:
> 
>     Locks are not inherited by a child process in a fork(2) system call.
> 
> This makes the idea much less attractive than I originally thought:
> a new backend would not automatically inherit a lock on the
> postmaster.pid file from the postmaster, but would have to open/lock it
> for itself.  That means there's a window where the new backend exists
> but would be invisible to a hypothetical new postmaster.
> 
> We could work around this with the following, very ugly protocol:
> 
> 1. Postmaster normally maintains fcntl read lock on its postmaster.pid
> file.  Each spawned backend immediately opens and read-locks
> postmaster.pid, too, and holds that file open until it dies.  (Thus
> wasting a kernel FD per backend, which is one of the less attractive
> things about this.)  If the backend is unable to obtain read lock on
> postmaster.pid, then it complains and dies.  We must use read locks
> here so that all these processes can hold them separately.
> 
> 2. If a newly started postmaster sees a pre-existing postmaster.pid
> file, it tries to obtain a *write* lock on that file.  If it fails,
> conclude that an old postmaster or backend is still alive; complain
> and quit.  If it succeeds, sit for say 1 second before deleting the file
> and creating a new one.  (The delay here is to allow any just-started
> old backends to fail to acquire read lock and quit.  A possible
> objection is that we have no way to guarantee 1 second is enough, though
> it ought to be plenty if the lock acquisition is just after the fork.)
> 
> One thing that worries me a little bit is that this means an fcntl
> read-lock request will exist inside the kernel for each active backend.
> Does anyone know of any performance problems or hard kernel limits we
> might run into with large numbers of backends (lots and lots of fcntl
> locks)?  At least the locks are on a file that we don't actually touch
> in the normal course of business.
> 
> A small savings is that the backends don't actually need to open new FDs
> for the postmaster.pid file; they can use the one they inherit from the
> postmaster, even though they do need to lock it again.  I'm not sure how
> much that saves inside the kernel, but at least something.
> 
> There are also the usual set of concerns about portability of flock,
> though this time we're locking a plain file and not a socket, so it
> shouldn't be as much trouble as it was before.
> 
> Comments?  Does anyone see a better way to do it?

Possibly...

What about encoding the shm id in the pidfile?  Then one can just ask
how many processes are attached to that segment?  (if it doesn't
exist, one can assume all backends have exited)

you want the field 'shm_nattch'

     The shmid_ds struct is defined as follows:

     struct shmid_ds {
         struct ipc_perm shm_perm;   /* operation permission structure */
         int             shm_segsz;  /* size of segment in bytes */
         pid_t           shm_lpid;   /* process ID of last shared memory op */
         pid_t           shm_cpid;   /* process ID of creator */
         short           shm_nattch; /* number of current attaches */
         time_t          shm_atime;  /* time of last shmat() */
         time_t          shm_dtime;  /* time of last shmdt() */
         time_t          shm_ctime;  /* time of last change by shmctl() */
         void           *shm_internal; /* sysv stupidity */
     };


--
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Tue, 6 Mar 2001 10:22:46 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
Alfred Perlstein <bright@wintelcom.net> writes:
> * Tom Lane <tgl@sss.pgh.pa.us> [010306 10:10] wrote:
>> The shmem key is driven primarily by port number
>> not data directory ...)

> This seems like a mistake.  

> I'm suprised you guys aren't just using some form of the FreeBSD
> ftok() algorithm for this:

This has been discussed before --- see the archives.  The conclusion was
that since ftok doesn't guarantee uniqueness, it adds nothing except
lack of predictability to the shmem key selection process.  We'd still
need logic to cope with key collisions, and given that, we might as well
select keys that have some obvious relationship to user-visible
parameters, viz the port number.  As is, you can fairly easily tell
which shmem segment belongs to which postmaster from the shmem key;
with ftok-derived keys, you couldn't tell a thing.

>> Comments?  Does anyone see a better way to do it?

> What about encoding the shm id in the pidfile?  Then one can just ask
> how many processes are attached to that segment?  (if it doesn't
> exist, one can assume all backends have exited)

Hmm ... that might actually be a pretty good idea.  A small problem is
that the shm key isn't yet selected at the time we initially create the
lockfile, but I can't think of any reason that we could not go back and
append the key to the lockfile afterwards.

> you want the field 'shm_nattch'

Are there any portability problems with relying on shm_nattch to be
available?  If not, I like this a lot...

			regards, tom lane

Tue, 06 Mar 2001 13:35:24 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
* Tom Lane <tgl@sss.pgh.pa.us> [010306 10:35] wrote:
> Alfred Perlstein <bright@wintelcom.net> writes:
> 
> > What about encoding the shm id in the pidfile?  Then one can just ask
> > how many processes are attached to that segment?  (if it doesn't
> > exist, one can assume all backends have exited)
> 
> Hmm ... that might actually be a pretty good idea.  A small problem is
> that the shm key isn't yet selected at the time we initially create the
> lockfile, but I can't think of any reason that we could not go back and
> append the key to the lockfile afterwards.
> 
> > you want the field 'shm_nattch'
> 
> Are there any portability problems with relying on shm_nattch to be
> available?  If not, I like this a lot...

Well it's available on FreeBSD and Solaris, I'm sure Redhat has
some deamon that resets the value to 0 periodically just for kicks
so it might not be viable... :)

Seriously, there's some dispute on the type that 'shm_nattch' is,
under Solaris it's "shmatt_t" (unsigned long afaik), under FreeBSD
it's 'short' (i should fix this. :)).

But since you're really only testing for 0'ness then it shouldn't
really be a problem.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Tue, 6 Mar 2001 10:44:47 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
Alfred Perlstein <bright@wintelcom.net> writes:
>> Are there any portability problems with relying on shm_nattch to be
>> available?  If not, I like this a lot...

> Well it's available on FreeBSD and Solaris, I'm sure Redhat has
> some deamon that resets the value to 0 periodically just for kicks
> so it might not be viable... :)

I notice that our BeOS and QNX emulations of shmctl() don't support
IPC_STAT, but that could be dealt with, at least to the extent of
stubbing it out.

This does raise the question of what to do if shmctl(IPC_STAT) fails
for a reason other than EINVAL.  I think the conservative thing to do
is refuse to start up.  On EPERM, for example, it's possible that there
is a postmaster running in your PGDATA but with a different userid.


> Seriously, there's some dispute on the type that 'shm_nattch' is,
> under Solaris it's "shmatt_t" (unsigned long afaik), under FreeBSD
> it's 'short' (i should fix this. :)).

> But since you're really only testing for 0'ness then it shouldn't
> really be a problem.

We need not copy the value anywhere, so as long as the struct is
correctly declared in the system header files I don't think it matters
what the field type is ...

			regards, tom lane

Tue, 06 Mar 2001 13:57:04 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Peter Eisentraut wrote:
> Well, if you have something clever you want to do if the postmaster
> doesn't come down after an orderly shutdown then please share it.  The
> current alternatives are 'leave running' or 'kill -9'.  I know I'd prefer
> the former.

Well, my preferences aren't really relevant here.  I have a job to do as
an initscript in the RPMish environment -- and I really have to meet my
obligations (using the first personal pronoun there to anthropomorph the
initscript to a person, allowing us to have a little sympathy for the
poor shell script's plight :-)).

My preference is to let it float in limbo -- if it's in limbo and won't
come out, then we have bigger issues.

However, I could do something really sneaky in the RedHat environment
and let init do the dirty work for me -- but, again, I am not at all
guaranteed that things will come down orderly -- if it is at all
possible for me to bring about an orderly (if slow) shutdown that does
terminate as the rest of the system needs it to do, then I'll attempt to
do so.

But, the immediate issue is preventing chaotic stops within the
initscript, so I'm going to experiment with things and see if I can make
the initscript hang -- if I can't, then I'll likely put in the 'killproc
postmaster -INT' with escalation to -TERM if it doesn't come down within
sixty seconds (and, no, I am not going to sleep 60 then check things --
I am going to sleep 1 and loop sixty times) -- no need to unnecessarily
delay system shutdown (and potential restart).  And I won't put in the
-KILL unless I can find a safe and thorough way to do so.

Or I may go ahead and pg_ctl-ize things and let pg_ctl do the dirty
work, as that IS what pg_ctl is supposed to accomplish.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 06 Mar 2001 14:07:32 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
* Tom Lane <tgl@sss.pgh.pa.us> [010306 11:03] wrote:
> Alfred Perlstein <bright@wintelcom.net> writes:
> >> Are there any portability problems with relying on shm_nattch to be
> >> available?  If not, I like this a lot...
> 
> > Well it's available on FreeBSD and Solaris, I'm sure Redhat has
> > some deamon that resets the value to 0 periodically just for kicks
> > so it might not be viable... :)
> 
> I notice that our BeOS and QNX emulations of shmctl() don't support
> IPC_STAT, but that could be dealt with, at least to the extent of
> stubbing it out.

Well since we already have spinlocks, I can't see why we can't
keep the refcount and spinlock in a special place in the shm
for all cases?

> This does raise the question of what to do if shmctl(IPC_STAT) fails
> for a reason other than EINVAL.  I think the conservative thing to do
> is refuse to start up.  On EPERM, for example, it's possible that there
> is a postmaster running in your PGDATA but with a different userid.

Yes, if possible a more meaningfull error message and pointer to
some docco would be nice or even a nice "i don't care, i killed
all the backends, just start darnit" flag, it's really no fun at
all to have to attempt to decypher some cryptic error message at
3am when the database/system is acting up. :)

> > Seriously, there's some dispute on the type that 'shm_nattch' is,
> > under Solaris it's "shmatt_t" (unsigned long afaik), under FreeBSD
> > it's 'short' (i should fix this. :)).
> 
> > But since you're really only testing for 0'ness then it shouldn't
> > really be a problem.
> 
> We need not copy the value anywhere, so as long as the struct is
> correctly declared in the system header files I don't think it matters
> what the field type is ...

Yup, my point exactly.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Tue, 6 Mar 2001 11:12:16 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
Alfred Perlstein writes:

> Seriously, there's some dispute on the type that 'shm_nattch' is,
> under Solaris it's "shmatt_t" (unsigned long afaik), under FreeBSD
> it's 'short' (i should fix this. :)).

What I don't like is that my /usr/include/sys/shm.h (through other
headers) has:

typedef unsigned long int shmatt_t;

/* Data structure describing a set of semaphores.  */
struct shmid_ds
  {
    struct ipc_perm shm_perm;           /* operation permission struct */
    size_t shm_segsz;                   /* size of segment in bytes */
    __time_t shm_atime;                 /* time of last shmat() */
    unsigned long int __unused1;
    __time_t shm_dtime;                 /* time of last shmdt() */
    unsigned long int __unused2;
    __time_t shm_ctime;                 /* time of last change by shmctl() */
    unsigned long int __unused3;
    __pid_t shm_cpid;                   /* pid of creator */
    __pid_t shm_lpid;                   /* pid of last shmop */
    shmatt_t shm_nattch;                /* number of current attaches */
    unsigned long int __unused4;
    unsigned long int __unused5;
  };

whereas /usr/src/linux/include/shm.h has:

struct shmid_ds {
        struct ipc_perm         shm_perm;       /* operation perms */
        int                     shm_segsz;      /* size of segment (bytes) */
        __kernel_time_t         shm_atime;      /* last attach time */
        __kernel_time_t         shm_dtime;      /* last detach time */
        __kernel_time_t         shm_ctime;      /* last change time */
        __kernel_ipc_pid_t      shm_cpid;       /* pid of creator */
        __kernel_ipc_pid_t      shm_lpid;       /* pid of last operator */
        unsigned short          shm_nattch;     /* no. of current attaches */
        unsigned short          shm_unused;     /* compatibility */
        void                    *shm_unused2;   /* ditto - used by DIPC */
        void                    *shm_unused3;   /* unused */
};


Not only note the shm_nattch type, but also shm_segsz, and the "unused"
fields in between.  I don't know a thing about the Linux kernel sources,
but this doesn't seem right.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Tue, 6 Mar 2001 20:19:12 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
Alfred Perlstein <bright@wintelcom.net> writes:
> * Tom Lane <tgl@sss.pgh.pa.us> [010306 11:03] wrote:
>> I notice that our BeOS and QNX emulations of shmctl() don't support
>> IPC_STAT, but that could be dealt with, at least to the extent of
>> stubbing it out.

> Well since we already have spinlocks, I can't see why we can't
> keep the refcount and spinlock in a special place in the shm
> for all cases?

No, we mustn't go there.  If the kernel isn't keeping the refcount
then it's worse than useless: as soon as some process crashes without
decrementing its refcount, you have a condition that you can't recover
from without reboot.

What I'm currently imagining is that the stub implementations will just
return a failure code for IPC_STAT, and the outer code will in turn fail
with a message along the lines of "It looks like there's a pre-existing
shmem block (id XXX) still in use.  If you're sure there are no old
backends still running, remove the shmem block with ipcrm(1), or just
delete $PGDATA/postmaster.pid."  I dunno what shmem management tools
exist on BeOS/QNX, but deleting the lockfile will definitely suppress
the startup interlock ;-).

> Yes, if possible a more meaningfull error message and pointer to
> some docco would be nice

Is the above good enough?

			regards, tom lane

Tue, 06 Mar 2001 14:24:54 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
Peter Eisentraut wrote:
> Not only note the shm_nattch type, but also shm_segsz, and the "unused"
> fields in between.  I don't know a thing about the Linux kernel sources,
> but this doesn't seem right.

Red Hat 7, right?  My RedHat 7 system isn't running RH 7 right now (it's
this notebook that I'm running Win95 on right now), but see which RPM's
own the two headers.  You may be in for a shock.  IIRC, the first system
include is from the 2.4 kernel, and the second in the kernel source tree
is from the 2.2 kernel.

Odd, but not really broken.  Should be fixed in the latest public beta
of RedHat, that actually has the 2.4 kernel.  I can't really say any
more about that, however.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 06 Mar 2001 14:27:12 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
* Tom Lane <tgl@sss.pgh.pa.us> [010306 11:30] wrote:
> Alfred Perlstein <bright@wintelcom.net> writes:
> > * Tom Lane <tgl@sss.pgh.pa.us> [010306 11:03] wrote:
> >> I notice that our BeOS and QNX emulations of shmctl() don't support
> >> IPC_STAT, but that could be dealt with, at least to the extent of
> >> stubbing it out.
> 
> > Well since we already have spinlocks, I can't see why we can't
> > keep the refcount and spinlock in a special place in the shm
> > for all cases?
> 
> No, we mustn't go there.  If the kernel isn't keeping the refcount
> then it's worse than useless: as soon as some process crashes without
> decrementing its refcount, you have a condition that you can't recover
> from without reboot.

Not if the postmaster outputs the following:

> What I'm currently imagining is that the stub implementations will just
> return a failure code for IPC_STAT, and the outer code will in turn fail
> with a message along the lines of "It looks like there's a pre-existing
> shmem block (id XXX) still in use.  If you're sure there are no old
> backends still running, remove the shmem block with ipcrm(1), or just
> delete $PGDATA/postmaster.pid."  I dunno what shmem management tools
> exist on BeOS/QNX, but deleting the lockfile will definitely suppress
> the startup interlock ;-).
> 
> > Yes, if possible a more meaningfull error message and pointer to
> > some docco would be nice
> 
> Is the above good enough?

Sure. :)

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Tue, 6 Mar 2001 11:34:32 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
Peter Eisentraut <peter_e@gmx.net> writes:
> What I don't like is that my /usr/include/sys/shm.h (through other
> headers) has [foo]
> whereas /usr/src/linux/include/shm.h has [bar]

Are those declarations perhaps bit-compatible?  Looks a tad endian-
dependent, though ...

			regards, tom lane

Tue, 06 Mar 2001 14:36:26 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
* Lamar Owen <lamar.owen@wgcr.org> [010306 11:39] wrote:
> Peter Eisentraut wrote:
> > Not only note the shm_nattch type, but also shm_segsz, and the "unused"
> > fields in between.  I don't know a thing about the Linux kernel sources,
> > but this doesn't seem right.
> 
> Red Hat 7, right?  My RedHat 7 system isn't running RH 7 right now (it's
> this notebook that I'm running Win95 on right now), but see which RPM's
> own the two headers.  You may be in for a shock.  IIRC, the first system
> include is from the 2.4 kernel, and the second in the kernel source tree
> is from the 2.2 kernel.
> 
> Odd, but not really broken.  Should be fixed in the latest public beta
> of RedHat, that actually has the 2.4 kernel.  I can't really say any
> more about that, however.

Y'know, I was only kidding about Linux going out of its way to
defeat the 'shm_nattch' trick... *sigh*

As a FreeBSD developer I'm wondering if Linux keeps compatibility
calls around for old binaries or not.  Any idea?

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Tue, 6 Mar 2001 11:49:45 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
* Tom Lane <tgl@sss.pgh.pa.us> [010306 11:49] wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > What I don't like is that my /usr/include/sys/shm.h (through other
> > headers) has [foo]
> > whereas /usr/src/linux/include/shm.h has [bar]
> 
> Are those declarations perhaps bit-compatible?  Looks a tad endian-
> dependent, though ...

Of course not, the size of the struct changed (short->unsigned
long, basically int16_t -> uint32_t), because the kernel and userland
in Linux are hardly in sync you have the fun of guessing if you
get:

old struct -> old syscall (ok)
new struct -> old syscall (boom)
old struct -> new syscall (boom)
new struct -> new syscall (ok)

Honestly I think this problem should be left to the vendor to fix
properly (if it needs fixing), the sysV API was published at least
6 years ago, they ought to have it mostly correct by now.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Tue, 6 Mar 2001 11:54:44 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
Alfred Perlstein <bright@wintelcom.net> writes:
> Of course not, the size of the struct changed (short->unsigned
> long, basically int16_t -> uint32_t), because the kernel and userland
> in Linux are hardly in sync you have the fun of guessing if you
> get:

> old struct -> old syscall (ok)
> new struct -> old syscall (boom)
> old struct -> new syscall (boom)
> new struct -> new syscall (ok)

Ugh.  However, it looks like it might be fairly fail-soft: if we
have the wrong declaration then we pick up some other field of the
struct, and probably end up complaining because nattch appears nonzero.
Recovery method (clean up the shm seg or delete lockfile) is the same.

I'm still inclined to go with this; it beats corrupting the WAL log,
and the fcntl(SETLK) alternative has its own set of portability
booby-traps.

			regards, tom lane

Tue, 06 Mar 2001 15:20:59 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: How to shoot yourself in the foot: kill -9 postmaster 
On Tue, Mar 06, 2001 at 08:19:12PM +0100, Peter Eisentraut wrote:
> Alfred Perlstein writes:
> 
> > Seriously, there's some dispute on the type that 'shm_nattch' is,
> > under Solaris it's "shmatt_t" (unsigned long afaik), under FreeBSD
> > it's 'short' (i should fix this. :)).
> 
> What I don't like is that my /usr/include/sys/shm.h (through other
> headers) has:
> 
> typedef unsigned long int shmatt_t;
> 
> /* Data structure describing a set of semaphores.  */
> struct shmid_ds
>   {
>     struct ipc_perm shm_perm;           /* operation permission struct */
>     size_t shm_segsz;                   /* size of segment in bytes */
>     __time_t shm_atime;                 /* time of last shmat() */
>     unsigned long int __unused1;
>     __time_t shm_dtime;                 /* time of last shmdt() */
>     unsigned long int __unused2;
>     __time_t shm_ctime;                 /* time of last change by shmctl() */
>     unsigned long int __unused3;
>     __pid_t shm_cpid;                   /* pid of creator */
>     __pid_t shm_lpid;                   /* pid of last shmop */
>     shmatt_t shm_nattch;                /* number of current attaches */
>     unsigned long int __unused4;
>     unsigned long int __unused5;
>   };
> 
> whereas /usr/src/linux/include/shm.h has:
> 
> struct shmid_ds {
>         struct ipc_perm         shm_perm;       /* operation perms */
>         int                     shm_segsz;      /* size of segment (bytes) */
>         __kernel_time_t         shm_atime;      /* last attach time */
>         __kernel_time_t         shm_dtime;      /* last detach time */
>         __kernel_time_t         shm_ctime;      /* last change time */
>         __kernel_ipc_pid_t      shm_cpid;       /* pid of creator */
>         __kernel_ipc_pid_t      shm_lpid;       /* pid of last operator */
>         unsigned short          shm_nattch;     /* no. of current attaches */
>         unsigned short          shm_unused;     /* compatibility */
>         void                    *shm_unused2;   /* ditto - used by DIPC */
>         void                    *shm_unused3;   /* unused */
> };
> 
> 
> Not only note the shm_nattch type, but also shm_segsz, and the "unused"
> fields in between.  I don't know a thing about the Linux kernel sources,
> but this doesn't seem right.

On Linux, /usr/src/linux/include is meaningless for anything in userland; 
it's meant only for building the kernel and kernel modules.  That Red Hat 
tends to expose it to user-level builds is a long-standing bug in Red 
Hat's distribution, in violation of the File Hierarchy Standard as well 
as explicit instructions from Linus & crew and from the maintainer of the 
C library.

User-level programs see what's in /usr/include, which only has to match 
what the C library wants.  It's the C library's job to do any mapping 
needed, and it does.  The C library is maintained very, very carefully
to keep binary compatibility with all old versions.  (One sometimes
encounters commercial programs that rely on a bug or undocumented/
usupported feature that disappears in a later library version.)

That is why there is no problem with version skew in the syscall
argument structures on a correctly-configured Linux system.  (On a
Red Hat system it is very easy to get them out of sync, but RH fans 
are used to problems.)

Nathan Myers
ncm@zembu.com

Tue, 6 Mar 2001 12:46:24 -0800
ncm@zembu.com (Nathan Myers)
Re: How to shoot yourself in the foot: kill -9 postmaster
Nathan Myers wrote:
> That is why there is no problem with version skew in the syscall
> argument structures on a correctly-configured Linux system.  (On a
> Red Hat system it is very easy to get them out of sync, but RH fans
> are used to problems.)

Is RedHat bashing really necessary here?  At least they are payrolling
Second Chair on the Linux kernel hierarchy.  And they are very
supportive of PostgreSQL (by shipping us with their distribution).
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 06 Mar 2001 16:20:13 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
On Tue, Mar 06, 2001 at 12:46:24PM -0800, Nathan Myers wrote:
> 
> On Linux, /usr/src/linux/include is meaningless for anything in userland; 
> it's meant only for building the kernel and kernel modules.  That Red Hat 
> tends to expose it to user-level builds is a long-standing bug in Red 
> Hat's distribution, in violation of the File Hierarchy Standard as well 
> as explicit instructions from Linus & crew and from the maintainer of the 
> C library.
> 
Red Hat's Fisher Beta has split the 2 includes, which caused an error trying
to compile a (I guess badly configured) kernel module.  The header files in
/usr/include now give an error if you try to build a kernel module that gets
header files from there.

So whether they were wrong in the past or not, they are now doing things the
way you say is proper.


Tue, 6 Mar 2001 13:56:25 -0800
Samuel Sieb <samuel@sieb.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
El Mar 06 Mar 2001 18:56, Samuel Sieb escribiï¿½:
> On Tue, Mar 06, 2001 at 12:46:24PM -0800, Nathan Myers wrote:
> > On Linux, /usr/src/linux/include is meaningless for anything in userland;
> > it's meant only for building the kernel and kernel modules.  That Red Hat
> > tends to expose it to user-level builds is a long-standing bug in Red
> > Hat's distribution, in violation of the File Hierarchy Standard as well
> > as explicit instructions from Linus & crew and from the maintainer of the
> > C library.
>
> Red Hat's Fisher Beta has split the 2 includes, which caused an error
> trying to compile a (I guess badly configured) kernel module.  The header
> files in /usr/include now give an error if you try to build a kernel module
> that gets header files from there.
>
> So whether they were wrong in the past or not, they are now doing things
> the way you say is proper.

I am very happy for seeing RedHat let out beta releases of there 
distribution. That's whats importante about it all.

-- 
System Administration: It's a dirty job, 
but someone told I had to do it.
-----------------------------------------------------------------
Martï¿½n Marquï¿½s			email: 	martin@math.unl.edu.ar
Santa Fe - Argentina		http://math.unl.edu.ar/~martin/
Administrador de sistemas en math.unl.edu.ar
-----------------------------------------------------------------

Tue, 6 Mar 2001 19:01:48 -0300
"Martin A. Marques" <martin@math.unl.edu.ar>
Re: How to shoot yourself in the foot: kill -9 postmaster
* Lamar Owen <lamar.owen@wgcr.org> [010306 13:27] wrote:
> Nathan Myers wrote:
> > That is why there is no problem with version skew in the syscall
> > argument structures on a correctly-configured Linux system.  (On a
> > Red Hat system it is very easy to get them out of sync, but RH fans
> > are used to problems.)
> 
> Is RedHat bashing really necessary here?  At least they are payrolling
> Second Chair on the Linux kernel hierarchy.  And they are very
> supportive of PostgreSQL (by shipping us with their distribution).

Just because they do some really nice things and have some really
nice stuff doesn't mean they should really get cut any slack for
doing things like shipping out of sync kernel/system headers, kill
-9'ing databases and having programs like 'tmpwatch' running on
the boxes.  It really shows a lack of understanding of how Unix is
supposed to run.

What they really need to do is hire some grey beards (old school
Unix folks) to QA the releases and keep stuff like this from
happening/shipping. 

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Tue, 6 Mar 2001 14:43:05 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: How to shoot yourself in the foot: kill -9 postmaster

   BeOS haven't this stat (I have a bunch of others but not this one).

   If I unsterstand correctly, you want to check if there is some backend 
still attached to shared mem segment of a given key ? In this case, I have an 
easy solution to fake the stat, because all segment have an encoded name 
containing this key, so I can count them.


    cyril

>
>Alfred Perlstein <bright@wintelcom.net> writes:
>>> Are there any portability problems with relying on shm_nattch to be
>>> available?  If not, I like this a lot...
>
>> Well it's available on FreeBSD and Solaris, I'm sure Redhat has
>> some deamon that resets the value to 0 periodically just for kicks
>> so it might not be viable... :)
>
>I notice that our BeOS and QNX emulations of shmctl() don't support
>IPC_STAT, but that could be dealt with, at least to the extent of
>stubbing it out.
>
>This does raise the question of what to do if shmctl(IPC_STAT) fails
>for a reason other than EINVAL.  I think the conservative thing to do
>is refuse to start up.  On EPERM, for example, it's possible that there
>is a postmaster running in your PGDATA but with a different userid.
>
>
>> Seriously, there's some dispute on the type that 'shm_nattch' is,
>> under Solaris it's "shmatt_t" (unsigned long afaik), under FreeBSD
>> it's 'short' (i should fix this. :)).
>
>> But since you're really only testing for 0'ness then it shouldn't
>> really be a problem.
>
>We need not copy the value anywhere, so as long as the struct is
>correctly declared in the system header files I don't think it matters
>what the field type is ...
>
>			regards, tom lane
>
>---------------------------(end of broadcast)---------------------------
>TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>


Wed, 07 Mar 2001 01:04:20 +0100
Cyril VELTER <cyril.velter@libertysurf.fr>
Re: How to shoot yourself in the foot: kill -9 postmaster 
> >Alfred Perlstein <bright@wintelcom.net> writes:
> >>> Are there any portability problems with relying on shm_nattch to be
> >>> available?  If not, I like this a lot...
> >
> >> Well it's available on FreeBSD and Solaris, I'm sure Redhat has
> >> some deamon that resets the value to 0 periodically just for kicks
> >> so it might not be viable... :)
> >
> >I notice that our BeOS and QNX emulations of shmctl() don't support
> >IPC_STAT, but that could be dealt with, at least to the extent of
> >stubbing it out.

* Cyril VELTER <cyril.velter@libertysurf.fr> [010306 16:15] wrote:
> 
>    BeOS haven't this stat (I have a bunch of others but not this one).
> 
>    If I unsterstand correctly, you want to check if there is some backend 
> still attached to shared mem segment of a given key ? In this case, I have an 
> easy solution to fake the stat, because all segment have an encoded name 
> containing this key, so I can count them.

We need to be able to take a single shared memory segment and
determine if any other process is using it.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Tue, 6 Mar 2001 16:22:10 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: How to shoot yourself in the foot: kill -9 postmaster
Alfred Perlstein wrote:
> What they really need to do is hire some grey beards (old school
> Unix folks) to QA the releases and keep stuff like this from
> happening/shipping.

Like the 250-strong RedHat Beta Team, of which I am a member? :-) I
can't disclose the discussions on that list, but, suffice to say the
traffic there is at least as great as the traffic on this one.

Of course, 7.1 hasn't shipped with a RedHat release yet -- and it's my
job to make sure the postmaster gets killed properly in my initscript
inside the package for 7.1 -- there will be no kill -9 unless it is an
emergency to do so for postmaster.

I've seen the advisories and the bug lists -- RedHat is not alone with
bugs -- not even unusual with bugs.  And every OS I know of (and you
too) has had a brown paper bag release before.  Even PostgreSQL, given
its high release quality standards, has had a brown paper bag release --
we all still make mistakes (I know -- I've made more than my share of
them).

Anyway, that's more than what the rest of the list wanted to read.
Replies to private e-mail, please. 
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 06 Mar 2001 19:29:41 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: How to shoot yourself in the foot: kill -9 postmaster
On Tue, Mar 06, 2001 at 04:20:13PM -0500, Lamar Owen wrote:
> Nathan Myers wrote:
> > That is why there is no problem with version skew in the syscall
> > argument structures on a correctly-configured Linux system.  (On a
> > Red Hat system it is very easy to get them out of sync, but RH fans
> > are used to problems.)
> 
> Is RedHat bashing really necessary here? 

I recognize that my last seven words above contributed nothing.
In the future I will only post strictly factual statements about
Red Hat and similarly charged topics, and keep the opinions to
myself.  I value the collegiality of this list too much to risk 
it further.  I offer my apologies for violating it.

By the way... do they call Red Hat "RedHat" at Red Hat? 

Nathan Myers
ncm@zembu.com

Tue, 6 Mar 2001 16:58:16 -0800
ncm@zembu.com (Nathan Myers)
Re: Red Hat bashing
Nathan Myers wrote:
> it further.  I offer my apologies for violating it.

Well, an apology is not really necessary -- but I do get a little tired
at the treatment a good open source company gets at the hands of open
source advocates.  Yes, they make mistakes. Everyone does.
 
> By the way... do they call Red Hat "RedHat" at Red Hat?

No, they don't.  I don't know how I got into the habit of leaving out
the space, but the space is supposed to be there -- unless you are on
the Red Hat CD, where you will find a directory called 'RedHat'.

Oh well. Totally off topic.  If the from header had your personal
address in it (Reply-All only lets me reply to the list for that
message) I wouldn't grieve the list further with it.

My last words on that subject.  Let's go on making PostgreSQL better. 
And preventing the kill -9 will make PostgreSQL better, even if it is
masking a certain amount of shortsightedness on a certain initscripts
author's part. :-)
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 06 Mar 2001 20:16:18 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Red Hat bashing
ncm@zembu.com (Nathan Myers) writes:

> On Linux, /usr/src/linux/include is meaningless for anything in userland; 
> it's meant only for building the kernel and kernel modules.  That Red Hat 
> tends to expose it to user-level builds is a long-standing bug in Red 
> Hat's distribution

1) it isn't this way anyore
2) this was so for most distributions for a loong time, not a "Red
   Hat" bug.

> in violation of the File Hierarchy Standard as well as explicit
> instructions from Linus & crew and from the maintainer of the C
> library.

Which obviously hasn't always been the case - the FHS isn't exactly old. 
Things have changed since then, we have followed.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

06 Mar 2001 20:50:52 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: How to shoot yourself in the foot: kill -9 postmaster
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> The interlock has to be tightly tied to the PGDATA directory, because
> what we're trying to protect is the files in and under that directory.
> It seems that something based on file(s) in that directory is the way
> to go.
> 
> The best idea I've seen so far is Hiroshi's idea of having all the
> backends hold fcntl locks on the same file (probably postmaster.pid
> would do fine).  Then the new postmaster can test whether any backends
> are still alive by trying to lock the old postmaster.pid file.
> Unfortunately, I read in the fcntl man page:
> 
>     Locks are not inherited by a child process in a fork(2) system call.
> 

Yes flock() works well here but fcntl() doesn't.

> This makes the idea much less attractive than I originally thought:
> a new backend would not automatically inherit a lock on the
> postmaster.pid file from the postmaster, but would have to open/lock it
> for itself.  That means there's a window where the new backend exists
> but would be invisible to a hypothetical new postmaster.
> 
> We could work around this with the following, very ugly protocol:
> 
> 1. Postmaster normally maintains fcntl read lock on its postmaster.pid
> file.  Each spawned backend immediately opens and read-locks
> postmaster.pid, too, and holds that file open until it dies.  (Thus
> wasting a kernel FD per backend, which is one of the less attractive
> things about this.)  If the backend is unable to obtain read lock on
> postmaster.pid, then it complains and dies.  We must use read locks
> here so that all these processes can hold them separately.
> 
> 2. If a newly started postmaster sees a pre-existing postmaster.pid
> file, it tries to obtain a *write* lock on that file.  If it fails,
> conclude that an old postmaster or backend is still alive; complain
> and quit.  If it succeeds, sit for say 1 second before deleting the file
> and creating a new one.  (The delay here is to allow any just-started
> old backends to fail to acquire read lock and quit.  A possible
> objection is that we have no way to guarantee 1 second is enough, though
> it ought to be plenty if the lock acquisition is just after the fork.)
> 

I have another idea. My main point is to not remove the existent
pidfile. For example
1) A newly started postmaster tries to obtain a write lock on the
   first byte of the pidfile. If it fails the postmaster quit.
2) The postmaster tries to obtain a write lock on the second byte
    of the pidfile. If it fails the postmaster quit.
3) The postmaster releases the lock of 2).
4) Each backend obtains a read-lock on the second byte of the
    pidfile.

Regards,
Hiroshi Inoue 

Wed, 7 Mar 2001 13:38:36 +0900
"Hiroshi Inoue" <Inoue@tpf.co.jp>
RE: How to shoot yourself in the foot: kill -9 postmaster 
Samuel Sieb <samuel@sieb.net> writes:

> On Tue, Mar 06, 2001 at 12:46:24PM -0800, Nathan Myers wrote:
> > 
> > On Linux, /usr/src/linux/include is meaningless for anything in userland; 
> > it's meant only for building the kernel and kernel modules.  That Red Hat 
> > tends to expose it to user-level builds is a long-standing bug in Red 
> > Hat's distribution, in violation of the File Hierarchy Standard as well 
> > as explicit instructions from Linus & crew and from the maintainer of the 
> > C library.
> > 
> Red Hat's Fisher Beta has split the 2 includes, which caused an error trying
> to compile a (I guess badly configured) kernel module.  

It was split in Red Hat Linux 7 as well.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

08 Mar 2001 11:43:16 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: How to shoot yourself in the foot: kill -9 postmaster