On FreeBSD 4.1.1 and above there's a sysctl tunable called
kern.ipc.shm_use_phys, when set to 1 it's supposed to
make the kernel's handling of shared memory much more
effecient at the expense or making the shm segment unpageable.

I tried to use this option with 7.0.3 and FreeBSD 4.2 but
for some reason spinlocks keep getting mucked up (there's
a log at the tail end of this message).

Anyone using Postgresql on FreeBSD probably wants this to work,
otherwise using extremely large chunks of shm and many backends
active can exhaust kernel memory.

I was wondering if any of the more experienced developers could
take a look at what's happenening here.

Here's the log, the number in parens is the address of the lock,
on tas() the value printed to the right is the value in _ret,
for the others, it's the value before the lock count is set.

S_INIT_LOCK: (0x30048008) -> 0
S_UNLOCK: (0x30048008) -> 0
S_INIT_LOCK: (0x3004800c) -> 0
S_UNLOCK: (0x3004800c) -> 0
S_INIT_LOCK: (0x30048010) -> 0
S_UNLOCK: (0x30048010) -> 0
S_INIT_LOCK: (0x30048011) -> 0
S_UNLOCK: (0x30048011) -> 0
S_INIT_LOCK: (0x30048012) -> 0
S_UNLOCK: (0x30048012) -> 0
S_INIT_LOCK: (0x30048018) -> 0
S_UNLOCK: (0x30048018) -> 0
S_INIT_LOCK: (0x3004801c) -> 0
S_UNLOCK: (0x3004801c) -> 0
S_INIT_LOCK: (0x3004801d) -> 1
S_UNLOCK: (0x3004801d) -> 1
S_INIT_LOCK: (0x3004801e) -> 0
S_UNLOCK: (0x3004801e) -> 0
S_INIT_LOCK: (0x30048024) -> 127
S_UNLOCK: (0x30048024) -> 127
S_INIT_LOCK: (0x30048028) -> 255
S_UNLOCK: (0x30048028) -> 255
S_INIT_LOCK: (0x30048029) -> 0
S_UNLOCK: (0x30048029) -> 0
S_INIT_LOCK: (0x3004802a) -> 0
S_UNLOCK: (0x3004802a) -> 0
S_INIT_LOCK: (0x30048030) -> 1
S_UNLOCK: (0x30048030) -> 1
S_INIT_LOCK: (0x30048034) -> 0
S_UNLOCK: (0x30048034) -> 0
S_INIT_LOCK: (0x30048035) -> 0
S_UNLOCK: (0x30048035) -> 0
S_INIT_LOCK: (0x30048036) -> 0
S_UNLOCK: (0x30048036) -> 0
S_INIT_LOCK: (0x3004803c) -> 50
S_UNLOCK: (0x3004803c) -> 50
S_INIT_LOCK: (0x30048040) -> 10
S_UNLOCK: (0x30048040) -> 10
S_INIT_LOCK: (0x30048041) -> 0
S_UNLOCK: (0x30048041) -> 0
S_INIT_LOCK: (0x30048042) -> 0
S_UNLOCK: (0x30048042) -> 0
S_INIT_LOCK: (0x30048048) -> 1
S_UNLOCK: (0x30048048) -> 1
S_INIT_LOCK: (0x3004804c) -> 80
S_UNLOCK: (0x3004804c) -> 80
S_INIT_LOCK: (0x3004804d) -> 1
S_UNLOCK: (0x3004804d) -> 1
S_INIT_LOCK: (0x3004804e) -> 0
S_UNLOCK: (0x3004804e) -> 0
S_INIT_LOCK: (0x30048054) -> 0
S_UNLOCK: (0x30048054) -> 0
S_INIT_LOCK: (0x30048058) -> 1
S_UNLOCK: (0x30048058) -> 1
S_INIT_LOCK: (0x30048059) -> 1
S_UNLOCK: (0x30048059) -> 1
S_INIT_LOCK: (0x3004805a) -> 0
S_UNLOCK: (0x3004805a) -> 0
S_INIT_LOCK: (0x30048060) -> 0
S_UNLOCK: (0x30048060) -> 0
S_INIT_LOCK: (0x30048064) -> 0
S_UNLOCK: (0x30048064) -> 0
S_INIT_LOCK: (0x30048065) -> 0
S_UNLOCK: (0x30048065) -> 0
S_INIT_LOCK: (0x30048066) -> 0
S_UNLOCK: (0x30048066) -> 0
S_INIT_LOCK: (0x3004806c) -> 0
S_UNLOCK: (0x3004806c) -> 0
S_INIT_LOCK: (0x30048070) -> 0
S_UNLOCK: (0x30048070) -> 0
S_INIT_LOCK: (0x30048071) -> 0
S_UNLOCK: (0x30048071) -> 0
S_INIT_LOCK: (0x30048072) -> 0
S_UNLOCK: (0x30048072) -> 0
S_INIT_LOCK: (0x30048078) -> 0
S_UNLOCK: (0x30048078) -> 0
S_INIT_LOCK: (0x3004807c) -> 0
S_UNLOCK: (0x3004807c) -> 0
S_INIT_LOCK: (0x3004807d) -> 0
S_UNLOCK: (0x3004807d) -> 0
S_INIT_LOCK: (0x3004807e) -> 0
S_UNLOCK: (0x3004807e) -> 0
tas (0x30048054) -> 0
tas (0x30048059) -> 0
tas (0x30048058) -> 0
S_UNLOCK: (0x30048054) -> 1
tas (0x30048048) -> 0
tas (0x3004804d) -> 0
tas (0x3004804c) -> 0
S_UNLOCK: (0x30048048) -> 1
tas (0x30048048) -> 0
S_UNLOCK: (0x3004804c) -> 1
S_UNLOCK: (0x3004804d) -> 1
S_UNLOCK: (0x30048048) -> 1
tas (0x30048048) -> 0
tas (0x3004804d) -> 0
tas (0x3004804c) -> 0
S_UNLOCK: (0x30048048) -> 1
tas (0x30048048) -> 0
S_UNLOCK: (0x3004804c) -> 1
S_UNLOCK: (0x3004804d) -> 1
S_UNLOCK: (0x30048048) -> 1
tas (0x30048048) -> 0
tas (0x3004804d) -> 4
tas (0x3004804d) -> 1
tas (0x3004804d) -> 1
tas (0x3004804d) -> 1
tas (0x3004804d) -> 1
tas (0x3004804d) -> 1
tas (0x3004804d) -> 1
tas (0x3004804d) -> 1

repeats (it's stuck)


-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."

Tue, 5 Dec 2000 07:14:58 -0800
Alfred Perlstein <bright@wintelcom.net>
Need help with phys backed shm segments (Postgresql+FreeBSD).
Alfred Perlstein <bright@wintelcom.net> writes:
> Here's the log, the number in parens is the address of the lock,
> on tas() the value printed to the right is the value in _ret,
> for the others, it's the value before the lock count is set.

This looks to be the trace of a SpinAcquire()
(see src/backend/storage/ipc/spin.c):

> tas (0x30048048) -> 0
> tas (0x3004804d) -> 0
> tas (0x3004804c) -> 0
> S_UNLOCK: (0x30048048) -> 1

followed by SpinRelease():

> tas (0x30048048) -> 0
> S_UNLOCK: (0x3004804c) -> 1
> S_UNLOCK: (0x3004804d) -> 1
> S_UNLOCK: (0x30048048) -> 1

followed by a failed attempt to reacquire the same SLock:

> tas (0x30048048) -> 0
> tas (0x3004804d) -> 4
> tas (0x3004804d) -> 1
> tas (0x3004804d) -> 1
> tas (0x3004804d) -> 1
> tas (0x3004804d) -> 1

And that looks completely broken :-( ... something's clobbered the
exlock field of the SLock struct, apparently.  Are you sure this
kernel feature you're trying to use actually works?

BTW, if you're wondering why an SLock needs to contain *three*
hardware spinlocks, the answer is that it doesn't.  This code has
been greatly simplified in current sources...

			regards, tom lane

Tue, 05 Dec 2000 10:43:10 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Need help with phys backed shm segments (Postgresql+FreeBSD). 
* Tom Lane <tgl@sss.pgh.pa.us> [001205 07:43] wrote:
> Alfred Perlstein <bright@wintelcom.net> writes:
> > Here's the log, the number in parens is the address of the lock,
> > on tas() the value printed to the right is the value in _ret,
> > for the others, it's the value before the lock count is set.
> 
> This looks to be the trace of a SpinAcquire()
> (see src/backend/storage/ipc/spin.c):

Yes, those are my debug printfs :).

> > tas (0x30048048) -> 0
> > tas (0x3004804d) -> 0
> > tas (0x3004804c) -> 0
> > S_UNLOCK: (0x30048048) -> 1
> 
> followed by SpinRelease():
> 
> > tas (0x30048048) -> 0
> > S_UNLOCK: (0x3004804c) -> 1
> > S_UNLOCK: (0x3004804d) -> 1
> > S_UNLOCK: (0x30048048) -> 1
> 
> followed by a failed attempt to reacquire the same SLock:
> 
> > tas (0x30048048) -> 0
> > tas (0x3004804d) -> 4
> > tas (0x3004804d) -> 1
> > tas (0x3004804d) -> 1
> > tas (0x3004804d) -> 1
> > tas (0x3004804d) -> 1
> 
> And that looks completely broken :-( ... something's clobbered the
> exlock field of the SLock struct, apparently.  Are you sure this
> kernel feature you're trying to use actually works?

No I'm not sure actually. :)  I'll look into it further, but I
was wondering if there was something I could do to debug the
locks better.  I think I'll add some S_MAGIC or something in
the struct to see if the whole thing is getting clobbered or
what...  If you have any suggestions let me know.

> BTW, if you're wondering why an SLock needs to contain *three*
> hardware spinlocks, the answer is that it doesn't.  This code has
> been greatly simplified in current sources...

It did look a bit strange...

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."

Tue, 5 Dec 2000 07:47:13 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: Need help with phys backed shm segments (Postgresql+FreeBSD).
Alfred Perlstein <bright@wintelcom.net> writes:
> No I'm not sure actually. :)  I'll look into it further, but I
> was wondering if there was something I could do to debug the
> locks better.  I think I'll add some S_MAGIC or something in
> the struct to see if the whole thing is getting clobbered or
> what...  If you have any suggestions let me know.

Seems like a plan.  In current sources I have moved the SLock struct
declaration out of header files and into spin.c; it doesn't really
need to be known anywhere else.  You could probably do the same in
7.0.*, which would greatly simplify changing the struct around to
see what's happening.

			regards, tom lane

Tue, 05 Dec 2000 11:01:53 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Need help with phys backed shm segments (Postgresql+FreeBSD). 
BTW, I just remembered that in 7.0.*, the SLocks that are managed by
SpinAcquire() all live in their own little shm segment.  On a machine
where slock_t is char, it'd likely only amount to 128 bytes or so.
Maybe you are seeing some bug in FreeBSD's handling of tiny shm
segments?

			regards, tom lane

Tue, 05 Dec 2000 11:12:32 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Need help with phys backed shm segments (Postgresql+FreeBSD). 
* Tom Lane <tgl@sss.pgh.pa.us> [001205 08:37] wrote:
> BTW, I just remembered that in 7.0.*, the SLocks that are managed by
> SpinAcquire() all live in their own little shm segment.  On a machine
> where slock_t is char, it'd likely only amount to 128 bytes or so.
> Maybe you are seeing some bug in FreeBSD's handling of tiny shm
> segments?

Good call, i think I found it! :)

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."

Tue, 5 Dec 2000 12:04:09 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: Need help with phys backed shm segments (Postgresql+FreeBSD).
* Alfred Perlstein <bright@wintelcom.net> [001205 12:30] wrote:
> * Tom Lane <tgl@sss.pgh.pa.us> [001205 08:37] wrote:
> > BTW, I just remembered that in 7.0.*, the SLocks that are managed by
> > SpinAcquire() all live in their own little shm segment.  On a machine
> > where slock_t is char, it'd likely only amount to 128 bytes or so.
> > Maybe you are seeing some bug in FreeBSD's handling of tiny shm
> > segments?
> 
> Good call, i think I found it! :)

Here's the patch I'm using on FreeBSD, it seems to work, if any
other FreeBSD'ers want to try it out, just apply the patch:
cd /usr/src/sys/vm ; patch < patchfile

and recompile and boot with a new kernel, then do this:

sysctl -w kern.ipc.shm_use_phys=1

or add:
kern.ipc.shm_use_phys=1 
to /etc/sysctl.conf

Let me know if it works.

thanks,
-Alfred

Index: phys_pager.c
===================================================================
RCS file: /home/ncvs/src/sys/vm/phys_pager.c,v
retrieving revision 1.3.2.1
diff -u -u -r1.3.2.1 phys_pager.c
--- phys_pager.c	2000/08/04 22:31:11	1.3.2.1
+++ phys_pager.c	2000/12/05 20:13:25
@@ -83,7 +83,7 @@
 		 * Allocate object and associate it with the pager.
 		 */
 		object = vm_object_allocate(OBJT_PHYS,
-			OFF_TO_IDX(foff + size));
+			OFF_TO_IDX(foff + PAGE_MASK + size));
 		object->handle = handle;
 		TAILQ_INSERT_TAIL(&phys_pager_object_list, object,
 		    pager_object_list);

Tue, 5 Dec 2000 13:04:45 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: Need help with phys backed shm segments (Postgresql+FreeBSD).
Alfred,

do you have any numbers with and without your patch ?
I mean performance. You may use pg_check utility.

	Oleg
On Tue, 5 Dec 2000, Alfred Perlstein wrote:

> Date: Tue, 5 Dec 2000 13:04:45 -0800
> From: Alfred Perlstein <bright@wintelcom.net>
> To: Tom Lane <tgl@sss.pgh.pa.us>
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Need help with phys backed shm segments (Postgresql+FreeBSD).
> 
> * Alfred Perlstein <bright@wintelcom.net> [001205 12:30] wrote:
> > * Tom Lane <tgl@sss.pgh.pa.us> [001205 08:37] wrote:
> > > BTW, I just remembered that in 7.0.*, the SLocks that are managed by
> > > SpinAcquire() all live in their own little shm segment.  On a machine
> > > where slock_t is char, it'd likely only amount to 128 bytes or so.
> > > Maybe you are seeing some bug in FreeBSD's handling of tiny shm
> > > segments?
> > 
> > Good call, i think I found it! :)
> 
> Here's the patch I'm using on FreeBSD, it seems to work, if any
> other FreeBSD'ers want to try it out, just apply the patch:
> cd /usr/src/sys/vm ; patch < patchfile
> 
> and recompile and boot with a new kernel, then do this:
> 
> sysctl -w kern.ipc.shm_use_phys=1
> 
> or add:
> kern.ipc.shm_use_phys=1 
> to /etc/sysctl.conf
> 
> Let me know if it works.
> 
> thanks,
> -Alfred
> 
> Index: phys_pager.c
> ===================================================================
> RCS file: /home/ncvs/src/sys/vm/phys_pager.c,v
> retrieving revision 1.3.2.1
> diff -u -u -r1.3.2.1 phys_pager.c
> --- phys_pager.c	2000/08/04 22:31:11	1.3.2.1
> +++ phys_pager.c	2000/12/05 20:13:25
> @@ -83,7 +83,7 @@
>  		 * Allocate object and associate it with the pager.
>  		 */
>  		object = vm_object_allocate(OBJT_PHYS,
> -			OFF_TO_IDX(foff + size));
> +			OFF_TO_IDX(foff + PAGE_MASK + size));
>  		object->handle = handle;
>  		TAILQ_INSERT_TAIL(&phys_pager_object_list, object,
>  		    pager_object_list);
> 

_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83


Wed, 6 Dec 2000 00:20:52 +0300 (GMT)
Oleg Bartunov <oleg@sai.msu.su>
Re: Need help with phys backed shm segments (Postgresql+FreeBSD).
Just as interesting

On Tue, 5 Dec 2000, Alfred Perlstein wrote:

> * Alfred Perlstein <bright@wintelcom.net> [001205 12:30] wrote:
> > * Tom Lane <tgl@sss.pgh.pa.us> [001205 08:37] wrote:
> > > BTW, I just remembered that in 7.0.*, the SLocks that are managed by
> > > SpinAcquire() all live in their own little shm segment.  On a machine
> > > where slock_t is char, it'd likely only amount to 128 bytes or so.
> > > Maybe you are seeing some bug in FreeBSD's handling of tiny shm
> > > segments?
> >
> > Good call, i think I found it! :)
>
> Here's the patch I'm using on FreeBSD, it seems to work, if any
> other FreeBSD'ers want to try it out, just apply the patch:
> cd /usr/src/sys/vm ; patch < patchfile
>
> and recompile and boot with a new kernel, then do this:
>
> sysctl -w kern.ipc.shm_use_phys=1
>
> or add:
> kern.ipc.shm_use_phys=1
> to /etc/sysctl.conf
>
> Let me know if it works.
>
> thanks,
> -Alfred
>
> Index: phys_pager.c
> ===================================================================
> RCS file: /home/ncvs/src/sys/vm/phys_pager.c,v
> retrieving revision 1.3.2.1
> diff -u -u -r1.3.2.1 phys_pager.c
> --- phys_pager.c	2000/08/04 22:31:11	1.3.2.1
> +++ phys_pager.c	2000/12/05 20:13:25
> @@ -83,7 +83,7 @@
>  		 * Allocate object and associate it with the pager.
>  		 */
>  		object = vm_object_allocate(OBJT_PHYS,
> -			OFF_TO_IDX(foff + size));
> +			OFF_TO_IDX(foff + PAGE_MASK + size));
>  		object->handle = handle;
>  		TAILQ_INSERT_TAIL(&phys_pager_object_list, object,
>  		    pager_object_list);
>
>

Randy Jonasz
Software Engineer
Click2net Inc.
Web:  http://www.click2net.com
Phone: (905) 271-3550

"You cannot possibly pay a philosopher what he's worth,
but try your best" -- Aristotle


Tue, 5 Dec 2000 16:36:25 -0500 (EST)
Randy Jonasz <rjonasz@click2net.com>
Re: Need help with phys backed shm segments (Postgresql+FreeBSD).
* Oleg Bartunov <oleg@sai.msu.su> [001205 13:33] wrote:
> Alfred,
> 
> do you have any numbers with and without your patch ?
> I mean performance. You may use pg_check utility.

Er, I just made the patch a couple of hours ago, and I'm also
dealing with some other FreeBSD issues right now.  I will report
on it as soon as I can.

Theoretically You'll only see performance gains when doing fork(),
the real intent here is to allow for giant segments, without
kern.ipc.shm_use_phys=1 running let's say 768meg (out of 1gig)
shared memory segments will probably cause performance problems
because of the amount of swap structures needed per-process to
manage swappable segments.

I'm going to be enabling this on one of our boxes and see if it
makes a noticeable difference.  I'll let you guys know.

> > Date: Tue, 5 Dec 2000 13:04:45 -0800
> > From: Alfred Perlstein <bright@wintelcom.net>
> > To: Tom Lane <tgl@sss.pgh.pa.us>
> > Cc: pgsql-hackers@postgresql.org
> > Subject: Re: [HACKERS] Need help with phys backed shm segments (Postgresql+FreeBSD).
> > 
> > Here's the patch I'm using on FreeBSD, it seems to work, if any
> > other FreeBSD'ers want to try it out, just apply the patch:
> > cd /usr/src/sys/vm ; patch < patchfile
> > 
> > and recompile and boot with a new kernel, then do this:
> > 
> > sysctl -w kern.ipc.shm_use_phys=1
> > 
> > or add:
> > kern.ipc.shm_use_phys=1 
> > to /etc/sysctl.conf
> > 
> > Let me know if it works.
> > 
> > thanks,
> > -Alfred

Tue, 5 Dec 2000 14:52:33 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: Need help with phys backed shm segments (Postgresql+FreeBSD).