I'm trying to sort out the documentation regarding the SysV IPC settings,
but I better understand them myself first. :)

We use three shared-memory segments: One is for the spin locks and is of
negligible size (144 bytes currently). The other two I don't know, but one
of them seems to be sized about 550kB + -B * BLCKSZ

My kernel has the following interesting-looking shared memory settings:

SHMMAX	-- max size per segment. Apparently must be >= 550kB + -B * BLCKSZ
SHMMNI	-- max number of segments system wide, better be >= 3
SHMSEG	-- max number of segments per process, also better be >= 3
SHMALL	-- max number of pages for shmem system wide. This seems to be
           fixed at some theoretical amount.

The most promising thing to promote here is evidently to raise SHMMAX.


For semaphores, we're using ceil(-N % 16) sets of 16 semaphores. In my
kernel I see:

SEMMNI	-- max number of semaphore "identifiers" (=sets?)
SEMMSL	-- max semaphores per set, this is explained in storage/proc.h
SEMMNS	-- max semaphores in system

So, SEMMNI and SEMMNS seem to be the most promising settings to change.

Is there any noteworthy relevance of some of the other parameters? I see
FAQ_BSDI talks about SEMUME and SEMMNU.


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Thu, 20 Jul 2000 00:46:17 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
About these IPC parameters
Peter Eisentraut <peter_e@gmx.net> writes:
> We use three shared-memory segments: One is for the spin locks and is of
> negligible size (144 bytes currently). The other two I don't know, but one
> of them seems to be sized about 550kB + -B * BLCKSZ

The shmem sizes depend on both -B and -N, but the dependence on -B is
much stronger.  Obviously there's 8K per -B for the buffer itself,
and there's also some allowance for hashtable entries for the buffer
indexing tables.  The -N number drives the size of the PROC table plus
some hashtables --- but a PROC entry isn't very big.

I believe there's no really fundamental reason why we use three shmem
segments and not just one.  I've toyed with the idea of trying to
combine them, but not done anything about it yet...

> My kernel has the following interesting-looking shared memory settings:

FWIW, HPUX does not have SHMALL --- and since HPUX began life as SysV
I would imagine a lot of other SysV derivatives don't either.  The
relevant parameters here seem to be

SEMA            Enable Sys V Semaphores
SEMAEM          Max Value for Adjust on Exit Semaphores
SEMMAP          Max Number of Semaphore Map Entries
SEMMNI          Number of Semaphore Identifiers
SEMMNS          Max Number of Semaphores
SEMMNU          Number of Semaphore Undo Structures
SEMUME          Semaphore Undo Entries per Process
SEMVMX          Semaphore Maximum Value
SHMEM           Enable Sys V Shared Memory
SHMMAX          Max Shared Mem Segment (bytes)
SHMMNI          Number of Shared Memory Identifiers
SHMSEG          Shared Memory Segments per Process

Other than shooting yourself in the foot by having SEMA or SHMEM be
0 (OFF), it looks like the parameters that could need raising on this
platform would be SEMMAP, SEMMNI, SEMMNS, SHMMAX.

> Is there any noteworthy relevance of some of the other parameters? I see
> FAQ_BSDI talks about SEMUME and SEMMNU.

AFAIK we don't use semaphore undo, so those are red herrings.

			regards, tom lane

Thu, 20 Jul 2000 00:50:56 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: About these IPC parameters 
On Thu, 20 Jul 2000, Tom Lane wrote:

> Peter Eisentraut <peter_e@gmx.net> writes:
> > We use three shared-memory segments: One is for the spin locks and is of
> > negligible size (144 bytes currently). The other two I don't know, but one
> > of them seems to be sized about 550kB + -B * BLCKSZ
> 
> The shmem sizes depend on both -B and -N, but the dependence on -B is
> much stronger.  Obviously there's 8K per -B for the buffer itself,
> and there's also some allowance for hashtable entries for the buffer
> indexing tables.  The -N number drives the size of the PROC table plus
> some hashtables --- but a PROC entry isn't very big.
> 
> I believe there's no really fundamental reason why we use three shmem
> segments and not just one.  I've toyed with the idea of trying to
> combine them, but not done anything about it yet...
> 
> > My kernel has the following interesting-looking shared memory settings:
> 
> FWIW, HPUX does not have SHMALL --- and since HPUX began life as SysV
> I would imagine a lot of other SysV derivatives don't either.  The
> relevant parameters here seem to be
> 
> SEMA            Enable Sys V Semaphores
> SEMAEM          Max Value for Adjust on Exit Semaphores
> SEMMAP          Max Number of Semaphore Map Entries
> SEMMNI          Number of Semaphore Identifiers
> SEMMNS          Max Number of Semaphores
> SEMMNU          Number of Semaphore Undo Structures
> SEMUME          Semaphore Undo Entries per Process
> SEMVMX          Semaphore Maximum Value
> SHMEM           Enable Sys V Shared Memory
> SHMMAX          Max Shared Mem Segment (bytes)
> SHMMNI          Number of Shared Memory Identifiers
> SHMSEG          Shared Memory Segments per Process
> 
> Other than shooting yourself in the foot by having SEMA or SHMEM be
> 0 (OFF), it looks like the parameters that could need raising on this
> platform would be SEMMAP, SEMMNI, SEMMNS, SHMMAX.
> 
> > Is there any noteworthy relevance of some of the other parameters? I see
> > FAQ_BSDI talks about SEMUME and SEMMNU.
> 
> AFAIK we don't use semaphore undo, so those are red herrings.

First off, this might be something we need a whole seperate FAQ for, since
I think the concepts are pretty much common across the various OSs?

for instance,  under FreeBSD, I have it set right now as:

====
options         SYSVSHM
options         SHMMAXPGS=4096
options         SHMSEG=256

options         SYSVSEM
options         SEMMNI=256
options         SEMMNS=512
options         SEMMNU=256
options         SEMMAP=256

options         SYSVMSG                 #SYSV-style message queues
====

To run three postmasters, one with '-B 256 -N 128', and the other two just
with '-N 16' ... the thing that I just don't get is how the settings ni my
kernel apply, and trying to find any info on taht is like pulling teeth :(
For instance, I'm allowing for up to 160 clients to connect, max .. does
that make for one semaphore identifier per client, so I need SEMMNI >=
160?  Or ... ?

I grab'd this off a Sun site dealing with Solaris, but it might also be of
aid:


  Name    Default Max             Brief Description
  ------  ------- -------------- -------------------------------------

  semmap  10      2147483647      Number of entries in semaphore map

  semmni  10      65535           Number of semaphore sets (identifiers)

  semmns  60      2147483647      Number of semaphores in the system
                  65535 (usage)

  semmnu  30      2147483647      Number of "undo" structures in the system

  semmsl  25      2147483647      Max number of semaphores, per semaphore id
                  65535 (usage)

  semopm  10      2147483647      Max number of operations, per semaphore call

  semume  10      2147483647      Max number of "undo" entries, per process

  semusz  96      *see below*     Size in bytes of "undo" structure

  semvmx  32767   2147483647      Semaphore maximum value
                  65535 (usage)

  semaem  16384   2147483647      Adjust on exit maximum value
                  32767 (usage)

  Detailed Descriptions
  ---------------------

  semmap

  Defines the size of the semaphore resource map;  each block of available,
  contiguous semaphores requires one entry in this map.  This is the pool from
  which semget(2) acquires semaphore sets.

  When a semaphore set is removed (deleted), if the block of semaphores to be
  freed is adjacent to a block of semaphores already in the resource map, the
  semaphores in the set being removed are added to the existing map entry;
  no new map entry is required.  If the semaphores in the removed set are not
  adjacent to those in an existing map entry, then a new map entry is required
  to track these semaphores;  if there are no more map entries available, the
  system has to discard an entry, 'permanently' losing a block of semaphores
  (permanence is relative;  a reboot fixes the problem).  If this should occur,
  a WARNING will be generated, the text of which will be something like
  "rmallocmap: rmap overflow, lost ...".   The end result is that a user could
  later get ENOSPC errors from semget(2) even though it doesn't look like all
  the semaphores are allocated.

  semmni

  Defines the number of semaphore sets (identifiers), system wide.  Every
  semaphore set in the system has a unique indentifier and control structure.
  The system pre-allocates kernel memory for semmni control structures;  each
  control structure is 84 bytes.  If no more identifiers are available,
  semget(2) returns ENOSPC.

  Attempting to set semmni to a value greater than 65535 will result in
  generation of a WARNING, and the value will be set to 65535.

  semmns

  Defines the number of semaphores in the system;  16 bytes of kernel memory is
  pre-allocated for each semaphores.  If there is not a large enough block of
  contiguous semaphores in the resource map (see semmap) to satisfy the request,
  semget(2) returns ENOSPC.

  Fragmentation of the semaphore map will result in ENOSPC errors, even though
  there may appear to be ample free semaphores.  Despite attempts by the system
  to merge free sets (see semmap), the size of the clusters of free semaphores
  generally decreases over time.  For this reason, semmns frequently must be set
  higher than the actual number of semaphores required.

  semmnu

  Defines the number of semaphore undo structures in the system.  semusz (see
  below) bytes of kernel memory are pre-allocated for each undo structure; one
  undo structure is required for every process for which undo information must
  be recorded.  semop() will return ENOSPC if it is requested to record undo
  information and there are no undo structures available.

  semmsl

  Limits the number of semaphores that can be created for a single semaphore id.
  If semget(2) returns EINVAL, this limit should be increased.  This parameter
  is only used to validate the argument passed to semget(2).  Logically, it
  should be less than or equal to semmns (see above).  Setting semmsl too high
  might allow a few identifiers to hog all the semaphores in the system.

  semopm

  Limits the number of operations that are allowed in a single semop(2) call.
  If semop(2) returns E2BIG, this limit should be increased.  This parameter is
  only used to validate the argument passed to semop(2).

  semume

  Limits the number of undo records that can exist for a process.  If semop(2)
  returns EINVAL, this limit should be increased.  In addition to its use in
  validating arguments to semop(2), this parameter is used to calculate the
  value of semusz (see below).

  semusz

  Defines the size of the semaphore undo structure.  Any attempt to modify this
  parameter directly will be ignored;  semusz is always calculated based upon
  the value of semume (see above);  semusz = 8 * (semume + 2).

  semvmx

  Limits the maximum value of a semaphore.  Due to the interaction with undo
  structures and semaem (see below), this tuneable should not be increased
  beyond its default value of 32767, unless you can guarantee that SEM_UNDO is
  never and will never be used.  It can be safely reduced, but doing so provides
  no savings.

  semaem

  Limits the maximum value of an adjust-on-exit undo element.  No system
  resources are allocated based on this value.


Thu, 20 Jul 2000 12:50:59 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: About these IPC parameters 
Tom Lane writes:

> Other than shooting yourself in the foot by having SEMA or SHMEM be
> 0 (OFF), it looks like the parameters that could need raising on this
> platform would be SEMMAP, SEMMNI, SEMMNS, SHMMAX.

Can you give me a couple of lines on how to change them (e.g., edit some
file and reboot) and perhaps a comment whether some of these tend to be
too low in the default configuration?


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Fri, 21 Jul 2000 20:29:53 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: About these IPC parameters 
The Hermit Hacker writes:

> First off, this might be something we need a whole seperate FAQ for, since
> I think the concepts are pretty much common across the various OSs?

Working on that...

> for instance,  under FreeBSD, I have it set right now as:

Is SysV IPC still off in stock FreeBSD kernels?

> For instance, I'm allowing for up to 160 clients to connect, max .. does
> that make for one semaphore identifier per client, so I need SEMMNI >=
> 160?  Or ... ?

SEMMNI = 10

> I grab'd this off a Sun site dealing with Solaris, but it might also be of
> aid:

Yes, that helped me a lot. I wrote a section about all this for the Admin
Guide. It should pop up in the next day or so.


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Fri, 21 Jul 2000 20:30:07 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: About these IPC parameters 
On Fri, 21 Jul 2000, Peter Eisentraut wrote:

> The Hermit Hacker writes:
> 
> > First off, this might be something we need a whole seperate FAQ for, since
> > I think the concepts are pretty much common across the various OSs?
> 
> Working on that...
> 
> > for instance,  under FreeBSD, I have it set right now as:
> 
> Is SysV IPC still off in stock FreeBSD kernels?

Checking the GENERIC config file, it is enabled by default now ...

> > For instance, I'm allowing for up to 160 clients to connect, max .. does
> > that make for one semaphore identifier per client, so I need SEMMNI >=
> > 160?  Or ... ?
> 
> SEMMNI = 10

Ouch ... so I'm running a bit high on values :)



Fri, 21 Jul 2000 15:34:15 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: About these IPC parameters 
At 3:34 PM -0300 7/21/00, The Hermit Hacker wrote:
>On Fri, 21 Jul 2000, Peter Eisentraut wrote:
> > Is SysV IPC still off in stock FreeBSD kernels?
>
>Checking the GENERIC config file, it is enabled by default now ...

It's on by default in NetBSD also.


Signature failed Preliminary Design Review.
Feasibility of a new signature is currently being evaluated.
h.b.hotz@jpl.nasa.gov, or hbhotz@oxy.edu

Fri, 21 Jul 2000 11:47:43 -0700
"Henry B. Hotz" <hotz@jpl.nasa.gov>
Re: About these IPC parameters
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> Other than shooting yourself in the foot by having SEMA or SHMEM be
>> 0 (OFF), it looks like the parameters that could need raising on this
>> platform would be SEMMAP, SEMMNI, SEMMNS, SHMMAX.

> Can you give me a couple of lines on how to change them (e.g., edit some
> file and reboot) and perhaps a comment whether some of these tend to be
> too low in the default configuration?

On HPUX the usual advice is "use SAM" (System Administration Manager).
It's a pretty decent point-and-drool tool.  You go into Kernel
Configuration / Configurable Parameters and double-click on the items
you don't like in the resulting list.  When you're done, hit Create
A New Kernel.  SAM used to have some memorable deficiencies (I still
recall that when I first used it, if you let it create a user's home
directory it would leave /users world-writable...) but it seems reliable
enough in HPUX 10.

If I've found the right file to look at, the factory defaults are

semmni              64          Number of Semaphore Identifiers       
semmns              128         Max Number of Semaphores           
shmmax              0x4000000   Max Shared Mem Segment (bytes) 
shmmni              200         Number of Shared Memory Identifiers 
shmseg              120         Shared Memory Segments per Process 

so you'd need to raise these to run a big installation (more than,
say, 100 backends) but not for a default-sized setup.

What I tend to want to raise are not the IPC parameters but

maxdsiz         0x04000000     Max Data Segment Size (bytes)        
maxssiz         0x00800000     Max Stack Segment Size (bytes)    
maxfiles        60             Soft File Limit per Process           
maxfiles_lim    1024           Hard File Limit per Process           
maxuprc         75             Max Number of User Processes (per user)
maxusers        32             Value of MAXUSERS macro
nfile           (16*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY))         Max Number of Open Files
ninode          ((NPROC+16+MAXUSERS)+32+(2*NPTY)+(10*NUM_CLIENTS))      Max Number of Open Inodes

In particular, the default maxuprc would definitely be a problem for
running a lot of backends, and you'd likely start running into nfile
or ninode limits too.

			regards, tom lane

Fri, 21 Jul 2000 14:59:26 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: About these IPC parameters 
> So, SEMMNI and SEMMNS seem to be the most promising settings to change.
> 
> Is there any noteworthy relevance of some of the other parameters? I see
> FAQ_BSDI talks about SEMUME and SEMMNU.

I wrote FAQ_BSDI because it was not trivial to figure out how to modify
those parameters.  I figured other OS's either don't need to do it, or
have an easier way of doing it.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 27 Jul 2000 09:06:32 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: About these IPC parameters

Peter ...

	Here is the 'latest and greatest' NOTES that one of the FreeBSD
guys has been working on for shared memory/semaphores ... not sure if it
helps or not, but I believe it was you that was working on "organizing
this"?

=============================
#####################################################################
# SYSV IPC KERNEL PARAMETERS
#
# Maximum number of entries in a semaphore map.
options         SEMMAP=31

# Maximum number of System V semaphores that can be used on the system at
# one time.
options         SEMMNI=11

# Total number of semaphores system wide
options         SEMMNS=61

# Total number of undo structures in system
options         SEMMNU=31

# Maximum number of System V semaphores that can be used by a single
process
# at one time.
options         SEMMSL=61

# Maximum number of operations that can be outstanding on a single System
V
# semaphore at one time.
options         SEMOPM=101

# Maximum number of undo operations that can be outstanding on a single
# System V semaphore at one time.
options         SEMUME=11

# Maximum number of shared memory pages system wide.
options         SHMALL=1025

# Maximum size, in bytes, of a single System V shared memory region.
options         SHMMAX="(SHMMAXPGS*PAGE_SIZE+1)"
options         SHMMAXPGS=1025

# Minimum size, in bytes, of a single System V shared memory region.
options         SHMMIN=2

# Maximum number of shared memory regions that can be used on the system
# at one time.
options         SHMMNI=33

# Maximum number of System V shared memory regions that can be attached to
# a single process at one time.
options         SHMSEG=9
========================================

On Thu, 27 Jul 2000, Bruce Momjian wrote:

> > So, SEMMNI and SEMMNS seem to be the most promising settings to change.
> > 
> > Is there any noteworthy relevance of some of the other parameters? I see
> > FAQ_BSDI talks about SEMUME and SEMMNU.
> 
> I wrote FAQ_BSDI because it was not trivial to figure out how to modify
> those parameters.  I figured other OS's either don't need to do it, or
> have an easier way of doing it.
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Thu, 27 Jul 2000 12:27:00 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: About these IPC parameters
The IPC killer is that different OS's have different methods for
changing kernel parameters, and some have different kernel parameter
names.

> Peter Eisentraut <peter_e@gmx.net> writes:
> > Tom Lane writes:
> >> Other than shooting yourself in the foot by having SEMA or SHMEM be
> >> 0 (OFF), it looks like the parameters that could need raising on this
> >> platform would be SEMMAP, SEMMNI, SEMMNS, SHMMAX.
> 
> > Can you give me a couple of lines on how to change them (e.g., edit some
> > file and reboot) and perhaps a comment whether some of these tend to be
> > too low in the default configuration?
> 
> On HPUX the usual advice is "use SAM" (System Administration Manager).
> It's a pretty decent point-and-drool tool.  You go into Kernel
> Configuration / Configurable Parameters and double-click on the items
> you don't like in the resulting list.  When you're done, hit Create
> A New Kernel.  SAM used to have some memorable deficiencies (I still
> recall that when I first used it, if you let it create a user's home
> directory it would leave /users world-writable...) but it seems reliable
> enough in HPUX 10.
> 
> If I've found the right file to look at, the factory defaults are
> 
> semmni              64          Number of Semaphore Identifiers       
> semmns              128         Max Number of Semaphores           
> shmmax              0x4000000   Max Shared Mem Segment (bytes) 
> shmmni              200         Number of Shared Memory Identifiers 
> shmseg              120         Shared Memory Segments per Process 
> 
> so you'd need to raise these to run a big installation (more than,
> say, 100 backends) but not for a default-sized setup.
> 
> What I tend to want to raise are not the IPC parameters but
> 
> maxdsiz         0x04000000     Max Data Segment Size (bytes)        
> maxssiz         0x00800000     Max Stack Segment Size (bytes)    
> maxfiles        60             Soft File Limit per Process           
> maxfiles_lim    1024           Hard File Limit per Process           
> maxuprc         75             Max Number of User Processes (per user)
> maxusers        32             Value of MAXUSERS macro
> nfile           (16*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY))         Max Number of Open Files
> ninode          ((NPROC+16+MAXUSERS)+32+(2*NPTY)+(10*NUM_CLIENTS))      Max Number of Open Inodes
> 
> In particular, the default maxuprc would definitely be a problem for
> running a lot of backends, and you'd likely start running into nfile
> or ninode limits too.
> 
> 			regards, tom lane
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 27 Jul 2000 14:34:43 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: About these IPC parameters