[ Send to hackers]

> I'd be willing to consider using mmap as a compile-time option if it
> can be shown to be a substantial performance win where it's available.
> (I suspect that's a very big "if".)  If it's not a substantial win,
> I don't think we should accept the change --- the portability risks and
> testing/maintenance costs loom too large for me.
> 

I was considering it because you can use a much larger amount of shared
memory without reconfiguring the kernel.

> BTW, how exactly is mmap a substitute for SysV shared memory?  AFAICT
> it's only defined to map a disk file into your address space, not to
> allow a shared memory region to be set up that's independent of any
> disk file.

It allows no backing store on disk.  It is the BSD solution to SysV
share memory.  Here are all the BSDi flags:

     MAP_ANON    Map anonymous memory not associated with any specific file.
                 The file descriptor used for creating MAP_ANON must be -1.
                 The offset parameter is ignored.

     MAP_FIXED   Do not permit the system to select a different address than
                 the one specified.  If the specified address cannot be used,
                 mmap will fail.  If MAP_FIXED is specified, addr must be a
                 multiple of the pagesize.  Use of this option is discouraged.

     MAP_PRIVATE
                 Modifications are private.

     MAP_SHARED  Modifications are shared.

We would use MAP_ANON|MAP_SHARED I guess.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 25 Feb 2001 21:00:41 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
[PATCHES] A patch for xlog.c
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> It allows no backing store on disk.  It is the BSD solution to SysV
> share memory.  Here are all the BSDi flags:

>      MAP_ANON    Map anonymous memory not associated with any specific file.
>                  The file descriptor used for creating MAP_ANON must be -1.
>                  The offset parameter is ignored.

Hmm.  Now that I read down to the "nonstandard extensions" part of the
HPUX man page for mmap(), I find

     If MAP_ANONYMOUS is set in flags:

          o    A new memory region is created and initialized to all zeros.
               This memory region can be shared only with descendants of
               the current process.

While I've said before that I don't think it's really necessary for
processes that aren't children of the postmaster to access the shared
memory, I'm not sure that I want to go over to a mechanism that makes it
*impossible* for that to be done.  Especially not if the only motivation
is to avoid having to configure the kernel's shared memory settings.

Besides, what makes you think there's not a limit on the size of shmem
allocatable via mmap()?

			regards, tom lane

Sun, 25 Feb 2001 23:28:46 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] A patch for xlog.c 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > It allows no backing store on disk.  It is the BSD solution to SysV
> > share memory.  Here are all the BSDi flags:
> 
> >      MAP_ANON    Map anonymous memory not associated with any specific file.
> >                  The file descriptor used for creating MAP_ANON must be -1.
> >                  The offset parameter is ignored.
> 
> Hmm.  Now that I read down to the "nonstandard extensions" part of the
> HPUX man page for mmap(), I find
> 
>      If MAP_ANONYMOUS is set in flags:
> 
>           o    A new memory region is created and initialized to all zeros.
>                This memory region can be shared only with descendants of
>                the current process.
> 
> While I've said before that I don't think it's really necessary for
> processes that aren't children of the postmaster to access the shared
> memory, I'm not sure that I want to go over to a mechanism that makes it
> *impossible* for that to be done.  Especially not if the only motivation
> is to avoid having to configure the kernel's shared memory settings.

Agreed.  It would make it impossible and a possible limitation.

> Besides, what makes you think there's not a limit on the size of shmem
> allocatable via mmap()?

I figured mmap() was different than SysV becuase mmap() is file based.

I have had this item on the TODO list for a while:

	* Use mmap() rather than SYSV shared memory(?)

Should I remove it?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 25 Feb 2001 23:48:11 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [PATCHES] A patch for xlog.c
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I have had this item on the TODO list for a while:
> 	* Use mmap() rather than SYSV shared memory(?)
> Should I remove it?

It's fine as long as it's got that question mark on it ;-).
I don't say we *shouldn't* do this, I'm just raising questions
that would need to be answered.

			regards, tom lane

Sun, 25 Feb 2001 23:58:45 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] A patch for xlog.c 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I have had this item on the TODO list for a while:
> > 	* Use mmap() rather than SYSV shared memory(?)
> > Should I remove it?
> 
> It's fine as long as it's got that question mark on it ;-).
> I don't say we *shouldn't* do this, I'm just raising questions
> that would need to be answered.

Yea, it is one of those question mark things.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 25 Feb 2001 23:59:48 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [PATCHES] A patch for xlog.c
On Sun, Feb 25, 2001 at 11:28:46PM -0500, Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > It allows no backing store on disk.  

I.e. it allows you to map memory without an associated inode; the memory
may still be swapped.  Of course, there is no problem with mapping an 
inode too, so that unrelated processes can join in.  Solarix has a flag
to pin the shared pages in RAM so they can't be swapped out.

> > It is the BSD solution to SysV
> > share memory.  Here are all the BSDi flags:
> 
> >      MAP_ANON    Map anonymous memory not associated with any specific
> >                  file.  The file descriptor used for creating MAP_ANON
> >                  must be -1.  The offset parameter is ignored.
> 
> Hmm.  Now that I read down to the "nonstandard extensions" part of the
> HPUX man page for mmap(), I find
> 
>      If MAP_ANONYMOUS is set in flags:
> 
>           o    A new memory region is created and initialized to all zeros.
>                This memory region can be shared only with descendants of
>                the current process.

This is supported on Linux and BSD, but not on Solarix 7.  It's not 
necessary; you can just map /dev/zero on SysV systems that don't 
have MAP_ANON.

> While I've said before that I don't think it's really necessary for
> processes that aren't children of the postmaster to access the shared
> memory, I'm not sure that I want to go over to a mechanism that makes it
> *impossible* for that to be done.  Especially not if the only motivation
> is to avoid having to configure the kernel's shared memory settings.

There are enormous advantages to avoiding the need to configure kernel 
settings.  It makes PG a better citizen.  PG is much easier to drop in 
and use if you don't need attention from the IT department.

But I don't know of any reason to avoid mapping an actual inode,
so using mmap doesn't necessarily mean giving up sharing among
unrelated processes.

> Besides, what makes you think there's not a limit on the size of shmem
> allocatable via mmap()?

I've never seen any mmap limit documented.  Since mmap() is how 
everybody implements shared libraries, such a limit would be equivalent 
to a limit on how much/many shared libraries are used.  mmap() with 
MAP_ANONYMOUS (or its SysV /dev/zero equivalent) is a common, modern 
way to get raw storage for malloc(), so such a limit would be a limit
on malloc() too.

The mmap architecture comes to us from the Mach microkernel memory
manager, backported into BSD and then copied widely.  Since it was
the fundamental mechanism for all memory operations in Mach, arbitrary
limits would make no sense.  That it worked so well is the reason it 
was copied everywhere else, so adding arbitrary limits while copying 
it would be silly.  I don't think we'll see any systems like that.

Nathan Myers
ncm@zembu.com

Mon, 26 Feb 2001 00:21:25 -0800
ncm@zembu.com (Nathan Myers)
Re: Re: [PATCHES] A patch for xlog.c
On Mon, 26 Feb 2001, Nathan Myers wrote:

> > While I've said before that I don't think it's really necessary for
> > processes that aren't children of the postmaster to access the shared
> > memory, I'm not sure that I want to go over to a mechanism that makes it
> > *impossible* for that to be done.  Especially not if the only motivation
> > is to avoid having to configure the kernel's shared memory settings.
>
> There are enormous advantages to avoiding the need to configure kernel
> settings.  It makes PG a better citizen.  PG is much easier to drop in
> and use if you don't need attention from the IT department.

Is there a reason why Oracle still uses shared memory and hasn't moved to
mmap()?  Are there advantages to it that we aren't seeing, or is oracle
just too much of a mahemouth for that sort of overhaul?  Don't go with the
quick answer either ...

> > Besides, what makes you think there's not a limit on the size of shmem
> > allocatable via mmap()?
>
> I've never seen any mmap limit documented.  Since mmap() is how
> everybody implements shared libraries, such a limit would be equivalent
> to a limit on how much/many shared libraries are used.

There are/will be limits based on how an admin sets his/her per user
datasize limits on their OS ...



Mon, 26 Feb 2001 08:37:35 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re: [PATCHES] A patch for xlog.c
> On Sun, Feb 25, 2001 at 11:28:46PM -0500, Tom Lane wrote:
> > Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > > It allows no backing store on disk.  
> 
> I.e. it allows you to map memory without an associated inode; the memory
> may still be swapped.  Of course, there is no problem with mapping an 
> inode too, so that unrelated processes can join in.  Solarix has a flag
> to pin the shared pages in RAM so they can't be swapped out.

We don't want to generate i/o to disk just for shared memory
modifications, that is why we can't use a disk file.

> 
> > > It is the BSD solution to SysV
> > > share memory.  Here are all the BSDi flags:
> > 
> > >      MAP_ANON    Map anonymous memory not associated with any specific
> > >                  file.  The file descriptor used for creating MAP_ANON
> > >                  must be -1.  The offset parameter is ignored.
> > 
> > Hmm.  Now that I read down to the "nonstandard extensions" part of the
> > HPUX man page for mmap(), I find
> > 
> >      If MAP_ANONYMOUS is set in flags:
> > 
> >           o    A new memory region is created and initialized to all zeros.
> >                This memory region can be shared only with descendants of
> >                the current process.
> 
> This is supported on Linux and BSD, but not on Solarix 7.  It's not 
> necessary; you can just map /dev/zero on SysV systems that don't 
> have MAP_ANON.

Oh, really.  Yes, I have seen people do that.

> > While I've said before that I don't think it's really necessary for
> > processes that aren't children of the postmaster to access the shared
> > memory, I'm not sure that I want to go over to a mechanism that makes it
> > *impossible* for that to be done.  Especially not if the only motivation
> > is to avoid having to configure the kernel's shared memory settings.
> 
> There are enormous advantages to avoiding the need to configure kernel 
> settings.  It makes PG a better citizen.  PG is much easier to drop in 
> and use if you don't need attention from the IT department.

One big advantage is that mmap() removes itself when all processes using
it exit, while SysV stays around and has to be cleaned up manually in
some cases.

> But I don't know of any reason to avoid mapping an actual inode,
> so using mmap doesn't necessarily mean giving up sharing among
> unrelated processes.

See above.

> 
> > Besides, what makes you think there's not a limit on the size of shmem
> > allocatable via mmap()?
> 
> I've never seen any mmap limit documented.  Since mmap() is how 
> everybody implements shared libraries, such a limit would be equivalent 
> to a limit on how much/many shared libraries are used.  mmap() with 
> MAP_ANONYMOUS (or its SysV /dev/zero equivalent) is a common, modern 
> way to get raw storage for malloc(), so such a limit would be a limit
> on malloc() too.
> 
> The mmap architecture comes to us from the Mach microkernel memory
> manager, backported into BSD and then copied widely.  Since it was
> the fundamental mechanism for all memory operations in Mach, arbitrary
> limits would make no sense.  That it worked so well is the reason it 
> was copied everywhere else, so adding arbitrary limits while copying 
> it would be silly.  I don't think we'll see any systems like that.

This is encouraging.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 26 Feb 2001 11:20:19 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: [PATCHES] A patch for xlog.c
ncm@zembu.com (Nathan Myers) writes:
> This is supported on Linux and BSD, but not on Solarix 7.  It's not 
> necessary; you can just map /dev/zero on SysV systems that don't 
> have MAP_ANON.

HPUX says:

     The mmap() function is supported for regular files.  Support for any
     other type of file is unspecified.

> But I don't know of any reason to avoid mapping an actual inode,

How about wasted I/O due to the kernel thinking it needs to reflect
writes to the memory region back out to the underlying file?

> Since mmap() is how everybody implements shared libraries,

Now *there's* a sweeping generalization.  Documentation of this
claim, please?

> The mmap architecture comes to us from the Mach microkernel memory
> manager, backported into BSD and then copied widely.

If everyone copied the Mach implementation, why is it they don't even
agree on the spellings of the user-visible flags?


This looks a lot like exchanging the devil we know (SysV shmem) for a
devil we don't know.  Do I need to remind you about, for example, the
mmap bugs in early Linux releases?  (I still vividly remember having to
abandon mmap on a project a few years back that needed to be portable
to Linux.  Perhaps that colors my opinions here.)  I don't think the
problems with shmem are sufficiently large to justify venturing into
a whole new terra incognita of portability issues and kernel bugs.

			regards, tom lane

Mon, 26 Feb 2001 11:23:25 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: [PATCHES] A patch for xlog.c 
Tom Lane <tgl@sss.pgh.pa.us> writes:

> > Since mmap() is how everybody implements shared libraries,
> 
> Now *there's* a sweeping generalization.  Documentation of this
> claim, please?

I've seen a lot of shared library implementations (I used to be the
GNU binutils maintainer), and Nathan is approximately correct.  Most
ELF systems use a dynamic linker inherited from the original SVR4
implementation, which uses mmap.  You can see this by running strace
on an SVR4 system.  The *BSD and GNU dynamic linker implementations
are of course independently derived, but they use mmap too.

mmap is the natural way to implement ELF style shared libraries.  The
basic operation you have to do is to map the shared library into the
process memory space, and then to process a few relocations.  Mapping
the shared library in can be done either using mmap, or using
open/read/close.  For a large file, mmap is going to be much faster
than open/read/close, because it doesn't require actually reading the
file.

There are, of course, many non-ELF shared libraries implementations.
SVR3 does not use mmap.  SunOS does use mmap (SunOS shared libraries
were taken into SVR4 and the ELF standard).  I don't know offhand
about AIX, Digital Unix, or Windows.

mmap is standardized by the most recent version of POSIX.1.

Ian

26 Feb 2001 08:57:08 -0800
Ian Lance Taylor <ian@airs.com>
Re: Re: [PATCHES] A patch for xlog.c
Hello Tom,

Tuesday, February 27, 2001, 12:23:25 AM, you wrote:

TL> This looks a lot like exchanging the devil we know (SysV shmem) for a
TL> devil we don't know.  Do I need to remind you about, for example, the
TL> mmap bugs in early Linux releases?  (I still vividly remember having to
TL> abandon mmap on a project a few years back that needed to be portable
TL> to Linux.  Perhaps that colors my opinions here.)  I don't think the
TL> problems with shmem are sufficiently large to justify venturing into
TL> a whole new terra incognita of portability issues and kernel bugs.

TL>                         regards, tom lane

the only problem is because if we need to tune Postermaster to use
large buffer while system havn't so many SYSV shared memory, in many
systemes, we need to recompile OS kernel, this is a small problem to install
PGSQL to product environment.

-- 
Best regards,
XuYifeng



Tue, 27 Feb 2001 10:27:22 +0800
jamexu <jamexu@telekbird.com.cn>
Re[2]: Re: [PATCHES] A patch for xlog.c
On Tue, 27 Feb 2001, jamexu wrote:

> Hello Tom,
>
> Tuesday, February 27, 2001, 12:23:25 AM, you wrote:
>
> TL> This looks a lot like exchanging the devil we know (SysV shmem) for a
> TL> devil we don't know.  Do I need to remind you about, for example, the
> TL> mmap bugs in early Linux releases?  (I still vividly remember having to
> TL> abandon mmap on a project a few years back that needed to be portable
> TL> to Linux.  Perhaps that colors my opinions here.)  I don't think the
> TL> problems with shmem are sufficiently large to justify venturing into
> TL> a whole new terra incognita of portability issues and kernel bugs.
>
> TL>                         regards, tom lane
>
> the only problem is because if we need to tune Postermaster to use
> large buffer while system havn't so many SYSV shared memory, in many
> systemes, we need to recompile OS kernel, this is a small problem to install
> PGSQL to product environment.

What?  You don't automatically recompile your OS kernel when you build a
system in the first place??  First step on any OS install of FreeBSD is to
rid myself of the 'extras' that are in the generic kernel, and enable
SharedMemory (even if I'm not using PgSQL on that machine) ...



Mon, 26 Feb 2001 23:00:05 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re[2]: Re: [PATCHES] A patch for xlog.c
>> the only problem is because if we need to tune Postermaster to use
>> large buffer while system havn't so many SYSV shared memory, in many
>> systemes, we need to recompile OS kernel, this is a small problem to install
>> PGSQL to product environment.

Of course, if you haven't got mmap(), a recompile won't help ...

I'd be somewhat more enthusiastic about mmap if I thought we could
abandon the SysV shmem support completely, but I don't foresee that
happening for a long while yet.

			regards, tom lane

Mon, 26 Feb 2001 22:05:52 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re[2]: Re: [PATCHES] A patch for xlog.c 
Hello The,

Tuesday, February 27, 2001, 11:00:05 AM, you wrote:

THH> On Tue, 27 Feb 2001, jamexu wrote:

>> Hello Tom,
>>
>> Tuesday, February 27, 2001, 12:23:25 AM, you wrote:
>>
>> TL> This looks a lot like exchanging the devil we know (SysV shmem) for a
>> TL> devil we don't know.  Do I need to remind you about, for example, the
>> TL> mmap bugs in early Linux releases?  (I still vividly remember having to
>> TL> abandon mmap on a project a few years back that needed to be portable
>> TL> to Linux.  Perhaps that colors my opinions here.)  I don't think the
>> TL> problems with shmem are sufficiently large to justify venturing into
>> TL> a whole new terra incognita of portability issues and kernel bugs.
>>
>> TL>                         regards, tom lane
>>
>> the only problem is because if we need to tune Postermaster to use
>> large buffer while system havn't so many SYSV shared memory, in many
>> systemes, we need to recompile OS kernel, this is a small problem to install
>> PGSQL to product environment.

THH> What?  You don't automatically recompile your OS kernel when you build a
THH> system in the first place??  First step on any OS install of FreeBSD is to
THH> rid myself of the 'extras' that are in the generic kernel, and enable
THH> SharedMemory (even if I'm not using PgSQL on that machine) ...

heihei, why do you think users always using FreeBSD and not other
UNIX systemes?
your assume is false.

---
Xu Yifeng



Tue, 27 Feb 2001 11:18:34 +0800
jamexu <jamexu@telekbird.com.cn>
Re[3]: Re: [PATCHES] A patch for xlog.c
On Tue, 27 Feb 2001, jamexu wrote:

> Hello The,
>
> Tuesday, February 27, 2001, 11:00:05 AM, you wrote:
>
> THH> On Tue, 27 Feb 2001, jamexu wrote:
>
> >> Hello Tom,
> >>
> >> Tuesday, February 27, 2001, 12:23:25 AM, you wrote:
> >>
> >> TL> This looks a lot like exchanging the devil we know (SysV shmem) for a
> >> TL> devil we don't know.  Do I need to remind you about, for example, the
> >> TL> mmap bugs in early Linux releases?  (I still vividly remember having to
> >> TL> abandon mmap on a project a few years back that needed to be portable
> >> TL> to Linux.  Perhaps that colors my opinions here.)  I don't think the
> >> TL> problems with shmem are sufficiently large to justify venturing into
> >> TL> a whole new terra incognita of portability issues and kernel bugs.
> >>
> >> TL>                         regards, tom lane
> >>
> >> the only problem is because if we need to tune Postermaster to use
> >> large buffer while system havn't so many SYSV shared memory, in many
> >> systemes, we need to recompile OS kernel, this is a small problem to install
> >> PGSQL to product environment.
>
> THH> What?  You don't automatically recompile your OS kernel when you build a
> THH> system in the first place??  First step on any OS install of FreeBSD is to
> THH> rid myself of the 'extras' that are in the generic kernel, and enable
> THH> SharedMemory (even if I'm not using PgSQL on that machine) ...
>
> heihei, why do you think users always using FreeBSD and not other
> UNIX systemes?
> your assume is false.

I don't ... I personally admin FreeBSD and Solaris boxen ... FreeBSD,
first step is to always recompile the kernel after an install, to get rid
of crud and add Shared Memory ... the Solaris boxes, you add a couple of
lines to /etc/system and reboot, and you have Shared Memory ...

I don't know about other 'commercial OSs', but I'd be shocked if a Linux
admin never does any kernel config cleanup befor egoing production *shrug*




Mon, 26 Feb 2001 23:41:30 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re[3]: Re: [PATCHES] A patch for xlog.c
> > the only problem is because if we need to tune Postermaster to use
> > large buffer while system havn't so many SYSV shared memory, in many
> > systemes, we need to recompile OS kernel, this is a small problem to install
> > PGSQL to product environment.
> 
> What?  You don't automatically recompile your OS kernel when you build a
> system in the first place??  First step on any OS install of FreeBSD is to
> rid myself of the 'extras' that are in the generic kernel, and enable
> SharedMemory (even if I'm not using PgSQL on that machine) ...

He is saying the machine is already in production.  Suppose he has run
PostgreSQL for a few months, then needs to increase number of buffers. 
He can't exceed the kernel limit unless he recompiles.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 26 Feb 2001 22:42:56 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re[2]: Re: [PATCHES] A patch for xlog.c
On Mon, 26 Feb 2001, Bruce Momjian wrote:

> > > the only problem is because if we need to tune Postermaster to use
> > > large buffer while system havn't so many SYSV shared memory, in many
> > > systemes, we need to recompile OS kernel, this is a small problem to install
> > > PGSQL to product environment.
> >
> > What?  You don't automatically recompile your OS kernel when you build a
> > system in the first place??  First step on any OS install of FreeBSD is to
> > rid myself of the 'extras' that are in the generic kernel, and enable
> > SharedMemory (even if I'm not using PgSQL on that machine) ...
>
> He is saying the machine is already in production.  Suppose he has run
> PostgreSQL for a few months, then needs to increase number of buffers.
> He can't exceed the kernel limit unless he recompiles.

Okay ... same applies to MMAP() though, I had to disappoint ... there are
kernel limits that, at least under FreeBSD, do require a kernel
recompile in order to exceed ... alot of them have been moved (maybe all
now) to sysctl settable values ... but, again, under some of the
commercial OSs, I don't think that anything but (as in Solaris) modifying
something like /etc/system and rebooting will fix ...



Mon, 26 Feb 2001 23:56:29 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re[2]: Re: [PATCHES] A patch for xlog.c
> Okay ... same applies to MMAP() though, I had to disappoint ... there are
> kernel limits that, at least under FreeBSD, do require a kernel
> recompile in order to exceed ... alot of them have been moved (maybe all
> now) to sysctl settable values ... but, again, under some of the
> commercial OSs, I don't think that anything but (as in Solaris) modifying
> something like /etc/system and rebooting will fix ...

But the mmap() limits are much larger than the SysV limits, aren't they,
to the point where you would never have to fiddle with the mmap() limits
to get 100mb of buffers, right?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 26 Feb 2001 23:04:42 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re[2]: Re: [PATCHES] A patch for xlog.c
On Mon, 26 Feb 2001, Bruce Momjian wrote:

> > Okay ... same applies to MMAP() though, I had to disappoint ... there are
> > kernel limits that, at least under FreeBSD, do require a kernel
> > recompile in order to exceed ... alot of them have been moved (maybe all
> > now) to sysctl settable values ... but, again, under some of the
> > commercial OSs, I don't think that anything but (as in Solaris) modifying
> > something like /etc/system and rebooting will fix ...
>
> But the mmap() limits are much larger than the SysV limits, aren't they,
> to the point where you would never have to fiddle with the mmap() limits
> to get 100mb of buffers, right?

Not necessarily ... it depends on the admin of the server ... then again,
I don't consider it a hassle to add a couple of lines to my kernel config
(or /etc/system) and reboot *shrug*  to me, its just part of the admin
process ...



Tue, 27 Feb 2001 00:26:16 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re[2]: Re: [PATCHES] A patch for xlog.c
> > But the mmap() limits are much larger than the SysV limits, aren't they,
> > to the point where you would never have to fiddle with the mmap() limits
> > to get 100mb of buffers, right?
> 
> Not necessarily ... it depends on the admin of the server ... then again,
> I don't consider it a hassle to add a couple of lines to my kernel config
> (or /etc/system) and reboot *shrug*  to me, its just part of the admin
> process ...

Are the kernel SysV defaults smaller than the mmap() kernel defaults?

I know it is easy for you, but the number of reports and problems we
hear about shows it is an issue for some.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 26 Feb 2001 23:27:48 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re[2]: Re: [PATCHES] A patch for xlog.c
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I know it is easy for you, but the number of reports and problems we
> hear about shows it is an issue for some.

We hear some reports, but not a lot.  We have no idea whatever what
problems might ensue if we used mmap instead.  I'm dubious that SysV
shmem creates enough problems to justify replacing it with a solution
of essentially unknown portability characteristics...

			regards, tom lane

Mon, 26 Feb 2001 23:45:18 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re[2]: Re: [PATCHES] A patch for xlog.c 
Hello Tom,

Tuesday, February 27, 2001, 12:45:18 PM, you wrote:

TL> Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> I know it is easy for you, but the number of reports and problems we
>> hear about shows it is an issue for some.

TL> We hear some reports, but not a lot.  We have no idea whatever what
TL> problems might ensue if we used mmap instead.  I'm dubious that SysV
TL> shmem creates enough problems to justify replacing it with a solution
TL> of essentially unknown portability characteristics...

TL>                         regards, tom lane

could anyone investigate mmap() in many modern UNIX systems to prove that
mmap() is so un-portable?

it seems mmap() is a portable problem like you said, but I think SYSV
shmem for PGSQL is a installation problem. you push some difficults to
end user, and take easy taskes for yourself.

Xu Yifeng



Tue, 27 Feb 2001 13:45:44 +0800
Xu Yifeng <jamexu@telekbird.com.cn>
Re[4]: Re: [PATCHES] A patch for xlog.c
On Tue, 27 Feb 2001, Xu Yifeng wrote:

> Hello Tom,
>
> Tuesday, February 27, 2001, 12:45:18 PM, you wrote:
>
> TL> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> I know it is easy for you, but the number of reports and problems we
> >> hear about shows it is an issue for some.
>
> TL> We hear some reports, but not a lot.  We have no idea whatever what
> TL> problems might ensue if we used mmap instead.  I'm dubious that SysV
> TL> shmem creates enough problems to justify replacing it with a solution
> TL> of essentially unknown portability characteristics...
>
> TL>                         regards, tom lane
>
> could anyone investigate mmap() in many modern UNIX systems to prove that
> mmap() is so un-portable?
>
> it seems mmap() is a portable problem like you said, but I think SYSV
> shmem for PGSQL is a installation problem. you push some difficults to
> end user, and take easy taskes for yourself.

Considering that, so far as I can tell, both you and Bruce are the only
ones that are really heavy on moving away from SysV ... how many ppl are
actually finding it to be that much more difficult? :)



Tue, 27 Feb 2001 08:28:43 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re[4]: Re: [PATCHES] A patch for xlog.c
> I don't know about other 'commercial OSs', but I'd be shocked if a Linux
> admin never does any kernel config cleanup befor egoing production *shrug*

oops...

                     - Thomas

Tue, 27 Feb 2001 14:31:35 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [PATCHES] A patch for xlog.c
> > could anyone investigate mmap() in many modern UNIX systems to prove that
> > mmap() is so un-portable?
> >
> > it seems mmap() is a portable problem like you said, but I think SYSV
> > shmem for PGSQL is a installation problem. you push some difficults to
> > end user, and take easy taskes for yourself.
> 
> Considering that, so far as I can tell, both you and Bruce are the only
> ones that are really heavy on moving away from SysV ... how many ppl are
> actually finding it to be that much more difficult? :)

I am not sure I would call myself _heavy_ on it.  I suggest researching
it on platforms that support anon. mmap() to reduce administration load
when increasing the number of buffers.  If it is not a big win, there is
no reason to add support for it.  We clearly will be keeping SysV for a
long time, so adding another shared memory system, mmap(), should only
be done for a very good reason.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 27 Feb 2001 11:23:44 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re[4]: Re: [PATCHES] A patch for xlog.c
The Hermit Hacker writes:

> I don't ... I personally admin FreeBSD and Solaris boxen ... FreeBSD,
> first step is to always recompile the kernel after an install, to get rid
> of crud and add Shared Memory ... the Solaris boxes, you add a couple of
> lines to /etc/system and reboot, and you have Shared Memory ...
>
> I don't know about other 'commercial OSs', but I'd be shocked if a Linux
> admin never does any kernel config cleanup befor egoing production *shrug*

Linux allows you to load and unload kernel modules, while the system is
running, to add and remove stuff as you need it.  But this is moot because
Linux also allows you to increase shared memory (up to the total
addressable memory)  while the system is running.  Recompiling Linux
kernels is a thing of the past with modern distributions.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Tue, 27 Feb 2001 17:25:35 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Re[3]: Re: [PATCHES] A patch for xlog.c
On Tue, 27 Feb 2001, Peter Eisentraut wrote:

> The Hermit Hacker writes:
>
> > I don't ... I personally admin FreeBSD and Solaris boxen ... FreeBSD,
> > first step is to always recompile the kernel after an install, to get rid
> > of crud and add Shared Memory ... the Solaris boxes, you add a couple of
> > lines to /etc/system and reboot, and you have Shared Memory ...
> >
> > I don't know about other 'commercial OSs', but I'd be shocked if a Linux
> > admin never does any kernel config cleanup befor egoing production *shrug*
>
> Linux allows you to load and unload kernel modules, while the system is
> running, to add and remove stuff as you need it.  But this is moot because
> Linux also allows you to increase shared memory (up to the total
> addressable memory)  while the system is running.  Recompiling Linux
> kernels is a thing of the past with modern distributions.

Actually, just found that out for FreeBSD too *sigh*  You do have to
enable SYSV* in the kernel itself, but increasing shared memory and
semaphores is a simple sysctl that can be run while the system is live ...



Tue, 27 Feb 2001 13:26:03 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re[3]: Re: [PATCHES] A patch for xlog.c