As I set out to do the Windows semaphore thing, I notice it can get quite ugly.

In the current CVS directory, there is pgsql/src/backend/port directory.

I propose that this become a separate subproject and library. The reason I want
this is because the semaphore support, specifically multiple semaphores
identified by a single key, has to be implemented with shared memory and
multiple semaphores. (Under Windows)

I also have to look at "ownership" issues with Windows processes and shared
objects. They may need to be own be a more persistent/independent module, a
DLL.

By creating a library like "libpgport.a" (or on Windows pgport.dll/pgport.lib)
little has to be changed for various ports. If we don't have the library then
the discrete object filed would need to be specified. The library will give the
port writer a greater amount of flexibility.

Comments?

Tue, 07 May 2002 08:42:15 -0400
mlw <markw@mohawksoft.com>
OK, lets talk portability.
mlw <markw@mohawksoft.com> writes:
> In the current CVS directory, there is pgsql/src/backend/port directory.

> I propose that this become a separate subproject and library.

Right offhand, that seems a pointless exercise in relabeling code that's
going to be the same either way.  What's the actual value?

> The reason I want this is because the semaphore support, specifically
> multiple semaphores identified by a single key, has to be implemented
> with shared memory and multiple semaphores. (Under Windows)

I think you are confusing issues that are now private to the SysV sema
implementation with things that you really need to do for Windows.
Take a look at port/posix_sema.c for a less cluttered view of the
semantics you actually need to support.  (I don't suppose there's any
chance that Gates & Co support POSIX semas, leaving you with no work?)

BTW, I have been able to test the named-semas variant of posix_sema.c
on OS X, and it works.  I don't have access to any platforms that
support unnamed POSIX semas, which is too bad because that seems much
the preferable variant.  Can anyone check it out?

			regards, tom lane

PS: there's a trivial little test program in port/ipc_test.c; if you
want a "smoke test" that's simpler than a full Postgres build, try that.

Tue, 07 May 2002 09:31:57 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: OK, lets talk portability. 
Tom Lane wrote:
> BTW, I have been able to test the named-semas variant of posix_sema.c
> on OS X, and it works.  I don't have access to any platforms that
> support unnamed POSIX semas, which is too bad because that seems much
> the preferable variant.  Can anyone check it out?

I did, and yes I was confused. Sorry. Your posix implementation assumes that
only a single process will have access to the semaphore list for deletion is
this correct? I guess I need to know how much access the child processes need
to have to the internal control structures, none? Some? All?

As I embark on my journey back to the dark side, here are my concerns for a
native Windows PostgreSQL. I think it is a whole lot more than originally
thought.

(The Matrix: Do not try to implement the fork() that would be impossible,
instead only try to realize the truth, there is no fork())

Cygwin does a lot of trickery to implement fork(), but it is not an OS level
thing. Without fork() we would have to have a way to start a postgres backend
and send it information about what it should be doing, and what it should be
doing it with.

With no fork(), information that would normally be copied on fork() is not
copied. Therefore, we need to know what that information is and how to
propagate it to the child process (under windows)

Files, Windows does not have a native open,close,read,write ,lseek support.
Yes, they have some notion of low I/O for compatibility, _open, _close, etc.
but the flags and permissions are not identical. The low file API for Windows
is CreateFile.

Semaphores, and shared memory are easy enough (If have written it in different
forms before), depending on the level of opaqueness to child processes.


The voice in the back of my head, says we need to define what the portability
issues are:

process control (fork()/spawn() etc.)
file operations (read, right, open, close, seek)
IPC constructs (shared memory, semaphores)
System interface (sync() etc)

Any others?

We should either use Apache's APR and augment it with semaphores, or come up
with a doc which defines how these various things are handled. Obviously, it
will grow as we find things that don't work.

Tue, 07 May 2002 10:03:07 -0400
mlw <markw@mohawksoft.com>
Re: OK, lets talk portability.
mlw <markw@mohawksoft.com> writes:
> I did, and yes I was confused. Sorry. Your posix implementation assumes that
> only a single process will have access to the semaphore list for deletion is
> this correct? I guess I need to know how much access the child processes need
> to have to the internal control structures, none? Some? All?

None.  The postmaster creates the semas, and the postmaster deletes 'em.
The children only use them via the PGSemaphore structs they find in
shared memory.  (The SysV implementation assumes that too btw.)

> With no fork(), information that would normally be copied on fork() is not
> copied. Therefore, we need to know what that information is and how to
> propagate it to the child process (under windows)

This will be a royal mess.  Three or four years ago, when PG actually
did fork/exec to start a backend, we were careful to arrange to pass
everything the backend needed to know as command-line parameters (or
else keep it in shared memory).  We have been busily breaking that
separation ever since we went over to fork-no-exec, however.  In the
current scheme of things there is no chance whatever of a backend
working unless it inherits the global/static variables of the
postmaster.

And no, I don't want to undo those changes.  Especially not if the
only reason for it is to not have to use Cygwin on Windows.  Most
of these changes made the startup code substantially simpler,
faster, and more reliable.

			regards, tom lane

Tue, 07 May 2002 10:32:28 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: OK, lets talk portability. 
Tom Lane wrote:
> And no, I don't want to undo those changes.  Especially not if the
> only reason for it is to not have to use Cygwin on Windows.  Most
> of these changes made the startup code substantially simpler,
> faster, and more reliable.

Then I think the notion of a pure Windows version is dead in the water. Writing
a fork()-like API for Windows is, of course, doable as evidenced by cygwin, and
from a general theory seems like a pretty straight forward thing to do (with a
few low level tricks of course) but the details are pretty scary.

Has anyone done a profile of PostgreSQL running on a windows box and identified
cygwin bottlenecks which we could augment with native code?

Tue, 07 May 2002 10:44:08 -0400
mlw <markw@mohawksoft.com>
Re: OK, lets talk portability.
On Tue, 7 May 2002, mlw wrote:

> Tom Lane wrote:
> > And no, I don't want to undo those changes.  Especially not if the
> > only reason for it is to not have to use Cygwin on Windows.  Most
> > of these changes made the startup code substantially simpler,
> > faster, and more reliable.
>
> Then I think the notion of a pure Windows version is dead in the water.
> Writing a fork()-like API for Windows is, of course, doable as evidenced
> by cygwin, and from a general theory seems like a pretty straight
> forward thing to do (with a few low level tricks of course) but the
> details are pretty scary.

How is Apache doing this?  I believe they do allow the pre-forked model to
work, so how are they getting around those limitations?



Tue, 7 May 2002 11:50:50 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: OK, lets talk portability.
"Marc G. Fournier" wrote:
> 
> On Tue, 7 May 2002, mlw wrote:
> 
> > Tom Lane wrote:
> > > And no, I don't want to undo those changes.  Especially not if the
> > > only reason for it is to not have to use Cygwin on Windows.  Most
> > > of these changes made the startup code substantially simpler,
> > > faster, and more reliable.
> >
> > Then I think the notion of a pure Windows version is dead in the water.
> > Writing a fork()-like API for Windows is, of course, doable as evidenced
> > by cygwin, and from a general theory seems like a pretty straight
> > forward thing to do (with a few low level tricks of course) but the
> > details are pretty scary.
> 
> How is Apache doing this?  I believe they do allow the pre-forked model to
> work, so how are they getting around those limitations?

Apache and PostgreSQL are quite different in their requirements of shared
memory. Apache (2.x) simply uses CreateProcess and passes duplicate file
handles.

Tue, 07 May 2002 10:58:29 -0400
mlw <markw@mohawksoft.com>
Re: OK, lets talk portability.
On 7 May 2002, Hannu Krosing wrote:

> On Tue, 2002-05-07 at 15:31, Tom Lane wrote:
> > mlw <markw@mohawksoft.com> writes:
> > > In the current CVS directory, there is pgsql/src/backend/port directory.
> >
> > > I propose that this become a separate subproject and library.
> >
> > Right offhand, that seems a pointless exercise in relabeling code that's
> > going to be the same either way.  What's the actual value?
> >
> > > The reason I want this is because the semaphore support, specifically
> > > multiple semaphores identified by a single key, has to be implemented
> > > with shared memory and multiple semaphores. (Under Windows)
> >
> > I think you are confusing issues that are now private to the SysV sema
> > implementation with things that you really need to do for Windows.
> > Take a look at port/posix_sema.c for a less cluttered view of the
> > semantics you actually need to support.  (I don't suppose there's any
> > chance that Gates & Co support POSIX semas, leaving you with no work?)
>
> A quick google search came up with
>
>  http://sources.redhat.com/pthreads-win32/announcement.html
>
>
> Unfortunately it seems to be the "wrong kind of free" software:
>
> Pthreads-win32 is free software, distributed under the GNU Lesser
> General Public License (LGPL).
>
> Or can we accept dependancies on LGPL libs for some ports.

What someone installs on their Windows box is their problem ... doesn't
mean we can't make use of it :)  Its not something that will be part of
the distribution itself, only something that needs to be availble  :)



Tue, 7 May 2002 13:10:13 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: OK, lets talk portability.
On Tue, 2002-05-07 at 15:31, Tom Lane wrote:
> mlw <markw@mohawksoft.com> writes:
> > In the current CVS directory, there is pgsql/src/backend/port directory.
> 
> > I propose that this become a separate subproject and library.
> 
> Right offhand, that seems a pointless exercise in relabeling code that's
> going to be the same either way.  What's the actual value?
> 
> > The reason I want this is because the semaphore support, specifically
> > multiple semaphores identified by a single key, has to be implemented
> > with shared memory and multiple semaphores. (Under Windows)
> 
> I think you are confusing issues that are now private to the SysV sema
> implementation with things that you really need to do for Windows.
> Take a look at port/posix_sema.c for a less cluttered view of the
> semantics you actually need to support.  (I don't suppose there's any
> chance that Gates & Co support POSIX semas, leaving you with no work?)

A quick google search acme up with

 http://sources.redhat.com/pthreads-win32/announcement.html


--------------
Hannu


07 May 2002 18:57:42 +0200
Hannu Krosing <hannu@tm.ee>
Re: OK, lets talk portability.
On Tue, 2002-05-07 at 15:31, Tom Lane wrote:
> mlw <markw@mohawksoft.com> writes:
> > In the current CVS directory, there is pgsql/src/backend/port directory.
> 
> > I propose that this become a separate subproject and library.
> 
> Right offhand, that seems a pointless exercise in relabeling code that's
> going to be the same either way.  What's the actual value?
> 
> > The reason I want this is because the semaphore support, specifically
> > multiple semaphores identified by a single key, has to be implemented
> > with shared memory and multiple semaphores. (Under Windows)
> 
> I think you are confusing issues that are now private to the SysV sema
> implementation with things that you really need to do for Windows.
> Take a look at port/posix_sema.c for a less cluttered view of the
> semantics you actually need to support.  (I don't suppose there's any
> chance that Gates & Co support POSIX semas, leaving you with no work?)

A quick google search came up with

 http://sources.redhat.com/pthreads-win32/announcement.html


Unfortunately it seems to be the "wrong kind of free" software: 

Pthreads-win32 is free software, distributed under the GNU Lesser
General Public License (LGPL).

Or can we accept dependancies on LGPL libs for some ports.

--------------
Hannu


07 May 2002 19:00:40 +0200
Hannu Krosing <hannu@tm.ee>
Re: OK, lets talk portability.
On 7 May 2002, Hannu Krosing wrote:

> On Tue, 2002-05-07 at 15:31, Tom Lane wrote:
> > mlw <markw@mohawksoft.com> writes:
> > > In the current CVS directory, there is pgsql/src/backend/port directory.
> >
> > > I propose that this become a separate subproject and library.
> >
> > Right offhand, that seems a pointless exercise in relabeling code that's
> > going to be the same either way.  What's the actual value?
> >
> > > The reason I want this is because the semaphore support, specifically
> > > multiple semaphores identified by a single key, has to be implemented
> > > with shared memory and multiple semaphores. (Under Windows)
> >
> > I think you are confusing issues that are now private to the SysV sema
> > implementation with things that you really need to do for Windows.
> > Take a look at port/posix_sema.c for a less cluttered view of the
> > semantics you actually need to support.  (I don't suppose there's any
> > chance that Gates & Co support POSIX semas, leaving you with no work?)
>
> A quick google search acme up with
>
>  http://sources.redhat.com/pthreads-win32/announcement.html

Damn ... doesn't implement fork(), but does implement semaphores :)
Sooooo close :)


  Semaphores
      ---------------------------
      sem_init
      sem_destroy
      sem_post
      sem_wait
      sem_trywait
      sem_timedwait
      sem_open               (returns an error ENOSYS)
      sem_close              (returns an error ENOSYS)
      sem_unlink             (returns an error ENOSYS)


Tue, 7 May 2002 14:45:52 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: OK, lets talk portability.
"Marc G. Fournier" wrote:
> 
> On 7 May 2002, Hannu Krosing wrote:
> 
> > On Tue, 2002-05-07 at 15:31, Tom Lane wrote:
> > > mlw <markw@mohawksoft.com> writes:
> > > > In the current CVS directory, there is pgsql/src/backend/port directory.
> > >
> > > > I propose that this become a separate subproject and library.
> > >
> > > Right offhand, that seems a pointless exercise in relabeling code that's
> > > going to be the same either way.  What's the actual value?
> > >
> > > > The reason I want this is because the semaphore support, specifically
> > > > multiple semaphores identified by a single key, has to be implemented
> > > > with shared memory and multiple semaphores. (Under Windows)
> > >
> > > I think you are confusing issues that are now private to the SysV sema
> > > implementation with things that you really need to do for Windows.
> > > Take a look at port/posix_sema.c for a less cluttered view of the
> > > semantics you actually need to support.  (I don't suppose there's any
> > > chance that Gates & Co support POSIX semas, leaving you with no work?)
> >
> > A quick google search acme up with
> >
> >  http://sources.redhat.com/pthreads-win32/announcement.html
> 
> Damn ... doesn't implement fork(), but does implement semaphores :)
> Sooooo close :)

Windows has semaphores, and looking at Tom's API, this is the least of our
problems. 

If we can come up with a fork() free PostgreSQL, the rest is easy.

We need to come up with a set of macros and function API that handle:

Native file operations.
Process control 
Shared memory, semaphores, and other IPC
Sockets (Some Windows specific crap will need to be written, but the berkeley
API is fine)
System interface, sync(), fdatasync() and such.

Tue, 07 May 2002 13:47:10 -0400
mlw <markw@mohawksoft.com>
Re: OK, lets talk portability.
On Tue, 2002-05-07 at 19:44, mlw wrote:
> Tom Lane wrote:
> > And no, I don't want to undo those changes.  Especially not if the
> > only reason for it is to not have to use Cygwin on Windows.  Most
> > of these changes made the startup code substantially simpler,
> > faster, and more reliable.
> 
> Then I think the notion of a pure Windows version is dead in the water. Writing
> a fork()-like API for Windows is, of course, doable as evidenced by cygwin, and
> from a general theory seems like a pretty straight forward thing to do (with a
> few low level tricks of course) but the details are pretty scary.

There is still another way - use threads. 

There you have of course the opposite problem - to determine what to
_not_ share, but AFAIK this has been done already at least once.

And there seems to be some consensus that doing things that would
eventually make it easier to use threaded model will probably increase
code quality in general.

---------------
Hannu



07 May 2002 23:28:39 +0500
Hannu Krosing <hannu@tm.ee>
Re: OK, lets talk portability.
> mlw <markw@mohawksoft.com> writes:
> > In the current CVS directory, there is pgsql/src/backend/port directory.
>
> BTW, I have been able to test the named-semas variant of posix_sema.c
> on OS X, and it works.  I don't have access to any platforms that
> support unnamed POSIX semas, which is too bad because that seems much
> the preferable variant.  Can anyone check it out?
>

They are supported on QNX, so I will check it. They are also faster than
named ones.

-- igor



Tue, 7 May 2002 15:55:50 -0500
"Igor Kovalenko" <Igor.Kovalenko@motorola.com>
Re: OK, lets talk portability. 