Why does postgres maintain a loader logic of its own? I can understand that
the dynamic_library_path is necessary in order to configure everything in
one single place. But why not just merge it with the LD_LIBRARY_PATH (or
PATH on Windows) and then let dlopen do the rest using a stripped filename?

The reason I ask is because I run into problems as soon as I have a module
that in turn depends on other shared libraries. It doesn't help much that
they are reachable through the dynamic_library_path since that path never is
made known to the OS loader mechanisms.

I'll be happty to submit some code to do the actual path merging (i.e
omitting duplicates etc.).

Kind regards,

Thomas Hallgren


Tue, 8 Jun 2004 10:21:05 +0200
"Thomas Hallgren" <thhal@mailblocks.com>
Question regarding dynamic_library_path
Thomas Hallgren wrote:
> Why does postgres maintain a loader logic of its own? I can understand that
> the dynamic_library_path is necessary in order to configure everything in
> one single place. But why not just merge it with the LD_LIBRARY_PATH (or
> PATH on Windows) and then let dlopen do the rest using a stripped filename?
> 
> The reason I ask is because I run into problems as soon as I have a module
> that in turn depends on other shared libraries. It doesn't help much that
> they are reachable through the dynamic_library_path since that path never is
> made known to the OS loader mechanisms.
> 
> I'll be happty to submit some code to do the actual path merging (i.e
> omitting duplicates etc.).

I think the idea is that you want to specify the path in the config
file, after the app has already started.  I don't think you can modify
the environment variable after the app has started, and even if you can,
it seems simpler to just do it in our code and specify the exact path
rather than having it poke around in whatever LD_LIBRARY_PATH is set to.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 8 Jun 2004 09:51:46 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Question regarding dynamic_library_path
Bruce Momjian wrote:

>Thomas Hallgren wrote:
>  
>
>>Why does postgres maintain a loader logic of its own? I can understand that
>>the dynamic_library_path is necessary in order to configure everything in
>>one single place. But why not just merge it with the LD_LIBRARY_PATH (or
>>PATH on Windows) and then let dlopen do the rest using a stripped filename?
>>
>>The reason I ask is because I run into problems as soon as I have a module
>>that in turn depends on other shared libraries. It doesn't help much that
>>they are reachable through the dynamic_library_path since that path never is
>>made known to the OS loader mechanisms.
>>
>>I'll be happty to submit some code to do the actual path merging (i.e
>>omitting duplicates etc.).
>>    
>>
>
>I think the idea is that you want to specify the path in the config
>file, after the app has already started.  I don't think you can modify
>the environment variable after the app has started, and even if you can,
>it seems simpler to just do it in our code and specify the exact path
>rather than having it poke around in whatever LD_LIBRARY_PATH is set to.
>
>  
>

And many people just hate playing games with LD_LIBRARY_PATH, and even 
more hate apps that do it.

cheers

andrew

Tue, 08 Jun 2004 10:32:29 -0400
Andrew Dunstan <andrew@dunslane.net>
Re: Question regarding dynamic_library_path
"Thomas Hallgren" <thhal@mailblocks.com> writes:
> Why does postgres maintain a loader logic of its own? I can understand that
> the dynamic_library_path is necessary in order to configure everything in
> one single place. But why not just merge it with the LD_LIBRARY_PATH (or
> PATH on Windows) and then let dlopen do the rest using a stripped filename?

What LD_LIBRARY_PATH?  The above statement is so full of system-specific
assumptions that it seems hopeless.

			regards, tom lane

Tue, 08 Jun 2004 10:37:13 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Question regarding dynamic_library_path 
> "Thomas Hallgren" <thhal@mailblocks.com> writes:
> > Why does postgres maintain a loader logic of its own? I can understand
that
> > the dynamic_library_path is necessary in order to configure everything
in
> > one single place. But why not just merge it with the LD_LIBRARY_PATH (or
> > PATH on Windows) and then let dlopen do the rest using a stripped
filename?
>
> What LD_LIBRARY_PATH?  The above statement is so full of system-specific
> assumptions that it seems hopeless.
>
> regards, tom lane
>
The LD_LIBRARY_PATH or PATH depending on system (Posix or Windows) that is
effective when the dlopen function is called. All OS'es where shared
libraries are possible have something similar.

The general idea is to let the OS find the shared library rather than have
the backend do it by itself. There's a flaw in the current design. IMHO, it
would be a good thing to improve it.

regards,

Thomas Hallgren



Tue, 8 Jun 2004 19:08:00 +0200
"Thomas Hallgren" <thhal@mailblocks.com>
Re: Question regarding dynamic_library_path 
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
>
> I think the idea is that you want to specify the path in the config
> file, after the app has already started.  I don't think you can modify
> the environment variable after the app has started, and even if you can,
> it seems simpler to just do it in our code and specify the exact path
> rather than having it poke around in whatever LD_LIBRARY_PATH is set to.
>
Ok, that makes sense. But I think most systems have the ability to change
the environment of a running process (setenv on Posix systems) and the
current design, simple as it is, doesn't work in all cases.

regards,

Thomas Hallgren



Tue, 8 Jun 2004 19:14:04 +0200
"Thomas Hallgren" <thhal@mailblocks.com>
Re: Question regarding dynamic_library_path
"Andrew Dunstan" <andrew@dunslane.net> wrote:
> And many people just hate playing games with LD_LIBRARY_PATH, and even
> more hate apps that do it.
>
The current design forces me and anyone else who wants a module that depends
on shared libraries to play these games. My suggestion is that we hide this
with a better implementation. I do *not* want to expose LD_LIBRARY_PATH nor
force anyone to use it. I simply want the dynamic_library_path propagated to
the loader in a correct fasion.

regards,

Thomas Hallgren


Tue, 8 Jun 2004 19:24:13 +0200
"Thomas Hallgren" <thhal@mailblocks.com>
Re: Question regarding dynamic_library_path
"Thomas Hallgren" <thhal@mailblocks.com> writes:
> The LD_LIBRARY_PATH or PATH depending on system (Posix or Windows) that is
> effective when the dlopen function is called. All OS'es where shared
> libraries are possible have something similar.

The variations among platforms are great enough that I don't think you
can make such an assertion.

(To take one example, HPUX does have a variable like this --- they call
it SHLIB_PATH --- but it *is not used* unless a flag enabling it was
provided when the shlib was linked, and I think most people avoid doing
that.  Certainly we don't enable SHLIB_PATH in the shlibs we build in
the standard Postgres distribution.  Furthermore, AFAICT shl_load()
requires an exact path spec for the initial shared library; it won't
do a path search for that, only for dependencies.)

> The general idea is to let the OS find the shared library rather than have
> the backend do it by itself. There's a flaw in the current design. IMHO, it
> would be a good thing to improve it.

I think we'd just be buying into a lot of pain and platform-specific
behavior.  Up to now we have not depended on knowing how to
environmentally influence the dynamic linker, but with your proposal
Postgres would be immediately broken anywhere that we didn't have that
right.  And there are places we *couldn't* get it right, see HPUX.
I'm much more concerned about whether shared library loading works at
all than about whether it's possible for one shlib to autoload another.

If you want $libdir to be part of LD_LIBRARY_PATH on your platform, why
don't you just set that up in the postmaster start script?

			regards, tom lane

Tue, 08 Jun 2004 13:47:34 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Question regarding dynamic_library_path 
> The variations among platforms are great enough that I don't think you
> can make such an assertion.
> 
> (To take one example, HPUX does have a variable like this --- they call
> it SHLIB_PATH --- but it *is not used* unless a flag enabling it was
> provided when the shlib was linked, and I think most people avoid doing
> that.  Certainly we don't enable SHLIB_PATH in the shlibs we build in
> the standard Postgres distribution.  Furthermore, AFAICT shl_load()
> requires an exact path spec for the initial shared library; it won't
> do a path search for that, only for dependencies.)
> 
Here's an excerpt from the HPUX manual pages on dlopen(3C).

"First, any directories specified by the environment variable
LD_LIBRARY_PATH are searched. Next, any directories specified by SHLIB_PATH
are searched. Then, any directories specified by a DT_RPATH entry in the
.dynamic section of the original program object are searched. Finally, the
directories /usr/lib/hpux32 in 32-bit mode and /usr/lib/hpux64 in 64-bit
mode are searched."

> > The general idea is to let the OS find the shared library rather than
> have
> > the backend do it by itself. There's a flaw in the current design. IMHO,
> it
> > would be a good thing to improve it.
> 
> I think we'd just be buying into a lot of pain and platform-specific
> behavior.  Up to now we have not depended on knowing how to
> environmentally influence the dynamic linker, but with your proposal
> Postgres would be immediately broken anywhere that we didn't have that
> right.  And there are places we *couldn't* get it right, see HPUX.
> I'm much more concerned about whether shared library loading works at
> all than about whether it's possible for one shlib to autoload another.
> 
I don't agree. Propagating the dynamic_library_path to the loader will make
it work the way people would expect it to. Failing to do so is confusing
since the path you specified appears to be incorrect. The concept of shared
libs that are dependent on other shared libs is not uncommon. I think that
following the platform specific behavior is what PostgreSQL *should* do in
this case. And HPUX, as you can see, is no obstacle. I actually sincerely
doubt that you'll find a platform where shared libraries are enabled and
this cannot be done.

> If you want $libdir to be part of LD_LIBRARY_PATH on your platform, why
> don't you just set that up in the postmaster start script?

Yes, there's always a workaround for the problem and this is what I do
today. My suggestion is based on the assumption that it would be nice if the
dynamic_library_path actually worked for libraries loading libraries. The
discussion is pointless if that assumption is incorrect.

Kind regards,

Thomas Hallgren




Tue, 8 Jun 2004 20:17:50 +0200
"Thomas Hallgren" <thhal@mailblocks.com>
Re: Question regarding dynamic_library_path 
"Thomas Hallgren" <thhal@mailblocks.com> writes:
>> ...  Furthermore, AFAICT shl_load()
>> requires an exact path spec for the initial shared library; it won't
>> do a path search for that, only for dependencies.)
>> 
> Here's an excerpt from the HPUX manual pages on dlopen(3C).

HPUX which?  There is no dlopen() in HPUX 10.20, which is what
I'm running here; nor do we use dlopen() in any HPUX version
given the current state of port/dynloader/hpux.c.

			regards, tom lane

Wed, 09 Jun 2004 01:24:13 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Question regarding dynamic_library_path 
Oh, sorry. This HP-UX 11.x. But you can get the same using shl_load (HP-UX
10.x) and pass the flag DYNAMIC_PATH provided the executable is linked with
+s. So it's still no showstopper.

If you do find that it is impossible on some older OS (HP-UX 11 arrived 4
years ago), then why not just disable this feature on that particular
version of the OS and make a note for the user that in order to get it, it's
time to upgrade?

IMO it's not a good policy to prevent new good features just to keep
customers sticking to old stuff from feeling bad not getting them. You need
to assure backward compatibility to keep everyone happy.

Kind regards,

Thomas Hallgren

----- Original Message ----- 
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Thomas Hallgren" <thhal@mailblocks.com>
Cc: <pgsql-hackers@postgresql.org>
Sent: Wednesday, June 09, 2004 7:24 AM
Subject: Re: [HACKERS] Question regarding dynamic_library_path


> "Thomas Hallgren" <thhal@mailblocks.com> writes:
> >> ...  Furthermore, AFAICT shl_load()
> >> requires an exact path spec for the initial shared library; it won't
> >> do a path search for that, only for dependencies.)
> >>
> > Here's an excerpt from the HPUX manual pages on dlopen(3C).
>
> HPUX which?  There is no dlopen() in HPUX 10.20, which is what
> I'm running here; nor do we use dlopen() in any HPUX version
> given the current state of port/dynloader/hpux.c.
>
> regards, tom lane
>



Wed, 9 Jun 2004 09:57:07 +0200
"Thomas Hallgren" <thhal@mailblocks.com>
Re: Question regarding dynamic_library_path 
Hi,

in linux you can change LD_LIBRARY_PATH in a running process, but it does not 
help. The library-loader initializes himself at process-startup and changing 
LD_LIBRARY_PATH do not change the actual path, the process is using for 
dlopen.

Tommi Mäkitalo

Am Dienstag, 8. Juni 2004 19:14 schrieb Thomas Hallgren:
> From: "Bruce Momjian" <pgman@candle.pha.pa.us>
>
> > I think the idea is that you want to specify the path in the config
> > file, after the app has already started.  I don't think you can modify
> > the environment variable after the app has started, and even if you can,
> > it seems simpler to just do it in our code and specify the exact path
> > rather than having it poke around in whatever LD_LIBRARY_PATH is set to.
>
> Ok, that makes sense. But I think most systems have the ability to change
> the environment of a running process (setenv on Posix systems) and the
> current design, simple as it is, doesn't work in all cases.
>
> regards,
>
> Thomas Hallgren
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 7: don't forget to increase your free space map settings

Wed, 9 Jun 2004 10:39:33 +0200
Tommi Maekitalo <t.maekitalo@epgmbh.de>
Re: Question regarding dynamic_library_path
"Tommi Maekitalo" <t.maekitalo@epgmbh.de> wrote:
> Hi,
>
> in linux you can change LD_LIBRARY_PATH in a running process, but it does
not
> help. The library-loader initializes himself at process-startup and
changing
> LD_LIBRARY_PATH do not change the actual path, the process is using for
> dlopen.
>
That's bad. My nice suggestion just scattered into pieces :-)

But then again, perhaps not. Isn't the fact that "dynamic_library_path" can
be changed at any time by a user a serious security flaw? It enables the
user to load a module from an arbitrary place and then execute what's in
that module with postmaster priviligies. I.e. there's nothing preventing
such a user to access all data in the database using low-level C-routines
and bypass the security imposed by PostgreSQL.

IMHO, the DB-admin should be able to limit what's loaded by defining loader
constraints. Regardless of the previous discussion I would suggest the
following:

1. Prohibit use of paths containing a directory component in SQL.
2. Make dynamic_library_path a PGC_POSTMASTER variable.

This would close the security hole and give the DB Admin full control over
what a user can and cannot load.

Now, back to my original suggestion.

Using the current desing, the postmaster would still not be able to set the
LD_LIBRARY_PATH from the dynamic_library_path on Unix since all it does when
a new connection is established is a fork. So on a Unix machine with Linux
behavior the postmaster would need to parse the config file on startup,
merge dynamic_library_path with LD_LIBRARY_PATH (or whatever is applicable),
set the environment, and then do re-exec using some flag indicating
"stage2". Stage 2 would do exactly what's done today but ignore the
dynamic_library_path setting completely.

I realize that this is stretching it a little :-). The part concerning the
security-leak is important though.

Kind regards,

Thomas Hallgren


Wed, 9 Jun 2004 13:04:20 +0200
"Thomas Hallgren" <thhal@mailblocks.com>
Re: Question regarding dynamic_library_path


Thomas Hallgren wrote:

>Isn't the fact that "dynamic_library_path" can
>be changed at any time by a user a serious security flaw? 
>  
>
It's not "a user". Only the superuser can change it.

cheers

andrew

Wed, 09 Jun 2004 07:56:13 -0400
Andrew Dunstan <andrew@dunslane.net>
Re: Question regarding dynamic_library_path
And only the super user can use directory components in a module name?

regards,

Thomas Hallgren

"Andrew Dunstan" <andrew@dunslane.net> wrote in message
news:40C6FADD.3000300@dunslane.net...
>
>
> Thomas Hallgren wrote:
>
> >Isn't the fact that "dynamic_library_path" can
> >be changed at any time by a user a serious security flaw?
> >
> >
> It's not "a user". Only the superuser can change it.
>
> cheers
>
> andrew
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>


Wed, 9 Jun 2004 14:35:04 +0200
"Thomas Hallgren" <thhal@mailblocks.com>
Re: Question regarding dynamic_library_path
"Thomas Hallgren" <thhal@mailblocks.com> writes:
> And only the super user can use directory components in a module name?

Only superusers can create C functions in the first place, so quibbling
about what paths they can use doesn't seem that useful ...

			regards, tom lane

Wed, 09 Jun 2004 09:56:18 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Question regarding dynamic_library_path 
I agree. I wasn't aware of that restriction.

regards,

Thomas Hallgren

----- Original Message ----- 
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Thomas Hallgren" <thhal@mailblocks.com>
Cc: <pgsql-hackers@postgresql.org>
Sent: Wednesday, June 09, 2004 15:56
Subject: Re: [HACKERS] Question regarding dynamic_library_path 


> "Thomas Hallgren" <thhal@mailblocks.com> writes:
> > And only the super user can use directory components in a module name?
> 
> Only superusers can create C functions in the first place, so quibbling
> about what paths they can use doesn't seem that useful ...
> 
> regards, tom lane
> 


Wed, 9 Jun 2004 16:37:56 +0200
"Thomas Hallgren" <thhal@mailblocks.com>
Re: Question regarding dynamic_library_path 