Problem:
PL/Java use a JVM. On some platforms and with some JVM's (Sun's in 
particular) a libzip.so is bundled that contains a 1.1.3 version of 
functions also provided in zlib (why they do this is beyond me, but they 
do so I'll have to live with it). PostgreSQL is linked with zlib by 
default. This resuls in a conflict which resuls in a JVM crash.

I can circumvein this crash by using LD_PRELOAD to force a load of the 
JVM bundled libzip.so but I suspect that might result in a crash as soon 
as PostgreSQL calls on zlib to do something. It's of course possible to 
configure postgresql with --without-zlib also provided you have accesst 
o the source.

Question:
 From what I can understand from the documentation, the only utility in 
PostgreSQL that actually uses zlib is pg_dump? If so, why is the 
postgres process linked with -lz?

Regards,

Thomas Hallgren


Thu, 23 Sep 2004 13:02:17 +0200
Thomas Hallgren <thhal@mailblocks.com>
Use of zlib
Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren:
>  From what I can understand from the documentation, the only utility in
> PostgreSQL that actually uses zlib is pg_dump? If so, why is the
> postgres process linked with -lz?

Because we are too lazy to fine-tune the build system for cases like this.  
The best solution would be to build zlib with symbol versioning.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

Thu, 23 Sep 2004 14:12:29 +0200
Peter Eisentraut <peter_e@gmx.net>
Re: Use of zlib
Peter Eisentraut wrote:

> Because we are too lazy to fine-tune the build system for cases like 
> this.
>
>The best solution would be to build zlib with symbol versioning.
>  
>
I'm not so sure. I think zlib is a commodity on most systems. You don't 
want to build it at all. Perhaps if I submit a patch that takes care of 
the fine-tuning?

- thomas



Thu, 23 Sep 2004 14:19:03 +0200
Thomas Hallgren <thhal@mailblocks.com>
Re: Use of zlib


Peter Eisentraut wrote:

>Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren:
>  
>
>> From what I can understand from the documentation, the only utility in
>>PostgreSQL that actually uses zlib is pg_dump? If so, why is the
>>postgres process linked with -lz?
>>    
>>
>
>Because we are too lazy to fine-tune the build system for cases like this.  
>The best solution would be to build zlib with symbol versioning.
>
>  
>

Or to be less lazy ;-) Is it such a huge task? There have been 
complaints before about our maximal linking.

cheers

andrew

Thu, 23 Sep 2004 09:44:50 -0400
Andrew Dunstan <andrew@dunslane.net>
Re: Use of zlib
Peter Eisentraut <peter_e@gmx.net> writes:
> Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren:
>> From what I can understand from the documentation, the only utility in
>> PostgreSQL that actually uses zlib is pg_dump? If so, why is the
>> postgres process linked with -lz?

> Because we are too lazy to fine-tune the build system for cases like this.  

I was just reading about a linker option (--as-needed or something like
that) that exists in more recent versions of the GNU linker.  It means
"only link the libraries actually referenced".  Applying this or local
equivalent where available would seem like a fairly painless way to
solve the problem, at least on some platforms.

I agree that trying to keep track of the set of libraries really needed
for each executable would be a hopeless task, but if we can make the
linker fix it for us ...

			regards, tom lane

Thu, 23 Sep 2004 10:48:54 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Use of zlib 


Tom Lane wrote:

>Peter Eisentraut <peter_e@gmx.net> writes:
>  
>
>>Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren:
>>    
>>
>>>From what I can understand from the documentation, the only utility in
>>>PostgreSQL that actually uses zlib is pg_dump? If so, why is the
>>>postgres process linked with -lz?
>>>      
>>>
>
>  
>
>>Because we are too lazy to fine-tune the build system for cases like this.  
>>    
>>
>
>I was just reading about a linker option (--as-needed or something like
>that) that exists in more recent versions of the GNU linker.  It means
>"only link the libraries actually referenced".  Applying this or local
>equivalent where available would seem like a fairly painless way to
>solve the problem, at least on some platforms.
>
>I agree that trying to keep track of the set of libraries really needed
>for each executable would be a hopeless task, but if we can make the
>linker fix it for us ...
>  
>

Only in version 2.15, which as you say is fairly new. And people who 
don't use GNU ld won't get it anyway. The docs ( 
http://sources.redhat.com/binutils/docs-2.15/ld/Options.html#Options ) say:

|--as-needed|

|--no-as-needed|
    This option affects ELF DT_NEEDED tags for dynamic libraries
    mentioned on the command line after the |--as-needed| option.
    Normally, the linker will add a DT_NEEDED tag for each dynamic
    library mentioned on the command line, regardless of whether the
    library is actually needed. |--as-needed| causes DT_NEEDED tags to
    only be emitted for libraries that satisfy some reference from
    regular objects. |--no-as-needed| restores the default behaviour.


I'm not sure I understand why keeping track of what we need for each 
executable is such a difficult task, though. I count 23 executables and 
a handful of libraries.  Is this such a herculean task?

cheers

andrew


Thu, 23 Sep 2004 11:41:16 -0400
Andrew Dunstan <andrew@dunslane.net>
Re: Use of zlib
Andrew Dunstan <andrew@dunslane.net> writes:
> I'm not sure I understand why keeping track of what we need for each 
> executable is such a difficult task, though. I count 23 executables and 
> a handful of libraries.  Is this such a herculean task?

"Handful"?  I count 32 AC_CHECK_LIB and AC_SEARCH_LIBS calls in
configure.in.

But that's only part of the problem.  Consider that many of these
libraries are interdependent to some extent.  The raw results from
Autoconf tell you that libfoo is available and we could link with it,
but not whether libfoo requires the previously-checked libbar to link.
We'd need to redo the whole library detection process for each
executable in order to be perfectly accurate about this.  Since the
Autoconf macros only cope with one LIBS list, that means throwing away
the Autoconf macros and writing our own.

Multiply that by N supported platforms, and you start to see why this is
a mess.  It's certainly work far out of proportion to the actual value
of not linking some unneeded libraries.

Adding "--as-needed" on platforms that support it seems like a
reasonable amount of work to expend in comparison to the value received.
Trying to do it manually does not.

			regards, tom lane

Thu, 23 Sep 2004 12:05:54 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Use of zlib 


Thomas Hallgren wrote:

> Problem:
> PL/Java use a JVM. On some platforms and with some JVM's (Sun's in 
> particular) a libzip.so is bundled that contains a 1.1.3 version of 
> functions also provided in zlib (why they do this is beyond me, but 
> they do so I'll have to live with it). PostgreSQL is linked with zlib 
> by default. This resuls in a conflict which resuls in a JVM crash.
>
> I can circumvein this crash by using LD_PRELOAD to force a load of the 
> JVM bundled libzip.so but I suspect that might result in a crash as 
> soon as PostgreSQL calls on zlib to do something. It's of course 
> possible to configure postgresql with --without-zlib also provided you 
> have accesst o the source.
>
> Question:
> From what I can understand from the documentation, the only utility in 
> PostgreSQL that actually uses zlib is pg_dump? If so, why is the 
> postgres process linked with -lz?
>
>

I did a small experiment by installing binutils 2.15 and adding 
-Wl,--as-needed to the LDFLAGS, as Tom had suggested might be useful.

This seemed to work quite well and trimmed back the needed libs quite a 
bit. However, when you configure --with-openssl, libz is again linked in.

Not sure where that leaves us.

cheers

andrew

Fri, 24 Sep 2004 10:12:59 -0400
Andrew Dunstan <andrew@dunslane.net>
Re: Use of zlib
Andrew Dunstan wrote:

>
>
> Thomas Hallgren wrote:
>
>> Problem:
>> PL/Java use a JVM. On some platforms and with some JVM's (Sun's in 
>> particular) a libzip.so is bundled that contains a 1.1.3 version of 
>> functions also provided in zlib (why they do this is beyond me, but 
>> they do so I'll have to live with it). PostgreSQL is linked with zlib 
>> by default. This resuls in a conflict which resuls in a JVM crash.
>>
>> I can circumvein this crash by using LD_PRELOAD to force a load of 
>> the JVM bundled libzip.so but I suspect that might result in a crash 
>> as soon as PostgreSQL calls on zlib to do something. It's of course 
>> possible to configure postgresql with --without-zlib also provided 
>> you have accesst o the source.
>>
>> Question:
>> From what I can understand from the documentation, the only utility 
>> in PostgreSQL that actually uses zlib is pg_dump? If so, why is the 
>> postgres process linked with -lz?
>>
>>
>
> I did a small experiment by installing binutils 2.15 and adding 
> -Wl,--as-needed to the LDFLAGS, as Tom had suggested might be useful.
>
> This seemed to work quite well and trimmed back the needed libs quite 
> a bit. However, when you configure --with-openssl, libz is again 
> linked in.

 From PL/Javas standpoint, I think it would be great if this change 
could be incorporated into the 8.0 release. The openssl issue is 
something I'll have to investigate. Perhaps it works using the libzip 
from the JRE if I use LD_PRELOAD. I'm happy to see that Sun has an open 
bug on the subject in their Bug Database (bug 4953050 if anyone is 
interested).

Regards,

Thomas Hallgren



Fri, 24 Sep 2004 17:17:13 +0200
Thomas Hallgren <thhal@mailblocks.com>
Re: Use of zlib

I am confused.  I just checked my backend binary and I don't see any
unusual libs required:

	$ ldd postgres
        libz.so => /usr/lib/libz.so (0x2833f000)
        libncurses.so.5 => /shlib/libncurses.so.5 (0x2834e000)
        libdl.so => /shlib/libdl.so (0x2838c000)
        libm.so => /shlib/libm.so.0.0 (0x2838f000)
        libgcc.so.1 => /shlib/libgcc.so.1 (0x283a0000)
        libc.so.2 => /shlib/libc.so.2 (0x283ac000)

The libncurses is a little unusual but I don't see libreadline in there
even though it is on the link line.

Are some OS's adding libs that aren't required by the binary?

---------------------------------------------------------------------------

Andrew Dunstan wrote:
> 
> 
> Thomas Hallgren wrote:
> 
> > Problem:
> > PL/Java use a JVM. On some platforms and with some JVM's (Sun's in 
> > particular) a libzip.so is bundled that contains a 1.1.3 version of 
> > functions also provided in zlib (why they do this is beyond me, but 
> > they do so I'll have to live with it). PostgreSQL is linked with zlib 
> > by default. This resuls in a conflict which resuls in a JVM crash.
> >
> > I can circumvein this crash by using LD_PRELOAD to force a load of the 
> > JVM bundled libzip.so but I suspect that might result in a crash as 
> > soon as PostgreSQL calls on zlib to do something. It's of course 
> > possible to configure postgresql with --without-zlib also provided you 
> > have accesst o the source.
> >
> > Question:
> > From what I can understand from the documentation, the only utility in 
> > PostgreSQL that actually uses zlib is pg_dump? If so, why is the 
> > postgres process linked with -lz?
> >
> >
> 
> I did a small experiment by installing binutils 2.15 and adding 
> -Wl,--as-needed to the LDFLAGS, as Tom had suggested might be useful.
> 
> This seemed to work quite well and trimmed back the needed libs quite a 
> bit. However, when you configure --with-openssl, libz is again linked in.
> 
> Not sure where that leaves us.
> 
> cheers
> 
> andrew
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sat, 25 Sep 2004 20:58:37 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Use of zlib
On Sat, 25 Sep 2004, Bruce Momjian wrote:

> I am confused.  I just checked my backend binary and I don't see any
> unusual libs required:
> 
> 	$ ldd postgres
>         libz.so => /usr/lib/libz.so (0x2833f000)
>         libncurses.so.5 => /shlib/libncurses.so.5 (0x2834e000)
>         libdl.so => /shlib/libdl.so (0x2838c000)
>         libm.so => /shlib/libm.so.0.0 (0x2838f000)
>         libgcc.so.1 => /shlib/libgcc.so.1 (0x283a0000)
>         libc.so.2 => /shlib/libc.so.2 (0x283ac000)
> 
> The libncurses is a little unusual but I don't see libreadline in there
> even though it is on the link line.
> 
> Are some OS's adding libs that aren't required by the binary?

This is the output in Fedora Core 2, and it includes readline:

$ ldd /usr/bin/postgres
        linux-gate.so.1 =>  (0x00240000)
        libpam.so.0 => /lib/libpam.so.0 (0x00cf1000)
        libssl.so.4 => /lib/libssl.so.4 (0x0014e000)
        libcrypto.so.4 => /lib/libcrypto.so.4 (0x00241000)
        libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x0052a000)
        libcom_err.so.2 => /lib/libcom_err.so.2 (0x00e56000)
        libz.so.1 => /usr/lib/libz.so.1 (0x00aca000)
        libreadline.so.4 => /usr/lib/libreadline.so.4 (0x00111000)
        libtermcap.so.2 => /lib/libtermcap.so.2 (0x00f25000)
        libcrypt.so.1 => /lib/libcrypt.so.1 (0x00406000)
        libresolv.so.2 => /lib/libresolv.so.2 (0x00f8a000)
        libnsl.so.1 => /lib/libnsl.so.1 (0x00215000)
        libdl.so.2 => /lib/libdl.so.2 (0x00908000)
        libm.so.6 => /lib/tls/libm.so.6 (0x008a5000)
        libc.so.6 => /lib/tls/libc.so.6 (0x00592000)
        libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00a57000)
        libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00ccd000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x004ca000)


-- 
/Dennis Bjï¿½rklund


Sun, 26 Sep 2004 06:28:15 +0200 (CEST)
Dennis Bjorklund <db@zigo.dhs.org>
Re: Use of zlib
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I am confused.  I just checked my backend binary and I don't see any
> unusual libs required:

> 	$ ldd postgres
>         libz.so => /usr/lib/libz.so (0x2833f000)
>         libncurses.so.5 => /shlib/libncurses.so.5 (0x2834e000)

I do not believe either of those are actually *used* by the backend.

			regards, tom lane

Sun, 26 Sep 2004 01:43:17 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Use of zlib 


Dennis Bjorklund wrote:

>This is the output in Fedora Core 2, and it includes readline:
>
>$ ldd /usr/bin/postgres
>        linux-gate.so.1 =>  (0x00240000)
>        libpam.so.0 => /lib/libpam.so.0 (0x00cf1000)
>        libssl.so.4 => /lib/libssl.so.4 (0x0014e000)
>        libcrypto.so.4 => /lib/libcrypto.so.4 (0x00241000)
>        libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x0052a000)
>        libcom_err.so.2 => /lib/libcom_err.so.2 (0x00e56000)
>        libz.so.1 => /usr/lib/libz.so.1 (0x00aca000)
>        libreadline.so.4 => /usr/lib/libreadline.so.4 (0x00111000)
>        libtermcap.so.2 => /lib/libtermcap.so.2 (0x00f25000)
>        libcrypt.so.1 => /lib/libcrypt.so.1 (0x00406000)
>        libresolv.so.2 => /lib/libresolv.so.2 (0x00f8a000)
>        libnsl.so.1 => /lib/libnsl.so.1 (0x00215000)
>        libdl.so.2 => /lib/libdl.so.2 (0x00908000)
>        libm.so.6 => /lib/tls/libm.so.6 (0x008a5000)
>        libc.so.6 => /lib/tls/libc.so.6 (0x00592000)
>        libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00a57000)
>        libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00ccd000)
>        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x004ca000)
>
>
>  
>

Now, to see the things postgres really needs, as opposed to things other 
libraries need or that we just throw in for good measure in case 
somebody needs them, run this:

  LDFLAGS=-Wl,--as-necessary ./configure

without any third-party options (like pam, openssl, kerberos ...), 
Alternatively, you can set the flag in src/Makefile.global

This should work on FC2, which I believe has binutils 2.15.

Almost every line above will disappear.

cheers

andrew

Sun, 26 Sep 2004 09:28:34 -0400
Andrew Dunstan <andrew@dunslane.net>
Re: Use of zlib

of course, I meant --as-needed rather than --as-necessary

Andrew Dunstan wrote:

>
>
> Dennis Bjorklund wrote:
>
>> This is the output in Fedora Core 2, and it includes readline:
>>
>> $ ldd /usr/bin/postgres
>>        linux-gate.so.1 =>  (0x00240000)
>>        libpam.so.0 => /lib/libpam.so.0 (0x00cf1000)
>>        libssl.so.4 => /lib/libssl.so.4 (0x0014e000)
>>        libcrypto.so.4 => /lib/libcrypto.so.4 (0x00241000)
>>        libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x0052a000)
>>        libcom_err.so.2 => /lib/libcom_err.so.2 (0x00e56000)
>>        libz.so.1 => /usr/lib/libz.so.1 (0x00aca000)
>>        libreadline.so.4 => /usr/lib/libreadline.so.4 (0x00111000)
>>        libtermcap.so.2 => /lib/libtermcap.so.2 (0x00f25000)
>>        libcrypt.so.1 => /lib/libcrypt.so.1 (0x00406000)
>>        libresolv.so.2 => /lib/libresolv.so.2 (0x00f8a000)
>>        libnsl.so.1 => /lib/libnsl.so.1 (0x00215000)
>>        libdl.so.2 => /lib/libdl.so.2 (0x00908000)
>>        libm.so.6 => /lib/tls/libm.so.6 (0x008a5000)
>>        libc.so.6 => /lib/tls/libc.so.6 (0x00592000)
>>        libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00a57000)
>>        libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00ccd000)
>>        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x004ca000)
>>
>>
>>  
>>
>
> Now, to see the things postgres really needs, as opposed to things 
> other libraries need or that we just throw in for good measure in case 
> somebody needs them, run this:
>
>  LDFLAGS=-Wl,--as-necessary ./configure
>
> without any third-party options (like pam, openssl, kerberos ...), 
> Alternatively, you can set the flag in src/Makefile.global
>
> This should work on FC2, which I believe has binutils 2.15.
>
> Almost every line above will disappear.
>
> cheers
>
> andrew
>

Sun, 26 Sep 2004 09:43:50 -0400
Andrew Dunstan <andrew@dunslane.net>
Re: Use of zlib