I have a potential patch for the glibc2 date problem; however I can't
test it because the snapshot won't build. 


Has this build error been cured recently, or do I have my own unique problem?


O/S: Linux 2.0.32
Glibc: 2.0.6
Gcc: 2.7.2.3
Flex: 2.5.4
Bison: 1.25
PostgreSQL: snapshot downloaded Jan 12

make[2]: Entering directory `/usr1/home/olly/mypackages/pgsql/src/backend/parse
r'
/usr/bin/bison -y -d gram.y
mv y.tab.c gram.c
mv y.tab.h parse.h
gcc -I../../include -I/usr/include/ncurses -I/usr/include/readline -O2 -m486  
-Wall -Wmissing-prototypes -I..  -Wno-error   -c analyze.c -o analyze.o
gcc -I../../include -I/usr/include/ncurses -I/usr/include/readline -O2 -m486  
-Wall -Wmissing-prototypes -I..  -Wno-error   -c gram.c -o gram.o
/usr/share/bison.simple: In function `yyparse':
/usr/share/bison.simple:327: warning: implicit declaration of function 
`yyerror'
/usr/share/bison.simple:387: warning: implicit declaration of function `yylex'
gcc -I../../include -I/usr/include/ncurses -I/usr/include/readline -O2 -m486  
-Wall -Wmissing-prototypes -I..  -Wno-error   -c keywords.c -o keywords.o

....

gcc -I../../include -I/usr/include/ncurses -I/usr/include/readline -O2 -m486  
-Wall -Wmissing-prototypes -I..  -Wno-error   -c scan.c -o scan.o
lex.yy.c:800: warning: no previous prototype for `yylex'
scan.l: In function `yylex':
scan.l:202: `ABORT' undeclared (first use this function)
scan.l:202: (Each undeclared identifier is reported only once
scan.l:202: for each function it appears in.)
scan.l: At top level:
scan.l:379: warning: no previous prototype for `yyerror'
scan.l: In function `yyerror':
scan.l:380: `ABORT' undeclared (first use this function)
lex.yy.c: At top level:
lex.yy.c:2103: warning: `yy_flex_realloc' defined but not used
make[2]: *** [scan.o] Error 1
make[2]: Leaving directory `/usr1/home/olly/mypackages/pgsql/src/backend/parser
'

scan.l version is:
$Header: /usr/local/cvsroot/pgsql/src/backend/parser/scan.l,v 1.34 1998/01/05 
16:39:19 momjian Exp $


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver

PGP key from public servers; key ID 32B8FAA1

Unsolicited email advertisements are not welcome; any person sending
such will be invoiced for telephone time used in downloading together
with a £25 administration charge.



Thu, 15 Jan 1998 12:45:41 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Cannot build recent snapshot
On Thu, 15 Jan 1998, Oliver Elphick wrote:

> I have a potential patch for the glibc2 date problem; however I can't
> test it because the snapshot won't build. 

	remove src/backend/parser/scan.c...that should fix your problem



Thu, 15 Jan 1998 08:06:46 -0500 (EST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Cannot build recent snapshot
The Hermit Hacker wrote:
  >On Thu, 15 Jan 1998, Oliver Elphick wrote:
  >
  >> I have a potential patch for the glibc2 date problem; however I can't
  >> test it because the snapshot won't build. 
  >
  >	remove src/backend/parser/scan.c...that should fix your problem
  >

Thanks, it did.


The patch for glibc2 dates is attached.  With this applied, a Linux system
with libc6 (glibc2) passes all the date and time related regression tests.




Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver

PGP key from public servers; key ID 32B8FAA1

Unsolicited email advertisements are not welcome; any person sending
such will be invoiced for telephone time used in downloading together
with a ï¿½25 administration charge.
Thu, 15 Jan 1998 18:02:25 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Patch for glibc2 date problems 
> The patch for glibc2 dates is attached.  With this applied, a Linux system
> with libc6 (glibc2) passes all the date and time related regression tests.

It looks as though this patch is a bit Linux-specific (or specific to some version of glibc which has only been tested on
Linux).

Can we wait until glibc2 settles down, or provide this as an add-on patch rather than merging it into the main tree? I hate
adding machine-specific code into otherwise general code...

Another possibility would be to add a new #define variable like HAVE_FUNNY_LIBRARY in config.h or in linux.h so we can
possibly use this with other ports if necessary in the future.

I'm planning on installing RH5.0 sometime soon (I have a clean disk so can fall back to RH4.2). I'm sure I'll sound more
sympathetic by then  :)

                                                    - Tom


Fri, 16 Jan 1998 02:16:32 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Patch for glibc2 date problems
Oliver Elphick wrote:
> 
> I have a potential patch for the glibc2 date problem; however I can't
> test it because the snapshot won't build.
> 
> Has this build error been cured recently, or do I have my own unique problem?
> 
> O/S: Linux 2.0.32
> Glibc: 2.0.6
> Gcc: 2.7.2.3
> Flex: 2.5.4
> Bison: 1.25
> PostgreSQL: snapshot downloaded Jan 12
> 
> make[2]: Entering directory `/usr1/home/olly/mypackages/pgsql/src/backend/parse
> r'
> /usr/bin/bison -y -d gram.y
> mv y.tab.c gram.c
> mv y.tab.h parse.h
> gcc -I../../include -I/usr/include/ncurses -I/usr/include/readline -O2 -m486
> -Wall -Wmissing-prototypes -I..  -Wno-error   -c analyze.c -o analyze.o
> gcc -I../../include -I/usr/include/ncurses -I/usr/include/readline -O2 -m486
> -Wall -Wmissing-prototypes -I..  -Wno-error   -c gram.c -o gram.o
> /usr/share/bison.simple: In function `yyparse':
> /usr/share/bison.simple:327: warning: implicit declaration of function
> `yyerror'
> /usr/share/bison.simple:387: warning: implicit declaration of function `yylex'
> gcc -I../../include -I/usr/include/ncurses -I/usr/include/readline -O2 -m486
> -Wall -Wmissing-prototypes -I..  -Wno-error   -c keywords.c -o keywords.o
> 
> ....
> 
> gcc -I../../include -I/usr/include/ncurses -I/usr/include/readline -O2 -m486
> -Wall -Wmissing-prototypes -I..  -Wno-error   -c scan.c -o scan.o
> lex.yy.c:800: warning: no previous prototype for `yylex'
> scan.l: In function `yylex':
> scan.l:202: `ABORT' undeclared (first use this function)
> scan.l:202: (Each undeclared identifier is reported only once
> scan.l:202: for each function it appears in.)
> scan.l: At top level:
> scan.l:379: warning: no previous prototype for `yyerror'
> scan.l: In function `yyerror':
> scan.l:380: `ABORT' undeclared (first use this function)
> lex.yy.c: At top level:
> lex.yy.c:2103: warning: `yy_flex_realloc' defined but not used
> make[2]: *** [scan.o] Error 1
> make[2]: Leaving directory `/usr1/home/olly/mypackages/pgsql/src/backend/parser
> '

My solution was to remove scan.c from that directory and let it be
rebuilt
from scan.l - the Makefile does not remove scan.c on "make clean"
I had a couple of warnings, but no premature termination of make.
(Linux 2.0.29, libc-5.4.33)

Hope this helps

Darrell

Thu, 15 Jan 1998 21:32:05 -0800
"Darrell A. Escola" <descola@ibm.net>
Re: [HACKERS] Cannot build recent snapshot
"Thomas G. Lockhart" wrote:
  >> The patch for glibc2 dates is attached.  With this applied, a Linux system
  >> with libc6 (glibc2) passes all the date and time related regression tests.
  >
  >It looks as though this patch is a bit Linux-specific (or specific to some v
      >ersion of glibc which has only been tested on
  >Linux).

I don't have experience of using glibc2 on any other type of machine.

However, isn't part of the point of it to remove inter-machine differences?

  >
  >Can we wait until glibc2 settles down, or provide this as an add-on patch ra
      >ther than merging it into the main tree? I hate
  >adding machine-specific code into otherwise general code...

I guess that's up to you.  
  >
  >Another possibility would be to add a new #define variable like HAVE_FUNNY_L
      >IBRARY in config.h or in linux.h so we can

Why isn't
 #if __GLIBC__ < 2
enough for this?
  >possibly use this with other ports if necessary in the future.
  >

I don't have experience of using glibc2 on any other type of machine.
However, isn't part of the point of it to remove inter-machine differences?

  >I'm planning on installing RH5.0 sometime soon (I have a clean disk so can f
      >all back to RH4.2). I'm sure I'll sound more
  >sympathetic by then  :)
  >
  >                                                    - Tom

My assumption was that any system using glibc2 would not have a broken 
rint() function; so the general change to TMODULO would be justified.

The change of the test of `var != 0' to `var != rint(var)' should not break
anything, even if var is non-zero.  It is merely saying, don't use
decimal points if there's no decimal part.

The remaining part of the patch is to force the undefinition of
HAVE_INT_TIMEZONE; again this is glibc2-specific, but I don't know
any reason to suppose it wouldn't be needed on any machine with glibc2.

It would really be helpful to have someone on a non-Linux machine test
it; but is there anyone?


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver

PGP key from public servers; key ID 32B8FAA1

Unsolicited email advertisements are not welcome; any person sending
such will be invoiced for telephone time used in downloading together
with a £25 administration charge.



Fri, 16 Jan 1998 06:42:44 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Patch for glibc2 date problems 
On Fri, 16 Jan 1998, Thomas G. Lockhart wrote:

> > The patch for glibc2 dates is attached.  With this applied, a Linux system
> > with libc6 (glibc2) passes all the date and time related regression tests.
 
> It looks as though this patch is a bit Linux-specific (or specific to
some version of glibc which has only been tested on Linux). 
 
> Can we wait until glibc2 settles down, or provide this as an add-on
patch rather than merging it into the main tree? I hate adding
machine-specific code into otherwise general code... 
 
> Another possibility would be to add a new #define variable like
HAVE_FUNNY_LIBRARY in config.h or in linux.h so we can possibly use this
with other ports if necessary in the future. 

How about making this #define 'HAVE_GLIBC' ???

Maarten

_____________________________________________________________________________
|     Maarten Boekhold, Faculty of Electrical Engineering TU Delft,   NL    |
|           Computer Architecture and Digital Technique section             |
|                          M.Boekhold@et.tudelft.nl                         |
-----------------------------------------------------------------------------


Fri, 16 Jan 1998 10:52:06 +0100 (MET)
Maarten Boekhold <maartenb@dutepp0.et.tudelft.nl>
Re: [HACKERS] Patch for glibc2 date problems
> 
> "Thomas G. Lockhart" wrote:
>   >> The patch for glibc2 dates is attached.  With this applied, a Linux system
>   >> with libc6 (glibc2) passes all the date and time related regression tests.
>   >
>   >It looks as though this patch is a bit Linux-specific (or specific to some v
>       >ersion of glibc which has only been tested on
>   >Linux).
> 
> I don't have experience of using glibc2 on any other type of machine.
> 
> However, isn't part of the point of it to remove inter-machine differences?
> 
>   >
>   >Can we wait until glibc2 settles down, or provide this as an add-on patch ra
>       >ther than merging it into the main tree? I hate
>   >adding machine-specific code into otherwise general code...
> 
> I guess that's up to you.  
>   >
>   >Another possibility would be to add a new #define variable like HAVE_FUNNY_L
>       >IBRARY in config.h or in linux.h so we can
> 
> Why isn't
>  #if __GLIBC__ < 2
> enough for this?
>   >possibly use this with other ports if necessary in the future.
>   >
> 
> I don't have experience of using glibc2 on any other type of machine.
> However, isn't part of the point of it to remove inter-machine differences?
> 
>   >I'm planning on installing RH5.0 sometime soon (I have a clean disk so can f
>       >all back to RH4.2). I'm sure I'll sound more
>   >sympathetic by then  :)
>   >
>   >                                                    - Tom
> 
> My assumption was that any system using glibc2 would not have a broken 
> rint() function; so the general change to TMODULO would be justified.
> 
> The change of the test of `var != 0' to `var != rint(var)' should not break
> anything, even if var is non-zero.  It is merely saying, don't use
> decimal points if there's no decimal part.
> 
> The remaining part of the patch is to force the undefinition of
> HAVE_INT_TIMEZONE; again this is glibc2-specific, but I don't know
> any reason to suppose it wouldn't be needed on any machine with glibc2.
> 
> It would really be helpful to have someone on a non-Linux machine test
> it; but is there anyone?

Our code is complicated enough without adding patches for OS bugs.  A
good place for the patch is the Linux FAQ.  If it really becomes a
problem, we can put the patch as a separate file in the distribution,
and mention it in the INSTALL instructions.  If you really want to get
fancy, as I did with the flex bug, you can run a test at compile time to
see if the bug exists.

-- 
Bruce Momjian
maillist@candle.pha.pa.us

Fri, 16 Jan 1998 09:17:47 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Patch for glibc2 date problems
"Thomas G. Lockhart" wrote:
  >The other good possibility is for Oliver to develop a patch kit (he has done
      > so
  >already) which we can include in the v6.3 distribution to be applied only to
  >linux/glibc2. When the beta settles down perhaps Oliver can generate a new p
      >atch
  >kit which we can include?

I'm happy to remake the patch in a week or so.  However I'd like to be clear
what you think is 'Linux/glibc-specific' (and therefore undesirable); obviously
it is better to make changes as general as possible.

I'm attaching the previous patch I made (which, of course, may not
actually work with the current release - I haven't tried yet) so as to comment
on it:

1.  I use `#if __GLIBC__ < 2' to test for glibc, where the change is only
    for glibc.  So I use it to force HAVE_INT_TIMEZONE to be undefined
    and to redefine TMODULO(); incidentally, I used the hint in the existing
    source that modf() was sometimes broken and chose to use glibc's
    unbroken modf(). 

    [ Incidentally, is modf() _still_ broken on whatever platforms it was
    broken on? ]

2.  The second set of changes are to do with unnecessary printing of n.00
    instead of n, where the test was whether fsec is non-zero.  With the
    glibc version, fsec is often non-zero but integer, so I substituted
    the test (fsec != rint(fsec)) for (fsec != 0).  This seems to me to
    be a completely general change that will work correctly on all
    implementations.

Now I can understand that you may choose to ignore the existence of glibc 
for the standard code (not that I agree, mind you) but I really don't see
how I could make the changes any more general than I have.


Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver

PGP key from public servers; key ID 32B8FAA1
Tue, 03 Feb 1998 12:02:08 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] (: JDBC+(Sun ~3:pm MST) CVS :) -also question about
	regression tests
> 1.  I use `#if __GLIBC__ < 2' to test for glibc, where the change is only
>     for glibc.  So I use it to force HAVE_INT_TIMEZONE to be undefined
>     and to redefine TMODULO(); incidentally, I used the hint in the existing
>     source that modf() was sometimes broken and chose to use glibc's
>     unbroken modf().
>
>     [ Incidentally, is modf() _still_ broken on whatever platforms it was
>     broken on? ]

Probably, since it was on Solaris, which should have been mature enough by now to not exhibit the problem :(

> 2.  The second set of changes are to do with unnecessary printing of n.00
>     instead of n, where the test was whether fsec is non-zero.  With the
>     glibc version, fsec is often non-zero but integer, so I substituted
>     the test (fsec != rint(fsec)) for (fsec != 0).  This seems to me to
>     be a completely general change that will work correctly on all
>     implementations.

The second set of changes does work in general, but is an extra math library call which is _only_ required for glibc2. This
is the math library bug I was alluding to earlier.

The extra overhead of this unnecessary function call should not be required for all platforms, since it only there as a bug
fix workaround for one platform. "fsec" is "fractional seconds", not "fractional seconds except for glibc2 where it could be
an integer too".

> Now I can understand that you may choose to ignore the existence of glibc
> for the standard code (not that I agree, mind you) but I really don't see
> how I could make the changes any more general than I have.

Heh, don't get me wrong, I _love_ glibc2 and am looking forward to being able to write thread-safe code using it. However,
the glibc2 you are using seems to be v2.0.5 or thereabouts and it still apparently has a few rough edges, this math rounding
problem being one of them.

As you know, the problem is not really with rint() but with whatever allowed the fsec variable to become a non-zero integer.
_That_ is the thing which should be patched with glibc2-specific #ifdefs. Between all other tested platforms (~10?) the
behavior is uniform and correct. So, I'll be at least somewhat happier with the patches if you identify the place where fsec
is going bad and fix that for glibc2, rather than coding a general workaround later in the code.

The more I think about it, the more I want to go back and rip out the general workaround I put in to fix Solaris. Then, we
can isolate up in the top of the file some #ifdef equivalent functions for Solaris and for glibc2.

Someday the glibc2 behavior will be fixed, and it should be easy to then remove the workaround code.

                                                - Tom


Tue, 03 Feb 1998 15:12:30 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] (: JDBC+(Sun ~3:pm MST) CVS :) -also question about
	regression tests
Bruce Momjian wrote:
  >I am noticing a lot of animal lovers in the group.

Yes, and I suppose I'm another one.

I'm 46 with a dog, three cats, a wife and five children, and we live
here on the Isle of Wight (Britain) and have done since 1996.  The
children range from 20 down to 13, and only three are now at home all
the time. My prime leisure activity (apart from computers...) is singing,
mainly in church.

I started my working life as an accountant, but then, as the ad says,
I discovered computers. For a number of years I worked with the PICK
database and one of its Unix copies, UniVerse.  Now I'm doing sufficient
consultancy to keep all the menagerie fed and housed, and doing my best
to promote Open Source software (note the new advertisement-speak for
free software!).

I maintain the PostgreSQL package for Debian GNU/Linux, but haven't
really found my way into the real meat of the source code yet. I am
just starting the design of a system for a pharmaceutical supplies
manufacturer, to replace his existing UniVerse-based one.


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver

PGP key from public servers; key ID 32B8FAA1



Fri, 20 Feb 1998 19:18:03 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Who is everyone? 
-----BEGIN PGP SIGNED MESSAGE-----

Then Bruce Momjian wrote:
> I am noticing a lot of animal lovers in the group.

I guess I'm sorta one, too (I'm just too allergic to actually keep
them at my own house).

I'm a 31 year old father of two (a third is due 29Mar).  I recently
finished my BS in Electrical and Computer Engineering at Carnegie
Mellon University, located in Pittsburgh, PA, USA.  As it happens,
I've been a full-time employee here for the last five years, doing
system administration.  Currently, I'm riding herd on a coupld of
HP9000s, a Sequent, two DG AViiONs, a VAXcluster, and an SP/2.  My
only real project in PostgreSQL was actually written against 4.2 in
QUEL (a helpdesk app that is still listed on the PostgreSQL home page
in spite of the fact that it hasn't been updated to keep it even
functional.  NOTE TO MYSELF: do something about that!).  I ride horse,
and watch fish.  Since the allergies started, I've had to stop playing
with the dogs and cats.

- -- 


=====================================================================
| "If you're all through trying to burn the field down, will you    |
| kindly get up and tell me why you're sitting in a fruit field,    |
| stark naked, frying peaches?"                                     |
=====================================================================
| Finger geek@andrew.cmu.edu for my public key.                     |
=====================================================================

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2

iQBVAwUBNO3deYdzVnzma+gdAQGZ3gH8DDD4TLacIyjf2Al30pUm1iIhJZUp8HYp
+sgoMDO4fuHfigWdrxBHOuGDyiQ+9i8vkt7lNfShinckzju0aIaGsg==
=2EQp
-----END PGP SIGNATURE-----


20 Feb 1998 14:46:01 -0500
<geek+@cmu.edu>
Re: [HACKERS] Who is everyone? 
Bruce Momjian wrote:
  >All tables are created with default permissions for SELECT to PUBLIC, so
  >views are no different.

Is this not contrary to the SQL standard?  I understood that SQL tables
are created with permissions for their creator only; any permissions for
other users must be granted explicitly.  According to "SQL The Standard
Handbook" (Cannan & Otten, 1993), the owner of the schema in which a table
is created is given a full set of privileges, and no other user can access
the table or even discover that it exists!

It certainly seems undesirable to give automatic access to data of unknown
sensitivity.  Surely the default permission should be for the table's
creator alone or for the owner of the PostgreSQL database (which I suppose 
is equivalent to the `schema').

I see that Jan Wieck has posted a method for preventing world readability;
perhaps this should just be flagged as a configurable option.


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver

PGP key from public servers; key ID 32B8FAA1



Mon, 23 Feb 1998 21:53:57 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Here it is - view permissions 

Oliver Elphick wrote:
>
> Bruce Momjian wrote:
>   >All tables are created with default permissions for SELECT to PUBLIC, =
> so
>   >views are no different.
>
> Is this not contrary to the SQL standard?  I understood that SQL tables
> are created with permissions for their creator only; any permissions for
> other users must be granted explicitly.  According to "SQL The Standard
> Handbook" (Cannan & Otten, 1993), the owner of the schema in which a tabl=
> e
> is created is given a full set of privileges, and no other user can acces=
> s
> the table or even discover that it exists!

                             ^^^^^^^^^^^^^^!!!

    Ha!

    The next table we must hide and create a view on :-)

    This time the view must check if the user has at least SELECT
    permission on the table/view and hide  rows.  More  tricky  -
    I'll try to work it out. But not doday - I'm tired and I know
    what can happen then (saying '... and make even  this  little
    thing'  at  23:00  to  reach the state of 22:59 at 04:00 :-).
    Good night to all!

    But a last word: There are  even  more  such  tables  as  the
    tables/views  are also reflected in pg_attributes, pg_rewrite
    and so on. Even if here only the Oid shows up.

    If we really want to get all this up to the highest level, we
    need  sometimes  a proacl field in pg_proc ... *Ack* - Bruce,
    *Outch* - no - not the pumpgun - *Help*

    :-)

>
> It certainly seems undesirable to give automatic access to data of unknow=
> n
> sensitivity.  Surely the default permission should be for the table's
> creator alone or for the owner of the PostgreSQL database (which I suppos=
> e =
>
> is equivalent to the `schema').
>
> I see that Jan Wieck has posted a method for preventing world readability=
> ;
> perhaps this should just be flagged as a configurable option.

    But configurable at compile  time  -  not  a  runtime  option
    please.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck@debis.com (Jan Wieck) #



Mon, 23 Feb 1998 23:27:59 +0100 (MET)
jwieck@debis.com (Jan Wieck)
Re: [HACKERS] Here it is - view permissions
 > 
> Bruce Momjian wrote:
>   >All tables are created with default permissions for SELECT to PUBLIC, so
>   >views are no different.
> 
> Is this not contrary to the SQL standard?  I understood that SQL tables
> are created with permissions for their creator only; any permissions for
> other users must be granted explicitly.  According to "SQL The Standard
> Handbook" (Cannan & Otten, 1993), the owner of the schema in which a table
> is created is given a full set of privileges, and no other user can access

Will be the default in 6.3, I think.
 
> the table or even discover that it exists!
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Not in 6.3, or maybe ever.  Too much OO stuff for that, I think.

> 
> It certainly seems undesirable to give automatic access to data of unknown
> sensitivity.  Surely the default permission should be for the table's
> creator alone or for the owner of the PostgreSQL database (which I suppose 
> is equivalent to the `schema').
> 
> I see that Jan Wieck has posted a method for preventing world readability;
> perhaps this should just be flagged as a configurable option.


-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

Mon, 23 Feb 1998 17:35:09 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Here it is - view permissions

So I've never gotten the distinction -- what makes postgreSQL an
object oriented database, aside from the oid attribute and class
inheritance (which could work a little better.. no way to find out the
child class of a tuple in a select from parent_class* query).

and what makes it relational?  the fact that it can do joins?

in confused delerium,
--brett

On Mon, 23 February 1998, at 17:35:09, Bruce Momjian wrote:

> Will be the default in 6.3, I think.
>  
> > the table or even discover that it exists!
>                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> Not in 6.3, or maybe ever.  Too much OO stuff for that, I think.
> 
> > 
> > It certainly seems undesirable to give automatic access to data of unknown
> > sensitivity.  Surely the default permission should be for the table's
> > creator alone or for the owner of the PostgreSQL database (which I suppose 
> > is equivalent to the `schema').
> > 
> > I see that Jan Wieck has posted a method for preventing world readability;
> > perhaps this should just be flagged as a configurable option.
> 
> 
> -- 
> Bruce Momjian                          |  830 Blythe Avenue
> maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
>   +  If your life is a hard drive,     |  (610) 353-9879(w)
>   +  Christ can be your backup.        |  (610) 853-3000(h)

Mon, 23 Feb 1998 15:06:32 -0800
Brett McCormick <brett@work.chicken.org>
Re: [HACKERS] Here it is - view permissions
> So I've never gotten the distinction -- what makes postgreSQL an
> object oriented database...

It is not. It is object-relational, which is relational with some object-oriented
features. The type/function extensibility is the most visible of these features.

> and what makes it relational?  the fact that it can do joins?

And allows one to use other aspects of relational algebra.

                                                 - Tom


Tue, 24 Feb 1998 02:37:02 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Here it is - view permissions
On Tue, 24 February 1998, at 02:37:02, Thomas G. Lockhart wrote:

> > So I've never gotten the distinction -- what makes postgreSQL an
> > object oriented database...
> 
> It is not. It is object-relational, which is relational with some object-oriented
> features. The type/function extensibility is the most visible of these features.

Whoops, that should have been a no-brainer (because I already knew that)

> 
> > and what makes it relational?  the fact that it can do joins?
> 
> And allows one to use other aspects of relational algebra.

what is relational algebra?  operations on entire tuples?

Mon, 23 Feb 1998 21:00:31 -0800
Brett McCormick <brett@work.chicken.org>
Re: [HACKERS] Here it is - view permissions
Last week, I wrote this, but no-one has answered. :(  
  >I am currently designing a database that I expect to put into use after
  >6.4 is released.  This makes use of inheritance, and I would like to
  >ask about how inheritance will relate to the handling of constraints.
  >
  >1. I would like to be able to say:
  >
  >     create table job
  >       (
  >          ...
  >          resid char(4) not null references resource*(id),
  >          ...
  >       )
  >
  >   to indicate that the foreign key constraint would be satisfied by the
  >   presence of the desired item in any class of the inheritance tree 
starting
  >   at resource.  The parser does not recognise this syntax at present.
  >   (This is parallel to `select ... from class*', by which we can currently
  >   list all items in an inheritance tree.)
  >
  >2. Will all constraints on a class be inherited along with the column
  >   definitions?
  >
  >   If constraints are inherited, there is the possibility of conflict or
  >   redefinition.
  >
  >   In single inheritance, could a constraint be redefined by being restated 
  >   in the descendent?
  >
  >   In multiple inheritance, a conflict of column types causes an error; how
  >   will a conflict of constraint names be handled, if the check condition
  >   is different?  (Options: error; drop the constraint; require a new
  >   definition of the constraint in the descendent class.)
  >
  >   At the moment, check constraints are inherited and are silently mangled
  >   by prefixing the class name; this can lead to an impossible combination
  >   of constraints, which could be solved if redefinition were possible.
  >
  >   Example:
  >
  >junk=> create table aa (id char(4) check (id > 'M'), name text);
  >CREATE
  >junk=> create table bb (id char(4) check (id < 'M'), qty int);
  >CREATE
  >junk=> create table ab (value money) inherits (aa, bb);
  >CREATE
  >junk=> insert into ab values ('ABCD', 5);
  >ERROR:  ExecAppend: rejected due to CHECK constraint aa_id
  >junk=> insert into ab values ('WXYZ', 5);
  >ERROR:  ExecAppend: rejected due to CHECK constraint bb_id
  >
  >   We could perhaps allow syntax such as:
  >
  >     create table ab (..., constraint id check (id > 'E' and id < 'Q'))
  >        inherits (aa, bb)
  >        undefine (constraint aa_id, constraint bb_id)
  >
  >   Is this feasible?
  >
  >   At present, primary key definitions are not inherited. Could they be?
  >   (either to share the same index or have a new one for each class, at
  >   the designer's option.)
  >
  

Having thought a bit more about it, I feel that all constraints should be
inherited, including primary key, and that the primary key index should
serve the entire tree, so that `select key from foo*' can be guaranteed
not to contain any duplicates.  Where there are non-identical constraints
of the same name, there ought to be a method for selecting the one to use
and also a method of redefining the constraint. (These ideas are taken
from the Eiffel language.)

  CREATE TABLE foobar (<field_definitions_and_constraints>)
    INHERITS (foo, bar)
    CHOOSE (constraint foo_id)
    REDEFINE (constraint id check (...))

The CHOOSE clause says that constraint foo_id is to be used in preference
to bar_id; the REDEFINE clause allows id (which would otherwise be foo_id, 
from the CHOOSE clause) to be redefined as something new.

The CHOOSE clause would only be valid if there were constraints of the
same name in more than one ancestor.  If the constraints were actually the
same, it would be legal but unnecessary.

The REDEFINE clause would be legal if any ancestor contained the redefined
constraint. 

I am thinking only of check constraints, at the moment.  As I said above,
I think that the primary key should apply to all descendants, using the same
index.  I am not sure about foreign keys, NOT NULL and UNIQUE.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver

PGP key from public servers; key ID 32B8FAA1

                 ========================================
    Come to me, all you who labour and are heavily laden, and I will
    give you rest. Take my yoke upon you, and learn from me; for I am
    meek and lowly in heart, and you shall find rest for your souls.
    For my yoke is easy and my burden is light.    (Matthew 11: 28-30)



Mon, 20 Apr 1998 21:45:08 +0200
"Oliver Elphick" <olly@lfix.co.uk>
Re: Constraints and inheritance 
Oliver Elphick wrote:
> 
> I am thinking only of check constraints, at the moment.  As I said above,
> I think that the primary key should apply to all descendants, using the same
> index.  I am not sure about foreign keys, NOT NULL and UNIQUE.

This can be done: we could store table' oid in btree index items
and keep index items for all tables in one index.

Vadim

Tue, 21 Apr 1998 09:18:31 +0800
"Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Re: [HACKERS] Re: Constraints and inheritance
I have a bug reported on the Debian package of 6.3.2:

 > videotapes=> grant all on tapes to www-data;
 > ERROR:  aclparse: non-existent user "www"

It is, in fact, impossible to create the user www-data:

  template1=> create user www-data;
  ERROR:  parser: parse error at or near "-"
  template1=> create user 'www-data';
  ERROR:  parser: parse error at or near "'"

So there are two problems:

1. The error message

	 `ERROR:  aclparse: non-existent user "www"'

   is incorrect.  The parser should actually object to the `-' character; it
   appears to be silently dropping the `-data'.

2. The range of possible user names is not the same as the range of possible
   Unix login names.  However, the manual pages do not define what characters
   are valid.  The SQL standard is silent on this point; it simply regards
   the current user name as an identifier supplied by the system. On the
   other hand, it is clear that PostgreSQL regards a user name as an SQL
   identifier, so that there is no distinction of case and no punctuation
   characters are allowed.

Is it possible to make the parser accept the full range of Unix login names,
including some punctuation characters and upper- and lower-case letters?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver

PGP key from public servers; key ID 32B8FAA1

                 ========================================
    Come to me, all you who labour and are heavily laden, and I will
    give you rest. Take my yoke upon you, and learn from me; for I am
    meek and lowly in heart, and you shall find rest for your souls.
    For my yoke is easy and my burden is light.    (Matthew 11: 28-30)



Sun, 26 Apr 1998 08:02:09 +0200
"Oliver Elphick" <olly@lfix.co.uk>
User names cannot contain `-'
> 
> I have a bug reported on the Debian package of 6.3.2:
> 
>  > videotapes=> grant all on tapes to www-data;
>  > ERROR:  aclparse: non-existent user "www"
> 
> It is, in fact, impossible to create the user www-data:
> 
>   template1=> create user www-data;
>   ERROR:  parser: parse error at or near "-"
>   template1=> create user 'www-data';
>   ERROR:  parser: parse error at or near "'"
> 
> So there are two problems:
> 
> 1. The error message
> 
> 	 `ERROR:  aclparse: non-existent user "www"'
> 
>    is incorrect.  The parser should actually object to the `-' character; it
>    appears to be silently dropping the `-data'.
> 
> 2. The range of possible user names is not the same as the range of possible
>    Unix login names.  However, the manual pages do not define what characters
>    are valid.  The SQL standard is silent on this point; it simply regards
>    the current user name as an identifier supplied by the system. On the
>    other hand, it is clear that PostgreSQL regards a user name as an SQL
>    identifier, so that there is no distinction of case and no punctuation
>    characters are allowed.

We allow undercores, but not dashes.

-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

Sun, 26 Apr 1998 10:11:00 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] User names cannot contain `-'
"Oliver Elphick" <olly@lfix.co.uk> writes:

> I have a bug reported on the Debian package of 6.3.2:
> 
>  > videotapes=> grant all on tapes to www-data;
>  > ERROR:  aclparse: non-existent user "www"
> 
> It is, in fact, impossible to create the user www-data:
> 
>   template1=> create user www-data;
>   ERROR:  parser: parse error at or near "-"
>   template1=> create user 'www-data';
>   ERROR:  parser: parse error at or near "'"

I believe createuser program did that for me (either that or it was
preinstalled; I don't recall.)  I did not issue a direct SQL command
to do it, so I think it is likely that createuser did it.

In any case:


template1=> select usename, usesysid, valuntil from pg_shadow;
usename |usesysid|valuntil                    
--------+--------+----------------------------
postgres|      31|Sat Jan 31 00:00:00 2037 CST
www-data|      33|Sat Jan 31 00:00:00 2037 CST
jgoerzen|    1000|Sat Jan 31 00:00:00 2037 CST
(3 rows)


-- 
John Goerzen        Linux, Unix programming           jgoerzen@complete.org |
Developer, Debian GNU/Linux (Free powerful OS upgrade)       www.debian.org |
----------------------------------------------------------------------------+
``You'll notice that this scanner, Bill [Gates]...''  <Blue Screen of Death>
``Whoa!''  <Applause>   ``Moving right along....''  -- Microsoft (Comdex
      video at: http://cnn.com/TECH/computing/9804/20/gates.comdex/index.html

27 Apr 1998 20:16:39 -0500
John Goerzen <jgoerzen@complete.org>
Re: User names cannot contain `-'
John Goerzen wrote:
 
  >>                 Were you able to create a user `www-data'?
  >> I get:
  >
  >I believe that the createuser script did it for me.  

Yes, it does.  Very inconsistent!

  >> I agree that there is a bug, but it is that the error message is wrong!
  >
  >If grant would permit the username to be quoted in ' characters, then
  >the problem ought to go away, I think.

Currently, it appears that the authors don't want to change, so it is
necessary to specify a postgres user-id when connecting.

However there is, as you say in another mail, no convenient way of
doing that automatically.  We need an environment variable or a
command-line option to specify the user and (optionally) password.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver

PGP key from public servers; key ID 32B8FAA1

                 ========================================
    Come to me, all you who labour and are heavily laden, and I will
    give you rest. Take my yoke upon you, and learn from me; for I am
    meek and lowly in heart, and you shall find rest for your souls.
    For my yoke is easy and my burden is light.    (Matthew 11: 28-30)



Tue, 28 Apr 1998 12:04:43 +0200
"Oliver Elphick" <olly@lfix.co.uk>
Re: Bug#21681: postgresql: Doesn't allow granting to www-data 
On 27 Apr 1998, John Goerzen wrote:

> "Oliver Elphick" <olly@lfix.co.uk> writes:
> 
> > I have a bug reported on the Debian package of 6.3.2:
> > 
> >  > videotapes=> grant all on tapes to www-data;
> >  > ERROR:  aclparse: non-existent user "www"
> > 
> > It is, in fact, impossible to create the user www-data:
> > 
> >   template1=> create user www-data;
> >   ERROR:  parser: parse error at or near "-"
> >   template1=> create user 'www-data';
> >   ERROR:  parser: parse error at or near "'"
> 
> I believe createuser program did that for me (either that or it was
> preinstalled; I don't recall.)  I did not issue a direct SQL command
> to do it, so I think it is likely that createuser did it.
> 
> In any case:
> 
> 
> template1=> select usename, usesysid, valuntil from pg_shadow;
> usename |usesysid|valuntil                    
> --------+--------+----------------------------
> postgres|      31|Sat Jan 31 00:00:00 2037 CST
> www-data|      33|Sat Jan 31 00:00:00 2037 CST
> jgoerzen|    1000|Sat Jan 31 00:00:00 2037 CST
> (3 rows)

	This might have already been gone over, but if this was an upgrade
from a previous release, its possible that this was created with a
'dump/reload'?



Tue, 28 Apr 1998 08:35:29 -0400 (EDT)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Re: User names cannot contain `-'
> >> Were you able to create a user `www-data'?
> >I believe that the createuser script did it for me.
> Yes, it does.  Very inconsistent!
> >> I agree that there is a bug, but it is that the error message is 
> >> wrong!
> >If grant would permit the username to be quoted in ' characters, then
> >the problem ought to go away, I think.
> 
> Currently, it appears that the authors don't want to change, so it is
> necessary to specify a postgres user-id when connecting.

"Don't want to change"? Probably not. We're trying to figure out how to
cope with an ever-increasing number of interested users _and_
developers, and don't always react quickly to good suggestions.

The topic just came up recently, as I recall, and your suggestions are
good. Do you really want the patch applied which disables the more
general user names, or do you want to move more slowly and try to get
full user names in v6.4 (we have several months to get this right; in
fact we may already have them; see below :)

> However there is, as you say in another mail, no convenient way of
> doing that automatically.  We need an environment variable or a
> command-line option to specify the user and (optionally) password.

Sorry, I didn't follow the whole discussion. Is the problem only with
explicit CREATE USER and GRANT commands in SQL, or are there other
interfaces which would show problems too (you mention command-line
options above, but I don't know to what).

Oh, I just tried something:

tgl=> create user "hi-there";
CREATE USER
tgl=> select usename from pg_user;
usename
--------
postgres
tgl
hi-there
(3 rows)

Isn't this what you want?? I haven't figured out how to get GRANT to
work, but it seems to swallow the double-quoted user name as it
should...

                        - Tom

btw, how is it going with the docs conversion, Oliver? I'd expect that
it would keep you out of trouble for a little while at least :)

Tue, 28 Apr 1998 12:38:29 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: Bug#21681: postgresql: Doesn't allow granting to
	www-data
The Hermit Hacker wrote:
  >> template1=> select usename, usesysid, valuntil from pg_shadow;
  >> usename |usesysid|valuntil                    
  >> --------+--------+----------------------------
  >> postgres|      31|Sat Jan 31 00:00:00 2037 CST
  >> www-data|      33|Sat Jan 31 00:00:00 2037 CST
  >...
  >
  >	This might have already been gone over, but if this was an upgrade
  >from a previous release, its possible that this was created with a
  >'dump/reload'?

Createuser does not use the CREATE USER command.  It updates the
system tables directly.  This enables it to be used to specify a user id,
which CREATE USER does not allow.  However, it also allows inconsistencies
to arise, as here. So createuser can put in user names that CREATE USER
cannot and that GRANT does not recognise.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver

PGP key from public servers; key ID 32B8FAA1

                 ========================================
    Come to me, all you who labour and are heavily laden, and I will
    give you rest. Take my yoke upon you, and learn from me; for I am
    meek and lowly in heart, and you shall find rest for your souls.
    For my yoke is easy and my burden is light.    (Matthew 11: 28-30)



Tue, 28 Apr 1998 16:47:20 +0200
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: User names cannot contain `-' 
"Thomas G. Lockhart" wrote:
  >"Don't want to change"? Probably not. We're trying to figure out how to
  >cope with an ever-increasing number of interested users _and_
  >developers, and don't always react quickly to good suggestions.
  
Don't take it personally!  I'm happy to accept the developers' decisions,
since you know the code much better.

  >The topic just came up recently, as I recall, and your suggestions are
  >good. Do you really want the patch applied which disables the more
  >general user names,

I would prefer to have Unix user names allowed throughout.  However, if
the developers decide not to do this, the patch to createuser is
required to maintain consistency.  From Bruce's original reply,  I had
thought that was the position.

  >                   or do you want to move more slowly and try to get
  >full user names in v6.4 (we have several months to get this right; in
  >fact we may already have them; see below :)

By all means, lets have them!  

  >> However there is, as you say in another mail, no convenient way of
  >> doing that automatically.  We need an environment variable or a
  >> command-line option to specify the user and (optionally) password.
  >
  >Sorry, I didn't follow the whole discussion. Is the problem only with
  >explicit CREATE USER and GRANT commands in SQL, or are there other
  >interfaces which would show problems too (you mention command-line
  >options above, but I don't know to what).

Sorry; that's what comes of running a three-way discussion.  The problem is
that you can't (I think) start a connection while supplying another
user-name than your login-name, except by the -u option to psql.  This
leads to an interactive prompt for name and password.  This is not
convenient for CGI scripts on web-servers (which is how the original
problem manifested itself.)  It seems to be desirable to be able to
specify the postgres user name while starting the connection.

  >Oh, I just tried something:
  >
  >tgl=> create user "hi-there";
  >CREATE USER

  >Isn't this what you want?? I haven't figured out how to get GRANT to
  >work, but it seems to swallow the double-quoted user name as it
  >should...

Yes it is; I hadn't tried double-quotes, because single-quotes are used
for strings - it didn't occur to me! (Incidentally, WHY double-quotes here
instead of single-quotes? Surely that's against SQL practice?) It doesn't
work for GRANT, though, with either kind of quote:

  bray->  grant all on address to www-data;
  ERROR:  aclparse: non-existent user "www"
  bray=> grant all on address to "www-data";
  ERROR:  aclparse: mode flags must use "arwR"
  bray=> grant all on address to 'www-data';
  ERROR:  parser: parse error at or near "'"


Overall, it seems to me that a user-name is just a string, that is used
as a key into pg_shadow.  The SQL92 definition allows it to be a
character string literal.  So there ought to be no problem in specifying
a string rather than an identifier in all the relevant places.
(I speak in happy ignorance of whatever the real problems may be!)


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver

PGP key from public servers; key ID 32B8FAA1

                 ========================================
    Come to me, all you who labour and are heavily laden, and I will
    give you rest. Take my yoke upon you, and learn from me; for I am
    meek and lowly in heart, and you shall find rest for your souls.
    For my yoke is easy and my burden is light.    (Matthew 11: 28-30)



Tue, 28 Apr 1998 16:47:25 +0200
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: Bug#21681: postgresql: Doesn't allow granting to
	www-data
> Don't take it personally!  I'm happy to accept the developers' 
> decisions, since you know the code much better.

No offense taken. It's just that you got an opinion, not a "decision".
When you are bouncing ideas off of the list, you might get an opinion
back, but if you really think it should be pursued then it's OK to push
the topic to see if it resonates with someone. For something big, it
does usually take a consensus among the core developers to get it
incorporated, but on smaller issues which fit in with how things already
are it isn't that complicated.

btw, we try to not be too thin-skinned; sometimes even polite people
need to be blunt to get their point across when we're being dense :)

>   >Do you really want the patch applied which disables the more
>   >general user names,
> I would prefer to have Unix user names allowed throughout.

Not exactly possible with SQL92 grammar; see below...

> (What started this thread?) The problem is
> that you can't (I think) start a connection while supplying another
> user-name than your login-name, except by the -u option to psql.  This
> leads to an interactive prompt for name and password.

This sounds like a feature to add to libpq and/or psql...

>   >tgl=> create user "hi-there";
>   >CREATE USER
> I hadn't tried double-quotes, because single-quotes are 
> used for strings - it didn't occur to me! (Incidentally, WHY 
> double-quotes here instead of single-quotes? Surely that's against SQL 
> practice?)

I don't think so; syntactically an authorization ID seems to resemble
other identifiers, not literal strings. Identifiers are allowed to be
"delimited", when means surrounded by double-quotes, to allow mixed case
and weird characters into an identifier.

> It doesn't work for GRANT, though, with either kind of quote:

Yeah, well, that is probably a smaller problem. Perhaps there is a query
buried in the backend which needs to have some double-quoting applied.
We can look into it, eh?

> Overall, it seems to me that a user-name is just a string, that is 
> used as a key into pg_shadow.  The SQL92 definition allows it to be a
> character string literal.

I don't see this in my reference books. A character string literal is
delimited by single-quotes; never run across it for a user name
(actually an "authorization ID" in SQL92).

> So there ought to be no problem in specifying
> a string rather than an identifier in all the relevant places.
> (I speak in happy ignorance of whatever the real problems may be!)

Yup :)

The SQL92 character set specifically allows only a few characters other
than [A-Za-z0-9] in non-delimited identifiers. And, it specifically
defines most other interesting characters (including "-") as explicit
delimiters.

I think we should concentrate on making the features work as well as
possible within the SQL92 framework, and within the limitations of our
lex/yacc grammar.

Delimited identifiers are afaik the way to do this...

                         - Tom

Wed, 29 Apr 1998 02:19:07 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: Bug#21681: postgresql: Doesn't allow granting to
	www-data
More info on the GRANT problem with user names containing a minus sign:

It turns out that "+-=" are used inside an ACL string constructed
internally in the backend. So, putting one of those characters into the
user name causes what follows to be misinterpreted.

 * aclparse
 *      Consumes and parses an ACL specification of the form:
 *              [group|user] [A-Za-z0-9]*[+-=][rwaR]*

I think that we would need to restructure this internal information to
make the user name field unambiguous no matter its contents.

Bruce, can you put this on the ToDo list? In the meantime I would
suggest _not_ restricting the allowable user names elsewhere, since this
is a bug fix kind of thing...

                     - Tom

Wed, 29 Apr 1998 02:58:53 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: Bug#21681: postgresql: Doesn't allow granting to
	www-data
> 
> More info on the GRANT problem with user names containing a minus sign:
> 
> It turns out that "+-=" are used inside an ACL string constructed
> internally in the backend. So, putting one of those characters into the
> user name causes what follows to be misinterpreted.
> 
>  * aclparse
>  *      Consumes and parses an ACL specification of the form:
>  *              [group|user] [A-Za-z0-9]*[+-=][rwaR]*
> 
> I think that we would need to restructure this internal information to
> make the user name field unambiguous no matter its contents.
> 
> Bruce, can you put this on the ToDo list? In the meantime I would
> suggest _not_ restricting the allowable user names elsewhere, since this
> is a bug fix kind of thing...

Added to TODO:

* Restructure storing of GRANT permission information to allow +-=


-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

Tue, 28 Apr 1998 23:56:21 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: Bug#21681: postgresql: Doesn't allow granting to
To bring everything back to base here...In one of the posts CC'd to
me, somebody wondered how this mess got started, so let me recap...

On Debian, a webserver runs under the user www-data.  I had created a
www-data user for PostgreSQL using createuser, so that CGI scripts
would be able to access the database (using ident authentication).
However, the newer Postgres releases default to not granting select to 
public (wise), so it did not work on the first try.  I looked into it, 
and remembered -- Ah ha! -- that I needed to grant all (in this case)
to www-data.  However, psql wouldn't parse my grant command due to the 
hyphen in the www-data username.  I had no problem creating the user
(although it appears now that the SQL way of creating the user may be
broken/strange).  I also had no problem getting the CGI to connect to
the database once www-data was able to access the tables.  (I was able 
to use a temporary workaround of granting all to public on the
specific tables, since nobody else happens to have a user account with 
this postgres installation).

The problem, then, was the GRANT command for me.  I reported this
using Debian's bug-tracking system as per usual procedure, and Oliver
sent it on up to y'all as per usual procedure.

"Oliver Elphick" <olly@lfix.co.uk> writes:

> However there is, as you say in another mail, no convenient way of
> doing that automatically.  We need an environment variable or a
> command-line option to specify the user and (optionally) password.

ident will work fine for the CGI.  It is just that I need to be able
to grant access to the ident'd user :-)

And finally, I hope you excuse this paragraph in the Debian logs
Oliver... :-)

FYI, while the -hackers are getting CCd anyway, I ought to mention
here how amazingly cool I think Postgres is.  At work, I am currently
writing a multiplatform Perl/Tk application using DBI::Pg as the
database driver, and Postgres running on BSDi.  (One of my jobs is at
an ISP, where BSDi is in heavy use.)  Besides being free (yay!  it's a 
smallish ISP), it also is the only one that runs natively on BSDi
(this is the BIG win), excepting the mSQLs of the world that we would
rather ignore :-)  Another person there is investigating moving all of 
the billing information over from flatfile ASCII databases to a
Postgres database, and I will be working with moving usage accounting
software from flatfile databases to Postgres, as well as my current
call tracking project.  Nobody has officially said that we will stick
with Postgres, but I think it is likely.  Good job guys!  (At home, I
do the relatively simple task of indexing my videotape collection with 
a couple of tables with no more than 700 rows each, compared with
15000/table with 10 tables at work <g>)

-- 
John Goerzen        Linux, Unix programming           jgoerzen@complete.org |
Developer, Debian GNU/Linux (Free powerful OS upgrade)       www.debian.org |
----------------------------------------------------------------------------+
``You'll notice that this scanner, Bill [Gates]...''  <Blue Screen of Death>
``Whoa!''  <Applause>   ``Moving right along....''  -- Microsoft (Comdex
      video at: http://cnn.com/TECH/computing/9804/20/gates.comdex/index.html

29 Apr 1998 00:13:52 -0500
John Goerzen <jgoerzen@complete.org>
Re: Bug#21681: postgresql: Doesn't allow granting to www-data
Hi all,

I think that PostgreSQL money type should be very useful if we could
remove the dollar sign. We can't use it with Lira/Peseta/Mark etc.
In europe now we have Euro. If we remove the $ it will be useful otherwise
we have to rename it to 'dollar'. ;-)
----
PS: Is there a reason to left justify it ?

select dollar from prova;
dollar
----------
$300.32
$302.21
$312.10
$12,312.10
$12,386.00
$12,312.00
                                         Thanks, Jose'


Mon, 11 May 1998 13:33:55 +0000 (UTC)
"Jose' Soares Da Silva" <sferac@bo.nettuno.it>
money or dollar type
"Jose' Soares Da Silva" wrote:
  >I think that PostgreSQL money type should be very useful if we could
  >remove the dollar sign. We can't use it with Lira/Peseta/Mark etc.
  >In europe now we have Euro. If we remove the $ it will be useful otherwise
  >we have to rename it to 'dollar'. ;-)
 
Compile with LANG support and set the LANG environment variable for the
postmaster. Restart the postmaster.

Then you get your own currency symbol:


junk=> select * from moneybag;
who|amount 
---+-------
A  |£250.00
(1 row)


But I don't like the fact that this has to be done in the backend.  It
means that the currency of money is tied to the LANG environment of the
postmaster, rather than to the data itself.  One of the characteristics of
money is the currency in which it is denominated; this ought to be part
of the datatype.  It would then be invalid to perform arithmetical
operations between different currencies, which would correctly reflect
the real world.

Therefore, I propose that the money type be extended to include a
currency definition, the default being that of the backend environment.
-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Search me, O God, and know my heart; try me, and know 
      my thoughts. And see if there be any wicked way in me,
      and lead me in the way everlasting."        
                  Psalms 139:23,24 



Mon, 11 May 1998 16:14:44 +0200
"Oliver Elphick" <olly@lfix.co.uk>
Re: [QUESTIONS] money or dollar type 
> I think that PostgreSQL money type should be very useful if we could
> remove the dollar sign. We can't use it with Lira/Peseta/Mark etc.
> In europe now we have Euro. If we remove the $ it will be useful 
> otherwise we have to rename it to 'dollar'. ;-)

Have you tried compiling with "USE_LOCALE" turned on and with the right
setting for LC_xxx? The code is supposed to use local conventions, but I
don't know if it works in the way you want. I agree that it should...

> PS: Is there a reason to left justify it ?

That is just an artifact of the column formatting; all columns are left
justified in psql afaik.

                  - Tom

Mon, 11 May 1998 14:29:52 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] money or dollar type
"Oliver Elphick" <olly@lfix.co.uk> writes:
> Therefore, I propose that the money type be extended to include a
> currency definition, the default being that of the backend environment.

This is not a bad idea; it would address some problems that I have in
my application too.  (What I was planning to do was store a separate
currency field associated with every money amount, but I think Oliver's
idea is better.)

However, what money *really* needs is more precision.  Has there been
any thought of working on the full SQL exact-numeric package?  (If I
read what I've seen correctly, that boils down to user-specifiable
decimal field widths, right?)  A variable-width money type including
a currency indicator would actually solve my problem...

			regards, tom lane

Mon, 11 May 1998 11:42:55 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [QUESTIONS] money or dollar type 
"Jose' Soares Da Silva" wrote:
  >On Mon, 11 May 1998, Oliver Elphick wrote:
  >
  >> "Jose' Soares Da Silva" wrote:
  >>   >I think that PostgreSQL money type should be very useful if we could
  >>   >remove the dollar sign. We can't use it with Lira/Peseta/Mark etc.
  >>   >In europe now we have Euro. If we remove the $ it will be useful otherw
      >ise
  >>   >we have to rename it to 'dollar'. ;-)
  >>  
  >> Compile with LANG support and set the LANG environment variable for the
  >> postmaster. Restart the postmaster.
  >> 
  >> Then you get your own currency symbol:
  >What's happening with EURO sign?
  
I guess that will need a tweak to libc to support it - I wonder if the
glibc developers have thought about it?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Search me, O God, and know my heart; try me, and know 
      my thoughts. And see if there be any wicked way in me,
      and lead me in the way everlasting."        
                  Psalms 139:23,24 



Mon, 11 May 1998 18:30:07 +0200
"Oliver Elphick" <olly@lfix.co.uk>
Re: [QUESTIONS] money or dollar type 
On Mon, 11 May 1998, Thomas G. Lockhart wrote:

> > I think that PostgreSQL money type should be very useful if we could
> > remove the dollar sign. We can't use it with Lira/Peseta/Mark etc.
> > In europe now we have Euro. If we remove the $ it will be useful 
> > otherwise we have to rename it to 'dollar'. ;-)
> 
> Have you tried compiling with "USE_LOCALE" turned on and with the right
> setting for LC_xxx? The code is supposed to use local conventions, but I
> don't know if it works in the way you want. I agree that it should...
Thanks Tom, I will try it.
> 
> > PS: Is there a reason to left justify it ?
> 
> That is just an artifact of the column formatting; all columns are left
> justified in psql afaik.
                    ^^^^^
Sorry Tom, I can't find the word 'afaik' on my dictionary.

Any way, seems that psql justify numbers to the right and text to the left, 
money is numeric then I expect that psql justify it to the right.
It has also a little problem justifying varchars, look:

prova=> select var as my_varchar from prova where var = '12';
my_varchar
----------
        12         <--right justified
(1 row)

prova=> select var as my_varchar from prova;
my_varchar
----------
12 		<--left justified, this time ???
a12
a12
(3 rows)
                                              Jose'


Mon, 11 May 1998 17:24:10 +0000 (UTC)
"Jose' Soares Da Silva" <sferac@bo.nettuno.it>
Re: [HACKERS] money or dollar type
On Mon, 11 May 1998, Oliver Elphick wrote:

> "Jose' Soares Da Silva" wrote:
>   >I think that PostgreSQL money type should be very useful if we could
>   >remove the dollar sign. We can't use it with Lira/Peseta/Mark etc.
>   >In europe now we have Euro. If we remove the $ it will be useful otherwise
>   >we have to rename it to 'dollar'. ;-)
>  
> Compile with LANG support and set the LANG environment variable for the
> postmaster. Restart the postmaster.
> 
> Then you get your own currency symbol:
What's happening with EURO sign?
> 
> 
> junk=> select * from moneybag;
> who|amount 
> ---+-------
> A  |�250.00
> (1 row)
> 
> 
> But I don't like the fact that this has to be done in the backend.  It
> means that the currency of money is tied to the LANG environment of the
> postmaster, rather than to the data itself.  One of the characteristics of
> money is the currency in which it is denominated; this ought to be part
> of the datatype.  It would then be invalid to perform arithmetical
> operations between different currencies, which would correctly reflect
> the real world.
> 
> Therefore, I propose that the money type be extended to include a
> currency definition, the default being that of the backend environment.
I agree, currently we can have only one currency definition. We can't
have for example Dollars and Pesetas in the same database.
                                                    Jose'


Mon, 11 May 1998 17:38:33 +0000 (UTC)
"Jose' Soares Da Silva" <sferac@bo.nettuno.it>
Re: [QUESTIONS] money or dollar type 
> However, what money *really* needs is more precision.  Has there been
> any thought of working on the full SQL exact-numeric package?

Yes. The problem is that afaik there is no variable-width exact numeric
package available. BCD arithmetic could work if a package were
available. The GNU extended precision package looks interesting, but we
would have to translate from a string to internal format for every
operation, or somehow store the internal representation in each tuple
which seems messy.

I'm thinking of moving the 64-bit integer contrib package I wrote into
the native backend as a foundation for the numeric/decimal data types.
We would need to get feedback from more of the supported platforms on
how to do 64-bit integers (a few processors have them as a "long" type,
and the GNU 32-bit compilers seem to allow a "long long" declaration,
but I don't know what other systems do for this). 

The only other thing which would need to be handled is how to pass along
the two value precision/scale parameters which are a part of the
declaration for these types. I've just finished working on the type
conversion algorithms so understand the current "atttypmod" field a bit
better, but have not decided how to extend it to multiple fields.

                       - Tom

Tue, 12 May 1998 05:17:43 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: [QUESTIONS] money or dollar type
On Mon, 11 May 1998, Thomas G. Lockhart wrote:

> > I think that PostgreSQL money type should be very useful if we could
> > remove the dollar sign. We can't use it with Lira/Peseta/Mark etc.
> > In europe now we have Euro. If we remove the $ it will be useful 
> > otherwise we have to rename it to 'dollar'. ;-)
> 
> Have you tried compiling with "USE_LOCALE" turned on and with the right
> setting for LC_xxx? The code is supposed to use local conventions, but I
> don't know if it works in the way you want. I agree that it should...
> 
> > PS: Is there a reason to left justify it ?
> 
> That is just an artifact of the column formatting; all columns are left
> justified in psql afaik.
> 

Seems there's some problems with type 'money'... I can't multiply or
divide 'money' types, and can't cast it properly to other data types.

prova=> select ename,job,hiredate, sal from employees;
ename |job       |  hiredate|sal
------+----------+----------+---------
ALLEN |SALESMAN  |1981-02-20|$1,600.00
BLAKE |MANAGER   |1981-05-01|$2,850.00
JONES |CLERK     |1981-12-03|$950.00
MILLER|SALESMAN  |1981-09-28|$1,250.00
CLARK |SALESMAN  |1981-09-08|$1,500.00
KING  |SALESMAN  |1981-02-22|$1,250.00
(6 rows)

prova=> select ename,job,hiredate, sal*1.1 as dream from employees;
ERROR:  There is no operator '*' for types 'money' and 'money'
        You will either have to retype this query using an explicit cast,
        or you will have to define the operator using CREATE OPERATOR

prova=> select ename,job,hiredate,sal, sal::float as dream from employees;
ename |job       |  hiredate|sal      |     dream
------+----------+----------+---------+----------
ALLEN |SALESMAN  |1981-02-20|$1,600.00|1079143604
BLAKE |MANAGER   |1981-05-01|$2,850.00|1079143508
JONES |CLERK     |1981-12-03|$950.00  |1079143412
MILLER|SALESMAN  |1981-09-28|$1,250.00|1079143316
CLARK |SALESMAN  |1981-09-08|$1,500.00|1079143220
KING  |SALESMAN  |1981-02-22|$1,250.00|1079143120
(6 rows)

Is this a bug ?
                                                           Jose'


Tue, 12 May 1998 10:07:53 +0000 (UTC)
"Jose' Soares Da Silva" <sferac@bo.nettuno.it>
Re: [HACKERS] money or dollar type
Thus spake Jose' Soares Da Silva
> I think that PostgreSQL money type should be very useful if we could
> remove the dollar sign. We can't use it with Lira/Peseta/Mark etc.
> In europe now we have Euro. If we remove the $ it will be useful otherwise
> we have to rename it to 'dollar'. ;-)

I have been trying to remove this from the code.  For some reason I can't
compile the system (something about wrong number of args to gettimeofday
in backend/tcop/postgres.c) but in the meantime, have you tried the
USE_LOCALE define?  That should at least switch it to your local money
indicator.

> PS: Is there a reason to left justify it ?

Not that I can think of but I'm not sure where you change that.

-- 
D'Arcy J.M. Cain <darcy@{druid|vex}.net>   |  Democracy is three wolves
http://www.druid.net/darcy/                |  and a sheep voting on
+1 416 424 2871     (DoD#0082)    (eNTP)   |  what's for dinner.

Tue, 12 May 1998 06:37:39 -0400 (EDT)
darcy@druid.net (D'Arcy J.M. Cain)
Re: [HACKERS] money or dollar type
Thus spake Thomas G. Lockhart
> > PS: Is there a reason to left justify it ?
> 
> That is just an artifact of the column formatting; all columns are left
> justified in psql afaik.

Ints are right formatted so it must be possible to do.

-- 
D'Arcy J.M. Cain <darcy@{druid|vex}.net>   |  Democracy is three wolves
http://www.druid.net/darcy/                |  and a sheep voting on
+1 416 424 2871     (DoD#0082)    (eNTP)   |  what's for dinner.

Tue, 12 May 1998 06:41:23 -0400 (EDT)
darcy@druid.net (D'Arcy J.M. Cain)
Re: [HACKERS] money or dollar type
> Seems there's some problems with type 'money'... I can't multiply or
> divide 'money' types, and can't cast it properly to other data types.
> Is this a bug ?

With the new type conversion code:

tgl=> create table mm (m money);
CREATE
tgl=> insert into mm values ('$1600.00');
INSERT 268105 1
tgl=> select m * 1.1 from mm;
?column?
---------
$1,760.00
(1 row)

But,

tgl=> select cast(m as float8) from mm;
    float8
----------
1077124288
(1 row)

So there is some funny interaction on the casting, the same as you found
in v6.3.2 (and presumably forever), which I will look into...

                     - Tom

Tue, 12 May 1998 13:38:39 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] money or dollar type
> prova=> select var as my_varchar from prova where var = '12';
> my_varchar
> ----------
>         12         <--right justified
> (1 row)
> 
> prova=> select var as my_varchar from prova;
> my_varchar
> ----------
> 12 		<--left justified, this time ???
> a12
> a12
> (3 rows)
>                                               Jose'

I can't reproduce this here.


-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

Tue, 12 May 1998 12:47:08 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] money or dollar type
> The only other thing which would need to be handled is how to pass along
> the two value precision/scale parameters which are a part of the
> declaration for these types. I've just finished working on the type
> conversion algorithms so understand the current "atttypmod" field a bit
> better, but have not decided how to extend it to multiple fields.
> 

I have thought about this.  Just bitmask the 16-bits to two 8-bit
quantities.  Give you max 256 length with 256 currencies.

The only place they are used is in the type-specific *.c function, so
you just us the mask there, or create a union of :8 and :8 and reference
it that way.

-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

Tue, 12 May 1998 12:54:09 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [QUESTIONS] money or dollar type
On Tue, 12 May 1998, Bruce Momjian wrote:

> > prova=> select var as my_varchar from prova where var = '12';
> > my_varchar
> > ----------
> >         12         <--right justified
> > (1 row)
> > 
> > prova=> select var as my_varchar from prova;
> > my_varchar
> > ----------
> > 12 		<--left justified, this time ???
> > a12
> > a12
> > (3 rows)
> >                                               Jose'
> 
> I can't reproduce this here.
Seems that PostgreSQL justify data based on data not on data type.
My environment is:
    PostgreSQL v6.3
    Linux 2.0.33

also Daniel A. Gauthier <3in7ifi@cmich.edu>
reported the same problem.

here my script:

create table prova ( my_varchar varchar );
CREATE
insert into prova values ('12');
INSERT 528521 1
insert into prova values ('a12');
INSERT 528522 1
select * from prova where my_varchar = '12';
my_varchar
----------
        12
(1 row)

select * from prova;
my_varchar
----------
12
a12
(2 rows)

EOF
                                                     Jose'


Wed, 13 May 1998 09:55:01 +0000 (UTC)
"Jose' Soares Da Silva" <sferac@bo.nettuno.it>
Re: [HACKERS] money or dollar type
> 
> create table prova ( my_varchar varchar );
> CREATE
> insert into prova values ('12');
> INSERT 528521 1
> insert into prova values ('a12');
> INSERT 528522 1
> select * from prova where my_varchar = '12';
> my_varchar
> ----------
>         12
> (1 row)
> 
> select * from prova;
> my_varchar
> ----------
> 12
> a12
> (2 rows)
> 

OK, I can reproduce this now.  I would love to know why it is happening.
Seems very strange to me.


-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

Wed, 13 May 1998 11:51:12 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] money or dollar type
Bruce Momjian <maillist@candle.pha.pa.us> writes:
> I have thought about this.  Just bitmask the 16-bits to two 8-bit
> quantities.  Give you max 256 length with 256 currencies.

Uh, no: what we were discussing was the total width and decimal place
position of exact numerics.  Probably, 255 numeric digits are enough
for practical purposes, so I don't feel an urgent need to make atttypmod
wider for this.  But if you want to make it 32 bits, that would
eliminate any concern --- we'd have room for 64k-digit numerics...

If we're going to associate currencies with the money datatype, the
currency needs to be part of the data, not part of the column type.
I need to be able to store amounts of different currencies in the same
column.  (Otherwise, a transaction log would need a separate column for
every possible currency, all but one of which would be null in any given
row.  Ick.)

			regards, tom lane

Wed, 13 May 1998 13:36:15 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [QUESTIONS] money or dollar type 
> 
> Bruce Momjian <maillist@candle.pha.pa.us> writes:
> > I have thought about this.  Just bitmask the 16-bits to two 8-bit
> > quantities.  Give you max 256 length with 256 currencies.
> 
> Uh, no: what we were discussing was the total width and decimal place
> position of exact numerics.  Probably, 255 numeric digits are enough
> for practical purposes, so I don't feel an urgent need to make atttypmod
> wider for this.  But if you want to make it 32 bits, that would
> eliminate any concern --- we'd have room for 64k-digit numerics...
> 
> If we're going to associate currencies with the money datatype, the
> currency needs to be part of the data, not part of the column type.
> I need to be able to store amounts of different currencies in the same
> column.  (Otherwise, a transaction log would need a separate column for
> every possible currency, all but one of which would be null in any given
> row.  Ick.)

Yep, good point.

-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

Wed, 13 May 1998 14:06:04 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [QUESTIONS] money or dollar type
Was looking through the new docs and noticed that the example for
creating a database in an alternate location has trouble:

   $ mkdir private_db
   $ initlocation ~/private_db
   Creating Postgres database system directory
/home/olly/private_db/base

   $ chmod a+rx private_db
   $ chmod a+rwx private_db/base
   $ psql
   ...

The chmod's are a Bad Idea (tm) since it blows the security assumptions
for Postgres. The protections are explicitly set by initlocation to lock
down these directories.

I guess that the alternate location setup (initlocation) was really
meant as a tool for the Postgres administrator, not for individual
users. If users create alternate locations, and then for example create
a database and then delete the directories from the file system rather
than through Postgres things will become ugly. The assumption is that
the administrator is likely to be more careful since she is likely to be
more aware of the issues.

I have (or had) some #ifdef code which _requires_ that environment
variables be used to specify alternate locations, rather than allowing
absolute paths also. This helps ensure that locations are used which
have been set up by the Postgres administrator, since the admin must
have defined the environment variables for the backend before it starts
up.

I'm not sure how to write an example which had initlocation being run by
someone other than the Postgres superuser while still being clear on
these security/integrity issues. What would you suggest?

                          - Tom

Fri, 15 May 1998 13:43:16 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
CREATE DATABASE
"Thomas G. Lockhart" wrote:
  >Was looking through the new docs and noticed that the example for
  >creating a database in an alternate location has trouble:
  >
  >   $ mkdir private_db
  >   $ initlocation ~/private_db
  >   Creating Postgres database system directory
  >/home/olly/private_db/base
  >
  >   $ chmod a+rx private_db
  >   $ chmod a+rwx private_db/base
  >   $ psql
  >   ...
  >
  >The chmod's are a Bad Idea (tm) since it blows the security assumptions
  >for Postgres. The protections are explicitly set by initlocation to lock
  >down these directories.

Well, I documented what I actually had to do to make it work.

I got the impression that this particular feature had not been properly
thought out!

  >
  >I guess that the alternate location setup (initlocation) was really
  >meant as a tool for the Postgres administrator, not for individual
  >users. If users create alternate locations, and then for example create
  >a database and then delete the directories from the file system rather
  >than through Postgres things will become ugly.

Shouldn't it cope with this possibility anyway?  (The mad systems
administrator....) What would happen to PostgreSQl if a database
directory disappeared from under it?

  >                                                 The assumption is that
  >the administrator is likely to be more careful since she is likely to be
  >more aware of the issues.
  
If so, the restriction should be placed in the code: only the postgres
administrator should be allowed to run the command.  That would get rid
of the permissions problems.  On the other hand, it would remove most of
the point of having a separate location (seeing that it could just as
well be implemented by symbolic links).

  >I have (or had) some #ifdef code which _requires_ that environment
  >variables be used to specify alternate locations, rather than allowing
  >absolute paths also. This helps ensure that locations are used which
  >have been set up by the Postgres administrator, since the admin must
  >have defined the environment variables for the backend before it starts
  >up.
  >
  >I'm not sure how to write an example which had initlocation being run by
  >someone other than the Postgres superuser while still being clear on
  >these security/integrity issues. What would you suggest?
  >

The first thing to do is decide whether this feature is required at all.
I'm trying to think of things in its favour:

1. It enables the administrator to direct a large database to a partition
   with sufficient space.

2. It allows especially sensitive data to be stored on an encrypted file
   system.

3. It allows a database to be created on removable media. (Is this a good
   thing?)


If run by a user:

4. It might give the chance of having data that is secure against the
   administrator, but only if a backend can be launched that is owned by
   the user rather than by the administrator.

Since 4 isn't possible at the moment, there seems to be no reason for 
allowing a user to run this command, even if he is otherwise allowed to
create databases.  Removing the ability from users means that the
documentation can be simplified by documenting it for administrative
use only.

Even simpler is to remove it altogether and let the administrator
handle 1-3 by unix commands and symbolic links.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Let not your heart be troubled: ye believe in God, 
      believe also in me."       John 14:1 



Fri, 15 May 1998 17:28:28 +0200
"Oliver Elphick" <olly@lfix.co.uk>
Re: CREATE DATABASE 
> Well, I documented what I actually had to do to make it work.

Well, not to cause trouble here but it will work as I intended, and
didn't work as you tried to use it. That's not so bad, and the usage is
documented in the administrator's guide. But your points below are good
and should be used to improve things.

> Shouldn't it cope with this possibility anyway?  (The mad systems
> administrator....) What would happen to PostgreSQl if a database
> directory disappeared from under it?

Postgres, via the master database, would think that the database
existed, but would not be able to connect and not be able to delete it.
The administrator would have to brute-force copy a database (template1
would do) into the removed area, at which time Postgres could remove it
and clean up internally.

>   > The assumption is that
>   >the administrator is likely to be more careful since she is likely 
>   >to be more aware of the issues.
> If so, the restriction should be placed in the code: only the postgres
> administrator should be allowed to run the command.  That would get 
> rid of the permissions problems.  On the other hand, it would remove 
> most of the point of having a separate location (seeing that it could 
> just as well be implemented by symbolic links).

No, there is a point to having a separate location; that and the
softlink kludge are two different things. The "initlocation" alternate
location can be used by any database user (with createdb privileges) to
create and maintain a new database, once set up by the Postgres
administrator.
Soft links don't do that; Postgres doesn't know about them and can't
create/delete/create databases in multiple locations by using them
afaik.

"initlocation" isn't actually the issue; the real issue is whether
alternate locations not created by the Postgres superuser should be
allowed at all. I'm thinking not, given these issues which would have to
be kept in mind by every user trying to use the command themselves.

> The first thing to do is decide whether this feature is required at 
> all. I'm trying to think of things in its favour:
> 1. It enables the administrator to direct a large database to a 
>    partition with sufficient space.

bingo. This is sufficient to make the feature desirable, but the
downsides should be addressed.

> 2. It allows especially sensitive data to be stored on an encrypted 
>    file system.
> 3. It allows a database to be created on removable media. (Is this a 
>    good thing?)

Removable media are not a good thing for this mechanism. Postgres used
to have direct support for databases on removable media (tape); this
should be resurrected if someone wants the feature.

> If run by a user:
> 4. It might give the chance of having data that is secure against the
>    administrator, but only if a backend can be launched that is owned 
>    by the user rather than by the administrator.

In that case the user would be the administrator. Postgres allows that,
and allows multiple servers on the same machine (the listening port
would need to be different for each one).

> Since 4 isn't possible at the moment, there seems to be no reason for
> allowing a user to run this command, even if he is otherwise allowed 
> to create databases.  Removing the ability from users means that the
> documentation can be simplified by documenting it for administrative
> use only.

I did document it for administrators only ;)

> Even simpler is to remove it altogether and let the administrator
> handle 1-3 by unix commands and symbolic links.

No, that is not a solid option.

At the moment, initlocation is just a shell utility which sets up a
directory area and changes permissions to be appropriate for Postgres
use *if it is run by the Postgres superuser*.

If we are concerned that alternate database locations can be misused and
should be under tighter control of the Postgres superuser, then it would
be easy to take out (or #ifdef) the code which allows absolute path
names and instead require that all paths for alternate locations be
specified as environment variables (that is already supported and was
always preferable imho).

That way the Postgres administrator can run initlocation, define an
environment variable pointing at that alternate location, and then start
up the backend. Users would have only those previously defined areas to
work with.

I'm being pretty direct here in my comments because you are raising very
good issues and we haven't had much discussion of them in the past. I am
not intending to be difficult or obstinant, just trying to clear things
up and get a plan for improving things.

Comments (on the issues, and/or whether I'm being a pain :)?

                       - Tom

Sat, 16 May 1998 02:00:07 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: CREATE DATABASE
"Thomas G. Lockhart" wrote:
  >> Well, I documented what I actually had to do to make it work.
  >
  >Well, not to cause trouble here but it will work as I intended, and
  >didn't work as you tried to use it. That's not so bad, and the usage is
  >documented in the administrator's guide.

I was actually working from the existing man pages, but now that I check,
it isn't sufficiently clear from the guide that initlocation is an
administrator-only command. (Sections II.7, III.22)  Even reading them in
the light of your comments, I cannot see that this restriction is stated.
(Or have you updated this since 6.3.2?)
...
  >"initlocation" isn't actually the issue; the real issue is whether
  >alternate locations not created by the Postgres superuser should be
  >allowed at all. I'm thinking not, given these issues which would have to
  >be kept in mind by every user trying to use the command themselves.

If initlocation is to be run only by the administrator, it should be owned
by the administrator login and have execute-permission only for the
owner.
...
  >> Since 4 isn't possible at the moment, there seems to be no reason for
  >> allowing a user to run this command, even if he is otherwise allowed 
  >> to create databases.  Removing the ability from users means that the
  >> documentation can be simplified by documenting it for administrative
  >> use only.
  >
  >I did document it for administrators only ;)

I still don't see it!  Sorry.
...
  >At the moment, initlocation is just a shell utility which sets up a
  >directory area and changes permissions to be appropriate for Postgres
  >use *if it is run by the Postgres superuser*.

So make it executable only by the superuser.  If some site wanted to have
multiple administrators, it would need some fancy code to check that
the postmaster listening on $PGPORT was owned by the same user as is
running initlocation. Then it could be made generally executable.
  
  >If we are concerned that alternate database locations can be misused and
  >should be under tighter control of the Postgres superuser, then it would
  >be easy to take out (or #ifdef) the code which allows absolute path
  >names and instead require that all paths for alternate locations be
  >specified as environment variables (that is already supported and was
  >always preferable imho).
  >
  >That way the Postgres administrator can run initlocation, define an
  >environment variable pointing at that alternate location, and then start
  >up the backend. Users would have only those previously defined areas to
  >work with.

How many alternate locations can be specified? I see PGDATA2 listed in your
documentation.  Can this be extended to PGDATAn?

  >
  >I'm being pretty direct here in my comments because you are raising very
  >good issues and we haven't had much discussion of them in the past. I am
  >not intending to be difficult or obstinant, just trying to clear things
  >up and get a plan for improving things.
  >
  >Comments (on the issues, and/or whether I'm being a pain :)?

No problem!  In this case, I was documenting from the point of view of a 
user, since I was running commands in my own name (with database creation
privileges).  The old man pages don't mention any restriction, so it didn't
occur to me that this was an administrator only command.  The way I read it,
it seemed to be designed for a user to set up his own private area.



-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "In my Father's house are many mansions: if it were not
      so, I would have told you. I go to prepare a place for
      you. And if I go and prepare a place for you, I will 
      come again, and receive you unto myself; that where I 
      am, there ye may be also."        John 14:2,3 



Sat, 16 May 1998 10:02:15 +0200
"Oliver Elphick" <olly@lfix.co.uk>
Re: CREATE DATABASE 
-----BEGIN PGP SIGNED MESSAGE-----

Then <lockhart@alumni.caltech.edu> spoke up and said:
> That way the Postgres administrator can run initlocation, define an
> environment variable pointing at that alternate location, and then start
> up the backend. Users would have only those previously defined areas to
> work with.

Personally, I don't care *how* alternate locations are
setup/specified, as long as
1)  With a single postmaster, I can access databases in all of the
locations. 
2)  Adding a new location should be reserved to the PostgreSQL
superuser.
3)  Adding a new location should *not* require restarting the
postmaster.

Additionally, it would be nice to have syntax available for spreading
a database across multiple locations (a la Ingres).
- -- 
=====================================================================
| JAVA must have been developed in the wilds of West Virginia.      |
| After all, why else would it support only single inheritance??    |
=====================================================================
| Finger geek@andrew.cmu.edu for my public key.                     |
=====================================================================

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2

iQBVAwUBNWAxr4dzVnzma+gdAQEeGQH/c2L/THi6ZpijGz+JyCfnLhgaVnf/CC0Z
w3iX7UEAp1C5Oc/UkmkX3cMZ3cEYkdYAYOBvGtKbhnxO9x90wp5+Mw==
=8QdB
-----END PGP SIGNATURE-----


18 May 1998 09:03:43 -0400
<geek+@cmu.edu>
Re: [HACKERS] Re: CREATE DATABASE
> ... and what worked for me was:  default (datetime(now())
> 
> So, the reason I'm annoying you now is this:  For some time now, I've
> been wondering what version of PostgreSQL my ISP has running.  He's
> never answered that particular query, and the version files are not
> readable by lowly me.
> 
> So, can you tell me in what version of PostgreSQL this worked/broke?

postgres=> create table t (i int,
postgres-> d datetime default datetime(text 'now'));
CREATE

Works now. Have you tried

postgres=> select version();
version
--------------------------------------------------------------
PostgreSQL 6.4.0 on i686-pc-linux-gnu, compiled by gcc 2.7.2.1
(1 row)

The CVS log shows something was added back in 1996. Don't know if it
actually worked back then though...
revision 1.1
date: 1996/08/28 01:57:23;  author: scrappy;

Fri, 07 Aug 1998 05:23:36 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: Default 'now'
Hi all

Could this be added to the FAQ for regular users? Something like "How do I know
what version I'm using?"

Just a thought.

On Aug 7,  5:23am, Thomas G. Lockhart wrote:
> Works now. Have you tried
>
> postgres=> select version();
> version
> --------------------------------------------------------------
> PostgreSQL 6.4.0 on i686-pc-linux-gnu, compiled by gcc 2.7.2.1
> (1 row)

-- 
Sincerely,

Jazzman (a.k.a. Justin Hickey)  e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand
==================================================================
People who think they know everything are very irritating to those
of us who do.  ---Anonymous

Jazz and Trek Rule!!!
==================================================================

Fri, 7 Aug 1998 14:55:23 +0000
"Justin Hickey" <jhickey@impact1.hpcc.nectec.or.th>
Re: [HACKERS] Re: Default 'now'
Added to FAQ.

> Hi all
> 
> Could this be added to the FAQ for regular users? Something like "How do I know
> what version I'm using?"
> 
> Just a thought.
> 
> On Aug 7,  5:23am, Thomas G. Lockhart wrote:
> > Works now. Have you tried
> >
> > postgres=> select version();
> > version
> > --------------------------------------------------------------
> > PostgreSQL 6.4.0 on i686-pc-linux-gnu, compiled by gcc 2.7.2.1
> > (1 row)
> 
> -- 
> Sincerely,
> 
> Jazzman (a.k.a. Justin Hickey)  e-mail: jhickey@hpcc.nectec.or.th
> High Performance Computing Center
> National Electronics and Computer Technology Center (NECTEC)
> Bangkok, Thailand
> ==================================================================
> People who think they know everything are very irritating to those
> of us who do.  ---Anonymous
> 
> Jazz and Trek Rule!!!
> ==================================================================
> 
> 


-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

Sat, 22 Aug 1998 07:41:25 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: Default 'now'
Hi,

I have successeed in porting PostgreSQL to Win NT. The patch that is
included is for development version from Sep 15, but I think that the
changes are version independent. The main difference from other port is
the renamed system table pg_version (vs. PG_VERSION) to pg_ver - Windows
file names are case insensitive :-) So this should be solved on a global
level perhaps in main sources. And also the communication through
AF_UNIX sockets is disabled. There are only two other changes:
- added some flag while opening directory with open() syscall
- changed flags for file descriptor in function pq_init()
and that's all :-)

the steps ;-) are:
do the steps that Joost wrote some time ago
patch <pgsql.diff (maybe by hand for newer versions of PostgreSQL)
make
make install
initdb
postmaster -i

I was able to run postmaster and two concurrent psql connections
yesterday. I will run the test later.

					Dan Horak


PS: where are you, Joost? the email for you is returning to me
Wed, 7 Oct 1998 14:03:35 +0200 
Horak Daniel <horak@mmp.plzen-city.cz>
NT port of PGSQL - success
> Hi,
> 
> I have successeed in porting PostgreSQL to Win NT. The patch that is
> included is for development version from Sep 15, but I think that the
> changes are version independent. The main difference from other port is
> the renamed system table pg_version (vs. PG_VERSION) to pg_ver - Windows

I thought Windows allowed any case, so you could open a file with
"PG_VERSION" or "pg_version" and it will open any file of any matching
case.

> file names are case insensitive :-) So this should be solved on a global
> level perhaps in main sources. And also the communication through
> AF_UNIX sockets is disabled. There are only two other changes:
> - added some flag while opening directory with open() syscall
> - changed flags for file descriptor in function pq_init()
> and that's all :-)
> 
> the steps ;-) are:
> do the steps that Joost wrote some time ago
> patch <pgsql.diff (maybe by hand for newer versions of PostgreSQL)
> make
> make install
> initdb
> postmaster -i
> 
> I was able to run postmaster and two concurrent psql connections
> yesterday. I will run the test later.

This is amazing.  You can actually run SQL statements on NT.

What would you like done with this patch?  Should it merged into the
tree, or just used for people testing things on NT, and later merged in
as you feel more comfortable?  You can make a 6.4 final patch, perhaps.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026


Wed, 7 Oct 1998 20:23:50 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] NT port of PGSQL - success
> Hi,
> 
> I have successeed in porting PostgreSQL to Win NT. The patch that is
> included is for development version from Sep 15, but I think that the
> changes are version independent. The main difference from other port is
> the renamed system table pg_version (vs. PG_VERSION) to pg_ver - Windows
> file names are case insensitive :-) So this should be solved on a global
> level perhaps in main sources. And also the communication through
> AF_UNIX sockets is disabled. There are only two other changes:
> - added some flag while opening directory with open() syscall
> - changed flags for file descriptor in function pq_init()
> and that's all :-)


I have removed the data/base/*/pg_version file because it was never
used.  We had removed the 'version' functions long ago, but
include/catalog/pg_version.h was still being processed by genbki.sh.  No
longer.  backend/command/version.c is also no longer compiled.

This should make your table changes unnecessary.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026


Thu, 8 Oct 1998 17:40:29 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] NT port of PGSQL - success
> Hello Bruce
> 
> On Oct 8,  5:40pm, Bruce Momjian wrote:
> > I have removed the data/base/*/pg_version file because it was never
> > used.  We had removed the 'version' functions long ago, but
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> Does this mean that the following from the FAQ is wrong?
> 
> 	3.25) How do I tell what PostgreSQL version I am running?
> 
> 	From psql, type select version();
> 
> If so then this question should probably be changed to point users to the
> PG_VERSION file.

No, the version command was removed.  I should not have said version
function.  select version() works just as always.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026


Thu, 8 Oct 1998 22:46:07 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] version functions (was NT port of PGSQL - success)
Hello Bruce

On Oct 8,  5:40pm, Bruce Momjian wrote:
> I have removed the data/base/*/pg_version file because it was never
> used.  We had removed the 'version' functions long ago, but
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Does this mean that the following from the FAQ is wrong?

	3.25) How do I tell what PostgreSQL version I am running?

	From psql, type select version();

If so then this question should probably be changed to point users to the
PG_VERSION file.

Just a thought

-- 
Sincerely,

Jazzman (a.k.a. Justin Hickey)  e-mail: jhickey@hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand
==================================================================
People who think they know everything are very irritating to those
of us who do.  ---Anonymous

Jazz and Trek Rule!!!
==================================================================

Fri, 9 Oct 1998 09:07:23 +0000
"Justin Hickey" <jhickey@impact1.hpcc.nectec.or.th>
Re: [HACKERS] version functions (was NT port of PGSQL - success)
Jazzman wrote:
>
> Hello Bruce
>
> On Oct 8,  5:40pm, Bruce Momjian wrote:
> > I have removed the data/base/*/pg_version file because it was never
> > used.  We had removed the 'version' functions long ago, but
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Does this mean that the following from the FAQ is wrong?
>
>    3.25) How do I tell what PostgreSQL version I am running?
>
>    From psql, type select version();
>
> If so then this question should probably be changed to point users to the
> PG_VERSION file.

    No, it is still correct. The version function is there and it
    returns the compiled in string from version.h.

    But take a look at version.c please. I think  it  should  use
    memcpy()  or  strncpy()  instead of strcpy(). As it is now it
    writes the null byte after the palloc'ed area.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck@debis.com (Jan Wieck) #



Fri, 9 Oct 1998 12:05:48 +0200 (MET DST)
jwieck@debis.com (Jan Wieck)
Re: [HACKERS] version functions (was NT port of PGSQL - success)
>     No, it is still correct. The version function is there and it
>     returns the compiled in string from version.h.
> 
>     But take a look at version.c please. I think  it  should  use
>     memcpy()  or  strncpy()  instead of strcpy(). As it is now it
>     writes the null byte after the palloc'ed area.

Yes, thanks.  Fixed using StrNCpy().

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026


Fri, 9 Oct 1998 12:39:17 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] version functions (was NT port of PGSQL - success)
I believe Tom Lane removed the double-backslash from the psql output by
modifying PQprint.

I recommend that is reversed, because psql escapes out the table column
delimiters with a single backslash, and now the format will be
ambigious.

	rtest=> create table test(x text);
	ERROR:  test relation already exists
	test=> create table test3(x text);
	CREATE
	test=> insert into test3 values ('\\x');
	INSERT 322185 1
	test=> select * from test3;
	x 
	--
	\x
	(1 row)
	

This used to show as:

	x 
	--
	\\x
	(1 row)

Comments?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026


Fri, 9 Oct 1998 16:24:14 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
backslash in psql output
> now the format will be ambigious.
>         test=> insert into test3 values ('\\x');
>         test=> select * from test3;
>         --
>         \x
> This used to show as:
>         --
>         \\x
> Comments?

Well, actually I've been thinking that this is closer to the behavior we
might want (though I haven't looked carefully at the new version). Of
course it bothered me more than it should have, since I misunderstood
where the re-escaping was happening; I had thought it was happening in
the backend.

psql could have an option to re-escape strings, but imho by default
should display what is stored, not what was typed in originally.

pg_dump _should_ re-escape everything, so that it reloads properly.

                     - $0.02 from Tom

Sat, 10 Oct 1998 02:31:07 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] backslash in psql output
> > now the format will be ambigious.
> >         test=> insert into test3 values ('\\x');
> >         test=> select * from test3;
> >         --
> >         \x
> > This used to show as:
> >         --
> >         \\x
> > Comments?
> 
> Well, actually I've been thinking that this is closer to the behavior we
> might want (though I haven't looked carefully at the new version). Of
> course it bothered me more than it should have, since I misunderstood
> where the re-escaping was happening; I had thought it was happening in
> the backend.
> 
> psql could have an option to re-escape strings, but imho by default
> should display what is stored, not what was typed in originally.
> 
> pg_dump _should_ re-escape everything, so that it reloads properly.

But what about backward compatability?  Aren't there people expecting
psql output to show double backslashes?  What do we do to display pipes
in the output?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026


Fri, 9 Oct 1998 23:04:13 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] backslash in psql output
> > psql could have an option to re-escape strings, but imho by default
> > should display what is stored, not what was typed in originally.
> > pg_dump _should_ re-escape everything, so that it reloads properly.
> But what about backward compatability?  Aren't there people expecting
> psql output to show double backslashes?  What do we do to display 
> pipes in the output?

We can include a command-line option to re-enable the escapes. Anyone
can use that form, and can define an alias for it if they need.

I understand your concern about backward compatibility, but imho the
convention it was using makes things more confusing, especially for a
new user. I'm not certain (assuming things have changed recently) that
the new behavior is what I would want, but the old behavior was not one
I would have guessed at.

Using psql to pipe output to another program is the best example of why
one might _not_ want to re-escape the strings. If the goal is to use
data from the database, then that data is what you will want to see in
the pipe.

Anyway, in the long run I'd like to consider moving toward a "no-escape"
default output for psql. If you think that it is too short notice to
fully understand the ramifications of a change this close to release, or
if your "faithful correspondents" have cottage cheese for brains, then
go ahead and revert it ;) But I like the new behavior in principle...

                    - Tom

Another possibility is to make it a configuration option:

  ./configure --enable-psql-string-escapes

or something like that...

Sat, 10 Oct 1998 04:21:32 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] backslash in psql output
> We can include a command-line option to re-enable the escapes. Anyone
> can use that form, and can define an alias for it if they need.
> 
> I understand your concern about backward compatibility, but imho the
> convention it was using makes things more confusing, especially for a
> new user. I'm not certain (assuming things have changed recently) that
> the new behavior is what I would want, but the old behavior was not one
> I would have guessed at.
> 
> Using psql to pipe output to another program is the best example of why
> one might _not_ want to re-escape the strings. If the goal is to use
> data from the database, then that data is what you will want to see in
> the pipe.
> 
> Anyway, in the long run I'd like to consider moving toward a "no-escape"
> default output for psql. If you think that it is too short notice to
> fully understand the ramifications of a change this close to release, or
> if your "faithful correspondents" have cottage cheese for brains, then
> go ahead and revert it ;) But I like the new behavior in principle...
> 
>                     - Tom
> 
> Another possibility is to make it a configuration option:
> 
>   ./configure --enable-psql-string-escapes
> 
> or something like that...
> 

I realize the double-backslash is confusing, but I don't think we can
make such a user-visible change at this time.  I think we need to open
discussion on this issue on the general list, and to include discussion
of NULL displays, and any other issues, as well as how to properly
output the column separation character if that appears in the data.

So, I think we have to put it back to the old way, and open discussion
about this after 6.4.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026


Sat, 10 Oct 1998 01:20:57 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] backslash in psql output
Bruce Momjian wrote:
  >> > now the format will be ambigious.
  >> >         test=> insert into test3 values ('\\x');
  >> >         test=> select * from test3;
  >> >         --
  >> >         \x
  >> > This used to show as:
  >> >         --
  >> >         \\x
  >> > Comments?
  >>...
  >But what about backward compatability?  Aren't there people expecting
  >psql output to show double backslashes?  What do we do to display pipes
  >in the output?

That change seems a good thing: the front-end ought to display what the
user wants.  Any manipulations should be done behind the scenes.  If I
store a DOS pathname, I don't want to see the backslashes doubled in it.
Even worse, I don't want to see them eliminated altogether, which is what
happens now if I don't remember to double them on input.

You mentioned that psql backslash-escapes the column delimiter character.
I think that this behaviour ought to be removed as well; it should be obvious
from the alignment with headings and other lines whether a pipe character is
part of the data or a column delimiter.  If it really matters, a user
can specify another character to use as delimiter. 

An unsophisticated user expects to type characters and have them
accepted; he should not have to know that certain characters need to be
doubled or escaped, nor that certain characters he sees in the output are
to be ignored.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "He that covereth his sins shall not prosper; but whoso
      confesseth and forsaketh them shall have mercy."      
                                         Proverbs 28:13 



Sat, 10 Oct 1998 08:01:55 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] backslash in psql output 
> Bruce Momjian wrote:
>   >> > now the format will be ambigious.
>   >> >         test=> insert into test3 values ('\\x');
>   >> >         test=> select * from test3;
>   >> >         --
>   >> >         \x
>   >> > This used to show as:
>   >> >         --
>   >> >         \\x
>   >> > Comments?
>   >>...
>   >But what about backward compatability?  Aren't there people expecting
>   >psql output to show double backslashes?  What do we do to display pipes
>   >in the output?
> 
> That change seems a good thing: the front-end ought to display what the
> user wants.  Any manipulations should be done behind the scenes.  If I
> store a DOS pathname, I don't want to see the backslashes doubled in it.
> Even worse, I don't want to see them eliminated altogether, which is what
> happens now if I don't remember to double them on input.
> 
> You mentioned that psql backslash-escapes the column delimiter character.
> I think that this behaviour ought to be removed as well; it should be obvious
> from the alignment with headings and other lines whether a pipe character is
> part of the data or a column delimiter.  If it really matters, a user
> can specify another character to use as delimiter. 

If the user is reading psql output into a program, it is very unclear
how to determine a valid column delimiter vs. a delimiter in the data. 
Yes, they can change delimiters, but they person has to choose one that
would never appear in the data stream, and that is sometimes impossible.

I don't know how many people do this type of thing, but I think we have
to ask the general users.

> 
> An unsophisticated user expects to type characters and have them
> accepted; he should not have to know that certain characters need to be
> doubled or escaped, nor that certain characters he sees in the output are
> to be ignored.

If we don't require double backslashes on input, we can no longer accept
C escape sequences like \r or \n, or octal values.  As it is now, octal
values are now interpreted, instead of returning their values:

	test=> insert into test3 values ('\253');
	INSERT 323237 1
	test=> select * from test3;
	x       
	--------
	nedd
	�       
	�       
	(3 rows)

I don't know if it always did this or not.

I can understand people wanting to change things, but we have to discuss
all the issues.

And what about the COPY command.  Do you want to change the display of
escape characters there too?  I hope not.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026


Sat, 10 Oct 1998 11:02:41 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] backslash in psql output
Bruce Momjian <maillist@candle.pha.pa.us> writes:
> I realize the double-backslash is confusing, but I don't think we can
> make such a user-visible change at this time.  I think we need to open
> discussion on this issue on the general list, and to include discussion
> of NULL displays, and any other issues, as well as how to properly
> output the column separation character if that appears in the data.
> So, I think we have to put it back to the old way, and open discussion
> about this after 6.4.

Well, actually there *was* public discussion of this issue, on the
pgsql-interfaces list around 12/13 August.  The consensus was that
unnecessary backslashing was a bad idea --- in fact, I didn't see
*anyone* arguing in favor of the old behavior, and the people who
actually had backslashes in their data definitely didn't want it.
Admittedly it was a pretty small sample (Tom Lockhart and I were
two of the primary complainers) but there wasn't any sentiment
for keeping the old behavior.

Keep in mind that what we are discussing here is the behavior of
PQprint(), not the behavior of FE/BE transport protocol or anything
else that affects data received by applications.  PQprint's goal in
life is to present data in a reasonably human-friendly way, *not*
to produce a completely unambiguous machine-readable syntax.  Its
output format is in fact very ambiguous.  Here's an example:

play=> create table test(id int4, val text);
play=> insert into test values (1, NULL);
play=> insert into test values (2, '    ');
play=> insert into test values (3, 'foobar');
play=> insert into test values (4, 'oneback\\slash');
play=> insert into test values (5, 'onevert|bar');
play=> select * from test;
id|val
--+-------------
 1|
 2|
 3|foobar
 4|oneback\slash
 5|onevert|bar
(5 rows)

You can't tell the difference between a NULL field and an all-blanks
value in this format; nor can you really be sure how many trailing
blanks there are in tuples 3 and 5.  So the goal is readability,
not lack of ambiguity.  Given that goal, I don't see the value of
printing backslash escapes.  Are you really having difficulty telling
the data vertical bar from the ones used as column separators?
Physical alignment is the cue the eye relies on, I think.

The only cases that PQprint inserted backslashes for were the column
separator char (unnecessary per above example), newlines (also not
exactly hard to recognize), and backslash itself.  All of these
seem unnecessary and confusing to me.

I'm sorry that this change sat in my to-do queue for so long, but
I don't see it as a last-minute thing.  The consensus to do it was
established two months ago.

			regards, tom lane

Sat, 10 Oct 1998 11:19:44 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] backslash in psql output 
Bruce Momjian wrote:...
  >And what about the COPY command.  Do you want to change the display of
  >escape characters there too?  I hope not.

No; but I wouldn't expect unsophisticated users to be using COPY.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "He that covereth his sins shall not prosper; but whoso
      confesseth and forsaketh them shall have mercy."      
                                         Proverbs 28:13 



Sat, 10 Oct 1998 23:06:22 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] backslash in psql output 
Bruce Momjian <maillist@candle.pha.pa.us> writes:
> If the user is reading psql output into a program, it is very unclear
> how to determine a valid column delimiter vs. a delimiter in the data. 

Quite true, but the PQprint output format has never been
program-friendly, as I pointed out previously.

I think the appropriate response to this gripe is to add another
command-line switch to psql, which would change its display of SELECT
data to something more suitable for program consumption, rather than
trying to make a single output format that's a bad compromise between
human readability and machine readability.

Perhaps the COPY data syntax would work for a program-friendly output
format?  (Basically tabs and newlines are field and row delimiters,
with (IIRC) backslash-escaping of tabs, newlines, backslash itself,
and maybe nulls and suchlike.  Also \N stands for a null field.)

> And what about the COPY command.  Do you want to change the display of
> escape characters there too?  I hope not.

I wasn't proposing any such thing, unless it proves necessary to ensure
we can copy arbitrary-8-bit text data out and back in.  But that's a
task for a future release.

			regards, tom lane

Sat, 10 Oct 1998 18:36:10 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] backslash in psql output 
> You can't tell the difference between a NULL field and an all-blanks
> value in this format; nor can you really be sure how many trailing
> blanks there are in tuples 3 and 5.  So the goal is readability,
> not lack of ambiguity.  Given that goal, I don't see the value of
> printing backslash escapes.  Are you really having difficulty telling
> the data vertical bar from the ones used as column separators?
> Physical alignment is the cue the eye relies on, I think.
> 
> The only cases that PQprint inserted backslashes for were the column
> separator char (unnecessary per above example), newlines (also not
> exactly hard to recognize), and backslash itself.  All of these
> seem unnecessary and confusing to me.

OK, I understand your point here, that we must maximize readability, and
that robustness is not as important.

OK, let's keep the removal of backslashes.  Can you recommend a nice
NULL display, perhaps '[NULL]' or '<NULL>'?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026


Sat, 10 Oct 1998 21:45:50 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] backslash in psql output
Thus spake Bruce Momjian
> OK, let's keep the removal of backslashes.  Can you recommend a nice
> NULL display, perhaps '[NULL]' or '<NULL>'?

I'd like to make at least one vote to keep the status quo.  Some of
us have come to depend on the existing behaviour.  At least make
it an option that you have to turn on.

-- 
D'Arcy J.M. Cain <darcy@{druid|vex}.net>   |  Democracy is three wolves
http://www.druid.net/darcy/                |  and a sheep voting on
+1 416 424 2871     (DoD#0082)    (eNTP)   |  what's for dinner.

Sun, 11 Oct 1998 00:40:24 -0400 (EDT)
darcy@druid.net (D'Arcy J.M. Cain)
Re: [HACKERS] backslash in psql output
> Thus spake Bruce Momjian
> > OK, let's keep the removal of backslashes.  Can you recommend a nice
> > NULL display, perhaps '[NULL]' or '<NULL>'?
> 
> I'd like to make at least one vote to keep the status quo.  Some of
> us have come to depend on the existing behaviour.  At least make
> it an option that you have to turn on.

Man, I can't win.  :-)

I vote for the status quo, and people want double backslashes removed. 
I want to add a NULL display, and people want status quo.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026


Sun, 11 Oct 1998 01:02:56 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] backslash in psql output
> 
> Bruce Momjian <maillist@candle.pha.pa.us> writes:
> > I realize the double-backslash is confusing, but I don't think we can
> > make such a user-visible change at this time.  I think we need to open
> > discussion on this issue on the general list, and to include discussion
> > of NULL displays, and any other issues, as well as how to properly
> > output the column separation character if that appears in the data.
> > So, I think we have to put it back to the old way, and open discussion
> > about this after 6.4.
> 
> Well, actually there *was* public discussion of this issue, on the
> pgsql-interfaces list around 12/13 August.  The consensus was that
> unnecessary backslashing was a bad idea --- in fact, I didn't see
> *anyone* arguing in favor of the old behavior, and the people who
> actually had backslashes in their data definitely didn't want it.
> Admittedly it was a pretty small sample (Tom Lockhart and I were
> two of the primary complainers) but there wasn't any sentiment
> for keeping the old behavior.
> 
> Keep in mind that what we are discussing here is the behavior of
> PQprint(), not the behavior of FE/BE transport protocol or anything
> else that affects data received by applications.  PQprint's goal in
> life is to present data in a reasonably human-friendly way, *not*
> to produce a completely unambiguous machine-readable syntax.  Its
> output format is in fact very ambiguous.  Here's an example:
> 
> play=> create table test(id int4, val text);
> play=> insert into test values (1, NULL);
> play=> insert into test values (2, '    ');
> play=> insert into test values (3, 'foobar');
> play=> insert into test values (4, 'oneback\\slash');
> play=> insert into test values (5, 'onevert|bar');
> play=> select * from test;
> id|val
> --+-------------
>  1|
>  2|
>  3|foobar
>  4|oneback\slash
>  5|onevert|bar
> (5 rows)
> 
> You can't tell the difference between a NULL field and an all-blanks
> value in this format; nor can you really be sure how many trailing
> blanks there are in tuples 3 and 5.  So the goal is readability,
> not lack of ambiguity.  Given that goal, I don't see the value of
> printing backslash escapes.  Are you really having difficulty telling
> the data vertical bar from the ones used as column separators?
> Physical alignment is the cue the eye relies on, I think.
> 
> The only cases that PQprint inserted backslashes for were the column
> separator char (unnecessary per above example), newlines (also not
> exactly hard to recognize), and backslash itself.  All of these
> seem unnecessary and confusing to me.
> 
> I'm sorry that this change sat in my to-do queue for so long, but
> I don't see it as a last-minute thing.  The consensus to do it was
> established two months ago.
> 
> 			regards, tom lane
> 
> 
> 

In my opinion we should privilege machine-readableness first and then provide
some user option to enable user-friendly conversion in psql output if one
really needs it.

In situations where data is processed by other programs it is very important
that there is no ambiguity in strings exchanged between the application and
the backend. This is already done for input, which supports C-like escape,
but not yet for output, which can produce ambiguous data when nulls, arrays
or non-printing characters are involved. This is the reason why I always use
my C-like output functions (contrib/string-io) in all my applications.

These arguments apply also to the copy command which uses the same output
functions. Consider the case where a text field contains a multi-line string
with newlines embedded; if you export the table into an external files the
field is split into many lines which are interpreted as separate records by
commonly used line-oriented filters like awk or grep.

I believe that the right way to handle all this stuff is the following:

  input:

           binary data            escaped data
                |                       |
        (user conversion)         (psql input)
                |                       |
                +-----------------------+
                |
          escaped query
                |
             (libpq)
                |
          escaped query            escaped data
                |                       |
        (parser unescape)      (copy-from unescape)
                |                       |
                +-----------------------+
                |
           binary data
                |
         (input function)
                |
          internal data


  output:

          internal data
                |
        (output function)
                |
          escaped data
                |
                +-----------------------+
                |                       |
             (libpq)                (copy-to)
                |                       |
          escaped data            escaped data
                |
                |
                +-----------------------+-----------------------+
                |                       |                       |
        (user conversion)         (psql output)           (psql unescape)
                |                       |                       |
           binary data            escaped data             binary data


In the above schema binary data means the external representation of data
containing non-printing or delimiters characters like quotes or newlines.
In this schema all the data exchanged with the backend should be escaped
in order to guarantee unambiguity to applications. The input and output
user conversion functions could be provided by libpq as utilities, and the
conversion could possibly be done automatically by libpq itself if some
global flag is set by the application.
Psql input should accept only escaped data while the output could be escaped
(default) or binary depending on a user supplied switch.
Files read or written by the copy command should be always escaped with
exactly one record for line. Pg_dump should produce escaped strings.
All this stuff requires the use of new output functions like those provided
in contrib/string-io.

There is still the problem of distinguishing between scalars and arrays which
is necessary for user output conversion. In my output functions I solved the
problem by escaping the first '{' of each field if it is not an array.
Another problem is that array input requires a double escaping, one for the
query parser and a second one for the array parser. Also nulls (\0) are not
handled by the input code. This should be fixed if we want true binary data.

I don't know if C-escapes violate the ansi sql standard but I believe they
makes life easier for the programmer. And if we add some global flag in
libpq we could also do automatic conversion to be compatible with ansi sql
and old applications. Note that arrays aren't ansi sql anyway.

Anyway a runtime switch is preferable to a configure switch.

-- 
Massimo Dal Zotto

+----------------------------------------------------------------------+
|  Massimo Dal Zotto                email:  dz@cs.unitn.it             |
|  Via Marconi, 141                 phone:  ++39-461-534251            |
|  38057 Pergine Valsugana (TN)     www:  http://www.cs.unitn.it/~dz/  |
|  Italy                            pgp:  finger dz@tango.cs.unitn.it  |
+----------------------------------------------------------------------+

Sun, 11 Oct 1998 14:24:14 +0200 (MET DST)
Massimo Dal Zotto <dz@cs.unitn.it>
Re: [HACKERS] backslash in psql output
On Sun, 11 Oct 1998, D'Arcy J.M. Cain wrote:

> I'd like to make at least one vote to keep the status quo.  Some of
> us have come to depend on the existing behaviour.  At least make
> it an option that you have to turn on.
> 
> -- 
> D'Arcy J.M. Cain <darcy@{druid|vex}.net>   |  Democracy is three wolves
> http://www.druid.net/darcy/                |  and a sheep voting on
> +1 416 424 2871     (DoD#0082)    (eNTP)   |  what's for dinner.
> 

To which I add my hearty approval (I'd like both options)!

Marc Zuckman
marc@fallon.classyad.com

_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
_     Visit The Home and Condo MarketPlace		      _
_          http://www.ClassyAd.com			      _
_							      _
_  FREE basic property listings/advertisements and searches.  _
_							      _
_  Try our premium, yet inexpensive services for a real	      _
_   selling or buying edge!				      _
_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_


Sun, 11 Oct 1998 13:10:41 -0400 (EDT)
Marc Howard Zuckman <marc@fallon.classyad.com>
Re: [HACKERS] backslash in psql output
(Sorry for being so slow to respond to this...)

Massimo Dal Zotto <dz@cs.unitn.it> writes:
> I believe that the right way to handle all this stuff is the following:

>   input:

>            binary data            escaped data
>                 |                       |
>         (user conversion)         (psql input)
>                 |                       |
>                 +-----------------------+
>                 |
>           escaped query
>                 |
>              (libpq)
>                 |
>           escaped query            escaped data
>                 |                       |
>         (parser unescape)      (copy-from unescape)
>                 |                       |
>                 +-----------------------+
>                 |
>            binary data
>                 |
>          (input function)
>                 |
>           internal data


>   output:

>           internal data
>                 |
>         (output function)
>                 |
>           escaped data
>                 |
>                 +-----------------------+
>                 |                       |
>              (libpq)                (copy-to)
>                 |                       |
>           escaped data            escaped data
>                 |
>                 |
>                 +-----------------------+-----------------------+
>                 |                       |                       |
>         (user conversion)         (psql output)           (psql unescape)
>                 |                       |                       |
>            binary data            escaped data             binary data


I disagree with this, at least for the output side.  The FE/BE protocol
for SELECT/FETCH results is already completely 8-bit clean.  There is
no reason to escape output data before passing it across the wire and
through libpq.  The application program might want to escape the data
for its own purposes, but that's not our concern.

As far as I can tell, COPY IN/OUT data is the only case where we really
have an issue.  Since the COPY protocol is inherently text-based, we
have to escape anything that won't do as text.  (Offhand, I think only
tab, newline, null, and of course backslash are essential to convert,
though we might also want to convert other nonprinting characters for
readability's sake.)  The conversions involved need to be nailed down
and documented as part of the FE/BE protocol.

Coping with array-valued fields is also a concern --- there needs to
be some reasonable way for an application to discover that a given field
is an array and what datatype it is an array of.  But I think the need
there is to extend the RowDescription information returned by SELECT,
not to modify the data representation.

			regards, tom lane

Sun, 18 Oct 1998 12:28:07 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] backslash in psql output 
> As far as I can tell, COPY IN/OUT data is the only case where we really
> have an issue.  Since the COPY protocol is inherently text-based, we
> have to escape anything that won't do as text.  (Offhand, I think only
> tab, newline, null, and of course backslash are essential to convert,
> though we might also want to convert other nonprinting characters for
> readability's sake.)  The conversions involved need to be nailed down
> and documented as part of the FE/BE protocol.
\. as end-of-input is also escaped.  Not sure that gets sent to the
backend, or is just used by the frontend protocol to signal
end-of-input.


> 
> Coping with array-valued fields is also a concern --- there needs to
> be some reasonable way for an application to discover that a given field
> is an array and what datatype it is an array of.  But I think the need
> there is to extend the RowDescription information returned by SELECT,
> not to modify the data representation.

Yes, arrays can be a problem.  Not sure.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 18 Oct 1998 14:29:13 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] backslash in psql output
Hi. I believe we are still shooting for a Nov 1 release, though without
reports of successful regression tests on more platforms I'm not sure we
can do that. I know that at least some of those listed below are the
active development platforms for some contributors, so those are
probably covered but I need confirmation. So, if you have a platform you
have tested or plan to have tested in the next few days please speak up.
Now. Or at least Soon :)

Here are the ones on the "currently supported" list (let me know if you
have something running on another platform. Any Ultrix people out there
still?):

_  AIX 4.1.x-4.2
_  BSDi
_  FreeBSD 2.2.x-3.x
_  NetBSD 1.3
_  NetBSD 1.3 NS32532
_  NetBSD 1.3 Sparc
_  NetBSD 1.3 VAX
_  DGUX 5.4R4.11 m88k
_  HPUX 10.20
_  IRIX 6.x
_  Digital 4.0
_  linux 2.0.x Alpha
x  linux 2.0.x x86
_  linux 2.0.x Sparc
x  mklinux PPC750
_  SCO
_  Solaris x86
_  Solaris 2.5.1-2.6 x86
_  SunOS 4.1.4 Sparc
x  SVR4 MIPS
_  SVR4 4.4 m88k
x  Unixware x86
x  Windows NT


The porting info goes into the Admin Guide in the docs. I plan to freeze
that one last, a few days before release to give Bruce et al a chance to
polish the installation and release notes.

The other docs will need to freeze earlier to give me a chance to
generate hardcopy for v6.4. So the freeze schedule will be (again
assuming a Nov 1 release, and I'm probably not giving myself enough
time):

Oct 26: freeze Programmer's Guide and Developer's Guide
Oct 27: freeze User's Guide and reference pages
Oct 28: freeze Admin Guide
Oct 29-30: finish hardcopy, generate html

I will be out of town Oct 31-Nov 1, so need to finish a day or two
early. As it is, I should have frozen some docs by now to get this stuff
done.

So, if you have anything else to contribute or update for docs, SEND IT
IN NOW. Or at least let me know it is coming soon. Or it will have to
wait for v6.5...

TIA

                     - Tom

Sun, 25 Oct 1998 04:29:38 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Last call?
> Here are the ones on the "currently supported" list (let me know if you
> have something running on another platform. Any Ultrix people out there
> still?):

Change:
	
	> _  BSDi
	
to
	
	> _  BSDI 3.x and 4.0


Also, the Windows NT item is of much interest.  Can we report this as
working, and have a binary of 6.4 built at the time of the 6.4 release,
November 1?  (I am CC'ing the NT person.)

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 25 Oct 1998 01:17:54 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Last call?
> Change:
>         > _  BSDi
> to
>         > _  BSDI 3.x and 4.0

The underscores meant that I don't know if the platform has been
regression tested yet. Exes meant I did. I was assuming that you had
already done BSDI (isn't that you're development platform?) and was
hoping to get a report from you saying to put an "x" for that one.

So the only platforms I've marked as being confirmed are NT, Unixware,
SVR4, mklinux, and linux. But I know there are others which are already
working, I'm just not certain exactly which ones...

                      - Tom

Sun, 25 Oct 1998 14:21:19 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [DOCS] Last call?
Hi Tom and all

I did not know what the '_' and 'x' meant the first time, as I recall the
one for 'Linux ix86' had an '_', or unknown, I can do this for you, should
I grab the Bata2 or the latest snapshot?

On Sun, 25 Oct 1998, Thomas G. Lockhart wrote:

> > Change:
> >         > _  BSDi
> > to
> >         > _  BSDI 3.x and 4.0
> 
> The underscores meant that I don't know if the platform has been
> regression tested yet. Exes meant I did. I was assuming that you had
> already done BSDI (isn't that you're development platform?) and was
> hoping to get a report from you saying to put an "x" for that one.
> 
> So the only platforms I've marked as being confirmed are NT, Unixware,
> SVR4, mklinux, and linux. But I know there are others which are already
> working, I'm just not certain exactly which ones...
> 
>                       - Tom
> 

Terry Mackintosh <terry@terrym.com>          http://www.terrym.com
sysadmin/owner  Please! No MIME encoded or HTML mail, unless needed.

Proudly powered by R H Linux 4.2, Apache 1.3, PHP 3, PostgreSQL 6.3
-------------------------------------------------------------------
Success Is A Choice ... book by Rick Patino, get it, read it!


Sun, 25 Oct 1998 10:32:51 -0500 (EST)
Terry Mackintosh <terry@terrym.com>
Re: [HACKERS] Re: [DOCS] Last call?
"Thomas G. Lockhart" wrote:
  >So the only platforms I've marked as being confirmed are NT, Unixware,
  >SVR4, mklinux, and linux. 

Tom, can you distinguish between linux with libc5 and with glibc (aka libc6)?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "If ye then be risen with Christ, seek those things 
      which are above, where Christ sitteth on the right 
      hand of God. Set your affection on things above, not 
      on things on the earth."              Colossians 3:1,2



Sun, 25 Oct 1998 16:03:47 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [DOCS] Last call? 
> I did not know what the '_' and 'x' meant the first time, as I recall 
> the one for 'Linux ix86' had an '_', or unknown

Thanks Terry, but I've already got that one done (it's my development
platform). The Linux Alpha and Sparc need testing, which is probably
what you remember seeing...

                     - Tom

Sun, 25 Oct 1998 16:09:57 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: [DOCS] Last call?
> can you distinguish between linux with libc5 and with glibc?

Yes, if necessary. I'm running glibc at work (with the RH 6.3.2 rpms)
and libc5 at home (with v6.4beta). I don't expect there to be any
significant differences; are you aware of any?

Also, I'm planning on doing a clean install of v6.4beta on my glibc
machine, so can verify it then.

Or are you just saying that we should mention both explicitly so that
people can know that it would work on their machine for sure?

                   - Tom

Sun, 25 Oct 1998 16:14:24 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [DOCS] Last call?
> > Change:
> >         > _  BSDi
> > to
> >         > _  BSDI 3.x and 4.0
> 
> The underscores meant that I don't know if the platform has been
> regression tested yet. Exes meant I did. I was assuming that you had
> already done BSDI (isn't that you're development platform?) and was
> hoping to get a report from you saying to put an "x" for that one.
> 
> So the only platforms I've marked as being confirmed are NT, Unixware,
> SVR4, mklinux, and linux. But I know there are others which are already
> working, I'm just not certain exactly which ones...

Oh, sorry.  Put an X for BSDI.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 25 Oct 1998 11:44:37 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Last call?
   Here are the ones on the "currently supported" list (let me know if you
   have something running on another platform. Any Ultrix people out there
   still?):

   x  NetBSD 1.3.2/i386

I'm running on NetBSD 1.3.2/i386 and as of yesterday the regressions
all passed.  I'm trying to redo them every few days to make sure
nothing creeps in so this should be a supported platform.

Cheers,
Brook

Sun, 25 Oct 1998 10:29:51 -0700 (MST)
Brook Milligan <brook@trillium.NMSU.Edu>
Re: Last call?
>    x  NetBSD 1.3.2/i386
> I'm running on NetBSD 1.3.2/i386 and as of yesterday the regressions
> all passed.  I'm trying to redo them every few days to make sure
> nothing creeps in so this should be a supported platform.

This is exactly what makes it a supported platform :)

                 - Tom

Sun, 25 Oct 1998 17:33:54 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: Last call?
> > 'Linux ix86' had an '_', or unknown
> Thanks Terry, but I've already got that one done

But Oliver was asking about glibc2 vs libc5. Do you happen to have a
glibc machine (RH 5.x?) available? If so we could use a direct report of
success since I'm still running RH4.2/libc5 for development...

                    - Tom

Sun, 25 Oct 1998 18:07:59 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: [DOCS] Last call?
I compiled it on RH 5.1... no problems.

Taral

> -----Original Message-----
> From: owner-pgsql-hackers@postgreSQL.org
> [mailto:owner-pgsql-hackers@postgreSQL.org]On Behalf Of Thomas G.
> Lockhart
> Sent: Sunday, October 25, 1998 12:08 PM
> To: Terry Mackintosh; PostgreSQL-development
> Subject: Re: [HACKERS] Re: [DOCS] Last call?
> 
> 
> > > 'Linux ix86' had an '_', or unknown
> > Thanks Terry, but I've already got that one done
> 
> But Oliver was asking about glibc2 vs libc5. Do you happen to have a
> glibc machine (RH 5.x?) available? If so we could use a direct report of
> success since I'm still running RH4.2/libc5 for development...
> 
>                     - Tom
> 

Sun, 25 Oct 1998 12:15:15 -0600
"Taral" <taral@mail.utexas.edu>
RE: [HACKERS] Re: [DOCS] Last call?
I'm sure Oliver runs a libc6 system. He is one of the debian core
developers.

-Egon

On Sun, 25 Oct 1998, Thomas G. Lockhart wrote:

> > > 'Linux ix86' had an '_', or unknown
> > Thanks Terry, but I've already got that one done
> 
> But Oliver was asking about glibc2 vs libc5. Do you happen to have a
> glibc machine (RH 5.x?) available? If so we could use a direct report of
> success since I'm still running RH4.2/libc5 for development...
> 
>                     - Tom
> 
> 


Sun, 25 Oct 1998 19:19:39 +0100 (MET)
Egon Schmid <eschmid@stuttgart.netsurf.de>
Re: [HACKERS] Re: [DOCS] Last call?
Hi Tom

Nope, RH4.2/libc5, same as you.
Still waiting for the dust to settle on the glibc thing:)

On Sun, 25 Oct 1998, Thomas G. Lockhart wrote:

> > > 'Linux ix86' had an '_', or unknown
> > Thanks Terry, but I've already got that one done
> 
> But Oliver was asking about glibc2 vs libc5. Do you happen to have a
> glibc machine (RH 5.x?) available? If so we could use a direct report of
> success since I'm still running RH4.2/libc5 for development...
> 
>                     - Tom


Terry Mackintosh <terry@terrym.com>          http://www.terrym.com
sysadmin/owner  Please! No MIME encoded or HTML mail, unless needed.

Proudly powered by R H Linux 4.2, Apache 1.3, PHP 3, PostgreSQL 6.3
-------------------------------------------------------------------
Success Is A Choice ... book by Rick Patino, get it, read it!


Sun, 25 Oct 1998 13:26:45 -0500 (EST)
Terry Mackintosh <terry@terrym.com>
Re: [HACKERS] Re: [DOCS] Last call?
I cannot believe, RH4.2 isn't a glibc2 system.

-Egon

On Sun, 25 Oct 1998, Terry Mackintosh wrote:

> Hi Tom
> 
> Nope, RH4.2/libc5, same as you.
> Still waiting for the dust to settle on the glibc thing:)
> 
> On Sun, 25 Oct 1998, Thomas G. Lockhart wrote:
> 
> > > > 'Linux ix86' had an '_', or unknown
> > > Thanks Terry, but I've already got that one done
> > 
> > But Oliver was asking about glibc2 vs libc5. Do you happen to have a
> > glibc machine (RH 5.x?) available? If so we could use a direct report of
> > success since I'm still running RH4.2/libc5 for development...
> > 
> >                     - Tom
> 
> 
> Terry Mackintosh <terry@terrym.com>          http://www.terrym.com
> sysadmin/owner  Please! No MIME encoded or HTML mail, unless needed.
> 
> Proudly powered by R H Linux 4.2, Apache 1.3, PHP 3, PostgreSQL 6.3
> -------------------------------------------------------------------
> Success Is A Choice ... book by Rick Patino, get it, read it!
> 
> 
> 


Sun, 25 Oct 1998 19:26:51 +0100 (MET)
Egon Schmid <eschmid@stuttgart.netsurf.de>
Re: [HACKERS] Re: [DOCS] Last call?

I just gave today's (Oct 25) snapshot a try on Sparc Linux.  
Looks good except datetime.   I'm getting failures due to this type 
of thing:

regression=> SELECT ('today'::datetime );
?column?
----------------------------
Sun Oct 25 00:00:00 1998 EDT
(1 row)

regression=> SELECT  ('tomorrow'::datetime - '1 day'::timespan);
?column?
----------------------------
Sun Oct 25 01:00:00 1998 EDT
(1 row)                                                                        


I *think* this may because we're not too far into EST yet.
Sound good?

My machine is Kernel is 2.0.29.  Libc 5.3.12.

Tom Szybist
szybist@boxhill.com




In message <3632A932.7FEB1733@alumni.caltech.edu>, "Thomas G. Lockhart" writes:
> Hi. I believe we are still shooting for a Nov 1 release, though without
> reports of successful regression tests on more platforms I'm not sure we
> can do that. I know that at least some of those listed below are the
> active development platforms for some contributors, so those are
> probably covered but I need confirmation. So, if you have a platform you
> have tested or plan to have tested in the next few days please speak up.
> Now. Or at least Soon :)
> 
> Here are the ones on the "currently supported" list (let me know if you
> have something running on another platform. Any Ultrix people out there
> still?):
> 
> _  AIX 4.1.x-4.2
> _  BSDi
> _  FreeBSD 2.2.x-3.x
> _  NetBSD 1.3
> _  NetBSD 1.3 NS32532
> _  NetBSD 1.3 Sparc
> _  NetBSD 1.3 VAX
> _  DGUX 5.4R4.11 m88k
> _  HPUX 10.20
> _  IRIX 6.x
> _  Digital 4.0
> _  linux 2.0.x Alpha
> x  linux 2.0.x x86
> _  linux 2.0.x Sparc
> x  mklinux PPC750
> _  SCO
> _  Solaris x86
> _  Solaris 2.5.1-2.6 x86
> _  SunOS 4.1.4 Sparc
> x  SVR4 MIPS
> _  SVR4 4.4 m88k
> x  Unixware x86
> x  Windows NT
> 
> 
> The porting info goes into the Admin Guide in the docs. I plan to freeze
> that one last, a few days before release to give Bruce et al a chance to
> polish the installation and release notes.
> 
> The other docs will need to freeze earlier to give me a chance to
> generate hardcopy for v6.4. So the freeze schedule will be (again
> assuming a Nov 1 release, and I'm probably not giving myself enough
> time):
> 
> Oct 26: freeze Programmer's Guide and Developer's Guide
> Oct 27: freeze User's Guide and reference pages
> Oct 28: freeze Admin Guide
> Oct 29-30: finish hardcopy, generate html
> 
> I will be out of town Oct 31-Nov 1, so need to finish a day or two
> early. As it is, I should have frozen some docs by now to get this stuff
> done.
> 
> So, if you have anything else to contribute or update for docs, SEND IT
> IN NOW. Or at least let me know it is coming soon. Or it will have to
> wait for v6.5...
> 
> TIA
> 
>                      - Tom
> 

Sun, 25 Oct 1998 14:24:13 -0500
"Thomas A. Szybist" <szybist@boxhill.com>
Re: [HACKERS] Last call? 
"Thomas G. Lockhart" wrote:
  >> can you distinguish between linux with libc5 and with glibc?
  >
  >Yes, if necessary. I'm running glibc at work (with the RH 6.3.2 rpms)
  >and libc5 at home (with v6.4beta). I don't expect there to be any
  >significant differences; are you aware of any?

No; there are minor textual differences in the math overflow messages
and, last time I tried, the geometry tests had differences of around
10 ^ -12.

  >...
  >Or are you just saying that we should mention both explicitly so that
  >people can know that it would work on their machine for sure?
 
Precisely.

The difference is such a major one that linux-libc5 and linux-glibc
should be regarded nearly as two different systems.
-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "If ye then be risen with Christ, seek those things 
      which are above, where Christ sitteth on the right 
      hand of God. Set your affection on things above, not 
      on things on the earth."              Colossians 3:1,2



Sun, 25 Oct 1998 21:31:23 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: [DOCS] Last call? 
"Thomas A. Szybist" wrote:
> 
> I just gave today's (Oct 25) snapshot a try on Sparc Linux.  
> Looks good except datetime.   I'm getting failures due to this type 
> of thing:
> 
> regression=> SELECT ('today'::datetime );
> ?column?
> ----------------------------
> Sun Oct 25 00:00:00 1998 EDT
> (1 row)
> 
> regression=> SELECT  ('tomorrow'::datetime - '1 day'::timespan);
> ?column?
> ----------------------------
> Sun Oct 25 01:00:00 1998 EDT
> (1 row)                                                                        
> 
> 
> I *think* this may because we're not too far into EST yet.
> Sound good?
> 

This is not a failure.  The date 24 hours (1 day) before 'Mon Oct 26 00:00:00 1998 EST' is 'Sun Oct 25 01:00:00 1998 EDT'.  You would think it should be 'Sun Oct 25 00:00:00 1998 EST', but thanks to Daylight Savings Time that datetime does not exist :-)
-- 
____       | Billy G. Allie    | Domain....: Bill.Allie@mug.org
|  /|      | 7436 Hartwell     | Compuserve: 76337,2061
|-/-|----- | Dearborn, MI 48126| MSN.......: B_G_Allie@email.msn.com
|/  |LLIE  | (313) 582-1540    | 



Sun, 25 Oct 1998 16:35:16 -0500
"Billy G. Allie" <Bill.Allie@mug.org>
Re: [HACKERS] Last call? 
> I just gave today's (Oct 25) snapshot a try on Sparc Linux.
> Looks good except datetime.

OK, picky picky. Run the test tomorrow instead. And I'll mark you down
as having tested and passed today :)

                  - Tom

Sun, 25 Oct 1998 22:56:11 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Last call?
> I compiled it on RH 5.1... no problems.

OK. Just to be unambiguous, I'd prefer a statement that includes "...
and passed all regression tests". It did, right?

                  - Tom

Sun, 25 Oct 1998 22:58:12 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: [DOCS] Last call?
> The difference is such a major one that linux-libc5 and linux-glibc
> should be regarded nearly as two different systems.

Sure. Well, I'm planning on building v6.4beta at work anyway, and will
try the regression tests. So we should have a firm confirmation for v6.4
then. Though it sounds like Oliver has tried something fairly recently?

Aside from the math rounding problems (I even see slight differences on
my libc5 machine when using the egcs compiler) I _really_ don't expect
to see a true failure.

                 - Tom

Sun, 25 Oct 1998 23:02:53 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: [DOCS] Last call?
Err.. umm... Updated 10/25 17:05 CST. The following tests fail on my system
(RH 5.1 - glibc):

int2, int4: Different error format
(Math result not representable is now Numerical result out of range)
float8: Weird... one query that was supposed to fail returned a bunch of
NaNs, another failed when it wasn't supposed to with an out of range error.
geometry: The results are only approximately right... but only in the first
sig. figure :(
datetime: CDT/CST thing
sanity_check: **backend aborts**
random: Fails because it returns a value outside the 80-120 range
misc: Rows out of order

I'm afraid this one can't go as 'supported'. Sorry.

Taral

> -----Original Message-----
> From: owner-pgsql-hackers@postgreSQL.org
> [mailto:owner-pgsql-hackers@postgreSQL.org]On Behalf Of Thomas G.
> Lockhart
> Sent: Sunday, October 25, 1998 4:58 PM
> To: Taral
> Cc: Terry Mackintosh; PostgreSQL-development
> Subject: Re: [HACKERS] Re: [DOCS] Last call?
>
>
> > I compiled it on RH 5.1... no problems.
>
> OK. Just to be unambiguous, I'd prefer a statement that includes "...
> and passed all regression tests". It did, right?
>
>                   - Tom
>


Sun, 25 Oct 1998 17:29:22 -0600
"Taral" <taral@mail.utexas.edu>
RE: [HACKERS] Re: [DOCS] Last call?
> sanity_check: **backend aborts**
> random: Fails because it returns a value outside the 80-120 range
> misc: Rows out of order

All passed when I reinitialized the db.

Taral

Sun, 25 Oct 1998 17:37:15 -0600
"Taral" <taral@mail.utexas.edu>
RE: [HACKERS] Re: [DOCS] Last call?
> > sanity_check: **backend aborts**
> > random: Fails because it returns a value outside the 80-120 range
> > misc: Rows out of order
> All passed when I reinitialized the db.

OK, good. 

btw, the random test will still occasionally fail, since a two or three
sigma result will have values outside the 80-120 range. But rerunning
should get something within range, usually.

                  - Tom

Mon, 26 Oct 1998 00:42:40 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: [DOCS] Last call?
> > > sanity_check: **backend aborts**
> > > random: Fails because it returns a value outside the 80-120 range
> > > misc: Rows out of order

Note that int2,int4,float8,geometry are still failing...

Why is geometry sooo far out?

Taral

Sun, 25 Oct 1998 19:17:28 -0600
"Taral" <taral@mail.utexas.edu>
RE: [HACKERS] Re: [DOCS] Last call?
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu> writes:
> _  HPUX 10.20

You can put down an X for both HPUX 9.03 and 10.20.

I discovered a number of minor problems when I tried to compile with
HP's cc instead of gcc like I usually do.  I just committed fixes for
those.

I am still getting a discrepancy in the "rules" regression test,
namely a difference in the order in which tuples are returned:

*** expected/rules.out	Fri Oct  2 12:28:01 1998
--- results/rules.out	Sun Oct 25 19:31:42 1998
***************
*** 315,322 ****
  pname |sysname
  ------+-------
  bm    |pluto  
- jwieck|orion  
  jwieck|notjw  
  (3 rows)
  
  QUERY: delete from rtest_system where sysname = 'orion';
--- 315,322 ----
  pname |sysname
  ------+-------
  bm    |pluto  
  jwieck|notjw  
+ jwieck|orion  
  (3 rows)
  
  QUERY: delete from rtest_system where sysname = 'orion';

----------------------

This happens on all four permutations of HPUX version and compiler.
Are other people really seeing the tuple order given in the "expected"
file?

			regards, tom lane

Sun, 25 Oct 1998 20:30:15 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Last call? 
In message <3633AC8B.756C31BB@alumni.caltech.edu>, "Thomas G. Lockhart" writes:
> > I just gave today's (Oct 25) snapshot a try on Sparc Linux.
> > Looks good except datetime.
> 
> OK, picky picky. Run the test tomorrow instead. And I'll mark you down
> as having tested and passed today :)
> 
>                   - Tom

Do I still get full credit ?:)

I noticed your list did not have Solaris / Sparc.  I gave that a spin
as well.  Aside from datetime, my results confirm the expected diffs
supplied.

Tom Szybist
szybist@boxhill.com

Sun, 25 Oct 1998 21:46:53 -0500
"Thomas A. Szybist" <szybist@boxhill.com>
Re: [HACKERS] Last call? 
I still need reports on:

> _  AIX 4.1.x-4.2
> _  FreeBSD 2.2.x-3.x
> _  NetBSD 1.3 NS32532
> _  NetBSD 1.3 Sparc
> _  NetBSD 1.3 VAX
> _  DGUX 5.4R4.11 (m88k or ? we need a new maintainer)
> _  IRIX 6.x      (Andrew, are you available?)
> _  Digital 4.0
> _  linux 2.0.x Alpha
> _  SCO           (Billy, can you confirm success now?)
> _  Solaris x86
> _  Solaris 2.5.1-2.6 Sparc
> _  SunOS 4.1.4 Sparc
> _  SVR4 4.4 m88k (I have v6.3 "confirmed with patching". better now?)
> x  Windows NT

Tatsuo, is your usual stable of machines available for reporting? It
would be nice to get confirmation with that big-endian/little-endian
mix.

How should I list WindowsNT? I have "mostly working, check web site for
patches" at the moment, but I don't know if we have more info available
now.

I would welcome confirming reports on other platforms too.

                    - Tom

Mon, 26 Oct 1998 15:12:40 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Last call?
On Mon, 26 Oct 1998, Thomas G. Lockhart wrote:

> I still need reports on:
> 
> > _  linux 2.0.x Alpha

 	I have gotten it to compile successfully again (fixes to our old
friend S_LOCK) and I should have a patch out in a few days (big test {at
college} on Wedesday, so it will later in the week). Most of the
regression tests are working, except the datetime ones (no, sorry, can't
blame daylight savings this time, that is unless the year is all of a
sudden 2136. :( ). I doubt I will have that working by Nov 2, as I have
been trying for a few months with out success to find these bugs. I will
post the patch and more detail on the datetime problems by the end of this
week to the pgsql-{ports,hacker,patch} mailing lists. TTYAL.

----------------------------------------------------------------------------
|   "For to me to live is Christ, and to die is gain."                     |
|                                            --- Philippians 1:21 (KJV)    |
----------------------------------------------------------------------------
|  Ryan Kirkpatrick  |  Boulder, Colorado  | rkirkpat@nag.cs.colorado.edu  |
----------------------------------------------------------------------------
|               http://www-ugrad.cs.colorado.edu/~rkirkpat/                |
----------------------------------------------------------------------------


Mon, 26 Oct 1998 09:45:12 -0600 (CST)
Ryan Kirkpatrick <rkirkpat@nag.cs.colorado.edu>
Re: [HACKERS] Last call?
Tom Lane wrote:

> I am still getting a discrepancy in the "rules" regression test,
> namely a difference in the order in which tuples are returned:
>
> *** expected/rules.out Fri Oct  2 12:28:01 1998
> --- results/rules.out  Sun Oct 25 19:31:42 1998
> ***************
> *** 315,322 ****
>   pname |sysname
>   ------+-------
>   bm    |pluto
> - jwieck|orion
>   jwieck|notjw
>   (3 rows)
>
>   QUERY: delete from rtest_system where sysname = 'orion';
> --- 315,322 ----
>   pname |sysname
>   ------+-------
>   bm    |pluto
>   jwieck|notjw
> + jwieck|orion
>   (3 rows)
>
>   QUERY: delete from rtest_system where sysname = 'orion';
>
> ----------------------
>
> This happens on all four permutations of HPUX version and compiler.
> Are other people really seeing the tuple order given in the "expected"
> file?

    I  think  they  should  be in the order given in the expected
    file.

    The rows inserted into the rtest_admin table (really a table,
    not a view) are:

        pname|sysname
        -----+-------
        jw   |orion
        jw   |notjw
        bm   |neptun

    Then  two  updates  are  performed.  The rules that are there
    would add parsetrees as if these queries are given:

        UPDATE rtest_admin SET sysname = 'pluto'
            WHERE rtest_system.sysname = 'neptun'
              AND rtest_admin.sysname = rtest_system.sysname;

        UPDATE rtest_admin SET pname = 'jwieck'
            WHERE rtest_person.pdesc = 'Jan Wieck'
              AND rtest_admin.pname = rtest_person.pdesc;

    These two queries will produce join plans. Since there are no
    indices  on  any of the tables, they should produce tuples in
    exactly the order they  where  entered  into  the  table.  An
    UPDATE  creates  a  new  tuple,  inserts  it and outdates the
    current by ctid.

    In rtest_system and rtest_person there are only  1  row  that
    matches  each of the given qualifications. So the question is
    why on HPUX the order of tuples  returned  in  the  resulting
    join  plans differs from other OS's. The SELECT that produces
    the wrong order above should result in a  SeqScan,  and  that
    must return the tuples in ctid order.

    The  first rule update on rtest_admin (fired at the UPDATE on
    rtest_system.sysname) doesn't change the order of the  tuples
    (or  did  you  omit this in your mail?). So why the hell does
    the second? Updated rows  always  appear  at  the  end  of  a
    SeqScan  in  the order they where updated. There is no vacuum
    between the updates, so the space from  the  outdated  tuples
    should not be reused here.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck@debis.com (Jan Wieck) #



Mon, 26 Oct 1998 19:44:02 +0100 (MET)
jwieck@debis.com (Jan Wieck)
Re: [HACKERS] Last call?
jwieck@debis.com (Jan Wieck) writes:
>     These two queries will produce join plans. Since there are no
>     indices  on  any of the tables, they should produce tuples in
>     exactly the order they  where  entered  into  the  table.

I modified the rules.sql test to perform an EXPLAIN of the query
that is generating the unexpected result, and it says:

QUERY: explain update rtest_person set pname = 'jwieck' where pdesc = 'Jan Wieck';
NOTICE:  QUERY PLAN:

Merge Join  (cost=0.00 size=1 width=42)
  ->  Seq Scan  (cost=0.00 size=0 width=0)
        ->  Sort  (cost=0.00 size=0 width=0)
              ->  Seq Scan on rtest_admin  (cost=0.00 size=0 width=30)
  ->  Seq Scan  (cost=0.00 size=0 width=0)
        ->  Sort  (cost=0.00 size=0 width=0)
              ->  Seq Scan on rtest_person  (cost=0.00 size=0 width=12)

NOTICE:  QUERY PLAN:

Seq Scan on rtest_person  (cost=0.00 size=0 width=18)

The thing that jumps out at me here is the "sort" step (which I assume
is on pname for rtest_admin and pdesc for rtest_person, those being the
fields to be joined).  Since the prior state of rtest_admin is

QUERY: select * from rtest_admin;
pname|sysname
-----+-------
jw   |orion  
jw   |notjw  
bm   |pluto  
(3 rows)

the two rows that will be updated have equal sort keys and therefore the
sort could legally return them in either order.  Does Postgres contain
its own sort algorithm for this kind of operation, or does it depend on
the system qsort?  System library qsorts don't normally guarantee
stability for equal keys.  It could be that we're looking at a byproduct
of some minor implementation difference between the qsorts on my machine
and yours.  If that's it, though, I'm surprised that I'm the only one
reporting a difference in the test's output.

BTW, I get the same query plan if I EXPLAIN the query that you say the
rule should expand to,

        UPDATE rtest_admin SET pname = 'jwieck'
            WHERE rtest_person.pdesc = 'Jan Wieck'
              AND rtest_admin.pname = rtest_person.pdesc;

so there doesn't seem to be anything wrong with the rule rewriter...

			regards, tom lane

Mon, 26 Oct 1998 18:07:31 -0500
Tom Lane <tgl@sss.pgh.pa.us>
rules regression test diff (was Re: [HACKERS] Last call?)
On Mon, 26 Oct 1998, Thomas G. Lockhart wrote:

> I still need reports on:
> 
> > _  AIX 4.1.x-4.2

	Duane out there was going to do this one for us, assuming that he
still had access to that machine...

> > _  FreeBSD 2.2.x-3.x

	Building now...

> > _  NetBSD 1.3 NS32532
> > _  NetBSD 1.3 Sparc
> > _  NetBSD 1.3 VAX
> > _  DGUX 5.4R4.11 (m88k or ? we need a new maintainer)
> > _  IRIX 6.x      (Andrew, are you available?)
> > _  Digital 4.0
> > _  linux 2.0.x Alpha
> > _  SCO           (Billy, can you confirm success now?)
> > _  Solaris x86
> > _  Solaris 2.5.1-2.6 Sparc

	Will rebuild and regress test both the x86/Sparc platforms for 2.6
during the day tomorrow...

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Mon, 26 Oct 1998 23:08:15 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Last call?
On Sun, 25 Oct 1998, Thomas G. Lockhart wrote:

> _  linux 2.0.x Alpha

Neither 6.4B2 nor the oct26 snapshot compile on my RH 5.0 Alpha Linux
(2.0.30) computer.

I don't have much time to work on it, but I can provide the gmake.log and
access to the machine if it would help.

Doug

------------------------------------------------------------------------
My views are actually owned by a small, strange, orange man from the
planet zikalikzutorabibian who can only communicate by making cryptic
oblong symbols in warm food.

If you have problems with my views, please take them up with him.
------------------------------------------------------------------------
For Hire:  Will write code/manage networks for obscene amounts of money.
------------------------------------------------------------------------
Doug Babst - Owner      | P.O. Box 103        |  (402) 463-3426 - Voice
TCG Computer Services   | Hastings, NE 68901  |  (402) 463-1754 - Fax
http://www.tcgcs.com    | http://www.babst.org/~dbabst/resume


Tue, 27 Oct 1998 00:10:39 -0600 (CST)
Douglas W Babst <dbabst@tcgcs.com>
Re: [HACKERS] Last call?
>Tatsuo, is your usual stable of machines available for reporting? It
>would be nice to get confirmation with that big-endian/little-endian
>mix.

I have run the regression tests of Oct26 Snapshot on some platforms
and a cross platform testing on MkLinux and FreeBSD.

Here are results.

(1) regression tests on some platforms

LinuxPPC on PowerMac:
	o PowerBook 2400c (PPC 603e) running 2.1.24 kernel

	  Note that 2.1.24 kernel doesn't support PCMCIA cards,
	  so we cannot use the network facility at all. sigh.
	  (Unix domain sockets are ok)
	  On the other hand, 2.1.1xx kernels do support PCMCIA,
	  unfortunately, these are broken in that using the Unix domain
	  sockets causes the system crash.
	  Anyway, these are not PostgreSQL's problem, of course.

	o regresion tests are almost good except the datetime test.

		SELECT ('tomorrow'::datetime = ('yesterday'::datetime
		 + '2 days'::timespan)) as "True"; --> shows 'false'
		SELECT ('current'::datetime = 'now'::datetime)
		as "True"; --> shows 'false'
		SELECT count(*) AS one FROM DATETIME_TBL WHERE 
		d1 = 'today'::datetime; --> no row selected
	
		They were ok on 6.3.2.

MkLinux on PowerMac:
	o PowerMac 7600 (PPC 750) running MkLinux DR3
	o Same failure of the datetime test as LinuxPPC

FreeBSD:
	o FreeBSD 2.2.6-RELEASE
	o datetime testing fails (seems same phenomenon as LinuxPPC)
	o int8 testing fails (is this normal?)

Seems there's something wrong with datetime. Comment?


(2) cross platform testing

I have run the test with FreeBSD 2.2.7 and Mklinux. Seems they are
happy to talk to each other.


Please let me know If you need addional testings.
--
Tatsuo Ishii
t-ishii@sra.co.jp

Tue, 27 Oct 1998 17:06:40 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] Last call? 
Tom Lane wrote:

> I modified the rules.sql test to perform an EXPLAIN of the query
> that is generating the unexpected result, and it says:
>
> QUERY: explain update rtest_person set pname = 'jwieck' where pdesc = 'Jan Wieck';
> NOTICE:  QUERY PLAN:
>
> Merge Join  (cost=0.00 size=1 width=42)
>   ->  Seq Scan  (cost=0.00 size=0 width=0)
>         ->  Sort  (cost=0.00 size=0 width=0)
>               ->  Seq Scan on rtest_admin  (cost=0.00 size=0 width=30)
>   ->  Seq Scan  (cost=0.00 size=0 width=0)
>         ->  Sort  (cost=0.00 size=0 width=0)
>               ->  Seq Scan on rtest_person  (cost=0.00 size=0 width=12)
>
> NOTICE:  QUERY PLAN:
>
> Seq Scan on rtest_person  (cost=0.00 size=0 width=18)

    Isn't it nice to have EXPLAIN doing the rewrite step?

> [...]
>
> the two rows that will be updated have equal sort keys and therefore the
> sort could legally return them in either order.  Does Postgres contain
> its own sort algorithm for this kind of operation, or does it depend on
> the system qsort?  System library qsorts don't normally guarantee
> stability for equal keys.  It could be that we're looking at a byproduct
> of some minor implementation difference between the qsorts on my machine
> and yours.  If that's it, though, I'm surprised that I'm the only one
> reporting a difference in the test's output.

    Could  be  the  reason.  createfirstrun() in psort.c is using
    qsort as a first try. Maybe we should  add  ORDER  BY  pname,
    sysname to the queries to avoid it.

>
> BTW, I get the same query plan if I EXPLAIN the query that you say the
> rule should expand to,
>
>         UPDATE rtest_admin SET pname = 'jwieck'
>             WHERE rtest_person.pdesc = 'Jan Wieck'
>               AND rtest_admin.pname = rtest_person.pdesc;
>
> so there doesn't seem to be anything wrong with the rule rewriter...

    Sure.  The  parsetree generated by the rule system is exactly
    that what the parser outputs for this  query.   I  hope  some
    people  understand  my  new documentation of the rule system.
    Sometimes the lonesome rule-rider needs a partner too.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck@debis.com (Jan Wieck) #



Tue, 27 Oct 1998 10:45:30 +0100 (MET)
jwieck@debis.com (Jan Wieck)
Re: rules regression test diff (was Re: [HACKERS] Last call?)
> LinuxPPC on PowerMac:
> o PowerBook 2400c (PPC 603e) running 2.1.24 kernel
> o regresion tests are almost good except the datetime test.
>                 They were ok on 6.3.2.
> 
> MkLinux on PowerMac:
> o PowerMac 7600 (PPC 750) running MkLinux DR3
> o Same failure of the datetime test as LinuxPPC
> 
> FreeBSD:
> o FreeBSD 2.2.6-RELEASE
> o datetime testing fails (seems same phenomenon as LinuxPPC)
> o int8 testing fails (is this normal?)
> 
> Seems there's something wrong with datetime. Comment?

Yes. I have learned to never ask for regression testing reports near a
daylight savings time boundary. I assume Japan set the clock back an
hour last Sunday? The explicit tests for 'yesterday', 'today',
'tomorrow' combined with date arithmetic fail since there is an hour
offset across that boundary. In a day or two the tests will succeed.

I'm not sure why FreeBSD has trouble with int8. It of course requires
support from the compiler, and configure tries to test for it. You don't
use gcc? If not, then perhaps you could check into 64-bit integer
support on your compiler. If you do use gcc, perhaps the test is having
trouble finding the complete set of libraries. I used to have a problem
on my Linux box with that; the 64-bit int subtraction routine didn't
make it into libc, but was hidden in some machine-specific library way
down the tree.

Perhaps you and Marc can look into the FreeBSD int8 problem?

> (2) cross platform testing
> I have run the test with FreeBSD 2.2.7 and Mklinux. Seems they are
> happy to talk to each other.

This is all good. Thanks.

                     - Tom

Tue, 27 Oct 1998 15:38:44 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Last call?
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> 	o regresion tests are almost good except the datetime test.
> 		SELECT ('tomorrow'::datetime = ('yesterday'::datetime
> 		 + '2 days'::timespan)) as "True"; --> shows 'false'

I assume you ran these tests during Monday (PST)?  That's a symptom
of the daylight savings time transition issue that was just discussed
to death on pg-hackers.  Not to worry --- it's just a shortcoming
of the datetime regression test, not a bug in Postgres.

> 		SELECT ('current'::datetime = 'now'::datetime)
> 		as "True"; --> shows 'false'
> 		SELECT count(*) AS one FROM DATETIME_TBL WHERE 
> 		d1 = 'today'::datetime; --> no row selected

These two worry me more.  They don't look like they should be
subject to the DST issue, and no one else reported seeing them
fail over the weekend.  Thomas, any thoughts?

> FreeBSD:
> 	o int8 testing fails (is this normal?)

It is if the platform does not have a 64-bit integer data type.

If FreeBSD's compiler and C library do support a 64-bit int type,
then there is a problem that we ought to fix.  Most likely,
configure doesn't know the name of the type to try, or isn't trying
the right format string for sprintf/sscanf of a long long int.

			regards, tom lane

Tue, 27 Oct 1998 11:04:47 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Last call? 
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu> writes:
> Yes. I have learned to never ask for regression testing reports near a
> daylight savings time boundary. I assume Japan set the clock back an
> hour last Sunday?

Japan doesn't use DST last I heard, but it doesn't matter.  The
regression tests are run with TZ=PST8PDT, so American DST boundaries are
the windows in which the datetime test will fail, anywhere in the world.

(Given correctly installed worldwide TZ info on the local system, of
course, but I suspect you can assume that most Unix systems will have
info about the American timezones...)

			regards, tom lane

Tue, 27 Oct 1998 11:39:19 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Last call? 
> Japan doesn't use DST last I heard, but it doesn't matter.  The
> regression tests are run with TZ=PST8PDT, so American DST boundaries 
> are the windows in which the datetime test will fail, anywhere in the 
> world.

Oh. Right.

                   - Tom

Tue, 27 Oct 1998 17:00:52 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Last call?
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu> writes:

> _  NetBSD 1.3

We normally write NetBSD/i386 1.3 (or 1.3.2, the latest version).

> _  NetBSD 1.3 Sparc

I just ran the regression tests on NetBSD/sparc 1.3H, which is an
interim (current) version between 1.3.2 and 1.4.  All tests pass
except float8, which generates the following extra error messages:

barsoom:tih> diff -c expected/float8-NetBSD.out results/float8.out 
*** expected/float8-NetBSD.out  Thu Oct  8 18:12:14 1998
--- results/float8.out  Tue Oct 27 20:07:18 1998
***************
*** 213,219 ****
--- 213,221 ----
  QUERY: INSERT INTO FLOAT8_TBL(f1) VALUES ('-10e400');
  ERROR:  Bad float8 input format '-10e400'
  QUERY: INSERT INTO FLOAT8_TBL(f1) VALUES ('10e-400');
+ ERROR:  Bad float8 input format '10e-400'
  QUERY: INSERT INTO FLOAT8_TBL(f1) VALUES ('-10e-400');
+ ERROR:  Bad float8 input format '-10e-400'
  QUERY: DELETE FROM FLOAT8_TBL;
  QUERY: INSERT INTO FLOAT8_TBL(f1) VALUES ('0.0');
  QUERY: INSERT INTO FLOAT8_TBL(f1) VALUES ('-34.84');
barsoom:tih> 

> _  NetBSD 1.3 VAX

Unfortunately, I can't run the regression tests under NetBSD/vax, as
my old VAX is having stability problems at the moment.  Things were
fine with 6.3, though, and since NetBSD/i386 and NetBSD/sparc both
like 6.4BETA, it's extremely probable that NetBSD/vax will as well.

-tih
-- 
Popularity is the hallmark of mediocrity.  --Niles Crane, "Frasier"

27 Oct 1998 20:41:34 +0100
Tom Ivar Helbekkmo <tih@Hamartun.Priv.NO>
Re: Last call?
jwieck@debis.com (Jan Wieck) writes:
> Tom Lane wrote:
>> the two rows that will be updated have equal sort keys and therefore the
>> sort could legally return them in either order.  Does Postgres contain
>> its own sort algorithm for this kind of operation, or does it depend on
>> the system qsort?  System library qsorts don't normally guarantee
>> stability for equal keys.  It could be that we're looking at a byproduct
>> of some minor implementation difference between the qsorts on my machine
>> and yours.  If that's it, though, I'm surprised that I'm the only one
>> reporting a difference in the test's output.

>     Could  be  the  reason.  createfirstrun() in psort.c is using
>     qsort as a first try. Maybe we should  add  ORDER  BY  pname,
>     sysname to the queries to avoid it.

I think this is the answer then.  Some poking around in HP's patch
documentation shows that they modified their version of qsort a while
back:

	PHCO_6780:
	qsort performs very badly on sorted blocks of data
	- customer found that qsort on a file with 100,000
	randomly sorted records took seconds, whereas a file
	of 100,000 records containing large sorted blocks
	took over an hour to sort.
	qsort needed to pick an alternate pivot point when
	detecting sorted or partially sorted data in order
	to improve poor performance.

So I guess it's not so surprising if HP's qsort has slightly different
behavior on equal-keyed data than everyone else's.

Does anyone object to modifying this regression test to sort the
tuples for display?  It seems that only the one query needs to be
changed...

			regards, tom lane

Tue, 27 Oct 1998 14:49:26 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: rules regression test diff (was Re: [HACKERS] Last call?) 
> I just ran the regression tests on NetBSD/sparc 1.3H, which is an
> interim (current) version between 1.3.2 and 1.4.  All tests pass
> except float8, which generates the following extra error messages:

OK, looks good...

                - Thomas

Tue, 27 Oct 1998 21:16:45 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: Last call?
Would like to get reports running the Postgres v6.4beta on these
remaining platforms in the next couple of days:

> _  DGUX (needs a new maintainer)
> _  IRIX 6.x (Andrew?)
> _  Digital 4.0
> _  linux 2.0.x Alpha (needs some work; someone have patches?)
> _  NetBSD 1.3 VAX (Tom H's machine is down; anyone else?)
> _  SCO (Billy, have you had any luck with this?)
> _  Solaris x86 (Marc?)
> _  SunOS 4.1.4 Sparc (Tatsuo?)
> _  SVR4 4.4 m88k

TIA

                     - Tom

Wed, 28 Oct 1998 17:31:33 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Last call?
On Wed, 28 Oct 1998, Thomas G. Lockhart wrote:

> Would like to get reports running the Postgres v6.4beta on these
> remaining platforms in the next couple of days:
> 
> > _  DGUX (needs a new maintainer)
> > _  IRIX 6.x (Andrew?)
> > _  Digital 4.0
> > _  linux 2.0.x Alpha (needs some work; someone have patches?)
> > _  NetBSD 1.3 VAX (Tom H's machine is down; anyone else?)
> > _  SCO (Billy, have you had any luck with this?)
> > _  Solaris x86 (Marc?)


sorry...yes...beautiful build and regression test...

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Wed, 28 Oct 1998 15:26:22 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Last call?
Hi, 

I just compiled beta4 on Solaris 7 (also known as 2.7) using egcs 1.1.

Everything went very smoothly and regression tests were passed.

Thomas G. Lockhart writes:
 > Would like to get reports running the Postgres v6.4beta on these
 > remaining platforms in the next couple of days:
 > 
 > > _  DGUX (needs a new maintainer)
 > > _  IRIX 6.x (Andrew?)
 > > _  Digital 4.0
 > > _  linux 2.0.x Alpha (needs some work; someone have patches?)
 > > _  NetBSD 1.3 VAX (Tom H's machine is down; anyone else?)
 > > _  SCO (Billy, have you had any luck with this?)
 > > _  Solaris x86 (Marc?)
 > > _  SunOS 4.1.4 Sparc (Tatsuo?)
 > > _  SVR4 4.4 m88k
 > 
 > TIA
 > 
 >                      - Tom
 > 
 > 

MfG/Regards
--
     /====                         Siemens AG
    /    Ridderbusch        / ,   ICP CS XS QM4
   /                       /./   Heinz Nixdorf Ring
  /=== /,== ,===/  /,==,  //    33106 Paderborn, Germany
 /    //   /   /  //   / / \   Tel.: (49) 5251-8-15211
/    /     `==/\ /    / /   \ Email: ridderbusch.pad@sni.de

Since I have taken all the Gates out of my computer, it finally works!!

Wed, 28 Oct 1998 21:28:23 +0100 (MET)
Frank Ridderbusch <ridderbusch.pad@sni.de>
Re: [HACKERS] Last call?
Hi,

here are my results for building beta3 on MIPS SVR4. The regression
tests are as good as they can be. There are a couple of changes
necessary, which I've described in a little README, which I've
appended below.

Thomas G. Lockhart writes:
 > Would like to get reports running the Postgres v6.4beta on these
 > remaining platforms in the next couple of days:
 > 
 > > _  DGUX (needs a new maintainer)
 > > _  IRIX 6.x (Andrew?)
 > > _  Digital 4.0
 > > _  linux 2.0.x Alpha (needs some work; someone have patches?)
 > > _  NetBSD 1.3 VAX (Tom H's machine is down; anyone else?)
 > > _  SCO (Billy, have you had any luck with this?)
 > > _  Solaris x86 (Marc?)
 > > _  SunOS 4.1.4 Sparc (Tatsuo?)
 > > _  SVR4 4.4 m88k
 > 
 > TIA
 > 
 >                      - Tom
 > 
 > 

Readme for building Postgresql 6.4 on Siemens RM Systems
========================================================

1. Overview
===========

This README describes the necessary steps to build the Postgresql
database system on SINIX/Reliant UNIX. Reliant UNIX (previously called
SINIX) is a SVR4 variant, which runs on Siemens RM400/RM600 servers.
These servers use the MIPS R3000/R4400/R10000 family of processors.

The following description is based on SINIX-P 5.42A10 running on a
RM600-xx with 4 processors (both the machine and the operating system
are pretty old, therefore your milage my vary on newer os versions).

2. Building
===========

You can not use the GCC version for this platform (2.7.2.3 from
ftp://ftp.mch.sni.de/sni/mr/pd/gnu). But you have to install it
anyway, since the GCC cpp must be used during an intermediate step,
when header files are produced. You have to use the Siemens
C-compilations environment (I used CDS 1.1A00) for the actual
compilation process. Reason is, that the postgresql backend is build
with several 'ld -R' passes. The linker expects the symbols to be
sorted in ELF object file, which this GCC port apparently does not do.

Apart from flex, bison, you also need GNU awk. Awk is used in
genbki.sh and the expression used in the shell script appears to be
too complex for the system awk. Therefore install GNU awk and make
sure, that this version is found before the system awk (ordering of
PATH variable).

I configured with

  ./configure --with-template=svr4 \
              --prefix=/home/tools/pgsql-6.4 \
              --with-includes=/home/tools/include \
              --with-libraries=/home/tools/lib

You might be tempted to run configure with the additional argument
--with-CC='cc -W0' to activate the native C-compiler. However, when I
did this, compilation stopped with this error message.

  cc -W0 -I../../../include -I../../../backend   -I/home/tools/include
  -I../..    -c istrat.c -o istrat.o
  istrat.c    496: [error]:     2324 Undefined: 'F_OIDEQ'
       2086 c1: errors: 1, warnings: 15

After that the following changes are necessary (the changes are
explicitly listed, since the changes might not be compatible with
other SVR4 platforms, which use the same files):

o    Add -lsocket before -lnsl in Makefile.global

o    edit Makefile.port and extend LDFLAGS with -lmutex, so that it
     like this

         LDFLAGS+= -lmutex -lc /usr/ucblib/libucb.a -LD-Blargedynsym

     And add a line to enable the native C-compiler

         CUSTOM_CC = cc -W0 -O2

o    configure does not correctly reqognize the number of arguments
     for gettimeofday. Therefore 'undef' HAVE_GETTIMEOFDAY_2_ARGS.

o    These two patches are necessary to fix a compiler bug.

     In backend/utils/adt/date.c around line 179 change

       EncodeTimeSpan(tm, 0, DateStyle, buf);

     to

       EncodeTimeSpan(tm, 0.0, DateStyle, buf);

     and in backend/utils/adt/dt.c around line 349 and 359 change

       tm2datetime(&tt, 0, NULL, &dt);

     to

       tm2datetime(&tt, 0.0, NULL, &dt);

     Patches in diff -u form are appended below.

o    In src/backend/port/Makefile remove the 'strcasecmp.o'.

3. Restrictions
===============

Connecting to the backend via unix domain sockets does not work and
the 64bit data type is not supported. Otherwise the regression test
shows no remarkable differences.

4. Patches
==========

Please remove the first blank column, if you apply the patches.

 diff -u src/backend/utils/adt/date.c~ src/backend/utils/adt/date.c
 --- src/backend/utils/adt/date.c~      Wed Oct 28 20:43:42 1998
 +++ src/backend/utils/adt/date.c       Wed Oct 28 20:43:42 1998
 @@ -176,7 +176,7 @@
         else
         {
                 reltime2tm(time, tm);
 -              EncodeTimeSpan(tm, 0, DateStyle, buf);
 +              EncodeTimeSpan(tm, 0.0, DateStyle, buf);
         }

         result = palloc(strlen(buf) + 1);
 diff -u src/backend/utils/adt/dt.c~ src/backend/utils/adt/dt.c
 --- src/backend/utils/adt/dt.c~        Wed Oct 28 20:45:42 1998
 +++ src/backend/utils/adt/dt.c Wed Oct 28 20:45:42 1998
 @@ -346,7 +346,7 @@
         if (DATETIME_IS_CURRENT(dt))
         {
                 GetCurrentTime(&tt);
 -              tm2datetime(&tt, 0, NULL, &dt);
 +              tm2datetime(&tt, 0.0, NULL, &dt);
                 dt = dt2local(dt, -CTimeZone);

  #ifdef DATEDEBUG
 @@ -356,7 +356,7 @@
         else
         {                                                      /* if (DATETIME_IS_EPOCH(dt1)) */
                 GetEpochTime(&tt);
 -              tm2datetime(&tt, 0, NULL, &dt);
 +              tm2datetime(&tt, 0.0, NULL, &dt);
  #ifdef DATEDEBUG
                 printf("SetDateTime- epoch time is %f\n", dt);
  #endif

MfG/Regards
--
     /====                         Siemens AG
    /    Ridderbusch        / ,   ICP CS XS QM4
   /                       /./   Heinz Nixdorf Ring
  /=== /,== ,===/  /,==,  //    33106 Paderborn, Germany
 /    //   /   /  //   / / \   Tel.: (49) 5251-8-15211
/    /     `==/\ /    / /   \ Email: ridderbusch.pad@sni.de

Since I have taken all the Gates out of my computer, it finally works!!

Wed, 28 Oct 1998 23:07:26 +0100 (MET)
Frank Ridderbusch <ridderbusch.pad@sni.de>
Re: [HACKERS] Last call?
"Thomas G. Lockhart" wrote:
> Would like to get reports running the Postgres v6.4beta on these
> remaining platforms in the next couple of days:
> 
[...]
> > _  SCO (Billy, have you had any luck with this?)

Tom,

Which SCO?

The only port I can support at this time is SCO UnixWare 7 (I no longer have 
UnixWare 2.x loaded on my machines).  The changes I made to support that port 
should also work for UnixWare 2.x if the UDK (SCO Universal Development Kit) 
is used to build it, but this has not been tested.  Those same changes may 
also work for SCO OpenServer 5.x, if the UDK is used to build it.

I have not done any work with the univel (aka UnixWare 2.x) port for 6.4.  I can not say if it will work with 6.4.

BTW.  Once 6.4 is offically released, I will supply a binary version that should execute on SCO OpenServer 5.x, UnixWare 2.x, and UnixWare 7.x.  The ability to generate binaries that will run across all of these platforms is the main reason the UDK exists.

-- 
____       | Billy G. Allie    | Domain....: Bill.Allie@mug.org
|  /|      | 7436 Hartwell     | Compuserve: 76337,2061
|-/-|----- | Dearborn, MI 48126| MSN.......: B_G_Allie@email.msn.com
|/  |LLIE  | (313) 582-1540    | 



Wed, 28 Oct 1998 23:41:37 -0500
"Billy G. Allie" <Bill.Allie@mug.org>
Re: [HACKERS] Last call? 
> > > _  SCO (Billy, have you had any luck with this?)
> Which SCO?
> The only port I can support at this time is SCO UnixWare 7 (I no 
> longer have UnixWare 2.x loaded on my machines).

Ah, I was confused, and had thought they were more distinct. Should I
list them separately, or should I just say SCO UnixWare 7 to cover all
current SCO products?

                - Tom

Thu, 29 Oct 1998 05:17:31 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Last call?
Frank Ridderbusch <ridderbusch.pad@sni.de> writes:
> You might be tempted to run configure with the additional argument
> --with-CC='cc -W0' to activate the native C-compiler. However, when I
> did this, compilation stopped with this error message.

>   cc -W0 -I../../../include -I../../../backend   -I/home/tools/include
>   -I../..    -c istrat.c -o istrat.o
>   istrat.c    496: [error]:     2324 Undefined: 'F_OIDEQ'
>        2086 c1: errors: 1, warnings: 15

I believe that is the first symptom you'd see when configure chooses
a cpp-from-stdin technique that doesn't actually work.  We went around
on that a couple of times in the past three or four days, and eventually
changed the shell scripts so that they don't need cpp from stdin.

So, with the current sources (or BETA4 when it's out) it might work to
specify --with-CC; would you try it and let us know?

This cpp mistake may also be the root of the apparent need to have gcc
installed --- please check and see if that's still true.

> After that the following changes are necessary (the changes are
> explicitly listed, since the changes might not be compatible with
> other SVR4 platforms, which use the same files):

I think all of these config changes could be handled with a special
Makefile.port for Siemens ... is that worth adding?

> o    configure does not correctly reqognize the number of arguments
>      for gettimeofday. Therefore 'undef' HAVE_GETTIMEOFDAY_2_ARGS.

This should be fixed; can you look into it and see why configure
is getting the wrong answer?  Look at configure.in --- there is a
small test program that the script tries to compile, and if there
is no compile error then it assumes gettimeofday has two args.
A first guess is that gettimeofday is not declared in sys/time.h
on your machine.  If we add wherever it is declared then perhaps
the test will work.

> o    In src/backend/port/Makefile remove the 'strcasecmp.o'.

Likewise, this should be fixable by improving configure's test
to see whether the system has strcasecmp.

			regards, tom lane

Thu, 29 Oct 1998 10:41:06 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Last call? 
Tom Lane writes:
 > Frank Ridderbusch <ridderbusch.pad@sni.de> writes:
 > > You might be tempted to run configure with the additional argument
....
 > 
 > This cpp mistake may also be the root of the apparent need to have gcc
 > installed --- please check and see if that's still true.

Once beta4 is ready I will definitely rebuild.

 > > After that the following changes are necessary (the changes are
 > > explicitly listed, since the changes might not be compatible with
 > > other SVR4 platforms, which use the same files):
 > 
 > I think all of these config changes could be handled with a special
 > Makefile.port for Siemens ... is that worth adding?

Well I'd say, this short before your expected release date, I wouldn't 
want to shake things up by adding another special Makefile.port with
the possible iterations to get it right. And although I would like it
to be otherwise, Siemens RM system have only a great market share in
Germany. I think, if Pyramid Nile users and, I think NEC has/had a MIPS
based SVR4, would speak up, then you should add a special MIPS based
Makefile.port 

 > > o    configure does not correctly reqognize the number of arguments
 > >      for gettimeofday. Therefore 'undef' HAVE_GETTIMEOFDAY_2_ARGS.
 > 
 > This should be fixed; can you look into it and see why configure
 > is getting the wrong answer?  Look at configure.in --- there is a
 > small test program that the script tries to compile, and if there
 > is no compile error then it assumes gettimeofday has two args.
 > A first guess is that gettimeofday is not declared in sys/time.h
 > on your machine.  If we add wherever it is declared then perhaps
 > the test will work.

As I say, my system is pretty old (still R3000) and ths os is just as
old. And indeed the missing prototype in sys/time.h is the
cause. Later OS version have it defined. Therefore I really see no
need for changes here.

 > > o    In src/backend/port/Makefile remove the 'strcasecmp.o'.
 > 
 > Likewise, this should be fixable by improving configure's test
 > to see whether the system has strcasecmp.
 > 
 > 			regards, tom lane

Well, this problem is originally caused by the need to link against
/usr/ucblib/libucb.a. Without libucb.a, I'm getting three undefined
symbols

 strncasecmp                         commands/SUBSYS.o
 alloca                              bootstrap/SUBSYS.o
 strcasecmp                          commands/SUBSYS.o

alloca is ok (GCC users have it build in). But strncasecmp and
strcasecmp are defined in the same archive member in
libucb.a. Therefore I get multiple defines if I link with strcasecmp.o 
from pgsql.

Come to think of it, shouldn't configure check also for strncasecmp if 
it checks for strcasecmp? But apparently, since no one complains, it
appears, that all other platforms do have strncasecmp and shouldn't
also need strcasecmp.

I general, I would say, leave the configure process as it is, once
beta4 is available. I'm quite happy as it stand, with all the
shortcommings of my dated hard- and software.

Regards,

	Frank

Thu, 29 Oct 1998 22:41:25 +0100 (MET)
Frank Ridderbusch <ridderbusch.pad@sni.de>
Re: [HACKERS] Last call? 
Frank Ridderbusch <ridderbusch.pad@sni.de> writes:
>> Likewise, this should be fixable by improving configure's test
>> to see whether the system has strcasecmp.

> Well, this problem is originally caused by the need to link against
> /usr/ucblib/libucb.a. Without libucb.a, I'm getting three undefined
> symbols
>  strncasecmp                         commands/SUBSYS.o
>  alloca                              bootstrap/SUBSYS.o
>  strcasecmp                          commands/SUBSYS.o
> alloca is ok (GCC users have it build in). But strncasecmp and
> strcasecmp are defined in the same archive member in
> libucb.a. Therefore I get multiple defines if I link with strcasecmp.o 
> from pgsql.

OK, the reason that configure is deciding strcasecmp.o is needed is that
it has no idea you are planning to link with /usr/ucblib/libucb.a.

Rather than editing the generated makefiles after the fact, you might
have better luck if you edit the template file you plan to use *before*
running configure.  I think adding "-L/usr/ucblib -lucb" to the LIBS
line in the template file would solve this particular problem.  I think
you had mentioned needing a few other unusual libraries as well --- you
should be able to take care of all of them that way.

> Come to think of it, shouldn't configure check also for strncasecmp if 
> it checks for strcasecmp? But apparently, since no one complains, it
> appears, that all other platforms do have strncasecmp and shouldn't
> also need strcasecmp.

Basically configure is assuming that your system library provides
both or neither.  I think that's a reasonable assumption.  Of course,
if we find any actual instances of systems with just one, we may have
to change it...

			regards, tom lane

Thu, 29 Oct 1998 19:34:42 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Last call? 
Hello!

I tried postgresql.v6.4-BETA3 and found some problems especially
with FreeBSD.
The following is my environment:
	FreeBSD 2.2.7-RELEASE
	GNU Make version 3.76.1, by Richard Stallman and Roland McGrath.
	gcc version 2.7.2.1
	GNU Bison version 1.25
	flex version 2.5.4

* makefiles/Makefile.freebsd
	It should have "-Bforcearchive" option just like as
	makefiles/Makefile.bsd (and it used to be).
	Without it, I got such errors in test/regress/results/misc.out.

========================================================================
ERROR:  Can't find function reverse_name in file /usr/local/src/postgresql.v6.4-BETA3/pgsql/src/test/regress/input/../regress.so
========================================================================

	# And the original Makefile.freebsd does not seem to work
	# as you intended.  As `\' overrides `#', the last 2 lines
	# are commented out together.


* Makefile.shlib
	FreeBSD is not supported.
	So no shared library for plpgsql is installed and regression
	test for it fails.

========================================================================
gmake -C src install
gmake[3]: Entering directory `/usr/local/src/postgresql.v6.4-BETA3/pgsql/src/pl/
plpgsql/src'
/usr/bin/install -c -m 644  /usr/local/pgsql/lib/plpgsql.so
usage: install [-CcDps] [-f flags] [-g group] [-m mode] [-o owner] file1 file2
       install [-CcDps] [-f flags] [-g group] [-m mode] [-o owner] file1 ...
             fileN directory
       install -d [-g group] [-m mode] [-o owner] directory ...
gmake[3]: *** [install] Error 64
========================================================================

	And no shared libraries for libecpg, libpq++ and libpq.a
	are made and installed.


* pg_dumpall
	Pg_dumpall command outputs "create database" statement such as:
========================================================================
create database with encoding='EUC_JP' DBNAME;
========================================================================
	But it should be:
========================================================================
create database DBNAME with encoding='EUC_JP';
========================================================================


Regards

-- 
ASCII CORPORATION
Technical Center
SHIOZAKI Takehiko
<takehi-s@ascii.co.jp>

Fri, 30 Oct 1998 10:09:21 +0900 (JST)
SHIOZAKI Takehiko <takehi-s@ascii.co.jp>
v6.4-BETA3 problems with FreeBSD
On Fri, 30 Oct 1998, SHIOZAKI Takehiko wrote:

> Hello!
> 
> I tried postgresql.v6.4-BETA3 and found some problems especially
> with FreeBSD.
> The following is my environment:
> 	FreeBSD 2.2.7-RELEASE
> 	GNU Make version 3.76.1, by Richard Stallman and Roland McGrath.
> 	gcc version 2.7.2.1
> 	GNU Bison version 1.25
> 	flex version 2.5.4
> 
> * makefiles/Makefile.freebsd
> 	It should have "-Bforcearchive" option just like as
> 	makefiles/Makefile.bsd (and it used to be).
> 	Without it, I got such errors in test/regress/results/misc.out.

	Under ELF, this does not work, so adding it back in "as is" isn't
an option.  Let me see if I can come up with something here that works...

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Thu, 29 Oct 1998 22:23:53 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD
>> I tried postgresql.v6.4-BETA3 and found some problems especially
>> with FreeBSD.
>> The following is my environment:
>> 	FreeBSD 2.2.7-RELEASE
>> 	GNU Make version 3.76.1, by Richard Stallman and Roland McGrath.
>> 	gcc version 2.7.2.1
>> 	GNU Bison version 1.25
>> 	flex version 2.5.4
>> 
>> * makefiles/Makefile.freebsd
>> 	It should have "-Bforcearchive" option just like as
>> 	makefiles/Makefile.bsd (and it used to be).
>> 	Without it, I got such errors in test/regress/results/misc.out.
>
>	Under ELF, this does not work, so adding it back in "as is" isn't
>an option.  Let me see if I can come up with something here that works...

Maybe we should have separate makefile/Makefiles.whatever for FreeBSD
3.x and 2.2.x ?
--
Tatsuo Ishii
t-ishii@sra.co.jp

Fri, 30 Oct 1998 12:13:36 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD 
On Fri, 30 Oct 1998, Tatsuo Ishii wrote:

> >> I tried postgresql.v6.4-BETA3 and found some problems especially
> >> with FreeBSD.
> >> The following is my environment:
> >> 	FreeBSD 2.2.7-RELEASE
> >> 	GNU Make version 3.76.1, by Richard Stallman and Roland McGrath.
> >> 	gcc version 2.7.2.1
> >> 	GNU Bison version 1.25
> >> 	flex version 2.5.4
> >> 
> >> * makefiles/Makefile.freebsd
> >> 	It should have "-Bforcearchive" option just like as
> >> 	makefiles/Makefile.bsd (and it used to be).
> >> 	Without it, I got such errors in test/regress/results/misc.out.
> >
> >	Under ELF, this does not work, so adding it back in "as is" isn't
> >an option.  Let me see if I can come up with something here that works...
> 
> Maybe we should have separate makefile/Makefiles.whatever for FreeBSD
> 3.x and 2.2.x ?

	Ewww...

	Waht does tools/ccsym show for you?  I'm guessing, but you don't
have a __ELF__ defined, correct?

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Thu, 29 Oct 1998 23:52:50 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD 
>> Maybe we should have separate makefile/Makefiles.whatever for FreeBSD
>> 3.x and 2.2.x ?
>
>	Ewww...
>
>	Waht does tools/ccsym show for you?  I'm guessing, but you don't
>have a __ELF__ defined, correct?

Right.
--
Tatsuo Ishii
t-ishii@sra.co.jp

Fri, 30 Oct 1998 12:57:34 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD 
On Fri, 30 Oct 1998, Tatsuo Ishii wrote:

> >> Maybe we should have separate makefile/Makefiles.whatever for FreeBSD
> >> 3.x and 2.2.x ?
> >
> >	Ewww...
> >
> >	Waht does tools/ccsym show for you?  I'm guessing, but you don't
> >have a __ELF__ defined, correct?
> 
> Right.

	And you don't have a /usr/include/elf.h, correct?  almost fixed,
just want to confirm my assumptions are correct, that is all :)

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Fri, 30 Oct 1998 00:03:29 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD 
>	And you don't have a /usr/include/elf.h, correct?  almost fixed,
>just want to confirm my assumptions are correct, that is all :)

We have /usr/include/elf.h (FreeBSD 2.2.6-RELEASE). What about yours,
Takehiko?
--
Tatsuo Ishii
t-ishii@sra.co.jp

Fri, 30 Oct 1998 13:12:02 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD 
On Fri, 30 Oct 1998, Tatsuo Ishii wrote:

> >	And you don't have a /usr/include/elf.h, correct?  almost fixed,
> >just want to confirm my assumptions are correct, that is all :)
> 
> We have /usr/include/elf.h (FreeBSD 2.2.6-RELEASE). What about yours,

	Oh, shit, there goes that idea :(  Hrmmm...back to the drawing
board :)

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Fri, 30 Oct 1998 00:17:32 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD 
[On Oct 30, Tatsuo Ishii <t-ishii@sra.co.jp> writes:]
>
>We have /usr/include/elf.h (FreeBSD 2.2.6-RELEASE). What about yours,
>Takehiko?

I also have /usr/include/elf.h in FreeBSD 2.2.7-RELEASE with RCS header:
	$Id: elf.h,v 1.1.2.1 1998/01/27 16:23:39 jdp Exp $
And I don't have it in FreeBSD 2.2.5-RELEASE.

-- 
$B$?$@$7!";d$O<*I!0v9"$,<e$$$N$G!"2q5D$O6X1l$K$7$F$/$@$5$$!#(B
--
$B1v:j(B $B5#I'(B(SHIOZAKI Takehiko)	<takehi-s@ascii.co.jp>
Technical Center,	ASCII CORPORATION

Fri, 30 Oct 1998 13:19:08 +0900 (JST)
SHIOZAKI Takehiko <takehi-s@ascii.co.jp>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD
SHIOZAKI Takehiko <takehi-s@ascii.co.jp> writes:
> * Makefile.shlib
> 	FreeBSD is not supported.

Can you provide an appropriate entry to add to Makefile.shlib for
FreeBSD?  It's probably much like one of the existing entries...

> gmake[3]: Entering directory `/usr/local/src/postgresql.v6.4-BETA3/pgsql/src/pl/
> plpgsql/src'
> /usr/bin/install -c -m 644  /usr/local/pgsql/lib/plpgsql.so
> usage: install [-CcDps] [-f flags] [-g group] [-m mode] [-o owner] file1 file2
>        install [-CcDps] [-f flags] [-g group] [-m mode] [-o owner] file1 ...
>              fileN directory
>        install -d [-g group] [-m mode] [-o owner] directory ...
> gmake[3]: *** [install] Error 64

Oops.  The plpgsql Makefile should exit a little more gracefully on
a system not supported by Makefile.shlib.  I'll do something about
that tomorrow.

			regards, tom lane

Thu, 29 Oct 1998 23:41:41 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD 
The Hermit Hacker <scrappy@hub.org> writes:
> On Fri, 30 Oct 1998, SHIOZAKI Takehiko wrote:
>> * makefiles/Makefile.freebsd
>> It should have "-Bforcearchive" option just like as
>> makefiles/Makefile.bsd (and it used to be).

> 	Under ELF, this does not work, so adding it back in "as is" isn't
> an option.  Let me see if I can come up with something here that works...

Can we teach configure how to tell the difference between
ELF and pre-ELF BSD?

			regards, tom lane

Thu, 29 Oct 1998 23:44:01 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD 
On Thu, 29 Oct 1998, Tom Lane wrote:

> The Hermit Hacker <scrappy@hub.org> writes:
> > On Fri, 30 Oct 1998, SHIOZAKI Takehiko wrote:
> >> * makefiles/Makefile.freebsd
> >> It should have "-Bforcearchive" option just like as
> >> makefiles/Makefile.bsd (and it used to be).
> 
> > 	Under ELF, this does not work, so adding it back in "as is" isn't
> > an option.  Let me see if I can come up with something here that works...
> 
> Can we teach configure how to tell the difference between
> ELF and pre-ELF BSD?

	Working on it now...:)

	Its not a "nice fix", but it will get us through the release.

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Fri, 30 Oct 1998 00:56:05 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD 
> > * makefiles/Makefile.freebsd
> > 	It should have "-Bforcearchive" option just like as
> > 	makefiles/Makefile.bsd (and it used to be).
> > 	Without it, I got such errors in test/regress/results/misc.out.
> 
> 	Under ELF, this does not work, so adding it back in "as is" isn't
> an option.  Let me see if I can come up with something here that works...

Take a look at bsd 4.0.  I had to do the same thing.  I defined a new
template file, and changed the DLSUFFIX, and checked that in
Makefile.shlib.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 30 Oct 1998 10:59:38 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD
[On Oct 29, Tom Lane <tgl@sss.pgh.pa.us> writes:]
>
>Can you provide an appropriate entry to add to Makefile.shlib for
>FreeBSD?  It's probably much like one of the existing entries...

O.K.
I tried snapshot(Oct30) and made some patches.
# I think that it is confused to manage both Makefile.shlib and
# makefiles/Makefile.*, don't you?

* configure
	Now FreeBSD 2.X is not supported..., so I added its entry.
	If ELF_SYSTEM is set, gmake treat it defined even though
	it is "false".  So nothing should be set to use "ifdef".
	BSD_SHLIB etc. may have same problems.

* Makefile.shlib
	As you said, FreeBSD entry is much like BSD's.
	I only added ELF_SYSTEM code.

* makefiles/Makefile.freebsd
	Ifdef/else/endif can not be indented with TABs.

========================================================================
*** configure.in.orig	Fri Oct 30 17:00:20 1998
--- configure.in	Sat Oct 31 10:07:38 1998
***************
*** 17,23 ****
     linux*) os=linux need_tas=no ;;
      bsdi*) os=bsdi need_tas=no ;;
   freebsd3*) os=freebsd need_tas=no elf=yes ;;
!  freebsd1*) os=freebsd need_tas=no ;;
    netbsd*|openbsd*) os=bsd need_tas=no ;;
      dgux*) os=dgux need_tas=no ;;
       aix*) os=aix need_tas=no ;;
--- 17,23 ----
     linux*) os=linux need_tas=no ;;
      bsdi*) os=bsdi need_tas=no ;;
   freebsd3*) os=freebsd need_tas=no elf=yes ;;
!  freebsd[12]*) os=freebsd need_tas=no ;;
    netbsd*|openbsd*) os=bsd need_tas=no ;;
      dgux*) os=dgux need_tas=no ;;
       aix*) os=aix need_tas=no ;;
***************
*** 52,58 ****
  then
  	ELF_SYS=true
  else
! 	ELF_SYS=false
  fi
  
  if test "X$need_tas" = "Xyes"
--- 52,58 ----
  then
  	ELF_SYS=true
  else
! 	ELF_SYS=
  fi
  
  if test "X$need_tas" = "Xyes"
*** configure.orig	Fri Oct 30 17:00:20 1998
--- configure	Sat Oct 31 10:07:00 1998
***************
*** 617,623 ****
     linux*) os=linux need_tas=no ;;
      bsdi*) os=bsdi need_tas=no ;;
   freebsd3*) os=freebsd need_tas=no elf=yes ;;
!  freebsd1*) os=freebsd need_tas=no ;;
    netbsd*|openbsd*) os=bsd need_tas=no ;;
      dgux*) os=dgux need_tas=no ;;
       aix*) os=aix need_tas=no ;;
--- 617,623 ----
     linux*) os=linux need_tas=no ;;
      bsdi*) os=bsdi need_tas=no ;;
   freebsd3*) os=freebsd need_tas=no elf=yes ;;
!  freebsd[12]*) os=freebsd need_tas=no ;;
    netbsd*|openbsd*) os=bsd need_tas=no ;;
      dgux*) os=dgux need_tas=no ;;
       aix*) os=aix need_tas=no ;;
***************
*** 652,658 ****
  then
  	ELF_SYS=true
  else
! 	ELF_SYS=false
  fi
  
  if test "X$need_tas" = "Xyes"
--- 652,658 ----
  then
  	ELF_SYS=true
  else
! 	ELF_SYS=
  fi
  
  if test "X$need_tas" = "Xyes"
*** Makefile.shlib.orig	Sat Oct 31 10:12:32 1998
--- Makefile.shlib	Sat Oct 31 10:14:47 1998
***************
*** 56,61 ****
--- 56,74 ----
  # Makefile.global (or really Makefile.port) to supply DLSUFFIX and other
  # symbols.
  
+ ifeq ($(PORTNAME), freebsd)
+   ifdef BSD_SHLIB
+     install-shlib-dep	:= install-shlib
+     shlib				:= lib$(NAME)$(DLSUFFIX).$(SO_MAJOR_VERSION).$(SO_MINOR_VERSION)
+     ifdef ELF_SYSTEM
+       LDFLAGS_SL			:= -x -Bshareable
+     else
+       LDFLAGS_SL			:= -x -Bshareable -Bforcearchive
+     endif
+     CFLAGS				+= $(CFLAGS_SL)
+   endif
+ endif
+ 
  ifeq ($(PORTNAME), bsd)
    ifdef BSD_SHLIB
      install-shlib-dep	:= install-shlib
*** makefiles/Makefile.freebsd.orig	Sat Oct 31 09:13:20 1998
--- makefiles/Makefile.freebsd	Sat Oct 31 09:13:45 1998
***************
*** 5,13 ****
  	@${AR} cq $@.pic `lorder $<.obj | tsort`
  	${RANLIB} $@.pic
  	@rm -f $@
! 	ifdef ELF_SYSTEM
! 		$(LD) -x -Bshareable -o $@ $@.pic
! 	else
! 	        $(LD) -x -Bshareable -Bforcearchive -o $@ $@.pic
! 	endif
  
--- 5,13 ----
  	@${AR} cq $@.pic `lorder $<.obj | tsort`
  	${RANLIB} $@.pic
  	@rm -f $@
! ifdef ELF_SYSTEM
! 	$(LD) -x -Bshareable -o $@ $@.pic
! else
! 	$(LD) -x -Bshareable -Bforcearchive -o $@ $@.pic
! endif
  
========================================================================

-- 
ASCII CORPORATION
Technical Center
SHIOZAKI Takehiko
<takehi-s@ascii.co.jp>

Sat, 31 Oct 1998 11:21:13 +0900 (JST)
SHIOZAKI Takehiko <takehi-s@ascii.co.jp>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD
On Sat, 31 Oct 1998, SHIOZAKI Takehiko wrote:

> [On Oct 29, Tom Lane <tgl@sss.pgh.pa.us> writes:]
> >
> >Can you provide an appropriate entry to add to Makefile.shlib for
> >FreeBSD?  It's probably much like one of the existing entries...
> 
> O.K.
> I tried snapshot(Oct30) and made some patches.
> # I think that it is confused to manage both Makefile.shlib and
> # makefiles/Makefile.*, don't you?

	Something ot look at for v6.5 .. right now, it works as is :)

> * configure
> 	Now FreeBSD 2.X is not supported..., so I added its entry.
> 	If ELF_SYSTEM is set, gmake treat it defined even though
> 	it is "false".  So nothing should be set to use "ifdef".
> 	BSD_SHLIB etc. may have same problems.

	My oops, actually...I meant to put 'freebsd2*', not 'freebsd1*' :)
I put in yours though, anyway...


Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Sat, 31 Oct 1998 00:06:37 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD
[On Oct 31, The Hermit Hacker <scrappy@hub.org> writes:]
>
>	My oops, actually...I meant to put 'freebsd2*', not 'freebsd1*' :)
>I put in yours though, anyway...

BETA5 stll needs the following small patch for FreeBSD 2.X.
# And my patch for nbtsearch.c is not applied yet.  Was it wrong?

========================================================================
*** configure.orig	Fri Oct 30 17:00:20 1998
--- configure	Sat Oct 31 10:07:00 1998
***************
*** 617,623 ****
     linux*) os=linux need_tas=no ;;
      bsdi*) os=bsdi need_tas=no ;;
   freebsd3*) os=freebsd need_tas=no elf=yes ;;
!  freebsd12*) os=freebsd need_tas=no ;;
    netbsd*|openbsd*) os=bsd need_tas=no ;;
      dgux*) os=dgux need_tas=no ;;
       aix*) os=aix need_tas=no ;;
--- 617,623 ----
     linux*) os=linux need_tas=no ;;
      bsdi*) os=bsdi need_tas=no ;;
   freebsd3*) os=freebsd need_tas=no elf=yes ;;
!  freebsd[12]*) os=freebsd need_tas=no ;;
    netbsd*|openbsd*) os=bsd need_tas=no ;;
      dgux*) os=dgux need_tas=no ;;
       aix*) os=aix need_tas=no ;;
*** makefiles/Makefile.freebsd.orig	Mon Nov  2 11:38:01 1998
--- makefiles/Makefile.freebsd	Mon Nov  2 11:38:08 1998
***************
*** 8,13 ****
  ifdef ELF_SYSTEM
  	$(LD) -x -Bshareable -o $@ $@.pic
  else
!         $(LD) -x -Bshareable -Bforcearchive -o $@ $@.pic
  endif
  
--- 8,13 ----
  ifdef ELF_SYSTEM
  	$(LD) -x -Bshareable -o $@ $@.pic
  else
! 	$(LD) -x -Bshareable -Bforcearchive -o $@ $@.pic
  endif
  
========================================================================

-- 
ASCII CORPORATION
Technical Center
SHIOZAKI Takehiko
<takehi-s@ascii.co.jp>

Mon, 2 Nov 1998 14:18:12 +0900 (JST)
SHIOZAKI Takehiko <takehi-s@ascii.co.jp>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD
On Mon, 2 Nov 1998, SHIOZAKI Takehiko wrote:

> [On Oct 31, The Hermit Hacker <scrappy@hub.org> writes:]
> >
> >	My oops, actually...I meant to put 'freebsd2*', not 'freebsd1*' :)
> >I put in yours though, anyway...
> 
> BETA5 stll needs the following small patch for FreeBSD 2.X.

Modifying configure directly isn't accetable, as it will change again once
autoconf is run.  Should have been written as:

	freebsd1*|freebsd2*) ... :(

Second patch applied...

> # And my patch for nbtsearch.c is not applied yet.  Was it wrong?

	Did I miss something?

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Mon, 2 Nov 1998 01:31:15 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD
[On Nov  2, The Hermit Hacker <scrappy@hub.org> writes:]
>
>> # And my patch for nbtsearch.c is not applied yet.  Was it wrong?
>
>	Did I miss something?

Please make sure whether this is correct.
http://www.PostgreSQL.ORG/mhonarc/pgsql-hackers/1998-10/msg01135.html

-- 
ASCII CORPORATION
Technical Center
SHIOZAKI Takehiko
<takehi-s@ascii.co.jp>

Mon, 2 Nov 1998 14:36:46 +0900 (JST)
SHIOZAKI Takehiko <takehi-s@ascii.co.jp>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD
On Mon, 2 Nov 1998, SHIOZAKI Takehiko wrote:

> [On Nov  2, The Hermit Hacker <scrappy@hub.org> writes:]
> >
> >> # And my patch for nbtsearch.c is not applied yet.  Was it wrong?
> >
> >	Did I miss something?
> 
> Please make sure whether this is correct.
> http://www.PostgreSQL.ORG/mhonarc/pgsql-hackers/1998-10/msg01135.html

	Bruce, can you check that one.  Basically, we try and pass:

	rel->rd_rel->relname

	instead of

	rel->rd_rel->relname.data

	around line 217 (elog(...)) in nbtsearch.c ...

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Mon, 2 Nov 1998 01:56:41 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD
> On Mon, 2 Nov 1998, SHIOZAKI Takehiko wrote:
> 
> > [On Nov  2, The Hermit Hacker <scrappy@hub.org> writes:]
> > >
> > >> # And my patch for nbtsearch.c is not applied yet.  Was it wrong?
> > >
> > >	Did I miss something?
> > 
> > Please make sure whether this is correct.
> > http://www.PostgreSQL.ORG/mhonarc/pgsql-hackers/1998-10/msg01135.html
> 
> 	Bruce, can you check that one.  Basically, we try and pass:
> 
> 	rel->rd_rel->relname
> 
> 	instead of
> 
> 	rel->rd_rel->relname.data
> 
> 	around line 217 (elog(...)) in nbtsearch.c ...

Oh, I see the problem now.  Originally, I thought it was just a question
of style, but now I see that the whole structure is being passed to
elog(), rather than the character pointer.

Patch applied.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 2 Nov 1998 10:27:14 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] v6.4-BETA3 problems with FreeBSD
RedHat 5.1 i386 with PostgreSQL v6.4 final version

./configure --with-tcl

after installing libpgtcl , cannot run pgaccess because library crypt
hasn't been linked into libpgtcl.so

must manually add -lcrypt in Makefile into src/interfaces/libpgtcl

Also, PgAccess version that has been included in 6.4 final version is
0.90 , but current stable version is 0.91
(ftp://ftp.flex.ro/pub/pgaccess/pgaccess-0.91.tar.gz)


Constantin Teodorescu
FLEX Consulting Braila, ROMANIA

Thu, 05 Nov 1998 16:48:52 +0000
Constantin Teodorescu <teo@flex.ro>
crypt not included when compiling libpgtcl !!!!!!!
> Also, PgAccess version that has been included in 6.4 final version is
> 0.90 , but current stable version is 0.91
> (ftp://ftp.flex.ro/pub/pgaccess/pgaccess-0.91.tar.gz)

Yes, it missed the code freeze cutoff and the bug fix cutoff. Next
release try to get the new version announced at least a couple of weeks
before the nominal release date, and a month before would be even
better. Should be in v6.4.1 but I don't think anyone has committed it
yet.

                   - Tom

Thu, 05 Nov 1998 17:29:59 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!!
Constantin Teodorescu <teo@flex.ro> writes:
> RedHat 5.1 i386 with PostgreSQL v6.4 final version
> after installing libpgtcl , cannot run pgaccess because library crypt
> hasn't been linked into libpgtcl.so
> must manually add -lcrypt in Makefile into src/interfaces/libpgtcl

I don't think that's the right approach.  libpgtcl doesn't depend on
libcrypt.  libpq does, and that's where we need to fix this, or else
we'll see the same problem with any other shared library that depends
on libpq.

Would you check whether it works to add the following patch to
src/interfaces/libpq's Makefile, and then build libpgtcl *without*
a reference to crypt?  (I can't test it here since crypt is part of
libc on my machine...)


*** Makefile.in~	Thu Nov  5 18:08:26 1998
--- Makefile.in	Thu Nov  5 18:11:43 1998
***************
*** 34,39 ****
--- 34,43 ----
  OBJS+= common.o wchar.o conv.o
  endif
  
+ # If crypt is a separate library, rather than part of libc,
+ # make sure it gets included in shared libpq.
+ SHLIB_LINK= $(findstring -lcrypt,$(LIBS))
+ 
  # Shared library stuff, also default 'all' target
  include $(SRCDIR)/Makefile.shlib
  


			regards, tom lane

Thu, 05 Nov 1998 18:16:27 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!! 
On Thu, 5 Nov 1998, Thomas G. Lockhart wrote:

> > Also, PgAccess version that has been included in 6.4 final version is
> > 0.90 , but current stable version is 0.91
> > (ftp://ftp.flex.ro/pub/pgaccess/pgaccess-0.91.tar.gz)
> 
> Yes, it missed the code freeze cutoff and the bug fix cutoff. Next
> release try to get the new version announced at least a couple of weeks
> before the nominal release date, and a month before would be even
> better. Should be in v6.4.1 but I don't think anyone has committed it
> yet.

	Working on it right now...sorry Constantin :(

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Thu, 5 Nov 1998 23:14:01 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!!
> On Thu, 5 Nov 1998, Thomas G. Lockhart wrote:
> 
> > > Also, PgAccess version that has been included in 6.4 final version is
> > > 0.90 , but current stable version is 0.91
> > > (ftp://ftp.flex.ro/pub/pgaccess/pgaccess-0.91.tar.gz)
> > 
> > Yes, it missed the code freeze cutoff and the bug fix cutoff. Next
> > release try to get the new version announced at least a couple of weeks
> > before the nominal release date, and a month before would be even
> > better. Should be in v6.4.1 but I don't think anyone has committed it
> > yet.
> 
> 	Working on it right now...sorry Constantin :(

I did pull down 0.90 about a month ago, and never went back to see if a
newer one existed.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 5 Nov 1998 23:37:14 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [INTERFACES] crypt not included when compiling
	libpgtcl !!!!!!!
Tom Lane wrote:
> 
> I don't think that's the right approach.  libpgtcl doesn't depend on
> libcrypt.  libpq does, and that's where we need to fix this, or else
> we'll see the same problem with any other shared library that depends
> on libpq.
> 
> Would you check whether it works to add the following patch to
> src/interfaces/libpq's Makefile, and then build libpgtcl *without*
> a reference to crypt?  (I can't test it here since crypt is part of
> libc on my machine...)

Completely delete postgresql from my machine and any trace of libpq and
libpgtcl
Untarred the 6.4 final distribution
apply the patch
compiled from scratch
installed

Got :

[teo@teo teo]$ Error in startup script: couldn't load file
"libpgtcl.so": /usr/local/pgsql/lib/libpgtcl.so: undefined symbol: crypt
    while executing
"load libpgtcl.so"
    (procedure "main" line 3)
    invoked from within
"main $argc $argv"
    (file "pgaccess.tcl" line 4813) 


When modify the Makefile from libpgtcl adding -lcrypt , recompile,
install . Everything it's ok.




> 
> *** Makefile.in~        Thu Nov  5 18:08:26 1998
> --- Makefile.in Thu Nov  5 18:11:43 1998
> ***************
> *** 34,39 ****
> --- 34,43 ----
>   OBJS+= common.o wchar.o conv.o
>   endif
> 
> + # If crypt is a separate library, rather than part of libc,
> + # make sure it gets included in shared libpq.
> + SHLIB_LINK= $(findstring -lcrypt,$(LIBS))
> +
>   # Shared library stuff, also default 'all' target
>   include $(SRCDIR)/Makefile.shlib
> 
> 
>                         regards, tom lane

-- 
Constantin Teodorescu
FLEX Consulting Braila, ROMANIA

Fri, 06 Nov 1998 08:18:57 +0000
Constantin Teodorescu <teo@flex.ro>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!!
Constantin Teodorescu <teo@flex.ro> writes:
> Tom Lane wrote:
>> Would you check whether it works to add the following patch to
>> src/interfaces/libpq's Makefile, and then build libpgtcl *without*
>> a reference to crypt?  (I can't test it here since crypt is part of
>> libc on my machine...)
>
> [ Still doesn't work ]

Odd.  Perhaps your system doesn't support libraries referring to other
shared libraries?  (Seems unlikely, but...)  In that case you'd be
getting the static libpq.a bound into libpgtcl.so, minus crypt of
course, and then this would happen.

If that's what's happening you should easily be able to tell from the
sizes of libpgtcl.so and libpq.so.  On my machine, libpgtcl is actually
smaller than libpq (about half as big, in fact).  If libpq is getting
bound into libpgtcl.so then libpgtcl.so would have to be bigger than
libpq.so.  Which one is bigger on your machine?

Also, do you have any sort of utility that shows directly what a
program's shared library requirements are?  For example, on HPUX
I can do

$ chatr /usr/local/pgsql/bin/pgtclsh
/usr/local/pgsql/bin/pgtclsh:
         shared executable
         shared library dynamic path search:
             SHLIB_PATH     disabled  second
             embedded path  enabled   first  /usr/local/pgsql/lib
         shared library list:
             static    pgtclsh
             dynamic   ../../interfaces/libpgtcl/libpgtcl.sl
             dynamic   ../../interfaces/libpq/libpq.sl
             dynamic   /usr/lib/libdld.sl
             dynamic   /lib/libcurses.sl
             dynamic   /lib/libc.sl
         shared library binding:
             deferred

$ chatr /usr/local/pgsql/lib/libpgtcl.sl
/usr/local/pgsql/lib/libpgtcl.sl:
         shared library
         shared library list:
             dynamic   ../libpq/libpq.sl


On Linux I think the corresponding command is "ldd".

			regards, tom lane

Fri, 06 Nov 1998 10:27:03 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!! 
Yes. libpgtcl is bigger than libpq

But ldd shows that libpq is statically linked :-( . Maybe this is the
problem ?

[root@teo libpgtcl]# ldd libpgtcl.so.2.0
        libcrypt.so.1 => /lib/libcrypt.so.1 (0x40014000)
        libc.so.6 => /lib/libc.so.6 (0x40041000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00000000)

[root@teo libpgtcl]# ldd ../libpq/libpq.so.2.0
        statically linked

[root@teo libpgtcl]# vdir ../libpq/libpq.so.2.0
-rwxr-xr-x   1 root     root        50014 Nov  6 10:08
../libpq/libpq.so.2.0

[root@teo libpgtcl]# vdir libpgtcl.so.2.0
-rwxr-xr-x   1 root     root        64725 Nov  6 10:19 libpgtcl.so.2.0  



-- 
Constantin Teodorescu
FLEX Consulting Braila, ROMANIA

Fri, 06 Nov 1998 15:49:59 +0000
Constantin Teodorescu <teo@flex.ro>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!!
Constantin Teodorescu <teo@flex.ro> writes:
> [root@teo libpgtcl]# ldd libpgtcl.so.2.0
>         libcrypt.so.1 => /lib/libcrypt.so.1 (0x40014000)
>         libc.so.6 => /lib/libc.so.6 (0x40041000)
>         /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00000000)

> [root@teo libpgtcl]# ldd ../libpq/libpq.so.2.0
>         statically linked

Well, that's pretty interesting.  Evidently you were able to persuade
the linker to dynamically link libpgtcl.so to libcrypt.so, but the patch
I suggested did *not* persuade the linker to dynamically link libpq.so
to libcrypt.so.  Just exactly what change did you make in libpgtcl's
Makefile, anyway?  I assumed it was simply adding -lcrypt, but now I
am not so sure.

It's also curious that libpq.so is not showing any dynamic dependency
on libc.  I think we must have the wrong linker options for libpq.

> [root@teo libpgtcl]# vdir ../libpq/libpq.so.2.0
> -rwxr-xr-x   1 root     root        50014 Nov  6 10:08
> ../libpq/libpq.so.2.0

> [root@teo libpgtcl]# vdir libpgtcl.so.2.0
> -rwxr-xr-x   1 root     root        64725 Nov  6 10:19 libpgtcl.so.2.0  

And we also need to figure out why the linker is including libpq.a into
libpgtcl.so, instead of creating a dynamic link from libpgtcl.so to
libpq.so like it did for libcrypt and libc.

(BTW, do you have a libcrypt.a in /lib, or just libcrypt.so?)

It seems clear from your ldd results that your machine is capable of
doing the right thing, but we aren't invoking the linker with the right
options.  Where I want to get to is:

	libpgtcl.so: dynamic dependency on libpq.so (and libc of course)

	libpq.so: dynamic dependency on libcrypt.so (and libc of course)

It might be worth extracting the part of the "make all" log that shows
what commands are being used to build each of these libraries.

			regards, tom lane

Fri, 06 Nov 1998 13:45:13 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!! 
Tom Lane wrote:
  
  >It's also curious that libpq.so is not showing any dynamic dependency
  >on libc.  I think we must have the wrong linker options for libpq.

To get the shared library dependency on libc you need to add -lc at the 
end of the link options.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Let all that you do be done in love."   
                                      1 Corinthians 16:14 



Fri, 06 Nov 1998 22:30:01 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!! 
Tom Lane wrote:
> 
> Constantin Teodorescu <teo@flex.ro> writes:
> > [root@teo libpgtcl]# ldd libpgtcl.so.2.0
> >         libcrypt.so.1 => /lib/libcrypt.so.1 (0x40014000)
> >         libc.so.6 => /lib/libc.so.6 (0x40041000)
> >         /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00000000)
> 
> > [root@teo libpgtcl]# ldd ../libpq/libpq.so.2.0
> >         statically linked
> 
> Well, that's pretty interesting.  Evidently you were able to persuade
> the linker to dynamically link libpgtcl.so to libcrypt.so, but the patch
> I suggested did *not* persuade the linker to dynamically link libpq.so
> to libcrypt.so.

The patch that you send me I have applied ONLY to Makefile.in from
src/interfaces/libpgtcl directory.
Should I apply it to Makefile.in from src/interfaces/libpq directory ?


>  Just exactly what change did you make in libpgtcl's
> Makefile, anyway?  I assumed it was simply adding -lcrypt, but now I
> am not so sure.

Yes. Only add -lcrypt


> And we also need to figure out why the linker is including libpq.a into
> libpgtcl.so, instead of creating a dynamic link from libpgtcl.so to
> libpq.so like it did for libcrypt and libc.
> 
> (BTW, do you have a libcrypt.a in /lib, or just libcrypt.so?)

in /usr/lib libcrypt.a and libcrypt.so
in /lib/libcrypt.so

-- 
Constantin Teodorescu
FLEX Consulting Braila, ROMANIA

Sat, 07 Nov 1998 08:20:25 +0000
Constantin Teodorescu <teo@flex.ro>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!!
Constantin Teodorescu <teo@flex.ro> writes:
> The patch that you send me I have applied ONLY to Makefile.in from
> src/interfaces/libpgtcl directory.
> Should I apply it to Makefile.in from src/interfaces/libpq directory ?

No no, that change was intended to be applied to libpq's Makefile *not*
libpgtcl's.  Sorry if I wasn't clear.

There's also the question of why the linker is making a dynamic
dependency on libc for libpgtcl but not libpq.  Oliver suggested
that -lc is needed as well as -lcrypt in the link step, but that
doesn't seem to explain all the facts --- libpgtcl doesn't have
-lc, and it seems to be getting built right.  Still, you might try
adding -lc to SHLIB_LINK in libpq's Makefile, and see if that
changes the ldd result.

			regards, tom lane

Sat, 07 Nov 1998 10:45:10 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!! 
Sferacarta Software <sferac@bo.nettuno.it> el dï¿½a Mon, 9 Nov 1998 14:33:06 
+0100, escribiï¿½:

>Hello Radhakrishnan,
>
>lunedï¿½, 9 novembre 98, you wrote:
>
>
>RCV> how can i limit the number of rows obtained from a select statement
>RCV> in postgreSQL to say, 10 rows while the select condition  actually
>RCV> matches more than that. in oracle we can use the ROW_NUM variable
>RCV> for this purpose but now i met such an issue with postgreSQL
>
>On v6.4 you can specify a limit for queries as:
>
>set QUERY_LIMIT TO '10';
>To have only the first 10 rows from a select;

this will limit _all_ the querys, right ?
wich is not very flexible.

I can't do something like

select * from news order by news_date limit 10

?

Sergio


Sat, 7 Nov 1998 13:39:27 -0300
Sergio <ser@perio.unlp.edu.ar>
Re: [INTERFACES] limiting the rows selected in postgresql

how can i limit the number of rows obtained from a select statement
in postgreSQL to say, 10 rows while the select condition  actually
matches more than that. in oracle we can use the ROW_NUM variable
for this purpose but now i met such an issue with postgreSQL

thanks in advance

CV Radhakrishnan

http://www.river-valley.com


Mon, 9 Nov 1998 18:22:58 +0530 (IST)
RADHAKRISHNAN C V <cvr@md2.vsnl.net.in>
limiting the rows selected in postgresql
Hello Radhakrishnan,

lunedï¿½, 9 novembre 98, you wrote:


RCV> how can i limit the number of rows obtained from a select statement
RCV> in postgreSQL to say, 10 rows while the select condition  actually
RCV> matches more than that. in oracle we can use the ROW_NUM variable
RCV> for this purpose but now i met such an issue with postgreSQL

On v6.4 you can specify a limit for queries as:

set QUERY_LIMIT TO '10';
To have only the first 10 rows from a select;

-Jose'-



Mon, 9 Nov 1998 14:33:06 +0100
Sferacarta Software <sferac@bo.nettuno.it>
Re: [INTERFACES] limiting the rows selected in postgresql
On Sat, 7 Nov 1998, Sergio wrote:

> Date: Sat, 7 Nov 1998 13:39:27 -0300
> From: Sergio <ser@perio.unlp.edu.ar>
> To: pgsql-interfaces@postgreSQL.org
> Subject: Re: [INTERFACES] limiting the rows selected in postgresql
> 
> Sferacarta Software <sferac@bo.nettuno.it> el dО©╫a Mon, 9 Nov 1998 14:33:06 
> +0100, escribiО©╫:
> 
> >Hello Radhakrishnan,
> >
> >lunedО©╫, 9 novembre 98, you wrote:
> >
> >
> >RCV> how can i limit the number of rows obtained from a select statement
> >RCV> in postgreSQL to say, 10 rows while the select condition  actually
> >RCV> matches more than that. in oracle we can use the ROW_NUM variable
> >RCV> for this purpose but now i met such an issue with postgreSQL
> >
> >On v6.4 you can specify a limit for queries as:
> >
> >set QUERY_LIMIT TO '10';
> >To have only the first 10 rows from a select;
> 
> this will limit _all_ the querys, right ?
> wich is not very flexible.
> 
> I can't do something like
> 
> select * from news order by news_date limit 10
> 
> ?

Sergio,

be patient. Jan Wieck is working on patch for support LIMIT in select query.
I'm using his patch (trial 2) with 6.4 and it works great for me.
With this patch you can limit output using LIMIT offset,count statement.
Hope, this patch will come to 6.4.1.

	Regards,

		Oleg

PS. Browse hackers mailing list for last month to read discussion on
    this subject.
> 
> Sergio
> 
> 

_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83



Mon, 9 Nov 1998 23:27:49 +0300 (MSK)
Oleg Bartunov <oleg@sai.msu.su>
Re: [INTERFACES] limiting the rows selected in postgresql

I did this with:-

declare tmp cursor for select * from table;fetch forward 10 in tmp;

But if someone else has a better way, please let me know....

On Sat, 7 Nov 1998, Sergio wrote:

> Sferacarta Software <sferac@bo.nettuno.it> el d���a Mon, 9 Nov 1998 14:33:06 
> +0100, escribi���:
> 
> >Hello Radhakrishnan,
> >
> >luned���, 9 novembre 98, you wrote:
> >
> >
> >RCV> how can i limit the number of rows obtained from a select statement
> >RCV> in postgreSQL to say, 10 rows while the select condition  actually
> >RCV> matches more than that. in oracle we can use the ROW_NUM variable
> >RCV> for this purpose but now i met such an issue with postgreSQL
> >
> >On v6.4 you can specify a limit for queries as:
> >
> >set QUERY_LIMIT TO '10';
> >To have only the first 10 rows from a select;
> 
> this will limit _all_ the querys, right ?
> wich is not very flexible.
> 
> I can't do something like
> 
> select * from news order by news_date limit 10
> 
> ?
> 
> Sergio
> 
> 

James (james@linuxrocks.co.uk)
Vortex Internet
My Windows unders~1 long filena~1, and yours?


Mon, 9 Nov 1998 22:06:11 +0000 (GMT)
A James Lewis <james@vrtx.net>
Re: [INTERFACES] limiting the rows selected in postgresql
pg_dump has improved a lot since 6.3.2, but here is one bug that relates
to inheritance, where a parent table has constraints.  

These are the original table definitions:

create table individual
(
        gender          char(1)         check (gender = 'M' or gender = 'F' or 
gender is null),
        born            datetime        check ((born >= '1 Jan 1880' and born 
<= 'today') or born is null),
        surname         text,
        forenames       text,
        title           text,
        old_surname     text,
        mobile          text,
        ni_no           text,

        constraint is_named check (not (surname isnull and forenames isnull))
)
        inherits (person)
;

create table outworker
(
        started         datetime                not null,
        finish          datetime

)
        inherits (individual)
;

This is the output from trying to reload the pg_dump output:

CREATE TABLE "individual" ("gender" char(1), "born" "datetime", "surname" 
"text", "forenames" "text", "title" "text", "old_surname" "text", "mobile" 
"text", "ni_no" "text",  CONSTRAINT is_named CHECK (NOT ( surname IS NULL AND 
forenames IS NULL )),  CONSTRAINT individual_born CHECK (( born >= '1 Jan 
1880' AND born <= 'today' ) OR born IS NULL),  CONSTRAINT individual_gender 
CHECK (gender = 'M' OR gender = 'F' OR gender IS NULL)) inherits ( "person");
CREATE

CREATE TABLE "outworker" ("started" "datetime" NOT NULL, "finish" "datetime",  
CONSTRAINT individual_gender CHECK (gender = 'M' OR gender = 'F' OR gender IS 
NULL),  CONSTRAINT individual_born CHECK (( born >= '1 Jan 1880' AND born <= 
'today' ) OR born IS NULL),  CONSTRAINT is_named CHECK (NOT ( surname IS NULL 
AND forenames IS NULL ))) inherits ( "individual");
ERROR:  DefineRelation: name (individual_gender) of CHECK constraint duplicated


The problem is that pg_dump is unnecessarily restating the constraints for
the parent table in its descendants.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Therefore being justified by faith, we have peace 
      with God through our Lord Jesus Christ."    Romans 5:1



Mon, 16 Nov 1998 11:50:34 +0000
"Oliver Elphick" <olly@lfix.co.uk>
pg-dump bug (at 6.4)
I have investigated further the bug in pg_dump relating to inherited
check constraints.  This arises in src/bin/pg_dump/pg_dump.c in getTables(), where the query recovers all the constraints for a table, whether or not
they are inherited:
           
   1477        sprintf(query, "SELECT rcname, rcsrc from pg_relcheck "   
   1478                       "where rcrelid = '%s'::oid ",              
   1479                           tblinfo[i].oid);                 

In the following example, a constraint is inherited from the
table `individual':

   bray=> select oid, relname from pg_class
          where oid in
              (select rcrelid from pg_relcheck
               where rcname = 'is_named')
	       order by oid desc;
     oid|relname   
   -----+----------
   67552|staff     
   67436|outworker 
   67111|individual
   (3 rows)


   bray=> select rcrelid, rcname, rcsrc from pg_relcheck
          where rcname = 'is_named'
          order by rcrelid desc;
   rcrelid|rcname  |rcsrc                                        
   -------+--------+---------------------------------------------
     67552|is_named|NOT ( surname IS NULL AND forenames IS NULL )
     67436|is_named|NOT ( surname IS NULL AND forenames IS NULL )
     67111|is_named|NOT ( surname IS NULL AND forenames IS NULL )
   (3 rows)


pg_dump writes all three constraints into its output, which causes the
table creation to fail on the inherited tables when the database is
restored.

We actually need to select a check constraint only if, for each constraint,
tblinfo[i].oid = min(rcrelid).  However, I cannot work out how
to write the query (not least because there is no min()
function for oids).

Can anyone take this further, please?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "For by grace are ye saved through faith; and that not
      of yourselves. It is the gift of God; not of works, 
      lest any man should boast."    Ephesians 2:8,9 



Wed, 18 Nov 1998 01:38:05 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] pg-dump bug (at 6.4) 
Still trying to fix the bug with inherited check constraints...

I have tried to create a min(oid) aggregate, but when I use it, I get
the message `ERROR:  fmgr_info: function 108994: cache lookup failed'.

What is the problem, please?

I created it thus:

create function oid4smaller (oid, oid) returns oid as
                '/home/olly/cprogs/oidcompare.so' language 'c';

create aggregate min (basetype = oid, sfunc1 = oid4smaller,
                      stype1 = oid, stype2 = oid);


The C file is compiled and linked thus (for Linux x86):

  $ gcc -o oidcompare.o -c -I/usr/include/postgresql oidcompare.c
  $ gcc -shared -o oidcompare.so oidcompare.o

and it says:
#include <postgresql/postgres.h>

Oid oid4smaller(Oid o1, Oid o2) {
        return (o1 < o2 ? o1 : o2);
}

Oid oid4larger(Oid o1, Oid o2) {
        return (o1 > o2 ? o1 : o2);
}



-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "If my people, which are called by my name, shall 
      humble themselves, and pray, and seek my face, and 
      turn from their wicked ways; then will I hear from 
      heaven, and will forgive their sin, and will heal 
      their land."               II Chronicles 7:14 



Fri, 20 Nov 1998 23:30:29 +0000
"Oliver Elphick" <olly@lfix.co.uk>
pg_dump bug - problems along the way
Hello Oliver,

sabato, 21 novembre 98, you wrote:

OE> Still trying to fix the bug with inherited check constraints...

OE> I have tried to create a min(oid) aggregate, but when I use it, I get
OE> the message `ERROR:  fmgr_info: function 108994: cache lookup failed'.

OE> What is the problem, please?

OE> I created it thus:

OE> create function oid4smaller (oid, oid) returns oid as
OE>                 '/home/olly/cprogs/oidcompare.so' language 'c';

OE> create aggregate min (basetype = oid, sfunc1 = oid4smaller,
OE>                       stype1 = oid, stype2 = oid);

Try this...it works...

create function oid4smaller (oid, oid) returns oid as
'
begin
     if $1 > $2 then
        return $2;
     else
        return $1;
     end if;
 end;
' language 'plpgsql';

create aggregate m (basetype = oid, sfunc1 = oid4smaller,
                      stype1 = oid, stype2 = oid);

prova=> select oid from a;
   oid
------
376064
376065
380064
380065
380066
380067
(6 rows)

prova=> select min(oid) from a;
   min
------
376064
(1 row)


-Jose'-



Mon, 23 Nov 1998 15:35:34 +0100
Sferacarta Software <sferac@bo.nettuno.it>
Re: [HACKERS] pg_dump bug - problems along the way
I think this is a bug in 6.4:

bray=> select rcname, rcsrc from pg_relcheck where rcrelid = '115404'::oid and 
rcrelid in (select min(rcrelid) from pg_relcheck group by rcname);
ERROR:  parser: Subselect has too many or too few fields.

The subselect only produces one column; so I think that the error message 
is wrong.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "The LORD is nigh unto all them that call upon him, to
      all that call upon him in truth."        
                                   Psalms 145:18



Mon, 23 Nov 1998 17:17:30 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Parser bug?
Sferacarta Software wrote:
  >Try this...it works...
  >
  >create function oid4smaller (oid, oid) returns oid as
  >'
  >begin
  >     if $1 > $2 then
  >        return $2;
  >     else
  >        return $1;
  >     end if;
  > end;
  >' language 'plpgsql';

I'm afraid it doesn't work for me; clearly the problem is elsewhere:
 
bray=> select min(oid) from europe;
ERROR:  fmgr_info: function 108994: cache lookup failed

**Idea** - try in another database -- it works, so it must be a database
corruption of some kind.

Thanks for your help.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "The LORD is nigh unto all them that call upon him, to
      all that call upon him in truth."        
                                   Psalms 145:18



Mon, 23 Nov 1998 17:17:31 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] pg_dump bug - problems along the way 
>
> Sferacarta Software wrote:
>   >Try this...it works...
>   >
>   >create function oid4smaller (oid, oid) returns oid as
>   >'
>   >begin
>   >     if $1 > $2 then
>   >        return $2;
>   >     else
>   >        return $1;
>   >     end if;
>   > end;
>   >' language 'plpgsql';
>
> I'm afraid it doesn't work for me; clearly the problem is elsewhere:
>
> bray=> select min(oid) from europe;
> ERROR:  fmgr_info: function 108994: cache lookup failed
>
> **Idea** - try in another database -- it works, so it must be a database
> corruption of some kind.

    Looks like you dropped and recreated the function used in the
    min(oid)  aggregate  without  dropping  and  recreating   the
    aggregate itself.

    Note  that  the functions used in an aggregate are referenced
    by OID, not by name.  In pg_aggregate the pg_proc tuple  with
    the  old  OID  is still referenced and cannot be found (cache
    lookup failed). Drop the agg and recreate it.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck@debis.com (Jan Wieck) #



Mon, 23 Nov 1998 19:12:53 +0100 (MET)
jwieck@debis.com (Jan Wieck)
Re: [HACKERS] pg_dump bug - problems along the way
Jan Wieck wrote:
  >> bray=> select min(oid) from europe;
  >> ERROR:  fmgr_info: function 108994: cache lookup failed
  >>
  >> **Idea** - try in another database -- it works, so it must be a database
  >> corruption of some kind.
  >
  >    Looks like you dropped and recreated the function used in the
  >    min(oid)  aggregate  without  dropping  and  recreating   the
  >    aggregate itself.
  >
  >    Note  that  the functions used in an aggregate are referenced
  >    by OID, not by name.  In pg_aggregate the pg_proc tuple  with
  >    the  old  OID  is still referenced and cannot be found (cache
  >    lookup failed). Drop the agg and recreate it.

Yes; that is what I did, which explains why it works in another
database where I hadn't been playing.  Thank you.

May I suggest that the error message be changed from

  `ERROR:  fmgr_info: function 108994: cache lookup failed'

to:

  `ERROR:  fmgr_info: function 108994 not found in pg_proc'

which would be a better explanation of what was wrong, and would have
let me diagnose the problem for myself (I hope!).  The existing message
suggests that the function itself is present but somehow faulty.

(src/backend/utils/fmgr/fmgr.c in fmgr_info())

In general, is it not better to say as explicitly as possible what is
wrong, rather than to use terms like "cache lookup" which are only
meaningful to someone who knows the program internals?  There are several
other error messages in that same file which could do with similar
clarification.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "The LORD is nigh unto all them that call upon him, to
      all that call upon him in truth."        
                                   Psalms 145:18



Mon, 23 Nov 1998 19:50:06 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] pg_dump bug - problems along the way 
> I think this is a bug in 6.4:
> 
> bray=> select rcname, rcsrc from pg_relcheck where rcrelid =
> '115404'::oid and rcrelid in (select min(rcrelid) from pg_relcheck
> group by rcname); ERROR:  parser: Subselect has too many or too
> few fields.
> 
> The subselect only produces one column; so I think that the
> error message is wrong.

What is the GROUP BY doing?

--
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 23 Nov 1998 15:10:15 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Parser bug?
Bruce Momjian wrote:
  >> I think this is a bug in 6.4:
  >> 
  >> bray=> select rcname, rcsrc from pg_relcheck where rcrelid =
  >> '115404'::oid and rcrelid in (select min(rcrelid) from pg_relcheck
  >> group by rcname); ERROR:  parser: Subselect has too many or too
  >> few fields.
  >> 
  >> The subselect only produces one column; so I think that the
  >> error message is wrong.
  >
  >What is the GROUP BY doing?

This relates to the bug in pg_dump which messes up inherited constraints.

The object is to find which is the table in an inheritance hierarchy for
which the check constraint is first defined, which must inevitably be the
one with the lowest numbered oid.  The GROUP BY operates with the aggregate
to return the low-numbered oid for each separate rcname.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "The LORD is nigh unto all them that call upon him, to
      all that call upon him in truth."        
                                   Psalms 145:18



Mon, 23 Nov 1998 23:18:26 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Parser bug? 
> Bruce Momjian wrote:
>   >> I think this is a bug in 6.4:
>   >> 
>   >> bray=> select rcname, rcsrc from pg_relcheck where rcrelid =
>   >> '115404'::oid and rcrelid in (select min(rcrelid) from pg_relcheck
>   >> group by rcname); ERROR:  parser: Subselect has too many or too
>   >> few fields.
>   >> 
>   >> The subselect only produces one column; so I think that the
>   >> error message is wrong.
>   >
>   >What is the GROUP BY doing?
> 
> This relates to the bug in pg_dump which messes up inherited constraints.
> 
> The object is to find which is the table in an inheritance hierarchy for
> which the check constraint is first defined, which must inevitably be the
> one with the lowest numbered oid.  The GROUP BY operates with the aggregate
> to return the low-numbered oid for each separate rcname.

Maybe I should be clearer.  You are grouping by a column that is not in
the target list.  If you try the subquery on its own, it should fail
with a better error message.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 23 Nov 1998 22:42:57 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Parser bug?
Bruce Momjian wrote:
...
  >>   >> bray=> select rcname, rcsrc from pg_relcheck where rcrelid =
  >>   >> '115404'::oid and rcrelid in (select min(rcrelid) from pg_relcheck
  >>   >> group by rcname); ERROR:  parser: Subselect has too many or too
  >>   >> few fields.
  >>   >> 
  >>   >> The subselect only produces one column; so I think that the
  >>   >> error message is wrong.
  >>   >
  >>   >What is the GROUP BY doing?
 
...

  >Maybe I should be clearer.  You are grouping by a column that is not in
  >the target list.  If you try the subquery on its own, it should fail
  >with a better error message.
 
It doesn't fail; it produces the results I want.

  bray=> select min(rcrelid) from pg_relcheck group by rcname;
     min
  ------
  115940
  115026
  115026
  115026
  ... etc ...

Any way, why should it be an error to group by a column that is not in the
results list, if the results list comprises aggregates only?

(Mind you, I think I have not yet got a reliable way of finding the
ultimate ancestor of an inherited constraint.  Is it actually possible to
do this with queries or do we have to add a boolean flag to pg_relcheck
to be set where the constraint is/is not inherited?)

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Jesus saith unto him, I am the way, the truth, and the
      life; no man cometh unto the Father, but by me."      
                                         John 14:6 



Tue, 24 Nov 1998 08:11:50 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Parser bug? 
> Bruce Momjian wrote:
> ...
>   >>   >> bray=> select rcname, rcsrc from pg_relcheck where rcrelid =
>   >>   >> '115404'::oid and rcrelid in (select min(rcrelid) from pg_relcheck
>   >>   >> group by rcname); ERROR:  parser: Subselect has too many or too
>   >>   >> few fields.
>   >>   >> 
>   >>   >> The subselect only produces one column; so I think that the
>   >>   >> error message is wrong.
>   >>   >
>   >>   >What is the GROUP BY doing?
>  
> ...
> 
>   >Maybe I should be clearer.  You are grouping by a column that is not in
>   >the target list.  If you try the subquery on its own, it should fail
>   >with a better error message.
>  
> It doesn't fail; it produces the results I want.
> 
>   bray=> select min(rcrelid) from pg_relcheck group by rcname;
>      min
>   ------
>   115940
>   115026
>   115026
>   115026
>   ... etc ...
> 
> Any way, why should it be an error to group by a column that is not in the
> results list, if the results list comprises aggregates only?
> 
> (Mind you, I think I have not yet got a reliable way of finding the
> ultimate ancestor of an inherited constraint.  Is it actually possible to
> do this with queries or do we have to add a boolean flag to pg_relcheck
> to be set where the constraint is/is not inherited?)

Gee, I didn't know we could do that.  Seems like doing that in a
subquery messes things up.  My guess is that the GROUP BY internally
carries the GROUP BY column, and that is not getting stripped when used
in a subquery, so it thinks the subquery returns two columns.  Perhaps
the junknode code needs to be added somewhere for subqueries?

Can anyone else comment on this possibility?


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 24 Nov 1998 10:22:39 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Parser bug?
hi.

I hope this is not a rehash of anything; a quick look at the mailing list
archives turned up similar, but not identical stories.

When I use the "-z" option (dump permissions)when dumping a database I have, I
get a segfault and no output. The other options are irrelevant (i.e., I can
specify any other option or options I like, it still happens). To the best of
my knowledge I have nothing tricky or complex in my database, just standard
types like varchar, bool and int, the refint stuff and a trigger or two.

It does NOT segfault with template1, but nor do I get any output, maybe this
is normal, I'm a total novice at this  :-)

This is not a critical issue for me, I can always set the permissions on my
tables manually, but a) it would be nice not to have to and b) I thought it
might interest someone... Seems to me that whatever the reasons for it,
pg_dump should not lose it to the extent of segfaulting :-)

Regards, K.

---
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Karl Auer (auer@kom.id.ethz.ch)              Geschaeft/work     +41-1-6327531
Kommunikation, ETHZ RZ                          Privat/home     +41-1-4517941
Clausiusstrasse 59                                      Fax     +41-1-6321225
CH-8092 ZUERICH Switzerland

Tue, 24 Nov 1998 16:49:43 +0100 (MET)
Karl Auer <auer@kom.id.ethz.ch>
pg_dump - segfault with -z option
[Charset iso-8859-1 unsupported, filtering to ASCII...]
> hi.
> 
> I hope this is not a rehash of anything; a quick look at the mailing list
> archives turned up similar, but not identical stories.
> 
> When I use the "-z" option (dump permissions)when dumping a database I have, I
> get a segfault and no output. The other options are irrelevant (i.e., I can
> specify any other option or options I like, it still happens). To the best of
> my knowledge I have nothing tricky or complex in my database, just standard
> types like varchar, bool and int, the refint stuff and a trigger or two.
> 
> It does NOT segfault with template1, but nor do I get any output, maybe this
> is normal, I'm a total novice at this  :-)
> 
> This is not a critical issue for me, I can always set the permissions on my
> tables manually, but a) it would be nice not to have to and b) I thought it
> might interest someone... Seems to me that whatever the reasons for it,
> pg_dump should not lose it to the extent of segfaulting :-)

I believe this will be fixed in the first 6.4 minor release.  Should we
schedule 6.4.1 soon, folks.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 24 Nov 1998 11:08:42 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] pg_dump - segfault with -z option
"Oliver Elphick" <olly@lfix.co.uk> writes:
> (Mind you, I think I have not yet got a reliable way of finding the
> ultimate ancestor of an inherited constraint.  Is it actually possible to
> do this with queries or do we have to add a boolean flag to pg_relcheck
> to be set where the constraint is/is not inherited?)

In fact, I was about to point out that the query you were describing
couldn't possibly give you a reliable answer, quite independent of
whether the backend is implementing it properly or not.

Consider

CREATE TABLE parent1 (i1 int4, CONSTRAINT c1 CHECK (i1 > 0));

CREATE TABLE child1 (CONSTRAINT c2 CHECK (i1 > 4)) INHERITS (parent1);

CREATE TABLE child2 (CONSTRAINT c2 CHECK (i1 > 4)) INHERITS (parent1);

This will give us a pg_relcheck like

	rcrelid		rcname	rcbin		rcsrc

	parent1		c1	gobbledegook	i1 > 0
	child1		c1	gobbledegook	i1 > 0
	child2		c1	gobbledegook	i1 > 0
	child1		c2	gobbledegook	i1 > 4
	child2		c2	gobbledegook	i1 > 4

(where I've written table names in place of numeric OIDs for rcrelid).
Now child2 did not inherit c2 from child1, but child1 has a lower OID
than child2, so your test would mistakenly omit c2 from child2's
definition.

It seems to me that the correct way to do this is to compare each of a
table's constraints against its immediate parent's constraints, and omit
from the child any constraints that have the same rcname AND the same
rcsrc as a constraint of the parent.  (You need not look at anything
other than the immediate parent, because constraints inherited from
more distant ancestors will also be listed for the parent.)

There is a case that pg_relcheck does not allow you to distinguish,
and that is whether or not the child definition was actually written
with a redundant constraint:

CREATE TABLE parentx (i1 int4, CONSTRAINT c1 CHECK (i1 > 0));

CREATE TABLE childx (CONSTRAINT c1 CHECK (i1 > 0)) INHERITS (parentx);

Unless we modify pg_relcheck, pg_dump will have to dump this as simply

CREATE TABLE parentx (i1 int4, CONSTRAINT c1 CHECK (i1 > 0));

CREATE TABLE childx () INHERITS (parentx);

since it cannot tell that childx's constraint wasn't simply inherited.
However, it's not clear to me that suppression of redundant constraints
is a bad thing ;-)

			regards, tom lane

Tue, 24 Nov 1998 11:17:58 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Parser bug? 
Karl Auer <auer@kom.id.ethz.ch> writes:
> When I use the "-z" option (dump permissions)when dumping a database I
> have, I get a segfault and no output.

Are you on Postgres 6.4?  I recall fixing several nasty problems with
permissions in pg_dump between 6.3.2 and 6.4.

If you are on 6.4, could you use gdb or something to get a backtrace
showing exactly where pg_dump dies?

FWIW, I currently use -z routinely, but my database is probably even
simpler than yours ... no triggers, for example.  My guess is pg_dump
doesn't work for permissions attached to a trigger, or something along
that line.

			regards, tom lane

Tue, 24 Nov 1998 11:25:46 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] pg_dump - segfault with -z option 
Thanks Tom.

Yes, I am on 6.4. As to gdb, I really wouldn't know how to get the info you
request. The segfault doesn't result in a core file either.

There is nothing sacred about my project (supporting RADIUS authentication),
I'll happily send you my database creation scripts if you want to try making
this happen yourself. This is 6.4 as stated, running on a SuSE 5.3 (Linux
2.0.34) distribution. No problems running the database and all the build
tests work fine.

I have no permissions attached to triggers as far as I know; and the
permissions aren't complicated either. The segfault occurs when running the
dump as user postgres, which should have godlike permissions anyway...

Regards, K.

Am 24-Nov-98 schrieb Tom Lane:
> Karl Auer <auer@kom.id.ethz.ch> writes:
>> When I use the "-z" option (dump permissions)when dumping a database I
>> have, I get a segfault and no output.
> 
> Are you on Postgres 6.4?  I recall fixing several nasty problems with
> permissions in pg_dump between 6.3.2 and 6.4.
> 
> If you are on 6.4, could you use gdb or something to get a backtrace
> showing exactly where pg_dump dies?

---
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Karl Auer (auer@kom.id.ethz.ch)              Geschaeft/work     +41-1-6327531
Kommunikation, ETHZ RZ                          Privat/home     +41-1-4517941
Clausiusstrasse 59                                      Fax     +41-1-6321225
CH-8092 ZUERICH Switzerland

Tue, 24 Nov 1998 17:38:40 +0100 (MET)
Karl Auer <auer@kom.id.ethz.ch>
Re: [HACKERS] pg_dump - segfault with -z option
Bruce Momjian <maillist@candle.pha.pa.us> writes:
> Should we schedule 6.4.1 soon, folks.

I still have about 10 things on my to-do-for-6.4.1 list, and no hope
of doing any of 'em before this weekend.  If "soon" means "a couple
of weeks", then fine ...

			regards, tom lane

Tue, 24 Nov 1998 11:54:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
6.4.1 schedule (was segfault with -z option)
On Tue, 24 Nov 1998, Bruce Momjian wrote:

> Date: Tue, 24 Nov 1998 11:08:42 -0500 (EST)
> From: Bruce Momjian <maillist@candle.pha.pa.us>
> To: auer@kom.id.ethz.ch
> Cc: pgsql-hackers@postgreSQL.org
> Subject: Re: [HACKERS] pg_dump - segfault with -z option
> 
> [Charset iso-8859-1 unsupported, filtering to ASCII...]
> > hi.
> 
> I believe this will be fixed in the first 6.4 minor release.  Should we
> schedule 6.4.1 soon, folks.
> 

Also, how about to include Jan's patch for LIMIT to 6.4.1 ?

	Regards,

		Oleg


> -- 
>   Bruce Momjian                        |  http://www.op.net/~candle
>   maillist@candle.pha.pa.us            |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
> 

_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83


Tue, 24 Nov 1998 20:05:45 +0300 (MSK)
Oleg Bartunov <oleg@sai.msu.su>
Re: [HACKERS] pg_dump - segfault with -z option
I have very simple database without triggers and also see
20:10[mira]:~/tmp>pg_dump -z guta > qq
Segmentation fault

Database    = guta
 +---------------------+----------------------------------------+
 | Relation            | Grant/Revoke Permissions               |
 +---------------------+----------------------------------------+
 | groups              | {"=","megera=arwR","er=r","httpd=arw"} |
 | groups_pkey         |                                        |
 | status              | {"=","megera=arwR","er=r","httpd=arw"} |
 | status_pkey         |                                        |
 | stidx_status_id     |                                        |
 | ugmidx_group_id     |                                        |
 | ugmidx_user_id      |                                        |
 | user_group_map      | {"=","megera=arwR","er=r","httpd=arw"} |
 | user_group_map_pkey |                                        |
 | users               | {"=","megera=arwR","er=r","httpd=arw"} |
 | users_pkey          |                                        |
 |

	Regards,

		Oleg

On Tue, 24 Nov 1998, Tom Lane wrote:

> Date: Tue, 24 Nov 1998 11:25:46 -0500
> From: Tom Lane <tgl@sss.pgh.pa.us>
> To: auer@kom.id.ethz.ch
> Cc: pgsql-hackers@postgreSQL.org
> Subject: Re: [HACKERS] pg_dump - segfault with -z option 
> 
> Karl Auer <auer@kom.id.ethz.ch> writes:
> > When I use the "-z" option (dump permissions)when dumping a database I
> > have, I get a segfault and no output.
> 
> Are you on Postgres 6.4?  I recall fixing several nasty problems with
> permissions in pg_dump between 6.3.2 and 6.4.
> 
> If you are on 6.4, could you use gdb or something to get a backtrace
> showing exactly where pg_dump dies?
> 
> FWIW, I currently use -z routinely, but my database is probably even
> simpler than yours ... no triggers, for example.  My guess is pg_dump
> doesn't work for permissions attached to a trigger, or something along
> that line.
> 
> 			regards, tom lane
> 

_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83


Tue, 24 Nov 1998 20:11:09 +0300 (MSK)
Oleg Bartunov <oleg@sai.msu.su>
Re: [HACKERS] pg_dump - segfault with -z option 
Bruce Momjian <maillist@candle.pha.pa.us> writes:
>> pg_dump should not lose it to the extent of segfaulting :-)

> I believe this will be fixed in the first 6.4 minor release.

If you meant that it has already been fixed, I don't think so.
I see only one post-6.4 patch so far against pg_dump, and it fixes
a wrong-output kind of problem, not a core-dump kind of problem.

BTW, is Oliver intending to try to fix pg_dump's inherited-constraints
problems in time for 6.4.1, or is that a longer-range project?

			regards, tom lane

Tue, 24 Nov 1998 12:21:48 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] pg_dump - segfault with -z option 
Tom Lane wrote:
  >"Oliver Elphick" <olly@lfix.co.uk> writes:
  >> (Mind you, I think I have not yet got a reliable way of finding the
  >> ultimate ancestor of an inherited constraint.  Is it actually possible to
  >> do this with queries or do we have to add a boolean flag to pg_relcheck
  >> to be set where the constraint is/is not inherited?)
  >
  >In fact, I was about to point out that the query you were describing
  >couldn't possibly give you a reliable answer, quite independent of
  >whether the backend is implementing it properly or not.

Yes, I had been using a concrete example; the results were not
sufficiently general.
 
[... skip example ...]

  >It seems to me that the correct way to do this is to compare each of a
  >table's constraints against its immediate parent's constraints, and omit
  >from the child any constraints that have the same rcname AND the same
  >rcsrc as a constraint of the parent.  (You need not look at anything
  >other than the immediate parent, because constraints inherited from
  >more distant ancestors will also be listed for the parent.)

That looks good.  I'll see if I can do it that way.

  >There is a case that pg_relcheck does not allow you to distinguish,
  >and that is whether or not the child definition was actually written
  >with a redundant constraint:
 
[...skip example...]

  >since it cannot tell that childx's constraint wasn't simply inherited.
  >However, it's not clear to me that suppression of redundant constraints
  >is a bad thing ;-)
 
It seems to be quite reasonable to drop them.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Jesus saith unto him, I am the way, the truth, and the
      life; no man cometh unto the Father, but by me."      
                                         John 14:6 



Tue, 24 Nov 1998 17:29:22 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Parser bug? 
On Tue, 24 Nov 1998, Tom Lane wrote:

> Bruce Momjian <maillist@candle.pha.pa.us> writes:
> > Should we schedule 6.4.1 soon, folks.
> 
> I still have about 10 things on my to-do-for-6.4.1 list, and no hope
> of doing any of 'em before this weekend.  If "soon" means "a couple
> of weeks", then fine ...

	What's on that scheduale?  Anything we can help with?

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Tue, 24 Nov 1998 13:29:36 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] 6.4.1 schedule (was segfault with -z option)
The Hermit Hacker <scrappy@hub.org> writes:
>> I still have about 10 things on my to-do-for-6.4.1 list, and no hope
>> of doing any of 'em before this weekend.  If "soon" means "a couple
>> of weeks", then fine ...

> 	What's on that scheduale?  Anything we can help with?

Here's my unvarnished to-do list.  Most of it is configuration checks
that I figured I was in a good position to test (particularly the HPUX
items), but if you feel that I'm holding things up then feel free to
take a couple...

Actually, what I was kind of hoping *you* would work on is the
process-title-setting issue, since you had once indicated you'd
have a go at that for 6.4.1.  I still think we need to snaffle
the process-title code from sendmail.  What we have now doesn't
manage to change the ps listing on HPUX, and a couple of other
OSes too if I recall the discussion properly.

			regards, tom lane


Auer reports segfault with pg_dump -z ... wtf?

HPUX FAQ!  Ought to document yacc issues for HPUX 9.  HP's lex doesn't
seem to work either, under both 9.* & 10.*.  If you don't install patch
PHSS_4630 on HPUX 9, rint() is broken which leads to weird bugs in
datetime.

Update FAQ_CVS, also web page if not same file --- 1.9 is obsolete.

postmaster.c's fflush(NULL) doesn't work on SUNOS ... seems safe to
just fflush stdout and stderr instead.  Worth making a configure test?
11/24 msgs

float8 regress test, pow, exp problems --- 11/18, 11/24 msgs

configure should unset USE_TK if X stuff not found.  Cf. situation on
csp linux box, which has Tcl but no X.

configure.in ought not have a hardwired set of assumptions about what
the template file can define ... why not just read what's there, for
pete's sake?  Proposed to list 11/22/98.

make install in doc failed on hpc200.  Likely cause: no "zcat" in path.
Need to use autoconf to look for gzcat or zcat.

bogus selection of template when .similar entry includes partial rev
number --- see ts' msg of 11/19 8:44.  Note that second try tries
to match without regard to version number, which is good ... it just
goes too far when there are multiple possibilities in .similar.
Note the wrong fix here could break HPUX --- might want to delete
rev number from hp line in .similar.  (Fixed 11/22, I think.)

hpux 10.01 does not have POSIX signals?  Perhaps we ought to use
autoconf to detect whether system has SA_RESTART...  trick here is
that Makefile.hpux needs to know it too, to adjust library link order.
So would need to set info in both Makefile.global and config.h.
Could get rid of -DHPUX_10, which'd be a good idea anyway (in fact,
it might be possible to stop calling uname at all in Makefile.hpux,
which'd be a performance win).
NOTE: make sure template file or Makefile.port can override autoconf decision
in either direction (or at least the no-posix direction)...  Makefile.port
can't do it because of config.h dependency.  See pgsql msgs of 10/30.

Why does configure want libcurses to be included?  Causes problems
in Makefile.hpux for HP10..  Could suppress libBSD instead of adding
an extra -lc pass if we got rid of curses.  (libBSD only has signal
stuff.)  Looks like libreadline requires either libtermcap or libcurses,
whichever is available on a given platform.  Safest thing might be to
just put -lBSD in front of -lc ...  Cf msgs 10/31 to 11/1.

%.sl rule in Makefile.hpux is probably not needed anymore?

Tue, 24 Nov 1998 12:41:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] 6.4.1 schedule (was segfault with -z option) 
Oleg Bartunov wrote:

> On Tue, 24 Nov 1998, Bruce Momjian wrote:
> > I believe this will be fixed in the first 6.4 minor release.  Should we
> > schedule 6.4.1 soon, folks.
> >
>
> Also, how about to include Jan's patch for LIMIT to 6.4.1 ?

    No!

    Should  be  put  out as a separate patch. We forgot about the
    initial v6.4-feature-patch I  created  over  that  branch/tag
    discussion.   I  still have it here - who can put it onto the
    ftp-server?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck@debis.com (Jan Wieck) #



Tue, 24 Nov 1998 18:55:47 +0100 (MET)
jwieck@debis.com (Jan Wieck)
Re: [HACKERS] pg_dump - segfault with -z option
On Tue, 24 Nov 1998, Jan Wieck wrote:

> Oleg Bartunov wrote:
> 
> > On Tue, 24 Nov 1998, Bruce Momjian wrote:
> > > I believe this will be fixed in the first 6.4 minor release.  Should we
> > > schedule 6.4.1 soon, folks.
> > >
> >
> > Also, how about to include Jan's patch for LIMIT to 6.4.1 ?
> 
>     No!
> 
>     Should  be  put  out as a separate patch. We forgot about the
>     initial v6.4-feature-patch I  created  over  that  branch/tag
>     discussion.   I  still have it here - who can put it onto the
>     ftp-server?

	You :)  ~pgsql/ftp/pub/patches is writable by those in group
pgsql, which is all those wiht comitter's access...gets wiped out after
each release is made though, as its only pertinent to the current
release...

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Tue, 24 Nov 1998 14:24:58 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] pg_dump - segfault with -z option
Tom Lane wrote:
  >BTW, is Oliver intending to try to fix pg_dump's inherited-constraints
  >problems in time for 6.4.1, or is that a longer-range project?
 
I hope to fix it quickly, since I want to back port a good pg_dump into
Debian's 6.3.2 before I release a 6.4 for Debian.

If anyone else can fix it, don't wait for me to find out how!

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Jesus saith unto him, I am the way, the truth, and the
      life; no man cometh unto the Father, but by me."      
                                         John 14:6 



Tue, 24 Nov 1998 18:33:01 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] pg_dump - segfault with -z option 
Tom Lane wrote:
...
  >In fact, I was about to point out that the query you were describing
  >couldn't possibly give you a reliable answer, quite independent of
  >whether the backend is implementing it properly or not.

I think this will do the job; can you please check it out:

select rcname,rcsrc from pg_relcheck, pg_inherits as i                          
        where rcrelid = '%s'::oid                                           
          and (not exists                                    -- for
		 (select * from pg_inherits                  -- non-inherited
                        where inhrel = pg_relcheck.rcrelid)  -- tables
            
          or  (not exists                                    --
		 (select * from pg_relcheck as c             -- for
                        where c.rcname = pg_relcheck.rcname  -- inherited
                          and c.rcrelid = i.inhparent)       -- tables
               and rcrelid = i.inhrel));                     --


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Jesus saith unto him, I am the way, the truth, and the
      life; no man cometh unto the Father, but by me."      
                                         John 14:6 



Tue, 24 Nov 1998 19:22:02 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Parser bug? 
"Oliver Elphick" wrote:
  >I think this will do the job; can you please check it out:
  
It seems to work, at least for my database, so here is a patch:

*** postgresql-6.4/src/bin/pg_dump/pg_dump.c	Tue Nov 24 23:01:27 1998
--- postgresql-6.4.orig/src/bin/pg_dump/pg_dump.c	Mon Oct 26 01:05:07 1998
***************
*** 1459,1504 ****
  		tblinfo[i].ncheck = atoi(PQgetvalue(res, i, i_relchecks));
  		tblinfo[i].ntrig = atoi(PQgetvalue(res, i, i_reltriggers));
  
! 		/* Exclude inherited CHECKs from CHECK constraints total */
! 		if (tblinfo[i].ncheck > 0)
! 		{
! 			PGresult	*res2;
! 			int		ntups2;
! 
! 			if (g_verbose)
! 				fprintf(stderr, "%s excluding inherited CHECK constraints "
! 						"for relation: '%s' %s\n",
! 						g_comment_start,
! 						tblinfo[i].relname,
! 						g_comment_end);
! 
! 			sprintf(query, "SELECT * from pg_relcheck, pg_inherits as i "
! 					"where rcrelid = '%s'::oid "
! 					" and exists "
! 					"  (select * from pg_relcheck as c "
! 					"    where c.rcname = pg_relcheck.rcname "
! 					"      and c.rcrelid = i.inhparent) "
! 					" and rcrelid = i.inhrel",
! 					tblinfo[i].oid);
! 			res2 = PQexec(g_conn, query);
! 			if (!res2 ||
! 				PQresultStatus(res2) != PGRES_TUPLES_OK)
! 			{
! 				fprintf(stderr, "getTables(): SELECT (for inherited CHECK) failed\n");
! 				exit_nicely(g_conn);
! 			}
! 			ntups2 = PQntuples(res2);
! 			tblinfo[i].ncheck -= ntups2;
! 			if (tblinfo[i].ncheck < 0)
! 			{
! 				fprintf(stderr, "getTables(): found more inherited CHECKs than total for "
! 						"relation %s\n",
! 						tblinfo[i].relname);
! 				exit_nicely(g_conn);
! 			}
! 		}
! 
! 		/* Get CHECK constraints originally defined for this table */
  		if (tblinfo[i].ncheck > 0)
  		{
  			PGresult   *res2;
--- 1459,1465 ----
  		tblinfo[i].ncheck = atoi(PQgetvalue(res, i, i_relchecks));
  		tblinfo[i].ntrig = atoi(PQgetvalue(res, i, i_reltriggers));
  
! 		/* Get CHECK constraints */
  		if (tblinfo[i].ncheck > 0)
  		{
  			PGresult   *res2;
***************
*** 1513,1531 ****
  						tblinfo[i].relname,
  						g_comment_end);
  
! 			sprintf(query, "SELECT DISTINCT rcname, rcsrc "
! 					"from pg_relcheck, pg_inherits as i "
! 					"where rcrelid = '%s'::oid "
! 					/* allow all checks from tables that do not inherit */
! 					"  and (not exists "
! 					"    (select * from pg_inherits "
! 					"       where inhrel = pg_relcheck.rcrelid)"
! 					/* and allow checks that are not inherited from other tables */
! 					"         or  (not exists "
! 					"           (select * from pg_relcheck as c "
! 					"             where c.rcname = pg_relcheck.rcname "
! 					"               and c.rcrelid = i.inhparent) "
! 					"         and rcrelid = i.inhrel))",
  					tblinfo[i].oid);
  			res2 = PQexec(g_conn, query);
  			if (!res2 ||
--- 1474,1481 ----
  						tblinfo[i].relname,
  						g_comment_end);
  
! 			sprintf(query, "SELECT rcname, rcsrc from pg_relcheck "
! 					"where rcrelid = '%s'::oid ",
  					tblinfo[i].oid);
  			res2 = PQexec(g_conn, query);
  			if (!res2 ||

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Jesus saith unto him, I am the way, the truth, and the
      life; no man cometh unto the Father, but by me."      
                                         John 14:6 


Tue, 24 Nov 1998 23:16:23 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Patch for pg_dump (6.4) inheritance bug 
> >       What's on that scheduale?  Anything we can help with?

I've made a few fixes, and have one more on my list for v6.4.x:

Update contrib/linux startup scripts (someone had modified them to give
both sh and csh examples, but the labeling is scrambled).

Won't get to it until next week, but it isn't critical to a release
either...

                      - Tom

Wed, 25 Nov 1998 02:27:15 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] 6.4.1 schedule (was segfault with -z option)
> I've made a few fixes, and have one more on my list for v6.4.x:

Never mind. I went ahead and committed the updates for this.

                  - Tom

Wed, 25 Nov 1998 02:58:25 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] 6.4.1 schedule (was segfault with -z option)
I tried posting a bug report to psql-ports@postgresql.org and it bounced.

I redirected the bounce message to hackers@postgreSQL.org (this list) and
it never showed up.

Hello?

Assuming this message got though I'd like to report my problems compiling
6.4 with KTH-KRB, and what I can say about how to fix it.

I'm still testing, but basically it looks like if you have kerberos 4 then
you need to disable use of the system crypt routines.  This *should* be
handled in the configure stuff.  I fixed it by modifying fe-auth.c and
fe-connect.c in libpq to not include <crypt.h> and by modifying
Makefile.global to include -lresolv instead of -lcrypt (that gives the
right load order).

The function des_encrypt exists in both the KTH kerberos and the system
crypt libraries with different arguments.

Signature failed Preliminary Design Review.
Feasibility of a new signature is currently being evaluated.
h.b.hotz@jpl.nasa.gov, or hbhotz@oxy.edu



Wed, 25 Nov 1998 12:32:08 -0800
"Henry B. Hotz" <hotz@jpl.nasa.gov>
Testing, Hello?
This is a Debian Linux bug report.

Size is still defined as unsigned int in 6.4.  Paul is suggesting that this
be changed to size_t for compatibility with alpha.

Is that correct?

------- Forwarded Message

Date:    Wed, 16 Sep 1998 21:03:44 +0200
From:    Paul Slootman <paul@wurtel.demon.nl>
To:      submit@bugs.debian.org
Subject: Bug#26778: postgresql-dev: postgresql/c.h typedefs Size as 'unsigned i
	  nt', should be 'size_t'

Package: postgresql-dev
Version: 6.3.2-11

Using the above wrong type for "Size" leads to conflicts with system
prototypes, e.g. palloc() which is apparently a define for malloc().
Replacing the "typedef unsigned int Size;" with "typedef size_t Size;"
leads to a usable system again; at least, on Alpha, where "unsigned int"
is NOT the same as "unsigned long", unlike i386.

Thanks,
Paul Slootman

- -- System Information
Debian Release: 2.0
Kernel Version: Linux alf 2.0.35 #1 Tue Aug 11 11:09:24 CEST 1998 alpha unknown

------- End of Forwarded Message


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Let us therefore come boldly unto the throne of grace,
      that we may obtain mercy, and find grace to help in 
      time of need."             Hebrews 4:16 



Wed, 25 Nov 1998 23:48:34 +0000
"Oliver Elphick" <olly@lfix.co.uk>
postgresql/c.h typedefs Size as 'unsigned int' (fwd)
"Henry B. Hotz" <hotz@jpl.nasa.gov> writes:
> I'm still testing, but basically it looks like if you have kerberos 4 then
> you need to disable use of the system crypt routines.  This *should* be
> handled in the configure stuff.

That's fairly unpleasant, since it's not out of the question that a
given site might need to support both auth methods to cope with varying
clients.

> The function des_encrypt exists in both the KTH kerberos and the system
> crypt libraries with different arguments.

Not everywhere --- there's no such routine in my crypt library, for
instance.  I would not like to see kerberos + crypt disabled everywhere
because it does not work on your machine.

Ideally we'd need an autoconf test to discover whether kerberos and
crypt libraries are compatible on a given machine, and an autoconf
--with switch to allow the user to decide which one to include if
they're not.  Do you have any ideas about a simple way to check whether
this problem exists on a given platform?

			regards, tom lane

Wed, 25 Nov 1998 19:18:37 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Testing, Hello? 
At 4:18 PM -0800 11/25/98, Tom Lane wrote:
>"Henry B. Hotz" <hotz@jpl.nasa.gov> writes:
>> I'm still testing, but basically it looks like if you have kerberos 4 then
>> you need to disable use of the system crypt routines.  This *should* be
>> handled in the configure stuff.
>
>That's fairly unpleasant, since it's not out of the question that a
>given site might need to support both auth methods to cope with varying
>clients.

Yeah.  I note that if you use the Solaris built-in kerberos support the
conflict should not exist.  For Postgres this problem is specific to the
KTH kerberos implementation I think, but it also exists with SSL.  I have
no information about MIT kerberos IV or V.

>> The function des_encrypt exists in both the KTH kerberos and the system
>> crypt libraries with different arguments.
>
>Not everywhere --- there's no such routine in my crypt library, for
>instance.  I would not like to see kerberos + crypt disabled everywhere
>because it does not work on your machine.

This is Solaris 2.5, presumably 2.6 and 7 have the same problem.

>Ideally we'd need an autoconf test to discover whether kerberos and
>crypt libraries are compatible on a given machine, and an autoconf
>--with switch to allow the user to decide which one to include if
>they're not.  Do you have any ideas about a simple way to check whether
>this problem exists on a given platform?

If you include <crypt.h> and <krb.h> from the system and
/usr/athena/include respectively then you get a compile error.

My problem may actually be a bit obscure.  I'm using the KTH implementation
of kerberos IV because I want to be able to use the JPL AFS kerberos
server.  (AFS kerberos is an incompatable variant of MIT kerberos IV for
those who don't know.  Solaris and NetBSD come with MIT kerberos IV support
built-in.  MIT kerberos V can support both kerberos IV variants, but
Postgres is a client.)

I will put in a plug for autoconf support for kerberos in any case.  We
need a --with-kerberos[={4,5}] option and --with-kerberos-include=..,
--with-kerberos-lib=.., and --with-kerberos-srvtab=.. options.

The administrator guide says support for kerberos IV will disappear when 5
is released.  I think there should be a fairly long delay in that.  Many
people will need to use kerberos IV in order to use an institutional
capability, like AFS accounting.  Many people should prefer to use the
built-in capabilities of their OS and all current bundled kerberos support
is at version IV.  This will take a *long* time.

Finally let me put in a big public thank-you to Tom Ivar Helbekkmo for
patiently explaining many things that I should have understood from the
documentation.

Signature failed Preliminary Design Review.
Feasibility of a new signature is currently being evaluated.
h.b.hotz@jpl.nasa.gov, or hbhotz@oxy.edu



Mon, 30 Nov 1998 09:53:28 -0800
"Henry B. Hotz" <hotz@jpl.nasa.gov>
Re: [HACKERS] Testing, Hello?
On Mon, 30 Nov 1998, Henry B. Hotz wrote:

> those who don't know.  Solaris and NetBSD come with MIT kerberos IV support
> built-in.  MIT kerberos V can support both kerberos IV variants, but
> Postgres is a client.)

Solaris 7 ships with Kerberos V support now.  I'll dig up a little info I
put together on Solaris 7 and send it on.

Dax Kelson
Internet Connect, Inc.


Mon, 30 Nov 1998 13:14:11 -0700 (MST)
Dax Kelson <dkelson@inconnect.com>
Re: [HACKERS] Testing, Hello?
"Henry B. Hotz" <hotz@jpl.nasa.gov> writes:
> At 4:18 PM -0800 11/25/98, Tom Lane wrote:
>> Ideally we'd need an autoconf test to discover whether kerberos and
>> crypt libraries are compatible on a given machine, and an autoconf
>> --with switch to allow the user to decide which one to include if
>> they're not.  Do you have any ideas about a simple way to check whether
>> this problem exists on a given platform?

> If you include <crypt.h> and <krb.h> from the system and
> /usr/athena/include respectively then you get a compile error.

OK, that seems pretty easy to check.

> I will put in a plug for autoconf support for kerberos in any case.  We
> need a --with-kerberos[={4,5}] option and --with-kerberos-include=..,
> --with-kerberos-lib=.., and --with-kerberos-srvtab=.. options.

Where does that information get entered now --- do you have to do it
manually after running configure?

I'd be willing to do the autoconf hacking, but since I have no kerberos
setup here, I can't test it; and I'm not familiar enough with kerberos
to expect to get it right the first time.  If you can test but don't
want to hack the code, let's get together off-list and work on it.

> The administrator guide says support for kerberos IV will disappear when 5
> is released.  I think there should be a fairly long delay in that.

As long as we have kerb4 support in the code (and I'm not hearing anyone
propose to take that out), it ought to be supported at the autoconf
level too.

			regards, tom lane

Tue, 01 Dec 1998 11:25:49 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Testing, Hello? 
At 8:25 AM -0800 12/1/98, Tom Lane wrote:
>"Henry B. Hotz" <hotz@jpl.nasa.gov> writes:
>> If you include <crypt.h> and <krb.h> from the system and
>> /usr/athena/include respectively then you get a compile error.
>
>OK, that seems pretty easy to check.

Specifically des_encrypt() is declared with a different number of arguments.

>> I will put in a plug for autoconf support for kerberos in any case.  We
>> need a --with-kerberos[={4,5}] option and --with-kerberos-include=..,
>> --with-kerberos-lib=.., and --with-kerberos-srvtab=.. options.
>
>Where does that information get entered now --- do you have to do it
>manually after running configure?

Yes.  It's documented in the administrator guide actually.  You edit
Makefile.global.  The comments in Makefile.global are pretty clear.

The autoconf support should just be a matter of copying the flags I
indicated into Makefile.global.

Signature failed Preliminary Design Review.
Feasibility of a new signature is currently being evaluated.
h.b.hotz@jpl.nasa.gov, or hbhotz@oxy.edu



Tue, 1 Dec 1998 09:44:34 -0800
"Henry B. Hotz" <hotz@jpl.nasa.gov>
Re: [HACKERS] Testing, Hello?
"Oliver Elphick" <olly@lfix.co.uk> writes:
> It seems to work, at least for my database, so here is a patch:
> [ patch to prevent dumping of inherited constraints snipped ]

I have applied this patch to both the main CVS tree and REL6_4 branch,
along with Constantin Teodorescu's suggestion to improve the formatting
of pg_dump's CREATE TABLE commands, and some work of my own to stop
an occasional coredump in pg_dump -z.

FYI, I was able to simplify your query for fetching non-inherited
checks; it now looks like:

sprintf(query, "SELECT rcname, rcsrc from pg_relcheck "
		"where rcrelid = '%s'::oid "
		"   and not exists "
		"  (select * from pg_relcheck as c, pg_inherits as i "
		"   where i.inhrel = pg_relcheck.rcrelid "
		"     and c.rcname = pg_relcheck.rcname "
		"     and c.rcsrc = pg_relcheck.rcsrc "
		"     and c.rcrelid = i.inhparent) ",
		tblinfo[i].oid);


			regards, tom lane

Sat, 05 Dec 1998 17:29:59 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Patch for pg_dump (6.4) inheritance bug 
On Thu, 5 Nov 1998, Constantin Teodorescu wrote:
> RedHat 5.1 i386 with PostgreSQL v6.4 final version
> 
> ./configure --with-tcl
> 
> after installing libpgtcl , cannot run pgaccess because library crypt
> hasn't been linked into libpgtcl.so
> 
> must manually add -lcrypt in Makefile into src/interfaces/libpgtcl

Hi Constantin. This is a problem only with systems running glibc, because
for legal reasons the FSF doesn't want to include encryption as part of
glibc. I assume you are doing this on Debian or on Red Hat 5.x?

Perhaps we  need an extra template for glibc?  

I came across this problem too today (I am currently packaging 6.4 into an
RPM for Red Hat 5.2). I have generated a patch file which fixes the lack
of -lcrypt in a number of places for glibc systems. After I have
thoroughly tested it I'll try to post it here.

> Also, PgAccess version that has been included in 6.4 final version is
> 0.90 , but current stable version is 0.91
> (ftp://ftp.flex.ro/pub/pgaccess/pgaccess-0.91.tar.gz)

Thanks for the pointer. I will incorporate that as another patch in
the .spec file for my Red Hat RPM. 

--
Eric Lee Green         eric@linux-hw.com     http://www.linux-hw.com/~eric
 "Linux represents a best-of-breed UNIX, that is trusted in mission
  critical applications..."   --  internal Microsoft memo


Sun, 6 Dec 1998 15:45:15 -0500 (EST)
Eric Lee Green <eric@linux-hw.com>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!!
Eric Lee Green <eric@linux-hw.com> writes:
> On Thu, 5 Nov 1998, Constantin Teodorescu wrote:
>> must manually add -lcrypt in Makefile into src/interfaces/libpgtcl

> I came across this problem too today (I am currently packaging 6.4 into an
> RPM for Red Hat 5.2). I have generated a patch file which fixes the lack
> of -lcrypt in a number of places for glibc systems.

I believe this is already fixed in the current Postgres sources.  Rather
than generating your own patch, would you verify that the current
sources are fixed?

You can get the current code from the Postgres CVS server --- see
http://www.postgresql.org/docs/faq-cvs.shtml.  Do the checkout with
"-r REL6_4" to get the soon-to-be-6.4.1 stable branch, rather than
the 6.5 development branch...

			regards, tom lane

Sun, 06 Dec 1998 16:06:04 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!! 
> (I am currently packaging 6.4 into an RPM for Red Hat 5.2).

Is this an alternative to the RedHat packaging, or are you using the
same layout and RPM setup that the RedHat 6.3 package had? Just curious;
I've been thinking about trying to help with RPM support but would be
happy that someone else is looking after it.

btw, we are currently looking at a date/time peculiarity in Postgres
with glibc2 which gives funny results for at least one test case. You
might want to stay in touch so we can fix that up for you also, once we
have a good workaround.

Actually, we should also test with RH5.2 to make sure the date/time
problem is still present, since it may have been due to a problem in
glibc2 which has since been fixed. I'm only running RH4.2 and RH5.1 so
can't directly test that myself...

                    - Tom

Sun, 06 Dec 1998 23:39:04 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!!
On Sun, 6 Dec 1998, Tom Lane wrote:
> Eric Lee Green <eric@linux-hw.com> writes:
> > On Thu, 5 Nov 1998, Constantin Teodorescu wrote:
> >> must manually add -lcrypt in Makefile into src/interfaces/libpgtcl
> 
> > I came across this problem too today (I am currently packaging 6.4 into an
> > RPM for Red Hat 5.2). I have generated a patch file which fixes the lack
> > of -lcrypt in a number of places for glibc systems.
> 
> I believe this is already fixed in the current Postgres sources.  Rather
> than generating your own patch, would you verify that the current
> sources are fixed?

Hi, Tom. I'll see if I can do that, but be forewarned that I will not be
able to actually run the "current" Postgres sources in order to actually
test it. I am working on a database application, not on Postgres, and I
have to maintain a stable Postgres on my development machine. The reason
I'm working on the 6.4 RPM for Red Hat 5.2 is for deployment purposes, not
for testing purposes, though of course I'll test it before deploying it.
(I want the latest stable version possible for deployment, because once
deployed, it will be out there for a LONG time).

BTW, congrats to everybody on an excellent job. I examined several
databases, including some commercial ones, and for everything but speed
PostgreSQL measured up nicely. And speed wasn't THAT bad, certainly quite
acceptable for my purposes (and definitely a lot faster than it used to
be!). 

--
Eric Lee Green         eric@linux-hw.com     http://www.linux-hw.com/~eric
 "Linux represents a best-of-breed UNIX, that is trusted in mission
  critical applications..."   --  internal Microsoft memo


Mon, 7 Dec 1998 00:18:48 -0500 (EST)
Eric Lee Green <eric@linux-hw.com>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!!
On Sun, 6 Dec 1998, Thomas G. Lockhart wrote:
> > (I am currently packaging 6.4 into an RPM for Red Hat 5.2).
> 
> Is this an alternative to the RedHat packaging, or are you using the
> same layout and RPM setup that the RedHat 6.3 package had? Just curious;

I am using the same layout and RPM setup that the Red Hat 6.3 package
had, but I had to make a fair number of changes to their setup to make
6.4 work.  I think I probably ended up changing about 1/3rd of the
"action" portions of their .spec file, and the patch file I used to
make it install and link cleanly was generated from scratch. I am not
doing this for Red Hat Software, I am doing it because I need to
deploy 6.4 and doing it via RPM is the easiest way to deploy on Red
Hat. And perhaps more importantly, I'm doing it because it's there
:-).

> I've been thinking about trying to help with RPM support but would be
> happy that someone else is looking after it.

Well, if you have some suggestions I'm certainly not loathe to ignore them!
Right now, the only suggestion I can think of that I'd like to implement
would be to somehow figure out how to compile the Python module at the same
time that I compile the rest of it, but I'm not sure that Python is capable
yet of doing that Perl trick of compiling a module against the Perl .so 
link library without having the source code around. I'll have to check
with the Python gurus on that one. 

> Actually, we should also test with RH5.2 to make sure the date/time
> problem is still present, since it may have been due to a problem in
> glibc2 which has since been fixed. I'm only running RH4.2 and RH5.1 so
> can't directly test that myself...

Well, RH5.2 is still using glibc 2.0.7, albeit patched up about 10
more times since RH 5.1 :-). But sure, if you have a test case to run,
drop it in the mail to me and I'll be happy to run it. 

One thought: Do you have a TZ environment variable set on your RH5.1?
I don't remember whether glibc needs one or not (I know there were other
programs sensitive to the TZ environment variable under RH5.1, like BRU, 
but it is 12:30am here on the east coast and my mind is not as clear as
it should be :-). 

--
Eric Lee Green         eric@linux-hw.com     http://www.linux-hw.com/~eric
 "Linux represents a best-of-breed UNIX, that is trusted in mission
  critical applications..."   --  internal Microsoft memo


Mon, 7 Dec 1998 00:33:36 -0500 (EST)
Eric Lee Green <eric@linux-hw.com>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!!
Eric Lee Green <eric@linux-hw.com> writes:
> I have to maintain a stable Postgres on my development machine. The reason
> I'm working on the 6.4 RPM for Red Hat 5.2 is for deployment purposes, not
> for testing purposes, though of course I'll test it before deploying it.
> (I want the latest stable version possible for deployment, because once
> deployed, it will be out there for a LONG time).

Fair enough.  I have not been around Postgres very long and may be out
of line to make this remark, but here goes anyway: as far as I've seen,
the minor releases (6.3.2, 6.4.1, etc) are likely to be your best bet
for a solid install-and-forget version.  Major releases are for feature
additions, minor releases are made to clean up whatever got broken. 
We do the best we can to test major releases before they go out, but
things do slip through the cracks sometimes.

In short, unless you need something in the next few days, the upcoming
6.4.1 release is likely to be a better bet for an RPM distribution
than 6.4.

Perhaps one of the senior Postgres guys would like to expound on the
release philosophy?  The above is just what I've observed, it may
not be the official policy...

			regards, tom lane

Mon, 07 Dec 1998 00:54:32 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [INTERFACES] crypt not included when compiling libpgtcl !!!!!!! 
Microsoft SQL Server v6.5 have SQL92 join syntax.  I don't have the
standard in front of me but here's what I remember.

join_clause :
 table_name|view_name|join_clause [alias ][LEFT |RIGHT |CROSS ] JOIN
table_name|view_name|join_clause [alias ]ON join_tatements

The allows for neat little tricks like (hope you can follow it):
  SELECT a3.name, a3.address, a3.city, a4.state_abbrev, a6.postal_code,
a9.country_code
   FROM (
          (
            (states_list a5
             JOIN postal_codes a6 ON (a5.stateid = a6.stateid)
            ) a4
           RIGHT JOIN 
            (clients a1 
             LEFT JOIN addresses a2 ON (a1.clientid = a2.clientid AND
a2.prefered = 1)
            ) a3 ON (a3.stateid = a4.stateid)
          ) a7 
         LEFT JOIN 
          countries a8 ON (a7.countryid = a8.countryid)
        ) a9

I'm not sure if Microsoft implemented it but I believe that subselects
would be a great addition the above. 

I can load up a Microsoft SQL server for any testing you need done.  I'm
pretty sure that the Help files have a run down of their supported
syntax but I never trust Microsoft to stick to a standard (even their
own).




> -----Original Message-----
> From: Dan Gowin [mailto:DGowin@avantec.net]
> Sent: Friday, December 11, 1998 7:26 AM
> To: 'Thomas G. Lockhart'; PGSQL HACKERS (E-mail)
> Subject: RE: [HACKERS] JOIN syntax. Examples?
>
>
> I run three HP minicomputers and two Sun Ultra 3000 all with
> Oracle 7.3 and one with Oracle 8.0 .
>
> Send it to me.
>
> D.
>
>
> -----Original Message-----
> From: Thomas G. Lockhart [mailto:lockhart@alumni.caltech.edu]
> Sent: Friday, December 11, 1998 1:36 AM
> To: Postgres Hackers List
> Subject: [HACKERS] JOIN syntax. Examples?
>
>
> Well, I've started looking through my books for info on
> joins. The cross
> join was pretty easy:
>
> postgres=> select * from (a cross join b);
> i|   j|i| k
> -+----+-+--
> 1|10.1|1|-1
> 2|20.2|1|-1
> 4|    |1|-1
> <snip>
>
> which I've put into my copy of the parser.
>
> Does anyone have a commercial installation which has good support for
> SQL92 joins? I'd like to send some small test cases to verify that I
> understand what the behavior should be.
>
> Also, if anyone has worked with join syntax, outer joins
> especially, it
> would be great to get some test case contributions...
>
>                       - Tom
> 

Fri, 11 Dec 1998 12:28:01 -0600
"Jackson, DeJuan" <djackson@cpsgroup.com>
RE: [HACKERS] JOIN syntax. Examples?
> Microsoft SQL Server v6.5 have SQL92 join syntax.  I don't have the
> standard in front of me but here's what I remember.

OK, it's pretty clear that Oracle doesn't implement SQL92-syntax on
outer joins (unless they support it as an alternative; does anyone find
"OUTER JOIN" in the syntax docs?).

Let's assume that M$ may be close to standard, but given that they don't
bother following standards in other areas (WHERE x = NULL, etc) we can't
use them as a truth generator.

We are looking for a system which supports syntax like DeJuan gave:

SELECT * FROM (A LEFT OUTER JOIN B USING (X));
or
SELECT * FROM (A LEFT OUTER JOIN B ON (A.X = B.X));

etc. if we are going to try for the SQL92 standard,

rather than the Oracle form:

SELECT * FROM A, B WHERE A.X = (+) B.X;

or the Informix form:

SELECT * FROM A, OUTER B WHERE A.X = B.X;
  (is the WHERE clause required here?)

Does anyone have a non-M$ RDBMS which implements SQL92 joins?

otoh, any system which can test the results of a query, even if the
query needs to be translated first, has some benefit. As/if I progress
I'll take some of you up on the offer to run queries.

                     - Tom

Fri, 11 Dec 1998 22:11:28 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] JOIN syntax. Examples?
"Thomas G. Lockhart" wrote:
  >Does anyone have a non-M$ RDBMS which implements SQL92 joins?
 
The book "The Practical SQL Handbook", which is often recommended on
these lists, uses the syntax `*=' and `=*' for left and right outer
joins (page 211).  I think we ought to support this syntax as well,
since it will save new users from confusion.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "The spirit of the Lord GOD is upon me; because the 
      LORD hath anointed me to preach good tidings unto the 
      meek; he hath sent me to bind up the brokenhearted, to
      proclaim liberty to the captives, and the opening of 
      the prison to them that are bound."                 
                                        Isaiah 61:1 



Fri, 11 Dec 1998 22:35:18 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] JOIN syntax. Examples? 
On Fri, 11 Dec 1998, Oliver Elphick wrote:
> The book "The Practical SQL Handbook", which is often recommended on
> these lists, uses the syntax `*=' and `=*' for left and right outer
> joins (page 211).  I think we ought to support this syntax as well,
> since it will save new users from confusion.

'A Guide to The SQL Standard" (4th Ed.) seems to indicate that the MS
syntax is fairly close.

ISBN 0-201-96426-0

-- 
| Matthew N. Dodd  | 78 280Z | 75 164E | 84 245DL | FreeBSD/NetBSD/Sprite/VMS |
| winter@jurai.net |      This Space For Rent     | ix86,sparc,m68k,pmax,vax  |
| http://www.jurai.net/~winter | Are you k-rad elite enough for my webpage?   |


Fri, 11 Dec 1998 18:39:58 -0500 (EST)
"Matthew N. Dodd" <winter@jurai.net>
Re: [HACKERS] JOIN syntax. Examples? 
> The book "The Practical SQL Handbook", which is often recommended on
> these lists, uses the syntax `*=' and `=*' for left and right outer
> joins (page 211).  I think we ought to support this syntax as well,
> since it will save new users from confusion.

This one conflicts with Postgres' operator extensibility features, since
it would look just like a legal operator.

The two books I have at hand (besides my old Ingres docs) are A Guide to
the SQL Standard by Date and Darwen and Understanding the New SQL by
Melton and Simon. Both focus on SQL standard syntax, and neither mention
the various outer join syntaxes accepted by Oracle, Informix, or Sybase.

An explanation for the lack of standards compliance by the big three
probably involves the fact that they predate the standard by a
significant number of years.

                    - Tom

Sat, 12 Dec 1998 04:02:38 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] JOIN syntax. Examples?
> This is a Debian Linux bug report.
> 
> Size is still defined as unsigned int in 6.4.  Paul is suggesting that this
> be changed to size_t for compatibility with alpha.
> 
> Is that correct?

I am going to change Size to size_t only in the CURRENT tree, but leave
6.4.1/RELEASE tree alone, OK?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 12 Dec 1998 22:42:36 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] postgresql/c.h typedefs Size as 'unsigned int' (fwd)
Bruce Momjian wrote:
  >> This is a Debian Linux bug report.
  >> 
  >> Size is still defined as unsigned int in 6.4.  Paul is suggesting that thi
      >s
  >> be changed to size_t for compatibility with alpha.
  >> 
  >> Is that correct?
  >
  >I am going to change Size to size_t only in the CURRENT tree, but leave
  >6.4.1/RELEASE tree alone, OK?
 
OK, I will change it in the Debian release of 6.4.1 and let you know if it
causes any problems (not that I expect any).

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "For thou art my hope, O Lord GOD; thou art my trust 
      from my youth."               Psalms 71:5 



Sun, 13 Dec 1998 10:04:37 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] postgresql/c.h typedefs Size as 'unsigned int' (fwd) 
Thomas G. Lockhart wrote:
> 
> > The book "The Practical SQL Handbook", which is often recommended on
> > these lists, uses the syntax `*=' and `=*' for left and right outer
> > joins (page 211).  I think we ought to support this syntax as well,
> > since it will save new users from confusion.
> 
> This one conflicts with Postgres' operator extensibility features, since
> it would look just like a legal operator.

so does =

Could it be possible to extend the operator extensibility features 
to achieve the behaviour of outer/cross joins ?

> The two books I have at hand (besides my old Ingres docs) are A Guide to
> the SQL Standard by Date and Darwen and Understanding the New SQL by
> Melton and Simon. Both focus on SQL standard syntax, and neither mention
> the various outer join syntaxes accepted by Oracle, Informix, or Sybase.

Has anybody tried out DB2 ?

I have downloaded it (for linux) but have not yet tried it.
 
> An explanation for the lack of standards compliance by the big three
> probably involves the fact that they predate the standard by a
> significant number of years.

Not to mention that both =* and =(+) are more concise and easier to 
follow, at least for one with my headshape.

The standard is probably the 'worst common denominator' or something 
like that :(

-----------------
Hannu

Mon, 14 Dec 1998 21:10:52 +0200
Hannu Krosing <hannu@trust.ee>
Re: [HACKERS] JOIN syntax. Examples?
> > This one conflicts with Postgres' operator extensibility features, 
> > since it would look just like a legal operator.
> so does =

But in fact its usage for joins matches the typical usage elsewhere.

> Has anybody tried out DB2 ?
> I have downloaded it (for linux) but have not yet tried it.

Just downloaded it this morning (and afternoon, it's a thin pipe at home
for 60MB of files :) Have you looked at what it takes to do an
installation yet?

> Not to mention that both =* and =(+) are more concise and easier to
> follow, at least for one with my headshape.
> The standard is probably the 'worst common denominator' or something
> like that :(

DeJuan points out a major strength of the SQL92 syntax, which allows
multiple outer joins in the same query. One of my books shows an
example:

  select * from
    q1 full outer join q2 on (q1.id = q2.id)
       full outer join q3 on (coalesce(q1.id,q2.id)=q3.id)
       full outer join q4 on (coalesce(q1.id,q2.id,q3.id)=q4.id)

I suppose one can do something similar using a *= operator by using
parentheses? Not sure though...

                       - Tom

Tue, 15 Dec 1998 01:01:01 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] JOIN syntax. Examples?
> > Has anybody tried out DB2 ?
> > I have downloaded it (for linux) but have not yet tried it.
> Just downloaded it this morning (and afternoon, it's a thin pipe at 
> home for 60MB of files :) Have you looked at what it takes to do an
> installation yet?

Well, I'll have to save it for later, at least at home. It's glibc2
only. Also, the tar file has a bunch of rpms but also other files. Don't
know what's up with that...

                       - Tom

Tue, 15 Dec 1998 01:20:47 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] JOIN syntax. Examples?
Hi all,

>> > Has anybody tried out DB2 ?
>> > I have downloaded it (for linux) but have not yet tried it.
>> Just downloaded it this morning (and afternoon, it's a thin pipe at 
>> home for 60MB of files :) Have you looked at what it takes to do an
>> installation yet?

Could someone tell me please where I can download DB2?
Thanks,
-Jose'-



Tue, 15 Dec 1998 14:43:20 +0100
Sferacarta Software <sferac@bo.nettuno.it>
Re[2]: [HACKERS] JOIN syntax. Examples?
Hello Thomas,

venerdï¿½, 11 dicembre 98, you wrote:

>> Microsoft SQL Server v6.5 have SQL92 join syntax.  I don't have the
>> standard in front of me but here's what I remember.

TGL> OK, it's pretty clear that Oracle doesn't implement SQL92-syntax on
TGL> outer joins (unless they support it as an alternative; does anyone find
TGL> "OUTER JOIN" in the syntax docs?).

TGL> Let's assume that M$ may be close to standard, but given that they don't
TGL> bother following standards in other areas (WHERE x = NULL, etc) we can't
TGL> use them as a truth generator.

TGL> We are looking for a system which supports syntax like DeJuan gave:

TGL> SELECT * FROM (A LEFT OUTER JOIN B USING (X));
TGL> or
TGL> SELECT * FROM (A LEFT OUTER JOIN B ON (A.X = B.X));

TGL> etc. if we are going to try for the SQL92 standard,

TGL> rather than the Oracle form:

TGL> SELECT * FROM A, B WHERE A.X = (+) B.X;

TGL> or the Informix form:

TGL> SELECT * FROM A, OUTER B WHERE A.X = B.X;
TGL>   (is the WHERE clause required here?)

TGL> Does anyone have a non-M$ RDBMS which implements SQL92 joins?

Download OCELOT for Win32 at http://ourworld.compuserve.com/homepages/OCELOTSQL
their database implements SQL92 joins.

Their home page says:

Ocelot makes the only Database Management System (DBMS) that supports
the full ANSI / ISO SQL Standard (1992).
...
This is also the only place on the Net where you can find documentation
that explains and provides examples of the full SQL-92 standard. This is version 1.0.

I'm trying it, is very interesting but it is only for M$-win.

-Jose'-



Mon, 11 Jan 1999 15:45:51 +0100
Sferacarta Software <sferac@bo.nettuno.it>
Re[2]: [HACKERS] JOIN syntax. Examples?
> Download OCELOT for Win32...
> their database implements SQL92 joins.
> I'm trying it, is very interesting but it is only for M$-win.

My linux system doesn't know how to boot or run M$ stuff. Funny, but my
Mac before that didn't know how either :)

                      - Tom

Tue, 12 Jan 1999 03:05:16 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] JOIN syntax. Examples?
Hello Thomas,

martedï¿½, 12 gennaio 99, you wrote:

>> Download OCELOT for Win32...
>> their database implements SQL92 joins.
>> I'm trying it, is very interesting but it is only for M$-win.

TGL> My linux system doesn't know how to boot or run M$ stuff. Funny, but my
TGL> Mac before that didn't know how either :)

TGL>                       - Tom

You are a very puritan, I'm glad for you ;)
Unfortunately I can't be 100% puritan like you :(

I tried some joins on Ocelot...seems nice.
If you want something more significant I can try it for you.

table P:

PNO     PNAME   COLOR   WEIGHT  CITY
-----------------------------------------
P1      NUT     RED     12      LONDON
P4      SCREW   RED     14      LONDON
P2      BOLT    GREEN   17      PARIS

table SP:
SNO     PNO     QTY
-----------------------
S1      P1      300
S1      P2      200
S1      P2      200

SELECT DISTINCT SP.PNO, P.CITY FROM SP NATURAL JOIN P;   
PNO     CITY
---------------
P1      LONDON
P2      PARIS

SELECT DISTINCT SP.PNO, P.CITY FROM SP LEFT OUTER JOIN P USING (PNO); 
PNO     CITY

---------------
P1      LONDON
P2      PARIS

SELECT DISTINCT SP.PNO, P.CITY FROM SP LEFT OUTER JOIN P ON (P.PNO = sp.pno);
PNO     CITY
---------------
P1      LONDON
P2      ?
P2      PARIS

SELECT DISTINCT SP.PNO, P.CITY FROM SP RIGHT OUTER JOIN P ON (P.PNO = sp.pno);
PNO     CITY
---------------
P1      LONDON
P2      PARIS
?       PARIS

SELECT DISTINCT SP.PNO, P.CITY FROM SP FULL OUTER JOIN P ON (P.PNO = sp.pno);
PNO     CITY
---------------
P1      LONDON
P2      ?
P2      PARIS
?       PARIS

SELECT DISTINCT SP.PNO, P.CITY FROM SP INNER JOIN P ON (P.PNO = sp.pno);
PNO     CITY
---------------
P1      LONDON
P2      PARIS


-Jose'-



Tue, 12 Jan 1999 15:15:35 +0100
Sferacarta Software <sferac@bo.nettuno.it>
Re[2]: [HACKERS] JOIN syntax. Examples?

Tatsuo, Vadim, Oleg, Scrappy,

Many thanks for the response.

A couple of you weren't convinced that this
is a Postgres problem so let me try to clear
the water a little bit. Maybe the use of 
Apache and mod_perl is confusing the issue -
the point I was trying to make is that if 
there are 49+ concurrent postgres processes
on a normal machine (i.e. where kernel 
parameters are the defaults, etc.) the 
postmaster dies in a nasty way with 
potentially damaging results. 

Here's a case without Apache/mod_perl that
causes exactly the same behaviour. Simply
enter the following 49 times:

kandinsky:patrick> psql template1 &

Note that I tried to automate this without
success: 

perl -e 'for ( 1..49 ) { system("/usr/local/pgsql/bin/psql template1 &"); }'

The 49th attempt to initiate a connection 
fails:

Connection to database 'template1' failed.
pqReadData() -- backend closed the channel unexpectedly.
        This probably means the backend terminated abnormally before or while processing the request.

and the error_log says:

InitPostgres
IpcSemaphoreCreate: semget failed (No space left on device) key=5432017, num=16, permission=600
proc_exit(3) [#0]
shmem_exit(3) [#0]
exit(3)
/usr/local/pgsql/bin/postmaster: reaping dead processes...
/usr/local/pgsql/bin/postmaster: CleanupProc: pid 1521 exited with status 768
/usr/local/pgsql/bin/postmaster: CleanupProc: sending SIGUSR1 to process 1518
NOTICE:  Message from PostgreSQL backend:
        The Postmaster has informed me that some other backend died abnormally and possibly corrupted shared memory.
        I have rolled back the current transaction and am going to terminate your database system connection and exit.
        Please reconnect to the database system and repeat your query.

FATAL: s_lock(dfebe065) at spin.c:125, stuck spinlock. Aborting.

FATAL: s_lock(dfebe065) at spin.c:125, stuck spinlock. Aborting.


Even if there is a hard limit there is no way that 
Postgres should die in this spectacular fashion.
I wouldn't have said that it was unreasonable for
some large applications to peak at >48 processes
when using powerful hardware with plenty of RAM.

The other point is that even if one had 1 GB RAM,
Postgres won't scale beyond 48 processes, using
probably less than 100 MB of RAM. Would it be
possible to make the 'MaxBackendId' configurable
for those who have the resources?

I have reproduced this behaviour on both 
FreeBSD 2.2.8 and Intel Solaris 2.6 using
version 6.4.x of PostgreSQL.

I'll try to change some of the parameters
suggested and see how far I get but the bottom 
line is Postgres shouldn't be dying like this.

Let me know if you need any more info.

Cheers.



Patrick

-- 

#===============================#
\  KAN Design & Publishing Ltd  /
/  T: +44 (0)1223 511134        \
\  F: +44 (0)1223 571968        /
/  E: mailto:patrick@kan.co.uk  \ 
\  W: http://www.kan.co.uk      /
#===============================#

Fri, 29 Jan 1999 16:05:28 +0000
Patrick Verdon <patrick@kan.co.uk>
Re: [HACKERS] Postmaster dies with many child processes
	(spinlock/semget failed)
Patrick Verdon wrote:
> 
> 
> Even if there is a hard limit there is no way that
> Postgres should die in this spectacular fashion.

[snip]

> I have reproduced this behaviour on both
> FreeBSD 2.2.8 and Intel Solaris 2.6 using
> version 6.4.x of PostgreSQL.
> 
> I'll try to change some of the parameters
> suggested and see how far I get but the bottom
> line is Postgres shouldn't be dying like this.

We definitely need a chapter on tuning postgres in some of the manuals.

It should contain not only the parameters that one can change in
PostgreSQL - for either better response or for taking a larger load -
but also the ways one can tune the underlying OS, being it Linux, *BSD, 
Solaris or whatever.

Even commercial databases (at least Oracle) tend to rebuild kernel 
during installation (obsereved with Oracle 7.1 on Solaris)

When I once needed the info about setting shared memory limits on 
solaris I cried out here and got the example lines (I actually had them 
already copied from a macine where oracle was running)

But the same info, and possibly more(increasing limits for max 
files per process/globally, shared mem config, ... whatever else 
is needed) seems to be essential part of setting up a serious DB 
server on any system.

---------------
Hannu

Fri, 29 Jan 1999 19:02:43 +0200
Hannu Krosing <hannu@trust.ee>
Re: [HACKERS] Postmaster dies with many child processes
	(spinlock/semget failed)
Thomas Metz <tmetz@gsf.de> writes:
> SELECT DISTINCT ON id id, name FROM test ORDER BY name;
> [doesn't work as expected]

There have been related discussions before on pg-hackers mail list;
you might care to check the list archives.  The conclusion I recall
is that it's not real clear how the combination of SELECT DISTINCT
on one column and ORDER BY on another *should* work.  Postgres'
current behavior is clearly wrong IMHO, but there isn't a unique
definition of right behavior, because it's not clear which tuples
should get selected for the sort.

This "SELECT DISTINCT ON attribute" option strikes me as even more
bogus.  Where did we get that from --- is it in the SQL92 standard?
If you SELECT DISTINCT on a subset of the attributes to be returned,
then there's no unique definition of which values get returned in the
other columns.  In Thomas' example:

> Assuming the table TEST as follows:
> ID     NAME
> - -----------------
> 1      Alex
> 2      Oliver
> 1      Thomas
> 2      Fenella

> SELECT DISTINCT ON id id, name FROM test;
> produces:
> ID     NAME
> - -----------------
> 1      Alex
> 2      Oliver

There's no justifiable reason for preferring this output over
	1      Thomas
	2      Oliver
or
	1      Alex
	2      Fenella
or
	1      Thomas
	2      Fenella

Any of these are "DISTINCT ON id", but it's purely a matter of
happenstance table order and unspecified implementation choices which
one will appear.  Do we really have (or want) a statement with
inherently undefined behavior?

Anyway, to answer Thomas' question, the way SELECT DISTINCT is
implemented is that first there's a sort on the DISTINCT columns,
then there's a pass that eliminates adjacent duplicates (like the Unix
uniq(1) program).  In the current backend, doing an ORDER BY on another
column overrides the sorting on the DISTINCT columns, so when the
duplicate-eliminator runs it will fail to get rid of duplicates that
don't happen to appear consecutively in its input.  That's pretty
broken, but then the entire concept of combining these two options
doesn't seem well defined; the SELECT DISTINCT doesn't make any promises
about which tuples (with the same DISTINCT columns) it's going to pick,
therefore the result of ordering by some other column isn't clear.

If you're willing to live with poorly defined behavior, the fix
is fairly obvious: run the sort and uniq passes for the DISTINCT
columns, *then* run the sort on the ORDER BY columns --- which
will use whichever tuple the DISTINCT phase selected at random
out of each set with the same DISTINCT value.

I think the issue got put on the back burner last time in hopes that
some definition with consistent behavior would come up, but I haven't
seen any hope that there is one.

			regards, tom lane

Fri, 29 Jan 1999 12:02:52 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: SELECT DISTINCT ON ... ORDER BY ...
Patrick Verdon <patrick@kan.co.uk> writes:
> the point I was trying to make is that if there are 49+ concurrent
> postgres processes on a normal machine (i.e. where kernel parameters
> are the defaults, etc.) the postmaster dies in a nasty way with
> potentially damaging results.

Right.  It looks to me like your problem is running out of SysV
semaphores:

> IpcSemaphoreCreate: semget failed (No space left on device) key=5432017, num=16, permission=600

(read the man page for semget(2):
          [ENOSPC]       A semaphore identifier is to be created, but the
                         system-imposed limit on the maximum number of
                         allowed semaphore identifiers system wide would be
                         exceeded.
Old bad habit of Unix kernel programmers: re-use closest available error
code, rather than deal with the hassle of inventing a new kernel errno.)

You can increase the kernel's number-of-semaphores parameter (on my box,
both SEMMNI and SEMMNS need to be changed), but it'll probably take a
kernel rebuild to do it.

> Even if there is a hard limit there is no way that 
> Postgres should die in this spectacular fashion.

Well, running out of resources is something that it's hard to guarantee
recovery from.  Postgres is designed on the assumption that it's better
to try to prevent corruption of the database than to try to limp along
after a failure --- so the crash recovery behavior is exactly what you
see, mutual mass suicide of all surviving backends.  Restarting all your
clients is a pain in the neck, agreed, but would you rather have
database corruption spreading invisibly?

> The other point is that even if one had 1 GB RAM,
> Postgres won't scale beyond 48 processes, using
> probably less than 100 MB of RAM. Would it be
> possible to make the 'MaxBackendId' configurable
> for those who have the resources?

MaxBackendId is 64 by default, so that's not the limit you're hitting.

It should be easier to configure MaxBackendId --- probably it should be
an option to the configure script.  I've put this on my personal to-do
list.  (I don't think it's a good idea to have *no* upper limit, even
if it were easy to do in the code --- otherwise an unfriendly person
could run you out of memory by starting more and more clients.  If he
stops just short of exhausting swap space, then Postgres is perfectly
happy, but all the rest of your system starts misbehaving ... not cool.)

Another thing we ought to look at is changing the use of semaphores so
that Postgres uses a fixed number of semaphores, not a number that
increases as more and more backends are started.  Kernels are
traditionally configured with very low limits for the SysV IPC
resources, so having a big appetite for semaphores is a Bad Thing.

Right now it looks like we use a sema per backend to support spinlocks.
Perhaps we could just use a single sema that all backends block on when
waiting for a spinlock?  This might be marginally slower, or it might
not, but hopefully one is not blocking on spinlocks too often anyway.
Or, given that the system seems to contain only a small fixed number of
spinlocks, maybe a sema per spinlock would work best.

			regards, tom lane

Fri, 29 Jan 1999 13:13:54 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Postmaster dies with many child processes
	(spinlock/semget failed)
Tom Lane wrote:
  >Thomas Metz <tmetz@gsf.de> writes:
  >> SELECT DISTINCT ON id id, name FROM test ORDER BY name;
  >> [doesn't work as expected]
  >
  >There have been related discussions before on pg-hackers mail list;
  >you might care to check the list archives.  The conclusion I recall
  >is that it's not real clear how the combination of SELECT DISTINCT
  >on one column and ORDER BY on another *should* work.  Postgres'
  >current behavior is clearly wrong IMHO, but there isn't a unique
  >definition of right behavior, because it's not clear which tuples
  >should get selected for the sort.
  >
  >This "SELECT DISTINCT ON attribute" option strikes me as even more
  >bogus.  Where did we get that from --- is it in the SQL92 standard?

I looked through the standard yesterday and couldn't find it.  It doesn't
seem to be a useful extension, since it does nothing that you can't do
with GROUP BY and seems much less well defined.  For the moment I have
added a brief description to the reference documentation for SELECT.

  >If you SELECT DISTINCT on a subset of the attributes to be returned,
  >then there's no unique definition of which values get returned in the
  >other columns.  In Thomas' example:
 ...
  >Any of these are "DISTINCT ON id", but it's purely a matter of
  >happenstance table order and unspecified implementation choices which
  >one will appear.  Do we really have (or want) a statement with
  >inherently undefined behavior?

We have it; I suggest we don't want it!
 
-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "My son, if sinners entice thee, consent thou not."    
                            Proverbs 1:10 



Fri, 29 Jan 1999 18:24:03 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: SELECT DISTINCT ON ... ORDER BY ... 
> MaxBackendId is 64 by default, so that's not the limit you're hitting.
> 
> It should be easier to configure MaxBackendId --- probably it should be
> an option to the configure script.  I've put this on my personal to-do
> list.  (I don't think it's a good idea to have *no* upper limit, even

Or even better, MaxBackendId can be set at the run time such as
postmaster's option. Also, it would be nice if we could monitor number
of backends currently running. Maybe we should have a new protocol for
this kind of puropose?

BTW, as I pointed out before, PostgreSQL will have serious problem
once hitting the MaxBackendId. My patches I proposed for this seem
still under discussion. I think we should solve the problem in the
next release in whatever way, however.
---
Tatsuo Ishii

Sat, 30 Jan 1999 10:18:52 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] Postmaster dies with many child processes
	(spinlock/semget failed)
On Fri, 29 Jan 1999, Patrick Verdon wrote:

> 
> Tatsuo, Vadim, Oleg, Scrappy,
> 
> Many thanks for the response.
> 
> A couple of you weren't convinced that this
> is a Postgres problem so let me try to clear
> the water a little bit. Maybe the use of 
> Apache and mod_perl is confusing the issue -
> the point I was trying to make is that if 
> there are 49+ concurrent postgres processes
> on a normal machine (i.e. where kernel 
> parameters are the defaults, etc.) the 
> postmaster dies in a nasty way with 
> potentially damaging results. 
> 
> Here's a case without Apache/mod_perl that
> causes exactly the same behaviour. Simply
> enter the following 49 times:
> 
> kandinsky:patrick> psql template1 &
> 
> Note that I tried to automate this without
> success: 
> 
> perl -e 'for ( 1..49 ) { system("/usr/local/pgsql/bin/psql template1 &"); }'
> 
> The 49th attempt to initiate a connection 
> fails:
> 
> Connection to database 'template1' failed.
> pqReadData() -- backend closed the channel unexpectedly.
>         This probably means the backend terminated abnormally before or while processing the request.
> 
> and the error_log says:
> 
> InitPostgres
> IpcSemaphoreCreate: semget failed (No space left on device) key=5432017, num=16, permission=600


this error indicates taht you are out of semaphores...you have enough
configures to allow for 48 processes, but not the 49th...

> I have reproduced this behaviour on both 
> FreeBSD 2.2.8 and Intel Solaris 2.6 using
> version 6.4.x of PostgreSQL.

Both of them have "default" settings for semaphores...I don't recall what
they are, but the error you are seeing about IPCSemaphoreCreate indicates
that you are exceeding it...

> I'll try to change some of the parameters
> suggested and see how far I get but the bottom 
> line is Postgres shouldn't be dying like this.

PostgreSQL cannot allocate past what the operating sytem has hardcoded as
the max...maybe a more graceful exit should be in order, though?  Or is
that what you mean?

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Sat, 30 Jan 1999 04:05:50 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Postmaster dies with many child processes
	(spinlock/semget failed)
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> BTW, as I pointed out before, PostgreSQL will have serious problem
> once hitting the MaxBackendId. My patches I proposed for this seem
> still under discussion.

Not sure why that didn't get applied before, but I just put it in,
and verified that you can start exactly MaxBackendId backends
(assuming that you don't hit any kernel resource limits on the way).

BTW, we do recover quite gracefully from hitting MAXUPRC (kernel
limit on processes for one userid) :-).  But that's just because the
postmaster's initial fork() fails.  A failure any later than that
in backend startup will be treated as a backend crash ...

I agree with Hannu Krosing's remark that we really need some
documentation about kernel parameters that have to be checked when
setting up a non-toy database server.  I've personally run into
NFILES limits, for instance, with not all that many backends running.

			regards, tom lane

Sat, 30 Jan 1999 15:18:15 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Postmaster dies with many child processes
	(spinlock/semget failed)
I said:
> Another thing we ought to look at is changing the use of semaphores so
> that Postgres uses a fixed number of semaphores, not a number that
> increases as more and more backends are started.  Kernels are
> traditionally configured with very low limits for the SysV IPC
> resources, so having a big appetite for semaphores is a Bad Thing.

I've been looking into this issue today, and it looks possible but messy.

The source of the problem is the lock manager
(src/backend/storage/lmgr/proc.c), which wants to be able to wake up a
specific process that is blocked on a lock.  I had first thought that it
would be OK to wake up any one of the processes waiting for a lock, but
after looking at the lock manager that seems a bad idea --- considerable
thought has gone into the queuing order of waiting processes, and we
don't want to give that up.  So we need to preserve this ability.

The way it's currently done is that each extant backend has its own
SysV-style semaphore, and when you want to wake up a particular backend
you just V() its semaphore.  (BTW, the semaphores get allocated in
chunks of 16, so an out-of-semaphores condition will always occur when
trying to start the 16*N+1'th backend...)  This is simple and reliable
but fails if you want to have more backends than the kernel has SysV
semaphores.  Unfortunately kernels are usually configured with not
very many semaphores --- 64 or so is typical.  Also, running the system
down to nearly zero free semaphores is likely to cause problems for
other subsystems even if Postgres itself doesn't run out.

What seems practical to do instead is this:
* At postmaster startup, allocate a fixed number of semaphores for
  use by all child backends.  ("Fixed" can really mean "configurable",
  of course, but the point is we won't ask for more later.)
* The semaphores aren't dedicated to use by particular backends.
  Rather, when a backend needs to block, it finds a currently free
  semaphore and grabs it for the duration of its wait.  The number
  of the semaphore a backend is using to wait with would be recorded
  in its PROC struct, and we'd also need an array of per-sema data
  to keep track of free and in-use semaphores.
* This works with very little extra overhead until we have more
  simultaneously-blocked backends than we have semaphores.  When that
  happens (which we hope is really seldom), we overload semaphores ---
  that is, we use the same sema to block two or more backends.  Then
  the V() operation by the lock's releaser might wake the wrong backend.
  So, we need an extra field in the LOCK struct to identify the intended
  wake-ee.  When a backend is released in ProcSleep, it has to look at
  the lock it is waiting on to see if it is supposed to be wakened
  right now.  If not, it V()s its shared semaphore a second time (to
  release the intended wakee), then P()s the semaphore again to go
  back to sleep itself.  There probably has to be a delay in here,
  to ensure that the intended wakee gets woken and we don't have its
  bed-mates indefinitely trading wakeups among the wrong processes.
  This is why we don't want this scenario happening often.

I think this could be made to work, but it would be a delicate and
hard-to-test change in what is already pretty subtle code.

A considerably more straightforward approach is just to forget about
incremental allocation of semaphores and grab all we could need at
postmaster startup.  ("OK, Mac, you told me to allow up to N backends?
Fine, I'm going to grab N semaphores at startup, and if I can't get them
I won't play.")  This would force the DB admin to either reconfigure the
kernel or reduce MaxBackendId to something the kernel can support right
off the bat, rather than allowing the problem to lurk undetected until
too many clients are started simultaneously.  (Note there are still
potential gotchas with running out of processes, swap space, or file
table slots, so we wouldn't have really guaranteed that N backends can
be started safely.)

If we make MaxBackendId settable from a postmaster command-line switch
then this second approach is probably not too inconvenient, though it
surely isn't pretty.

Any thoughts about which way to jump?  I'm sort of inclined to take
the simpler approach myself...

			regards, tom lane

Sat, 30 Jan 1999 19:11:54 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Reducing sema usage (was Postmaster dies with many child processes)
I said:
> Any thoughts about which way to jump?  I'm sort of inclined to take
> the simpler approach myself...

A further thought: we could leave the semaphore management as-is,
and instead try to make running out of semaphores a less catastrophic
failure.  I'm thinking that the postmaster could be the one to try
to allocate more semaphores whenever there are none left, just before
trying to fork a new backend.  (The postmaster has access to the same
shared memory as the backends, right?  So no reason it couldn't do this.)
If the allocation fails, it can simply refuse the connection request,
rather than having to proceed as though we'd had a full-fledged backend
crash.  This only works because we can predict the number of semas
needed by an additional backend -- but we can: one.

			regards, tom lane

Sat, 30 Jan 1999 19:42:25 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Reducing sema usage (was Postmaster dies with many child
	processes)
> A considerably more straightforward approach is just to forget about
> incremental allocation of semaphores and grab all we could need at
> postmaster startup.  ("OK, Mac, you told me to allow up to N backends?
> Fine, I'm going to grab N semaphores at startup, and if I can't get them
> I won't play.")  This would force the DB admin to either reconfigure the
> kernel or reduce MaxBackendId to something the kernel can support right
> off the bat, rather than allowing the problem to lurk undetected until
> too many clients are started simultaneously.  (Note there are still
> potential gotchas with running out of processes, swap space, or file
> table slots, so we wouldn't have really guaranteed that N backends can
> be started safely.)
> 
> If we make MaxBackendId settable from a postmaster command-line switch
> then this second approach is probably not too inconvenient, though it
> surely isn't pretty.
> 
> Any thoughts about which way to jump?  I'm sort of inclined to take
> the simpler approach myself...

Semaphore are hard enough without overloading them.  I say just gram
them on startup.  They are cheap.  Many databases use semaphores for
every row/page they lock, and boy that can be a lot of semaphores.  We
are only getting a few.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 30 Jan 1999 20:42:56 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Reducing sema usage (was Postmaster dies with many
	child processes)
> I said:
> > Any thoughts about which way to jump?  I'm sort of inclined to take
> > the simpler approach myself...
> 
> A further thought: we could leave the semaphore management as-is,
> and instead try to make running out of semaphores a less catastrophic
> failure.  I'm thinking that the postmaster could be the one to try
> to allocate more semaphores whenever there are none left, just before
> trying to fork a new backend.  (The postmaster has access to the same
> shared memory as the backends, right?  So no reason it couldn't do this.)
> If the allocation fails, it can simply refuse the connection request,
> rather than having to proceed as though we'd had a full-fledged backend
> crash.  This only works because we can predict the number of semas
> needed by an additional backend -- but we can: one.

If they asked for 64 backends, we better be able go give them to them,
and not crash or fail under a load.  64 semaphores is nothing.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 30 Jan 1999 20:45:23 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: Reducing sema usage (was Postmaster dies with many
	child processes)
On Sat, 30 Jan 1999, Tom Lane wrote:

> I said:
> > Another thing we ought to look at is changing the use of semaphores so
> > that Postgres uses a fixed number of semaphores, not a number that
> > increases as more and more backends are started.  Kernels are
> > traditionally configured with very low limits for the SysV IPC
> > resources, so having a big appetite for semaphores is a Bad Thing.
> 
> I've been looking into this issue today, and it looks possible but messy.
> 
> The source of the problem is the lock manager
> (src/backend/storage/lmgr/proc.c), which wants to be able to wake up a
> specific process that is blocked on a lock.  I had first thought that it
> would be OK to wake up any one of the processes waiting for a lock, but
> after looking at the lock manager that seems a bad idea --- considerable
> thought has gone into the queuing order of waiting processes, and we
> don't want to give that up.  So we need to preserve this ability.
> 
> The way it's currently done is that each extant backend has its own
> SysV-style semaphore, and when you want to wake up a particular backend
> you just V() its semaphore.  (BTW, the semaphores get allocated in
> chunks of 16, so an out-of-semaphores condition will always occur when
> trying to start the 16*N+1'th backend...)  This is simple and reliable
> but fails if you want to have more backends than the kernel has SysV
> semaphores.  Unfortunately kernels are usually configured with not
> very many semaphores --- 64 or so is typical.  Also, running the system
> down to nearly zero free semaphores is likely to cause problems for
> other subsystems even if Postgres itself doesn't run out.
> 
> What seems practical to do instead is this:
> * At postmaster startup, allocate a fixed number of semaphores for
>   use by all child backends.  ("Fixed" can really mean "configurable",
>   of course, but the point is we won't ask for more later.)
> * The semaphores aren't dedicated to use by particular backends.
>   Rather, when a backend needs to block, it finds a currently free
>   semaphore and grabs it for the duration of its wait.  The number
>   of the semaphore a backend is using to wait with would be recorded
>   in its PROC struct, and we'd also need an array of per-sema data
>   to keep track of free and in-use semaphores.
> * This works with very little extra overhead until we have more
>   simultaneously-blocked backends than we have semaphores.  When that
>   happens (which we hope is really seldom), we overload semaphores ---
>   that is, we use the same sema to block two or more backends.  Then
>   the V() operation by the lock's releaser might wake the wrong backend.
>   So, we need an extra field in the LOCK struct to identify the intended
>   wake-ee.  When a backend is released in ProcSleep, it has to look at
>   the lock it is waiting on to see if it is supposed to be wakened
>   right now.  If not, it V()s its shared semaphore a second time (to
>   release the intended wakee), then P()s the semaphore again to go
>   back to sleep itself.  There probably has to be a delay in here,
>   to ensure that the intended wakee gets woken and we don't have its
>   bed-mates indefinitely trading wakeups among the wrong processes.
>   This is why we don't want this scenario happening often.
> 
> I think this could be made to work, but it would be a delicate and
> hard-to-test change in what is already pretty subtle code.
> 
> A considerably more straightforward approach is just to forget about
> incremental allocation of semaphores and grab all we could need at
> postmaster startup.  ("OK, Mac, you told me to allow up to N backends?
> Fine, I'm going to grab N semaphores at startup, and if I can't get them
> I won't play.")  This would force the DB admin to either reconfigure the
> kernel or reduce MaxBackendId to something the kernel can support right
> off the bat, rather than allowing the problem to lurk undetected until
> too many clients are started simultaneously.  (Note there are still
> potential gotchas with running out of processes, swap space, or file
> table slots, so we wouldn't have really guaranteed that N backends can
> be started safely.)
> 
> If we make MaxBackendId settable from a postmaster command-line switch
> then this second approach is probably not too inconvenient, though it
> surely isn't pretty.
> 
> Any thoughts about which way to jump?  I'm sort of inclined to take
> the simpler approach myself...

I'm inclined to agree...get rid of the 'hard coded' max, make it a
settable option on run time, and 'reserve the semaphores' on startup...

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Sat, 30 Jan 1999 21:52:42 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Reducing sema usage (was Postmaster dies with many
	child processes)
Bruce Momjian <maillist@candle.pha.pa.us> writes:
>> A further thought: we could leave the semaphore management as-is,
>> and instead try to make running out of semaphores a less catastrophic
>> failure.

> If they asked for 64 backends, we better be able go give them to them,
> and not crash or fail under a load.  64 semaphores is nothing.

That argument would be pretty convincing if pre-grabbing the semaphores
was sufficient to ensure we could start N backends, but of course it's
not sufficient.  The system could also run out of processes or file
descriptors, and I doubt that it's reasonable to grab all of those
instantly at postmaster startup.

The consensus seems clear not to go for the complex solution I described
at first.  But I'm still vacillating whether to do pre-reservation of
semaphores or just fix the postmaster to reject a connection cleanly if
no more can be gotten.  An advantage of the latter is that it would more
readily support on-the-fly changes of the max backend limit.  (Which I
am *not* proposing to support now; I only plan to make it settable at
postmaster startup; but someday we might want to change it on the fly.)

			regards, tom lane

Sun, 31 Jan 1999 13:00:23 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: Reducing sema usage (was Postmaster dies with many
	child processes)
Tom Lane wrote:
  >Bruce Momjian <maillist@candle.pha.pa.us> writes:
  >> If they asked for 64 backends, we better be able go give them to them,
  >> and not crash or fail under a load.  64 semaphores is nothing.
  >
  >That argument would be pretty convincing if pre-grabbing the semaphores
  >was sufficient to ensure we could start N backends, but of course it's
  >not sufficient.  The system could also run out of processes or file
  >descriptors, and I doubt that it's reasonable to grab all of those
  >instantly at postmaster startup.
 
The major problem at the moment is not that a new backend fails, but
that it brings down everything else with it.  How about having a new
backend set a one-byte flag in shared memory when it has
finished setting itself up? as long as the flag is unset, the
backend is still starting itself up, and a failure will not require
other backends to be brought down.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Jesus saith unto him, I am the way, the truth, and the
      life; no man cometh unto the Father, but by me."      
                                John 14:6 



Sun, 31 Jan 1999 21:33:35 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: Reducing sema usage (was Postmaster dies with many
	child processes)
"Oliver Elphick" <olly@lfix.co.uk> writes:
> The major problem at the moment is not that a new backend fails, but
> that it brings down everything else with it.

Agreed.

> How about having a new backend set a one-byte flag in shared memory
> when it has finished setting itself up? as long as the flag is unset,
> the backend is still starting itself up, and a failure will not
> require other backends to be brought down.

Not much win to be had there, I suspect.  The main problem is that as
soon as a new backend starts altering shared memory, you have potential
corruption issues to worry about if it goes down.  And there's not
really very much the new backend can do before it alters shared memory.
In fact, it can't do much of *anything* until it's made an entry for
itself in the lock manager's PROC array, because it cannot find out
anything interesting without locking shared structures.

Hmm.  If that's true, then the failure to get a sema would occur very
early in the new backend's lifetime, before it's had a chance to create
any trouble.  Maybe the very easiest solution to the sema issue is to
make the new backend send a failure report to its client and then
exit(0) instead of exit(1), so that the postmaster considers it a clean
exit rather than a crash...

			regards, tom lane

Sun, 31 Jan 1999 18:02:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: Reducing sema usage (was Postmaster dies with many
	child processes)
Tom Lane wrote:
> 
> I said:
> > Another thing we ought to look at is changing the use of semaphores so
> > that Postgres uses a fixed number of semaphores, not a number that
> > increases as more and more backends are started.  Kernels are
> > traditionally configured with very low limits for the SysV IPC
> > resources, so having a big appetite for semaphores is a Bad Thing.
> 
...
> 
> Any thoughts about which way to jump?  I'm sort of inclined to take
> the simpler approach myself...

Could we use sigpause (or something like this) to block
and some signal to wake up?

Vadim

Mon, 01 Feb 1999 09:42:57 +0700
Vadim Mikheev <vadim@krs.ru>
Re: [HACKERS] Reducing sema usage (was Postmaster dies with many
	child processes)
> Hmm.  If that's true, then the failure to get a sema would occur very
> early in the new backend's lifetime, before it's had a chance to 
> create any trouble.  Maybe the very easiest solution to the sema issue 
> is to make the new backend send a failure report to its client and 
> then exit(0) instead of exit(1), so that the postmaster considers it a 
> clean exit rather than a crash...

Sounds like the cleanest solution too. If it pans out, I like it...

                       - Tom

Mon, 01 Feb 1999 12:46:21 +0000
"Thomas G. Lockhart" <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: Reducing sema usage (was Postmaster dies with many
	child processes)
Oliver Elphick ha scritto:

> Tom Lane wrote:
>   >Thomas Metz <tmetz@gsf.de> writes:
>   >> SELECT DISTINCT ON id id, name FROM test ORDER BY name;
>   >> [doesn't work as expected]
>   >
>   >There have been related discussions before on pg-hackers mail list;
>   >you might care to check the list archives.  The conclusion I recall
>   >is that it's not real clear how the combination of SELECT DISTINCT
>   >on one column and ORDER BY on another *should* work.  Postgres'
>   >current behavior is clearly wrong IMHO, but there isn't a unique
>   >definition of right behavior, because it's not clear which tuples
>   >should get selected for the sort.
>   >
>   >This "SELECT DISTINCT ON attribute" option strikes me as even more
>   >bogus.  Where did we get that from --- is it in the SQL92 standard?
>
> I looked through the standard yesterday and couldn't find it.  It doesn't
> seem to be a useful extension, since it does nothing that you can't do
> with GROUP BY and seems much less well defined.  For the moment I have
> added a brief description to the reference documentation for SELECT.
>
>   >If you SELECT DISTINCT on a subset of the attributes to be returned,
>   >then there's no unique definition of which values get returned in the
>   >other columns.  In Thomas' example:
>  ...
>   >Any of these are "DISTINCT ON id", but it's purely a matter of
>   >happenstance table order and unspecified implementation choices which
>   >one will appear.  Do we really have (or want) a statement with
>   >inherently undefined behavior?
>
> We have it; I suggest we don't want it!

Yes, seems that SELECT DISTINCT ON is not part of SQL92 but it is very
interesting and I think it is something missing to Standard.
I don't know how to do the following, if we take off DISTINCT ON from
PostgreSQL:

db=> select distinct cognome, nome,via from membri where cap = '41010';
cognome|nome            |via
-------+----------------+--------------------------
FIORANI|ELISABETTA      |VIA PRETI PARTIGIANI, 63
FIORANI|GASTONE         |VIA PRETI PARTIGIANI, 63
FIORANI|MATTIA          |VIA PRETI PARTIGIANI, 63
FIORANI|SIMONE          |VIA PRETI PARTIGIANI, 63
GOZZI  |LILIANA         |VIA MAGNAGHI, 39
GOZZI  |MATTEO          |VIA MAGNAGHI, 39
RUSSO  |DAVIDE          |STRADA CORLETTO SUD, 194/1
RUSSO  |ELENA TERESA    |STRADA CORLETTO SUD, 194/1
RUSSO  |FORTUNATO       |STRADA CORLETTO SUD, 194/1
RUSSO  |MAURIZIO ANTONIO|STRADA CORLETTO SUD, 194/1
(10 rows)

db=> select distinct on cognome cognome, nome,via from membri where cap =
'41010';
cognome|nome            |via
-------+----------------+--------------------------
FIORANI|GASTONE         |VIA PRETI PARTIGIANI, 63
GOZZI  |LILIANA         |VIA MAGNAGHI, 39
RUSSO  |MAURIZIO ANTONIO|STRADA CORLETTO SUD, 194/1
(3 rows)



 
Oliver Elphick ha scritto:
Tom Lane wrote:
  >Thomas Metz <tmetz@gsf.de> writes:
  >> SELECT DISTINCT ON id id, name FROM test ORDER BY name;
  >> [doesn't work as expected]
  >
  >There have been related discussions before on pg-hackers mail
list;
  >you might care to check the list archives.  The conclusion
I recall
  >is that it's not real clear how the combination of SELECT DISTINCT
  >on one column and ORDER BY on another *should* work. 
Postgres'
  >current behavior is clearly wrong IMHO, but there isn't a unique
  >definition of right behavior, because it's not clear which
tuples
  >should get selected for the sort.
  >
  >This "SELECT DISTINCT ON attribute" option strikes me as even
more
  >bogus.  Where did we get that from --- is it in the SQL92
standard?
I looked through the standard yesterday and couldn't find it. 
It doesn't
seem to be a useful extension, since it does nothing that you can't
do
with GROUP BY and seems much less well defined.  For the moment
I have
added a brief description to the reference documentation for SELECT.
  >If you SELECT DISTINCT on a subset of the attributes to be returned,
  >then there's no unique definition of which values get returned
in the
  >other columns.  In Thomas' example:
 ...
  >Any of these are "DISTINCT ON id", but it's purely a matter
of
  >happenstance table order and unspecified implementation choices
which
  >one will appear.  Do we really have (or want) a statement
with
  >inherently undefined behavior?
We have it; I suggest we don't want it!
Yes, seems that SELECT DISTINCT ON is not part of SQL92 but it is very
interesting and I think it is something missing to Standard.
I don't know how to do the following, if we take off DISTINCT ON
from PostgreSQL:
db=> select distinct cognome, nome,via from membri where cap = '41010';
cognome|nome           
|via
-------+----------------+--------------------------
FIORANI|ELISABETTA      |VIA PRETI PARTIGIANI,
63
FIORANI|GASTONE        
|VIA PRETI PARTIGIANI, 63
FIORANI|MATTIA         
|VIA PRETI PARTIGIANI, 63
FIORANI|SIMONE         
|VIA PRETI PARTIGIANI, 63
GOZZI  |LILIANA        
|VIA MAGNAGHI, 39
GOZZI  |MATTEO         
|VIA MAGNAGHI, 39
RUSSO  |DAVIDE         
|STRADA CORLETTO SUD, 194/1
RUSSO  |ELENA TERESA    |STRADA CORLETTO SUD,
194/1
RUSSO  |FORTUNATO       |STRADA
CORLETTO SUD, 194/1
RUSSO  |MAURIZIO ANTONIO|STRADA CORLETTO SUD, 194/1
(10 rows)
db=> select distinct on cognome cognome, nome,via from membri where
cap = '41010';
cognome|nome           
|via
-------+----------------+--------------------------
FIORANI|GASTONE        
|VIA PRETI PARTIGIANI, 63
GOZZI  |LILIANA        
|VIA MAGNAGHI, 39
RUSSO  |MAURIZIO ANTONIO|STRADA CORLETTO SUD, 194/1
(3 rows)
Mon, 01 Feb 1999 15:57:52 +0100
"jose' soares" <sferac@bo.nettuno.it>
Re: [SQL] Re: [HACKERS] Re: SELECT DISTINCT ON ... ORDER BY ...
"jose' soares" wrote:
  >Yes, seems that SELECT DISTINCT ON is not part of SQL92 but it is very
  >interesting and I think it is something missing to Standard.
  >I don't know how to do the following, if we take off DISTINCT ON from
  >PostgreSQL:
  >
  >db=> select distinct cognome, nome,via from membri where cap = '41010';
  >cognome|nome            |via
  >-------+----------------+--------------------------
  >FIORANI|ELISABETTA      |VIA PRETI PARTIGIANI, 63
  >FIORANI|GASTONE         |VIA PRETI PARTIGIANI, 63
  >FIORANI|MATTIA          |VIA PRETI PARTIGIANI, 63
  >FIORANI|SIMONE          |VIA PRETI PARTIGIANI, 63
  >GOZZI  |LILIANA         |VIA MAGNAGHI, 39
  >GOZZI  |MATTEO          |VIA MAGNAGHI, 39
  >RUSSO  |DAVIDE          |STRADA CORLETTO SUD, 194/1
  >RUSSO  |ELENA TERESA    |STRADA CORLETTO SUD, 194/1
  >RUSSO  |FORTUNATO       |STRADA CORLETTO SUD, 194/1
  >RUSSO  |MAURIZIO ANTONIO|STRADA CORLETTO SUD, 194/1
  >(10 rows)
  >
  >db=> select distinct on cognome cognome, nome,via from membri where cap =
  >'41010';
  >cognome|nome            |via
  >-------+----------------+--------------------------
  >FIORANI|GASTONE         |VIA PRETI PARTIGIANI, 63
  >GOZZI  |LILIANA         |VIA MAGNAGHI, 39
  >RUSSO  |MAURIZIO ANTONIO|STRADA CORLETTO SUD, 194/1
  >(3 rows)

This gives the same results:

junk=> select cognome, nome, via from membri where cap = '41010' group by cognome;
cognome|nome      |via                       
-------+----------+--------------------------
FIORANI|ELISABETTA|VIA PRETI PARTIGIANI, 63  
GOZZI  |LILIANA   |VIA MAGNAGHI, 39          
RUSSO  |DAVIDE    |STRADA CORLETTO SUD, 194/1
(3 rows)

The particular values returned for nome and via are different from yours
but the same as I get using DISTINCT ON.  Since nome and via are not
aggregated, the value returned for those columns is unpredictable and 
therefore not useful.  I think that it is actually a bug that you are
able to name them at all.

In fact, if you add an aggregate column to the column list, GROUP BY does
not then allow columns that are neither grouped nor aggregated:

junk=> select  cognome, nome,via, max(age) from membri where cap = '41010' group by cognome;
ERROR:  parser: illegal use of aggregates or non-group column in target list
junk=> select  cognome, max(age) from membri where cap = '41010' group by cognome;
cognome|max
-------+---
FIORANI| 54
GOZZI  | 76
RUSSO  | 45
(3 rows)

which definitely suggests that it is a bug to allow such fields when no
aggregate is specified.

DISTINCT ON fails with an aggregate, even if no other columns are named:

junk=> select distinct on cognome cognome, max(age) from membri where cap = '41010';
ERROR:  parser: illegal use of aggregates or non-group column in target list

which makes it even less useful!

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "And be not conformed to this world; but be ye  
      transformed by the renewing of your mind, that ye may 
      prove what is that good, and acceptable, and perfect, 
      will of God."             Romans 12:2 



Mon, 01 Feb 1999 16:40:37 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [SQL] Re: [HACKERS] Re: SELECT DISTINCT ON ... ORDER BY ... 
Oliver Elphick ha scritto:

> "jose' soares" wrote:
>   >Yes, seems that SELECT DISTINCT ON is not part of SQL92 but it is very
>   >interesting and I think it is something missing to Standard.
>   >I don't know how to do the following, if we take off DISTINCT ON from
>   >PostgreSQL:
>   >
>   >db=> select distinct cognome, nome,via from membri where cap = '41010';
>   >cognome|nome            |via
>   >-------+----------------+--------------------------
>   >FIORANI|ELISABETTA      |VIA PRETI PARTIGIANI, 63
>   >FIORANI|GASTONE         |VIA PRETI PARTIGIANI, 63
>   >FIORANI|MATTIA          |VIA PRETI PARTIGIANI, 63
>   >FIORANI|SIMONE          |VIA PRETI PARTIGIANI, 63
>   >GOZZI  |LILIANA         |VIA MAGNAGHI, 39
>   >GOZZI  |MATTEO          |VIA MAGNAGHI, 39
>   >RUSSO  |DAVIDE          |STRADA CORLETTO SUD, 194/1
>   >RUSSO  |ELENA TERESA    |STRADA CORLETTO SUD, 194/1
>   >RUSSO  |FORTUNATO       |STRADA CORLETTO SUD, 194/1
>   >RUSSO  |MAURIZIO ANTONIO|STRADA CORLETTO SUD, 194/1
>   >(10 rows)
>   >
>   >db=> select distinct on cognome cognome, nome,via from membri where cap =
>   >'41010';
>   >cognome|nome            |via
>   >-------+----------------+--------------------------
>   >FIORANI|GASTONE         |VIA PRETI PARTIGIANI, 63
>   >GOZZI  |LILIANA         |VIA MAGNAGHI, 39
>   >RUSSO  |MAURIZIO ANTONIO|STRADA CORLETTO SUD, 194/1
>   >(3 rows)
>
> This gives the same results:
>
> junk=> select cognome, nome, via from membri where cap = '41010' group by cognome;
> cognome|nome      |via
> -------+----------+--------------------------
> FIORANI|ELISABETTA|VIA PRETI PARTIGIANI, 63
> GOZZI  |LILIANA   |VIA MAGNAGHI, 39
> RUSSO  |DAVIDE    |STRADA CORLETTO SUD, 194/1

This is very interesting and useful, I thought it wasn't possible. Seems that standard allows
only the "order by" column(s)
and the aggregate function(s) on target list.
I tried the same query on Informix, also on Ocelot but it gives me an error.

On Informix and Ocelot
queries like:
   select  cognome, max(age) from membri where cap = '41010' group by cognome;
are allowed.
but
queries like:
   select cognome, nome, via from membri where cap = '41010' group by cognome;
aren't allowed.

-Jose'-



 
Oliver Elphick ha scritto:
"jose' soares" wrote:
  >Yes, seems that SELECT DISTINCT ON is not part of SQL92
but it is very
  >interesting and I think it is something missing to Standard.
  >I don't know how to do the following, if we take off DISTINCT
ON from
  >PostgreSQL:
  >
  >db=> select distinct cognome, nome,via from membri where
cap = '41010';
  >cognome|nome           
|via
  >-------+----------------+--------------------------
  >FIORANI|ELISABETTA      |VIA PRETI
PARTIGIANI, 63
  >FIORANI|GASTONE        
|VIA PRETI PARTIGIANI, 63
  >FIORANI|MATTIA         
|VIA PRETI PARTIGIANI, 63
  >FIORANI|SIMONE         
|VIA PRETI PARTIGIANI, 63
  >GOZZI  |LILIANA        
|VIA MAGNAGHI, 39
  >GOZZI  |MATTEO         
|VIA MAGNAGHI, 39
  >RUSSO  |DAVIDE         
|STRADA CORLETTO SUD, 194/1
  >RUSSO  |ELENA TERESA    |STRADA CORLETTO
SUD, 194/1
  >RUSSO  |FORTUNATO      
|STRADA CORLETTO SUD, 194/1
  >RUSSO  |MAURIZIO ANTONIO|STRADA CORLETTO SUD, 194/1
  >(10 rows)
  >
  >db=> select distinct on cognome cognome, nome,via from
membri where cap =
  >'41010';
  >cognome|nome           
|via
  >-------+----------------+--------------------------
  >FIORANI|GASTONE        
|VIA PRETI PARTIGIANI, 63
  >GOZZI  |LILIANA        
|VIA MAGNAGHI, 39
  >RUSSO  |MAURIZIO ANTONIO|STRADA CORLETTO SUD, 194/1
  >(3 rows)
This gives the same results:
junk=> select cognome, nome, via from membri where cap = '41010'
group by cognome;
cognome|nome      |via
-------+----------+--------------------------
FIORANI|ELISABETTA|VIA PRETI PARTIGIANI, 63
GOZZI  |LILIANA   |VIA MAGNAGHI, 39
RUSSO  |DAVIDE    |STRADA CORLETTO SUD, 194/1
This is very interesting and useful, I thought it wasn't possible. Seems
that standard allows only the "order by" column(s)
and the aggregate function(s) on target list.
I tried the same query on Informix, also on Ocelot but it gives me
an error.
On Informix and Ocelot
queries like:
   select  cognome, max(age) from membri where cap
= '41010' group by cognome;
are allowed.
but
queries like:
   select cognome, nome, via from membri where cap =
'41010' group by cognome;
aren't allowed.
-Jose'-
Tue, 02 Feb 1999 16:25:19 +0100
"jose' soares" <sferac@bo.nettuno.it>
Re: [SQL] Re: [HACKERS] Re: SELECT DISTINCT ON ... ORDER BY ...
At 17:25 +0200 on 02/02/1999, jose' soares wrote:


> This gives the same results:
>
> junk=> select cognome, nome, via from membri where cap = '41010'
> group by cognome;
> cognome|nomeï¿½ï¿½ï¿½ï¿½ï¿½ |via
> -------+----------+--------------------------
> FIORANI|ELISABETTA|VIA PRETI PARTIGIANI, 63
> GOZZIï¿½ |LILIANAï¿½ï¿½ |VIA MAGNAGHI, 39
> RUSSOï¿½ |DAVIDEï¿½ï¿½ï¿½ |STRADA CORLETTO SUD, 194/1
>
>  This is very interesting and useful, I thought it wasn't possible. Seems
>that standard allows only the "order by" column(s)
> and the aggregate function(s) on target list.
> I tried the same query on Informix, also on Ocelot but it gives me an error.

And with good reason, too. The above query has the same drawback as the
"select distinct on", which is: it does not fully specify which value
should be selected for the "nome" and "via" fields.

Thus, running this same query on a table that has the same data but was,
for example, filled in a different order, gives a different result. That's
bad, because order should not make a difference for output. Tables are
taken to be unordered sets.

If you want to have a representative of the "nome" and "via" fields, and it
doesn't matter which representative, then min(nome) or max(nome) should do
the trick. And this query (select cognome, min(nome), min(via)... group by
cognome) should give you the same result on all databases, no matter which
rows were inserted first.

If it was up to me, I wouldn't use the above form, and frankly, I am
surprised the Postgres allows this.

Herouth

--
Herouth Maoz, Internet developer.
Open University of Israel - Telem project
http://telem.openu.ac.il/~herutma



Tue, 2 Feb 1999 17:57:08 +0200
Herouth Maoz <herouth@oumail.openu.ac.il>
Re: [SQL] Re: [HACKERS] Re: SELECT DISTINCT ON ... ORDER BY ...
I get this error when I try to create a function using plpgsql:

ERROR:  Unrecognized language specified in a CREATE FUNCTION: 'plpgsql'.
Recognized languages are sql, C, internal and the created procedural
languages.

Do I need to specify another flag when I compile Postgresql?

Adam
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Si hoc legere scis nimium eruditionis habes.



Mon, 10 May 1999 16:32:09 -0400
"Adam H. Pendleton" <apendleton@vgsinc.com>
plpgsql error
Edit: /usr/src/pgsql/postgresql-6.4.2/src/pl/plpgsql/src/mklang.sql

Change: as '${exec_prefix}/lib/plpgsql.so'
to: as '/usr/local/pgsql/lib/plpgsql.so'

Then: psql your_db < mklang.sql

This should really be part of the documentation as I wasted two days on
this same problem a few weeks back.

Have fun

Andy

On Mon, 10 May 1999, Adam H. Pendleton wrote:

> I get this error when I try to create a function using plpgsql:
> 
> ERROR:  Unrecognized language specified in a CREATE FUNCTION: 'plpgsql'.
> Recognized languages are sql, C, internal and the created procedural
> languages.
> 
> Do I need to specify another flag when I compile Postgresql?
> 
> Adam
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Si hoc legere scis nimium eruditionis habes.
> 
> 
> 




Mon, 10 May 1999 16:05:22 -0500 (CDT)
Andy Lewis <alewis@themecca.net>
Re: [SQL] plpgsql error

Yes, this clearly looks broken.  In mklang.sql.in, @libdir@ is replaced
with ${exec_prefix} in mklang.sql.  Seems it should be something else.


> Edit: /usr/src/pgsql/postgresql-6.4.2/src/pl/plpgsql/src/mklang.sql
> 
> Change: as '${exec_prefix}/lib/plpgsql.so'
> to: as '/usr/local/pgsql/lib/plpgsql.so'
> 
> Then: psql your_db < mklang.sql
> 
> This should really be part of the documentation as I wasted two days on
> this same problem a few weeks back.
> 
> Have fun
> 
> Andy
> 
> On Mon, 10 May 1999, Adam H. Pendleton wrote:
> 
> > I get this error when I try to create a function using plpgsql:
> > 
> > ERROR:  Unrecognized language specified in a CREATE FUNCTION: 'plpgsql'.
> > Recognized languages are sql, C, internal and the created procedural
> > languages.
> > 
> > Do I need to specify another flag when I compile Postgresql?
> > 
> > Adam
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Si hoc legere scis nimium eruditionis habes.
> > 
> > 
> > 
> 
> 
> 
> 
> 


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 10 May 1999 17:38:16 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [SQL] plpgsql error
Bruce Momjian <maillist@candle.pha.pa.us> writes:
> Yes, this clearly looks broken.  In mklang.sql.in, @libdir@ is replaced
> with ${exec_prefix} in mklang.sql.  Seems it should be something else.

Oh ... OK, that looks like a garden-variety configure bug (one too many
levels of quoting, or some such).  I can look at this if no one else
beats me to it.

			regards, tom lane

Mon, 10 May 1999 17:55:23 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error 
> Bruce Momjian <maillist@candle.pha.pa.us> writes:
> > Yes, this clearly looks broken.  In mklang.sql.in, @libdir@ is replaced
> > with ${exec_prefix} in mklang.sql.  Seems it should be something else.
> 
> Oh ... OK, that looks like a garden-variety configure bug (one too many
> levels of quoting, or some such).  I can look at this if no one else
> beats me to it.

It replaces @libdir@ with ${exec_prefix}/lib.  It appears the
configure code expects the replacement to occour in a Makefile, so
${exec_prefix} can be replaced in Makefile.global.  However,
$exec_prefix is not in Makefile.global, so maybe it is just a problem
with configure that $exec_prefix is replace before @libdir@, and
libdir's that contain exec_prefix have a problem.

However, it appears the default value of libdir contains exec_prefix, so
you would think they would have found such a problem themselves in
testing.

I am confused.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 10 May 1999 20:22:20 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error
>
> Edit: /usr/src/pgsql/postgresql-6.4.2/src/pl/plpgsql/src/mklang.sql
>
> Change: as '${exec_prefix}/lib/plpgsql.so'
> to: as '/usr/local/pgsql/lib/plpgsql.so'
>
> Then: psql your_db < mklang.sql
>
> This should really be part of the documentation as I wasted two days on
> this same problem a few weeks back.

    And this became IMHO an FAQ. Should we avoid it by installing
    PL/pgSQL and PL/Tcl (if built) by default  in  the  template1
    database  during  intidb?   Installing  it in template1 would
    have the  effect  that  it  will  be  available  after  every
    createdb.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck@debis.com (Jan Wieck) #



Tue, 11 May 1999 09:35:27 +0200 (MET DST)
jwieck@debis.com (Jan Wieck)
Re: [SQL] plpgsql error
Bruce Momjian <maillist@candle.pha.pa.us> writes:
>> Oh ... OK, that looks like a garden-variety configure bug (one too many
>> levels of quoting, or some such).  I can look at this if no one else
>> beats me to it.

> It replaces @libdir@ with ${exec_prefix}/lib.  It appears the
> configure code expects the replacement to occour in a Makefile, so
> ${exec_prefix} can be replaced in Makefile.global.  However,
> $exec_prefix is not in Makefile.global, so maybe it is just a problem
> with configure that $exec_prefix is replace before @libdir@, and
> libdir's that contain exec_prefix have a problem.

configure is designed to generate makefiles that look like this:

	exec_prefix=/usr/local
	bindir=${exec_prefix}/bin
	libdir=${exec_prefix}/lib

with the notion that this will simplify after-the-fact hand tweaking
of install destinations in the makefile if you feed a need to do that.
So that's why libdir's default definition looks the way it does.

Now, that works OK in makefiles and in shell scripts, where the
reference to the exec_prefix variable can get expanded when the file
is read.  But it falls down for mklang.sql, where the value of libdir
is substituted into an SQL command --- Postgres ain't gonna expand the
variable reference.

What we need is to substitute a "fully expanded" version of libdir into
this file, instead of a version that might depend on other variables.

Any shell-scripting gurus on the list?  I thought this would be an easy
fix, but I'm having some difficulty getting the configure script to
produce a fully-expanded value for libdir.  Given a shell variable that
may contain $-references to other variables, the requirement is to
assign to a new variable an expanded value containing no $-references.
I tried
	expanded_libdir="$libdir"
but that just gets you an exact copy, no recursive expansion.  A few
other ideas didn't work either; the Bourne shell doesn't seem to want
to re-expand text it's already expanded.  Suggestions?

			regards, tom lane

Tue, 11 May 1999 10:18:16 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error 
> >
> > Edit: /usr/src/pgsql/postgresql-6.4.2/src/pl/plpgsql/src/mklang.sql
> >
> > Change: as '${exec_prefix}/lib/plpgsql.so'
> > to: as '/usr/local/pgsql/lib/plpgsql.so'
> >
> > Then: psql your_db < mklang.sql
> >
> > This should really be part of the documentation as I wasted two days on
> > this same problem a few weeks back.
> 
>     And this became IMHO an FAQ. Should we avoid it by installing
>     PL/pgSQL and PL/Tcl (if built) by default  in  the  template1
>     database  during  intidb?   Installing  it in template1 would
>     have the  effect  that  it  will  be  available  after  every
>     createdb.

Sure, why not?


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 11 May 1999 10:56:42 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [SQL] plpgsql error
> What we need is to substitute a "fully expanded" version of libdir into
> this file, instead of a version that might depend on other variables.
> 
> Any shell-scripting gurus on the list?  I thought this would be an easy
> fix, but I'm having some difficulty getting the configure script to
> produce a fully-expanded value for libdir.  Given a shell variable that
> may contain $-references to other variables, the requirement is to
> assign to a new variable an expanded value containing no $-references.
> I tried



> 	expanded_libdir="$libdir"
> but that just gets you an exact copy, no recursive expansion.  A few
> other ideas didn't work either; the Bourne shell doesn't seem to want
> to re-expand text it's already expanded.  Suggestions?

Please try:

	expanded_libdir="`eval echo $libdir`"

Then I assume you have to do a:

sed "s/@libdir@/$expanded_libdir/g" <mklang.sql.template >mklang.sql

I can take it if you commit what you have.  The one item I am not sure
about is having it generate mklang.sql when the configure values change.
When they run configure, I think we have to generate a new file, so the
Makefile can see the change in datestamp and generate a new mklang.sql. 
Sounds like we need mklang.template.in, mklang.template, and mklang.sql
and a rule in the makefile that mklang.sql depends on mklang.template.

You can complete it, or I will take a crack at it.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 11 May 1999 11:21:48 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error
Tom Lane wrote:
  >Any shell-scripting gurus on the list?  I thought this would be an easy
  >fix, but I'm having some difficulty getting the configure script to
  >produce a fully-expanded value for libdir.  Given a shell variable that
  >may contain $-references to other variables, the requirement is to
  >assign to a new variable an expanded value containing no $-references.
  >I tried
  >	expanded_libdir="$libdir"
  >but that just gets you an exact copy, no recursive expansion.  A few
  >other ideas didn't work either; the Bourne shell doesn't seem to want
  >to re-expand text it's already expanded.  Suggestions?
 
Use eval:

$ v1=DF_\$EIFFEL_GTK  
$ echo $v1
DF_$EIFFEL_GTK
$ v2=$v1
$ echo $v2
DF_$EIFFEL_GTK
$ eval v2=$v1
$ echo $v2
DF_/usr/lib/eiffel-gtk
$ 


but if it gets too complicated, you might have to change to Perl

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Search me, O God, and know my heart; try me, and know 
      my thoughts. And see if there be any wicked way in me,
      and lead me in the way everlasting."        
                  Psalms 139:23,24 



Tue, 11 May 1999 16:52:24 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: [SQL] plpgsql error 
"Oliver Elphick" <olly@lfix.co.uk> writes:
> Use eval:
> $ eval v2=$v1
> $ echo $v2
> DF_/usr/lib/eiffel-gtk

Looks good.  Thanks for the clue.

> but if it gets too complicated, you might have to change to Perl

If configure depended on Perl, you couldn't build Postgres at all
without having a working Perl installation... somehow I doubt that
would go over well...

			regards, tom lane

Tue, 11 May 1999 14:08:04 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error 
   Any shell-scripting gurus on the list?  I thought this would be an easy
   fix, but I'm having some difficulty getting the configure script to
   produce a fully-expanded value for libdir.  Given a shell variable that
   may contain $-references to other variables, the requirement is to
   assign to a new variable an expanded value containing no $-references.
   I tried
	   expanded_libdir="$libdir"
   but that just gets you an exact copy, no recursive expansion.  A few
   other ideas didn't work either; the Bourne shell doesn't seem to want
   to re-expand text it's already expanded.  Suggestions?

Isn't the correct solution to have the Makefile contain a rule that
creates the file from a template (e.g., with sed -e
's/@xxx@/${xxx}/g')?  That way make resolves the variable references
and you needn't worry about it.  You can have the rule depend on
something like Makefile or Makefile.global or wherever the relevant
variables are set so that if local tweaks are made the files get
remade automatically.

Cheers,
Brook



Tue, 11 May 1999 15:59:34 -0600 (MDT)
Brook Milligan <brook@trillium.NMSU.Edu>
Re: [HACKERS] Re: [SQL] plpgsql error
>    Any shell-scripting gurus on the list?  I thought this would be an easy
>    fix, but I'm having some difficulty getting the configure script to
>    produce a fully-expanded value for libdir.  Given a shell variable that
>    may contain $-references to other variables, the requirement is to
>    assign to a new variable an expanded value containing no $-references.
>    I tried
> 	   expanded_libdir="$libdir"
>    but that just gets you an exact copy, no recursive expansion.  A few
>    other ideas didn't work either; the Bourne shell doesn't seem to want
>    to re-expand text it's already expanded.  Suggestions?
> 
> Isn't the correct solution to have the Makefile contain a rule that
> creates the file from a template (e.g., with sed -e
> 's/@xxx@/${xxx}/g')?  That way make resolves the variable references
> and you needn't worry about it.  You can have the rule depend on
> something like Makefile or Makefile.global or wherever the relevant
> variables are set so that if local tweaks are made the files get
> remade automatically.

Yes, that is what we were saying.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 11 May 1999 18:04:40 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error
>> Isn't the correct solution to have the Makefile contain a rule that

> Yes, that is what we were saying.

The problem is simply a matter of failing to expand indirect references
in that substitution process.  I just committed a fix.

			regards, tom lane

Tue, 11 May 1999 19:01:25 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error 
Brook Milligan <brook@trillium.NMSU.Edu> writes:
> Isn't the correct solution to have the Makefile contain a rule that
> creates the file from a template (e.g., with sed -e
> 's/@xxx@/${xxx}/g')?  That way make resolves the variable references
> and you needn't worry about it.

(after further thought...)  Oh, right, I see what you're saying: don't
generate mklang.sql in configure at all, but let pl/plpgsql/src/Makefile
be responsible for it.  Yeah, that'd be a cleaner solution.  However,
what I just committed works ;-).  If you feel like improving it, be
my guest; I have other items on the to-do list...

			regards, tom lane

Tue, 11 May 1999 19:26:15 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error 
> Bruce Momjian <maillist@candle.pha.pa.us> writes:
> >> Oh ... OK, that looks like a garden-variety configure bug (one too many
> >> levels of quoting, or some such).  I can look at this if no one else
> >> beats me to it.
> 
> > It replaces @libdir@ with ${exec_prefix}/lib.  It appears the
> > configure code expects the replacement to occour in a Makefile, so
> > ${exec_prefix} can be replaced in Makefile.global.  However,
> > $exec_prefix is not in Makefile.global, so maybe it is just a problem
> > with configure that $exec_prefix is replace before @libdir@, and
> > libdir's that contain exec_prefix have a problem.
> 
> configure is designed to generate makefiles that look like this:
> 
> 	exec_prefix=/usr/local
> 	bindir=${exec_prefix}/bin
> 	libdir=${exec_prefix}/lib
> 
> with the notion that this will simplify after-the-fact hand tweaking
> of install destinations in the makefile if you feed a need to do that.
> So that's why libdir's default definition looks the way it does.

Tom, I like your fix in configure.in better than adding a silly
Makefile�rule.  Yours is much cleaner.  You just created an
expanded_libdir in configure.in and let that be expanded in the *.in
files.  Great.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 12 May 1999 00:39:11 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error
>
> Brook Milligan <brook@trillium.NMSU.Edu> writes:
> > Isn't the correct solution to have the Makefile contain a rule that
> > creates the file from a template (e.g., with sed -e
> > 's/@xxx@/${xxx}/g')?  That way make resolves the variable references
> > and you needn't worry about it.
>
> (after further thought...)  Oh, right, I see what you're saying: don't
> generate mklang.sql in configure at all, but let pl/plpgsql/src/Makefile
> be responsible for it.  Yeah, that'd be a cleaner solution.  However,
> what I just committed works ;-).  If you feel like improving it, be
> my guest; I have other items on the to-do list...

    I've  just  committed  a  little  change  to  initdb and it's
    Makefile. The initdb Makefile now expands  __DLSUFFIX__  into
    it  and  initdb uses $PGLIB/plpgsql__DLSUFFIX__ to test if it
    is there  and  then  runs  the  appropriate  queries  against
    template1. Same for PL/Tcl.

    If  anyone  agrees we can get rid of these mklang.sql scripts
    totally.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck@debis.com (Jan Wieck) #



Wed, 12 May 1999 12:37:24 +0200 (MET DST)
jwieck@debis.com (Jan Wieck)
Re: [HACKERS] Re: [SQL] plpgsql error
> >
> > Brook Milligan <brook@trillium.NMSU.Edu> writes:
> > > Isn't the correct solution to have the Makefile contain a rule that
> > > creates the file from a template (e.g., with sed -e
> > > 's/@xxx@/${xxx}/g')?  That way make resolves the variable references
> > > and you needn't worry about it.
> >
> > (after further thought...)  Oh, right, I see what you're saying: don't
> > generate mklang.sql in configure at all, but let pl/plpgsql/src/Makefile
> > be responsible for it.  Yeah, that'd be a cleaner solution.  However,
> > what I just committed works ;-).  If you feel like improving it, be
> > my guest; I have other items on the to-do list...
> 
>     I've  just  committed  a  little  change  to  initdb and it's
>     Makefile. The initdb Makefile now expands  __DLSUFFIX__  into
>     it  and  initdb uses $PGLIB/plpgsql__DLSUFFIX__ to test if it
>     is there  and  then  runs  the  appropriate  queries  against
>     template1. Same for PL/Tcl.
> 
>     If  anyone  agrees we can get rid of these mklang.sql scripts
>     totally.

Sure.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 12 May 1999 08:50:17 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error
   Tom, I like your fix in configure.in better than adding a silly
   Makefile�rule.  Yours is much cleaner.  You just created an
   expanded_libdir in configure.in and let that be expanded in the *.in
   files.  Great.

The problem with solutions like this is that we end up proliferating
expanded and unexpanded versions of the same variables; hence,
maintenance problems with coordinating the various uses of this
information.

We've already had this discussion with some of the other cases (the
perl or tcl interfaces come to mind but I'm not sure).  It would be
unfortunate to make configure that much more complex when we don't
really need to at all.  It seems we should be using Makefile rules to
do what they are best at: automatically generating files based on
known rules that depend on a specific database of local configuration
options; configure's role should simply be to create that database.

Cheers,
Brook



Wed, 12 May 1999 08:08:27 -0600 (MDT)
Brook Milligan <brook@trillium.NMSU.Edu>
Re: [HACKERS] Re: [SQL] plpgsql error
Bruce Momjian <maillist@candle.pha.pa.us> writes:
> Tom, I like your fix in configure.in better than adding a silly
> Makefile rule.  Yours is much cleaner.  You just created an
> expanded_libdir in configure.in and let that be expanded in the *.in
> files.

Not really --- did you see what I had to do to get the thing expanded
properly?  Ick...  Brook's approach would be cleaner.  But I don't want
to spend more time on it now.

			regards, tom lane

Wed, 12 May 1999 10:12:18 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error 
jwieck@debis.com (Jan Wieck) writes:
>     I've  just  committed  a  little  change  to  initdb and it's
>     Makefile. The initdb Makefile now expands  __DLSUFFIX__  into
>     it  and  initdb uses $PGLIB/plpgsql__DLSUFFIX__ to test if it
>     is there  and  then  runs  the  appropriate  queries  against
>     template1. Same for PL/Tcl.
>     If  anyone  agrees we can get rid of these mklang.sql scripts
>     totally.

Well, I hate to be a party-pooper, but I don't agree: I like having
the flexibility of installing plpgsql into just selected databases.
If it's automatically included into template1 then I no longer have
any choice in the matter.

Perhaps this could be driven by a configuration switch?

			regards, tom lane

Wed, 12 May 1999 10:22:37 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error 
>
> Bruce Momjian <maillist@candle.pha.pa.us> writes:
> > Tom, I like your fix in configure.in better than adding a silly
> > Makefile rule.  Yours is much cleaner.  You just created an
> > expanded_libdir in configure.in and let that be expanded in the *.in
> > files.
>
> Not really --- did you see what I had to do to get the thing expanded
> properly?  Ick...  Brook's approach would be cleaner.  But I don't want
> to spend more time on it now.

    And it's not required for PL/pgSQL or PL/Tcl any more. initdb
    now installs them in template1 (if their shared  objects  are
    installed  in  the  libdir),  so we can remove the mklang.sql
    scripts.  So concentrate on your other items.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck@debis.com (Jan Wieck) #



Wed, 12 May 1999 17:03:36 +0200 (MET DST)
jwieck@debis.com (Jan Wieck)
Re: [HACKERS] Re: [SQL] plpgsql error
>
> jwieck@debis.com (Jan Wieck) writes:
> >     I've  just  committed  a  little  change  to  initdb and it's
> >     Makefile. The initdb Makefile now expands  __DLSUFFIX__  into
> >     it  and  initdb uses $PGLIB/plpgsql__DLSUFFIX__ to test if it
> >     is there  and  then  runs  the  appropriate  queries  against
> >     template1. Same for PL/Tcl.
> >     If  anyone  agrees we can get rid of these mklang.sql scripts
> >     totally.
>
> Well, I hate to be a party-pooper, but I don't agree: I like having
> the flexibility of installing plpgsql into just selected databases.
> If it's automatically included into template1 then I no longer have
> any choice in the matter.
>
> Perhaps this could be driven by a configuration switch?

    Or maybe some switch to initdb and createdb?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck@debis.com (Jan Wieck) #



Wed, 12 May 1999 17:06:09 +0200 (MET DST)
jwieck@debis.com (Jan Wieck)
Re: [HACKERS] Re: [SQL] plpgsql error
jwieck@debis.com (Jan Wieck) writes:
>> Perhaps this could be driven by a configuration switch?

>     Or maybe some switch to initdb and createdb?

That's a good idea; it would save having to propagate the value out of
configure and into the places where it'd be needed.

			regards, tom lane

Wed, 12 May 1999 11:20:52 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [SQL] plpgsql error 
>
> jwieck@debis.com (Jan Wieck) writes:
> >> Perhaps this could be driven by a configuration switch?
>
> >     Or maybe some switch to initdb and createdb?
>
> That's a good idea; it would save having to propagate the value out of
> configure and into the places where it'd be needed.

    I've thought a little more about that one and I don't like it
    anymore. It's a bad idea that enabling a language in  such  a
    looser-friendly  way is only possible during db creation, not
    for existing databases.

    I'll remove that from initdb again and instead  add  two  new
    utilities  "installpl"  and  "removepl".  That  way it's also
    possible to automate the process in  scripts  but  it  is  as
    flexible as can.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck@debis.com (Jan Wieck) #



Wed, 12 May 1999 17:58:16 +0200 (MET DST)
jwieck@debis.com (Jan Wieck)
Re: [HACKERS] Re: [SQL] plpgsql error
Restoring a full dump from 6.4.2 into 6.5:

When the owner of a database does not have usesuper privilege in pg_shadow,
it is not possible to recreate items that were created with superuser
privilege, even if the superuser is running the script:

======== example ==============
\connect template1
select datdba into table tmp_pg_shadow       from pg_database where datname = 
'template1';
delete from pg_shadow where usesysid <> tmp_pg_shadow.datdba;
drop table tmp_pg_shadow;
copy pg_shadow from stdin;
...
user1   1005    t       t       f       t               \N
\.
\connect template1 user1
create database morejunk;
\connect morejunk user1
...
CREATE FUNCTION "plpgsql_call_handler" ( ) RETURNS opaque AS 
'/usr/lib/postgresql/lib/plpgsql.so' LANGUAGE 'C';
...

Result:
QUERY: CREATE FUNCTION "plpgsql_call_handler" ( ) RETURNS opaque AS 
'/usr/lib/postgresql/lib/plpgsql.so' LANGUAGE 'C';
ERROR:  Only users with Postgres superuser privilege are permitted to create a 
function in the 'C' language.  Others may use the 'sql' language or the 
created procedural languages.
======================

It would seem that there should be some command that operates with 
superuser privilege, whatever the nominal state indicated by the data.
Since the above script was being run by postgres, it should all have
been capable of being executed.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "There is a way that seems right to a man, but in the 
      end it leads to death."       
                                        Proverbs 16:25 



Tue, 08 Jun 1999 00:32:54 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Problem when reloading data from older version
"Oliver Elphick" <olly@lfix.co.uk> writes:
> Restoring a full dump from 6.4.2 into 6.5:
> When the owner of a database does not have usesuper privilege in pg_shadow,
> it is not possible to recreate items that were created with superuser
> privilege, even if the superuser is running the script:

> QUERY: CREATE FUNCTION "plpgsql_call_handler" ( ) RETURNS opaque AS 
> '/usr/lib/postgresql/lib/plpgsql.so' LANGUAGE 'C';
> ERROR:  Only users with Postgres superuser privilege are permitted to create
> a function in the 'C' language.  Others may use the 'sql' language or the 
> created procedural languages.

Hmm.  This seems wrong; if the function was created by the superuser
then it should have proowner set to the superuser, and pg_dump looks
like it does the right thing about reconnecting as the function owner
(assuming you used -z, which is now default but wasn't in 6.4.2...).

Is it possible that "plpgsql_call_handler" was somehow marked as being
owned by the database owner rather than the superuser?  If so, I'd think
that that is the real bug.

> It would seem that there should be some command that operates with 
> superuser privilege, whatever the nominal state indicated by the data.

I'd be worried about security holes if it's not designed very carefully...

> Since the above script was being run by postgres, it should all have
> been capable of being executed.

I wonder whether we need a notion of "effective" and "real" user ID,
such as most Unix systems have.  Then it'd be possible for the system
to know "I may be creating objects on behalf of user X, but I really
am the superuser" and apply protection checks appropriately.  This'd
be a much more elegant solution than \connect for pg_dump scripts,
since the whole script would run in a single superuser session and just
do a SET VARIABLE or something to indicate which user would be the owner
of created objects.

However, that's not going to happen for 6.5.  For a short-term fix, we
need to look at why pg_dump didn't reconnect as superuser before trying
to create that C function.

			regards, tom lane

Mon, 07 Jun 1999 20:13:21 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Problem when reloading data from older version 
Tom Lane wrote:
  >Hmm.  This seems wrong; if the function was created by the superuser
  >then it should have proowner set to the superuser, and pg_dump looks
  >like it does the right thing about reconnecting as the function owner
  >(assuming you used -z, which is now default but wasn't in 6.4.2...).
  
Ah... looking back, I see that I did not use -z.

Using -z, it works OK.

  >I wonder whether we need a notion of "effective" and "real" user ID,
  >such as most Unix systems have.  Then it'd be possible for the system
  >to know "I may be creating objects on behalf of user X, but I really
  >am the superuser" and apply protection checks appropriately.  This'd
  >be a much more elegant solution than \connect for pg_dump scripts,
  >since the whole script would run in a single superuser session and just
  >do a SET VARIABLE or something to indicate which user would be the owner
  >of created objects.
 
I definitely agree with that.  It's also needed in order to restrict
password manipulation of other users' passwords to the superuser alone.

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Fear not, for I am with thee; be not dismayed, 
      for I am thy God. I will strengthen thee and I will 
      help thee; yea, I will uphold thee with the right hand
      of my righteousness."         Isaiah 41:10 



Tue, 08 Jun 1999 07:01:48 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Problem when reloading data from older version 
> Tom Lane wrote:
>   >Hmm.  This seems wrong; if the function was created by the superuser
>   >then it should have proowner set to the superuser, and pg_dump looks
>   >like it does the right thing about reconnecting as the function owner
>   >(assuming you used -z, which is now default but wasn't in 6.4.2...).
>   
> Ah... looking back, I see that I did not use -z.
> 
> Using -z, it works OK.

-z is now default in 6.5.


> 
>   >I wonder whether we need a notion of "effective" and "real" user ID,
>   >such as most Unix systems have.  Then it'd be possible for the system
>   >to know "I may be creating objects on behalf of user X, but I really
>   >am the superuser" and apply protection checks appropriately.  This'd
>   >be a much more elegant solution than \connect for pg_dump scripts,
>   >since the whole script would run in a single superuser session and just
>   >do a SET VARIABLE or something to indicate which user would be the owner
>   >of created objects.
>  
> I definitely agree with that.  It's also needed in order to restrict
> password manipulation of other users' passwords to the superuser alone.
> 
> -- 
>       Vote against SPAM: http://www.politik-digital.de/spam/
>                  ========================================
> Oliver Elphick                                Oliver.Elphick@lfix.co.uk
> Isle of Wight                              http://www.lfix.co.uk/oliver
>                PGP key from public servers; key ID 32B8FAA1
>                  ========================================
>      "Fear not, for I am with thee; be not dismayed, 
>       for I am thy God. I will strengthen thee and I will 
>       help thee; yea, I will uphold thee with the right hand
>       of my righteousness."         Isaiah 41:10 
> 
> 
> 
> 


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 8 Jun 1999 11:40:18 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Problem when reloading data from older version
This patch should enable 6.5 to build on Motorola 68000 architecture.  It comes
from Roman Hodek <Roman.Hodek@informatik.uni-erlangen.de>.




      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "I love them that love me; and those that seek me early
      shall find me."           Proverbs 8:17
Thu, 10 Jun 1999 16:29:11 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Patch for m68k architecture 
Applied.  You man want to give us a fix for /template/.similar for that
platform, if needed to configure guesses the proper platform.

> This patch should enable 6.5 to build on Motorola 68000 architecture.  It comes
> from Roman Hodek <Roman.Hodek@informatik.uni-erlangen.de>.
> 
> 
Content-Description: ol

[Attachment, skipping...]

>       Vote against SPAM: http://www.politik-digital.de/spam/
>                  ========================================
> Oliver Elphick                                Oliver.Elphick@lfix.co.uk
> Isle of Wight                              http://www.lfix.co.uk/oliver
>                PGP key from public servers; key ID 32B8FAA1
>                  ========================================
>      "I love them that love me; and those that seek me early
>       shall find me."           Proverbs 8:17


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 10 Jun 1999 18:58:22 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [PORTS] Patch for m68k architecture
At 8:29 AM -0700 6/10/99, Oliver Elphick wrote:
>This patch should enable 6.5 to build on Motorola 68000 architecture.  It
>comes
>from Roman Hodek <Roman.Hodek@informatik.uni-erlangen.de>.

Has anyone compared the Linux/m68k patch with the NetBSD/m68k patch (which
I believe was already included in 6.5)?

Also I have been trying to cross-post some traffic on the
port-mac68k@NetBSD.org list to the PG-ports list and it hasn't been
appearing afaict.  Am I just not looking carefully enough or is something
screwy?

Signature failed Preliminary Design Review.
Feasibility of a new signature is currently being evaluated.
h.b.hotz@jpl.nasa.gov, or hbhotz@oxy.edu

Fri, 11 Jun 1999 10:59:28 -0700
"Henry B. Hotz" <hotz@jpl.nasa.gov>
Re: [PORTS] Patch for m68k architecture
>At 8:29 AM -0700 6/10/99, Oliver Elphick wrote:
>>This patch should enable 6.5 to build on Motorola 68000 architecture.  It
>>comes
>>from Roman Hodek <Roman.Hodek@informatik.uni-erlangen.de>.
>
>Has anyone compared the Linux/m68k patch with the NetBSD/m68k patch (which
>I believe was already included in 6.5)?

yes.

>Also I have been trying to cross-post some traffic on the
>port-mac68k@NetBSD.org list to the PG-ports list and it hasn't been
>appearing afaict.  Am I just not looking carefully enough or is something
>screwy?

I have tried 6.4beta4 on NetBSD 1.3.3/m68k. It failed while running
initdb:

Creating template database in /usr/local/pgsql/data/base/template1

FATAL: s_lock(001bbea3) at bufmgr.c:1992, stuck spinlock. Aborting.

FATAL: s_lock(001bbea3) at bufmgr.c:1992, stuck spinlock. Aborting.

Seems something really bad is going on...
--
Tatsuo Ishii

Sat, 12 Jun 1999 17:47:18 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] Re: [PORTS] Patch for m68k architecture 
>>At 8:29 AM -0700 6/10/99, Oliver Elphick wrote:
>>>This patch should enable 6.5 to build on Motorola 68000 architecture.  It
>>>comes
>>>from Roman Hodek <Roman.Hodek@informatik.uni-erlangen.de>.
>>
>>Has anyone compared the Linux/m68k patch with the NetBSD/m68k patch (which
>>I believe was already included in 6.5)?
>
>yes.
>
>>Also I have been trying to cross-post some traffic on the
>>port-mac68k@NetBSD.org list to the PG-ports list and it hasn't been
>>appearing afaict.  Am I just not looking carefully enough or is something
>>screwy?
>
>I have tried 6.4beta4 on NetBSD 1.3.3/m68k. It failed while running
>initdb:
>
>Creating template database in /usr/local/pgsql/data/base/template1
>
>FATAL: s_lock(001bbea3) at bufmgr.c:1992, stuck spinlock. Aborting.
>
>FATAL: s_lock(001bbea3) at bufmgr.c:1992, stuck spinlock. Aborting.
>
>Seems something really bad is going on...

I reverted back the patch for include/storage/s_lock.h and seems
NetBSD/m68k port begins to work again.

I think we should revert back the linux/m68k patches and leave them
for 6.5.1.  Objection?
--
Tatsuo Ishii


Sun, 13 Jun 1999 00:39:19 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] Re: [PORTS] Patch for m68k architecture 
Tatsuo Ishii wrote:
  >>>At 8:29 AM -0700 6/10/99, Oliver Elphick wrote:
  >>>>This patch should enable 6.5 to build on Motorola 68000 architecture. 
...
  >>Seems something really bad is going on...
  >
  >I reverted back the patch for include/storage/s_lock.h and seems
  >NetBSD/m68k port begins to work again.
  >
  >I think we should revert back the linux/m68k patches and leave them
  >for 6.5.1.  Objection?

That seems sensible; presumably no other current users are on linux_m68k
or this would have been sorted already.  I will keep it in the Debian
version where there can't be any conflict with NetBSD users.

It seems that the patch needs to depend not only on being m68k but also
on being linux.  What defined variable can we use to distinguish between
the two?

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "And he said unto Jesus, Lord, remember me when thou 
      comest into thy kingdom. And Jesus said unto him, 
      Verily I say unto thee, To day shalt thou be with me 
      in paradise."       Luke 23:42,43 



Sat, 12 Jun 1999 21:02:06 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: [PORTS] Patch for m68k architecture 
>  >I reverted back the patch for include/storage/s_lock.h and seems
>  >NetBSD/m68k port begins to work again.
>  >
>  >I think we should revert back the linux/m68k patches and leave them
>  >for 6.5.1.  Objection?
>
>That seems sensible; presumably no other current users are on linux_m68k
>or this would have been sorted already.  I will keep it in the Debian
>version where there can't be any conflict with NetBSD users.
>
>It seems that the patch needs to depend not only on being m68k but also
>on being linux.  What defined variable can we use to distinguish between
>the two?

I have changed 
#if defined(__mc68000__)
to:
#if defined(__mc68000__) && defined(__linux__)
in s_lock.h.
--
Tatsuo Ishii

Sun, 13 Jun 1999 09:10:17 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] Re: [PORTS] Patch for m68k architecture 
At 1:47 AM -0700 6/12/99, Tatsuo Ishii wrote:
>I have tried 6.4beta4 on NetBSD 1.3.3/m68k. It failed while running
>initdb:
>
>Creating template database in /usr/local/pgsql/data/base/template1
>
>FATAL: s_lock(001bbea3) at bufmgr.c:1992, stuck spinlock. Aborting.
>
>FATAL: s_lock(001bbea3) at bufmgr.c:1992, stuck spinlock. Aborting.
>
>Seems something really bad is going on...

That certainly seems bad enough, but I did not see that problem.  Are you
talking about way back when 6.4 was still in beta?  Or did you mean 6.5?

As I tried to post earlier: when I built 6.4.2 using the patches it built
fine and initdb worked.  Most regression tests seemed ok-ish, but one of
them noticed that 'now' - 'current' was more than 200 days.

I had a problem building 6.5, but I think it was related to my
configuration rather than to Postgres.

Signature failed Preliminary Design Review.
Feasibility of a new signature is currently being evaluated.
h.b.hotz@jpl.nasa.gov, or hbhotz@oxy.edu

Sun, 13 Jun 1999 17:39:23 -0700
"Henry B. Hotz" <hotz@jpl.nasa.gov>
Re: [HACKERS] Re: [PORTS] Patch for m68k architecture
At 8:39 AM -0700 6/12/99, Tatsuo Ishii wrote:
>I reverted back the patch for include/storage/s_lock.h and seems
>NetBSD/m68k port begins to work again.
>
>I think we should revert back the linux/m68k patches and leave them
>for 6.5.1.  Objection?

I would like to support linux/m68k, but on the Mac 68K platform the NetBSD
folks are more numerous.  Unless linux outnumbers NetBSD on some 68k
platforms other than Mac (and I think Amiga) I don't think you should break
a (mostly) working port in order to support a less widely used one.

Just my $0.02.

Signature failed Preliminary Design Review.
Feasibility of a new signature is currently being evaluated.
h.b.hotz@jpl.nasa.gov, or hbhotz@oxy.edu

Sun, 13 Jun 1999 17:50:50 -0700
"Henry B. Hotz" <hotz@jpl.nasa.gov>
Re: [HACKERS] Re: [PORTS] Patch for m68k architecture
>>I reverted back the patch for include/storage/s_lock.h and seems
>>NetBSD/m68k port begins to work again.
>>
>>I think we should revert back the linux/m68k patches and leave them
>>for 6.5.1.  Objection?
>
>I would like to support linux/m68k, but on the Mac 68K platform the NetBSD
>folks are more numerous.  Unless linux outnumbers NetBSD on some 68k
>platforms other than Mac (and I think Amiga) I don't think you should break
>a (mostly) working port in order to support a less widely used one.

I think the change I made in s_lock.h should make both NetBSD/m68k and 
Linux/m68k happy. Can some Linux/m68k folks confirm it?
--
Tatsuo Ishii

Mon, 14 Jun 1999 10:03:31 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] Re: [PORTS] Patch for m68k architecture 
The regression tests no longer seem to be using the "alternative" expected
files should they exist. I have run out of time looking for the cause, but
the story so far is in going from version 1.28-1.29 of regress.sh,
SYSTEM has gone from 
...   printf"%s-%s", $1, a[1] }'
to
...   printf"%s-%s", $portname, a[1] }'
which means an example of output has changed from
i386-netbsd
to
i386-unknown-netbsd1.4-netbsd

Now, portname comes from PORTNAME=${os} in configure, which it appears ought
to be set in my case to

  netbsd*)
        os=bsd need_tas=no
        case "$host_cpu" in
          powerpc) elf=yes ;;
        esac ;;

"bsd", so I would expect SYSTEM to be set to "bsd-netbsd" ?! which doesn't
seem right either...

Maybe "someone" could take another look?

Cheers,

Patrick


Mon, 14 Jun 1999 12:48:56 +0100 (BST)
"Patrick Welche" <prlw1@newn.cam.ac.uk>
regress.sh
>
> The regression tests no longer seem to be using the "alternative" expected
> files should they exist. I have run out of time looking for the cause, but
> the story so far is in going from version 1.28-1.29 of regress.sh,
> SYSTEM has gone from
> ...   printf"%s-%s", $1, a[1] }'
> to
> ...   printf"%s-%s", $portname, a[1] }'
> which means an example of output has changed from
> i386-netbsd
> to
> i386-unknown-netbsd1.4-netbsd
>
> Now, portname comes from PORTNAME=${os} in configure, which it appears ought
> to be set in my case to
>
>   netbsd*)
>         os=bsd need_tas=no
>         case "$host_cpu" in
>           powerpc) elf=yes ;;
>         esac ;;
>
> "bsd", so I would expect SYSTEM to be set to "bsd-netbsd" ?! which doesn't
> seem right either...
>
> Maybe "someone" could take another look?

    Ouch  -  looks  like  my  recent change made while adding the
    NUMERIC regression tests.

    Looking at the actual sources I wonder why it can  cause  any
    problems.  At the very beginning I've added

        portname=$1
        export portname
        shift

    That  variable  is  used  ONLY  ONCE  in  the awk line you're
    quoting above. Prior to my changes, $1 was directly  used  as
    argument  to  awk  and all remaining args ignored silently by
    regress.sh.

    Is it required that variables local in regress.sh have  upper
    case? If so, why?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #



Mon, 14 Jun 1999 15:40:12 +0200 (MET DST)
wieck@debis.com (Jan Wieck)
Re: [HACKERS] regress.sh
Jan Wieck wrote:
> 
>     Looking at the actual sources I wonder why it can  cause  any
>     problems.  At the very beginning I've added
> 
>         portname=$1
>         export portname
>         shift
> 
>     That  variable  is  used  ONLY  ONCE  in  the awk line you're
>     quoting above. Prior to my changes, $1 was directly  used  as
>     argument  to  awk  and all remaining args ignored silently by
>     regress.sh.

Ah! portname=$1 means take $1 command line argument that the script
was called by.

awk -F\- '{ split($3,a,/[0-9]/); printf"%s-%s", $1, a[1] }'

$1 here is the 1st variable from the line split by awk. ie., $1 in the
first case is "sh" syntax, $1 in second case is "awk" syntax.

So now that I know there is no intentional magic, we can go back successfully
with

39c39
< SYSTEM=`../../config.guess | awk -F\- '{ split($3,a,/[0-9]/); printf"%s-%s", $portname, a[1] }'`
---
> SYSTEM=`../../config.guess | awk -F\- '{ split($3,a,/[0-9]/); printf"%s-%s", $1, a[1] }'`

the only remaining query being:

*** expected/random.out Sun Aug 30 19:50:58 1998
--- results/random.out  Mon Jun 14 15:18:04 1999
***************
*** 19,23 ****
    WHERE random NOT BETWEEN 80 AND 120;
  random
  ------
! (0 rows)
  
--- 19,24 ----
    WHERE random NOT BETWEEN 80 AND 120;
  random
  ------
!    124
! (1 row)

?

Cheers,

Patrick

Mon, 14 Jun 1999 15:22:09 +0100 (BST)
"Patrick Welche" <prlw1@newn.cam.ac.uk>
Re: [HACKERS] regress.sh
wieck@debis.com (Jan Wieck) writes:
>     Is it required that variables local in regress.sh have  upper
>     case? If so, why?

Nope, you just plain broke it.  The only use of the script's $1
parameter is *above* where you inserted portname=$1 (the test to
see if on windows).

The $1 in the awk script is awk's own meaning of $1.  Since it is
inside single quotes, the shell doesn't substitute for it.

I strongly suggest patching this before 6.5 ...

			regards, tom lane

Mon, 14 Jun 1999 10:22:54 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] regress.sh 
> the only remaining query being:
> *** expected/random.out Sun Aug 30 19:50:58 1998
> --- results/random.out  Mon Jun 14 15:18:04 1999
> ***************
> *** 19,23 ****
>     WHERE random NOT BETWEEN 80 AND 120;
>   random
>   ------
> ! (0 rows)
> 
> --- 19,24 ----
>     WHERE random NOT BETWEEN 80 AND 120;
>   random
>   ------
> !    124
> ! (1 row)

Well, sometimes random is too random. I'll bet if you run again you
will see a different result; I'd hope that *usually* you will see the
hoped-for result. I didn't want to make the criteria too loose so that
we would miss real problems. But sometimes the test fails, even on my
machine.

                  - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Mon, 14 Jun 1999 14:54:51 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] regress.sh
>
> wieck@debis.com (Jan Wieck) writes:
> >     Is it required that variables local in regress.sh have  upper
> >     case? If so, why?
>
> Nope, you just plain broke it.  The only use of the script's $1
> parameter is *above* where you inserted portname=$1 (the test to
> see if on windows).

    Uhhhh - and there where times where I wrote awk scripts who's
    output got piped  into  'groff  -p'  to  produce  statistical
    reports  with  graphics  -  hurts  to  see  that  I'm no real
    programmer anymore :-(

>
> The $1 in the awk script is awk's own meaning of $1.  Since it is
> inside single quotes, the shell doesn't substitute for it.
>
> I strongly suggest patching this before 6.5 ...

    No comment other that "sorry".


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #



Mon, 14 Jun 1999 16:59:07 +0200 (MET DST)
wieck@debis.com (Jan Wieck)
Re: [HACKERS] regress.sh
Thomas Lockhart wrote:
> 
> Well, sometimes random is too random.

?! So all is alright then :)

Cheers,

Patrick

Mon, 14 Jun 1999 16:18:09 +0100 (BST)
"Patrick Welche" <prlw1@newn.cam.ac.uk>
Re: [HACKERS] regress.sh
Thomas Lockhart wrote:
  >> the only remaining query being:
  >> *** expected/random.out Sun Aug 30 19:50:58 1998
  >> --- results/random.out  Mon Jun 14 15:18:04 1999
  >> ***************
  >> *** 19,23 ****
  >>     WHERE random NOT BETWEEN 80 AND 120;
  >>   random
  >>   ------
  >> ! (0 rows)
  >> 
  >> --- 19,24 ----
  >>     WHERE random NOT BETWEEN 80 AND 120;
  >>   random
  >>   ------
  >> !    124
  >> ! (1 row)
  >
  >Well, sometimes random is too random. I'll bet if you run again you
  >will see a different result; I'd hope that *usually* you will see the
  >hoped-for result. I didn't want to make the criteria too loose so that
  >we would miss real problems. But sometimes the test fails, even on my
  >machine.
 
Every time I have run it I have had a 124 row. That's about 8 times.
(glibc2/linux/i386)

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "I beseech you therefore, brethren, by the mercies of 
      God, that ye present your bodies a living sacrifice, 
      holy, acceptable unto God, which is your reasonable 
      service."       Romans 12:1 



Mon, 14 Jun 1999 20:29:37 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] regress.sh 
I'm trying to do some Linux rpms for the v6.5 release, and want to
include more interfaces than RH has done in the past. The pygres
installation instructions make no sense to me. There is a mention of a
"make redhat", but no Makefile, and a mention of "building python",
which is already on my system. Can anyone lead me through what
*really* needs to be done? I'm guessing that README.linux is out of
date, but don't know enough to be sure...

                      - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Sun, 20 Jun 1999 04:34:24 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
pygres/python installation
Thomas Lockhart wrote:
  >I'm trying to do some Linux rpms for the v6.5 release, and want to
  >include more interfaces than RH has done in the past. The pygres
  >installation instructions make no sense to me. There is a mention of a
  >"make redhat", but no Makefile, and a mention of "building python",
  >which is already on my system. Can anyone lead me through what
  >*really* needs to be done? I'm guessing that README.linux is out of
  >date, but don't know enough to be sure...

I believe it is.

Here are the instructions I use for building Pygresql for the Debian
distribution, which I put together after asking for help on Debian lists:

build-python:
   cd src/interfaces/python && \
      cp /usr/lib/python1.5/config/Makefile.pre.in `pwd`
   cd src/interfaces/python && \
      echo *shared* > Setup
   cd src/interfaces/python && \
      echo _pg pgmodule.c -I../../include -I../libpq \
         -L../libpq -lpq -lcrypt \
         >> Setup
   cd src/interfaces/python && \
      $(MAKE) -f Makefile.pre.in boot
   cd src/interfaces/python && \
      $(MAKE)
   touch build-python


-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "There is therefore now no condemnation to them which 
      are in Christ Jesus, who walk not after the flesh, but
      after the Spirit."              Romans 8:1 



Sun, 20 Jun 1999 07:19:43 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] pygres/python installation 
I mentioned this the other day on another list. I want to reiterate it
here because I can't seem to get anywhere.

I create a temporary table
=> create temp table foo (bar text);
CREATE
=> insert into foo values ('hi');
ERROR:  pg_temp.29112.0: Permission denied.

This apparently happens if and only if the user that executes this has
pg_shadow.usecatupd = 'f'.

I have tried this with the 6.5.1 source rpm bundle, fresh after initdb and
also with a 6.5.0 tar ball installation -- same result. (both on RH Linux
5.2-ish)

A potential reason that this has gone unnoticed so far is that when you
create a user thus:
=> create user joe;
the usecatupd defaults to true (why?).

Also this does not have anything to do with superuser status, the ability
to create and use regular tables, the ability to create databases, the
datatypes in the temp table, any hba stuff, or anything else I could think
of.

Anyone got a clue?

Regards,
	Peter

-- 
Peter Eisentraut
PathWay Computing, Inc.


Fri, 30 Jul 1999 13:22:19 -0400 (EDT)
Peter Eisentraut <peter@pathwaynet.com>
Cannot insert into temp tables
> I mentioned this the other day on another list. I want to reiterate it
> here because I can't seem to get anywhere.
> 
> I create a temporary table
> => create temp table foo (bar text);
> CREATE
> => insert into foo values ('hi');
> ERROR:  pg_temp.29112.0: Permission denied.
> 
> This apparently happens if and only if the user that executes this has
> pg_shadow.usecatupd = 'f'.
> 
> I have tried this with the 6.5.1 source rpm bundle, fresh after initdb and
> also with a 6.5.0 tar ball installation -- same result. (both on RH Linux
> 5.2-ish)
> 
> A potential reason that this has gone unnoticed so far is that when you
> create a user thus:
> => create user joe;
> the usecatupd defaults to true (why?).
> 
> Also this does not have anything to do with superuser status, the ability
> to create and use regular tables, the ability to create databases, the
> datatypes in the temp table, any hba stuff, or anything else I could think
> of.

OK, you have good points.  usecatupd should not be set by default. 
Making changes to the system tables can mess things up for everyone. 
Initdb will give the postgres superuser permissions, but now createuser
and the SQL command CREATE USER will not give this permission.  Also, I
have fixed the code so temp tables, which are acutally named pg_temp,
can be updated by normal users without usecatupd permissions.

Attached is a patch.  I will apply it to the current tree.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

? src/log
? src/config.log
? src/config.cache
? src/config.status
? src/GNUmakefile
? src/Makefile.global
? src/Makefile.custom
? src/backend/fmgr.h
? src/backend/parse.h
? src/backend/postgres
? src/backend/global1.bki.source
? src/backend/local1_template1.bki.source
? src/backend/global1.description
? src/backend/local1_template1.description
? src/backend/bootstrap/bootparse.c
? src/backend/bootstrap/bootstrap_tokens.h
? src/backend/bootstrap/bootscanner.c
? src/backend/catalog/genbki.sh
? src/backend/catalog/global1.bki.source
? src/backend/catalog/global1.description
? src/backend/catalog/local1_template1.bki.source
? src/backend/catalog/local1_template1.description
? src/backend/port/Makefile
? src/backend/utils/Gen_fmgrtab.sh
? src/backend/utils/fmgr.h
? src/backend/utils/fmgrtab.c
? src/bin/cleardbdir/cleardbdir
? src/bin/createdb/createdb
? src/bin/createlang/createlang
? src/bin/createuser/createuser
? src/bin/destroydb/destroydb
? src/bin/destroylang/destroylang
? src/bin/destroyuser/destroyuser
? src/bin/initdb/initdb
? src/bin/initlocation/initlocation
? src/bin/ipcclean/ipcclean
? src/bin/pg_dump/Makefile
? src/bin/pg_dump/pg_dump
? src/bin/pg_id/pg_id
? src/bin/pg_passwd/pg_passwd
? src/bin/pg_version/Makefile
? src/bin/pg_version/pg_version
? src/bin/pgtclsh/mkMakefile.tcldefs.sh
? src/bin/pgtclsh/mkMakefile.tkdefs.sh
? src/bin/pgtclsh/Makefile.tkdefs
? src/bin/pgtclsh/Makefile.tcldefs
? src/bin/pgtclsh/pgtclsh
? src/bin/pgtclsh/pgtksh
? src/bin/psql/Makefile
? src/bin/psql/psql
? src/include/version.h
? src/include/config.h
? src/interfaces/ecpg/lib/Makefile
? src/interfaces/ecpg/lib/libecpg.so.3.0.0
? src/interfaces/ecpg/lib/libecpg.so.3.0.1
? src/interfaces/ecpg/preproc/ecpg
? src/interfaces/libpgtcl/Makefile
? src/interfaces/libpgtcl/libpgtcl.so.2.0
? src/interfaces/libpq/Makefile
? src/interfaces/libpq/libpq.so.2.0
? src/interfaces/libpq++/Makefile
? src/interfaces/libpq++/libpq++.so.3.0
? src/interfaces/odbc/GNUmakefile
? src/interfaces/odbc/Makefile.global
? src/lextest/lex.yy.c
? src/lextest/lextest
? src/pl/plpgsql/src/Makefile
? src/pl/plpgsql/src/mklang.sql
? src/pl/plpgsql/src/pl_gram.c
? src/pl/plpgsql/src/pl.tab.h
? src/pl/plpgsql/src/pl_scan.c
? src/pl/plpgsql/src/libplpgsql.so.1.0
? src/pl/tcl/mkMakefile.tcldefs.sh
? src/pl/tcl/Makefile.tcldefs
Index: src/backend/catalog/aclchk.c
===================================================================
RCS file: /usr/local/cvsroot/pgsql/src/backend/catalog/aclchk.c,v
retrieving revision 1.26
diff -c -r1.26 aclchk.c
*** src/backend/catalog/aclchk.c	1999/07/17 20:16:47	1.26
--- src/backend/catalog/aclchk.c	1999/07/30 17:58:38
***************
*** 392,397 ****
--- 392,398 ----
  	 */
  	if (((mode & ACL_WR) || (mode & ACL_AP)) &&
  		!allowSystemTableMods && IsSystemRelationName(relname) &&
+ 		strncmp(relname,"pg_temp.", strlen("pg_temp.")) != 0 &&
  		!((Form_pg_shadow) GETSTRUCT(tuple))->usecatupd)
  	{
  		elog(DEBUG, "pg_aclcheck: catalog update to \"%s\": permission denied",
Index: src/backend/commands/user.c
===================================================================
RCS file: /usr/local/cvsroot/pgsql/src/backend/commands/user.c,v
retrieving revision 1.32
diff -c -r1.32 user.c
*** src/backend/commands/user.c	1999/07/17 20:16:54	1.32
--- src/backend/commands/user.c	1999/07/30 17:58:38
***************
*** 169,175 ****
  	snprintf(sql, SQL_LENGTH,
  			 "insert into %s (usename,usesysid,usecreatedb,usetrace,"
  			 "usesuper,usecatupd,passwd,valuntil) "
! 			 "values('%s',%d,'%c','t','%c','t',%s%s%s,%s%s%s)",
  			 ShadowRelationName,
  			 stmt->user,
  			 max_id + 1,
--- 169,175 ----
  	snprintf(sql, SQL_LENGTH,
  			 "insert into %s (usename,usesysid,usecreatedb,usetrace,"
  			 "usesuper,usecatupd,passwd,valuntil) "
! 			 "values('%s',%d,'%c','f','%c','f',%s%s%s,%s%s%s)",
  			 ShadowRelationName,
  			 stmt->user,
  			 max_id + 1,
Index: src/bin/createuser/createuser.sh
===================================================================
RCS file: /usr/local/cvsroot/pgsql/src/bin/createuser/createuser.sh,v
retrieving revision 1.11
diff -c -r1.11 createuser.sh
*** src/bin/createuser/createuser.sh	1999/01/31 05:04:25	1.11
--- src/bin/createuser/createuser.sh	1999/07/30 17:58:45
***************
*** 218,224 ****
  QUERY="insert into pg_shadow \
          (usename, usesysid, usecreatedb, usetrace, usesuper, usecatupd) \
         values \
!          ('$NEWUSER', $SYSID, '$CANCREATE', 't', '$CANADDUSER','t')"
  
  RES=`$PSQL -c "$QUERY" template1`
  
--- 218,224 ----
  QUERY="insert into pg_shadow \
          (usename, usesysid, usecreatedb, usetrace, usesuper, usecatupd) \
         values \
!          ('$NEWUSER', $SYSID, '$CANCREATE', 'f', '$CANADDUSER','f')"
  
  RES=`$PSQL -c "$QUERY" template1`
Fri, 30 Jul 1999 14:08:09 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Cannot insert into temp tables
Bruce Momjian wrote:
...
  >OK, you have good points.  usecatupd should not be set by default. 
  >Making changes to the system tables can mess things up for everyone. 
  >Initdb will give the postgres superuser permissions, but now createuser
  >and the SQL command CREATE USER will not give this permission.  Also, I
  >have fixed the code so temp tables, which are acutally named pg_temp,
  >can be updated by normal users without usecatupd permissions.
  >
  >Attached is a patch.  I will apply it to the current tree.

Bruce, this change has some other implications.  I tested
the effect of the patch by altering the rights of my own account (setting
usecatupd to false).  I cannot now create other users: although usesuper is
true, the attempt to update pg_shadow with the new user's row fails:

  olly@linda$ createuser fred
  Enter user's postgres ID -> 999
  Is user "fred" allowed to create databases (y/n) n
  Is user "fred" a superuser? (y/n) n
  ERROR:  pg_shadow: Permission denied.
  createuser: fred was NOT added successfully

so I think your change needs to be extended to allow pg_shadow to be
updated when a user is created; in this case, usesuper should
override usecatupd.

On the other hand, a user with usecreatedb true is able to modify
pg_database outside the context of a create database command.  This also
seems to be undesirable.  I think that create user, alter user and create
database should work even though the user does not have usecatupd, but the
user should be able to change the affected tables only through those
commands and not by direct manipulation, unless he has usecatupd in
addition to other privileges.

I regret that I can only point out these problems rather than provide 
a fix...



-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Have not I commanded thee? Be strong and of a good 
      courage; be not afraid, neither be thou dismayed; for 
      the LORD thy God is with thee whithersoever thou 
      goest."                        Joshua 1:9 



Sat, 31 Jul 1999 18:26:22 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Cannot insert into temp tables 
I've posted a tarball of new man pages at

  ftp://postgresql.org/pub/doc/man.tar.gz

which were generated from the sgml sources used in the other docs.
There are a few new pages corresponding to applications like pgtclsh,
ipcclean, etc which were not documented in reference pages previously.

Also, the old man pages have been removed from the main branch of the
cvs repository (to take effect for the v6.6 release). All information
from those man pages appears somewhere else in the new docs; usually
in the reference pages but sometimes in a more suitable place.

Please let me know if you see any problems with these.

                     - Thomas

btw, the man tarball will also appear in a source distribution of
Postgres, much like the tarballs of html documentation. We may choose
to build the docs on the fly for a release, making the tarballs
superfluous, but I decided to do it this way to get us started...

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Tue, 10 Aug 1999 02:40:08 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
New man pages
> I've posted a tarball of new man pages at
> 
>   ftp://postgresql.org/pub/doc/man.tar.gz
> 
> which were generated from the sgml sources used in the other docs.
> There are a few new pages corresponding to applications like pgtclsh,
> ipcclean, etc which were not documented in reference pages previously.
> 
> Also, the old man pages have been removed from the main branch of the
> cvs repository (to take effect for the v6.6 release). All information
> from those man pages appears somewhere else in the new docs; usually
> in the reference pages but sometimes in a more suitable place.

Great.  Good job.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 9 Aug 1999 22:54:49 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [ANNOUNCE] New man pages
Bruce Momjian wrote:
> 
> > All information
> > from those man pages appears somewhere else in the new docs; usually
> > in the reference pages but sometimes in a more suitable place.
> Great.  Good job.

Thanks. So, at the moment we have man pages for section one (1) and
section ell (l). Is this the right section numbering for the future?
How did we choose the "ell"? Should we consider having some utilities
like initdb documented in, say, section eight (8)? How about having
sections like one-sql (1sql) or something similar?

It's easy to move things around, and I'm wondering if the current
scheme works for people. I don't have a strong opinion on this...

                   - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Tue, 10 Aug 1999 03:04:37 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [ANNOUNCE] New man pages
> Bruce Momjian wrote:
> > 
> > > All information
> > > from those man pages appears somewhere else in the new docs; usually
> > > in the reference pages but sometimes in a more suitable place.
> > Great.  Good job.
> 
> Thanks. So, at the moment we have man pages for section one (1) and
> section ell (l). Is this the right section numbering for the future?
> How did we choose the "ell"? Should we consider having some utilities

It is very confusing.  I think they chose 'l' for language.  Does
someone want to suggest a better section number.  Maybe just put them
all in 1.

> like initdb documented in, say, section eight (8)? How about having
> sections like one-sql (1sql) or something similar?

Oh, I get it.  Can everyone handle multi-character man sections?

> 
> It's easy to move things around, and I'm wondering if the current
> scheme works for people. I don't have a strong opinion on this...
> 

I would like to use existing sections, rather than do our own.  I found
I had to modify the man page search to look in a manl, and others may
have the same problem.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 9 Aug 1999 23:08:10 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [ANNOUNCE] New man pages
> Oh, I get it.  Can everyone handle multi-character man sections?

That is how, for example, the X system does their man pages. There are
sections "1x", etc. Except that now that I look on my RH linux system
they are squirreled away in /usr/X11/man/man1/, etc so I must have
seen that on another system. Perhaps my old Alpha boxes??

> I would like to use existing sections, rather than do our own.  I found
> I had to modify the man page search to look in a manl, and others may
> have the same problem.

Yes, that is a consideration. It is easy to automate adding a new
section (like "1sql" or "1p" or ??) for packages, but is something the
admin needs to remember to do on a from-source installation. 

otoh, it does eliminate the possibility of man page pollution if we
manage to have the same man page name as some other existing page.
*That* would be a bad thing. And in general adding ~75 man pages to
existing sections is a pretty big load...

                   - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Tue, 10 Aug 1999 03:21:06 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [ANNOUNCE] New man pages
> > Oh, I get it.  Can everyone handle multi-character man sections?
> 
> That is how, for example, the X system does their man pages. There are
> sections "1x", etc. Except that now that I look on my RH linux system
> they are squirreled away in /usr/X11/man/man1/, etc so I must have
> seen that on another system. Perhaps my old Alpha boxes??

SCO does that.  Section 1M.

> 
> > I would like to use existing sections, rather than do our own.  I found
> > I had to modify the man page search to look in a manl, and others may
> > have the same problem.
> 
> Yes, that is a consideration. It is easy to automate adding a new
> section (like "1sql" or "1p" or ??) for packages, but is something the
> admin needs to remember to do on a from-source installation. 
> 
> otoh, it does eliminate the possibility of man page pollution if we
> manage to have the same man page name as some other existing page.
> *That* would be a bad thing. And in general adding ~75 man pages to
> existing sections is a pretty big load...

Oh, good point.  How do we get around that.  I don't see another
existing section that looks appropriate for SQL commands.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 9 Aug 1999 23:29:06 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [ANNOUNCE] New man pages
Thomas Lockhart wrote:
  >> Oh, I get it.  Can everyone handle multi-character man sections?
  >
  >That is how, for example, the X system does their man pages. There are
  >sections "1x", etc. Except that now that I look on my RH linux system
  >they are squirreled away in /usr/X11/man/man1/, etc so I must have
  >seen that on another system. Perhaps my old Alpha boxes??
  
Pages from multi-character sections are stored in the directory for the
first character.  For instance: /usr/man/man7/select.7l.gz

  >> I would like to use existing sections, rather than do our own.  I found
  >> I had to modify the man page search to look in a manl, and others may
  >> have the same problem.
  
For Debian, I have relocated the SQL pages to section 7l and commands such
as psql and createuser go in section 1.  Policy requires me to use one of
the numbered sections (1-8), though I can use a suffix to ensure uniqueness.

On Debian GNU/Linux, the sections are:
1 User commands
2 System calls
3 Library routines
4 Devices
5 File formats
6 Games
7 Miscellaneous
8 System administration

...
  >otoh, it does eliminate the possibility of man page pollution if we
  >manage to have the same man page name as some other existing page.

As of course we do; for example, select is also in section 2.

  >*That* would be a bad thing. And in general adding ~75 man pages to
  >existing sections is a pretty big load...

I'm not sure that's much of a problem. These are the figures from my
system for /usr/man, /usr/share/man, /usr/X11R6/man and /usr/local/man
combined:

Section   Count
1          2258
2           236
3          6554
4            39
5           236
6            26
7           128
8           517

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "If ye abide in me, and my words abide in you, ye shall
      ask what ye will, and it shall be done unto you."     
                                      John 15:7 



Tue, 10 Aug 1999 12:31:04 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: [ANNOUNCE] New man pages 
> Pages from multi-character sections are stored in the directory for the
> first character.  For instance: /usr/man/man7/select.7l.gz

Oh! afaik that is one option; the man system in general could also
handle man7l/select.7.gz right? You would update /etc/man.config to
add, say, "7l" to the list of sections.

But is is against Debian policy to invent new directories for pages? I
see that my RH linux system actually does about the same as Debian;
there are some ".1x" files in the /usr/man/man1 directory.

>   >> I would like to use existing sections, rather than do our own.  I found
>   >> I had to modify the man page search to look in a manl, and others may
>   >> have the same problem.
> For Debian, I have relocated the SQL pages to section 7l and commands such
> as psql and createuser go in section 1.  Policy requires me to use one of
> the numbered sections (1-8), though I can use a suffix to ensure uniqueness.
> On Debian GNU/Linux, the sections are:
> 1 User commands
> 2 System calls
> 3 Library routines
> 4 Devices
> 5 File formats
> 6 Games
> 7 Miscellaneous
> 8 System administration

Same for Linux ("man 7 man" has a summary).

>   >otoh, it does eliminate the possibility of man page pollution if we
>   >manage to have the same man page name as some other existing page.
> As of course we do; for example, select is also in section 2.

A near miss, since we weren't likely to have chosen section 2 for
*our* select. But it does illustrate the risk.

>   >*That* would be a bad thing. And in general adding ~75 man pages to
>   >existing sections is a pretty big load...
> I'm not sure that's much of a problem. These are the figures from my
> system for /usr/man, /usr/share/man, /usr/X11R6/man and /usr/local/man
> combined:

Right.

So, do Oliver's conventions make sense for most platforms? istm that
they do. Would folks have problems with a mapping similar to what
Oliver uses? We would use section one (1) and section seven (7), with
a qualifier of ell (l) on each of the man page names. I won't do
anything about it right now, but would like to get a consensus now
that the subject has come up. Speak up now or forever hold your...

                     - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Tue, 10 Aug 1999 12:47:53 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: [ANNOUNCE] New man pages
Thomas Lockhart wrote:
  >> Pages from multi-character sections are stored in the directory for the
  >> first character.  For instance: /usr/man/man7/select.7l.gz
  >
  >Oh! afaik that is one option; the man system in general could also
  >handle man7l/select.7.gz right? You would update /etc/man.config to
  >add, say, "7l" to the list of sections.
  >
  >But is is against Debian policy to invent new directories for pages?

6.1 Manual pages 

You must install manual pages in nroff source form, in appropriate places
under /usr/share/man. You should only use sections 1 to 9 (see the FHS
                                                       ^^^
                            FHS only defines 1 to 8; this may be an error

for more details). You must not install a preformatted `cat page'. 

The FHS has a section (4.8.2) on manual pages, which we ought to follow
if possible: <http://www.pathname.com/fhs/2.0/fhs-4.8.2.html>


-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "If ye abide in me, and my words abide in you, ye shall
      ask what ye will, and it shall be done unto you."     
                                      John 15:7 



Tue, 10 Aug 1999 14:16:04 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: [ANNOUNCE] New man pages 
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> Oh, I get it.  Can everyone handle multi-character man sections?
>
> That is how, for example, the X system does their man pages. There are
> sections "1x", etc. Except that now that I look on my RH linux system
> they are squirreled away in /usr/X11/man/man1/, etc so I must have
> seen that on another system. Perhaps my old Alpha boxes??

HPUX, as usual, is off in left field somewhere: they use 1m for sysadmin
commands, but everything else just goes into the single-digit-named
subdirectories (man1, man3, etc).  There is no separate namespace for
section 3c vs. section 3m, for example --- all those man pages live in
man3.  And sections named by a bare letter don't work at all.  AFAICT
this section search logic is implemented by hardwired hacks in the guts
of man(1) --- there is no way to affect it with MANPATH, for example,
because MANPATH determines where the manual root directories are, not
which subdirectories get looked at.

Newer implementations of man(1) are probably cleaner, but I fear that
HPUX's may be representative of what you'll find on older Unixes.

I'd like to see us change away from putting SQL commands in section l
(ell), simply because that doesn't work on HPUX.  Something like 8l
or 8s would work a lot better for me.  However, I'm not sure which
major section to use --- there doesn't seem to be very much cross-
platform standardization about the meanings of the sections beyond 4.
On BSD, section 8 seems to contain admin programs (the stuff HPUX
keeps in 1m).  I don't see any sections on either my HPUX box or
a nearby BSD box that contain pages for individual keywords of
a programming language...

> otoh, it does eliminate the possibility of man page pollution if we
> manage to have the same man page name as some other existing page.
> *That* would be a bad thing. And in general adding ~75 man pages to
> existing sections is a pretty big load...

As long as we install into /usr/local/pgsql/man/man*, naming conflicts
with other packages aren't too big a deal --- there's no physical file
conflict, and people can just add or remove /usr/local/pgsql/man/ in
their MANPATH settings to see or not see Postgres manpages.

			regards, tom lane

Tue, 10 Aug 1999 10:27:29 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [ANNOUNCE] New man pages 
> I'd like to see us change away from putting SQL commands in section l
> (ell), simply because that doesn't work on HPUX.  Something like 8l
> or 8s would work a lot better for me.  However, I'm not sure which
> major section to use --- there doesn't seem to be very much cross-
> platform standardization about the meanings of the sections beyond 4.
> On BSD, section 8 seems to contain admin programs (the stuff HPUX
> keeps in 1m).  I don't see any sections on either my HPUX box or
> a nearby BSD box that contain pages for individual keywords of
> a programming language...

Oliver uses section 7 (miscellaneous...) for SQL commands, which seems
to be the right choice given the guidelines for Debian, RedHat, FHS,
and my imprecise impression of what is typical.

> > otoh, it does eliminate the possibility of man page pollution if we
> > manage to have the same man page name as some other existing page.
> > *That* would be a bad thing. And in general adding ~75 man pages to
> > existing sections is a pretty big load...
> As long as we install into /usr/local/pgsql/man/man*, naming conflicts
> with other packages aren't too big a deal --- there's no physical file
> conflict, and people can just add or remove /usr/local/pgsql/man/ in
> their MANPATH settings to see or not see Postgres manpages.

Yeah, but "we" don't always install into /usr/local/pgsql, though we
can suggest that as a possibility.

btw, on my Solaris boxes MANPATH is worse than useless; if you specify
it then none of the other paths mentioned in /etc/man.config (or
wherever that is on Solaris) get used. So you have to recreate all of
the default MANPATH settings in your environment variable. Of course,
now that I've whined about this perhaps someone knows a way around
this?

Is there any concern about using "l" (ell) for the section
discriminator?

So, I'll count you as not objecting to sections "1l" (one-ell) and
"7l" (seven-ell), ok?

                        - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Tue, 10 Aug 1999 14:44:21 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: [ANNOUNCE] New man pages
On Tue, Aug 10, 1999 at 12:47:53PM +0000, Thomas Lockhart wrote:
> 
> So, do Oliver's conventions make sense for most platforms? istm that
> they do. Would folks have problems with a mapping similar to what
> Oliver uses? We would use section one (1) and section seven (7), with
> a qualifier of ell (l) on each of the man page names. I won't do
> anything about it right now, but would like to get a consensus now
> that the subject has come up. Speak up now or forever hold your...
> 

Well, they make sense for _me_ but then, I run Debian and use Oliver's 
packages ;-) As to the general proposal: put them in the nubmered sections
witha unique suffix. I'd suggest, however, at least a two character suffix:
how about pg? or sql? (tcl/tk uses section 3 with foo.3tcl and bar.3tk)

Ross
-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005

Tue, 10 Aug 1999 09:44:22 -0500
"Ross J. Reedstrom" <reedstrm@wallace.ece.rice.edu>
Re: [HACKERS] Re: [ANNOUNCE] New man pages
   So, do Oliver's conventions make sense for most platforms? istm that
   they do. Would folks have problems with a mapping similar to what
   Oliver uses? We would use section one (1) and section seven (7), with
   a qualifier of ell (l) on each of the man page names. I won't do
   anything about it right now, but would like to get a consensus now
   that the subject has come up. Speak up now or forever hold your...

OK, I'll speak up. :)

This doesn't make too much sense to me based on my experience with
zillions of other man pages.

A quick look through all of my system-supplied, X11, and locally
installed man pages in section 1 shows none with any suffix other than
.1 or .1.gz.  What exactly is the point of a .1l or whatever?  Is it
just to avoid name collisions?  If so, why not use something more
meaningful, like .1sql?  Note that the downside of any "odd" suffix,
though, is that the man system will likely need reconfiguring so as to
recognize it.  This will add an extra installation step, one that
probably cannot be easily automated.  Perhaps a relevant question here
is, how likely is a name collision anyway?  Is it likely enough to
require reconfiguration of the man system for all users?

As far as sections go, I think the following conventions apply for
sections of relevance to PostgreSQL:

1 - most of the commands which comprise the user environment
3 - an overview of the library functions, their error returns and
    other common definitions and concepts
5 - documentation on binary and configuration file formats
8 - information related to system operation and maintenance

Thus, it seems that documentation for user commands, e.g., CREATE
TABLE, SELECT, UPDATE, ... should go into section 1, the API to
libpq/libpq++/libpgtcl/... should go into section 3, configuration
files like pg_hba.conf should go into section 5, and admin commands,
e.g., createdb, createuser, pg_dump, ... should go into section 8.

Cheers,
Brook

Tue, 10 Aug 1999 08:48:15 -0600 (MDT)
Brook Milligan <brook@biology.nmsu.edu>
Re: [HACKERS] Re: [ANNOUNCE] New man pages
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> On BSD, section 8 seems to contain admin programs (the stuff HPUX
>> keeps in 1m).  I don't see any sections on either my HPUX box or
>> a nearby BSD box that contain pages for individual keywords of
>> a programming language...

> Oliver uses section 7 (miscellaneous...) for SQL commands, which seems
> to be the right choice given the guidelines for Debian, RedHat, FHS,
> and my imprecise impression of what is typical.

7 works for me; there's not much there except a few kernel device
driver manpages on my box.

> Is there any concern about using "l" (ell) for the section
> discriminator?
> So, I'll count you as not objecting to sections "1l" (one-ell) and
> "7l" (seven-ell), ok?

How about "s" for "SQL"?  ell is too easily mistaken for one; we'd
have to put "no, it's not section eleven" in the FAQ ...

> btw, on my Solaris boxes MANPATH is worse than useless; if you specify
> it then none of the other paths mentioned in /etc/man.config (or
> wherever that is on Solaris) get used. So you have to recreate all of
> the default MANPATH settings in your environment variable. Of course,
> now that I've whined about this perhaps someone knows a way around
> this?

Er, why not "MANPATH=/usr/local/pgsql/man:$MANPATH" ?

			regards, tom lane

Tue, 10 Aug 1999 11:01:32 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [ANNOUNCE] New man pages 
Tom Lane wrote:
  >As long as we install into /usr/local/pgsql/man/man*, naming conflicts
  >with other packages aren't too big a deal --- there's no physical file
  >conflict, and people can just add or remove /usr/local/pgsql/man/ in
  >their MANPATH settings to see or not see Postgres manpages.

But do remember us poor distribution maintainers.  It makes life a lot
easier for us if upstream writers make an effort not to conflict with the
rest of the world!

If some local user or administrator puts PostgreSQL on his machine, it
may be appropriate to use /usr/local, but a PostgreSQL package installed as 
part of a distribution should never use it.

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "If ye abide in me, and my words abide in you, ye shall
      ask what ye will, and it shall be done unto you."     
                                      John 15:7 



Tue, 10 Aug 1999 16:04:12 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: [ANNOUNCE] New man pages 
"Ross J. Reedstrom" <reedstrm@wallace.ece.rice.edu> writes:
> I'd suggest, however, at least a two character suffix:

Doesn't work with the standard man(1) on either HPUX or SunOS...

			regards, tom lane

Tue, 10 Aug 1999 11:18:12 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [ANNOUNCE] New man pages 
------- Start of forwarded message -------
   So, do Oliver's conventions make sense for most platforms? istm that
   they do. Would folks have problems with a mapping similar to what
   Oliver uses? We would use section one (1) and section seven (7), with
   a qualifier of ell (l) on each of the man page names. I won't do
   anything about it right now, but would like to get a consensus now
   that the subject has come up. Speak up now or forever hold your...

OK, I'll speak up. :)

This doesn't make too much sense to me based on my experience with
zillions of other man pages.

A quick look through all of my system-supplied, X11, and locally
installed man pages in section 1 shows none with any suffix other than
.1 or .1.gz.  What exactly is the point of a .1l or whatever?  Is it
just to avoid name collisions?  If so, why not use something more
meaningful, like .1sql?  Note that the downside of any "odd" suffix,
though, is that the man system will likely need reconfiguring so as to
recognize it.  This will add an extra installation step, one that
probably cannot be easily automated.  Perhaps a relevant question here
is, how likely is a name collision anyway?  Is it likely enough to
require reconfiguration of the man system for all users?

As far as sections go, I think the following conventions apply for
sections of relevance to PostgreSQL:

1 - most of the commands which comprise the user environment
3 - an overview of the library functions, their error returns and
    other common definitions and concepts
5 - documentation on binary and configuration file formats
8 - information related to system operation and maintenance

Thus, it seems that documentation for user commands, e.g., CREATE
TABLE, SELECT, UPDATE, ... should go into section 1, the API to
libpq/libpq++/libpgtcl/... should go into section 3, configuration
files like pg_hba.conf should go into section 5, and admin commands,
e.g., createdb, createuser, pg_dump, ... should go into section 8.

Cheers,
Brook

Tue, 10 Aug 1999 10:30:09 -0600 (MDT)
Brook Milligan <brook@biology.nmsu.edu>
Re: [HACKERS] Re: [ANNOUNCE] New man pages
> > Pages from multi-character sections are stored in the directory for the
> > first character.  For instance: /usr/man/man7/select.7l.gz
> 
> Oh! afaik that is one option; the man system in general could also
> handle man7l/select.7.gz right? You would update /etc/man.config to
> add, say, "7l" to the list of sections.
> 
> But is is against Debian policy to invent new directories for pages? I
> see that my RH linux system actually does about the same as Debian;
> there are some ".1x" files in the /usr/man/man1 directory.

I have never seen a 'name.1x' or anything with a more than
single-character file prefix, and once it is formatted, it becomes
name.0.  I don't see it buys us anything to do this.  What we could do
is to put throw them in section 7, assuming there is no conflict.  I
only have two pgp man pages in my 7.

> 
> >   >> I would like to use existing sections, rather than do our own.  I found
> >   >> I had to modify the man page search to look in a manl, and others may
> >   >> have the same problem.
> > For Debian, I have relocated the SQL pages to section 7l and commands such
> > as psql and createuser go in section 1.  Policy requires me to use one of
> > the numbered sections (1-8), though I can use a suffix to ensure uniqueness.
> > On Debian GNU/Linux, the sections are:
> > 1 User commands
> > 2 System calls
> > 3 Library routines
> > 4 Devices
> > 5 File formats
> > 6 Games
> > 7 Miscellaneous
> > 8 System administration
> 
> Same for Linux ("man 7 man" has a summary).
> 
> >   >otoh, it does eliminate the possibility of man page pollution if we
> >   >manage to have the same man page name as some other existing page.
> > As of course we do; for example, select is also in section 2.
> 
> A near miss, since we weren't likely to have chosen section 2 for
> *our* select. But it does illustrate the risk.
> 
> >   >*That* would be a bad thing. And in general adding ~75 man pages to
> >   >existing sections is a pretty big load...
> > I'm not sure that's much of a problem. These are the figures from my
> > system for /usr/man, /usr/share/man, /usr/X11R6/man and /usr/local/man
> > combined:
> 
> Right.
> 
> So, do Oliver's conventions make sense for most platforms? istm that
> they do. Would folks have problems with a mapping similar to what
> Oliver uses? We would use section one (1) and section seven (7), with
> a qualifier of ell (l) on each of the man page names. I won't do
> anything about it right now, but would like to get a consensus now
> that the subject has come up. Speak up now or forever hold your...

I agree with the 7, but see no need for the additional qualifier.  I
think that could cause more problems than it is worth.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 10 Aug 1999 12:43:29 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [ANNOUNCE] New man pages
> btw, on my Solaris boxes MANPATH is worse than useless; if you specify
> it then none of the other paths mentioned in /etc/man.config (or
> wherever that is on Solaris) get used. So you have to recreate all of
> the default MANPATH settings in your environment variable. Of course,
> now that I've whined about this perhaps someone knows a way around
> this?
> 
> Is there any concern about using "l" (ell) for the section
> discriminator?
> 
> So, I'll count you as not objecting to sections "1l" (one-ell) and
> "7l" (seven-ell), ok?

Why not just put it in section 7, not 7l.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 10 Aug 1999 13:00:09 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [ANNOUNCE] New man pages
Hackers: has anyone any insight on this one, please?

Craig Sanders wrote:
  >one of my clients is having problems with postgresql. i've upgraded to
  >the latest version 6.5.1-5 hoping that might fix the problem..no luck.
  >
  >any ideas?
  >
  >
  >BTW, i turned on debugging in postmaster.init and this is a sample of
  >what it shows - "ERROR: vacuum: can't destroy lock file!". there are 2
  >or 3 instances of this in the logs.
  >
...
  >ERROR:  vacuum: can't destroy lock file!
  >AbortCurrentTransaction
...

This does seem to be a problem with 6.5.1.  I have got a similar one coming up
in the regression test database.  Very interestingly, it has arisen since
my last clean vacuum and I have not touched the database since then. I
wonder if it is possible that vacuum has itself corrupted the database.

I have found no useful information in the logs; the actual error seems to
indicate that the vc_shutdown() routine is being called a second time for
the same database, but I cannot yet see why (if you want to investigate,
it is in src/backend/commands/vacuum.c).

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "If ye abide in me, and my words abide in you, ye shall
      ask what ye will, and it shall be done unto you."     
                                      John 15:7 



Tue, 10 Aug 1999 18:27:35 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: (fwd) Problems with Postgres 
"Oliver Elphick" wrote:
  >Hackers: has anyone any insight on this one, please?
 
This is where things are going wrong:

regression=> vacuum verbose analyze;
NOTICE:  --Relation pg_type--
NOTICE:  Pages 4: Changed 0, Reapped 1, Empty 0, New 0; Tup 248: Vac 0, 
Keep/VTL
 0/0, Crash 0, UnUsed 31, MinLen 105, MaxLen 109; Re-using: Free/Avail. Space 
41
16/0; EndEmpty/Avail. Pages 0/0. Elapsed 0/0 sec.
NOTICE:  Index pg_type_typname_index: Pages 4; Tuples 248: Deleted 0. Elapsed 
0/
0 sec.
NOTICE:  Index pg_type_oid_index: Pages 2; Tuples 248: Deleted 0. Elapsed 0/0 
se
c.
NOTICE:  --Relation pg_attribute--
NOTICE:  Pages 22: Changed 0, Reapped 1, Empty 0, New 0; Tup 1666: Vac 0, 
Keep/V
TL 0/0, Crash 0, UnUsed 50, MinLen 97, MaxLen 100; Re-using: Free/Avail. Space 
5
072/0; EndEmpty/Avail. Pages 0/0. Elapsed 0/0 sec.
NOTICE:  Index pg_attribute_attrelid_index: Pages 10; Tuples 1666: Deleted 0. 
El
apsed 0/0 sec.
NOTICE:  Index pg_attribute_relid_attnum_index: Pages 12; Tuples 1666: Deleted 
0
. Elapsed 0/0 sec.
NOTICE:  Index pg_attribute_relid_attnam_index: Pages 28; Tuples 1666: Deleted 
0
. Elapsed 0/0 sec.
NOTICE:  --Relation pg_proc--
NOTICE:  Pages 24: Changed 0, Reapped 0, Empty 0, New 0; Tup 1070: Vac 0, 
Keep/V
TL 0/0, Crash 0, UnUsed 0, MinLen 145, MaxLen 2369; Re-using: Free/Avail. 
Space
0/0; EndEmpty/Avail. Pages 0/0. Elapsed 0/0 sec.
NOTICE:  Index pg_proc_prosrc_index: Pages 10; Tuples 1070. Elapsed 0/0 sec.
NOTICE:  Index pg_proc_proname_narg_type_index: Pages 17; Tuples 1070. Elapsed 
0
/0 sec.
NOTICE:  Index pg_proc_oid_index: Pages 5; Tuples 1070. Elapsed 0/0 sec.
NOTICE:  --Relation pg_class--
NOTICE:  Pages 3: Changed 0, Reapped 2, Empty 0, New 0; Tup 210: Vac 0, 
Keep/VTL
 0/0, Crash 0, UnUsed 14, MinLen 102, MaxLen 132; Re-using: Free/Avail. Space 
14
32/0; EndEmpty/Avail. Pages 0/0. Elapsed 0/0 sec.
NOTICE:  Index pg_class_relname_index: Pages 5; Tuples 210: Deleted 0. Elapsed 
0
/0 sec.
NOTICE:  Index pg_class_oid_index: Pages 2; Tuples 210: Deleted 0. Elapsed 0/0 
s
ec.

The relation now being vacuumed is bt_text_heap

the pg_vlock file gets deleted in vc_abort with this backtrace:
(gdb) bt
#0  vc_abort () at vacuum.c:252
#1  0x8078f28 in TransactionIdAbort (transactionId=156164) at transam.c:578
#2  0x80e0269 in XactLockTableWait (xid=156164) at lmgr.c:332
#3  0x806c9a9 in heap_delete (relation=0x8216ac8, tid=0x8225c48, ctid=0x0) at 
heapam.c:1149
#4  0x808ce04 in vc_delhilowstats (relid=2073242, attcnt=0, attnums=0x0) at 
vacuum.c:2484
#5  0x80897fc in vc_vacone (relid=2073242, analyze=1, va_cols=0x0) at 
vacuum.c:510
#6  0x80891fe in vc_vacuum (VacRelP=0x0, analyze=1 '\001', va_cols=0x0) at 
vacuum.c:279
#7  0x80890df in vacuum (vacrel=0x0, verbose=1, analyze=1 '\001', va_spec=0x0) 
at vacuum.c:164
#8  0x80e7ee0 in ProcessUtility (parsetree=0x8237e08, dest=Remote) at 
utility.c:638
#9  0x80e4e66 in pg_exec_query_dest (query_string=0xbfffa2b4 "vacuum verbose 
analyze;", dest=Remote, aclOverride=0) at postgres.c:727
#10 0x80e4d74 in pg_exec_query (query_string=0xbfffa2b4 "vacuum verbose 
analyze;") at postgres.c:656
#11 0x80e5edf in PostgresMain (argc=6, argv=0xbffff42c, real_argc=10, 
real_argv=0xbffff9b4) at postgres.c:1647
#12 0x80cec5f in DoBackend (port=0x81d5140) at postmaster.c:1628
#13 0x80ce73a in BackendStartup (port=0x81d5140) at postmaster.c:1373
#14 0x80cde59 in ServerLoop () at postmaster.c:823
#15 0x80cd989 in PostmasterMain (argc=10, argv=0xbffff9b4) at postmaster.c:616
#16 0x80a4245 in main (argc=10, argv=0xbffff9b4) at main.c:97

  >Craig Sanders wrote:
  >  >one of my clients is having problems with postgresql. i've upgraded to
  >  >the latest version 6.5.1-5 hoping that might fix the problem..no luck.
  >  >
  >  >any ideas?
  >  >
  >  >
  >  >BTW, i turned on debugging in postmaster.init and this is a sample of
  >  >what it shows - "ERROR: vacuum: can't destroy lock file!". there are 2
  >  >or 3 instances of this in the logs.
  >  >
  >...
  >  >ERROR:  vacuum: can't destroy lock file!
  >  >AbortCurrentTransaction
  >...
  >
  >This does seem to be a problem with 6.5.1.  I have got a similar one coming 
      >up
  >in the regression test database.  Very interestingly, it has arisen since
  >my last clean vacuum and I have not touched the database since then. I
  >wonder if it is possible that vacuum has itself corrupted the database.
  >
  >I have found no useful information in the logs; the actual error seems to
  >indicate that the vc_shutdown() routine is being called a second time for
  >the same database, but I cannot yet see why

In fact vc_abort unlinks pg_vlock, but then the vacuum keeps running and
vc_shutdown prints the error when it tries to unlink pg_vlock again.

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "If ye abide in me, and my words abide in you, ye shall
      ask what ye will, and it shall be done unto you."     
                                      John 15:7 



Tue, 10 Aug 1999 20:42:58 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: (fwd) Problems with Postgres 
"Oliver Elphick" <olly@lfix.co.uk> writes:
> This is where things are going wrong:
> the pg_vlock file gets deleted in vc_abort with this backtrace:
> (gdb) bt
> #0  vc_abort () at vacuum.c:252
> #1  0x8078f28 in TransactionIdAbort (transactionId=156164) at transam.c:578
> #2  0x80e0269 in XactLockTableWait (xid=156164) at lmgr.c:332
> #3  0x806c9a9 in heap_delete (relation=0x8216ac8, tid=0x8225c48, ctid=0x0) at heapam.c:1149

Ah-hah, I *knew* that code was bogus: TransactionIdAbort() has no
business calling vc_abort().  I fixed that about two days ago
in both current and REL6_5 branches...

			regards, tom lane

Tue, 10 Aug 1999 17:21:47 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: (fwd) Problems with Postgres 
Brook Milligan <brook@biology.nmsu.edu> writes:
> Thus, it seems that documentation for user commands, e.g., CREATE
> TABLE, SELECT, UPDATE, ... should go into section 1,

No.  Section 1 is exclusively for *programs*, ie, commands available
at a shell command line.  There isn't really anyplace in the standard
man conventions to put a separate man page for each command accepted
by a single program, which is what SQL language constructs are.
We're usurping more "man page namespace" than we ought to by putting
each SQL construct on its own man page.  However, one man page for
the whole of SQL isn't too appealing, so we have little choice.
The proposal to put 'em in section 7 sounded reasonable to me.

> the API to
> libpq/libpq++/libpgtcl/... should go into section 3, configuration
> files like pg_hba.conf should go into section 5, and admin commands,
> e.g., createdb, createuser, pg_dump, ... should go into section 8.

These would make sense, but you start to run into some of the
cross-platform idiosyncracies in section usage as soon as you go past
section 3.  For example, on HPUX file format docs live in section 4,
and admin commands live in section 1m.  I don't think HP invented those
conventions on their own --- they are probably common to a lot of
old-line SysV-derived Unixes.  The Debian conventions that Oliver
mentioned look like they descend from BSD Unix.

I agree with putting the libpq etc. APIs in section 3, assuming we still
have man pages for them at all (at one time there was talk of dropping
those manpages in favor of the chapters in the SGML docs).  I'd be
inclined to just put createdb and friends in section 1, rather than
worrying about where they should go to classify them as admin commands...

			regards, tom lane

Tue, 10 Aug 1999 17:32:52 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [ANNOUNCE] New man pages 
Tom Lane wrote:
  >Ah-hah, I *knew* that code was bogus: TransactionIdAbort() has no
  >business calling vc_abort().  I fixed that about two days ago
  >in both current and REL6_5 branches...
 
Tom, can you give me a patch, or instructions to fix it? I don't want to
release the whole REL6_5 branch until it's officially released.

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "If ye abide in me, and my words abide in you, ye shall
      ask what ye will, and it shall be done unto you."     
                                      John 15:7 



Tue, 10 Aug 1999 22:45:16 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Re: (fwd) Problems with Postgres 
> > btw, on my Solaris boxes MANPATH is worse than useless; if you specify
> > it then none of the other paths mentioned in /etc/man.config (or
> > wherever that is on Solaris) get used. So you have to recreate all of
> > the default MANPATH settings in your environment variable. Of course,
> > now that I've whined about this perhaps someone knows a way around
> > this?
> Er, why not "MANPATH=/usr/local/pgsql/man:$MANPATH" ?

'Cause MANPATH is not defined to start with. My point was that man
does a great job just using its configuration file, but if you start
using MANPATH you have to (apparently) figure out what paths were in
the config file and put those in too...

                     - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Wed, 11 Aug 1999 03:10:39 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Re: [ANNOUNCE] New man pages
> > > btw, on my Solaris boxes MANPATH is worse than useless; if you specify
> > > it then none of the other paths mentioned in /etc/man.config (or
> > > wherever that is on Solaris) get used. So you have to recreate all of
> > > the default MANPATH settings in your environment variable. Of course,
> > > now that I've whined about this perhaps someone knows a way around
> > > this?
> > Er, why not "MANPATH=/usr/local/pgsql/man:$MANPATH" ?
> 
> 'Cause MANPATH is not defined to start with. My point was that man
> does a great job just using its configuration file, but if you start
> using MANPATH you have to (apparently) figure out what paths were in
> the config file and put those in too...

Yes, I have seen this too.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 10 Aug 1999 23:36:18 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [ANNOUNCE] New man pages
I've been getting a strange error from the vacuum command. When I try
'vacuum verbose analyze' the vacuum goes through the tables fine until
just after finishing one particular table. Then I get the error:

NOTICE: AbortTransaction and not in in-progress state
pqReadData() -- backend closed the channel unexpectedly.
        This probably means the backend terminated abnormally
        before or while processing the request.
We have lost the connection to the backend, so further processing is
impossible. Terminating.


When I try to vacuum the tables individually, I get no problems with
aborted backends.

Does anyone know what is going on here?

-Tony

PostgreSQL 6.5.1 on RH Linux 6.0, PII/400 MHz, 512 Meg RAM, 18 Gig HD




Thu, 12 Aug 1999 16:33:32 -0700
"G. Anthony Reina" <reina@nsi.edu>
Aborted Transaction During Vacuum
"G. Anthony Reina" <reina@nsi.edu> writes:
> I've been getting a strange error from the vacuum command. When I try
> 'vacuum verbose analyze' the vacuum goes through the tables fine until
> just after finishing one particular table. Then I get the error:

> NOTICE: AbortTransaction and not in in-progress state
> pqReadData() -- backend closed the channel unexpectedly.

Interesting.  Is there any additional message appearing in the
postmaster log?  Is a core file being generated?  (look in the
data/base/ subdirectory for the database in question)  If there
is a corefile, a debugger backtrace from it would be helpful.

			regards, tom lane

Thu, 12 Aug 1999 21:28:24 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Aborted Transaction During Vacuum 
Tom Lane wrote:

>
> Interesting.  Is there any additional message appearing in the
> postmaster log?  Is a core file being generated?  (look in the
> data/base/ subdirectory for the database in question)  If there
> is a corefile, a debugger backtrace from it would be helpful.
>
>                         regards, tom lane

Tom,

    I tried the 'vacuum verbose analyze' again today. I get the same error
with the AbortTransaction. There is a core file generated but no pg_vlock
file. The core is over 1 Gig in size (38 Megs gzipped) so I'm not so sure
you'd want to get that  (you can have it if you want it though!). Plus,
there seems to be nothing written to the postmaster.log file (I re-started
the postmaster before the vacuum using 'nohup postmaster -i -B 15000 -o -F
> postmaster.log&').

    Oliver Elphick (Oliver.Elphick@lfix.co.uk) told me that this sounds
like like a bug you patched for him a few days ago:

> It does sound very like the bug that was found a few days back where the

> pg_vlock file gets deleted by  a mistaken call to vc_abort().  This only

> gets called in the analyze code.

We're probably talking about the same bug. When I try just 'vacuum verbose'
without the analyze, the vacuum completes just fine. So it must be within
the analyze code.

-Tony



Fri, 13 Aug 1999 10:28:10 -0700
"G. Anthony Reina" <reina@nsi.edu>
Re: [HACKERS] Aborted Transaction During Vacuum
"G. Anthony Reina" <reina@nsi.edu> writes:
>     I tried the 'vacuum verbose analyze' again today. I get the same error
> with the AbortTransaction. There is a core file generated but no pg_vlock
> file. The core is over 1 Gig in size (38 Megs gzipped) so I'm not so sure
> you'd want to get that  (you can have it if you want it though!).

It wouldn't do me any good anyway without a copy of the executable and a
copy of gdb built for whatever platform you are on.  I was hoping you
could run gdb on the corefile there and just send the backtrace output
("gdb postgres-executable core-file" and then say "bt").

>     Oliver Elphick (Oliver.Elphick@lfix.co.uk) told me that this sounds
> like like a bug you patched for him a few days ago:

I doubt it's the same bug --- the error message generated by the other
bug was different...

			regards, tom lane

Fri, 13 Aug 1999 13:37:19 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Aborted Transaction During Vacuum 
Tom Lane wrote:
  >>     Oliver Elphick (Oliver.Elphick@lfix.co.uk) told me that this sounds
  >> like like a bug you patched for him a few days ago:
  >
  >I doubt it's the same bug --- the error message generated by the other
  >bug was different...

The error message I posted was the log output at level 3, which he doesn't
seem to be running; if you switch to logging at that level, look for
messages that mention pg_vlock.

As far as I recall, the only error message he has listed so far is from
psql, which merely records the unlink failure at the end of the vacuum.
Everything else looks remarkably like the vc_abort bug.

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Watch ye therefore, and pray always, that ye may be 
      accounted worthy to escape all these things that shall
      come to pass, and to stand before the Son of man."    
                               Luke 21:36 



Fri, 13 Aug 1999 19:04:43 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Aborted Transaction During Vacuum 
"Oliver Elphick" <olly@lfix.co.uk> writes:
> As far as I recall, the only error message he has listed so far is from
> psql, which merely records the unlink failure at the end of the vacuum.
> Everything else looks remarkably like the vc_abort bug.

But the vc_abort problem didn't cause a backend coredump --- it just
reported an error and failed to complete the vacuum, no?

			regards, tom lane

Fri, 13 Aug 1999 14:09:01 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Aborted Transaction During Vacuum 
Tom Lane wrote:
  >"Oliver Elphick" <olly@lfix.co.uk> writes:
  >> As far as I recall, the only error message he has listed so far is from
  >> psql, which merely records the unlink failure at the end of the vacuum.
  >> Everything else looks remarkably like the vc_abort bug.
  >
  >But the vc_abort problem didn't cause a backend coredump --- it just
  >reported an error and failed to complete the vacuum, no?
  >

I got a coredump too; I never mentioned it, because I found the proximate
cause and curing that made it go away.  When unlink failed in vc_shutdown,
it called ELOG and a segfault occurred a little later.

I forgot about that when your patch fixed the original problem.

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Watch ye therefore, and pray always, that ye may be 
      accounted worthy to escape all these things that shall
      come to pass, and to stand before the Son of man."    
                               Luke 21:36 



Fri, 13 Aug 1999 19:28:31 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Aborted Transaction During Vacuum 
Tom Lane wrote:

> But the vc_abort problem didn't cause a backend coredump --- it just
> reported an error and failed to complete the vacuum, no?
>
>                         regards, tom lane

Here's the error message again:

NOTICE:  --Relation ex_ellipse_proc--
NOTICE:  --Relation ex_ellipse_proc--
NOTICE:  Pages 2545: Changed 0, Reapped 0, Empty 0, New 0; Tup 30535: Vac 0,
Keep/VTL 0/0, Crash 0, UnUsed 0,
MinLen 660, MaxLen 660; Re-using: Free/Avail. Space 0/0; EndEmpty/Avail.
Pages 0/0. Elapsed 1/0 sec.
NOTICE:  Pages 2545: Changed 0, Reapped 0, Empty 0, New 0; Tup 30535: Vac 0,
Keep/VTL 0/0, Crash 0, UnUsed 0,
MinLen 660, MaxLen 660; Re-using: Free/Avail. Space 0/0; EndEmpty/Avail.
Pages 0/0. Elapsed 1/0 sec.
NOTICE:  Index pkex_ellipse_proc: Pages 138; Tuples 30535. Elapsed 0/0 sec.
NOTICE:  Index pkex_ellipse_proc: Pages 138; Tuples 30535. Elapsed 0/0 sec.
NOTICE:  --Relation ex_ellipse_cell--
NOTICE:  --Relation ex_ellipse_cell--
NOTICE:  Pages 370: Changed 0, Reapped 0, Empty 0, New 0; Tup 6109: Vac 0,
Keep/VTL 0/0, Crash 0, UnUsed 0,
MinLen 80, MaxLen 2736; Re-using: Free/Avail. Space 0/0; EndEmpty/Avail.
Pages 0/0. Elapsed 0/0 sec.
NOTICE:  Pages 370: Changed 0, Reapped 0, Empty 0, New 0; Tup 6109: Vac 0,
Keep/VTL 0/0, Crash 0, UnUsed 0,
MinLen 80, MaxLen 2736; Re-using: Free/Avail. Space 0/0; EndEmpty/Avail.
Pages 0/0. Elapsed 0/0 sec.
NOTICE:  Index pkex_ellipse_cell: Pages 42; Tuples 6109. Elapsed 0/0 sec.
NOTICE:  Index pkex_ellipse_cell: Pages 42; Tuples 6109. Elapsed 0/0 sec.
NOTICE:  --Relation ex_ellipse_opto--
NOTICE:  --Relation ex_ellipse_opto--
NOTICE:  Pages 26356: Changed 0, Reapped 0, Empty 0, New 0; Tup 71475: Vac
0, Keep/VTL 0/0, Crash 0, UnUsed 0,
MinLen 1760, MaxLen 6108; Re-using: Free/Avail. Space 0/0; EndEmpty/Avail.
Pages 0/0. Elapsed 3/2 sec.
NOTICE:  Pages 26356: Changed 0, Reapped 0, Empty 0, New 0; Tup 71475: Vac
0, Keep/VTL 0/0, Crash 0, UnUsed 0,
MinLen 1760, MaxLen 6108; Re-using: Free/Avail. Space 0/0; EndEmpty/Avail.
Pages 0/0. Elapsed 3/2 sec.
NOTICE:  Index pkex_ellipse_opto: Pages 357; Tuples 71475. Elapsed 0/0 sec.
NOTICE:  Index pkex_ellipse_opto: Pages 357; Tuples 71475. Elapsed 0/0 sec.
NOTICE:  --Relation ex_ellipse_opto_proc--
NOTICE:  --Relation ex_ellipse_opto_proc--
NOTICE:  Pages 14742: Changed 0, Reapped 0, Empty 0, New 0; Tup 30535: Vac
0, Keep/VTL 0/0, Crash 0, UnUsed 0,
MinLen 1816, MaxLen 5900; Re-using: Free/Avail. Space 0/0; EndEmpty/Avail.
Pages 0/0. Elapsed 1/1 sec.
NOTICE:  Pages 14742: Changed 0, Reapped 0, Empty 0, New 0; Tup 30535: Vac
0, Keep/VTL 0/0, Crash 0, UnUsed 0,
MinLen 1816, MaxLen 5900; Re-using: Free/Avail. Space 0/0; EndEmpty/Avail.
Pages 0/0. Elapsed 1/1 sec.
NOTICE:  Index pkex_ellipse_opto_proc: Pages 138; Tuples 30535. Elapsed 0/0
sec.
NOTICE:  Index pkex_ellipse_opto_proc: Pages 138; Tuples 30535. Elapsed 0/0
sec.
ERROR:  vacuum: can't destroy lock file!
NOTICE:  AbortTransaction and not in in-progress state
NOTICE:  AbortTransaction and not in in-progress state
pqReadData() -- backend closed the channel unexpectedly.
        This probably means the backend terminated abnormally
        before or while processing the request.
We have lost the connection to the backend, so further processing is
impossible.  Terminating.
[postgres@bigred ~]$



-Tony



Fri, 13 Aug 1999 11:38:12 -0700
"G. Anthony Reina" <reina@nsi.edu>
Re: [HACKERS] Aborted Transaction During Vacuum
"Oliver Elphick" <olly@lfix.co.uk> writes:
>> But the vc_abort problem didn't cause a backend coredump --- it just
>> reported an error and failed to complete the vacuum, no?

> I got a coredump too; I never mentioned it, because I found the proximate
> cause and curing that made it go away.  When unlink failed in vc_shutdown,
> it called ELOG and a segfault occurred a little later.

Ah, I wish I'd known that.  So what Tony is seeing is exactly the same
behavior you observed.  OK, I feel better now --- I thought the coredump
was probably some platform-specific misbehavior that only Tony was seeing.

We still need to figure out what is causing it, because I can see no
reason for a coredump after vc_shutdown elog()s.  Something is being
clobbered that should not be.  But it sounds like installing the
vc_abort patch will get Tony on his feet, and then we can look for the
secondary bug at our leisure.

			regards, tom lane

Fri, 13 Aug 1999 17:31:47 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Aborted Transaction During Vacuum 
On Tue, 10 Aug 1999, Tom Lane wrote:

> > manage to have the same man page name as some other existing page.
> > *That* would be a bad thing. And in general adding ~75 man pages to
> > existing sections is a pretty big load...
> 
> As long as we install into /usr/local/pgsql/man/man*, naming conflicts
> with other packages aren't too big a deal --- there's no physical file
> conflict, and people can just add or remove /usr/local/pgsql/man/ in
> their MANPATH settings to see or not see Postgres manpages.

Jumping in mid-stream and a week late (love holidays, eh? *grin*) ... My
opinion is to make the default ${PREFIX}/man and have a --manpath=
variable set to configure to move it to a different place...

I *believe* that Oracle installs its man pages under its install
directory, but can't confirm right now...

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Mon, 16 Aug 1999 00:00:49 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Re: [ANNOUNCE] New man pages 
On Tue, 10 Aug 1999, Brook Milligan wrote:

> ------- Start of forwarded message -------
>    So, do Oliver's conventions make sense for most platforms? istm that
>    they do. Would folks have problems with a mapping similar to what
>    Oliver uses? We would use section one (1) and section seven (7), with
>    a qualifier of ell (l) on each of the man page names. I won't do
>    anything about it right now, but would like to get a consensus now
>    that the subject has come up. Speak up now or forever hold your...
> 
> OK, I'll speak up. :)
> 
> This doesn't make too much sense to me based on my experience with
> zillions of other man pages.
> 
> A quick look through all of my system-supplied, X11, and locally

Just a quick note, but, under FreeBSD *at least*, X11 puts its man pages
in /usr/X11R6/man/* ... so it doesn't conflict with "system" man pages...

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Mon, 16 Aug 1999 00:03:15 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Re: [ANNOUNCE] New man pages
On Wed, 11 Aug 1999, Thomas Lockhart wrote:

> > > btw, on my Solaris boxes MANPATH is worse than useless; if you specify
> > > it then none of the other paths mentioned in /etc/man.config (or
> > > wherever that is on Solaris) get used. So you have to recreate all of
> > > the default MANPATH settings in your environment variable. Of course,
> > > now that I've whined about this perhaps someone knows a way around
> > > this?
> > Er, why not "MANPATH=/usr/local/pgsql/man:$MANPATH" ?
> 
> 'Cause MANPATH is not defined to start with. My point was that man
> does a great job just using its configuration file, but if you start
> using MANPATH you have to (apparently) figure out what paths were in
> the config file and put those in too...

Okay, I run Solaris at work (2.5.x -> 7) and have yet to find a
/etc/man.config file...I've always used MANPATH here :(

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Mon, 16 Aug 1999 00:04:50 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Re: [ANNOUNCE] New man pages
> Bruce Momjian wrote:
> ...
>   >OK, you have good points.  usecatupd should not be set by default. 
>   >Making changes to the system tables can mess things up for everyone. 
>   >Initdb will give the postgres superuser permissions, but now createuser
>   >and the SQL command CREATE USER will not give this permission.  Also, I
>   >have fixed the code so temp tables, which are acutally named pg_temp,
>   >can be updated by normal users without usecatupd permissions.
>   >
>   >Attached is a patch.  I will apply it to the current tree.
> 
> Bruce, this change has some other implications.  I tested
> the effect of the patch by altering the rights of my own account (setting
> usecatupd to false).  I cannot now create other users: although usesuper is
> true, the attempt to update pg_shadow with the new user's row fails:


OK, I have committed a fix for this.  I now give update catalog rights
to anyone who has createdb or super-user rights.  This is in the current
tree only.  Let me know if you see any additional problems.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 27 Sep 1999 12:43:09 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Cannot insert into temp tables
Hmm. I see a problem coming up for myself :(

I'm working on JOIN syntax and (hopefully) OUTER JOIN capabilities for
the next release. That will take a lot of time afaik. We also have
lots of great new features (almost certainly) coming up, such as WAL
and RI. Lots of stuff, and taken together it should lead to a v7.0
release.

However! I've been waiting for v7.0 to do internal remapping of the
datetime types (and possibly other types; I'm not remembering right
now) so that the 4-byte TIMESTAMP becomes, internally, what is
currently DATETIME and so that the current DATETIME becomes a synonym
for TIMESTAMP. If we are releasing v7.0 rather than v6.6, then I
should work on that now to ensure it is completed. If we might release
as v6.6, I should continue to work on OUTER JOIN, since the type
changes shouldn't go in yet.

Any opinions/comments on the right path to take? With code changes and
docs changes I would think that the datetime stuff is two or a few
weeks work with my off-work time, which I suppose wouldn't impace a
release schedule too much...

                  - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Fri, 08 Oct 1999 15:07:33 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Features for next release
Thomas Lockhart     wrote:

>
> Hmm. I see a problem coming up for myself :(
>
> I'm working on JOIN syntax and (hopefully) OUTER JOIN capabilities for
> the next release. That will take a lot of time afaik. We also have
> lots of great new features (almost certainly) coming up, such as WAL
> and RI. Lots of stuff, and taken together it should lead to a v7.0
> release.

    What  I  actually  see  when starting/stopping the postmaster
    lets me think that Vadim alread has the WAL target in  sight.

    And for RI I can say that we'll have at least FOREIGN KEY for
    MATCH FULL without PENDANT  in  the  next  release.  The  key
    features  are  in place and there's plenty of time left to do
    all the stuff around.

>
> However! I've been waiting for v7.0 to do internal remapping of the
> datetime types (and possibly other types; I'm not remembering right
> now) so that the 4-byte TIMESTAMP becomes, internally, what is
> currently DATETIME and so that the current DATETIME becomes a synonym
> for TIMESTAMP. If we are releasing v7.0 rather than v6.6, then I
> should work on that now to ensure it is completed. If we might release
> as v6.6, I should continue to work on OUTER JOIN, since the type
> changes shouldn't go in yet.
>
> Any opinions/comments on the right path to take? With code changes and
> docs changes I would think that the datetime stuff is two or a few
> weeks work with my off-work time, which I suppose wouldn't impace a
> release schedule too much...

    Break TIMESTAMP right now. IMHO it would  be  THE  reason  (a
    functional  incompatibility  to v6.5) we where looking for to
    be 100% sure the next release must be v7.0.

    I clearly vote for v7.0 - even if I personally hoped we would
    have   a   little  chance  to  produce  the  ultimate  devils
    PostgreSQL-v6.6.6 someday.

    Apropos devil: tomorrow Braunschweig Lions vs.  Hamburg  Blue
    Devils here in Hamburg - German bowl! Go Devils go!


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #



Fri, 8 Oct 1999 18:25:50 +0200 (MET DST)
wieck@debis.com (Jan Wieck)
Re: [HACKERS] Features for next release
> Hmm. I see a problem coming up for myself :(
> 
> I'm working on JOIN syntax and (hopefully) OUTER JOIN capabilities for
> the next release. That will take a lot of time afaik. We also have
> lots of great new features (almost certainly) coming up, such as WAL
> and RI. Lots of stuff, and taken together it should lead to a v7.0
> release.
> 
> However! I've been waiting for v7.0 to do internal remapping of the
> datetime types (and possibly other types; I'm not remembering right
> now) so that the 4-byte TIMESTAMP becomes, internally, what is
> currently DATETIME and so that the current DATETIME becomes a synonym
> for TIMESTAMP. If we are releasing v7.0 rather than v6.6, then I
> should work on that now to ensure it is completed. If we might release
> as v6.6, I should continue to work on OUTER JOIN, since the type
> changes shouldn't go in yet.
> 
> Any opinions/comments on the right path to take? With code changes and
> docs changes I would think that the datetime stuff is two or a few
> weeks work with my off-work time, which I suppose wouldn't impace a
> release schedule too much...

My guess is 7.0.  With the minor releases still being made, Marc is in
no rush.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 8 Oct 1999 13:06:10 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Features for next release
> Thomas Lockhart     wrote:
> 
> >
> > Hmm. I see a problem coming up for myself :(
> >
> > I'm working on JOIN syntax and (hopefully) OUTER JOIN capabilities for
> > the next release. That will take a lot of time afaik. We also have
> > lots of great new features (almost certainly) coming up, such as WAL
> > and RI. Lots of stuff, and taken together it should lead to a v7.0
> > release.
> 
>     What  I  actually  see  when starting/stopping the postmaster
>     lets me think that Vadim alread has the WAL target in  sight.

Yes, looks very impressive:
	
	DEBUG:  Data Base System is starting up at Fri Oct  8 01:38:04 1999
	DEBUG:  Data Base System was shutdowned at Fri Oct  8 01:37:42 1999
	DEBUG:  CheckPoint record at (0, 968)
	DEBUG:  Redo record at (0, 968); Undo record at (0, 0)
	DEBUG:  NextTransactionId: 20995; NextOid: 0
	DEBUG:  Invalid NextTransactionId/NextOid
	DEBUG:  Data Base System is in production state at Fri Oct  8 01:38:04 1999



-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 8 Oct 1999 13:15:56 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Features for next release
Found this quote by Philip Greenspun (who is unabashedly in love with
Oracle) in a LinuxWorld article:
"> The open source purist's only realistic choice for an RDBMS
>  is PostgreSQL (see Resources for the URL). In some ways,
>  PostgreSQL has more advanced features than any commercial
>  RDBMS. Most important, the loosely organized, unpaid 
>  developers of PostgreSQL were able to convert to an 
>  Oracle-style multiversion concurrency system (see below),
>  leaving all the rest of the commercial competition 
>  deadlocked in the dust. If you've decided to accept John
>  Ousterhout as your personal savior, you'll be delighted
>  to learn that you can run Tcl procedures inside the 
>  PostgreSQL database. And if your business can't live without
>  commercial support for your RDBMS, you can buy it 
>  (see Resources for a link). 

Cheers!  (the full series of articles, which are about AOLserver, is
available at http://linuxworld.com)

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Fri, 08 Oct 1999 13:28:56 -0400
Lamar Owen <lamar.owen@wgcr.org>
Interesting Quote you might enjoy about PGSQL.
> Found this quote by Philip Greenspun (who is unabashedly in love with
> Oracle) in a LinuxWorld article:
> "> The open source purist's only realistic choice for an RDBMS
> >  is PostgreSQL (see Resources for the URL). In some ways,
> >  PostgreSQL has more advanced features than any commercial
> >  RDBMS. Most important, the loosely organized, unpaid 
> >  developers of PostgreSQL were able to convert to an 
> >  Oracle-style multiversion concurrency system (see below),
> >  leaving all the rest of the commercial competition 
> >  deadlocked in the dust. If you've decided to accept John

That is am amazing quotation.  Vince, can we get a link to it on the
webpage?



> >  Ousterhout as your personal savior, you'll be delighted
> >  to learn that you can run Tcl procedures inside the 
> >  PostgreSQL database. And if your business can't live without
> >  commercial support for your RDBMS, you can buy it 
> >  (see Resources for a link). 
> 
> Cheers!  (the full series of articles, which are about AOLserver, is
> available at http://linuxworld.com)
> 
> --
> Lamar Owen
> WGCR Internet Radio
> 1 Peter 4:11
> 
> ************
> 


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 8 Oct 1999 14:25:36 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Interesting Quote you might enjoy about PGSQL.]
On Fri, Oct 08, 1999 at 06:25:50PM +0200, Jan Wieck wrote:
>     I clearly vote for v7.0 - even if I personally hoped we would

Me too. Although it means I have to check whether there is more stuff I have
to add to ECPG before 7.0 comes out.

>     have   a   little  chance  to  produce  the  ultimate  devils
>     PostgreSQL-v6.6.6 someday.

It seems we are too superstitous to allow this number. :-)

>     Apropos devil: tomorrow Braunschweig Lions vs.  Hamburg  Blue
>     Devils here in Hamburg - German bowl! Go Devils go!

Jan, now you surprise me. You are a football fan? I never expected another
German on this list to love football (american style that is). Maybe we are
able to meet at a football games. But then I cannot come to the German Bowl. 

Do you go to NFLE games too?

Michael

P.S.: Sorry for being off-topic for this list in the last part of my mail.
-- 
Michael Meskes                         | Go SF 49ers!
Th.-Heuss-Str. 61, D-41812 Erkelenz    | Go Rhein Fire!
Tel.: (+49) 2431/72651                 | Use Debian GNU/Linux!
Email: Michael@Fam-Meskes.De           | Use PostgreSQL!

Fri, 8 Oct 1999 20:57:43 +0200
Michael Meskes <meskes@postgresql.org>
Re: [HACKERS] Features for next release
It seems people are thinking winter or early spring (northern hemisphere
that is ;-)) for the next major release, and by then I think there will
be enough cool stuff done that we can call it 7.0.  The only really big
to-do item that no one seems to be committed to doing in this cycle is
tuples bigger than a disk block, and maybe once the dust settles for
long queries someone will feel like tackling that...

I have another reason for calling it 7.0, which is that if we fix
the function-call interface the way I want to, we will break existing
user-written loadable modules that contain C-language functions.
Better to do that in a "7.0" than in a "6.6", no?

			regards, tom lane

Fri, 08 Oct 1999 19:14:24 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Features for next release 
Lamar Owen wrote:
> 
> Found this quote by Philip Greenspun (who is unabashedly in love with
> Oracle) in a LinuxWorld article:

It's nice to know he is a fan. btw, Philip was inquiring about the
feasibility of making PostgreSQL more compatible with Oracle to allow
him to port a large software project he built (in fact, you referred
us to his web site on some other topic recently). Until we have outer
joins it is a non-starter for him, but it was nice he asked :)

                   - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Sat, 09 Oct 1999 02:05:55 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Interesting Quote you might enjoy about PGSQL.
> > ... PostgreSQL has more advanced features than any commercial
> >  RDBMS. Most important, the loosely organized, unpaid
> >  developers of PostgreSQL were able to convert to an
> >  Oracle-style multiversion concurrency system (see below),
> >  leaving all the rest of the commercial competition
> >  deadlocked in the dust.

There we go, riding Vadim's coattails to glory :))

                    - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Sat, 09 Oct 1999 02:18:39 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Interesting Quote you might enjoy about PGSQL.
> > > ... PostgreSQL has more advanced features than any commercial
> > >  RDBMS. Most important, the loosely organized, unpaid
> > >  developers of PostgreSQL were able to convert to an
> > >  Oracle-style multiversion concurrency system (see below),
> > >  leaving all the rest of the commercial competition
> > >  deadlocked in the dust.
> 
> There we go, riding Vadim's coattails to glory :))

Yep.  The article makes it sound like we all did MVCC.  No one told him
is was just one person.

I was mentioning to the MySQL folks that we have some brilliant
developers.  Certainly most database companies have only a few really
good developers.  Because we have the entire Internet to get people, we
draw the best database developers in the world.

Added to developers page:

We don't hire people to develop PostgreSQL.  We have the entire Internet
to get people, so we draw the best database developers in the world.    

[Jan, can we get a better image for the developers page?]

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 8 Oct 1999 22:46:43 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Interesting Quote you might enjoy about PGSQL.
> > > ... PostgreSQL has more advanced features than any commercial
> > >  RDBMS. Most important, the loosely organized, unpaid
> > >  developers of PostgreSQL were able to convert to an
> > >  Oracle-style multiversion concurrency system (see below),
> > >  leaving all the rest of the commercial competition
> > >  deadlocked in the dust.
> 
> There we go, riding Vadim's coattails to glory :))
> 

New text:

We don't hire developers.  We reach across the Internet, drawing the
best database developers in the world to PostgreSQL.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 8 Oct 1999 22:53:35 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Interesting Quote you might enjoy about PGSQL.
> > > ... PostgreSQL has more advanced features than any commercial
> > >  RDBMS. Most important, the loosely organized, unpaid
> > >  developers of PostgreSQL were able to convert to an
> > >  Oracle-style multiversion concurrency system (see below),
> > >  leaving all the rest of the commercial competition
> > >  deadlocked in the dust.
> 
> There we go, riding Vadim's coattails to glory :))

Wow, that web page looks good now, with the quote at the bottom.  Jan,
we need the nicer world image.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 8 Oct 1999 22:55:08 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Interesting Quote you might enjoy about PGSQL.
Bruce Momjian wrote:

>
> > > > ... PostgreSQL has more advanced features than any commercial
> > > >  RDBMS. Most important, the loosely organized, unpaid
> > > >  developers of PostgreSQL were able to convert to an
> > > >  Oracle-style multiversion concurrency system (see below),
> > > >  leaving all the rest of the commercial competition
> > > >  deadlocked in the dust.
> >
> > There we go, riding Vadim's coattails to glory :))
>
> Wow, that web page looks good now, with the quote at the bottom.  Jan,
> we need the nicer world image.

    You mean one with the mountains - no?

    Well, I'll spend some time, polish up the Povray sources etc.
    so Vince can easily maintain the map after  -  only  that  he
    needs  Povray  3.1  and  maybe  Tcl/Tk 8.0 to do it, but that
    shouldn't be a problem since both are  freely  available  and
    easily to install.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #



Mon, 11 Oct 1999 12:42:23 +0200 (MET DST)
wieck@debis.com (Jan Wieck)
Re: [HACKERS] Interesting Quote you might enjoy about PGSQL.
On Mon, 11 Oct 1999, Jan Wieck wrote:

> Bruce Momjian wrote:
> 
> >
> > > > > ... PostgreSQL has more advanced features than any commercial
> > > > >  RDBMS. Most important, the loosely organized, unpaid
> > > > >  developers of PostgreSQL were able to convert to an
> > > > >  Oracle-style multiversion concurrency system (see below),
> > > > >  leaving all the rest of the commercial competition
> > > > >  deadlocked in the dust.
> > >
> > > There we go, riding Vadim's coattails to glory :))
> >
> > Wow, that web page looks good now, with the quote at the bottom.  Jan,
> > we need the nicer world image.
> 
>     You mean one with the mountains - no?
> 
>     Well, I'll spend some time, polish up the Povray sources etc.
>     so Vince can easily maintain the map after  -  only  that  he
>     needs  Povray  3.1  and  maybe  Tcl/Tk 8.0 to do it, but that
>     shouldn't be a problem since both are  freely  available  and
>     easily to install.

Not a problem.  I may have some questions re Povray's setup (I've 
never used it).

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH   email: vev@michvhf.com   flame-mail: /dev/null
  # include <std/disclaimers.h>       Have you seen http://www.pop4.net?
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Mon, 11 Oct 1999 07:40:20 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: [HACKERS] Interesting Quote you might enjoy about PGSQL.
> > Wow, that web page looks good now, with the quote at the bottom.  Jan,
> > we need the nicer world image.
> 
>     You mean one with the mountains - no?
> 
>     Well, I'll spend some time, polish up the Povray sources etc.
>     so Vince can easily maintain the map after  -  only  that  he
>     needs  Povray  3.1  and  maybe  Tcl/Tk 8.0 to do it, but that
>     shouldn't be a problem since both are  freely  available  and
>     easily to install.

Yes.  I would also like to see the sources.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 11 Oct 1999 09:45:54 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Interesting Quote you might enjoy about PGSQL.
> > > Wow, that web page looks good now, with the quote at the bottom.  Jan,
> > > we need the nicer world image.
> > 
> >     You mean one with the mountains - no?
> > 
> >     Well, I'll spend some time, polish up the Povray sources etc.
> >     so Vince can easily maintain the map after  -  only  that  he
> >     needs  Povray  3.1  and  maybe  Tcl/Tk 8.0 to do it, but that
> >     shouldn't be a problem since both are  freely  available  and
> >     easily to install.

I have tcl 8.0.5 and povray here too.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 11 Oct 1999 09:46:17 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Interesting Quote you might enjoy about PGSQL.
Thomas Lockhart wrote:
> 
> Lamar Owen wrote:
> >
> > Found this quote by Philip Greenspun (who is unabashedly in love with
> > Oracle) in a LinuxWorld article:
> 
> It's nice to know he is a fan. btw, Philip was inquiring about the
> feasibility of making PostgreSQL more compatible with Oracle to allow
> him to port a large software project he built (in fact, you referred
> us to his web site on some other topic recently). Until we have outer
> joins it is a non-starter for him, but it was nice he asked :)

That would be the ArsDigita Community System -- ACS for short.  This is
an absolutely wonderful community system for database-backed web sites
-- www.arsdigita.com for more info. Version 7 territory! I have played
with ACS against the free development-only Oracle 8i for Linux -- very
comprehensive. However, Oracle for a database-backed web site is priced
sky-high -- I saw one quote for $25,000 to back a website with Oracle 8i
on a 500MHz Pentium III (their license factors server speed into the
cost -- a 386-16 might only cost $1,000 on that scale...;-)) 
Compatibility with Oracle is IMHO a pretty good thing, as long as we
don't become an Oracle clone -- PostgreSQL has too many nice extras for
that.

Yeah, it was _because_ of AOLserver and Philip Greenspun that I got into
PostgreSQL in the first place.  

BTW: RedHat 6.1 shipped, with the last RPM's I posted.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 11 Oct 1999 11:24:38 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] Interesting Quote you might enjoy about PGSQL.
On Fri, 8 Oct 1999, Tom Lane wrote:

> It seems people are thinking winter or early spring (northern hemisphere
> that is ;-)) for the next major release, and by then I think there will
> be enough cool stuff done that we can call it 7.0.  The only really big
> to-do item that no one seems to be committed to doing in this cycle is
> tuples bigger than a disk block, and maybe once the dust settles for
> long queries someone will feel like tackling that...
> 
> I have another reason for calling it 7.0, which is that if we fix
> the function-call interface the way I want to, we will break existing
> user-written loadable modules that contain C-language functions.
> Better to do that in a "7.0" than in a "6.6", no?

Based on all the comments flyng around, v7.0 it is. to be released...some
day :)

Bruce, you asked for a v6.5.3 to be released...anything outstanding that
should prevent me from doing that tomorrow afternoon?  

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Tue, 12 Oct 1999 01:07:57 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Features for next release 
> > I have another reason for calling it 7.0, which is that if we fix
> > the function-call interface the way I want to, we will break existing
> > user-written loadable modules that contain C-language functions.
> > Better to do that in a "7.0" than in a "6.6", no?
> 
> Based on all the comments flyng around, v7.0 it is. to be released...some
> day :)
> 
> Bruce, you asked for a v6.5.3 to be released...anything outstanding that
> should prevent me from doing that tomorrow afternoon?  

Not that I know of.  I was waiting to see if we could come up with other
patches, but I don't think that is going to happen anytime soon.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 09:56:10 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Features for next release
>> Bruce, you asked for a v6.5.3 to be released...anything outstanding that
>> should prevent me from doing that tomorrow afternoon?  

> Not that I know of.  I was waiting to see if we could come up with other
> patches, but I don't think that is going to happen anytime soon.

On the other hand, is there a reason to be in a rush to put out 6.5.3?
I didn't think we had many important changes from 6.5.2 yet.

I imagine that we will acquire a few more fixes for 6.5.* as time goes
on, and it looks like 7.0 will be months away.  Maybe a 6.5.* update
every month or two (if anything's been patched) would be the way to
proceed.

			regards, tom lane

Tue, 12 Oct 1999 10:14:17 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Features for next release 
> >> Bruce, you asked for a v6.5.3 to be released...anything outstanding that
> >> should prevent me from doing that tomorrow afternoon?  
> 
> > Not that I know of.  I was waiting to see if we could come up with other
> > patches, but I don't think that is going to happen anytime soon.
> 
> On the other hand, is there a reason to be in a rush to put out 6.5.3?
> I didn't think we had many important changes from 6.5.2 yet.
> 
> I imagine that we will acquire a few more fixes for 6.5.* as time goes
> on, and it looks like 7.0 will be months away.  Maybe a 6.5.* update
> every month or two (if anything's been patched) would be the way to
> proceed.
> 

Missing pgaccess source code.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 10:41:24 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Features for next release
On Tue, 12 Oct 1999, Tom Lane wrote:

> >> Bruce, you asked for a v6.5.3 to be released...anything outstanding that
> >> should prevent me from doing that tomorrow afternoon?  
> 
> > Not that I know of.  I was waiting to see if we could come up with other
> > patches, but I don't think that is going to happen anytime soon.
> 
> On the other hand, is there a reason to be in a rush to put out 6.5.3?
> I didn't think we had many important changes from 6.5.2 yet.

v6.5.3, I believe, was because PgAccess somehow got removed in v6.5.2 :(


Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Tue, 12 Oct 1999 11:44:29 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Features for next release 
> On Tue, 12 Oct 1999, Tom Lane wrote:
> 
> > >> Bruce, you asked for a v6.5.3 to be released...anything outstanding that
> > >> should prevent me from doing that tomorrow afternoon?  
> > 
> > > Not that I know of.  I was waiting to see if we could come up with other
> > > patches, but I don't think that is going to happen anytime soon.
> > 
> > On the other hand, is there a reason to be in a rush to put out 6.5.3?
> > I didn't think we had many important changes from 6.5.2 yet.
> 
> v6.5.3, I believe, was because PgAccess somehow got removed in v6.5.2 :(

OK, it was me, I didn't want to add it to 6.5.2 because it was not a
major bugfix, I got it 24 hours before release, it was a completely new
source tree layout, I asked the author to check my work, I didn't check
my work, and it was not committed properly.  I feel better now.  :-)

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 10:48:09 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Features for next release
The Hermit Hacker <scrappy@hub.org> writes:
>> On the other hand, is there a reason to be in a rush to put out 6.5.3?
>> I didn't think we had many important changes from 6.5.2 yet.

> v6.5.3, I believe, was because PgAccess somehow got removed in v6.5.2 :(

Ah, right, I'd forgotten that.  OK, we need a 6.5.3.

It sounds like Peter wants another day to check JDBC though...

			regards, tom lane

Tue, 12 Oct 1999 11:24:22 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Features for next release 
Bruce Momjian wrote:
  >> Bruce, you asked for a v6.5.3 to be released...anything outstanding that
  >> should prevent me from doing that tomorrow afternoon?  
  >
  >Not that I know of.  I was waiting to see if we could come up with other
  >patches, but I don't think that is going to happen anytime soon.

Will this include a patch to let NUMERIC fields be indexed?

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Blessed is the man who makes the LORD his trust, 
      who does not look to the proud, to those who turn 
      aside to false gods."            Psalms 40:4 



Tue, 12 Oct 1999 17:03:46 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Features for next release 
> Bruce Momjian wrote:
>   >> Bruce, you asked for a v6.5.3 to be released...anything outstanding that
>   >> should prevent me from doing that tomorrow afternoon?  
>   >
>   >Not that I know of.  I was waiting to see if we could come up with other
>   >patches, but I don't think that is going to happen anytime soon.
> 
> Will this include a patch to let NUMERIC fields be indexed?

Not sure.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 12:08:12 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Features for next release
>
> > > > Wow, that web page looks good now, with the quote at the bottom.  Jan,
> > > > we need the nicer world image.
> > >
> > >     You mean one with the mountains - no?
> > >
> > >     Well, I'll spend some time, polish up the Povray sources etc.
> > >     so Vince can easily maintain the map after  -  only  that  he
> > >     needs  Povray  3.1  and  maybe  Tcl/Tk 8.0 to do it, but that
> > >     shouldn't be a problem since both are  freely  available  and
> > >     easily to install.
>
> I have tcl 8.0.5 and povray here too.

    I've setup an example for the new developers page at

    <http://www.PostgreSQL.ORG/~wieck>

    The image size is adjusted for the page width.

    To   maintain   the   hotspots  I  made  a  little,  slightly
    overspecialized, Tcl/Tk application that creates the imagemap
    so it can easily be pasted into the page.

    I'll  contact you and Vince via private mail after packing it
    up.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #



Tue, 12 Oct 1999 18:20:18 +0200 (MET DST)
wieck@debis.com (Jan Wieck)
New developer globe (was: Re: [HACKERS] Interesting Quote you might
	enjoy about PGSQL.)
>
> > Bruce Momjian wrote:
> >   >> Bruce, you asked for a v6.5.3 to be released...anything outstanding that
> >   >> should prevent me from doing that tomorrow afternoon?
> >   >
> >   >Not that I know of.  I was waiting to see if we could come up with other
> >   >patches, but I don't think that is going to happen anytime soon.
> >
> > Will this include a patch to let NUMERIC fields be indexed?
>
> Not sure.

    Not  FOR  sure.  Adding  the  default  operator class etc. is
    adding catalog entries. Thus it requires "initdb" and  that's
    a NONO for bugfix releases according to our release policy.

    It's done in the v7.0 tree.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #



Tue, 12 Oct 1999 18:27:47 +0200 (MET DST)
wieck@debis.com (Jan Wieck)
Re: [HACKERS] Features for next release
> >
> > > > > Wow, that web page looks good now, with the quote at the bottom.  Jan,
> > > > > we need the nicer world image.
> > > >
> > > >     You mean one with the mountains - no?
> > > >
> > > >     Well, I'll spend some time, polish up the Povray sources etc.
> > > >     so Vince can easily maintain the map after  -  only  that  he
> > > >     needs  Povray  3.1  and  maybe  Tcl/Tk 8.0 to do it, but that
> > > >     shouldn't be a problem since both are  freely  available  and
> > > >     easily to install.
> >
> > I have tcl 8.0.5 and povray here too.
> 
>     I've setup an example for the new developers page at
> 
>     <http://www.PostgreSQL.ORG/~wieck>
> 
>     The image size is adjusted for the page width.
> 
>     To   maintain   the   hotspots  I  made  a  little,  slightly
>     overspecialized, Tcl/Tk application that creates the imagemap
>     so it can easily be pasted into the page.
> 
>     I'll  contact you and Vince via private mail after packing it
>     up.

Yikes, that's amazingly good looking.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 12:34:25 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might enjoy about PGSQL.)u
> > I have tcl 8.0.5 and povray here too.
> 
>     I've setup an example for the new developers page at
> 
>     <http://www.PostgreSQL.ORG/~wieck>
> 
>     The image size is adjusted for the page width.
> 
>     To   maintain   the   hotspots  I  made  a  little,  slightly
>     overspecialized, Tcl/Tk application that creates the imagemap
>     so it can easily be pasted into the page.
> 
>     I'll  contact you and Vince via private mail after packing it
>     up.

I just realized I can put my mouse over a pin, and the name appears. 
That is great.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 12:35:28 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might enjoy about PGSQL.)
Here is my proposal for an outline for a PostgreSQL book.  Many of us
have been asked by publishers about writing a book.  Here is what I
think would be a good outline for the book.

I am interested in whether this is a good outline for a PostgreSQL book,
how our existing documentation matches this outline, where our existing
documentation can be managed into a published book, etc.

Any comments would be welcome.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
...................................................................

The attached document is in both web page and text formats.
View the one which looks best.




PostgreSQL Book Proposal











PostgreSQL Book Proposal
Bruce Momjian



1.
Introduction
2.
Installation


(a)
Getting POSTGRESQL
(b)
Compiling
(c)
Initialization
(d)
Starting the server
(e)
Creating a database
(f)
Issuing database commands 
3.
Introduction to SQL



(a)
Why a database?
(b)
Creating tables 
(c)
Adding data with INSERT
(d)
Viewing data with SELECT
(e)
Removing data with DELETE
(f)
Modifying data with UPDATE
(g)
Restriction with WHERE
(h)
Sorting data with ORDER BY
(i)
Usage of NULL values
4.
Advanced SQL Commands



(a)
Inserting data from a SELECT
(b)
Aggregates: COUNT, SUM, etc.
(c)
GROUP BY with aggregates
(d)
HAVING with aggregates
(e)
Joining tables
(f)
Using table aliases
(g)
UNION clause
(h)
Subqueries
(i)
Transactions
(j)
Cursors
(k)
Indexing
(l)
Table defaults
(m)
Primary/Foreign keys
(n)
AND/OR usage 
(o)
LIKE clause usage
(p)
Temporary tables
(q)
Importing data
5.
POSTGRESQL'S Unique Features



(a)
Object ID'S (OID)
(b)
Multi-version Concurrency Control (MVCC)
(c)
Locking and Deadlocks
(d)
Vacuum
(e)
Views
(f)
Rules
(g)
Sequences
(h)
Triggers
(i)
Large Objects(BLOBS)
(j)
Adding User-defined Functions
(k)
Adding User-defined Operators
(l)
Adding User-defined Types
(m)
Exotic Preinstalled Types
(n)
Arrays
(o)
Inheritance
6.
Interfacing to the POSTGRESQL Database



(a)
C Language API
(b)
Embedded C
(c)
C++
(d)
JAVA
(e)
ODBC
(f)
PERL
(g)
TCL/TK
(h)
PYTHON
(i)
Web access (PHP)
(j)
Server-side programming (PLPGSQL and SPI)
7.
POSTGRESQL Adminstration



(a)
Creating users and databases
(b)
Backup and restore
(c)
Performance tuning
(d)
Troubleshooting
(e)
Customization options
(f)
Setting access permissions
8.
Additional Resources



(a)
Frequently Asked Questions (FAQ'S)
(b)
Mailing list support
(c)
Supplied documentation
(d)
Commercial support
(e)
Modifying the source code






                          PostgreSQL Book Proposal
                                      
                               Bruce Momjian
   
   1.
          Introduction
   2.
          Installation
        (a)
               Getting POSTGRESQL
        (b)
               Compiling
        (c)
               Initialization
        (d)
               Starting the server
        (e)
               Creating a database
        (f)
               Issuing database commands
   3.
          Introduction to SQL
        (a)
               Why a database?
        (b)
               Creating tables
        (c)
               Adding data with INSERT
        (d)
               Viewing data with SELECT
        (e)
               Removing data with DELETE
        (f)
               Modifying data with UPDATE
        (g)
               Restriction with WHERE
        (h)
               Sorting data with ORDER BY
        (i)
               Usage of NULL values
   4.
          Advanced SQL Commands
        (a)
               Inserting data from a SELECT
        (b)
               Aggregates: COUNT, SUM, etc.
        (c)
               GROUP BY with aggregates
        (d)
               HAVING with aggregates
        (e)
               Joining tables
        (f)
               Using table aliases
        (g)
               UNION clause
        (h)
               Subqueries
        (i)
               Transactions
        (j)
               Cursors
        (k)
               Indexing
        (l)
               Table defaults
        (m)
               Primary/Foreign keys
        (n)
               AND/OR usage
        (o)
               LIKE clause usage
        (p)
               Temporary tables
        (q)
               Importing data
   5.
          POSTGRESQL'S Unique Features
        (a)
               Object ID'S (OID)
        (b)
               Multi-version Concurrency Control (MVCC)
        (c)
               Locking and Deadlocks
        (d)
               Vacuum
        (e)
               Views
        (f)
               Rules
        (g)
               Sequences
        (h)
               Triggers
        (i)
               Large Objects(BLOBS)
        (j)
               Adding User-defined Functions
        (k)
               Adding User-defined Operators
        (l)
               Adding User-defined Types
        (m)
               Exotic Preinstalled Types
        (n)
               Arrays
        (o)
               Inheritance
   6.
          Interfacing to the POSTGRESQL Database
        (a)
               C Language API
        (b)
               Embedded C
        (c)
               C++
        (d)
               JAVA
        (e)
               ODBC
        (f)
               PERL
        (g)
               TCL/TK
        (h)
               PYTHON
        (i)
               Web access (PHP)
        (j)
               Server-side programming (PLPGSQL and SPI)
   7.
          POSTGRESQL Adminstration
        (a)
               Creating users and databases
        (b)
               Backup and restore
        (c)
               Performance tuning
        (d)
               Troubleshooting
        (e)
               Customization options
        (f)
               Setting access permissions
   8.
          Additional Resources
        (a)
               Frequently Asked Questions (FAQ'S)
        (b)
               Mailing list support
        (c)
               Supplied documentation
        (d)
               Commercial support
        (e)
               Modifying the source code
Tue, 12 Oct 1999 13:16:35 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Outline for PostgreSQL book
Jan Wieck wrote:
>     I've setup an example for the new developers page at
> 
>     <http://www.PostgreSQL.ORG/~wieck>
> 
>     The image size is adjusted for the page width.
> 
>     To   maintain   the   hotspots  I  made  a  little,  slightly
>     overspecialized, Tcl/Tk application that creates the imagemap
>     so it can easily be pasted into the page.

I get a Javascript error:
JavaScript Error: http://www.PostgreSQL.ORG/~wieck/, line 51:

pg_dot_start is not defined.

Netscape Communicator 4.61.

Otherwise, this globe is stunning.

There are a couple of typos (to be pedantic...): (;-))
Under Thomas, 'documentation' is incorrectly spelled.
Under Tom Lane, 'He has works on the optimizer', unless some arcane
usage of works is in vogue, should be either 'He works on' or 'He has
worked on', with the latter fitting with the tense already used in his
description.

If Vince wants to add my name as RPM maintainer, and no one objects, I
volunteer to continue to maintain the RedHat Linux RPM set (the last
revision of which shipped with RedHat 6.1).

Again, THIS GLOBE IS STUNNING!

Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 12 Oct 1999 13:21:57 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might enjoy about PGSQL.)
> Again, THIS GLOBE IS STUNNING!
> 

Yep.  That's the word for it.  I will fix the wording once Jan is
finished.  The mistakes are mine.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 13:26:25 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might enjoy about PGSQL.)
On Tue, 12 Oct 1999, Lamar Owen wrote:

> If Vince wants to add my name as RPM maintainer, and no one objects, I
> volunteer to continue to maintain the RedHat Linux RPM set (the last
> revision of which shipped with RedHat 6.1).

Vince always thought that was Bruce's page.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH   email: vev@michvhf.com   flame-mail: /dev/null
  # include <std/disclaimers.h>       Have you seen http://www.pop4.net?
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Tue, 12 Oct 1999 13:26:43 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might  enjoy about PGSQL.)
Lamar Owen wrote:

>
> Jan Wieck wrote:
> >     I've setup an example for the new developers page at
> >
> >     <http://www.PostgreSQL.ORG/~wieck>
> >
> >     The image size is adjusted for the page width.
> >
> >     To   maintain   the   hotspots  I  made  a  little,  slightly
> >     overspecialized, Tcl/Tk application that creates the imagemap
> >     so it can easily be pasted into the page.
>
> I get a Javascript error:
> JavaScript Error: http://www.PostgreSQL.ORG/~wieck/, line 51:
>
> pg_dot_start is not defined.

    Ooops  -  thanks and corrected, try again. I forgot to remove
    the onLoad() in the <body>.  That  blinking-dot  stuff  isn't
    required  anymore  since  the globe looks already nice enough
    without animation.

> Again, THIS GLOBE IS STUNNING!


    Tnx :-)


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #



Tue, 12 Oct 1999 19:26:59 +0200 (MET DST)
wieck@debis.com (Jan Wieck)
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might
> On Tue, 12 Oct 1999, Lamar Owen wrote:
> 
> > If Vince wants to add my name as RPM maintainer, and no one objects, I
> > volunteer to continue to maintain the RedHat Linux RPM set (the last
> > revision of which shipped with RedHat 6.1).
> 
> Vince always thought that was Bruce's page.
> 

Sure, I'll add you once Jan finishes.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 13:27:04 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might enjoy about PGSQL.)
Jan Wieck wrote:
> > pg_dot_start is not defined.
> 
>     Ooops  -  thanks and corrected, try again. I forgot to remove
>     the onLoad() in the <body>.  That  blinking-dot  stuff  isn't
>     required  anymore  since  the globe looks already nice enough
>     without animation.

Ah, yes.  Much better.

--
Lamar Owen
WGCR Internet Radio
Pisgah Forest, North Carolina
1 Peter 4:11

Tue, 12 Oct 1999 13:40:05 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might
On Tue, 12 Oct 1999, Bruce Momjian wrote:

> > > I have tcl 8.0.5 and povray here too.
> > 
> >     I've setup an example for the new developers page at
> > 
> >     <http://www.PostgreSQL.ORG/~wieck>
> > 
> >     The image size is adjusted for the page width.
> > 
> >     To   maintain   the   hotspots  I  made  a  little,  slightly
> >     overspecialized, Tcl/Tk application that creates the imagemap
> >     so it can easily be pasted into the page.
> > 
> >     I'll  contact you and Vince via private mail after packing it
> >     up.
> 
> I just realized I can put my mouse over a pin, and the name appears. 
> That is great.

most cool...one thing though...considering teh width of those tags, how
about adding location info in there also?  For instance "Edmund, Mergyl:
Stuttgart, Germany" ... :)

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Tue, 12 Oct 1999 14:43:54 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might enjoy about PGSQL.)
> > >     I'll  contact you and Vince via private mail after packing it
> > >     up.
> > 
> > I just realized I can put my mouse over a pin, and the name appears. 
> > That is great.
> 
> most cool...one thing though...considering teh width of those tags, how
> about adding location info in there also?  For instance "Edmund, Mergyl:
> Stuttgart, Germany" ... :)

Marc, let's not get Jan upset.  :-)

Their location is already in the lower page next to their name anyway.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 13:45:15 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might enjoy about PGSQL.)
>
> > > >     I'll  contact you and Vince via private mail after packing it
> > > >     up.
> > >
> > > I just realized I can put my mouse over a pin, and the name appears.
> > > That is great.
> >
> > most cool...one thing though...considering teh width of those tags, how
> > about adding location info in there also?  For instance "Edmund, Mergyl:
> > Stuttgart, Germany" ... :)
>
> Marc, let's not get Jan upset.  :-)

    Bruce, to upset me Marc needs alot of more efford!

    And  why not? If you UPDATE you'll see that we can get rid of
    most of the entire body. Remember - a picture says more  than
    a thousand words :-)

    Instead  there  could  be more important information like our
    release policy  (no  features  in  bufix  releases,  backward
    compatibility    in    minor   releases),   how   to   submit
    patches/improvements, where to find documentation on  how  to
    enhance/hack PostgreSQL etc.

>
> Their location is already in the lower page next to their name anyway.



Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #



Tue, 12 Oct 1999 21:11:23 +0200 (MET DST)
wieck@debis.com (Jan Wieck)
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might
Hi,

who is a maintainer of Developers List. 
I understand that I'm not a postgres coder but is't
possible to mention somewhere:

Bartunov, Oleg in Moscow, Russia (oleg@sai.msu.su)
has introduced locale support.

It was rather long ago and was very limited but
I still proud for this little hack many people
used. (take into account I'm not a C-programmer at all :-)


	Regards,

		Oleg

_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83


Tue, 12 Oct 1999 23:28:48 +0400 (MSD)
Oleg Bartunov <oleg@sai.msu.su>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might enjoy about PGSQL.)
> > Marc, let's not get Jan upset.  :-)
> 
>     Bruce, to upset me Marc needs alot of more efford!
> 
>     And  why not? If you UPDATE you'll see that we can get rid of
>     most of the entire body. Remember - a picture says more  than
>     a thousand words :-)

Wow, that is cool.  No need for text at bottom, except to list names and
e-mail addresses.

>     Instead  there  could  be more important information like our
>     release policy  (no  features  in  bufix  releases,  backward
>     compatibility    in    minor   releases),   how   to   submit
>     patches/improvements, where to find documentation on  how  to
>     enhance/hack PostgreSQL etc.

That's an interesting idea, though I think the documentation or download
page may be a better place for that.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 15:37:39 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might
> Hi,
> 
> who is a maintainer of Developers List. 
> I understand that I'm not a postgres coder but is't
> possible to mention somewhere:
> 
> Bartunov, Oleg in Moscow, Russia (oleg@sai.msu.su)
> has introduced locale support.
> 
> It was rather long ago and was very limited but
> I still proud for this little hack many people
> used. (take into account I'm not a C-programmer at all :-)

Sure, let me do that when Jan is done.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 15:38:22 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might enjoy about PGSQL.)
Bruce Momjian wrote:
  >Here is my proposal for an outline for a PostgreSQL book.  Many of us
  >have been asked by publishers about writing a book.  Here is what I
  >think would be a good outline for the book.
  >
  >I am interested in whether this is a good outline for a PostgreSQL book,
  >how our existing documentation matches this outline, where our existing
  >documentation can be managed into a published book, etc.
  >
  >Any comments would be welcome.
 


  >                          PostgreSQL Book Proposal
  >                                      
  >                               Bruce Momjian
  >   
  >   1.
  >          Introduction
  >   2.
  >          Installation
...
  >   3.
  >          Introduction to SQL
...

It looks good; I have a comment on the order of chapters, however.

I suggest that Installlation goes in an Appendix.  More and more, people
will be coming to machines that already have software installed, or have
them installed as packages (.rpm or .deb).

Installation is the administrator's job, and not of interest to `normal'
users, so it should not be placed in the book as if every user had to do
it.

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Blessed is the man who makes the LORD his trust, 
      who does not look to the proud, to those who turn 
      aside to false gods."            Psalms 40:4 



Tue, 12 Oct 1999 21:49:55 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Outline for PostgreSQL book 
Bruce Momjian wrote:
  >> > Marc, let's not get Jan upset.  :-)
  >> 
  >>     Bruce, to upset me Marc needs alot of more efford!
  >> 
  >>     And  why not? If you UPDATE you'll see that we can get rid of
  >>     most of the entire body. Remember - a picture says more  than
  >>     a thousand words :-)
  >
  >Wow, that is cool.  No need for text at bottom, except to list names and
  >e-mail addresses.
 
Don't forget the people who use Lynx and other such browsers.

In Lynx the JavaScript stuff is not visible at all, so there ought to be
normal text for those who cannot display images.

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Blessed is the man who makes the LORD his trust, 
      who does not look to the proud, to those who turn 
      aside to false gods."            Psalms 40:4 



Tue, 12 Oct 1999 21:53:56 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: New developer globe (was: Re: [HACKERS] Interesting Quote you
	might
> It looks good; I have a comment on the order of chapters, however.
> 
> I suggest that Installlation goes in an Appendix.  More and more, people
> will be coming to machines that already have software installed, or have
> them installed as packages (.rpm or .deb).

> 
> Installation is the administrator's job, and not of interest to `normal'
> users, so it should not be placed in the book as if every user had to do
> it.

Done.  Certainly better to get that at the end.  That chapter is going
to be a mess to look at.

New version attached.  No PDF version this time.  Do people like PDF?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
...................................................................

The attached document is in both web page and text formats.
View the one which looks best.




PostgreSQL Book Proposal











PostgreSQL Book Proposal
Bruce Momjian



1.
Introduction


(a)
History of POSTGRESQL
(b)
Open source software
(c)
When to use a database
2.
Issuing database commands



(a)
Starting a database session
(b)
Controlling a session
(c)
Sending queries
(d)
Getting help
3.
Introduction to SQL



(a)
Creating tables 
(b)
Adding data with INSERT
(c)
Viewing data with SELECT
(d)
Removing data with DELETE
(e)
Modifying data with UPDATE
(f)
Restricting with WHERE
(g)
Sorting data with ORDER BY
(h)
Using NULL values
4.
Advanced SQL Commands



(a)
Inserting data from a SELECT
(b)
Aggregates: COUNT, SUM, ...
(c)
GROUP BY with aggregates
(d)
HAVING with aggregates
(e)
Joining tables
(f)
Using table aliases
(g)
UNION clause
(h)
Subqueries
(i)
Transactions
(j)
Cursors
(k)
Indexing
(l)
Column defaults
(m)
Primary/foreign keys
(n)
AND/OR usage 
(o)
LIKE clause usage
(p)
Temporary tables
(q)
Importing data
5.
POSTGRESQL'S Unique Features



(a)
Object ID'S (OID'S)
(b)
Multi-Version Concurrency Control
(c)
Locking and deadlocks
(d)
Vacuum
(e)
Views
(f)
Rules
(g)
Sequences
(h)
Triggers
(i)
Large objects(BLOBS)
(j)
Adding user-defined functions
(k)
Adding user-defined operators
(l)
Adding user-defined types
(m)
Exotic pre-installed types
(n)
Arrays
(o)
Inheritance
6.
Interfacing to the POSTGRESQL Database



(a)
C Language API
(b)
Embedded C
(c)
C++
(d)
JAVA
(e)
ODBC
(f)
PERL
(g)
TCL/TK
(h)
PYTHON
(i)
Web access (PHP)
(j)
Server-side programming (PLPGSQL and SPI)
7.
POSTGRESQL Administration



(a)
Creating users and databases
(b)
Backup and restore
(c)
Performance
(d)
Troubleshooting
(e)
Customization
(f)
Setting access permissions
(g)
International character encodings
8.
Additional Resources



(a)
Frequently Asked Questions (FAQ'S)
(b)
Mailing list support
(c)
Supplied documentation
(d)
Commercial support
(e)
Modifying the source code
9.
Appendix: Installation



(a)
Getting POSTGRESQL
(b)
Compiling
(c)
Initialization
(d)
Starting the server
(e)
Creating a database
10.
Annotated Bibliography






                          PostgreSQL Book Proposal
                                      
                               Bruce Momjian
   
   1.
          Introduction
        (a)
               History of POSTGRESQL
        (b)
               Open source software
        (c)
               When to use a database
   2.
          Issuing database commands
        (a)
               Starting a database session
        (b)
               Controlling a session
        (c)
               Sending queries
        (d)
               Getting help
   3.
          Introduction to SQL
        (a)
               Creating tables
        (b)
               Adding data with INSERT
        (c)
               Viewing data with SELECT
        (d)
               Removing data with DELETE
        (e)
               Modifying data with UPDATE
        (f)
               Restricting with WHERE
        (g)
               Sorting data with ORDER BY
        (h)
               Using NULL values
   4.
          Advanced SQL Commands
        (a)
               Inserting data from a SELECT
        (b)
               Aggregates: COUNT, SUM, ...
        (c)
               GROUP BY with aggregates
        (d)
               HAVING with aggregates
        (e)
               Joining tables
        (f)
               Using table aliases
        (g)
               UNION clause
        (h)
               Subqueries
        (i)
               Transactions
        (j)
               Cursors
        (k)
               Indexing
        (l)
               Column defaults
        (m)
               Primary/foreign keys
        (n)
               AND/OR usage
        (o)
               LIKE clause usage
        (p)
               Temporary tables
        (q)
               Importing data
   5.
          POSTGRESQL'S Unique Features
        (a)
               Object ID'S (OID'S)
        (b)
               Multi-Version Concurrency Control
        (c)
               Locking and deadlocks
        (d)
               Vacuum
        (e)
               Views
        (f)
               Rules
        (g)
               Sequences
        (h)
               Triggers
        (i)
               Large objects(BLOBS)
        (j)
               Adding user-defined functions
        (k)
               Adding user-defined operators
        (l)
               Adding user-defined types
        (m)
               Exotic pre-installed types
        (n)
               Arrays
        (o)
               Inheritance
   6.
          Interfacing to the POSTGRESQL Database
        (a)
               C Language API
        (b)
               Embedded C
        (c)
               C++
        (d)
               JAVA
        (e)
               ODBC
        (f)
               PERL
        (g)
               TCL/TK
        (h)
               PYTHON
        (i)
               Web access (PHP)
        (j)
               Server-side programming (PLPGSQL and SPI)
   7.
          POSTGRESQL Administration
        (a)
               Creating users and databases
        (b)
               Backup and restore
        (c)
               Performance
        (d)
               Troubleshooting
        (e)
               Customization
        (f)
               Setting access permissions
        (g)
               International character encodings
   8.
          Additional Resources
        (a)
               Frequently Asked Questions (FAQ'S)
        (b)
               Mailing list support
        (c)
               Supplied documentation
        (d)
               Commercial support
        (e)
               Modifying the source code
   9.
          Appendix: Installation
        (a)
               Getting POSTGRESQL
        (b)
               Compiling
        (c)
               Initialization
        (d)
               Starting the server
        (e)
               Creating a database
   10.
          Annotated Bibliography
Tue, 12 Oct 1999 16:56:04 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Outline for PostgreSQL book

On 12-Oct-99 Bruce Momjian wrote:
>> It looks good; I have a comment on the order of chapters, however.
>> 
>> I suggest that Installlation goes in an Appendix.  More and more, people
>> will be coming to machines that already have software installed, or have
>> them installed as packages (.rpm or .deb).
> 
>> 
>> Installation is the administrator's job, and not of interest to `normal'
>> users, so it should not be placed in the book as if every user had to do
>> it.
> 
> Done.  Certainly better to get that at the end.  That chapter is going
> to be a mess to look at.
> 
> New version attached.  No PDF version this time.  Do people like PDF?

Not at this early stage.  Wait till it's further along.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH   email: vev@michvhf.com   flame-mail: /dev/null
  # include <std/disclaimers.h>       Have you seen http://www.pop4.net?
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================



Tue, 12 Oct 1999 17:18:11 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: [HACKERS] Outline for PostgreSQL book
Bruce Momjian <maillist@candle.pha.pa.us> writes:

>                           PostgreSQL Book Proposal
>                                       
>                                Bruce Momjian

[...]

>    4.
>           Advanced SQL Commands

[...]


>    6.
>           Interfacing to the POSTGRESQL Database
>         (a)
>                C Language API
>         (b)
>                Embedded C
>         (c)
>                C++
>         (d)
>                JAVA
>         (e)
>                ODBC
>         (f)
>                PERL
>         (g)
>                TCL/TK
>         (h)
>                PYTHON
>         (i)
>                Web access (PHP)
>         (j)
>                Server-side programming (PLPGSQL and SPI)

Isn't (j) logically part of chapter 4?  (Or 5, if it's PostgreSQL
specific.)  Or am I completely confused?  (Where can I read about
PLPGSQL and/or SPI, other than in the forthcoming book?)

If it came to a choice between having very short sections in chapter
6, and having two or three of them covered in more depth, I'd go for
the latter.  

(Of course, you'll inevitably choose two or three which don't match
what many readers will want (whichever two or three you choose), but
even so, I think I'd get more out of a reasonably thorough coverage of
a couple of languages that I won't use than superficial coverage of
all of them which doesn't really reveal anything useful.)

12 Oct 1999 23:08:05 +0100
Bruce Stephens <bruce@cenderis.demon.co.uk>
Re: [DOCS] Outline for PostgreSQL book
At 16:56 12-10-99 -0400, Bruce Momjian wrote: 
>
> > Installation is the administrator's job, and not of interest to `normal'
> > users, so it should not be placed in the book as if every user had to do
> > it.
> Done.  Certainly better to get that at the end.  That chapter is going
> to be a mess to look at.


I agree on this particular item BUT...

wouldn't it be nice if the book would be a 'hard' book?
there already are lots of database tutorials, introductions to sql etc.

focus on db developers, go into (maybe not that deep) how and why postgres
is/was developed the way it is/was.

'hard' books are a 'good thing' (tm) and it would be too bad if this would
become another entry-level booklet.

OTOH an entry-level book is probably required to get as big a user-base as
possible.



Wed, 13 Oct 1999 01:27:29 +0200
gravity <z.nijmeyers@cable.a2000.nl>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
At 16:56 12-10-99 -0400, Bruce Momjian wrote: 
> Installation is the administrator's job, and not of interest to `normal'
> users, so it should not be placed in the book as if every user had to do
> it.
Done.  Certainly better to get that at the end.  That chapter is going
to be a mess to look at.

I agree on this particular item BUT...

wouldn't it be nice if the book would be a 'hard' book?
there already are lots of database tutorials, introductions to sql etc.

focus on db developers, go into (maybe not that deep) how and why postgres
is/was developed the way it is/was.

'hard' books are a 'good thing' (tm) and it would be too bad if this would
become another entry-level booklet.

OTOH an entry-level book is probably required to get as big a user-base as
possible.



Wed, 13 Oct 1999 01:41:27 +0200
gravity <gravity@dds.nl>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book

On 12-Oct-99 Bruce Momjian wrote:
>> It looks good; I have a comment on the order of chapters, however.
>> 
>> I suggest that Installlation goes in an Appendix.  More and more, people
>> will be coming to machines that already have software installed, or have
>> them installed as packages (.rpm or .deb).
> 
>> 
>> Installation is the administrator's job, and not of interest to `normal'
>> users, so it should not be placed in the book as if every user had to do
>> it.
> 
> Done.  Certainly better to get that at the end.  That chapter is going
> to be a mess to look at.

It doesn't have to be a mess.  

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH   email: vev@michvhf.com   flame-mail: /dev/null
  # include <std/disclaimers.h>       Have you seen http://www.pop4.net?
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================



Tue, 12 Oct 1999 19:56:11 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
RE: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> I agree on this particular item BUT...
> 
> wouldn't it be nice if the book would be a 'hard' book?
> there already are lots of database tutorials, introductions to sql etc.
> 
> focus on db developers, go into (maybe not that deep) how and why postgres
> is/was developed the way it is/was.
> 
> 'hard' books are a 'good thing' (tm) and it would be too bad if this would
> become another entry-level booklet.
> 
> OTOH an entry-level book is probably required to get as big a user-base as
> possible.

Publishers have already talked to me about multiple books.  I think we
need to start with an newbie book, with the chapters clearly arranged so
experienced people can skip newbie chapters.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 20:02:35 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> 
> On 12-Oct-99 Bruce Momjian wrote:
> >> It looks good; I have a comment on the order of chapters, however.
> >> 
> >> I suggest that Installlation goes in an Appendix.  More and more, people
> >> will be coming to machines that already have software installed, or have
> >> them installed as packages (.rpm or .deb).
> > 
> >> 
> >> Installation is the administrator's job, and not of interest to `normal'
> >> users, so it should not be placed in the book as if every user had to do
> >> it.
> > 
> > Done.  Certainly better to get that at the end.  That chapter is going
> > to be a mess to look at.
> 
> It doesn't have to be a mess.  
> 

Frankly, if I can get away with having whole sections that point to URL
files, so much the better.  If I can just point them to the INSTALL
file, and be done with it, great.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 20:03:47 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book

On 13-Oct-99 Bruce Momjian wrote:
>> 
>> On 12-Oct-99 Bruce Momjian wrote:
>> >> It looks good; I have a comment on the order of chapters, however.
>> >> 
>> >> I suggest that Installlation goes in an Appendix.  More and more, people
>> >> will be coming to machines that already have software installed, or have
>> >> them installed as packages (.rpm or .deb).
>> > 
>> >> 
>> >> Installation is the administrator's job, and not of interest to `normal'
>> >> users, so it should not be placed in the book as if every user had to do
>> >> it.
>> > 
>> > Done.  Certainly better to get that at the end.  That chapter is going
>> > to be a mess to look at.
>> 
>> It doesn't have to be a mess.  
>> 
> 
> Frankly, if I can get away with having whole sections that point to URL
> files, so much the better.  If I can just point them to the INSTALL
> file, and be done with it, great.

The installation can be quick and painless.  I'll explain more tomorrow.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH   email: vev@michvhf.com   flame-mail: /dev/null
  # include <std/disclaimers.h>       Have you seen http://www.pop4.net?
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================



Tue, 12 Oct 1999 20:55:59 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
Bruce Momjian wrote:

> > OTOH an entry-level book is probably required to get as big a user-base as
> > possible.
>
> Publishers have already talked to me about multiple books.  I think we
> need to start with an newbie book, with the chapters clearly arranged so
> experienced people can skip newbie chapters.

    I  don't  think it really matters that much if the first book
    about PostgreSQL is more for a newbie than a professional  or
    vice  versa.  What  count's  is  that  it  is  up to date and
    correct. If I go to a book store and find only one book on  a
    topic,  it's  usually  not  the one "I" was looking for.  But
    what would make other authors write another book on the  same
    topic  -  most likely the authors who write details I haven't
    known before? It's the success of the former one.

    Think about it a little.

    The first book has to be  successful.  Therefore  it  has  to
    address  most of the interested people. Those who know how to
    get the information they need out of manpages, RFC's and  W3C
    recommendations  aren't  the ppl who to address in this case.
    So let it please be a newbie book, and  the  hard  ones  will
    follow.

    Another  problem  is  that during the last release cycles, it
    wasn't  that  easy  to  follow  all  the   changes   in   the
    capabilities  of  PostgreSQL.   Not  even for me, and I'm not
    counting myself to the outermost circle.  Now what chance  do
    you  give a book that's written based on v6.5 if we are about
    to release v7.1 some months ahead? And more important, if  it
    happens  this  way,  does our "aggressive" development invite
    other authors to take a chance on the  same  topic?  I  don't
    think so.

    If  we  really  want professional publishing about PostgreSQL
    (we want - no?), the core team has  to  co-operate  with  the
    authors  of  those  books in a way, that they can write their
    book based on the upcoming release and  sell  it  with  a  CD
    where that release is included.  At the time it is published,
    there should only be bugfixes available  on  the  net  -  not
    already two newer releases.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #



Wed, 13 Oct 1999 04:15:07 +0200 (MET DST)
wieck@debis.com (Jan Wieck)
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> Bruce Momjian wrote:
> 
> > > OTOH an entry-level book is probably required to get as big a user-base as
> > > possible.
> >
> > Publishers have already talked to me about multiple books.  I think we
> > need to start with an newbie book, with the chapters clearly arranged so
> > experienced people can skip newbie chapters.
> 
>     I  don't  think it really matters that much if the first book
>     about PostgreSQL is more for a newbie than a professional  or
>     vice  versa.  What  count's  is  that  it  is  up to date and
>     correct. If I go to a book store and find only one book on  a
>     topic,  it's  usually  not  the one "I" was looking for.  But
>     what would make other authors write another book on the  same
>     topic  -  most likely the authors who write details I haven't
>     known before? It's the success of the former one.
> 
>     Think about it a little.
> 
>     The first book has to be  successful.  Therefore  it  has  to
>     address  most of the interested people. Those who know how to
>     get the information they need out of manpages, RFC's and  W3C
>     recommendations  aren't  the ppl who to address in this case.
>     So let it please be a newbie book, and  the  hard  ones  will
>     follow.



That was my thought too.


>     Another  problem  is  that during the last release cycles, it
>     wasn't  that  easy  to  follow  all  the   changes   in   the
>     capabilities  of  PostgreSQL.   Not  even for me, and I'm not
>     counting myself to the outermost circle.  Now what chance  do
>     you  give a book that's written based on v6.5 if we are about
>     to release v7.1 some months ahead? And more important, if  it
>     happens  this  way,  does our "aggressive" development invite
>     other authors to take a chance on the  same  topic?  I  don't
>     think so.
> 
>     If  we  really  want professional publishing about PostgreSQL
>     (we want - no?), the core team has  to  co-operate  with  the
>     authors  of  those  books in a way, that they can write their
>     book based on the upcoming release and  sell  it  with  a  CD
>     where that release is included.  At the time it is published,
>     there should only be bugfixes available  on  the  net  -  not
>     already two newer releases.

I have a list of interested publishers, and am going to post it so
people can get involved and start writing.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Oct 1999 22:36:54 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
You should put one special section to talk about DATE. (datetime,
abstime, etc..)
That is the most asked by the mailing list.
You could put tons of samples on how to use and manipulate DATE.
ie. How to get the time, how to get the minute, how to get the hour,
    how to get the month in number (1,2,3,4),
    how to get the month in word (JAN, FEB, MAR)

The most important thing is to have alot of samples.
Practical samples are more useful than just syntax.

Regards,
Chairudin Sentosa

> Bruce Momjian wrote:
> 
> Here is my proposal for an outline for a PostgreSQL book.  Many of us
> have been asked by publishers about writing a book.  Here is what I
> think would be a good outline for the book.
> 
> I am interested in whether this is a good outline for a PostgreSQL
> book,
> how our existing documentation matches this outline, where our
> existing
> documentation can be managed into a published book, etc.
> 
> Any comments would be welcome.
> 
> --
>   Bruce Momjian                        |  http://www.op.net/~candle
>   maillist@candle.pha.pa.us            |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania
> 19026
> ...................................................................
> 
> The attached document is in both web page and text formats.
> View the one which looks best.
> 
>     ---------------------------------------------------------------
> 
>                        PostgreSQL Book Proposal
> 
>                              Bruce Momjian
> 
> 1.   Introduction
> 2.   Installation
> 
>      (a)  Getting POSTGRESQL
>      (b)  Compiling
>      (c)  Initialization
>      (d)  Starting the server
>      (e)  Creating a database
>      (f)  Issuing database commands
> 3.   Introduction to SQL
> 
>      (a)  Why a database?
>      (b)  Creating tables
>      (c)  Adding data with INSERT
>      (d)  Viewing data with SELECT
>      (e)  Removing data with DELETE
>      (f)  Modifying data with UPDATE
>      (g)  Restriction with WHERE
>      (h)  Sorting data with ORDER BY
>      (i)  Usage of NULL values
> 4.   Advanced SQL Commands
> 
>      (a)  Inserting data from a SELECT
>      (b)  Aggregates: COUNT, SUM, etc.
>      (c)  GROUP BY with aggregates
>      (d)  HAVING with aggregates
>      (e)  Joining tables
>      (f)  Using table aliases
>      (g)  UNION clause
>      (h)  Subqueries
>      (i)  Transactions
>      (j)  Cursors
>      (k)  Indexing
>      (l)  Table defaults
>      (m)  Primary/Foreign keys
>      (n)  AND/OR usage
>      (o)  LIKE clause usage
>      (p)  Temporary tables
>      (q)  Importing data
> 5.   POSTGRESQL'S Unique Features
> 
>      (a)  Object ID'S (OID)
>      (b)  Multi-version Concurrency Control (MVCC)
>      (c)  Locking and Deadlocks
>      (d)  Vacuum
>      (e)  Views
>      (f)  Rules
>      (g)  Sequences
>      (h)  Triggers
>      (i)  Large Objects(BLOBS)
>      (j)  Adding User-defined Functions
>      (k)  Adding User-defined Operators
>      (l)  Adding User-defined Types
>      (m)  Exotic Preinstalled Types
>      (n)  Arrays
>      (o)  Inheritance
> 6.   Interfacing to the POSTGRESQL Database
> 
>      (a)  C Language API
>      (b)  Embedded C
>      (c)  C++
>      (d)  JAVA
>      (e)  ODBC
>      (f)  PERL
>      (g)  TCL/TK
>      (h)  PYTHON
>      (i)  Web access (PHP)
>      (j)  Server-side programming (PLPGSQL and SPI)
> 7.   POSTGRESQL Adminstration
> 
>      (a)  Creating users and databases
>      (b)  Backup and restore
>      (c)  Performance tuning
>      (d)  Troubleshooting
>      (e)  Customization options
>      (f)  Setting access permissions
> 8.   Additional Resources
> 
>      (a)  Frequently Asked Questions (FAQ'S)
>      (b)  Mailing list support
>      (c)  Supplied documentation
>      (d)  Commercial support
>      (e)  Modifying the source code
> 
>     ---------------------------------------------------------------
> 
>                           PostgreSQL Book Proposal
> 
>                                Bruce Momjian
> 
>    1.
>           Introduction
>    2.
>           Installation
>         (a)
>                Getting POSTGRESQL
>         (b)
>                Compiling
>         (c)
>                Initialization
>         (d)
>                Starting the server
>         (e)
>                Creating a database
>         (f)
>                Issuing database commands
>    3.
>           Introduction to SQL
>         (a)
>                Why a database?
>         (b)
>                Creating tables
>         (c)
>                Adding data with INSERT
>         (d)
>                Viewing data with SELECT
>         (e)
>                Removing data with DELETE
>         (f)
>                Modifying data with UPDATE
>         (g)
>                Restriction with WHERE
>         (h)
>                Sorting data with ORDER BY
>         (i)
>                Usage of NULL values
>    4.
>           Advanced SQL Commands
>         (a)
>                Inserting data from a SELECT
>         (b)
>                Aggregates: COUNT, SUM, etc.
>         (c)
>                GROUP BY with aggregates
>         (d)
>                HAVING with aggregates
>         (e)
>                Joining tables
>         (f)
>                Using table aliases
>         (g)
>                UNION clause
>         (h)
>                Subqueries
>         (i)
>                Transactions
>         (j)
>                Cursors
>         (k)
>                Indexing
>         (l)
>                Table defaults
>         (m)
>                Primary/Foreign keys
>         (n)
>                AND/OR usage
>         (o)
>                LIKE clause usage
>         (p)
>                Temporary tables
>         (q)
>                Importing data
>    5.
>           POSTGRESQL'S Unique Features
>         (a)
>                Object ID'S (OID)
>         (b)
>                Multi-version Concurrency Control (MVCC)
>         (c)
>                Locking and Deadlocks
>         (d)
>                Vacuum
>         (e)
>                Views
>         (f)
>                Rules
>         (g)
>                Sequences
>         (h)
>                Triggers
>         (i)
>                Large Objects(BLOBS)
>         (j)
>                Adding User-defined Functions
>         (k)
>                Adding User-defined Operators
>         (l)
>                Adding User-defined Types
>         (m)
>                Exotic Preinstalled Types
>         (n)
>                Arrays
>         (o)
>                Inheritance
>    6.
>           Interfacing to the POSTGRESQL Database
>         (a)
>                C Language API
>         (b)
>                Embedded C
>         (c)
>                C++
>         (d)
>                JAVA
>         (e)
>                ODBC
>         (f)
>                PERL
>         (g)
>                TCL/TK
>         (h)
>                PYTHON
>         (i)
>                Web access (PHP)
>         (j)
>                Server-side programming (PLPGSQL and SPI)
>    7.
>           POSTGRESQL Adminstration
>         (a)
>                Creating users and databases
>         (b)
>                Backup and restore
>         (c)
>                Performance tuning
>         (d)
>                Troubleshooting
>         (e)
>                Customization options
>         (f)
>                Setting access permissions
>    8.
>           Additional Resources
>         (a)
>                Frequently Asked Questions (FAQ'S)
>         (b)
>                Mailing list support
>         (c)
>                Supplied documentation
>         (d)
>                Commercial support
>         (e)
>                Modifying the source code

Wed, 13 Oct 1999 11:22:01 +0700
Chairudin Sentosa Harjo <chai@prima.net.id>
Re: [HACKERS] Outline for PostgreSQL book
> Here is my proposal for an outline for a PostgreSQL book.  Many of us
> have been asked by publishers about writing a book.  Here is what I
> think would be a good outline for the book.
> 
> I am interested in whether this is a good outline for a PostgreSQL book,
> how our existing documentation matches this outline, where our existing
> documentation can be managed into a published book, etc.
> 
> Any comments would be welcome.

A chapter following the Introduction discussing the philosopy behind
Postgres development, the benefits of Postgres,  and an outline of the
architecture would be useful.

For the rest - I can't wait :)

--------
Regards
Theo

Wed, 13 Oct 1999 09:53:45 +0200
Theo Kramer <theo@flame.co.za>
Re: [HACKERS] Outline for PostgreSQL book

On 13-Oct-99 Vince Vielhaber wrote:
> 
> On 13-Oct-99 Bruce Momjian wrote:

2BOOK Authors:
Please, try to keep rights for translating this book into another
languages by you self, not by publisher.


I may ask some St.Pitersburg's publishing company 
to make russian translation of this book, but some publishers
like O'Reilly have too hard license policy 
and too long reaction time. 


---
Dmitry Samersoff, dms@wplus.net, ICQ:3161705
http://devnull.wplus.net
* There will come soft rains ...

Wed, 13 Oct 1999 12:06:59 +0400 (MSD)
Dmitry Samersoff <dms@wplus.net>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
Dmitry Samersoff wrote:
> 
> On 13-Oct-99 Vince Vielhaber wrote:
> >
> > On 13-Oct-99 Bruce Momjian wrote:
> 
> 2BOOK Authors:
> Please, try to keep rights for translating this book into another
> languages by you self, not by publisher.
> 
> I may ask some St.Pitersburg's publishing company
> to make russian translation of this book, but some publishers
> like O'Reilly have too hard license policy
> and too long reaction time.
> 

I may ask some Indonesian's publishing company to make
Indonesian translation of this book too.
I may help the translation from English to Indonesian language.

Regards,
Chairudin Sentosa

Wed, 13 Oct 1999 15:21:35 +0700
Chairudin Sentosa Harjo <chai@prima.net.id>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
Chairudin Sentosa Harjo wrote:
> 
> You should put one special section to talk about DATE. (datetime,
> abstime, etc..)
> That is the most asked by the mailing list.
> You could put tons of samples on how to use and manipulate DATE.
> ie. How to get the time, how to get the minute, how to get the hour,
>     how to get the month in number (1,2,3,4),
>     how to get the month in word (JAN, FEB, MAR)
    how to select all individuals that are more than 50 years old today
at 12:00PM

And ouf course about TimeZones (why sometimes I insert one date and 
get out another ;-p)

And of course mention that mostly it is not PostgreSQL's problem but
just a very hairy subject ;(

And perhaps some discussion about Date/Time types also in the section 
for programming languages (what you get from date field, how to be 
sure that what you give to postgresql is understood correctly)

> The most important thing is to have alot of samples.
> Practical samples are more useful than just syntax.

Agreed.

Maybe "Advanced SQL Commands" could also contain:

* Outer Joins
* Casts (bot AS and ::, with some explanations on why and when)

Not sure if Views and Triggers should go under Uniqe features - they are 
supported on most commercial SQL databases.

"Interfacing to the POSTGRESQL Database" could also contain more options
than PHP,
at least as references - there are much more of both established (CGI,
PyApache,mod_perl) 
and emerging (like the recent pgxml announcement) technologies for that
purpos.

Server side programming should also mention PL/Tcl 

Under which section would requirements the restrictions (db size, field
size/count, record size/count) 
or lackof them :) go?
(I recently checked Oracle8i for Linux, and it quoted 128MB RAM as
minimum and 256 
as recommended - I think we are a lot less demanding, I guess PG can
reasonably run in 16MB ?)

-----------------
Hannu

Wed, 13 Oct 1999 12:43:55 +0300
Hannu Krosing <hannu@trust.ee>
Re: [HACKERS] Outline for PostgreSQL book


> Bruce Momjian wrote:
> 
> 
> New version attached.  No PDF version this time.  Do people like PDF?
> 

There could be also a chapter on client side tools - pgAccess et.al.
and also small (but working ;) examples for using ODBC from MS Access, 
MS Query, Delphi, JBuilder, Visual Cafe - each about 2-5 pages to get 
people started.

--------------
Hannu

Wed, 13 Oct 1999 12:48:58 +0300
Hannu Krosing <hannu@trust.ee>
Re: [HACKERS] Outline for PostgreSQL book
On Tue, 12 Oct 1999, Vince Vielhaber wrote:

> 
> On 13-Oct-99 Bruce Momjian wrote:
> > Frankly, if I can get away with having whole sections that point to URL
> > files, so much the better.  If I can just point them to the INSTALL
> > file, and be done with it, great.
> 
> The installation can be quick and painless.  I'll explain more tomorrow.

Ok it's tomorrow.  This will apply mainly to the actual installation 
process, but will also apply to the instructions for the book.

1) Add to configure: --with-postgres-user=  and  --with-postgres-group=
   with the default being postgres:postgres.

2) When generating the makefiles, use the -g and -o parameters to install.
   These, of course, will come from configure.

At this point, anyone can build PostgreSQL, but only root and the 
postgres user can install it.  Either way, the ownership will be correct
since we added the -g and -o to install during the configuration process.

Currently, the only difference between the way I do it and the above is
that I override INSTALL in Makefile.custom (which I believe Tom Lockhart
tipped me off to).  I build as myself and use sudo to install (as many
other admins also do).

3) gmake

4) gmake install (or sudo gmake install)

Alternately, steps 3 and 4 can be done this way:

3a) gmake > gmake.out 2>&1 &
    tail -f gmake.out

4a) gmake install >> gmake.out 2>&1 &
    tail -f gmake.out

But I don't usually bother.

The final step may not be necessary if steps 1 and 2 are done right.  I
do it just to make sure the directories have the right permissions:

5) cd /usr/local   (or whatever the directory is just below postgres')
   chown -R postgres:postgres pgsql  (substitute as necessary).

Then do whatever initdb's and createdb's etc as necessary/desired.

Even doing step 5 (and doing the custom makefile), on a 450 P-III running
FreeBSD 3.2-RELEASE, I have the entire system installed and running in 
about 20-30 minutes.


So how's that?  There may be some minor differences between systems, 
I seem to recall HP having some with install(1), but I believe there
was a workaround script that fixed it for most things called bsd.install
or something like that.  I'll verify HPs install on systems from 8-10.2
later this mourning - think I have a couple SGIs I can check too.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH   email: vev@michvhf.com   flame-mail: /dev/null
  # include <std/disclaimers.h>       Have you seen http://www.pop4.net?
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Wed, 13 Oct 1999 05:53:12 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
[Charset KOI8-R unsupported, filtering to ASCII...]
> 
> On 13-Oct-99 Vince Vielhaber wrote:
> > 
> > On 13-Oct-99 Bruce Momjian wrote:
> 
> 2BOOK Authors:
> Please, try to keep rights for translating this book into another
> languages by you self, not by publisher.
> 
> 
> I may ask some St.Pitersburg's publishing company 
> to make russian translation of this book, but some publishers
> like O'Reilly have too hard license policy 
> and too long reaction time. 

Actually, I want to make sure the book is accessible on the web.  I will
keep your translation idea in mind.  Good point.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 13 Oct 1999 07:09:58 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
[Charset KOI8-R unsupported, filtering to ASCII...]
> 
> On 13-Oct-99 Vince Vielhaber wrote:
> > 
> > On 13-Oct-99 Bruce Momjian wrote:
> 
> 2BOOK Authors:
> Please, try to keep rights for translating this book into another
> languages by you self, not by publisher.
> 
> 
> I may ask some St.Pitersburg's publishing company 
> to make russian translation of this book, but some publishers
> like O'Reilly have too hard license policy 
> and too long reaction time. 
> 

FYI, I just did bibliography, and got:

  (a) The Practical SQL Handbook, Bowman et al., Addison Wesley
  (b) Web Development with PHP and PostgreSQL, \ldots{}, Addison Wesley
  (c) A Guide to The SQL Standard, C.J. Date, Addison Wesley
  (d) An Introduction to Database Systems, C.J. Date, Addison Wesley
  (e) SQL For Smarties, Joe Celko, Morgan, Kaufmann

Looks like Addision Wesley is the winner.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 13 Oct 1999 07:11:34 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> And perhaps some discussion about Date/Time types also in the section 
> for programming languages (what you get from date field, how to be 
> sure that what you give to postgresql is understood correctly)
> 
> > The most important thing is to have alot of samples.
> > Practical samples are more useful than just syntax.
> 
> Agreed.
> 
> Maybe "Advanced SQL Commands" could also contain:
> 
> * Outer Joins

But we don't have them yet.

> * Casts (bot AS and ::, with some explanations on why and when)
> 
> Not sure if Views and Triggers should go under Uniqe features - they are 
> supported on most commercial SQL databases.

Yes, I know, but Triggers are specific to the database, and I guess I
have to move views to the other section.

> 
> "Interfacing to the POSTGRESQL Database" could also contain more options
> than PHP,
> at least as references - there are much more of both established (CGI,
> PyApache,mod_perl) 

That's probably too advanced for this book.  Not sure how much detail I
am going to give each interface anyway.  Probably just two examples of
each.

> and emerging (like the recent pgxml announcement) technologies for that
> purpos.
> 
> Server side programming should also mention PL/Tcl 
> 

Added.


> Under which section would requirements the restrictions (db size, field
> size/count, record size/count) 
> or lackof them :) go?

Under Administration, or Installation.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 13 Oct 1999 07:20:13 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Outline for PostgreSQL book
> 
> 
> > Bruce Momjian wrote:
> > 
> > 
> > New version attached.  No PDF version this time.  Do people like PDF?
> > 
> 
> There could be also a chapter on client side tools - pgAccess et.al.
> and also small (but working ;) examples for using ODBC from MS Access, 
> MS Query, Delphi, JBuilder, Visual Cafe - each about 2-5 pages to get 
> people started.

I wasn't sure where to put pgaccess. I will add it to interfaces.  The
others are probably better left to another book, Interfaces PostgreSQL
to PC's.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 13 Oct 1999 07:21:43 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
On Wed, 13 Oct 1999, Bruce Momjian wrote:

> > And perhaps some discussion about Date/Time types also in the section 
> > for programming languages (what you get from date field, how to be 
> > sure that what you give to postgresql is understood correctly)
> > 
> > > The most important thing is to have alot of samples.
> > > Practical samples are more useful than just syntax.
> > 
> > Agreed.
> > 
> > Maybe "Advanced SQL Commands" could also contain:
> > 
> > * Outer Joins
> 
> But we don't have them yet.

Yes, but we will (or should) by the time the book comes out.  

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH   email: vev@michvhf.com   flame-mail: /dev/null
  # include <std/disclaimers.h>       Have you seen http://www.pop4.net?
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Wed, 13 Oct 1999 07:35:21 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: [HACKERS] Outline for PostgreSQL book
> > 2BOOK Authors:
> > Please, try to keep rights for translating this book into another
> > languages by you self, not by publisher.
> > 
> > 
> > I may ask some St.Pitersburg's publishing company 
> > to make russian translation of this book, but some publishers
> > like O'Reilly have too hard license policy 
> > and too long reaction time. 
> 
> Actually, I want to make sure the book is accessible on the web.  I will
> keep your translation idea in mind.  Good point.

What about translating the regular docs first? Are you planning on using a
lot of the existing documentation for the book? Perhaps one could organize
the distributed documentation as a sort of abridged version of the
official book, similar to what is being done with Perl. Just an idea.

And what kind of timespan did you have in mind? 

I had the insane plan of providing a German documentation (at least the
User's Guide) in time for 7.0. (big market) On the other hand, I have lots
of insane plans ...

	-Peter


Wed, 13 Oct 1999 13:49:30 +0200 (MET DST)
Peter Eisentraut <e99re41@csd.uu.se>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> > Actually, I want to make sure the book is accessible on the web.  I will
> > keep your translation idea in mind.  Good point.
> 
> What about translating the regular docs first? Are you planning on using a
> lot of the existing documentation for the book? Perhaps one could organize
> the distributed documentation as a sort of abridged version of the
> official book, similar to what is being done with Perl. Just an idea.
> 
> And what kind of timespan did you have in mind? 
> 
> I had the insane plan of providing a German documentation (at least the
> User's Guide) in time for 7.0. (big market) On the other hand, I have lots
> of insane plans ...

Not sure on a timespan.  I think I could do most if it in one month if I
did nothing else.  My wife seems supportive of the idea.

Not sure how to merge current documentation into it.  I would like to
point them to URL locations as much as possible.  I thought if I give
them enough to get started, and to understand how the current docs fit
together, that would be good.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 13 Oct 1999 07:54:02 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
On Wed, 13 Oct 1999, Bruce Momjian wrote:

> Not sure how to merge current documentation into it.  I would like to
> point them to URL locations as much as possible.  I thought if I give
> them enough to get started, and to understand how the current docs fit
> together, that would be good.

Personally, I always think that computer books that point you to URLs to
get the complete information are less than desirable. The very point of
reading the book is that you don't have to get up to your computer all the
time. Books should be self-contained and add to the existing documentation
since otherwise I won't need it.

Also, think about the fact that the online documentation might change more
quickly than a book is published. In a magazine you can do that, but in a
book that's questionable. I have a few books that are only about two years
old and the information in them is still very valid, but the URLs with all
the examples and all don't work anymore. Who knows why, I don't have the
time to find out.

But the publishers are probably a lot smarter in that area than I am.

	-Peter



Wed, 13 Oct 1999 14:08:01 +0200 (MET DST)
Peter Eisentraut <e99re41@csd.uu.se>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> On Wed, 13 Oct 1999, Bruce Momjian wrote:
> 
> > Not sure how to merge current documentation into it.  I would like to
> > point them to URL locations as much as possible.  I thought if I give
> > them enough to get started, and to understand how the current docs fit
> > together, that would be good.
> 
> Personally, I always think that computer books that point you to URLs to
> get the complete information are less than desirable. The very point of
> reading the book is that you don't have to get up to your computer all the
> time. Books should be self-contained and add to the existing documentation
> since otherwise I won't need it.
> 
> Also, think about the fact that the online documentation might change more
> quickly than a book is published. In a magazine you can do that, but in a
> book that's questionable. I have a few books that are only about two years
> old and the information in them is still very valid, but the URLs with all
> the examples and all don't work anymore. Who knows why, I don't have the
> time to find out.
> 

I just don't want to produce a 500 page book to cover these topics.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 13 Oct 1999 08:51:12 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
>Here is my proposal for an outline for a PostgreSQL book.  Many of us
>have been asked by publishers about writing a book.  Here is what I
>think would be a good outline for the book.
>
>I am interested in whether this is a good outline for a PostgreSQL book,
>how our existing documentation matches this outline, where our existing
>documentation can be managed into a published book, etc.
>
>Any comments would be welcome.

FYI, here is the table of contents from my PostgreSQL book published
in Japan at the beginning of this year (I do not guarantee the
accuracy of translation to English, however).

Note that the second edition about to be released will contain many more 
topics including MVCC, transactions, views, rules, triggers...
---
Tatsuo Ishii

-----------------------------------------------------------
Chapter 1

Introduction to PostgreSQL

1.1 History of PostgreSQL
1.2 Advantages of PostgreSQL

Chapter 2

Installation

2.1 Before installation
2.2 Preparing for installation
2.3 Compiling and installation
2.4 Setting your environment
2.5 Initialization
2.6 Adding users and creating database
2.7 Using psql
2.8 Security

Chapter 3

Learning PostgreSQL

3.1 Processes and modules
3.2 Source tree
3.3 Data types
3.4 User defined functions
3.5 User defined operators
3.6 User defined types

Chapter 4

Make applications

4.1 Tcl/Tk
4.2 C
4.3 PHP
4.4 Perl
4.5 Java

Chapter 5

Tips for PostgreSQL

5.1 Backuping database
5.2 Benchmark tests
5.3 Performance tuning
5.4 Troubles shooting
5.5 New functionalities (while writing this book 6.4 was about to release)
5.6 Developers and future plans
5.7 Resources on the Internet

Appendix

Total pages: 309

Wed, 13 Oct 1999 23:37:56 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] Outline for PostgreSQL book 
On Wed, 13 Oct 1999, Peter Eisentraut wrote:

> I had the insane plan of providing a German documentation (at least the
> User's Guide) in time for 7.0. (big market) On the other hand, I have lots
> of insane plans ...

Hi Peter,

I translated the FAQ and Linux-FAQ to german recently. I consider to
translate the online doc, too. But it is so much for one person, so I
didn't started yet. What about a cooperation? 
(Warning: I really need some advices to set up my sgml system ;))

have fun!
Karsten



Wed, 13 Oct 1999 16:52:11 +0200 (CEST)
Karsten Schulz <schulz@Linux-Systemhaus.de>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
On Wed, 13 Oct 1999, Bruce Momjian wrote:

> > Maybe "Advanced SQL Commands" could also contain:
> > 
> > * Outer Joins
> 
> But we don't have them yet.

As someone else mentioned previously, due to time required to write,
publish and get this out on the shelf, shouldn't we be revolving this
around the upcoming v7 release, which I believe Thomas has Outer Joins
target'd for?

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Wed, 13 Oct 1999 12:31:09 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] Outline for PostgreSQL book
Hello,

I just joined this list and am getting into the middle of this thread,
but I did go back and read the archives.

Just a few comments.

To answer a previous question, I like PDFs.

I like the approach that Bruce Eckel has taken to writing his last
couple of books. It was somewhat of an opensource approach. He wrote the
book and kept it on his site in PDF format and released regular updates
based on either his additions to the book or corrections of errors and
such by readers. It helped him to develop a good rapport with his
audience. I do not believe the book being available on the web really
impacted the sales of the published book that much. I know I printed one
copy and purchased one copy.
http://www.BruceEckel.com/

One thing to consider when writing the book is keeping it on target
about PostgreSQL. I was browsing Amazon and SQL books and was reading
about the MySQL book by O'Reilly. It got some really bad reviews based
it being to general on databases and not specific about MySQL. Many said
great db book, poor MySQL book.
Read about it at, http://www.amazon.com/exec/obidos/ASIN/1565924347 .

Their are many books which cover the basics of databases and database
design and you list a few below. What is needed is a PostgreSQL book.
Their will be overlap, but the emphasis should be on database
development with PostgreSQL. That is what will differentiate it from
others.

Addison Wesley is an excellent publisher as is O'Reilly. I would
recommend trying the above approach when writing the book and find a
publisher friendly to the idea. Bruce's Thinking in Java was published
by Prentice Hall. I don't know about any of the other publishers.

Jimmie Houchin


Bruce Momjian wrote:
[snip]
> FYI, I just did bibliography, and got:
> 
>   (a) The Practical SQL Handbook, Bowman et al., Addison Wesley
>   (b) Web Development with PHP and PostgreSQL, \ldots{}, Addison Wesley
>   (c) A Guide to The SQL Standard, C.J. Date, Addison Wesley
>   (d) An Introduction to Database Systems, C.J. Date, Addison Wesley
>   (e) SQL For Smarties, Joe Celko, Morgan, Kaufmann
> 
> Looks like Addision Wesley is the winner.
> 
> --
>   Bruce Momjian                        |  http://www.op.net/~candle
>   maillist@candle.pha.pa.us            |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
> 
> ************

Wed, 13 Oct 1999 10:50:43 -0500
Jimmie Houchin <jhouchin@texoma.net>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
Bruce Momjian wrote:
> Actually, I want to make sure the book is accessible on the web.  I will
> keep your translation idea in mind.  Good point.

Talk to Philip Greenspun.  Morgan-Kaufman cut him a deal where his book,
"Philip and Alex's Guide to Web Publishing" is also available free on
the web (http://photo.net/wtr/thebook).  His e-mail is philg@mit.edu.

And you gotta see the quality of his book!  Amazon carries it. 
(although, I must say, some of his choices for pictures were a little
over the top....)

--
Lamar Owen
WGCR Internet Radio
Pisgah Forest, North Carolina
1 Peter 4:11

Wed, 13 Oct 1999 11:52:41 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> On Wed, 13 Oct 1999, Peter Eisentraut wrote:
> 
> > I had the insane plan of providing a German documentation (at least the
> > User's Guide) in time for 7.0. (big market) On the other hand, I have lots
> > of insane plans ...
> 
> Hi Peter,
> 
> I translated the FAQ and Linux-FAQ to german recently. I consider to
> translate the online doc, too. But it is so much for one person, so I
> didn't started yet. What about a cooperation? 
> (Warning: I really need some advices to set up my sgml system ;))

I got sgmltools 2.0.1 and installed it on BSDI with no problems.  Only
issue was that I needed install GNU m4.  See SGML docs for sgmltools
install notes.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 13 Oct 1999 12:43:45 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> On Wed, 13 Oct 1999, Bruce Momjian wrote:
> 
> > > Maybe "Advanced SQL Commands" could also contain:
> > > 
> > > * Outer Joins
> > 
> > But we don't have them yet.
> 
> As someone else mentioned previously, due to time required to write,
> publish and get this out on the shelf, shouldn't we be revolving this
> around the upcoming v7 release, which I believe Thomas has Outer Joins
> target'd for?

Yes, for 7.0, but no sense in writing stuff until it is completed.  I
don't even have WAL mentioned.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 13 Oct 1999 12:46:17 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Outline for PostgreSQL book
Hello,

I agree with your comments here. When I posted my message I had forgotten
about Philip's book. Excellent book and website, photos, many are great and
many are as you say "a little over the top".

Based on his book which I learned about from his articles on LinuxWorld I
have decided to deploy my website with AOLserver. I currently am planning
to use PostgreSQL for the database.

He does a great job of discussing how to develop excellent websites.

Jimmie Houchin


At 11:52 AM -0400 10/13/99, Lamar Owen wrote:
>Bruce Momjian wrote:
>> Actually, I want to make sure the book is accessible on the web.  I will
>> keep your translation idea in mind.  Good point.
>
>Talk to Philip Greenspun.  Morgan-Kaufman cut him a deal where his book,
>"Philip and Alex's Guide to Web Publishing" is also available free on
>the web (http://photo.net/wtr/thebook).  His e-mail is philg@mit.edu.
>
>And you gotta see the quality of his book!  Amazon carries it.
>(although, I must say, some of his choices for pictures were a little
>over the top....)
>
>--
>Lamar Owen
>WGCR Internet Radio
>Pisgah Forest, North Carolina
>1 Peter 4:11
>
>************


Wed, 13 Oct 1999 11:48:24 -0500
Jimmie Houchin <jhouchin@texoma.net>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> Hello,
> 
> I just joined this list and am getting into the middle of this thread,
> but I did go back and read the archives.
> 
> Just a few comments.
> 
> To answer a previous question, I like PDFs.
> 
> I like the approach that Bruce Eckel has taken to writing his last
> couple of books. It was somewhat of an opensource approach. He wrote the
> book and kept it on his site in PDF format and released regular updates
> based on either his additions to the book or corrections of errors and
> such by readers. It helped him to develop a good rapport with his
> audience. I do not believe the book being available on the web really
> impacted the sales of the published book that much. I know I printed one
> copy and purchased one copy.
> http://www.BruceEckel.com/

Yes, I would like to do that, and the publisher I was talking to today
seemed to think it was fine.  He said it helps sell books.  I would be
very unhappy if the book could not be put on the web.  In fact, he is
subscribed to the hackers list, so he may be reading this.


> One thing to consider when writing the book is keeping it on target
> about PostgreSQL. I was browsing Amazon and SQL books and was reading
> about the MySQL book by O'Reilly. It got some really bad reviews based
> it being to general on databases and not specific about MySQL. Many said
> great db book, poor MySQL book.
> Read about it at, http://www.amazon.com/exec/obidos/ASIN/1565924347 .

Good point.  I will keep it in mind.

> Their are many books which cover the basics of databases and database
> design and you list a few below. What is needed is a PostgreSQL book.
> Their will be overlap, but the emphasis should be on database
> development with PostgreSQL. That is what will differentiate it from
> others.


Good.  And I will show actual PostgreSQL examples in the book, with
PostgreSQL output from psql, etc.

> Addison Wesley is an excellent publisher as is O'Reilly. I would
> recommend trying the above approach when writing the book and find a
> publisher friendly to the idea. Bruce's Thinking in Java was published
> by Prentice Hall. I don't know about any of the other publishers.

Yes, I am a big Addison Wesley fan.  I realized this when I found most
of the books I like were from them.  I am not a big O'Reilly fan, though
I have a few of their books.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 13 Oct 1999 12:52:23 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> Bruce Momjian wrote:
> > Actually, I want to make sure the book is accessible on the web.  I will
> > keep your translation idea in mind.  Good point.
> 
> Talk to Philip Greenspun.  Morgan-Kaufman cut him a deal where his book,
> "Philip and Alex's Guide to Web Publishing" is also available free on
> the web (http://photo.net/wtr/thebook).  His e-mail is philg@mit.edu.
> 
> And you gotta see the quality of his book!  Amazon carries it. 
> (although, I must say, some of his choices for pictures were a little
> over the top....)

Again, having it on the web will be done.  No better way to have it
available to everyone, and it helps sell books, and the publisher thinks
that is fine.

Morgan-Kaufmann is good, but I don't have many of their books.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 13 Oct 1999 12:53:33 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> Bruce Momjian <maillist@candle.pha.pa.us> writes:
> 
> >                           PostgreSQL Book Proposal
> >                                       
> >                                Bruce Momjian
> 
> [...]
> 
> >    4.
> >           Advanced SQL Commands
> 
> [...]
> 
> 
> >    6.
> >           Interfacing to the POSTGRESQL Database
> >         (a)
> >                C Language API
> >         (b)
> >                Embedded C
> >         (c)
> >                C++
> >         (d)
> >                JAVA
> >         (e)
> >                ODBC
> >         (f)
> >                PERL
> >         (g)
> >                TCL/TK
> >         (h)
> >                PYTHON
> >         (i)
> >                Web access (PHP)
> >         (j)
> >                Server-side programming (PLPGSQL and SPI)
> 
> Isn't (j) logically part of chapter 4?  (Or 5, if it's PostgreSQL
> specific.)  Or am I completely confused?  (Where can I read about
> PLPGSQL and/or SPI, other than in the forthcoming book?)
> 
> If it came to a choice between having very short sections in chapter
> 6, and having two or three of them covered in more depth, I'd go for
> the latter.  

Not sure.  They are properly 'programming' to me, so I put them there. 
They address a similar programmatic need in the database.

> (Of course, you'll inevitably choose two or three which don't match
> what many readers will want (whichever two or three you choose), but
> even so, I think I'd get more out of a reasonably thorough coverage of
> a couple of languages that I won't use than superficial coverage of
> all of them which doesn't really reveal anything useful.)

I was going to do a newbie thing and show the advantages of each one. 
Not sure I want to go into great depth on any of them.  Just enough to
get people started, and using the documentation.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 13 Oct 1999 14:31:54 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Re: [DOCS] Outline for PostgreSQL book
Bruce Stephens wrote:
> 
> Bruce Momjian <maillist@candle.pha.pa.us> writes:
> 
> >                           PostgreSQL Book Proposal
> >
> >                                Bruce Momjian
> 
> [...]
> 
> >    4.
> >           Advanced SQL Commands
> 
> [...]
> 
> >    6.
> >           Interfacing to the POSTGRESQL Database
> >         (a)
> >                C Language API
> >         (b)
> >                Embedded C
> >         (c)
> >                C++
> >         (d)
> >                JAVA
> >         (e)
> >                ODBC
> >         (f)
> >                PERL
> >         (g)
> >                TCL/TK
> >         (h)
> >                PYTHON
> >         (i)
> >                Web access (PHP)
> >         (j)
> >                Server-side programming (PLPGSQL and SPI)
> 
> Isn't (j) logically part of chapter 4?  (Or 5, if it's PostgreSQL
> specific.)  Or am I completely confused?  (Where can I read about
> PLPGSQL and/or SPI, other than in the forthcoming book?)
> 
> If it came to a choice between having very short sections in chapter
> 6, and having two or three of them covered in more depth, I'd go for
> the latter.
> 
> (Of course, you'll inevitably choose two or three which don't match
> what many readers will want (whichever two or three you choose), but
> even so, I think I'd get more out of a reasonably thorough coverage of
> a couple of languages that I won't use than superficial coverage of
> all of them which doesn't really reveal anything useful.)
> 
> ************


I second this opinion.

Thu, 14 Oct 1999 08:24:28 +0700
Chairudin Sentosa Harjo <chai@prima.net.id>
Re: [HACKERS] Re: [DOCS] Outline for PostgreSQL book

Bruce,

Were you going to do this as a "group" project or
is it a "solo" venture?

If its a group project, I assume you are the "owner"
of the "uber-contract" until it sells to a book 
publisher.  To do this right, you would divide
this contract into small peices, or sub-contracts.
Then, as people contribute, their time is tracked
against these components.  Right now, for instance,
you have a book-outline project which people are 
working on; it could be decided that the book-outline
sub contract is worth ".5%" of the book ownership.
Then, as people track their time against these
sub-contracts, a summary (once a week) is presented
to you (the owner of the contract) for approval
of their hours.  If approved, then the contributer
owns a portion of the sub-contract, and thus a 
portion of the overall book.   How compensation is
done is up to you if you specify it before hand,
or it is up to the owners if you wait until the book
sells.  The compensation schedule may be something 
like this:
  
Up to | Compensation
------+-------------------
 0.1% | Listed as "helper"
 0.5% | Listed as "contributor", including a 
      | very small biography; perhaps including
      | company name for free advertising. 
 2.0% | Listed as "major contributor", including
      | a full biography and a picture.
10.0% | Listed as "co-author", entitiled to 
      | the same percentage of royalty stream.
30.0% | Listed as "editor", entitied to a 
      | percentage of any book signing bonus plus
      | the same percentage of royalty stream

Thus, the total royalty stream allocated will
be less than 100%, and will be divided among,
at most 9 people, which is managable.  The remainder
of the royalty stream can be donated to the
site maintenance or can be used to fund a bonus
pool for future projects / documentation maintance.

...

However, if it is a "solo" deal, then you you should 
make it pretty clear -- otherwise I see heart-ache and
bad feelings.

...

Why do it this way?  Beacuse PostgreSQL is a community
effort, and a solo book sucks.  Also, you want the
book done, so direct rewards for contribution work
does wonders!  To use parts of the documentation, 
Thomas Lockheart would have to be given a percentage 
in the book, etc.  Anyway, if the system is clear
you will get far more help than otherwise...

I'll be glad to maintain the "contract hierarchy"
(per your direction) and to accept timesheets
from people, preparing a weelky summary for you
to approve.  Then, I will maintain a page with 
progress on each sub-contract (or sub-sub-contract)
so that we can go about this in a very organized way.  

I'm in it to see a kick ass book and to see
a collaborative effort win.

What do you think?  Thomas? 

Clark



Thu, 14 Oct 1999 10:04:32 -0400 (EDT)
"Clark C. Evans" <clark.evans@manhattanproject.com>
Business Plan for PostgreSQL book?
Last night Marc and i picked up 

Object Relation DBMS
Michael Stonebraker & Paul Brown
Pub: Morgan Kaufman
ISBN  1 55860 452 9

It was reccommented by a potential client, i'll put up 
a review when i finish it.

Jeff

On Wed, 13 Oct 1999, Bruce Momjian wrote:

> [Charset KOI8-R unsupported, filtering to ASCII...]
> > 
> > On 13-Oct-99 Vince Vielhaber wrote:
> > > 
> > > On 13-Oct-99 Bruce Momjian wrote:
> > 
> > 2BOOK Authors:
> > Please, try to keep rights for translating this book into another
> > languages by you self, not by publisher.
> > 
> > 
> > I may ask some St.Pitersburg's publishing company 
> > to make russian translation of this book, but some publishers
> > like O'Reilly have too hard license policy 
> > and too long reaction time. 
> > 
> 
> FYI, I just did bibliography, and got:
> 
>   (a) The Practical SQL Handbook, Bowman et al., Addison Wesley
>   (b) Web Development with PHP and PostgreSQL, \ldots{}, Addison Wesley
>   (c) A Guide to The SQL Standard, C.J. Date, Addison Wesley
>   (d) An Introduction to Database Systems, C.J. Date, Addison Wesley
>   (e) SQL For Smarties, Joe Celko, Morgan, Kaufmann
> 
> Looks like Addision Wesley is the winner.
> 
> 
> -- 
>   Bruce Momjian                        |  http://www.op.net/~candle
>   maillist@candle.pha.pa.us            |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
> 
> ************
> 

Jeff MacDonald
jeff@hub.org

===================================================================
 So long as the Universe had a beginning, we can suppose it had a 
creator, but if the Universe is completly self contained , having 
no boundry or edge, it would neither be created nor destroyed
 It would simply be.
===================================================================



Thu, 14 Oct 1999 11:07:52 -0300 (ADT)
Jeff MacDonald <jeff@hub.org>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
At 7:04 AM -0700 10/14/99, Clark C. Evans wrote:
>Bruce,
>
>Were you going to do this as a "group" project or
>is it a "solo" venture?
>
>If its a group project, I assume you are the "owner"
>of the "uber-contract" until it sells to a book
>publisher.  To do this right, you would divide
>this contract into small peices, or sub-contracts.
>Then, as people contribute, their time is tracked
>against these components.  Right now, for instance,
>you have a book-outline project which people are
>working on; it could be decided that the book-outline
>sub contract is worth ".5%" of the book ownership.
>Then, as people track their time against these
>sub-contracts, a summary (once a week) is presented
>to you (the owner of the contract) for approval
>of their hours.  If approved, then the contributer

The way this is normally done is to do a page or word count of the finished
product.  Each author owns a "chapter".  The editor negotiates with the
publisher and takes a cut off the top.  Counting time is a bad idea IMHO.
Too much uncertainty and inequity.

Of course in this case there is a lot of pre-existing text which makes
counting pages hard.  Is there any way we can make this whole thing owned
by postgresql.org and just use the proceeds for the project?  I haven't
been tracking the legal status.

Signature failed Preliminary Design Review.
Feasibility of a new signature is currently being evaluated.
h.b.hotz@jpl.nasa.gov, or hbhotz@oxy.edu

Thu, 14 Oct 1999 09:07:12 -0700
"Henry B. Hotz" <hotz@jpl.nasa.gov>
Re: [DOCS] Business Plan for PostgreSQL book?

Thanks Bruce, just thought that I'd put in
the suggestion.  I'm sure with your experience
it will be a fantastic book!

;) Clark

On Thu, 14 Oct 1999, Bruce Momjian wrote:
>
> I am taking this on as a solo project.
> 


Thu, 14 Oct 1999 13:51:16 -0400 (EDT)
"Clark C. Evans" <clark.evans@manhattanproject.com>
Re: [DOCS] Business Plan for PostgreSQL book?
> 
> Bruce,
> 
> Were you going to do this as a "group" project or
> is it a "solo" venture?

I am taking this on as a solo project.

> However, if it is a "solo" deal, then you you should 
> make it pretty clear -- otherwise I see heart-ache and
> bad feelings.

I have hesitated to talk about this in any more detail until I have
discussed this via telephone with Marc, Thomas, and Tom Lane.  I feel I
need to get clearance from them on this.  Vadim, unfortunately, is too
far away to telephone.


> Why do it this way?  Beacuse PostgreSQL is a community
> effort, and a solo book sucks.  Also, you want the
> book done, so direct rewards for contribution work
> does wonders!  To use parts of the documentation, 
> Thomas Lockheart would have to be given a percentage 
> in the book, etc.  Anyway, if the system is clear
> you will get far more help than otherwise...

Well, I don't think a "group-written" book is going to read very well. 
Everyone has a different style, and a mis-mash of writing styles in a
book will not work.  It also will take too long to produce a book in
that way.

The book will be available via the Web and in PDF format even before it
is completed.  (I am writing it using LyX/LaTeX).  I have written the
first two chapters, and will be putting them out for everyone to read
and use very soon.  This book project will clearly be a win for all
PostgreSQL users, whether they buy the book or not.

That doesn't mean I will not be including significant amount of our
existing documentation.  For example, I would probably include the
'manual' pages at the end of the book, like many computer books.

As far a money, let me mention something.  While making $0 with
PostgreSQL (I don't use it in my work, or even at home to store any
data.), I have always offered to put money into the project because I
think it is only fair that the costs be born fairly by the people
involved.  I have sent money to support our server, I have offered to
send more in the past, and have offered to host the PostgreSQL Award
around-the-world tour by including checks to pay for every leg of the
trip.  I have done other monetary gifts for PostgreSQL.

So, if there need for some money for PostgreSQL, let me know.  With or
without the book, I am always interested in helping.

Once I talk to everyone, I will be saying more.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 14 Oct 1999 14:05:55 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Business Plan for PostgreSQL book?

Well Henery, the collaborative book idea is dead.
But, here are my comments anyway.   BTW, I didn't
get your e-mail... is there a "docs" list and not
a "pgsql-docs" list?  If so, where are the archives?

On Thu, 14 Oct 1999, Henery B. Hotz wrote:
> >Then, as people contribute, their time is tracked
> >against these components.  Right now, for instance,
> >you have a book-outline project which people are
> >working on; it could be decided that the book-outline
> >sub contract is worth ".5%" of the book ownership.
> >Then, as people track their time against these
> >sub-contracts, a summary (once a week) is presented
> >to you (the owner of the contract) for approval
> >of their hours. 
>
> The way this is normally done is to do a page or word count 
> of the finished product.  Each author owns a "chapter".  The 
> editor negotiates with the publisher and takes a cut off 
> the top.  Counting time is a bad idea IMHO.
> Too much uncertainty and inequity.

I don't believe this is the case at all.  For two reasons:

1. When someone decides they want to contribute, their hours
   can be multipled by a factor commensurate with their 
   relevant expeience.  This multiplier can be wrong at first,
   but as the project moves along, the owner of the contract will
   have insentive to "promote" the multiplier for those who have
   really demonstrated that they are valid contributors.
   This will work out organically... besides most people will 
   be very honest and submit a multiplier commensurate with their
   experience.  The (nonexistent?) few that don't will be quickly
   identified... it really isn't all that hard.

2. If someone logs a ton of hours and tries to get them approved,
   the contract owner would have a good laugh and simply deny the
   hours.  I doubt this would happen in reality; in fact you might
   have the opposite problem -- people not turning in an adequate
   reflection of the time they have spent.

As for counting words or basing it on chapters -- this really limits 
the ability of the vast number of contributers to feel like they 
are a part of the process.  A tiny, but juicy kernel of wisdom is 
worth a ton.  

Pity nobody want's to do it.  Oh well.  We all have such
little faith in other people.  Damn shame.

Clark





Thu, 14 Oct 1999 14:12:33 -0400 (EDT)
"Clark C. Evans" <clark.evans@manhattanproject.com>
Re: [DOCS] Business Plan for PostgreSQL book?
> >If its a group project, I assume you are the "owner"
> >of the "uber-contract" until it sells to a book
> >publisher.  To do this right, you would divide
> >this contract into small peices, or sub-contracts.
> >Then, as people contribute, their time is tracked
> >against these components.  Right now, for instance,
> >you have a book-outline project which people are
> >working on; it could be decided that the book-outline
> >sub contract is worth ".5%" of the book ownership.
> >Then, as people track their time against these
> >sub-contracts, a summary (once a week) is presented
> >to you (the owner of the contract) for approval
> >of their hours.  If approved, then the contributer
> 
> The way this is normally done is to do a page or word count of the finished
> product.  Each author owns a "chapter".  The editor negotiates with the
> publisher and takes a cut off the top.  Counting time is a bad idea IMHO.
> Too much uncertainty and inequity.
> 
> Of course in this case there is a lot of pre-existing text which makes
> counting pages hard.  Is there any way we can make this whole thing owned
> by postgresql.org and just use the proceeds for the project?  I haven't
> been tracking the legal status.

The first two chapters are Intro/History, like the one I wrote for the
BSD magazine, and chapter 2 is using psql, which is all new.

We really don't have much hand-holding stuff.  Chapters 1-4 are going to
be all new.  The later chapters may use some existing stuff.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 14 Oct 1999 14:14:02 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Business Plan for PostgreSQL book?
 
Bruce,

On the slim chance that you may change your mind,
I've decided to address some of your points.

On Thu, 14 Oct 1999, Bruce Momjian wrote:
> 
> Well, I don't think a "group-written" book is going to read very well. 

You already started one.  It hardly looks like a "solo" 
project to me.  There is _nothing_ saying that it has to be
design-by-committee.

In fact, I assumed that you would run the book as a 
"dictatorship" where you keep complete artistic
control over every element... including the voice.

There is a ton other people can do... think: 
 
    "What can I delegate".   

It will bring you amazing power -- you might end up owning only
40% of the book.... however, the book will be 1000% better
and sell thousands more copies.

> Everyone has a different style, and a mis-mash of writing 
> styles in a book will not work.

Compare to:  Everyone has a different style, a mis-match of 
             programming styles in a computer program will not work.

Answer:      Modulize, design your book so that this will 
             be a non-issue.   1/2 of the book will be dreaming
             up and presenting 'cool' examples.  The little bit
             of text that is in a personal voice can be re-written.

> It also will take too long to produce a book in that way.

Compare to:  It will take too long to make a computer program that way.
 
Answer:      Once the outline is made; and the project is broken
             down into modules much of it can be done in parallel.
             Debugging can also be done in parallel (as Eric Raymond
             so clearly writes).  So what if a complete re-write 
             is needed at the end:  Plan to throw one away.

> The book will be available via the Web and in PDF format even before it
> is completed.  (I am writing it using LyX/LaTeX).  I have written the
> first two chapters, and will be putting them out for everyone to read

comment, debug, suggest improvements on, revise, help with,

> and use very soon.

Yes, I know.   You want to leverage the collaborative PostgreSQL 
community process when every possible.  Amazing how well it works!

> This book project will clearly be a win for all
> PostgreSQL users, whether they buy the book or not.

This much is true, but it's not the issue.

> That doesn't mean I will not be including significant amount of our
> existing documentation.  For example, I would probably include the
> 'manual' pages at the end of the book, like many computer books.

They are seveal hundred pages and will be a great
resource when writing the book -- they must have taken
hundreds of hours to generate.  I doubt that they
would be all that useful verbadim at the end of the book.
  
> As far a money, let me mention something.  While making $0 with
> PostgreSQL (I don't use it in my work, or even at home to store any
> data.), I have always offered to put money into the project because I
> think it is only fair that the costs be born fairly by the people
> involved.  I have sent money to support our server, I have offered to
> send more in the past, and have offered to host the PostgreSQL Award
> around-the-world tour by including checks to pay for every leg of the
> trip.  I have done other monetary gifts for PostgreSQL.

Yes, I know.  Most committed free software developers are in a similar
boat and I wish there was a more equitable way of developing software
like PostgreSQL.

> So, if there need for some money for PostgreSQL, let me know.  With or
> without the book, I am always interested in helping.

Bruce, I'm not questioning your integrety; if anything it should be
the other way around as I am not a significant contributor and 
as an 'familar outsider' really don't have the right to question
your actions.  Infact, I admire you a TON and that's why I'm 
spending _my_ time authoring this e-mail.

...

   Please understand.   I'm suggesting an alternative way of doing
   things; that maybye, just mabye could turn out to be bigger and 
   more useful than expected.  Consider this as a "small project"  
   to see if the members of PostgreSQL community can deliver as
   a cohesive unit; not as an individual.
   
   If an accountable process like this were to work for a book --
   then I would bet solid money that it would work for
   application software development for profit.  And this, 
   being able to generate a profit should be the goal that we
   as a community should be striving for.

...

   Imagine the book titled:
   "PostgreSQL: The Definitive Guide, by Bruce Momjian"
   
   Now imagine you going into Crysler corporation trying to 
   bid on a production control system.  Do you think you
   will get it with the Oracle representative right next to you?

   Imagine instead the book titled:

   "PostgreSQL:  The Definitive Guide
   A collaborative work by the PostgreSQL Community,   
   Edited by Bruce Momjian"

   Put yourself back in the board room at Crysler with
   Oracle sales person next to you.  Do you think you
   are in a better position?  I think so.  The first book
   says you are a lone wolf.  The second one shows
   you are a leader.  It also demonstrates all too clearly
   that you an muster the entire PostgreSQL community 
   behind you, ready to deliver on your promises.

   That is way powerful.  Far more powerful than the 
   lone wolf approach chosen by Larry Wall.
 
...

So, you mentioned the $ word.   Is this about $?  Yes.
However, it is not about the immediate money nor about
your right to profit from PostgreSQL.  It is about a 
key juncture for the PostgreSQL community; we can either
fragment off as individuals... going Solo.   Or,
we can develop a business model that lets us move
together as a community.   Oracle isn't scared about 
the first one.  Its petrified about the second.

Best Wishes,

Clark


Thu, 14 Oct 1999 15:01:55 -0400 (EDT)
"Clark C. Evans" <clark.evans@manhattanproject.com>
Re: [DOCS] Business Plan for PostgreSQL book?
> 2. If someone logs a ton of hours and tries to get them approved,
>    the contract owner would have a good laugh and simply deny the
>    hours.  I doubt this would happen in reality; in fact you might
>    have the opposite problem -- people not turning in an adequate
>    reflection of the time they have spent.
> 
> As for counting words or basing it on chapters -- this really limits 
> the ability of the vast number of contributers to feel like they 
> are a part of the process.  A tiny, but juicy kernel of wisdom is 
> worth a ton.  
> 
> Pity nobody want's to do it.  Oh well.  We all have such
> little faith in other people.  Damn shame.

I guess I figured that if we were going to do a collaborative book, we
would have done that with our documentation already.  But in fact, it
usually takes one person and lots of time.  In our current Doc's case,
it is Thomas.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 14 Oct 1999 15:34:39 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Business Plan for PostgreSQL book?
> 
> Thanks Bruce, just thought that I'd put in
> the suggestion.  I'm sure with your experience
> it will be a fantastic book!
> 

It was a good suggestion.  I just don't think writing scales to many
people very well.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 14 Oct 1999 15:35:21 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Business Plan for PostgreSQL book?


On Thu, 14 Oct 1999, Bruce Momjian wrote:
> I guess I figured that if we were going to do a collaborative book, we
> would have done that with our documentation already. 

Seveal Reasons:

(a) we didn't have the goal to write a book for profit, (b) we didn't 
have a business model, (c) we didn't have a leader.

Seems we have all three now:

(a) Lots of people seem to have latched on to the goal, and it is 
apparant that there is significant demand. (b) The business model 
is there, it may not be perfect, but we can nail that down once we 
have experience.   Everyone I've met in PostgreSQL is resonable; 
the inital agreement will be loose, as the book progresses the 
agreement can be tightened down as needed.  (c) We have a *sweet* 
leader, Bruce Momjian, he has proven to be on top of things -- a 
man with not only great vision, but the drive to make things tick.

Are you ready to delegate?

Clark




Thu, 14 Oct 1999 15:47:01 -0400 (EDT)
"Clark C. Evans" <clark.evans@manhattanproject.com>
Re: [DOCS] Business Plan for PostgreSQL book?

My apologies.  

On Thu, 14 Oct 1999, The Hermit Hacker wrote:
> IMHO, if you feel that you need to continue along this vein, please take
> it offlist and with Bruce privately...if someone else wishes to put the
> time, effort and "risk of maritial status" up on the block and work on a
> collaborative effort, so be it...but I personally am getting a major
> distaste in my mouth from watching you trying to pressure Bruce into a
> direction that he has already stated no  desire to go.  If ppl don't like
> that, fine...don't buy the book when its done.  Personally, I know where
> my book buying dollars are going when the time comes...



Thu, 14 Oct 1999 16:23:24 -0400 (EDT)
"Clark C. Evans" <clark.evans@manhattanproject.com>
Re: [DOCS] Business Plan for PostgreSQL book?
On Thu, 14 Oct 1999, Clark C. Evans wrote:

> 
> 
> On Thu, 14 Oct 1999, Bruce Momjian wrote:
> > I guess I figured that if we were going to do a collaborative book, we
> > would have done that with our documentation already. 
> 
> Seveal Reasons:
> 
> (a) we didn't have the goal to write a book for profit, (b) we didn't 
> have a business model, (c) we didn't have a leader.
> 
> Seems we have all three now:
> 
> (a) Lots of people seem to have latched on to the goal, and it is 
> apparant that there is significant demand. (b) The business model 
> is there, it may not be perfect, but we can nail that down once we 
> have experience.   Everyone I've met in PostgreSQL is resonable; 
> the inital agreement will be loose, as the book progresses the 
> agreement can be tightened down as needed.  (c) We have a *sweet* 
> leader, Bruce Momjian, he has proven to be on top of things -- a 
> man with not only great vision, but the drive to make things tick.
> 
> Are you ready to delegate?

Delegate what?  Bruce is doing this as a solo project, what would he need
to delegate? *confused look*  Now, the way I see it, he *could* have gone
off and written the book withuot letting any of us know about it, and
wihtout asking any of us for input as to how we'd like to see it look...he
didn't.  

IMHO, if you feel that you need to continue along this vein, please take
it offlist and with Bruce privately...if someone else wishes to put the
time, effort and "risk of maritial status" up on the block and work on a
collaborative effort, so be it...but I personally am getting a major
distaste in my mouth from watching you trying to pressure Bruce into a
direction that he has already stated no  desire to go.  If ppl don't like
that, fine...don't buy the book when its done.  Personally, I know where
my book buying dollars are going when the time comes...


Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Thu, 14 Oct 1999 17:56:07 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: [DOCS] Business Plan for PostgreSQL book?
> 
> My apologies.  
> 
> On Thu, 14 Oct 1999, The Hermit Hacker wrote:
> > IMHO, if you feel that you need to continue along this vein, please take
> > it offlist and with Bruce privately...if someone else wishes to put the
> > time, effort and "risk of maritial status" up on the block and work on a
> > collaborative effort, so be it...but I personally am getting a major
> > distaste in my mouth from watching you trying to pressure Bruce into a
> > direction that he has already stated no  desire to go.  If ppl don't like
> > that, fine...don't buy the book when its done.  Personally, I know where
> > my book buying dollars are going when the time comes...

Thomas mentioned that I need to request more than 20 free copies from
the publisher.  I will either get more for free, or pay for them myself
and have them delivered to the main devlopers.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 14 Oct 1999 17:42:35 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Business Plan for PostgreSQL book?
For external reasons I didn't get as much done as I wanted. Originally, I
planned for 4 weeks and I think I can keep that.

The source is posted at http://www.pathwaynet.com/~peter/psql3.tar.gz

CHANGELOG
* Accommodated object descriptions in \d* commands.
* Re-wrote \dd
* Re-wrote \d "table"
* \pset command as generic way for setting printing options (e.g.,
  \pset format html, \pset null "(null)")
* Proliferated use of const char * and enums
* Rewrote \di, \dt, \ds, \dS. Say hello to \dv for views, which are
  now recognized correctly. You can also call, e.g., \dtvi for a list
  of indices, tables, and views. The possibilities are endless ...
  (where "endless" = 325)

Also I wrote new printing routines which are better abstracted so that one
could easily throw in new formats. Here are some examples. Let me know if
you hate them.

(Sorry that this is a little long.)

peter@localhost:5432 play=> \pset format aligned
peter@localhost:5432 play=> \pset border 0
peter@localhost:5432 play=> select * from foo;                 
 first      second   
-------- ------------
       2 
       0 -----
       0 hi
      34 
      -1 -2
         yo &&& <> ho
99999999 this is text
(7 rows)
peter@localhost:5432 play=> \pset border 1
peter@localhost:5432 play=> select * from foo;
  first   |    second    
----------+--------------
        2 | 
        0 | -----
        0 | hi
       34 | 
       -1 | -2
          | yo &&& <> ho
 99999999 | this is text
(7 rows)
peter@localhost:5432 play=> \pset border 2
peter@localhost:5432 play=> select * from foo;
+----------+--------------+
|  first   |    second    |
+----------+--------------+
|        2 |              |
|        0 | -----        |
|        0 | hi           |
|       34 |              |
|       -1 | -2           |
|          | yo &&& <> ho |
| 99999999 | this is text |
+----------+--------------+
(7 rows)
peter@localhost:5432 play=> \pset format unaligned
peter@localhost:5432 play=> \pset fieldsep ",,"
peter@localhost:5432 play=> select * from foo;
first,,second
2,,
0,,-----
0,,hi
34,,
-1,,-2
,,yo &&& <> ho
99999999,,this is text
(7 rows)
peter@localhost:5432 play=> \t
Turned on only tuples
peter@localhost:5432 play=> \x
Turned on expanded table representation
peter@localhost:5432 play=> select * from foo;

first,,2
second,,

first,,0
second,,-----

first,,0
second,,hi

first,,34
second,,

first,,-1
second,,-2

first,,
second,,yo &&& <> ho

first,,99999999
second,,this is text
peter@localhost:5432 play=> \pset border 1
peter@localhost:5432 play=> \pset format html
peter@localhost:5432 play=> \pset expanded
Turned off expanded display
peter@localhost:5432 play=> select * from foo;
<table border=1>
  <tr valign=top>
    <td align=right>2</td>
    <td align=left>&nbsp;</td>
  </tr>
  <tr valign=top>
    <td align=right>0</td>
    <td align=left>-----</td>
  </tr>
  <tr valign=top>
    <td align=right>0</td>
    <td align=left>hi</td>
  </tr>
  <tr valign=top>
    <td align=right>34</td>
    <td align=left>&nbsp;</td>
  </tr>
  <tr valign=top>
    <td align=right>-1</td>
    <td align=left>-2</td>
  </tr>
  <tr valign=top>
    <td align=right>&nbsp;</td>
    <td align=left>yo &amp;&amp;&amp; &lt;&gt; ho</td>
  </tr>
  <tr valign=top>
    <td align=right>99999999</td>
    <td align=left>this is text</td>
  </tr>
</table>
peter@localhost:5432 play=> \t
Turned off only tuples
peter@localhost:5432 play=> \pset border 2
peter@localhost:5432 play=> \pset null "(null)"
peter@localhost:5432 play=> \pset format latex
peter@localhost:5432 play=> select * from foo;
\begin{tabular}{|r|l|}
\hline
first & second \\
\hline
2 & (null) \\
0 & ----- \\
0 & hi \\
34 & (null) \\
-1 & -2 \\
(null) & yo &&& <> ho \\       % Yes, this needs to be escaped.
99999999 & this is text \\
\hline
\end{tabular}

(7 rows) \\
peter@localhost:5432 play=> \pset null
Current null display is "(null)".
peter@localhost:5432 play=> \pset fieldsep
Current field separator is ",,".


(Oh, you actually read all the way down to here? Good.)

-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Tue, 19 Oct 1999 21:41:55 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
psql Week 3
> For external reasons I didn't get as much done as I wanted. Originally, I
> planned for 4 weeks and I think I can keep that.
> 
> The source is posted at http://www.pathwaynet.com/~peter/psql3.tar.gz
> 
> CHANGELOG
> * Accommodated object descriptions in \d* commands.
> * Re-wrote \dd
> * Re-wrote \d "table"
> * \pset command as generic way for setting printing options (e.g.,
>   \pset format html, \pset null "(null)")
> * Proliferated use of const char * and enums
> * Rewrote \di, \dt, \ds, \dS. Say hello to \dv for views, which are
>   now recognized correctly. You can also call, e.g., \dtvi for a list
>   of indices, tables, and views. The possibilities are endless ...
>   (where "endless" = 325)
> 

Very cool.  Nice new formats.  Man, I will have to add them to this
book.  I am going to have to have pre-7.0 psql backslash command
listings, and 7.0 backslash listings.  This improvement is long overdue.
psql has always been one of our nifty features.  It just got niftier.



-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 19 Oct 1999 17:49:58 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] psql Week 3
Bruce Momjian wrote:
  >> For external reasons I didn't get as much done as I wanted. Originally, I
  >> planned for 4 weeks and I think I can keep that.
  >> 
  >> The source is posted at http://www.pathwaynet.com/~peter/psql3.tar.gz
  >> 
  >> CHANGELOG
  >> * Accommodated object descriptions in \d* commands.
  >> * Re-wrote \dd
  >> * Re-wrote \d "table"
  >> * \pset command as generic way for setting printing options (e.g.,
  >>   \pset format html, \pset null "(null)")
  >> * Proliferated use of const char * and enums
  >> * Rewrote \di, \dt, \ds, \dS. Say hello to \dv for views, which are
  >>   now recognized correctly. You can also call, e.g., \dtvi for a list
  >>   of indices, tables, and views. The possibilities are endless ...
  >>   (where "endless" = 325)
  >> 
  >
  >Very cool.  Nice new formats.  Man, I will have to add them to this
  >book.  I am going to have to have pre-7.0 psql backslash command
  >listings, and 7.0 backslash listings.  This improvement is long overdue.
  >psql has always been one of our nifty features.  It just got niftier.
 
My previous database experience was with PICK, where the data dictionaries
include an output format and the command language allows for column
formatting 'on the fly'.  This is something that I have missed with
PostgreSQL, though I believe the SQL standard does not cover it.

What I would like would be the ability to attach a format to a column, so
that text could be truncated at 25 characters or floats lined up with a
specified number of decimal places.  (May be we can already and I've missed
it?)  I would particularly like to be able to do text wrapping within a 
column and totalling of numeric columns:

-----+---------------------+--------
id   | description         |     qty
-----+---------------------+--------
abc1 | text that rambles   |   35.43
     | on and on about     |
     | something or other  |
def2 | more useless text   |    2.00
hgf3 | and yet more text   |  355.10
     | to read             |
-----+---------------------+--------
     |                     |  392.53
-----+---------------------+--------
(3 rows)

Do you think there's any place for this in PostgreSQL?  Perhaps it needs a
separate front-end tool.




-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Commit thy way unto the LORD; trust also in him and 
      he shall bring it to pass."          Psalms 37:5 



Tue, 19 Oct 1999 23:52:04 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] psql Week 3 
hi...

> What I would like would be the ability to attach a format to a column, so
> that text could be truncated at 25 characters or floats lined up with a
> specified number of decimal places.  (May be we can already and I've missed
> it?)  I would particularly like to be able to do text wrapping within a 
> column and totalling of numeric columns:

i miss this from oracle as well...

> 
> Do you think there's any place for this in PostgreSQL?  Perhaps it needs a
> separate front-end tool.

psql would seem the proper place for this?

-- 
Aaron J. Seigo
Sys Admin

Tue, 19 Oct 1999 17:06:12 -0600
"Aaron J. Seigo" <aaron@gtv.ca>
Re: [HACKERS] psql Week 3
>   >Very cool.  Nice new formats.  Man, I will have to add them to this
>   >book.  I am going to have to have pre-7.0 psql backslash command
>   >listings, and 7.0 backslash listings.  This improvement is long overdue.
>   >psql has always been one of our nifty features.  It just got niftier.
>  
> My previous database experience was with PICK, where the data dictionaries
> include an output format and the command language allows for column
> formatting 'on the fly'.  This is something that I have missed with
> PostgreSQL, though I believe the SQL standard does not cover it.

I had that with Progress.  Yes, it was handy.

> 
> What I would like would be the ability to attach a format to a column, so
> that text could be truncated at 25 characters or floats lined up with a

char(25)?

> specified number of decimal places.  (May be we can already and I've missed

numeric(16,2)?

> it?)  I would particularly like to be able to do text wrapping within a 
> column and totalling of numeric columns:


> 
> -----+---------------------+--------
> id   | description         |     qty
> -----+---------------------+--------
> abc1 | text that rambles   |   35.43
>      | on and on about     |
>      | something or other  |
> def2 | more useless text   |    2.00
> hgf3 | and yet more text   |  355.10
>      | to read             |
> -----+---------------------+--------
>      |                     |  392.53
> -----+---------------------+--------
> (3 rows)
> 
> Do you think there's any place for this in PostgreSQL?  Perhaps it needs a
> separate front-end tool.

That's a job for pgaccess.  I thought it did stuff like that.  If you
want printing like that, we need a report-writer, which is an important
application.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 19 Oct 1999 19:10:04 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] psql Week 3
On Tue, 19 Oct 1999, Oliver Elphick wrote:

> My previous database experience was with PICK, where the data dictionaries
> include an output format and the command language allows for column
> formatting 'on the fly'.  This is something that I have missed with
> PostgreSQL, though I believe the SQL standard does not cover it.

Hmm. Well, the idea of keeping formatting data in the backend doesn't
sound too exciting to me. The data type already contains a fair amount of
formatting information in itself.

But psql is not supposed to be a report generator. It's also not a
typesetting engine. The idea was to sort of write a shell, but one that
doesn't use the OS kernel but a database server.

> 
> What I would like would be the ability to attach a format to a column, so
> that text could be truncated at 25 characters or floats lined up with a
> specified number of decimal places.  (May be we can already and I've missed

Okay, lining up floats is sort of on the wish list (as distinct from todo
list). I'll keep that in mind.

> it?)  I would particularly like to be able to do text wrapping within a 
> column and totalling of numeric columns:

Wrapping text also seemed like a nice idea to me, but psql is supposed to
be a query interpreter. Wrapping text is a whole different animal and
there are specialty programs out there for that.

And totalling numeric columns is the job of a report generator. All psql
does is send a query and output the results, and it wants to make that job
as fun as possible along the way. It doesn't know anything about what's in
the query results. That is probably an important line to keep.

> 
> -----+---------------------+--------
> id   | description         |     qty
> -----+---------------------+--------
> abc1 | text that rambles   |   35.43
>      | on and on about     |
>      | something or other  |
> def2 | more useless text   |    2.00
> hgf3 | and yet more text   |  355.10
>      | to read             |
> -----+---------------------+--------
>      |                     |  392.53
> -----+---------------------+--------
> (3 rows)
> 
> Do you think there's any place for this in PostgreSQL?  Perhaps it needs a
> separate front-end tool.

I hear pg_access has a report generator. If you want a text-based report
generator, then you're seemingly on your own.


	-Peter

-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Wed, 20 Oct 1999 12:16:11 +0200 (MET DST)
Peter Eisentraut <e99re41@csd.uu.se>
Re: [HACKERS] psql Week 3 
> Bruce Momjian wrote:
> > Actually, I want to make sure the book is accessible on the web.  I will
> > keep your translation idea in mind.  Good point.
> 
> Talk to Philip Greenspun.  Morgan-Kaufman cut him a deal where his book,
> "Philip and Alex's Guide to Web Publishing" is also available free on
> the web (http://photo.net/wtr/thebook).  His e-mail is philg@mit.edu.

I have this from Addison Wesley.  It is online now.

> And you gotta see the quality of his book!  Amazon carries it. 
> (although, I must say, some of his choices for pictures were a little
> over the top....)

Much thanks for sending me this suggestion.

I read it at:

	http://photo.net/wtr/dead-trees/story.html

First, it is very long, so let me give you a quick summary.  He did his
first book for MacMillan, which is like Sams, Waite, etc.  Basically your
fat book for dummies, and had a pretty terrible experience.  You have to
read it to appreciate it.  This reminds me of Thomas's comment that we
need a good, quality publisher for our books.  If the first book that
came out on PostgreSQL was "PostgreSQL for Dummies", Marc would have a
fit.  And I have seen him in fits -- it is not pretty.

He did his second book for Morgan Kaufmann, who is a quality publisher
like Addison Wesley.  Totally different experience, and a different
quality book.

This is required reading for anyone who is interested in the book
writing experience or is considering writing a book.  People, we need to
stay with the quality publishers, if possible.

I am sure there will be a "PostgreSQL Unleashed" book one day, with tons
of graphic arrows and very little content, but let's put it off as long
as we can.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 20 Oct 1999 23:49:25 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book

If I may add..

I would highly recommend that you all read Phil Greenspun's story about his
experience in publishing his book with MacMillan.  Read between the lines,
however.  Where ever reference is made to MacMillan you could substitue
almost <any publisher>. Greenspun is a great story teller so the material is
a good read. I'm sure he embellished a lot on his experiences.  He strikes
me as being a puckish kind of guy, somewhat rare amongst engineers. Read
some of his other material. A Day at the Zoo is a good one.

Paul

Paul W. Becker
Addison Wesley Longman
671 Andover Road
Valley Cottage, NY  10989

914-268-8003 (v)
914-268-3874 (f)

Assistant: Ross Venables 781-944-3700  x2501




-----Original Message-----
From: Bruce Momjian [mailto:maillist@candle.pha.pa.us]
Sent: Wednesday, October 20, 1999 11:49 PM
To: Lamar Owen
Cc: Dmitry Samersoff; Vince Vielhaber; Oliver Elphick;
PostgreSQL-documentation; "PostgreSQL-development"@candle.pha.pa.us;
pgsql-hackers@postgreSQL.org; Jan Wieck
Subject: Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book


> Bruce Momjian wrote:
> > Actually, I want to make sure the book is accessible on the web.  I will
> > keep your translation idea in mind.  Good point.
>
> Talk to Philip Greenspun.  Morgan-Kaufman cut him a deal where his book,
> "Philip and Alex's Guide to Web Publishing" is also available free on
> the web (http://photo.net/wtr/thebook).  His e-mail is philg@mit.edu.

I have this from Addison Wesley.  It is online now.

> And you gotta see the quality of his book!  Amazon carries it.
> (although, I must say, some of his choices for pictures were a little
> over the top....)

Much thanks for sending me this suggestion.

I read it at:

	http://photo.net/wtr/dead-trees/story.html

First, it is very long, so let me give you a quick summary.  He did his
first book for MacMillan, which is like Sams, Waite, etc.  Basically your
fat book for dummies, and had a pretty terrible experience.  You have to
read it to appreciate it.  This reminds me of Thomas's comment that we
need a good, quality publisher for our books.  If the first book that
came out on PostgreSQL was "PostgreSQL for Dummies", Marc would have a
fit.  And I have seen him in fits -- it is not pretty.

He did his second book for Morgan Kaufmann, who is a quality publisher
like Addison Wesley.  Totally different experience, and a different
quality book.

This is required reading for anyone who is interested in the book
writing experience or is considering writing a book.  People, we need to
stay with the quality publishers, if possible.

I am sure there will be a "PostgreSQL Unleashed" book one day, with tons
of graphic arrows and very little content, but let's put it off as long
as we can.

--
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026



Thu, 21 Oct 1999 10:06:16 -0400
"Paul Becker" <paul.becker@awl.com>
RE: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
Paul Becker wrote:
> a good read. I'm sure he embellished a lot on his experiences.  He strikes
> me as being a puckish kind of guy, somewhat rare amongst engineers. Read
> some of his other material. A Day at the Zoo is a good one.

And "Travels with Samantha" is a hoot.  His coding style is very similar
to his writing style -- lispish.  But, then again, he's an old-hand MIT
LISP hacker.  You should read his tcl one day -- I've laughed till I've
cried over some of his code.  (I don't know if that's more of a
statement about his coding style, or about my reading habits.....)

Philip is certainly an interesting writer.

Oh, BTW, Bruce, "You're Welcome."

Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Thu, 21 Oct 1999 11:09:37 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
> Paul Becker wrote:
> > a good read. I'm sure he embellished a lot on his experiences.  He strikes
> > me as being a puckish kind of guy, somewhat rare amongst engineers. Read
> > some of his other material. A Day at the Zoo is a good one.
> 
> And "Travels with Samantha" is a hoot.  His coding style is very similar
> to his writing style -- lispish.  But, then again, he's an old-hand MIT
> LISP hacker.  You should read his tcl one day -- I've laughed till I've
> cried over some of his code.  (I don't know if that's more of a
> statement about his coding style, or about my reading habits.....)
> 
> Philip is certainly an interesting writer.

Yes, he was interesting.  He clearly caused some of his own problems
with MacMillan, and I agreed with MacMillan on a number of issues.

What I found interesting is something I had suspected for quite some
time.  I found that I have some great computer books, but when I go to
the computer section of a book store, most books there are junk.

Then I ordered Knuth's "Art of Computer Programming" directly from
Addison Wesley, and I started receiving their quarterly book bulletins. 
I said, "Hey, I read that book, and that one, and that one..."  I
realized that most of my books are from a handful of book publishers,
Addison Wesley being the most popular in my bookshelf.

Basically, I realized that not all the publishers are the same.  Some
produce quality, and others take a much more marketing slant in book
production, usually producing poor quality books.

One of the reasons I am posting this to the list is so people
considering book writing ( and I know some publishers have been lurking
in the past months), be careful who you sign with.  It can affect your
whole outlook on the process, and in the end, it is your name that is on
the cover of the book, not the publisher.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 21 Oct 1999 12:55:56 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
I'm currently studying for a Masters at Brunel University, London. I was
looking for a thesis and since I had an interest in Opensource development
and databases I thought that I would like to work on PGsql.  As I also had
an interest in trying to understand CORBA and other related technologies it
made sense that I try and work on a project that ties PGsql and CORBA
together in some way.

Marc G. Fournier and Bruce Momjian were contacted with regards to this and
they seemed to think that there was room for me to work on this, although at
this stage I hadn't gone through the mailing list.

since then I've finally got PGsql up and working and I'm now embarking on
understand how to system actual works. I thought a good place to start with
my research would be to go through the mailing lists to try and see the
current status for development in this area. I did this for several hours
trying to get the current take on CORBA with regards to PGsql. It seems
there was a lot of initial talk back in Nov 1998 on the hackers list with
the first thoughts of some sort of CORBA implementation. The conversation
orientated over 2 main areas; which ORB and implementation methods, with
some peoples offering to work on this. It was then suggested that the
converstaion take place in the interfaces mailing list. I've since been over
some of this but find it very difficult to understand the current status
with regards to PGsql and CORBA. I've seen many references to people who
have developed
a project that allows them to work with PGsql via CORBA, but non of this
seems to be part of the main project or system. There is some reference to
Micheal Robinson working on an implementation but again I'm not sure how
this fits
in-http://www.postgresql.org/mhonarc/pgsql-interfaces/1998-11/msg00090.html

Is there room for me to work on this project in such a way that it is
adequate for my masters. If anyone is working on this, or has a good
knowledge of the current status of the CORBA implementation for PGsql can
you please let me know, so I can know whether to get started on this or not.
The reference thread for my initial point of contact with Marc G. Fournier
and Bruce Momjian and how they think I should attack the project is -
http://www.postgresql.org/mhonarc/pgsql-hackers/1999-09/msg00076.html

Regards

Mark Proctor
email : m.proctor@bigfoot.com






I'm currently studying for a Masters at 
Brunel University, London. I was looking 
for a thesis and since I had an interest 
in Opensource development and databases I thought that I would like to work 
on PGsql.  As I also had an interest 
in trying to understand CORBA and other 
related technologies it made sense that I 
try and work on a project that ties PGsql 
and CORBA together in some way.Marc G. Fournier and Bruce Momjian were 
contacted with regards to this and they 
seemed to think that there was room for me 
to work on this, although at this stage I 
hadn't gone through the mailing list.since then I've finally got PGsql 
up and working and I'm now embarking on 
understand how to system actual works. I 
thought a good place to start with my
research would be to go through the mailing lists to try and see the current status for development 
in this area. I did this for several hours 
trying to get the current take on CORBA 
with regards to PGsql. It seems there was 
a lot of initial talk back in Nov 1998 on
the hackers list with the first thoughts of some sort of CORBA implementation. The conversation 
orientated over 2 main areas; which ORB 
and implementation methods, with some 
peoples offering to work on this. It was 
then suggested that the converstaion take place
in the interfaces mailing list. I've since been over some of this but find it very difficult to 
understand the current status with regards 
to PGsql and CORBA. I've seen many 
references to people who have developeda project that allows them to work 
with PGsql via CORBA, but non of this 
seems to be part of the main project or system. There is some reference to Micheal 
Robinson working on an implementation but 
again I'm not sure how this fits in-http://www.postgresql.org/mhonarc/pgsql-interfaces/1998-11/msg00090.html
Is there 
room for me to work on this project in such a
way that it is adequate for my masters. If anyone is working on this, or has a good knowledge of 
the current status of the CORBA 
implementation for PGsql can you please 
let me know, so I can know whether to get 
started on this or not. The reference thread for
my initial point of contact with Marc G. Fournier and Bruce Momjian and how they think I should 
attack the project is - http://www.postgresql.org/mhonarc/pgsql-hackers/1999-09/msg00076.htmlRegardsMark 
Proctoremail : m.proctor@bigfoot.com
Tue, 9 Nov 1999 01:38:35 -0000
"Mark Proctor" <mail-list@molnir.com>
CORBA STATUS
> Is there room for me to work on this project in such a way that it is
> adequate for my masters. If anyone is working on this, or has a good
> knowledge of the current status of the CORBA implementation for PGsql can
> you please let me know, so I can know whether to get started on this or not.
> The reference thread for my initial point of contact with Marc G. Fournier
> and Bruce Momjian and how they think I should attack the project is -
> http://www.postgresql.org/mhonarc/pgsql-hackers/1999-09/msg00076.html

I know of no one working on it.  There were two ideas as I remember. 
One was to replace our existing client/server communication with corba,
and the second was to have a corba server that accepted corba requests
and sent them to a PostgreSQL server.  I prefer this second approach as
I think CORBA may be too much overhead for people who don't need it. 
Our current client/server communication is quite efficient.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 8 Nov 1999 21:09:51 -0500 (EST)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] CORBA STATUS
> > Is there room for me to work on this project in such a way that it is
> > adequate for my masters. If anyone is working on this, or has a good
> > knowledge of the current status of the CORBA implementation for PGsql can
> > you please let me know, so I can know whether to get started on this or not.
> > The reference thread for my initial point of contact with Marc G. Fournier
> > and Bruce Momjian and how they think I should attack the project is -
> > http://www.postgresql.org/mhonarc/pgsql-hackers/1999-09/msg00076.html
> I know of no one working on it.

Right. No one is working on it, or if they are they haven't told
anyone. It's all yours ;)

> There were two ideas as I remember.
> One was to replace our existing client/server communication with corba,
> and the second was to have a corba server that accepted corba requests
> and sent them to a PostgreSQL server.  I prefer this second approach as
> I think CORBA may be too much overhead for people who don't need it.
> Our current client/server communication is quite efficient.

Actually, our current protocol is about the best you can do assuming
that you don't have something as powerful as Corba to do it right.

In the time since Corba was first brought up wrt Postgres, I have been
involved with extensive Corba development for a family of systems at
work. It is a tremendously powerful standard, though just
re-implementing the existing interfaces using Corba would probably
defeat the power and flexibility Corba can give you.

Postgres currently avoids endian and other data representation issues
between client and server by converting all data to strings. Corba can
efficiently pass binary info back and forth, automatically handling
endian issues *if necessary*. This alone should make a Corba-based
interface using native internal representations of data types more
efficient both in speed and size than our current scheme.

Using Corba's DII, we might even cope with Postgres' type
extensibility features in a transparent manner.

One trick will be to choose a Corba ORB to use on the server side. It
should probably be a C implementation, though Corba more naturally
maps to an OO language such as C++. It will be a trick to find an ORB
which is supported on as many platforms as Postgres is. One of the
most portable ORBs is TAO, which we are using at work, but that is C++
and involves 1.3GB of disk space to fully build!! But the runtime
sizes are pretty reasonable, adding just a few megabytes of shared
libraries to a plain-vanilla client/server application.

imho it will be extremely difficult to find an ORB which could be
injected directly into the Postgres server. It would likely reduce the
number of platforms Postgres runs on, and would not be considered
acceptable.

For a first cut, you might consider Bruce's "plan B", which involves
writing a new client to the Postgres backend, which would be a Corba
server to other clients. That would allow you to start working with
Corba without hacking up the backend early on.

At the extreme end, fully Corba-ized Postgres server is an intriguing
thought, allowing backend to be broken up into distributable modules.

Have fun thinking about the possibilities...

                    - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Tue, 09 Nov 1999 07:55:37 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] CORBA STATUS
Then <lockhart@alumni.caltech.edu> spoke up and said:
> imho it will be extremely difficult to find an ORB which could be
> injected directly into the Postgres server. It would likely reduce the
> number of platforms Postgres runs on, and would not be considered
> acceptable.

On the other hand, this is where the power of configure comes into
play.  Assuming we have servers for multiple ORBs, configure can look
to see what's installed (TAO, Orbit, whatever) and then build only
that server.

-- 
=====================================================================
| JAVA must have been developed in the wilds of West Virginia.      |
| After all, why else would it support only single inheritance??    |
=====================================================================
| Finger geek@cmu.edu for my public key.                            |
=====================================================================
9 Nov 1999 08:15:30 -0500
Brian E Gallew <geek+@cmu.edu>
Re: [HACKERS] CORBA STATUS
On 9 Nov 1999, Brian E Gallew wrote:

> Then <lockhart@alumni.caltech.edu> spoke up and said:
> > imho it will be extremely difficult to find an ORB which could be
> > injected directly into the Postgres server. It would likely reduce the
> > number of platforms Postgres runs on, and would not be considered
> > acceptable.
> 
> On the other hand, this is where the power of configure comes into
> play.  Assuming we have servers for multiple ORBs, configure can look
> to see what's installed (TAO, Orbit, whatever) and then build only
> that server.

ACtually, I believe Thomas was referring to those platforms that we
currently support that have no ORBs available to them...being a "purely C"
server so far, how many of our currently supported platforms are we going
to cut off with this? 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Tue, 9 Nov 1999 10:12:56 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] CORBA STATUS
Then <scrappy@hub.org> spoke up and said:
> ACtually, I believe Thomas was referring to those platforms that we
> currently support that have no ORBs available to them...being a "purely C"
> server so far, how many of our currently supported platforms are we going
> to cut off with this? 

But that's just it: we're not cutting anybody off.  We may not have a
CORBA server available for that platform, but the standard server will
continue to work quite nicely.  I guess that I'm looking at this more
as contrib code than as mainstream, required functionality.  Besides,
once the CORBA server is built on a platform that can support it,
clients can run from (basically) anywhere.

-- 
=====================================================================
| JAVA must have been developed in the wilds of West Virginia.      |
| After all, why else would it support only single inheritance??    |
=====================================================================
| Finger geek@cmu.edu for my public key.                            |
=====================================================================
9 Nov 1999 09:44:27 -0500
Brian E Gallew <geek+@cmu.edu>
Re: [HACKERS] CORBA STATUS
> > > imho it will be extremely difficult to find an ORB which could be
> > > injected directly into the Postgres server. It would likely reduce the
> > > number of platforms Postgres runs on, and would not be considered
> > > acceptable.
> > On the other hand, this is where the power of configure comes into
> > play.  Assuming we have servers for multiple ORBs, configure can look
> > to see what's installed (TAO, Orbit, whatever) and then build only
> > that server.

In the long run, that would be neat. In the short run, the details of
each ORB vary considerably wrt, for example, the names and numbers of
#include files. So it would complicate the code to try bringing along
two ORBs at the beginning. We might expect the ORBs to converge a bit
over time, so this will be easier later.

> ACtually, I believe Thomas was referring to those platforms that we
> currently support that have no ORBs available to them...being a "purely C"
> server so far, how many of our currently supported platforms are we going
> to cut off with this?

Don't know, and it doesn't matter (yet). We shouldn't avoid the issue
without someone looking at it further just because we *might* lose
some platforms; better to push it farther as a demonstration at least
before deciding that it isn't a possibility.

Anyway, I know that at least one ORB, TAO, runs on many more types of
platforms than Postgres does (e.g. VxWorks, Lynx, Solaris, NT, ...),
though Postgres runs on more Unix-style platforms. But that particular
ORB is not a good candidate for us, for reasons I already mentioned
(C++, large build size, poor configure support).

                    - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Tue, 09 Nov 1999 15:06:53 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] CORBA STATUS

On 09-Nov-99 Thomas Lockhart wrote:
>> > > imho it will be extremely difficult to find an ORB which could be
>> > > injected directly into the Postgres server. It would likely reduce the
>> > > number of platforms Postgres runs on, and would not be considered
>> > > acceptable.
>> > On the other hand, this is where the power of configure comes into
>> > play.  Assuming we have servers for multiple ORBs, configure can look
>> > to see what's installed (TAO, Orbit, whatever) and then build only
>> > that server.
> 
> In the long run, that would be neat. In the short run, the details of
> each ORB vary considerably wrt, for example, the names and numbers of
>#include files. So it would complicate the code to try bringing along
> two ORBs at the beginning. We might expect the ORBs to converge a bit
> over time, so this will be easier later.
> 
>> ACtually, I believe Thomas was referring to those platforms that we
>> currently support that have no ORBs available to them...being a "purely C"
>> server so far, how many of our currently supported platforms are we going
>> to cut off with this?
> 
> Don't know, and it doesn't matter (yet). We shouldn't avoid the issue
> without someone looking at it further just because we *might* lose
> some platforms; better to push it farther as a demonstration at least
> before deciding that it isn't a possibility.
> 
> Anyway, I know that at least one ORB, TAO, runs on many more types of
> platforms than Postgres does (e.g. VxWorks, Lynx, Solaris, NT, ...),
> though Postgres runs on more Unix-style platforms. But that particular
> ORB is not a good candidate for us, for reasons I already mentioned
> (C++, large build size, poor configure support).

IMHO, There has no ideal ORB for all platforms.
we use ORBacus (http://www.ooc.com), 
because it's the only known for me ORB, working without threads 
so its really faster and more stable than another ones under FreeBSD,
but it's not free.

May be it is better make directory CORBA under interfaces subtree
and time-to-time put objects for differend ORB's inside, 
into separate directory.

Probably, It's better to make separate configure for 
some parts of postgres distributions to allow users to build/upgrade
parts of postgres i.e psql or perl interface 




---
Dmitry Samersoff, dms@wplus.net, ICQ:3161705
http://devnull.wplus.net
* There will come soft rains ...

Tue, 09 Nov 1999 20:56:19 +0400 (MSK)
Dmitry Samersoff <dms@wplus.net>
Re: [HACKERS] CORBA STATUS

I have done some research into slapping
a CORBA interface onto the PgSQL server.
I have looked at ORBit but my findings 
seems to apply to most CORBA implementations.

I found that there is a fundamental problem
concerning the difference in process models
in pgsql and the POA (Portable Object Adaptor)
in CORBA implementations.

AFAICS, POA assumes a threaded server while
PgSQL uses a traditional forking model.

I see 3 ways to resolve this:

1. Adapt the PgSQL server to use threads instead of forking.
   I am not sure we want to do this since it is a major
   undertaking and will not help the rock solid stability
   we expect from a professional RDBMS.

2. Write a Forking Object Adaptor (FOA).
   Would be a big job but it would benefit other similar
   projects. A FOA would have to be rewritten for every
   ORB. If you want to do this I suggest starting with
   ORBit since it is all C and thereby easier to slap
   onto PgSQL.

3. Extend postmaster to proxy CORBA requests.
   Postmaster could fork of postgres processes
   on incoming connections. Postmaster will keep 
   listening and proxying request and responses 
   to/from the postgres process through some kind
   of IPC. This will make the postmaster multithreaded 
   and the postgres processes still singlethreaded.

It is doubtful if the gains:
- standard-based protocols
- less code to maintain (?)
justifies the amount of work required.

An easier way out is to, as suggested by others,
implement the CORBA-service outside the server
itself. It will in some cases give some latency
related performance degrading compared to a 
in-server implementation.

One effort to do something like that
(and more) is gnome-db. Check out
http://www.gnome.org/gnome-db/

regards,
-----------------
Gï¿½ran Thyni
On quiet nights you can hear Windows NT reboot!

Tue, 09 Nov 1999 18:15:06 +0100
Goran Thyni <goran@kirra.net>
Re: [HACKERS] CORBA STATUS
> ACtually, I believe Thomas was referring to those platforms that we
> currently support that have no ORBs available to them...being a "purely C"
> server so far, how many of our currently supported platforms are we going
> to cut off with this?

OmniORB http://www.uk.research.att.com/omniORB/omniORBForm.html runs on
most
Unix-like platforms. It is free and it is pretty fast. I used it a about
18 months ago and it was already pretty good then.

Adriaan

Tue, 09 Nov 1999 19:15:17 +0200
Adriaan Joubert <a.joubert@albourne.com>
Re: [HACKERS] CORBA STATUS
On Tue, 9 Nov 1999, Dmitry Samersoff wrote:
> > Anyway, I know that at least one ORB, TAO, runs on many more types of
> > platforms than Postgres does (e.g. VxWorks, Lynx, Solaris, NT, ...),
> > though Postgres runs on more Unix-style platforms. But that particular
> > ORB is not a good candidate for us, for reasons I already mentioned
> > (C++, large build size, poor configure support).
> 
> IMHO, There has no ideal ORB for all platforms.
> we use ORBacus (http://www.ooc.com), 
> because it's the only known for me ORB, working without threads 
> so its really faster and more stable than another ones under FreeBSD,
> but it's not free.

   FNORB - http://www.dstc.edu.au/Products/Fnorb/
   Fnorb is a CORBA 2.0 ORB written in Python. Free for non-commercial
use.

Oleg.
---- 
    Oleg Broytmann     http://members.xoom.com/phd2/     phd2@earthling.net
           Programmers don't die, they just GOSUB without RETURN.


Tue, 9 Nov 1999 18:25:23 +0000 (GMT)
Oleg Broytmann <phd@phd.russ.ru>
Re: [HACKERS] CORBA STATUS

Whomever does this implementation, or works on this, needs to bear in mind
that there are several different ORBs out there, and the code should be
written in such a way that *I* can use MICO, Oleg here can use Fnorb,
someone  else can use ORBit, etc...I personally don't want to have to go
ut and grab <INSERT ORB of the Day here> just so that I can use CORBA,
when I already have Mico installed for other software...


On Tue, 9 Nov 1999, Oleg Broytmann wrote:

> On Tue, 9 Nov 1999, Dmitry Samersoff wrote:
> > > Anyway, I know that at least one ORB, TAO, runs on many more types of
> > > platforms than Postgres does (e.g. VxWorks, Lynx, Solaris, NT, ...),
> > > though Postgres runs on more Unix-style platforms. But that particular
> > > ORB is not a good candidate for us, for reasons I already mentioned
> > > (C++, large build size, poor configure support).
> > 
> > IMHO, There has no ideal ORB for all platforms.
> > we use ORBacus (http://www.ooc.com), 
> > because it's the only known for me ORB, working without threads 
> > so its really faster and more stable than another ones under FreeBSD,
> > but it's not free.
> 
>    FNORB - http://www.dstc.edu.au/Products/Fnorb/
>    Fnorb is a CORBA 2.0 ORB written in Python. Free for non-commercial
> use.
> 
> Oleg.
> ---- 
>     Oleg Broytmann     http://members.xoom.com/phd2/     phd2@earthling.net
>            Programmers don't die, they just GOSUB without RETURN.
> 
> 
> ************
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Tue, 9 Nov 1999 17:39:20 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] CORBA STATUS
> Whomever does this implementation, or works on this, needs to bear in mind
> that there are several different ORBs out there, and the code should be
> written in such a way that *I* can use MICO, Oleg here can use Fnorb,
> someone  else can use ORBit, etc...I personally don't want to have to go
> ut and grab <INSERT ORB of the Day here> just so that I can use CORBA,
> when I already have Mico installed for other software...

Then forget it. Corba implementations don't work this way (yet).

Sorry you're not interested...

                    - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Wed, 10 Nov 1999 03:01:59 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] CORBA STATUS
On Wed, 10 Nov 1999, Thomas Lockhart wrote:

> > Whomever does this implementation, or works on this, needs to bear in mind
> > that there are several different ORBs out there, and the code should be
> > written in such a way that *I* can use MICO, Oleg here can use Fnorb,
> > someone  else can use ORBit, etc...I personally don't want to have to go
> > ut and grab <INSERT ORB of the Day here> just so that I can use CORBA,
> > when I already have Mico installed for other software...
> 
> Then forget it. Corba implementations don't work this way (yet).

Wait...when we talked about this months back, I swore that one of the
conclusions *was* that this was possible...it would involve us doing
wrapper functions in our code that were defined in an include file based
on which ORB implementation was used...?

Basically...

pg_<corba function> maps to <insert mico corba function here>
			 or <insert orbit corba function here>
		  	 or <insert other implementation function here>

Has this ability changed? *raised eyebrow*


Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Wed, 10 Nov 1999 00:07:13 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] CORBA STATUS
> Wait...when we talked about this months back, I swore that one of the
> conclusions *was* that this was possible...it would involve us doing
> wrapper functions in our code that were defined in an include file based
> on which ORB implementation was used...?
> Basically...
> pg_<corba function> maps to <insert mico corba function here>
>                          or <insert orbit corba function here>
>                          or <insert other implementation function here>
> Has this ability changed? *raised eyebrow*

No, this probably is not necessary since the C or C++ mappings for
function calls in Corba are very well defined. 

What is not fully specified in the Corba standard is, for example,
which header files (and by what names) will be generated by the IDL
stubber, so each Orb has, or might have, different conventions for
include files. This probably impacts server-side code a bit more than
clients.

There is some interest for some Orbs to try lining up the header file
names, but I don't know how feasible it is in the short term.

We could probably isolate this into Postgres-specific header files,
but there will probably be Orb-specific #ifdef blocks in those
headers.

                    - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Wed, 10 Nov 1999 04:25:27 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] CORBA STATUS
On Wed, 10 Nov 1999, Thomas Lockhart wrote:

> > Wait...when we talked about this months back, I swore that one of the
> > conclusions *was* that this was possible...it would involve us doing
> > wrapper functions in our code that were defined in an include file based
> > on which ORB implementation was used...?
> > Basically...
> > pg_<corba function> maps to <insert mico corba function here>
> >                          or <insert orbit corba function here>
> >                          or <insert other implementation function here>
> > Has this ability changed? *raised eyebrow*
> 
> No, this probably is not necessary since the C or C++ mappings for
> function calls in Corba are very well defined. 
> 
> What is not fully specified in the Corba standard is, for example,
> which header files (and by what names) will be generated by the IDL
> stubber, so each Orb has, or might have, different conventions for
> include files. This probably impacts server-side code a bit more than
> clients.
> 
> There is some interest for some Orbs to try lining up the header file
> names, but I don't know how feasible it is in the short term.
> 
> We could probably isolate this into Postgres-specific header files,
> but there will probably be Orb-specific #ifdef blocks in those
> headers.

Is there any reason configure couldn't handle this?

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH   email: vev@michvhf.com   flame-mail: /dev/null
  # include <std/disclaimers.h>       Have you seen http://www.pop4.net?
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Wed, 10 Nov 1999 06:22:45 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [HACKERS] CORBA STATUS
On Wed, 10 Nov 1999, Thomas Lockhart wrote:

> > Wait...when we talked about this months back, I swore that one of the
> > conclusions *was* that this was possible...it would involve us doing
> > wrapper functions in our code that were defined in an include file based
> > on which ORB implementation was used...?
> > Basically...
> > pg_<corba function> maps to <insert mico corba function here>
> >                          or <insert orbit corba function here>
> >                          or <insert other implementation function here>
> > Has this ability changed? *raised eyebrow*
> 
> No, this probably is not necessary since the C or C++ mappings for
> function calls in Corba are very well defined. 
> 
> What is not fully specified in the Corba standard is, for example,
> which header files (and by what names) will be generated by the IDL
> stubber, so each Orb has, or might have, different conventions for
> include files. This probably impacts server-side code a bit more than
> clients.
> 
> There is some interest for some Orbs to try lining up the header file
> names, but I don't know how feasible it is in the short term.
> 
> We could probably isolate this into Postgres-specific header files,
> but there will probably be Orb-specific #ifdef blocks in those
> headers.

Right, which is something that I thought we had pseudo-agreed upon the
last time through, that we woudl most likely require this...hadn't
realized it was for 'Orb-header files', but, IMHO, that's no worse then
putting in HAVE_MICO vs HAVE_ORBIT blocks and having it a configure
option...

See...I am interested, just not interested in having us tied to one
"vendor"...:)

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Wed, 10 Nov 1999 10:58:12 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] CORBA STATUS
On Wed, 10 Nov 1999, Vince Vielhaber wrote:

> On Wed, 10 Nov 1999, Thomas Lockhart wrote:
> 
> > > Wait...when we talked about this months back, I swore that one of the
> > > conclusions *was* that this was possible...it would involve us doing
> > > wrapper functions in our code that were defined in an include file based
> > > on which ORB implementation was used...?
> > > Basically...
> > > pg_<corba function> maps to <insert mico corba function here>
> > >                          or <insert orbit corba function here>
> > >                          or <insert other implementation function here>
> > > Has this ability changed? *raised eyebrow*
> > 
> > No, this probably is not necessary since the C or C++ mappings for
> > function calls in Corba are very well defined. 
> > 
> > What is not fully specified in the Corba standard is, for example,
> > which header files (and by what names) will be generated by the IDL
> > stubber, so each Orb has, or might have, different conventions for
> > include files. This probably impacts server-side code a bit more than
> > clients.
> > 
> > There is some interest for some Orbs to try lining up the header file
> > names, but I don't know how feasible it is in the short term.
> > 
> > We could probably isolate this into Postgres-specific header files,
> > but there will probably be Orb-specific #ifdef blocks in those
> > headers.
> 
> Is there any reason configure couldn't handle this?

As simple as a '--with-corba=mico' configure option ... or, I would think?


Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Wed, 10 Nov 1999 10:58:47 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] CORBA STATUS
The Hermit Hacker wrote:
  >> Is there any reason configure couldn't handle this?
  >
  >As simple as a '--with-corba=mico' configure option ... or, I would think?
 
>From the point of view of a package maintainer, I would much prefer a
solution that separated the choice of Orb from the main build.

If the choice goes into configure, I will have to pick a single Orb and
build the Debian package for that, or else make packages for every
Orb that Debian supports.  It would be better if I could build a generic
Orb-enabled PostgreSQL and produce a little pg-orb connection library
for each Debian-supported Orb.


-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "But thanks be to God, which giveth us the victory 
      through our Lord Jesus Christ."     
                              I Corinthians 15:57 



Wed, 10 Nov 1999 15:42:53 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] CORBA STATUS 
> See...I am interested, just not interested in having us tied to one
> "vendor"...:)

I know. That still doesn't keep me from being in a bad mood after
spending 3.5 hours at the dentist yesterday :((

On the Corba fork vs thread issue:

It is true that the server process would need to be handed off to the
client in a different manner from the postmaster; with Corba you don't
just fork onto a different port and be done with it.

However, the postmaster *could* start up a server process and return
an IOR to the client, which givs the client a direct handle to the
server. The client would then initiate contact directly with the
server, and the postmaster is no longer involved.

afaik you could still fork in the postmaster, though whether our
streamlined tricks would work with every Orb is not certain. But that
is an optimization for our specific forking implementation, not a
fundamental feature.

As I mentioned, the real performance benefits come from having an
optimized query connection which bypasses the *expensive* string
conversions we currently use to pass data around.

                     - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Wed, 10 Nov 1999 15:43:46 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] CORBA STATUS
On Wed, 10 Nov 1999, Oliver Elphick wrote:
> The Hermit Hacker wrote:
>   >> Is there any reason configure couldn't handle this?
>   >
>   >As simple as a '--with-corba=mico' configure option ... or, I would think?
>  
> From the point of view of a package maintainer, I would much prefer a
> solution that separated the choice of Orb from the main build.
> 
> If the choice goes into configure, I will have to pick a single Orb and
> build the Debian package for that, or else make packages for every
> Orb that Debian supports.  It would be better if I could build a generic
> Orb-enabled PostgreSQL and produce a little pg-orb connection library
> for each Debian-supported Orb.

The same issue is true for the RPM's -- which ORB?  If I'm on RedHat Linux, the
choice of ORB is going to depend upon the choice of desktops -- KDE or GNOME. 
ORBit is packaged standard for GNOME -- KDE 2 is going to use something else --
now, my understanding of CORBA is quite limited -- Thomas, you have far more
experience, as you are actively developing CORBA stuff.  If I choose to install
just KDE, KDE's ORB is going to be installed -- if I install GNOME, ORBit is
going to be installed. If I install both (the default), both ORB's will be
resident.

I can force the installation of a particular ORB through dependencies, but
that seems messy to me.

I CAN produce multiple sets of packages -- but that's going to cause all
manner of confusion for users.

Is it possible in the CORBA context to do what Oliver mentioned with a 'pg_orb'
abstraction layer to a generic CORBA-enabled postgreSQL??

It may seem like Oliver and I are getting the cart before the horse, but
the strategic decision of how to integrate CORBA into the system is going to
have wide-ranging repercussions for integrators.

 --
 Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Wed, 10 Nov 1999 10:57:51 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] CORBA STATUS
> The same issue is true for the RPM's -- which ORB?  If I'm on RedHat Linux, the
> choice of ORB is going to depend upon the choice of desktops -- KDE or GNOME.
> ORBit is packaged standard for GNOME -- KDE 2 is going to use something else --
> now, my understanding of CORBA is quite limited -- Thomas, you have far more
> experience, as you are actively developing CORBA stuff.  If I choose to install
> just KDE, KDE's ORB is going to be installed -- if I install GNOME, ORBit is
> going to be installed. If I install both (the default), both ORB's will be
> resident.

Right.

> I can force the installation of a particular ORB through dependencies, but
> that seems messy to me.

Not so bad, but I understand your point.

> I CAN produce multiple sets of packages -- but that's going to cause all
> manner of confusion for users.

Right. Not worth the effort.

> Is it possible in the CORBA context to do what Oliver mentioned with a 'pg_orb'
> abstraction layer to a generic CORBA-enabled postgreSQL??

Maybe, as a second step. The first step (which we are a *long* ways
away from; getting a consensus on how to proceed will take a miracle)
will be to get an implementation using one Orb with the feature set we
want.

> It may seem like Oliver and I are getting the cart before the horse, but
> the strategic decision of how to integrate CORBA into the system is going to
> have wide-ranging repercussions for integrators.

This may not be what people want to hear, and may not be what turns
out, but imho and imle ("little experience" ;)...

Corba is really intended to allow clients and servers implemented with
different Orb products to interoperate transparently. It has very
carefully stayed away from over-specifying exactly *how* a particular
client or server would be implemented for a particular Orb.

The header file conventions, or lack thereof, is a good example of
this. I'm familiar with a couple of the C++ Orbs. Mico produces a
single header file per interface, while TAO produces two. Hmm, it just
dawned on me that I might be able to jigger the output file names from
TAO's IDL compiler to make the names line up with Mico. Will get back
to you on this detail :)

Anyway, if Corba is in our future, I would think that we would work
with a single Orb for the server-side implementation, at least at
first. Once we are up and running, then we talk about how to slip in
someone's favorite other Orb.

For clients, we will have to pick and choose depending on the language
and features required. For example, TAO has portability and some
realtime features and optimizations that make it the *only* choice for
our realtime systems at work. But Mico has a nice TCL binding, so we
are using that to implement some TCL GUIs for commanding and telemetry
interfaces.

Not a big deal, and we quickly got over the *flap arms all over* "Oh
no! This Orb doesn't support language X!!!!".

btw, the Orb which has more language bindings than any other is ILU.
ILU predates Corba by several years, but it has evolved to support the
Corba standard in many areas.

Corba was primarily intended to decouple clients from servers.
Inter-orb transportability at the source code level was a secondary
concern, though the Corba standard, or at least some of the
conventions used in the open-source implementations, may be converging
a bit to help with the source portability. And the biggest
source-level portability concern, the call-level interfaces, is not a
problem.

As we are introducing Corba to new users at work (we've got O(20)
programmers who will be using it on our testbeds and ground
implementations for optical interferometers), I emphasize the
following:

1) Corba makes distributed computing easy, in that clients (the
calling programs) call servers (the subroutines) as though they were
local to the client. But in fact they could reside anywhere.

2) Specifying interfaces through IDL is a *great* way to design
systems. If you have the interface, then you know what you need to
implement. From then on, clients and servers, or callers and call-ees,
can be implemented independently. If we end up with Corba in our
server, then we could/should start specifying our internal interfaces
with IDL also.

3) Since clients and servers are decoupled through well-defined
interfaces, and since these interfaces can be decoupled "on the wire",
you have great flexibility in how you mix and match Orb products to
implement clients and servers. But afaik all of the Orbs have a "short
circuit" which will allow you to build Corba-enabled routines written
with that Orb into the same image, without taking a hit at runtime to
marshall/unmarshall/network/etc.

One in an occasional series... ;)

                 - Thomas

btw, I'm guessing that the way to get Corba going is to have a code
freeze/fork, and have a few people work on demonstrating Corba using
that frozen version. Then we re-merge later if the Corba demo was a
success *and* if Corba is what we want in the main-line product. That
could happen during our 7.x series of releases, and if the world wants
Corba, we could mainstream it for v8.0.

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Wed, 10 Nov 1999 17:23:47 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] CORBA STATUS
"Oliver Elphick" <olly@lfix.co.uk> writes:

> From the point of view of a package maintainer, I would much prefer
> a solution that separated the choice of Orb from the main build.
> 
> If the choice goes into configure, I will have to pick a single Orb
> and build the Debian package for that, or else make packages for
> every Orb that Debian supports.  It would be better if I could build
> a generic Orb-enabled PostgreSQL and produce a little pg-orb
> connection library for each Debian-supported Orb.

It probably makes sense to try to get things working with one ORB, and
then see if it's worth generalising.  I'd guess ORBit is a good one to
start with since it's C-based, and it's pretty small: if I have to
install an extra package on my machine just to run PostgreSQL, I'm not
going to mind ORBit, whereas TAO might annoy me (IIRC, TAO is quite
big; I may be thinking of another ORB, though).  ORBit does IIOP, I
believe, so that covers GNOME and KDE people.

10 Nov 1999 23:37:38 +0000
Bruce Stephens <bruce@cenderis.demon.co.uk>
Re: [INTERFACES] Re: [HACKERS] CORBA STATUS
> It probably makes sense to try to get things working with one ORB, and
> then see if it's worth generalising.  I'd guess ORBit is a good one to
> start with since it's C-based, and it's pretty small: if I have to
> install an extra package on my machine just to run PostgreSQL, I'm not
> going to mind ORBit, whereas TAO might annoy me (IIRC, TAO is quite
> big; I may be thinking of another ORB, though).  ORBit does IIOP, I
> believe, so that covers GNOME and KDE people.

Yeah, TAO might annoy you; it takes 1.3GB of disk space to build,
though much less once built ;)

                     - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Thu, 11 Nov 1999 15:35:58 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [INTERFACES] Re: [HACKERS] CORBA STATUS
Hello,
I am attempting to recover a database previously secured with pg_dump,
however on attempting to restore using 

pgsql < db_security_file 

I get the following error(s)


ERROR:  tbl_breeders relation already exists

I have removed all data tables and user from the database,
what am I over looking?

Stephen 



Thu, 11 Nov 1999 13:00:57 -0800
"Stephen Martin" <stephen@sealteam.demon.co.uk>
Error on db recovery..
"Stephen Martin" <stephen@sealteam.demon.co.uk> el dï¿½a Thu, 11 Nov 1999 
13:00:57 -0800, escribiï¿½:

>Hello,
>I am attempting to recover a database previously secured with pg_dump,
>however on attempting to restore using 
>
>pgsql < db_security_file 

bad, bad ...

go to /usr/doc/postgres and read ...


Sergio


Thu, 11 Nov 1999 19:04:20 -0300
"Sergio A. Kessler" <ser@perio.unlp.edu.ar>
Re: [INTERFACES] Error on db recovery..
> Dmitry Samersoff wrote:
> > 
> > On 13-Oct-99 Vince Vielhaber wrote:
> > >
> > > On 13-Oct-99 Bruce Momjian wrote:
> > 
> > 2BOOK Authors:
> > Please, try to keep rights for translating this book into another
> > languages by you self, not by publisher.
> > 
> > I may ask some St.Pitersburg's publishing company
> > to make russian translation of this book, but some publishers
> > like O'Reilly have too hard license policy
> > and too long reaction time.
> > 
> 
> I may ask some Indonesian's publishing company to make
> Indonesian translation of this book too.
> I may help the translation from English to Indonesian language.

Addison-Wesley does a lot of foreign rights sales.  Let me know when you
want information.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 29 Nov 1999 20:31:46 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [DOCS] Re: [HACKERS] Outline for PostgreSQL book
Mark Proctor wrote:

>  I'm currently studying for a Masters at BrunelUniversity, London. I
> was looking for a thesis andsince I had an interest inOpensource
> development anddatabases I thought that I would like to work onPGsql.
> As I also had an interest in trying tounderstand CORBA and other
> related technologies itmade sense that I try and work on a project
> that tiesPGsql and CORBA together in some way.
>
> Marc G. Fournier and Bruce Momjian were contacted withregards to this
> and they seemed to think that therewas room for me to work on this,
> although at thisstage I hadn't gone through the mailing list.
>
> since then I've finally got PGsql up and working andI'm now embarking
> on understand how to system actualworks. I thought a good place to
> start with myresearch would be to go through the mailing lists totry
> and see the current status for development in thisarea. I did this for
> several hours trying to get thecurrent take on CORBA with regards to
> PGsql. It seemsthere was a lot of initial talk back in Nov 1998 onthe
> hackers list with the first thoughts of some sortof CORBA
> implementation. The conversation orientatedover 2 main areas; which
> ORB and implementationmethods, with some peoples offering to work on
> this.It was then suggested that the converstaion take placein the
> interfaces mailing list. I've since been oversome of this but find it
> very difficult to understandthe current status with regards to PGsql
> and CORBA.I've seen many references to people who have developed
> a project that allows them to work with PGsql viaCORBA, but non of
> this seems to be part of the mainproject or system. There is some
> reference to Micheal Robinsonworking on an implementation but again
> I'm not surehow this fits
> in-http://www.postgresql.org/mhonarc/pgsql-interfaces/1998-11/msg00090.html
>
> Is there room for me to work on this project in such away that it is
> adequate for my masters. If anyone isworking on this, or has a good
> knowledge of thecurrent status of the CORBA implementation for
> PGsqlcan you please let me know, so I can know whether toget started
> on this or not. The reference thread formy initial point of contact
> with Marc G. Fournier andBruce Momjian and how they think I should
> attack theproject is
> -http://www.postgresql.org/mhonarc/pgsql-hackers/1999-09/msg00076.html
>
> Regards
>
> Mark Proctor
> email : m.proctor@bigfoot.com

Dear Mark,

as this reply stuck between my brain and my fingers for two weeks now,
it might come too late.
But Corba defines a Facility (or a Service) to access object oriented
databases via a standardized interface. Since Postgres is an OORDBMS
(there is a definitely shorter abbreviation), it might suit it well to
offer such an interface. I never looked too deep into the standard
document but if you are interested, I'll look up the exact location.

Since this does not interfere with Postgres' internals it should be much
easier to do. And since a redesign (for speed) of the backend interface
would not provide new features, I would suggest taking a look into this
area.

IMHO the Corba specs should get more attention. I really like these
standardization efforts, though they rarely affect everyday (free)
programming environments.

Regards
  Christof
 (which was tempted to implement them some month ago but decided to
build actual programs on now-available and (to me) well known technology
(ecpg driven Corba objects)).


Tue, 30 Nov 1999 05:44:10 +0100
Christof Petig <christof.petig@wtal.de>
Re: [HACKERS] CORBA STATUS
In my digging around in initdb, I came to the --username option, which
supposedly allows you to initialize the database system with another
username. Not a bad idea, really.

Obviously, you'd have to be root in that case, because you want to create
files in someone else's name. But if you are root, the backends will
refuse to execute, wisely so. So this option is totally broken.

I propose that I remove it, and that it instead be possible that you can
do

root# su -l postgres -c 'initdb ...'

if you are not a fan of logging in and out of user accounts during the
install process.

That would also remove a whole bunch of the username/id checking logic,
since you just use the user you're logged in as ($UID, $USER), and if the
backend doesn't like it, it will tell you. pg_id adieu.


Another question: Is there a reason why the system views in initdb are all
created with a CREATE TABLE and then with a CREATE RULE, instead of using
CREATE VIEW? Is that a left over from the time before there were views as
such?


Question 3: Is there a reason why the template1 database is vacuumed twice
in the process? Once before all the views are created (no analyze) and
once at the very end (with analyze).

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Thu, 9 Dec 1999 01:18:37 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
More initdb follies
Peter Eisentraut <peter_e@gmx.net> writes:
> In my digging around in initdb, I came to the --username option, which
> supposedly allows you to initialize the database system with another
> username. Not a bad idea, really.
>
> Obviously, you'd have to be root in that case, because you want to create
> files in someone else's name. But if you are root, the backends will
> refuse to execute, wisely so. So this option is totally broken.
>
> I propose that I remove it,

Makes sense to me --- I was saying more or less the same thing, I think,
when I said that initdb should pay attention to the effective UID it's
run under *and nothing else* to determine the postgres user name/ID.
In particular, if you want to be able to run it via "su", it mustn't
assume that environment variables like LOGNAME or USER are set correctly
for the postgres user.  If it needs the user name it should look it up
from the EUID.

> Another question: Is there a reason why the system views in initdb are all
> created with a CREATE TABLE and then with a CREATE RULE, instead of using
> CREATE VIEW? Is that a left over from the time before there were views as
> such?

Probably, but it's before my time.

> Question 3: Is there a reason why the template1 database is vacuumed twice
> in the process? Once before all the views are created (no analyze) and
> once at the very end (with analyze).

I've wondered about that too.  There are some comments in initdb
suggesting that other orderings might fail, but I wouldn't be surprised
if those are obsolete.  Have you tried altering the procedure?

			regards, tom lane

Wed, 08 Dec 1999 20:38:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] More initdb follies 
> Question 3: Is there a reason why the template1 database is vacuumed twice
> in the process? Once before all the views are created (no analyze) and
> once at the very end (with analyze).

Yes, there is a reason, though I can't remember why.  We need the
analyze at the end so the system tables are completely optimized, but we
need the earlier vacuum to set some table statistics that don't get set
by the raw load used by initdb.  You can try removing the first one to
see if it works.

Seems it works without the first initdb here.  I will apply a patch to
remove the first initdb.  I think we needed it long ago for some reason.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 8 Dec 1999 21:54:16 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] More initdb follies
> Question 3: Is there a reason why the template1 database is vacuumed twice
> in the process? Once before all the views are created (no analyze) and
> once at the very end (with analyze).

I see the line before the first vacuum:

# If the COPY is first, the VACUUM generates an error, so we vacuum first

That may have been me adding this.  Seems it is no longer an issue. 
Removed.



-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 8 Dec 1999 21:55:35 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] More initdb follies
Peter Eisentraut wrote:
  >In my digging around in initdb, I came to the --username option, which
  >supposedly allows you to initialize the database system with another
  >username. Not a bad idea, really.
  >
  >Obviously, you'd have to be root in that case, because you want to create
  >files in someone else's name. But if you are root, the backends will
  >refuse to execute, wisely so. So this option is totally broken.
  >
  >I propose that I remove it, and that it instead be possible that you can
  >do
  >
  >root# su -l postgres -c 'initdb ...'

You can already do this;  this example is from the Debian package installation
(which runs as root, of course):

        su  postgres -c "cd ${PGHOME}; . ./${PROFILE}; initdb -e ${Encoding} -l
${PGLIB} -r ${PGDATA} -u postgres"

It can be quite tricky, though, since there are a number of different su
versions around.  I would prefer it if root were able to run initdb directly
and set the ownerships as part of the process.  The ability to run as root
should only apply to initdb.

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Let not sin therefore reign in your mortal body, that 
      ye should obey it in the lusts thereof."            
                                   Romans 6:12 



Thu, 09 Dec 1999 11:14:28 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] More initdb follies 
On 1999-12-09, Oliver Elphick mentioned:

>   >I propose that I remove it, and that it instead be possible that you can
>   >do
>   >
>   >root# su -l postgres -c 'initdb ...'
> 
> You can already do this;  this example is from the Debian package installation

> It can be quite tricky, though, since there are a number of different su
> versions around.  I would prefer it if root were able to run initdb directly

Of course the user would actually invoke the su himself, so he better know
what his does.

> and set the ownerships as part of the process.  The ability to run as root
> should only apply to initdb.

I don't think there is a good (let alone portable) way to set ownership of
a shell script at run time. While you could do all kinds of funny business
with file ownerships, the postgres backends will still refuse to execute.

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden




Sat, 11 Dec 1999 01:20:38 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] More initdb follies 
On 1999-12-08, Tom Lane mentioned:

> Makes sense to me --- I was saying more or less the same thing, I think,
> when I said that initdb should pay attention to the effective UID it's
> run under *and nothing else* to determine the postgres user name/ID.
> In particular, if you want to be able to run it via "su", it mustn't
> assume that environment variables like LOGNAME or USER are set correctly
> for the postgres user.  If it needs the user name it should look it up
> from the EUID.

(The odd thing is, that our installation instructions suggest to su as
postgres and do the install that way, yet this seems to work anyway. On my
system (GNU sh-utils), USER is only set if you 'su -', but it seems
relying on the user doing this right is way too much to ask.)

Hmm, portability issues. Can you be sure EUID is implemented everywhere?
(I guess so.) How do you determine the user name from a UID? That would be
the inverse of what pg_id does now.

An idea I just had would be to use
* First resort: `id -n -u`
* Second resort: `whoami`
* Third resort: --username option

The first two don't distinguish between su and su -, at least here. Not
sure if #3 is necessary though. One of those you should have.

On the other hand, one more thing to think about is that the only place
the superuser's UID and name are actually used, is to initialize the
catalogs. All the files will happily be created under which ever user you
run this as (as they should). Using the above three step approach, you can
choose to name your _database_ superuser whatever you want, independent of
the Unix filesystem concerns. Of course this is nothing to encourage, but
it's possible. I could install the system in my home directory in some
computing lab under my own user name, yet still name the superuser
postgres to have a standard environment within the database. I guess the
--username option does kind of work, just not as expected.

(The next thing would be the ability to choose the database superuser's
usesysid as well, which might not be so far-fetched either, because if
1) you already have 40000 users on your system when you install PostgreSQL
2) you assign the Unix user postgres the uid 400001
3) you add your 40000 users to the database
you end up with usesysid's 40000-80000 (+/- 1). Not the end of the world
but kind of dumb. The you could simply assign 0 (or 1 if zero is not
allowed) to the superuser at initdb time.)

There seem to be a lot of misconceptions (possibly caused by restrictions
in the past) about all the user names and ids of the installation files,
the data directory, the server process, the database users, the database
super user, etc. In fact, most of these can be chosen freely, the only
requirement is that the server process can access the data directory.

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Sat, 11 Dec 1999 01:20:49 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] More initdb follies 
On Wed, 08 Dec 1999, I wrote:
> I have a table with a field varchar(5), filed with right alligned numbers.
> Ordering was fine, just like we expected compared with nummeric.
> 
> Starting from RedHat version 6.1 the ordening seems to remove the leading
> blanco's, what was not for use. I try different versions of postgres, as
> there are 6.5.2-1, 6.5.3-1, 6.5.3-2
> I also try to change varchar in char, and remove the index on the varchar
> field but nothing helps.

Today i take up again and see the ordering for me (third time i install
RH6.1), now 
with postgres version 6.5.2-1
	the problem is always the same

When i do the following
CREATE TABLE BLANK (column1 varchar(5));
INSERT  INTO BLANK (column1) VALUES ('    1');
INSERT  INTO BLANK (column1) VALUES ('   11');
INSERT  INTO BLANK (column1) VALUES ('  100');
INSERT  INTO BLANK (column1) VALUES ('    2');

then:
SELECT * FROM BLANK order by column1;

I received

  1
 100
 11
  2		--> mark also a not aligned output.

and I expected
    1
    2
   11
  100


Anybody has an idea?? 

Frans


Tue, 14 Dec 1999 23:46:47 +0100
Frans Van Elsacker <fve@atbib.be>
ordering RH6.1
Frans Van Elsacker <fve@atbib.be> writes:
> When i do the following
> CREATE TABLE BLANK (column1 varchar(5));
> INSERT  INTO BLANK (column1) VALUES ('    1');
> INSERT  INTO BLANK (column1) VALUES ('   11');
> INSERT  INTO BLANK (column1) VALUES ('  100');
> INSERT  INTO BLANK (column1) VALUES ('    2');
> then:
> SELECT * FROM BLANK order by column1;

> I received
>   1
>  100
>  11
>   2		--> mark also a not aligned output.

> and I expected
>     1
>     2
>    11
>   100

> Anybody has an idea?? 

Bizarre.  I see the expected results under both 6.5.3 and current
development sources:

play=> SELECT * FROM BLANK order by column1;
column1
-------
      1
      2
     11
    100
(4 rows)

I wonder if this could be a LOCALE or MULTIBYTE issue.  Do you have
either feature enabled in your copy, and if so what locale/encoding
do you use?  (I'm running plain vanilla no-USE_LOCALE, no-MULTIBYTE
code, so that might be why I don't see anything funny...)

			regards, tom lane

Tue, 14 Dec 1999 18:28:57 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] ordering RH6.1 
Tom Lane wrote:
  >Bizarre.  I see the expected results under both 6.5.3 and current
  >development sources:
  >
  >play=> SELECT * FROM BLANK order by column1;
  >column1
  >-------
  >      1
  >      2
  >     11
  >    100
  >(4 rows)
  >
  >I wonder if this could be a LOCALE or MULTIBYTE issue.  Do you have
  >either feature enabled in your copy, and if so what locale/encoding
  >do you use?  (I'm running plain vanilla no-USE_LOCALE, no-MULTIBYTE
  >code, so that might be why I don't see anything funny...)

I've tried this with both locale and multibyte enabled and with
LANG=en_GB.  The results are correct.  (Version = 6.5.3).
-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "The fear of the LORD is the instruction of wisdom, and
      before honour is humility."      Proverbs 15:33 



Wed, 15 Dec 1999 10:29:09 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] ordering RH6.1 
Tom Lane wrote:
 
> I wonder if this could be a LOCALE or MULTIBYTE issue.  Do you have
> either feature enabled in your copy, and if so what locale/encoding
> do you use?  (I'm running plain vanilla no-USE_LOCALE, no-MULTIBYTE
> code, so that might be why I don't see anything funny...)

He's running the RPM distribution, which at that release has
--enable-locale but no multibyte.

Using the no-locale RPM's I last built, I can't reproduce his results.

Frans, try out the no-locale rpm set and see if the result changes, if
you please.  (using wget, you would do: wget
http://www.ramifordistat.net/postgres/RPMS/redhat-6.x/postgresql*-2nl.i386.rpm
)

This will verify whether it is locale-related or not.  I would install
the locale RPMs and test for you right now, but my 6.1 machine is at
home.  If it is inconvenient for you to download this, let me know, and
I'll try to test tonight at home -- although, I've been meaning to do
just that for nearly a week now, but I haven't even fired up the machine
at home in the last week.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Wed, 15 Dec 1999 10:55:28 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] ordering RH6.1
I download all the rpm from
(http://www.ramifordistat.net/postgres/RPMS/redhat-6.x/postgresql*-2nl.i386.
rpm)
and have install each of them in redhat 6.1.

But we received the same bad results as before.

	- my redhat was downloaded from a mirror site as a cd-image
	- and there where some older version of postgres (v 6.5.2-1) installed on
our test system before. They were first uninstalled.

This are the only two packages that we have running on our machine. I've
choose the half-automatic graphic install of redhat. Selected some standard
tools like ftp, network tool,... and choose a belgian keyboard. I have left
the timezone selection as default.

Strange things!

I see only the following posibilities :
		- Our redhat cd-image is different from yours (Why ??)
		- influence of the older versions 
		- keyboard settings ???

This is our result :

[postgres@dekatest pgsql]$ psql test
Welcome to the POSTGRESQL interactive sql monitor:
  Please read the file COPYRIGHT for copyright terms of POSTGRESQL
[PostgreSQL 6.5.3 on i586-pc-linux-gnu, compiled by gcc egcs-2.91.66]

   type \? for help on slash commands
   type \q to quit
   type \g or terminate with semicolon to execute query
 You are currently connected to the database: test

test=> CREATE TABLE BLANK (column1 varchar(5));
CREATE
test=> INSERT  INTO BLANK (column1) VALUES ('    1');
INSERT 587145 1
test=> INSERT  INTO BLANK (column1) VALUES ('   11');
INSERT 587146 1
test=> INSERT  INTO BLANK (column1) VALUES ('  100');
INSERT 587147 1
test=> INSERT  INTO BLANK (column1) VALUES ('    2');
INSERT 587148 1
test=> SELECT * FROM BLANK order by column1;
column1
-------
      1
    100
     11
      2
(4 rows)
          
Any Idea ?

greetings,
Frans


At 10:55 15/12/99 -0500, you wrote:
>Tom Lane wrote:
> 
>> I wonder if this could be a LOCALE or MULTIBYTE issue.  Do you have
>> either feature enabled in your copy, and if so what locale/encoding
>> do you use?  (I'm running plain vanilla no-USE_LOCALE, no-MULTIBYTE
>> code, so that might be why I don't see anything funny...)
>
>He's running the RPM distribution, which at that release has
>--enable-locale but no multibyte.
>
>Using the no-locale RPM's I last built, I can't reproduce his results.
>
>Frans, try out the no-locale rpm set and see if the result changes, if
>you please.  (using wget, you would do: wget
>http://www.ramifordistat.net/postgres/RPMS/redhat-6.x/postgresql*-2nl.i386.
rpm
>)
>
>This will verify whether it is locale-related or not.  I would install
>the locale RPMs and test for you right now, but my 6.1 machine is at
>home.  If it is inconvenient for you to download this, let me know, and
>I'll try to test tonight at home -- although, I've been meaning to do
>just that for nearly a week now, but I haven't even fired up the machine
>at home in the last week.
>
>--
>Lamar Owen
>WGCR Internet Radio
>1 Peter 4:11
>
>


Thu, 16 Dec 1999 23:53:45 +0100
Frans Van Elsacker <fve@atbib.be>
Re: [HACKERS] ordering RH6.1
[Cristian, Jeff: we have a problem here.  RedHat 6.1 Install versus
RedHat 6.0 upgraded to 6.1 behaves differently. Ideas of where to start
looking?]

Frans Van Elsacker wrote:
> But we received the same bad results as before.
> column1
> -------
>       1
>     100
>      11
>       2
> (4 rows)
> 
> Any Idea ?

Ok, I bit the bullet and spent the whole day (plus or minus a couple of
hours) putting together a test bed.  I have three machines in this
testbed:
1.)	My production server, running Mandrake 5.3, Postgresql 6.5.3-2nl;
2.)	My backup server, running RedHat 6.0, Postgresql 6.5.3-2nl;
3.)	My development server, freshly installed with RH 6.1 + all updates,
PostgreSQL 6.5.3-2nl.

I have now reproduced the results.  HOWEVER, my home machine didn't
reproduce the earlier results, and it is RedHat 6.1 (an upgrade from RH
6.0).

For Mandrake 5.3:

column1
-------
      1
      2
     11
    100
(4 rows)

For RH 6.0: ditto Mandrake 5.3.

for RH 6.1 (fresh install):
column1
-------
      1
    100
     11
      2

So, I moved the physical database structure over from the 6.1 machine to
the 6.0 machine and redid the select: the right results.

The RedHat 6.0 machine is running the same exact postgres binaries that
the RedHat 6.1 machine is running -- the 6.5.3-2nl rpms were built on my
home RedHat 6.1 machine. The Mandrake 5.3 machine is running the RedHat
5.2 binaries built by the alternate boot set on the development server
(which is why it took most of the day to set things up....).

Ok, hackers:

What library routine is used to do the order by in this case?

I'm going to retry this exact set of queries again at home -- I wasn't
able to reproduce the last set of results -- but we'll see what happens
here.

Strange.

I'll see what I can find -- this also explains some strange regression
results I was mailed awhile back.  In fact, let's try regression on the
RH 6.1 fresh install.... AND I AM GETTING FAILURES THAT I HAVE NEVER
GOTTEN AT HOME ON MY UPGRADE REDHAT 6.1!  

Recap while I'm waiting for regression to finish:
The fresh install of RedHat 6.1 is from the exact same CD that I
upgraded my home box from RH 6.0.  The ONLY difference is the fresh
install versus the upgrade -- same versions of PostgreSQL. I am going to
double check regression at home, but I have not seen these results
before, and I distinctly remember running regression at home.  I'll keep
you all updated.

[Nine minutes later]

Failures: float8, geometry, select implicit, select having, and select
views.  The regress.out and regression.diffs are attached. Float8 and
geometry are normal.

Looking at the regression diffs, it is obvious that there is a collation
problem here.  But where is this collation sequence problem coming from?
(Note that the 6.5.3-2nl RPMs are built without locale support.)

I'm going to go digging into a diff of my home machine versus this new
RH 6.1 install.
--
Lamar Owen
WGCR Internet Radio
Thu, 16 Dec 1999 19:20:47 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] ordering RH6.1
At 07:20 PM 12/16/99 -0500, Lamar Owen wrote:

>I'll see what I can find -- this also explains some strange regression
>results I was mailed awhile back.  In fact, let's try regression on the
>RH 6.1 fresh install.... AND I AM GETTING FAILURES THAT I HAVE NEVER
>GOTTEN AT HOME ON MY UPGRADE REDHAT 6.1!  

I know that AOLserver works on 6.0 and has problems on 6.1 (a 2.2.12
kernel) and works again if you upgrade your 6.1 to a 2.2.13 kernel.
(have to use "-i" on 6.1 or it won't work, though I forget offhand
what "-i" does for AOLserver)

This ain't specific help but at least PostgreSQL's not alone in 
having weird problems with RH 6.* releases.



- Don Baccus, Portland OR <dhogaza@pacifier.com>
  Nature photos, on-line guides, Pacific Northwest
  Rare Bird Alert Service and other goodies at
  http://donb.photo.net.

Thu, 16 Dec 1999 17:06:01 -0800
Don Baccus <dhogaza@pacifier.com>
Re: [HACKERS] ordering RH6.1
On Thu, 16 Dec 1999, Lamar Owen wrote:
 
> [Cristian, Jeff: we have a problem here.  RedHat 6.1 Install versus
> RedHat 6.0 upgraded to 6.1 behaves differently. Ideas of where to start
> looking?]
> I'm going to retry this exact set of queries again at home -- I wasn't
> able to reproduce the last set of results -- but we'll see what happens
> here.

Ok, confirmation.  On my home machine, which was upgraded to RedHat 6.1 from
RedHat 6.0, I get the correct results:
column1
------
      1
     11
    100
      2
(4 rows)

> Recap while I'm waiting for regression to finish:
> The fresh install of RedHat 6.1 is from the exact same CD that I
> upgraded my home box from RH 6.0.  The ONLY difference is the fresh
> install versus the upgrade -- same versions of PostgreSQL. I am going to
> double check regression at home, but I have not seen these results
> before, and I distinctly remember running regression at home.  I'll keep
> you all updated.

Update: regression tests that fail on my 6.0-6.1 home machine: float8 and
geometry -- which are normal to fail on RedHat any version.  IOW, no collation
problems at home! Oh, I'm running the exact same postgresql binary RPM's at
home as I am running on the fresh RH 6.1 install at work.

Time to dig into date and time stamps on installed RPMs versus updated RPMs.

Frans, try installing RedHat 6.0 on a box, then upgrading to RH 6.1, then rerun
your tests and see what happens.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Thu, 16 Dec 1999 20:53:49 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] ordering RH6.1
Lamar Owen <lamar.owen@wgcr.org> writes:
> For RH 6.0: [ correct results ]

> for RH 6.1 (fresh install):
> column1
> -------
>       1
>     100
>      11
>       2

> So, I moved the physical database structure over from the 6.1 machine to
> the 6.0 machine and redid the select: the right results.

> The RedHat 6.0 machine is running the same exact postgres binaries that
> the RedHat 6.1 machine is running -- the 6.5.3-2nl rpms were built on my
> home RedHat 6.1 machine.

Wow.  Same data files, same binaries, different results.  Sure looks
like the finger is pointing at 6.1's libc.  (I'm assuming that the
binaries make use of a shared-library libc, not statically-linked-in
routines, right?)

> Ok, hackers:
> What library routine is used to do the order by in this case?

If you compiled with USE_LOCALE, it's strcoll(); if not, strncmp().
See varstr_cmp() in src/backend/utils/adt/varlena.c.

> Looking at the regression diffs, it is obvious that there is a collation
> problem here.  But where is this collation sequence problem coming from?
> (Note that the 6.5.3-2nl RPMs are built without locale support.)

OK...

Your regression failures show collation problems in all three of bpchar,
varchar, and name.  (But curiously, not for text ... hmm ...).  bpchar
and varchar both use varstr_cmp(), but namelt just calls strncmp
unconditionally --- see adt/name.c.  So the evidence is looking very
strong that strncmp has got some kind of problem on RH 6.1.

			regards, tom lane

Thu, 16 Dec 1999 21:04:25 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] ordering RH6.1 
On Thu, 16 Dec 1999, Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> Wow.  Same data files, same binaries, different results.  Sure looks
> like the finger is pointing at 6.1's libc.  (I'm assuming that the
> binaries make use of a shared-library libc, not statically-linked-in
> routines, right?)

Right.

> Your regression failures show collation problems in all three of bpchar,
> varchar, and name.  (But curiously, not for text ... hmm ...).  bpchar
> and varchar both use varstr_cmp(), but namelt just calls strncmp
> unconditionally --- see adt/name.c.  So the evidence is looking very
> strong that strncmp has got some kind of problem on RH 6.1.

More information: the LOCALE enabled-binaries act the same way.  So, there's an
issue with both strcoll and strncmp.  What gets me is that it works perfectly
fine on my RedHat 6.1 box that was upgraded from RedHat 6.0 -- but it does not
work fine at all on a box that I did a fresh install on today -- from the same
CD I did the upgrade.

Hmmm....

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

> 
> 			regards, tom lane

Thu, 16 Dec 1999 21:56:22 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] ordering RH6.1
On Thu, 16 Dec 1999, Lamar Owen wrote:
> On Thu, 16 Dec 1999, Lamar Owen wrote:
>  
> > [Cristian, Jeff: we have a problem here.  RedHat 6.1 Install versus
> > RedHat 6.0 upgraded to 6.1 behaves differently. Ideas of where to start
> > looking?]
> > I'm going to retry this exact set of queries again at home -- I wasn't
> > able to reproduce the last set of results -- but we'll see what happens
> > here.
> 
> Ok, confirmation.  On my home machine, which was upgraded to RedHat 6.1 from
> RedHat 6.0, I get the correct results:

More information: it seems that the i18n support is the cause of this.  If you
remove or rename the file /etc/sysconfig/i18n and restart, then even the fresh
RedHat 6.1 install provides the correct results for this query.  This file, I
think, is created during a fresh installation of RH 6.1 -- it doesn't seem to
belong to any RPM.  An upgrade wouldn't create this file..... (Jeff? Cristian?
am I right on this?)

Running regression,  I get the float8 and geometry failures, but I now get an
opr_sanity failure -- but the other collation failures are gone.  The opr_sanity
failure diff: 

*** expected/opr_sanity.out	Wed May 12 11:02:34 1999
--- results/opr_sanity.out	Thu Dec 16 22:39:45 1999
***************
*** 48,56 ****
      (p1.proargtypes[0] < p2.proargtypes[0]);
  proargtypes|proargtypes
  -----------+-----------
-          25|       1043
         1042|       1043
! (2 rows)
  
  QUERY: SELECT DISTINCT p1.proargtypes[1], p2.proargtypes[1]
  FROM pg_proc AS p1, pg_proc AS p2
--- 48,55 ----
      (p1.proargtypes[0] < p2.proargtypes[0]);
  proargtypes|proargtypes
  -----------+-----------
         1042|       1043
! (1 row)
  
  QUERY: SELECT DISTINCT p1.proargtypes[1], p2.proargtypes[1]
  FROM pg_proc AS p1, pg_proc AS p2


HOWEVER, after doing an initdb and rerunning regression, this test no longer
fails.  FWIW.

I seems that charmap (i18n) rears its ugly head even if locale doesn't.

--
Lamar Owen
WGCR Internet Radio

Thu, 16 Dec 1999 22:35:48 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] ordering RH6.1
On Thu, 16 Dec 1999, Lamar Owen wrote:

> More information: the LOCALE enabled-binaries act the same way.  So, there's an
> issue with both strcoll and strncmp.  What gets me is that it works perfectly
> fine on my RedHat 6.1 box that was upgraded from RedHat 6.0 -- but it does not
> work fine at all on a box that I did a fresh install on today -- from the same
> CD I did the upgrade.

Any differences in the environment variables maybe?

Cristian
--
----------------------------------------------------------------------
Cristian Gafton    --     gafton@redhat.com     --       Red Hat, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 UNIX is user friendly. It's just selective about who its friends are.




Thu, 16 Dec 1999 22:52:23 -0500 (EST)
Cristian Gafton <gafton@redhat.com>
Re: [HACKERS] ordering RH6.1
On Thu, 16 Dec 1999, Cristian Gafton wrote:
> On Thu, 16 Dec 1999, Lamar Owen wrote:
> 
> > More information: the LOCALE enabled-binaries act the same way.  So, there's an
> > issue with both strcoll and strncmp.  What gets me is that it works perfectly
> > fine on my RedHat 6.1 box that was upgraded from RedHat 6.0 -- but it does not
> > work fine at all on a box that I did a fresh install on today -- from the same
> > CD I did the upgrade.
> 
> Any differences in the environment variables maybe?

In a nutshell, yes.  /etc/sysconfig/i18n on the fresh install sets LANG,
LC_ALL, and LINGUAS all to be "en_US".  The upgraded machine at home doesn't
have an /etc/sysconfig/i18n -- nor does the RH 6.0 box.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Thu, 16 Dec 1999 22:58:21 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] ordering RH6.1
>
> Failures: float8, geometry, select implicit, select having, and select
> views.  The regress.out and regression.diffs are attached. Float8 and
> geometry are normal.

More problem data...

We installed RH 6.1 fresh, then installed pgsql 6.5.2 from tar.gz.  Failures on the following regression tests:  float8, geometry, opr_sanity, sanity_check, random, and misc.

On a hunch from this thread, I then removed all postgresql-related RPM packages (with 'rpm -e'), rebuilt pgsql 6.5.2, and all regression tests passed (except float8 and geometry, which is normal).

I've also noticed some new (and possibly related?) RH 6.1 wierdness with a fairly mature perl module, Date::Manip-5.35, that wasn't showing up on RH 6.0.  It is now failing the first time it attempts to ascertain the timezone, and then appears to succeed everytime thereafter for a given process (and TZ is clearly set in the configuration for the module as well as showing up with the
'date' command).  The RPM removal above had no effect on that problem.

Cheers,
Ed Loehr





Thu, 16 Dec 1999 23:20:05 -0600
Ed Loehr <ELOEHR@austin.rr.com>
Re: [HACKERS] ordering RH6.1
Lamar Owen wrote:

> More information: it seems that the i18n support is the cause of this.  If you
> remove or rename the file /etc/sysconfig/i18n and restart, then even the fresh
> RedHat 6.1 install provides the correct results for this query.  This file, I
> think, is created during a fresh installation of RH 6.1 -- it doesn't seem to
> belong to any RPM.  An upgrade wouldn't create this file..... (Jeff? Cristian?
> am I right on this?)

Still more data...

After renaming the file /etc/sysconfig/i18n and rebooting, the perl module
Date::Manip timezone lookup failure described previously has ceased.

It seems there may be at least two issues, possibly related.  My pgsql regression
tests were fixed by nuking the pgsql-related RPMs, but that didn't fix the
Date::Manip perl module problem.  Renaming i18n did.  I didn't test the
SELECT query in question prior to making these changes, but that SELECT query does
indeed now return expected results.

Cheers,
Ed Loehr



Thu, 16 Dec 1999 23:54:36 -0600
Ed Loehr <ELOEHR@austin.rr.com>
Re: [HACKERS] ordering RH6.1
> After renaming the file /etc/sysconfig/i18n and rebooting, the perl module
> Date::Manip timezone lookup failure described previously has ceased.

I spoke too soon.  Timezone problem is not fixed by this as it first appeared.

Cheers,
Ed Loehr


Fri, 17 Dec 1999 00:09:45 -0600
Ed Loehr <ELOEHR@austin.rr.com>
Re: [HACKERS] ordering RH6.1
May I humbly suggest a TODO item (or two)?

On the general list a while back someone wanted to implement
a view where the records visible to the client were
dependent upon an application defined user id. I, too, am in
a similar circumstance. However, one solution would be to do
something like this:

SELECT authenticate(<userid>, <password>);

where <userid> and <password> are submitted by the client
application as input from the user. The authenticate()
routine would be a 'C' language routine which would validate
the userid and password by checking some sort of password
table (the database equivalent of shadowed passwords) and
then either set or clear an environmental variable, say,
CLIENTID and returning either 0 or 1. Then, one could have a
view such as:

CREATE VIEW 'mypayroll' AS SELECT * FROM payroll WHERE
employeeid = clientid();

and the clientid() function simply returns the value of the
environmental variable CLIENTID.  Of course this would only
be useful for single-connection client applications. The
client application would initially connect to PostgreSQL
using a PostgreSQL userid/password which would only have
SELECT privileges on the 'mypayroll' table.

At any rate. the whole point of this deal is that anyone
wanting to write a 'C' function which needs to access tuples
has to use the SPI interface. And at the moment, that means
they need the source tree to the backend. I wrote Lamar a
note a while back regarding this and he said to check the
dependencies on spi.h.  Well, a 'make depend' for spi.c
yielded 86 headers, so if that's any indication, it would
mean a substantial number of additional headers would be
required for properly allowing users with binary
distributions to write SPI code.  Quite frankly, I don't
know why spi.h is even being shipped with various packages
at the moment.  As a result, no one can use the contrib code
without the backend source, nor can one write 'C' functions
that access tables.  So I was wondering if this is something
that might ever be addressed? Of course, the whole issue
disappears if there is ever a PostgreSQL equivalent of a
'setuid' attribute and grant/revoke privileges for
functions....

Just some thoughts...

Mike Mascari



Fri, 17 Dec 1999 20:32:02 -0500
Mike Mascari <mascarm@mascari.com>
SPI header dependencies
Mike Mascari <mascarm@mascari.com> writes:
> On the general list a while back someone wanted to implement
> a view where the records visible to the client were
> dependent upon an application defined user id. I, too, am in
> a similar circumstance. However, one solution would be to do
> something like this:
> SELECT authenticate(<userid>, <password>);
> where <userid> and <password> are submitted by the client
> application as input from the user.

This seems like a completely redundant mechanism to me.
What is wrong with using the *existing* user authentication
mechanisms, and then using getpgusername() or CURRENT_USER
in your queries?

> At any rate. the whole point of this deal is that anyone
> wanting to write a 'C' function which needs to access tuples
> has to use the SPI interface. And at the moment, that means
> they need the source tree to the backend.

I can't really see anyone writing SPI functions without access to
a source tree; there's too much stuff that's most readily learned
by looking at the code.  But I think you are right that the installed
include tree is probably insufficient to *compile* the average SPI
function, and that's not good.  We haven't been paying much attention
to the question of which headers need to be installed.  There are
probably some installed that needn't be anymore, as well as vice versa.

Proposed TODO:
* Re-examine list of header files that get installed, add/delete as needed

			regards, tom lane

Sat, 18 Dec 1999 12:20:50 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] SPI header dependencies 
Tom Lane wrote:
  >Proposed TODO:
  >* Re-examine list of header files that get installed, add/delete as needed

I attach the list of extra include files that I needed to include in Debian's
postgresql-dev package

access/funcindex.h
access/heapam.h
access/htup.h
access/ibit.h
access/itup.h
access/relscan.h
access/sdir.h
access/skey.h
access/strat.h
access/transam.h
access/tupdesc.h
access/tupmacs.h
access/xact.h
catalog/catname.h
catalog/pg_am.h
catalog/pg_attribute.h
catalog/pg_class.h
catalog/pg_index.h
catalog/pg_language.h
catalog/pg_proc.h
catalog/pg_type.h
executor/execdefs.h
executor/execdesc.h
executor/executor.h
executor/hashjoin.h
executor/tuptable.h
lib/fstack.h
nodes/execnodes.h
nodes/memnodes.h
nodes/nodes.h
nodes/params.h
nodes/parsenodes.h
nodes/pg_list.h
nodes/plannodes.h
nodes/primnodes.h
nodes/relation.h
parser/parse_node.h
parser/parse_type.h
rewrite/prs2lock.h
storage/block.h
storage/buf.h
storage/buf_internals.h
storage/bufmgr.h
storage/bufpage.h
storage/fd.h
storage/ipc.h
storage/item.h
storage/itemid.h
storage/itemptr.h
storage/lmgr.h
storage/lock.h
storage/off.h
storage/page.h
storage/shmem.h
storage/sinvaladt.h
storage/spin.h
tcop/dest.h
tcop/pquery.h
tcop/tcopprot.h
tcop/utility.h
utils/array.h
utils/builtins.h
utils/datetime.h
utils/datum.h
utils/dt.h
utils/fcache.h
utils/hsearch.h
utils/inet.h
utils/int8.h
utils/mcxt.h
utils/memutils.h
utils/nabstime.h
utils/numeric.h
utils/portal.h
utils/rel.h
utils/syscache.h
utils/tqual.h

-- 
      Vote against SPAM: http://www.politik-digital.de/spam/
                 ========================================
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "In the beginning was the Word, and the Word was with 
      God, and the Word was God. The same was in the  
      beginning with God. All things were made by him; and 
      without him was not any thing made that was made."    
                                John 1:1-3 



Sat, 18 Dec 1999 18:31:05 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] SPI header dependencies 
Tom Lane wrote:
> 
> Mike Mascari <mascarm@mascari.com> writes:
> > SELECT authenticate(<userid>, <password>);
> > where <userid> and <password> are submitted by the client
> > application as input from the user.
> 
> This seems like a completely redundant mechanism to me.
> What is wrong with using the *existing* user authentication
> mechanisms, and then using getpgusername() or CURRENT_USER
> in your queries?

I agree. I imagine the poster's development probably took
the same course as mine - first he was using PostgreSQL as a
backend to a web server, like Apache. He then probably using
Basic authentication with something like mod_auth_pgsql. In
order to authenticate web pages using something like
mod_auth_pgsql, the httpd user (www, nobody, etc.) would
connect to the database and check the user name and
encrypted password submitted against a user-specified table.
Since the only application that is going to be connecting to
PostgreSQL is the webserver, one is tempted (including me)
to create and manage fake webuser id's and passwords, and
only have a single real PostgreSQL user id connect to the
database...particularly when the webuser list numbers in the
thousands. That's why I attributed the LRU file descriptor
exhaustion problem I reported about a month ago to kernel
problems instead of the password authentication leak - 90%
of our users use the web-server. The httpd process runs as a
user id which does not have a shell account, and can only
connect to the database on localhost. This whole scheme
looks good at first, until you find yourself developing
Windows-based clients...You either have to shoe-horn in a
hack (like the above) or bite the bullet and migrate your
core authentication mechanism to PostgreSQL's.

> Proposed TODO:
> * Re-examine list of header files that get installed, add/delete as needed
> 
>                         regards, tom lane

Sounds great. Although hopefully not needed in the next
release :-) , the most annoying thing in the past was the
inability to build a refint.so from the various binary
distributions...

Mike Mascari

Sat, 18 Dec 1999 18:50:57 -0500
Mike Mascari <mascarm@mascari.com>
Re: [HACKERS] SPI header dependencies
   > Proposed TODO:
   > * Re-examine list of header files that get installed, add/delete as needed

   Sounds great. Although hopefully not needed in the next
   release :-) , the most annoying thing in the past was the
   inability to build a refint.so from the various binary
   distributions...

Absolutely needed!  There is more to SPI than refint.  The foreign
keys cannot remove the need for that interface to be complete as
installed.

Same problem happens when trying to include trigger.h, so those
dependencies need looking at too.

Cheers,
Brook

Sat, 18 Dec 1999 17:07:25 -0700 (MST)
Brook Milligan <brook@biology.nmsu.edu>
Re: [HACKERS] SPI header dependencies
Oliver Elphick wrote:
> 
> Tom Lane wrote:
>   >Proposed TODO:
>   >* Re-examine list of header files that get installed, add/delete as needed
> 
> I attach the list of extra include files that I needed to include in Debian's
> postgresql-dev package
[snip]

I owe you again, Oliver.  This list gives the postgres-dev package
everything needed to do SPI development??  Good thing I haven't done my
planned 6.5.3-3 release yet.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 20 Dec 1999 12:13:18 -0500
Lamar Owen <admin@wgcr.org>
Re: [HACKERS] SPI header dependencies
Oliver Elphick wrote:
> 
> Tom Lane wrote:
>   >Proposed TODO:
>   >* Re-examine list of header files that get installed, add/delete as needed
> 
> I attach the list of extra include files that I needed to include in Debian's
> postgresql-dev package
[snip]

I owe you again, Oliver.  This list gives the postgres-dev package
everything needed to do SPI development??  Good thing I haven't done my
planned 6.5.3-3 release yet.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 20 Dec 1999 13:29:20 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] SPI header dependencies
There appear to have been changes in the shared library libpq.

The default library from 6.5.3 with psql from current tree gives:

  olly@linda$ psql template1
  ...
  psql: error in loading shared libraries: psql: undefined symbol:   
createPQExpBuffer

  olly@linda$ LD_PRELOAD=/usr/local/pgsql/lib/libpq.so.2.0 psql template1
  ...
  template1=>

Since the library has changed, it needs to have a new version number.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Be patient therefore, brethren, unto the coming of the
      Lord...Be patient; strengthen your hearts, for 
      the coming of the Lord draweth nigh."           
                                   James 5:7,8 



Mon, 10 Jan 2000 14:11:30 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Shared library version 
> There appear to have been changes in the shared library libpq.
> 
> The default library from 6.5.3 with psql from current tree gives:
> 
>   olly@linda$ psql template1
>   ...
>   psql: error in loading shared libraries: psql: undefined symbol:   
> createPQExpBuffer
> 
>   olly@linda$ LD_PRELOAD=/usr/local/pgsql/lib/libpq.so.2.0 psql template1
>   ...
>   template1=>
> 
> Since the library has changed, it needs to have a new version number.

Seems I should just kick up every minor version number for 7.0 for all
interfaces.  OK?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 10 Jan 2000 10:06:05 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Shared library version
"Oliver Elphick" <olly@lfix.co.uk> writes:
> There appear to have been changes in the shared library libpq.
> Since the library has changed, it needs to have a new version number.

You're right, we need to bump the number before release (and I hope we
remember!).  Past practice has not been to bump the number during
development cycles, since we'd shortly have ridiculously high version
numbers if we incremented them at every development change.

libpq++ has also had API changes requiring a new version number before
release, I think --- any others?

			regards, tom lane

Mon, 10 Jan 2000 10:22:44 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Shared library version 
On Mon, Jan 10, 2000 at 10:22:44AM -0500, Tom Lane wrote:
> You're right, we need to bump the number before release (and I hope we
> remember!).  Past practice has not been to bump the number during
> development cycles, since we'd shortly have ridiculously high version
> numbers if we incremented them at every development change.

Ehem... I do exactly that with libecpg. For simply changes not involving API
changes I increment just the patch level. That's why libecpg has major
version, minor version and patchlevel. 

I find it very difficult to keep track of the changes without changing
version number.

So please, do not change this number upon release.

Michael
-- 
Michael Meskes                         | Go SF 49ers!
Th.-Heuss-Str. 61, D-41812 Erkelenz    | Go Rhein Fire!
Tel.: (+49) 2431/72651                 | Use Debian GNU/Linux!
Email: Michael@Fam-Meskes.De           | Use PostgreSQL!

Mon, 10 Jan 2000 21:18:17 +0100
Michael Meskes <meskes@postgresql.org>
Re: [HACKERS] Shared library version
Tom Lane wrote:
  >"Oliver Elphick" <olly@lfix.co.uk> writes:
  >> There appear to have been changes in the shared library libpq.
  >> Since the library has changed, it needs to have a new version number.
  >
  >You're right, we need to bump the number before release (and I hope we
  >remember!).  Past practice has not been to bump the number during
  >development cycles, since we'd shortly have ridiculously high version
  >numbers if we incremented them at every development change.

Yes, but it should be bumped the first time it changes; I agree that it 
need not be increased during later development of the same release.

A patch would be superfluous.  The necessary change is simply to
increment SO_MINOR_VERSION in src/interfaces/libpq/Makefile

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Enter into his gates with thanksgiving, and into his 
      courts with praise. Be thankful unto him, and bless 
      his name."           Psalms 100:4 



Tue, 11 Jan 2000 09:40:56 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Shared library version 
On 2000-01-10, Tom Lane mentioned:

> "Oliver Elphick" <olly@lfix.co.uk> writes:
> > There appear to have been changes in the shared library libpq.
> > Since the library has changed, it needs to have a new version number.
> 
> You're right, we need to bump the number before release (and I hope we
> remember!).  Past practice has not been to bump the number during
> development cycles, since we'd shortly have ridiculously high version
> numbers if we incremented them at every development change.
> 
> libpq++ has also had API changes requiring a new version number before
> release, I think --- any others?

It would at least be fair to bump the minor version number when you do the
branch for a new version, so now we'd be at 2.1. IIRC the dynamic linker
will pick the one with the higher minor version. Since we do not have any
incompatible changes (?) we shouldn't bump the major version.

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Tue, 11 Jan 2000 14:26:45 +0100
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Shared library version 
On Tue, 11 Jan 2000, Peter Eisentraut wrote:

> On 2000-01-10, Tom Lane mentioned:
> 
> > "Oliver Elphick" <olly@lfix.co.uk> writes:
> > > There appear to have been changes in the shared library libpq.
> > > Since the library has changed, it needs to have a new version number.
> > 
> > You're right, we need to bump the number before release (and I hope we
> > remember!).  Past practice has not been to bump the number during
> > development cycles, since we'd shortly have ridiculously high version
> > numbers if we incremented them at every development change.
> > 
> > libpq++ has also had API changes requiring a new version number before
> > release, I think --- any others?
> 
> It would at least be fair to bump the minor version number when you do the
> branch for a new version, so now we'd be at 2.1. IIRC the dynamic linker
> will pick the one with the higher minor version. Since we do not have any
> incompatible changes (?) we shouldn't bump the major version.

libpq++ got a major number bump on my first sweep.  This time through 
it should only need a minor since I don't forsee any operational changes,
just additional functionality and bug fixes.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
   128K ISDN: $24.95/mo or less - 56K Dialup: $17.95/mo or less at Pop4
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Tue, 11 Jan 2000 09:22:35 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [HACKERS] Shared library version 
Using the cvs version updated this morning, this query kills the backend,
with no explanation in the log (-d 3):

  create table junk (id char(4) primary key, name text not null)

If the primary key constraint is omitted, it is OK.

This worked yesterday.  Is this a solved problem, or do I need to trace it?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Enter into his gates with thanksgiving, and into his 
      courts with praise. Be thankful unto him, and bless 
      his name."           Psalms 100:4 



Tue, 11 Jan 2000 22:29:28 +0000
"Oliver Elphick" <olly@lfix.co.uk>
CREATE TABLE ... PRIMARY KEY kills backend
> Using the cvs version updated this morning, this query kills the backend,
> with no explanation in the log (-d 3):
> 
>   create table junk (id char(4) primary key, name text not null)
> 
> If the primary key constraint is omitted, it is OK.
> 
> This worked yesterday.  Is this a solved problem, or do I need to trace it?
> 

Works for me on current sources:

test=>   create table junk (id char(4) primary key, name text not null);
NOTICE:  CREATE TABLE/PRIMARY KEY will create implicit index 'junk_pkey'
for table 'junk'
CREATE

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 11 Jan 2000 19:01:17 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] CREATE TABLE ... PRIMARY KEY kills backend
"Oliver Elphick" <olly@lfix.co.uk> writes:
> Using the cvs version updated this morning, this query kills the backend,
> with no explanation in the log (-d 3):
>
>   create table junk (id char(4) primary key, name text not null)

Works for me:

regression=# create table junk (id char(4) primary key, name text not null);
NOTICE:  CREATE TABLE/PRIMARY KEY will create implicit index 'junk_pkey' for table 'junk'
CREATE

Are you sure you have a complete update of the INDEX_MAX_KEYS changes?
I committed the last of them about 1am EST (6am GMT) this morning, and
it was a change to config.h.in ---- you would need to do a *full*
configure, build, initdb cycle to be sure you have working code.

If that doesn't do it for you, there may be a platform-dependent bug
still lurking; can you provide a debugger backtrace of the crashed
backend?

I'd also suggest running the regress tests ... they pass here, with
the exception of the 'array' test ...

			regards, tom lane

Tue, 11 Jan 2000 19:09:00 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] CREATE TABLE ... PRIMARY KEY kills backend 
Tom Lane wrote:
  >"Oliver Elphick" <olly@lfix.co.uk> writes:
  >> Using the cvs version updated this morning, this query kills the backend,
  >> with no explanation in the log (-d 3):
  >>
  >>   create table junk (id char(4) primary key, name text not null)
  >
  >Works for me:
  >
  >regression=# create table junk (id char(4) primary key, name text not null);
  >NOTICE:  CREATE TABLE/PRIMARY KEY will create implicit index 'junk_pkey' for
      > table 'junk'
  >CREATE
  >
  >Are you sure you have a complete update of the INDEX_MAX_KEYS changes?
  >I committed the last of them about 1am EST (6am GMT) this morning, and
  >it was a change to config.h.in ---- you would need to do a *full*
  >configure, build, initdb cycle to be sure you have working code.
  
./configure --with-tcl --with-mb=UNICODE --with-odbc --enable-locale 
--with-maxbackends=64 --with-pgport=5431 --program-prefix=pg7.

(The program-prefix doesn't seem to do anything.)

Database destoyed and initdb run...

  >If that doesn't do it for you, there may be a platform-dependent bug
  >still lurking; can you provide a debugger backtrace of the crashed
  >backend?

Segmentation fault (in the end):
#0  0x400f068d in _IO_default_xsputn () from /lib/libc.so.6
#1  0x400e0126 in vfprintf () from /lib/libc.so.6
#2  0x400edf23 in vsnprintf () from /lib/libc.so.6
#3  0x80a8e82 in appendStringInfo ()
#4  0x80c244d in _outTypeName ()
#5  0x80c43da in _outNode ()
#6  0x80c2391 in _outColumnDef ()
... 
#157128 0x80c23f6 in _outColumnDef ()
#157129 0x80c43ca in _outNode ()
#157130 0x80c407c in _outNode ()
#157131 0x80c3f7a in _outConstraint ()
#157132 0x80c475a in _outNode ()
#157133 0x80c407c in _outNode ()
#157134 0x80c23f6 in _outColumnDef ()
#157135 0x80c43ca in _outNode ()
#157136 0x80c407c in _outNode ()
#157137 0x80c219c in _outCreateStmt ()
#157138 0x80c43aa in _outNode ()
#157139 0x80c2578 in _outQuery ()
#157140 0x80c43fa in _outNode ()
#157141 0x80c47fd in nodeToString ()
#157142 0x80ed791 in pg_parse_and_plan ()
#157143 0x80eda46 in pg_exec_query_dest ()
#157144 0x80eda01 in pg_exec_query ()
#157145 0x80eeb82 in PostgresMain ()
#157146 0x80d7ee7 in DoBackend ()
#157147 0x80d7abe in BackendStartup ()
#157148 0x80d6cc9 in ServerLoop ()
#157149 0x80d66ae in PostmasterMain ()
#157150 0x80ae2cb in main ()
#157151 0x400bc7e2 in __libc_start_main () from /lib/libc.so.6

I don't have any line number info, so I'll have to rebuild in order to
do more detailed tracing.

  >I'd also suggest running the regress tests ... they pass here, with
  >the exception of the 'array' test ...

The regression test doesn't seem to work at all with multibyte enabled:

=============== creating new regression database...   =================
createdb: "UNICODE" is not a valid encoding name.
createdb failed
ACTUAL RESULTS OF REGRESSION TEST ARE NOW IN FILE regress.out

The reason is that regress.sh uses createdb; createdb has a bad test for
the encoding value (I have sent a patch separately).  So I assume no-one
has run regression tests with multibyte-encoding enabled?

I fixed the createdb bug and ran the regression test.  The constraints
test failed when trying to create a table with a primary key.  Every
test thereafter failed immediately [pqReadData() -- backend closed the
channel unexpectedly]; it appears that the primary key error messes up
the postmaster in some way.


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "For the LORD is good; his mercy is everlasting; and 
      his truth endureth to all generations."         
                                     Psalms 100:5 



Wed, 12 Jan 2000 10:31:43 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] CREATE TABLE ... PRIMARY KEY kills backend 
"Oliver Elphick" wrote:
  >I don't have any line number info, so I'll have to rebuild in order to
  >do more detailed tracing.

QUERY: create table oljunk (id char(2) primary key, name text);

(gdb) n
134             _outNode(str, node->raw_default);
(gdb) p *str
$25 = {
  data = 0x81dd788 "{ QUERY :command 5  :create oljunk { CREATE :relname oljunk  :istemp false \t:columns ({ COLUMNDEF :colname id :typename { TYPENAME :name bpchar :timezone false :setof false typmod 6 :arrayBounds  :arr"..., len = 263, 
  maxlen = 512}
(gdb) n
135             appendStringInfo(str, " :cooked_default %s :constraints ",
(gdb) n
137             _outNode(str, node->constraints);
(gdb) n

Program received signal SIGSEGV, Segmentation fault.
0x400f068a in _IO_default_xsputn () from /lib/libc.so.6

This is the backtrace before calling _outNode() at 137:
#0  _outColumnDef (str=0xbfffe8cc, node=0x81dd610) at outfuncs.c:137
        str = 0xbfffe8cc
        node = (ColumnDef *) 0x81dd610
#1  0x80c43da in _outNode (str=0xbfffe8cc, obj=0x81dd610) at outfuncs.c:1355
        str = 0xbfffe8cc
        obj = (void *) 0x81dd610
#2  0x80c408c in _outNode (str=0xbfffe8cc, obj=0x81dd970) at outfuncs.c:1336
        l = (List *) 0x81dd970
        str = 0xbfffe8cc
        obj = (void *) 0x81dd970
#3  0x80c21ac in _outCreateStmt (str=0xbfffe8cc, node=0x81dd7b8)
    at outfuncs.c:74
        str = 0xbfffe8cc
        node = (CreateStmt *) 0x81dd7b8
#4  0x80c43ba in _outNode (str=0xbfffe8cc, obj=0x81dd7b8) at outfuncs.c:1348
        str = 0xbfffe8cc
        obj = (void *) 0x81dd7b8
#5  0x80c2588 in _outQuery (str=0xbfffe8cc, node=0x81dd8e8) at outfuncs.c:185
        str = 0xbfffe8cc
        node = (Query *) 0x81dd8e8
#6  0x80c440a in _outNode (str=0xbfffe8cc, obj=0x81dd8e8) at outfuncs.c:1364
        str = 0xbfffe8cc
        obj = (void *) 0x81dd8e8
#7  0x80c480d in nodeToString (obj=0x81dd8e8) at outfuncs.c:1570
        obj = (void *) 0x81dd8e8
        str = {
  data = 0x81ddc20 "{ QUERY :command 5  :create oljunk { CREATE :relname oljunk  :istemp false \t:columns ({ COLUMNDEF :colname id :typename { TYPENAME :name bpchar :timezone false :setof false typmod 6 :arrayBounds  :arr"..., len = 298, 
  maxlen = 512}
#8  0x80ed7a1 in pg_parse_and_plan (
    query_string=0x8184da0 "create table oljunk (id char(2) primary key, name text)", typev=0x0, nargs=0, queryListP=0xbfffe97c, dest=Remote, 
    aclOverride=0 '\000') at postgres.c:435
        query_string = 0x81dd8e8 "X\002"
        aclOverride = 0 '\000'
        querytree_list = (List *) 0x81dd8e8
        plan_list = (List *) 0x0
        querytree_list_item = (List *) 0x81dda60
        querytree = (Query *) 0x81dd8e8
        plan = (Plan *) 0x81dd8e8
        new_list = (List *) 0x0
        rewritten = (List *) 0xf5


Now we run on a bit, and we go into a recursive loop inside _outNode:

_outNode (str=0xbfffe8cc, obj=0x81dd9a0) at outfuncs.c:1323
1323            if (obj == NULL)
(gdb) p *str
$21 = {
  data = 0x81dde28 "{ QUERY :command 5  :create oljunk { CREATE :relname oljunk  :istemp false \t:columns ({ COLUMNDEF :colname id :typename { TYPENAME :name bpchar :timezone false :setof false typmod 6 :arrayBounds  :arrayBounds <>} :is_not_null true :is_sequence false :raw_default <> :cooked_default <> :constraints ({ oljunk_pkey :type PRIMARY KEY ({ COLUMNDEF :colname id :typename { TYPENAME :name bpchar :timezone false :setof false typmod 6 :arrayBounds  :arrayBounds <>} :is_not_null true :is_sequence false :raw_default <> :cooked_default <> :constraints ({ oljunk_pkey :type PRIMARY KEY ({ COLUMNDEF :colname id :typename { TYPENAME :name bpchar :timezone false :setof false typmod 6 :arrayBounds  :arrayBounds <>} :is_not_null true :is_sequence false :raw_default <> :cooked_default <> :constraints ({ oljunk_pkey :type PRIMARY KEY ", len = 823, maxlen = 1024}
(gdb) 

(gdb) bt
#0  _outNode (str=0xbfffe8cc, obj=0x81dd9a0) at outfuncs.c:1323
#1  0x80c3f8a in _outConstraint (str=0xbfffe8cc, node=0x81dd5e8)
    at outfuncs.c:1283
#2  0x80c476a in _outNode (str=0xbfffe8cc, obj=0x81dd5e8) at outfuncs.c:1528
#3  0x80c408c in _outNode (str=0xbfffe8cc, obj=0x81dd660) at outfuncs.c:1336
#4  0x80c2406 in _outColumnDef (str=0xbfffe8cc, node=0x81dd610)
    at outfuncs.c:137
#5  0x80c43da in _outNode (str=0xbfffe8cc, obj=0x81dd610) at outfuncs.c:1355
#6  0x80c408c in _outNode (str=0xbfffe8cc, obj=0x81dd9a0) at outfuncs.c:1336
#7  0x80c3f8a in _outConstraint (str=0xbfffe8cc, node=0x81dd5e8)
    at outfuncs.c:1283
#8  0x80c476a in _outNode (str=0xbfffe8cc, obj=0x81dd5e8) at outfuncs.c:1528
#9  0x80c408c in _outNode (str=0xbfffe8cc, obj=0x81dd660) at outfuncs.c:1336
#10 0x80c2406 in _outColumnDef (str=0xbfffe8cc, node=0x81dd610)
    at outfuncs.c:137
#11 0x80c43da in _outNode (str=0xbfffe8cc, obj=0x81dd610) at outfuncs.c:1355
#12 0x80c408c in _outNode (str=0xbfffe8cc, obj=0x81dd9a0) at outfuncs.c:1336
#13 0x80c3f8a in _outConstraint (str=0xbfffe8cc, node=0x81dd5e8)
    at outfuncs.c:1283
#14 0x80c476a in _outNode (str=0xbfffe8cc, obj=0x81dd5e8) at outfuncs.c:1528
#15 0x80c408c in _outNode (str=0xbfffe8cc, obj=0x81dd660) at outfuncs.c:1336
#16 0x80c2406 in _outColumnDef (str=0xbfffe8cc, node=0x81dd610)
    at outfuncs.c:137
#17 0x80c43da in _outNode (str=0xbfffe8cc, obj=0x81dd610) at outfuncs.c:1355
#18 0x80c408c in _outNode (str=0xbfffe8cc, obj=0x81dd970) at outfuncs.c:1336
#19 0x80c21ac in _outCreateStmt (str=0xbfffe8cc, node=0x81dd7b8)
    at outfuncs.c:74
#20 0x80c43ba in _outNode (str=0xbfffe8cc, obj=0x81dd7b8) at outfuncs.c:1348
#21 0x80c2588 in _outQuery (str=0xbfffe8cc, node=0x81dd8e8) at outfuncs.c:185
#22 0x80c440a in _outNode (str=0xbfffe8cc, obj=0x81dd8e8) at outfuncs.c:1364
#23 0x80c480d in nodeToString (obj=0x81dd8e8) at outfuncs.c:1570
#24 0x80ed7a1 in pg_parse_and_plan (
    query_string=0x8184da0 "create table oljunk (id char(2) primary key, name text)", typev=0x0, nargs=0, queryListP=0xbfffe97c, dest=Remote, 
    aclOverride=0 '\000') at postgres.c:435


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "For the LORD is good; his mercy is everlasting; and 
      his truth endureth to all generations."         
                                     Psalms 100:5 



Wed, 12 Jan 2000 12:24:14 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] CREATE TABLE ... PRIMARY KEY kills backend 
> ./configure --with-tcl --with-mb=UNICODE --with-odbc --enable-locale 
> --with-maxbackends=64 --with-pgport=5431 --program-prefix=pg7.

I didn't see your problem here. My configuratuion is:

	./configure --with-mb=EUC_JP

> The reason is that regress.sh uses createdb; createdb has a bad test for
> the encoding value (I have sent a patch separately).  So I assume no-one
> has run regression tests with multibyte-encoding enabled?

Thank you for the fix. I have just committed your changes.

I already have fixed that too with my private working files, but
forgot to commit.
--
Tatsuo Ishii

Wed, 12 Jan 2000 22:14:01 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] CREATE TABLE ... PRIMARY KEY kills backend 
"Oliver Elphick" <olly@lfix.co.uk> writes:
> Segmentation fault (in the end):
> #0  0x400f068d in _IO_default_xsputn () from /lib/libc.so.6
> #1  0x400e0126 in vfprintf () from /lib/libc.so.6
> #2  0x400edf23 in vsnprintf () from /lib/libc.so.6
> #3  0x80a8e82 in appendStringInfo ()
> #4  0x80c244d in _outTypeName ()
> #5  0x80c43da in _outNode ()
> #6  0x80c2391 in _outColumnDef ()
> ... 
> #157128 0x80c23f6 in _outColumnDef ()
> #157129 0x80c43ca in _outNode ()
> #157130 0x80c407c in _outNode ()
   ^^^^^^

Hmm, I take it this is a stack-growth-limit-exceeded failure, although
your system isn't reporting it that way.

This is a known bug that's been there for quite a while: PRIMARY KEY
generates a parse tree with circular references, so if you have parse
tree dumping turned on, you get an infinite recursion in the dumper
code.  It needs to be fixed, but hasn't gotten to the top of anyone's
to-do list (and a clean way to fix it isn't obvious).

> I fixed the createdb bug and ran the regression test.  The constraints
> test failed when trying to create a table with a primary key.

If you had -d set high enough, it would...

> Every
> test thereafter failed immediately [pqReadData() -- backend closed the
> channel unexpectedly]; it appears that the primary key error messes up
> the postmaster in some way.

The WAL postmaster takes a few seconds to recover before it will allow
new connections (I have a proposal on the table that it should just
delay accepting the connections, instead of rejecting 'em, but there
hasn't been any discussion about that).  I usually see that the next
three or four regression tests fail after a crash, but if your machine
is fast enough it might be that they all do.

I have noticed that recent versions of libpq fail to display the
connection-refused error message that I assume the postmaster is
returning.  That useta work ... someone broke it ...

			regards, tom lane

Wed, 12 Jan 2000 10:31:29 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] CREATE TABLE ... PRIMARY KEY kills backend 
[Version: CVS as of yesterday]
When I create a table that inherits from another table that uses foreign
keys, I get something like this:

  ERROR:  cache lookup of attribute 10 in relation 124171 failed

This is happening in get_attribute_name() of backend/utils/adt/ruleutils.c


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "For the LORD is good; his mercy is everlasting; and 
      his truth endureth to all generations."         
                                     Psalms 100:5 



Thu, 13 Jan 2000 22:14:39 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Problem with foreign keys and inheritance
"Oliver Elphick" wrote:
  >When I create a table that inherits from another table that uses foreign
  >keys, I get something like this:
  >
  >  ERROR:  cache lookup of attribute 10 in relation 124171 failed
  >
  >This is happening in get_attribute_name() of backend/utils/adt/ruleutils.c

I'm still trying to track this down; it seems to be happening when the
backend is trying to fetch details of the ancestor class, in
deparse_expression().

However, I cannot find relation 124171; is there any way to find out
where a relation is, given only its oid?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "For I know that my redeemer liveth, and that he shall 
      stand at the latter day upon the earth"             
                                       Job 19:25 



Sat, 15 Jan 2000 21:44:42 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Problem with foreign keys and inheritance 
Oliver Elphick wrote:
> 
> However, I cannot find relation 124171; is there any way to find out
> where a relation is, given only its oid?

This might give you a pretty good hint...

	select * from pg_attribute where attrelid = 124171;

Cheers,
Ed Loehr

Sat, 15 Jan 2000 17:26:27 -0600
Ed Loehr <eloehr@austin.rr.com>
Re: [HACKERS] Problem with foreign keys and inheritance
Ed Loehr wrote:
  >Oliver Elphick wrote:
  >> 
  >> However, I cannot find relation 124171; is there any way to find out
  >> where a relation is, given only its oid?
  >
  >This might give you a pretty good hint...
  >
  >	select * from pg_attribute where attrelid = 124171;

Nothing.
 
I tried looking for the oid in every system table listed by \dS - no joy :-(

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "For I know that my redeemer liveth, and that he shall 
      stand at the latter day upon the earth"             
                                       Job 19:25 



Sat, 15 Jan 2000 23:45:59 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Problem with foreign keys and inheritance 
"Oliver Elphick" <olly@lfix.co.uk> writes:
>>> However, I cannot find relation 124171; is there any way to find out
>>> where a relation is, given only its oid?
>> 
>> This might give you a pretty good hint...
>> 
>> select * from pg_attribute where attrelid = 124171;

Actually, "select * from pg_class where oid = 124171" is the canonical
answer; if that doesn't produce anything, you have no such table.
 
> I tried looking for the oid in every system table listed by \dS - no joy :-(

Is it possible that you dropped the table in question since that try?
If you recreated it, it wouldn't have the same OID the second time.

Another possibility is that the rule dumper is picking up a completely
wrong number for some reason.  I thought that code was pretty solid by
now, but it might still have some glitches left.

If you provided an SQL script that reproduces the problem, more people
might be motivated to look for it...

			regards, tom lane

Sat, 15 Jan 2000 20:31:56 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Problem with foreign keys and inheritance 
"Oliver Elphick" wrote:
  >[Version: CVS as of yesterday]
  >When I create a table that inherits from another table that uses foreign
  >keys, I get something like this:
  >
  >  ERROR:  cache lookup of attribute 10 in relation 124171 failed
  >
  >This is happening in get_attribute_name() of backend/utils/adt/ruleutils.c

Here is an SQL script that makes this happen:

========================================================
create database newj with encoding = 'SQL_ASCII';
\connect newj
create table person
(
        id              char(10)        primary key,
        name            text            not null,
        address         int,
        salutation      text            default 'Dear Sir',
        envelope        text,
        email           text,
        www             text
);

create table individual
(
        gender          char(1)         check (gender = 'M' or gender = 'F'
                                                 or gender is null),
        born            datetime        check ((born >= '1 Jan 1880'
                                           and born <= 'today') or born is null),
        surname         text,
        forenames       text,
        title           text,
        old_surname     text,
        mobile          text,
        ni_no           text,
        constraint is_named check (not (surname isnull and forenames isnull))
)
        inherits (person);

create table organisation
(
        contact         char(10)        references individual (id) match full,
        structure       char(1)         check (structure='L' or structure='C'
                                              or structure='U' or structure='O')
)
        inherits (person);

create table customer
(
        acs_code        char(8),
        acs_addr        int,        class           char(1)         default '',
        type            char(2),
        area            char(2),
        country         char(2),
        vat_class       char(1),
        vat_number      char(12),
        discount        numeric(6,3)      check (discount >= -50.0::numeric(6,3)
                                           and discount <= 50.0)::numeric(6,3),
        commission      bool            default 'f',
        status          char(1)         default '',
        deliver_to      int,
        factor_code     text
)
        inherits (organisation);
========================================================

Table customer does not get created; instead, I get:

  ERROR:  cache lookup of attribute 10 in relation <some_oid> failed

-- 



Sun, 16 Jan 2000 15:10:34 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Problem with foreign keys and inheritance 
"Oliver Elphick" <olly@lfix.co.uk> writes:
>> [Version: CVS as of yesterday]
>> When I create a table that inherits from another table that uses foreign
>> keys, I get something like this:
>> 
>> ERROR:  cache lookup of attribute 10 in relation 124171 failed

Ah, I see it.  It's got nothing to do with foreign keys, just inherited
constraints.  We're trying to deparse the inherited constraint
expressions at a time that the relation-in-process-of-being-created
isn't yet officially visible.  So trying to look up its attributes is
failing.  Need another CommandCounterIncrement() in there to make it
work.

This must have been busted for a good while, I think.  I rewrote that
module months ago and probably broke it then.  Probably should add
a regress test case that uses inherited constraints...

			regards, tom lane

Sun, 16 Jan 2000 13:10:41 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Problem with foreign keys and inheritance 
[Using current cvs] I have a problem with COPY when called like this:

  psql -e bray </tmp/ol

with one particular file, containing commands and data like this:

========================================
copy address from stdin;
1  Some place, Regal Way     Somewhere Oxon AB1 3CF  [ Tel: 01367 888888 ]   No
 Deliveries Pm Fridays        GB \N \N \N \N \N \N \N \N \N \N \N \N
...
1000     73 Some Road      London      SW1 1ZZ  GB 44 81 999   9999     \N
   \N \N \N \N \N \N \N
\.
-- 1000 records written

select count(*) from address;

copy address from stdin;
1001...
... and so on up to 3916 records in total, divided into 1000 record chunks
========================================

psql or libpq seems to choke on the data, so that some spurious error
arises, such as null input into a non-null field.  Thereafter, libpq
seems to get stuck in a COPY state:

copy address from stdin;
-- 1000 records written
select count(*) from address;
PQexec: you gotta get out of a COPY state yourself.

(The comment in PQexec says it does this to preserve backwards
compatibility, but getting stuck in COPY state is not backwards
compatible!)

If I remove the SQL commands from the input file, go into psql and do:

  copy address from '/tmp/ol';

all 3916 records are added correctly.  This seems to indicate that
the problem is not in the backend.

I found that if I broke the first 1000 records into 2 equal parts, all
of them were added correctly without error; so I conclude that data
is being buffered and lost somewhere in psql or libpq, and the problem is
dependent on the amount of data being copied.

This began to happen within the last week, but I don't know which
recent change is responsible.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Neither is there salvation in any other; for there is 
      none other name under heaven given among men, whereby 
      we must be saved."               Acts 4:12 



Thu, 20 Jan 2000 15:21:24 +0000
"Oliver Elphick" <olly@lfix.co.uk>
COPY problems with psql / libpq
"Oliver Elphick" <olly@lfix.co.uk> writes:
> I found that if I broke the first 1000 records into 2 equal parts, all
> of them were added correctly without error; so I conclude that data
> is being buffered and lost somewhere in psql or libpq, and the problem is
> dependent on the amount of data being copied.

I have the following note in my (much too long) to-do list:

: psql.c doesn't appear to cope correctly with quoted newlines in COPY data;
: if one falls just after a buffer boundary, trouble!
: Does fe-exec.c work either??

(This note is some months old, and may or may not still apply since
Peter's rework of psql.)  It could be that your dataset is hitting this
problem or a similar one.  A buffer-boundary problem would explain why
the error seems to be so dataset-specific.

> copy address from stdin;
> -- 1000 records written
> select count(*) from address;
> PQexec: you gotta get out of a COPY state yourself.

It sure sounds like psql is failing to recognize the trailing \.
of the COPY data.

			regards, tom lane

Thu, 20 Jan 2000 11:02:31 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] COPY problems with psql / libpq 
On Thu, Jan 20, 2000 at 11:02:31AM -0500, Tom Lane wrote:
> 
> It sure sounds like psql is failing to recognize the trailing \.
> of the COPY data.

Precisely what I saw yesterday (cf Subject: pg_dump disaster) - but what
does one do about it?

Cheers,

Patrick

Thu, 20 Jan 2000 16:40:25 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] COPY problems with psql / libpq
* Patrick Welche <prlw1@newn.cam.ac.uk> [000120 09:10] wrote:
> On Thu, Jan 20, 2000 at 11:02:31AM -0500, Tom Lane wrote:
> > 
> > It sure sounds like psql is failing to recognize the trailing \.
> > of the COPY data.
> 
> Precisely what I saw yesterday (cf Subject: pg_dump disaster) - but what
> does one do about it?

Is this with a recent snapshot or 6.5.3 using libpq?  
Either way, you should check the contents of the send buffer, please let
me know if there is data queued in it.  You can include the 'internal'
header for libpq (libpq-int.h?) to get at the send buffer.

-Alfred

Thu, 20 Jan 2000 09:22:16 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: [HACKERS] COPY problems with psql / libpq
On Thu, Jan 20, 2000 at 09:22:16AM -0800, Alfred Perlstein wrote:
> * Patrick Welche <prlw1@newn.cam.ac.uk> [000120 09:10] wrote:
> > On Thu, Jan 20, 2000 at 11:02:31AM -0500, Tom Lane wrote:
> > > 
> > > It sure sounds like psql is failing to recognize the trailing \.
> > > of the COPY data.
> > 
> > Precisely what I saw yesterday (cf Subject: pg_dump disaster) - but what
> > does one do about it?
> 
> Is this with a recent snapshot or 6.5.3 using libpq?  

For me, it's using yesterday's cvs'd source - but I obviously can't speak
for Oliver.

> Either way, you should check the contents of the send buffer, please let
> me know if there is data queued in it.  You can include the 'internal'
> header for libpq (libpq-int.h?) to get at the send buffer.

That will take a while. In the meantime, just pg_dumpall something and try
to read the output back in. I do have ^M's in some of the text columns if
that matters.

Cheers,

Patrick

Thu, 20 Jan 2000 17:26:59 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] COPY problems with psql / libpq
Patrick Welche wrote:
  >On Thu, Jan 20, 2000 at 09:22:16AM -0800, Alfred Perlstein wrote:
  >> * Patrick Welche <prlw1@newn.cam.ac.uk> [000120 09:10] wrote:
  >> > On Thu, Jan 20, 2000 at 11:02:31AM -0500, Tom Lane wrote:
  >> > > 
  >> > > It sure sounds like psql is failing to recognize the trailing \.
  >> > > of the COPY data.
  >> > 
  >> > Precisely what I saw yesterday (cf Subject: pg_dump disaster) - but what
  >> > does one do about it?
  >> 
  >> Is this with a recent snapshot or 6.5.3 using libpq?  
  >
  >For me, it's using yesterday's cvs'd source - but I obviously can't speak
  >for Oliver.
  
This morning's.

  >> Either way, you should check the contents of the send buffer, please let
  >> me know if there is data queued in it.  You can include the 'internal'
  >> header for libpq (libpq-int.h?) to get at the send buffer.
  >
  >That will take a while. In the meantime, just pg_dumpall something and try
  >to read the output back in. I do have ^M's in some of the text columns if
  >that matters.

I can't do that, because pg_dump seems to be broken if there are tables
with foreign key constraints.   (See a separate message.)

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Neither is there salvation in any other; for there is 
      none other name under heaven given among men, whereby 
      we must be saved."               Acts 4:12 



Thu, 20 Jan 2000 18:44:51 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] COPY problems with psql / libpq 
Tom Lane wrote:
  >"Oliver Elphick" <olly@lfix.co.uk> writes:
  >> I found that if I broke the first 1000 records into 2 equal parts, all
  >> of them were added correctly without error; so I conclude that data
  >> is being buffered and lost somewhere in psql or libpq, and the problem is
  >> dependent on the amount of data being copied.
  >
  >I have the following note in my (much too long) to-do list:
  >
  >: psql.c doesn't appear to cope correctly with quoted newlines in COPY data;
  >: if one falls just after a buffer boundary, trouble!
  >: Does fe-exec.c work either??
  
New-lines are not the problem in this particular case, since the data
does not contain any.

 
-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Neither is there salvation in any other; for there is 
      none other name under heaven given among men, whereby 
      we must be saved."               Acts 4:12 



Thu, 20 Jan 2000 18:55:40 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] COPY problems with psql / libpq 
On 2000-01-20, Tom Lane mentioned:

> "Oliver Elphick" <olly@lfix.co.uk> writes:
> > I found that if I broke the first 1000 records into 2 equal parts, all
> > of them were added correctly without error; so I conclude that data
> > is being buffered and lost somewhere in psql or libpq, and the problem is
> > dependent on the amount of data being copied.

The buffering is line-based though and the default buffer is 8192. If
your line is longer than that you're in all sorts of other troubles.

> 
> I have the following note in my (much too long) to-do list:
> 
> : psql.c doesn't appear to cope correctly with quoted newlines in COPY data;

What's a quoted newline?
a) "<newline>"
b) "\n"
c) \<newline>

Earlier you also mentioned to me something in general about control
characters messing up COPY. Could you give me some details on that so I
can look into it?

> : if one falls just after a buffer boundary, trouble!
> : Does fe-exec.c work either??
> 
> (This note is some months old, and may or may not still apply since
> Peter's rework of psql.)  It could be that your dataset is hitting this

I haven't touched that code.

> problem or a similar one.  A buffer-boundary problem would explain why
> the error seems to be so dataset-specific.

At first I thunk a PQExpBuffer based solution would be the answer, but
as I said above, if you overflow the buffer, you're in trouble anyway.

> 
> > copy address from stdin;
> > -- 1000 records written
> > select count(*) from address;
> > PQexec: you gotta get out of a COPY state yourself.
> 
> It sure sounds like psql is failing to recognize the trailing \.
> of the COPY data.

The last call in the function psql/copy.c:handleCopyIn is

	return !PQendcopy(conn);

and there is no way it can exit earlier. Also the connection seems to be
good, since that's checked right after it returns. The calls to PQputvalue
are not checked for return values, so problems might get missed there, but
that would in any case still point to a problem elsewhere. Gotta pass the
buck to libpq ...

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden




Thu, 20 Jan 2000 22:54:56 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] COPY problems with psql / libpq 
Peter Eisentraut <peter_e@gmx.net> writes:
>> : psql.c doesn't appear to cope correctly with quoted newlines in COPY data;

> What's a quoted newline?
> a) "<newline>"
> b) "\n"
> c) \<newline>

(c).  That's how a newline appearing in the data is supposed to be
represented.  IIRC, I was worried that if the \ falls at the end of
a bufferload and the newline at the start of the next, psql and/or
libpq would fail to recognize the pattern; if so, they'd probably
think the newline is a record boundary.

Patrick could be falling victim to this, but Oliver sez he has no
newlines in his data, so there's at least one other problem.

> that would in any case still point to a problem elsewhere. Gotta pass the
> buck to libpq ...

Could be.  I think Alfred is on the hook here...

			regards, tom lane

Thu, 20 Jan 2000 17:35:45 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] COPY problems with psql / libpq 
I would like to work on improving implementation of inheritance,
especially with regard to referential integrity.   I suspect there are
a number of issues that may be related and will need to be done together.
In addition, this will be my first attempt to do anything serious in
the PostgreSQL code itself,  so I would like to get some hints as
to what I haven't even thought about!

First, I would like to change the definition of the foreign key
constraints to allow the inheritance star to follow a table name.
This would mean that, for RI purposes, the named table would be
aggregated with its descendants.  So "REFERENCES tbl" would mean that
the foreign key must exist in tbl, but "REFERENCES tbl*" would allow it
to exist either in tbl or in any of tbl's descendants.

Implications: where * is used, dropping a descendant table is OK, so
long as the parent continues to exist.  ON DELETE actions would apply
to all the relations in the table to be dropped; to reduce complexity,
this should be broken down into:
  `DELETE FROM descendant; DROP TABLE descendant'
and the whole should be treated as atomic.  If any one relation could
not be deleted, the whole operation would fail.

Use of ON DELETE or ON UPDATE implies there must be an index on the
referring column, to enable checking or deletion to be done speedily.
This doesn't seem to happen at the moment.  If the reference is to
an inheritance group, it would seem to be appropriate that all the
tables in the group should use the same index.  Similarly, where
a unique or primary key constraint is inherited, it may be desirable
to use a single index to manage the constraint.  The implication of
this would be that there must be a check when a table is dropped
to make sure that a grouped index is not dropped until the last
table in the group is dropped.

Is this feasible, or would it require too many changes elsewhere?

Another item I would like to get fixed is to make sure that all
constraints are inherited when a descendant table is created; this
is a current TODO item.  It will also be necessary to ensure that
added constraints get inherited, when ALTER TABLE ... ADD/DROP
CONSTRAINT gets implemented.

====== Design proposal =======

I think that the implications of inheritance have never been fully
explored and I would like to establish the framework in which future
work that involves inheritance will be done.

It seems to me that declaring a table to inherit from another, and
enabling both to be read together by the table* syntax, together
imply certain things about an inheritance group:

1. All tables in the group must possess all the columns of their
ancestor, and all those columns must be of the same type.

2. Some constraints at least must be shared - primary key is the most
obvious example; I think that _all_ constraints on inherited columns
should be shared.  It is probably not practicable to force table
constraints to be shared upwards.

3. There seems to be no need to enforce similar restrictions on
GRANT.  In fact it is quite likely that different permissions could
apply to different tables in the hierarchy.

4. Dropping a table implies dropping all its descendants.

==============================

I would like to consider the implications of this proposal in the light
of the ALTER TABLE commands that have recently been added.

The grammar for ALTER TABLE allows either `ALTER TABLE table ...' or
`ALTER TABLE table* ...'.  I would like to suggest that an alteration
to a parent table must necessarily involve all its descendants and
that alterations to inherited columns must be done in the appropriate
parent.  So, given this hierarchy of tables:

  t1 (c1 char(2) primary key,
      c2 text)

  t2 (c3 int not null
      c4 timestamp default current_timestamp) inherits (t1)

  t3 (c5 text not null) inherits (t2)

adding a column to t1, means the same column must be added to t2 and t3
and must appear before any columns originating in t2; columns c1 to c4
cannot be dropped from table t3 unless they are also dropped from the
parents. Alterations to c2 must be done in t1, and alterations to c4
must be done in t2.  Any table constraint applied to t1 would automatically
be inherited by t2 and t3, a new constraint added to t2 would be
inherited by t3 but would not affect t1.

Attempts to use ALTER TABLE to bypass these restrictions should be
disallowed.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "If anyone has material possessions and sees his
      brother in need but has no pity on him, how can the
      love of God be in him?"
                                    I John 3:17 



Mon, 24 Jan 2000 21:52:56 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Inheritance, referential integrity and other constraints

As long as you're working on this area you could fix the problem where
if you do ALTER table* ADD COLUMN ... pg_dump no longer works because
the column orders have changed in different inherited tables.

Oliver Elphick wrote:
> 
> I would like to work on improving implementation of inheritance,
> especially with regard to referential integrity.   I suspect there are
> a number of issues that may be related and will need to be done together.
> In addition, this will be my first attempt to do anything serious in
> the PostgreSQL code itself,  so I would like to get some hints as
> to what I haven't even thought about!
> 
> First, I would like to change the definition of the foreign key
> constraints to allow the inheritance star to follow a table name.
> This would mean that, for RI purposes, the named table would be
> aggregated with its descendants.  So "REFERENCES tbl" would mean that
> the foreign key must exist in tbl, but "REFERENCES tbl*" would allow it
> to exist either in tbl or in any of tbl's descendants.
> 
> Implications: where * is used, dropping a descendant table is OK, so
> long as the parent continues to exist.  ON DELETE actions would apply
> to all the relations in the table to be dropped; to reduce complexity,
> this should be broken down into:
>   `DELETE FROM descendant; DROP TABLE descendant'
> and the whole should be treated as atomic.  If any one relation could
> not be deleted, the whole operation would fail.
> 
> Use of ON DELETE or ON UPDATE implies there must be an index on the
> referring column, to enable checking or deletion to be done speedily.
> This doesn't seem to happen at the moment.  If the reference is to
> an inheritance group, it would seem to be appropriate that all the
> tables in the group should use the same index.  Similarly, where
> a unique or primary key constraint is inherited, it may be desirable
> to use a single index to manage the constraint.  The implication of
> this would be that there must be a check when a table is dropped
> to make sure that a grouped index is not dropped until the last
> table in the group is dropped.
> 
> Is this feasible, or would it require too many changes elsewhere?
> 
> Another item I would like to get fixed is to make sure that all
> constraints are inherited when a descendant table is created; this
> is a current TODO item.  It will also be necessary to ensure that
> added constraints get inherited, when ALTER TABLE ... ADD/DROP
> CONSTRAINT gets implemented.
> 
> ====== Design proposal =======
> 
> I think that the implications of inheritance have never been fully
> explored and I would like to establish the framework in which future
> work that involves inheritance will be done.
> 
> It seems to me that declaring a table to inherit from another, and
> enabling both to be read together by the table* syntax, together
> imply certain things about an inheritance group:
> 
> 1. All tables in the group must possess all the columns of their
> ancestor, and all those columns must be of the same type.
> 
> 2. Some constraints at least must be shared - primary key is the most
> obvious example; I think that _all_ constraints on inherited columns
> should be shared.  It is probably not practicable to force table
> constraints to be shared upwards.
> 
> 3. There seems to be no need to enforce similar restrictions on
> GRANT.  In fact it is quite likely that different permissions could
> apply to different tables in the hierarchy.
> 
> 4. Dropping a table implies dropping all its descendants.
> 
> ==============================
> 
> I would like to consider the implications of this proposal in the light
> of the ALTER TABLE commands that have recently been added.
> 
> The grammar for ALTER TABLE allows either `ALTER TABLE table ...' or
> `ALTER TABLE table* ...'.  I would like to suggest that an alteration
> to a parent table must necessarily involve all its descendants and
> that alterations to inherited columns must be done in the appropriate
> parent.  So, given this hierarchy of tables:
> 
>   t1 (c1 char(2) primary key,
>       c2 text)
> 
>   t2 (c3 int not null
>       c4 timestamp default current_timestamp) inherits (t1)
> 
>   t3 (c5 text not null) inherits (t2)
> 
> adding a column to t1, means the same column must be added to t2 and t3
> and must appear before any columns originating in t2; columns c1 to c4
> cannot be dropped from table t3 unless they are also dropped from the
> parents. Alterations to c2 must be done in t1, and alterations to c4
> must be done in t2.  Any table constraint applied to t1 would automatically
> be inherited by t2 and t3, a new constraint added to t2 would be
> inherited by t3 but would not affect t1.
> 
> Attempts to use ALTER TABLE to bypass these restrictions should be
> disallowed.
> 
> --
> Oliver Elphick                                Oliver.Elphick@lfix.co.uk
> Isle of Wight                              http://www.lfix.co.uk/oliver
>                PGP key from public servers; key ID 32B8FAA1
>                  ========================================
>      "If anyone has material possessions and sees his
>       brother in need but has no pity on him, how can the
>       love of God be in him?"
>                                     I John 3:17
> 
> ************

Tue, 25 Jan 2000 10:33:07 +1100
Chris Bitmead <chris@bitmead.com>
Re: [HACKERS] Inheritance,
 referential integrity and other constraints
Chris Bitmead wrote:
  >As long as you're working on this area you could fix the problem where
  >if you do ALTER table* ADD COLUMN ... pg_dump no longer works because
  >the column orders have changed in different inherited tables.

It seems that this might be quite a problem; I would not like to have
to do a physical insert into every row in a huge table.  Would it be
feasible to add a column order attribute to pg_attribute for tables
altered in this way?  A null entry in that would indicate the table was unaltered from its creation.

Perhaps this could be combined with the idea of column hiding: a zero
column number would mean it was hidden.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "My little children, let us not love in word, neither 
      in tongue; but in deed and in truth."          
                                            I John 3:18 



Tue, 25 Jan 2000 08:06:34 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Inheritance, referential integrity and other
	constraints
I took up the issue of

* Alter TABLE ADD COLUMN does not honor DEFAULT, add CONSTRAINT

and chances are excellent that this will get all done in the next day or
three.

First a syntactical issue. We currently allow the following statement,
although it is not legal SQL:

create table test (
	a int4,
	b int4 check (a>b)
);

It's not legal because the column constraint for "b" may only reference
column "b". Instead you could legally write

create table test (
	a int4,
	b int4,
	check (a>b)
);

because the check constraint is now a table constraint. Big deal. Now the
problem is that because I reuse the same syntactical elements, the
following will work:

create table test (a int4);
alter table test add column b int4 check (a>b);

No harm done, but how about:

create table test (a int4, b int4);
alter table test add column c text check (a>b);

I guess this would be sort of equivalent to saying

alter table test add column c text;
alter table test add constraint check (a>b);

So, I guess what I'm saying is whether you want to allow the mentioned
weirdness or not.


Secondly, an internal question. If I use SearchSysCacheTuple() on a
condition with several potential matches, what is the defined behaviour?
Can I call it again to get the next tuple?


Thirdly, about TODO item

* ALTER TABLE ADD COLUMN to inherited table put column in wrong place

Actually, according to what I would expect out of the blue, it puts it
into the *right* place. Even good ol' SQL, although they naturally do not
know about inheritance, seems to agree:

"... the degree of [the table] is increased by 1 and the ordinal position
of that [new] column is equal to the new degree of [the table] ..."
(11.11)

What that says to me is that if you add a column to a table (during create
or alter) then the new column gets placed after all the others. Thus,
we're in compliance without even knowing it. 

Or maybe look at it this way:
create table test1 (a int4);
create table test2 (b int4) inherits (test1);
                    ^ col #1          ^ col #2
alter table test1* add column c int4;
                              ^ col #3

Everything has its order and it's not like the inheritance as such is
broken.

Surely, trying to stick the column in between is going to be three times
as much work as dropping columns would be, whichever way you do it. (And
making attributes invisible ain't gonna help you. ;)

What do you all say?

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Wed, 26 Jan 2000 00:15:35 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Column ADDing issues
Peter Eisentraut <peter_e@gmx.net> writes:
> Or maybe look at it this way:
> create table test1 (a int4);
> create table test2 (b int4) inherits (test1);
>                     ^ col #1          ^ col #2
> alter table test1* add column c int4;
>                               ^ col #3

> Everything has its order and it's not like the inheritance as such is
> broken.

Yes, a whole bunch of stuff is broken after this happens.  Go back and
consult the archives --- or maybe Chris Bitmead will fill you in; he's
got plenty of scars to show for this set of problems.  (All I recall
offhand is that pg_dump and reload can fail to generate a working
database.)  The bottom line is that it would be a lot nicer if column c
had the same column position in both the parent table and the child
table(s).

I suggest you be very cautious about messing with ALTER TABLE until you
understand why inheritance makes it such a headache ;-)

			regards, tom lane

Tue, 25 Jan 2000 19:10:04 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Column ADDing issues 
Peter Eisentraut wrote:
  >Thirdly, about TODO item
  >
  >* ALTER TABLE ADD COLUMN to inherited table put column in wrong place
  >
  >Actually, according to what I would expect out of the blue, it puts it
  >into the *right* place. Even good ol' SQL, although they naturally do not
  >know about inheritance, seems to agree:
  >
  >"... the degree of [the table] is increased by 1 and the ordinal position
  >of that [new] column is equal to the new degree of [the table] ..."
  >(11.11)
  >
  >What that says to me is that if you add a column to a table (during create
  >or alter) then the new column gets placed after all the others. Thus,
  >we're in compliance without even knowing it. 
  >
  >Or maybe look at it this way:
  >create table test1 (a int4);
  >create table test2 (b int4) inherits (test1);
  >                    ^ col #1          ^ col #2
  >alter table test1* add column c int4;
  >                              ^ col #3
  >
  >Everything has its order and it's not like the inheritance as such is
  >broken.
  >
  >Surely, trying to stick the column in between is going to be three times
  >as much work as dropping columns would be, whichever way you do it. (And
  >making attributes invisible ain't gonna help you. ;)

It is:
create table test1 (a int4);
create table test2 (b int4) inherits (test1);
                    ^ col #2          ^ col #1
alter table test1* add column c int4;
                              ^ col #3 but needs to be #2, since it _is_
                                #2 of test1

As far as inheritance goes, all the descendants are treated as one table,
including those created on a different branch from test2.  All of them
have to return the right columns for a single query; the two options for
dealing with this seem to be logical column numbering, or rewriting the
descendant tables.  (But I haven't spent enough time in the code to be
sure of that.)

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "My little children, let us not love in word, neither 
      in tongue; but in deed and in truth."          
                                            I John 3:18 



Wed, 26 Jan 2000 00:45:20 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Column ADDing issues 
On 2000-01-25, Chris Bitmead mentioned:

> As long as you're working on this area you could fix the problem where
> if you do ALTER table* ADD COLUMN ... pg_dump no longer works because
> the column orders have changed in different inherited tables.

This should be fixed in pg_dump then. As I see it, ALTER table* ADD COLUMN
does exactly the right thing.

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Wed, 26 Jan 2000 19:34:29 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Inheritance,
 referential integrity and other constraints
On 2000-01-24, Oliver Elphick mentioned:

> I would like to work on improving implementation of inheritance,
> especially with regard to referential integrity.   I suspect there are
> a number of issues that may be related and will need to be done together.

What I really consider a problem, and it would be great if you could
tackle that, is that there is no real standard that all of this does or
even could follow. For example, I wrote the other day that depending on
which way you see it, the behaviour of alter table x* add colum might be
considered right. Also I just looked into item 'Disallow inherited columns
with the same name as new columns' and it seems that someone actually made
provisions for this to be allowed, meaning that
create table test1 (x int);
create table test2 (x int) inherits (test1);
would result in test2 looking exactly like test1. No one knows what the
motivation was. (I removed it anyway.)

> It will also be necessary to ensure that
> added constraints get inherited, when ALTER TABLE ... ADD/DROP
> CONSTRAINT gets implemented.

I assume the semantics of ADD CONSTRAINT will be exactly the same as of
all the other alter table commands, in that if you specify a star then it
gets inherited, if not then not. But the problem with ADD CONSTRAINT is of
course that the entire table needs to be verified against the constraint
before allowing it to be added. This is fine if you do ADD CONSTRAINT
UNIQUE (a, b), because the index will take care of it, but it's trickier
if you add a trigger based constraint. The former might get into 7.0 if I
hurry, the latter most likely not.

What needs discussion is whether indexes should be shared between
inherited tables, or whether each new descendant table needs a new
one. Not sure if this just made sense, though.


> I think that the implications of inheritance have never been fully
> explored and I would like to establish the framework in which future
> work that involves inheritance will be done.

Precisely.

 
> It seems to me that declaring a table to inherit from another, and
> enabling both to be read together by the table* syntax, together
> imply certain things about an inheritance group:
> 
> 1. All tables in the group must possess all the columns of their
> ancestor, and all those columns must be of the same type.

Isn't it this way now?

> 
> 2. Some constraints at least must be shared - primary key is the most
> obvious example; I think that _all_ constraints on inherited columns
> should be shared.  It is probably not practicable to force table
> constraints to be shared upwards.

Not sure about this one. See the ranting about the shared indexes
above. Might be a great pain.

> 
> 4. Dropping a table implies dropping all its descendants.

Actually what it does now is to refuse dropping when descendants
exist. What seems to be the proper solution to this is to implement the
proper DROP TABLE SQL syntax by adding a RESTRICT/CASCADE at the
end. Restrict refuses dropping if anything (descendants, views,
etc.) references the table, cascade drops everything else as
well. Implementing this could be your first step to glory ;) since it
seems it's more a matter of man hours than conceptual difficulty. Then
again, I could be wrong.


> The grammar for ALTER TABLE allows either `ALTER TABLE table ...' or
> `ALTER TABLE table* ...'.  I would like to suggest that an alteration
> to a parent table must necessarily involve all its descendants and
> that alterations to inherited columns must be done in the appropriate
> parent.  So, given this hierarchy of tables:

It's been a while since I looked into C++, but when you alter a descendant
(such as making a formerly public method private) you surely do not affect
the parents. The other way around I think the choice of star-or-not should
be given to the user. But this is again one of the issues that have no
point of reference, so I'm glad you bring it up for discussion.

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Wed, 26 Jan 2000 19:35:14 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Inheritance,
 referential integrity and other constraints
At 07:35 PM 1/26/00 +0100, Peter Eisentraut wrote:
>On 2000-01-24, Oliver Elphick mentioned:
>
>> I would like to work on improving implementation of inheritance,
>> especially with regard to referential integrity.   I suspect there are
>> a number of issues that may be related and will need to be done together.
>
>What I really consider a problem, and it would be great if you could
>tackle that, is that there is no real standard that all of this does or
>even could follow. For example, I wrote the other day that depending on
>which way you see it, the behaviour of alter table x* add colum might be
>considered right.

Are you basing this on your earlier comment:

"
Or maybe look at it this way:
create table test1 (a int4);
create table test2 (b int4) inherits (test1);
                    ^ col #1          ^ col #2
alter table test1* add column c int4;
                              ^ col #3

"?

If so, I thought Oliver pointed out that you had the numbering wrong.
I thought so, too...

Which is right?



- Don Baccus, Portland OR <dhogaza@pacifier.com>
  Nature photos, on-line guides, Pacific Northwest
  Rare Bird Alert Service and other goodies at
  http://donb.photo.net.

Wed, 26 Jan 2000 11:05:39 -0800
Don Baccus <dhogaza@pacifier.com>
Re: [HACKERS] Inheritance,
 referential integrity and other constraints
On Wed, Jan 26, 2000 at 07:34:29PM +0100, Peter Eisentraut wrote:
> On 2000-01-25, Chris Bitmead mentioned:
> 
> > As long as you're working on this area you could fix the problem where
> > if you do ALTER table* ADD COLUMN ... pg_dump no longer works because
> > the column orders have changed in different inherited tables.
> 
> This should be fixed in pg_dump then. As I see it, ALTER table* ADD COLUMN
> does exactly the right thing.

No, the problem is that right now, the order of columns in a child table
depends on the exact history of how all the columns got into each table.
Ideally, we want to be able to describe all the tables without reference
to history, only to (meta)content.  The exact order of columns in a table
really isn't much use to users, in any case (even though it is visible,
technically.  This had got to be a backward compatability feature of
the original SQL, isn't it?)

Ross
-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005

Wed, 26 Jan 2000 14:21:16 -0600
"Ross J. Reedstrom" <reedstrm@wallace.ece.rice.edu>
Re: [HACKERS] Inheritance,
 referential integrity and other constraints
Peter Eisentraut wrote:
  >On 2000-01-24, Oliver Elphick mentioned:
  >
  >> I would like to work on improving implementation of inheritance,
  >> especially with regard to referential integrity.   I suspect there are
  >> a number of issues that may be related and will need to be done together.
  >
  >What I really consider a problem, and it would be great if you could
  >tackle that, is that there is no real standard that all of this does or
  >even could follow.

That is the point of this thread: to settle the design.

  >                    For example, I wrote the other day that depending on
  >which way you see it, the behaviour of alter table x* add colum might be
  >considered right. Also I just looked into item 'Disallow inherited columns
  >with the same name as new columns' and it seems that someone actually made
  >provisions for this to be allowed, meaning that
  >create table test1 (x int);
  >create table test2 (x int) inherits (test1);
  >would result in test2 looking exactly like test1. No one knows what the
  >motivation was. (I removed it anyway.)

That's a relief!  Unless you have actually removed the ability to do
repeated inheritance?

  >> It will also be necessary to ensure that
  >> added constraints get inherited, when ALTER TABLE ... ADD/DROP
  >> CONSTRAINT gets implemented.
  >
  >I assume the semantics of ADD CONSTRAINT will be exactly the same as of
  >all the other alter table commands, in that if you specify a star then it
  >gets inherited, if not then not. 

This is the point of policy that needs deciding.  The fact that we can
say `SELECT ... FROM table*' implies to me that inheritance is a
permanent relationship between tables.  That is why we cannot DROP an
inherited column.  The question is, how close is the relationship?
We have to decide what model of inheritance we are using, because a
lot of design will flow automatically from that decision.

We can choose to follow a language model, but we must then decide which
language - Smalltalk, Eiffel, C++?  The fact that multiple inheritance
is possible seems to exclude Smalltalk; C++ is a conceptual mess (OK,
you can guess I'm an Eiffel fan!).  As a matter of fact, I don't think
that language models are very useful to PostgreSQL - an RDBMS with
inheritance is a unique animal!  I think we must devise a consistent and
useful scheme and not trouble overmuch about fitting it into a theoretical
framework, not least because the amount of work involved in a pure 
implementation is likely to be horrendous.  At present, PostgreSQL
supports multiple, repeated inheritance in reading tables, and partially
supports it in creating and altering them.  This scheme needs tidying
and completing.

The question to answer, then, is what inheritance is useful for; those are
the uses to be catered for.  I see its main use as being in the division
of similar data by kind.  I have used it like this:

                                       /----  customer
                /------- organisation <
               /                       \----  supplier
       person <
               \                       /----  staff
                \------- individual   <
                                       \----  contact

the idea being that each sub-level gives a more specialised view.

I want to be able to say `REFERENCES person*' to refer to the whole
group, or `REFERENCES organisation*' for a sub-group, or
`REFERENCES customer' for a single table.  Each is a valid use according
to how much specialisation is required in the individual case.
(The data is only at the lowest level descendant tables of this group
In Eiffel terms, person, organisation and individual would be deferred
classes, that cannot be used directly but must have at least one
descendant.)

With this kind of scheme, some constraints can, perhaps, be allowed to
differ, but I feel that PRIMARY KEY and REFERENCES, at the very least,
should be inherited.  UNIQUE should probably be inherited, and CHECK
constraints, DEFAULT and NOT NULL, can quite likely be allowed to differ.
What do you all think about this?

If we do allow differences, I think that they should not depend on the
user's remembering to add * to the table name.  I think that an
alteration to a parent table alone should require a UNINHERITED keyword
to make the intention explicit. (After all, the user may not realise
that the table is a parent; I think the RDBMS should protect him against
obvious traps.)

  >                                 But the problem with ADD CONSTRAINT is of
  >course that the entire table needs to be verified against the constraint
  >before allowing it to be added. This is fine if you do ADD CONSTRAINT
  >UNIQUE (a, b), because the index will take care of it, but it's trickier
  >if you add a trigger based constraint. The former might get into 7.0 if I
  >hurry, the latter most likely not.
  >
  >What needs discussion is whether indexes should be shared between
  >inherited tables, or whether each new descendant table needs a new
  >one. Not sure if this just made sense, though.
  
Perhaps we need a concept of grouped indexes to go with the grouped
tables that inheritance creates.  Clearly this is one of the issues
that the original designers didn't think through.  If we consider the
uses of an index, we can see that it is used first for fast access to
tuples and second to enforce uniqueness.  If (as I am suggesting)
the constraints that require an index (PRIMARY KEY, REFERENCES and UNIQUE)
are forced to be group-wide, it will follow that the corresponding
indexes should also be group-wide.  On the other hand, a user-created
index for fast access could apply to a single table in the group.

  >> I think that the implications of inheritance have never been fully
  >> explored and I would like to establish the framework in which future
  >> work that involves inheritance will be done.
  >
  >Precisely.
  >
  > 
  >> It seems to me that declaring a table to inherit from another, and
  >> enabling both to be read together by the table* syntax, together
  >> imply certain things about an inheritance group:
  >> 
  >> 1. All tables in the group must possess all the columns of their
  >> ancestor, and all those columns must be of the same type.
  >
  >Isn't it this way now?

Not if you allow columns to be dropped from or added to an individual
table, after it has become a parent, without enforcing the same change
on its descendants.  I am suggesting that this must be disallowed.  I am
also suggesting that adding columns to a parent requires either logical
column numbering or else physical insertion into the descendants in the
correct sequence.
  
  >> 2. Some constraints at least must be shared - primary key is the most
  >> obvious example; I think that _all_ constraints on inherited columns
  >> should be shared.  It is probably not practicable to force table
  >> constraints to be shared upwards.
  >
  >Not sure about this one. See the ranting about the shared indexes
  >above. Might be a great pain.

I fear it will be; but I suspect it is necessary in at least some cases
(see below).

  >> 4. Dropping a table implies dropping all its descendants.
  >
  >Actually what it does now is to refuse dropping when descendants
  >exist. What seems to be the proper solution to this is to implement the
  >proper DROP TABLE SQL syntax by adding a RESTRICT/CASCADE at the
  >end. Restrict refuses dropping if anything (descendants, views,
  >etc.) references the table, cascade drops everything else as
  >well. Implementing this could be your first step to glory ;) since it
  >seems it's more a matter of man hours than conceptual difficulty. Then
  >again, I could be wrong.

In this case, why not simply require `DROP TABLE table*', if table has
descendants?  I'm not at all sure that allowing a CASCADE option for DROP
TABLE is a good idea; someone could end up wiping out most of the database
with an ill-considered command; and RESTRICT should be the normal case.

  >> The grammar for ALTER TABLE allows either `ALTER TABLE table ...' or
  >> `ALTER TABLE table* ...'.  I would like to suggest that an alteration
  >> to a parent table must necessarily involve all its descendants and
  >> that alterations to inherited columns must be done in the appropriate
  >> parent.  So, given this hierarchy of tables:
  >
  >It's been a while since I looked into C++, but when you alter a descendant
  >(such as making a formerly public method private) you surely do not affect
  >the parents. The other way around I think the choice of star-or-not should
  >be given to the user. But this is again one of the issues that have no
  >point of reference, so I'm glad you bring it up for discussion.

Here, my point is that `SELECT * FROM table*' must be able to get a
consistent view throughout the inheritance group.  If an inherited
column is altered, the alteration may be one that would break that view.
The question to be decided is how far we go in enforcing similarity in
the columns that are shared.

Some things cannot be allowed: renaming columns must only be done to
the group as a whole; inherited columns can only be dropped from the 
whole group; a column cannot change its type in a descendant.

However, some differences are going to be allowed.
Consider this, as a case in point:

  a (id char2 primary key, name text not null)
  b (tp char(1) not null default 'B', supplier text) inherits (a);
  c (tp char(1) not null default 'C', customer text) inherits (a);

It seems quite a sensible use of inheritance to allow different defaults
for tp in tables b and c.  However, we then have difficulty here:

  d (c1 text) inherits (b,c)

Which tp is to be inherited?  At present, PostgreSQL avoids the problem
by not inheriting any constraints.  We need something like:

  d (c1 text) inherits (b,c) using b.tp


Now I have finished writing this, I can see that I have changed my mind
about the necessity of rigorously enforcing column sharing.  I think this
shows that I am still confused about what we want from inheritance; we
probably need to discuss this quite a bit more thoroughly before we
can come up with a design that we can all be happy with and that will
last.



Final note: I have just realised that most of what I am using inheritance
for could be done with views and unions, provided that we can REFERENCE a
view (which I haven't tested).  One really radical option would be to strip
out inheritance altogether!

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Wash me thoroughly from mine iniquity, and cleanse me 
      from my sin. For I acknowledge my transgressions; and 
      my sin is ever before me. Against thee, thee only, 
      have I sinned, and done this evil in thy sight..."
                                   Psalms 51:2-4 



Wed, 26 Jan 2000 23:58:35 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Inheritance, referential integrity and other
	constraints
Peter Eisentraut wrote:

> What needs discussion is whether indexes should be shared between
> inherited tables, or whether each new descendant table needs a new
> one. Not sure if this just made sense, though.

Shared indexes should definitely be allowed, if not the default. When
you've got deep hierarchies of inheritance it makes queries slow to have
to consult a whole lot of indexes.

> > 1. All tables in the group must possess all the columns of their
> > ancestor, and all those columns must be of the same type.
> 
> Isn't it this way now?

No, because you can do an ALTER TABLE (without the *) on the base, and
it doesn't get propagated to the descendants. Possibly this should be
disallowed, although it needs more thought.

BTW, if I remember right, Informix/Illustra has made the "*" also
include subclasses syntax the default. In other words you DON'T use the
*. If you only want a particular class and not sub-classes you have to
write "ONLY <classname>", or something. IMHO this is the RIGHT THING.
For almost everything (eg ALTER TABLE above) you always want to include
subclasses. Same goes for any random query. This is the OO way, you
don't think about subclasses unless you are doing something strange.

This is a pretty big change, but IMHO it should be made at some time.
"*" syntax should be eliminated and made default and something like ONLY
be added for when you really only want that one table. This won't affect
anyone using postgres as a RDBMS, only those people using it as ORDBMS.


> It's been a while since I looked into C++, but when you alter a descendant
> (such as making a formerly public method private) you surely do not affect
> the parents.

I don't think we're talking about descendants. Rather parents.

> The other way around I think the choice of star-or-not should
> be given to the user.

But then you can create a hierachy using ALTER that you couldn't have
created using plain CREATEs. This is bad I think and also has never been
done in any object system/language I've ever heard of.

Thu, 27 Jan 2000 11:28:00 +1100
Chris Bitmead <chris@bitmead.com>
Re: [HACKERS] Inheritance,
 referential integrity and other constraints
Oliver Elphick wrote:

> If we do allow differences, I think that they should not depend on the
> user's remembering to add * to the table name.  I think that an
> alteration to a parent table alone should require a UNINHERITED keyword
> to make the intention explicit. (After all, the user may not realise
> that the table is a parent; I think the RDBMS should protect him against
> obvious traps.)

I agree, and think that the "*" is a trap in general. Which is why I
suggest we go the Informix/Illustra route and dump "*" altogether,
replacing it with "ONLY" or some such, when you don't want inherited.

> Perhaps we need a concept of grouped indexes to go with the grouped
> tables that inheritance creates.  Clearly this is one of the issues
> that the original designers didn't think through.  If we consider the
> uses of an index, we can see that it is used first for fast access to
> tuples and second to enforce uniqueness.  If (as I am suggesting)
> the constraints that require an index (PRIMARY KEY, REFERENCES and UNIQUE)
> are forced to be group-wide, it will follow that the corresponding
> indexes should also be group-wide.  On the other hand, a user-created
> index for fast access could apply to a single table in the group.

I think indexes too should be inherited (physically as well as
logically) unless you choose the ONLY keyword.

>   a (id char2 primary key, name text not null)
>   b (tp char(1) not null default 'B', supplier text) inherits (a);
>   c (tp char(1) not null default 'C', customer text) inherits (a);
> 
> It seems quite a sensible use of inheritance to allow different defaults
> for tp in tables b and c.  However, we then have difficulty here:
> 
>   d (c1 text) inherits (b,c)
> 
> Which tp is to be inherited?  At present, PostgreSQL avoids the problem
> by not inheriting any constraints.  We need something like:
> 
>   d (c1 text) inherits (b,c) using b.tp

Hmmm. I don't think that's right at all. For example tp might be a
different type in b and c, and code might depend on that. It would be
logically unreasonable to have an inherited "d" not have BOTH tp from b
and c. I think from memory, Eiffel solves this by renaming doesn't it? I
think you need either renaming or scope resolving syntax. This would
probably get very messy, and I think it's probably quite sufficient to
force the user to not inherit the same name from b and C. If you want
that, you have to rename tp to be something else in b and/or c.

> Final note: I have just realised that most of what I am using inheritance
> for could be done with views and unions, provided that we can REFERENCE a
> view (which I haven't tested).  One really radical option would be to strip
> out inheritance altogether!

Please no! Yep, inheritance in SELECT is actually implemented as a UNION
internally. But don't dump it!

Thu, 27 Jan 2000 13:01:29 +1100
Chris Bitmead <chris@bitmead.com>
Re: [HACKERS] Inheritance,
 referential integrity and other constraints
Chris Bitmead wrote:
  >Oliver Elphick wrote:
...
  >>   a (id char2 primary key, name text not null)
  >>   b (tp char(1) not null default 'B', supplier text) inherits (a);
  >>   c (tp char(1) not null default 'C', customer text) inherits (a);
  >> 
  >> It seems quite a sensible use of inheritance to allow different defaults
  >> for tp in tables b and c.  However, we then have difficulty here:
  >> 
  >>   d (c1 text) inherits (b,c)
  >> 
  >> Which tp is to be inherited?  At present, PostgreSQL avoids the problem
  >> by not inheriting any constraints.  We need something like:
  >> 
  >>   d (c1 text) inherits (b,c) using b.tp
  >
  >Hmmm. I don't think that's right at all. For example tp might be a
  >different type in b and c, and code might depend on that.

No, the inheritance system doesn't allow them to be different types.
You get an error if you try to create such a table:

junk=> \d d
Table    = d
+----------------------------------+----------------------------------+-------+
|              Field               |              Type                | Length|
+----------------------------------+----------------------------------+-------+
| id                               | char()                           |     1 |
| words                            | text                             |   var |
| nu                               | float8                           |     8 |
+----------------------------------+----------------------------------+-------+
junk=> \d b
Table    = b
+----------------------------------+----------------------------------+-------+
|              Field               |              Type                | Length|
+----------------------------------+----------------------------------+-------+
| id                               | char()                           |     1 |
| words                            | text                             |   var |
| nu                               | numeric                          |  8.2  |
+----------------------------------+----------------------------------+-------+
junk=> \d d
Table    = d
+----------------------------------+----------------------------------+-------+
|              Field               |              Type                | Length|
+----------------------------------+----------------------------------+-------+
| id                               | char()                           |     1 |
| words                            | text                             |   var |
| nu                               | float8                           |     8 |
+----------------------------------+----------------------------------+-------+
junk=> create table e (x text) inherits (b,d);
ERROR:  float8 and numeric conflict for nu

And this is right because `SELECT nu FROM b*' and `SELECT nu FROM d*'
both need to work.

  >                                                          It would be
  >logically unreasonable to have an inherited "d" not have BOTH tp from b
  >and c.

Because the column names are identical, they are overlaid and treated
as the same column.  This is so whether or not they ultimately derive
from the same parent, so it isn't strictly a case of repeated inheritance
as in Eiffel. (There, repeatedly inherited features of the same parent
are silently combined, but identical names from unrelated classes are
conflicts.)

  >       I think from memory, Eiffel solves this by renaming doesn't it? I
  >think you need either renaming or scope resolving syntax. This would
  >probably get very messy, and I think it's probably quite sufficient to
  >force the user to not inherit the same name from b and C. If you want
  >that, you have to rename tp to be something else in b and/or c.
  
But we do allow this at the moment; identically named and typed columns
are taken to be the same column.  This is so, even if they don't appear
in the same order:

junk=> \d m
Table    = m
+----------------------------------+----------------------------------+-------+
|              Field               |              Type                | Length|
+----------------------------------+----------------------------------+-------+
| c1                               | char()                           |     2 |
| c2                               | int4                             |     4 |
| c3                               | text                             |   var |
| c4                               | numeric                          |  8.2  |
+----------------------------------+----------------------------------+-------+

so it looks as if the recent discussion about column ordering and
inheritance was off the point!

  >> Final note: I have just realised that most of what I am using inheritance
  >> for could be done with views and unions, provided that we can REFERENCE a
  >> view (which I haven't tested).  One really radical option would be to stri
      >p
  >> out inheritance altogether!
  >
  >Please no! Yep, inheritance in SELECT is actually implemented as a UNION
  >internally. But don't dump it!
  
Well no; I didn't really mean it!



-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Come now, and let us reason together, saith the LORD; 
      though your sins be as scarlet, they shall be as white
      as snow; though they be red like crimson, they shall 
      be as wool."                     Isaiah 1:18 



Thu, 27 Jan 2000 11:08:54 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Inheritance, referential integrity and other
	constraints
On 2000-01-25, Tom Lane mentioned:

> > Everything has its order and it's not like the inheritance as such is
> > broken.
> 
> Yes, a whole bunch of stuff is broken after this happens.  Go back and
> consult the archives --- or maybe Chris Bitmead will fill you in; he's
> got plenty of scars to show for this set of problems.  (All I recall
> offhand is that pg_dump and reload can fail to generate a working
> database.)  The bottom line is that it would be a lot nicer if column c
> had the same column position in both the parent table and the child
> table(s).

This should be fixed in pg_dump by infering something via the oids of the
pg_attribute entries. No need to mess up the backend for it.

Maybe pg_dump should optionally dump schemas in terms of insert into
pg_something commands rather than actual DDL. ;)

> 
> I suggest you be very cautious about messing with ALTER TABLE until you
> understand why inheritance makes it such a headache ;-)

I'm just trying to get the defaults and constraints working. If
inheritance stays broken the way it previously was, it's beyond my
powers. But I get the feeling that people rather not alter their tables
unless they have *perfect* alter table commands. I don't feel like arguing
with them, they'll just have to do without then.


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Thu, 27 Jan 2000 18:41:45 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Column ADDing issues 
On 2000-01-26, Oliver Elphick mentioned:

> As far as inheritance goes, all the descendants are treated as one table,
> including those created on a different branch from test2.  All of them
> have to return the right columns for a single query; the two options for
> dealing with this seem to be logical column numbering, or rewriting the
> descendant tables.  (But I haven't spent enough time in the code to be
> sure of that.)

Logical column ordering seems like a rather clean solution. The system
could also make educated decisions such as storing fixed size attributes
before variables sized ones. Kind of like a Cluster within the table.

I still think that fixing this in pg_dump might be the path of least
resistance, but we've got until autumn(?) to figure it out.

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Thu, 27 Jan 2000 18:41:55 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Column ADDing issues 
> > I suggest you be very cautious about messing with ALTER TABLE until you
> > understand why inheritance makes it such a headache ;-)
> 
> I'm just trying to get the defaults and constraints working. If
> inheritance stays broken the way it previously was, it's beyond my
> powers. But I get the feeling that people rather not alter their tables
> unless they have *perfect* alter table commands. I don't feel like arguing
> with them, they'll just have to do without then.

OK, so am I hearing we don't want ALTER TABLE DROP COLUMN without it
working for inhertance.  Is this really the way we want things?  May as
well disable ADD COLUMN too because that doesn't work for inheritance
either.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 27 Jan 2000 12:52:43 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Column ADDing issues
On Thu, Jan 27, 2000 at 12:52:43PM -0500, Bruce Momjian wrote:
> > > I suggest you be very cautious about messing with ALTER TABLE until you
> > > understand why inheritance makes it such a headache ;-)
> > 
> > I'm just trying to get the defaults and constraints working. If
> > inheritance stays broken the way it previously was, it's beyond my
> > powers. But I get the feeling that people rather not alter their tables
> > unless they have *perfect* alter table commands. I don't feel like arguing
> > with them, they'll just have to do without then.
> 
> OK, so am I hearing we don't want ALTER TABLE DROP COLUMN without it
> working for inhertance.  Is this really the way we want things?  May as
> well disable ADD COLUMN too because that doesn't work for inheritance
> either.

Bruce, I hope your playing devil's advocate here. What I'm hearing,
from this discussion, is a number of people interested in getting psql's
object features defined in a useful way. As far as impacting Peter's work
on getting ALTER commands working, I hope he understands that getting
the commands working for the SQL92 case, and leaving inheritance broken
(as it currently is) is far preferable to holding off for the *perfect*
problem definition. I interpreted his last sentence to mean "they'll
just have to do without *perfect* alter table commands", not "I'm not
going to work on this at all anymore". At least, I sure that's what I
hope he means :-)

If you meant the later, Peter, let me say that, in my opinion, very
few people are currently using postgres's inheritence features, and are
already having to manage with the broken state they're in. I'm glad to
see interest in improving them, but I see that as post 7.0 work. Heck,
If Oliver & Co. come up with an interesting, consistent object model,
that'd be reason enough for an 8.0 release. ;-) (No, please, not another
version number thread!) Certainly might be worth a long range development
fork in the CVS, at least.

Ross

-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005

Thu, 27 Jan 2000 12:55:44 -0600
"Ross J. Reedstrom" <reedstrm@wallace.ece.rice.edu>
Re: [HACKERS] Column ADDing issues
> On Thu, Jan 27, 2000 at 12:52:43PM -0500, Bruce Momjian wrote:
> > > > I suggest you be very cautious about messing with ALTER TABLE until you
> > > > understand why inheritance makes it such a headache ;-)
> > > 
> > > I'm just trying to get the defaults and constraints working. If
> > > inheritance stays broken the way it previously was, it's beyond my
> > > powers. But I get the feeling that people rather not alter their tables
> > > unless they have *perfect* alter table commands. I don't feel like arguing
> > > with them, they'll just have to do without then.
> > 
> > OK, so am I hearing we don't want ALTER TABLE DROP COLUMN without it
> > working for inhertance.  Is this really the way we want things?  May as
> > well disable ADD COLUMN too because that doesn't work for inheritance
> > either.
> 
> Bruce, I hope your playing devil's advocate here. What I'm hearing,
> from this discussion, is a number of people interested in getting psql's
> object features defined in a useful way. As far as impacting Peter's work
> on getting ALTER commands working, I hope he understands that getting
> the commands working for the SQL92 case, and leaving inheritance broken
> (as it currently is) is far preferable to holding off for the *perfect*
> problem definition. I interpreted his last sentence to mean "they'll
> just have to do without *perfect* alter table commands", not "I'm not
> going to work on this at all anymore". At least, I sure that's what I
> hope he means :-)

I interpret it the other way.  ALTER TABLE DROP is currently disabled in
gram.y, and I believe he thinks that unless it is 100%, we don't want
it.  Now, I believe that is very wrong, and I think it is fine as it is,
but I can see why he would think that after the hard time he was given.

This whole thing has wrapped around, and now I am not sure what signal
we are sending Peter.  I personally like what he has done, seeing that
he did exactly what I suggested when he asked on the list months ago.  I
don't want to do a phantom attribute thing at this point with very
little payback.  I also am not terribly concerned about inheritance
either as it needs work in many areas.

However, I am only one voice, and no one is giving direction to him.

We had better decide what we want in this area.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 27 Jan 2000 17:15:20 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Column ADDing issues
On 2000-01-26, Don Baccus mentioned:

> If so, I thought Oliver pointed out that you had the numbering wrong.
> I thought so, too...

Someone made the good point that, independent of which numbering you might
prefer, using add column gives you a setup which you could not achieve
using only create table. That makes sense to me, so I'm withdrawing my
argument. ;)

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Thu, 27 Jan 2000 23:28:16 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Inheritance,
	referential integrity and other  constraints
On 2000-01-26, Oliver Elphick mentioned:

>   >considered right. Also I just looked into item 'Disallow inherited columns
>   >with the same name as new columns' and it seems that someone actually made
>   >provisions for this to be allowed, meaning that
>   >create table test1 (x int);
>   >create table test2 (x int) inherits (test1);
>   >would result in test2 looking exactly like test1. No one knows what the
>   >motivation was. (I removed it anyway.)
> 
> That's a relief!  Unless you have actually removed the ability to do
> repeated inheritance?

Ugh, I just realized that of course you _have_ to allow duplicate column
names, e.g. in a scheme like
        b
    a <   > d
        c

the columns of "a" would arrive duplicated at "d" and the above logic
would merge them. I haven't finished that fix yet, so I better scrap it
now. Seem like this TODO item was really a non-starter.


> Final note: I have just realised that most of what I am using inheritance
> for could be done with views and unions, provided that we can REFERENCE a
> view (which I haven't tested).  One really radical option would be to strip
> out inheritance altogether!

Sure, I could live with that. It's not like it ever worked (in its
entirety). And any

table a (a1, a2, a3)
table b (b1, b2) inherits (a)

can also be implemented as

table a (a_id, a1, a2, a3)
table b_bare (b_id, b1, b2)
create view b as
   select a1, a2, a3, b1, b2 from outer join a, b on a_id = b_id
{or whatever that syntax was}

plus an insert rule or two.

It would make the rest of the code soooo much easier. (Sarcasm intended,
but a glimpse of truth as well.)

+++
Slashdot: "Self-proclaimed most advanced open-source database drops
object-oriented facilities to simplify code base"
AC reply: "Now when's KDE moving to C?"
+++

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Thu, 27 Jan 2000 23:28:24 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Inheritance,
	referential integrity and other  constraints
Oliver Elphick wrote:

> No, the inheritance system doesn't allow them to be different types.
> You get an error if you try to create such a table:

Hmm. While it might allow it, I can't see the logic in it. Can't think
of any OO language that thinks this way. All other languages you get
two different variables either with :: scope resolution in C++ or
renaming in Eiffel.

> Because the column names are identical, they are overlaid and treated
> as the same column.  This is so whether or not they ultimately derive
> from the same parent, so it isn't strictly a case of repeated inheritance
> as in Eiffel. (There, repeatedly inherited features of the same parent
> are silently combined, but identical names from unrelated classes are
> conflicts.)

Which seems like the right thing to me.

Fri, 28 Jan 2000 11:10:39 +1100
Chris Bitmead <chris@bitmead.com>
Re: [HACKERS] Inheritance,
 referential integrity and other constraints


Bruce Momjian wrote:

> > > I suggest you be very cautious about messing with ALTER TABLE until you
> > > understand why inheritance makes it such a headache ;-)
> >
> > I'm just trying to get the defaults and constraints working. If
> > inheritance stays broken the way it previously was, it's beyond my
> > powers. But I get the feeling that people rather not alter their tables
> > unless they have *perfect* alter table commands. I don't feel like arguing
> > with them, they'll just have to do without then.
>
> OK, so am I hearing we don't want ALTER TABLE DROP COLUMN without it
> working for inhertance.  Is this really the way we want things?  May as
> well disable ADD COLUMN too because that doesn't work for inheritance
> either.
>

I think this is not a good idea. Many of us doesn't interest inheritance.
ALTER ADD COLUMN is not complete but it is better than nothing.

> --
>   Bruce Momjian                        |  http://www.op.net/~candle
>   pgman@candle.pha.pa.us               |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
>
> ************

--
Jose' Soares
Bologna, Italy                     Jose@sferacarta.com



Fri, 28 Jan 2000 15:55:58 +0100
Jose Soares <jose@sferacarta.com>
Re: [HACKERS] Column ADDing issues
Oliver Elphick wrote:

> I would like to work on improving implementation of inheritance,
> especially with regard to referential integrity.   I suspect there are
> a number of issues that may be related and will need to be done together.
> In addition, this will be my first attempt to do anything serious in
> the PostgreSQL code itself,  so I would like to get some hints as
> to what I haven't even thought about!
>
> First, I would like to change the definition of the foreign key
> constraints to allow the inheritance star to follow a table name.
> This would mean that, for RI purposes, the named table would be
> aggregated with its descendants.  So "REFERENCES tbl" would mean that
> the foreign key must exist in tbl, but "REFERENCES tbl*" would allow it
> to exist either in tbl or in any of tbl's descendants.

    I  haven't  thought  about it in depth up to now, but I think
    that would cause much trouble in the RI triggers. They  don't
    even  have the full functionality and must be tested well for
    7.0.

    Can we wait with such an issue until after 7.0.

> Use of ON DELETE or ON UPDATE implies there must be an index on the
> referring column, to enable checking or deletion to be done speedily.
> This doesn't seem to happen at the moment.  If the reference is to
> an inheritance group, it would seem to be appropriate that all the
> tables in the group should use the same index.  Similarly, where
> a unique or primary key constraint is inherited, it may be desirable
> to use a single index to manage the constraint.  The implication of
> this would be that there must be a check when a table is dropped
> to make sure that a grouped index is not dropped until the last
> table in the group is dropped.

    Yes and yes. I thought about checking if there  is  a  unique
    index at the time, the referencing table is created (or later
    the constraint added).  But there is no way,  except  blowing
    up the DROP INDEX, to prevent someone from removing it later.
    And doing so would  prevent  then  from  fixing  a  corrupted
    index, so I'd be the first to vote against.

> Another item I would like to get fixed is to make sure that all
> constraints are inherited when a descendant table is created; this
> is a current TODO item.  It will also be necessary to ensure that
> added constraints get inherited, when ALTER TABLE ... ADD/DROP
> CONSTRAINT gets implemented.

    Yepp.

    But  please  don't  start  on  it  before 7.0. I would expect
    touching it right now could become a showstopper.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #



Fri, 28 Jan 2000 18:26:16 +0100 (CET)
wieck@debis.com (Jan Wieck)
Re: [HACKERS] Inheritance,
 referential integrity and other constraints
I'm down to the point where the parallel tests mostly work with a small
SI buffer --- but they do still sometimes fail.  I've realized that
there is a whole class of bugs along the following lines:

There are plenty of routines that do two or more SearchSysCacheTuple
calls to get the information they need.  As the code stands, it is
unsafe to continue accessing the tuple returned by SearchSysCacheTuple
after making a second such call, because the second call could possibly
cause an SI cache reset message to be processed, thereby flushing the
contents of the caches.

heap_open and CommandCounterIncrement are other routines that could
cause cache entries to be dropped.

This is a very insidious kind of bug because the probability of
occurrence is very low (at normal SI buffer size a reset is unlikely,
and even if it happens, you won't observe a failure unless the
pfree'd tuple is actually overwritten before you're done with it).
So we cannot hope to catch these things by testing.

I am not sure what to do about it.  One solution path is to make
all the potential trouble spots do SearchSysCacheTupleCopy and then
pfree the copied tuple when done.  However, that adds a nontrivial
amount of overhead, and it'd be awfully easy to miss some trouble
spots or to introduce new ones in the future.

Another possibility is to introduce some sort of notion of a reference
count, and to make the standard usage pattern be
	tuple = SearchSysCacheTuple(...);
	... use tuple ...
	ReleaseSysCacheTuple(tuple);
The idea here is that a tuple with positive refcount would not be
deleted during a cache reset, but would simply be removed from its
cache, and then finally deleted when released (or during elog
recovery).

This might allow us to get rid of SearchSysCacheTupleCopy, too,
since the refcount should be just as good as palloc'ing one's own
copy for most purposes.

I haven't looked at the callers of SearchSysCacheTuple to see whether
this would be a practical change to make.  I was wondering if anyone
had any comments or better ideas...

			regards, tom lane

Sun, 30 Jan 2000 10:41:13 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Another nasty cache problem
> I'm down to the point where the parallel tests mostly work with a small
> SI buffer --- but they do still sometimes fail.  I've realized that
> there is a whole class of bugs along the following lines:
> 
> There are plenty of routines that do two or more SearchSysCacheTuple
> calls to get the information they need.  As the code stands, it is
> unsafe to continue accessing the tuple returned by SearchSysCacheTuple
> after making a second such call, because the second call could possibly
> cause an SI cache reset message to be processed, thereby flushing the
> contents of the caches.

Yes, I have always been aware of this problem.  The issue is that since
cache entries are removed on a oldest-removed-first basis, I never
thought that several cache lookups would be a problem.  If you do many
cache lookups and expect very old ones to still exist, that could be a
problem.

However, a full reset of the cache could cause major problems.  Is there
a way to re-load the cache after the reset with the most recently cached
entries?  Seems that would be easier.  However, your issue is probably
that the new cache entries would have different locations from the old
entries.  Is it possible to delay the cache reset of the five most
recent cache entries, and do them later?  I don't see many places where
more than 2-3 cache entries are kept.  Maybe we need to keep them around
somehow during cache reset.

> I am not sure what to do about it.  One solution path is to make
> all the potential trouble spots do SearchSysCacheTupleCopy and then
> pfree the copied tuple when done.  However, that adds a nontrivial
> amount of overhead, and it'd be awfully easy to miss some trouble
> spots or to introduce new ones in the future.

Sounds like a lot of overhead to do the copy.

> 
> Another possibility is to introduce some sort of notion of a reference
> count, and to make the standard usage pattern be
> 	tuple = SearchSysCacheTuple(...);
> 	... use tuple ...
> 	ReleaseSysCacheTuple(tuple);
> The idea here is that a tuple with positive refcount would not be
> deleted during a cache reset, but would simply be removed from its
> cache, and then finally deleted when released (or during elog
> recovery).

If you can do that, can't you just keep the few most recent ones by
default.  Seems that would be very clean.

> This might allow us to get rid of SearchSysCacheTupleCopy, too,
> since the refcount should be just as good as palloc'ing one's own
> copy for most purposes.

Yes, that would be nice.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 30 Jan 2000 12:57:58 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Another nasty cache problem
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Yes, I have always been aware of this problem.  The issue is that since
> cache entries are removed on a oldest-removed-first basis, I never
> thought that several cache lookups would be a problem.

They're not, under normal circumstances...

> However, a full reset of the cache could cause major problems.  Is there
> a way to re-load the cache after the reset with the most recently cached
> entries?  Seems that would be easier.  However, your issue is probably
> that the new cache entries would have different locations from the old
> entries.  Is it possible to delay the cache reset of the five most
> recent cache entries, and do them later?

I don't think that's a good answer; what if one of those entries is the
one that the SI messages wanted us to update?  With a scheme like that,
you might be protecting a cache entry that actually isn't being used
anymore.  With a refcount you'd at least know whether it was safe to
throw the entry away.

Of course this just begs the question of what to do when an SI update
message arrives for a tuple that is locked down by refcount.  Maybe
we have to kick out an elog(ERROR) then.  Could be messy.

			regards, tom lane

Sun, 30 Jan 2000 13:22:05 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Yes, I have always been aware of this problem.  The issue is that since
> > cache entries are removed on a oldest-removed-first basis, I never
> > thought that several cache lookups would be a problem.
> 
> They're not, under normal circumstances...
> 
> > However, a full reset of the cache could cause major problems.  Is there
> > a way to re-load the cache after the reset with the most recently cached
> > entries?  Seems that would be easier.  However, your issue is probably
> > that the new cache entries would have different locations from the old
> > entries.  Is it possible to delay the cache reset of the five most
> > recent cache entries, and do them later?
> 
> I don't think that's a good answer; what if one of those entries is the
> one that the SI messages wanted us to update?  With a scheme like that,
> you might be protecting a cache entry that actually isn't being used
> anymore.  With a refcount you'd at least know whether it was safe to
> throw the entry away.
> 
> Of course this just begs the question of what to do when an SI update
> message arrives for a tuple that is locked down by refcount.  Maybe
> we have to kick out an elog(ERROR) then.  Could be messy.

Yep, that was my question.  You can re-load it, but if it is the one
that just got invalidated, what do you reload?  My guess is that you
keep using the same cache entry until the current transaction finishes,
at which point you can throw it away.

Now, if we did proper locking, no SI message could arrive for such an
entry.

My assumption is that these are mostly system cache entries, and they
rarely change, right?  If someone is operating on a table that gets an
SI entry, odds are that later on the system will fail because the table
is changed in some way, right?

Actually, don't we have a transaction id for the transaction that loaded
that cache entry.  We can add a transaction id to the cache record that
shows the transaction that last accessed that cache entry.  Then we can
say if any SI message comes in for a cache entry that was accessed by
the current transaction, we throw an elog.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 30 Jan 2000 14:25:29 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Another nasty cache problem
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Now, if we did proper locking, no SI message could arrive for such an
> entry.

> My assumption is that these are mostly system cache entries, and they
> rarely change, right?  If someone is operating on a table that gets an
> SI entry, odds are that later on the system will fail because the table
> is changed in some way, right?

If the tuple is actually *changed* then that's true (and locking should
have prevented it anyway).  But we also issue cache flushes against
whole system tables in order to handle VACUUM of a system table.  There,
the only thing that's actually been modified is the tuple's physical
location (ctid).  We don't want to blow away transactions that are just
looking at cache entries when a VACUUM happens.

Perhaps the caches shouldn't store ctid?  Not sure.

			regards, tom lane

Sun, 30 Jan 2000 16:54:14 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Now, if we did proper locking, no SI message could arrive for such an
> > entry.
> 
> > My assumption is that these are mostly system cache entries, and they
> > rarely change, right?  If someone is operating on a table that gets an
> > SI entry, odds are that later on the system will fail because the table
> > is changed in some way, right?
> 
> If the tuple is actually *changed* then that's true (and locking should
> have prevented it anyway).  But we also issue cache flushes against
> whole system tables in order to handle VACUUM of a system table.  There,
> the only thing that's actually been modified is the tuple's physical
> location (ctid).  We don't want to blow away transactions that are just
> looking at cache entries when a VACUUM happens.
> 
> Perhaps the caches shouldn't store ctid?  Not sure.

I am guilt of that.  There are a few place where I grab the tuple from
the cache, then use that to update the heap.  I thought it was a nifty
solution at the time.  I thought I used the CacheCopy calls for that,
but I am not positive.  Even if I did, that doesn't help because the
copy probably has an invalid tid at that point, thought I have opened
the table. Maybe I have to make sure I open the table before geting the
tid from the cache.

Is it only the tid that is of concern.  If so, that can probably be
fixed somehow.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 30 Jan 2000 16:56:14 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Another nasty cache problem
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Perhaps the caches shouldn't store ctid?  Not sure.

> I am guilt of that.  There are a few place where I grab the tuple from
> the cache, then use that to update the heap.  I thought it was a nifty
> solution at the time.  I thought I used the CacheCopy calls for that,
> but I am not positive.  Even if I did, that doesn't help because the
> copy probably has an invalid tid at that point, thought I have opened
> the table. Maybe I have to make sure I open the table before geting the
> tid from the cache.

I believe we worked that out and fixed it a few months ago: it's safe
to use the cache to find a tuple you want to update, if you open and
lock the containing table *before* doing the cache lookup.  Then you
know VACUUM's not running on that table (since you have it locked)
and you have an up-to-date TID for the tuple (since the open+lock
would have processed any pending shared-inval messages).  I went
around and made sure that's true everywhere.

What I was thinking about was adding code to the caches that would
(a) maintain refcounts on cached tuples, (b) reread rather than
discard a tuple if it is invalidated while refcount > 0, and (c)
kick out an error if the reread shows that the tuple has in fact
changed.  It seems that we would need to ignore the TID when deciding
if a tuple has changed, however.

			regards, tom lane

Sun, 30 Jan 2000 20:52:22 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
> I believe we worked that out and fixed it a few months ago: it's safe
> to use the cache to find a tuple you want to update, if you open and
> lock the containing table *before* doing the cache lookup.  Then you
> know VACUUM's not running on that table (since you have it locked)
> and you have an up-to-date TID for the tuple (since the open+lock
> would have processed any pending shared-inval messages).  I went
> around and made sure that's true everywhere.

Good.

> What I was thinking about was adding code to the caches that would
> (a) maintain refcounts on cached tuples, (b) reread rather than
> discard a tuple if it is invalidated while refcount > 0, and (c)
> kick out an error if the reread shows that the tuple has in fact
> changed.  It seems that we would need to ignore the TID when deciding
> if a tuple has changed, however.

Yes, that is one solution.  We can do it the same way heap_fetch works.
It requires a Buffer pointer which it uses to return a value that calls
ReleaseBuffer() when completed.

However, would just throwing an elog on any cache invalidate on a cache
row looked up in the current transaction/command counter make more
sense?  Sometimes you are using that cache oid in some later actions
that really can't be proper unlocked at the end?  Would be less code.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 30 Jan 2000 21:18:51 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Another nasty cache problem
On Sun, 30 Jan 2000, Tom Lane wrote:

> There are plenty of routines that do two or more SearchSysCacheTuple
> calls to get the information they need.  As the code stands, it is
> unsafe to continue accessing the tuple returned by SearchSysCacheTuple
> after making a second such call, because the second call could possibly
> cause an SI cache reset message to be processed, thereby flushing the
> contents of the caches.
> 
> heap_open and CommandCounterIncrement are other routines that could
> cause cache entries to be dropped.

This sort of thing should be documented, at least in the comment on top of
the function. From the developer's FAQ I gathered something like that
these tuples can be used for a short while, which is of course very exact.

Anyway, I just counted 254 uses of SearchSysCacheTuple in the backend tree
and a majority of these are probably obviously innocent. Since I don't
have any more developing planned, I would volunteer to take a look at all
of those and look for violations of second cache look up, heap_open, and
CommandCounterIncrement, fixing them where possible, or at least pointing
them out to more experienced people. That might save you from going out of
your way and instituting some reference count or whatever, and it would be
an opportunity for me to read some code.


-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Mon, 31 Jan 2000 13:55:15 +0100 (MET)
Peter Eisentraut <e99re41@DoCS.UU.SE>
Re: [HACKERS] Another nasty cache problem
On Sun, 30 Jan 2000, Bruce Momjian wrote:

> > Perhaps the caches shouldn't store ctid?  Not sure.
> 
> I am guilt of that.  There are a few place where I grab the tuple from
> the cache, then use that to update the heap.  I thought it was a nifty
> solution at the time.  I thought I used the CacheCopy calls for that,
> but I am not positive.  Even if I did, that doesn't help because the
> copy probably has an invalid tid at that point, thought I have opened
> the table. Maybe I have to make sure I open the table before geting the
> tid from the cache.

Urgh, I better check my code for that as well ... :(

> 
> Is it only the tid that is of concern.  If so, that can probably be
> fixed somehow.
> 
> 
> 

-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Mon, 31 Jan 2000 13:57:48 +0100 (MET)
Peter Eisentraut <e99re41@DoCS.UU.SE>
Re: [HACKERS] Another nasty cache problem
Peter Eisentraut <e99re41@DoCS.UU.SE> writes:
> This sort of thing should be documented,

... or changed ...

> Anyway, I just counted 254 uses of SearchSysCacheTuple in the backend tree
> and a majority of these are probably obviously innocent. Since I don't
> have any more developing planned, I would volunteer to take a look at all
> of those and look for violations of second cache look up, heap_open, and
> CommandCounterIncrement, fixing them where possible, or at least pointing
> them out to more experienced people. That might save you from going out of
> your way and instituting some reference count or whatever, and it would be
> an opportunity for me to read some code.

I appreciate the offer, but I don't really want to fix it that way.
If that's how things have to work, then the code will be *extremely*
fragile --- any routine that opens a relation or looks up a cache tuple
will potentially break its callers as well as itself.  And since the
probability of failure is so low, we'll never find it; we'll just keep
getting the occasional irreproducible failure report from the field.
I think we need a designed-in solution rather than a restrictive coding
rule.

Also, I am not sure that the existing uses are readily fixable.  For
example, I saw a number of crashes in the parser last night, most of
which traced to uses of Operator or Type pointers --- which are really
SearchSysCacheTuple results, but the parser passes them around with wild
abandon.  I don't see any easy way of restructuring that code to avoid
this.

I am starting to think that Bruce's idea might be the way to go: lock
down any cache entry that's been referenced since the last transaction
start or CommandCounterIncrement, and elog() if it's changed by
invalidation.  Then the only coding rule needed is "cached tuples don't
stay valid across CommandCounterIncrement", which is relatively
simple to check for.

			regards, tom lane

Mon, 31 Jan 2000 10:24:23 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
Tom Lane wrote:
> I'm down to the point where the parallel tests mostly work with a small
> SI buffer --- but they do still sometimes fail.

Have you committed your changes? I tried the parallel tests with cvs of around
5pm GMT 31 Jan, and they were all fine (I just ran out of procs at one point).
This is much better than last week! Thanks! I also tried that nonsensical
join from the other day, and it failed in the same way again:

newnham=# select * from crsids,"tblPerson" where
newnham-# crsids.crsid != "tblPerson"."CRSID";
Backend sent B message without prior T
D21Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.

After \. :

Unknown protocol character 'M' read from backend.  (The protocol character is the first character the backend sends in response to a query it receives).
PQendcopy: resetting connection
Asynchronous NOTIFY 'ndropoulou' from backend with pid '1818589281' received.
Asynchronous NOTIFY 'ndropoulou' from backend with pid '1818589281' received.


pq_flush: send() failed: Broken pipe
FATAL: pq_endmessage failed: errno=32

but no NOTICEs about SI anywhere any more, in fact no messages at all until
the "Unknown protocol character" bit above. The psql frontend process grows to
about 120Mb in size before this if that matters (200Mb swap free).

(Looking at why pg_dumpall creates unique indices for each different type
of index at the moment...)

Cheers,

Patrick

Mon, 31 Jan 2000 19:13:56 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] Another nasty cache problem
Does anyone know what is happening here?  There is no other user; the
CREATE TABLE is part of a much larger script that was working OK last
Thursday.  The script drops and recreates an entire database.

bray=# \d country
        Table "country"
 Attribute |  Type   | Modifier 
-----------+---------+----------
 id        | char(2) | not null
 name      | text    | not null
 region    | text    | 
 telcode   | text    | 
Index: country_pkey
Constraint: (id ~ '[A-Z]{2}'::text)

bray=# create table country_ccy
(
   country     char(2)     references country (id) match full,
   ccy      char(3)     references currency (symbol) match full,
   primary key (country, ccy)
)
;
NOTICE:  CREATE TABLE/PRIMARY KEY will create implicit index 
'country_ccy_pkey'
for table 'country_ccy'
NOTICE:  CREATE TABLE will create implicit trigger(s) for FOREIGN KEY check(s)
ERROR:  RelationClearRelation: relation 21645 modified while in use

bray=# select relname from pg_class where oid = 21645;
 relname 
---------
 country
(1 row)

country was referenced in a previous table's foreign key.  In view of
the notes on RelationClearRelation, I am wondering if refcount wasn't 
decremented after that table was created.


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "My son, if sinners entice thee, consent thou not."    
                            Proverbs 1:10 



Mon, 31 Jan 2000 22:18:19 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Recent RI changes have broken something
> I am starting to think that Bruce's idea might be the way to go: lock
> down any cache entry that's been referenced since the last transaction
> start or CommandCounterIncrement, and elog() if it's changed by
> invalidation.  Then the only coding rule needed is "cached tuples don't
> stay valid across CommandCounterIncrement", which is relatively
> simple to check for.

Yea, I had a good idea ...

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 31 Jan 2000 20:54:27 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Another nasty cache problem
Patrick Welche <prlw1@newn.cam.ac.uk> writes:
> Tom Lane wrote:
>> I'm down to the point where the parallel tests mostly work with a small
>> SI buffer --- but they do still sometimes fail.

> Have you committed your changes? I tried the parallel tests with cvs
> of around 5pm GMT 31 Jan, and they were all fine (I just ran out of
> procs at one point).  This is much better than last week! Thanks!

Yes, I committed what I had last night (about 04:35 GMT 1/31).

There are cache-flush-related bugs still left to deal with, but they
seem to be far lower in probability than the ones squashed so far.
I'm finding that even with MAXNUMMESSAGES set to 8, the parallel tests
usually pass; so it seems we need some other way of testing to nail down
the remaining problems.

> I also tried that nonsensical join from the other day, and it failed in
> the same way again:
> newnham=# select * from crsids,"tblPerson" where
> newnham-# crsids.crsid != "tblPerson"."CRSID";
> Backend sent B message without prior T

Hmm.  Can you provide a self-contained test case (a script to build the
failing tables, preferably)?

			regards, tom lane

Mon, 31 Jan 2000 21:02:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
"Oliver Elphick" <olly@lfix.co.uk> writes:
> ERROR:  RelationClearRelation: relation 21645 modified while in use

This is probably my fault.  Can you provide a simple test case?
The two table declarations might be enough.

			regards, tom lane

Mon, 31 Jan 2000 21:06:45 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Recent RI changes have broken something 
Tom Lane wrote:
  >"Oliver Elphick" <olly@lfix.co.uk> writes:
  >> ERROR:  RelationClearRelation: relation 21645 modified while in use
  >
  >This is probably my fault.  Can you provide a simple test case?
  >The two table declarations might be enough.
 
They seem to need to have the data loaded too.  The attached gzipped
tar is the minimum extract that will trigger the bug.  It contains
one example script (psql -d template1 -e <example) and 2 data files.
You will have to amend the script to put in the correct path of the
data files for your backend to find them.



Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "And be not conformed to this world; but be ye  
      transformed by the renewing of your mind, that ye may 
      prove what is that good, and acceptable, and perfect, 
      will of God."             Romans 12:2
Tue, 01 Feb 2000 09:29:31 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Recent RI changes have broken something 
>> "Oliver Elphick" <olly@lfix.co.uk> writes:
>>> ERROR:  RelationClearRelation: relation 21645 modified while in use
>> 
>> This is probably my fault.  Can you provide a simple test case?
>> The two table declarations might be enough.

I think this is fixed now.  Hopefully I didn't break SELECT FOR UPDATE
while I was at it.

			regards, tom lane

Wed, 02 Feb 2000 19:04:39 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Recent RI changes have broken something 
On Mon, Jan 31, 2000 at 09:02:30PM -0500, Tom Lane wrote:
> Patrick Welche <prlw1@newn.cam.ac.uk> writes:
> > Tom Lane wrote:
> 
> There are cache-flush-related bugs still left to deal with, but they
> seem to be far lower in probability than the ones squashed so far.
> I'm finding that even with MAXNUMMESSAGES set to 8, the parallel tests
> usually pass; so it seems we need some other way of testing to nail down
> the remaining problems.
> 
> > I also tried that nonsensical join from the other day, and it failed in
> > the same way again:
> > newnham=# select * from crsids,"tblPerson" where
> > newnham-# crsids.crsid != "tblPerson"."CRSID";
> > Backend sent B message without prior T
> 
> Hmm.  Can you provide a self-contained test case (a script to build the
> failing tables, preferably)?

It seems this is a memory exhaustion thing: I have 128Mb real memory.
Attached below is the C program used to create some random data in
tables test and test2 of database test (which needs to exist). Executing
the non-sensical query

  select * from test,test2 where test.i!=test2.i;

should result in 2600*599=1557400 (ie lots of) rows to be returned.
The process's memory consumption during this select grows to 128Mb, and after
a moment or two:

Backend sent D message without prior T
Backend sent D message without prior T
...

Which isn't quite the same message as before, but is of the same type.

 59 processes:  2 running, 57 sleeping
CPU states:  2.3% user, 86.4% nice,  9.3% system,  0.0% interrupt,  1.9% idle
Memory: 74M Act, 37M Inact, 184K Wired, 364K Free, 95M Swap, 262M Swap free

  PID USERNAME PRI NICE   SIZE   RES STATE   TIME   WCPU    CPU COMMAND
 1547 prlw1     50    0   128M  516K run     1:28 59.28% 59.28% psql
 1552 postgres  50    0  1920K  632K run     1:37 24.32% 24.32% postgres

later, while the "Backend sent..." messages appear

 1547 prlw1     -5    0   128M   68M sleep   1:41 23.00% 23.00% psql
 1552 postgres   2    0  1920K    4K sleep   1:41 141.00%  6.88% <postgres>

Note that there is still plenty of swap space. The 128Mb number seems to be
more than a coincidence (how to prove?)

So, is this only happening to me? How can lack of real memory affect timing
of interprocess communication?

Cheers,

Patrick

==========================================================================

#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>

#include "libpq-fe.h"

const char *progname;

PGresult *send_query(PGconn *db, const char *query)
{
  PGresult *res;

  res=PQexec(db,query);
  switch(PQresultStatus(res))
    {
      case PGRES_EMPTY_QUERY:
        printf("PGRES_EMPTY_QUERY: %s\n",query);
        break;
      case PGRES_COMMAND_OK:
        printf("PGRES_COMMAND_OK: %s\n",query);
        break;
      case PGRES_TUPLES_OK:
        printf("PGRES_TUPLES_OK: %s\n",query);
        break;
      case PGRES_COPY_OUT:
        printf("PGRES_COPY_OUT: %s\n",query);
        break;
      case PGRES_COPY_IN:
        printf("PGRES_COPY_IN: %s\n",query);
        break;
      case PGRES_BAD_RESPONSE:
        printf("PGRES_BAD_RESPONSE: %s\n",query);
        exit(1);
        break;
      case PGRES_NONFATAL_ERROR:
        printf("PGRES_NONFATAL_ERROR: %s\n",query);
        break;
      case PGRES_FATAL_ERROR:
        printf("PGRES_FATAL_ERROR: %s\n",query);
        exit(1);
        break;
      default:
        fprintf(stderr,"Error from %s: Unknown response from "\
          "PQresultStatus()\n",progname);
        exit(1);
        break;
    }

  return res;
}

char get_letter(void)
{
  int c;

  do c=(int)random()%128;
  while(!(isascii(c)&&isalpha(c)));

  return (char)tolower(c);
}

unsigned int get_num(void)
{
  return random()%100;
}

int main(int argc, char* argv[])
{
  char id[7],query[2048];
  int i;
  PGconn *db;
  PGresult *res;

  progname=argv[0];

  srandom(42); /* same data each time hopefully */

  db=PQconnectdb("dbname=test");
  if(PQstatus(db)==CONNECTION_BAD)
    {
      fprintf(stderr,"Error from %s: Unable to connect to database \"test\".\n",
        progname);
      exit(1);
    }

  res=send_query(db,"create table test (txt text,var varchar(7),i integer)");
  PQclear(res);
  res=send_query(db,"create table test2(txt text,var varchar(7),i integer)");
  PQclear(res);

  for(i=1;i<=2600;++i)
    {
      sprintf(id,"%c%c%c%c%03u",get_letter(),get_letter(),get_letter(),
        get_letter(),get_num());

      sprintf(query,"insert into test values ('%s','%s','%i')",id,id,i);
      res=send_query(db,query);
      PQclear(res);
    }

  for(i=1;i<=600;++i)
    {
      sprintf(id,"%c%c%c%c%03u",get_letter(),get_letter(),get_letter(),
        get_letter(),get_num());

      sprintf(query,"insert into test2 values ('%s','%s','%i')",id,id,i);
      res=send_query(db,query);
      PQclear(res);
    }

  PQfinish(db);

  return 0;
}

Thu, 3 Feb 2000 11:24:34 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] Another nasty cache problem
On Thu, Feb 03, 2000 at 11:24:34AM +0000, Patrick Welche wrote:
>   char id[7],query[2048];
            ^
  should be 8 to be safe..

Thu, 3 Feb 2000 12:14:26 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] Another nasty cache problem
Tom Lane wrote:
  >>> "Oliver Elphick" <olly@lfix.co.uk> writes:
  >>>> ERROR:  RelationClearRelation: relation 21645 modified while in use
  >>> 
  >>> This is probably my fault.  Can you provide a simple test case?
  >>> The two table declarations might be enough.
  >
  >I think this is fixed now.  Hopefully I didn't break SELECT FOR UPDATE
  >while I was at it.
 
Yes, it is fixed from my point of view.  I can't say about SELECT FOR
UPDATE...

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "O come, let us worship and bow down; let us kneel 
      before the LORD our maker."            Psalms 95:6 



Thu, 03 Feb 2000 15:06:42 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Recent RI changes have broken something 
Patrick Welche <prlw1@newn.cam.ac.uk> writes:
> Note that there is still plenty of swap space. The 128Mb number seems to be
> more than a coincidence (how to prove?)

The majority of Unix systems have a process size limit kernel parameter,
which is normally set to less than the amount of available swap space
(you don't want a single process running wild to chew up all your swap
and make other stuff start failing for lack of swap...)  Check your
kernel parameters.

It sounds to me like the backend has hit the size limit and is not
reacting gracefully to failure of malloc() to allocate more space.
It ought to exit with an elog(FATAL), probably.  Sigh, time to take
another pass through the code to cast a suspicious eye on everyplace
that calls malloc() directly.

There's a separate question about *why* such a simple query is chewing
up so much memory.  What query plan does EXPLAIN show for your test
query?

You said this was with current sources, right?

			regards, tom lane

Thu, 03 Feb 2000 12:00:21 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
Tom Lane wrote:
  >There's a separate question about *why* such a simple query is chewing
  >up so much memory.  What query plan does EXPLAIN show for your test
  >query?
  
I can show a similar problem.

  >You said this was with current sources, right?
 
This is with current sources: I managed to kill the backend before
it had used up all swap.  If left to run on 6.5.3 or CVS as of 2
weeks back it would kill the whole machine; I haven't let it get that
far today.

bray=# explain select * from pg_operator as a, pg_operator as b;
NOTICE:  QUERY PLAN:

Nested Loop  (cost=12604.88 rows=258064 width=162)
  ->  Seq Scan on pg_operator b  (cost=24.76 rows=508 width=81)
  ->  Seq Scan on pg_operator a  (cost=24.76 rows=508 width=81)

EXPLAIN


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "O come, let us worship and bow down; let us kneel 
      before the LORD our maker."            Psalms 95:6 



Thu, 03 Feb 2000 18:41:28 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Another nasty cache problem 
On Thu, Feb 03, 2000 at 12:00:21PM -0500, Tom Lane wrote:
>
> The majority of Unix systems have a process size limit kernel parameter,
> which is normally set to less than the amount of available swap space
> (you don't want a single process running wild to chew up all your swap
> and make other stuff start failing for lack of swap...)  Check your
> kernel parameters.

Probably to do with the shell limit:

memoryuse       125460 kbytes
 
> There's a separate question about *why* such a simple query is chewing
> up so much memory.  What query plan does EXPLAIN show for your test
> query?

test=# explain select * from test,test2 where test.i!=test2.i;
NOTICE:  QUERY PLAN:

Nested Loop  (cost=64104.80 rows=1559400 width=56)
  ->  Seq Scan on test2  (cost=24.80 rows=600 width=28)
  ->  Seq Scan on test  (cost=106.80 rows=2600 width=28)

EXPLAIN

> You said this was with current sources, right?

They're about 2 days old now. (Well, after your SI buffer overrun fixes)

Cheers,

Patrick

Thu, 3 Feb 2000 18:47:07 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] Another nasty cache problem
"Oliver Elphick" <olly@lfix.co.uk> writes:
> Tom Lane wrote:
>> There's a separate question about *why* such a simple query is chewing
>> up so much memory.  What query plan does EXPLAIN show for your test
>> query?
  
> I can show a similar problem.

> bray=# explain select * from pg_operator as a, pg_operator as b;
> NOTICE:  QUERY PLAN:

> Nested Loop  (cost=12604.88 rows=258064 width=162)
>  -> Seq Scan on pg_operator b  (cost=24.76 rows=508 width=81)
>  -> Seq Scan on pg_operator a  (cost=24.76 rows=508 width=81)

OK, I sussed this one --- there's a (longstanding) memory leak in
catcache.c.  When entering a system-table tuple into the cache,
it forgot to free the copy of the tuple that had been created in
transaction-local memory context.  Cause enough cache entries to
be created within one transaction, and you'd start to notice the
leak.  The above query exhibits the problem because it produces
about 250K tuples each with six regproc columns, and each regprocout
call does a cache lookup to convert regproc OID to procedure name.
Since you're cycling through 500-plus different procedure names,
and the cache only keeps ~ 300 entries, there's going to be a
fresh cache entry made every time :-(

With the fix I just committed, current sources execute the above query
in constant backend memory space.  psql's space usage still goes to the
moon, of course, since it's trying to buffer the whole query result :-(
... but there's no way around that short of a major redesign of libpq's
API.  When and if we switch over to CORBA, we really need to rethink
the client access API so that buffering the query result in the client-
side library is an option not a requirement.

I do not think this is the same problem that Patrick Welche is
complaining of, unfortunately.

			regards, tom lane

Thu, 03 Feb 2000 23:17:03 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
Tom Lane wrote:

> With the fix I just committed, current sources execute the above query
> in constant backend memory space.  psql's space usage still goes to the
> moon, of course, since it's trying to buffer the whole query result :-(
> ... but there's no way around that short of a major redesign of libpq's
> API.  When and if we switch over to CORBA, we really need to rethink
> the client access API so that buffering the query result in the client-
> side library is an option not a requirement.

What about portals? Doesn't psql use portals?

Fri, 04 Feb 2000 15:39:59 +1100
Chris Bitmead <chrisb@nimrod.itg.telstra.com.au>
Re: [HACKERS] Another nasty cache problem
Patrick Welche <prlw1@newn.cam.ac.uk> writes:
>   PID USERNAME PRI NICE   SIZE   RES STATE   TIME   WCPU    CPU COMMAND
>  1547 prlw1     50    0   128M  516K run     1:28 59.28% 59.28% psql
>  1552 postgres  50    0  1920K  632K run     1:37 24.32% 24.32% postgres

Sigh, I shoulda read this closely enough to notice that you were
complaining of psql memory overrun, not backend memory overrun :-(

The major problem here is that libpq's API is designed on the assumption
that libpq will buffer the whole query result in application memory
before letting the app see any of it.  I see no way around that without
a fundamental redesign of the API.  Which will happen someday, but not
today.

The minor problem is that libpq doesn't react very gracefully to running
out of memory.  It detects it OK, but then aborts query processing,
which means it gets out of step with the backend.  It needs to be fixed
so that it continues to absorb tuples (but drops them on the floor)
until the backend is done.  I've known of this problem for some time,
but have had many higher-priority problems to worry about.  Perhaps
someone else would like to take it on...

			regards, tom lane

Fri, 04 Feb 2000 00:26:00 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
Chris Bitmead <chrisb@nimrod.itg.telstra.com.au> writes:
> What about portals? Doesn't psql use portals?

No ... portals are a backend concept ...

			regards, tom lane

Fri, 04 Feb 2000 00:42:58 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
Tom Lane wrote:
> 
> Chris Bitmead <chrisb@nimrod.itg.telstra.com.au> writes:
> > What about portals? Doesn't psql use portals?
> 
> No ... portals are a backend concept ...

Since when?

According to the old doco you do...

select portal XX * from table_name where ...;

fetch 20 into XX.

If the PQexec() is called with "fetch 20" at a time
wouldn't this mean that you wouldn't exhaust front-end
memory with a big query?

Fri, 04 Feb 2000 16:57:54 +1100
Chris Bitmead <chrisb@nimrod.itg.telstra.com.au>
Re: [HACKERS] Another nasty cache problem
Chris Bitmead <chrisb@nimrod.itg.telstra.com.au> writes:
>> No ... portals are a backend concept ...

> Since when?

> According to the old doco you do...

> select portal XX * from table_name where ...;

> fetch 20 into XX.

That still works if you spell it in the SQL-approved way,
DECLARE CURSOR followed by FETCH.

> If the PQexec() is called with "fetch 20" at a time
> wouldn't this mean that you wouldn't exhaust front-end
> memory with a big query?

Sure, and that's how you work around the problem.  Nonetheless
this requires the user to structure his queries to avoid sucking
up a lot of data in a single query.  If the user doesn't have any
particular reason to need random access into a query result, it'd
be nicer to be able to read the result in a streaming fashion
without buffering it anywhere *or* making arbitrary divisions in it.

In any case, psql doesn't (and IMHO shouldn't) convert a SELECT
into a series of FETCHes for you.

			regards, tom lane

Fri, 04 Feb 2000 01:33:45 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
It seems that I am still tracking problems, but each time they turn out to
have a different cause: A slight variant on the select that caused memory
to run out gives


newnham=# select crsids.surname, "tblPerson"."Surname" from crsids,"tblPerson" where crsids.usn="tblPerson"."USN"::int4;
pqReadData() -- backend closed the channel unexpectedly.
        This probably means the backend terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

Nested Loop  (cost=66496.62 rows=34359 width=40)
  ->  Seq Scan on tblPerson  (cost=157.62 rows=2625 width=24)
  ->  Seq Scan on crsids  (cost=25.27 rows=584 width=16)

this is the table I based the memory hog on (2600*600). The backend closes
instantly ie., no memory usage! And, as before, it is hard to find a test case
that will do the same as repeatably (ie., test case never crashes, the
above case crashes every single time). "tblPerson", as its strange
capitalisation suggests, was imported from M$ access via ODBC.

select test.txt,test2.var from test,test2 where test2.i=test.var::int4;

Nested Loop  (cost=63504.80 rows=2600 width=40)
  ->  Seq Scan on test2  (cost=24.80 rows=600 width=16)
  ->  Seq Scan on test  (cost=105.80 rows=2600 width=24)

works fine.

Any thoughts on where to look?

Cheers,

Patrick

Fri, 4 Feb 2000 17:11:53 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] Another nasty cache problem
Patrick Welche <prlw1@newn.cam.ac.uk> writes:
> newnham=# select crsids.surname, "tblPerson"."Surname" from crsids,"tblPerson" where crsids.usn="tblPerson"."USN"::int4;
> pqReadData() -- backend closed the channel unexpectedly.
>         This probably means the backend terminated abnormally
>         before or while processing the request.
> The connection to the server was lost. Attempting reset: Failed.

> Any thoughts on where to look?

Is there anything in the postmaster log?  Is there a core file (look
in the database subdirectory, ie .../data/base/yourdatabase/core)?
If so, compiling the backend with -g and extracting a backtrace from
the resulting corefile with gdb would be very useful info.

			regards, tom lane

Fri, 04 Feb 2000 15:58:57 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
On Fri, Feb 04, 2000 at 03:58:57PM -0500, Tom Lane wrote:
> 
> > Any thoughts on where to look?
> 
> Is there anything in the postmaster log?

DEBUG:  Data Base System is in production state at Fri Feb  4 17:11:05 2000
Server process (pid 3588) exited with status 11 at Fri Feb  4 17:14:57 2000
Terminating any active server processes...
Server processes were terminated at Fri Feb  4 17:14:57 2000
Reinitializing shared memory and semaphores

>  Is there a core file (look
> in the database subdirectory, ie .../data/base/yourdatabase/core)?

But no core file ... so who knows what the sigsegv comes from. (don't worry
coredumpsize    unlimited)


> If so, compiling the backend with -g and extracting a backtrace from
> the resulting corefile with gdb would be very useful info.

(already have the -g..)

Still looking...

Cheers,

Patrick

Sat, 5 Feb 2000 14:35:15 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] Another nasty cache problem
Patrick Welche <prlw1@newn.cam.ac.uk> writes:
>> Is there anything in the postmaster log?

> DEBUG:  Data Base System is in production state at Fri Feb  4 17:11:05 2000
> Server process (pid 3588) exited with status 11 at Fri Feb  4 17:14:57 2000

> But no core file ... so who knows what the sigsegv comes from. (don't worry
> coredumpsize    unlimited)

There sure oughta be a corefile after a SIGSEGV.  Hmm.  How are you
starting the postmaster --- is it from a system startup script?
It might work better to start it from an ordinary user process.
I discovered the other day on a Linux box that the system just plain
would not dump a core file from a process started by root, even though
the process definitely had nonzero "ulimit -c" and had set its euid
to a nonprivileged userid.  But start the same process by hand from an
unprivileged login, and it would dump a core file.  Weird.  Dunno if
your platform behaves the same way, but it's worth trying.

			regards, tom lane

Sat, 05 Feb 2000 12:18:29 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
Someone mentioned recently that a timezone style of "GMT+0800" was on
their FreeBSD machine as an allowed time zone, that its behavior was
the same as the usual ISO8601 timezone of "-0800", and that this
conformed to some sort of Posix standard.

I had posted patches for this, and have just modified the patch to be
cleaner and more robust.

Before committing this (or at least before completing our upcoming
beta period), I'd like confirmation that this actually matches
expected behavior for a machine implementing a "GMT+0800" (or similar)
time zone, and that it is indeed a Posix standard? Anyone??

                   - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Sun, 06 Feb 2000 22:24:55 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Need confirmation of "Posix time standard" on FreeBSD
Thomas Lockhart wrote:
  >Someone mentioned recently that a timezone style of "GMT+0800" was on
  >their FreeBSD machine as an allowed time zone, that its behavior was
  >the same as the usual ISO8601 timezone of "-0800", and that this
  >conformed to some sort of Posix standard.
  >
  >I had posted patches for this, and have just modified the patch to be
  >cleaner and more robust.
  >
  >Before committing this (or at least before completing our upcoming
  >beta period), I'd like confirmation that this actually matches
  >expected behavior for a machine implementing a "GMT+0800" (or similar)
  >time zone, and that it is indeed a Posix standard? Anyone??
  >
This seems to be the case for me.

Debian GNU/Linux, using libc6 version 2.1.2:

linda:olly $ date
Sun Feb  6 23:55:52 GMT 2000
linda:olly $ TZ=GMT+8
linda:olly $ date
Sun Feb  6 15:56:26 GMT 2000
linda:olly $ TZ=posix/Etc/GMT+8
linda:olly $ date
Sun Feb  6 15:59:22 GMT+8 2000


 

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Lift up your heads, O ye gates; and be ye lift up, ye 
      everlasting doors; and the King of glory shall come 
      in. Who is this King of glory? The LORD strong and 
      mighty, the LORD mighty in battle."           
                                   Psalms 24:7,8 



Mon, 07 Feb 2000 00:00:19 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Need confirmation of "Posix time standard" on FreeBSD 
Thomas Lockhart writes:
> Someone mentioned recently that a timezone style of "GMT+0800" was on
> their FreeBSD machine as an allowed time zone, that its behavior was
> the same as the usual ISO8601 timezone of "-0800", and that this
> conformed to some sort of Posix standard.
> 
> I had posted patches for this, and have just modified the patch to be
> cleaner and more robust.
> 
> Before committing this (or at least before completing our upcoming
> beta period), I'd like confirmation that this actually matches
> expected behavior for a machine implementing a "GMT+0800" (or similar)
> time zone, and that it is indeed a Posix standard? Anyone??

I can confirm that it is a POSIX standard. Section 8.1.1 "Extensions
to Time Functions" of POSIX 1003.1-1988 says TZ can be of the form
    :characters
for implementation-defined behaviour or else
    std offset [dst [offset][,start[/time],end[/time]]]
(spaces for readability only) where std is three or more bytes
designating the standard time zone (any characters except a leading
colon, digits, comma, minus, plus or NUL allowed) and offset is the
value one must add to the local time to arrive at Coordinated
Universal Time. offset is of the form hh[:mm[:ss]] with hh required
and may be a single digit. Followed by gory details about the rest of
the fields. Phew.

--Malcolm

-- 
Malcolm Beattie <mbeattie@sable.ox.ac.uk>
Unix Systems Programmer
Oxford University Computing Services

Mon, 7 Feb 2000 12:24:48 +0000 (GMT)
Malcolm Beattie <mbeattie@sable.ox.ac.uk>
Re: [HACKERS] Need confirmation of "Posix time standard" on FreeBSD
> I can confirm that it is a POSIX standard. Section 8.1.1 "Extensions
> to Time Functions" of POSIX 1003.1-1988 says TZ can be of the form
>     :characters
> for implementation-defined behaviour or else
>     std offset [dst [offset][,start[/time],end[/time]]]
> (spaces for readability only) where std is three or more bytes
> designating the standard time zone (any characters except a leading
> colon, digits, comma, minus, plus or NUL allowed) and offset is the
> value one must add to the local time to arrive at Coordinated
> Universal Time. offset is of the form hh[:mm[:ss]] with hh required
> and may be a single digit. Followed by gory details about the rest of
> the fields. Phew.

Thanks for the info. How do they define "the standard time zone"? Is
it *any* time zone, or "GMT", or some other set of choices?

                 - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Mon, 07 Feb 2000 14:59:45 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Need confirmation of "Posix time standard" on FreeBSD
Thomas Lockhart writes:
> > I can confirm that it is a POSIX standard. Section 8.1.1 "Extensions
> > to Time Functions" of POSIX 1003.1-1988 says TZ can be of the form
> >     :characters
> > for implementation-defined behaviour or else
> >     std offset [dst [offset][,start[/time],end[/time]]]
> > (spaces for readability only) where std is three or more bytes
> > designating the standard time zone (any characters except a leading
> > colon, digits, comma, minus, plus or NUL allowed) and offset is the
> > value one must add to the local time to arrive at Coordinated
> > Universal Time. offset is of the form hh[:mm[:ss]] with hh required
> > and may be a single digit. Followed by gory details about the rest of
> > the fields. Phew.
> 
> Thanks for the info. How do they define "the standard time zone"? Is
> it *any* time zone, or "GMT", or some other set of choices?

It's "standard" in the sense of not-summer/not-daylight-savings rather
than in the "POSIX compliance" sense. In other words, std can be any
three bytes you like subject to the not-leading-colon, not-digits etc.
constraints above. Later in the section it says that summer time is
assumed to be one hour ahead of standard time if no offset follows
dst. Also,
    If [an offset is] preceded by a "-"; the time zone shall be east of
    the Prime Meridian; otherwise it shall be west (which may be
    indicated by an optional preceding "+").
That's the bit which shows that the "+" is OK. Aha, but I've just
looked back at your original message and it refers to "GMT+0800"
whereas POSIX requires a ":" between the hours and minutes. So in
fact, "GMT+0800" is *not* legal and it should be "GMT+08:00" or
"GMT+08" or "GMT+8" (single digit hours are allowed).

--Malcolm

-- 
Malcolm Beattie <mbeattie@sable.ox.ac.uk>
Unix Systems Programmer
Oxford University Computing Services

Mon, 7 Feb 2000 15:12:08 +0000 (GMT)
Malcolm Beattie <mbeattie@sable.ox.ac.uk>
Re: [HACKERS] Need confirmation of "Posix time standard" on FreeBSD
From: "Malcolm Beattie" <mbeattie@sable.ox.ac.uk>
> Thomas Lockhart writes:
> > Before committing this (or at least before completing our upcoming
> > beta period), I'd like confirmation that this actually matches
> > expected behavior for a machine implementing a "GMT+0800" (or similar)
> > time zone, and that it is indeed a Posix standard? Anyone??
>
> I can confirm that it is a POSIX standard. Section 8.1.1 "Extensions
> to Time Functions" of POSIX 1003.1-1988 says TZ can be of the form
>     :characters
> for implementation-defined behaviour or else
>     std offset [dst [offset][,start[/time],end[/time]]]

It probably won't affect anything, but some implementations (FreeBSD most
notably) have a bug in parsing TZ light savings string. The M notation gives
one day off for switching to/from light savings. Actually, it incorrectly
assumes Sunday as 0 for Zeller Congruence when it's Saturday.

Gene Sokolov.



Mon, 7 Feb 2000 19:07:12 +0300
"Gene Sokolov" <hook@aktrad.ru>
Re: [HACKERS] Need confirmation of "Posix time standard" on FreeBSD
> > Thanks for the info. How do they define "the standard time zone"? Is
> > it *any* time zone, or "GMT", or some other set of choices?
> It's "standard" in the sense of not-summer/not-daylight-savings rather
> than in the "POSIX compliance" sense. In other words, std can be any
> three bytes you like subject to the not-leading-colon, not-digits etc.
> constraints above. Later in the section it says that summer time is
> assumed to be one hour ahead of standard time if no offset follows
> dst. Also,
>     If [an offset is] preceded by a "-"; the time zone shall be east of
>     the Prime Meridian; otherwise it shall be west (which may be
>     indicated by an optional preceding "+").
> That's the bit which shows that the "+" is OK. Aha, but I've just
> looked back at your original message and it refers to "GMT+0800"
> whereas POSIX requires a ":" between the hours and minutes. So in
> fact, "GMT+0800" is *not* legal and it should be "GMT+08:00" or
> "GMT+08" or "GMT+8" (single digit hours are allowed).

OK. I'll need to generalize the current code, which looks specifically
for "gmt". Possibly, we'll have just the "GMT+/-####" case handled for
7.0, but if I get time.

And we'll allow a superset of the Posix standard, so "GMT+0800" will
be legal (otherwise, it would disallow the ISO8601 standard which imho
should take precedence).

                        - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Tue, 08 Feb 2000 18:24:44 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Need confirmation of "Posix time standard" on FreeBSD
On 2000-02-08, Thomas Lockhart mentioned:

> OK. I'll need to generalize the current code, which looks specifically
> for "gmt". Possibly, we'll have just the "GMT+/-####" case handled for
> 7.0, but if I get time.

I believe it should preferrably be called "UTC".

> And we'll allow a superset of the Posix standard, so "GMT+0800" will

He mentioned earlier that it has to be GMT+08:00 or GMT+8.

> be legal (otherwise, it would disallow the ISO8601 standard which imho
> should take precedence).

Oh please, it should. Is it just me or is this notation not making any
sense? If GMT+08:00 means "you need to add 8 hours to your local time zone
to get to GMT", then x = a + b means "you need to add 'b' to 'x' in order
to get 'a'". Darn standards. How about NOON+01:30 to indicate 10:30(am)?

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Thu, 10 Feb 2000 02:14:30 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Need confirmation of "Posix time standard" on FreeBSD
> > OK. I'll need to generalize the current code, which looks specifically
> > for "gmt". Possibly, we'll have just the "GMT+/-####" case handled for
> > 7.0, but if I get time.
> I believe it should preferrably be called "UTC".

The specific case we are solving is an interaction with the zinc
timezone database available on (at least) Linux and FreeBSD. Both
platforms have "GMT+/-n" zones defined, and we'll want to correctly
parse them.

> > And we'll allow a superset of the Posix standard, so "GMT+0800" will
> He mentioned earlier that it has to be GMT+08:00 or GMT+8.

Right. That's why I thought I'd mention that we'll do a superset.

> > be legal (otherwise, it would disallow the ISO8601 standard which imho
> > should take precedence).
> Oh please, it should. Is it just me or is this notation not making any
> sense? If GMT+08:00 means "you need to add 8 hours to your local time zone
> to get to GMT", then x = a + b means "you need to add 'b' to 'x' in order
> to get 'a'". Darn standards. How about NOON+01:30 to indicate 10:30(am)?

Not in this lifetime. afaict there is a common thread to date/time
representation in these two standards, as they both involve using a
"+/-" notation to represent time zones.

It is certainly annoying that there is a sign flip on the numeric
fields for the two standards (Posix and ISO8601). But only one, Posix,
has the preceding alpha time zone, so I should be able to figure it
out.

The annoying thing is that my token parsing is getting trickier, since
I also allow "January-1 2000" as a date specification, which from a
*token* standpoint is pretty similar to "gmt+8"...

                     - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Thu, 10 Feb 2000 06:52:10 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Need confirmation of "Posix time standard" on FreeBSD
The killer query was:

select crsids.surname,"tblPerson"."Surname" from crsids,"tblPerson" where crsids.usn="tblPerson"."USN"::int4;

and the reason for the SIGSEGV, is that somehow, text_int4(text *string) in
src/backend/utils/adt/int.c is called with string=(text *)0x0, so obviously
this is a problem!

crsids.usn is integer, "tblPerson"."USN" is varchar(9).

Oddly enough, text_int4 is called from fmgr.c:136 which is in the case
statement for n_arguments=2, yet that should be 1

(gdb) print {FmgrInfo}0x8221a30
$4 = {fn_addr = 0x80f9dbc <text_int4>, fn_plhandler = 0, fn_oid = 1620, 
  fn_nargs = 1}

unless gdb is reporting the wrong line number. values->data[0]=0=string.

I have a backtrace and a pretty printed copy of the query tree if useful...

Still trying to make a small test case...

Any suggestions appreciated!

Cheers,

Patrick

(source of 31st Jan)

On Sat, Feb 05, 2000 at 12:18:29PM -0500, Tom Lane wrote:
> Patrick Welche <prlw1@newn.cam.ac.uk> writes:
> >> Is there anything in the postmaster log?
> 
> > DEBUG:  Data Base System is in production state at Fri Feb  4 17:11:05 2000
> > Server process (pid 3588) exited with status 11 at Fri Feb  4 17:14:57 2000
> 
> > But no core file ... so who knows what the sigsegv comes from. (don't worry
> > coredumpsize    unlimited)
> 
> There sure oughta be a corefile after a SIGSEGV.  Hmm.  How are you
> starting the postmaster --- is it from a system startup script?
> It might work better to start it from an ordinary user process.
> I discovered the other day on a Linux box that the system just plain
> would not dump a core file from a process started by root, even though
> the process definitely had nonzero "ulimit -c" and had set its euid
> to a nonprivileged userid.  But start the same process by hand from an
> unprivileged login, and it would dump a core file.  Weird.  Dunno if
> your platform behaves the same way, but it's worth trying.
> 
> 			regards, tom lane

Fri, 11 Feb 2000 21:04:42 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] Another nasty cache problem
Patrick Welche <prlw1@newn.cam.ac.uk> writes:
> and the reason for the SIGSEGV, is that somehow, text_int4(text *string) in
> src/backend/utils/adt/int.c is called with string=(text *)0x0, so obviously
> this is a problem!

Um.  Probably you have a NULL value in "tblPerson"."USN" somewhere?

There are a lot of functions without adequate defenses against NULL
inputs :-( --- we've been cleaning them up slowly, but evidently you
found another one.

			regards, tom lane

Fri, 11 Feb 2000 18:18:32 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
On Fri, Feb 11, 2000 at 06:18:32PM -0500, Tom Lane wrote:
> Patrick Welche <prlw1@newn.cam.ac.uk> writes:
> > and the reason for the SIGSEGV, is that somehow, text_int4(text *string) in
> > src/backend/utils/adt/int.c is called with string=(text *)0x0, so obviously
> > this is a problem!
> 
> Um.  Probably you have a NULL value in "tblPerson"."USN" somewhere?

Yes of course! Naturally I was looking for something far too complicated and
the trees got in the way.. And that's why my test case didn't work.

> There are a lot of functions without adequate defenses against NULL
> inputs :-( --- we've been cleaning them up slowly, but evidently you
> found another one.

So the trouble is, if the function returns and int, and you want to say
return null, there really isn't a value that can be stuck into the int
that represents null?

In the meantime, I think this might help, so I would have seen:

newnham=# select crsids.surname,"tblPerson"."Surname" from crsids,"tblPerson" where crsids.usn="tblPerson"."USN"::int4;
ERROR:  Trying to convert NULL text to integer (int4)

Cheers,

Patrick



Index: int.c
===================================================================
RCS file: /usr/local/cvsroot/pgsql/src/backend/utils/adt/int.c,v
retrieving revision 1.32
diff -c -r1.32 int.c
*** int.c       2000/01/26 05:57:14     1.32
--- int.c       2000/02/14 11:22:32
***************
*** 277,282 ****
--- 277,285 ----
        int                     len;
        char       *str;
  
+       if (!string)
+               elog(ERROR, "Trying to convert NULL text to integer (int2)");
+ 
        len = (VARSIZE(string) - VARHDRSZ);
  
        str = palloc(len + 1);
***************
*** 317,322 ****
--- 320,328 ----
  
        int                     len;
        char       *str;
+ 
+       if (!string)
+               elog(ERROR, "Trying to convert NULL text to integer (int4)");
  
        len = (VARSIZE(string) - VARHDRSZ);


Mon, 14 Feb 2000 11:23:56 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] Another nasty cache problem
Patrick Welche <prlw1@newn.cam.ac.uk> writes:
  
> +       if (!string)
> +               elog(ERROR, "Trying to convert NULL text to integer (int2)");

This is unreasonable behavior.  The correct patch is just

	if (!string)
		return 0;

which will allow the function manager to plow ahead with returning the
NULL that it's going to return anyway.  See the past pghackers threads
about redesigning the function manager interface if you don't understand
what's going on here.

			regards, tom lane

Mon, 14 Feb 2000 11:00:16 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Another nasty cache problem 
On Mon, Feb 14, 2000 at 11:00:16AM -0500, Tom Lane wrote:
> Patrick Welche <prlw1@newn.cam.ac.uk> writes:
>   
> > +       if (!string)
> > +               elog(ERROR, "Trying to convert NULL text to integer (int2)");
> 
> This is unreasonable behavior.  The correct patch is just
> 
> 	if (!string)
> 		return 0;
> 
> which will allow the function manager to plow ahead with returning the
> NULL that it's going to return anyway.  See the past pghackers threads
> about redesigning the function manager interface if you don't understand
> what's going on here.

Off top of head, that means that null and the string "0" both return 0..
OK - I'll look for the mail thread.

Cheers,

Patrick

Mon, 14 Feb 2000 16:14:44 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] Another nasty cache problem
I've just committed changes to "reunify" the date/time types.
"timestamp" and "interval" are now the two primary date/time types for
users. Also, I've changed the default date style to "ISO" (not just in
time for Y2K, but we'll be ready for "Y3K").

Also, I made some changes to have NUMERIC be a "known" type for
purposes of implicit type coersion, but have not tested to see if the
underlying conversion functions are available.

initdb required (and enforced by a catalog version change).

Regression tests pass, except for the rules test due to ongoing rules
formatting work.

                  - Thomas

The detailed change log:

Make NUMERIC a known native type for purposes of type coersion. Not
tested.
Make ISO date style (e.g. "2000-02-16 09:33") the default.
Implement "date/time grand unification".
 Transform datetime and timespan into timestamp and interval.
 Deprecate datetime and timespan, though translate to new types in
gram.y.
 Transform all datetime and timespan catalog entries into new types.
 Make "INTERVAL" reserved word allowed as a column identifier in
gram.y.
 Remove dt.h, dt.c files, and retarget datetime.h, datetime.c as
utility
  routines for all date/time types.
 date.{h,c} now deals with date, time types.
 timestamp.{h,c} now deals with timestamp, interval types.
 nabstime.{h,c} now deals with abstime, reltime, tinterval types.
All regression tests pass except for rules.sql (unrelated).

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Wed, 16 Feb 2000 17:41:23 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Date/time types: big change
On Wed, 16 Feb 2000, Thomas Lockhart wrote:

> I've just committed changes to "reunify" the date/time types.
> "timestamp" and "interval" are now the two primary date/time types for
> users. Also, I've changed the default date style to "ISO" (not just in
> time for Y2K, but we'll be ready for "Y3K").

I still don't like our Y2038 status. ;)

Anyway, the question I have is what did you do with functions such as
datetimein() or comparison functions and such for the old types? Did you
remove them? What if some, say, user-defined trigger function uses them?

The reason I'm asking is that I would like to see the floating point types
converted to SQL in a similar fashion, but when I rename, say, float4eq to
realeq it might break user applications. Or not? This is all hypothetical
of course.

-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Wed, 16 Feb 2000 19:37:57 +0100 (MET)
Peter Eisentraut <e99re41@DoCS.UU.SE>
Re: [HACKERS] Date/time types: big change
> I've just committed changes to "reunify" the date/time types.
> "timestamp" and "interval" are now the two primary date/time types for
> users. Also, I've changed the default date style to "ISO" (not just in
> time for Y2K, but we'll be ready for "Y3K").
> 

I think we need a consensus on this.  I think this may be a problem for
some people.  Comments?

	test=> create table x ( y date);
	CREATE
	test=> insert into x values ('02/01/99');
	INSERT 18697 1
	test=> select * from x;
	     y      
	------------
	 02-01-1999
	(1 row)
	
	test=> set datestyle to 'iso';
	SET VARIABLE
	test=> select * from x;
	     y      
	------------
	 1999-02-01
	(1 row)


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 16 Feb 2000 13:55:14 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Date/time types: big changeu
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Also, I've changed the default date style to "ISO" (not just in
>> time for Y2K, but we'll be ready for "Y3K").

> I think we need a consensus on this.  I think this may be a problem for
> some people.  Comments?

Good point.  Perhaps there should be a way to select the default date
style at configure or initdb time.  I don't mind if the "default default"
is ISO, but if I had apps that were dependent on the old default setting
I'd sure be annoyed by this change...

Has anyone thought much about the fact that beginning next year,
heuristics to guess which field is the year will become nearly useless?
Quick, when is '01/02/03'?  I suspect a lot of people who got away with
not thinking hard about datestyles will suddenly realize that they need
to set the default datestyle to whatever they are accustomed to using.

			regards, tom lane

Wed, 16 Feb 2000 17:09:08 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Date/time types: big changeu 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> Also, I've changed the default date style to "ISO" (not just in
> >> time for Y2K, but we'll be ready for "Y3K").
> 
> > I think we need a consensus on this.  I think this may be a problem for
> > some people.  Comments?
> 
> Good point.  Perhaps there should be a way to select the default date
> style at configure or initdb time.  I don't mind if the "default default"
> is ISO, but if I had apps that were dependent on the old default setting
> I'd sure be annoyed by this change...
> 
> Has anyone thought much about the fact that beginning next year,
> heuristics to guess which field is the year will become nearly useless?
> Quick, when is '01/02/03'?  I suspect a lot of people who got away with
> not thinking hard about datestyles will suddenly realize that they need
> to set the default datestyle to whatever they are accustomed to using.

Wow, that is an excellent point.  I was doing it for 2000, and was
thinking, gee, that's not too hard.  I can see it getting much more
confusing next year, as you said.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 16 Feb 2000 17:13:42 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Date/time types: big changeu
Tom Lane wrote:
  >Bruce Momjian <pgman@candle.pha.pa.us> writes:
  >>> Also, I've changed the default date style to "ISO" (not just in
  >>> time for Y2K, but we'll be ready for "Y3K").
  >
  >> I think we need a consensus on this.  I think this may be a problem for
  >> some people.  Comments?
  >
  >Good point.  Perhaps there should be a way to select the default date
  >style at configure or initdb time.  I don't mind if the "default default"
  >is ISO, but if I had apps that were dependent on the old default setting
  >I'd sure be annoyed by this change...
  >
  >Has anyone thought much about the fact that beginning next year,
  >heuristics to guess which field is the year will become nearly useless?
  >Quick, when is '01/02/03'?  I suspect a lot of people who got away with
  >not thinking hard about datestyles will suddenly realize that they need
  >to set the default datestyle to whatever they are accustomed to using.
 
I have code to let the installer choose the default datestyle in Debian's installation script for PostgreSQL.  It makes its own best guess on
the basis of the timezone and then asks the user with its own guess as
the presented default.

See the attached script; I don't know how generalisable the timezone
guessing would be.




Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "But as many as received him, to them gave he power to 
      become the sons of God, even to them that believe on 
      his name"        John 1:12
Wed, 16 Feb 2000 23:20:26 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Date/time types: big changeu 
> > I've just committed changes to "reunify" the date/time types.
> > "timestamp" and "interval" are now the two primary date/time types for
> > users. Also, I've changed the default date style to "ISO" (not just in
> > time for Y2K, but we'll be ready for "Y3K").
> I still don't like our Y2038 status. ;)

Yeah. Like the ntp rfc doc says: "we'll expect the solution to appear
before it is needed" or something to that effect.

> Anyway, the question I have is what did you do with functions such as
> datetimein() or comparison functions and such for the old types? Did you
> remove them? What if some, say, user-defined trigger function uses them?

Then they are SOL. I had originally implemented datetime and timespan
as an experiment to see if a floating point number could behave well
enough to represent dates (I was worried about rounding and the
.999999 problem, especially with the wide range of platforms we
support).

So it turns out that they work. In the meantime, someone contributed a
timestamp type, but did not fully implement it and chose a 4 byte
representation, which is fundamentally flawed imho.

I've been waiting a year or two to do this upgrade, and the major rev
bump is the time and place to do it.

One reason why I didn't carry along both datetime *and* timestamp is
the large number of related functions and operators. It would have
significantly increased the size of the catalogs (mostly because
timestamp didn't have much to start with).

> The reason I'm asking is that I would like to see the floating point types
> converted to SQL in a similar fashion, but when I rename, say, float4eq to
> realeq it might break user applications. Or not? This is all hypothetical
> of course.

Lots of work for not much gain imho. For the date/time stuff, it made
sense because timestamp needed to be replaced. There isn't the same
underlying need for the floating point types afaik.

On the other hand, 7.0 (or 8.0, but that may be another 4 years ;) is
the time to do it. Does anyone else see this as an issue?

                       - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Thu, 17 Feb 2000 06:04:45 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Date/time types: big change
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> But, I'd have no objection to a configure or initdb option; I *would*
> suggest that the old default (and it is the default mostly because
> original Postgres95 had no other styles implemented) is a relatively
> poor choice, and that ISO should be the default choice in the absence
> of an explicit configure or initdb switch.

As I said, I have no objection to making ISO the new "standard default";
I just think some people will need a way to change the default in their
installations.

			regards, tom lane

Thu, 17 Feb 2000 01:12:14 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Date/time types: big changeu 
> >> Also, I've changed the default date style to "ISO" (not just in
> >> time for Y2K, but we'll be ready for "Y3K").
> > I think we need a consensus on this.  I think this may be a problem for
> > some people.  Comments?
> Good point.  Perhaps there should be a way to select the default date
> style at configure or initdb time.  I don't mind if the "default default"
> is ISO, but if I had apps that were dependent on the old default setting
> I'd sure be annoyed by this change...

I've been talking about this for quite some time, but there *really*
is no excuse to not go to the ISO date/time standard. Every other date
style is prone to misinterpretation, and the ISO standard is commonly
used in other instances where reliable date reporting is needed.

I've waited until a major rev to do this, and the groundwork has been
there for a year or two. There are some good summaries of the issues
on the web.

But, I'd have no objection to a configure or initdb option; I *would*
suggest that the old default (and it is the default mostly because
original Postgres95 had no other styles implemented) is a relatively
poor choice, and that ISO should be the default choice in the absence
of an explicit configure or initdb switch.

                    - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Thu, 17 Feb 2000 06:14:23 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Date/time types: big changeu
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> The reason I'm asking is that I would like to see the floating point types
>> converted to SQL in a similar fashion, but when I rename, say, float4eq to
>> realeq it might break user applications. Or not? This is all hypothetical
>> of course.

> Lots of work for not much gain imho. For the date/time stuff, it made
> sense because timestamp needed to be replaced. There isn't the same
> underlying need for the floating point types afaik.

> On the other hand, 7.0 (or 8.0, but that may be another 4 years ;) is
> the time to do it. Does anyone else see this as an issue?

I think it's too late in the 7.0 cycle to start thinking about renaming
the numeric types.  While you implemented the date/time changes at
almost the last minute, the changes had been discussed and agreed to
long ago, and you knew exactly what you needed to do.  I don't think
that constitutes a precedent for a hurried revision of the numeric
types...

We've already postponed 7.0 beta twice.  Seems to me it's time to
start raising the bar for what we will accept into this revision.

			regards, tom lane

Thu, 17 Feb 2000 01:29:40 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Date/time types: big change 
On Thu, 17 Feb 2000, Tom Lane wrote:

> I think it's too late in the 7.0 cycle to start thinking about renaming
> the numeric types.

I didn't mean that this should happen now or even soon. It was more of a
policy/practice inquiry.

-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Thu, 17 Feb 2000 13:00:17 +0100 (MET)
Peter Eisentraut <e99re41@DoCS.UU.SE>
Re: [HACKERS] Date/time types: big change 
Peter Eisentraut <e99re41@DoCS.UU.SE> writes:
> On Thu, 17 Feb 2000, Tom Lane wrote:
>> I think it's too late in the 7.0 cycle to start thinking about renaming
>> the numeric types.

> I didn't mean that this should happen now or even soon. It was more of a
> policy/practice inquiry.

OK, fair enough.  What I'm thinking at the moment is let's wait and see
how painful or painless the transition is for the date/time types.
The number of squawks we hear about that should give us a clue whether
we want to be in a hurry to rename the numeric types...

			regards, tom lane

Thu, 17 Feb 2000 11:08:13 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Date/time types: big change 
> I've been talking about this for quite some time, but there *really*
> is no excuse to not go to the ISO date/time standard. Every other date
> style is prone to misinterpretation, and the ISO standard is commonly
> used in other instances where reliable date reporting is needed.
> 
> I've waited until a major rev to do this, and the groundwork has been
> there for a year or two. There are some good summaries of the issues
> on the web.
> 
> But, I'd have no objection to a configure or initdb option; I *would*
> suggest that the old default (and it is the default mostly because
> original Postgres95 had no other styles implemented) is a relatively
> poor choice, and that ISO should be the default choice in the absence
> of an explicit configure or initdb switch.

Well, no one is objecting to this yet, so it may be a good choice.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 17 Feb 2000 11:58:10 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Date/time types: big changeu
On Thu, Feb 17, 2000 at 06:14:23AM +0000, Thomas Lockhart wrote:
> I've been talking about this for quite some time, but there *really*
> is no excuse to not go to the ISO date/time standard. Every other date

Yes, please let's go to this standard. It's an awful lot of work to fix apps
just because they expect US notation and most people in Germany cannot even
think about typing it that way.

> poor choice, and that ISO should be the default choice in the absence
> of an explicit configure or initdb switch.

Completely agree.

Michael
-- 
Michael Meskes                         | Go SF 49ers!
Th.-Heuss-Str. 61, D-41812 Erkelenz    | Go Rhein Fire!
Tel.: (+49) 2431/72651                 | Use Debian GNU/Linux!
Email: Michael@Fam-Meskes.De           | Use PostgreSQL!

Thu, 17 Feb 2000 19:41:07 +0100
Michael Meskes <meskes@postgresql.org>
Re: [HACKERS] Date/time types: big changeu
On Wed, Feb 16, 2000 at 11:20:26PM +0000, Oliver Elphick wrote:
> I have code to let the installer choose the default datestyle in Debian's installation script for PostgreSQL.  It makes its own best guess on
> the basis of the timezone and then asks the user with its own guess as
> the presented default.

Yes, Oliver's script works nicely on all Debian machines.

Michael

-- 
Michael Meskes                         | Go SF 49ers!
Th.-Heuss-Str. 61, D-41812 Erkelenz    | Go Rhein Fire!
Tel.: (+49) 2431/72651                 | Use Debian GNU/Linux!
Email: Michael@Fam-Meskes.De           | Use PostgreSQL!

Thu, 17 Feb 2000 19:42:33 +0100
Michael Meskes <meskes@postgreSQL.org>
Re: [HACKERS] Date/time types: big changeu
>   >>> Also, I've changed the default date style to "ISO" (not just in
>   >>> time for Y2K, but we'll be ready for "Y3K").
>   >... Perhaps there should be a way to select the default date
>   >style at configure or initdb time.  I don't mind if the "default default"
>   >is ISO, but if I had apps that were dependent on the old default setting
>   >I'd sure be annoyed by this change...

Well, that is the joy of a major release; not all backward
compatibility is guaranteed. This has been a *documented change* for
at least a year or two; check the chapter on date/time data types for
more info.

However, istm that we could/should have more "default settings"
traveling in the pg_database table. We've got the encoding, which if
set for template1 will be set for every db. We've got the database
location, which can point to an alternate location.

Wouldn't it be reasonable to allow a "default datestyle", or something
more general to help with other defaults? Hmm, could be a text field
which allows something like "PGDATESTYLE='ISO',LANGUAGE='french',..."
so that it is extensible, but maybe that detail is a bad idea because
it is a bit fragile.

What fields would be appropriate for v7.0? The datestyle and timezone
are two obvious candidates, and if we add them now then we could make
use of them later.

Later, we can get things like 

  ALTER DATABASE SET DEFAULT DATESTYLE='ISO';

etc etc.

For v7.1, I'm hoping to work with Tatsuo and others to get closer to
the general character sets and collation sequences allowed by SQL92.
At that point, the MULTIBYTE hardcoded differences in the backend
might go away and we will need these configurable default values.

                    - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Thu, 17 Feb 2000 18:48:11 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Date/time types: big changeu
> >   >>> Also, I've changed the default date style to "ISO" (not just in
> >   >>> time for Y2K, but we'll be ready for "Y3K").
> >   >... Perhaps there should be a way to select the default date
> >   >style at configure or initdb time.  I don't mind if the "default default"
> >   >is ISO, but if I had apps that were dependent on the old default setting
> >   >I'd sure be annoyed by this change...
> 
> Well, that is the joy of a major release; not all backward
> compatibility is guaranteed. This has been a *documented change* for
> at least a year or two; check the chapter on date/time data types for
> more info.

> 
> However, istm that we could/should have more "default settings"
> traveling in the pg_database table. We've got the encoding, which if
> set for template1 will be set for every db. We've got the database
> location, which can point to an alternate location.

But we have to store this information in the database because it is
related to how the data is stored.  Do the date/time fields also have
that assumption _in_ that stored data?  If so, we need it stored in the
database, if not, it seems some SET command or psql startup file setting
is enough.  Many people work on the same database from different
locations and may need different settings.  I would only store database
settings that relate to the data, not how the data is displayed.  That
stuff belongs outside the database, I think.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 17 Feb 2000 23:52:39 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Date/time types: big changeu
-----------------------------------------------------------------------

On Wed, 16 Feb 2000, Thomas Lockhart wrote:

> I've just committed changes to "reunify" the date/time types.
> "timestamp" and "interval" are now the two primary date/time types for
> users. Also, I've changed the default date style to "ISO" (not just in
> time for Y2K, but we'll be ready for "Y3K").
> 
> Also, I made some changes to have NUMERIC be a "known" type for
> purposes of implicit type coersion, but have not tested to see if the
> underlying conversion functions are available.
> 
> initdb required (and enforced by a catalog version change).
> 
> Regression tests pass, except for the rules test due to ongoing rules
> formatting work.


Great, you fix my formatting code for timestamp. Thanks Thomas!

But conversion timestam to 'tm' struct is not Y2038 ready 
(POSIX 'tm' limitation?):

test=# select to_char('Fri Feb 18 11:57:47 2038 CET'::timestamp, 'HH:MI:SS YYYY');
    to_char
---------------
 10:57:47 2038
(1 row)

Or simple:

test=# select 'Fri Feb 18 11:57:47 2038 CET'::timestamp;
         ?column?
--------------------------
 Thu Feb 18 10:57:47 2038
(1 row)


Or am I something leave out?

							Karel




Fri, 18 Feb 2000 12:27:04 +0100 (CET)
Karel Zak - Zakkr <zakkr@zf.jcu.cz>
Re: [HACKERS] Date/time types: big change
On Thu, 17 Feb 2000, Thomas Lockhart wrote:

> However, istm that we could/should have more "default settings"
> traveling in the pg_database table. We've got the encoding, which if
> set for template1 will be set for every db. We've got the database
> location, which can point to an alternate location.

I don't think this should be a per database setting. Why not use an
environment variable PGDATESTYLE for it. That's easy enough for now.
Before we throw all kinds of per database defaults around, I'd like to see
some sort of a concept where exactly a "database" stands versus "schema",
etc. What happens if one day queries across databases are allowed?

> For v7.1, I'm hoping to work with Tatsuo and others to get closer to
> the general character sets and collation sequences allowed by SQL92.

Excellent.

-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Fri, 18 Feb 2000 15:29:34 +0100 (MET)
Peter Eisentraut <e99re41@DoCS.UU.SE>
Re: [HACKERS] Date/time types: big changeu
> But conversion timestam to 'tm' struct is not Y2038 ready
> (POSIX 'tm' limitation?):
>     to_char
> ---------------
>  10:57:47 2038
> (1 row)
> Or am I something leave out?

No, that is the expected behavior. In most of the world (certainly in
the US), time zones and daylight savings time were both very nebulous
things until around the turn of the century. I recall reading that in
the US building the continental railroads in the 1860's provoked
thinking about standardizing time zone.

There are also minor changes in time zone and DST behavior in recent
history; in the US we had a year or two in the 1970's which ran DST
year round due to the oil shortage.

So, since the actual time zone behavior for years past 2038 is
uncertain, and since the Unix time support routines don't support
anything past 2038 anyway, I omit time zone calculations after
2038-01-18 and before 1901-12-14. Everything is carried as equivalent
to GMT, but no time zone adjustment is carried out.

btw, there *may* be some edge effects which are, um, unexpected; e.g.
having a time zone adjustment as you enter a date w/o an explicit tz
into the database, to which the same adjustment is *not* applied as
the date is read back out. Feel free to test it out...

                       - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Fri, 18 Feb 2000 14:48:46 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Date/time types: big change
Peter Eisentraut <e99re41@DoCS.UU.SE> writes:
> I don't think this should be a per database setting. Why not use an
> environment variable PGDATESTYLE for it.

We already have that, and I wouldn't have brought up the issue if I
thought it was sufficient.  The case where you really want to be able
to set a default at the database or installation level is when you have
a ton of client apps running on a bunch of different machines, and you
can't or don't want to fix them all at once.  A client-side fix doesn't
get the job done for a dbadmin faced with that kind of situation.

Or were you talking about a server-side env variable?  That could work
I guess, but I thought you were intent on eliminating env-var
dependencies in initdb and postmaster startup ... for good reasons ...

> Before we throw all kinds of per database defaults around, I'd like to see
> some sort of a concept where exactly a "database" stands versus "schema",
> etc. What happens if one day queries across databases are allowed?

Presumably a client doing that would make sure to request the same
datestyle (or whatever) from each database.  You're right though
that we could use some global thinking about what parameters need
to be settable from where, and what their scopes need to be.

			regards, tom lane

Fri, 18 Feb 2000 10:35:01 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Date/time types: big changeu 
> Or were you talking about a server-side env variable?

fwiw, we've already got that one...

                 - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Fri, 18 Feb 2000 15:48:29 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Date/time types: big changeu
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> Or were you talking about a server-side env variable?

> fwiw, we've already got that one...

We do?  (... examines code ...)  By golly, you're right.

OK, I agree with Peter: this is enough to save the day for a dbadmin
who really really doesn't want to switch to default ISO datestyle,
so I withdraw my complaint.

I do, however, suggest that the backend env var needs to be documented
more prominently than it is now.  One might also ask why its set of
allowed values is inconsistent with the SET command's (probably
postgres.c ought to just call a routine in variable.c, rather than
having its own parsing code)?

			regards, tom lane

Fri, 18 Feb 2000 10:52:48 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Date/time types: big changeu 
> I do, however, suggest that the backend env var needs to be documented
> more prominently than it is now.

Hmm, I thought it was in the Admin Guide, but in fact it shows up only
in the Data Types chapter and in the release notes. Should be added to
runtime.sgml just before (?) "Starting postmaster".

> One might also ask why its set of
> allowed values is inconsistent with the SET command's (probably
> postgres.c ought to just call a routine in variable.c, rather than
> having its own parsing code)?

I'm vaguely recalling that there was a "chicken and egg" problem with
the backend firing up... Ah, in fact I think (still from my sometimes
faulty memory) that it had to do with whether the Postgres memory
management stuff (palloc et al) was available at the time postgres.c
needed to make the call.

Feel free to review it though and make sweeping or small changes.

                  - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Fri, 18 Feb 2000 16:28:37 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Date/time types: big changeu
On 2000-02-18, Tom Lane mentioned:

> Or were you talking about a server-side env variable?  That could work
> I guess, but I thought you were intent on eliminating env-var
> dependencies in initdb and postmaster startup ... for good reasons ...

Yes, as you noticed.

I don't mind postmaster startup environment variables that much. The ones
for initdb were much more evil. This really seems to be an item for the
Grand Unified Configuration File, but until that happens it's easier to
have a dozen of orthogonal environment variables than having to reorganize
this whole thing later on.

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Sat, 19 Feb 2000 15:12:44 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Date/time types: big changeu 
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> One might also ask why its set of
>> allowed values is inconsistent with the SET command's (probably
>> postgres.c ought to just call a routine in variable.c, rather than
>> having its own parsing code)?

> I'm vaguely recalling that there was a "chicken and egg" problem with
> the backend firing up... Ah, in fact I think (still from my sometimes
> faulty memory) that it had to do with whether the Postgres memory
> management stuff (palloc et al) was available at the time postgres.c
> needed to make the call.

Yup, your memory is still working...

> Feel free to review it though and make sweeping or small changes.

OK, I tweaked the code in variable.c to not depend on palloc(), and
made the change.  In the course of doing so, I noticed what I assume
is a bug: RESET DateStyle and SET DateStyle = 'DEFAULT' were still
setting to Postgres style.  Presumably they should reset to ISO style
in the brave new world, no?

What I actually did was to make them reset to whatever the backend's
startup setting is.  Thus, if a postmaster PGDATESTYLE environment
variable exists, it will determine the result of RESET DateStyle as
well as the state of a new backend.  (A client-side PGDATESTYLE setting
cannot affect RESET, of course, since it just causes a SET command to
be issued.)  I think this is appropriate behavior, but it might be open
to debate.

BTW, here is an interesting corner case for you: what happens when
the postmaster is started with PGDATESTYLE="DEFAULT"?  You get ISO
now, but I almost committed code that would have gone into infinite
recursion...

			regards, tom lane

Sat, 19 Feb 2000 17:25:35 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Date/time types: big change
Does anybody know how to disable "pager" of psql? It's really annoying
when I use psql in my emacs's shell buffer...
--
Tatsuo Ishii


Tue, 22 Feb 2000 15:57:30 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
psql and pager
Tatsuo Ishii wrote:
> 
> Does anybody know how to disable "pager" of psql? It's really annoying
> when I use psql in my emacs's shell buffer...

One way is to unset the PAGER environment variable.

-- 
Chris Bitmead
mailto:chris@bitmead.com

Tue, 22 Feb 2000 21:09:55 +1100
Chris <chris@bitmead.com>
Re: [HACKERS] psql and pager
> > Does anybody know how to disable "pager" of psql? It's really annoying
> > when I use psql in my emacs's shell buffer...
> 
> One way is to unset the PAGER environment variable.

New psql seems to try to use more even PAGER is not set.
--
Tatsuo Ishii

Tue, 22 Feb 2000 19:13:25 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] psql and pager
Tatsuo Ishii wrote:
  >Does anybody know how to disable "pager" of psql? It's really annoying
  >when I use psql in my emacs's shell buffer...
 
Before you start psql, do `export PAGER=cat'

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "The LORD bless thee, and keep thee; The LORD make his
      face shine upon thee, and be gracious unto thee; The 
      LORD lift up his countenance upon thee, and give thee 
      peace."              Numbers 6:24-26 



Tue, 22 Feb 2000 10:32:05 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] psql and pager 
On Tue, 22 Feb 2000, Tatsuo Ishii wrote:

> Does anybody know how to disable "pager" of psql? It's really annoying
> when I use psql in my emacs's shell buffer...

\pset pager

Put it in .psqlrc if you like. (Or in .psqlrc-7.0.0 to not affect the old
one.)

-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Tue, 22 Feb 2000 13:20:07 +0100 (MET)
Peter Eisentraut <e99re41@DoCS.UU.SE>
Re: [HACKERS] psql and pager
> > Does anybody know how to disable "pager" of psql? It's really annoying
> > when I use psql in my emacs's shell buffer...
> 
> \pset pager
> 
> Put it in .psqlrc if you like. (Or in .psqlrc-7.0.0 to not affect the old
> one.)

Thansk. I have read the man page of psql. However I misunderstood:

>Toggles the list of a pager to do table output. If the environment
>variable PAGER is set, the output is piped to the specified program.
>Otherwise more is used. 
>
>In any case, psql only uses the pager if it seems appropriate. 

I thought psql may use a pager even if \pset toggles off a pager.  I
need to learn English more:-)
--
Tatsuo Ishii


Tue, 22 Feb 2000 21:30:07 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] psql and pager
On Tue, 22 Feb 2000, Tatsuo Ishii wrote:

> I thought psql may use a pager even if \pset toggles off a pager. 

No, only if it's toggled on.

In the old version the use of the pager was determined by whether or not
the environment variable PAGER was set. This behaviour is not very
cooperative with other programs that might use the same variable.

-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Tue, 22 Feb 2000 13:46:51 +0100 (MET)
Peter Eisentraut <e99re41@DoCS.UU.SE>
Re: [HACKERS] psql and pager

-----Pďż˝vodnďż˝ zprďż˝va-----
Od: Tom Lane <tgl@sss.pgh.pa.us>
Komu: Oliver Elphick <olly@lfix.co.uk>
Kopie: pgsql-hackers@postgreSQL.org <pgsql-hackers@postgreSQL.org>;
Vladimir.Benes@pvt.cz <Vladimir.Benes@pvt.cz>
Datum: 22. ďż˝nora 2000 18:06
Pďż˝edmďż˝t: Re: [HACKERS] Out of memory problem (forwarded bug report)


>"Oliver Elphick" <olly@lfix.co.uk> writes:
>> Can someone advise, please, how to deal with this problem in 6.5.3?
>

>My guess is that the cause is memory leaks during expression evaluation;
>but without seeing the complete view definitions and underlying table
>definitions, it's impossible to know what processing is being invoked
>by this query...
>
> regards, tom lane



    Well, I will append views and underlying table definition:

1) Once again - failure query:
select comm_type,name,tot_bytes,tot_packets
from flow_sums_days_send_200002_view
where day='2000-02-21' and name not like '@%'
union all
select comm_type,name,tot_bytes,tot_packets
from flow_sums_days_receive_200002_view
where day='2000-02-21' and name not like '@%'

2) views definition:
create view flow_sums_days_send_200002_view as
select
  'send'::varchar as comm_type, date_trunc('day',start) as day,
  src_name as name, sum(bytes) as tot_bytes, sum(packets) as tot_packets
from flow_sums_200002
group by day, src_name

create view flow_sums_days_receive_200002_view as
select
  'receive'::varchar as comm_type, date_trunc('day',start) as day,
  dst_name as name, sum(bytes) as tot_bytes, sum(packets) as tot_packets
from flow_sums_200002
group by day, dst_name


I wanted create only one usefull view:

create view flow_sums_days_200002_view as
select
  'send'::varchar as comm_type, date_trunc('day',start) as day,
  src_name as name, sum(bytes) as tot_bytes, sum(packets) as tot_packets
from flow_sums_200002
group by day, src_name
UNION ALL
select
  'receive'::varchar as comm_type, date_trunc('day',start) as day,
  dst_name as name, sum(bytes) as tot_bytes, sum(packets) as tot_packets
from flow_sums_200002
group by day, dst_name

...but Postgres cann't use clause UNION ALL at view definition. So I created
two views mentioned above and I wanted use this ones with UNION ALL clause
only.

3) underlaying table definition:
create table flow_sums_200002 (
    primary_collector varchar(50) not null,
    start datetime not null,
    end_period datetime not null,
    dead_time_rel float4 not null,
    src_name varchar(50) not null,
    dst_name varchar(50) not null,
    bytes int8 not null,
    packets int4 not null
)

    Today this table has about 3 000 000 rows and the select command
mentioned above returns 190 + 255 rows.


    Now I don't use clause "UNION ALL" and the program executes two queryes
and then adds both result to new result. I reduced time increment of number
rows to flow_sums_200002 table (three times less). This table contains data
of February 2000 and the program will create table flow_sums_200003 with
relevant views next month.
    Well, now this solution solve my problem but always depends on number of
rows - I only moved limit of rows count.


            Thank You, V. Benes

P.S.: I append part of top on my system while the query is running:

CPU states: 98.6% user,  1.3% system,  0.0% nice,  0.0% idle
Mem:  127256K av, 124316K used,   2940K free,  29812K shrd,   2620K buff
Swap: 128516K av,  51036K used,  77480K free                  7560K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
2942 postgres  20   0  141M  99M 17348 R       0 99.0 80.4   1:22 postmaster

=> postmaster later took 80 - 95% of memory, free memory decressed to 2 MB,
CPU was overloaded (0% idle and 99% by user process of postmaster). Have You
ever seen something similar :-) ?



Wed, 23 Feb 2000 08:26:11 +0100
"=?iso-8859-2?B?VmxhZGlt7XIgQmVuZbk=?=" <Vladimir.Benes@pvt.cz>
Re: [HACKERS] Out of memory problem (forwarded bug report) 
Vladimir,
  Thanks for the details.  I think you are undoubtedly running into
expression evaluation memory leaks.  Basically, any expression that
yields a non-pass-by-value data type consumes memory that is not
reclaimed until end of statement --- so when you process a few million
rows, that memory starts to add up.  (Yes, I realize this is a horrible
misfeature.  It's on our TO-DO list to fix it, but it probably won't
happen until 7.1 or 7.2.)  In the meantime the best I can offer you
is workarounds.

  I think the major problems here are coming from the
"date_trunc('day',start)" calculation (because its datetime result is
pass-by-reference) and to a lesser extent from the sum(bytes)
calculation (because int8 is pass-by-reference).  You could easily
replace "date_trunc('day',start)" with "date(start)"; since date is
a pass-by-value type, that won't leak memory, and it should give
equivalent results.  The int8 sum is not quite so easy to fix.
I assume you can't get away with switching to int4 --- probably
your sum would overflow an int4?  It may be that just fixing the
inefficient date_trunc calc will reduce your memory requirements
enough to get by.  If not, the only good news I have is that release
7.0 does fix the memory-leak problem for internal calculations of
aggregate functions like sum().  You can get the first beta release
for 7.0 now.

			regards, tom lane


"=?iso-8859-2?B?VmxhZGlt7XIgQmVuZbk=?=" <Vladimir.Benes@pvt.cz> writes:
> -----P�vodn� zpr�va-----
> Od: Tom Lane <tgl@sss.pgh.pa.us>
> Komu: Oliver Elphick <olly@lfix.co.uk>
> Kopie: pgsql-hackers@postgreSQL.org <pgsql-hackers@postgreSQL.org>;
> Vladimir.Benes@pvt.cz <Vladimir.Benes@pvt.cz>
> Datum: 22. �nora 2000 18:06
> P�edm�t: Re: [HACKERS] Out of memory problem (forwarded bug report)


>> "Oliver Elphick" <olly@lfix.co.uk> writes:
>>> Can someone advise, please, how to deal with this problem in 6.5.3?
>> 

>> My guess is that the cause is memory leaks during expression evaluation;
>> but without seeing the complete view definitions and underlying table
>> definitions, it's impossible to know what processing is being invoked
>> by this query...
>> 
>> regards, tom lane



>     Well, I will append views and underlying table definition:

> 1) Once again - failure query:
> select comm_type,name,tot_bytes,tot_packets
> from flow_sums_days_send_200002_view
> where day='2000-02-21' and name not like '@%'
> union all
> select comm_type,name,tot_bytes,tot_packets
> from flow_sums_days_receive_200002_view
> where day='2000-02-21' and name not like '@%'

> 2) views definition:
> create view flow_sums_days_send_200002_view as
> select
>   'send'::varchar as comm_type, date_trunc('day',start) as day,
>   src_name as name, sum(bytes) as tot_bytes, sum(packets) as tot_packets
> from flow_sums_200002
> group by day, src_name

> create view flow_sums_days_receive_200002_view as
> select
>   'receive'::varchar as comm_type, date_trunc('day',start) as day,
>   dst_name as name, sum(bytes) as tot_bytes, sum(packets) as tot_packets
> from flow_sums_200002
> group by day, dst_name


> I wanted create only one usefull view:

> create view flow_sums_days_200002_view as
> select
>   'send'::varchar as comm_type, date_trunc('day',start) as day,
>   src_name as name, sum(bytes) as tot_bytes, sum(packets) as tot_packets
> from flow_sums_200002
> group by day, src_name
> UNION ALL
> select
>   'receive'::varchar as comm_type, date_trunc('day',start) as day,
>   dst_name as name, sum(bytes) as tot_bytes, sum(packets) as tot_packets
> from flow_sums_200002
> group by day, dst_name

> ...but Postgres cann't use clause UNION ALL at view definition. So I created
> two views mentioned above and I wanted use this ones with UNION ALL clause
> only.

> 3) underlaying table definition:
> create table flow_sums_200002 (
>     primary_collector varchar(50) not null,
>     start datetime not null,
>     end_period datetime not null,
>     dead_time_rel float4 not null,
>     src_name varchar(50) not null,
>     dst_name varchar(50) not null,
>     bytes int8 not null,
>     packets int4 not null
> )

>     Today this table has about 3 000 000 rows and the select command
> mentioned above returns 190 + 255 rows.


>     Now I don't use clause "UNION ALL" and the program executes two queryes
> and then adds both result to new result. I reduced time increment of number
> rows to flow_sums_200002 table (three times less). This table contains data
> of February 2000 and the program will create table flow_sums_200003 with
> relevant views next month.
>     Well, now this solution solve my problem but always depends on number of
> rows - I only moved limit of rows count.


>             Thank You, V. Benes

> P.S.: I append part of top on my system while the query is running:

> CPU states: 98.6% user,  1.3% system,  0.0% nice,  0.0% idle
> Mem:  127256K av, 124316K used,   2940K free,  29812K shrd,   2620K buff
> Swap: 128516K av,  51036K used,  77480K free                  7560K cached

>   PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
> 2942 postgres  20   0  141M  99M 17348 R       0 99.0 80.4   1:22 postmaster

> => postmaster later took 80 - 95% of memory, free memory decressed to 2 MB,
> CPU was overloaded (0% idle and 99% by user process of postmaster). Have You
> ever seen something similar :-) ?


Thu, 24 Feb 2000 00:45:56 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Out of memory problem (forwarded bug report) 
Tom Lane wrote:
...
  >your sum would overflow an int4?  It may be that just fixing the
  >inefficient date_trunc calc will reduce your memory requirements
  >enough to get by.  If not, the only good news I have is that release
  >7.0 does fix the memory-leak problem for internal calculations of
  >aggregate functions like sum().  You can get the first beta release
  >for 7.0 now.

I'm putting together a Debian release of the beta at the moment. 

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Thy word is a lamp unto my feet, and a light unto my 
      path."           Psalms 119:105 



Thu, 24 Feb 2000 11:22:38 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] Out of memory problem (forwarded bug report) 
Don't know if you know this already, but since april 23, you've been
on SecurityFocus.com for the cleartext passwords in pg_shadow:

    http://www.securityfocus.com/bid/1139

I know it has been discussed at least a couple of times before, but in
my opinion this is an issue that needs a solution.

The problem with cleartext passwords is not just that root, postgres
super user or anyone who has legally or illegally got access to the
system can see the passwords a user uses to log in to PostgreSQL.  The
problem lies in the well known fact that we tend to use the same
password several places, if not everywhere.  With all the passwords
needed these days, that is how it _has_ to be.

The first PostgreSQL based site that gets cracked, will make headlines
stating that passwords have got into the wrong hands.  Do we (or you)
want that?


Sverre.

-- 
<URL:mailto:sverrehu@online.no>
<URL:http://home.sol.no/~sverrehu/>          Echelon bait: semtex, bin Laden,
                                         plutonium, North Korea, nuclear bomb

Sat, 6 May 2000 00:40:24 +0200
"Sverre H. Huseby" <sverrehu@online.no>
You're on SecurityFocus.com for the cleartext passwords.
On Sat, 6 May 2000, Sverre H. Huseby wrote:

> Don't know if you know this already, but since april 23, you've been
> on SecurityFocus.com for the cleartext passwords in pg_shadow:
> 
>     http://www.securityfocus.com/bid/1139
> 
> I know it has been discussed at least a couple of times before, but in
> my opinion this is an issue that needs a solution.
> 
> The problem with cleartext passwords is not just that root, postgres
> super user or anyone who has legally or illegally got access to the
> system can see the passwords a user uses to log in to PostgreSQL.  The
> problem lies in the well known fact that we tend to use the same
> password several places, if not everywhere.  With all the passwords
> needed these days, that is how it _has_ to be.
> 
> The first PostgreSQL based site that gets cracked, will make headlines
> stating that passwords have got into the wrong hands.  Do we (or you)
> want that?

You've lost me here ... the only person(s) that can get at those passwords
are those that have compromised the system already.  Even if the passwords
*weren't* in cleartext, there is nothing that stops me from downloading
the data/* directory down to my computer and running pg_upgrade to "make
it my own", removing the passwords ... 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Fri, 5 May 2000 20:25:10 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: You're on SecurityFocus.com for the cleartext passwords.
On Fri, 5 May 2000, The Hermit Hacker wrote:
> You've lost me here ... the only person(s) that can get at those passwords
> are those that have compromised the system already.  Even if the passwords
> *weren't* in cleartext, there is nothing that stops me from downloading
> the data/* directory down to my computer and running pg_upgrade to "make
> it my own", removing the passwords ... 

You don't get it. Its one of most basic things about security of the
password databases: Cleartext must not be available for anyone, not even
the administrators. The damage one can do with list of 10000 passwords
far exceeds damage you can do to the database which contain these
passwords. Why? Because people tend to use same password everywhere. 

(Yes, I know that they shouldn't, however, you must take good care of
passwords users entrusted to you). 

There is no excuse for not storing it as a hash or at least in crypt(3)
way.

-alex


Fri, 5 May 2000 19:39:15 -0400 (EDT)
Alex Pilosov <alex@pilosoft.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
On Fri, 5 May 2000, The Hermit Hacker wrote:

> On Sat, 6 May 2000, Sverre H. Huseby wrote:
> 
> > Don't know if you know this already, but since april 23, you've been
> > on SecurityFocus.com for the cleartext passwords in pg_shadow:
> > 
> >     http://www.securityfocus.com/bid/1139
> > 
> > I know it has been discussed at least a couple of times before, but in
> > my opinion this is an issue that needs a solution.
> > 
> > The problem with cleartext passwords is not just that root, postgres
> > super user or anyone who has legally or illegally got access to the
> > system can see the passwords a user uses to log in to PostgreSQL.  The
> > problem lies in the well known fact that we tend to use the same
> > password several places, if not everywhere.  With all the passwords
> > needed these days, that is how it _has_ to be.
> > 
> > The first PostgreSQL based site that gets cracked, will make headlines
> > stating that passwords have got into the wrong hands.  Do we (or you)
> > want that?
> 
> You've lost me here ... the only person(s) that can get at those passwords
> are those that have compromised the system already.  Even if the passwords
> *weren't* in cleartext, there is nothing that stops me from downloading
> the data/* directory down to my computer and running pg_upgrade to "make
> it my own", removing the passwords ... 

Same defense I used when I responded to the BugTRAQ post.  Even tho I 
understand the possible ramifications of cleartext passwords, I still
stand by my previous comments, an admin needs to properly maintain and
protect the systems they're entrusted to.  However after reading about
the www.apache.org compromise details earlier today I'm of the opinion
now that we should look into encrypting the passwords.  I'm also of the
opinion that I should volunteer to at least help in the fixing of it.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Fri, 5 May 2000 21:12:56 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
Vince Vielhaber <vev@michvhf.com> writes:
> ... I'm of the opinion now that we should look into encrypting the
> passwords.

I think it'd be a reasonable thing to work on.  I don't particularly
intend to be stampeded into doing something about it by "public
relations" pressure from people who would rather make inflated claims
than get their hands dirty by contributing a solution ;-).  (And, yes,
these claims are inflated.  If you don't trust your dbadmin, the
security of your password is the least of your worries --- the data
in your database may well be far more critical info than anything the
dbadmin could find in your personal account.  The general opinion on
the pghackers list has been that password-based security is the least
desirable of the authentication options we offer, anyway.  A security-
conscious site wouldn't even be using database passwords.)

The main potential hazard I see is portability.  Is crypt(3) available
on *all* the platforms Postgres runs on?  Does it give the same answers
on all those platforms?  If not, what shall we use instead?  Don't
forget that the frontend libraries have to have it too (or are you going
to keep transmitting passwords in cleartext?).  So that means you'd
better have it for Win, Mac, BeOS, etc, not just for dozens of Unix
variants --- and they *must* all give the same results.

There are also lesser worries about patents and US export regulations.
If we include an encryption package in the distribution we could
eliminate the portability problem, only to find ourselves facing
headaches in those departments :-(

So, by all means let's look for a solution ... but I suspect that
the cost/benefit ratio of fixing this is a lot higher than is being
claimed in some quarters.

			regards, tom lane

Sat, 06 May 2000 01:47:01 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
I wrote:
> The main potential hazard I see is portability.  Is crypt(3) available
> on *all* the platforms Postgres runs on?

Waitasec, what am I saying?  We already *do* have crypt password
support, at least on those platforms where crypt(3) is available.

As near as I can tell, the crypt option transmits an encrypted password
across the wire (good), but the comparison at the server end is done by
taking the cleartext password stored in pg_shadow, crypt()ing it on
the fly, and comparing that to what was sent by the client.

This does have the advantage that the same pg_shadow entry will support
both cleartext-password and crypted-password connections, but we could
get that another way.  Assuming that the server has crypt(), the
password could be stored always encrypted instead of always not.
Cleartext-password connections would be handled just by crypting the
received password before comparing.  (Before you ask, no I don't think
we should remove the option of cleartext-password connections.  What of
clients running on platforms with neither crypt() nor anything better
like Kerberos?  Should they be forced to drop down to no security at
all?  I think not.)

This'd take some rejiggering in (at least) CREATE USER and ALTER USER,
but it seems doable.  I withdraw the complaint about portability...

			regards, tom lane

Sat, 06 May 2000 03:09:15 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
[Tom Lane]

|   If you don't trust your dbadmin, the security of your password is
|   the least of your worries --- the data in your database may well
|   be far more critical info than anything the dbadmin could find in
|   your personal account.

It may, and then again, it may not.  There are lots of databases out
there that do not contain secret or critical data.  All databases I
have made fall into this category.  But the password I use on my
PostgreSQL account is (or used to be, until I discovered the cleartext
passwords) the same password I use most other places. I don't care if
anyone reads the data, as long as they don't start testing my password
on all other sites they may guess I have access to.  I have my
PostgreSQL database on an ISP on the other side of the globe.  Why
should I trust those people more than, say, my neighbour?

|   The main potential hazard I see is portability.  Is crypt(3) available
|   on *all* the platforms Postgres runs on?  Does it give the same answers
|   on all those platforms?  If not, what shall we use instead?

I implemented MD5 in Java a couple of years ago.  I'm sure me or
someone else will be able to convert it to C.  I'll make the license
anything you want it to be if you care to use it.

|   There are also lesser worries about patents and US export regulations.
|   If we include an encryption package in the distribution we could
|   eliminate the portability problem, only to find ourselves facing
|   headaches in those departments :-(

AFAIK, MD5 is not restricted, as it can't be used for
encryption/decryption.  It is a one way hashing function only.  Please
correct me if I am wrong, I never understood those stupid export
regulations anyway.


Sverre - who really do not want _anyone_ to know his passwords.

-- 
<URL:mailto:sverrehu@online.no>
<URL:http://home.sol.no/~sverrehu/>          Echelon bait: semtex, bin Laden,
                                         plutonium, North Korea, nuclear bomb

Sat, 6 May 2000 09:09:33 +0200
"Sverre H. Huseby" <sverrehu@online.no>
Re: You're on SecurityFocus.com for the cleartext passwords.
> I wrote:
> > The main potential hazard I see is portability.  Is crypt(3) available
> > on *all* the platforms Postgres runs on?
> 
> Waitasec, what am I saying?  We already *do* have crypt password
> support, at least on those platforms where crypt(3) is available.
> 
> As near as I can tell, the crypt option transmits an encrypted password
> across the wire (good), but the comparison at the server end is done by
> taking the cleartext password stored in pg_shadow, crypt()ing it on
> the fly, and comparing that to what was sent by the client.
> 
> This does have the advantage that the same pg_shadow entry will support
> both cleartext-password and crypted-password connections, but we could
> get that another way.  Assuming that the server has crypt(), the
> password could be stored always encrypted instead of always not.
> Cleartext-password connections would be handled just by crypting the
> received password before comparing.  (Before you ask, no I don't think
> we should remove the option of cleartext-password connections.  What of
> clients running on platforms with neither crypt() nor anything better
> like Kerberos?  Should they be forced to drop down to no security at
> all?  I think not.)
> 
> This'd take some rejiggering in (at least) CREATE USER and ALTER USER,
> but it seems doable.  I withdraw the complaint about portability...

Yes, agreed.  Doing it in the backend only is the way to go.  We already
have wire crypting.

I think the only problem is moving dumps from on machine to another. 
The crypt version may not exist or be different on different machines.

However, I now remember there was a bigger issue.  I think the actual
password has to be crypted based on the salt used supplied to the
client.  We can't do that based on the crypted version because we don't
know the client can generate that version.

Now, at the time, we were looking at Unix-style crypting of the
password, which is one-way.  This will not work.  We need something that
we can uncrypt in the backend before applying the client-supplied salt
to see if the passwords match.

The goal here was to make wire sniffing unproductive, and because the
server supplied the salt to be used by the client, you can't just
re-use a sniffed password you saw on the wire.

At least this is my recollection of the problem.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 10:25:38 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
On Sat, 6 May 2000, Bruce Momjian wrote:

> > This'd take some rejiggering in (at least) CREATE USER and ALTER USER,
> > but it seems doable.  I withdraw the complaint about portability...
> 
> Yes, agreed.  Doing it in the backend only is the way to go.  We already
> have wire crypting.
> 
> I think the only problem is moving dumps from on machine to another. 
> The crypt version may not exist or be different on different machines.
> 
> However, I now remember there was a bigger issue.  I think the actual
> password has to be crypted based on the salt used supplied to the
> client.  We can't do that based on the crypted version because we don't
> know the client can generate that version.
> 
> Now, at the time, we were looking at Unix-style crypting of the
> password, which is one-way.  This will not work.  We need something that
> we can uncrypt in the backend before applying the client-supplied salt
> to see if the passwords match.
> 
> The goal here was to make wire sniffing unproductive, and because the
> server supplied the salt to be used by the client, you can't just
> re-use a sniffed password you saw on the wire.
> 
> At least this is my recollection of the problem.
> 
> 

We can do it with MD5.  Sverre has offered up a java version of it
that he wrote, I can convert it to C and make sure it at least runs
on FreeBSD, IRIX, DOS/Windows, and HPUX 8-10.  If it runs in unix then
it should also run in OS/2.  If we roll our own we should be safe.  I
can even include a simple test to make sure it works for all platforms
we support.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sat, 6 May 2000 10:53:25 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
> > The goal here was to make wire sniffing unproductive, and because the
> > server supplied the salt to be used by the client, you can't just
> > re-use a sniffed password you saw on the wire.
> > 
> > At least this is my recollection of the problem.
> > 
> > 
> 
> We can do it with MD5.  Sverre has offered up a java version of it
> that he wrote, I can convert it to C and make sure it at least runs
> on FreeBSD, IRIX, DOS/Windows, and HPUX 8-10.  If it runs in unix then
> it should also run in OS/2.  If we roll our own we should be safe.  I
> can even include a simple test to make sure it works for all platforms
> we support.

Yes, I seem to remember that was the issue.  If we only did crypting on
the server, and allowed passwords to come cleartext from clients, then
we only needed crypting on the server.  If we crypt in a one-way fashion
on the client before coming to the server using a random salt, we have
to do the other part of the crypting on the client too.

In other words, it is the one-way nature of the password crypt we used
on the client that caused us to need the _exact_ same input string to
go into that crypt on the client and server, so we would need the same
crypt process in both places.

Now, let me ask another, better question:

Right now the password receives a random salt from the server, it uses
that salt to crypt the password, then send that back for comparison with
the clear-text password we store in the system.

What if we:
	store the password in pg_shadow like a unix-style password with salt
	pass the random salt and the salt from pg_shadow to the client
	client crypts the password twice through the routine:
		once using the pg_shadow salt
		another time using the random salt

and passes that back to the server.  The server can use the pg_shadow
copy of the password, use the random salt make a new version, and
compare the result.

This has the huge advantage of not requiring any new crypting methods on
the client.  It only requires the crypt to happen twice using two
different salts.

Sounds like a winner.  Comments?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 11:17:52 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
On Sat, 6 May 2000, Bruce Momjian wrote:

> > We can do it with MD5.  Sverre has offered up a java version of it
> > that he wrote, I can convert it to C and make sure it at least runs
> > on FreeBSD, IRIX, DOS/Windows, and HPUX 8-10.  If it runs in unix then
> > it should also run in OS/2.  If we roll our own we should be safe.  I
> > can even include a simple test to make sure it works for all platforms
> > we support.
> 
> Yes, I seem to remember that was the issue.  If we only did crypting on
> the server, and allowed passwords to come cleartext from clients, then
> we only needed crypting on the server.  If we crypt in a one-way fashion
> on the client before coming to the server using a random salt, we have
> to do the other part of the crypting on the client too.
> 
> In other words, it is the one-way nature of the password crypt we used
> on the client that caused us to need the _exact_ same input string to
> go into that crypt on the client and server, so we would need the same
> crypt process in both places.
> 
> Now, let me ask another, better question:
> 
> Right now the password receives a random salt from the server, it uses
> that salt to crypt the password, then send that back for comparison with
> the clear-text password we store in the system.
> 
> What if we:
> 	store the password in pg_shadow like a unix-style password with salt
> 	pass the random salt and the salt from pg_shadow to the client
> 	client crypts the password twice through the routine:
> 		once using the pg_shadow salt
> 		another time using the random salt
> 
> and passes that back to the server.  The server can use the pg_shadow
> copy of the password, use the random salt make a new version, and
> compare the result.
> 
> This has the huge advantage of not requiring any new crypting methods on
> the client.  It only requires the crypt to happen twice using two
> different salts.
> 
> Sounds like a winner.  Comments?

Overlycomplicated?

What was your objection to MD5 again?

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sat, 6 May 2000 11:38:03 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
> > This has the huge advantage of not requiring any new crypting methods on
> > the client.  It only requires the crypt to happen twice using two
> > different salts.
> > 
> > Sounds like a winner.  Comments?
> 
> Overlycomplicated?
> 
> What was your objection to MD5 again?

Not really.  We are using our same unix crypt code, which works on all
platforms.  The change to each interface is minimal.  Not much testing
will be required.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 11:48:41 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
> > Sounds like a winner.  Comments?
> 
> Overlycomplicated?
> 
> What was your objection to MD5 again?

Also, MD5 is not ideal for passwords.  Seems the standard unix-style
password crypting is the standard, so it should be used to crypt our own
passwords in pg_shadow.  I am sure someone would find some problem with
us using md5 for password storage.

We already use the unix-style password crypt to send passwords over the
wire.  Why not use it for storage too?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 11:54:03 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
On Sat, 6 May 2000, Bruce Momjian wrote:

> > > Sounds like a winner.  Comments?
> > 
> > Overlycomplicated?
> > 
> > What was your objection to MD5 again?
> 
> Also, MD5 is not ideal for passwords.  Seems the standard unix-style
> password crypting is the standard, so it should be used to crypt our own
> passwords in pg_shadow.  I am sure someone would find some problem with
> us using md5 for password storage.

FreeBSD uses MD5 by default since at least ver 2.2, possibly earlier.
 
> We already use the unix-style password crypt to send passwords over the
> wire.  Why not use it for storage too?

Can ALL clients we support use it over the wire?  

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sat, 6 May 2000 12:09:13 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
> On Sat, 6 May 2000, Bruce Momjian wrote:
> 
> > > > Sounds like a winner.  Comments?
> > > 
> > > Overlycomplicated?
> > > 
> > > What was your objection to MD5 again?
> > 
> > Also, MD5 is not ideal for passwords.  Seems the standard unix-style
> > password crypting is the standard, so it should be used to crypt our own
> > passwords in pg_shadow.  I am sure someone would find some problem with
> > us using md5 for password storage.
> 
> FreeBSD uses MD5 by default since at least ver 2.2, possibly earlier.

Oh, I didn't know that.  Interesting.

>  
> > We already use the unix-style password crypt to send passwords over the
> > wire.  Why not use it for storage too?
> 
> Can ALL clients we support use it over the wire?  

That is an excellent question.  Any client that can use passwords has to
do this, so yes, I think they all do.  I can say for sure Java has it,
and that is usually the hardest.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 12:10:54 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
> > Also, MD5 is not ideal for passwords.  Seems the standard unix-style
> > password crypting is the standard, so it should be used to crypt our own
> > passwords in pg_shadow.  I am sure someone would find some problem with
> > us using md5 for password storage.
> 
> FreeBSD uses MD5 by default since at least ver 2.2, possibly earlier.
>  
> > We already use the unix-style password crypt to send passwords over the
> > wire.  Why not use it for storage too?
> 
> Can ALL clients we support use it over the wire?  

Yes, I think so.  Java has its own, and the others use libpq do to it. 
The beauty of my suggesting is that all we have to do is pass the
pg_shadow salt along with the random salt, and call the crypt code
twice, first with the pg_shadow salt, then with the random salt.

The server pass the pg_shadow version through the random salt crypt, and
compares.

Now, I we want to move all the stuff to use MD5 rather than the standard
unix password crypt, that is another option, though I am not sure what
value it would have.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 12:28:53 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
[Bruce Momjian]

|   	store the password in pg_shadow like a unix-style password with salt
|   	pass the random salt and the salt from pg_shadow to the client
|   	client crypts the password twice through the routine:
|   		once using the pg_shadow salt
|   		another time using the random salt

That's close to what I thought of a couple of days ago too, except I
would have used MD5, since I already have that implemented. :) (It
seems you already have crypt, so you wouldn't need MD5.)

Does anyone here really _know_ (and I mean KNOW)
security/cryptography?  If so, could you please comment on this
scheme?  And while you're at it, whats better of MD5 and Unix crypt
(triple DES ++, isn't it?) from a security perspective?


Sverre.

-- 
<URL:mailto:sverrehu@online.no>
<URL:http://home.sol.no/~sverrehu/>          Echelon bait: semtex, bin Laden,
                                         plutonium, North Korea, nuclear bomb

Sat, 6 May 2000 18:45:26 +0200
"Sverre H. Huseby" <sverrehu@online.no>
Re: You're on SecurityFocus.com for the cleartext passwords.
On Sat, 6 May 2000, Bruce Momjian wrote:

> > > Also, MD5 is not ideal for passwords.  Seems the standard unix-style
> > > password crypting is the standard, so it should be used to crypt our own
> > > passwords in pg_shadow.  I am sure someone would find some problem with
> > > us using md5 for password storage.
> > 
> > FreeBSD uses MD5 by default since at least ver 2.2, possibly earlier.
> >  
> > > We already use the unix-style password crypt to send passwords over the
> > > wire.  Why not use it for storage too?
> > 
> > Can ALL clients we support use it over the wire?  
> 
> Yes, I think so.  Java has its own, and the others use libpq do to it. 
> The beauty of my suggesting is that all we have to do is pass the
> pg_shadow salt along with the random salt, and call the crypt code
> twice, first with the pg_shadow salt, then with the random salt.
> 
> The server pass the pg_shadow version through the random salt crypt, and
> compares.
> 
> Now, I we want to move all the stuff to use MD5 rather than the standard
> unix password crypt, that is another option, though I am not sure what
> value it would have.
> 
> 

How about ODBC?  This is from the ODBC driver source connection.c:
                                
     self->errormsg = "Password crypt authentication not supported";

Is that because of the platform it's running on or what it's talking
to?

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sat, 6 May 2000 12:56:57 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
> > Now, I we want to move all the stuff to use MD5 rather than the standard
> > unix password crypt, that is another option, though I am not sure what
> > value it would have.
> > 
> > 
> 
> How about ODBC?  This is from the ODBC driver source connection.c:
>                                 
>      self->errormsg = "Password crypt authentication not supported";
> 
> Is that because of the platform it's running on or what it's talking
> to?

Seems we don't have crypt support, so you can't send crypt passwords
from an ODBC client.  That is news to me.

 From looking there, and looking at pg_hba.conf, we have both 'password'
and 'crypt' authentication in there.  

However, this is not a problem because we can still do backend-only
crypting when comparing client-sent cleartext passwords to pg_shadow
passwords.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 13:02:47 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
Vince Vielhaber <vev@michvhf.com> writes:

> FreeBSD uses MD5 by default since at least ver 2.2, possibly
> earlier.

So does Red Hat Linux, and probably Linux distributions as well. 

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

06 May 2000 13:03:10 -0400
teg@redhat.com (Trond Eivind=?iso-8859-1?q?_Glomsr=F8d?=)
Re: You're on SecurityFocus.com for the cleartext passwords.
on 5/6/00 12:45 PM, Sverre H. Huseby at sverrehu@online.no wrote:

> Does anyone here really _know_ (and I mean KNOW)
> security/cryptography?  If so, could you please comment on this
> scheme?  And while you're at it, whats better of MD5 and Unix crypt
> (triple DES ++, isn't it?) from a security perspective?

Finally something I can comment on with a tiny bit of authority :)

The unix crypt command is a sneaky version of DES (I've never heard of
Triple-DES being used for this). Your password is transformed into a DES key
which is then used to encrypt a block of 0's. The result is what's stored in
the password file. Poor Man's Hash, in a sense :)

MD5 is quite standard (as hashing algs go) and much more secure. It allows
for longer passwords, and it's quite fast (easily tens of thousands of MD5
hashes per second on today's midlevel processors). I strongly recommend you
use that.

|       store the password in pg_shadow like a unix-style password with salt
|       pass the random salt and the salt from pg_shadow to the client
|       client crypts the password twice through the routine:
|           once using the pg_shadow salt
|           another time using the random salt

My first impression of this scheme is that it's quite good. Use MD5 instead
of crypt, and it's great. You've got a good challenge-response setup here,
and with MD5 you can even make your salt much longer than the 2 bytes of
unix crypt salt, thus much more secure.

I like it!

-Ben


Sat, 06 May 2000 13:17:22 -0400
Benjamin Adida <ben@mit.edu>
Re: You're on SecurityFocus.com for the cleartext
	passwords.
On Sat, 6 May 2000, Bruce Momjian wrote:

> > > Now, I we want to move all the stuff to use MD5 rather than the standard
> > > unix password crypt, that is another option, though I am not sure what
> > > value it would have.
> > > 
> > > 
> > 
> > How about ODBC?  This is from the ODBC driver source connection.c:
> >                                 
> >      self->errormsg = "Password crypt authentication not supported";
> > 
> > Is that because of the platform it's running on or what it's talking
> > to?
> 
> Seems we don't have crypt support, so you can't send crypt passwords
> from an ODBC client.  That is news to me.
> 
> >From looking there, and looking at pg_hba.conf, we have both 'password'
> and 'crypt' authentication in there.  
> 
> However, this is not a problem because we can still do backend-only
> crypting when comparing client-sent cleartext passwords to pg_shadow
> passwords.

But what I'm proposing will let ALL clients send an encrypted password
over the wire and we can also store them encrypted.  By comparing twice
we can maintain backward compatibility.  The backend would compare the
password received with the stored md5 password and compare the received
password after md5ing it in case it was sent clear-text.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sat, 6 May 2000 13:19:16 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
> But what I'm proposing will let ALL clients send an encrypted password
> over the wire and we can also store them encrypted.  By comparing twice
> we can maintain backward compatibility.  The backend would compare the
> password received with the stored md5 password and compare the received
> password after md5ing it in case it was sent clear-text.

But you can do that with our current system.  Store them in pg_shadow
using unix password format.  If a cleartext password comes in, crypt it
using the pg_shadow salt and compare them.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 13:21:16 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
> on 5/6/00 12:45 PM, Sverre H. Huseby at sverrehu@online.no wrote:
> 
> > Does anyone here really _know_ (and I mean KNOW)
> > security/cryptography?  If so, could you please comment on this
> > scheme?  And while you're at it, whats better of MD5 and Unix crypt
> > (triple DES ++, isn't it?) from a security perspective?
> 
> Finally something I can comment on with a tiny bit of authority :)
> 
> The unix crypt command is a sneaky version of DES (I've never heard of
> Triple-DES being used for this). Your password is transformed into a DES key
> which is then used to encrypt a block of 0's. The result is what's stored in
> the password file. Poor Man's Hash, in a sense :)
> 
> MD5 is quite standard (as hashing algs go) and much more secure. It allows
> for longer passwords, and it's quite fast (easily tens of thousands of MD5
> hashes per second on today's midlevel processors). I strongly recommend you
> use that.
> 
> |       store the password in pg_shadow like a unix-style password with salt
> |       pass the random salt and the salt from pg_shadow to the client
> |       client crypts the password twice through the routine:
> |           once using the pg_shadow salt
> |           another time using the random salt
> 
> My first impression of this scheme is that it's quite good. Use MD5 instead
> of crypt, and it's great. You've got a good challenge-response setup here,
> and with MD5 you can even make your salt much longer than the 2 bytes of
> unix crypt salt, thus much more secure.
> 
> I like it!
> 

Good.  I only recommend our current setup because we already have code
in most interfaces to handle it.  I have no problem moving to md5, but
this should be done for _all_ crypting.  I just see no reason to mix
standard password crypt with md5 and try to keep two crypts working on
all interfaces.  The easy way would be to use our current crypt stuff to
get it working, then move to md5 if we can get it working on all our
interfaces.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 13:23:52 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext     passwords.
On Sat, 6 May 2000, Bruce Momjian wrote:

> > But what I'm proposing will let ALL clients send an encrypted password
> > over the wire and we can also store them encrypted.  By comparing twice
> > we can maintain backward compatibility.  The backend would compare the
> > password received with the stored md5 password and compare the received
> > password after md5ing it in case it was sent clear-text.
> 
> But you can do that with our current system.  Store them in pg_shadow
> using unix password format.  If a cleartext password comes in, crypt it
> using the pg_shadow salt and compare them.

You missed half of it.  Platforms that don't have crypt would use our
MD5 so eventually all of them would be sending encrypted passwords 
over the wire.  I'm trying to accomplish two things here.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sat, 6 May 2000 13:25:18 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>>>> The goal here was to make wire sniffing unproductive, and because the
>>>> server supplied the salt to be used by the client, you can't just
>>>> re-use a sniffed password you saw on the wire.

Good point.  Are we trying to make the password challenge proof against
wire sniffing?  I think that might be an unreasonable goal.  People who
are concerned about sniffing attacks ought to be using something like
SSL to encrypt the *entire* connection (and/or a better auth protocol
than passwords, in the first place...).  I repeat my prior comment that
the data in the database is likely to be just as valuable as the
password.

> Right now the client receives a random salt from the server, it uses
> that salt to crypt the password, then send that back for comparison with
> the clear-text password we store in the system.

> What if we:
> 	store the password in pg_shadow like a unix-style password with salt
> 	pass the random salt and the salt from pg_shadow to the client
> 	client crypts the password twice through the routine:
> 		once using the pg_shadow salt
> 		another time using the random salt

> and passes that back to the server.  The server can use the pg_shadow
> copy of the password, use the random salt make a new version, and
> compare the result.

Use the random salt to make a new version of what, exactly?  If the
server doesn't have the cleartext password, it can't make a crypted
password that will correspond to any randomly chosen salt either.

> This has the huge advantage of not requiring any new crypting methods on
> the client.  It only requires the crypt to happen twice using two
> different salts.

A serious objection to both this and the MD5 proposal is that it'd
create a cross-version incompatibility between clients and servers.
We were just fending off complaints about 6.5-to-7.0 incompatibilities
that were considerably less serious than being unable to connect at all,
so how well do you think this'd go over?

I think we should try to stick to the current protocol: one salt sent
by the server, one crypted password sent back.  The costs of changing
the protocol will probably outweigh any real-world security gain.

We could get some of the benefits of a random salt if we were willing
to enlarge the pg_shadow entry a little bit.  Suppose that we allow
N different salt values to be sent by the server (crypt(3) only allows
4096 possible salts anyway, but I'm thinking N in the range of 100).
When a password is set, crypt the password with each of these and store
*all* the results in pg_shadow.  Then we have the right crypted password
available to compare to the client response, and an attacker still has
a relatively low probability of having sniffed the right password.

BTW, I hear "MD5" being chanted like a mantra, but someone will have
to explain to me what it does that avoids the sniffed-crypted-password
problem...

			regards, tom lane

Sat, 06 May 2000 13:29:01 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
> On Sat, 6 May 2000, Bruce Momjian wrote:
> 
> > > But what I'm proposing will let ALL clients send an encrypted password
> > > over the wire and we can also store them encrypted.  By comparing twice
> > > we can maintain backward compatibility.  The backend would compare the
> > > password received with the stored md5 password and compare the received
> > > password after md5ing it in case it was sent clear-text.
> > 
> > But you can do that with our current system.  Store them in pg_shadow
> > using unix password format.  If a cleartext password comes in, crypt it
> > using the pg_shadow salt and compare them.
> 
> You missed half of it.  Platforms that don't have crypt would use our
> MD5 so eventually all of them would be sending encrypted passwords 
> over the wire.  I'm trying to accomplish two things here.

That is fine:  We need crypted passwords in pg_shadow, and MD5 is
probably better than our current setup.

But we have tons of interfaces, all of which use the old stuff.  If you
think you can do both at the same time, go ahead.  MD5 has salt
capability, so you can move it right into our current client dialog
setup, and do double-MD5 as I suggested.

You still need double-MD5 because you have to crypt the password based
on the random salt passed to the client by the server.  If you can make
the salt larger than 2 bytes at the same time, so much the better.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 13:30:24 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
I said:
> I think we should try to stick to the current protocol: one salt sent
> by the server, one crypted password sent back.  The costs of changing
> the protocol will probably outweigh any real-world security gain.

Actually, since libpq handles the authentication phase of connection
via a state-machine, it'd be possible for the postmaster to send two
successive authentication challenge packets with different salts, and
libpq would respond correctly to each one.  This is a little bit shaky
because the current protocol document does not say that clients should
loop at the challenge point of the protocol, so there might be non-libpq
clients that wouldn't cope.  But it's possible we could do it without
breaking compatibility with old clients.

However, I still fail to see what it buys us to challenge the frontend
with two salts.  If the password is stored crypted, the *only* thing
we can validate is that password with the same salt it was stored
with.  It doesn't sound like MD5 changes this at all.

			regards, tom lane

Sat, 06 May 2000 14:14:13 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
on 5/6/00 2:14 PM, Tom Lane at tgl@sss.pgh.pa.us wrote:

> However, I still fail to see what it buys us to challenge the frontend
> with two salts.  If the password is stored crypted, the *only* thing
> we can validate is that password with the same salt it was stored
> with.  It doesn't sound like MD5 changes this at all.

The MD5 definitely doesn't change anything except overall security strength
of the algorithm. The additional random salt prevents someone from sniffing
the communication between client and server and then simply log in by
sending the known hash of the password. The challenge-response means that
sniffing one login doesn't allow you to fake the next one.

-Ben


Sat, 06 May 2000 14:21:10 -0400
Benjamin Adida <ben@mit.edu>
Re: You're on SecurityFocus.com for the cleartext
	passwords.
On Sat, 06 May 2000, Trond Eivind Glomsr�d wrote:
> Vince Vielhaber <vev@michvhf.com> writes:
> 
> > FreeBSD uses MD5 by default since at least ver 2.2, possibly
> > earlier.
> 
> So does Red Hat Linux, and probably Linux distributions as well. 
> 
> -- 
> Trond Eivind Glomsr�d
> Red Hat, Inc.

There is some good information about crypt and MD5 at www.php.net in the
documentation: String Functions/crypt

http://www.php.net/manual/function.crypt.php

It explains that many systems have updated crypt() to use MD5 and how to
check what hash algorithm your system's crypt() actually uses.

-- 
Robert B. Easter
reaster@comptechnews.com

Sat, 6 May 2000 14:26:09 -0400
"Robert B. Easter" <reaster@comptechnews.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
Benjamin Adida <ben@mit.edu> writes:
>> It doesn't sound like MD5 changes this at all.

> The MD5 definitely doesn't change anything except overall security strength
> of the algorithm.

OK, understood.  So it seems that switching to MD5 would offer (a) more
portability to platforms without crypt(3), and (b) better security,
at the costs of (a) implementation effort and (b) cross-version
compatibility problems.  We probably ought to keep that discussion
separate from the one about how the challenge protocol works.

> The additional random salt prevents someone from sniffing
> the communication between client and server and then simply log in by
> sending the known hash of the password. The challenge-response means that
> sniffing one login doesn't allow you to fake the next one.

How so?  The server sends out one fixed salt (the one stored for that
user's password in pg_shadow) and one randomly-chosen salt.  The client
sends back two crypted passwords.  The server can check one of them.
What can it do with the other?  Nothing that I can see, so where is the
security gain?  A sniffer can still get in by sending back the same
pair of crypted passwords next time, no matter what random salt is
presented.

			regards, tom lane

Sat, 06 May 2000 14:29:22 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
on 5/6/00 2:29 PM, Tom Lane at tgl@sss.pgh.pa.us wrote:

> How so?  The server sends out one fixed salt (the one stored for that
> user's password in pg_shadow) and one randomly-chosen salt.  The client
> sends back two crypted passwords.  The server can check one of them.
> What can it do with the other?  Nothing that I can see, so where is the
> security gain?  A sniffer can still get in by sending back the same
> pair of crypted passwords next time, no matter what random salt is
> presented.

Okay, my understanding was that the protocol would work as follows:

- client requests login
- server sends stored salt c1, and random salt c2.
- client performs hash_c2(hash_c1(password)) and sends result to server.
- server performs hash_c2(stored_pg_shadow) and compares with client
submission.
- if there's a match, there's successful login.

This protocol will truly create a challenge-response where the communication
is different at each login, and where sniffing one
hash_c2(hash_c1(password)) doesn't give you any way to log in with a
different c2.

-Ben


Sat, 06 May 2000 14:32:24 -0400
Benjamin Adida <ben@mit.edu>
Re: You're on SecurityFocus.com for the cleartext
	passwords.
On Sat, 6 May 2000, Tom Lane wrote:

> Benjamin Adida <ben@mit.edu> writes:
> >> It doesn't sound like MD5 changes this at all.
> 
> > The MD5 definitely doesn't change anything except overall security strength
> > of the algorithm.
> 
> OK, understood.  So it seems that switching to MD5 would offer (a) more
> portability to platforms without crypt(3), and (b) better security,
> at the costs of (a) implementation effort and (b) cross-version
> compatibility problems.  We probably ought to keep that discussion
> separate from the one about how the challenge protocol works.

I agree.
 
> > The additional random salt prevents someone from sniffing
> > the communication between client and server and then simply log in by
> > sending the known hash of the password. The challenge-response means that
> > sniffing one login doesn't allow you to fake the next one.
> 
> How so?  The server sends out one fixed salt (the one stored for that
> user's password in pg_shadow) and one randomly-chosen salt.  The client
> sends back two crypted passwords.  The server can check one of them.
> What can it do with the other?  Nothing that I can see, so where is the
> security gain?  A sniffer can still get in by sending back the same
> pair of crypted passwords next time, no matter what random salt is
> presented.

Off hand here is the only way I can see that this can work.

1) client gets password from user and md5's it.
2) upon connecting, the client receives a random salt from the server.
3) the client md5's the already md5'd password with this new salt.
4) the client sends the resulting hash to the server.
5) the server takes the md5'd password from pg_shadow and md5's it
   with the same random salt it sent to the client.
6) if it matches, the server sends yet another salt to the client.
7) repeat steps 3, 4 and 5.
8) if it matches the client's in.

Why should this work?  Because the next time the client tries to connect
it will be given a different salt.   But why twice?  It seems that once
would be enough since it's a random salt to begin with and the client
should never be getting that salt twice.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sat, 6 May 2000 14:40:41 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
on 5/6/00 2:40 PM, Vince Vielhaber at vev@michvhf.com wrote:

> Why should this work?  Because the next time the client tries to connect
> it will be given a different salt.   But why twice?  It seems that once
> would be enough since it's a random salt to begin with and the client
> should never be getting that salt twice.

No, the reason why you would have "two" hashes is so that the server doesn't
have to store the cleartext password. The server stores an already-hashed
version of the password, so the client must hash the cleartext twice, once
with a long-term salt, once with a random, one-time salt.

-Ben


Sat, 06 May 2000 14:41:57 -0400
Benjamin Adida <ben@mit.edu>
Re: You're on SecurityFocus.com for the cleartext
	passwords.
Benjamin Adida <ben@mit.edu> writes:
> Okay, my understanding was that the protocol would work as follows:

> - client requests login
> - server sends stored salt c1, and random salt c2.
> - client performs hash_c2(hash_c1(password)) and sends result to server.
> - server performs hash_c2(stored_pg_shadow) and compares with client
> submission.
> - if there's a match, there's successful login.

Oh, now I see.  OK, that looks like it would work.  It would definitely
mean a change of algorithm on the client side.

Probably the way to attack this would be to combine MD5 and this double
password-munging algorithm as a new authentication protocol type to add
to the ones we already support.  That way old clients don't have to be
updated instantly.

OTOH, if the password stored in pg_shadow is MD5-encrypted, then we lose
the ability to support the old crypt-based auth method, don't we?
Old clients could be successfully authenticated with cleartext password
challenge (server MD5's the transmitted password and compares to
pg_shadow), but we couldn't do anything with a crypt()-encrypted
password.  Is that enough reason to stay with crypt() as the underlying
hashing engine?  Maybe not, but we gotta consider the tradeoffs...

			regards, tom lane

Sat, 06 May 2000 14:43:37 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
> > The additional random salt prevents someone from sniffing
> > the communication between client and server and then simply log in by
> > sending the known hash of the password. The challenge-response means that
> > sniffing one login doesn't allow you to fake the next one.
> 
> How so?  The server sends out one fixed salt (the one stored for that
> user's password in pg_shadow) and one randomly-chosen salt.  The client
> sends back two crypted passwords.  The server can check one of them.
> What can it do with the other?  Nothing that I can see, so where is the
> security gain?  A sniffer can still get in by sending back the same
> pair of crypted passwords next time, no matter what random salt is
> presented.

No, you crypt the user-supplied password twice.

	'fred' -> crypt with fixed -> crypt with random

Server does:

	pg_shadow password -> crypt with random

Then check to see they match.

Does that help?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 14:50:59 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
> Okay, my understanding was that the protocol would work as follows:
> 
> - client requests login
> - server sends stored salt c1, and random salt c2.
> - client performs hash_c2(hash_c1(password)) and sends result to server.
> - server performs hash_c2(stored_pg_shadow) and compares with client
> submission.
> - if there's a match, there's successful login.
> 
> This protocol will truly create a challenge-response where the communication
> is different at each login, and where sniffing one
> hash_c2(hash_c1(password)) doesn't give you any way to log in with a
> different c2.

Yes.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 14:51:25 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext     passwords.
Let me comment:

> > How so?  The server sends out one fixed salt (the one stored for that
> > user's password in pg_shadow) and one randomly-chosen salt.  The client
> > sends back two crypted passwords.  The server can check one of them.
> > What can it do with the other?  Nothing that I can see, so where is the
> > security gain?  A sniffer can still get in by sending back the same
> > pair of crypted passwords next time, no matter what random salt is
> > presented.
> 
> Off hand here is the only way I can see that this can work.
> 
> 1) client gets password from user and md5's it.

No, no md5 yet.

> 2) upon connecting, the client receives a random salt from the server.
> 3) the client md5's the already md5'd password with this new salt.

md5's plaintext password using pg_shadow salt, and random salt.

> 4) the client sends the resulting hash to the server.
> 5) the server takes the md5'd password from pg_shadow and md5's it
>    with the same random salt it sent to the client.

Yes.

> 6) if it matches, the server sends yet another salt to the client.
> 7) repeat steps 3, 4 and 5.
> 8) if it matches the client's in.
> 
> Why should this work?  Because the next time the client tries to connect
> it will be given a different salt.   But why twice?  It seems that once
> would be enough since it's a random salt to begin with and the client
> should never be getting that salt twice.

No, once with pg_shadow salt, then random salt.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 14:54:22 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
> on 5/6/00 2:40 PM, Vince Vielhaber at vev@michvhf.com wrote:
> 
> > Why should this work?  Because the next time the client tries to connect
> > it will be given a different salt.   But why twice?  It seems that once
> > would be enough since it's a random salt to begin with and the client
> > should never be getting that salt twice.
> 
> No, the reason why you would have "two" hashes is so that the server doesn't
> have to store the cleartext password. The server stores an already-hashed
> version of the password, so the client must hash the cleartext twice, once
> with a long-term salt, once with a random, one-time salt.
> 

Yeah, right!

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 14:54:52 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext     passwords.
"Robert B. Easter" <reaster@comptechnews.com> writes:
> http://www.php.net/manual/function.crypt.php
> It explains that many systems have updated crypt() to use MD5 and how to
> check what hash algorithm your system's crypt() actually uses.

Oh, that's interesting.  If that's correct, we *already* have a cross-
platform compatibility problem: a client compiled on a machine with
DES-derived crypt() will be unable to authenticate itself under "crypt"
protocol to a server using MD5-based crypt(), or vice versa, because the
wrong hashed password will be sent.  Can someone with access to two such
machines check this?

If that's true, it seriously weakens the backwards-compatibility
argument for sticking with crypt(), IMHO.  Old clients on another
platform may already fail to talk to your server...

			regards, tom lane

Sat, 06 May 2000 14:55:42 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
> Probably the way to attack this would be to combine MD5 and this double
> password-munging algorithm as a new authentication protocol type to add
> to the ones we already support.  That way old clients don't have to be
> updated instantly.

Not sure that will work because once we use md5 on the server side for
pg_shadow, we have to be able to do md5 on the client, I think, for
crypting because the md5 has to be done _before_ the random salt crypt.

> 
> OTOH, if the password stored in pg_shadow is MD5-encrypted, then we lose
> the ability to support the old crypt-based auth method, don't we?

Yes.

> Old clients could be successfully authenticated with cleartext password
> challenge (server MD5's the transmitted password and compares to
> pg_shadow), but we couldn't do anything with a crypt()-encrypted
> password.  Is that enough reason to stay with crypt() as the underlying
> hashing engine?  Maybe not, but we gotta consider the tradeoffs...

Not sure.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 14:57:38 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Probably the way to attack this would be to combine MD5 and this double
>> password-munging algorithm as a new authentication protocol type to add
>> to the ones we already support.  That way old clients don't have to be
>> updated instantly.

> Not sure that will work because once we use md5 on the server side for
> pg_shadow, we have to be able to do md5 on the client, I think, for
> crypting because the md5 has to be done _before_ the random salt crypt.

We can still support old clients under the cleartext-password protocol:
client sends password in clear, server MD5's it using salt from
pg_shadow and compares result.  This is vulnerable to sniffing but no
more so than it was before.  What we would lose is backwards
compatibility to the crypt-password protocol.  We should still choose
a new Authentication typecode for the MD5/double-hash protocol, just to
make sure no one gets confused about which protocol is being requested.

If these reports are correct that some platforms already have MD5, not
DES, inside crypt(3) then I'm definitely leaning towards going with MD5.
The best reason to stick with crypt as the hash engine would be to
preserve support for the existing crypt-based protocol, but if that's
already broken cross-platform then the value of continuing to support it
looks pretty dubious.  (After all, the clients on your own box are
probably getting updated at the same time as the server --- it's clients
on other boxes that you're really worried about backwards compatibility
for.)

			regards, tom lane

Sat, 06 May 2000 15:06:53 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 

Would public/private key pair authentication (like GPG) or SSL-like solutions
work? If the backend could use SSL, it would have the ability to protect
passwords and all data too from being seen on the network.  Somekind of SSL
ability would solve all security problems.  Can't OpenSSL be used on top of the
client/backend connection?  This way, the backend/database could just store
hashed passwords anyway it wants and the client only needs to support the
SSL-like layer in the connection.  I guess this would mean adding SSL into the
libraries (libpq etc) for the functions that make the backend connection.  It
would be nice if after user authentication, that the protocol could optionally
renegotiate back to an unencrypted connection for speed.  A security option
could be added to databases that allows the DBA to specify whether or not
access to the database requires a secure connection to protect the sensitive
info.

I'm probably not understanding everything here, but if system crypt() is used,
it looks like you have to go with the least common denominator algorithm
that is on all platforms, which might be the old 2-byte salt DES.  But if you
embed all this in the libpq etc, then you use whatever you want.

Also, isn't the salt the first x bytes of the hashed string, x depending on
which algorithm used?  Wouldn't things work like this:

1. Server sends the first 2 (or x bytes) of the hashed password (e.g., the salt
used to make the hashed password.
2. The client hashes the password with the salt and sends it back to the server.
3. The server compares what the client sent with the hash it has stored.  If
they match the user is let in.  I didn't think there was any need for this
random salt and double hashing thats been discussed.

If you have to implement something into the backend and the client libraries,
why not go for an SSL type solution?

(people might say, it sounds fine, why don't YOU do it :)

 Robert Easter


Sat, 6 May 2000 15:18:35 -0400
"Robert B. Easter" <reaster@comptechnews.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
Bruce Momjian wrote:
> 
> Now, I we want to move all the stuff to use MD5 rather than the standard
> unix password crypt,

AFAIK, MD5 is one of "the standard password crypt"'s ;)

> that is another option, though I am not sure what
> value it would have.

One advantage would be passwords with more than 8 characters that
matter.

IMO the salt part in the "old" crypt code is there only to make it 
harder for people to accidentally discover that other people have 
the same password with them, which could easily be avoided by 
including the username as kind of supersalt in the md5 string, 
so the value passed over wire (and stored in DB would be
MD5('<username>:<passwd>'). 
If we want to make password hijacking real hard, we could store 
the above but ask the client for 
MD5(<server-supplied-salt>+MD5(<username>+':'+<passwd>))
and compare that

-------------
Hannu

Sat, 06 May 2000 22:36:21 +0300
Hannu Krosing <hannu@tm.ee>
Re: You're on SecurityFocus.com for the cleartext passwords.
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I think the only problem is moving dumps from on machine to another. 
> The crypt version may not exist or be different on different machines.

This is something I was going to bring up, but I see Bruce already did.
How will dump/restore and upgrades cope with crypted passwords?

If we standardize on MD5 encryption then there shouldn't be a cross-
platform problem with incompatible hashes, but we've still got troubles.

Currently, pg_dumpall handles saving and loading of pg_shadow as a
simple table COPY operation.  That'd work OK once you have the passwords
stored in MD5-encrypted form, but it will surely not work for upgrading
from 7.0 to 7.1 (assume 7.1 is when we'll have this stuff ready).

I've never cared for dumping pg_shadow that way anyway, since it makes
cross-version changes in the format of pg_shadow nearly impossible;
this password storage issue is just one case of a larger problem.  What
pg_dumpall should really be doing is emitting CREATE USER commands to
reload the contents of pg_shadow.

To do it that way, we'd need two variants of CREATE USER:

CREATE USER ... WITH PASSWORD 'foo'

(the existing syntax).  'foo' is cleartext and it gets hashed on its
way into the table.  The hashing step includes choosing a random salt.

CREATE USER ... WITH ENCRYPTED PASSWORD 'bar'

Here 'bar' is the already-encrypted password form (with salt value
embedded); it'd be dropped into pg_shadow unchanged, although the
command ought to do whatever it can to check the validity of the
encryption format.

pg_dumpall would generate this second form, inserting the crypted
password it had read from the pg_shadow table being dumped.

(Probably, there should be an ALTER USER SET ENCRYPTED PASSWORD as
well, for transferring already-crypted passwords, but that's not
essential for the purpose at hand.)

That solves our problem going forward, but we're still stuck for
how to get 7.0 password data into 7.1.  One possible avenue is to make
sure that it is possible to distinguish whether an existing database
contains crypted or cleartext passwords (maybe this comes for free,
or maybe we have to change the name of the pg_shadow password column
or some such).  Then pg_dumpall could be made to dump out either
WITH PASSWORD 'foo' or WITH ENCRYPTED PASSWORD 'foo' depending on
whether it sees that it is reading cleartext or crypted passwords
from the source database.  Then we tell people that they have to
use 7.1's pg_dumpall to dump a 7.0 database in preparation for
updating to 7.1, or else expect to have to reset all their passwords.

Is there a better way?

			regards, tom lane

Sat, 06 May 2000 16:09:04 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
Vince Vielhaber <vev@michvhf.com> writes:
> How about ODBC?  This is from the ODBC driver source connection.c:
>    self->errormsg = "Password crypt authentication not supported";
> Is that because of the platform it's running on or what it's talking
> to?

I think the ODBC authors didn't want to assume that libcrypt() is
available on the client side (which is probably right for Windows and
Mac at least).  Standardizing on our own implementation of MD5 would
sidestep that problem quite neatly.

Depending on libcrypt is pretty painful even in Unix environments;
have you seen what we have to do to get it to work in shared-library
contexts, on machines where libcrypt is a separate shlib and not part of
libc?  Yech.  We could get rid of a bunch of cruft in the makefiles by
abandoning crypt() ...

			regards, tom lane

Sat, 06 May 2000 16:15:42 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> Probably the way to attack this would be to combine MD5 and this double
> >> password-munging algorithm as a new authentication protocol type to add
> >> to the ones we already support.  That way old clients don't have to be
> >> updated instantly.
> 
> > Not sure that will work because once we use md5 on the server side for
> > pg_shadow, we have to be able to do md5 on the client, I think, for
> > crypting because the md5 has to be done _before_ the random salt crypt.
> 
> We can still support old clients under the cleartext-password protocol:
> client sends password in clear, server MD5's it using salt from
> pg_shadow and compares result.  This is vulnerable to sniffing but no
> more so than it was before.  What we would lose is backwards
> compatibility to the crypt-password protocol.  We should still choose
> a new Authentication typecode for the MD5/double-hash protocol, just to
> make sure no one gets confused about which protocol is being requested.

Yes, got it.  I was confused.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 16:19:14 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
> Depending on libcrypt is pretty painful even in Unix environments;
> have you seen what we have to do to get it to work in shared-library
> contexts, on machines where libcrypt is a separate shlib and not part of
> libc?  Yech.  We could get rid of a bunch of cruft in the makefiles by
> abandoning crypt() ...

Agreed.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 16:22:03 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I think the only problem is moving dumps from on machine to another. 
> > The crypt version may not exist or be different on different machines.
> 
> This is something I was going to bring up, but I see Bruce already did.
> How will dump/restore and upgrades cope with crypted passwords?
> 
> If we standardize on MD5 encryption then there shouldn't be a cross-
> platform problem with incompatible hashes, but we've still got troubles.
> 
> Currently, pg_dumpall handles saving and loading of pg_shadow as a
> simple table COPY operation.  That'd work OK once you have the passwords
> stored in MD5-encrypted form, but it will surely not work for upgrading
> from 7.0 to 7.1 (assume 7.1 is when we'll have this stuff ready).
> 
> I've never cared for dumping pg_shadow that way anyway, since it makes
> cross-version changes in the format of pg_shadow nearly impossible;
> this password storage issue is just one case of a larger problem.  What
> pg_dumpall should really be doing is emitting CREATE USER commands to
> reload the contents of pg_shadow.

Hey, pg_dumpall is only a shell script.  It does what it can.  :-)

> 
> To do it that way, we'd need two variants of CREATE USER:
> 
> CREATE USER ... WITH PASSWORD 'foo'
> 
> (the existing syntax).  'foo' is cleartext and it gets hashed on its
> way into the table.  The hashing step includes choosing a random salt.
> 
> CREATE USER ... WITH ENCRYPTED PASSWORD 'bar'
> 
> Here 'bar' is the already-encrypted password form (with salt value
> embedded); it'd be dropped into pg_shadow unchanged, although the
> command ought to do whatever it can to check the validity of the
> encryption format.
> 
> pg_dumpall would generate this second form, inserting the crypted
> password it had read from the pg_shadow table being dumped.
> 
> (Probably, there should be an ALTER USER SET ENCRYPTED PASSWORD as
> well, for transferring already-crypted passwords, but that's not
> essential for the purpose at hand.)
> 
> That solves our problem going forward, but we're still stuck for
> how to get 7.0 password data into 7.1.  One possible avenue is to make
> sure that it is possible to distinguish whether an existing database
> contains crypted or cleartext passwords (maybe this comes for free,
> or maybe we have to change the name of the pg_shadow password column
> or some such).  Then pg_dumpall could be made to dump out either
> WITH PASSWORD 'foo' or WITH ENCRYPTED PASSWORD 'foo' depending on
> whether it sees that it is reading cleartext or crypted passwords
> from the source database.  Then we tell people that they have to
> use 7.1's pg_dumpall to dump a 7.0 database in preparation for
> updating to 7.1, or else expect to have to reset all their passwords.
> 
> Is there a better way?

If we add this WITH PASSWORD 'foo' to 7.0.X, then 7.1 can read that
format, know it is in cleartext, hash it, and load it in.  7.1 can dump
its table out as WITH ENCRYPTED PASSWORD 'foo'.

With 6.5.X and earlier, we can just tell people they have to manually
update them.  We can emit a warning if the pg_shadow password field does
contain an md5 format password.

Another idea is to add code to 7.1 to convert non-md5 shadow password
fields to md5 format.  Since we already have special handling to do
pg_pwd, we could do it there.  Seems like a plan.  MD5 format is all
hex digits of a specific length.  No way to get that confused with a
real password.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 16:34:47 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
on 5/6/00 3:18 PM, Robert B. Easter at reaster@comptechnews.com wrote:

> 
> Would public/private key pair authentication (like GPG) or SSL-like solutions
> work? If the backend could use SSL, it would have the ability to protect
> passwords and all data too from being seen on the network.  Somekind of SSL
> ability would solve all security problems.  Can't OpenSSL be used on top of
> the
> client/backend connection?

While SSL could probably be an option for people dealing with tremendously
sensitive data that shouldn't go in the clear over their internal network
(we're not talking about passwords here, just the SQL queries and
responses), I think it's overkill to impose SSL for everything.

The key exchange and constant encryption overhead would significantly affect
performance, so this doesn't seem like something to impose on everyone.

-Ben


Sat, 06 May 2000 17:50:46 -0400
Benjamin Adida <ben@mit.edu>
Re: You're on SecurityFocus.com for the cleartext
	passwords.
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Another idea is to add code to 7.1 to convert non-md5 shadow password
> fields to md5 format.  Since we already have special handling to do
> pg_pwd, we could do it there.  Seems like a plan.  MD5 format is all
> hex digits of a specific length.  No way to get that confused with a
> real password.

Well, if you're willing to depend on that, then there's no need for the
WITH ENCRYPTED PASSWORD variant syntax: the existing syntax WITH
PASSWORD could do it all, just by checking to see if the supplied
password string looks like it's already been md5-ified.

The real trick would be to get this to happen during a COPY into
pg_shadow --- if we did that, then dumps generated by 7.0 pg_dumpall
would still work.  Perhaps a trigger on pg_shadow insert/update is
the right place to check and md5-ify the password?  (If that's in place
then neither CREATE nor ALTER USER would need to do anything special!)

			regards, tom lane

Sat, 06 May 2000 17:57:10 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
"Robert B. Easter" <reaster@comptechnews.com> writes:
> Would public/private key pair authentication (like GPG) or SSL-like solutions
> work?

We already have SSL support --- that's why I wasn't especially excited
about making the password challenge itself be proof against sniffing.
Anybody who's afraid of sniffing attacks ought to be SSL-ifying his
entire database connection, not just trying to protect the password.

			regards, tom lane

Sat, 06 May 2000 18:02:07 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
Benjamin Adida <ben@mit.edu> writes:
> I think it's overkill to impose SSL for everything.

Agreed, and in any case we are not going to require people to install
SSL before they can use Postgres.  It's an appropriate tool for some
people to use depending on what their security situation is.

I think we are converging on a plan that involves switching from crypt
to MD5 as our password-hashing algorithm, so given that we are going to
need a client upgrade anyway, we can throw in the double hashing (two
salt) method you proposed without any extra pain.  Might as well protect
the password against sniffing if we can...

			regards, tom lane

Sat, 06 May 2000 18:12:54 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
On Sat, 06 May 2000, Benjamin Adida wrote:
> While SSL could probably be an option for people dealing with tremendously
> sensitive data that shouldn't go in the clear over their internal network
> (we're not talking about passwords here, just the SQL queries and
> responses), I think it's overkill to impose SSL for everything.
> 
> The key exchange and constant encryption overhead would significantly affect
> performance, so this doesn't seem like something to impose on everyone.
> 
> -Ben

I agree that it should not be active all the time.  Just active for databases
that have been setup to require it if the dba sets the option for it.  My idea
is that it would work like this:

1. Client connects to server.  The initial connection is automatically SSL.
2. The user is authenticated.
3. The client and server renegotiate the connection to drop out of SSL and to a
normal unencrpyted connection by default.  However, if the database has been set
to require a secure connection by the database owner, then the SSL connection
will remain.  This adds some overhead to connecting to the server, but when
people need performance, they use persistent connections.

The dba would have to set the database to require the SSL connection to remain
by running commands something like:

CREATE DATABASE mydb SECURE;  -- creates it initially secure.
ALTER DATABASE mydb ADD|DROP SECURE;  -- alters the secure option.
(some proposed Postgres extensions:)

OpenSSL is under the BSD license (www.openssl.org).  Its source code can be
integrated into the PostgreSQL source code so that users need know nothing
about it.  It would just get used internal to Postgres and the client
libraries.

OpenSSL also contains an MD5 routine that can be used on the passwords.

So far, no one is excited about this so I will not push it anymore.

-- 
Robert B. Easter
reaster@comptechnews.com

Sat, 6 May 2000 22:02:04 -0400
"Robert B. Easter" <reaster@comptechnews.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
On Sat, 6 May 2000, Robert B. Easter wrote:

> OpenSSL is under the BSD license (www.openssl.org).  Its source code can be
> integrated into the PostgreSQL source code so that users need know nothing
> about it.  It would just get used internal to Postgres and the client
> libraries.
Please do not 'integrate' code from OpenSSL into the tree. Its huge (2M
compressed source tree).

Negotiating security protocols and reconnecting seems like a hassle, just
having autoconf detect presence of openssl libraries (automatically or
--with-openssl) is perfect. The best (as in, simplest and most
transparent) way to integrate SSL support is to do it like http/https:
provide another port on which connections will be only accepted using SSL
protocol. Security-minded administrators should have an option of
disabling non-encrypted port.  On client side, use fairly simple (to my
memory, you use {tls|ssl}_connect instead of connect)

A flag for databases that would disallow their usage if the connection is
unencrypted would be nice though, for those people who wish to have both
encrypted and unencrypted connections.

> OpenSSL also contains an MD5 routine that can be used on the passwords.
MD5 is extremely simple, about 50 lines of code.

-alex



Sat, 6 May 2000 22:41:09 -0400 (EDT)
Alex Pilosov <alex@pilosoft.com>
Re: You're on SecurityFocus.com for the cleartext passwords.

So we're in agreement on using MD5.   Sverre, is the offer still open
for the java MD5 you wrote?  I'll translate it to C and make sure it
will compile/run/give-correct-results on as many platforms as possible
including DOS/Windows, hpux, FreeBSD and IRIX.  

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sat, 6 May 2000 23:06:42 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
So we're in agreement.... 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Another idea is to add code to 7.1 to convert non-md5 shadow password
> > fields to md5 format.  Since we already have special handling to do
> > pg_pwd, we could do it there.  Seems like a plan.  MD5 format is all
> > hex digits of a specific length.  No way to get that confused with a
> > real password.
> 
> Well, if you're willing to depend on that, then there's no need for the
> WITH ENCRYPTED PASSWORD variant syntax: the existing syntax WITH
> PASSWORD could do it all, just by checking to see if the supplied
> password string looks like it's already been md5-ified.

Yes, I would like that.  Our syntax for CREATE USER is already pretty
large.  Why not do it automatically?

> The real trick would be to get this to happen during a COPY into
> pg_shadow --- if we did that, then dumps generated by 7.0 pg_dumpall
> would still work.  Perhaps a trigger on pg_shadow insert/update is
> the right place to check and md5-ify the password?  (If that's in place
> then neither CREATE nor ALTER USER would need to do anything special!)

We already have a trigger for pg_shadow updates to recreate pg_pwd.  
Not sure if that happens from COPY however.  I sure hope it does.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 23:10:47 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
> Benjamin Adida <ben@mit.edu> writes:
> > I think it's overkill to impose SSL for everything.
> 
> Agreed, and in any case we are not going to require people to install
> SSL before they can use Postgres.  It's an appropriate tool for some
> people to use depending on what their security situation is.
> 
> I think we are converging on a plan that involves switching from crypt
> to MD5 as our password-hashing algorithm, so given that we are going to
> need a client upgrade anyway, we can throw in the double hashing (two
> salt) method you proposed without any extra pain.  Might as well protect
> the password against sniffing if we can...

That was my logic.  Pretty cheap to do it.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 23:11:35 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
> OpenSSL is under the BSD license (www.openssl.org).  Its source code can be
> integrated into the PostgreSQL source code so that users need know nothing
> about it.  It would just get used internal to Postgres and the client
> libraries.
> 
> OpenSSL also contains an MD5 routine that can be used on the passwords.
> 
> So far, no one is excited about this so I will not push it anymore.

Seems like that could be a solution, but we are trying not to bloat the
PostgreSQL tarball.  Shipping SSH would certainly do that.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 23:14:52 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
> 
> So we're in agreement on using MD5.   Sverre, is the offer still open
> for the java MD5 you wrote?  I'll translate it to C and make sure it
> will compile/run/give-correct-results on as many platforms as possible
> including DOS/Windows, hpux, FreeBSD and IRIX.  

Yes, MD5, double-crypt with pg_shadow salt and random salt.  Sounds like
a winner all around.

And finally, we need a trigger to somehow update non-md5 strings in the
pg_shadow password field.  No one is sure how to do that yet.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 23:21:11 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: So we're in agreement....

So, we're going to go with less security then is available on most Unix
OSs?  

if we are going to do this, *please* just use the regular system
crypt() function ... for those that are using MD5 for their passwords, at
least as it is under FreeBSD, crypt() does either MD5 or DES depending on
the system ...



On Sat, 6 May 2000, Bruce Momjian wrote:

> > 
> > So we're in agreement on using MD5.   Sverre, is the offer still open
> > for the java MD5 you wrote?  I'll translate it to C and make sure it
> > will compile/run/give-correct-results on as many platforms as possible
> > including DOS/Windows, hpux, FreeBSD and IRIX.  
> 
> Yes, MD5, double-crypt with pg_shadow salt and random salt.  Sounds like
> a winner all around.
> 
> And finally, we need a trigger to somehow update non-md5 strings in the
> pg_shadow password field.  No one is sure how to do that yet.
> 
> -- 
>   Bruce Momjian                        |  http://www.op.net/~candle
>   pgman@candle.pha.pa.us               |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Sun, 7 May 2000 00:28:17 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: So we're in agreement....
On Sat, 06 May 2000, Benjamin Adida wrote:
> on 5/6/00 2:14 PM, Tom Lane at tgl@sss.pgh.pa.us wrote:
> 
> > However, I still fail to see what it buys us to challenge the frontend
> > with two salts.  If the password is stored crypted, the *only* thing
> > we can validate is that password with the same salt it was stored
> > with.  It doesn't sound like MD5 changes this at all.
> 
> The MD5 definitely doesn't change anything except overall security strength
> of the algorithm. The additional random salt prevents someone from sniffing
> the communication between client and server and then simply log in by
> sending the known hash of the password. The challenge-response means that
> sniffing one login doesn't allow you to fake the next one.
> 
> -Ben

I see.  This protects the hash, which is an effective password, from being
gotten by sniffers.  But a cracker who has stolen the hashes out of Postgres can
still get in no matter what until you change the passwords.

I guess hashed password authentication is really not designed for use over an
untrusted connection.  You get the hash becomes effective password problem. 
Its very important that the hashed passwords stored in Postgres cannot be read
by anyone except the Postgres superuser.

I'm I getting this right?

Crypto 101 - I'm learning. :)
-- 
Robert B. Easter
reaster@comptechnews.com

Sat, 6 May 2000 23:29:05 -0400
"Robert B. Easter" <reaster@comptechnews.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
On Sun, 7 May 2000, The Hermit Hacker wrote:

> 
> So, we're going to go with less security then is available on most Unix
> OSs?  

Whoever said MD5 is less secure than standard DES has handed you a line
so long....  

If you want the entire story, go back and reread the thread.  If you want
it in a nutshell, MD5 isn't reversable so it's not considered encryption
as defined in the US Govt's eyes.  DES has been broken more than once.  I
don't know what you're using on hub, but MD5 is the default in FreeBSD
unless you actually choose DES.  Also the FreeBSD folks feel that MD5 is
the better choice.

> if we are going to do this, *please* just use the regular system
> crypt() function ... for those that are using MD5 for their passwords, at
> least as it is under FreeBSD, crypt() does either MD5 or DES depending on
> the system ...

crypt() requires libcrypt which isn't available on all platforms.  I 
think we all feel we can do better.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sat, 6 May 2000 23:42:01 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: So we're in agreement....
> 
> So, we're going to go with less security then is available on most Unix
> OSs?  
> 
> if we are going to do this, *please* just use the regular system
> crypt() function ... for those that are using MD5 for their passwords, at
> least as it is under FreeBSD, crypt() does either MD5 or DES depending on
> the system ...
> 

We have to use MD5 with double-encription.  We have to do that because
all clients need MD5 too, and we can't assume they all have DES.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 6 May 2000 23:43:06 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: So we're in agreement....
The Hermit Hacker <scrappy@hub.org> writes:
> So, we're going to go with less security then is available on most Unix
> OSs?  

What's your evidence for that assertion?  Garfinkel & Spafford's
_Practical Unix and Internet Security_ recommends MD5 as a *more*
secure method for storing passwords than crypt() (page 720 in my
copy).  DES is almost 20 years older than MD5, so I'm not sure
why you'd assume that it must be more secure.

> if we are going to do this, *please* just use the regular system
> crypt() function

Half of the argument for touching the issue at all is that we have a
lot of problems with crypt() --- not available on some platforms,
inconsistent results across platforms (not proven yet, but seems likely)
and a serious pain in the neck for our shared libraries to boot.
If we have to stick with crypt I'm not sure it's worth doing anything.


BTW, Vince, I see no need to reverse-engineer a Java implementation
into C.  The original spec includes a C implementation ... and it
looks to have a reasonably BSDish license.  See RFC 1321, eg at 
http://www.faqs.org/rfcs/rfc1321.html

			regards, tom lane

Sun, 07 May 2000 00:03:16 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: So we're in agreement.... 
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> So far, no one is excited about this so I will not push it anymore.

> Seems like that could be a solution, but we are trying not to bloat the
> PostgreSQL tarball.  Shipping SSH would certainly do that.

We already have the ability to work with an externally provided SSL
library.  Right at the moment I don't see the merit of folding SSL
into the Postgres distribution instead.

			regards, tom lane

Sun, 07 May 2000 00:10:02 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
> I see.  This protects the hash, which is an effective password, from being
> gotten by sniffers.  But a cracker who has stolen the hashes out of Postgres can
> still get in no matter what until you change the passwords.
> 
> I guess hashed password authentication is really not designed for use over an
> untrusted connection.  You get the hash becomes effective password problem. 
> Its very important that the hashed passwords stored in Postgres cannot be read
> by anyone except the Postgres superuser.
> 
> I'm I getting this right?

Good point.  Though they can't see the original password, they can have
a pgsql client use it to connect to the database.

Anyone have a fix for that one?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 7 May 2000 00:17:28 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
On Sun, 7 May 2000, Tom Lane wrote:

> BTW, Vince, I see no need to reverse-engineer a Java implementation
> into C.  The original spec includes a C implementation ... and it
> looks to have a reasonably BSDish license.  See RFC 1321, eg at 
> http://www.faqs.org/rfcs/rfc1321.html

Got it!!  Thanks!!

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sun, 7 May 2000 00:18:08 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: So we're in agreement.... 
>> I see.  This protects the hash, which is an effective password, from
>> being gotten by sniffers.  But a cracker who has stolen the hashes
>> out of Postgres can still get in no matter what until you change the
>> passwords.

What's your point?  Stealing a password is stealing a password,
whatever form it's represented in.  More to the point, a cracker
who can get to the stored passwords in Postgres has already
thoroughly broken the database's security; he doesn't need any
more access to the db than he's already got.

>> Its very important that the hashed passwords stored in Postgres
>> cannot be read by anyone except the Postgres superuser.

No different from the current system, where the cleartext passwords
mustn't be readable by anyone except the superuser, either.  That's
not the objective of this exercise.  The objective is to ensure that
getting hold of the (hashed) Postgres passwords doesn't let you into
*other* systems that a database user might have used the same
(cleartext) password for.  We're trying to provide some security
for other people's barns in the event that our own horses have already
been stolen.

			regards, tom lane

Sun, 07 May 2000 00:50:06 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
On Sun, 07 May 2000, Tom Lane wrote:
> not the objective of this exercise.  The objective is to ensure that
> getting hold of the (hashed) Postgres passwords doesn't let you into
> *other* systems that a database user might have used the same
> (cleartext) password for.  We're trying to provide some security
> for other people's barns in the event that our own horses have already
> been stolen.

Ok, now I feel like this discussion has gone full circle and it is all very
clear to me.  The objectives are satisfied by the MD5 double salt scheme.  The
password itself is protected.  That doesn't protect the user.

I guess my point is about protecting your login and keeping others from getting
in there and doing things as you, in short, protecting the user. Sure, if the
cracker got the hashes, he got everything, but the accountability/blame
doesn't lie with one of the database users. The blame lies on the poor security
of the postgres superuser only.  When hashes alone are used for authentication,
it should not matter if someone gets the hashes - the hashes alone shouldn't be
enough let them in. Ordinarily, a cleartext password would go straight
through to the other end over a path that isn't sniffable or isn't even over a
network, like on a unix host when you login to a console.  But in Postgres'
case, you have the hash sent as an effective password from the client - it
doesn't send the actual password.  Anytime people really want a secure network
login, they have to use ssh, ssl, or pgp (pub/private key) software.  Those are
the only things that really protect user and keeps in control of their login. 
Your password might be secure, but your login still isn't.  Someone could get
the hashes and you might not ever know it.  They could login to the database
system as you for a long long time stealing your information and doing things
as you, but they'd never get your password.  The user could end up getting
blamed for some serious stuff that is not his fault.  The user doesn't have
control of his login.  The dba might realize that he had a security breach and
Bob's hash was stolen and used by a cracker to login as Bob and do serious
damage.  The db just shrugs it off and blames it on Bob, saying that Bob must
have compromised his clear text password.

I'd say under the scheme proposed, you really have to trust your dba and change
your password frequently.  Anyone with access to the hashes can login as you and
make you look bad.

-- 
Robert B. Easter
reaster@comptechnews.com

Sun, 7 May 2000 01:17:50 -0400
"Robert B. Easter" <reaster@comptechnews.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
"Robert B. Easter" <reaster@comptechnews.com> writes:
> I'd say under the scheme proposed, you really have to trust your dba
> and change your password frequently.  Anyone with access to the hashes
> can login as you and make you look bad.

Again, what's your point?  The dbadmin can do whatever he wants *inside
the database*, including altering data that you might nominally be
responsible for.  He doesn't need your password for that, any more than
your local Unix sysadmin needs anything but root privileges to alter
your files.

The point of this change is to make sure that the dbadmin can't get
at your cleartext password, which might allow him to pose as you for
non-database purposes (if you are so foolish as to use that same
cleartext password for non-database purposes).

			regards, tom lane

Sun, 07 May 2000 02:32:54 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
[Vince Vielhaber]

|   So we're in agreement on using MD5.   Sverre, is the offer still open
|   for the java MD5 you wrote?

Yes, of course!

|   I'll translate it to C and make sure it will
|   compile/run/give-correct-results on as many platforms as possible
|   including DOS/Windows, hpux, FreeBSD and IRIX.

I started translating it yesterday.  I'll do some commenting and
testing today, if I get the time I need from my family. :) It's been
three years since I last wrote a C program, so someone should probably
peek thru it looking for pointer problems. :)

If I don't finish it today, I'll send you the half finished stuff
tonight (it's morning here now). OK?


Sverre.

-- 
<URL:mailto:sverrehu@online.no>
<URL:http://home.sol.no/~sverrehu/>          Echelon bait: semtex, bin Laden,
                                         plutonium, North Korea, nuclear bomb

Sun, 7 May 2000 08:34:20 +0200
"Sverre H. Huseby" <sverrehu@online.no>
Re: So we're in agreement....
On Sun, 7 May 2000, Sverre H. Huseby wrote:

> [Vince Vielhaber]
> 
> |   So we're in agreement on using MD5.   Sverre, is the offer still open
> |   for the java MD5 you wrote?
> 
> Yes, of course!
> 
> |   I'll translate it to C and make sure it will
> |   compile/run/give-correct-results on as many platforms as possible
> |   including DOS/Windows, hpux, FreeBSD and IRIX.
> 
> I started translating it yesterday.  I'll do some commenting and
> testing today, if I get the time I need from my family. :) It's been
> three years since I last wrote a C program, so someone should probably
> peek thru it looking for pointer problems. :)
> 
> If I don't finish it today, I'll send you the half finished stuff
> tonight (it's morning here now). OK?

Works for me.

BTW, you may want to look at your email configuration.  The BCC was 
leaking thru.  

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sun, 7 May 2000 02:41:10 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: So we're in agreement....
Bruce Momjian wrote:
> 
> Another idea is to add code to 7.1 to convert non-md5 shadow password
> fields to md5 format.  Since we already have special handling to do
> pg_pwd, we could do it there.  Seems like a plan.  MD5 format is all
> hex digits of a specific length.  No way to get that confused with a
> real password.

One way to approach it in a semi-transparent way would be to add a
column md5passwd to pg_shadow and set up a trigger to automatically 
update it whenever passwd is inserted/updated (and for 
security-concious people the same trigger would empty the passwd 
field itself, or set it to some special value that disables 
crypt/cleartext logins)

the WITH ENCRYPTED PASSWORD would then update md5passwd directly, and 
reset the passwd field.

I still think that the easiest way to get unique hashes would be to use 
the username as salt when generating the value for md5passwd .

----------------
Hannu

Sun, 07 May 2000 10:11:21 +0300
Hannu Krosing <hannu@tm.ee>
Re: You're on SecurityFocus.com for the cleartext passwords.
Bruce Momjian wrote:
> 
> >
> > So we're in agreement on using MD5.   Sverre, is the offer still open
> > for the java MD5 you wrote?  I'll translate it to C and make sure it
> > will compile/run/give-correct-results on as many platforms as possible
> > including DOS/Windows, hpux, FreeBSD and IRIX.
> 
> Yes, MD5, double-crypt with pg_shadow salt and random salt.  Sounds like
> a winner all around.

why pg_shadow salt ? for md5 we will need to store it separately anyway.
why not MD5(<server-supplied-random-salt> || MD5(<username> ||
<password>))
that way we would overcome the original need for salt (accidental
discovery 
of similar passwords) and would have no need for storing the salt.

actually we would probably need some kind of separator as well to avoid
the scenario of <user>+<password> and <userpa>+<ssword> being the same 
and thus having the same md5 hash. so the escheme could be

MD5(<server-supplied-random-salt> || '\n' || MD5(<username> || '\n' ||
<password>))

AFAIK there is no easy way to have a newline inside password. 

> And finally, we need a trigger to somehow update non-md5 strings in the
> pg_shadow password field.  No one is sure how to do that yet.

see my separate mail which I was unable to send yesterday as my phone 
line went down ;(

--------------
Hannu

Sun, 07 May 2000 10:21:56 +0300
Hannu Krosing <hannu@tm.ee>
Re: So we're in agreement....
The Hermit Hacker wrote:
> 
> So, we're going to go with less security then is available on most Unix
> OSs?

No, the general consensus seems to be using MD5, not the weaker DES
crypt.

> if we are going to do this, *please* just use the regular system
> crypt() function ... for those that are using MD5 for their passwords, at
> least as it is under FreeBSD, crypt() does either MD5 or DES depending on
> the system ...

What does it do to portbility ?

-----------
Hannu

Sun, 07 May 2000 10:27:01 +0300
Hannu Krosing <hannu@tm.ee>
Re: So we're in agreement....
[Vince Vielhaber]

|   Works for me.

Ok, here's the source.  Please note the following:

* You should change the typedefs in md5.c.  I guess the PostgreSQL
  source has types that are correct for various platforms.  I've run
  this on Linux (x86) only.

* I use the Artistic License.  Feel free to replace it with the
  license used in the rest of PostgreSQL.

* I tested my implementation (or rather, the translation to C) by
  generating MD5 sums for all files in my /usr/local/bin, and
  comparing the sums with those generated by GNU's md5sum program.
  You may want to do further tests.

* Although I've written tens of thousands of lines of C code in
  the past, it is almost three years since the last time I did.  Java
  programming has made me rather relaxed when it comes to memory
  handling, so you should look for pointer problems and failure to
  free allocated memory.  :-)

* Salting is left to the caller.  You will probably want to build
  convenience functions on top of md5_hash.

I really hope you will find the code useful, as it is my secret wish
to contribute to PostgreSQL.  :-)

Please keep me informed on what you do with the code, even if you
choose to throw it in /dev/null.


Sverre.

-- 
<URL:mailto:sverrehu@online.no>
<URL:http://home.sol.no/~sverrehu/>          Echelon bait: semtex, bin Laden,
                                         plutonium, North Korea, nuclear bomb
Sun, 7 May 2000 14:39:32 +0200
"Sverre H. Huseby" <sverrehu@online.no>
Re: So we're in agreement....
> > Yes, MD5, double-crypt with pg_shadow salt and random salt.  Sounds like
> > a winner all around.
> 
> why pg_shadow salt ? for md5 we will need to store it separately anyway.
> why not MD5(<server-supplied-random-salt> || MD5(<username> ||
> <password>))
> that way we would overcome the original need for salt (accidental
> discovery 
> of similar passwords) and would have no need for storing the salt.
> 
> actually we would probably need some kind of separator as well to avoid
> the scenario of <user>+<password> and <userpa>+<ssword> being the same 
> and thus having the same md5 hash. so the escheme could be
> 
> MD5(<server-supplied-random-salt> || '\n' || MD5(<username> || '\n' ||
> <password>))
> 
> AFAIK there is no easy way to have a newline inside password. 

Well, unix passwords don't use the username as salt, so why should we?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 7 May 2000 09:08:45 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: So we're in agreement....
On Sun, 07 May 2000, you wrote:
> So, if someone can see those hashes, why don't they just create
> themselves a new user, grant it full privileges to the database and
> play?

I know, they can do anything.  But creating a new user is something very
obvious that the admin will see.  The breach of security would be detectable. 
If they can get in with the hashes, they can be very sneaky and it would take a
long time to detect.  The cracker shouldn't able to compromise a current
users account without having to even change the password on it.  Its better to
force the cracker have to create an account than to let him do bad things as
you whenever he wants. Would you like the feeling of never knowing that maybe
someone has your hash and is able to get in without you knowing?  Your
password becomes useless.  Really, sensitive information in the database could
be insecure over a long period of time and it would never be detectable.  You'd
just have to change your password frequently to ensure that you are the only
one that can get in.  Its better to make a security system where the alarm will
go off.

If your competitor is able to get into the database as you, because he got your
hash after hiring some cracker to get it, he can learn all your trade secrets
and always find a way to have the advantage.  You and your company might have a
hard time figuring out whats going on because, so to speak, the security on the
database has no alarm.

I agree that the MD5 double hash solution fixes the immediate problem.  Its
just not going to be a complete security solution.

-- 
Robert B. Easter
reaster@comptechnews.com

Sun, 7 May 2000 10:37:28 -0400
"Robert B. Easter" <reaster@comptechnews.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
Hannu Krosing <hannu@tm.ee> writes:
> One way to approach it in a semi-transparent way would be to add a
> column md5passwd to pg_shadow and set up a trigger to automatically 
> update it whenever passwd is inserted/updated (and for 
> security-concious people the same trigger would empty the passwd 
> field itself, or set it to some special value that disables 
> crypt/cleartext logins)

I don't think it's optional to get rid of the cleartext password;
kindly recall the original complaint we are trying to address
(see subject line of this thread ;-)).  So there's little value in
storing two columns.

Also, by having just one password field we can deal with either
cleartext or pre-encrypted incoming passwords fairly easily.
The trigger either reformats the field, or not; no upstream code
needs to worry about whether the password is already encrypted.
So we don't need the "WITH ENCRYPTED PASSWORD" variant syntax,
which is a good thing IMHO.

> I still think that the easiest way to get unique hashes would be to use 
> the username as salt when generating the value for md5passwd .

No, I don't think that's an improvement.  Please recall that the
original reason for inventing salt was to make sure that it wouldn't be
obvious whether the same user was using the same password on multiple
machines.

Since MD5 can take an arbitrarily long input phrase, we could possibly
run the calculation as MD5(password || username || random salt), but
there *must* be some randomness in there.

I doubt that it'd be all that great an idea to include the username.
The biggest objection to it is that renaming a user would break his
password (nonobviously, too).  The only reason in favor of it is that
it wouldn't be apparent when two different users share the same password
--- but the random salt covers that problem and does more too.

			regards, tom lane

Sun, 07 May 2000 11:28:40 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
On Sun, 7 May 2000, Bruce Momjian wrote:

> > > Yes, MD5, double-crypt with pg_shadow salt and random salt.  Sounds like
> > > a winner all around.
> > 
> > why pg_shadow salt ? for md5 we will need to store it separately anyway.
> > why not MD5(<server-supplied-random-salt> || MD5(<username> ||
> > <password>))
> > that way we would overcome the original need for salt (accidental
> > discovery 
> > of similar passwords) and would have no need for storing the salt.
> > 
> > actually we would probably need some kind of separator as well to avoid
> > the scenario of <user>+<password> and <userpa>+<ssword> being the same 
> > and thus having the same md5 hash. so the escheme could be
> > 
> > MD5(<server-supplied-random-salt> || '\n' || MD5(<username> || '\n' ||
> > <password>))
> > 
> > AFAIK there is no easy way to have a newline inside password. 
> 
> Well, unix passwords don't use the username as salt, so why should we?

It could add a level of security.  The client knows the username.  If
the client were to only send LOGIN or something like that to the server
without sending the username and the server only replied with the random
salt, the client would know that the username was the fixed salt and could
use that with random salt received from the server.  So it's really a
hidden salt.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sun, 7 May 2000 12:16:45 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: So we're in agreement....
Vince Vielhaber <vev@michvhf.com> writes:
> It could add a level of security.  The client knows the username.  If
> the client were to only send LOGIN or something like that to the server
> without sending the username and the server only replied with the random
> salt, the client would know that the username was the fixed salt and could
> use that with random salt received from the server.  So it's really a
> hidden salt.

Hidden from whom?  The client *must* send the username to the server,
so a sniffer who is able to see both sides of the conversation will
still have all the same pieces.  If the sniffer only sees one side of
the conversation, he's still in trouble: he'll get the random salt, or
the hashed password, but not both.  So I still don't see what the
username is adding to the process that will make up for rendering it
much more difficult to rename users.

			regards, tom lane

Sun, 07 May 2000 13:15:20 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: So we're in agreement.... 
On Sun, 7 May 2000, Tom Lane wrote:

> Vince Vielhaber <vev@michvhf.com> writes:
> > It could add a level of security.  The client knows the username.  If
> > the client were to only send LOGIN or something like that to the server
> > without sending the username and the server only replied with the random
> > salt, the client would know that the username was the fixed salt and could
> > use that with random salt received from the server.  So it's really a
> > hidden salt.
> 
> Hidden from whom?  The client *must* send the username to the server,
> so a sniffer who is able to see both sides of the conversation will
> still have all the same pieces.  If the sniffer only sees one side of
> the conversation, he's still in trouble: he'll get the random salt, or
> the hashed password, but not both.  So I still don't see what the
> username is adding to the process that will make up for rendering it
> much more difficult to rename users.

My intent was not to send the username, but let the server figure it 
out by the response.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sun, 7 May 2000 13:21:54 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: So we're in agreement.... 
Vince Vielhaber <vev@michvhf.com> writes:
> My intent was not to send the username, but let the server figure it 
> out by the response.

That would be a neat trick.  How will you do it?  MD5 is not reversible.

Actually you could do it, but *not* by folding the username into the
password.  Instead try this:

1. Client chooses random saltC, sends saltC and MD5(username, saltC)
to server in initial request.

2. Server runs through all available usernames, computing
MD5(thisusername, saltC) for each one and looking for a match.

3. Server chooses random saltS, sends it to client along with saltP
stored in pg_shadow entry for user.

4. Client computes MD5(MD5(password, saltP), saltS) and sends to server.
Server compares this against MD5(stored hashed password, saltS).

This would prevent a sniffer from finding out anything about the valid
usernames, which'd be a useful improvement, but I'm not convinced that
it's worth breaking the initial protocol for.  (Not to mention the
additional postmaster CPU time --- step 2 above is not cheap if there
are lots of usernames.)  So far we have just been talking about adding
a different type of authentication challenge to the set we already
support; that doesn't break old clients as long as they aren't
challenged that way.  Modifying the initial connection request packet
is a different story.

I'm still of the opinion that anyone who is really concerned about
sniffing attacks ought to be using SSL, because protecting just their
password and not the data that will be exchanged later in the session is
unwise.  So I'm not really excited about adding anti-sniffing frammishes
like this one.  We've got a good scheme for the password; let's be
careful about adding "improvements" that won't carry their weight in
the real world.  There's no such thing as a single security scheme that
addresses every possible vulnerability.  Extending one part of your
security arsenal to partially solve problems that are better solved
by a different tool is just wasting time.

			regards, tom lane

Sun, 07 May 2000 13:40:14 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: So we're in agreement.... 
On Sun, 7 May 2000, Tom Lane wrote:

> Vince Vielhaber <vev@michvhf.com> writes:
> > My intent was not to send the username, but let the server figure it 
> > out by the response.
> 
> That would be a neat trick.  How will you do it?  MD5 is not reversible.

CLIENT: md5(salt_from_server + md5(username + md5(password)))

SERVER: md5(salt_from_server + md5(username + stored_password))

The server runs thru all available usernames using the above algorithm.

> I'm still of the opinion that anyone who is really concerned about
> sniffing attacks ought to be using SSL, because protecting just their
> password and not the data that will be exchanged later in the session is
> unwise.  So I'm not really excited about adding anti-sniffing frammishes
> like this one.  We've got a good scheme for the password; let's be
> careful about adding "improvements" that won't carry their weight in
> the real world.  There's no such thing as a single security scheme that
> addresses every possible vulnerability.  Extending one part of your
> security arsenal to partially solve problems that are better solved
> by a different tool is just wasting time.

Agreed.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sun, 7 May 2000 14:45:44 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: So we're in agreement.... 
Vince Vielhaber <vev@michvhf.com> writes:
>>>> My intent was not to send the username, but let the server figure it 
>>>> out by the response.
>> 
>> That would be a neat trick.  How will you do it?  MD5 is not reversible.

> CLIENT: md5(salt_from_server + md5(username + md5(password)))

> SERVER: md5(salt_from_server + md5(username + stored_password))

> The server runs thru all available usernames using the above algorithm.

No, that doesn't work unless stored passwords contain no random salt
at all (you could use the username alone, but as I previously said
that's no substitute for random salt, and of dubious value anyway).
That'd be a distinct *loss* in security, not an improvement.

To have salt in the stored passwords, the server must receive the
username first so that it can look up the pg_shadow entry and find
which stored salt to send to the client (along with the randomly
generated per-transaction salt).  You could cloak the username as
I suggested before, but there have to be two messages.

			regards, tom lane

Sun, 07 May 2000 15:16:51 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: So we're in agreement.... 
On Sun, 7 May 2000, Tom Lane wrote:

> Vince Vielhaber <vev@michvhf.com> writes:
> >>>> My intent was not to send the username, but let the server figure it 
> >>>> out by the response.
> >> 
> >> That would be a neat trick.  How will you do it?  MD5 is not reversible.
> 
> > CLIENT: md5(salt_from_server + md5(username + md5(password)))
> 
> > SERVER: md5(salt_from_server + md5(username + stored_password))
> 
> > The server runs thru all available usernames using the above algorithm.
> 
> No, that doesn't work unless stored passwords contain no random salt
> at all (you could use the username alone, but as I previously said
> that's no substitute for random salt, and of dubious value anyway).
> That'd be a distinct *loss* in security, not an improvement.
> 
> To have salt in the stored passwords, the server must receive the
> username first so that it can look up the pg_shadow entry and find
> which stored salt to send to the client (along with the randomly
> generated per-transaction salt).  You could cloak the username as
> I suggested before, but there have to be two messages.

You're right, it wouldn't work.  It should've been like this:

CLIENT: md5(salt_from_server + md5(username + password)))

SERVER: md5(salt_from_server + stored_password) 

The "salt_from_server" is your random salt.  The fixed salt is the
username.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sun, 7 May 2000 15:23:37 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: So we're in agreement.... 
Tom Lane wrote:
> 
> Hannu Krosing <hannu@tm.ee> writes:
> > One way to approach it in a semi-transparent way would be to add a
> > column md5passwd to pg_shadow and set up a trigger to automatically
> > update it whenever passwd is inserted/updated (and for
> > security-concious people the same trigger would empty the passwd
> > field itself, or set it to some special value that disables
> > crypt/cleartext logins)
> 
> I don't think it's optional to get rid of the cleartext password;
> kindly recall the original complaint we are trying to address
> (see subject line of this thread ;-)).  So there's little value in
> storing two columns.

there is some value in

A. a smooth transition path via dump/restore
B. backwards compatibility for those that need it more than security -
   we can still do DES-crypt authentication if we choose to

later (say in 7.2) we can drop password and have only md5password

> Also, by having just one password field we can deal with either
> cleartext or pre-encrypted incoming passwords fairly easily.
> The trigger either reformats the field, or not; no upstream code
> needs to worry about whether the password is already encrypted.
> So we don't need the "WITH ENCRYPTED PASSWORD" variant syntax,
> which is a good thing IMHO.

But how will you know if the data in the field is md5 hashed ?

I know no way to tell for sure if an arbitrary string is a md5 
hash or not. 

Of course we could choose a format like {MD5}:hashedstring
and disallow cleartext passwords which start with {MD5}: like Zope does

> > I still think that the easiest way to get unique hashes would be to use
> > the username as salt when generating the value for md5passwd .
> 
> No, I don't think that's an improvement.  Please recall that the
> original reason for inventing salt was to make sure that it wouldn't be
> obvious whether the same user was using the same password on multiple
> machines.

Ok. My previous impression was that it was in order to make sure that 
two users on the same machine would not find out theat the other is 
using the same passord as she is

> Since MD5 can take an arbitrarily long input phrase, we could possibly
> run the calculation as MD5(password || username || random salt), but
> there *must* be some randomness in there.
> 
> I doubt that it'd be all that great an idea to include the username.
> The biggest objection to it is that renaming a user would break his
> password (nonobviously, too). 

Can we really rename users ?

hannu=# create user foo;
CREATE USER
hannu=# alter user foo rename to bar;
ERROR:  parser: parse error at or near "rename"

Why would one do it in the first place ?

> The only reason in favor of it is that
> it wouldn't be apparent when two different users share the same password
> --- but the random salt covers that problem and does more too.

True. I did not think of the one user/multiple computers scenario.

-----------
Hannu

Sun, 07 May 2000 22:29:39 +0300
Hannu Krosing <hannu@tm.ee>
Re: You're on SecurityFocus.com for the cleartext passwords.
Vince Vielhaber <vev@michvhf.com> writes:
> You're right, it wouldn't work.  It should've been like this:

> CLIENT: md5(salt_from_server + md5(username + password)))

> SERVER: md5(salt_from_server + stored_password) 

> The "salt_from_server" is your random salt.  The fixed salt is the
> username.

You're still not getting the point.  I refer you to Ben Adida's
original, correct description of the way to do this:

> - client requests login
> - server sends stored salt c1, and random salt c2.
> - client performs hash_c2(hash_c1(password)) and sends result to server.
> - server performs hash_c2(stored_pg_shadow) and compares with client
>   submission.
> - if there's a match, there's successful login.

There have to be *two* random salts involved, one chosen when the
password was set (and used to cloak the stored password against people
with access to pg_shadow) and one chosen for the duration of this
password challenge (and used to cloak the challenge transaction against
people sniffing the packet traffic).  If you give up either one of those
bits of randomness then you lose a great deal.

Using the username instead of an independent random value to salt the
stored password is not a small change, it is a fundamental weakening of
the security system.  If you don't see that this is so then you don't
understand anything about cryptography.

			regards, tom lane

Sun, 07 May 2000 15:34:09 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: So we're in agreement.... 
Tom Lane wrote:
> 
> Vince Vielhaber <vev@michvhf.com> writes:
> > You're right, it wouldn't work.  It should've been like this:
> 
> > CLIENT: md5(salt_from_server + md5(username + password)))
> 
> > SERVER: md5(salt_from_server + stored_password)
> 
> > The "salt_from_server" is your random salt.  The fixed salt is the
> > username.
> 
> You're still not getting the point.  I refer you to Ben Adida's
> original, correct description of the way to do this:
> 
> > - client requests login
> > - server sends stored salt c1, and random salt c2.
> > - client performs hash_c2(hash_c1(password)) and sends result to server.
> > - server performs hash_c2(stored_pg_shadow) and compares with client
> >   submission.
> > - if there's a match, there's successful login.
> 
> There have to be *two* random salts involved, one chosen when the
> password was set (and used to cloak the stored password against people
> with access to pg_shadow) and one chosen for the duration of this
> password challenge (and used to cloak the challenge transaction against
> people sniffing the packet traffic).  If you give up either one of those
> bits of randomness then you lose a great deal.
> 
> Using the username instead of an independent random value to salt the
> stored password is not a small change, it is a fundamental weakening of
> the security system. 

It allows one of the salts to never be sent, thereby strengthening that 
part against _anyone_sniffing_the_traffic_ (just a little) as he sees 
only one hash, different each time.

It allows _a_user_with_access_to_ pg_shadow _on_two_or_more_machines_ 
see the fact that a user has the same password on both of them (which 
info he can then useto guess the password in two tries, as often seen 
in movies ;)

> If you don't see that this is so then you don't understand anything 
> about cryptography.

It is too easy to think that you do ;).

BTW, I don't claim to "understand cryptography" . 
What I said above is just plain common sense ;)

And you will never get good security by cryptography only, not even 
by using SSL or SSH which are the right way to go if you want to protect 
against sniffing. 

The current thread started from a simple the need to hide passwords 
from PG superusers and system ROOT's. For that we have two schemes:

store MD5(username+passwd)
 - hidden from sniffing but easily guessable salt (as most users are
called 'bob')

store MD5(random_salt+password) , or more likely
random_salt+MD5(random_salt+passord) 
  or we will never find out the salt again ;)
 - both salts are known to sniffer who is still unable to do anything
with them

the difference between the two in mainly the fact that in first case the
user 
already knows the salt and in the second case it must be transferred to
her 
over fire - this makes the first one stronger, at least in case when the 
username is chosen as creatively as the password ;)

otoh, in the second case three things should match username, salt and
hash
which of course makes the second case stronger. 

If you understand which one is stronger you are smarter than I am ;)

--------------
Hannu

Sun, 07 May 2000 23:11:34 +0300
Hannu Krosing <hannu@tm.ee>
Re: So we're in agreement....
"Robert B. Easter" wrote:
> 
> On Sun, 07 May 2000, Hannu Krosing wrote:
> >
> > But how will you know if the data in the field is md5 hashed ?
> 
> I think they begin with $1$ and that the salt in the hashed string is like this:

how do you distinguish it from a plaintext password thet starts with $1$
?

> $1$<salt>$   -- a total of 12 characters of salt if you include the $1$$
> characters. <salt> is 9 characters.  Someone can correct me if this is not
> true. I'm not an expert. :)

Well in Zope they begin with {MD5} for MD5 hash. The md5 hash itself
knows 
nothing about salt - it is just fed to the function before the password.
And the digest can begin with anything, possibly even \0 if not
{uu|base64}encoded

------------
Hannu

Sun, 07 May 2000 23:19:04 +0300
Hannu Krosing <hannu@tm.ee>
Re: You're on SecurityFocus.com for the cleartext passwords.
On Sun, 07 May 2000, Hannu Krosing wrote:
> 
> But how will you know if the data in the field is md5 hashed ?

I think they begin with $1$ and that the salt in the hashed string is like this:

$1$<salt>$   -- a total of 12 characters of salt if you include the $1$$
characters. <salt> is 9 characters.  Someone can correct me if this is not
true. I'm not an expert. :)

Robert B. Easter
reaster@comptechnews.com

Sun, 7 May 2000 17:04:52 -0400
"Robert B. Easter" <reaster@comptechnews.com>
Re: You're on SecurityFocus.com for the cleartext passwords.
Hannu Krosing <hannu@tm.ee> writes:
> Tom Lane wrote:
>> I don't think it's optional to get rid of the cleartext password;
>> kindly recall the original complaint we are trying to address
>> (see subject line of this thread ;-)).  So there's little value in
>> storing two columns.

> there is some value in

> A. a smooth transition path via dump/restore

True, but if we set up the translation to occur in a trigger that
checks for already-hashed data, then we've got that licked.

> B. backwards compatibility for those that need it more than security -
>    we can still do DES-crypt authentication if we choose to

That is a definite loss, but on the other hand I now realize that the
crypt-based authentication has its own compatibility problems: it may
fail with a client running on another machine already!  Not to mention
that some popular client platforms and/or interfaces (think ODBC) never
have supported crypt authentication.  So it's questionable that there
are very many people using it in cross-platform situations where
backwards compatibility with old clients is critical.  Furthermore,
it's not like the people who do get burnt have no option at all: they
can switch to cleartext password authentication or one of the other
already-supported methods until they can bring their clients up to
speed.  On the whole, I think the advantages of moving to MD5 clearly
outweigh this single disadvantage.

>> Also, by having just one password field we can deal with either
>> cleartext or pre-encrypted incoming passwords fairly easily.
>> The trigger either reformats the field, or not; no upstream code
>> needs to worry about whether the password is already encrypted.
>> So we don't need the "WITH ENCRYPTED PASSWORD" variant syntax,
>> which is a good thing IMHO.

> But how will you know if the data in the field is md5 hashed ?

Easily.  We will need, say, 32 bits of random salt plus the 128-bit
MD5 hash value.  Represent these in a string that consists of, say,
8 hex digits, a '/' sign, and 32 more hex digits; use only uppercase
A-F, not lowercase, as hex digits.  Now, are you going to say that
	23B3C990/652B8383A48348CDEF57298289A882DD
is plausible as a cleartext password?  I don't agree...

>> No, I don't think that's an improvement.  Please recall that the
>> original reason for inventing salt was to make sure that it wouldn't be
>> obvious whether the same user was using the same password on multiple
>> machines.

> Ok. My previous impression was that it was in order to make sure that 
> two users on the same machine would not find out theat the other is 
> using the same passord as she is

It does that too.  But remember that the point of not storing cleartext
passwords is to prevent using a password stolen from User A to break
into User A's other accounts, not User B's accounts.  So same username,
different machine is definitely a case we should consider.

> Can we really rename users ?

update pg_shadow set usename = 'bar' where usename = 'foo';

An ALTER USER syntax would be handier, but you don't really need it...

			regards, tom lane

Sun, 07 May 2000 17:28:39 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
Tom Lane writes:

> How will dump/restore and upgrades cope with crypted passwords?

We could distribute a sed or awk script that you have to run on the dumped
file to convert the copy to create user commands. Shouldn't be hard to
write, it's just a question of whether people want to put up with it. It
seems cleaner than any of the "magic hooks" that have been proposed.

Actually, I have some ideas in the pipe that would indeed change the
layout of pg_shadow slightly, so this might have to happen anyway.

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Sun, 7 May 2000 23:34:58 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: You're on SecurityFocus.com for the cleartext passwords.
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:

>> How will dump/restore and upgrades cope with crypted passwords?

> We could distribute a sed or awk script that you have to run on the dumped
> file to convert the copy to create user commands. Shouldn't be hard to
> write, it's just a question of whether people want to put up with it. It
> seems cleaner than any of the "magic hooks" that have been proposed.

To my mind the real advantage of doing it in a trigger is that
CREATE USER WITH PASSWORD and ALTER USER SET PASSWORD can accept
*either* cleartext or already-hashed password data.  That seems
nicer than forcing the user to deal with two syntaxes, upgrade
scripts, etc.

> Actually, I have some ideas in the pipe that would indeed change the
> layout of pg_shadow slightly, so this might have to happen anyway.

How far down the pipe?  It'd be nice if we could fix pg_dumpall to
dump CREATE USER commands a version before we actually need it ;-).
I'd like to change the script for 7.1 (or maybe even 7.0.1) but keep
backwards compatibility for the old-style dump scripts until 7.2.
(At the moment I'm kind of kicking myself for not having fixed the
problem when I saw it, but there was no talk of pg_shadow changes
in the air at the time.)

			regards, tom lane

Sun, 07 May 2000 17:53:12 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
On Sun, 7 May 2000, Tom Lane wrote:

> Using the username instead of an independent random value to salt the
> stored password is not a small change, it is a fundamental weakening of
> the security system.  

That's what I was doing, substituting the original random salt for the 
username.  

>                         If you don't see that this is so then you don't
> understand anything about cryptography.

Was this smartass comment really necessary, Tom?

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
 128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Sun, 7 May 2000 18:56:09 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: So we're in agreement.... 
Vince Vielhaber <vev@michvhf.com> writes:
>> If you don't see that this is so then you don't
>> understand anything about cryptography.

> Was this smartass comment really necessary, Tom?

Probably not.  My sincerest apologies.

			regards, tom lane

Sun, 07 May 2000 18:58:52 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: So we're in agreement.... 
My understanding is that what you get from crypt(pw, salt) =

	$1$<salt>$<hashed password>

Please correct me if I wrong.  Again, not an expert.

On Sun, 07 May 2000, you wrote:
> "Robert B. Easter" wrote:
> > 
> > On Sun, 07 May 2000, Hannu Krosing wrote:
> > >
> > > But how will you know if the data in the field is md5 hashed ?
> > 
> > I think they begin with $1$ and that the salt in the hashed string is like this:
> 
> how do you distinguish it from a plaintext password thet starts with $1$
> ?
> 
> > $1$<salt>$   -- a total of 12 characters of salt if you include the $1$$
> > characters. <salt> is 9 characters.  Someone can correct me if this is not
> > true. I'm not an expert. :)
> 
> Well in Zope they begin with {MD5} for MD5 hash. The md5 hash itself
> knows 
> nothing about salt - it is just fed to the function before the password.
> And the digest can begin with anything, possibly even \0 if not
> {uu|base64}encoded
> 
> ------------
> Hannu
-- 
Robert B. Easter
reaster@comptechnews.com

Sun, 7 May 2000 19:08:33 -0400
"Robert B. Easter" <reaster@comptechnews.com>
Re: [HACKERS] You're on SecurityFocus.com for the cleartext
 passwords.
At 07:08 PM 5/7/00 -0400, Robert B. Easter wrote:
>My understanding is that what you get from crypt(pw, salt) =
>
>         $1$<salt>$<hashed password>

I thought it was only $<salt - 2 chars>$<hashed password>


Regards,
Stephan
--
Stephan Richter - (901) 573-3308 - srichter@cbu.edu
CBU - Physics & Chemistry; Framework Web - Web Design & Development
PGP Key: 735E C61E 5C64 F430 4F9C 798E DCA2 07E3 E42B 5391


Sun, 07 May 2000 18:21:15 -0500
Stephan Richter <srichter@cbu.edu>
Re: Re: [HACKERS] You're on SecurityFocus.com for
	the cleartext passwords.
At 15:34 7/05/00 -0400, Tom Lane wrote:
>
>You're still not getting the point.  I refer you to Ben Adida's
>original, correct description of the way to do this:
>
>> - client requests login
>> - server sends stored salt c1, and random salt c2.
>> - client performs hash_c2(hash_c1(password)) and sends result to server.
>> - server performs hash_c2(stored_pg_shadow) and compares with client
>>   submission.
>> - if there's a match, there's successful login.
>

I may well have missed something here, but it seems to me that the above
scheme is also not particularly secure since someone who has managed to get
access to the pg_shadow file will be able to fake a login by using a custom
client. ie:

 - client requests login
 - server sends stored salt c1, and random salt c2.
 - client ignores c1 and performs hash_c2(some_hash_from_the_file) and
sends result to server.
 - server performs hash_c2(stored_pg_shadow) and compares with client
   submission.

Have I missed somehting here? Obviously it depends on getting a copy of
pg_shadow.

It seems that there are at least two problems to solve in the whole
password & authentication problem:

1. How to store passwords so they can't be decrypted.

2. How to perform a secure handshake with a client.

You have already solved (1) by using MD5 (or even SHA, which is faster to
compute).

To solve (2) it seems to me that a slightly more complex interaction must
be undertaken using a public key algorithm:

 - Client sends [username] to server
 - Server sends [public key] to client
 - Client sends [enc(public key, password)] to server. 
 - server uses dec(secret key,enc) and computes MD5 hash of password,
comparing it to pg_shadow.

This would require a decent large integer library (which certainly exist).
For speed, a key pair would need to be stored on the server, since key
generation is quite slow.

In order for a man-in-the-middle attack to work, the attacker would also
need the secret key off the server. The risk could be reduced, at the
expense of computation, by generating a new key pair for each client,
although that would be *very* expensive.

Additionally, it may be good to allow the entire client/server comms to be
done as an encrypted interaction, since a man-in-the-middle may not be able
to read the password, but they will be able to read the data...

FWIW, I'd be willing to write the password and handshaking code, if no-one
else were interested.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: +61-03-5367 7422            |                 _________  \
Fax: +61-03-5367 7430            |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Tue, 09 May 2000 01:38:40 +1000
Philip Warner <pjw@rhyme.com.au>
Re: So we're in agreement.... 
on 5/8/00 11:38 AM, Philip Warner at pjw@rhyme.com.au wrote:

> I may well have missed something here, but it seems to me that the above
> scheme is also not particularly secure since someone who has managed to get
> access to the pg_shadow file will be able to fake a login by using a custom
> client. ie:

Yes, absolutely, but someone who gets the pg_shadow file can also alter the
database however he/she wants. The protocol defined prevents any knowledge
gained from sniffing, and prevents discovery of plaintext passwords from
looking at pg_shadow.

However, it does not prevent logins, as you mention, because once you have
the pg_shadow file, you've got everything anyways.

-Ben


Mon, 08 May 2000 11:59:10 -0400
Benjamin Adida <ben@mit.edu>
Re: So we're in agreement.... 
Philip Warner <pjw@rhyme.com.au> writes:
> To solve (2) it seems to me that a slightly more complex interaction must
> be undertaken using a public key algorithm:

>  - Client sends [username] to server
>  - Server sends [public key] to client
>  - Client sends [enc(public key, password)] to server. 
>  - server uses dec(secret key,enc) and computes MD5 hash of password,
> comparing it to pg_shadow.

Hmm.  The main problem with this is that once we get into having actual
encryption/decryption code in Postgres, we are going to run afoul of US
export regulations and other headaches.  MD5 doesn't pose that problem
because it's only a hashing algorithm not an encryptor.  I see your
point though, that requiring the client to send something one step
upstream from what's stored in pg_shadow would make it harder to do
anything useful by stealing pg_shadow.  Can we get the same result with
just MD5 operations?

One possibility that comes to mind is that we store MD5(MD5(password))
in pg_shadow, and expect the client to transmit MD5(password).
Of course that needs a cloaking scheme if you want to protect against
password sniffing, but offhand it seems that the same scheme Ben Adida
proposed should still work...

> Additionally, it may be good to allow the entire client/server comms to be
> done as an encrypted interaction, since a man-in-the-middle may not be able
> to read the password, but they will be able to read the data...

We have SSL capability already.  I don't feel an urge to reinvent SSL.

			regards, tom lane

Mon, 08 May 2000 12:02:30 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: So we're in agreement.... 
At 12:02 8/05/00 -0400, Tom Lane wrote:
>Philip Warner <pjw@rhyme.com.au> writes:
>
>Hmm.  The main problem with this is that once we get into having actual
>encryption/decryption code in Postgres, we are going to run afoul of US
>export regulations and other headaches.  

I thought that they had been relaxed recently, but I take the point: it's
probably not been relaxed enough. Then again, maybe 48 bit (or 56 or 112 or
whatever) is sufficient.


>One possibility that comes to mind is that we store MD5(MD5(password))
>in pg_shadow, and expect the client to transmit MD5(password).
>Of course that needs a cloaking scheme if you want to protect against
>password sniffing, but offhand it seems that the same scheme Ben Adida
>proposed should still work...

Yes. This seems like a much simpler solution - getting the pg_shadow file
is almost useless, as long as the effectiveness of dictionary attacks is
reduced by a sufficiently large salt. 

It still does not protect against a man-in-the-middle attack, since the new
'password' is just a 160 bit value, rather than clear text. Unless SSL is
used, of course.

>
>We have SSL capability already.  I don't feel an urge to reinvent SSL.
>

Sounds pretty reasonable to me. But if SSL is already there, don't you have
import/export problems?


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: +61-03-5367 7422            |                 _________  \
Fax: +61-03-5367 7430            |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Tue, 09 May 2000 02:34:43 +1000
Philip Warner <pjw@rhyme.com.au>
Re: So we're in agreement.... 
Philip Warner <pjw@rhyme.com.au> writes:
>> We have SSL capability already.  I don't feel an urge to reinvent SSL.

> Sounds pretty reasonable to me. But if SSL is already there, don't you have
> import/export problems?

No, because we don't include SSL in the distribution.  There are just
hooks to call it.

(At one time there were export regs against even having hooks for crypto
:-(, but I believe they finally agreed that was pretty silly...)

			regards, tom lane

Mon, 08 May 2000 12:40:32 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: So we're in agreement.... 
On Tue, 9 May 2000, Philip Warner wrote:

> At 12:02 8/05/00 -0400, Tom Lane wrote:
> >Philip Warner <pjw@rhyme.com.au> writes:
> >
> >Hmm.  The main problem with this is that once we get into having actual
> >encryption/decryption code in Postgres, we are going to run afoul of US
> >export regulations and other headaches.  
> 
> I thought that they had been relaxed recently, but I take the point: it's
> probably not been relaxed enough. Then again, maybe 48 bit (or 56 or 112 or
> whatever) is sufficient.

Being a Canadian based project, like OpenBSD, I do not believe that these
issues apply ... even FreeBSD, based in California, appears to be getting
around them now ... something about making you download rsaref seperately,
they can now ship with OpenSSH included as part of the system?



Mon, 8 May 2000 13:54:26 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: So we're in agreement.... 
The Hermit Hacker <scrappy@hub.org> writes:
> Being a Canadian based project, like OpenBSD, I do not believe that these
> issues apply ...

If we had to do it that way we could, but I'd just as soon not create
any questions for US mirror sites.

> even FreeBSD, based in California, appears to be getting
> around them now ... something about making you download rsaref seperately,

Yes, I think this is the cleanest current answer: you can connect to a
separately-distributed crypto engine and then it's not your problem.

MD5 (or SHA1 if people like that better) isn't a crypto engine according
to the rules, so including that in our distribution is a non-issue,
but a reversible encryptor might be an issue.

			regards, tom lane

Mon, 08 May 2000 13:08:32 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: So we're in agreement.... 
Stephan Richter wrote:
> 
> At 07:08 PM 5/7/00 -0400, Robert B. Easter wrote:
> >My understanding is that what you get from crypt(pw, salt) =
> >
> >         $1$<salt>$<hashed password>

That's for DES crypt (and without the $$)

---------
Hannu

Mon, 08 May 2000 22:48:35 +0300
Hannu Krosing <hannu@tm.ee>
Re: Re: [HACKERS] You're on SecurityFocus.com forthe cleartext
	passwords.
Tom Lane writes:

> We already have the ability to work with an externally provided SSL
> library.

Does it actually work? Has anybody tried it? Is it documented anywhere?


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Mon, 8 May 2000 23:37:25 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: You're on SecurityFocus.com for the cleartext passwords.
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> We already have the ability to work with an externally provided SSL
>> library.

> Does it actually work? Has anybody tried it? Is it documented anywhere?

Picky, picky ;-)

It looks like you compile with USE_SSL (which ought to be listed as an
available option in config.h.in, but isn't; someday it should be a
configure option, perhaps) and then add "-l" to the postmaster switches.
At least "-l" is documented.

There are some interactions between SSL-client-and-non-SSL-server, etc,
which you can read about in the pghackers archives from last year, if
not in the docs.  Also, I thought there was supposed to be a postmaster
option to refuse non-SSL connections, but I don't see it now...

As for whether it works, damifino; I don't have SSL installed here.
I do have a note in my todo list that speculates that the recent changes
for non-blocking connect in libpq may have broken its SSL support, and I
asked the pghackers list about that in January.  But I didn't get around
to looking at it, and no one else picked up on it either.

			regards, tom lane

Mon, 08 May 2000 18:50:25 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
Tom Lane wrote:

> One possibility that comes to mind is that we store MD5(MD5(password))
> in pg_shadow, and expect the client to transmit MD5(password).
> Of course that needs a cloaking scheme if you want to protect against
> password sniffing, but offhand it seems that the same scheme Ben Adida
> proposed should still work...

That would be pretty close to the RFC 2617 Digest Authentication. Why
don't we use that? Using a existing, widespread standard is good in
terms of portability, and saves on validating the principal algorithm.

Sevo

-- 
sevo@ip23.net

Tue, 09 May 2000 13:44:39 +0200
Sevo Stille <sevo@ip23.net>
Re: So we're in agreement....
Tom Lane writes:

> > Actually, I have some ideas in the pipe that would indeed change the
> > layout of pg_shadow slightly, so this might have to happen anyway.
> 
> How far down the pipe?

It would have to do with the access control work which I had planned to
look at. With the summer and all coming up and the hopefully shorter
release cycle I'm not sure whether I'm going to get to it. The
configuration and build clean-up should happen first anyway.

> It'd be nice if we could fix pg_dumpall to dump CREATE USER commands a
> version before we actually need it ;-).

The problem is that CREATE USER doesn't cover all fields of pg_shadow, in
particular usecatupd. Though perhaps this field is obscure enough to not
bother. Also this will be pretty tricky to get to work for groups. (That
pg_group table really needs a redesign.)


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Tue, 9 May 2000 22:51:52 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: You're on SecurityFocus.com for the cleartext passwords.
Peter Eisentraut <peter_e@gmx.net> writes:
>> It'd be nice if we could fix pg_dumpall to dump CREATE USER commands a
>> version before we actually need it ;-).

> The problem is that CREATE USER doesn't cover all fields of pg_shadow, in
> particular usecatupd. Though perhaps this field is obscure enough to not
> bother.

Well, we'd also want to make sure that CREATE and/or ALTER USER can be
used to set everything in pg_shadow.  A few more optional clauses
shouldn't be a big deal ...

> Also this will be pretty tricky to get to work for groups. (That
> pg_group table really needs a redesign.)

True.  I'm inclined to think that should be looked at in the context
of the schema support that people have been muttering about --- maybe
groups can be replaced by schemas somehow?  (Just a thought, maybe a
half-baked one.)

			regards, tom lane

Tue, 09 May 2000 16:56:32 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords. 
If I understand the original objection it's that passwords are stored 
in cleartext on the postmaster machine.  That's not much of an 
objection since you have to have your secrets available in the clear 
on both ends of a connection if you want the traffic on the 
connection secured.  Both the Solaris and NetBSD ppp implementations 
have this characteristic.  Kerberos IV encrypts the passwords on the 
server, but the master password for that encryption is still stored 
in "/.k" so it's the same in principle.

If I understand the current password implementation the postmaster 
chooses the salt, sends it to the client which does a unix crypt and 
returns the encrypted password for verification.  This is a nice, 
simple system which provides what I would consider a reasonable 
minimum of security.

But it's not *really* secure.  For one thing a bad guy could 
intercept the encrypted password and feed it to one of the 
password-guessing programs, like crack.  It's not very robust to 
man-in-the-middle attacks, either.  Do we know how predictable the 
salt-choosing algorithm is?  What if a counterfeit server requested 
authentication with a carefully-chosen salt (like 0)?

We are not in the business of creating security protocols.  IMHO we 
should leave that to the people who are.  If we want something better 
than the password scheme we have then we should adopt an existing 
standard.  Chap, as used with ppp, comes to mind.  It might be a good 
substitute for the password protocol.

Remember that we already support kerberos.  Also there is no reason 
you can't run the connection over an ssh tunnel.  That solution would 
protect the data as well as the passwords.  There are applications 
where the important information is not what's in the database, but 
what information did someone want from it and when did they ask.

Out of curiosity does SecurityFocus.com also criticise pppd for the 
same "problem"?  The problem is not us storing the passwords in the 
clear, but rather are we careful about the access permissions on the 
file that contains them, and are we careful about how they get passed 
around.

Signature failed Preliminary Design Review.
Feasibility of a new signature is currently being evaluated.
h.b.hotz@jpl.nasa.gov, or hbhotz@oxy.edu

Tue, 9 May 2000 14:02:17 -0700
"Henry B. Hotz" <hotz@jpl.nasa.gov>
Re: You're on SecurityFocus.com for the cleartext
 passwords.
"Henry B. Hotz" <hotz@jpl.nasa.gov> writes:
> But it's not *really* secure.  For one thing a bad guy could 
> intercept the encrypted password and feed it to one of the 
> password-guessing programs, like crack.  It's not very robust to 
> man-in-the-middle attacks, either.  Do we know how predictable the 
> salt-choosing algorithm is?  What if a counterfeit server requested 
> authentication with a carefully-chosen salt (like 0)?

I doubt the latter is a problem; AFAIK there are no weak salt values
in crypt() --- remember the salt is not a key.

However, the relatively small number of legal salt values (4096 IIRC)
is a weakness; an attacker who'd sniffed one encrypted password could
hope to get in by repeatedly connecting until he's challenged with
that same salt, and then he just gives the captured encrypted password.
If the salt-choosing code has any predictability then it might take
much less than ~4K tries, but that number is too small anyway.

I thought one of the major reasons for switching to a new protocol
is that we could include much wider random salt values in it, so
as to render that sort of attack impractical.

As for man-in-the-middle attacks, stealing passwords is the least
of our worries in that scenario --- the attacker could just wait
for login to complete and then insert his own queries into the
conversation.  I think we have to rely on end-to-end encryption
like SSH or SSL to defend against that sort of thing.

> We are not in the business of creating security protocols.  IMHO we 
> should leave that to the people who are.  If we want something better 
> than the password scheme we have then we should adopt an existing 
> standard.

Aw, that's no fun :-).  But you're right, we should look to see if there
are existing standards that meet all the criteria we are looking for.

> Out of curiosity does SecurityFocus.com also criticise pppd for the 
> same "problem"?

I checked and in fact there is nothing official about this "criticism";
it's just one message posted on a web bbs by someone with no obvious
credentials.  Still, given the other headaches that reliance on crypt()
causes us, it seems to make sense to work on a replacement password
scheme that deals with more problems than just cleartext password
storage.

			regards, tom lane

Tue, 09 May 2000 17:27:44 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: You're on SecurityFocus.com for the cleartext passwords.
Tom Lane wrote:
  >> Also this will be pretty tricky to get to work for groups. (That
  >> pg_group table really needs a redesign.)
  >
  >True.  I'm inclined to think that should be looked at in the context
  >of the schema support that people have been muttering about --- maybe
  >groups can be replaced by schemas somehow?  (Just a thought, maybe a
  >half-baked one.)

Please don't lose group support; it is a separate issue from schemas.

If schemas were available, I would use them to separate or subdivide
projects.  For example, a financial accounting system would logically
divide into modules, some of which would be universally required
(general ledger, base support, etc.) and some of which would be 
of interest to smaller groups of customers (sales ledger, order processing,
inventory control, payroll).  Each of these modules would contain a
logically separable set of data elements (tables, views, etc.).
However, add-on modules are very likely to need to access data from
other modules.  I would use schemas to implement these modules and
they could cross-refer to each other with the specification 
`schema.table.column'.

It should be possible for separate schemas to be housed on different
machines, linked by networking.

Different sets of users need to access different parts of the data, in
different ways.  This is not necessarily related to the separation of
modules.  For example, the directors of a company might have private
loan accounts in the general ledger module, and their pay would be
handled by the payroll module; it is likely that access to their 
records would be heavily restricted.  This is more easily done by 
group privileges, by giving restricted access to a "directors" group,
than by trying to split the database schema down further.  Similarly,
where there is extensive separation of duties, one group may
have write access for payments but not for invoices.

Finally, use of groups makes it far easier to handle the privileges of
new employees.  It is much easier to make a new employee a member of
the "buyers" group than to grant access rights individually to each of
a number of tables, spread across several schemas.


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
               PGP key from public servers; key ID 32B8FAA1
                 ========================================
     "Therefore, my beloved brethren, be ye stedfast,  
      unmoveable, always abounding in the work of the Lord, 
      forasmuch as ye know that your labour is not in vain 
      in the Lord."      I Corinthians 15:58 



Tue, 09 May 2000 22:33:36 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Groups and schemas (was: You're on SecurityFocus.com...)
When creating a child (through CREATE TABLE ... INHERIT (parent)) it
seems the child gets all of the parent's contraints _except_ its PRIMARY
KEY. Is this normal? Should I add a PRIMARY KEY(id) statement each time
I create an inherited table?

Cheers,

-- 
Louis-David Mitterrand - ldm@apartia.org - http://www.apartia.fr

If at first you don't succeed, redefine success.

Sat, 3 Jun 2000 17:22:56 +0200
Louis-David Mitterrand <cunctator@apartia.ch>
child table doesn't inherit PRIMARY KEY?
On Sat, Jun 03, 2000 at 05:22:56PM +0200, Louis-David Mitterrand wrote:
> When creating a child (through CREATE TABLE ... INHERIT (parent)) it
> seems the child gets all of the parent's contraints _except_ its PRIMARY
> KEY. Is this normal? Should I add a PRIMARY KEY(id) statement each time
> I create an inherited table?

Following up to my previous message, I found that one can't explicitely
add a PRIMARY KEY on child table referencing a field on the parent
table, for instance:

    CREATE TABLE auction (
    id SERIAL PRIMARY KEY,
    title text,
    ... etc...
    );

then 

    CREATE TABLE auction_dvd (
    zone int4,
    PRIMARY KEY("id")
    ) inherits("auction");

doesn't work:
    ERROR:  CREATE TABLE: column 'id' named in key does not exist

But the aution_dvd table doesn't inherit the auction table's PRIMARY
KEY, so I can insert duplicates.

Solutions:

1) don't use PRIMARY KEY, use UNIQUE NOT NULL (which will be inherited?)
but the I lose the index,

2) use the OID field, but it's deprecated by PG developers?

What would be the best solution?

TIA

-- 
Louis-David Mitterrand - ldm@apartia.org - http://www.apartia.fr

Veni, Vidi, VISA.

Sat, 3 Jun 2000 17:51:56 +0200
Louis-David Mitterrand <cunctator@apartia.ch>
Re: child table doesn't inherit PRIMARY KEY?
Louis-David Mitterrand writes:

> When creating a child (through CREATE TABLE ... INHERIT (parent)) it
> seems the child gets all of the parent's contraints _except_ its PRIMARY
> KEY. Is this normal?

It's kind of a bug.


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Sun, 4 Jun 2000 03:46:53 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: child table doesn't inherit PRIMARY KEY?
On Sun, Jun 04, 2000 at 03:46:53AM +0200, Peter Eisentraut wrote:
> Louis-David Mitterrand writes:
> 
> > When creating a child (through CREATE TABLE ... INHERIT (parent)) it
> > seems the child gets all of the parent's contraints _except_ its PRIMARY
> > KEY. Is this normal?
> 
> It's kind of a bug.

Is it a well-known bug or have I discovered it? ;-)

(I am sending a copy of the bug report to -hackers)

Thanks,

-- 
Louis-David Mitterrand - ldm@apartia.org - http://www.apartia.fr

   "God is a mathematician of very high order, and he used very
   advanced mathematics in constructing the universe."  (Dirac)

Sun, 4 Jun 2000 09:20:16 +0200
Louis-David Mitterrand <cunctator@apartia.ch>
Re: child table doesn't inherit PRIMARY KEY?

This is purely a 'we missed some stuff' release, with the only change
being:

Changes
-------
Added documentation to tarball.


So, if you are already running, you don't need this ... this is just to
help Lamar out with the RPMs more then anything ...



Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Mon, 5 Jun 2000 15:23:47 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
[v7.0.2] Very small cleanup release ...
The Hermit Hacker wrote:
  >
  >This is purely a 'we missed some stuff' release, with the only change
  >being:
 
but where is it?  ftp.postgresql.org/pub/latest still points to 7.0.1

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Let your conversation be without covetousness; and be 
      content with such things as ye have. For he hath said,
      I will never leave thee, nor forsake thee."      
                                             Hebrews 13:5 



Mon, 05 Jun 2000 21:42:48 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [HACKERS] [v7.0.2] Very small cleanup release ... 
Oliver Elphick wrote:
> 
> The Hermit Hacker wrote:
>   >
>   >This is purely a 'we missed some stuff' release, with the only change
>   >being:
 
> but where is it?  ftp.postgresql.org/pub/latest still points to 7.0.1

ftp.postgresql.org/pub/source/v7.0.2 was where I pulled it from.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 05 Jun 2000 16:51:59 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] [v7.0.2] Very small cleanup release ...

fixed ...

On Mon, 5 Jun 2000, Oliver Elphick wrote:

> The Hermit Hacker wrote:
>   >
>   >This is purely a 'we missed some stuff' release, with the only change
>   >being:
>  
> but where is it?  ftp.postgresql.org/pub/latest still points to 7.0.1
> 
> -- 
> Oliver Elphick                                Oliver.Elphick@lfix.co.uk
> Isle of Wight                              http://www.lfix.co.uk/oliver
> PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
> GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
>                  ========================================
>      "Let your conversation be without covetousness; and be 
>       content with such things as ye have. For he hath said,
>       I will never leave thee, nor forsake thee."      
>                                              Hebrews 13:5 
> 
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Tue, 6 Jun 2000 00:50:26 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: [HACKERS] [v7.0.2] Very small cleanup release ... 
Hi folks,

This is my first post to your list. I've been reading it for about a week. I like the quality of the developers here and think 
this portends well for the future of Postgres.

Anyway, an idea. Not sure if RDBMSs internally already implement this technique. But in case them don't and in case 
you've never thought of it here something I just thought of:

CHAR fields have different sorting (aka collation) rules for each code page. eg the very fact that A comes before B is 
something that the collation info for a given code page has to specify. Well, just because a character has a lower value 
than another character in its encoding in a given code page doesn't mean it gets sorted first. 

So let me cut to the chase: I'm thinking that rather than store the actual character sequence of each field (or some 
subset of a field) in an index why not translate the characters into their collation sequence values and store _those_ in 
the index? 

The idea is to reduce the number of times that string has to be converted to its mathematical sorting order representation. 
Don't do it every time two strings get compared. Do it when a record is inserted or that field is updated.

Is this already done? Or is it not such a good idea for some reason? 

I'd consider this idea of greater value in something like Unicode. For 16 bit Unicode the lookup table to find each 
character's ordinal value (or sorting value, whatever its called) is 128k, right? Doing a bunch of look-ups into that has to 
not be good for L1 and L2 cache in a processor. 





Wed, 21 Jun 2000 13:14:58 -0700
"Randall Parker" <randall@nls.net>
An idea on faster CHAR field indexing

> So let me cut to the chase: I'm thinking that rather than store the
> actual character sequence of each field (or some subset of a field)
> in an index why not translate the characters into their collation
> sequence values and store _those_ in the index?

This is not an obvious win, since:

1. some collations rules require multiple passes over the data

2. POSIX strxfrm() will convert strings of characters to a form that
   can be compared by strcmp() [i.e. single pass] but tends to greatly
   increase memory requirements

   I've only data for one implementation of strxfrm(), but the memory
   usage startled me.  In my application it was faster to use
   strcoll() directly for collation than to pre-expand the data with
   strxfrm().

Regards,

Giles


Thu, 22 Jun 2000 06:59:06 +1000
Giles Lean <giles@nemeton.com.au>
Re: An idea on faster CHAR field indexing 
Giles,

I'm curious as to why the need for multiple passes. Is that true even in Latin 1 code pages? If not, this optimization 
could at least be used for code pages that don't require multiple passes.

As for memory usage: I don't see the issue here. The translation to some collation sequence has to be done anyhow. 
Writing one's own routine to do look-ups into a collation sequence table is a fairly 
trivial exercise. 

One would have the option with SBCS code pages to either translate to 8 bit collation values or to translate them into 
master Unicode collation values. Not sure what the advantage would be of doing the 
latter. I only see it as useful if you have different rows storing text in different code pages and then only if the RDBMS 
can know for a given field on a per row basis what its code page is.

On Thu, 22 Jun 2000 06:59:06 +1000, Giles Lean wrote:

>
>> So let me cut to the chase: I'm thinking that rather than store the
>> actual character sequence of each field (or some subset of a field)
>> in an index why not translate the characters into their collation
>> sequence values and store _those_ in the index?
>
>This is not an obvious win, since:
>
>1. some collations rules require multiple passes over the data
>
>2. POSIX strxfrm() will convert strings of characters to a form that
>   can be compared by strcmp() [i.e. single pass] but tends to greatly
>   increase memory requirements
>
>   I've only data for one implementation of strxfrm(), but the memory
>   usage startled me.  In my application it was faster to use
>   strcoll() directly for collation than to pre-expand the data with
>   strxfrm().
>
>Regards,
>
>Giles
>




Wed, 21 Jun 2000 14:15:30 -0700
"Randall Parker" <randall@nls.net>
Re: An idea on faster CHAR field indexing
I've seen a number of RDBMSs that require that an entire database (whatever they call a database) 
has to be in the same code page. There are disadvantages in this. With those disadvantages in mind 
and with the idea that it would be worth examining options other than Unicode I typed up a list of 
some of the ways that databases could handle lots of different national languages and code pages. 

Suppose you have to develop a database that will store text strings for a large number of languages 
cutting across several code pages. Well, you have a few choices:

1) Make the entire database Unicode (providing that your RDBMS supports Unicode).
   At least this way you know that you can store any text that you have. Whether it is Swedish, 
Russian, Japanese, German, and English is not a problem. 
   The problem is that all CHAR fields in all tables are now Unicode. That makes all text storage take 
up more room. It also makes sorting and indexing take more time. It also requires translation between 
Unicode and the Code Page that any particular client PC is using. 
    Aside: DB2 has a feature whereby the client PC can declare itself to the RDBMS as being in a 
particular code page and then DB2 does all the back-end to client-end code page translation going in 
both directions. 

2) Allow each table to be in a particular code page.
   This is not too satisfying. How can you do foreign keys between tables when two tables are in two 
different code pages? You'd have to ignore the differences and assume that only the first 127 chars 
are used and that they are the same in the code pages of two different code pages. 

3) Allow individual columns to have code pages to be declared for them. 
   This is okay. But it has downsides:
   A) For every national language or at least for every code page you want to support you have to 
create a new column. 
        Picture a translation table. You might have 30 languages you translate into. So what do you do? 
Make 30 columns? You'd have a column that was a TokenID that describes what phrase or word the 
row represents. Then all the translation columns. 
    Having a column per language results in more columns than if you have a column per code page. 
But if you have a column per code page that results in more rows. The reason is that you might put 
French, German, English, and several other languages in ISO 8859-1. Well, they each need to go on 
a different line. But if only Japanese goes into the SHIFT-JIS (eg CP932 or CP942) column then only 
one of those rows has Japanese in it. Do you put the Japanese on the same row as the English 
translation? Or do you put it on its own row? 
    You end up with a sparse matrix appearance if you do one column for each code page. But you 
end up with a lot more columns if you do one column for each language. Then you run into limits of 
how many columns and how many bytes a particular RDBMS can support. 
   
4) Mix code pages in a single column. Different rows may have different code pages. 
   I've done this in DB2. One gives up the ability to do indexing of that column. After all, whose code 
page collation rules do you use? 
   That limitation is okay if the table is a translation table for token ids that are used in the 
development of software. Basically, the programmers writing some app use Token ID constants in 
their code to specify what text gets displayed. Then the database contains the mappings from those 
token ids to the various national languages. In the case where I did this the database is populated 
from some outside source of data that comes from translators who don't even have direct access to 
the database. The table then is just a storehouse that is indexed on the token id and national 
language fields. 
   Of course, one could come up with some scheme whereby the RDBMS would somehow know for 
each row and field what its code page is. One would need to have a way to declare a field as having a 
variable code page and then to have a way to set a code page value along with the text for a given 
field. I'm not arguing for this approach. Just pointing it out for completeness. 

Note that in order to support the mixing of code pages in a column you have to have one of two 
conditions: 
   A) The column is declared in some code page. But the RDBMS can not enforce the requirement 
that all text going into that column be in the set of encodings that are legal in that code page. Some 
of the code pages that will be put in rows in that column may use encodings that are not legal in the 
declared code page.
   B) Have a binary character type that is not in any code page. eg see DB2's CLOB, DBCLOB, 
VARGRAPHIC and other similar fields. I forget which one of those I used but I have used one of them. 





Wed, 21 Jun 2000 14:25:44 -0700
"Randall Parker" <randall@nls.net>
Thoughts on multiple simultaneous code page support

> I'm curious as to why the need for multiple passes. Is that true
> even in Latin 1 code pages?

Yes.  Some locales want strings to be ordered first by ignoring any
accents on chracters, then using a tie-break on equal strings by doing
a comparison that includes the accents.

To take another of your points out of order: this is an obstacle that
Unicode doesn't resolve.  Unicode gives you a character set capable of
representing characters from many different locales, but collation
order will remain locale specific.

> If not, this optimization could at least
> be used for code pages that don't require multiple passes.

... but due to the increased memory/disk space, this is likely not an
optimisation.  Measurements needed, I'd suggest.

My only experience of this was tuning a sort utility, where the extra
time to convert the strings with strxfrm() and the large additional
memory requirement killed any advantage strcmp() had over strcoll().
Whether this would be the case for database indexes in general or
ideed ever I don't know.

> As for memory usage: I don't see the issue here. The translation to
> some collation sequence has to be done anyhow. 

No; you can do the comparisons in multiple passes instead without
extra storage allocation.  Using multiple passes will be efficient if
the comparisons mostly don't need the second pass, which I suspect is
typical.

> Writing one's own routine to do look-ups into a collation sequence
> table is a fairly trivial exercise.

True.  But if you can't do character-by-character comparisons then
such a simplistic implementation will fail.

I hadn't mentioned this time around (but see the archives for the
recent discussion of LIKE) that there are locales with 2:1 and 1:2
mappings of characters too.

Regards,

Giles


Thu, 22 Jun 2000 11:12:54 +1000
Giles Lean <giles@nemeton.com.au>
Re: An idea on faster CHAR field indexing 

> 1) Make the entire database Unicode
> ...
> It also makes sorting and indexing take more time.

Mentioned in my other email, but what collation order were you
proposing to use?  Binary might be OK for unique keys but that doesn't
help you for '<', '>' etc.

My expectation (not the same as I'd like to see, necessarily, and not
that my opinion counts -- I'm not a developer) would be that each
database have a locale, and that this locale's collation order be used
for indexing, LIKE, '<', '>' etc.  If you want to store data from
multiple human languages using a locale that has Unicode for its
character set would be appropriate/necessary.

Regards,

Giles


Thu, 22 Jun 2000 11:17:14 +1000
Giles Lean <giles@nemeton.com.au>
Re: Thoughts on multiple simultaneous code page support 
Giles,

On Thu, 22 Jun 2000 11:12:54 +1000, Giles Lean wrote:

>Yes.  Some locales want strings to be ordered first by ignoring any
>accents on chracters, then using a tie-break on equal strings by doing
>a comparison that includes the accents.

I guess I don't see how this is really any different. Why order first by the character and second by the accent? For instance, 
if you know the relative order of the various forms of "o" then just give them all successive numbers and do a single pass 
sort. You just have to make sure that all the numbers in that set of numbers are greater than the number you assign to "m" 
and less than the number you assign to "p".

>To take another of your points out of order: this is an obstacle that
>Unicode doesn't resolve.  Unicode gives you a character set capable of
>representing characters from many different locales, but collation
>order will remain locale specific.

With Unicode you have to have a collation order that cuts across what use to be separate character sets in separate code 
pages. 

>... but due to the increased memory/disk space, this is likely not an
>optimisation.  Measurements needed, I'd suggest.

But why is there increased memory and disk space? Do the fields that go into an index not now already get stored twice? 
Does the index just contain a series of references to records and that is it? 






Wed, 21 Jun 2000 18:45:20 -0700
"Randall Parker" <randall@nls.net>
Re: An idea on faster CHAR field indexing
"Randall Parker" <randall@nls.net> writes:
> On Thu, 22 Jun 2000 11:12:54 +1000, Giles Lean wrote:
>> Yes.  Some locales want strings to be ordered first by ignoring any
>> accents on chracters, then using a tie-break on equal strings by doing
>> a comparison that includes the accents.

> I guess I don't see how this is really any different. Why order first
> by the character and second by the accent? For instance, if you know
> the relative order of the various forms of "o" then just give them all
> successive numbers and do a single pass sort. You just have to make
> sure that all the numbers in that set of numbers are greater than the
> number you assign to "m" and less than the number you assign to "p".

Nope.  Would it were that easy.  I don't have a keyboard that will
let me type a proper example, but consider

1.	a  o-with-type-1-accent  c

2.	a  o-with-type-2-accent  b

If type-1 accent sorts before type-2 then your proposal will consider
string 1 less than string 2.  But the correct answer (in these locales)
is the other way round, because you mustn't look at the accents at all
unless you discover that the strings are otherwise equal.  The
determining comparison here is that b < c, therefore string 2 < string 1.

			regards, tom lane

Wed, 21 Jun 2000 23:01:16 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: An idea on faster CHAR field indexing 
Giles Lean <giles@nemeton.com.au> writes:
> My only experience of this was tuning a sort utility, where the extra
> time to convert the strings with strxfrm() and the large additional
> memory requirement killed any advantage strcmp() had over strcoll().
> Whether this would be the case for database indexes in general or
> ideed ever I don't know.

Interesting.  That certainly suggests strxfrm could be a loser for
a database index too, but I agree it'd be nice to see some actual
measurements rather than speculation.

What locale(s) were you using when testing your sort code?  I suspect
the answers might depend on locale quite a bit...

			regards, tom lane

Wed, 21 Jun 2000 23:18:19 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: An idea on faster CHAR field indexing 

> Interesting.  That certainly suggests strxfrm could be a loser for
> a database index too, but I agree it'd be nice to see some actual
> measurements rather than speculation.
> 
> What locale(s) were you using when testing your sort code?  I suspect
> the answers might depend on locale quite a bit...

I did a little more measurement today.  It's still only annecdotal
evidence -- I wasn't terribly rigorous -- but here are my results.

My data file consisted of ~660,000 lines and a total size of ~200MB.
Each line had part descriptions in German and some uninteresting
fields.  I stripped out the uninteresting fields and read the file
calling calling strxfrm() for each line.  I recorded the total input
bytes and the total bytes returned by strxfrm().

HP-UX 11.00 de_DE.roman8 locale:
input bytes:   179647811
result bytes: 1447833496 (increase factor 8.05)

Solaris 2.6 de_CH locale:
input bytes:   179647811 
result bytes: 1085875122 (increase factor 6.04)

I didn't time the test program on Solaris, but on HP-UX this program
took longer to run than a simplistic qsort() using strcoll() does, and
my comparison sort program has to write the data out as well, which
the strxfrm() calling program didn't do.

Regards,

Giles

Thu, 22 Jun 2000 18:47:43 +1000
Giles Lean <giles@nemeton.com.au>
Re: An idea on faster CHAR field indexing 

[ I hope this is useful for the archives, even though it's pretty much
  been covered already. --giles ]

> With Unicode you have to have a collation order that cuts across
> what use to be separate character sets in separate code pages.

 From the glossary at http://www.unicode.org/glossary/index.html:

Collation
    The process of ordering units of textual information. Collation is
    usually specific to a particular language. Also known as
    alphabetizing or alphabetic sorting. Unicode Technical Report #10,
    "Unicode Collation Algorithm," defines a complete, unambiguous,
    specified ordering for all characters in the Unicode Standard.

The "Unicode Technical Report #10" is accessible at:

http://www.unicode.org/unicode/reports/tr10/

This technical report provides a way to specify "the tailoring for any
particular country" which together with the standard ordering will
allow different Unicode implementations to sort any particular input
identically.

Summary is that Unicode still has locales and we still have to know
not only the character set/code page but also the locale ("country
specific tailoring") an index was created with to use it.

Regards,

Giles


Thu, 22 Jun 2000 21:41:40 +1000
Giles Lean <giles@nemeton.com.au>
Re: An idea on faster CHAR field indexing 
Are we interested in adding Try/Catch exception code to PostgreSQL. 
This looks interesting:

	http://www.cs.berkeley.edu/~amc/cexcept/

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 26 Jun 2000 13:02:01 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
C exception code
Oh, well.  At least I asked.  :-)


> 
> > Are we interested in adding Try/Catch exception code to PostgreSQL. 
> > This looks interesting:
> > 
> > 	http://www.cs.berkeley.edu/~amc/cexcept/
> 
> IMHO using the C pre-processor to make C look like some other language:
> 
> - makes the code harder to read as readers have to learn the dialect
>   first
> 
> - makes the code harder to debug, since debugging tools don't know the
>   dialect but only the C it is translated into
> 
> This exception implementation has the obvious(?) problem of using
> setjump()/longjmp() where sigsetjmp()/siglongjmp() would probably be
> necessary for postgresql.
> 
> There are places too where this implementation would just plain not
> work and so couldn't be used: setjmp(), longjmp(), sigsetjump(), and
> siglongjmp() are not async safe signal functions and so can't be
> called in signal handlers, for a start.
> 
> Regards,
> 
> Giles
> 
> 
> 
> 


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 26 Jun 2000 17:34:48 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: C exception code

> Are we interested in adding Try/Catch exception code to PostgreSQL. 
> This looks interesting:
> 
> 	http://www.cs.berkeley.edu/~amc/cexcept/

IMHO using the C pre-processor to make C look like some other language:

- makes the code harder to read as readers have to learn the dialect
  first

- makes the code harder to debug, since debugging tools don't know the
  dialect but only the C it is translated into

This exception implementation has the obvious(?) problem of using
setjump()/longjmp() where sigsetjmp()/siglongjmp() would probably be
necessary for postgresql.

There are places too where this implementation would just plain not
work and so couldn't be used: setjmp(), longjmp(), sigsetjump(), and
siglongjmp() are not async safe signal functions and so can't be
called in signal handlers, for a start.

Regards,

Giles




Tue, 27 Jun 2000 07:36:53 +1000
Giles Lean <giles@nemeton.com.au>
Re: C exception code 
I just spent a rather frustrating hour trying to debug a backend startup
failure --- and getting nowhere because I couldn't catch the failure in
a debugger, or even step to where I thought it might be.  I've seen this
sort of difficulty before, and always had to resort to expedients like
putting in printf's.  But tonight I finally realized what the problem is.

The early stages of startup are run under signal mask BlockSig, which we
initialize to include *EVERY SIGNAL* (except SIGUSR1 for some reason).
In particular SIGTRAP is blocked, which prevents debugger breakpoints
from working.  Even sillier, normally-fatal signals like SIGSEGV are
blocked.  I now know by observation that HPUX, at least, takes this
literally: for example, if you've blocked SEGV you don't hear about bus
errors, you just keep going.  Possibly rather slowly, if every attempted
instruction execution causes the hardware to fault to the kernel, but
by golly the system will keep trying to run your code.

Needless to say I find this braindead in the extreme.  Will anyone
object if I change the signal masks so that we never ever block
SIGABRT, SIGILL, SIGSEGV, SIGBUS, SIGTRAP, SIGCONT, SIGSYS?  Any
other candidates?  Are there any systems that do not define all
of these signal names?

BTW, once I turned this silliness off, I was able to home in on
my bug within minutes...

			regards, tom lane

PS: The postmaster spends most of its time running under BlockSig too.
Good thing we haven't had many postmaster bugs lately.

Mon, 26 Jun 2000 20:44:19 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Is *that* why debugging backend startup is so hard!?
> Needless to say I find this braindead in the extreme.  Will anyone
> object if I change the signal masks so that we never ever block
> SIGABRT, SIGILL, SIGSEGV, SIGBUS, SIGTRAP, SIGCONT, SIGSYS?  Any
> other candidates?  Are there any systems that do not define all
> of these signal names?
> 
> BTW, once I turned this silliness off, I was able to home in on
> my bug within minutes...

Go ahead.  Current setup sound very broken.  Why do they even bother
doing all this.  Seems we should identify the signals we want to block,
and just block those.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 26 Jun 2000 23:47:33 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Is *that* why debugging backend startup is so hard!?
Bruce Momjian wrote:
> 
> Are we interested in adding Try/Catch exception code to PostgreSQL.
> This looks interesting:
> 
>         http://www.cs.berkeley.edu/~amc/cexcept/

How tricky is the error handling in Postgres?

As an aside, I have just started working on a Java project, nearly done
for a company where they have not used the Java exception model. I.e.
there are error codes, setErrorCode, and ifError everywhere. A bigger
mess you will not see. So I'm partial to a decent exception model, and
might even use the above in a project of my own.

Tue, 27 Jun 2000 16:45:35 +1000
Chris Bitmead <chrisb@nimrod.itg.telstra.com.au>
Re: C exception code

> Needless to say I find this braindead in the extreme.

Wow, definitely braindead.  Trapping some of them on systems that can
programmatically generate a stack backtrace might be useful -- it
would help reporting what happened.

Blocking them and continuing seems about the most dangerous thing that
could be done; if we've just got SIGSEGV or similar the code is
confused isn't to be trusted to safely modify data!

> Will anyone object if I change the signal masks so that we never
> ever block SIGABRT, SIGILL, SIGSEGV, SIGBUS, SIGTRAP, SIGCONT,
> SIGSYS?  Any other candidates?  Are there any systems that do not
> define all of these signal names?

I'd expect these everywhere; certainly they're all defined in the
"Single Unix Specification, version 2".  Some of them don't exist in
ANSI C, if that matters.

Usually it's easy enough to wrap code that cares in

#ifdef SIGABRT
...
#endif

so when/if a platform shows up that lacks one or more it's easy to
fix.

Potential additions to your list:

SIGFPE
SIGSTOP (can't be blocked)

Regards,

Giles


Tue, 27 Jun 2000 18:27:26 +1000
Giles Lean <giles@nemeton.com.au>
Re: Is *that* why debugging backend startup is so hard!? 
Tom Lane writes:

> I just spent a rather frustrating hour trying to debug a backend startup
> failure --- and getting nowhere because I couldn't catch the failure in
> a debugger, or even step to where I thought it might be.  I've seen this
> sort of difficulty before, and always had to resort to expedients like
> putting in printf's.  But tonight I finally realized what the problem is.

Could that be contributing to the Heisenbug I decribed on Sunday in "Pid
file magically disappears"?


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Tue, 27 Jun 2000 20:11:59 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Is *that* why debugging backend startup is so hard!?
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> I just spent a rather frustrating hour trying to debug a backend startup
>> failure --- and getting nowhere because I couldn't catch the failure in
>> a debugger, or even step to where I thought it might be.  I've seen this
>> sort of difficulty before, and always had to resort to expedients like
>> putting in printf's.  But tonight I finally realized what the problem is.

> Could that be contributing to the Heisenbug I decribed on Sunday in "Pid
> file magically disappears"?

Hm.  Maybe.  I haven't tried to reproduce the pid-file issue here
(I'm up to my eyebrows in memmgr at the moment).  But the blocking
of SEGV and friends could certainly lead to some odd behavior, due
to code plowing on after getting an error that should have crashed it.

Depending on how robust your local implementation of abort(3) is,
it's even possible that the code would fall through a failed
Assert() test...

			regards, tom lane

Tue, 27 Jun 2000 17:05:49 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Is *that* why debugging backend startup is so hard!? 
Locale does not work on BSD/OS because it does not have LC_MESSAGES.  I
just got this back from BSDI technical support.  Comments?

---------------------------------------------------------------------------

In message <200006100115.VAA06505@candle.pha.pa.us>, Bruce Momjian writes:
>I assume the symbol is supposed to be defined in locale.h.  HP-UX and
>other Unix's have this symbol, but BSDI doesn't.  It is only used when
>PostgreSQL is compile with locale enabled.

>I see no mention of LC_MESSAGES in the include file or manual page.

LC_MESSAGES is apparently an extension; it's not in the POSIX spec, and it's
not in the C spec I have handy.  (Admittedly, that's C99, but I don't think
anything like this was removed since C89.)  So, recommend you #ifdef that;

	#ifdef LC_MESSAGES
	...[interact with LC_MESSAGES]
	#endif

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 28 Jun 2000 10:58:22 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
LC_MESSAGES and BSD/OS

> Locale does not work on BSD/OS because it does not have LC_MESSAGES.  I
> just got this back from BSDI technical support.  Comments?

...

| LC_MESSAGES is apparently an extension; it's not in the POSIX spec, and it's
| not in the C spec I have handy.  (Admittedly, that's C99, but I don't think
| anything like this was removed since C89.)  So, recommend you #ifdef that;
| 
| 	#ifdef LC_MESSAGES
| 	...[interact with LC_MESSAGES]
| 	#endif

POSIX and ANSI C have the following categories:

LC_ALL
LC_COLLATE
LC_CTYPE
LC_MONETARY
LC_NUMERIC
LC_TIME

"Implementation defined additional categories" are allowed, and it
looks like many (most?) implementations use LC_MESSAGES, but they
don't have to, I guess.

Regards,

Giles

Thu, 29 Jun 2000 04:43:26 +1000
Giles Lean <giles@nemeton.com.au>
Re: LC_MESSAGES and BSD/OS 
OK, I am willing to make them #ifdef'ed, if someone would explain what
LC_MESSAGES is used for.


> 
> > Locale does not work on BSD/OS because it does not have LC_MESSAGES.  I
> > just got this back from BSDI technical support.  Comments?
> 
> ...
> 
> | LC_MESSAGES is apparently an extension; it's not in the POSIX spec, and it's
> | not in the C spec I have handy.  (Admittedly, that's C99, but I don't think
> | anything like this was removed since C89.)  So, recommend you #ifdef that;
> | 
> | 	#ifdef LC_MESSAGES
> | 	...[interact with LC_MESSAGES]
> | 	#endif
> 
> POSIX and ANSI C have the following categories:
> 
> LC_ALL
> LC_COLLATE
> LC_CTYPE
> LC_MONETARY
> LC_NUMERIC
> LC_TIME
> 
> "Implementation defined additional categories" are allowed, and it
> looks like many (most?) implementations use LC_MESSAGES, but they
> don't have to, I guess.
> 
> Regards,
> 
> Giles
> 


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 28 Jun 2000 14:46:13 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: LC_MESSAGES and BSD/OS
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> OK, I am willing to make them #ifdef'ed, if someone would explain what
> LC_MESSAGES is used for.

The HPUX locale man pages say:

     LC_CTYPE determines the interpretation of text as single and/or
     multi-byte characters, the classification of characters as printable,
     and the characters matched by character class expressions in regular
     expressions.

     LC_COLLATE provides collation sequence definition
     for relative ordering between collating elements (single- and
     multi-character collating elements) in the locale.

     LC_MESSAGES determines the locale that should be used to affect the
     format and content of diagnostic messages written to standard error,
     and informative messages written to standard output.

     LC_MONETARY defines the rules and symbols used to
     format monetary numeric information.

     LC_NUMERIC defines rules and symbols used to format
     non-monetary numeric information.

     LC_TIME defines the rules for generating locale-specific formatted
     date strings.

     LC_ALL, when set to a non-empty string value, overrides the values of
     all other internationalization variables.

Dunno which LC_foo variable corresponds to LC_MESSAGES on BSDI.

			regards, tom lane

Wed, 28 Jun 2000 21:13:30 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: LC_MESSAGES and BSD/OS 
OK, I have put #ifdef around the LC_MESSAGES to BSDI will work with
locale. Thanks.


> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > OK, I am willing to make them #ifdef'ed, if someone would explain what
> > LC_MESSAGES is used for.
> 
> The HPUX locale man pages say:
> 
>      LC_CTYPE determines the interpretation of text as single and/or
>      multi-byte characters, the classification of characters as printable,
>      and the characters matched by character class expressions in regular
>      expressions.
> 
>      LC_COLLATE provides collation sequence definition
>      for relative ordering between collating elements (single- and
>      multi-character collating elements) in the locale.
> 
>      LC_MESSAGES determines the locale that should be used to affect the
>      format and content of diagnostic messages written to standard error,
>      and informative messages written to standard output.
> 
>      LC_MONETARY defines the rules and symbols used to
>      format monetary numeric information.
> 
>      LC_NUMERIC defines rules and symbols used to format
>      non-monetary numeric information.
> 
>      LC_TIME defines the rules for generating locale-specific formatted
>      date strings.
> 
>      LC_ALL, when set to a non-empty string value, overrides the values of
>      all other internationalization variables.
> 
> Dunno which LC_foo variable corresponds to LC_MESSAGES on BSDI.
> 
> 			regards, tom lane
> 


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 28 Jun 2000 21:19:04 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: LC_MESSAGES and BSD/OS
Tim Perdue wrote:

> On wednesday or thursday, I'm going to be publishing my article on MySQL
> vs. Postgres on PHPBuilder.com.
>
> Before I do that I want to confirm the major problem I had w/postgres:
> the 8K tuple limit. When trying to import some tables from MySQL,
> postgres kept choking because MySQL has no such limit on the size of a
> row in the database (text fields on MySQL can be multi-megabyte).

This is beeing fixed: http://www.postgresql.org/projects/devel-toast.html

>
>
> Is it even possible to import large text fields into postgres? If not,
> how in the world can anyone use this to store message board posts,
> resumes, etc? Do you have to use pgsql-specific large-object
> import/export commands?

I'm currently building a newspaper system and I just split the articles into
8K sections. This is just a workaround until the TOAST project is finished.

>
>
> I actually intended the article to be a win for Postgres, as I've used
> it and had good luck with it for such a long time, but if you look at
> the results below, it seems very positive for MySQL.
>
> Performace/Scalability:
>
> MySQL was About 50-60% faster in real-world web serving, but it crumbles
> under a real load. Postgres on the other hand scaled 3x higher than
> MySQL before it started to crumble on the same machine. Unfortunately,
> Postgres would probably still lose on a high-traffic website because
> MySQL can crank out the pages so much faster, number of concurrent
> connections is hard to compare. MySQL also seems to make better use of
> multiple-processor machines like the quad-xeon I tested on. Postgres
> never saturated all 4 processors as MySQL did.
>
> Tools:
> MySQL has some nice admin tools that allow you to watch individual
> connections and queries as they progress and tools to recover from
> corruption. I haven't seem any similar tools for postgres.

Have you looked at pgAdmin? http://www.pgadmin.freeserve.co.uk/
There is also a tool called pgAccess.

>
> Long-term stability:
> Postgres is undoubtably the long-run winner in stability, whereas MySQL
> will freak out or die when left running for more than a month at a time.
> But if you ever do have a problem with postgres, you generally have to
> nuke the database and recover from a backup, as there are no known tools
> to fix index and database corruption. For a long-running postgres
> database, you will occasionally have to drop indexes and re-create them,
> causing downtime.
>
> Usability:
> Both databases use a similar command-line interface. Postgres uses
> "slash commands" to help you view database structures. MySQL uses a more
> memorable, uniform syntax like "Show Tables; Show Databases; Describe
> table_x;" and has better support for altering/changing tables, columns,
> and even databases.
>
> Features:
> Postgres is undoubtedly far, far more advanced than MySQL is. Postgres
> now supports foreign keys, which can help with referential integrity.
> Postgres supports subselects and better support for creating tables as
> the result of queries. The "transaction" support that MySQL lacks is
> included in Postgres, although you'll never miss it on a website, unless
> you're building something for a bank, and if you're doing that, you'll
> use oracle.

Not true. Transactions are used to make atomic database operations. We use
transactions more than 60 times in our application (we use Cold Fusion).

>
>
> Tim
>
> --
> Founder - PHPBuilder.com / Geocrawler.com
> Lead Developer - SourceForge
> VA Linux Systems
> 408-542-5723

Poul L. Christiansen
Dynamic Paper


Tue, 04 Jul 2000 21:15:22 +0200
"Poul L. Christiansen" <plc@faroenet.fo>
Re: Article on MySQL vs. Postgres
On wednesday or thursday, I'm going to be publishing my article on MySQL
vs. Postgres on PHPBuilder.com.

Before I do that I want to confirm the major problem I had w/postgres:
the 8K tuple limit. When trying to import some tables from MySQL,
postgres kept choking because MySQL has no such limit on the size of a
row in the database (text fields on MySQL can be multi-megabyte).

Is it even possible to import large text fields into postgres? If not,
how in the world can anyone use this to store message board posts,
resumes, etc? Do you have to use pgsql-specific large-object
import/export commands?

I actually intended the article to be a win for Postgres, as I've used
it and had good luck with it for such a long time, but if you look at
the results below, it seems very positive for MySQL.

Performace/Scalability:

MySQL was About 50-60% faster in real-world web serving, but it crumbles
under a real load. Postgres on the other hand scaled 3x higher than
MySQL before it started to crumble on the same machine. Unfortunately,
Postgres would probably still lose on a high-traffic website because
MySQL can crank out the pages so much faster, number of concurrent
connections is hard to compare. MySQL also seems to make better use of
multiple-processor machines like the quad-xeon I tested on. Postgres
never saturated all 4 processors as MySQL did.

Tools:
MySQL has some nice admin tools that allow you to watch individual
connections and queries as they progress and tools to recover from
corruption. I haven't seem any similar tools for postgres.

Long-term stability:
Postgres is undoubtably the long-run winner in stability, whereas MySQL
will freak out or die when left running for more than a month at a time.
But if you ever do have a problem with postgres, you generally have to
nuke the database and recover from a backup, as there are no known tools
to fix index and database corruption. For a long-running postgres
database, you will occasionally have to drop indexes and re-create them,
causing downtime.

Usability:
Both databases use a similar command-line interface. Postgres uses
"slash commands" to help you view database structures. MySQL uses a more
memorable, uniform syntax like "Show Tables; Show Databases; Describe
table_x;" and has better support for altering/changing tables, columns,
and even databases.

Features:
Postgres is undoubtedly far, far more advanced than MySQL is. Postgres
now supports foreign keys, which can help with referential integrity.
Postgres supports subselects and better support for creating tables as
the result of queries. The "transaction" support that MySQL lacks is
included in Postgres, although you'll never miss it on a website, unless
you're building something for a bank, and if you're doing that, you'll
use oracle.

Tim

-- 
Founder - PHPBuilder.com / Geocrawler.com
Lead Developer - SourceForge
VA Linux Systems
408-542-5723

Tue, 04 Jul 2000 12:42:31 -0700
Tim Perdue <tperdue@valinux.com>
Article on MySQL vs. Postgres
Tim Perdue wrote:
> 
> On wednesday or thursday, I'm going to be publishing my article on MySQL
> vs. Postgres on PHPBuilder.com.

Cool!

> Features:
> Postgres is undoubtedly far, far more advanced than MySQL is. Postgres
> now supports foreign keys, which can help with referential integrity.
> Postgres supports subselects and better support for creating tables as
> the result of queries. The "transaction" support that MySQL lacks is
> included in Postgres, although you'll never miss it on a website, unless
> you're building something for a bank, and if you're doing that, you'll
> use oracle.

Since MySQL version 3.23.16 it supports transactions with sleepycats DB3
and since version 3.23.19 it is under the GPL.

-Egon

-- 
SIX Offene Systeme GmbH ï¿½ Stuttgart  -  Berlin  -  New York
Sielminger Straï¿½e 63   ï¿½    D-70771 Leinfelden-Echterdingen
Fon +49 711 9909164 ï¿½ Fax +49 711 9909199 http://www.six.de
PHP-Stand auf Europas grï¿½sster Linux-Messe: 'LinuxTag 2001'
weitere Infos @ http://www.dynamic-webpages.de/

Tue, 04 Jul 2000 21:51:11 +0200
Egon Schmid <eschmid@php.net>
Re: Article on MySQL vs. Postgres
Tim Perdue wrote:
> On wednesday or thursday, I'm going to be publishing my article on MySQL
> vs. Postgres on PHPBuilder.com.
>
> Before I do that I want to confirm the major problem I had w/postgres:
> the 8K tuple limit. When trying to import some tables from MySQL,
> postgres kept choking because MySQL has no such limit on the size of a
> row in the database (text fields on MySQL can be multi-megabyte).

    I  just committed the first portion of TOAST. Enabling lztext
    fields to hold multi-megabytes too. But it's not  the  answer
    to  such  big  objects.   I  have plans to add an Oracle like
    large object handling in a future version.

> I actually intended the article to be a win for Postgres, as I've used
> it and had good luck with it for such a long time, but if you look at
> the results below, it seems very positive for MySQL.

    It's never a good plan to have an initial intention which  of
    the  competitors  should  finally  look  good.  It's  visible
    between the lines.

> Performace/Scalability:
>
> MySQL was About 50-60% faster in real-world web serving, but it crumbles
> under a real load. Postgres on the other hand scaled 3x higher than
> MySQL before it started to crumble on the same machine. Unfortunately,
> Postgres would probably still lose on a high-traffic website because
> MySQL can crank out the pages so much faster, number of concurrent
> connections is hard to compare. MySQL also seems to make better use of
> multiple-processor machines like the quad-xeon I tested on. Postgres
> never saturated all 4 processors as MySQL did.

    The  question  in  this  case  is  "what  is  real-world  web
    serving"?  To  spit  out  static  HTML  pages  loaded  into a
    database? To handle discussion forums like OpenACS with  high
    concurrency and the need for transactions?

    Web  applications  differ  in  database  usage as much as any
    other type of application. From huge amounts of static, never
    changing   data   to   complex   data  structures  with  many
    dependencies constantly in motion.   There  is  no  such  one
    "real world web scenario".

> Tools:
> MySQL has some nice admin tools that allow you to watch individual
> connections and queries as they progress and tools to recover from
> corruption. I haven't seem any similar tools for postgres.

    Yepp, we need alot more nice tools.

> Long-term stability:
> Postgres is undoubtably the long-run winner in stability, whereas MySQL
> will freak out or die when left running for more than a month at a time.
> But if you ever do have a problem with postgres, you generally have to
> nuke the database and recover from a backup, as there are no known tools
> to fix index and database corruption. For a long-running postgres
> database, you will occasionally have to drop indexes and re-create them,
> causing downtime.

    Not true IMHO. We had some problems with indices in the past.
    But you can drop/recreate them online and someone  running  a
    query  concurrently  might  just use a sequential scan during
    that time. All other corruptions need  backup  and  recovery.
    WAL is on it's way.

> Usability:
> Both databases use a similar command-line interface. Postgres uses
> "slash commands" to help you view database structures. MySQL uses a more
> memorable, uniform syntax like "Show Tables; Show Databases; Describe
> table_x;" and has better support for altering/changing tables, columns,
> and even databases.

    Since professional application development starts with a data
    design, such "describe" commands  and  "alter"  features  are
    unimportant.  The  more  someone  needs them, the more I know
    that he isn't well educated.

    Productional installations don't need any "alter" command  at
    all.  New  features  are  developed  in the development area,
    tested with real life data in the test environment and  moved
    to  the  production  server  including  a maybe required data
    conversion step during a downtime.

    24/7  scenarios  require  hot  standby,  online  synchronized
    databases  with  hardware takeover. All that is far away from
    our scope by now.

> Features:
> Postgres is undoubtedly far, far more advanced than MySQL is. Postgres
> now supports foreign keys, which can help with referential integrity.
> Postgres supports subselects and better support for creating tables as
> the result of queries. The "transaction" support that MySQL lacks is
> included in Postgres, although you'll never miss it on a website, unless
> you're building something for a bank, and if you're doing that, you'll
> use oracle.

    FOREIGN KEY  doesn't  help  with  referential  integrity,  it
    guarantees  it.   No  application  must ever worry if it will
    find the customer when it has a problem  report.  It  does  a
    SELECT  and  has  it  or  it would've never found the problem
    report first - period.

    And for big, functional expanding web sites, it does so  even
    if  one  of  a  dozen  programmers  forgot  it  once.  If the
    constraint says you cannot delete a customer who payed  until
    end  of  the year, the database won't let you, even if one of
    the 7 CGI programs that can delete customers doesn't check.

    Transactions are the base for any data integrity.  Especially
    in the web environment. Almost every web server I've seen has
    some timeout for CGI, ADP, ASP or whatever they call  it.  As
    soon  as  your  page needs to update more than one table, you
    run the risk of getting aborted  just  between,  leaving  the
    current  activity half done. No matter if a database supports
    FOREIGN KEY. I could live without it,  but  transactions  are
    essential.

    Fortunately  the MySQL team has changed it's point of view on
    that detail and made some noticeable advantage into that area
    by  integrating BDB. The lates BETA does support transactions
    including rollback as they announced. As far as I see it, the
    integration  of  BDB only buys them transactions, on the cost
    of performance and maintainence efford. So the  need  for  it
    cannot be that small as you think.

    Final notes:

    I  hate  these "MySQL" vs. "PostgreSQL" articles that want to
    say "this one is the better". Each one  has  it's  advantages
    and disadvantages. Both have a long TODO.

    Your  article  might  better  analyze  a  couple of different
    "real-world web services", telling what DB usage profile they
    have  and  then  suggesting which of the two databases is the
    better choice in each case.

    MySQL is a tool and PostgreSQL is a tool. But as  with  other
    tools, a hammer doesn't help if you need a screw driver.

    Please  don't intend to tell anyone either of these databases
    is "the best". You'd do both  communities  a  bad  job.  Help
    people  to  choose the right database for their current needs
    and tell them to reevaluate their choice for the next project
    instead  of blindly staying with the same database. We'll end
    up with alot of customers using both databases  parallel  for
    different needs.

    At  the  bottom  line  both  teams  share the same idea, open
    source. Anyone who pays a license fee is a loss (looser?) for
    all of us.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Wed, 5 Jul 2000 00:39:04 +0200 (MEST)
JanWieck@t-online.de (Jan Wieck)
Re: Article on MySQL vs. Postgres
on 7/4/00 3:42 PM, Tim Perdue at tperdue@valinux.com wrote:

> Before I do that I want to confirm the major problem I had w/postgres:
> the 8K tuple limit. When trying to import some tables from MySQL,
> postgres kept choking because MySQL has no such limit on the size of a
> row in the database (text fields on MySQL can be multi-megabyte).

It's possible in the current version to up your tuple limit to 16K before
compilation, and you can use lztext, the compressed text type, which should
give you up to 32K of storage. Netscape's textarea limit is 32K, so that's a
good basis for doing a number of web-based things. Anything that is
multi-megabyte is really not something I'd want to store in an RDBMS.

> I actually intended the article to be a win for Postgres, as I've used
> it and had good luck with it for such a long time, but if you look at
> the results below, it seems very positive for MySQL.

Jan said that each tool has its value, and that's true. I recommend you
define your evaluation context before you write this. Is this for running a
serious mission-critical web site? Is it for logging web site hits with
tolerance for data loss and a need for doing simple reporting?

> Performace/Scalability:
> 
> MySQL was About 50-60% faster in real-world web serving, but it crumbles
> under a real load. Postgres on the other hand scaled 3x higher than
> MySQL before it started to crumble on the same machine. Unfortunately,
> Postgres would probably still lose on a high-traffic website because
> MySQL can crank out the pages so much faster, number of concurrent
> connections is hard to compare. MySQL also seems to make better use of
> multiple-processor machines like the quad-xeon I tested on. Postgres
> never saturated all 4 processors as MySQL did.

What kind of queries did you perform? Did you use connection pooling (a lot
of PHP apps don't, from what I've seen)? How does the performance get
affected when a query in Postgres with subselects has to be split into 4
different queries in MySQL? Postgres is process-based, each connection
resulting in one process. If you use connection pooling with at least as
many connections as you have processors, you should see it scale quite well.
In fact, for serious load-testing, you should have 10-15 pooled connections.

I *strongly* question your intuition on Postgres running web sites. MySQL's
write performance is very poor, which forces excessive caching (see sites
like Slashdot) to prevent updates from blocking entire web site serving.
Yes, the BDB addition might be useful. Let's see some performance tests
using BDB tables.

> Postgres is undoubtably the long-run winner in stability, whereas MySQL
> will freak out or die when left running for more than a month at a time.
> But if you ever do have a problem with postgres, you generally have to
> nuke the database and recover from a backup, as there are no known tools
> to fix index and database corruption. For a long-running postgres
> database, you will occasionally have to drop indexes and re-create them,
> causing downtime.

Dropping indexes and recreating them does not cause downtime. I've run a
couple of postgres-backed web sites for months on end with no issues. I've
survived a heavy slashdotting on my dual Pentium II-400, with Postgres
WRITES and READS on every Slashdot-referred hit, resulting in perfectly
respectable serving times (less than 3-4 seconds to serve > 20K of data on
each hit). No caching optimization of any kind on the app layer. And I'd
forgotten to vacuum my database for a few days.

> Features:
> Postgres is undoubtedly far, far more advanced than MySQL is. Postgres
> now supports foreign keys, which can help with referential integrity.
> Postgres supports subselects and better support for creating tables as
> the result of queries. The "transaction" support that MySQL lacks is
> included in Postgres, although you'll never miss it on a website, unless
> you're building something for a bank, and if you're doing that, you'll
> use oracle.

I'm just shocked at this. Where did this "transactions aren't necessary"
school of thinking originate? I've been developing database-backed web sites
for 5 years now, and I can't conceive of building a serious web site without
transactions. How do you guarantee that a record and its children records
are all stored together successfully? Do you run on a magic power grid that
never fails? Do you never have code-related error conditions that require
rolling back a series of database edits?

One quick point: while you may well be personally unbiased, VA Linux just
endorsed and funded MySQL. SourceForge uses MySQL. How do you expect to
convince readers that you're being objective in this comparison?

-Ben


Tue, 04 Jul 2000 19:37:25 -0400
Benjamin Adida <ben@mit.edu>
Re: Article on MySQL vs. Postgres
Tim Perdue writes:

> the 8K tuple limit.

BLCKSZ in src/include/config.h -- But it's being worked on these very
days.

> Postgres never saturated all 4 processors as MySQL did.

Blame that on your operating system?

> MySQL has some nice admin tools that allow you to watch individual
> connections and queries as they progress

ps
tail -f <serverlog>

> and tools to recover from corruption. I haven't seem any similar tools
> for postgres.

I always like this one -- "tools to recover from corruption". If your
database is truly corrupted then there's nothing you can do about it, you
need a backup. If your database engine just creates garbage once in a
while then the solution is to fix the database engine, not to provide
external tools to clean up after it.

> as there are no known tools to fix index

REINDEX

> Both databases use a similar command-line interface. Postgres uses
> "slash commands" to help you view database structures. MySQL uses a more
> memorable, uniform syntax like "Show Tables; Show Databases; Describe
> table_x;"

Yeah, but once you have memorized ours then it will be shorter to type. :)
And you get tab completion. And what's so non-uniform about ours?

> The "transaction" support that MySQL lacks is included in Postgres,
> although you'll never miss it on a website,

Think again. Transactions and multi-version concurrency control are
essential for any multi-user web site that expects any writes at all. I'll
reiterate the old Bugzilla bug: User A issues a search that "takes
forever". User B wants to update some information in the database, waits
for user A. Now *every* user in the system, reading or writing, is blocked
waiting for A (and B).

But you don't even have to go that far. What if you just update two
separate tables at once?

If your web site is truly read only, yes, you don't need transactions. But
then you don't need a database either. If your web site does writes, you
need transactions, or you're really not trying hard enough.


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Wed, 5 Jul 2000 02:12:41 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Article on MySQL vs. Postgres
Benjamin Adida wrote:
> Jan said that each tool has its value, and that's true. I recommend you
> define your evaluation context before you write this. Is this for running a
> serious mission-critical web site? Is it for logging web site hits with
> tolerance for data loss and a need for doing simple reporting?

This is for what most people do with PHP and databases - run
semi-critical medium-traffic sites. Anyone running a mission-critical
site would have to look elsewhere for true robustness. I would not at
this time recommend any serious, life-threatening app run On either
database.

> > Performace/Scalability:
> >
> > MySQL was About 50-60% faster in real-world web serving, but it crumbles
> > under a real load. Postgres on the other hand scaled 3x higher than
> > MySQL before it started to crumble on the same machine. Unfortunately,
> > Postgres would probably still lose on a high-traffic website because
> > MySQL can crank out the pages so much faster, number of concurrent
> > connections is hard to compare. MySQL also seems to make better use of
> > multiple-processor machines like the quad-xeon I tested on. Postgres
> > never saturated all 4 processors as MySQL did.
> 
> What kind of queries did you perform? 

I took a real-world page from our site
<http://sourceforge.net/forum/forum.php?forum_id=1> and made it portable
to both databases. Of course, I could not import the "body" of the
message into postgres because of the 8k limitation, so the body had to
be dropped from both databases.

The "nested" view of this page requires joins against three tables and
some recursion to show submessages.

The test was conducted with "ab" (apache benchmark software) using
varying numbers of concurrent connections and 1000 total page views.

The "10% inserts" test is most realistic, as about 10% of all page views
in a discussion forum involve posting to the database. I used a
random-number generator in the PHP script to insert a row into the table
10% of the time. If you look at the results, you'll see that MySQL was
actually harmed somewhat more by the writes than postgres was.

Here are the actual results I saw on my quad-xeon machine:

postgres:

concurrency w/pconnects:
10 cli - 10.27 pg/sec 333.69 kb/s
20 cli - 10.24 pg/sec 332.86 kb/s
30 cli - 10.25 pg/sec 333.01 kb/s
40 cli - 10.0 pg/sec 324.78 kb/s
50 cli - 10.0 pg/sec 324.84 kb/s
75 cli - 9.58 pg/sec 311.43 kb/s
90 cli - 9.48 pg/sec 307.95 kb/s
100 cli - 9.23 pg/sec 300.00 kb/s
110 cli - 9.09 pg/sec 295.20 kb/s
120 cli - 9.28 pg/sec 295.02 kb/s (2.2% failure)

concurrency w/10% inserts & pconnects:
30 cli - 9.97 pg/sec 324.11 kb/s
40 cli - 10.08 pg/sec 327.40 kb/s
75 cli - 9.51 pg/sec 309.13 kb/s

MySQL:

Concurrency Tests w/pconnects:
30 cli - 16.03 pg/sec   521.01 kb/s
40 cli - 15.64 pg/sec   507.18 kb/s *failures
50 cli - 15.43 pg/sec   497.88 kb/s *failures
75 cli - 14.70 pg/sec   468.64 kb/s *failures
90 - mysql dies
110 - mysql dies
120 - mysql dies

Concurrency Tests w/o pconnects:
10 cli - 16.55 pg/sec 537.63 kb/s
20 cli - 15.99 pg/sec 519/51 kb/s
30 cli - 15.55 pg/sec 505.19 kb/s
40 cli - 15.46 pg/sec 490.01 kb/s 4.7% failure
50 cli - 15.59 pg/sec 482.24 kb/s 8.2% failure
75 cli - 17.65 pg/sec 452.08 kb/s 36.3% failure
90 cli - mysql dies

concurrency w/10% inserts & pconnects:
20 cli - 16.37 pg/sec 531.79 kb/s
30 cli - 16.15 pg/sec 524.64 kb/s
40 cli - 22.04 pg/sec 453.82 kb/sec 37.8% failure


> Did you use connection pooling (a lot

I used persistent connections, yes. Without them, Postgres' showing was
far poorer, with mysql showing about 2x the performance.



> of PHP apps don't, from what I've seen)? How does the performance get
> affected when a query in Postgres with subselects has to be split into 4
> different queries in MySQL?

I'd really love to see a case where a real-world page view requires 4x
the queries on MySQL. If you are doing subselects like that on a website
in real-time you've got serious design problems and postgres would
fold-up and quit under the load anyway.


> Postgres is process-based, each connection
> resulting in one process. If you use connection pooling with at least as
> many connections as you have processors, you should see it scale quite well.
> In fact, for serious load-testing, you should have 10-15 pooled connections.
> 
> I *strongly* question your intuition on Postgres running web sites. MySQL's

Specifically, what is the problem with my "intuition"? All I did in the
prior message was report my results and ask for feedback before I post
it.


> write performance is very poor, which forces excessive caching (see sites
> like Slashdot) to prevent updates from blocking entire web site serving.
> Yes, the BDB addition might be useful. Let's see some performance tests
> using BDB tables.

I wouldn't use BDB tables as MySQL 3.23.x isn't stable and I wouldn't
use it until it is.


> > Postgres is undoubtably the long-run winner in stability, whereas MySQL
> > will freak out or die when left running for more than a month at a time.
> > But if you ever do have a problem with postgres, you generally have to
> > nuke the database and recover from a backup, as there are no known tools
> > to fix index and database corruption. For a long-running postgres
> > database, you will occasionally have to drop indexes and re-create them,
> > causing downtime.
> 
> Dropping indexes and recreating them does not cause downtime. I've run a
> couple of postgres-backed web sites for months on end with no issues. I've
> survived a heavy slashdotting on my dual Pentium II-400, with Postgres
> WRITES and READS on every Slashdot-referred hit, resulting in perfectly
> respectable serving times (less than 3-4 seconds to serve > 20K of data on
> each hit). No caching optimization of any kind on the app layer. And I'd
> forgotten to vacuum my database for a few days.

Not sure why you're arguing with this as this was a clear win for
postgres.


> Do you run on a magic power grid that
> never fails?

Reality is that postgres is as likely - or more likely - to wind up with
corrupted data than MySQL. I'm talking physical corruption where I have
to destroy the database and recover from a dump. Just a couple months
ago I sent a message about "Eternal Vacuuming", in which case I had to
destroy and recover a multi-gigabyte database.

Further, I have had situations where postgres actually had DUPLICATE ids
in a primary key field, probably due to some abort or other nasty
situation in the middle of a commit. How did I recover from That? Well,
I had to run a count(*) next to each ID and select out the rows where
there was more than one of each "unique" id, then reinsert those rows
and drop and rebuild the indexes and reset the sequences.

I've only been using MySQL for about a year (as compared to 2 years for
postgres), but I have never seen either of those problems with MySQL.


> Do you never have code-related error conditions that require
> rolling back a series of database edits?

Personally, I check every query in my PHP code. On the rare occasion
that it fales, I show an error and get out. Even with postgres, I have
always checked success or failure of a query and shown an appropriate
error. Never in two years of programming PHP/postgres have I ever used
commit/rollback, and I have written some extremely complex web apps
(sourceforge being a prime example). Geocrawler.com runs on postgres and
again, I NEVER saw any need for any kind of rollback at all.

The statelessness of the web pretty much obviates the needs for
locks/rollbacks as each process is extremely quick and runs from start
to finish instantly. It's not like the old days where you pull data down
into a local application, work on it, then upload it again.

Only now, with some extremely complex stuff that we're doing on
SourceForge would I like to see locks and rollbacks (hence my recent
interest in benchmarking and comparing the two). Your average web
programmer will almost never run into that in the short term.

 
> One quick point: while you may well be personally unbiased, VA Linux just
> endorsed and funded MySQL. SourceForge uses MySQL. How do you expect to
> convince readers that you're being objective in this comparison?

Your own strong biases are shown in your message. I do this stuff
because I'm curious and want to find out for myself. Most readers will
find it interesting as I did. Few will switch from MySQL to postgres or
vice versa because of it.

Another clarification: PHPBuilder is owned by internet.com, a competitor
of VA Linux/Andover.

Tim

-- 
Founder - PHPBuilder.com / Geocrawler.com
Lead Developer - SourceForge
VA Linux Systems
408-542-5723

Tue, 04 Jul 2000 17:30:51 -0700
Tim Perdue <tperdue@valinux.com>
Re: Article on MySQL vs. Postgres
Tim Perdue wrote:
> 
> This is for what most people do with PHP and databases - run
> semi-critical medium-traffic sites. Anyone running a mission-critical
> site would have to look elsewhere for true robustness. I would not at
> this time recommend any serious, life-threatening app run On either
> database.
> 

I've seen problems with block read errors in large Oracle
databases which fail their alleged CRC check -- intermittent core
dumps which required a dump/restore of 25 years of insurance
claims data (40 gig - it was a lot at the time). After being down
for days and restoring on a new box, the same errors occured. 

> 
> I'd really love to see a case where a real-world page view requires 4x
> the queries on MySQL. If you are doing subselects like that on a website
> in real-time you've got serious design problems and postgres would
> fold-up and quit under the load anyway.

This can be true for Internet sites, of course. But with
corporate Intranet sites that dish-out and process ERP data, the
queries can become quite complex while concurrency is limited to
< 1000 users.

> Further, I have had situations where postgres actually had DUPLICATE ids
> in a primary key field, probably due to some abort or other nasty
> situation in the middle of a commit. How did I recover from That? Well,
> I had to run a count(*) next to each ID and select out the rows where
> there was more than one of each "unique" id, then reinsert those rows
> and drop and rebuild the indexes and reset the sequences.

Umm...

DELETE FROM foo WHERE EXISTS 
(SELECT f.key FROM foo f WHERE f.key = foo.key AND f.oid >
foo.oid);

I believe there's even a purely SQL (non-oid) method of doing
this as well.

> Personally, I check every query in my PHP code. On the rare occasion
> that it fales, I show an error and get out. Even with postgres, I have
> always checked success or failure of a query and shown an appropriate
> error. Never in two years of programming PHP/postgres have I ever used
> commit/rollback, and I have written some extremely complex web apps
> (sourceforge being a prime example). Geocrawler.com runs on postgres and
> again, I NEVER saw any need for any kind of rollback at all.

This is the nature of the application. In the same example above,
how can I "charge" a cost center for the purchase of products in
an in-house distribution center and "deduct" the resulting
quantity from the distribution center's on-hand inventory sanely
without transactions?

Mike Mascari

Tue, 04 Jul 2000 20:51:30 -0400
Mike Mascari <mascarm@mascari.com>
Re: Article on MySQL vs. Postgres
Tim Perdue wrote:

> I'd really love to see a case where a real-world page view requires 4x
> the queries on MySQL. If you are doing subselects like that on a website
> in real-time you've got serious design problems and postgres would
> fold-up and quit under the load anyway.

Why? There are some subselect queries that have no problems running in
real-time. There are some non-subselect queries which one should never
attempt in real time. There is nothing fundamentally wrong with using
subselects for page views if it works for you. Nor is there anything
necessarily wrong with a design that requires subselects.

> > Do you run on a magic power grid that
> > never fails?
> 
> Reality is that postgres is as likely - or more likely - to wind up with
> corrupted data than MySQL.

What do you base this statement on? With your sample size of one
corrupted postgres database? Also do you include inconsistent data in
your definition of corrupted data?

> Never in two years of programming PHP/postgres have I ever used
> commit/rollback, and I have written some extremely complex web apps
> (sourceforge being a prime example). 

I would humbly suggest that you are doing it wrong then.

> Geocrawler.com runs on postgres and
> again, I NEVER saw any need for any kind of rollback at all.

So what do you do when you get an error and "get out" as you put it?
Leave the half-done work in the database?

> The statelessness of the web pretty much obviates the needs for
> locks/rollbacks as each process is extremely quick and runs from start
> to finish instantly. It's not like the old days where you pull data down
> into a local application, work on it, then upload it again.

Even in the "old days" you should never keep a transaction open while
you "work on it". Transactions should *always* be short, and the web
changes nothing.

Really, REALLY there is nothing different about the web to traditional
applications as far as the db is concerned.

Wed, 05 Jul 2000 12:24:12 +1000
Chris Bitmead <chrisb@nimrod.itg.telstra.com.au>
Re: Article on MySQL vs. Postgres
On Tue, 4 Jul 2000, Tim Perdue wrote:

> Performace/Scalability:
> 
> MySQL was About 50-60% faster in real-world web serving, but it
> crumbles under a real load. Postgres on the other hand scaled 3x
> higher than MySQL before it started to crumble on the same machine.
> Unfortunately, Postgres would probably still lose on a high-traffic
> website because MySQL can crank out the pages so much faster

	Actually, this one depends alot on how the site is
setup/programmed.  I did work with a friend several months ago using the
newest released versions of MySQL and PostgreSQL ... we loaded (with some
massaging) the exact same data/tables onto both on the *exact* same
machine, and the exact same operating system.  When we ran their existing
web site, without modifications, on both MySQL and PgSQL, the MySQL was
substantially faster ... when we spent a little bit of time looking at the
queries used, we found that due to MySQLs lack of sub-queries, each page
being loaded had to do multiple queries to get the same information that
we could get out of PgSQL using one.  Once we optimized the queries, our
timings to load the page went from something like 3sec for MySQL and 1sec
for PgSQL ... (vs something like, if I recall correctly, 19sec for
PgSQL) ...

	Same with some recent work I did with UDMSearch ... by default,
UDMSearch does 2+n queries to the database to get the information it
requires ... by re-writing the 'n' queries that are performed as an IN
query, I was able to cut down searches from taking ~1sec*n queries down to
a 3sec query ...

	The point being that if you do a 1:1 comparison, MySQL will be
faster ... if you use features in PgSQL that don't exist in MySQL, you can
knock that speed difference down considerably, if not surpass MySQL,
depending on the circumstance ...



Tue, 4 Jul 2000 23:31:32 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Article on MySQL vs. Postgres
>     Final notes:
> 
>     I  hate  these "MySQL" vs. "PostgreSQL" articles that want to
>     say "this one is the better". Each one  has  it's  advantages
>     and disadvantages. Both have a long TODO.

Also, none of the 'comparisons' take the time to deal with the fact that
ones "disadvantages" can generally be overcome using its
"advantages" (ie. speed issues with PostgreSQL can generally be overcome
by making use of its high end features (ie. subselects)) ...



Wed, 5 Jul 2000 00:19:45 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Article on MySQL vs. Postgres
On Tue, 4 Jul 2000, Benjamin Adida wrote:

> Dropping indexes and recreating them does not cause downtime. I've run a

Just got hit with a 'bits moved;recreate index' on the PostgreSQL search
engine ... drop'd and re-created index on the fly, no server shut down ...

> couple of postgres-backed web sites for months on end with no issues. I've
> survived a heavy slashdotting on my dual Pentium II-400, with Postgres
> WRITES and READS on every Slashdot-referred hit, resulting in perfectly
> respectable serving times (less than 3-4 seconds to serve > 20K of data on
> each hit). No caching optimization of any kind on the app layer. And I'd
> forgotten to vacuum my database for a few days.

We had a *very* old version of PostgreSQL running on a Pentium acting as
an accounting/authentication backend to a RADIUS server for an ISP
... uptime for the server itself was *almost* 365 days (someone hit the
power switch by accident, meaning to power down a different machine
*sigh*) ... PostgreSQL server had been up for something like 6 months
without any problems, with the previous downtime being to upgrade the
server ...

 > > > Features:
> > Postgres is undoubtedly far, far more advanced than MySQL is. Postgres
> > now supports foreign keys, which can help with referential integrity.
> > Postgres supports subselects and better support for creating tables as
> > the result of queries. The "transaction" support that MySQL lacks is
> > included in Postgres, although you'll never miss it on a website, unless
> > you're building something for a bank, and if you're doing that, you'll
> > use oracle.
> 
> I'm just shocked at this. Where did this "transactions aren't necessary"
> school of thinking originate? 

Ummm, hate to disparage someone else, and I may actually be incorrect, but
I'm *almost* certain that MySQL docs, at one time, had this in it
... where they were explaining why they didn't have and never would have
transaction support.  Obviously this mentality has changed since, with
the recent addition of transactions through a third-party database product
(re: Berkeley DB) ...

> I've been developing database-backed web sites for 5 years now, and I
> can't conceive of building a serious web site without transactions.
> How do you guarantee that a record and its children records are all
> stored together successfully? Do you run on a magic power grid that
> never fails? Do you never have code-related error conditions that
> require rolling back a series of database edits?

Actually, hate to admit it, but it wasn't until recently that I clued into
what transaction were for and how they wre used :(  I now use them for
just about everything I do, and couldn't imagine doing without them ...



Wed, 5 Jul 2000 00:28:36 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Article on MySQL vs. Postgres
On Wed, 5 Jul 2000, Peter Eisentraut wrote:

> If your web site is truly read only, yes, you don't need transactions. But
> then you don't need a database either. If your web site does writes, you
> need transactions, or you're really not trying hard enough.

	... or not popular enough :)



Wed, 5 Jul 2000 00:30:19 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Article on MySQL vs. Postgres
On Tue, 4 Jul 2000, Tim Perdue wrote:

> Benjamin Adida wrote:
> > Jan said that each tool has its value, and that's true. I recommend you
> > define your evaluation context before you write this. Is this for running a
> > serious mission-critical web site? Is it for logging web site hits with
> > tolerance for data loss and a need for doing simple reporting?
> 
> This is for what most people do with PHP and databases - run
> semi-critical medium-traffic sites. Anyone running a mission-critical
> site would have to look elsewhere for true robustness. I would not at
> this time recommend any serious, life-threatening app run On either
> database.

Someone want to give me an example of something that would be
life-threatening that would run on a database?  I can think of loads of
mission critical stuff, but life threatening?  As for mission critical,
mission critical is in the eye of the end-user ... all my clients run
PostgreSQL for their backend needs, and I can guarantee you that each and
every one of them considers it a mission critical element to their sites
... then again, I have 3+ years of personal experience with PostgreSQL to
back me up ..

> I took a real-world page from our site
> <http://sourceforge.net/forum/forum.php?forum_id=1> and made it portable
> to both databases. Of course, I could not import the "body" of the

did you take the time to optimize the queries to take advantage of
features that MySQL doesn't have, or just straight plug-n-play?

> > of PHP apps don't, from what I've seen)? How does the performance get
> > affected when a query in Postgres with subselects has to be split into 4
> > different queries in MySQL?
> 
> I'd really love to see a case where a real-world page view requires 4x
> the queries on MySQL. If you are doing subselects like that on a website
> in real-time you've got serious design problems and postgres would
> fold-up and quit under the load anyway.

Odd, I'll have to let one of my clients know that their site has design
flaws ... wait, no, they had 3x the queries in MySQL as in PgSQL, so that
probably doesnt' apply ...

> > Do you run on a magic power grid that
> > never fails?
> 
> Reality is that postgres is as likely - or more likely - to wind up with
> corrupted data than MySQL. I'm talking physical corruption where I have
> to destroy the database and recover from a dump. 

Odd, in my 3+ years of PostgreSQL development, I've yet to have a
project/database corrupt such that I had to restore from backups *knock on
wood*  INDEX corruption, yup ... 'DROP INDEX/CREATE INDEX' fixes that
though.  Physical database corruption, nope ...

> Further, I have had situations where postgres actually had DUPLICATE
> ids in a primary key field, probably due to some abort or other nasty
> situation in the middle of a commit. How did I recover from That?
> Well, I had to run a count(*) next to each ID and select out the rows
> where there was more than one of each "unique" id, then reinsert those
> rows and drop and rebuild the indexes and reset the sequences.

Odd, were you using transactions here, or transactionless?

> > Do you never have code-related error conditions that require
> > rolling back a series of database edits?
> 
> Personally, I check every query in my PHP code. On the rare occasion
> that it fales, I show an error and get out. Even with postgres, I have
> always checked success or failure of a query and shown an appropriate
> error. Never in two years of programming PHP/postgres have I ever used
> commit/rollback, and I have written some extremely complex web apps
> (sourceforge being a prime example). Geocrawler.com runs on postgres and
> again, I NEVER saw any need for any kind of rollback at all.

Wait ... how does checking every query help if QUERY2 fails after QUERY1
is sent, and you aren't using transactions?

> Only now, with some extremely complex stuff that we're doing on
> SourceForge would I like to see locks and rollbacks (hence my recent
> interest in benchmarking and comparing the two). Your average web
> programmer will almost never run into that in the short term.

Cool, at least I'm not considered average :)  I *always* use transactions
in my scripts ... *shrug*  then again, I'm heavily into 'the rules of
normalization', so tend to not crowd everything into one table.  



Wed, 5 Jul 2000 00:40:18 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Article on MySQL vs. Postgres
The Hermit Hacker wrote:
> > Further, I have had situations where postgres actually had DUPLICATE
> > ids in a primary key field, probably due to some abort or other nasty
> > situation in the middle of a commit. How did I recover from That?
> > Well, I had to run a count(*) next to each ID and select out the rows
> > where there was more than one of each "unique" id, then reinsert those
> > rows and drop and rebuild the indexes and reset the sequences.
> 
> Odd, were you using transactions here, or transactionless?

Does it matter? I suppose it was my programming error that somehow I got
duplicate primary keys in a table in the database where that should be
totally impossible under any circumstance? Another stupid
transactionless program I'm sure.

At any rate, it appears that the main problem I had with postgres (the
8K tuple limit) is being fixed and I will mention that in my writeup.

Tim

-- 
Founder - PHPBuilder.com / Geocrawler.com
Lead Developer - SourceForge
VA Linux Systems
408-542-5723

Tue, 04 Jul 2000 21:08:36 -0700
Tim Perdue <tperdue@valinux.com>
Re: Article on MySQL vs. Postgres
----- Original Message -----
From: "Tim Perdue" <tperdue@valinux.com>

> Before I do that I want to confirm the major problem I had w/postgres:
> the 8K tuple limit.

    Just wanted to point out that this is not *exactly* true.  While the
default limit is 8k, all that is required to change it to 32k is to change
one line of text in config.h (blcksz from 8k to 32k).  This is pointed out
in the FAQ.  So I would really consider the *default* to be 8k and the
*limit* to be 32k.  IMHO 32k is good enough for 99% of tuples in a typical
bulletin-board-like application.  It is not unreasonable to reject posts >
32k in size.  Though you might want to evaluate performance using the 32k
tuples; might increase or decrease depending on application.

                  -Mike


Wed, 5 Jul 2000 00:50:06 -0400
"Michael Mayo" <michael-a-mayo@worldnet.att.net>
Re: Article on MySQL vs. Postgres
Tim Perdue wrote:
> 
> MySQL was About 50-60% faster in real-world web serving ...

Sorry if I didn't noticed, but I searched all the messages in the thread
for an information about the PostgreSQL version used in the test and
didn't found anything.

Tim, what version of PostgreSQL did you used? Hope it's 7.x.

Constantin Teodorescu
FLEX Consulting Braila, ROMANIA

Wed, 05 Jul 2000 09:11:11 +0300
Constantin Teodorescu <teo@flex.ro>
Re: Article on MySQL vs. Postgres
Constantin Teodorescu wrote:
> Tim, what version of PostgreSQL did you used? Hope it's 7.x.

Yes, 7.0.2

Tim

-- 
Founder - PHPBuilder.com / Geocrawler.com
Lead Developer - SourceForge
VA Linux Systems
408-542-5723

Tue, 04 Jul 2000 23:40:44 -0700
Tim Perdue <tperdue@valinux.com>
Re: Article on MySQL vs. Postgres
The Hermit Hacker wrote:
> 
> On Tue, 4 Jul 2000, Tim Perdue wrote:
> 
> > Further, I have had situations where postgres actually had DUPLICATE
> > ids in a primary key field, probably due to some abort or other nasty
> > situation in the middle of a commit. How did I recover from That?
> > Well, I had to run a count(*) next to each ID and select out the rows
> > where there was more than one of each "unique" id, then reinsert those
> > rows and drop and rebuild the indexes and reset the sequences.
> 
> Odd, were you using transactions here, or transactionless?

Actully I think I remember a recent bug report about some condition that 
failed the uniqueness check when inside a transaction ;(

I think the report came with a fix ;)

------------
Hannu

Wed, 05 Jul 2000 11:39:39 +0300
Hannu Krosing <hannu@tm.ee>
Re: Article on MySQL vs. Postgres
Tim Perdue wrote:
> 
> The Hermit Hacker wrote:
> > > Further, I have had situations where postgres actually had DUPLICATE
> > > ids in a primary key field, probably due to some abort or other nasty
> > > situation in the middle of a commit. How did I recover from That?
> > > Well, I had to run a count(*) next to each ID and select out the rows
> > > where there was more than one of each "unique" id, then reinsert those
> > > rows and drop and rebuild the indexes and reset the sequences.
> >
> > Odd, were you using transactions here, or transactionless?
> 
> Does it matter? I suppose it was my programming error that somehow I got
> duplicate primary keys in a table in the database where that should be
> totally impossible under any circumstance? Another stupid
> transactionless program I'm sure.
> 
> At any rate, it appears that the main problem I had with postgres (the
> 8K tuple limit) is being fixed and I will mention that in my writeup.

Currently (as of 7.0.x) you could use BLKSIZE=32K + lztext datatype and 
get text fields about 64-128K depending on data if you are desperately 
after big textfields.

-----------
Hannu

Wed, 05 Jul 2000 11:46:52 +0300
Hannu Krosing <hannu@tm.ee>
Re: Article on MySQL vs. Postgres
The Hermit Hacker wrote:
> On Tue, 4 Jul 2000, Tim Perdue wrote:
>
> > I took a real-world page from our site
> > <http://sourceforge.net/forum/forum.php?forum_id=1> and made it portable
> > to both databases. Of course, I could not import the "body" of the
>
> did you take the time to optimize the queries to take advantage of
> features that MySQL doesn't have, or just straight plug-n-play?
>

    What a "real-world", one single URL, whow.

    The  "made  it portable to both" lets me think it is stripped
    down to the common denominator that both  databases  support.
    That is no transactions, no subqueries, no features.

    That's  no  "comparision",  it's  BS  - sorry. If you want to
    write  a  good  article,  take  a  couple  of  existing   web
    applications  and  analyze the complexity of their underlying
    data model, what features are important/unimportant for  them
    and  what  could  be  done better in them with each database.
    Then make suggestions  which  application  should  use  which
    database and explain why you think so.

> > Further, I have had situations where postgres actually had DUPLICATE
> > ids in a primary key field, probably due to some abort or other nasty
> > situation in the middle of a commit. How did I recover from That?
> > Well, I had to run a count(*) next to each ID and select out the rows
> > where there was more than one of each "unique" id, then reinsert those
> > rows and drop and rebuild the indexes and reset the sequences.
>
> Odd, were you using transactions here, or transactionless?

    Mark,  you  cannot  use Postgres transactionless. Each single
    statement run outside of a transaction  block  has  it's  own
    transaction.

    Anyway,  what  version  of  Postgres  was it? How big was the
    indexed field?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Wed, 5 Jul 2000 10:55:11 +0200 (MEST)
JanWieck@t-online.de (Jan Wieck)
Re: Article on MySQL vs. Postgres
The Hermit Hacker wrote:
> Someone want to give me an example of something that would be
> life-threatening that would run on a database?  I can think of loads of
> mission critical stuff, but life threatening? 

How soon we forget the Y2K horror story warnings....

Pharmacy dispensary systems <-No meds, folks die.
Medical needs supply chain systems <- No Meds or Gauze or Tools, folks die.
Surgery scheduling systems <- No doctors or rooms for surgery, folks die
Military bombing flight-path systems <-Bad data for bomb location...
Weapons Design specifications storage <- Poorly designed systems killing
the testers and military users
Powergrid billing info <-No power, on assisted living (life support)
Banking/Financial account data <-No money, slow death of hunger
Food Shipping systems <- No food
Water distribution/management systems <- No water (I live in a desert)

Just off of the top of my head, yes, it's possible to kill people
with bad data.

-Bop

--
Brought to you from iBop the iMac, a MacOS, Win95, Win98, LinuxPPC machine,
which is currently in MacOS land.  Your bopping may vary.

Wed, 05 Jul 2000 02:02:36 -0700
Ron Chmara <ron@Opus1.COM>
Re: Article on MySQL vs. Postgres

> Someone want to give me an example of something that would be
> life-threatening that would run on a database?

Medical records: I've stored blood type, HIV status, general pathology
results, and radiology results in a database.

A government site I know about stores court records about domestic
violence orders.  Access to this information is required on short
notice and its absence can definitely be life threatening.

Life-threatening doesn't have to be realtime.

Regards,

Giles




Wed, 05 Jul 2000 19:41:20 +1000
Giles Lean <giles@nemeton.com.au>
Re: Article on MySQL vs. Postgres 
Hannu Krosing wrote:
> 
> Tim Perdue wrote:
> >
> > The Hermit Hacker wrote:
> > > > Further, I have had situations where postgres actually had DUPLICATE
> > > > ids in a primary key field, probably due to some abort or other nasty
> > > > situation in the middle of a commit. How did I recover from That?
> > > > Well, I had to run a count(*) next to each ID and select out the rows
> > > > where there was more than one of each "unique" id, then reinsert those
> > > > rows and drop and rebuild the indexes and reset the sequences.

There a bug report that allowed tuplicate ids in an uniqe field when 
SELECT FOR UPDATE was used. Could this be your case ?

---8<-------8<-------8<-------8<-------8<-------8<-------8<-------8<----
gamer=# create table test(i int primary key);
NOTICE:  CREATE TABLE/PRIMARY KEY will create implicit index 'test_pkey'
for table 'test'
CREATE
gamer=# insert into test values(1);
INSERT 18860 1
gamer=# begin;
BEGIN
gamer=# select * from test for update;
 i 
---
 1
(1 row)

gamer=# insert into test values(1);
INSERT 18861 1
gamer=# commit;
COMMIT
gamer=# select * from test;
 i 
---
 1
 1
(2 rows)

gamer=# insert into test values(1);
ERROR:  Cannot insert a duplicate key into unique index test_pkey
---8<-------8<-------8<-------8<-------8<-------8<-------8<-------8<----

IIRC the fix was also provided, so it could be fixed in current CVS (the
above 
is from 7.0.2, worked the same in 6.5.3)

> > > Odd, were you using transactions here, or transactionless?

Ironically the above has to be using transactions as select for update
works 
like this only inside transactions and is thus ineffectif if 
transaction=statement;

As multi-command statements are run as a single transaction 
(which can't be done from psql as it does its own splittng ;()
so a command like 'select * from test for update;insert into test
values(1);'
has the same effect 

> > Does it matter? I suppose it was my programming error that somehow I got
> > duplicate primary keys in a table in the database where that should be
> > totally impossible under any circumstance? Another stupid
> > transactionless program I'm sure.

constraints and transactions are quite different (though connected)
things.

lack of some types of constraints (not null, in (1,2,3)) can be overcome 
with careful programming, others like foreign keys or unique can't
unless 
transactions are used)

no amount of careful programming will overcome lack of transactions
(except 
implementing transactions yourself ;)

 
-----------
Hannu

Wed, 05 Jul 2000 14:24:42 +0300
Hannu Krosing <hannu@tm.ee>
Re: Article on MySQL vs. Postgres
"Robert B. Easter" wrote:
> 
> 
> While it is slow, I've been able to store unlimited amounts of text into
> the database by using the following code. 

Thanks for a really nice exaple !

> I've tested inserting over 4
> megabytes from a TEXTAREA web form using PHP.  When inserting such massive
> amounts of text, you will have to wait a while, but it will eventually succeed
> if you don't run out of memory.  If you do run out of memory, the backend
> terminates gracefully and the transaction aborts/rollsback.
> 
> -- Load the PGSQL procedural language
> -- This could also be done with the createlang script/program.
> -- See man createlang.
> CREATE FUNCTION plpgsql_call_handler()
>         RETURNS OPAQUE AS '/usr/local/pgsql/lib/plpgsql.so'
>         LANGUAGE 'C';
> 
> CREATE TRUSTED PROCEDURAL LANGUAGE 'plpgsql'
>         HANDLER plpgsql_call_handler
>         LANCOMPILER 'PL/pgSQL';

You probably meant pl/tcl as all your code is using that ?

---------
Hannu

Wed, 05 Jul 2000 14:51:39 +0300
Hannu Krosing <hannu@tm.ee>
Re: Article on MySQL vs. Postgres
On Wed, 5 Jul 2000, Jan Wieck wrote:

> > Odd, were you using transactions here, or transactionless?
> 
>     Mark,  you  cannot  use Postgres transactionless. Each single
>     statement run outside of a transaction  block  has  it's  own
>     transaction.

Sorry, but 'transactionless' I mean no BEGIN/END ... from what I've been
gathering from Tim, his code goes something like:

do query 1
do query 2
if query 2 fails "oops"

vs

do query 1
do query 2
if query 2 fails, abort and auto-rollback query 1

Then again, Tim might be being even more simple then that:

do query 1
exit




Wed, 5 Jul 2000 08:54:35 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Article on MySQL vs. Postgres
On Wed, 05 Jul 2000, Hannu Krosing wrote:
> Tim Perdue wrote:
> > 
> > The Hermit Hacker wrote:
> > > > Further, I have had situations where postgres actually had DUPLICATE
> > > > ids in a primary key field, probably due to some abort or other nasty
> > > > situation in the middle of a commit. How did I recover from That?
> > > > Well, I had to run a count(*) next to each ID and select out the rows
> > > > where there was more than one of each "unique" id, then reinsert those
> > > > rows and drop and rebuild the indexes and reset the sequences.
> > >
> > > Odd, were you using transactions here, or transactionless?
> > 
> > Does it matter? I suppose it was my programming error that somehow I got
> > duplicate primary keys in a table in the database where that should be
> > totally impossible under any circumstance? Another stupid
> > transactionless program I'm sure.
> > 
> > At any rate, it appears that the main problem I had with postgres (the
> > 8K tuple limit) is being fixed and I will mention that in my writeup.
> 
> Currently (as of 7.0.x) you could use BLKSIZE=32K + lztext datatype and 
> get text fields about 64-128K depending on data if you are desperately 
> after big textfields.
> 
> -----------
> Hannu

While it is slow, I've been able to store unlimited amounts of text into
the database by using the following code.  I've tested inserting over 4
megabytes from a TEXTAREA web form using PHP.  When inserting such massive
amounts of text, you will have to wait a while, but it will eventually succeed
if you don't run out of memory.  If you do run out of memory, the backend
terminates gracefully and the transaction aborts/rollsback.

-- Load the PGSQL procedural language
-- This could also be done with the createlang script/program.
-- See man createlang.
CREATE FUNCTION plpgsql_call_handler()
	RETURNS OPAQUE AS '/usr/local/pgsql/lib/plpgsql.so'
	LANGUAGE 'C';

CREATE TRUSTED PROCEDURAL LANGUAGE 'plpgsql'
	HANDLER plpgsql_call_handler
	LANCOMPILER 'PL/pgSQL';
          	
--------------------------------------------------------------------------------
--
-- Large Text storage
--


-- 	putlgtext -	generic function to store text into the
--			specified text storage table.
--		The table specified in $1 should have the following
--		fields:
--			id, text_seq, text_block
--
-- $1 is the name of the table into which $3 is stored
-- $2 is the id of the text and references id in another table
-- $3 is the text to store, which is broken into chunks.
-- returns 0 on success
-- nonzero otherwise
CREATE FUNCTION putlgtext (TEXT, INTEGER, TEXT) RETURNS INTEGER AS '
	set i_table $1
	set i_id $2
	set i_t {}
	regsub -all {([\\''\\\\])} $3 {\\\\\\1} i_t
	set i_seq 0
	while { $i_t != {} } {
		set i_offset 0	
		set tblock [string range $i_t 0 [expr 7000 + $i_offset]]
		# Do not split string at a backslash
		while { [string range $tblock end end] == "\\\\" && $i_offset < 1001 } {
			set i_offset [expr $i_offset + 1]
			set tblock [string range $i_t 0 [expr 7000 + $i_offset]]
		}
		set i_t [string range $i_t [expr 7000 + [expr $i_offset + 1]] end]
		spi_exec "INSERT INTO $i_table (id, text_seq, text_block) VALUES ( $i_id , $i_seq , ''$tblock'' )"
		incr i_seq
	}
	return 0
' LANGUAGE 'pltcl';

-- 		getlgtext - like putlgtext, this is a generic
--				function that does the opposite of putlgtext
-- $1 is the table from which to get TEXT
-- $2 is the id of the text to get
-- returns the text concatenated from one or more rows
CREATE FUNCTION getlgtext(TEXT, INTEGER) RETURNS TEXT AS '
	set o_text {}
	spi_exec -array q_row "SELECT text_block FROM $1 WHERE id = $2 ORDER BY text_seq" {
		append o_text $q_row(text_block)
	}
	return $o_text
' LANGUAGE 'pltcl';

-- largetext exists just to hold an id and a dummy 'lgtext' attribute.
-- This table's trigger function provides for inserting and updating
-- into largetext_block.  The text input to lgtext actually gets
-- broken into chunks and stored in largetext_block.
-- Deletes to this table will chain to largetext_block automatically
-- by referential integrity on the id attribute.
-- Selects have to be done using the getlgtext function.
CREATE TABLE largetext (
	id				INTEGER PRIMARY KEY,
	lgtext		TEXT -- dummy field
);
COMMENT ON TABLE largetext IS 'Holds large text';

-- This table must have the field names as they are.
-- These attribute names are expected by put/getlgtext.
CREATE TABLE largetext_block (
	id					INTEGER NOT NULL
						REFERENCES largetext
						ON DELETE CASCADE,
						
	text_seq			INTEGER NOT NULL,
	
	text_block		TEXT,
	
	PRIMARY KEY (id, text_seq)
);
COMMENT ON TABLE largetext_block IS 'Holds blocks of text for table largetext';
CREATE SEQUENCE largetext_seq;

-- SELECT:
-- SELECT id AS the_id FROM largetext;
-- SELECT getlgtext('largetext_block', id) FROM largetext WHERE id = the_id;

-- INSERT:
-- INSERT INTO largetext (lgtext) values ('.......');

-- DELETE:
-- DELETE FROM largetext WHERE id = someid;
-- deletes from largetext and by referential
-- integrity, from largetext_text all associated block rows.
CREATE FUNCTION largetext_trigfun() RETURNS OPAQUE AS '
	set i_t {}
	regsub -all {([\\''\\\\])} $NEW($2) {\\\\\\1} i_t
	switch $TG_op {
		INSERT {
			spi_exec "SELECT nextval(''largetext_seq'') AS new_id"
			set NEW($1) $new_id
			spi_exec "SELECT putlgtext(''largetext_block'', $new_id, ''$i_t'') AS rcode"
			if { $rcode != 0 } then { return SKIP }
		}
		UPDATE {
			if { $NEW($2) != {} } then {
				spi_exec "DELETE FROM largetext_text WHERE id = $OLD($1)"
				spi_exec "SELECT putlgtext(''largetext_block'', $OLD($1), ''$NEW($2)'') AS rcode"
				if { $rcode != 0 } then { return SKIP }
			}
		}
	}
	set NEW($2) "ok"
	return [array get NEW]
' LANGUAGE 'pltcl';

-- Set the function as trigger for table largetext
CREATE TRIGGER largetext_trig BEFORE INSERT OR UPDATE
ON largetext FOR EACH ROW EXECUTE
PROCEDURE largetext_trigfun(id,lgtext);



I had to use the regsub function calls to replace the \ escaping on literal
'\'s.  What a pain!  If anyone can try this code and suggest ways to improve
its speed, I'd be happy.

 -- 
			Robert

Wed, 5 Jul 2000 08:27:07 -0400
"Robert B. Easter" <reaster@comptechnews.com>
Re: Article on MySQL vs. Postgres
What ? sleepycat DB3 is now GPL ? That would be a change of philosophy.

Peter

----- Original Message -----
From: "Egon Schmid" <eschmid@php.net>
To: "Tim Perdue" <tperdue@valinux.com>
Cc: <pgsql-hackers@hub.org>
Sent: Tuesday, July 04, 2000 7:51 PM
Subject: Re: [HACKERS] Article on MySQL vs. Postgres


Tim Perdue wrote:
>
> On wednesday or thursday, I'm going to be publishing my article on MySQL
> vs. Postgres on PHPBuilder.com.

Cool!

> Features:
> Postgres is undoubtedly far, far more advanced than MySQL is. Postgres
> now supports foreign keys, which can help with referential integrity.
> Postgres supports subselects and better support for creating tables as
> the result of queries. The "transaction" support that MySQL lacks is
> included in Postgres, although you'll never miss it on a website, unless
> you're building something for a bank, and if you're doing that, you'll
> use oracle.

Since MySQL version 3.23.16 it supports transactions with sleepycats DB3
and since version 3.23.19 it is under the GPL.

-Egon

--
SIX Offene Systeme GmbH ï¿½ Stuttgart  -  Berlin  -  New York
Sielminger Straï¿½e 63   ï¿½    D-70771 Leinfelden-Echterdingen
Fon +49 711 9909164 ï¿½ Fax +49 711 9909199 http://www.six.de
PHP-Stand auf Europas grï¿½sster Linux-Messe: 'LinuxTag 2001'
weitere Infos @ http://www.dynamic-webpages.de/



Wed, 5 Jul 2000 13:00:53 -0000
"Peter Galbavy" <peter.galbavy@knowledge.com>
Re: Article on MySQL vs. Postgres
Peter Galbavy wrote:
> 
> What ? sleepycat DB3 is now GPL ? That would be a change of philosophy.
> 
> Peter

Not to my understanding. If you sell a commercial solution
involving MySQL, you have to pay Sleepycat a licensing fee. For
non-commercial use, its free. Oh, what a tangled web we weave
when we bail from BSD.

Mike Mascari

Wed, 05 Jul 2000 09:06:16 -0400
Mike Mascari <mascarm@mascari.com>
Re: Article on MySQL vs. Postgres
On Wed, 05 Jul 2000, Hannu Krosing wrote:
> "Robert B. Easter" wrote:
> > -- Load the PGSQL procedural language
> > -- This could also be done with the createlang script/program.
> > -- See man createlang.
> > CREATE FUNCTION plpgsql_call_handler()
> >         RETURNS OPAQUE AS '/usr/local/pgsql/lib/plpgsql.so'
> >         LANGUAGE 'C';
> > 
> > CREATE TRUSTED PROCEDURAL LANGUAGE 'plpgsql'
> >         HANDLER plpgsql_call_handler
> >         LANCOMPILER 'PL/pgSQL';
> 
> You probably meant pl/tcl as all your code is using that ?

Yes, I mean't to say this:

-- Load the TCL procedural language
-- This could also be done with the createlang script/program.
-- See man createlang.
CREATE FUNCTION pltcl_call_handler()
	RETURNS OPAQUE AS '/usr/local/pgsql/lib/pltcl.so'
	LANGUAGE 'C';
	
CREATE TRUSTED PROCEDURAL LANGUAGE 'pltcl'
	HANDLER pltcl_call_handler
	LANCOMPILER 'PL/tcl';


-- 
			Robert

Wed, 5 Jul 2000 09:14:48 -0400
"Robert B. Easter" <reaster@comptechnews.com>
Re: Article on MySQL vs. Postgres
on 7/4/00 8:30 PM, Tim Perdue at tperdue@valinux.com wrote:

> This is for what most people do with PHP and databases - run
> semi-critical medium-traffic sites. Anyone running a mission-critical
> site would have to look elsewhere for true robustness. I would not at
> this time recommend any serious, life-threatening app run On either
> database.

To the person who owns the web site, data is always critical. Does
www.yahoo.com store "life-threatening" information? Not really, but if you
lose your yahoo.com email, the "oh sorry, our database doesn't support
transactions" excuse doesn't cut it.

> I took a real-world page from our site
> <http://sourceforge.net/forum/forum.php?forum_id=1> and made it portable
> to both databases. Of course, I could not import the "body" of the
> message into postgres because of the 8k limitation, so the body had to
> be dropped from both databases.
> 
> The "nested" view of this page requires joins against three tables and
> some recursion to show submessages.

Some recursion? That is interesting. Do you mean multiple queries to the
database? I don't see any reason to have multiple queries to the database to
show nested messages in a forum. Using stored procedures to create sort keys
at insertion or selection time is the efficient way to do this. Ah, but
MySQL doesn't have stored procedures.

>> Did you use connection pooling (a lot
> 
> I used persistent connections, yes. Without them, Postgres' showing was
> far poorer, with mysql showing about 2x the performance.

Well, there must be some issue with your setup, because 10 requests per
second on Postgres on reads only is far from what I've seen on much wimpier
boxes than yours. Maybe I should look some more into how pconnect really
handles connection pooling, I have heard bad things that need to be
verified.

> I'd really love to see a case where a real-world page view requires 4x
> the queries on MySQL. If you are doing subselects like that on a website
> in real-time you've got serious design problems and postgres would
> fold-up and quit under the load anyway.

I believe the "design problems" come up if you need subselects and you're
using MySQL. I've used Illustra/Informix, Oracle, and now Postgres to build
database-backed web sites, and subselects are a vital part of any
somewhat-complex web app. How exactly do subselects constitute a design
problem in your opinion?

> Specifically, what is the problem with my "intuition"? All I did in the
> prior message was report my results and ask for feedback before I post
> it.

Your intuition is that Postgres will be slower because it is slower than
MySQL at reads. I contend that:
    - Postgres 7.0 is much faster at reads than the numbers you've shown.
I've seen it be much faster on smaller boxes.
    - The slowdown you're seeing is probably due in no small part to the
implementation of pconnect(), the number of times it actually connects vs.
the number of times it goes to the pool, how large that pool gets, etc...
    - The write-inefficiencies of MySQL will, on any serious web site, cut
performance so significantly that it is simply not workable. I'm thinking of
the delayed updates on Slashdot, the 20-25 second page loads on SourceForge
for permission updating and such...

> Personally, I check every query in my PHP code. On the rare occasion
> that it fales, I show an error and get out. Even with postgres, I have
> always checked success or failure of a query and shown an appropriate
> error. Never in two years of programming PHP/postgres have I ever used
> commit/rollback, and I have written some extremely complex web apps
> (sourceforge being a prime example). Geocrawler.com runs on postgres and
> again, I NEVER saw any need for any kind of rollback at all.

Geez. So you never have two inserts or updates you need to perform at once?
*ever*? What happens if your second one fails? Do you manually attempt to
backtrack on the changes you've made?

> The statelessness of the web pretty much obviates the needs for
> locks/rollbacks as each process is extremely quick and runs from start
> to finish instantly. It's not like the old days where you pull data down
> into a local application, work on it, then upload it again.
> 
> Only now, with some extremely complex stuff that we're doing on
> SourceForge would I like to see locks and rollbacks (hence my recent
> interest in benchmarking and comparing the two). Your average web
> programmer will almost never run into that in the short term.

This is simply false. If you're not using commit/rollbacks, you're either
cutting back on the functionality of your site, creating potential error
situations by the dozen, or you've got some serious design issues in your
system. Commit/Rollback is not an "advanced" part of building web sites. It
is a basic building block.

Telling your "average web programmer" to ignore transactions is like telling
your programmers not to free memory in your C programs because, hey, who
cares, you've got enough RAM for small programs, and they can learn to clean
up memory when they build "real" systems!

Of all things, this is precisely the type of thinking that crushes the
credibility of the open-source community. Enterprise IT managers understand
in great detail the need for transactions. Web sites actually need *more*
reliable technology, because you don't have that stateful session: you
sometimes need to recreate rollback mechanisms across pages by having
cleanup processes. Building this on a substrate that doesn't support the
basic transaction construct is impossible and irresponsible.

> Your own strong biases are shown in your message. I do this stuff
> because I'm curious and want to find out for myself. Most readers will
> find it interesting as I did. Few will switch from MySQL to postgres or
> vice versa because of it.

My bias? Well, my company doesn't have a vested interest in promoting
Postgres or MySQL. Before I started using Postgres, I looked into MySQL.
You're right if you think my evaluation didn't take too long. If I have
preferences, they're based purely on engineering decisions. That's not the
same as "my company just publicly endorsed MySQL, and check it out, we think
MySQL is better than Postgres."

Note that I am *not* saying that you're doing this on purpose, I'm just
saying that you're going to have a really hard time proving your
objectivity.

> Another clarification: PHPBuilder is owned by internet.com, a competitor
> of VA Linux/Andover.

PHP folks have a bias, too: PHP was built with MySQL in mind, it even ships
with MySQL drivers (and not Postgres). PHP's mediocre connection pooling
limits Postgres performance.

I'm happy to continue this discussion, but here's what I've noticed from
having had this argument many many times: if you don't believe that
transactions are useful or necessary, that subselects and enforced foreign
key constraints are hugely important, then this discussion will lead
nowhere. We simply begin with different assumptions.

I only suggest that you begin your evaluation article by explaining:
    - your assumptions
    - the fact that the page you used for benchmarking was originally built
for MySQL, and thus makes no use of more advanced Postgres features.

-Ben


Wed, 05 Jul 2000 10:48:01 -0400
Benjamin Adida <ben@mit.edu>
Re: Article on MySQL vs. Postgres
Tim Perdue wrote:
> 
> Benjamin Adida wrote:
> 
> ...useless rant about all MySQL users being stupid inept programmers
> deleted....
> 
> > PHP folks have a bias, too: PHP was built with MySQL in mind, it even ships
> > with MySQL drivers (and not Postgres). PHP's mediocre connection pooling
> > limits Postgres performance.
> 
> Well the point of this article is obviously in relation to PHP. Yes,
> Rasmus Lerdorf himself uses MySQL and I'm sure Jan would say he's a
> "wannabee", not a "real developer".

Rather he is probably a _web_ developer and not a _database_ developer, as 
most developers with DB background abhor lack of transactions, as you have 
surely noticed by now, and would not use MySQL fro R/W access ;)

> Yes I'm sure that PHP was designed to make Postgres look bad. All
> benchmarks are designed to make postgres look bad. All web designers
> build everything in just that special way that makes postgres look bad,
> and they all do it because they're inept and stupid,

Or just irresponsible. 

That's how most websites grow -
 at first no writes - MySQL is great (a filesystem with SQL interface
performance-wize)
 then some writes in real time, when popularity grows bad things start to
happen.
 then delayed writes a la Slashdot to keep the performance and integrity of
database.

> unlike the small crowd of postgres users.

That could be part of the problem ;)

SQL is a complex beast and a programmer experienced in procedural languages 
takes some time to learn to use it effectively. Until then he just tries to 
use his C/Pascal/java/whatever knowledge and simple selects - and this is
where MySQL excels.

----------------
Hannu

Wed, 05 Jul 2000 18:13:02 +0300
Hannu Krosing <hannu@tm.ee>
Re: Article on MySQL vs. Postgres
Benjamin Adida wrote:

...useless rant about all MySQL users being stupid inept programmers
deleted....


> PHP folks have a bias, too: PHP was built with MySQL in mind, it even ships
> with MySQL drivers (and not Postgres). PHP's mediocre connection pooling
> limits Postgres performance.

Well the point of this article is obviously in relation to PHP. Yes,
Rasmus Lerdorf himself uses MySQL and I'm sure Jan would say he's a
"wannabee", not a "real developer". 

Yes I'm sure that PHP was designed to make Postgres look bad. All
benchmarks are designed to make postgres look bad. All web designers
build everything in just that special way that makes postgres look bad,
and they all do it because they're inept and stupid, unlike the small
crowd of postgres users.

Tim

-- 
Founder - PHPBuilder.com / Geocrawler.com
Lead Developer - SourceForge
VA Linux Systems
408-542-5723

Wed, 05 Jul 2000 08:37:58 -0700
Tim Perdue <tperdue@valinux.com>
Re: Article on MySQL vs. Postgres
> Yes I'm sure that PHP was designed to make Postgres look bad. All
> benchmarks are designed to make postgres look bad. All web designers
> build everything in just that special way that makes postgres look bad,
> and they all do it because they're inept and stupid, unlike the small
> crowd of postgres users.

I don't believe that your sarcasm is unwarranted, BUT, and this is a big but
(just like mine :), I have found that the popularity of free software is
sometimes iversly proportional to it's complexity. Complexity in turn
sometimes, but not always, implies that the software has more features and
is better thought out. There are exceptions to this, but it has proven true
for many of the packages I have worked with.

MySQL is used by Linux folks (generalising), probably because the learning
curve is not too steep. And the otherway round for other DB + OS
combinations.

The problem I think that many folk have with printed benchmarks is the
apples to oranges comparisons. To make the comparison look valid, you have
to either reduce or ignore the differences of the fruit and just look at a
limited set of values. In the case of the apples and oranges, "average
diameter" may be valid, while "green-ness" is not. The eater of the fruit
actually wanted to know "which tastes better".

Peter


Wed, 5 Jul 2000 15:55:13 -0000
"Peter Galbavy" <peter.galbavy@knowledge.com>
Re: Article on MySQL vs. Postgres
on 7/5/00 11:37 AM, Tim Perdue at tperdue@valinux.com wrote:

> ...useless rant about all MySQL users being stupid inept programmers
> deleted....

Hmmm, okay, well, I guess my invitation to continue the conversation while
admitting a difference in assumptions is declined. Yes, my response was
harsh, but harsh on MySQL. I didn't attack MySQL programmers. I attacked the
product.

Is there a way to do this without incurring the wrath of MySQL users? If you
look at the Postgres mailing list, your worries (the duplicate key thing)
were addressed immediately by Postgres programmers, because they (the
Postgres team, which *doesn't* include me) understand the need to improve
the product.

And no, benchmarks aren't built to make Postgres look bad. But PHP is built
around an inefficient connection pooling system, which doesn't appear much
under MySQL because MySQL has extremely fast connection setup, while every
other RDBMS on the market (Oracle, Sybase, Informix, Postgres) does not.
That's the cost of setting up a transaction environment, it takes a bit of
time. Thus, PHP's pconnect() crushes performance on all databases except
MySQL.

But anyhow, I've clearly hit a nerve. You asked a question, I answered
truthfully, honestly, and logically. And you're absolutely right that I come
out strongly against MySQL. Proceed with this information as you see fit...

-Ben


Wed, 05 Jul 2000 11:56:45 -0400
Benjamin Adida <ben@mit.edu>
Re: Article on MySQL vs. Postgres
> Yes I'm sure that PHP was designed to make Postgres look bad. All
> benchmarks are designed to make postgres look bad. All web designers
> build everything in just that special way that makes postgres look bad,
> and they all do it because they're inept and stupid, unlike the small
> crowd of postgres users.

Another happy customer... ;)

Tim, one of the apparent "discriminators" between typical MySQL users
and typical Postgres users is their perception of the importance of
transactions and its relevance in application design.

For myself, coming from other commercial databases and having built
large data handling systems using those, doing without transactions is
difficult to accept. And we'd like for others to see the light too.
Hopefully the light will be a bit closer soon, since, apparently,
transactions are coming to the MySQL feature set.

You mentioned a speed difference in Postgres vs MySQL. The anecdotal
reports are quite often in this direction, but we typically see
comparable or better performance with Postgres when we actually look at
the app or benchmark. Would it be possible to see the test case and to
reproduce it here?

Regards.

                     - Thomas

Wed, 05 Jul 2000 16:03:22 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Article on MySQL vs. Postgres
Tim Perdue wrote:
> 
> Yes I'm sure that PHP was designed to make Postgres look bad. All
> benchmarks are designed to make postgres look bad. All web designers
> build everything in just that special way that makes postgres look bad,
> and they all do it because they're inept and stupid, unlike the small
> crowd of postgres users.

Tim, don't be so upset.

I'm not an english fluently speaker so I hope I can make myself clearly
understood.
Noone wants you to write a good article for PostgreSQL just because they
are developing PostgreSQL.
Noone hates MySQL.
Noone tries to make PostgreSQL look better as it is. We don't sell it
:-)
It's just a couple of things that are important in database benchmarks
and the PostgreSQL developers knows them better.
That's why I consider that you have done a good thing telling us about
your article and I sincerely hope that you don't feel sorry for that.
I agree with you that they were some replies to your message rather ...
violent I can say.
Definitely, MySQL and PostgreSQL has their own application preferences
and they are making a good job each of them.
It's so difficult to compare them as it would be comparing two cars
(theyu have 4 wheels, 4 doors, an engine) and we could pick for example
the VW Beetle and a Mercedes A-class.

So, I would say to write your article about using MySQL or PostgreSQL on
PHP applications and let other know your results. Now, when MySQL is
GPL, it's a good thing to make such a comparisson. But please, don't pe
angry and upset on the PostgreSQL developers and community. They just
tried to give a hand of help revealing some important features of
PostgreSQL.

hope it helps,
Best regards,
Constantin Teodorescu
FLEX Consulting Braila, ROMANIA

Wed, 05 Jul 2000 19:04:18 +0300
Constantin Teodorescu <teo@flex.ro>
Re: Article on MySQL vs. Postgres
The Hermit Hacker wrote:

> We had a *very* old version of PostgreSQL running on a Pentium acting as
> an accounting/authentication backend to a RADIUS server for an ISP
> ... uptime for the server itself was *almost* 365 days (someone hit the
> power switch by accident, meaning to power down a different machine
> *sigh*) ... PostgreSQL server had been up for something like 6 months
> without any problems, with the previous downtime being to upgrade the
> server ...

At a previous employer, there is still a database running that has not
seen a crash downtime ever since early 1996 - the only few downtimes it
ever saw were for a rare few postgres, OS and hardware upgrades. As
there have been no cries for help on any database or reboot issue ever
since I left (I still am appointed as the DB admin in case of any
trouble), it must be getting close to two years uptime by now, and that
literally unattended. 

Sevo

-- 
sevo@ip23.net

Wed, 05 Jul 2000 18:13:51 +0200
Sevo Stille <sevo@ip23.net>
Re: Article on MySQL vs. Postgres
Hannu Krosing <hannu@tm.ee> writes:
> There a bug report that allowed tuplicate ids in an uniqe field when 
> SELECT FOR UPDATE was used. Could this be your case ?
> [snip]
> IIRC the fix was also provided, so it could be fixed in current CVS (the
> above is from 7.0.2, worked the same in 6.5.3)

It does seem to be fixed in current CVS:

regression=#  create table test(i int primary key);
NOTICE:  CREATE TABLE/PRIMARY KEY will create implicit index 'test_pkey' for table 'test'
CREATE
regression=# insert into test values(1);
INSERT 145215 1
regression=# begin;
BEGIN
regression=# select * from test for update;
 i
---
 1
(1 row)

regression=# insert into test values(1);
ERROR:  Cannot insert a duplicate key into unique index test_pkey
regression=#


			regards, tom lane

Wed, 05 Jul 2000 12:16:30 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Article on MySQL vs. Postgres 
On Wed, 5 Jul 2000, Benjamin Adida wrote:

> > Another clarification: PHPBuilder is owned by internet.com, a competitor
> > of VA Linux/Andover.
> 
> PHP folks have a bias, too: PHP was built with MySQL in mind, it even ships
> with MySQL drivers (and not Postgres). PHP's mediocre connection pooling
> limits Postgres performance.

Careful here ... PHP was not built with MySQL in mind ... hell, I used PHP
ages before it even *had* MySQL support (hell, before I even know about
Postgres95 *or* MySQL) ... also, if I recall reading on the PHP site, the
MySQL support that is included is limited, but I don't recall where I read
it.  There is a recommendation *somewhere* that if you want to use all the
features, you ahve to install the MySQL libraries first ...

Just to defend PHP, cause, well ... I like it :)


Wed, 5 Jul 2000 13:30:51 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Article on MySQL vs. Postgres
Thomas Lockhart wrote:
> You mentioned a speed difference in Postgres vs MySQL. The anecdotal
> reports are quite often in this direction, but we typically see
> comparable or better performance with Postgres when we actually look at
> the app or benchmark. Would it be possible to see the test case and to
> reproduce it here?

Finally a sensible reply from one of the core guys.

http://www.perdue.net/benchmarks.tar.gz

To switch between postgres and mysql, copy postgres.php to database.php,
change the line of SQL with the LIMIT statement in forum.php. 

To move to mysql, copy mysql.php to database.php and change the line of
SQL in forum.php

No bitching about the "bad design" of the forum using recursion to show
submessages. It can be done in memory in PHP, but I chose to hit the
database instead. This page is a good example of one that hits the
database hard. It's one of the worst on our site.

At any rate, I wish someone would write an article that explains what
the benefits of transactions are, and how to use them effectively in a
web app, skipping the religious fervor surrounding pgsql vs. myql.
There's a lot of people visiting PHPBuilder who just want to expand
their knowledge of web development, and many of them would find that
interesting.

Tim

-- 
Founder - PHPBuilder.com / Geocrawler.com
Lead Developer - SourceForge
VA Linux Systems
408-542-5723

Wed, 05 Jul 2000 09:34:05 -0700
Tim Perdue <tperdue@valinux.com>
Re: Article on MySQL vs. Postgres
On Wed, 5 Jul 2000, Tim Perdue wrote:

> Benjamin Adida wrote:
> 
> ...useless rant about all MySQL users being stupid inept programmers
> deleted....
> 
> 
> > PHP folks have a bias, too: PHP was built with MySQL in mind, it even ships
> > with MySQL drivers (and not Postgres). PHP's mediocre connection pooling
> > limits Postgres performance.
> 
> Well the point of this article is obviously in relation to PHP. Yes,
> Rasmus Lerdorf himself uses MySQL and I'm sure Jan would say he's a
> "wannabee", not a "real developer". 

I would seriously doubt that Jan wuld consider Rasmus a 'wannabee'
... Rasmus essentially built a Web optimized, HTML embedded language that
I imagine a *large* percentage of the sites on the 'Net rely on.  My
experience with the language is that it is clean and *very* easy to pick
up for simple stuff, with some nice, advanced tools for the more complex
issues ...

I use PHP with PgSQL almost exclusively now for my frontends, since its
got some *nice* features for retrieving the results of queries (ie. I love
being able to do a SELECT * and being able to retrive the results by the
field name instead of having to know the ordering) ...



Wed, 5 Jul 2000 13:35:16 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Article on MySQL vs. Postgres
On Wed, 5 Jul 2000, Tim Perdue wrote:

> At any rate, I wish someone would write an article that explains what
> the benefits of transactions are, and how to use them effectively in a
> web app, skipping the religious fervor surrounding pgsql vs. myql.
> There's a lot of people visiting PHPBuilder who just want to expand
> their knowledge of web development, and many of them would find that
> interesting.

I couldn't write to save my life, but if you want to try and co-write
something, I'm more then willing to try and provide required input ... 



Wed, 5 Jul 2000 13:40:02 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Article on MySQL vs. Postgres

Tim,

I'm sorry if I came off harsh in my previous comments. I'm a fervent
supporter of open-source software, and have hit massive pushback from
enterprise people because they see all the open-source sites using MySQL,
and that is outrageous to them. Although MySQL has a few, important niches
to fill, it's been used in places where I think it's hurt the credibility of
open-source web developers. I've been trying to talk to MySQL
developer/users about how we got to where we are, but with little success
(and what I've told you is by far the nastiest I've ever been in this
respect).

I hope that we can have a meaningful exchange about these issues. I'm a fan
of Postgres, but by no means a religious supporter of it. I *am* a religious
supporter of transactions, subselects, and such.

If you'd like to find out more about transactions, you can check out Philip
Greenspun's http://www.arsdigita.com/asj/aolserver/introduction-2.html which
has a paragraph about "Why Oracle?" which explains the reasons for choosing
an ACID-compliant RDBMS.

I'm also happy to write up a "why transactions are good" article.

-Ben

on 7/5/00 12:34 PM, Tim Perdue at tperdue@valinux.com wrote:

> Thomas Lockhart wrote:
>> You mentioned a speed difference in Postgres vs MySQL. The anecdotal
>> reports are quite often in this direction, but we typically see
>> comparable or better performance with Postgres when we actually look at
>> the app or benchmark. Would it be possible to see the test case and to
>> reproduce it here?
> 
> Finally a sensible reply from one of the core guys.
> 
> http://www.perdue.net/benchmarks.tar.gz
> 
> To switch between postgres and mysql, copy postgres.php to database.php,
> change the line of SQL with the LIMIT statement in forum.php.
> 
> To move to mysql, copy mysql.php to database.php and change the line of
> SQL in forum.php
> 
> No bitching about the "bad design" of the forum using recursion to show
> submessages. It can be done in memory in PHP, but I chose to hit the
> database instead. This page is a good example of one that hits the
> database hard. It's one of the worst on our site.
> 
> At any rate, I wish someone would write an article that explains what
> the benefits of transactions are, and how to use them effectively in a
> web app, skipping the religious fervor surrounding pgsql vs. myql.
> There's a lot of people visiting PHPBuilder who just want to expand
> their knowledge of web development, and many of them would find that
> interesting.
> 
> Tim


Wed, 05 Jul 2000 12:43:05 -0400
Benjamin Adida <ben@mit.edu>
Re: Article on MySQL vs. Postgres
I have been looking at using the lztext type and I have some
questions/observations.   Most of my experience comes from attempting to
compress text records in a different database (CTREE), but I think the
experience is transferable.

My typical table consists of variable length text records.  The average
length record is around 1K bytes.  I would like to compress my records
to save space and improve I/O performance (smaller records means more
records fit into the file system cache which means less I/O - or so the
theory goes).  I am not too concerned about CPU as we are using a 4-way
Sun Enterprise class server.  So compress seems like a good idea to me.

My experience with attempting to compress such a relatively small
(around 1K) text string is that the compression ration is not very
good.  This is because the string is not long enough for the LZ
compression algorithm to establish really good compression patterns and
the fact that the de-compression table has to be built into each
record.  What I have done in the past to get around these problems is
that I have "taught" the compression algorithm the patterns ahead of
time and stored the de-compression patterns in an external table.  Using
this technique, I have achieved *much* better compression ratios.

So my questions/comments are:

    - What are the typical compression rations on relatively small (i.e.
around 1K) strings seen with lztext?
    - Does anyone see a need/use for a generalized string compression
type that can be "trained" external to the individual records?
    - Am I crazy in even attempting to compress strings of this relative
size?  My largest table correct contains about 2 million entries of
roughly 1k size strings or about 2Gig of data.  If I could compress this
to about 33% of it's original size (not unreasonable with a trained LZ
compression), I would save a lot of disk space (not really important)
and a lot of file system cache space (very important) and be able to fit
the entire table into memory (very, very important).

Thank you,
Jeff



Wed, 05 Jul 2000 12:59:12 -0400
Jeffery Collins <collins@onyx-technologies.com>
lztext and compression ratios...
The Hermit Hacker wrote:
> On Wed, 5 Jul 2000, Tim Perdue wrote:
>
> > Benjamin Adida wrote:
> >
> > ...useless rant about all MySQL users being stupid inept programmers
> > deleted....
> >
> >
> > > PHP folks have a bias, too: PHP was built with MySQL in mind, it even ships
> > > with MySQL drivers (and not Postgres). PHP's mediocre connection pooling
> > > limits Postgres performance.
> >
> > Well the point of this article is obviously in relation to PHP. Yes,
> > Rasmus Lerdorf himself uses MySQL and I'm sure Jan would say he's a
> > "wannabee", not a "real developer".
>
> I would seriously doubt that Jan wuld consider Rasmus a 'wannabee'
> .... Rasmus essentially built a Web optimized, HTML embedded language that
> I imagine a *large* percentage of the sites ...

    NEVER!

    Once  I've built a PG based middle tear with an apache module
    that could in cooperation be a complete virtual  host  inside
    of  a DB. Including inbound Tcl scripting, DB-access, dynamic
    images and whatnot. Never finished that work until AOL-Server
    3.0   appeared,  at  which  point  I  considered  my  product
    "trashwork".

    Some of the sources I looked at (and learned alot  from)  was
    the PHP module. So I know what kind of programmer built that.

    Maybe someone of the PG  community  should  spend  some  time
    building  a  better  PHP  coupling  and  contribute  to  that
    project. And there are more such projects  out  that  need  a
    helping hand from our side.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Wed, 5 Jul 2000 20:01:54 +0200 (MEST)
JanWieck@t-online.de (Jan Wieck)
Re: Article on MySQL vs. Postgres
Jeffery Collins <collins@onyx-technologies.com> writes:
> My experience with attempting to compress such a relatively small
> (around 1K) text string is that the compression ration is not very
> good.  This is because the string is not long enough for the LZ
> compression algorithm to establish really good compression patterns and
> the fact that the de-compression table has to be built into each
> record.  What I have done in the past to get around these problems is
> that I have "taught" the compression algorithm the patterns ahead of
> time and stored the de-compression patterns in an external table.  Using
> this technique, I have achieved *much* better compression ratios.

(Puts on compression-guru hat...)

There is much in what you say.  Perhaps we should consider keeping the
lztext type around (currently it's slated for doom in 7.1, since the
TOAST feature will make plain text do everything lztext does and more)
and having it be different from text in that a training sample is
supplied when the column is defined.  Not quite sure how that should
look or where to store the sample, but it could be a big win for tables
having a large number of moderate-sized text entries.

			regards, tom lane

Wed, 05 Jul 2000 14:12:37 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: lztext and compression ratios... 
I was sitting here reading about TOAST and 7.1 and another question came
into my mind.. I looked on the web page for some information about features
to be added in 7.1 and an approximate 7.1 release date.. I found information
on the TOAST feature as well as Referential Integrity  but nothing else.. If
I missed it please just point me in the right direction, if not I'd like to
as about the approximate time of 7.1 arriving in a stable form and if full
text indexing is going to be a part of 7.1..

I ask because if 7.1 is going to include full text indexing natively and is
going to arrive pretty soon, I might not continue on this project as I have
been (the new full text index trigger)..

Thanks!!!

-Mitch


Wed, 5 Jul 2000 14:39:17 -0400
"Mitch Vincent" <mitch@venux.net>
PostgreSQL 7.1 

figure sometime in October before v7.1 is released ... i believe the last
talk talked about a beta starting sometime in September, but nothing is
ever "set in stone" until we actually do it :)


On Wed, 5 Jul 2000, Mitch Vincent wrote:

> I was sitting here reading about TOAST and 7.1 and another question came
> into my mind.. I looked on the web page for some information about features
> to be added in 7.1 and an approximate 7.1 release date.. I found information
> on the TOAST feature as well as Referential Integrity  but nothing else.. If
> I missed it please just point me in the right direction, if not I'd like to
> as about the approximate time of 7.1 arriving in a stable form and if full
> text indexing is going to be a part of 7.1..
> 
> I ask because if 7.1 is going to include full text indexing natively and is
> going to arrive pretty soon, I might not continue on this project as I have
> been (the new full text index trigger)..
> 
> Thanks!!!
> 
> -Mitch
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Wed, 5 Jul 2000 15:49:59 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: PostgreSQL 7.1 
"Mitch Vincent" <mitch@venux.net> writes:
> I was sitting here reading about TOAST and 7.1 and another question came
> into my mind.. I looked on the web page for some information about features
> to be added in 7.1 and an approximate 7.1 release date.. I found information
> on the TOAST feature as well as Referential Integrity  but nothing else.. If
> I missed it please just point me in the right direction, if not I'd like to
> as about the approximate time of 7.1 arriving in a stable form and if full
> text indexing is going to be a part of 7.1..

AFAIK there are no plans on the table to do more with full text
indexing; at least none of the core developers are working on it.
(If someone else is and I've forgotten, my apologies.)

Currently the plans for 7.1 look like this:

	* WAL (assuming Vadim gets it done soon)
	* TOAST (pretty far along already)
	* new fmgr (fmgr itself done, still need to turn crank on
		converting built-in functions)
	* better memory management, no more intraquery memory leaks
	  (about half done)
	* some other stuff, but I think those are all the "big features"
	  that anyone has committed to get done
	* whatever else gets done meanwhile

We were shooting for going beta around Aug 1 with release around Sep 1,
but don't hold us to that ;-).

Notably missing from this list is OUTER JOIN and other things that
depend on querytree redesign (such as fixing all the restrictions on
views).  We've agreed to put that off till 7.2 in hopes of keeping the
7.1 development cycle reasonably short.  Not sure what else will be
in 7.2, but the querytree work will be.

> I ask because if 7.1 is going to include full text indexing natively and is
> going to arrive pretty soon, I might not continue on this project as I have
> been (the new full text index trigger)..

Seems like you should keep at it.

			regards, tom lane

Wed, 05 Jul 2000 16:02:43 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: PostgreSQL 7.1 
Mitch Vincent wrote:
> I was sitting here reading about TOAST and 7.1 and another question came
> into my mind.. I looked on the web page for some information about features
> to be added in 7.1 and an approximate 7.1 release date.. I found information
> on the TOAST feature as well as Referential Integrity  but nothing else.. If
> I missed it please just point me in the right direction, if not I'd like to
> as about the approximate time of 7.1 arriving in a stable form and if full
> text indexing is going to be a part of 7.1..

    The  project page you've seen was an idea I had several month
    ago.  I had a big success with it since I  got  co-developers
    for  FOREIGN  KEY.   It  was the first time I something in an
    open source project with related developers, and  the  result
    was stunning - even for me.

    Unfortunately,  none  of  the other "inner-circle" developers
    catched up that idea. Maybe all they're doing is not  of  the
    nature    that    they   would   gain   anything   from   co-
    developers/volunteers.

    Not an answer to your real question, more like a kick in  the
    A.. of my colleagues.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Wed, 5 Jul 2000 22:41:18 +0200 (MEST)
JanWieck@t-online.de (Jan Wieck)
Re: PostgreSQL 7.1
Jeffery Collins wrote:
> I have been looking at using the lztext type and I have some
> questions/observations.   Most of my experience comes from attempting to
> compress text records in a different database (CTREE), but I think the
> experience is transferable.

    First   of   all  I  welcome  any  suggestions/input  to  the
    compression code I've implemented. Seems  you're  experienced
    in  this  area so keep going and hammer down any of my points
    below.

    You should know that the "lztext" type  will  disappear  soon
    and  be  replaced  with  the  general  "all varlena types are
    potentially compressable" approach of TOAST.

> My typical table consists of variable length text records.  The average
> length record is around 1K bytes.  I would like to compress my records
> to save space and improve I/O performance (smaller records means more
> records fit into the file system cache which means less I/O - or so the
> theory goes).  I am not too concerned about CPU as we are using a 4-way
> Sun Enterprise class server.  So compress seems like a good idea to me.
>
> My experience with attempting to compress such a relatively small
> (around 1K) text string is that the compression ration is not very
> good.  This is because the string is not long enough for the LZ
> compression algorithm to establish really good compression patterns and
> the fact that the de-compression table has to be built into each
> record.  What I have done in the past to get around these problems is
> that I have "taught" the compression algorithm the patterns ahead of
> time and stored the de-compression patterns in an external table.  Using
> this technique, I have achieved *much* better compression ratios.

    The compression algorithm used in "lztext" (and so  in  TOAST
    in  the  future)  doesn't have a de-compression table at all.
    It's  based  on  Adisak  Pochanayon's  SLZ  algorithm,  using
    <literal_char>  or  <token>.  A <token> just tells how far to
    go back in the OUTPUT-buffer and how many bytes to copy  from
    OUTPUT to OUTPUT. Look at the code for details.

    My  design  rules for the compression inside of Postgres have
    been

        -   beeing fast on decompression
        -   beeing good for relatively small values
        -   beeing fast on compression

    The first rule is met by  the  implementation  itself.  Don't
    underestimate  this  design rule! Usually you don't update as
    often as you query. And the implementation of TOAST  requires
    a super fast decompression.

    The   second   rule   is  met  by  not  needing  any  initial
    decompression table inside of the stored value.

    The third rule is controlled by the default strategy  of  the
    algorithm,        (unfortunately)        hardcoded       into
    utils/adt/pg_lzcompress.c.  It'll never try to compress items
    smaller  than 256 bytes. It'll fallback to plain storage (for
    speed advantage while decompressing a value) if less than 20%
    of  compression  is  gained.  It'll  stop  match loookup if a
    backward match of 128 or more bytes is found.

> So my questions/comments are:
>
>     - What are the typical compression rations on relatively small (i.e.
> around 1K) strings seen with lztext?

    Don't have that small items handy. But a table  containing  a
    file path and it's content. All files where HTML files.

        From - To      | count(*) | avg(length) | avg(octet_length)
        ---------------+----------+-------------+------------------
        1024 - 2047    |       14 |        1905 |              1470
        2048 - 4095    |       67 |        3059 |              1412
        4096 - 8191    |       45 |        5384 |              2412
        8192 -         |       25 |       17200 |              6323
        ---------------+----------+-------------+------------------
        all            |      151 |        5986 |              2529

>     - Does anyone see a need/use for a generalized string compression
> type that can be "trained" external to the individual records?

    Yes, of course. Maybe "lztext" can be a framework for you and
    we just tell the toaster "never apply your lousy  compression
    on that" (it's prepared for).

>     - Am I crazy in even attempting to compress strings of this relative
> size?  My largest table correct contains about 2 million entries of
> roughly 1k size strings or about 2Gig of data.  If I could compress this
> to about 33% of it's original size (not unreasonable with a trained LZ
> compression), I would save a lot of disk space (not really important)
> and a lot of file system cache space (very important) and be able to fit
> the entire table into memory (very, very important).

    Noone is crazy attempting to improve something. It might turn
    out not to work well, or beeing brain damaged from the start.
    But someone who never tries will miss all his chances.

    Final note:

    One  thing  to  keep  in mind is that the LZ algorithm you're
    thinking of must be distributable under the terms of the  BSD
    license.  If it's copyrighted or patented by any third party,
    not agreeing to these terms, it's out of discussion and  will
    never  appear in the Postgres source tree.  Especially the LZ
    algorithm used in GIF is one of these show-stoppers.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Thu, 6 Jul 2000 00:16:43 +0200 (MEST)
JanWieck@t-online.de (Jan Wieck)
Re: lztext and compression ratios...
JanWieck@t-online.de (Jan Wieck) writes:
>     One  thing  to  keep  in mind is that the LZ algorithm you're
>     thinking of must be distributable under the terms of the  BSD
>     license.  If it's copyrighted or patented by any third party,
>     not agreeing to these terms, it's out of discussion and  will
>     never  appear in the Postgres source tree.  Especially the LZ
>     algorithm used in GIF is one of these show-stoppers.

As long as you brought it up: how sure are you that the method you've
used is not subject to any patents?  The mere fact that you learned
it from someone who didn't patent it does not guarantee anything ---
someone else could have invented it independently and filed for a
patent.

If you can show that this method uses no ideas not found in zlib,
then I'll feel reassured --- a good deal of legal research went into
zlib to make sure it didn't fall foul of any patents, and zlib has
now been around long enough that it'd be tough for anyone to get a
new patent on one of its techniques.  But if SLZ has any additional
ideas in it, then we could be in trouble.  There are an awful lot of
compression patents.

			regards, tom lane

Wed, 05 Jul 2000 18:40:31 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: lztext and compression ratios... 
Benjamin Adida wrote:

> Some recursion? That is interesting. Do you mean multiple queries to the
> database? I don't see any reason to have multiple queries to the database to
> show nested messages in a forum. Using stored procedures to create sort keys
> at insertion or selection time is the efficient way to do this. Ah, but
> MySQL doesn't have stored procedures.

Can you be more specific on how you would support arbitrary nesting and
correct sorting of a threaded discussion in postgres? I've thought about
this problem but didn't come up with anything except to re-implement the
old recursive " retrieve* " from the old postgres.

Thu, 06 Jul 2000 09:47:31 +1000
Chris Bitmead <chrisb@nimrod.itg.telstra.com.au>
Re: Article on MySQL vs. Postgres
Jan,

Thank you for your comments on lztext.  They were very informative.  I hope
you (and the rest of the list) don't mind my ramblings on compression, but I
think there is some performance/space advantage if compression can integrated
into a table.  I admit to not being a compression expert.  All that I know
has been learn from reading some zlib source.

The compression that I have been working with (and am more familiar with) is
Huffman compression.  You are probably familiar with it and may have even
evaluated it when you were looking at compression techniques.  In my limited
understanding of LZ compression, there are some advantages and disadvantages
to Huffman compression.

The disadvantages seem to be:
    - In normal Huffman compression, the string to be compressed is examined
twice.  Once to build the translation table and once to do the translation.
    - The translation table must be written with the string.  This makes the
result larger and limits it's effectiveness with small strings.
    - I don't know for sure, but I wouldn't be surprised if uncompression is
slower with Huffman than LZ compression.

To get around these disadvantages, I modified my Huffman support in the
following way:
    - I trained the translation table and stored the table separately.  I was
able to do this because I knew before hand the type of text that would be in
my columns.
    - I "improved" the compression by giving the translation table the
ability to look for well known substrings in addition to byte values.  For
example the string "THE" in an English text might have it's own Huffman
translation value instead of relying on the individual T, H and E
translations.  Again I was able to do this because I knew the type of text I
would be storing in the column.

Because the translation table is external to the string, it is no longer
included in the string.  This improves the compression ratio and helps the
uncompression as the table does not need to be read and interpreted.  This
approach also allows for and takes advantage of cross-row commonalities.
After doing all of this, I was able to get the average compression ratio to
about 3.5 (i.e. orig size / new size) on text columns of 1K or less.

Of course the disadvantage to this is that if the dataset changes overtime,
the compression translations to not change with it and the compression ratio
will get worse.  In the very worst case, where the dataset totally changes
(say the language changed from English to Russian) the "compressed" string
could actually get larger than the original string because everything the
compression algorithm thought it knew was totally wrong.

As I said, I did this with a different database (CTREE).  My company is now
converting from CTREE to postgresql and I would like to find some fairly
elegant way to include this type of compression support.  My CTREE
implementation was rather a hack, but now that we are moving to a better
database, I would like to have a better solution for compression.

Originally I latched onto the idea of using lztext or something like lztext,
but I agree it is better if the column type is something standard and the
compression is hidden by the backend.  I guess my vision would be to be able
to define either a column type or a column attribute that would allow me to
indicate the type of compression to have and possibly the compression
algorithm and translation set to use.  Actually the real vision is to have it
all hidden and have the database learn about the column as rows are added and
modify the compression transalation to adapt to the changing column data.

Jeff




Wed, 05 Jul 2000 20:07:54 -0400
Jeffery Collins <collins@onyx-technologies.com>
Re: lztext and compression ratios...
Tom Lane wrote:
> JanWieck@t-online.de (Jan Wieck) writes:
> >     One  thing  to  keep  in mind is that the LZ algorithm you're
> >     thinking of must be distributable under the terms of the  BSD
> >     license.  If it's copyrighted or patented by any third party,
> >     not agreeing to these terms, it's out of discussion and  will
> >     never  appear in the Postgres source tree.  Especially the LZ
> >     algorithm used in GIF is one of these show-stoppers.
>
> As long as you brought it up: how sure are you that the method you've
> used is not subject to any patents?  The mere fact that you learned
> it from someone who didn't patent it does not guarantee anything ---
> someone else could have invented it independently and filed for a
> patent.

    Now that you ask for it: I'm not sure. Could be.

> If you can show that this method uses no ideas not found in zlib,
> then I'll feel reassured --- a good deal of legal research went into
> zlib to make sure it didn't fall foul of any patents, and zlib has
> now been around long enough that it'd be tough for anyone to get a
> new patent on one of its techniques.  But if SLZ has any additional
> ideas in it, then we could be in trouble.  There are an awful lot of
> compression patents.

    To  do  so  I  don't know enough about the algorithms used in
    zlib.  Is there someone out here who could verify that  if  I
    detailed enough describe what our compression code does?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Thu, 6 Jul 2000 10:47:24 +0200 (MEST)
JanWieck@t-online.de (Jan Wieck)
Re: lztext and compression ratios...
JanWieck@t-online.de (Jan Wieck) writes:
>> As long as you brought it up: how sure are you that the method you've
>> used is not subject to any patents?

>     Now that you ask for it: I'm not sure. Could be.

>> If you can show that this method uses no ideas not found in zlib,
>> then I'll feel reassured

>     To  do  so  I  don't know enough about the algorithms used in
>     zlib.  Is there someone out here who could verify that  if  I
>     detailed enough describe what our compression code does?

After a quick look at the code, I don't think there is anything
problematic about the data representation or the decompression
algorithm.  The compression algorithm is another story, and it's
not real well commented :-(.  The important issues are how you
search for matches in the past text and how you decide which match
is the best one to use.  Please update the code comments to describe
that, and I'll take another look.

			regards, tom lane

Thu, 06 Jul 2000 10:22:55 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: lztext and compression ratios... 
Tom Lane wrote:
> JanWieck@t-online.de (Jan Wieck) writes:
> >> As long as you brought it up: how sure are you that the method you've
> >> used is not subject to any patents?
>
> >     Now that you ask for it: I'm not sure. Could be.
>
> >> If you can show that this method uses no ideas not found in zlib,
> >> then I'll feel reassured
>
> >     To  do  so  I  don't know enough about the algorithms used in
> >     zlib.  Is there someone out here who could verify that  if  I
> >     detailed enough describe what our compression code does?
>
> After a quick look at the code, I don't think there is anything
> problematic about the data representation or the decompression
> algorithm.  The compression algorithm is another story, and it's
> not real well commented :-(.  The important issues are how you
> search for matches in the past text and how you decide which match
> is the best one to use.  Please update the code comments to describe
> that, and I'll take another look.

    Done. You'll find a new section in the top comments.

    While  writing  it  I  noticed  that  the algorithm is really
    expensive for big items. The history lookup  table  allocated
    is  8  times (on 32 bit architectures) the size of the input.
    So if you want to have 1MB compressed, it'll allocate 8MB for
    the  history.  It  hit  me  when  I  was hunting a bug in the
    toaster earlier today. Doing an update to a toasted  item  of
    5MB, resulting in a new value of 10MB, the backend blew up to
    290MB of virtual memory - oh boy. I definitely need  to  make
    that smarter.

    When  I wrote it I never thought about items that big. It was
    before we had the idea of TOAST.

    This all might  open  another  discussion  I'll  start  in  a
    separate thread.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Thu, 6 Jul 2000 23:09:27 +0200 (MEST)
JanWieck@t-online.de (Jan Wieck)
Re: [SQL] Re: lztext and compression ratios...
> 
> figure sometime in October before v7.1 is released ... i believe the last
> talk talked about a beta starting sometime in September, but nothing is
> ever "set in stone" until we actually do it :)

I agree.  August is a very useful month because a lot of people are slow
at work during that month, and that gives them time to work on
PostgreSQL.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 6 Jul 2000 17:35:13 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: PostgreSQL 7.1
JanWieck@t-online.de (Jan Wieck) writes:
> Tom Lane wrote:
>> After a quick look at the code, I don't think there is anything
>> problematic about the data representation or the decompression
>> algorithm.  The compression algorithm is another story, and it's
>> not real well commented :-(.  The important issues are how you
>> search for matches in the past text and how you decide which match
>> is the best one to use.  Please update the code comments to describe
>> that, and I'll take another look.

>     Done. You'll find a new section in the top comments.

I think we are probably OK.  Anyone who wants to check the issue might
want to start with http://www.faqs.org/faqs/compression-faq/,
particularly part 1 item 8.  The two patents I'm most concerned about
are Waterworth's (4701745) and Stac's (5016009) which both cover basic
variants of LZ77 (for full text of these or any other US patent consult
http://patent.womplex.ibm.com/).  But it looks to me like this
particular variant can be argued not to fall under either patent.
For example, Waterworth's patent specifies using 3-byte hash values
while you use 4-byte, and stupid as that is it's sufficient to get
you out from under.  (A more technically valid reason is that he
doesn't use hash collision chains.)  The Stac patent also specifies
a data structure considerably different from your hash chains.

To give you an idea of what's involved here, I attach an old message
from Jean-loup Gailly, author of gzip and zlib, explaining why he
believes gzip doesn't fall foul of these two patents.  Not all of
his reasons apply to your method, but I think enough do.  (BTW,
Gailly's a good guy --- I worked with him on the PNG project.
I think I'll write to him and see if he's got time to review this
code for patent problems.)

>     While  writing  it  I  noticed  that  the algorithm is really
>     expensive for big items. The history lookup  table  allocated
>     is  8  times (on 32 bit architectures) the size of the input.
>     So if you want to have 1MB compressed, it'll allocate 8MB for
>     the  history.  It  hit  me  when  I  was hunting a bug in the
>     toaster earlier today. Doing an update to a toasted  item  of
>     5MB, resulting in a new value of 10MB, the backend blew up to
>     290MB of virtual memory - oh boy. I definitely need  to  make
>     that smarter.

Yes, you need some smarter method for choosing the size of the
hash table... a braindead but possibly sufficient answer is to
have some hard limit on the size... or you could just use a
hard-wired constant size to begin with.  I think it's usually
considered wise to use a prime for the table size and reduce
the raw input bits modulo the prime.

			regards, tom lane


Article: 14604 of gnu.misc.discuss
Path: chorus!octave.chorus.fr!jloup
From: jloup@chorus.fr (Jean-loup Gailly)
Newsgroups: gnu.misc.discuss
Subject: Re: LPF, NY Times, GNU and compression algorithms
Message-ID: <7989@chorus.chorus.fr>
Date: 10 Mar 94 10:10:03 GMT
References: <RXN106.94Mar6103732@wilbur.cac.psu.edu> <HSU.94Mar7204324@laphroaig.cs.hut.fi> <MARC.94Mar8090631@marc.watson.ibm.com>
Sender: news@chorus.chorus.fr
Distribution: gnu
Lines: 201

Marc Auslander <marc@watson.ibm.com> writes:

>   Two Stac patents in the case are 4701745 and 5016009.
>
>   From 4601745: [typo, meant: 4701745]
>
>   the data processing means including circuit means operable to check
>   whether a sequence of successive bytes to be processed identical with
>   a sequence of bytes already processed, and including a hash generating
>   means responsive to the application of a predetermined number of bytes ...
>
>   From 5016009
>
>   in order to perform the hashing function, a data compression system
>   includes certain hash data structures including a history array
>   pointer ...
>
>   also:
>
>   ... is found ..., encoding said matching string ... a variable length
>   indicator ..., said predetermined strategy ensuring that a matching
>   string of two characters of said input data stream is compressed to
>   less than said two characters of said input data stream.
>
>   So - on the face of it, why isn't gzip covered by these patents?

Let's take each patent in turn. A clarification first: the Stac patent
4,701,745 was not invented by Stac. It was initially owned by Ferranti
(UK) and only recently bought by Stac. This was a very profitable
acquisition (assuming it cost Stac far less than the $120 million they
won by using this patent against Microsoft).

(a) 4,701,745

This algorithm is now known as LZRW1, because Ross Williams reinvented
it independently later and posted it on comp.compression on April 22,
1991. Exactly the same algorithm has also been patented by Gibson and
Graybill (5,049,881). The patent office failed to recognize that
it was the same algorithm.

This algorithm uses LZ77 with hashing but no collision chains and
outputs unmatched bytes directly without further compression.  gzip
uses collisions chains of arbitrary length, and uses Huffman encoding
on the unmatched bytes:

- Claim 1 of the patent is restricted to (emphasis added by me):

    output means operable to APPLY to a transfer medium each byte of data
      not forming part of such an identical sequence; and

    ENCODING means responsive to the identification of such a sequence
      to APPLY to the transfer medium an identification signal which
      identifies both the location in the input store of the previous
      occurrence of the sequence of bytes and the number of bytes
      contained in the sequence.

  The claim thus makes a clear distinction between "encoding" and
  "applying to the transfer medium". A system which compresses the
  unmatched bytes does not infringe this patent.

- The description of the patent and claim 2 make clear that the check
  for identity of the sequences of bytes is to be made only once (no
  hash collision chains). Gzip performs an arbitrary number of such
  checks. The "means" enumerated in claim 1 specify what the hash
  table consists of, and this does not include any means for storing
  hash collision chains.

- Claim 2 also requires that *all* bytes participating in the hash
  function should be compared:

     A system as claimed in claim 1 in which the circuit means also
      includes check means operable to check for identity between EACH
      of the said predetermined number of bytes in sequence and EACH of
      a similar sequence of bytes contained in the input store at a
      location defined by a pointer read out from the temporary store at
      said address

  [in plain English, this is the check for hash collision]

      and to check whether identity exists between succeeding bytes in
      each sequence of bytes, and a byte counter operable to count the
      number of identical bytes in each sequence.

  [this is the determination of the match length]

  Gzip never checks for equality of the third byte used in the hash
  function. The hash function is such that on a hash hit with equality
  of the first two bytes, the third byte necessarily matches.

- In addition, gzip uses a "lazy" evaluation of string matches.  Even
  when a match is found, gzip may encode (with Huffman coding) a single
  unmatched byte. This is done when gzip determines that it is more
  beneficial to parse the input string differently because a longer
  match follows. In the Waterworth patent, a string match is always
  encoded as a (length, pointer) pair.

All other claims of the patent are dependent on claim 1
("a system as claimed in claim 1 in which ..."). Since gzip
does not infringe claim 1 it does not infringe the other claims.
In particular, claim 6 explicitly states that unmatched strings
are not compressed:

      A system as claimed in claim 5 in which the data receiving means
      includes decoder means operable to separate UNCOMPRESSED bytes of
      data from identification signals.

The gzip decoder never receives uncompressed bytes since all input is
compressed with Huffman coding [both literals and (length, offset) pairs].


The only "invention" in the Waterworth patent is the absence of hash
collision chains. The original description of the LZ77 algorithm
required searching for the true longest match, not just checking the
length of one particular match.  Using hashing for string searching
was very well known at the time of the patent application (March 86).
The patent description specifies explicitly that "Hash techniques are
well known and many differents forms of hash will be suitable".

The --fast option of gzip was on purpose made slower than possible
precisely because of the existence of the Waterworth patent.
See in particular the following part of the gzip TODO file:

  Add a super-fast compression method, suitable for implementing
  file systems with transparent compression. One problem is that the
  best candidate (lzrw1) is patented twice (Waterworth 4,701,745
  and Gibson & Graybill 5,049,881).


(b) 5,016,009

This is standard LZ77 with hashing, and collisions resolved using
linked lists. There are several important restrictions which let gzip
escape from the patent:

- the linked lists are implemented only with offsets. The end of
  a chain is detected by adding together all offsets, until the
  sum becomes greater than the size of the history buffer. gzip uses
  direct indices, and so detects the end of the chains differently.
  The exact wording of claim 1 of the patent is:

   ... said data compression system comprising ... an offset array means
   ... said method comprising the steps of ...

       calculating a difference between said history array pointer
          and said pointer obtained from said hash table means,
       storing said difference into said offset array means entry
          pointed to by said history array pointer, ...

   gzip never calculates such a difference and does not have any offset
   array.

- unmatched strings are emitted as literal bytes without any
  compression. gzip uses Huffman encoding on the unmatched strings.
  This is the same argument as for the Waterworth patent.

- unmatched strings are preceded by
   ... a "raw" data tag indicating that no matching data string was found

  gzip does not use such a tag because it uses a single Huffman table for
  both string literals and match lengths. It is only the prefix
  property of Huffman codes which allows the decoder to distinguish
  the two cases. So there is not a unique "raw" tag preceding all
  literals. This is not a minor point. It is one of the reasons
  giving gzip superior compression to that obtained with the Stac
  algorithm.

- a string match is always encoded as a (length, pointer) pair.
  Gzip uses a "lazy" evaluation of string matches as described
  above for the Waterworth patent.

All other claims of the patent are dependent on claim 1 ("the method
of claim 1 wherein ..."). Since gzip does not infringe claim 1 it does
not infringe the other claims.  In any case, I have studied in detail
all the 77 claims to make sure that gzip does not infringe.

Unrelated note: this Stac patent is the only one where I found an
original and non obvious idea. The hash table is refreshed using a
incremental mechanism, so that the refresh overhead is distributed
among all input bytes. This allows the real response time necessary in
disk compressors such as Stacker (the Stac product). gzip does not use
this idea, and refreshes the hash table in a straightforward manner
every 32K bytes.


One final comment: I estimate that I have now spent more time studying
data compression patents than actually implementing data compression
algorithms.  I have a partial list of 318 data compression patents,
even though for the moment I restrict myself mostly to lossless
algorithms (ignoring most image compression patents).  Richard
Stallman has been *extremely* careful before accepting gzip as the GNU
compressor.  I continue to study new patents regularly. I would of
course very much prefer spending what's left of my spare time
improving the gzip compression algorithm instead of studying patents.
Some improvements that I would have liked to put in gzip have not been
incorporated because of patents.

In short, every possible precaution has been taken to make sure that
gzip isn't covered by patents.

Jean-loup Gailly, author of gzip.
jloup@chorus.fr

Fri, 07 Jul 2000 01:30:40 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: lztext and compression ratios... 
Maybe you just want to use zlib. Let other guys hammer out the details.

On Thu, 6 Jul 2000, Jan Wieck wrote:

>     While  writing  it  I  noticed  that  the algorithm is really
>     expensive for big items. The history lookup  table  allocated
>     is  8  times (on 32 bit architectures) the size of the input.
>     So if you want to have 1MB compressed, it'll allocate 8MB for
>     the  history.  It  hit  me  when  I  was hunting a bug in the
>     toaster earlier today. Doing an update to a toasted  item  of
>     5MB, resulting in a new value of 10MB, the backend blew up to
>     290MB of virtual memory - oh boy. I definitely need  to  make
>     that smarter.
> 
>     When  I wrote it I never thought about items that big. It was
>     before we had the idea of TOAST.
> 
>     This all might  open  another  discussion  I'll  start  in  a
>     separate thread.
> 
> 
> Jan
> 
> --
> 
> #======================================================================#
> # It's easier to get forgiveness for being wrong than for being right. #
> # Let's break this rule - forgive me.                                  #
> #================================================== JanWieck@Yahoo.com #
> 
> 
> 

-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Fri, 7 Jul 2000 09:14:07 -0400 (EDT)
eisentrp@csis.gvsu.edu
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios...
eisentrp@csis.gvsu.edu writes:
> Maybe you just want to use zlib. Let other guys hammer out the details.

I've been wondering about that myself.  Jan, have you actually
benchmarked decompression speed on this method vs. stock zlib?
Unless the differential is huge we might want to just skip the
patent worries.

zlib has a BSD-style license, so we could include it in our
distribution to avoid worries about whether it's installed.

I've written to Jean-loup for his advice.  He's pretty busy these
days (CTO at some startup, I believe) but we'll see if he has time
to comment.

			regards, tom lane

Fri, 07 Jul 2000 11:56:26 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios... 
> eisentrp@csis.gvsu.edu writes:
> > Maybe you just want to use zlib. Let other guys hammer out the details.
> 
> I've been wondering about that myself.  Jan, have you actually
> benchmarked decompression speed on this method vs. stock zlib?
> Unless the differential is huge we might want to just skip the
> patent worries.

If we found later that there was a patent problem, could we just issue a
new release with a new compression algorithm?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 7 Jul 2000 12:02:30 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios...
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Unless the differential is huge we might want to just skip the
>> patent worries.

> If we found later that there was a patent problem, could we just issue a
> new release with a new compression algorithm?

Yeah, we could, and it could presumably even be a fully compatible
dot-release with no change to the on-disk representation.  That
representation and consequently the decompression algorithm are safe
enough, it's the details of the compressor's search for matching
patterns that are a patent minefield.

However, changing the code after-the-fact might not be enough to keep us
from being sued :-(.  I'd rather use something that's pretty well
established as being in the clear...

			regards, tom lane

Fri, 07 Jul 2000 12:52:29 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios... 
eisentrp@csis.gvsu.edu wrote:
> Maybe you just want to use zlib. Let other guys hammer out the details.
>

    We  cannot  assume that zlib is available everywhere. Thus it
    must be determined during configure and where it isn't, TOAST
    can  only  move  off  values  to make tuples fit into blocks.
    Since decompression of already in memory items is alot faster
    than doing an index scan on the TOAST table, I expect this to
    make installations without zlib damned slow.

    And how should binary distributions like RPM's handle  it?  I
    assume  that  this  problem is already on it's way because of
    the integration of zlib into pg_dump. The only way I  see  is
    having  different  RPM's  for  each  possible  combination of
    available helper libs.  Or  is  there  another  way  to  work
    around?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Fri, 7 Jul 2000 21:47:10 +0200 (MEST)
JanWieck@t-online.de (Jan Wieck)
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios...
JanWieck@t-online.de (Jan Wieck) writes:
> eisentrp@csis.gvsu.edu wrote:
>> Maybe you just want to use zlib. Let other guys hammer out the details.

>     We  cannot  assume that zlib is available everywhere.

We can if we include it in our distribution --- which we could; it's
pretty small and uses a BSD-style license.  I can assure you the zlib
guys would be happy with that.  And it's certainly as portable as our
own code.  The real question is, is a custom compressor enough better
than zlib for our purposes to make it worth taking any patent risks?

We could run zlib at a low compression setting (-z1 to -z3 maybe)
to make compression relatively fast, and since that also doesn't
generate a custom Huffman tree, the overhead in the compressed data
is minor even for short strings.  And its memory footprint is
certainly no worse than Jan's method...

The real question is whether zlib decompression is markedly slower
than Jan's code.  Certainly Jan's method is a lot simpler and *should*
be faster --- but on the other hand, zlib has had a heck of a lot
of careful performance tuning put into it over the years.  The speed
difference might not be as bad as all that.

I think it's worth taking a look at the option.

			regards, tom lane

Fri, 07 Jul 2000 17:56:12 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios... 
Jan Wieck writes:

> > Maybe you just want to use zlib. Let other guys hammer out the details.

>     And how should binary distributions like RPM's handle  it?

They got all their fancy dependency mechanisms for that (which never work
well, but anyway).


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Sat, 8 Jul 2000 02:16:18 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios...
At 21:47 7/07/00 +0200, Jan Wieck wrote:
>
>    And how should binary distributions like RPM's handle  it?  I
>    assume  that  this  problem is already on it's way because of
>    the integration of zlib into pg_dump. The only way I  see  is
>    having  different  RPM's  for  each  possible  combination of
>    available helper libs.  Or  is  there  another  way  to  work
>    around?

This remoinded me of a question I wanted to ask Unix people: other OSs I
use allow for dynamic linking, at runtime and in code, against shared
libraries, and I know Unix must allow this. The places where zlib is used
are pretty limited, so it might be worth considering doing the 'HAVE_ZLIB'
kinds of checks at runtime. Then one binary fits all...

Is this hard or easy - at least on machines with a libz.so?

Is it worth doing?

I guess the alternative on rpm is to create both: pg_dump.zlib and
pg_dump.nozlib, and install the right one?


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sat, 08 Jul 2000 11:16:36 +1000
Philip Warner <pjw@rhyme.com.au>
Re: Re: [SQL] Re: [GENERAL] lztext and compression
  ratios...
> This remoinded me of a question I wanted to ask Unix people: other OSs I
> use allow for dynamic linking, at runtime and in code, against shared
> libraries, and I know Unix must allow this. The places where zlib is used
> are pretty limited, so it might be worth considering doing the 'HAVE_ZLIB'
> kinds of checks at runtime. Then one binary fits all...
> 
> Is this hard or easy - at least on machines with a libz.so?
> 
> Is it worth doing?
> 
> I guess the alternative on rpm is to create both: pg_dump.zlib and
> pg_dump.nozlib, and install the right one?

We do dynamic loading for functions.  Not sure if we want to load zlib
dynamically if we can help it.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 7 Jul 2000 21:19:50 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: [SQL] Re: [GENERAL] lztext and compression  ratios...
>> This remoinded me of a question I wanted to ask Unix people: other OSs I
>> use allow for dynamic linking, at runtime and in code, against shared
>> libraries, and I know Unix must allow this. The places where zlib is used
>> are pretty limited, so it might be worth considering doing the 'HAVE_ZLIB'
>> kinds of checks at runtime. Then one binary fits all...

> We do dynamic loading for functions.  Not sure if we want to load zlib
> dynamically if we can help it.

Not bloody likely!  Do you want to be in a position where you restart
your postmaster and suddenly chunks of your database are inaccessible?
That's what could happen to you if someone moves or deletes libz.so.

I don't mind being dynamically linked to standard system shared libs;
if libc.so is busted then whether Postgres launches is the least of
your worries.  But dynamic dependence on an optional package that's
probably living in /usr/local strikes me as exceedingly risky.

If we do go with using zlib instead of homegrown code, I would recommend
building and statically linking to our own copy even if there is a copy
available on the system.  This will prevent cross-version compatibility
problems as well as where'd-my-library-go syndrome.  We cannot afford
those sorts of risks for something that could prevent us from reading
our database.

			regards, tom lane

Fri, 07 Jul 2000 21:49:53 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios...
> If we do go with using zlib instead of homegrown code, I would recommend
> building and statically linking to our own copy even if there is a copy
> available on the system.  This will prevent cross-version compatibility
> problems as well as where'd-my-library-go syndrome.  We cannot afford
> those sorts of risks for something that could prevent us from reading
> our database.

I don't know much about zlib, but it is hard to imagine it would have
the flexibility and optimization tuning of Jan's code.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 7 Jul 2000 21:54:54 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios...
At 21:49 7/07/00 -0400, Tom Lane wrote:
>
>Not bloody likely!  Do you want to be in a position where you restart
>your postmaster and suddenly chunks of your database are inaccessible?
>That's what could happen to you if someone moves or deletes libz.so.

My question was limited to it's use in pg_dump; rather than basing
pg_dump's compression bahaviour on configure, base it on it's runtime
environment. But my guess is you still would be inclined, rather strongly,
against it.


>If we do go with using zlib instead of homegrown code

This begs the obvious question: should pg_dump be using Jan's compression
code? In all cases/when zlib is not available?


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sat, 08 Jul 2000 12:00:31 +1000
Philip Warner <pjw@rhyme.com.au>
Re: Re: [SQL] Re: [GENERAL] lztext and compression
  ratios...
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I don't know much about zlib, but it is hard to imagine it would have
> the flexibility and optimization tuning of Jan's code.

Uh, I think you have that backwards ...

			regards, tom lane

PS: I do know a few things about zlib.

Fri, 07 Jul 2000 22:18:53 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios... 
Philip Warner <pjw@rhyme.com.au> writes:
>> Not bloody likely!  Do you want to be in a position where you restart
>> your postmaster and suddenly chunks of your database are inaccessible?
>> That's what could happen to you if someone moves or deletes libz.so.

> My question was limited to it's use in pg_dump; rather than basing
> pg_dump's compression bahaviour on configure, base it on it's runtime
> environment. But my guess is you still would be inclined, rather strongly,
> against it.

pg_dump is rather a different case.  I would want to see a runtime
option *not* to use compression, in case you know you are going to
need to restore on another system where zlib (or whatever) isn't
available.  But making an uncompressed dump today doesn't invalidate
the compressed dump you made yesterday, nor vice versa.

>> If we do go with using zlib instead of homegrown code

> This begs the obvious question: should pg_dump be using Jan's compression
> code? In all cases/when zlib is not available?

Good point.  If we include zlib in the distribution it would be pretty
silly for pg_dump not to use it.  If we don't, then Peter's remarks
about not liking an environment-determined feature set are relevant.

			regards, tom lane

Fri, 07 Jul 2000 22:28:01 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios... 
At 22:28 7/07/00 -0400, Tom Lane wrote:
>
>pg_dump is rather a different case.  I would want to see a runtime
>option *not* to use compression

That's there already; using -Z0 produces the same output as a mchine
without zlib.

>But making an uncompressed dump today doesn't invalidate
>the compressed dump you made yesterday, nor vice versa.

Not sure what you mean here.


>Good point.  If we include zlib in the distribution it would be pretty
>silly for pg_dump not to use it.  If we don't, then Peter's remarks
>about not liking an environment-determined feature set are relevant.

zlib seems to have been ported to zillions of architectures (must be a good
design ;-}), so it's probably pretty safe to put in the source tree, and
I'm sure any porting problems we encounter would be useful to the zlib
maintainers.

If we don't want to use zlib, then fefore using Jan's compression code in
pg_dump, I suppose I'd need to know that the output is compatible across
32/64 bit architectures and is not sensitive to other issues like endian-ness.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sat, 08 Jul 2000 12:56:46 +1000
Philip Warner <pjw@rhyme.com.au>
Re: Re: [SQL] Re: [GENERAL] lztext and compression
  ratios...
I get the following when building cvs:

gcc -I../../include  -O2 -g -Wall -Wmissing-prototypes
-Wmissing-declarations   -c pqformat.c -o pqformat.o
pqformat.c: In function `pq_endmessage':
pqformat.c:219: `errno' undeclared (first use in this function)
pqformat.c:219: (Each undeclared identifier is reported only once
pqformat.c:219: for each function it appears in.)
make: *** [pqformat.o] Error 1

Any ideas?

----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sat, 08 Jul 2000 14:44:39 +1000
Philip Warner <pjw@rhyme.com.au>
'errno' undefined?
Philip Warner <pjw@rhyme.com.au> writes:
> I get the following when building cvs:

Is that with the be-pqexec removal I just did?  I pulled out a couple
of header inclusions that I thought were no longer necessary --- and
they weren't on my platform.  But maybe something in there was needed
on yours.  Where does errno get declared in your system's headers?

			regards, tom lane

Sat, 08 Jul 2000 00:57:04 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: 'errno' undefined? 
At 00:57 8/07/00 -0400, Tom Lane wrote:
>Philip Warner <pjw@rhyme.com.au> writes:
>> I get the following when building cvs:
>
>Is that with the be-pqexec removal I just did?  I pulled out a couple
>of header inclusions that I thought were no longer necessary --- and
>they weren't on my platform.  But maybe something in there was needed
>on yours.  Where does errno get declared in your system's headers?

errno.h


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sat, 08 Jul 2000 15:02:05 +1000
Philip Warner <pjw@rhyme.com.au>
Re: 'errno' undefined? 
Philip Warner <pjw@rhyme.com.au> writes:
>> they weren't on my platform.  But maybe something in there was needed
>> on yours.  Where does errno get declared in your system's headers?

> errno.h

Hmm.  The stuff I just removed doesn't look like it would cause an
inclusion of errno.h.  Which other system headers include errno.h
on your box?

The answer probably is that pqformat.h needs to include <errno.h>,
but I'm just curious to understand why it didn't before ...

			regards, tom lane

Sat, 08 Jul 2000 01:17:15 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: 'errno' undefined? 
At 01:17 8/07/00 -0400, Tom Lane wrote:
>Hmm.  The stuff I just removed doesn't look like it would cause an
>inclusion of errno.h.  Which other system headers include errno.h
>on your box?
>
>The answer probably is that pqformat.h needs to include <errno.h>,
>but I'm just curious to understand why it didn't before ...
>

pjw@Cerberus2:~/work/postgresql-cvs/pgsql > grep errno.h /usr/include/*.h
/usr/include/argz.h:#include <errno.h>
/usr/include/envz.h:#include <errno.h>
/usr/include/errno.h: * ISO C Standard: 4.1.3 Errors    <errno.h>
/usr/include/errno.h:#endif /* errno.h  */
/usr/include/errnos.h:# include <linux/errno.h>
/usr/include/pthread.h:#include <errno.h>

pjw@Cerberus2:~/work/postgresql-cvs/pgsql > grep errno.h /usr/include/*/*.h
/usr/include/X11/Xlibint.h:#include <errno.h>
/usr/include/X11/Xos.h:#include <errno.h>
/usr/include/asm/unistd.h:/* user-visible error numbers are in the range -1
- -122: see <asm-i386/errno.h> */
/usr/include/linux/errno.h:#include <asm/errno.h>
/usr/include/linux/isdn.h:#include <linux/errno.h>
/usr/include/linux/mm.h:#include <linux/errno.h>
/usr/include/linux/notifier.h:#include <linux/errno.h>
/usr/include/linux/quota.h:#include <linux/errno.h>
/usr/include/mysql/my_sys.h:#include <errno.h>                  /* errno is
a define */
/usr/include/python1.5/Python.h:#include <errno.h>
/usr/include/rpcsvc/bootparam.h:#include <sys/errno.h>
/usr/include/rpcsvc/bootparam_prot.h:#include <sys/errno.h>
/usr/include/sys/errno.h:#include <errno.h>

no matches in local/include
----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sat, 08 Jul 2000 15:21:58 +1000
Philip Warner <pjw@rhyme.com.au>
Re: 'errno' undefined? 
>> The answer probably is that pqformat.h needs to include <errno.h>,
>> but I'm just curious to understand why it didn't before ...

Well, I noticed that <stdlib.h> pulls in <errno.h> on my box, so that
explains why I didn't see a problem.  I still don't see the connection
between the includes I killed and <errno.h> on your box, though...
curious.  Anyway I committed the change to pqformat.c.

			regards, tom lane

Sat, 08 Jul 2000 01:36:49 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: 'errno' undefined? 
Peter Eisentraut wrote:
  >>     And how should binary distributions like RPM's handle  it?
  >
  >They got all their fancy dependency mechanisms for that (which never work
  >well, but anyway).

Debian's work very well, because the Debian repository is centralised
under uniform quality control.

As a package maintainer, I can allow a package to depend on zlib -- in
which case the dependency will be satisfied by any version -- or I can
specify a versioned dependency.  I might have to provide the correct
library myself, if the zlib maintainer didn't want to maintain an old
version.  At the moment Debian provides zlib1g; if the major version
changes, I would expect to see this package remain and a new zlib2
package be created.

I would prefer a solution that did not _require_ me to maintain zlib
as well, if the dependency can be satisfied from another package.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Ask, and it shall be given you; seek, and ye shall
      find; knock, and it shall be opened unto you."        
                                  Matthew 7:7 



Sat, 08 Jul 2000 07:43:19 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Re: [SQL] Re: [GENERAL] lztext and compression
 ratios...
Philip Warner wrote:
> At 21:49 7/07/00 -0400, Tom Lane wrote:
> >
> >Not bloody likely!  Do you want to be in a position where you restart
> >your postmaster and suddenly chunks of your database are inaccessible?
> >That's what could happen to you if someone moves or deletes libz.so.
>
> My question was limited to it's use in pg_dump; rather than basing
> pg_dump's compression bahaviour on configure, base it on it's runtime
> environment. But my guess is you still would be inclined, rather strongly,
> against it.
>
>
> >If we do go with using zlib instead of homegrown code
>
> This begs the obvious question: should pg_dump be using Jan's compression
> code? In all cases/when zlib is not available?

    It can't. My code is designed for in-memory attribute values.
    It doesn't support streaming - which I assume is required for
    pg_dump.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Sat, 8 Jul 2000 13:29:17 +0200 (MEST)
JanWieck@t-online.de (Jan Wieck)
Re: Re: [SQL] Re: [GENERAL] lztext and compression  ratios....

> Where does errno get declared in your system's headers?

The ANSI C standard says errno is declared in <errno.h>.  Since ANSI C
also says that the standard header files are independent, it is poor
form for your system to have included <errno.h> via <stdlib.h>.

(Yeah, since you've already added <errno.h> this is a bit pedantic.)

Ciao,

Giles


Sun, 09 Jul 2000 08:38:07 +1000
Giles Lean <giles@nemeton.com.au>
Re: 'errno' undefined? 
Hi!

How can I count the number of rows affected by UPDATE or DELETE?

Przem


Mon, 10 Jul 2000 09:51:28 +0200
"Przem Kowalczyk" <przem@sprintnet.pl>
Counting affected rows

> How can I count the number of rows affected by UPDATE or DELETE?

This depends on the interface you are using.

* The psql SQL monitor always reports the number of affected rows
  (it's the 2nd number)

* In DBI "do" returns the number of affected rows.

* In JDBC "executeUpdate" does the same. (By the way, is there a way
  to get the number of rows from a executeQuery without counting them?).

Regards,
Mit freundlichem Gruï¿½,
	Holger Klawitter
--
Holger Klawitter                                    +49 (0)251 484 0637
holger@klawitter.de                            http://www.klawitter.de/


Mon, 10 Jul 2000 11:36:19 +0200
Holger Klawitter <holger@klawitter.de>
Re: Counting affected rows
Tom Lane wrote:
> JanWieck@t-online.de (Jan Wieck) writes:
> > eisentrp@csis.gvsu.edu wrote:
> >> Maybe you just want to use zlib. Let other guys hammer out the details.
>
> >     We  cannot  assume that zlib is available everywhere.
>
> We can if we include it in our distribution --- which we could; it's
> pretty small and uses a BSD-style license.  I can assure you the zlib
> guys would be happy with that.  And it's certainly as portable as our
> own code.  The real question is, is a custom compressor enough better
> than zlib for our purposes to make it worth taking any patent risks?

    Good,  we  shouldn't  worry about that anymore. If we want to
    use zlib, I vote for including it into our  distribution  and
    link static against the one shipped with our code.

    If we want to ...

> We could run zlib at a low compression setting (-z1 to -z3 maybe)
> to make compression relatively fast, and since that also doesn't
> generate a custom Huffman tree, the overhead in the compressed data
> is minor even for short strings.  And its memory footprint is
> certainly no worse than Jan's method...

    Definitely  not,  it's  memory  footprint  is  actually  much
    smaller.  Thus, I need  to  recreate  the  comparision  below
    again  after  making  the  history table of fixed size with a
    wrap around mechanism to get a small  footprint  on  multi-MB
    inputs too.

> The real question is whether zlib decompression is markedly slower
> than Jan's code.  Certainly Jan's method is a lot simpler and *should*
> be faster --- but on the other hand, zlib has had a heck of a lot
> of careful performance tuning put into it over the years.  The speed
> difference might not be as bad as all that.
>
> I think it's worth taking a look at the option.

    Some quick numbers though:

    I  simply  stripped  down pg_lzcompress.c to call compress2()
    and uncompress() instead of doing  anything  itself  (what  a
    nice,  small  source  file  :-). There might be some room for
    improvement  using  static   zlib   stream   allocaions   and
    deflateReset(),  inflateReset()  or  the  like.  But  I don't
    expect a significant difference from that.

    The test is a Tcl (pgtclsh) script doing the following:

    -   Loading 151 HTML files into a table t1 of structure (path
        text, content lztext).

    -   SELECT  *  FROM  t1  and checking for correct result set.
        Each file is read again during the check.

    -   UPDATE t1 SET content = upper(content).

    ï¿½   SELECT * FROM t1 and checking  for  correct  result  set.
        Each  file  is  read again, converted to upper case using
        Tcl's "string toupper" function for comparision.

    -   SELECT path FROM t1. Loop over result set  to  UPDATE  t1
        SET content = <value> WHERE path = <path>.  All files are
        read again and converted to lower case before UPDATE.

    -   SELECT * FROM t1 and check for correct result set.  Files
        are  again  reread  and  lower  case converted in Tcl for
        comparision.

    -   Doing 20 SELECT * FROM t1 to have  alot  more  decompress
        than compress cycles.

    Of course, there's an index on path. Here are the timings and
    sizes:

    Compressor | level | heap size | toastrel | toastidx | seconds
               |       |           |   size   |   size   |
    -----------+-------+-----------+----------+----------+--------
    PGLZ       |   -   |   425,984 |  950,272 |   32,768 |    5.20
    zlib       |   1   |   499,712 |  614,400 |   16,384 |    6.85
    zlib       |   3   |   499,712 |  557,056 |   16,384 |    6.75
    zlib       |   6   |   491,520 |  524,288 |   16,384 |    7.10
    zlib       |   9   |   491,520 |  524,288 |   16,384 |    7.21

    Seconds is an average over multiple runs. Interesting is that
    compression  level  3  seems  to  be  faster than 1. I double
    checked it because it was so surprising.

    Also, increasing the number of SELECT * at the end  increases
    the  difference. So the PGLZ decompressor does a perfect job.

    And what must be taken into account too is that  the  script,
    running  on  the  same  processor  and doing all the overhead
    (reading files, doing case conversions, quoting  values  with
    regsub  and  comparisions),  along  with  the normal Postgres
    query execution (parsing,  planning,  optimizing,  execution)
    occupies  a  substantial  portion  of the bare runtime. Still
    PGLZ is about 25% faster than the best zlib compression level
    I'm  seeing, while zlib gains a much better compression ratio
    (factor 1.7 at least).

    As I see it:

    If replacing the compressor/decompressor can cause a  runtime
    difference  of  25%  in  such a scenario, the pure difference
    between the two methods must be alot.

    PGLZ is what I mentioned in the comments. Optimized for speed
    on the cost of compression ratio.

    What I suggest:

    Leave  PGLZ  in place as the default compressor for toastable
    types.  Speed is what all benchmarks talk  about  -  on  disk
    storage size is seldom a minor note.

    Fix  it's history allocation for huge values and have someone
    (PgSQL Inc.?)  patenting the compression algorithm, so  we're
    safe at some point in the future. If there's a patent problem
    in it, we are already running the risk to get sued, the  PGLZ
    code got shipped with 7.0, used in lztext.

    We  can  discuss  about  enabling  zlib  as  a  per attribute
    configurable alternative further. But is the  confusion  this
    might cause worth it all?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Mon, 10 Jul 2000 17:11:51 +0200 (MEST)
JanWieck@t-online.de (Jan Wieck)
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios...
I've been collecting a few ideas for integrating the distribution making
process into the build system. (I did take a look at the mk-release and
mk-snapshot scripts on hub.org as well.) I have a trial implementation
which works well, except that it doesn't build the documentation.

1. Instead of the release_prep script we have a new target `distprep'.
Those makefiles that want to build something for the distribution can
simply implement this target. This is nice because you have the distprep
and the correspondng distclean target in the same file. To make a
distribution-ready tree, you do

./configure
make distprep
make distclean

2. In order to get rid of the CVS directories (and anything else you don't
want to ship, like _deadcode) and get the directory name straight,
*without* clobbering your checked out tree, I create a directory
"postgresql-$(VERSION)" and copy the files I want to distribute in there.
The conceptual procedure is

# use your already configured tree
make distprep
mkdir postgresql-$(VERSION)
copy all files from . into postgresql-$(VERSION), expect those you don't want
make -C postgresql-$(VERSION) distclean

This is done by `make distdir'.

3. `make dist' depends on distdir, tars up the prepared tree, and leaves a
file postgresql-$(VERSION).tar.gz, which you can give to your friends.

4. If you got an extra half hour you can run `make distcheck', which:

* makes a distribution, using make dist
* unpacks the distribution
* runs configure
* runs make -q distprep, to check whether the files you just prepared for
  distribution are really still up to date
* builds and installs everything
* runs make uninstall and checks whether it really uninstalled everything
* makes another distribution from this test tree
* checks whether this distribution is sufficiently similar to the previous
  one (i.e., same files, same size)

This approach should guard against the common tarball making problems:
misnamed top-level directory, missing files, funky timestamps, etc.


If there's any interest in this I can commit it so you can take a look. It
doesn't affect anything else (including release_prep).

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Mon, 10 Jul 2000 20:18:01 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Distribution making

On Mon, 10 Jul 2000, Peter Eisentraut wrote:

> 2. In order to get rid of the CVS directories (and anything else you don't
> want to ship, like _deadcode) and get the directory name straight,

 Need we still "_deadcode", if we have CVS? BTW. --- who compile backend
with "#define NOT_USED"?

> 3. `make dist' depends on distdir, tars up the prepared tree, and leaves a
> file postgresql-$(VERSION).tar.gz, which you can give to your friends.

 And what ".orig.tar.gz"? We Debian's user love it :-) (and IMHO it is 
really not bad idea.)

	 				Karel

PS. not flame please...


Mon, 10 Jul 2000 20:33:37 +0200 (CEST)
Karel Zak <zakkr@zf.jcu.cz>
Re: Distribution making
>     Compressor | level | heap size | toastrel | toastidx | seconds
>                |       |           |   size   |   size   |
>     -----------+-------+-----------+----------+----------+--------
>     PGLZ       |   -   |   425,984 |  950,272 |   32,768 |    5.20
>     zlib       |   1   |   499,712 |  614,400 |   16,384 |    6.85
>     zlib       |   3   |   499,712 |  557,056 |   16,384 |    6.75
>     zlib       |   6   |   491,520 |  524,288 |   16,384 |    7.10
>     zlib       |   9   |   491,520 |  524,288 |   16,384 |    7.21

Consider that the 25% slowness gets us a 35% disk reduction, and that
translates to fewer buffer blocks and disk accesses.  Seems there is a
clear tradeoff there.

>     If replacing the compressor/decompressor can cause a  runtime
>     difference  of  25%  in  such a scenario, the pure difference
>     between the two methods must be alot.
> 
>     Leave  PGLZ  in place as the default compressor for toastable
>     types.  Speed is what all benchmarks talk  about  -  on  disk
>     storage size is seldom a minor note.

True, disk storage is not the issue, but disk access are an issue.

>     We  can  discuss  about  enabling  zlib  as  a  per attribute
>     configurable alternative further. But is the  confusion  this
>     might cause worth it all?

I think we have to choose one proposal.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 10 Jul 2000 17:32:35 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: [SQL] Re: [GENERAL] lztext and compression ratios...
Sounds pretty good to me... but Marc's the guy who would normally be
using it, so it's his wishes you gotta cater to...

			regards, tom lane

Mon, 10 Jul 2000 18:26:38 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Distribution making 
Karel Zak writes:

> > 3. `make dist' depends on distdir, tars up the prepared tree, and leaves a
> > file postgresql-$(VERSION).tar.gz, which you can give to your friends.
> 
>  And what ".orig.tar.gz"? We Debian's user love it :-) (and IMHO it is 
> really not bad idea.)

What's that?

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Tue, 11 Jul 2000 00:26:47 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Distribution making

Sounds great to me ... what we have now is a major kludge ...

Basically, what I'd have to do for a release is CVSUP the latest code on
that branch, do a 'make dist' and that's it?  

Is this something that can be 'extended' to do the RPM also? :)

On Mon, 10 Jul 2000, Peter Eisentraut wrote:

> I've been collecting a few ideas for integrating the distribution making
> process into the build system. (I did take a look at the mk-release and
> mk-snapshot scripts on hub.org as well.) I have a trial implementation
> which works well, except that it doesn't build the documentation.
> 
> 1. Instead of the release_prep script we have a new target `distprep'.
> Those makefiles that want to build something for the distribution can
> simply implement this target. This is nice because you have the distprep
> and the correspondng distclean target in the same file. To make a
> distribution-ready tree, you do
> 
> ./configure
> make distprep
> make distclean
> 
> 2. In order to get rid of the CVS directories (and anything else you don't
> want to ship, like _deadcode) and get the directory name straight,
> *without* clobbering your checked out tree, I create a directory
> "postgresql-$(VERSION)" and copy the files I want to distribute in there.
> The conceptual procedure is
> 
> # use your already configured tree
> make distprep
> mkdir postgresql-$(VERSION)
> copy all files from . into postgresql-$(VERSION), expect those you don't want
> make -C postgresql-$(VERSION) distclean
> 
> This is done by `make distdir'.
> 
> 3. `make dist' depends on distdir, tars up the prepared tree, and leaves a
> file postgresql-$(VERSION).tar.gz, which you can give to your friends.
> 
> 4. If you got an extra half hour you can run `make distcheck', which:
> 
> * makes a distribution, using make dist
> * unpacks the distribution
> * runs configure
> * runs make -q distprep, to check whether the files you just prepared for
>   distribution are really still up to date
> * builds and installs everything
> * runs make uninstall and checks whether it really uninstalled everything
> * makes another distribution from this test tree
> * checks whether this distribution is sufficiently similar to the previous
>   one (i.e., same files, same size)
> 
> This approach should guard against the common tarball making problems:
> misnamed top-level directory, missing files, funky timestamps, etc.
> 
> 
> If there's any interest in this I can commit it so you can take a look. It
> doesn't affect anything else (including release_prep).
> 
> -- 
> Peter Eisentraut                  Sernanders v�g 10:115
> peter_e@gmx.net                   75262 Uppsala
> http://yi.org/peter-e/            Sweden
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Mon, 10 Jul 2000 20:00:13 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Distribution making
The Hermit Hacker wrote:
> Is this something that can be 'extended' to do the RPM also? :)

Yes.  Do you want that to happen?  It won't be too hard -- the
requirements would be simply having an 'RPM' directory in the tarball
that would contain the spec file and the required patches and ancilliary
programs.  Then, you simply issue (on a RedHat system :-)) the
appropriate 'rpm -tb' (or 'rpm -ta') command. The binary RPM's will be
built to the usual place, and all is well with the world.

Although the spec file might need to be in the tarball's top-level
directory to work -- I'll have to investigate this.

So, the RPM-building would go like:
download postgresql-$version.tar.gz to %{_topdir}/SOURCES
(in %{_topdir}/SOURCES, execute 'rpm -tb' (or -ta).
Pick up your RPM's in %{_topdir}/RPMS/%{_arch}
Install them with rpm -Uvh.

No sweat.  See, the RPM building process is already highly automated --
once the patches are made, that is.

Although, I must admit -- it would be very nice to not have those
patches.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 10 Jul 2000 20:09:43 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Distribution making
On Mon, 10 Jul 2000, Lamar Owen wrote:

> The Hermit Hacker wrote:
> > Is this something that can be 'extended' to do the RPM also? :)
> 
> Yes.  Do you want that to happen?  It won't be too hard -- the
> requirements would be simply having an 'RPM' directory in the tarball
> that would contain the spec file and the required patches and ancilliary
> programs.  Then, you simply issue (on a RedHat system :-)) the
> appropriate 'rpm -tb' (or 'rpm -ta') command. The binary RPM's will be
> built to the usual place, and all is well with the world.
> 
> Although the spec file might need to be in the tarball's top-level
> directory to work -- I'll have to investigate this.
> 
> So, the RPM-building would go like:
> download postgresql-$version.tar.gz to %{_topdir}/SOURCES
> (in %{_topdir}/SOURCES, execute 'rpm -tb' (or -ta).
> Pick up your RPM's in %{_topdir}/RPMS/%{_arch}
> Install them with rpm -Uvh.
> 
> No sweat.  See, the RPM building process is already highly automated --
> once the patches are made, that is.
> 
> Although, I must admit -- it would be very nice to not have those
> patches.

If this can be worked in along side/on top of what Peter is doing, I think
that would be great ...



Mon, 10 Jul 2000 23:30:14 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Distribution making

On Tue, 11 Jul 2000, Peter Eisentraut wrote:

> Karel Zak writes:
> 
> > > 3. `make dist' depends on distdir, tars up the prepared tree, and leaves a
> > > file postgresql-$(VERSION).tar.gz, which you can give to your friends.
> > 
> >  And what ".orig.tar.gz"? We Debian's user love it :-) (and IMHO it is 
> > really not bad idea.)
> 
> What's that?

 This is the original source package. But 'tar.gz' is some common archive
file. See any debian's ftp....

	 					Karel


Tue, 11 Jul 2000 10:10:12 +0200 (CEST)
Karel Zak <zakkr@zf.jcu.cz>
Re: Distribution making
On Mon, 10 Jul 2000, The Hermit Hacker wrote:

> Is this something that can be 'extended' to do the RPM also? :)

 Into standard *source* tree? I was long time glad of PG's devevelopment,
that create dateless and OS independent source.

 Well, we will have support for RPM, do you want support for Debian too?
If yes, do you accept /Debian directory in top source tree? 
Do you accept different directory philosophy for several dists.?

...etc.

 IMHO to *source* tree belong to matter for binary making only. A
distribution must be out of source.

                                        Karel

PS. is it this equation right: "OS = Linux = RH"? I hope that not.


Tue, 11 Jul 2000 11:05:56 +0200 (CEST)
Karel Zak <zakkr@zf.jcu.cz>
Re: Distribution making
Peter Eisentraut wrote:
  >Karel Zak writes:
  >
  >> > 3. `make dist' depends on distdir, tars up the prepared tree, and leaves
      > a
  >> > file postgresql-$(VERSION).tar.gz, which you can give to your friends.
  >> 
  >>  And what ".orig.tar.gz"? We Debian's user love it :-) (and IMHO it is 
  >> really not bad idea.)
  >
  >What's that?

A Debian source package consists of the upstream source (named
package.orig.tar.gz) together with a diff file and a control file.  These
should be all that are needed to rebuild the binary package from the source.

Some upstream developers make it hard for us, though. :-(

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "I waited patiently for the LORD; and he inclined unto 
      me, and heard my cry. He brought me up also out of an 
      horrible pit, out of the miry clay, and set my feet 
      upon a rock, and established my goings. And he hath 
      put a new song in my mouth, even praise unto our God.
      Many shall see it, and fear, and shall trust in the 
      LORD."                 Psalms 40:1-3 



Tue, 11 Jul 2000 11:30:04 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Distribution making 
The Hermit Hacker wrote:
> On Mon, 10 Jul 2000, Lamar Owen wrote:
> > The Hermit Hacker wrote:
> > > Is this something that can be 'extended' to do the RPM also? :)

> > Yes.  Do you want that to happen?  It won't be too hard -- the

> If this can be worked in along side/on top of what Peter is doing, I think
> that would be great ...

Well, it can be done.  However, the RPM' have historically had a tighter
cycle than the official release -- We have 7.0.2-1 and -2 RPM's (soon to
be up to -6 or -7) a problems are found in the packaging. Of course, as
I try to synchronize things, maybe that cycle time can go away. :-)

Now, I know that, in the past, feature patches were released on
postgresql.org (as well as bugfix patches) that were inbetween releases
-- that could be done for the RPM stuff easily enough.  But, I'd want
some comments from RPM distribution users before going that route.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 11 Jul 2000 11:10:01 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Distribution making
Karel Zak wrote:
> On Mon, 10 Jul 2000, The Hermit Hacker wrote:
> > Is this something that can be 'extended' to do the RPM also? :)
 
>  Into standard *source* tree? I was long time glad of PG's devevelopment,
> that create dateless and OS independent source.
 
>  Well, we will have support for RPM, do you want support for Debian too?
> If yes, do you accept /Debian directory in top source tree?
> Do you accept different directory philosophy for several dists.?

I don't mind one bit to have multiple distribution support in the
tarball.  
 
> ...etc.
 
>  IMHO to *source* tree belong to matter for binary making only. A
> distribution must be out of source.

There are more files in the RPM set, for instance, than just the
binaries -- there is the spec file, which controls the building of the
RPM's; there is a patch file (to patch around some madness in the source
that breaks the RPM package in some way); there are a couple of scripts
(startup and upgrade); there is a man page for  the upgrade script; as
well as other things.  Now, getting the *source* of the RPM distribution
packaging into the tarball might be OK -- I'm certainly not advocating
packaging binaries in the tarball!

RPM is used on many more systems than just RedHat Linux -- and I am
actively trying to get the *source* RPM to be buildable on as many
RPM-based systems as possible -- including being buildable on a Solaris
system that might have RPM installed, or a Tru64 system with RPM
installed (they DO exist).  Unfortunately there are a number of
RedHat-isms in the current RPMset that are having to be worked around --
although, this situation promises to improve in the near future, if I
have anything to do with it.

Therefore, a separate source RPM would not need to be distributed -- a
person can just download the tarball, execute a single command, and have
properly built RPM's for their system ready to install.  Can't get much
easier than that!

> PS. is it this equation right: "OS = Linux = RH"? I hope that not.

No, and neither is RPM == "RedHat Linux Only."

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 11 Jul 2000 11:58:25 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Distribution making

On Tue, 11 Jul 2000, Lamar Owen wrote:

> >  IMHO to *source* tree belong to matter for binary making only. A
> > distribution must be out of source.
> 
> There are more files in the RPM set, for instance, than just the
> binaries -- there is the spec file, which controls the building of the
> RPM's; there is a patch file (to patch around some madness in the source
> that breaks the RPM package in some way); there are a couple of scripts
> (startup and upgrade); there is a man page for  the upgrade script; as
> well as other things.  Now, getting the *source* of the RPM distribution
> packaging into the tarball might be OK -- I'm certainly not advocating
> packaging binaries in the tarball!

 I good known how act RH packaging. And I probably understant you.

> Therefore, a separate source RPM would not need to be distributed -- a
> person can just download the tarball, execute a single command, and have
> properly built RPM's for their system ready to install.  Can't get much
> easier than that!

Yes, I know. But it expect that in the *common-original-source* must be
.spec file. Or not? 

 I'm not enemy of RH, I only not sure if is good "foul" original source.

					Karel 



Tue, 11 Jul 2000 18:22:30 +0200 (CEST)
Karel Zak <zakkr@zf.jcu.cz>
Re: Distribution making
The Hermit Hacker writes:

> Basically, what I'd have to do for a release is CVSUP the latest code on
> that branch, do a 'make dist' and that's it?  

Yes. You'll just have to copy the documentation files in there first. Of
course eventually that should work better as well.


> Is this something that can be 'extended' to do the RPM also? :)

I'd have to look at how the RPMs are made, but in theory why not.


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Wed, 12 Jul 2000 02:23:48 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Distribution making
   From: Karel Zak <zakkr@zf.jcu.cz>
   Date: Tue, 11 Jul 2000 18:22:30 +0200 (CEST)

Hi Karel,

   > Yes, I know. But it expect that in the *common-original-source* must
   > be .spec file. Or not?

not, of course. But if it is there, you can easily built RPM via rpm -ta
source*tar.gz. Only Source RPMS (aka SRPM) must have .spec file built in.

   >  I'm not enemy of RH, I only not sure if is good "foul" original
   >  source.

The same for me. I do hate packagers who just rename my package.tar.gz to
package.orig.tar.gz and built tens of patches into one big .diff.gz which
is crazy to maintain :-)) No flamewars please.

BTW - Searching of mailing-lists is not working on postgresql.org. Try this
very long link :-))

http://www.postgresql.org/mhonarc/pgsql-hackers/search.cgi?ps=10&ul=http%3A%2F%2Fwww.postgresql.org%2Fmhonarc%2Fpgsql-hackers%2F%25&q=CREATE+SNAPSHOT&ps=10&o=0&m=all
-- 
Pavel Janďż˝k ml.
Pavel.Janik@linux.cz

Wed, 12 Jul 2000 12:42:47 +0200
Pavel.Janik@linux.cz (Pavel =?iso-8859-2?q?Jan=EDk?= ml.)
Re: Distribution making
JanWieck@t-online.de (Jan Wieck) writes:
>     Some quick numbers though:
>     I  simply  stripped  down pg_lzcompress.c to call compress2()
>     and uncompress() instead of doing  anything  itself  (what  a
>     nice,  small  source  file  :-).

I went at it in a different way: pulled out pg_lzcompress into a
standalone source program that could also call zlib.  These numbers
represent the pure compression or decompression time for memory-to-
memory processing, no other overhead at all.  Each run was iterated
1000 times to make it long enough to time accurately (so you can
read the times as "milliseconds per operation", though they're
really seconds).

A small text file of about 1K:

LZ compressed 990 bytes to 717 bytes in 0.35 u 0.00 sys sec
LZ decompressed 717 bytes to 990 bytes in 0.04 u 0.00 sys sec
zlib(3) compressed 990 bytes to 527 bytes in 0.73 u 0.00 sys sec
zlib uncompressed 527 bytes to 990 bytes in 0.21 u 0.00 sys sec
zlib(6) compressed 990 bytes to 513 bytes in 0.86 u 0.00 sys sec
zlib uncompressed 513 bytes to 990 bytes in 0.21 u 0.00 sys sec
zlib(9) compressed 990 bytes to 513 bytes in 0.86 u 0.00 sys sec
zlib uncompressed 513 bytes to 990 bytes in 0.20 u 0.00 sys sec

A larger text file, a bit under 100K:

LZ compressed 92343 bytes to 54397 bytes in 49.00 u 0.05 sys sec
LZ decompressed 54397 bytes to 92343 bytes in 3.74 u 0.00 sys sec
zlib(3) compressed 92343 bytes to 40524 bytes in 38.48 u 0.02 sys sec
zlib uncompressed 40524 bytes to 92343 bytes in 8.58 u 0.00 sys sec
zlib(6) compressed 92343 bytes to 38002 bytes in 68.04 u 0.03 sys sec
zlib uncompressed 38002 bytes to 92343 bytes in 8.13 u 0.00 sys sec
zlib(9) compressed 92343 bytes to 37961 bytes in 71.99 u 0.03 sys sec
zlib uncompressed 37961 bytes to 92343 bytes in 8.14 u 0.00 sys sec

It looks like the ultimate speed difference is roughly 2:1 on
decompression and less than that on compression, though zlib suffers
from greater setup overhead that makes it look worse on short strings.

Here is a much different example, a small JPEG image:

LZ compressed 5756 bytes to 5764 bytes in 1.48 u 0.00 sys sec
LZ decompressed 5764 bytes to 5756 bytes in 0.01 u 0.00 sys sec
zlib(3) compressed 5756 bytes to 5629 bytes in 3.18 u 0.00 sys sec
zlib uncompressed 5629 bytes to 5756 bytes in 0.75 u 0.00 sys sec
zlib(6) compressed 5756 bytes to 5625 bytes in 3.78 u 0.00 sys sec
zlib uncompressed 5625 bytes to 5756 bytes in 0.75 u 0.00 sys sec
zlib(9) compressed 5756 bytes to 5625 bytes in 3.78 u 0.00 sys sec
zlib uncompressed 5625 bytes to 5756 bytes in 0.75 u 0.00 sys sec

The PG-LZ code realizes it cannot compress this file and falls back to
straight bytewise copy, yielding very fast "decompress".  zlib manages
to eke out a few bytes' savings, so it goes ahead and compresses
instead of just storing literally, resulting in a big loss on
decompression speed for just a fractional space savings.  If we use
zlib, we'd probably be well advised to accept a compressed result only
if it's at least, say, 25% smaller than uncompressed to avoid this
low-return-on-investment situation.

>     There might be some room for
>     improvement  using  static   zlib   stream   allocaions   and
>     deflateReset(),  inflateReset()  or  the  like.  But  I don't
>     expect a significant difference from that.

It turns out you can get a noticeable win from overriding zlib's
default memory allocator, which calls calloc() even though it has
no need for pre-zeroed storage.  Just making it use malloc() instead
saves about 25% of the runtime on 1K-sized strings (the above numbers
include this improvement).  We'd probably want to make it use palloc
instead of malloc anyway, so this won't cost us any extra work.  I
have not pushed on it to see what else might be gained by tweaking.

>     What I suggest:

>     Leave  PGLZ  in place as the default compressor for toastable
>     types.  Speed is what all benchmarks talk  about  -  on  disk
>     storage size is seldom a minor note.

True, but the other side of the coin is that better compression and
smaller disk space will mean less I/O, better use of shared-memory
disk buffers, etc.  So to some extent, better compression helps pay
for itself.

>     Fix  it's history allocation for huge values and have someone
>     (PgSQL Inc.?)  patenting the compression algorithm, so  we're
>     safe at some point in the future.

That would be a really *bad* idea.  What will people say if we say
"Postgres contains patented algorithms, but we'll let you use them
for free" ?  They'll say "no thanks, I remember Unisys' repeatedly
broken promises about the GIF patent" and stay away in droves.
There is a *lot* of bad blood in the air about compression patents
of any sort.  We mustn't risk tainting Postgres' reputation with
that mess.

(In any case, one would hope you couldn't get a patent on this
method, though I suppose it never pays to overestimate the competence
of the USPTO...)

>     If there's a patent problem
>     in it, we are already running the risk to get sued, the  PGLZ
>     code got shipped with 7.0, used in lztext.

But it hasn't been documented or advertised.  If we take it out
again in 7.1, I think our exposure to potential lawsuits from it is
negligible.  Not that I think there is any big risk there anyway,
but we ought to consider the possibility.

My feeling is that going with zlib is probably the right choice.
The technical case for using a homebrew compressor instead isn't
very compelling, and the advantages of using a standardized,
known-patent-free library are not to be ignored.

			regards, tom lane

Thu, 13 Jul 2000 03:07:57 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: lztext and compression ratios... 
Tom Lane wrote:
> 
> JanWieck@t-online.de (Jan Wieck) writes:
> >     Some quick numbers though:
> >     I  simply  stripped  down pg_lzcompress.c to call compress2()
> >     and uncompress() instead of doing  anything  itself  (what  a
> >     nice,  small  source  file  :-).
> 
> I went at it in a different way: pulled out pg_lzcompress into a
> standalone source program that could also call zlib.  These numbers
> represent the pure compression or decompression time for memory-to-
> memory processing, no other overhead at all.  Each run was iterated
> 1000 times to make it long enough to time accurately (so you can
> read the times as "milliseconds per operation", though they're
> really seconds).

We could just make this part extensible as well, like the rest of 
postgres, so we would have directory tree like 

/compressors
  /nullcompressor
  /lzcompress
  /zlib
  /lzo
  /bzip2
  /my_new_supercompressor
  /classic_huffman_for_uppercase_american_english

and select the desired compressor at compilt time, or even better, on 
field by field basis at runtime, so that field that stores mainly 
tar.gz-s at compression level 9 will use nullcompressor, and others 
will use what is best for them.

> 
> >     Fix  it's history allocation for huge values and have someone
> >     (PgSQL Inc.?)  patenting the compression algorithm, so  we're
> >     safe at some point in the future.
> 
> That would be a really *bad* idea.  What will people say if we say
> "Postgres contains patented algorithms, but we'll let you use them
> for free" ?  They'll say "no thanks, I remember Unisys' repeatedly
> broken promises about the GIF patent" and stay away in droves.
> There is a *lot* of bad blood in the air about compression patents
> of any sort.  We mustn't risk tainting Postgres' reputation with
> that mess.
> (In any case, one would hope you couldn't get a patent on this
> method, though I suppose it never pays to overestimate the competence
> of the USPTO...)

And AFAIK (IANAL ;) you can only patent previously _unpublished_ work,
even by the patent applicant.

> 
> >     If there's a patent problem
> >     in it, we are already running the risk to get sued, the  PGLZ
> >     code got shipped with 7.0, used in lztext.
> 
> But it hasn't been documented or advertised.  If we take it out
> again in 7.1, I think our exposure to potential lawsuits from it is
> negligible.  Not that I think there is any big risk there anyway,
> but we ought to consider the possibility.
> 
> My feeling is that going with zlib is probably the right choice.
> The technical case for using a homebrew compressor instead isn't
> very compelling,

Speed seems to be a good reason, if we can keep it up.

> and the advantages of using a standardized,
> known-patent-free library are not to be ignored.

OTOH, there are possibly patents on other part of postgres, 
like indexing, storage methods, the mere fact that something is 
stored in another relation, using 'Z' as a protocol character, etc. 
etc. So using a patent-free compression library does not help much.

So if PgSQL Inc. has lots of lawyers with nothing to do, they could do
some patent research and scare all developers away with their findings
;)

-------------
Hannu

Thu, 13 Jul 2000 10:39:36 +0300
Hannu Krosing <hannu@tm.ee>
Re: lztext and compression ratios...
Some platforms (OSF/cc, HPUX) are already using -rpath or equivalent, so
you don't have to specify a shared library search path at runtime. I think
that a lot more platforms could use this. Can people comment on whether
and how it works on their platform? Essentially,

LDFLAGS+=-rpath '$(libdir)'

might do the trick for most.


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Tue, 18 Jul 2000 20:18:14 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Shared library search paths
For the SGI Irix 6.5, "man ld" gives:

     ....

     -rpath library_path
               Adds the library_path to the search path for DSOs.  Each
               library path is appended to the list of directories at the
               time the executable or DSO is loaded.  This option directs
               rld(5) to look in the named directories, but to look only
               for DSOs, and to stop looking when the correct one is found.

               This option can be specified only when the -shared or
               -call_shared options are also in effect.  For more
               information, see the rld(5) man page.  (C, C++, F77, F90)

    ....

     -shared   Produces a DSO, creates all of the tables for run-time
               linking, and resolves references to other specified shared
               objects.  The object created can be used by the linker to
               make dynamic executables.  (C, C++, F77, F90)

    ....

Hope this helps.
Mark

Peter Eisentraut wrote:

> Some platforms (OSF/cc, HPUX) are already using -rpath or equivalent, so
> you don't have to specify a shared library search path at runtime. I think
> that a lot more platforms could use this. Can people comment on whether
> and how it works on their platform? Essentially,
>
> LDFLAGS+=-rpath '$(libdir)'
>
> might do the trick for most.
>
> --
> Peter Eisentraut                  Sernanders vï¿½g 10:115
> peter_e@gmx.net                   75262 Uppsala
> http://yi.org/peter-e/            Sweden

--
Mark Dalphin                          email: mdalphin@amgen.com
Mail Stop: 29-2-A                     phone: +1-805-447-4951 (work)
One Amgen Center Drive                       +1-805-375-0680 (home)
Thousand Oaks, CA 91320                 fax: +1-805-499-9955 (work)




Tue, 18 Jul 2000 14:55:35 -0700
Mark Dalphin <mdalphin@amgen.com>
Re: Shared library search paths
Peter Eisentraut wrote:
  >Some platforms (OSF/cc, HPUX) are already using -rpath or equivalent, so
  >you don't have to specify a shared library search path at runtime. I think
  >that a lot more platforms could use this. Can people comment on whether
  >and how it works on their platform? Essentially,
  >
  >LDFLAGS+=-rpath '$(libdir)'
  >
  >might do the trick for most.

As far as Debian is concerned, use of rpath is a bug.  Here's a quote from
some Debian system documentation:

  libtool automatically inserts `-rpath' settings when compiling your
  program. But `-rpath' can cause big problems if the referenced
  libraries get updated. Therefore, no Debian package should use the
  `-rpath' option.

  libtool also refuses to link shared libraries against other shared
  libraries.  Debian packages have to at least link against libc (with
  "-lc"), so that the dynamic linker knows whether to use the
  libc5-compat libraries or not.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "For God so loved the world, that he gave his only 
      begotten Son, that whosoever believeth in him should 
      not perish, but have everlasting life."     John 3:16 



Tue, 18 Jul 2000 23:47:38 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Shared library search paths 

for all the stuff I'm doign lately, I just do:

setenv LDFLAGS "-R/usr/local/pgsql/lib -R/usr/local/lib"

and let configure handle the rest ...


On Tue, 18 Jul 2000, Mark Dalphin wrote:

> For the SGI Irix 6.5, "man ld" gives:
> 
>      ....
> 
>      -rpath library_path
>                Adds the library_path to the search path for DSOs.  Each
>                library path is appended to the list of directories at the
>                time the executable or DSO is loaded.  This option directs
>                rld(5) to look in the named directories, but to look only
>                for DSOs, and to stop looking when the correct one is found.
> 
>                This option can be specified only when the -shared or
>                -call_shared options are also in effect.  For more
>                information, see the rld(5) man page.  (C, C++, F77, F90)
> 
>     ....
> 
>      -shared   Produces a DSO, creates all of the tables for run-time
>                linking, and resolves references to other specified shared
>                objects.  The object created can be used by the linker to
>                make dynamic executables.  (C, C++, F77, F90)
> 
>     ....
> 
> Hope this helps.
> Mark
> 
> Peter Eisentraut wrote:
> 
> > Some platforms (OSF/cc, HPUX) are already using -rpath or equivalent, so
> > you don't have to specify a shared library search path at runtime. I think
> > that a lot more platforms could use this. Can people comment on whether
> > and how it works on their platform? Essentially,
> >
> > LDFLAGS+=-rpath '$(libdir)'
> >
> > might do the trick for most.
> >
> > --
> > Peter Eisentraut                  Sernanders v�g 10:115
> > peter_e@gmx.net                   75262 Uppsala
> > http://yi.org/peter-e/            Sweden
> 
> --
> Mark Dalphin                          email: mdalphin@amgen.com
> Mail Stop: 29-2-A                     phone: +1-805-447-4951 (work)
> One Amgen Center Drive                       +1-805-375-0680 (home)
> Thousand Oaks, CA 91320                 fax: +1-805-499-9955 (work)
> 
> 
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 


Tue, 18 Jul 2000 22:48:33 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: [PORTS] Shared library search paths
> Some platforms (OSF/cc, HPUX) are already using -rpath or equivalent, so
> you don't have to specify a shared library search path at runtime. I think
> that a lot more platforms could use this. Can people comment on whether
> and how it works on their platform? Essentially,
> LDFLAGS+=-rpath '$(libdir)'

For linux (at least gcc 2.7.x and 2.95.2 systems):

if specified in the compilation step,

  -Wl,-rpath $(libdir)

or if specified directly to the linker

  -rpath $(libdir)

                     - Thomas

Wed, 19 Jul 2000 01:52:10 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [PORTS] Shared library search paths
Oliver Elphick writes:

> As far as Debian is concerned, use of rpath is a bug.  Here's a quote from
> some Debian system documentation:
> 
>   libtool automatically inserts `-rpath' settings when compiling your
>   program.

I don't think so.

> But `-rpath' can cause big problems if the referenced
>   libraries get updated. Therefore, no Debian package should use the
>   `-rpath' option.

I'm not sure I buy that. All -rpath does is add a directory to the search
path that the program consults at runtime for its shared libraries. So
it's just an alternative in place of

hard-coded into dynamic linker
/etc/ld.so.conf
LD_LIBRARY_PATH

but it's the terminally accurate alternative.

What does happen if the referenced library gets updated? Nothing. -rpath
doesn't reference any libraries, it just suggests to the runtime linker
where it might look for one. I don't want to use it to find system
libraries, I just want psql to find libpq, and the right libpq, and I want
to relieve installers from having to fiddle around with these settings.

>   libtool also refuses to link shared libraries against other shared
>   libraries.

I don't think so.


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Thu, 20 Jul 2000 00:46:01 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Shared library search paths 
Peter Eisentraut wrote:
  >Oliver Elphick writes:
  >
  >> As far as Debian is concerned, use of rpath is a bug.  Here's a quote from
  >> some Debian system documentation:
  >> 
  >>   libtool automatically inserts `-rpath' settings when compiling your
  >>   program.
  >
  >I don't think so.
  >
  >> But `-rpath' can cause big problems if the referenced
  >>   libraries get updated. Therefore, no Debian package should use the
  >>   `-rpath' option.
  >
  >I'm not sure I buy that. All -rpath does is add a directory to the search
  >path that the program consults at runtime for its shared libraries. 

I'm referring this back to the Debian mailing lists; perhaps this
documentation is out of date.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Behold, what manner of love the Father hath bestowed 
      upon us, that we should be called the sons of God..." 
                                I John 3:1 



Thu, 20 Jul 2000 08:53:27 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Shared library search paths 
"Oliver Elphick" wrote:
  >I'm referring this back to the Debian mailing lists; perhaps this
  >documentation is out of date.
 
Here's some responses.  It seems that I shall probably have to disable
use of -rpath for the Debian build.

=========================================================================
Date: Fri, 21 Jul 2000 09:09:32 +0200
From: "J.H.M. Dassen (Ray)" <jhm@cistron.nl>
To: debian-devel@lists.debian.org
Subject: Re: Use of -rpath

In my experience, -rpath is a big PITA. For example, Red Hat used to ship a
libc5 CDE package that was linked -rpath /usr/X11R6/lib. In that time,
Debian's X was already libc6, and libc5 versions of the X libraries were in
/usr/lib/libc5-compat. This directory was in /etc/ld.so.conf, and the
dynamic loader was smart enough to know which library to use, /except/ when
-rpath was used. The result was that a libc5 binary, compiled for libc5 X
libraris, got loaded against libc6 X libraries, and segfaulted.

Now if -rpath's semantics were changed to it adding _to the end_ of the
dynamic loader's directory search path, it might actually be useful.

Ray
=========================================================================

To: debian-devel@lists.debian.org
Subject: Re: Use of -rpath
From: Brian May <bam@debian.org>
Date: 21 Jul 2000 17:35:39 +1000

    >> From: Peter Eisentraut <peter_e@gmx.net>
    Mike> * * *
    >> > libtool also refuses to link shared libraries against other
    >> shared > libraries.
    >> 
    >> I don't think so.

    Mike> Peter seems clearly right on this:

    Mike> guardian:~ ldd /usr/lib/libpq.so.2.0 libc.so.6 =>
    Mike> /lib/libc.so.6 (0x40011000) libcrypt.so.1 =>
    Mike> /lib/libcrypt.so.1 (0x400ee000) /lib/ld-linux.so.2 =>
    Mike> /lib/ld-linux.so.2 (0x80000000)

That should read "libtool 1.3 refuses to link shared libraries against
other *uninstalled* shared libraries".

ie libtool 1.3 will only allow it if both libraries are installed.

libtool 1.4 will fix this serious limitation. At least, it is serious
for packages like Kerberos (both implementations), which come with
many libraries that must (read: should) link to each other in the same
source structure.

However, AFAIK, this has nothing to do with -rpath??????
=========================================================================
Date: Fri, 21 Jul 2000 09:39:43 +0200
From: "Marcelo E. Magallon" <mmagallo@debian.org>
To: debian-devel@lists.debian.org
Subject: Re: Use of -rpath

>> Peter Eisentraut <peter_e@gmx.net> writes:

 > > As far as Debian is concerned, use of rpath is a bug.  Here's a quote from
 > > some Debian system documentation:
 > > 
 > >   libtool automatically inserts `-rpath' settings when compiling your
 > >   program.
 > 
 > I don't think so.
 
 Nowadays the situation might have changed...  let me check...  Nope,
 1.3.5 still uses -rpath whenever shared libraries are going to be
 used.

 > > But `-rpath' can cause big problems if the referenced
 > >   libraries get updated. Therefore, no Debian package should use the
 > >   `-rpath' option.
 > 
 > I'm not sure I buy that. All -rpath does is add a directory to the search
 > path that the program consults at runtime for its shared libraries. So
 > it's just an alternative in place of

 The rpath in the library tells the linker where the library is
 installed.  This is hardcoded into the linked program.  From ld.info:

 >   Add a directory to the runtime library search path.  This is used
 >   when linking an ELF executable with shared objects.  All `-rpath'
 >   arguments are concatenated and passed to the runtime linker, which
 >   uses them to locate shared objects at runtime.  The `-rpath'
 >   option is also used when locating shared objects which are needed
 >   by shared objects explicitly included in the link; see the
 >   description of the `-rpath-link' option.  If `-rpath' is not used
 >   when linking an ELF executable, the contents of the environment
 >   variable `LD_RUN_PATH' will be used if it is defined.

 This is particularly nasty when you -rpath things like /usr/lib.

 > >   libtool also refuses to link shared libraries against other shared
 > >   libraries.
 > 
 > I don't think so.

 He's right.  See (libtool.info)Inter-library dependencies.  Note
 particularly:

 >  The simple-minded inter-library dependency tracking code of libtool
 >  releases prior to 1.2 was disabled because it was not clear when it
 >  was possible to link one library with another, and complex failures
 >  would occur.  A more complex implementation of this concept was
 >  re-introduced before release 1.3, but it has not been ported to all
 >  platforms that libtool supports.  The default, conservative behavior
 >  is to avoid linking one library with another, introducing their
 >  inter-dependencies only when a program is linked with them.  

 i.e., you have to ask libtool to do this, else it won't.

 Greetings,

                                        Marcelo
=========================================================================

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Greater love hath no man than this, that a man lay 
      down his life for his friends."      John 15:13 



Fri, 21 Jul 2000 16:51:20 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Shared library search paths 
I noticed that INHERITS doesn't propogate indecies, It'd be nice
if there was an toption to do so.

thanks,
-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."

Mon, 28 Aug 2000 12:00:07 -0700
Alfred Perlstein <bright@wintelcom.net>
INHERITS doesn't offer enough functionality
Alfred Perlstein wrote:
> 
> I noticed that INHERITS doesn't propogate indecies, It'd be nice
> if there was an toption to do so.

Yep it would. Are you volunteering?

Tue, 29 Aug 2000 11:09:18 +1100
Chris <chrisb@nimrod.itg.telstra.com.au>
Re: INHERITS doesn't offer enough functionality
* Chris <chrisb@nimrod.itg.telstra.com.au> [000828 17:15] wrote:
> Alfred Perlstein wrote:
> > 
> > I noticed that INHERITS doesn't propogate indecies, It'd be nice
> > if there was an toption to do so.
> 
> Yep it would. Are you volunteering?

Ha, right now I'm doing a major rewrite of my own code to hack
around vacuum, it'll have to wait but thanks for the vote of
confidence. :-)

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."

Mon, 28 Aug 2000 17:20:38 -0700
Alfred Perlstein <bright@wintelcom.net>
Re: INHERITS doesn't offer enough functionality
I've brought this up before, but maybe if I sell it as a bug someone will
agree. :-)

It would be desirable for several reasons that PostgreSQL can be installed
safely with --prefix=/usr/local, or some other such shared location.  
(One reason:  no PATH, MANPATH, LD_LIBRARY_PATH to set up.)

The main problem is that we'd clutter the include/ subtree hopelessly with
files such as <c.h>, <os.h>, <config.h>, which has a serious chance of
breaking other autoconfiscated packages.  Not to mention the other dozen
or so files that will be spread out evenly.

My proposal is to set includedir=${prefix}/include/postgresql (instead of
${prefix}/include) in such cases where the prefix is shared, i.e., it does
not contain something like "pgsql" already. (precise pattern t.b.d.)  
This is similar to the existing RPM's, and compatible with FHS, GNU fs
std., and BSD hier(7).  Apache and Perl also have a similar behaviour in
their installation process.  Additionally, one can now use `pg-config
--includedir` to find the right include directory anywhere.

Marc objected that he liked "everything in one place".  But doing that is
exactly what's causing the problem here.  Additionally I ask why then the
default is prefix is /usr/local/pgsql in the first place, which is
certainly not in support of that notion.  I think having a cooperative
installation layout should be prioritized.

Secondly, I'd like to do the same thing with the share/ subtree.  That
tree is only read by the PG programs anyway so no one has to know about
it.  It's common practice for every package to have its own tree under
share and not to write into it directly.  That would also help associate a
file like `global.description' to where it belongs.

Finally, the doc tree should also get this treatment.  Otherwise we'd get
files like /usr/local/doc/{admin,tutorial}/index.html, which do not
indicate at all what package they belong to and they could be confused
with documentation of the operating system proper.  Users would only have
to update their bookmarks, but I doubt that installations into shared
prefixes are large scale anyway.

Comments?  Better objections? :-)

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Wed, 27 Sep 2000 12:43:23 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Installation layout is still hazardous for shared prefixes
Peter Eisentraut <peter_e@gmx.net> writes:
> It would be desirable for several reasons that PostgreSQL can be installed
> safely with --prefix=/usr/local, or some other such shared location.  
> ...
> My proposal is to set includedir=${prefix}/include/postgresql (instead of
> ${prefix}/include) in such cases where the prefix is shared, i.e., it does
> not contain something like "pgsql" already. (precise pattern t.b.d.)  

Hmm, so basically you propose an install setup whereby 'bin' and 'lib'
files can go directly into /usr/local/bin and /usr/local/lib, but
everything else still lives in postgres-specific directories?

To do that without creating problems, we'd have to go back to making
sure that all the programs we install have 'pg'-prefixed names.  The
scripts (createdb and so forth) don't at the moment, and names like
'createuser' clearly have potential for confusion if they are in non-
PG-specific directories.

I think it would be a real bad idea to put the postmaster and postgres
executables right in /usr/local/bin.  Perhaps it is time to think about
a separate 'sbin' directory for programs that aren't supposed to be
invoked by normal users.  Those two, initdb, initlocation, and ipcclean
could certainly go to sbin, also pg_id, maybe the create/drop scripts
if you feel those are admin-only.  Perhaps using a private sbin directory
could eliminate the issue of needing to rename stuff.

The stuff that's going into lib doesn't look like it'd cause any big
conflicts, and I agree that not having to run ldconfig or equivalent
would eliminate a lot of install headaches.

			regards, tom lane

Wed, 27 Sep 2000 10:53:43 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Installation layout is still hazardous for shared prefixes 
On Wed, Sep 27, 2000 at 10:53:43AM -0400, Tom Lane wrote:
> I think it would be a real bad idea to put the postmaster and postgres
> executables right in /usr/local/bin.  Perhaps it is time to think about
> a separate 'sbin' directory for programs that aren't supposed to be
> invoked by normal users.  Those two, initdb, initlocation, and ipcclean
> could certainly go to sbin, also pg_id, maybe the create/drop scripts
> if you feel those are admin-only.  Perhaps using a private sbin directory
> could eliminate the issue of needing to rename stuff.

generally there is a /usr/local/sbin or /usr/local/libexec for things like
postgres and postmaster.

at least on freebsd.

if we are gonna go this route, i  would prefer that we not have any "special"
directories for postgres, other than the data directory.

a layout of:

/usr/local/bin
/usr/local/include/pgsql
/usr/local/lib
/usr/local/libexec
/usr/local/pgsql/data or /usr/local/pgsql

the last one could also be /home/pgsql or whatever.

-- 
[ Jim Mercer                 jim@reptiles.org              +1 416 410-5633 ]
[          Reptilian Research -- Longer Life through Colder Blood          ]
[  Don't be fooled by cheap Finnish imitations; BSD is the One True Code.  ]

Wed, 27 Sep 2000 11:02:06 -0400
Jim Mercer <jim@reptiles.org>
Re: Installation layout is still hazardous for shared prefixes
Tom Lane writes:

> Hmm, so basically you propose an install setup whereby 'bin' and 'lib'
> files can go directly into /usr/local/bin and /usr/local/lib, but
> everything else still lives in postgres-specific directories?

Yes.

In detail, for those who cry "do it like Debian", we have these categories
of installation directories (not all actually used by PostgreSQL):

bin	- user programs
sbin	- administrator programs
libexec	- programs used only by other programs

lib	- libraries
include	- headers

share	- architecture independent read-only data
etc	- single-host read-only data
com	- architecture independent writeable data
var	- single-host writeable data

doc	- documentation
man	- manual pages

These do not actually have to exist with these names (although they
usually do), but the point is that each directory has a fairly orthogonal
purpose, thus enabling the local installer to easily adopt the layout to
his local convention.

Now the convention is that if you have a lot of files to put into one of
these directories you create a private subdirectory, which is what I'm
proposing here.  However, the bin, sbin, lib, and man directories are
obviously exempted from this convention, because otherwise the
shell/linker/man system won't find the files without the same contortions
we're trying to avoid here.

> To do that without creating problems, we'd have to go back to making
> sure that all the programs we install have 'pg'-prefixed names.

Yeah, that again.  That would be a true incompatibility, though, so it's
more complex.  But let's consider that complementary, not prerequisite to
the issue at hand.

> I think it would be a real bad idea to put the postmaster and postgres
> executables right in /usr/local/bin.

Why?

> Perhaps it is time to think about a separate 'sbin' directory for
> programs that aren't supposed to be invoked by normal users.

Good idea as well, but again a compatibility break that needs to be
thought about.

> Perhaps using a private sbin directory could eliminate the issue of
> needing to rename stuff.

I don't think private sbin directories are conventional.  But again,
that's really a complementary issue.


-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 28 Sep 2000 15:08:28 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Installation layout is still hazardous for shared
 prefixes
Tom Lane wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > My proposal is to set includedir=${prefix}/include/postgresql (instead of
> > ${prefix}/include) in such cases where the prefix is shared, i.e., it does
> > not contain something like "pgsql" already. (precise pattern t.b.d.)
 
> Hmm, so basically you propose an install setup whereby 'bin' and 'lib'
> files can go directly into /usr/local/bin and /usr/local/lib, but
> everything else still lives in postgres-specific directories?

Peter: Please do this....
 
> To do that without creating problems, we'd have to go back to making
> sure that all the programs we install have 'pg'-prefixed names.  The
> scripts (createdb and so forth) don't at the moment, and names like
> 'createuser' clearly have potential for confusion if they are in non-
> PG-specific directories.

RedHat includes PostgreSQL, with executables in /usr/bin.  There have
been no namespace collisions as yet, with as many packages as RedHat
ships.
 
> I think it would be a real bad idea to put the postmaster and postgres
> executables right in /usr/local/bin.  Perhaps it is time to think about
> a separate 'sbin' directory for programs that aren't supposed to be
> invoked by normal users.  Those two, initdb, initlocation, and ipcclean

This is doable, but not really necessary.  However, if this is the
direction things are going..... I can certainly work with it.  In fact,
I may go ahead with 7.1's RPMset and do that, popping those executables
in /usr/sbin -- not a big change, by any means, except to the scripts
that are bundled with the RPM.

A good, usable, shared prefix would make my job much easier.  Great gobs
of code in the spec file would go away as PostgreSQL loses the
'/usr/local/pgsql'-centric thinking and gets more in the step of what is
standard for packaging.  And this would help even on system other than
Linux FHS-compliant distributions.  And it would not cause any problems
for those who still want to use a prefix of /usr/local/pgsql.

Thanks for the thinking, Peter.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Thu, 28 Sep 2000 12:44:54 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Installation layout is still hazardous for shared prefixes
Lamar Owen wrote:
  >Tom Lane wrote:
  >> To do that without creating problems, we'd have to go back to making
  >> sure that all the programs we install have 'pg'-prefixed names.  The
  >> scripts (createdb and so forth) don't at the moment, and names like
  >> 'createuser' clearly have potential for confusion if they are in non-
  >> PG-specific directories.
  >
  >RedHat includes PostgreSQL, with executables in /usr/bin.  There have
  >been no namespace collisions as yet, with as many packages as RedHat
  >ships.
 
The same applies to Debian, with something like 4000 binary packages in
the current development tree.

  >> I think it would be a real bad idea to put the postmaster and postgres
  >> executables right in /usr/local/bin.  Perhaps it is time to think about
  >> a separate 'sbin' directory for programs that aren't supposed to be
  >> invoked by normal users.  Those two, initdb, initlocation, and ipcclean
  >
  >This is doable, but not really necessary.  However, if this is the
  >direction things are going..... I can certainly work with it.  In fact,
  >I may go ahead with 7.1's RPMset and do that, popping those executables
  >in /usr/sbin -- not a big change, by any means, except to the scripts
  >that are bundled with the RPM.
  
In the Debian package, I have put the administrator programs in
/usr/lib/postgresql/bin.  The postgres user has that directory in its path
so that all works properly.  Since root cannot run these, I don't think it
appropriate to put them in /usr/sbin.

  >A good, usable, shared prefix would make my job much easier.  Great gobs
  >of code in the spec file would go away as PostgreSQL loses the
  >'/usr/local/pgsql'-centric thinking and gets more in the step of what is
  >standard for packaging.  And this would help even on system other than
  >Linux FHS-compliant distributions.  And it would not cause any problems
  >for those who still want to use a prefix of /usr/local/pgsql.

Agreed.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Let not your heart be troubled; ye believe in God, 
      believe also in me."          John 14:1 



Fri, 29 Sep 2000 10:29:48 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Installation layout is still hazardous for shared
 prefixes
"Oliver Elphick" <olly@lfix.co.uk> writes:
>> Tom Lane wrote:
>>> I think it would be a real bad idea to put the postmaster and postgres
>>> executables right in /usr/local/bin.  Perhaps it is time to think about
>>> a separate 'sbin' directory for programs that aren't supposed to be
>>> invoked by normal users.  Those two, initdb, initlocation, and ipcclean
  
> In the Debian package, I have put the administrator programs in
> /usr/lib/postgresql/bin.  The postgres user has that directory in its path
> so that all works properly.  Since root cannot run these, I don't think it
> appropriate to put them in /usr/sbin.

That seems like a good compromise to me.  From a more general
perspective I guess that install would see it as two separate target
directories for executables, which I suppose we'd describe as "user"
and "dbadmin" bin directories.  When installing one should have a
choice of a "traditional" setup (both user and admin exes go into
a single directory, eg /usr/local/pgsql/bin) or a "shared" setup
(user exes to a shared dir like /usr/local/bin, admin exes still go
to a private dir like /usr/local/pgsql/bin).

Offhand I'd make the division be user:

createdb dropdb ecpg pg-config pgaccess pgtclsh pgtksh psql

and admin:

createlang createuser droplang dropuser initdb initlocation ipcclean
pg_ctl pg_dump pg_dumpall pg_id pg_passwd pg_restore pg_upgrade postgres
postmaster vacuumdb

(Not sure about pg_dump/pg_dumpall/pg_restore; are these of any
significant use to non-superusers?)  This would keep createuser/dropuser
out of the shared bin directory, which certainly seem like the names
most likely to cause conflicts.

The man pages probably need to adopt the same division as the exes,
ie some to /usr/local/man and some to /usr/local/pgsql/man.

Note that it'd be a real bad idea to abandon the option of the
"traditional" install-tree configuration.  For people like me, with
three or four versions of Postgres hanging around on the same machine,
it's critical to be able to install everything into a single private
directory tree.

Comments?

			regards, tom lane

Fri, 29 Sep 2000 10:15:27 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Installation layout is still hazardous for shared prefixes 
Tom Lane wrote:

[useful an complete discussion of sbin-style programs and their place
snipped]

> (Not sure about pg_dump/pg_dumpall/pg_restore; are these of any
> significant use to non-superusers?)  This would keep createuser/dropuser
> out of the shared bin directory, which certainly seem like the names
> most likely to cause conflicts.

pg_dump, yes, as a user might want to dump his own database.
 
> The man pages probably need to adopt the same division as the exes,
> ie some to /usr/local/man and some to /usr/local/pgsql/man.

Currently, since there is no collision in the executables there have
been no collisions in the man pages.  But, I had a radical idea about
the man pages -- why not package a 'man database' as a dump, let someone
restore that dump into a database, then you can use SQL to access your
man pages.  Of course, you still need docs outside the database, but,
with TOAST, this is possible.

Comments?
 
> Note that it'd be a real bad idea to abandon the option of the
> "traditional" install-tree configuration.  For people like me, with
> three or four versions of Postgres hanging around on the same machine,
> it's critical to be able to install everything into a single private
> directory tree.

No one is advocating removing the 'traditional' packaging from the
options -- least of all me.  Choice and flexibility are my bywords. 
Currently, the PostgreSQL installation is very inflexible WRT the
directories under the installation dir.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Fri, 29 Sep 2000 10:47:33 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Installation layout is still hazardous for shared prefixes
Tom Lane writes:

> > In the Debian package, I have put the administrator programs in
> > /usr/lib/postgresql/bin.  The postgres user has that directory in its path
> > so that all works properly.  Since root cannot run these, I don't think it
> > appropriate to put them in /usr/sbin.

That's okay as far as Debian goes, because users do not *ever* have to run
initdb, postmaster, etc. because that's done by the package tool and/or
the rc.* stuff, but for a PostgreSQL source distribution (which is what
we're talking about here, binary packagers can always make their own
arrangements) initdb and postmaster do get run by the user so they have to
be in some kind of obvious place.

> choice of a "traditional" setup (both user and admin exes go into
> a single directory, eg /usr/local/pgsql/bin) or a "shared" setup
> (user exes to a shared dir like /usr/local/bin, admin exes still go
> to a private dir like /usr/local/pgsql/bin).

Um, now that would go against the "meta" file system standard that
Autoconf gives us, and I think it's very important to adhere by that
because users expect a configure script to behave in certain ways.

An installation directory looks like PREFIX/SUBTREE/OPT-PRIVATE, where
prefix would be /usr/local/pgsql or /usr/local, subtree is bin, man,
include, etc., and OPT-PRIVATE would be "postgresql".  But you can only
have one prefix, if it's /usr/local then you can't use /usr/local/pgsql
for some other part.

What we could compatibly do is

/usr/local/bin/psql
/usr/local/sbin/initdb

or 

/usr/local/bin/psql
/usr/local/sbin/postgresql/initdb

but the latter is going to be pretty annoying because I've never seen
somebody make subdirectories in bin or sbin.

 
> Offhand I'd make the division be user:
> 
> createdb dropdb ecpg pg-config pgaccess pgtclsh pgtksh psql
> 
> and admin:
> 
> createlang createuser droplang dropuser initdb initlocation ipcclean
> pg_ctl pg_dump pg_dumpall pg_id pg_passwd pg_restore pg_upgrade postgres
> postmaster vacuumdb

Hmm, I'd rather see createuser, dropuser, vacuumdb, pg_dump, pg_restore in
the first category because that's the client/server split -- the second
category only needs to be installed on the server, the first on clients
and servers.  That might be more useful in more ways and does not make too
many presumptions about usage pattern.

> This would keep createuser/dropuser out of the shared bin directory,
> which certainly seem like the names most likely to cause conflicts.

I'm not so concerned about that anymore, considering that RPM and Debian
are not concerned.  Hiding away executables is probably not the terminally
elegant plan, because users will put them in their PATH anyway before
long.  If someone else can show some other package having a createuser
program, then we need to act, but probably rather by renaming ours.


> Note that it'd be a real bad idea to abandon the option of the
> "traditional" install-tree configuration.  For people like me, with
> three or four versions of Postgres hanging around on the same machine,
> it's critical to be able to install everything into a single private
> directory tree.

That's definitely not going to happen of course.  To make clear what I
want, it's this:

--prefix=/usr/local/pgsql (default)

/usr/local/pgsql/bin/psql ...
/usr/local/pgsql/lib/libpq.a ...
/usr/local/pgsql/include/libpq-fe.h ...
/usr/local/pgsql/share/template1.bki ...

--prefix=/usr/local	!~ /(postgres)|(pgsql)/

/usr/local/bin/psql ...
/usr/local/lib/libpq.a ...
/usr/local/include/postgresql/libpq-fe.h ...
/usr/local/share/postgresql/template1.bki ...

Whether or not we want to have a separate sbin tree is independent of
that, but let's not invent a completely new file system standard for that.


-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Fri, 29 Sep 2000 18:24:16 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Installation layout is still hazardous for shared
 prefixes
Can someone comment on this?

[ Charset ISO-8859-1 unsupported, converting... ]
> Louis-David Mitterrand writes:
> 
> > When creating a child (through CREATE TABLE ... INHERIT (parent)) it
> > seems the child gets all of the parent's contraints _except_ its PRIMARY
> > KEY. Is this normal?
> 
> It's kind of a bug.
> 
> 
> -- 
> Peter Eisentraut                  Sernanders v?g 10:115
> peter_e@gmx.net                   75262 Uppsala
> http://yi.org/peter-e/            Sweden
> 
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 2 Oct 2000 12:29:34 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: child table doesn't inherit PRIMARY KEY?
I wrote:

> My proposal is to set includedir=${prefix}/include/postgresql (instead of
> ${prefix}/include) in such cases where the prefix is shared, i.e., it does
> not contain something like "pgsql" already. (precise pattern t.b.d.)  

I think that most people agreed to this step, or at least no one
explicitly disagreed.  Could we move ahead with this?

The default prefix will stay at /usr/local/pgsql so if you are concerned
about createuser, etc. clashing, you don't have to use it.  The decision
about sbin or no sbin can be considered separately.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Wed, 4 Oct 2000 22:13:46 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Installation layout is still hazardous for shared prefixes
> I have created the following patch with applies to the current source
> tree.  However, there have been significant changes since 7.0, and even
> my patch does not incorporate all the other changes.
> 
> I am not sure what to recommend.  It would be optimal if you could take
> your patch, grab our current snapshot, and submit a new patch that works
> with our current code.  Another option if you are too busy is to have
> one of our folks massage my patch to more properly merge into the
> existing code. 
> 
> Please let me know how you would like to proceed.  My version of your patch
> is attached.

First of all, thank you for taking the time to work on my submission!
I would prefer if someone who's already familiar with what's changed since
7.0 would make whatever adjustments are needed.  I'm occupied with
other projects at the moment.  If it turns out that no one else has time
to work on it, I'll be the fallback.

Thu, 12 Oct 2000 14:09:03 -0400
"David J. MacKenzie" <djm@web.us.uu.net>
Re: [PATCHES] PostgreSQL virtual hosting support 
> Alfred Perlstein wrote:
> > 
> > I noticed that INHERITS doesn't propogate indecies, It'd be nice
> > if there was an toption to do so.
> 
> Yep it would. Are you volunteering?
> 

Added to TODO:

	* Allow inherited tables to inherit index

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 16 Oct 2000 11:55:14 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: INHERITS doesn't offer enough functionality
* Bruce Momjian <pgman@candle.pha.pa.us> [001016 08:55] wrote:
> > Alfred Perlstein wrote:
> > > 
> > > I noticed that INHERITS doesn't propogate indecies, It'd be nice
> > > if there was an toption to do so.
> > 
> > Yep it would. Are you volunteering?
> > 
> 
> Added to TODO:
> 
> 	* Allow inherited tables to inherit index

Thank you, it's not a big problem that this doesn't happen, but it'd
be nice to see it as an option when creating a table via inheritance.

What about RULEs?  I wouldn't really have a use for that but others
might.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."

Mon, 16 Oct 2000 09:42:23 -0700
Alfred Perlstein <bright@wintelcom.net>
Re: INHERITS doesn't offer enough functionality
Alfred Perlstein wrote:

> Thank you, it's not a big problem that this doesn't happen, but it'd
> be nice to see it as an option when creating a table via inheritance.
> 
> What about RULEs?  I wouldn't really have a use for that but others
> might.

Actually it's a reasonably big deal. Apart from the obvious performance
penalty for a deep inheritance hierarchy it affects the implementation
of unique keys, referential integrity for inheritance and the
transparancy of extending an inheritance hierarchy.

Tue, 17 Oct 2000 21:21:44 +1100
Chris <chris@bitmead.com>
Re: INHERITS doesn't offer enough functionality
Bruce Momjian wrote:
  >> Alfred Perlstein wrote:
  >> > 
  >> > I noticed that INHERITS doesn't propogate indecies, It'd be nice
  >> > if there was an toption to do so.
  >> 
  >> Yep it would. Are you volunteering?
  >> 
  >
  >Added to TODO:
  >
  >	* Allow inherited tables to inherit index

What is the spec for this?  

Do you mean that inheriting tables should share a single index with their
ancestors, or that each descendant should get a separate index on the
same pattern as its ancestors'?  

With the former, the inherited index could be used to enforce a primary
key over a whole inheritance hierarchy, and would presumable make it
easier to implement RI against an inheritance hierarchy.  Is this what
you have in mind?



-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Delight thyself also in the LORD; and he shall give 
      thee the desires of thine heart."          Psalms 37:4



Wed, 18 Oct 2000 12:57:40 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: INHERITS doesn't offer enough functionality 
* Oliver Elphick <olly@lfix.co.uk> [001018 04:59] wrote:
> Bruce Momjian wrote:
>   >> Alfred Perlstein wrote:
>   >> > 
>   >> > I noticed that INHERITS doesn't propogate indecies, It'd be nice
>   >> > if there was an toption to do so.
>   >> 
>   >> Yep it would. Are you volunteering?
>   >> 
>   >
>   >Added to TODO:
>   >
>   >	* Allow inherited tables to inherit index
> 
> What is the spec for this?  
> 
> Do you mean that inheriting tables should share a single index with their
> ancestors, or that each descendant should get a separate index on the
> same pattern as its ancestors'?  
> 
> With the former, the inherited index could be used to enforce a primary
> key over a whole inheritance hierarchy, and would presumable make it
> easier to implement RI against an inheritance hierarchy.  Is this what
> you have in mind?

Not really, it's more of a convience issue for me, a 'derived table'
should inherit the attributes of the 'base table' (including indecies),
having an index shared between two tables is an interesting idea but
not what I had in mind.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."

Wed, 18 Oct 2000 08:40:31 -0700
Alfred Perlstein <bright@wintelcom.net>
Re: INHERITS doesn't offer enough functionality
Alfred Perlstein wrote:
  >* Oliver Elphick <olly@lfix.co.uk> [001018 04:59] wrote:
  >> Do you mean that inheriting tables should share a single index with their
  >> ancestors, or that each descendant should get a separate index on the
  >> same pattern as its ancestors'?  
  >> 
  >> With the former, the inherited index could be used to enforce a primary
  >> key over a whole inheritance hierarchy, and would presumable make it
  >> easier to implement RI against an inheritance hierarchy.  Is this what
  >> you have in mind?
  >
  >Not really, it's more of a convience issue for me, a 'derived table'
  >should inherit the attributes of the 'base table' (including indecies),
  >having an index shared between two tables is an interesting idea but
  >not what I had in mind.

Well then, what will happen if I do

 SELECT * FROM table* WHERE inherited_unique_indexed_field = some_value;

would I expect to get back multiple rows?  Are all the separate indexes
candidates for use in the selection?

I think you are highlighting the fact that we still haven't satisfactorily
defined the semantics of inheritance in PostgreSQL; is it merely a
template system or is it something more meaningful?  What inheritance
specifications are we going to work towards?


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Delight thyself also in the LORD; and he shall give 
      thee the desires of thine heart."          Psalms 37:4



Wed, 18 Oct 2000 20:46:34 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: INHERITS doesn't offer enough functionality 
It's pretty clear to me that an inherited index should be only one
index. There may be a case for optional non-inherited indexes (CREATE
INDEX ON ONLY foobar), but if the index is inherited, it is just one
index.

At the end of the day though, the reason is only performance. The
semantics should be the same no matter whether implemented as multiple
indexes or not. Performance is much better with one index though.(*)

(*) Assuming you use inheritance in the queries, which I have found is
the most common thing. That's reflected in the 7.1 defaults where
inheritance is the default.

Oliver Elphick wrote:
> 
> Alfred Perlstein wrote:
>   >* Oliver Elphick <olly@lfix.co.uk> [001018 04:59] wrote:
>   >> Do you mean that inheriting tables should share a single index with their
>   >> ancestors, or that each descendant should get a separate index on the
>   >> same pattern as its ancestors'?
>   >>
>   >> With the former, the inherited index could be used to enforce a primary
>   >> key over a whole inheritance hierarchy, and would presumable make it
>   >> easier to implement RI against an inheritance hierarchy.  Is this what
>   >> you have in mind?
>   >
>   >Not really, it's more of a convience issue for me, a 'derived table'
>   >should inherit the attributes of the 'base table' (including indecies),
>   >having an index shared between two tables is an interesting idea but
>   >not what I had in mind.
> 
> Well then, what will happen if I do
> 
>  SELECT * FROM table* WHERE inherited_unique_indexed_field = some_value;
> 
> would I expect to get back multiple rows?  Are all the separate indexes
> candidates for use in the selection?
> 
> I think you are highlighting the fact that we still haven't satisfactorily
> defined the semantics of inheritance in PostgreSQL; is it merely a
> template system or is it something more meaningful?  What inheritance
> specifications are we going to work towards?
> 
> --
> Oliver Elphick                                Oliver.Elphick@lfix.co.uk
> Isle of Wight                              http://www.lfix.co.uk/oliver
> PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
> GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
>                  ========================================
>      "Delight thyself also in the LORD; and he shall give
>       thee the desires of thine heart."          Psalms 37:4

-- 
Chris Bitmead
mailto:chris@bitmead.com

Thu, 19 Oct 2000 18:33:24 +1100
Chris <chris@bitmead.com>
Re: INHERITS doesn't offer enough functionality


Chris wrote:

> It's pretty clear to me that an inherited index should be only one
> index. There may be a case for optional non-inherited indexes (CREATE
> INDEX ON ONLY foobar), but if the index is inherited, it is just one
> index.
>
> At the end of the day though, the reason is only performance. The
> semantics should be the same no matter whether implemented as multiple
> indexes or not. Performance is much better with one index though.(*)
>

Is it true ?
How to guarantee the uniqueness using multiple indexes ?

Regards.
Hiroshi Inoue


Thu, 19 Oct 2000 18:38:47 +0900
Hiroshi Inoue <Inoue@tpf.co.jp>
Re: INHERITS doesn't offer enough functionality
Hiroshi Inoue wrote:

> > At the end of the day though, the reason is only performance. The
> > semantics should be the same no matter whether implemented as multiple
> > indexes or not. Performance is much better with one index though.(*)
> >
> 
> Is it true ?
> How to guarantee the uniqueness using multiple indexes ?

Well you'd have to check every index in the hierarchy. As I said,
inefficient.

Thu, 19 Oct 2000 22:58:14 +1100
Chris <chris@bitmead.com>
Re: INHERITS doesn't offer enough functionality


Hiroshi Inoue schrieb:
> 
> Chris wrote:
> 
> > It's pretty clear to me that an inherited index should be only one
> > index. There may be a case for optional non-inherited indexes (CREATE
> > INDEX ON ONLY foobar), but if the index is inherited, it is just one
> > index.
> >
> > At the end of the day though, the reason is only performance. The
> > semantics should be the same no matter whether implemented as multiple
> > indexes or not. Performance is much better with one index though.(*)
> >
> 
> Is it true ?
> How to guarantee the uniqueness using multiple indexes ?
> 

 Sorry to say, but you all should really think about, what inheritance
should mean !!!!

 In the classic mapping strategy (OO-rdbms mapping) it's said, that 
each class is mapped to ONE table ! This is the classic mapping
strategy, which is mentioned in every literature.

 The point is: this is classic, but noone does it like this if
your really have a larger hierarchy of classes. You'll not get 
any good performance, when solving an association in your oo
program, because the framework has to query against each 
table: 6 tables - 6 queries !!! :-(((((

 With the PostgreSQL approach one can send ONE query against
the tables and one would get one result ... which will be
much faster (I hope so ... that has to be prooved ..).

--

 I'm not sure, that inherited indices should be really ONE
index. There are very well reasons NOT to build ONE larger
index.

 Actually one should think about: why do I really want to 
have inheritance in the oo-rdbms ? Actually I could put
all columns (of all classes in this hierarchy into one table 
and that's it). I would like to have inheritance in this
database system, because the tables are getting smaller
and queries against special classes (eh tables) are becoming
faster.

 Actually the inserts will be much faster also because you
have several smaller indices.

 I've run tests here with ONE large table (5 columns and 
5 indices) holding data for about 17 classes and the result 
is: the insert/update path is the problem and not the 
select-path. insert-performance is decreasing in a 
linear fashon ... very, very bad.


 Marten

Thu, 19 Oct 2000 18:48:50 +0100
M.Feldtmann@t-online.de (Marten Feldtmann)
Re: INHERITS doesn't offer enough functionality

>  The point is: this is classic, but noone does it 
> like this if your really have a larger hierarchy of 
> classes. You'll not get any good performance, when 
> solving an association in your oo
> program, because the framework has to query against 
> each table: 6 tables - 6 queries !!! :-(((((
> 
>  With the PostgreSQL approach one can send ONE query 
> against the tables and one would get one result ... 
> which will be much faster (I hope so ... that has to 
> be prooved ..).=

You'll still have to do 6 queries in postgres because it does not return
fields in sub-classes. Imagine the root of the hierarchy is abstract
with no fields. You query this class and you get 100 tuples with no
columns! This is the aspect I'm hoping to fix but I'm waiting for Tom to
re-do the query data structures before I do changes that are thrown
away.

>  Actually one should think about: why do I really want to
> have inheritance in the oo-rdbms ? Actually I could put
> all columns (of all classes in this hierarchy into one table
> and that's it).

Ouch. That way lies madness.

Fri, 20 Oct 2000 18:35:41 +1100
Chris <chris@bitmead.com>
Re: INHERITS doesn't offer enough functionality


Chris schrieb:
> 
> >  The point is: this is classic, but noone does it
> > like this if your really have a larger hierarchy of
> > classes. You'll not get any good performance, when
> > solving an association in your oo
> > program, because the framework has to query against
> > each table: 6 tables - 6 queries !!! :-(((((
> >
> >  With the PostgreSQL approach one can send ONE query
> > against the tables and one would get one result ...
> > which will be much faster (I hope so ... that has to
> > be prooved ..).=
> 
> You'll still have to do 6 queries in postgres because it does not return
> fields in sub-classes. 

 Practically this is not such a big problem as one might think.

 WHEN you have a persistance framework you tell your framework, 
that every attribut is located (mapped or stored or however you 
may see it) in the superclass and then your top class (table)
helds all attributes your "lowest" subclass has.

 But that puts another question to be answered: are the defined
contrained also inheritate ??? Actually I would say: no and
therefore we have the same handling as with indices.

 Most of the attributes may have NULL, but who cares ? The 
framework actually has to interpret the data coming from
the database and will throw him away.

 Therefore I can get around the limitations of PostgreSQL
in this case. If PostgreSQL can handle this in addition
this would be very nice ... but before the basic stuff has
to be fixed and it has to be very solid.

 But I have to admit: my point is a viewpoint from a programmer
using an object oriented language and I only want to store
my objects into a database. People using PHP, pearl or
other "low-level" languages may have a different view or
need, because they do not have a framework doing the work
for them.

 I can only tell you, what will be an improvement for me as
a persistance framework programmer and will not help me.

 What will not help me:

 * that the database generates OID

 * that the database generates "clsss" OID (one may want to
   have that, because to recognize which table the data
   comes from..)

 * special features to solve very special problems

 What will help me:

 * all the stuff to reduce the number (!) of queries send 
   to database to get my data

 * a way to insert VERY quickly a larger amount of data 
   into a table.

 * a good, speedy database
 
Marten

Fri, 20 Oct 2000 19:02:47 +0100
M.Feldtmann@t-online.de (Marten Feldtmann)
Re: INHERITS doesn't offer enough functionality
Marten Feldtmann wrote:

> > You'll still have to do 6 queries in postgres because it does not return
> > fields in sub-classes.
> 
>  Practically this is not such a big problem as one might think.

>  WHEN you have a persistance framework you tell your framework,
> that every attribut is located (mapped or stored or however you
> may see it) in the superclass and then your top class (table)
> helds all attributes your "lowest" subclass has.

I don't understand what you're saying. There is no query which will
bring back a set of objects of different types without truncating the
sub-class fields. Therefore it's a big problem for persistance
frameworks that use inheritance.

>  I can only tell you, what will be an improvement for me as
> a persistance framework programmer and will not help me.
> 
>  What will not help me:
> 
>  * that the database generates OID
> 
>  * that the database generates "clsss" OID (one may want to
>    have that, because to recognize which table the data
>    comes from..)

You don't seem to be thinking much in terms of an Object Data Management
Group style persistence framework. That's a shame since it's becoming
increasingly important. Sun seems to be endorsing it for Java in some
way too.

> 
>  * special features to solve very special problems
> 
>  What will help me:
> 
>  * all the stuff to reduce the number (!) of queries send
>    to database to get my data
> 
>  * a way to insert VERY quickly a larger amount of data
>    into a table.
> 
>  * a good, speedy database
> 
> Marten

Tue, 24 Oct 2000 08:13:28 +1100
Chris <chris@bitmead.com>
Re: INHERITS doesn't offer enough functionality

Tried --enable-multibyte for grins.  Bad Move.  

Top part of config.status:
#! /bin/sh
# Generated automatically by configure.
# Run this file to recreate the current configuration.
# This directory was configured as follows,
# on host lerami.lerctr.org:
#
# ./configure  --prefix=/home/ler/pg-test --enable-syslog --with-CXX --with-perl --enable-multibyte --with-includes=/usr/local/include --with-libs=/usr/local/lib
#
# Compiler output produced by configure, useful for debugging
# configure, is in ./config.log if it exists.


Bottom of make output...
UX:acomp: WARNING: "mbutils.c", line 178: argument #1 incompatible with prototype: strlen()
UX:acomp: WARNING: "mbutils.c", line 195: argument #1 incompatible with prototype: strlen()
cc -c -I/usr/local/include -I../../../../src/include -O -K inline -o wchar.o wchar.c
cc -c -I/usr/local/include -I../../../../src/include -O -K inline -o wstrcmp.o wstrcmp.c
cc -c -I/usr/local/include -I../../../../src/include -O -K inline -o wstrncmp.o wstrncmp.c
cc -c -I/usr/local/include -I../../../../src/include -O -K inline -o big5.o big5.c
/bin/ld -r -o SUBSYS.o common.o conv.o mbutils.o wchar.o wstrcmp.o wstrncmp.o big5.o
gmake[4]: Leaving directory `/home/ler/pg-dev/pgsql/src/backend/utils/mb'
/bin/ld -r -o SUBSYS.o fmgrtab.o adt/SUBSYS.o cache/SUBSYS.o error/SUBSYS.o fmgr/SUBSYS.o hash/SUBSYS.o init/SUBSYS.o misc/SUBSYS.o mmgr/SUBSYS.o sort/SUBSYS.o time/SUBSYS.o mb/SUBSYS.o
gmake[3]: Leaving directory `/home/ler/pg-dev/pgsql/src/backend/utils'
cc -O -K inline -o postgres access/SUBSYS.o bootstrap/SUBSYS.o catalog/SUBSYS.o parser/SUBSYS.o commands/SUBSYS.o executor/SUBSYS.o lib/SUBSYS.o libpq/SUBSYS.o main/SUBSYS.o nodes/SUBSYS.o optimizer/SUBSYS.o port/SUBSYS.o postmaster/SUBSYS.o regex/SUBSYS.o rewrite/SUBSYS.o storage/SUBSYS.o tcop/SUBSYS.o utils/SUBSYS.o -L/usr/local/lib -lz -lgen -lld -lnsl -lsocket -ldl -lm -lreadline -ltermcap -lcurses  -Wl,-Bexport
Undefined			first referenced
symbol  			    in file
reset_client_encoding               tcop/SUBSYS.o
UX:ld: ERROR: Symbol referencing errors. No output written to postgres
gmake[2]: *** [postgres] Error 1
gmake[2]: Leaving directory `/home/ler/pg-dev/pgsql/src/backend'
gmake[1]: *** [all] Error 2
gmake[1]: Leaving directory `/home/ler/pg-dev/pgsql/src'
gmake: *** [all] Error 2

-- 
Larry Rosenman                      http://www.lerctr.org/~ler
Phone: +1 972-414-9812 (voice) Internet: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Thu, 26 Oct 2000 07:07:53 -0500
Larry Rosenman <ler@lerctr.org>
--enable-multibyte dies (UnixWare 7.1.1)/Current Sources
Larry Rosenman <ler@lerctr.org> writes:
> Tried --enable-multibyte for grins.  Bad Move.  

> Undefined			first referenced
> symbol  			    in file
> reset_client_encoding               tcop/SUBSYS.o

Ooops, I guess I broke that yesterday :-(.  Will fix.

			regards, tom lane

Thu, 26 Oct 2000 10:51:35 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: --enable-multibyte dies (UnixWare 7.1.1)/Current Sources 
Tom Lane wrote:
  >Larry Rosenman <ler@lerctr.org> writes:
  >> Tried --enable-multibyte for grins.  Bad Move.  
  >
  >> Undefined			first referenced
  >> symbol  			    in file
  >> reset_client_encoding               tcop/SUBSYS.o
  >
  >Ooops, I guess I broke that yesterday :-(.  Will fix.

It hit me as well (on Linux).

reset_client_encoding is wrongly declared as static in 
src/backend/commands/variable.c

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Whosoever therefore shall be ashamed of me and of my 
      words in this adulterous and sinful generation; of him
      also shall the Son of man be ashamed, when he cometh 
      in the glory of his Father with the holy angels."     
                                 Mark 8:38 



Thu, 26 Oct 2000 22:17:49 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: --enable-multibyte dies (UnixWare 7.1.1)/Current
 Sources
ive got the backend stuff to compile on sco with the sdk had to add
-lsocket 
to get rid of unresolved var gethostbyaddress. made it as far as
compiling 
epcg compiles but fails with unresolved var
nocachegetattr in pgc.o
is this a yacc/lex issue if so what would be min version requirements
for 
bison/flex replacments, easiest to port to sco 5.0.5

-- 
My opinions are my own and not that of my employer even if I am self
employed

Thu, 30 Nov 2000 07:04:34 -0600
"Arno A. Karner" <karner@tnss.com>
compiling pg 7.0.3 on sco 5.0.5
"Arno A. Karner" <karner@tnss.com> writes:
> epcg compiles but fails with unresolved var
> nocachegetattr in pgc.o

This is a header bug (there's a backend header file that some bright
soul put a static function declaration into :-( ... and the function
can't link outside the backend ... and ecpg includes that header,
even though it has no use for the particular function).

I'd suggest trying to remove the #define DISABLE_COMPLEX_MACRO from
port/sco.h.  If it compiles and passes regress tests that way, you're
better off without the #define anyhow.

There was another discussion about this on pghackers just recently...
see the archives.

			regards, tom lane

Thu, 30 Nov 2000 10:15:31 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: compiling pg 7.0.3 on sco 5.0.5 
Tom Lane wrote:
> This is a header bug (there's a backend header file that some bright
> soul put a static function declaration into :-( ... and the function

Actually, it's a static function, not a declaration.  The DISABLE_COMPLEX_MACRO
definition was originally put in to work around a macro size limitation of the 
UnixWare 2.1 C compiler (and later the SCO UDK (Universal Development Kit)).  
If the gnu C compiler is being used it should not be defined.  The function 
used to replace the macro was placed in the header and defined as static so 
that the UnixWare compiler would compile the function in-line where ever it 
was used.

> can't link outside the backend ... and ecpg includes that header,
> even though it has no use for the particular function).
> 
> I'd suggest trying to remove the #define DISABLE_COMPLEX_MACRO from
> port/sco.h.  If it compiles and passes regress tests that way, you're
> better off without the #define anyhow.

-- 
____       | Billy G. Allie    | Domain....: Bill.Allie@mug.org
|  /|      | 7436 Hartwell     | Compuserve: 76337,2061
|-/-|----- | Dearborn, MI 48126| MSN.......: B_G_Allie@email.msn.com
|/  |LLIE  | (313) 582-1540    |
Sun, 03 Dec 2000 02:25:26 -0500
"Billy G. Allie" <bga@mug.org>
Re: compiling pg 7.0.3 on sco 5.0.5 
"Billy G. Allie" <bga@mug.org> writes:
> ... The DISABLE_COMPLEX_MACRO definition was originally put in to work
> around a macro size limitation of the UnixWare 2.1 C compiler (and
> later the SCO UDK (Universal Development Kit)).  If the gnu C compiler
> is being used it should not be defined.

Hm.  Is anyone likely to still be using a version of that compiler that
still has such limitations?

I ask because we recently pulled "#define DISABLE_COMPLEX_MACRO" from
port/sco.h, on the grounds that various people were seeing more harm
than good from it.  But I'm suddenly wondering whether those people
might've been using gcc.  I wonder if

	#ifndef __GNUC__
	#define DISABLE_COMPLEX_MACRO
	#endif

in port/sco.h would be the smart way to go.

			regards, tom lane

Mon, 04 Dec 2000 01:00:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: compiling pg 7.0.3 on sco 5.0.5 
* Tom Lane <tgl@sss.pgh.pa.us> [001204 09:27]:
> "Billy G. Allie" <bga@mug.org> writes:
> > ... The DISABLE_COMPLEX_MACRO definition was originally put in to work
> > around a macro size limitation of the UnixWare 2.1 C compiler (and
> > later the SCO UDK (Universal Development Kit)).  If the gnu C compiler
> > is being used it should not be defined.
> 
> Hm.  Is anyone likely to still be using a version of that compiler that
> still has such limitations?
> 
> I ask because we recently pulled "#define DISABLE_COMPLEX_MACRO" from
> port/sco.h, on the grounds that various people were seeing more harm
> than good from it.  But I'm suddenly wondering whether those people
> might've been using gcc.  I wonder if
> 
> 	#ifndef __GNUC__
> 	#define DISABLE_COMPLEX_MACRO
> 	#endif
> 
> in port/sco.h would be the smart way to go.
Based on my running both CURRENT UDK and GCC on my UnixWare 7 boxes
with CURRENT sources, I think we may need to see if anyone complains. 

LER
> 
> 			regards, tom lane
-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Mon, 4 Dec 2000 09:33:58 -0600
Larry Rosenman <ler@lerctr.org>
Re: compiling pg 7.0.3 on sco 5.0.5
Tom Lane wrote:

> "Billy G. Allie" <bga@mug.org> writes:
> 
>> ... The DISABLE_COMPLEX_MACRO definition was originally put in to work
>> around a macro size limitation of the UnixWare 2.1 C compiler (and
>> later the SCO UDK (Universal Development Kit)).  If the gnu C compiler
>> is being used it should not be defined.
> 
> 
> Hm.  Is anyone likely to still be using a version of that compiler that
> still has such limitations?
> 
> I ask because we recently pulled "#define DISABLE_COMPLEX_MACRO" from
> port/sco.h, on the grounds that various people were seeing more harm
> than good from it.  But I'm suddenly wondering whether those people
> might've been using gcc.  I wonder if
> 
> 	#ifndef __GNUC__
> 	#define DISABLE_COMPLEX_MACRO
> 	#endif
> 
> in port/sco.h would be the smart way to go.
> 
> 			regards, tom lane

Well I recompilied with the stock cc shipped in the SCO development 
package for OpenServer 5. It was released in 97'.


Mon, 04 Dec 2000 10:36:35 -0500
Dave Smith <dave@candata.com>
Re: compiling pg 7.0.3 on sco 5.0.5
Tom Lane writes:

> I ask because we recently pulled "#define DISABLE_COMPLEX_MACRO" from
> port/sco.h, on the grounds that various people were seeing more harm
> than good from it.  But I'm suddenly wondering whether those people
> might've been using gcc.

We can be fairly certain that they weren't, unless GCC started accepting
SCO's compiler flags (or someone altered the compiler flags and filed a
*very* incomplete bug report).

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Mon, 4 Dec 2000 18:31:17 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: compiling pg 7.0.3 on sco 5.0.5 
In the creation below, the column parent_id has ended up with a NOT NULL
constraint that I didn't ask for and don't want.

This is 7.1, updated today from cvs.

=======================================================================

[... other tables created ...]

CREATE TABLE person
(
   ptype        SMALLINT,
   id           CHAR(10),
   name         TEXT            NOT NULL,
   address      INTEGER,
   salutation   TEXT            DEFAULT 'Dear Sir',
   envelope     TEXT,
   email        TEXT,
   www          TEXT,
   CONSTRAINT person_ptype CHECK (ptype >= 0 AND ptype <= 8)
,
PRIMARY KEY (id),
FOREIGN KEY (id, address) REFERENCES address(person, id)
   ON UPDATE CASCADE
   ON DELETE RESTRICT
)
NOTICE:  CREATE TABLE/PRIMARY KEY will create implicit index 'person_pkey' for 
table 'person'
NOTICE:  CREATE TABLE will create implicit trigger(s) for FOREIGN KEY check(s)
CREATE

[... other tables created ...]

CREATE TABLE organisation
(
   contact      CHAR(10)   REFERENCES individual (id)
                                ON UPDATE CASCADE
                                ON DELETE NO ACTION,
   structure    CHAR(1)    CHECK (structure='L' OR
                                  structure='C' OR
                                  structure='U' OR
                                  structure='O'),
   department   TEXT,
   parent_id    CHAR(10),
   CONSTRAINT dept_parent CHECK ((department IS NULL AND parent_id IS NULL) OR
               (department > '' AND parent_id > '')),
   CONSTRAINT organisation_ptype CHECK ((ptype >= 2 AND ptype <=4) OR ptype = 
8)
,
PRIMARY KEY (id),
FOREIGN KEY (id, address) REFERENCES address(person, id)
   ON UPDATE CASCADE
   ON DELETE RESTRICT
)
   INHERITS (person)
NOTICE:  CREATE TABLE/PRIMARY KEY will create implicit index 
'organisation_pkey' for table 'organisation'
NOTICE:  CREATE TABLE will create implicit trigger(s) for FOREIGN KEY check(s)
CREATE

[... skip various COMMENT declarations ...]

\d+ organisation
                              Table "organisation"
 Attribute  |     Type      |      Modifier      |         Description
------------+---------------+--------------------+-----------------------------
-
 ptype      | smallint      |                    |
 id         | character(10) | not null           | Identifier
 name       | text          | not null           | Name
 address    | integer       |                    | Primary address id
 salutation | text          | default 'Dear Sir' | Salutation in a letter
 envelope   | text          |                    | Form of name on envelope
 email      | text          |                    | Email address
 www        | text          |                    | Web URL
 contact    | character(10) |                    | Id of primary contact person
 structure  | character(1)  |                    | Legal structure code
 department | text          |                    | Name of this department
 parent_id  | character(10) | not null           | Parent organisation id
Index: organisation_pkey
Constraints: ((ptype >= 0) AND (ptype <= 8))
             ((((structure = 'L'::bpchar) OR (structure = 'C'::bpchar)) OR 
(structure = 'U'::bpchar)) OR (structure = 'O'::bpchar))
             (((department ISNULL) AND (parent_id ISNULL)) OR (((department > 
''::text) AND (parent_id NOTNULL)) AND (parent_id > ''::bpchar)))
             (((ptype >= 2) AND (ptype <= 4)) OR (ptype = 8))

=======================================================================
-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "The fear of the LORD is the instruction of wisdom, and
      before honour is humility."      Proverbs 15:33 



Fri, 15 Dec 2000 17:53:13 +0000
"Oliver Elphick" <olly@lfix.co.uk>
7.1 (current) unwanted NOT NULL constraint inserted
"Oliver Elphick" wrote:
  >In the creation below, the column parent_id has ended up with a NOT NULL
  >constraint that I didn't ask for and don't want.
 
If I change the order of the columns, the NOT NULL constraint always ends
up on the last column.

This is not the only table to be affected.

Every table that is affected has the last column incorrectly set to
NOT NULL.  Every such table is inherited; however, not every inherited
table is affected.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "The fear of the LORD is the instruction of wisdom, and
      before honour is humility."      Proverbs 15:33 



Fri, 15 Dec 2000 18:43:01 +0000
"Oliver Elphick" <olly@lfix.co.uk>
7.1 (current) unwanted NOT NULL constraint inserted (more)
I can't reproduce this --- I get

Constraints: ((ptype >= 0) AND (ptype <= 8))
             ((((structure = 'L'::bpchar) OR (structure = 'C'::bpchar)) OR (structure = 'U'::bpchar)) OR (structure = 'O'::bpchar))
             (((department ISNULL) AND (parent_id ISNULL)) OR ((department > ''::text) AND (parent_id > ''::bpchar)))
             (((ptype >= 2) AND (ptype <= 4)) OR (ptype = 8))

However, I had to guess about the referenced tables, and possibly I
guessed wrong.  Could you supply their declarations too?

			regards, tom lane

Fri, 15 Dec 2000 15:41:54 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: 7.1 (current) unwanted NOT NULL constraint inserted 
OK, I see the problem.  You have:

CREATE TABLE person (
   id           CHAR(10)
);

CREATE TABLE organisation (
   ...,
PRIMARY KEY (id)
)
   INHERITS (person);

ie, a PRIMARY KEY declaration on an inherited column.  Normally a
PRIMARY KEY declaration causes the key column to become marked
NOT NULL --- but if the key column is an inherited one then the
code misapplies the mark to the last non-inherited column ... or
coredumps if there are no non-inherited columns :-(.  See line 995
in parse/analyze.c.

While it's easy enough to avoid the mis-marking of the last column,
causing the right thing to happen instead is much less easy.  What
we really want is for the key column to be marked NOT NULL,
but during analyze.c there isn't a set of ColumnDefs for the inherited
columns, and so there's no place to put the mark.

Short of a major restructuring of inherited-column creation, I see
no good solution to this.  I see two bad solutions:

1. Require that the referenced column be marked NOT NULL already,
so that the constraint will be inherited properly from the parent.
In other words you couldn't say PRIMARY KEY for an inherited column
unless it is NOT NULL (or a fortiori, PRIMARY KEY) in the parent table.

2. Do nothing, in effect silently dropping the NOT NULL constraint
for such a column.  (Actually we don't have to be silent about it;
we could emit a NOTICE when the parent doesn't have NOT NULL.)

IMHO, #1 is a little less bad, but I'm not firmly committed to it.
Comments anyone?

			regards, tom lane

Fri, 15 Dec 2000 16:56:53 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: 7.1 (current) unwanted NOT NULL constraint inserted 
Tom Lane wrote:
...
  >Short of a major restructuring of inherited-column creation, I see
  >no good solution to this.  I see two bad solutions:
  >
  >1. Require that the referenced column be marked NOT NULL already,
  >so that the constraint will be inherited properly from the parent.
  >In other words you couldn't say PRIMARY KEY for an inherited column
  >unless it is NOT NULL (or a fortiori, PRIMARY KEY) in the parent table.
  >
  >2. Do nothing, in effect silently dropping the NOT NULL constraint
  >for such a column.  (Actually we don't have to be silent about it;
  >we could emit a NOTICE when the parent doesn't have NOT NULL.)
  >
  >IMHO, #1 is a little less bad, but I'm not firmly committed to it.
  >Comments anyone?

In the absence of properly working inheritance, I would vote for 1. (I
am only declaring PRIMARY KEY on the inherited column because that
constraint doesn't get inherited as (I think) it should.)  Option 2 would
give a wrongly-defined table.


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "The fear of the LORD is the instruction of wisdom, and
      before honour is humility."      Proverbs 15:33 



Fri, 15 Dec 2000 23:31:41 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: 7.1 (current) unwanted NOT NULL constraint inserted 
Peter, I thought I saw a recent CVS commit from you stating that you now
can generate the HISTORY file from the SGML source.  Have you tested it?
Is it being done automatically?

Also, I would like to get the release dates into the HISTORY file.  Do
you know an easy way to do that?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 16 Dec 2000 15:15:26 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Generating HISTORY file
Bruce Momjian writes:

> Peter, I thought I saw a recent CVS commit from you stating that you now
> can generate the HISTORY file from the SGML source.  Have you tested it?

Yes.

> Is it being done automatically?

No.

It's basically a semi-simplified way of creating an HTML page with the
release notes and saving it with Netscape.  It was more or less a
by-product of automating the INSTALL build.  It's still nothing to get
excited about as you would still have to manually polish the resulting
text file in some ways.

If you want to try it out run gmake HISTORY in doc/src/sgml.  I've also
tried lynx and w3m for HTML to text conversions but their results where
inferior in several ways.

> Also, I would like to get the release dates into the HISTORY file.  Do
> you know an easy way to do that?

I can't think of anything more particular than just writing it in there
somewhere.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Wed, 20 Dec 2000 00:12:11 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Generating HISTORY file
> Bruce Momjian writes:
> 
> > Peter, I thought I saw a recent CVS commit from you stating that you now
> > can generate the HISTORY file from the SGML source.  Have you tested it?
> 
> Yes.
> 
> > Is it being done automatically?
> 
> No.
> 
> It's basically a semi-simplified way of creating an HTML page with the
> release notes and saving it with Netscape.  It was more or less a
> by-product of automating the INSTALL build.  It's still nothing to get
> excited about as you would still have to manually polish the resulting
> text file in some ways.
> 
> If you want to try it out run gmake HISTORY in doc/src/sgml.  I've also
> tried lynx and w3m for HTML to text conversions but their results where
> inferior in several ways.

OK, same old way I did it long ago.

> 
> > Also, I would like to get the release dates into the HISTORY file.  Do
> > you know an easy way to do that?
> 
> I can't think of anything more particular than just writing it in there
> somewhere.

I have it in the SGML.  I just need it transfered into the HISTORY file.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 19 Dec 2000 22:20:35 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: Generating HISTORY file
> > I can't think of anything more particular than just writing it in there
> > somewhere.
> I have it in the SGML.  I just need it transfered into the HISTORY file.

You might recall Bruce that I've done this with Applixware for the last
several releases...

                      - Thomas

Wed, 20 Dec 2000 07:17:00 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Generating HISTORY file
> > > I can't think of anything more particular than just writing it in there
> > > somewhere.
> > I have it in the SGML.  I just need it transfered into the HISTORY file.
> 
> You might recall Bruce that I've done this with Applixware for the last
> several releases...

So you regenerate the HISTORY file using Applixware.  I thought it
looked poor, and I had to re-do it, or am I remembering something else.


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 20 Dec 2000 12:41:49 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: Generating HISTORY file
> > So you regenerate the HISTORY file using Applixware.  I thought it
> > looked poor, and I had to re-do it, or am I remembering something else.
> 
> *shrug*

I just looked at the CVS logs and I think you fixed Applix to look
better.  Please update HISTORY as part of your documentation changes. 
Thanks.  I added release dates to release.sgml, so it should have the
dates when you are done.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 20 Dec 2000 13:02:10 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: Generating HISTORY file
> So you regenerate the HISTORY file using Applixware.  I thought it
> looked poor, and I had to re-do it, or am I remembering something else.

*shrug*

                   - Thomas

Wed, 20 Dec 2000 18:18:01 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Generating HISTORY file
Some SQL92 functionality is missing from the BIT and VARBIT types.

It should be possible to enter hexadecimal values as:

  B'[<bit>...]'[{<separator>...'[<bit...]'}...]
  X'[<hexdigit>...]'[{<separator>...'[<hexdigit...]'}...]

(Cannan and Otten: SQL - The Standard Handbook, p.38)

but the hexadeximal form is not accepted.

For example:

bray=# \d junk
            Table "junk"
 Attribute |     Type     | Modifier 
-----------+--------------+----------
 id        | character(4) | not null
 flag1     | bit(1)       | 
 flags     | bit(8)       | 
Index: junk_pkey

bray=# insert into junk values ('BBBB',B'0',X'FF');
ERROR:  Attribute 'flags' is of type 'bit' but expression is of type 'int4'
	You will need to rewrite or cast the expression

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "For a child will be born to us, a son will be given to
      us; And the government will rest on His shoulders; And
      His name will be called Wonderful Counsellor, Mighty 
      God, Eternal Father, Prince of Peace." 
                                        Isaiah 9:6 



Thu, 21 Dec 2000 21:33:51 +0000
"Oliver Elphick" <olly@lfix.co.uk>
(7.1) BIT datatype
Oliver Elphick writes:

> Some SQL92 functionality is missing from the BIT and VARBIT types.
>
> It should be possible to enter hexadecimal values as:
>
>   B'[<bit>...]'[{<separator>...'[<bit...]'}...]
>   X'[<hexdigit>...]'[{<separator>...'[<hexdigit...]'}...]
>
> (Cannan and Otten: SQL - The Standard Handbook, p.38)
>
> but the hexadeximal form is not accepted.

This was omitted because in SQL99 the X'1001' notation also serves as a
binary large object value under certain circumstances.  Unfortunately,
it's not exactly known what those circumstances are.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 21 Dec 2000 23:11:25 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: (7.1) BIT datatype
Peter Eisentraut wrote:
  >Oliver Elphick writes:
  >
  >> Some SQL92 functionality is missing from the BIT and VARBIT types.
  >>
  >> It should be possible to enter hexadecimal values as:
...
  >This was omitted because in SQL99 the X'1001' notation also serves as a
  >binary large object value under certain circumstances.  Unfortunately,
  >it's not exactly known what those circumstances are.

It seems a pity that we don't support the SQL92 version at least.
How much more convenient it is to enter X'FE1D' than B'1111111000011101'
(did I get that right?)

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "For a child will be born to us, a son will be given to
      us; And the government will rest on His shoulders; And
      His name will be called Wonderful Counsellor, Mighty 
      God, Eternal Father, Prince of Peace." 
                                        Isaiah 9:6 



Thu, 21 Dec 2000 22:29:45 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: (7.1) BIT datatype 
> Some SQL92 functionality is missing from the BIT and VARBIT types.
>
> It should be possible to enter hexadecimal values as:
>
>   B'[<bit>...]'[{<separator>...'[<bit...]'}...]
>   X'[<hexdigit>...]'[{<separator>...'[<hexdigit...]'}...]
>
> (Cannan and Otten: SQL - The Standard Handbook, p.38)
>
> but the hexadeximal form is not accepted.


I have been using the BIT and VARBIT types in Postgres 7.0.3 (undocumented I
believe), and I note that the _input_ format is as follows:

update blah set flags='b101001';  -- Binary
update blah set flags='xff45';  -- Hex

But the _output_ format (for varbit) is always:

B'1010110'

Has any of this changed in 7.1?

Chris


Fri, 22 Dec 2000 09:18:47 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
RE: (7.1) BIT datatype
"Christopher Kings-Lynne" wrote:
  >I have been using the BIT and VARBIT types in Postgres 7.0.3 (undocumented I
  >believe), and I note that the _input_ format is as follows:
  >
  >update blah set flags='b101001';  -- Binary

That is still accepted.

  >update blah set flags='xff45';  -- Hex
  
That is not.

  >But the _output_ format (for varbit) is always:
  >
  >B'1010110'


bray=# select * from junk;
  id  | flag1 |  flags   | flags2 
------+-------+----------+--------
 AAAA | 1     | 11000101 | 
 BBBB | 0     | 00111010 | 
 cccc | 0     | 01101100 | 11001
 dddd | 0     | 01100000 | 

  >Has any of this changed in 7.1?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "And there were in the same country shepherds abiding 
      in the field, keeping watch over their flock by night.
      And, lo, the angel of the Lord came upon them, and the
      glory of the Lord shone around them; and they were 
      sore afraid. And the angel said unto them, " Fear not;
      for behold I bring you good tidings of great joy which
      shall be to all people. For unto you is born this day 
      in the city of David a Saviour, which is Christ the 
      Lord."        Luke 2:8-11 



Fri, 22 Dec 2000 07:15:28 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: (7.1) BIT datatype 
Hi,
  I saw the thread from a few days ago about Linux/Alpha and 7.1. I
believe I'm seeing the same problems with DEC/Alpha (Tru64Unix 4.0D).

I noticed the following in the postmaster.log, which occurs, as the
Linux/Alpha bug report states, during the misc regression test.

  DEBUG:  copy: line 293, XLogWrite: had to create new log file - you probably should do checkpoints more often
  Server process (pid 24954) exited with status 139 at Fri Dec 22 17:15:48 2000
  Terminating any active server processes...
  Server processes were terminated at Fri Dec 22 17:15:48 2000
  Reinitializing shared memory and semaphores
  DEBUG:  starting up
  DEBUG:  database system was interrupted at 2000-12-22 17:15:47
  DEBUG:  CheckPoint record at (0, 316624)
  DEBUG:  Redo record at (0, 316624); Undo record at (0, 0); Shutdown TRUE

the full src/test/regress/log/postmaster.log can be snagged from
http://www.rcfile.org/postmaster.log

in addition to this, compiling on DEC/Alpha with gcc does not work,
without some shameful hackery :) as __INTERLOCKED_TESTBITSS_QUAD() is 
a builtin that gcc does not know about. The DEC cc builds pg properly.
either way pg is built the test results are much the same, esp the
FAILURE of misc regression test.

If there is anything else I can do to help get this working, please
let me know.

  Brent Verner

Fri, 22 Dec 2000 20:27:10 -0500
Brent Verner <brent@rcfile.org>
7.1 on DEC/Alpha
On 22 Dec 2000 at 20:27 (-0500), Brent Verner wrote:

observation:

  commenting out the queries with 'FROM person* p' causes the misc
  regression test to pass.

    SELECT p.name, p.hobbies.name FROM person* p;

  Brent

| Hi,
|   I saw the thread from a few days ago about Linux/Alpha and 7.1. I
| believe I'm seeing the same problems with DEC/Alpha (Tru64Unix 4.0D).
| 
| I noticed the following in the postmaster.log, which occurs, as the
| Linux/Alpha bug report states, during the misc regression test.
| 
|   DEBUG:  copy: line 293, XLogWrite: had to create new log file - you probably should do checkpoints more often
|   Server process (pid 24954) exited with status 139 at Fri Dec 22 17:15:48 2000
|   Terminating any active server processes...
|   Server processes were terminated at Fri Dec 22 17:15:48 2000
|   Reinitializing shared memory and semaphores
|   DEBUG:  starting up
|   DEBUG:  database system was interrupted at 2000-12-22 17:15:47
|   DEBUG:  CheckPoint record at (0, 316624)
|   DEBUG:  Redo record at (0, 316624); Undo record at (0, 0); Shutdown TRUE
| 
| the full src/test/regress/log/postmaster.log can be snagged from
| http://www.rcfile.org/postmaster.log
| 
| in addition to this, compiling on DEC/Alpha with gcc does not work,
| without some shameful hackery :) as __INTERLOCKED_TESTBITSS_QUAD() is 
| a builtin that gcc does not know about. The DEC cc builds pg properly.
| either way pg is built the test results are much the same, esp the
| FAILURE of misc regression test.
| 
| If there is anything else I can do to help get this working, please
| let me know.
| 
|   Brent Verner

Fri, 22 Dec 2000 21:58:35 -0500
Brent Verner <brent@rcfile.org>
Re: 7.1 on DEC/Alpha
On 22 Dec 2000 at 21:58 (-0500), Brent Verner wrote:
| On 22 Dec 2000 at 20:27 (-0500), Brent Verner wrote:
| 
| observation:
| 
|   commenting out the queries with 'FROM person* p' causes the misc
|   regression test to pass.

that's not what I meant to say. the misc test still FAILS, but it 
no longer causes pg to die.

  b

Fri, 22 Dec 2000 22:02:51 -0500
Brent Verner <brent@rcfile.org>
Re: 7.1 on DEC/Alpha

here's a post-mortem.

#0  0x1200ce58c in ExecEvalFieldSelect (fselect=0x1401615c0, 
    econtext=0x14016a030, isNull=0x14016ab31 "", isDone=0x0) at execQual.c:1096
#1  0x1200ceafc in ExecEvalExpr (expression=0x1401615f0, econtext=0x0, 
    isNull=0x14016ab31 "", isDone=0x0) at execQual.c:1234
#2  0x1200cdd74 in ExecEvalFuncArgs (fcache=0x14016aa70, argList=0x14016a030, 
    econtext=0x14016a030) at execQual.c:603
#3  0x1200cde54 in ExecMakeFunctionResult (fcache=0x14016aa70, 
    arguments=0x1401616d0, econtext=0x14016a030, isNull=0x11fffdf88 "", 
    isDone=0x0) at execQual.c:654
#4  0x1200ce224 in ExecEvalOper (opClause=0x1401615f0, econtext=0x14016a030, 
    isNull=0x11fffdf88 "", isDone=0x0) at execQual.c:841
#5  0x1200cea24 in ExecEvalExpr (expression=0x1401615f0, econtext=0x0, 
    isNull=0x14016ab31 "", isDone=0x0) at execQual.c:1204
#6  0x1200cec54 in ExecQual (qual=0x14016a1a0, econtext=0x14016a030)
    at execQual.c:1356
#7  0x1200cf2a8 in ExecScan (node=0x14016a1d0, accessMtd=0x1200d8320 <SeqNext>)
    at execScan.c:129
#8  0x1200d846c in ExecSeqScan (node=0x1401615f0) at nodeSeqscan.c:138
#9  0x1200cc280 in ExecProcNode (node=0x14016a1d0, parent=0x14016a1d0)
    at execProcnode.c:284
#10 0x1200ca8c0 in ExecutePlan (estate=0x14016a310, plan=0x14016a1d0, 
    numberTuples=1, direction=ForwardScanDirection, destfunc=0x140020c20)
    at execMain.c:959
#11 0x1200c9b50 in ExecutorRun (queryDesc=0x1401615f0, estate=0x14016a310, 
    count=0) at execMain.c:199
#12 0x1200d1140 in postquel_getnext (es=0x140160630) at functions.c:324
#13 0x1200d1300 in postquel_execute (es=0x140160630, fcinfo=0x1401604a0, 
    fcache=0x140160590) at functions.c:417
#14 0x1200d14d8 in fmgr_sql (fcinfo=0x1401604a0) at functions.c:542
#15 0x1200ce09c in ExecMakeFunctionResult (fcache=0x140160480, 
    arguments=0x14015e810, econtext=0x140119cd0, isNull=0x140160350 "", 
    isDone=0x11fffe258) at execQual.c:712
#16 0x1200ce2c4 in ExecEvalFunc (funcClause=0x1401615f0, econtext=0x140119cd0, 
    isNull=0x140160350 "", isDone=0x11fffe258) at execQual.c:883
#17 0x1200cea3c in ExecEvalExpr (expression=0x1401615f0, econtext=0x0, 
    isNull=0x14016ab31 "", isDone=0x0) at execQual.c:1208
#18 0x1200c8e10 in ExecEvalIter (iterNode=0x1401615f0, econtext=0x14016a030, 
    isNull=0x1 <Error reading address 0x1: Invalid argument>, isDone=0x0)
    at execFlatten.c:56
#19 0x1200ce9b0 in ExecEvalExpr (expression=0x1401615f0, econtext=0x0, 
    isNull=0x14016ab31 "", isDone=0x0) at execQual.c:1183
#20 0x1200cdd74 in ExecEvalFuncArgs (fcache=0x140160290, argList=0x14016a030, 
    econtext=0x140119cd0) at execQual.c:603
#21 0x1200cde54 in ExecMakeFunctionResult (fcache=0x140160290, 
    arguments=0x14015e840, econtext=0x140119cd0, isNull=0x11fffe3a0 "", 
    isDone=0x11fffe468) at execQual.c:654
#22 0x1200ce2c4 in ExecEvalFunc (funcClause=0x1401615f0, econtext=0x140119cd0, 
    isNull=0x11fffe3a0 "", isDone=0x11fffe468) at execQual.c:883
#23 0x1200cea3c in ExecEvalExpr (expression=0x1401615f0, econtext=0x0, 
    isNull=0x14016ab31 "", isDone=0x0) at execQual.c:1208
#24 0x1200ce574 in ExecEvalFieldSelect (fselect=0x14015e720, 
    econtext=0x14016a030, isNull=0x11fffe3a0 "", isDone=0x0) at execQual.c:1091
#25 0x1200ceafc in ExecEvalExpr (expression=0x1401615f0, econtext=0x0, 
    isNull=0x14016ab31 "", isDone=0x0) at execQual.c:1234
#26 0x1200c8e10 in ExecEvalIter (iterNode=0x1401615f0, econtext=0x14016a030, 
    isNull=0x1 <Error reading address 0x1: Invalid argument>, isDone=0x0)
    at execFlatten.c:56
#27 0x1200ce9b0 in ExecEvalExpr (expression=0x1401615f0, econtext=0x0, 
    isNull=0x14016ab31 "", isDone=0x0) at execQual.c:1183
#28 0x1200ceea4 in ExecTargetList (targetlist=0x14015e870, 
    targettype=0x140160000, values=0x140160260, econtext=0x140119cd0, 
    isDone=0x11fffe5a8) at execQual.c:1528
#29 0x1200cf1a8 in ExecProject (projInfo=0x0, isDone=0x1) at execQual.c:1751
#30 0x1200d8074 in ExecResult (node=0x14015e5b0) at nodeResult.c:167
#31 0x1200cc238 in ExecProcNode (node=0x14015e5b0, parent=0x14015e5b0)
    at execProcnode.c:272
#32 0x1200ca8c0 in ExecutePlan (estate=0x14015eab0, plan=0x14015e5b0, 
    numberTuples=0, direction=ForwardScanDirection, destfunc=0x1401603a0)
    at execMain.c:959
#33 0x1200c9b50 in ExecutorRun (queryDesc=0x1401615f0, estate=0x14015eab0, 
    count=0) at execMain.c:199
#34 0x12013e5c0 in ProcessQuery (parsetree=0x14015ea80, plan=0x140160000)
    at pquery.c:305
#35 0x12013c568 in pg_exec_query_string (
    query_string=0x140115310 "SELECT p.hobbies.equipment.name, p.hobbies.name, p.name FROM person* p;", parse_context=0x1400c5c60) at postgres.c:817
#36 0x12013dd10 in PostgresMain (argv=0x11fffe9a8, real_argv=0x11ffffae8, 
    username=0x1400b72f9 "pgadmin") at postgres.c:1827
#37 0x12011aef0 in DoBackend (port=0x1400b7080) at postmaster.c:2021
#38 0x12011a888 in BackendStartup (port=0x1400b7080) at postmaster.c:1798
#39 0x12011938c in ServerLoop () at postmaster.c:957
#40 0x120118c10 in PostmasterMain (argv=0x11ffffae8) at postmaster.c:664
#41 0x1200e5980 in main (argv=0x11ffffae8) at main.c:138


Sat, 23 Dec 2000 23:49:40 -0500
Brent Verner <brent@rcfile.org>
Re: 7.1 on DEC/Alpha
Brent Verner <brent@rcfile.org> writes:
> here's a post-mortem.

> #0  0x1200ce58c in ExecEvalFieldSelect (fselect=0x1401615c0, 
>     econtext=0x14016a030, isNull=0x14016ab31 "", isDone=0x0) at execQual.c:1096

Looks reasonable as far as it goes.  Evidently the crash is in the
heap_getattr macro call at line 1096 of src/backend/executor/execQual.c.
We need to look at the data structures that macro uses.
What do you get from

p *fselect

p *econtext

p *resSlot->val

p *resSlot->ttc_tupleDescriptor

BTW, if you didn't configure with --enable-cassert, it'd be a good idea
to go back and try it that way...

			regards, tom lane

Sun, 24 Dec 2000 01:00:32 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: 7.1 on DEC/Alpha 
On 24 Dec 2000 at 01:00 (-0500), Tom Lane wrote:
| Brent Verner <brent@rcfile.org> writes:
| > here's a post-mortem.
| 
| > #0  0x1200ce58c in ExecEvalFieldSelect (fselect=0x1401615c0, 
| >     econtext=0x14016a030, isNull=0x14016ab31 "", isDone=0x0) at execQual.c:1096
| 
| Looks reasonable as far as it goes.  Evidently the crash is in the
| heap_getattr macro call at line 1096 of src/backend/executor/execQual.c.
| We need to look at the data structures that macro uses.
| What do you get from
| 
| p *fselect

$1 = {type = T_FieldSelect, arg = 0x140169d40, fieldnum = 1, resulttype = 25, 
  resulttypmod = -1}

| p *econtext

$2 = {type = T_ExprContext, ecxt_scantuple = 0x14016a568, 
  ecxt_innertuple = 0x0, ecxt_outertuple = 0x0, 
  ecxt_per_query_memory = 0x1400c5df0, ecxt_per_tuple_memory = 0x1400c6670, 
  ecxt_param_exec_vals = 0x0, ecxt_param_list_info = 0x140141760, 
  ecxt_aggvalues = 0x0, ecxt_aggnulls = 0x0}

| p *resSlot->val

Error accessing memory address 0x40141838: Invalid argument.
 
| p *resSlot->ttc_tupleDescriptor

Error accessing memory address 0x40141848: Invalid argument.


additionally:

(gdb) p result
$4 = 1075058736

(gdb) p *resSlot
Error accessing memory address 0x40141830: Invalid argument.


| BTW, if you didn't configure with --enable-cassert, it'd be a good idea
| to go back and try it that way...

will reconfig/rebuild shortly.

  brent

Sun, 24 Dec 2000 01:13:19 -0500
Brent Verner <brent@rcfile.org>
Re: Re: 7.1 on DEC/Alpha
Brent Verner <brent@rcfile.org> writes:
> (gdb) p *resSlot
> Error accessing memory address 0x40141830: Invalid argument.

Oooh.  resSlot has been truncated to 32 bits --- judging by the other
nearby pointer values, it almost certainly should have been 0x140141830.
Now we have a lead.

I am guessing that the truncation happened somewhere in
executor/functions.c, but don't see it right away...

			regards, tom lane

Sun, 24 Dec 2000 01:19:34 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: 7.1 on DEC/Alpha 
On 24 Dec 2000 at 00:47 (-0500), Tom Lane wrote:
| 
| > I'll send the patch that allows me to
| > cleanly build with gcc. right now, s_lock.h does the wrong thing
| > when compiling on Alpha/OSF with gcc.
| 
| Roger, we want to build with either.

The attached patch _seems_ to do the right thing. could someone
who knows Alpha assembly check it out (please).

for more info on Alpha assembly, this link may help.
http://tru64unix.compaq.com/faqs/publications/base_doc/DOCUMENTATION/V40D_HTML/APS31DTE/TITLE.HTM

  brent 'who learned too much today'
Sun, 24 Dec 2000 15:14:19 -0500
Brent Verner <brent@rcfile.org>
Re: Re: 7.1 on DEC/Alpha
On 24 Dec 2000 at 01:19 (-0500), Tom Lane wrote:
| Brent Verner <brent@rcfile.org> writes:
| > (gdb) p *resSlot
| > Error accessing memory address 0x40141830: Invalid argument.
| 
| Oooh.  resSlot has been truncated to 32 bits --- judging by the other
| nearby pointer values, it almost certainly should have been 0x140141830.
| Now we have a lead.

FWIW, saying 'set econtext->ecxt_param_list_info->value 0x14014183' in
geb allows the process to not SEGV where it _was_ destined to do so, 
though it does SEGV in a later return to the function. I've tried to
determine where this value is originating, and where it is subsequently
modified, but have not been able to do so. lost in gdb. 

Q: I tried doing 'watch <address>', but this (appeared) to just hang.
  is there some trick to using 'watch' on addresses that I might be
  overlooking?

| I am guessing that the truncation happened somewhere in
| executor/functions.c, but don't see it right away...

more observations WRT sql that blows up postgres on Alpha.

works:
  SELECT p.hobbies.equipment.name, p.hobbies.name, p.name 
    FROM ONLY person p;

breaks:
  SELECT p.hobbies.equipment.name, p.hobbies.name, p.name 
    FROM person p;
  SELECT p.hobbies.equipment.name, p.hobbies.name, p.name 
    FROM person* p;

whatever it is that ONLY causes, avoids the breakage. I've spent the
past two days in a gdb-hole, going in circles. I just think don't know 
enough (about gdb or postgres) to make any further progress. anyway, 
if someone could tell me what difference the ONLY keyword makes WRT
pg internally, it might help me quit running in circles.

thanks.
  brent


Mon, 25 Dec 2000 21:01:06 -0500
Brent Verner <brent@rcfile.org>
Re: 7.1 on DEC/Alpha
Brent Verner <brent@rcfile.org> writes:
> more observations WRT sql that blows up postgres on Alpha.
> works:
>   SELECT p.hobbies.equipment.name, p.hobbies.name, p.name 
>     FROM ONLY person p;
> breaks:
>   SELECT p.hobbies.equipment.name, p.hobbies.name, p.name 
>     FROM person p;
>   SELECT p.hobbies.equipment.name, p.hobbies.name, p.name 
>     FROM person* p;

OK, I see the problem.  The breakage actually is present in 7.0.* and
prior versions as well, it just doesn't happen to be exposed by the
regress tests --- until now.

The trouble is the way that entire-tuple function arguments are handled.
Tuple types are declared in pg_type as being the same size as Oid, ie,
4 bytes.  This reflects situations where a tuple value is represented by
an Oid reference to a row in a table.  (I am not sure whether there is
any code left that depends on that ... in any case I'm nervous about
changing it during beta.)  But the expression evaluator's implementation
of a tuple argument is that the Datum value contains a pointer to a
TupleTableSlot.  This works fine as long as the Datum is just passed
around as a Datum, but if anyone tries to form a tuple containing that
Datum, only 4 bytes get stored into the tuple.  Result: failure on
machines where pointers are wider than 4 bytes.

The reason this shows up in this particular regression test now, and
not before, is that 7.1 does the function evaluations at the top of
the Append plan that implements inheritance union, whereas 7.0 did it
at the bottom.  That means that in 7.1, the TupleTableSlot Datum gets
inserted into a tuple that becomes part of the Append output before
it gets to the function execution.  7.0 would still show the bug
under the right circumstances --- a join would do it, for example.

I think that there may still be cases where an Oid is the correct
representation of a tuple type; anyway I'm afraid to foreclose that
possibility.  What I'm thinking about doing is setting typmod of
an entire-tuple function argument to sizeof(Pointer), rather than
the default -1, to indicate that a pointer representation is being
used.  Comments, hackers?

			regards, tom lane

Tue, 26 Dec 2000 11:26:14 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Tuple-valued datums on Alpha (was Re: 7.1 on DEC/Alpha)
I wrote:
> ... What I'm thinking about doing is setting typmod of
> an entire-tuple function argument to sizeof(Pointer), rather than
> the default -1, to indicate that a pointer representation is being
> used.  Comments, hackers?

Here is a patch to current sources along this line.  I have not
committed it, since I'm not sure it does the job.  It doesn't break
the regress tests on my machine, but does it fix them on Alphas?
Please apply it locally and let me know what you find.

			regards, tom lane


*** src/backend/parser/parse_func.c.orig	Fri Dec 15 14:22:03 2000
--- src/backend/parser/parse_func.c	Tue Dec 26 14:17:09 2000
***************
*** 442,451 ****
  
  			/*
  			 * for func(relname), the param to the function is the tuple
! 			 * under consideration.  we build a special VarNode to reflect
  			 * this -- it has varno set to the correct range table entry,
  			 * but has varattno == 0 to signal that the whole tuple is the
! 			 * argument.
  			 */
  			if (rte->relname == NULL)
  				elog(ERROR,
--- 442,453 ----
  
  			/*
  			 * for func(relname), the param to the function is the tuple
! 			 * under consideration.  We build a special VarNode to reflect
  			 * this -- it has varno set to the correct range table entry,
  			 * but has varattno == 0 to signal that the whole tuple is the
! 			 * argument.  Also, it has typmod set to sizeof(Pointer) to
! 			 * signal that the runtime representation will be a pointer
! 			 * not an Oid.
  			 */
  			if (rte->relname == NULL)
  				elog(ERROR,
***************
*** 453,459 ****
  			toid = typenameTypeId(rte->relname);
  
  			/* replace it in the arg list */
! 			lfirst(i) = makeVar(vnum, 0, toid, -1, sublevels_up);
  		}
  		else if (!attisset)
  			toid = exprType(arg);
--- 455,465 ----
  			toid = typenameTypeId(rte->relname);
  
  			/* replace it in the arg list */
! 			lfirst(i) = makeVar(vnum,
! 								InvalidAttrNumber,
! 								toid,
! 								sizeof(Pointer),
! 								sublevels_up);
  		}
  		else if (!attisset)
  			toid = exprType(arg);
*** src/backend/access/common/tupdesc.c.orig	Thu Nov 16 17:30:15 2000
--- src/backend/access/common/tupdesc.c	Tue Dec 26 14:16:06 2000
***************
*** 352,358 ****
  
  	AssertArg(!PointerIsValid(desc->attrs[attributeNumber - 1]));
  
- 
  	/* ----------------
  	 *	allocate storage for this attribute
  	 * ----------------
--- 352,357 ----
***************
*** 362,368 ****
  	desc->attrs[attributeNumber - 1] = att;
  
  	/* ----------------
! 	 *	initialize some of the attribute fields
  	 * ----------------
  	 */
  	att->attrelid = 0;			/* dummy value */
--- 361,367 ----
  	desc->attrs[attributeNumber - 1] = att;
  
  	/* ----------------
! 	 *	initialize the attribute fields
  	 * ----------------
  	 */
  	att->attrelid = 0;			/* dummy value */
***************
*** 372,378 ****
  	else
  		MemSet(NameStr(att->attname), 0, NAMEDATALEN);
  
- 
  	att->attdispersion = 0;		/* dummy value */
  	att->attcacheoff = -1;
  	att->atttypmod = typmod;
--- 371,376 ----
***************
*** 414,421 ****
  		att->atttypid = InvalidOid;
  		att->attlen = (int16) 0;
  		att->attbyval = (bool) 0;
- 		att->attstorage = 'p';
  		att->attalign = 'i';
  		return false;
  	}
  
--- 412,419 ----
  		att->atttypid = InvalidOid;
  		att->attlen = (int16) 0;
  		att->attbyval = (bool) 0;
  		att->attalign = 'i';
+ 		att->attstorage = 'p';
  		return false;
  	}
  
***************
*** 427,468 ****
  	typeForm = (Form_pg_type) GETSTRUCT(tuple);
  
  	att->atttypid = tuple->t_data->t_oid;
- 	att->attalign = typeForm->typalign;
  
! 	/* ------------------------
! 	   If this attribute is a set, what is really stored in the
! 	   attribute is the OID of a tuple in the pg_proc catalog.
! 	   The pg_proc tuple contains the query string which defines
! 	   this set - i.e., the query to run to get the set.
! 	   So the atttypid (just assigned above) refers to the type returned
! 	   by this query, but the actual length of this attribute is the
! 	   length (size) of an OID.
! 
! 	   Why not just make the atttypid point to the OID type, instead
! 	   of the type the query returns?  Because the executor uses the atttypid
! 	   to tell the front end what type will be returned (in BeginCommand),
! 	   and in the end the type returned will be the result of the query, not
! 	   an OID.
! 
! 	   Why not wait until the return type of the set is known (i.e., the
! 	   recursive call to the executor to execute the set has returned)
! 	   before telling the front end what the return type will be?  Because
! 	   the executor is a delicate thing, and making sure that the correct
! 	   order of front-end commands is maintained is messy, especially
! 	   considering that target lists may change as inherited attributes
! 	   are considered, etc.  Ugh.
! 	   -----------------------------------------
! 	   */
  	if (attisset)
  	{
  		att->attlen = sizeof(Oid);
  		att->attbyval = true;
  		att->attstorage = 'p';
  	}
  	else
  	{
  		att->attlen = typeForm->typlen;
  		att->attbyval = typeForm->typbyval;
  		att->attstorage = typeForm->typstorage;
  	}
  
--- 425,487 ----
  	typeForm = (Form_pg_type) GETSTRUCT(tuple);
  
  	att->atttypid = tuple->t_data->t_oid;
  
! 	/*------------------------
! 	 * There are a couple of cases where we must override the information
! 	 * stored in pg_type.
! 	 *
! 	 * First: if this attribute is a set, what is really stored in the
! 	 * attribute is the OID of a tuple in the pg_proc catalog.
! 	 * The pg_proc tuple contains the query string which defines
! 	 * this set - i.e., the query to run to get the set.
! 	 * So the atttypid (just assigned above) refers to the type returned
! 	 * by this query, but the actual length of this attribute is the
! 	 * length (size) of an OID.
! 	 *
! 	 * (Why not just make the atttypid point to the OID type, instead
! 	 * of the type the query returns?  Because the executor uses the atttypid
! 	 * to tell the front end what type will be returned (in BeginCommand),
! 	 * and in the end the type returned will be the result of the query, not
! 	 * an OID.)
! 	 *
! 	 * (Why not wait until the return type of the set is known (i.e., the
! 	 * recursive call to the executor to execute the set has returned)
! 	 * before telling the front end what the return type will be?  Because
! 	 * the executor is a delicate thing, and making sure that the correct
! 	 * order of front-end commands is maintained is messy, especially
! 	 * considering that target lists may change as inherited attributes
! 	 * are considered, etc.  Ugh.)
! 	 *
! 	 * Second: if we are dealing with a complex type (a tuple type), then
! 	 * pg_type will say that the representation is the same as Oid.  But
! 	 * if typmod is sizeof(Pointer) then the internal representation is
! 	 * actually a pointer to a TupleTableSlot, and we have to substitute
! 	 * that information.
! 	 *
! 	 * A set of complex type is first and foremost a set, so its
! 	 * representation is Oid not pointer.  So, test that case first.
! 	 *-----------------------------------------
! 	 */
  	if (attisset)
  	{
  		att->attlen = sizeof(Oid);
  		att->attbyval = true;
+ 		att->attalign = 'i';
+ 		att->attstorage = 'p';
+ 	}
+ 	else if (typeForm->typtype == 'c' && typmod == sizeof(Pointer))
+ 	{
+ 		att->attlen = sizeof(Pointer);
+ 		att->attbyval = true;
+ 		att->attalign = 'd';	/* kluge to work with 8-byte pointers */
+ 		/* XXX ought to have a separate attalign value for pointers ... */
  		att->attstorage = 'p';
  	}
  	else
  	{
  		att->attlen = typeForm->typlen;
  		att->attbyval = typeForm->typbyval;
+ 		att->attalign = typeForm->typalign;
  		att->attstorage = typeForm->typstorage;
  	}
  
*** src/backend/executor/execTuples.c.orig	Sat Nov 11 19:36:57 2000
--- src/backend/executor/execTuples.c	Tue Dec 26 14:17:30 2000
***************
*** 835,841 ****
  	return tupType;
  }
  
! /*
  TupleDesc
  ExecCopyTupType(TupleDesc td, int natts)
  {
--- 835,841 ----
  	return tupType;
  }
  
! #ifdef NOT_USED
  TupleDesc
  ExecCopyTupType(TupleDesc td, int natts)
  {
***************
*** 852,881 ****
  		}
  	return newTd;
  }
! */
  
  /* ----------------------------------------------------------------
   *		ExecTypeFromTL
   *
   *		Currently there are about 4 different places where we create
   *		TupleDescriptors.  They should all be merged, or perhaps
   *		be rewritten to call BuildDesc().
-  *
-  *	old comments
-  *		Forms attribute type info from the target list in the node.
-  *		It assumes all domains are individually specified in the target list.
-  *		It fails if the target list contains something like Emp.all
-  *		which represents all the attributes from EMP relation.
-  *
-  *		Conditions:
-  *			The inner and outer subtrees should be initialized because it
-  *			might be necessary to know the type infos of the subtrees.
   * ----------------------------------------------------------------
   */
  TupleDesc
  ExecTypeFromTL(List *targetList)
  {
! 	List	   *tlcdr;
  	TupleDesc	typeInfo;
  	Resdom	   *resdom;
  	Oid			restype;
--- 852,874 ----
  		}
  	return newTd;
  }
! #endif
  
  /* ----------------------------------------------------------------
   *		ExecTypeFromTL
   *
+  *		Generate a tuple descriptor for the result tuple of a targetlist.
+  *		Note that resjunk columns, if any, are included in the result.
+  *
   *		Currently there are about 4 different places where we create
   *		TupleDescriptors.  They should all be merged, or perhaps
   *		be rewritten to call BuildDesc().
   * ----------------------------------------------------------------
   */
  TupleDesc
  ExecTypeFromTL(List *targetList)
  {
! 	List	   *tlitem;
  	TupleDesc	typeInfo;
  	Resdom	   *resdom;
  	Oid			restype;
***************
*** 897,910 ****
  	typeInfo = CreateTemplateTupleDesc(len);
  
  	/* ----------------
! 	 * notes: get resdom from (resdom expr)
! 	 *		  get_typbyval comes from src/lib/l-lisp/lsyscache.c
  	 * ----------------
  	 */
! 	tlcdr = targetList;
! 	while (tlcdr != NIL)
  	{
! 		TargetEntry *tle = lfirst(tlcdr);
  
  		if (tle->resdom != NULL)
  		{
--- 890,901 ----
  	typeInfo = CreateTemplateTupleDesc(len);
  
  	/* ----------------
! 	 * scan list, generate type info for each entry
  	 * ----------------
  	 */
! 	foreach(tlitem, targetList)
  	{
! 		TargetEntry *tle = lfirst(tlitem);
  
  		if (tle->resdom != NULL)
  		{
***************
*** 920,926 ****
  							   0,
  							   false);
  
! /*
  			ExecSetTypeInfo(resdom->resno - 1,
  							typeInfo,
  							(Oid) restype,
--- 911,917 ----
  							   0,
  							   false);
  
! #ifdef NOT_USED
  			ExecSetTypeInfo(resdom->resno - 1,
  							typeInfo,
  							(Oid) restype,
***************
*** 929,941 ****
  							NameStr(*resdom->resname),
  							get_typbyval(restype),
  							get_typalign(restype));
! */
  		}
  		else
  		{
  			Resdom	   *fjRes;
  			List	   *fjTlistP;
! 			List	   *fjList = lfirst(tlcdr);
  
  #ifdef SETS_FIXED
  			TargetEntry *tle;
--- 920,933 ----
  							NameStr(*resdom->resname),
  							get_typbyval(restype),
  							get_typalign(restype));
! #endif
  		}
  		else
  		{
+ 			/* XXX this branch looks fairly broken ... tgl 12/2000 */
  			Resdom	   *fjRes;
  			List	   *fjTlistP;
! 			List	   *fjList = lfirst(tlitem);
  
  #ifdef SETS_FIXED
  			TargetEntry *tle;
***************
*** 953,959 ****
  							   fjRes->restypmod,
  							   0,
  							   false);
! /*
  			ExecSetTypeInfo(fjRes->resno - 1,
  							typeInfo,
  							(Oid) restype,
--- 945,951 ----
  							   fjRes->restypmod,
  							   0,
  							   false);
! #ifdef NOT_USED
  			ExecSetTypeInfo(fjRes->resno - 1,
  							typeInfo,
  							(Oid) restype,
***************
*** 962,968 ****
  							(char *) fjRes->resname,
  							get_typbyval(restype),
  							get_typalign(restype));
! */
  
  			foreach(fjTlistP, lnext(fjList))
  			{
--- 954,960 ----
  							(char *) fjRes->resname,
  							get_typbyval(restype),
  							get_typalign(restype));
! #endif
  
  			foreach(fjTlistP, lnext(fjList))
  			{
***************
*** 978,984 ****
  								   0,
  								   false);
  
! /*
  				ExecSetTypeInfo(fjRes->resno - 1,
  								typeInfo,
  								(Oid) fjRes->restype,
--- 970,976 ----
  								   0,
  								   false);
  
! #ifdef NOT_USED
  				ExecSetTypeInfo(fjRes->resno - 1,
  								typeInfo,
  								(Oid) fjRes->restype,
***************
*** 987,997 ****
  								(char *) fjRes->resname,
  								get_typbyval(fjRes->restype),
  								get_typalign(fjRes->restype));
! */
  			}
  		}
- 
- 		tlcdr = lnext(tlcdr);
  	}
  
  	return typeInfo;
--- 979,987 ----
  								(char *) fjRes->resname,
  								get_typbyval(fjRes->restype),
  								get_typalign(fjRes->restype));
! #endif
  			}
  		}
  	}
  
  	return typeInfo;
*** src/backend/executor/execUtils.c.orig	Thu Nov 16 17:30:20 2000
--- src/backend/executor/execUtils.c	Tue Dec 26 14:17:26 2000
***************
*** 274,289 ****
  {
  	List	   *targetList;
  	TupleDesc	tupDesc;
- 	int			len;
  
  	targetList = node->targetlist;
  	tupDesc = ExecTypeFromTL(targetList);
! 	len = ExecTargetListLength(targetList);
! 
! 	if (len > 0)
! 		ExecAssignResultType(commonstate, tupDesc);
! 	else
! 		ExecAssignResultType(commonstate, (TupleDesc) NULL);
  }
  
  /* ----------------
--- 274,283 ----
  {
  	List	   *targetList;
  	TupleDesc	tupDesc;
  
  	targetList = node->targetlist;
  	tupDesc = ExecTypeFromTL(targetList);
! 	ExecAssignResultType(commonstate, tupDesc);
  }
  
  /* ----------------
***************
*** 582,589 ****
  }
  
  /* ----------------
!  *		ExecFreeTypeInfo frees the array of attrbutes
!  *		created by ExecMakeTypeInfo and returned by ExecTypeFromTL...
   * ----------------
   */
  void
--- 576,583 ----
  }
  
  /* ----------------
!  *		ExecFreeTypeInfo frees the array of attributes
!  *		created by ExecMakeTypeInfo and returned by ExecTypeFromTL
   * ----------------
   */
  void
Tue, 26 Dec 2000 14:41:33 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Tuple-valued datums on Alpha (was Re: 7.1 on DEC/Alpha) 
On 26 Dec 2000 at 14:41 (-0500), Tom Lane wrote:
| I wrote:
| > ... What I'm thinking about doing is setting typmod of
| > an entire-tuple function argument to sizeof(Pointer), rather than
| > the default -1, to indicate that a pointer representation is being
| > used.  Comments, hackers?
| 
| Here is a patch to current sources along this line.  I have not
| committed it, since I'm not sure it does the job.  It doesn't break
| the regress tests on my machine, but does it fix them on Alphas?
| Please apply it locally and let me know what you find.

results _look_ the same from 'make check'. I'm gonna get back into
the debugger on this (I've learned a few tricks that I didn't know
when last I gdb'd on the Alpha).

  brent

Tue, 26 Dec 2000 15:47:37 -0500
Brent Verner <brent@rcfile.org>
Re: Tuple-valued datums on Alpha (was Re: 7.1 on DEC/Alpha)
On 26 Dec 2000 at 14:41 (-0500), Tom Lane wrote:
| I wrote:
| > ... What I'm thinking about doing is setting typmod of
| > an entire-tuple function argument to sizeof(Pointer), rather than
| > the default -1, to indicate that a pointer representation is being
| > used.  Comments, hackers?
| 
| Here is a patch to current sources along this line.  I have not
| committed it, since I'm not sure it does the job.  It doesn't break
| the regress tests on my machine, but does it fix them on Alphas?
| Please apply it locally and let me know what you find.

what I'm seeing now is much the same. FWIW, it looks like we're picking
up the cruft around 
  
  functions.c:354    paramLI->value = fcinfo->arg[paramLI->id - 1];

(both of which are type Datum)

i've been in circles trying to figure out where fcinfo->arg is filled.
can you point me toward that?

thanks for your help.
  brent

Tue, 26 Dec 2000 18:03:42 -0500
Brent Verner <brent@rcfile.org>
Re: Tuple-valued datums on Alpha (was Re: 7.1 on DEC/Alpha)
Brent Verner <brent@rcfile.org> writes:
> | Please apply it locally and let me know what you find.

> what I'm seeing now is much the same.

Drat.  More to do, then.

> i've been in circles trying to figure out where fcinfo->arg is filled.
> can you point me toward that?

See src/backend/utils/fmgr/README and src/backend/utils/fmgr/fmgr.c.
But fmgr is probably only the carrier of disease, not the source...

			regards, tom lane

Tue, 26 Dec 2000 23:41:20 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
On 26 Dec 2000 at 23:41 (-0500), Tom Lane wrote:
| Brent Verner <brent@rcfile.org> writes:
| > | Please apply it locally and let me know what you find.
| 
| > what I'm seeing now is much the same.

sorry, I sent the previous email w/o the details of the different 
behavior. Inside ExecEvalFieldSelect(), result is now 303, instead
of 110599844 (...or whatever is was). I'm not sure if this gives 
you any additional clues.

thanks.
  brent

Wed, 27 Dec 2000 00:28:49 -0500
Brent Verner <brent@rcfile.org>
Re: Re: Tuple-valued datums on Alpha (was Re: 7.1 on DEC/Alpha)
On 26 Dec 2000 at 23:41 (-0500), Tom Lane wrote:
| Brent Verner <brent@rcfile.org> writes:
| > | Please apply it locally and let me know what you find.
| 
| > what I'm seeing now is much the same.
| 
| Drat.  More to do, then.
| 
| > i've been in circles trying to figure out where fcinfo->arg is filled.
| > can you point me toward that?
| 
| See src/backend/utils/fmgr/README and src/backend/utils/fmgr/fmgr.c.
| But fmgr is probably only the carrier of disease, not the source...

ok, I've tracked this further (in the right direction I hope:).

these are the steps leading up the the assignment of the fscked
fcache->fcinfo.arg[i] at execQual.c:603, which is what will eventually
blow up ExecEvalFieldSelect.


Breakpoint 4, ExecMakeFunctionResult (fcache=0x14014e700, 
    arguments=0x14014c850, econtext=0x140127ae0, isNull=0x14014e390 "", 
    isDone=0x11fffde78) at execQual.c:652
652             if (fcache->fcinfo.nargs > 0 && !fcache->argsValid)
(gdb) print fcache->fcinfo
$56 = {flinfo = 0x14014e700, context = 0x0, resultinfo = 0x14014e7d0, 
  isnull = 0 '\000', nargs = 1, arg = {0 <repeats 16 times>}, 
  argnull = '\000' <repeats 15 times>}
(gdb) cont
Breakpoint 6, ExecEvalVar (variable=0x14014c820, econtext=0x140127ae0, 
    isNull=0x14014e7c0 "") at execQual.c:298
298             switch (variable->varno)
(gdb) print *variable
$57 = {type = T_Var, varno = 65001, varattno = 1, vartype = 21220, 
  vartypmod = 8, varlevelsup = 0, varnoold = 1, varoattno = 0}
(gdb) print *econtext
$58 = {type = T_ExprContext, ecxt_scantuple = 0x14014cc58, 
  ecxt_innertuple = 0x0, ecxt_outertuple = 0x14014cc58, 
  ecxt_per_query_memory = 0x1400e6370, ecxt_per_tuple_memory = 0x1400e66a0, 
  ecxt_param_exec_vals = 0x0, ecxt_param_list_info = 0x0, 
  ecxt_aggvalues = 0x0, ecxt_aggnulls = 0x0}
(gdb) break 313
(gdb) cont
(gdb) print *slot
$60 = {type = T_TupleTableSlot, val = 0x14014e430, ttc_shouldFree = 0 '\000', 
  ttc_descIsNew = 1 '\001', ttc_tupleDescriptor = 0x14014ded0, ttc_buffer = 0}
(gdb) break 353
(gdb) cont
(gdb) print *heapTuple
$73 = {t_len = 48, t_self = {ip_blkid = {bi_hi = 65535, bi_lo = 65535}, 
    ip_posid = 0}, t_tableOid = 0, t_datamcxt = 0x1400e6370, 
  t_data = 0x14014e450}
(gdb) print attnum
$74 = 1
(gdb) print *tuple_type
$75 = {natts = 2, attrs = 0x14014df00, constr = 0x0}
(gdb) print isNull
$76 = (bool *) 0x14014e7c0 ""
(gdb) break 359
(gdb) cont
# after heap_getattr, we have the smashed value.
(gdb) print result
$79 = 303


is this nearing the problem, or still simply witnessing symptoms?

  brent 'delirious from sleep dep.'


Wed, 27 Dec 2000 04:06:11 -0500
Brent Verner <brent@rcfile.org>
Re: Re: Tuple-valued datums on Alpha (was Re: 7.1 on DEC/Alpha)
On 26 Dec 2000 at 23:41 (-0500), Tom Lane wrote:
| Brent Verner <brent@rcfile.org> writes:
| > | Please apply it locally and let me know what you find.
| 
| > what I'm seeing now is much the same.
| 
| Drat.  More to do, then.

after hours in the gdb-hole, I see this... maybe a clue? :)

src/include/access/common/heaptuple.c:

450     {
451 
452       /*
453        * Fix me when going to a machine with more than a four-byte
454        * word!
455        */
456       off = att_align(off, att[j]->attlen, att[j]->attalign);
457 
458       att[j]->attcacheoff = off;
459 
460       off = att_addlength(off, att[j]->attlen, tp + off);
461     }

I'm pretty sure I don't know best how to fix this, but I've got some
randomly entered code compiling now :)  If it passes the regression 
tests I'll send it along.

  brent 'glad the coffee shop in the backyard is open now :)'


Wed, 27 Dec 2000 06:29:16 -0500
Brent Verner <brent@rcfile.org>
Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
Christopher Kings-Lynne wrote:
> 
> > Some SQL92 functionality is missing from the BIT and VARBIT types.
> >
> > It should be possible to enter hexadecimal values as:
> >
> >   B'[<bit>...]'[{<separator>...'[<bit...]'}...]
> >   X'[<hexdigit>...]'[{<separator>...'[<hexdigit...]'}...]
> >
> > (Cannan and Otten: SQL - The Standard Handbook, p.38)
> >
> > but the hexadeximal form is not accepted.

As Peter noted: the standard does not say whether X'..' should be a
blob, a bit or a varbit type. Converting it into an integer seems to me
to be the least reasonable solution, albeit the historical one, as
larger bitmasks will not fit. With TOAST the bit type can contain quite
large bit strings, so a case could be made for converting to bit
(especially as the blob implementation has reputedly got some problems). 
> 
> I have been using the BIT and VARBIT types in Postgres 7.0.3 (undocumented I
> believe), and I note that the _input_ format is as follows:
> 
> update blah set flags='b101001';  -- Binary
> update blah set flags='xff45';  -- Hex

Yes, that was done due to limitations in the parser. These have been
fixed and this format should not be used any longer.

> 
> But the _output_ format (for varbit) is always:
> 
> B'1010110'

The SQL standard says nothing about the output of the BIT datatypes. The
C-routines to interpret both the B'..' and X'..' formats, as well as
output routines to generate both are implemented and included. The
problem is that a default had to be chosen, and the B'..' format seemed
more useful for people using small bit masks. 

I don't know whether a function was defined to return an X'..' string of
a bit mask. I don't have one of the more recent Postgres snapshots down
at the moment. Peter E. may know, as he did all the integration.

An alternative may be to add a 'SET variable' to psql to govern the
output format, but there seem to be too many of those already.

Adriaan

Wed, 27 Dec 2000 16:11:30 +0200
Adriaan Joubert <a.joubert@albourne.com>
Re: (7.1) BIT datatype
Brent Verner <brent@rcfile.org> writes:
> after hours in the gdb-hole, I see this... maybe a clue? :)

I don't think that comment means anything.  Possibly it's a leftover
from a time when there was something unportable there.  But if att_align
were broken on Alphas, you'd have a lot worse problems than what you're
seeing.

			regards, tom lane

Wed, 27 Dec 2000 10:08:29 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
Brent Verner <brent@rcfile.org> writes:
> these are the steps leading up the the assignment of the fscked
> fcache->fcinfo.arg[i] at execQual.c:603, which is what will eventually
> blow up ExecEvalFieldSelect.

That looks OK as far as it goes.  Inside ExecEvalVar, you need to look
at the tuple_type data structure in more detail, specifically
	p *tuple_type->attrs[0]
	p *tuple_type->attrs[1]
(I think the leading * is correct here, try omitting it if gdb gets
unhappy.)

> (gdb) print *variable
> $57 = {type = T_Var, varno = 65001, varattno = 1, vartype = 21220, 
>   vartypmod = 8, varlevelsup = 0, varnoold = 1, varoattno = 0}

That part looks promising --- vartypmod is sizeof(Pointer) not -1,
so the front-end part of my patch seems to be working.  What I suspect
we'll find is that the tupledesc doesn't show sizeof the first field to
be 8 the way we want.  Which would imply that I missed a place (or
multiple places :-() that needs to know about the convention for typmod
of a tuple datatype.

			regards, tom lane

Wed, 27 Dec 2000 10:18:45 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Tuple-valued datums on Alpha (was Re: 7.1 on DEC/Alpha) 
After further study, I realized that fetchatt() and a number of other
places were not prepared to cope with 8-byte pass-by-value datatypes.
Most of them weren't checking for cases they couldn't handle, either.

Here is a revised patch for you to try (this includes yesterday's patch
plus more changes, so you'll need to reverse out the prior patch before
applying this one).  NOTE you will need to do a full reconfigure and
rebuild to make this fly --- I'd suggest "make distclean" to start.

			regards, tom lane
Wed, 27 Dec 2000 16:50:10 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
On 27 Dec 2000 at 16:50 (-0500), Tom Lane wrote:
| After further study, I realized that fetchatt() and a number of other
| places were not prepared to cope with 8-byte pass-by-value datatypes.
| Most of them weren't checking for cases they couldn't handle, either.
| 
| Here is a revised patch for you to try (this includes yesterday's patch
| plus more changes, so you'll need to reverse out the prior patch before
| applying this one).  NOTE you will need to do a full reconfigure and
| rebuild to make this fly --- I'd suggest "make distclean" to start.

excellent!

this patch fixes the SEGV problem in the regression tests. the only 
remaining failures, which are not due to SEGV, are:

  oid                  ... FAILED
  float8               ... FAILED
  geometry             ... FAILED

initial comments WRT failures:
  float8    fails only when building with gcc.
  oid       recall seeing one-liner change to correct this. will try.

many thanks,
  brent


Wed, 27 Dec 2000 18:05:55 -0500
Brent Verner <brent@rcfile.org>
Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
Brent Verner <brent@rcfile.org> writes:
> this patch fixes the SEGV problem in the regression tests. the only 
> remaining failures, which are not due to SEGV, are:
>   oid                  ... FAILED
>   float8               ... FAILED
>   geometry             ... FAILED

What are the regression diffs, exactly?

			regards, tom lane

Wed, 27 Dec 2000 18:10:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
On 27 Dec 2000 at 18:10 (-0500), Tom Lane wrote:
| Brent Verner <brent@rcfile.org> writes:
| > this patch fixes the SEGV problem in the regression tests. the only 
| > remaining failures, which are not due to SEGV, are:
| >   oid                  ... FAILED
| >   float8               ... FAILED
| >   geometry             ... FAILED
| 
| What are the regression diffs, exactly?

see attachment.

  brent
Wed, 27 Dec 2000 18:35:30 -0500
Brent Verner <brent@rcfile.org>
Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
Brent Verner <brent@rcfile.org> writes:
>   INSERT INTO OID_TBL(f1) VALUES ('-1040');
> + ERROR:  oidin: error reading "-1040": Error 0 occurred.

Hm.  I thought I'd fixed that.  Are you up to date on
src/backend/utils/adt/oid.c ?  Current CVS has rev 1.42.

  
> *** ./expected/float8-fp-exception.out	Thu Mar 30 02:46:00 2000
> --- ./results/float8.out	Wed Dec 27 18:27:15 2000
> ***************
> *** 214,220 ****
>      SET f1 = FLOAT8_TBL.f1 * '-1'
>      WHERE FLOAT8_TBL.f1 > '0.0';
>   SELECT '' AS bad, f.f1 * '1e200' from FLOAT8_TBL f;
> ! ERROR:  floating point exception! The last floating point operation either exceeded legal ranges or was a divide by zero
>   SELECT '' AS bad, f.f1 ^ '1e200' from FLOAT8_TBL f;
>   ERROR:  pow() result is out of range
>   SELECT '' AS bad, ln(f.f1) from FLOAT8_TBL f where f.f1 = '0.0' ;
> --- 214,220 ----
>      SET f1 = FLOAT8_TBL.f1 * '-1'
>      WHERE FLOAT8_TBL.f1 > '0.0';
>   SELECT '' AS bad, f.f1 * '1e200' from FLOAT8_TBL f;
> ! ERROR:  Bad float8 input format -- overflow
>   SELECT '' AS bad, f.f1 ^ '1e200' from FLOAT8_TBL f;
>   ERROR:  pow() result is out of range
>   SELECT '' AS bad, ln(f.f1) from FLOAT8_TBL f where f.f1 = '0.0' ;

It would appear that Alpha no longer needs the special
float8-fp-exception.out comparison file.  Try removing the line

	float8/alpha.*-dec-osf=float8-fp-exception

from src/test/regress/resultmap.

The geometry diffs also look like Alpha may be more nearly in sync with
the rest of the world than it used to be.  Do any of the other geometry
comparison files match what you are getting as results/geometry.out?

			regards, tom lane

Wed, 27 Dec 2000 18:44:42 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
On 27 Dec 2000 at 18:44 (-0500), Tom Lane wrote:
| Brent Verner <brent@rcfile.org> writes:
| >   INSERT INTO OID_TBL(f1) VALUES ('-1040');
| > + ERROR:  oidin: error reading "-1040": Error 0 occurred.
| 
| Hm.  I thought I'd fixed that.  Are you up to date on
| src/backend/utils/adt/oid.c ?  Current CVS has rev 1.42.

yup. got that version -- 1.42 2000/12/22 21:36:09 tgl

| It would appear that Alpha no longer needs the special
| float8-fp-exception.out comparison file.  Try removing the line
| 
| 	float8/alpha.*-dec-osf=float8-fp-exception

  cc w/o line above:  FAIL
  cc w/ line above:   ok
  gcc w/o line above: ???    (will retest later)
  gcc w/ line above:  FAIL

| The geometry diffs also look like Alpha may be more nearly in sync with
| the rest of the world than it used to be.  Do any of the other geometry
| comparison files match what you are getting as results/geometry.out?

none match.

| 			regards, tom lane

Wed, 27 Dec 2000 19:38:05 -0500
Brent Verner <brent@rcfile.org>
Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
Brent Verner <brent@rcfile.org> writes:
> | float8-fp-exception.out comparison file.  Try removing the line
> | 
> | 	float8/alpha.*-dec-osf=float8-fp-exception

>   cc w/o line above:  FAIL
>   cc w/ line above:   ok
>   gcc w/o line above: ???    (will retest later)
>   gcc w/ line above:  FAIL

OK, then it should work for both cases if you do

float8/alpha.*-dec-osf.*:cc=float8-fp-exception

			regards, tom lane

Wed, 27 Dec 2000 20:02:27 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
Brent Verner <brent@rcfile.org> writes:
> | Hm.  I thought I'd fixed that.  Are you up to date on
> | src/backend/utils/adt/oid.c ?  Current CVS has rev 1.42.

> yup. got that version -- 1.42 2000/12/22 21:36:09 tgl

You're right, it was still broken :-(.  I think I've got it now, though.

Oliver Elphick was kind enough to arrange access to an Alpha running
Debian Linux, and I find that current-as-of-this-moment sources pass
all regression tests in either serial or parallel test mode on that
system.  Curiously, however, the system fails when you try to shut
it down:

Smart Shutdown request at Thu Dec 28 02:41:49 2000
DEBUG:  shutting down
FATAL 2:  Checkpoint lock is busy while data base is shutting down
Shutdown failed - abort

I have no idea why this should be.  Evidently there's something wrong
with the TAS() macro --- yet it seems to work fine elsewhere.  Ideas
anyone?

			regards, tom lane

Wed, 27 Dec 2000 21:45:06 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] Re: Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
On 27 Dec 2000 at 21:45 (-0500), Tom Lane wrote:
| Brent Verner <brent@rcfile.org> writes:
| > | Hm.  I thought I'd fixed that.  Are you up to date on
| > | src/backend/utils/adt/oid.c ?  Current CVS has rev 1.42.
| 
| > yup. got that version -- 1.42 2000/12/22 21:36:09 tgl
| 
| You're right, it was still broken :-(.  I think I've got it now, though.

i'll check it tomorrow.

| Oliver Elphick was kind enough to arrange access to an Alpha running
| Debian Linux, and I find that current-as-of-this-moment sources pass
| all regression tests in either serial or parallel test mode on that
| system.  Curiously, however, the system fails when you try to shut
| it down:

good. I'm glad you guys linked up :)

| Smart Shutdown request at Thu Dec 28 02:41:49 2000
| DEBUG:  shutting down
| FATAL 2:  Checkpoint lock is busy while data base is shutting down
| Shutdown failed - abort

I'm not seeing this with my latest revision of the TAS() asm.

Smart Shutdown request at Wed Dec 27 19:25:45 2000
DEBUG:  shutting down
DEBUG:  MoveOfflineLogs: remove 0000000000000000
DEBUG:  database system is shut down

| I have no idea why this should be.  Evidently there's something wrong
| with the TAS() macro --- yet it seems to work fine elsewhere.  Ideas
| anyone?

re-evaluating the asm stuff now.

thanks.
  brent

Thu, 28 Dec 2000 00:05:42 -0500
Brent Verner <brent@rcfile.org>
Re: [PATCHES] Re: Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
Tom Lane wrote:
...
  >system.  Curiously, however, the system fails when you try to shut
  >it down:
  >
  >Smart Shutdown request at Thu Dec 28 02:41:49 2000
  >DEBUG:  shutting down
  >FATAL 2:  Checkpoint lock is busy while data base is shutting down
  >Shutdown failed - abort
  >
  >I have no idea why this should be.  Evidently there's something wrong
  >with the TAS() macro --- yet it seems to work fine elsewhere.  Ideas
  >anyone?
 
It's not just on Alpha; I've seen that on my i386 Linux system.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "For God shall bring every work into judgment, 
      with every secret thing, whether it be good, or  
      whether it be evil."               Ecclesiastes 12:14 



Thu, 28 Dec 2000 07:59:06 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [PATCHES] Re: Re: Tuple-valued datums on Alpha (was Re:
	7.1 on DEC/Alpha)
"Oliver Elphick" <olly@lfix.co.uk> writes:
>> Smart Shutdown request at Thu Dec 28 02:41:49 2000
>> DEBUG:  shutting down
>> FATAL 2:  Checkpoint lock is busy while data base is shutting down
>> Shutdown failed - abort
 
> It's not just on Alpha; I've seen that on my i386 Linux system.

Oooh, that's interesting.  I was just blindly assuming that it was
a problem with the Alpha spinlock code (we've sure heard plenty of
discussion of same).  But maybe there's an actual logic bug in the
checkpoint code.  I don't see one in a quick scan though.

FWIW, I do *not* see this behavior on HPUX.  It seems perfectly
reproducible on the Debian Alpha box.  Is it reproducible on your
i386 box, or only sometimes?

Vadim, any ideas?

			regards, tom lane

Thu, 28 Dec 2000 03:10:11 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] Re: Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
Tom Lane wrote:
  >"Oliver Elphick" <olly@lfix.co.uk> writes:
  >>> FATAL 2:  Checkpoint lock is busy while data base is shutting down

  >> It's not just on Alpha; I've seen that on my i386 Linux system.

  >FWIW, I do *not* see this behavior on HPUX.  It seems perfectly
  >reproducible on the Debian Alpha box.  Is it reproducible on your
  >i386 box, or only sometimes?


Hmm. I'm just waking up a bit more.  Now I'm thinking slightly more
clearly, I saw the problem yesterday when I was doing an Alpha build
on faure.debian.org; so I think it was actually on Alpha, not i386 after
all.  Sorry for the red herring.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "For God shall bring every work into judgment, 
      with every secret thing, whether it be good, or  
      whether it be evil."               Ecclesiastes 12:14 



Thu, 28 Dec 2000 08:24:56 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [PATCHES] Re: Re: Tuple-valued datums on Alpha (was Re:
	7.1 on DEC/Alpha)
On Wed, 27 Dec 2000, Tom Lane wrote:

> After further study, I realized that fetchatt() and a number of other
> places were not prepared to cope with 8-byte pass-by-value datatypes.
> Most of them weren't checking for cases they couldn't handle, either.
> 
> Here is a revised patch for you to try (this includes yesterday's patch
> plus more changes, so you'll need to reverse out the prior patch before
> applying this one).  NOTE you will need to do a full reconfigure and
> rebuild to make this fly --- I'd suggest "make distclean" to start.

	Good news is that it solves the 'misc' regression test failure. It
now passes with flying colors! The bad news is that the 'oid' regression
test is still broken (with the exact same problem as before). I think
Brent hit the same problem... I guess, verify that your oid fix actually
hit the CVS tree, and if it did, rethink the solution. :(
	For testing I used the snapshot from ftp.postgresql.org:/pub/dev/
dated yesterday on my Alpha XLT366 running Debian GNU/Linux 2.2r0, kernel
2.2.17. Though I found the 'configure' file actually a copy of 'config.in'
and had to run the latter file through autoconf to get the correct version
of the former file. Weird.
	Also, I tested a patches source tree on an Linux/x86 box, and it
passed all regression tests w/o problems. I can test the patched source
tree on a Linux/Sparc machine if you want (bit more effort required to do
so). 
	Overall, it looks like we are making progress! Thanks to both you
and Brent for looking deeper into these problems. TTYL.

---------------------------------------------------------------------------
|   "For to me to live is Christ, and to die is gain."                    |
|                                            --- Philippians 1:21 (KJV)   |
---------------------------------------------------------------------------
|   Ryan Kirkpatrick  |  Boulder, Colorado  |  http://www.rkirkpat.net/   |
---------------------------------------------------------------------------


Thu, 28 Dec 2000 18:55:30 -0700 (MST)
Ryan Kirkpatrick <pgsql@rkirkpat.net>
Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
Ryan Kirkpatrick <pgsql@rkirkpat.net> writes:
> 	Good news is that it solves the 'misc' regression test failure. It
> now passes with flying colors! The bad news is that the 'oid' regression
> test is still broken (with the exact same problem as before). I think
> Brent hit the same problem... I guess, verify that your oid fix actually
> hit the CVS tree, and if it did, rethink the solution. :(

I believe that is fixed as of src/backend/utils/adt/oid.c v 1.43,
committed at Thu Dec 28 01:51:15 2000 UTC.  It should have been in
Thursday morning's snapshot.  If you've got 1.43 and it still fails
the regress test, let me know.

			regards, tom lane

Thu, 28 Dec 2000 21:47:25 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: Tuple-valued datums on Alpha (was Re: 7.1 on
	DEC/Alpha)
pg_restore has some options that are supposed to allow restoring some or
all indexes/tables/triggers/etc.  For example

pg_restore --table

restores all tables and

pg_restore --table=my_table

restores only the named table.  The equivalent short option is -t but it
does not allow restoring all tables, since it requires an argument.  I
suggest that an argument of '*' also means to restore all tables.

Also, if you just call pg_restore without arguments it hangs waiting for
input from stdin.  This is a bit confusing.  I suggest that stdin is used
if and only if the file name argument is '-'.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 6 Jan 2001 22:05:08 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
pg_restore options issues
At 22:05 6/01/01 +0100, Peter Eisentraut wrote:
>
>restores only the named table.  The equivalent short option is -t but it
>does not allow restoring all tables, since it requires an argument.  I
>suggest that an argument of '*' also means to restore all tables.

Sounds fine. 


>Also, if you just call pg_restore without arguments it hangs waiting for
>input from stdin.  This is a bit confusing.  I suggest that stdin is used
>if and only if the file name argument is '-'.

Also sounds reasonable.


I'm working on pg_dump at the moment, so if there is anything else, let me
know.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sun, 07 Jan 2001 11:48:05 +1100
Philip Warner <pjw@rhyme.com.au>
Re: pg_restore options issues
Peter Eisentraut wrote:
  >pg_restore has some options that are supposed to allow restoring some or
  >all indexes/tables/triggers/etc.  For example
  >
  >pg_restore --table
  >
  >restores all tables and
  >
  >pg_restore --table=my_table
  >
  >restores only the named table.  The equivalent short option is -t but it
  >does not allow restoring all tables, since it requires an argument.  I
  >suggest that an argument of '*' also means to restore all tables.

I don't like that; it will need to be escaped with \ or the shell will
substitute the contents of the current directory for the *.

Why not use `-t all' or `-A'?

You should also change the help text a bit:

  -t [table], --table[=table] 	 dump for this table only

gives no hint that omitting `=table' will restore all tables.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Blessed are the pure in heart, for they shall see 
      God."                    Matthew 5:8 



Sun, 07 Jan 2001 13:42:16 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: pg_restore options issues 
Oliver Elphick writes:

> I don't like that; it will need to be escaped with \ or the shell will
> substitute the contents of the current directory for the *.
>
> Why not use `-t all'

Because there might be a table called "all".  (Okay, there could be a
table called "*", but really...)

> or `-A'?

We'd need an option letter for tables, triggers, indexes, functions, etc.

> You should also change the help text a bit:
>
>   -t [table], --table[=table] 	 dump for this table only
>
> gives no hint that omitting `=table' will restore all tables.

Okay, when we decide how to handle it.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sun, 7 Jan 2001 17:02:53 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: pg_restore options issues 
Ok, I have a first set of 7.1beta3 RPMs uploading now.  These RPMs pass
regression on my home RedHat 6.2 machine, which has all locale environment
variables disabled (/etc/sysconfig/i18n deleted and a reboot).

It may take a few minutes to a few hours for the changes I uploaded to
propagate to the public ftp server.

NOTE: BETA RPMS ARE JUST EXACTLY THAT! These RPM's are not in a finished state
-- they are for the express purpose of allowing people to test 7.1beta in an
RPM setup.  Upgrading from a prior version is explicitly NOT supported at this
time.

There ARE KNOWN packaging problems -- I am working on them.

NOTE: there have been some changes in the RPM install structure. 
/usr/lib/pgsql is gone, replaced by /usr/share/postgresql.  See the file lists
from 'rpm -qlp postgresql*7.1beta3-1.i386.rpm' for more details.

README.rpm-dist has not been updated for the 7.1beta3 distribution yet.

To run regression tests, make sure the postgresql-test RPM is installed, then
make sure that the locale environment is set properly.  Regression tests may be
run by first, as root, starting a postmaster: /etc/rc.d/init.d/postgresql start
 (if there are any errors, make sure you are not upgrading!)

Then, as root:
chown -R postgres.postgres /usr/share/postgresql/test/regress

(yes, I know the RPMset should do this for you.  It is being fixed.)

Then, su to postgres and cd to /usr/share/postgresql/test/regress.  Execute:
./pg_regress --schedule=parallel_schedule

Any failures indicate an error somewhere -- most likely your locale settings.

Look for another release this week, with things streamlined somewhat.

Also, if you wish to rebuild from the source RPM, please read the comments and
conditionals in the spec file BEFORE rebuilding -- you can conditionally
rebuild just a few of the packages instead of having to rebuild everything.

Please check all the client interfaces you are equipped to check.  I need
someone to work over the perl, python, tcl, and tk stuff.

If someone wants to contribute built jars of the newest JDBC, I would be most
grateful, as I don't have a JDK on my devel machine yet.  The jars distributed
are the 7.0 JDBC jars -- which are known to need some recent patches.

pgaccess currently will not run unless you reconfigure to use -i in the
startup.  This is also being fixed in the RPMset -- there is a change necessary
in postgresql.config, I just have to do the change.

PL/perl is being worked on -- many thanks to Karl DeBisschop for his
perserverence in this.

RPMS are in at:
ftp.postgresql.org/pub/dev/test-rpms
and are built for RedHat 6.2/Intel ONLY.

Once again, at the risk of sounding like a broken record -- these are BETA
QUALITY RPMS -- BY REQUEST.  Please don't make me regret not waiting until a
release candidate :-).
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 15 Jan 2001 02:24:19 -0500
Lamar Owen <lamar.owen@wgcr.org>
RPMS for 7.1beta3 being uploaded.
Lamar Owen wrote:
  >Ok, I have a first set of 7.1beta3 RPMs uploading now. 
...  >
  >pgaccess currently will not run unless you reconfigure to use -i in the
  >startup.  This is also being fixed in the RPMset -- there is a change necess
      >ary
  >in postgresql.config, I just have to do the change.
 
In my experience, pgaccess will use the Unix socket if the hostname is
left blank.  Is this not the case with your RPMs?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "For I know that my redeemer liveth, and that he shall 
      stand at the latter day upon the earth"             
                                       Job 19:25 



Mon, 15 Jan 2001 12:31:07 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: RPMS for 7.1beta3 being uploaded. 
Lamar Owen writes:

> Ok, I have a first set of 7.1beta3 RPMs uploading now.  These RPMs pass
> regression on my home RedHat 6.2 machine, which has all locale environment
> variables disabled (/etc/sysconfig/i18n deleted and a reboot).

Some thoughts:

Re: rpm-pgsql-7.1beta3.patch

| diff -uNr postgresql-7.1beta3.orig/src/Makefile.shlib postgresql-7.1beta3/src/Makefile.shlib
| --- postgresql-7.1beta3.orig/src/Makefile.shlib Wed Dec  6 14:37:08 2000
| +++ postgresql-7.1beta3/src/Makefile.shlib      Mon Jan 15 01:50:04 2001
|@@ -160,7 +160,7 @@
|
|  ifeq ($(PORTNAME), linux)
|    shlib                        := lib$(NAME)$(DLSUFFIX).$(SO_MAJOR_VERSION).$(SO_MINOR_VERSION)
| -  LINK.shared          = $(COMPILER) -shared -Wl,-soname,$(soname)
| +  LINK.shared          = $(COMPILER) -shared -Wl
|  endif

This cannot possibly be right.

| diff -uNr postgresql-7.1beta3.orig/src/Makefile.shlib~ postgresql-7.1beta3/src/Makefile.shlib~

???

| -#!/usr/local/bin/perl -w
| +#!/usr/bin/perl -w
(and more of these for Python)

I think this should be fixed to read

#! /usr/bin/env perl

Any comments?


Re: spec file

| # I hope this works....
|
| %ifarch ia64
| ln -s linux_i386 src/template/linux
| %endif

It definitely won't...

| # If libtool installed, copy some files....
| if [ -d /usr/share/libtool ]
| then
|         cp /usr/share/libtool/config.* .
| fi

This is useless (because the config.* files are not in src/ anymore) and
(if it were fixed) not recommendable because config.{guess,sub} is not
compatible to itself, *especially* in terms of Linux recognition.  You
really should use the ones PostgreSQL comes with.

| %ifarch ppc
| NEW_CFLAGS=`echo $CFLAGS|xargs -n 1|grep -v "\-O"|xargs -n 100`
| NEW_CFLAGS="$NEW_CFLAGS -O0"

This is no longer necessary.

| ./configure --enable-hba --enable-locale  --with-CXX --prefix=/usr\

There is no option called '--enable-hba'.  And I think you're supposed to
use %{configure}.

| %if %tkpkg
|         --with-tk --with-x \
| %endif

There is no '--with-x'.  '--with-tk' is the default if '--with-tcl' was
given; you should use '--without-tk' if you don't want it.

| %if %jdbc
|         --with-java \
| %endif

There is no such option.

| %ifarch alpha
|         --with-template=linux_alpha \
| %endif

This won't work and is not necessary.

| make COPT="$NEW_CFLAGS" DESTDIR=$RPM_BUILD_ROOT/usr all

You should set CFLAGS when you run configure. (%{configure} will do that.)
DESTDIR is only useful when you run 'make install'.  And DESTDIR should
not include /usr.

| make all PGDOCS=unpacked -C doc

Not sure what this is supposed to do, but I don't think it will do what
you expect.  The docs are installed automatically.

| mkdir -p $RPM_BUILD_ROOT/usr/{include/pgsql,lib,bin}
| mkdir -p $RPM_BUILD_ROOT%{_mandir}

You don't need that, the directories are made automatically.

| make DESTDIR=$RPM_BUILD_ROOT -C src install

No '-C src'.

| # copy over the includes needed for SPI development.
| pushd src/include
| /lib/cpp -M -I. -I../backend executor/spi.h | \
|         xargs -n 1| \
|         grep \\W| \
|         grep -v ^/| \
|         grep -v spi.o | \
|         grep -v spi.h | \
|         sort | \
|         cpio -pdu $RPM_BUILD_ROOT/usr/include/pgsql

I think the standard installed set of headers is sufficient.

| %if %pgaccess
|         # pgaccess installation

Pgaccess is installed automatically when you configure --with-tcl.

| # Move the PL's to the right place
| mv $RPM_BUILD_ROOT/usr/lib/pl*.so $RPM_BUILD_ROOT/usr/share/postgresql

You should not put architecture specific files into share/.  I'm sure this
is in violation of FHS.  (I'm amazed createlang finds it there.)


Re: sub-packages

* pg_id should be in server

* What were the last thoughts about renaming the <nothing> package to -clients?

* pg_upgrade won't work, so you might as well not install it.  It will
  probably be disabled before we release.

* You're missing pg_config in the -devel package.


These are the things I could find at first glance. ;-)

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Mon, 15 Jan 2001 18:50:39 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: RPMS for 7.1beta3 being uploaded.
Peter Eisentraut wrote:
> 
> Lamar Owen writes:
> 
> > Ok, I have a first set of 7.1beta3 RPMs uploading now.  These RPMs pass
> > regression on my home RedHat 6.2 machine, which has all locale environment
> > variables disabled (/etc/sysconfig/i18n deleted and a reboot).
> 
> Some thoughts:

snip

> | -#!/usr/local/bin/perl -w
> | +#!/usr/bin/perl -w
> (and more of these for Python)
> 
> I think this should be fixed to read
> 
> #! /usr/bin/env perl
> 
> Any comments?

What assurance do you have that 'env' is in /usr/bin? Linux FHS
(http://www.pathname.com/fhs/2.1/fhs-4.2.html) says that when perl
is in some location other than /usr/bin, then symlinks should be
provided to /usr/bin/perl. You don't have this assurance with
/usr/bin/env. Maybe there are some unices that do not have perl in
/usr/bin - but maybe there are unices that put 'env' into /bin
instead of /usr/bin. If you follow the current FHS, /usr/bin/perl
may be better -- but I'm not sure if that is entirely true of the
real world of linux boxes out there.

> | # copy over the includes needed for SPI development.
> | pushd src/include
> | /lib/cpp -M -I. -I../backend executor/spi.h | \
> |         xargs -n 1| \
> |         grep \\W| \
> |         grep -v ^/| \
> |         grep -v spi.o | \
> |         grep -v spi.h | \
> |         sort | \
> |         cpio -pdu $RPM_BUILD_ROOT/usr/include/pgsql
> 
> I think the standard installed set of headers is sufficient.

I haven't installed the RPM yet. But I dissent unless things have
changed from 7.0 in that respect. I copmile against executor/spi.h,
based on the examples in the docs. Should I be using something else?
(I'll have to go look at the 7.1 source, but I just wanted to
register at least some comfusion, if not dissent) 

-- 
Karl DeBisschop                       
kdebisschop@alert.infoplease.com
Learning Network Reference             http://www.infoplease.com
Netsaint Plugin Developer             
kdebisschop@users.sourceforge.net

Mon, 15 Jan 2001 13:37:58 -0500
Karl DeBisschop <kdebisschop@alert.infoplease.com>
Re: RPMS for 7.1beta3 being uploaded.
(Sorry of this double posts - I'm having alias troubles on the
list.)

Peter Eisentraut wrote:
> 
> Lamar Owen writes:
> 
> > Ok, I have a first set of 7.1beta3 RPMs uploading now.  These RPMs pass
> > regression on my home RedHat 6.2 machine, which has all locale environment
> > variables disabled (/etc/sysconfig/i18n deleted and a reboot).
> 
> Some thoughts:

snip

> | -#!/usr/local/bin/perl -w
> | +#!/usr/bin/perl -w
> (and more of these for Python)
> 
> I think this should be fixed to read
> 
> #! /usr/bin/env perl
> 
> Any comments?

What assurance do you have that 'env' is in /usr/bin? Linux FHS
(http://www.pathname.com/fhs/2.1/fhs-4.2.html) says that when perl
is in some location other than /usr/bin, then symlinks should be
provided to /usr/bin/perl. You don't have this assurance with
/usr/bin/env. Maybe there are some unices that do not have perl in
/usr/bin - but maybe there are unices that put 'env' into /bin
instead of /usr/bin. If you follow the current FHS, /usr/bin/perl
may be better -- but I'm not sure if that is entirely true of the
real world of linux boxes out there.

> | # copy over the includes needed for SPI development.
> | pushd src/include
> | /lib/cpp -M -I. -I../backend executor/spi.h | \
> |         xargs -n 1| \
> |         grep \\W| \
> |         grep -v ^/| \
> |         grep -v spi.o | \
> |         grep -v spi.h | \
> |         sort | \
> |         cpio -pdu $RPM_BUILD_ROOT/usr/include/pgsql
> 
> I think the standard installed set of headers is sufficient.

I haven't installed the RPM yet. But I dissent unless things have
changed from 7.0 in that respect. I copmile against executor/spi.h,
based on the examples in the docs. Should I be using something else?
(I'll have to go look at the 7.1 source, but I just wanted to
register at least some comfusion, if not dissent) 

-- 
Karl DeBisschop              kdebisschop@alert.infoplease.com
Learning Network/Reference   http://www.infoplease.com
Netsaint Plugin Developer    kdebisschop@users.sourceforge.net

Mon, 15 Jan 2001 13:54:24 -0500
Karl DeBisschop <kdebisschop@alert.infoplease.com>
Re: RPMS for 7.1beta3 being uploaded.
Peter Eisentraut wrote:
> Lamar Owen writes:
> > Ok, I have a first set of 7.1beta3 RPMs uploading now.  These RPMs pass
> > regression on my home RedHat 6.2 machine, which has all locale environment
> > variables disabled (/etc/sysconfig/i18n deleted and a reboot).
 
> Some thoughts:
 
> Re: rpm-pgsql-7.1beta3.patch
 
> | diff -uNr postgresql-7.1beta3.orig/src/Makefile.shlib postgresql-7.1beta3/src/Makefile.shlib
> | -  LINK.shared          = $(COMPILER) -shared -Wl,-soname,$(soname)
> | +  LINK.shared          = $(COMPILER) -shared -Wl
> |  endif
 
> This cannot possibly be right.

It's what you recommended a while back.  See the discussions on -soname
from the libpq.so.2.1 versus libpq.so.2.0 thread awhile back.
 
> | diff -uNr postgresql-7.1beta3.orig/src/Makefile.shlib~ postgresql-7.1beta3/src/Makefile.shlib~
 
> ???

Leftover Kedit baggage.  Those are easy to forget to rm during patch
build, particularly at 2:00AM -- but there's no harm in it being there
for now.  And it won't be there in -2.

> | -#!/usr/local/bin/perl -w
> | +#!/usr/bin/perl -w
> (and more of these for Python)
 
> I think this should be fixed to read
> 
> #! /usr/bin/env perl

No, for a RedHat or any other Linux distribution, /usr/bin is where perl
and python (or their symlinks) will always live.

Although you missed the redundant patches in the regression tree :-).

> Re: spec file
 
> | # I hope this works....
> |
> | %ifarch ia64
> | ln -s linux_i386 src/template/linux
> | %endif
> 
> It definitely won't...

7.0 required it.  Baggage from the previous build.
 
> | # If libtool installed, copy some files....
> | if [ -d /usr/share/libtool ]
> | then
> |         cp /usr/share/libtool/config.* .
> | fi
 
> This is useless (because the config.* files are not in src/ anymore) and
> (if it were fixed) not recommendable because config.{guess,sub} is not
> compatible to itself, *especially* in terms of Linux recognition.  You
> really should use the ones PostgreSQL comes with.

Trond can answer that one more effectively than can I, as that's his
insertion.

Of course, I've got to reorg the destination to match the source tree's
reorg.
 
> | %ifarch ppc
> | NEW_CFLAGS=`echo $CFLAGS|xargs -n 1|grep -v "\-O"|xargs -n 100`
> | NEW_CFLAGS="$NEW_CFLAGS -O0"
 
> This is no longer necessary.

Depends on the convolutions the particular build of rpm itself is
doing.  This is a fix for the broken rpm setup found on Linux-PPC, as
found by Tom Lane.  It would be marvelous if this would be expendable at
this juncture.
 
> | ./configure --enable-hba --enable-locale  --with-CXX --prefix=/usr\
 
> There is no option called '--enable-hba'.  And I think you're supposed to
> use %{configure}.

If it works now, I'll use it.  Version 7.0 and prior wouldn't work with
%{configure}.

And --enable-hba has existed at some point in time.
 
> | %if %tkpkg
> |         --with-tk --with-x \
> | %endif
> 
> There is no '--with-x'.  '--with-tk' is the default if '--with-tcl' was
> given; you should use '--without-tk' if you don't want it.

There was in the past a --with-x.  So I need to change that to check for
the negation of tkpkg and use --without-tk if so.
 
> | %if %jdbc
> |         --with-java \
> | %endif
 
> There is no such option.

Hmmm.  I don't remember when that one got placed there....
 
> | %ifarch alpha
> |         --with-template=linux_alpha \
> | %endif
 
> This won't work and is not necessary.

More 7.0 and prior baggage.  The patches for alpha at one point (6.5
through 7.0.3) have required this -- of course, with the need for the
alpha patches gone, the need for the special config step is also gone. 
One more piece of baggage I missed.
 
> | make COPT="$NEW_CFLAGS" DESTDIR=$RPM_BUILD_ROOT/usr all
 
> You should set CFLAGS when you run configure. (%{configure} will do that.)
> DESTDIR is only useful when you run 'make install'.  And DESTDIR should
> not include /usr.

Yes, if you'll notice I fixed the DESTDIR in the install. But, of
course, it's not needed (nor is it used) in the build itself.  Again,
I'll use %{configure} when I verify that it works properly (if it does,
that will be a very Good Thing for all involved).  But, again, you're
seeing baggage that 7.0 and prior required in order to build (well,
except DESTDIR).
 
> | make all PGDOCS=unpacked -C doc
 
> Not sure what this is supposed to do, but I don't think it will do what
> you expect.  The docs are installed automatically.

Well, they are _now_.  But 7.0 and prior..... again, more old baggage.
 
> | mkdir -p $RPM_BUILD_ROOT/usr/{include/pgsql,lib,bin}
> | mkdir -p $RPM_BUILD_ROOT%{_mandir}
 
> You don't need that, the directories are made automatically.

They are _now_.  But before, when the make install put things in the
'wrong' place, it was required to make the directories before doing the
copies and moves necessary.
 
> | make DESTDIR=$RPM_BUILD_ROOT -C src install
 
> No '-C src'.

Not anymore, at least.  *sigh*  There has been alot of baggage
accumulate in the spec due to 7.0 and prior's slightly brain-dead
build.  I got rid of a lot of it -- but obviously I missed some.
 
> | # copy over the includes needed for SPI development.
> | pushd src/include
> | /lib/cpp -M -I. -I../backend executor/spi.h | \
> |         xargs -n 1| \
> |         grep \\W| \
> |         grep -v ^/| \
> |         grep -v spi.o | \
> |         grep -v spi.h | \
> |         sort | \
> |         cpio -pdu $RPM_BUILD_ROOT/usr/include/pgsql
 
> I think the standard installed set of headers is sufficient.

Is it?  It _wasn't_ sufficient for SPI development at 7.0.  Have the
headers and the headers install been fixed to install _all_ necessary
development headers, SPI included?  If so, this step won't add any
headers anyway -- but it would be nice to see the difference.
 
> | %if %pgaccess
> |         # pgaccess installation
 
> Pgaccess is installed automatically when you configure --with-tcl.

It is now.  It wasn't previously. At least not to the proper place.
 
> | # Move the PL's to the right place
> | mv $RPM_BUILD_ROOT/usr/lib/pl*.so $RPM_BUILD_ROOT/usr/share/postgresql
 
> You should not put architecture specific files into share/.  I'm sure this
> is in violation of FHS.  (I'm amazed createlang finds it there.)

It finds it there because I tell it to :-P.  No, this objection is
definitely correct -- /usr/lib is the right place, and is where it will
live in the future.  Previously, that line moved them to /usr/lib/pgsql.
 
> Re: sub-packages
 
> * pg_id should be in server
 
> * What were the last thoughts about renaming the <nothing> package to -clients?

My thoughts are to leave it the way it is.  I am not alone.  But it's
not set in stone.
 
> * pg_upgrade won't work, so you might as well not install it.  It will
>   probably be disabled before we release.

Yes, this one needs to not be installed for this release.
 
> * You're missing pg_config in the -devel package.

Yes, I am.  I noticed that just after I uploaded the rpms from a 28.8
dialup at home.  So, I'll be releasing a -2 set this week to fix that
problem, amongst others.
 
> These are the things I could find at first glance. ;-)

Some first glance :-)....

Thanks for the feedback.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 15 Jan 2001 15:50:04 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: RPMS for 7.1beta3 being uploaded.
On older versions of PG,  7.0 included,  in the $PDGATA/base folder you
could see the names of the databases for that $PGDATA.  Now all I see is:

$ ls -l
total 16
drwx------  2 postgres  wheel  1536 Jan 12 15:42 1
drwx------  2 postgres  wheel  1536 Jan 12 15:41 18719
drwx------  2 postgres  wheel  1536 Jan 12 15:42 18720
drwx------  2 postgres  wheel  1536 Jan 15 15:59 18721

Is there a way to relate this to the names of the databases?  Why the
change?  Or am I missing something key here..

- Brandon

b. palmer,  bpalmer@crimelabs.net
pgp:  www.crimelabs.net/bpalmer.pgp5


Mon, 15 Jan 2001 16:01:47 -0500 (EST)
bpalmer <bpalmer@crimelabs.net>
$PGDATA/base/???
Lamar Owen <lamar.owen@wgcr.org> writes:

> Peter Eisentraut wrote:
> > Lamar Owen writes:
> > > Ok, I have a first set of 7.1beta3 RPMs uploading now.  These RPMs pass
> > > regression on my home RedHat 6.2 machine, which has all locale environment
> > > variables disabled (/etc/sysconfig/i18n deleted and a reboot).
>  
> > Some thoughts:
>  
> > Re: rpm-pgsql-7.1beta3.patch
>  
> > | diff -uNr postgresql-7.1beta3.orig/src/Makefile.shlib postgresql-7.1beta3/src/Makefile.shlib
> > | -  LINK.shared          = $(COMPILER) -shared -Wl,-soname,$(soname)
> > | +  LINK.shared          = $(COMPILER) -shared -Wl
> > |  endif
>  
> > This cannot possibly be right.
> 
> It's what you recommended a while back.  See the discussions on -soname
> from the libpq.so.2.1 versus libpq.so.2.0 thread awhile back.

Not having a soname doesn't really solve the problem, though... it
should be used, but correctly (major, minor)

 
> > Re: spec file
>  
> > | # I hope this works....
> > |
> > | %ifarch ia64
> > | ln -s linux_i386 src/template/linux
> > | %endif
> > 
> > It definitely won't...

It did... we've successfully run queries etc on it.

> > | # If libtool installed, copy some files....
> > | if [ -d /usr/share/libtool ]
> > | then
> > |         cp /usr/share/libtool/config.* .
> > | fi
>  
> > This is useless (because the config.* files are not in src/ anymore) and
> > (if it were fixed) not recommendable because config.{guess,sub} is not
> > compatible to itself, *especially* in terms of Linux recognition.  You
> > really should use the ones PostgreSQL comes with.

We have a libtool tuned to work with lots of platforms, like ia64,
s390 etc... this makes sure it's used.

> 

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

15 Jan 2001 16:03:43 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: RPMS for 7.1beta3 being uploaded.
bpalmer wrote:
> 
> On older versions of PG,  7.0 included,  in the $PDGATA/base folder you
> could see the names of the databases for that $PGDATA.  Now all I see is:

No longer.
 
> Is there a way to relate this to the names of the databases?  Why the
> change?  Or am I missing something key here..

See the thread on the renaming in the archives.  In short, this is part
of Vadim's work on WAL -- the new naming makes certain things easier for
WAL.

Utilities to relate the new names to the actual database/table names
_do_ need to be written, however.  The information exists in one of the
system catalogs now -- it just has to be made accessible.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 15 Jan 2001 16:04:48 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: $PGDATA/base/???
Trond Eivind Glomsrï¿½d wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > Peter Eisentraut wrote:
> > > Re: rpm-pgsql-7.1beta3.patch
> > > | diff -uNr postgresql-7.1beta3.orig/src/Makefile.shlib postgresql-7.1beta3/src/Makefile.shlib
> > > | -  LINK.shared          = $(COMPILER) -shared -Wl,-soname,$(soname)
> > > | +  LINK.shared          = $(COMPILER) -shared -Wl
> > > |  endif

> > > This cannot possibly be right.

> > It's what you recommended a while back.  See the discussions on -soname
> > from the libpq.so.2.1 versus libpq.so.2.0 thread awhile back.
 
> Not having a soname doesn't really solve the problem, though... it
> should be used, but correctly (major, minor)

Ok, what's your recommendation for the patch?  I'll look again. 7.0.3-2
shipped with the equivalent patch.  I think I know what you have in
mind, but it will be later tonight before I have time to try it.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 15 Jan 2001 16:08:20 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: RPMS for 7.1beta3 being uploaded.
Lamar Owen <lamar.owen@wgcr.org> writes:
>> | %ifarch ppc
>> | NEW_CFLAGS=`echo $CFLAGS|xargs -n 1|grep -v "\-O"|xargs -n 100`
>> | NEW_CFLAGS="$NEW_CFLAGS -O0"
 
>> This is no longer necessary.

> Depends on the convolutions the particular build of rpm itself is
> doing.  This is a fix for the broken rpm setup found on Linux-PPC, as
> found by Tom Lane.  It would be marvelous if this would be expendable at
> this juncture.

It is.  7.1 builds cleanly on PPC without any CFLAGS hackery.  I think
we can even survive the -fsigned-char stupidity now ;-)


>> I think the standard installed set of headers is sufficient.

> Is it?  It _wasn't_ sufficient for SPI development at 7.0.  Have the
> headers and the headers install been fixed to install _all_ necessary
> development headers, SPI included?

No, nothing's been done about that AFAIK.  I'm not sure the RPMs should
be taking it on themselves to solve the problem, however.

			regards, tom lane

Mon, 15 Jan 2001 16:11:57 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: RPMS for 7.1beta3 being uploaded. 
Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > doing.  This is a fix for the broken rpm setup found on Linux-PPC, as
> > found by Tom Lane.  It would be marvelous if this would be expendable at
> > this juncture.
 
> It is.  7.1 builds cleanly on PPC without any CFLAGS hackery.  I think
> we can even survive the -fsigned-char stupidity now ;-)

Oh, good.  Makes it much cleaner.  Care to test that theory? :-)
 
> >> I think the standard installed set of headers is sufficient.
 
> > Is it?  It _wasn't_ sufficient for SPI development at 7.0.  Have the
> > headers and the headers install been fixed to install _all_ necessary
> > development headers, SPI included?
 
> No, nothing's been done about that AFAIK.  I'm not sure the RPMs should
> be taking it on themselves to solve the problem, however.

Just trying to make the postgresql-devel rpm complete, as per request. 
Since the folk who build from source usually still have the source tree
around to do SPI development in, it's not as big of an issue for that
install.

Of course, if the consensus is that the RPM's simply track what the
source tarball does, then that can also be arranged.  But, the precedent
of the RPMset fixing difficulties with the source install has already
been set with the upgrading procedure.

Arguments about why those wishing to do SPI development should install a
full source tree aside, I'm simply providing an RPM-specific requested
feature.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 15 Jan 2001 16:21:10 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: RPMS for 7.1beta3 being uploaded.
Lamar Owen <lamar.owen@wgcr.org> writes:
>> It is.  7.1 builds cleanly on PPC without any CFLAGS hackery.  I think
>> we can even survive the -fsigned-char stupidity now ;-)

> Oh, good.  Makes it much cleaner.  Care to test that theory? :-)

I already did, I believe, but just in plain builds from source.

Let me know when you think the 7.1 RPM specfile is stable enough to be
worth testing, and I'll try to build PPC RPMs.

			regards, tom lane

Mon, 15 Jan 2001 16:26:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: RPMS for 7.1beta3 being uploaded. 
Tom Lane wrote:
> Let me know when you think the 7.1 RPM specfile is stable enough to be
> worth testing, and I'll try to build PPC RPMs.

Ok.  Should be coincident with -2. I'm planning to have a -2 out later
this week.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 15 Jan 2001 16:29:12 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: RPMS for 7.1beta3 being uploaded.
> > Is there a way to relate this to the names of the databases?  Why the
> > change?  Or am I missing something key here..
> 
> See the thread on the renaming in the archives.  In short, this is part
> of Vadim's work on WAL -- the new naming makes certain things easier for
> WAL.
> 
> Utilities to relate the new names to the actual database/table names
> _do_ need to be written, however.  The information exists in one of the
> system catalogs now -- it just has to be made accessible.

Yes, I am hoping to write this utility before 7.1 final.  Maybe it will
have to be in /contrib.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 15 Jan 2001 16:55:51 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: $PGDATA/base/???
Trond Eivind Glomsrï¿½d writes:

> We have a libtool tuned to work with lots of platforms, like ia64,
> s390 etc... this makes sure it's used.

We don't use libtool.  Nor does libtool care about the processor.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Mon, 15 Jan 2001 23:02:54 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: RPMS for 7.1beta3 being uploaded.
Peter Eisentraut <peter_e@gmx.net> writes:

> Trond Eivind Glomsrï¿½d writes:
> 
> > We have a libtool tuned to work with lots of platforms, like ia64,
> > s390 etc... this makes sure it's used.
> 
> We don't use libtool.  

Doing so would be a good thing.

> Nor does libtool care about the processor.

As you can see from the actual code segment, only the
config.{guess,sub} files are copied.
-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

15 Jan 2001 17:05:14 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: RPMS for 7.1beta3 being uploaded.
Peter Eisentraut wrote:
> The patch I recommended was
 
> -  LDFLAGS_SL            := -Bdynamic -shared -soname $(shlib)
> +  LDFLAGS_SL            := -Bdynamic -shared -soname lib$(NAME)$(DLSUFFIX).$(SO_MAJOR_VERSION)

And that is what was in 7.0.3.
 
> but that's not what your patch does.  The issue is fixed, you shouldn't
> patch anything.

Oh, OK.  Easy enough to delete that section and see where the chips
fall.

> > > #! /usr/bin/env perl

> > No, for a RedHat or any other Linux distribution, /usr/bin is where perl
> > and python (or their symlinks) will always live.
 
> I was thinking in terms of fixing this in the source tree.

Oh.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 15 Jan 2001 17:15:14 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: RPMS for 7.1beta3 being uploaded.
Lamar Owen writes:

> > | diff -uNr postgresql-7.1beta3.orig/src/Makefile.shlib postgresql-7.1beta3/src/Makefile.shlib
> > | -  LINK.shared          = $(COMPILER) -shared -Wl,-soname,$(soname)
> > | +  LINK.shared          = $(COMPILER) -shared -Wl
> > |  endif
>
> > This cannot possibly be right.
>
> It's what you recommended a while back.  See the discussions on -soname
> from the libpq.so.2.1 versus libpq.so.2.0 thread awhile back.

The patch I recommended was

-  LDFLAGS_SL            := -Bdynamic -shared -soname $(shlib)
+  LDFLAGS_SL            := -Bdynamic -shared -soname lib$(NAME)$(DLSUFFIX).$(SO_MAJOR_VERSION)

but that's not what your patch does.  The issue is fixed, you shouldn't
patch anything.

> > I think this should be fixed to read
> >
> > #! /usr/bin/env perl
>
> No, for a RedHat or any other Linux distribution, /usr/bin is where perl
> and python (or their symlinks) will always live.

I was thinking in terms of fixing this in the source tree.

> > There is no '--with-x'.  '--with-tk' is the default if '--with-tcl' was
> > given; you should use '--without-tk' if you don't want it.
>
> There was in the past a --with-x.

But it never did anything.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Mon, 15 Jan 2001 23:16:19 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: RPMS for 7.1beta3 being uploaded.
Peter Eisentraut wrote:
> Trond Eivind Glomsrï¿½d writes:
> > We have a libtool tuned to work with lots of platforms, like ia64,
> > s390 etc... this makes sure it's used.
 
> We don't use libtool.  Nor does libtool care about the processor.

'We' has a lot of different meanings.  In your sentence, of course, 'We'
is 'the PostgreSQL Global Development Group.'  In my case, I have to use
a 'We' comprised of the PGDG, RedHat, SuSE, TurboLinux, Mandrake,
Caldera, GreatBridge, etc.

The RPM-based Linux distributions have their own consistency
requirements that may at times conflict with 'our' own.  In the case of
packages the are being distributed with these distributions, it behooves
'us' to play by the distribution's rules when those rules don't
negatively affect 'our' primary source distribution.

By having this only in the RPM spec file, it obviates the need for 'us'
to change anything, and it allows 'our' package to be more easily
integrated in the distribution in question.

In particular, this was and is a RedHat-made change. It does not break
anything that I am aware of, and allows the distributions to do their
thing as well.

Therefore, the conditional.  If the distribution doesn't care, then the
files won't exist, and they won't overwrite ours. RedHat 6.1, for
instance.

Is there a technical reason that this would cause a failed or even
non-optimal build?  Non-functional binaries?  Problems at the user
level?

The PPC shenanigans will be gone -- but the libtool stuff -- until
libtoolize works properly on 'our' build (I seem to recall a
distribution that successfully libtollized PostgreSQL -- I'll have to
check.....), this seems reasonable to me in my limited knowledge.

So, Trond, what sort of tunings have been performed that make the files
in question need to be copied?  I'm sure you have a good reason; I am
just curious as to what it is.

Likewise, Peter, I'm sure that from your point of view you have good
reasons -- I'd like to see them as well, for the same reasons as I'd
like to see Trond's.

We're early in the RPM beta cycle here -- many things can and will
change before final.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 15 Jan 2001 17:16:35 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: RPMS for 7.1beta3 being uploaded.
Peter Eisentraut <peter_e@gmx.net> writes:

> Trond Eivind Glomsrï¿½d writes:
> 
> > > We don't use libtool.
> >
> > Doing so would be a good thing.
> 
> Not if our code is more portable than libtool's.

And this is the case? libtool covers pretty much everything... and you
don't need to use it for every target.
 
> > > Nor does libtool care about the processor.
> >
> > As you can see from the actual code segment, only the
> > config.{guess,sub} files are copied.
> 
> But you argued that this is because your config.guess supports s390 and
> ia64 (which ours does as well)

It may do so now, but I'm pretty sure it hasn't always done so... and
even if it does, it doesn't hurt.
> 

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

15 Jan 2001 17:17:37 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: RPMS for 7.1beta3 being uploaded.
Trond Eivind Glomsrï¿½d writes:

> > We don't use libtool.
>
> Doing so would be a good thing.

Not if our code is more portable than libtool's.

> > Nor does libtool care about the processor.
>
> As you can see from the actual code segment, only the
> config.{guess,sub} files are copied.

But you argued that this is because your config.guess supports s390 and
ia64 (which ours does as well), but that cannot possibly be libtool
related.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Mon, 15 Jan 2001 23:18:27 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: RPMS for 7.1beta3 being uploaded.
Lamar Owen <lamar.owen@wgcr.org> writes:

> Peter Eisentraut wrote:
> > Trond Eivind Glomsrï¿½d writes:
> > > We have a libtool tuned to work with lots of platforms, like ia64,
> > > s390 etc... this makes sure it's used.
>  
> > We don't use libtool.  Nor does libtool care about the processor.
>
> In particular, this was and is a RedHat-made change. It does not break
> anything that I am aware of, and allows the distributions to do their
> thing as well.

Note that this wasn't included in Red Hat Linux 7... it's been done
since then, and I don't remember doing it myself (which of course
doesn't mean I didn't do it :) - it might have been done for the S/390
port, by the people working on that. 
 
> So, Trond, what sort of tunings have been performed that make the files
> in question need to be copied?  I'm sure you have a good reason; I am
> just curious as to what it is.

For most apps, it's just a question of configure working vs. configure
failing on IA64 (there is no "tuning" as such, my choice of words
wasn't too good). There may be something similar for S/390.


-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

15 Jan 2001 17:25:50 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: RPMS for 7.1beta3 being uploaded.
Trond Eivind Glomsrï¿½d wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > In particular, this was and is a RedHat-made change. It does not break
> > anything that I am aware of, and allows the distributions to do their
> > thing as well.
 
> Note that this wasn't included in Red Hat Linux 7... it's been done
> since then, and I don't remember doing it myself (which of course
> doesn't mean I didn't do it :) - it might have been done for the S/390
> port, by the people working on that.

A non-conditional version (the conditional is my change) was included as
far back as RedHat 6.2.
 
> For most apps, it's just a question of configure working vs. configure
> failing on IA64 (there is no "tuning" as such, my choice of words
> wasn't too good). There may be something similar for S/390.

Can we test both ways (after your current project is done)?
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 15 Jan 2001 17:32:26 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: RPMS for 7.1beta3 being uploaded.
Lamar Owen writes:

> Likewise, Peter, I'm sure that from your point of view you have good
> reasons -- I'd like to see them as well, for the same reasons as I'd
> like to see Trond's.

Two points of view here:

1.  The config.* files were specifically updated because the old ones did
not recognize certain Linux(!) setups correctly.  This probably affects
all config.*'s before August 2000.  This will simply cause the build to
fail.

2.  If you use the %{configure} macro then this will be pointless because
that macro selects the host system type itself:

| %configure \
| CFLAGS="${CFLAGS:-%optflags}" ; export CFLAGS ; \
| CXXFLAGS="${CXXFLAGS:-%optflags}" ; export CXXFLAGS ; \
| FFLAGS="${FFLAGS:-%optflags}" ; export FFLAGS ; \
| %{?__libtoolize:[ -f configure.in ] && %{__libtoolize} --copy --force} ; \
| ./configure %{_target_platform} \\\
...

(This is subtly wrong as well, but that's something for the RPM folks to
deal with.)

So in short, don't do it.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Tue, 16 Jan 2001 17:42:06 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: RPMS for 7.1beta3 being uploaded.
Hello,

I've selected postgresql 7.0.3 for our (critical) application and while
doing my first experiments I've found a bug which makes me worry very
much.

The problem is that a SELECT with a certain LIKE condition in combination
with a GROUP BY does not find the proper records when there is an index on
the particular column present.  When the index is removed the SELECT *does*
return the right answer.

Fortunately I managed to strip down our database and create a simple
single table with which the bug can be easily reproduced.

I've been searching in the Postgres bug-database and this problem
might be related to this report:

        http://www.postgresql.org/bugs/bugs.php?4~111

Below you find a psql-session that demonstrates the bug.

I've made a dump of the test-database available as:
        
        http://dutepp0.et.tudelft.nl/~robn/demo.dump.bz2

(it is 46100 bytes long in compressed form but 45 MB when uncompressed,
 I tried to trim it down but then the bug isn't reproducable anymore !)

The table is filled with all Spaces execpt for the "town" column.


Sysinfo:  
--------
        - well-maintained Linux Red Hat 6.2
	- kernel 2.2.18
        - Intel Pentium III
        - postgresql-7.0.3-2 RPMs from the Postgresql site
          (the problem also occurs with locally rebuilt Source RPM)

Any help is much appreciated !
        
        Friendly greetings,
        Rob van Nieuwkerk


psql session:
***********************************************************************
demo=> \d 
     List of relations
    Name    | Type  | Owner 
------------+-------+-------
 demo_table | table | robn
(1 row)

demo=> \d demo_table    
       Table "demo_table"
 Attribute |   Type   | Modifier 
-----------+----------+----------
 postcode  | char(7)  |
 odd_even  | char(1)  |
 low       | char(5)  |
 high      | char(5)  |
 street    | char(24) | 
 town      | char(24) |   
 area      | char(1)  |

demo=> \di
No relations found.
demo=> SELECT town FROM demo_table WHERE town  LIKE 'ZWO%' GROUP BY town;
           town           
--------------------------
 ZWOLLE
(1 row)

demo=> SELECT town FROM demo_table WHERE town  LIKE 'Z%' GROUP BY town;

    <<<<<< here 86 towns are correctly found (output removed) >>>>>>

demo=> CREATE INDEX demo_table_town_idx ON demo_table(town);
CREATE
demo=> SELECT town FROM demo_table WHERE town  LIKE 'Z%' GROUP BY town;
 town
------
(0 rows)
        <<<<<< This is wrong !!!!!! >>>>>>>

demo=> SELECT town FROM demo_table WHERE town  LIKE 'ZWO%' GROUP BY town;
           town
--------------------------
 ZWOLLE
(1 row)

demo=> DROP INDEX demo_table_town_idx;
DROP
demo=> SELECT town FROM demo_table WHERE town  LIKE 'Z%' GROUP BY town;

    <<<<<< here 86 towns are correctly found again >>>>>>
***********************************************************************

Thu, 18 Jan 2001 15:13:02 +0000 (UTC)
robn@verdi.et.tudelft.nl (Rob van Nieuwkerk)
7.0.3 reproduceable serious select error
robn@verdi.et.tudelft.nl (Rob van Nieuwkerk) writes:
> The problem is that a SELECT with a certain LIKE condition in combination
> with a GROUP BY does not find the proper records when there is an index on
> the particular column present.  When the index is removed the SELECT *does*
> return the right answer.

Are you running the postmaster in a non-ASCII locale?  This sounds like
the old LIKE index optimization problem that we've struggled with for
quite a while now.  7.1 works around it by disabling the optimization
in non-ASCII locales, which is unpleasant but at least it gives right
answers ...

			regards, tom lane

Thu, 18 Jan 2001 16:13:17 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: 7.0.3 reproduceable serious select error 
Tom Lane wrote:
> 
> robn@verdi.et.tudelft.nl (Rob van Nieuwkerk) writes:
> > The problem is that a SELECT with a certain LIKE condition in combination
> > with a GROUP BY does not find the proper records when there is an index on
> > the particular column present.  When the index is removed the SELECT *does*
> > return the right answer.
> 
> Are you running the postmaster in a non-ASCII locale?  This sounds like
> the old LIKE index optimization problem that we've struggled with for
> quite a while now.  7.1 works around it by disabling the optimization
> in non-ASCII locales, which is unpleasant but at least it gives right
> answers ...

Hi Tom,

I don't think I'm running postmaster in a non-ASCII locale.
At least I did not explicitly do anything to accomplish it.
I'm running with the default settings from the RPMs and didn't
change any default setting.

I peeked in some manual pages but couldn't find info quickly about
this setting.  Please tell me how to check it if you want to know !

Thank you for your reaction.

	greetings,
	Rob van Nieuwkerk

Thu, 18 Jan 2001 22:23:26 +0100 (CET)
Rob van Nieuwkerk <robn@verdi.et.tudelft.nl>
Re: 7.0.3 reproduceable serious select error
Rob van Nieuwkerk wrote:
I tried to reproduce this bug on 7.0.2 and 7.0.3 with both 8K and 32K block
sizes, and could not reproduce the error.

I am running RedHat 6.2 kernel 2.2.16.

I don't know enough to even be close, but I wonder if there are any subtle
differences between the way characters are treated for indexes vs the way they
are treated for table scans? If there are even slight differences in the way
this happens, a misinterpretation of ascii conversions for instance, (I am
assuming you may be using ascii characters above 0x7F), it could behave
something like this, and explain why I wouldn't see it. .Like I said, however,
I don't know  much so don't read too much into what I say.



> Hello,
>
> I've selected postgresql 7.0.3 for our (critical) application and while
> doing my first experiments I've found a bug which makes me worry very
> much.
>
> The problem is that a SELECT with a certain LIKE condition in combination
> with a GROUP BY does not find the proper records when there is an index on
> the particular column present.  When the index is removed the SELECT *does*
> return the right answer.
>
> Fortunately I managed to strip down our database and create a simple
> single table with which the bug can be easily reproduced.
>
> I've been searching in the Postgres bug-database and this problem
> might be related to this report:
>
>         http://www.postgresql.org/bugs/bugs.php?4~111
>
> Below you find a psql-session that demonstrates the bug.
>
> I've made a dump of the test-database available as:
>
>         http://dutepp0.et.tudelft.nl/~robn/demo.dump.bz2
>
> (it is 46100 bytes long in compressed form but 45 MB when uncompressed,
>  I tried to trim it down but then the bug isn't reproducable anymore !)
>
> The table is filled with all Spaces execpt for the "town" column.
>
> Sysinfo:
> --------
>         - well-maintained Linux Red Hat 6.2
>         - kernel 2.2.18
>         - Intel Pentium III
>         - postgresql-7.0.3-2 RPMs from the Postgresql site
>           (the problem also occurs with locally rebuilt Source RPM)
>
> Any help is much appreciated !
>
>         Friendly greetings,
>         Rob van Nieuwkerk
>
> psql session:
> ***********************************************************************
> demo=> \d
>      List of relations
>     Name    | Type  | Owner
> ------------+-------+-------
>  demo_table | table | robn
> (1 row)
>
> demo=> \d demo_table
>        Table "demo_table"
>  Attribute |   Type   | Modifier
> -----------+----------+----------
>  postcode  | char(7)  |
>  odd_even  | char(1)  |
>  low       | char(5)  |
>  high      | char(5)  |
>  street    | char(24) |
>  town      | char(24) |
>  area      | char(1)  |
>
> demo=> \di
> No relations found.
> demo=> SELECT town FROM demo_table WHERE town  LIKE 'ZWO%' GROUP BY town;
>            town
> --------------------------
>  ZWOLLE
> (1 row)
>
> demo=> SELECT town FROM demo_table WHERE town  LIKE 'Z%' GROUP BY town;
>
>     <<<<<< here 86 towns are correctly found (output removed) >>>>>>
>
> demo=> CREATE INDEX demo_table_town_idx ON demo_table(town);
> CREATE
> demo=> SELECT town FROM demo_table WHERE town  LIKE 'Z%' GROUP BY town;
>  town
> ------
> (0 rows)
>         <<<<<< This is wrong !!!!!! >>>>>>>
>
> demo=> SELECT town FROM demo_table WHERE town  LIKE 'ZWO%' GROUP BY town;
>            town
> --------------------------
>  ZWOLLE
> (1 row)
>
> demo=> DROP INDEX demo_table_town_idx;
> DROP
> demo=> SELECT town FROM demo_table WHERE town  LIKE 'Z%' GROUP BY town;
>
>     <<<<<< here 86 towns are correctly found again >>>>>>
> ***********************************************************************


Thu, 18 Jan 2001 16:24:48 -0500
mlw <markw@mohawksoft.com>
Re: 7.0.3 reproduceable serious select error
> I don't think I'm running postmaster in a non-ASCII locale.
> At least I did not explicitly do anything to accomplish it.

Did you have LANG, LOCALE, or any of the LC_xxx family of
environment variables set when you started the postmaster?
Some Linux distros tend to set those in system profile scripts ...

			regards, tom lane

Thu, 18 Jan 2001 16:50:19 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: 7.0.3 reproduceable serious select error 

Hi Mark,

I just checked: the "demo.dump" file does not contain any characters
above 0x7F; it's just plain ASCII.  So that can't be the reason.

        greetings,
        Rob van Nieuwkerk


> Rob van Nieuwkerk wrote:

Ehm .., *you* wrote this ! :-)

> I tried to reproduce this bug on 7.0.2 and 7.0.3 with both 8K and 32K block
> sizes, and could not reproduce the error.
> 
> I am running RedHat 6.2 kernel 2.2.16.
> 
> I don't know enough to even be close, but I wonder if there are any subtle
> differences between the way characters are treated for indexes vs the way they
> are treated for table scans? If there are even slight differences in the way
> this happens, a misinterpretation of ascii conversions for instance, (I am
> assuming you may be using ascii characters above 0x7F), it could behave
> something like this, and explain why I wouldn't see it. .Like I said, however,
> I don't know  much so don't read too much into what I say.


> > Hello,
> >
> > I've selected postgresql 7.0.3 for our (critical) application and while
> > doing my first experiments I've found a bug which makes me worry very
> > much.
> >
> > The problem is that a SELECT with a certain LIKE condition in combination
> > with a GROUP BY does not find the proper records when there is an index on
> > the particular column present.  When the index is removed the SELECT *does*
> > return the right answer.
> >
> > Fortunately I managed to strip down our database and create a simple
> > single table with which the bug can be easily reproduced.
> >
> > I've been searching in the Postgres bug-database and this problem
> > might be related to this report:
> >
> >         http://www.postgresql.org/bugs/bugs.php?4~111
> >
> > Below you find a psql-session that demonstrates the bug.
> >
> > I've made a dump of the test-database available as:
> >
> >         http://dutepp0.et.tudelft.nl/~robn/demo.dump.bz2
> >
> > (it is 46100 bytes long in compressed form but 45 MB when uncompressed,
> >  I tried to trim it down but then the bug isn't reproducable anymore !)
> >
> > The table is filled with all Spaces execpt for the "town" column.
> >
> > Sysinfo:
> > --------
> >         - well-maintained Linux Red Hat 6.2
> >         - kernel 2.2.18
> >         - Intel Pentium III
> >         - postgresql-7.0.3-2 RPMs from the Postgresql site
> >           (the problem also occurs with locally rebuilt Source RPM)
> >
> > Any help is much appreciated !
> >
> >         Friendly greetings,
> >         Rob van Nieuwkerk
> >
> > psql session:
> > ***********************************************************************
> > demo=> \d
> >      List of relations
> >     Name    | Type  | Owner
> > ------------+-------+-------
> >  demo_table | table | robn
> > (1 row)
> >
> > demo=> \d demo_table
> >        Table "demo_table"
> >  Attribute |   Type   | Modifier
> > -----------+----------+----------
> >  postcode  | char(7)  |
> >  odd_even  | char(1)  |
> >  low       | char(5)  |
> >  high      | char(5)  |
> >  street    | char(24) |
> >  town      | char(24) |
> >  area      | char(1)  |
> >
> > demo=> \di
> > No relations found.
> > demo=> SELECT town FROM demo_table WHERE town  LIKE 'ZWO%' GROUP BY town;
> >            town
> > --------------------------
> >  ZWOLLE
> > (1 row)
> >
> > demo=> SELECT town FROM demo_table WHERE town  LIKE 'Z%' GROUP BY town;
> >
> >     <<<<<< here 86 towns are correctly found (output removed) >>>>>>
> >
> > demo=> CREATE INDEX demo_table_town_idx ON demo_table(town);
> > CREATE
> > demo=> SELECT town FROM demo_table WHERE town  LIKE 'Z%' GROUP BY town;
> >  town
> > ------
> > (0 rows)
> >         <<<<<< This is wrong !!!!!! >>>>>>>
> >
> > demo=> SELECT town FROM demo_table WHERE town  LIKE 'ZWO%' GROUP BY town;
> >            town
> > --------------------------
> >  ZWOLLE
> > (1 row)
> >
> > demo=> DROP INDEX demo_table_town_idx;
> > DROP
> > demo=> SELECT town FROM demo_table WHERE town  LIKE 'Z%' GROUP BY town;
> >
> >     <<<<<< here 86 towns are correctly found again >>>>>>
> > ***********************************************************************
> 



Thu, 18 Jan 2001 22:56:51 +0100
Rob van Nieuwkerk <robn@verdi.et.tudelft.nl>
Re: 7.0.3 reproduceable serious select error 

Tom Lane wrote:

> > I don't think I'm running postmaster in a non-ASCII locale.
> > At least I did not explicitly do anything to accomplish it.
> 
> Did you have LANG, LOCALE, or any of the LC_xxx family of
> environment variables set when you started the postmaster?
> Some Linux distros tend to set those in system profile scripts ...

Checking whith ps and looking in /proc reveiled that postmaster indeed
had LANG set to "en_US" in its environment.  I disabled the system script
that makes this setting, restarted postgres/postmaster and reran my tests.

The problem query returns the *right* answer now !
Turning LANG=en_US back on gives the old buggy behaviour.

I know very little about this LANG, LOCALE etc. stuff.
But for our application it is very important to support "weird" characters
like "���� ..." etc. for names.  Basically we need all letter symbols
in ISO-8859-1 (Latin 1).  A quick experiment shows that without the
LANG setting I can still insert & select strings containing these
symbols.

Do I lose any postgresql functionality by just getting rid of the LANG
environment variable ?  Will I be able to use full ISO-8859-1 in table
fields without problems ?

Please tell if you want me to do any other tests !

	greetings,
	Rob van Nieuwkerk

Fri, 19 Jan 2001 00:15:10 +0100
Rob van Nieuwkerk <robn@verdi.et.tudelft.nl>
Re: 7.0.3 reproduceable serious select error 
Rob van Nieuwkerk <robn@verdi.et.tudelft.nl> writes:

> I know very little about this LANG, LOCALE etc. stuff.
> But for our application it is very important to support "weird" characters
> like "ï¿½ï¿½ï¿½ï¿½ ..." etc. for names.  Basically we need all letter symbols
> in ISO-8859-1 (Latin 1). 

en_US is latin1 - this is what distinguishes it from POSIX/C.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

18 Jan 2001 18:41:43 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: 7.0.3 reproduceable serious select error
Rob van Nieuwkerk <robn@verdi.et.tudelft.nl> writes:
> Checking whith ps and looking in /proc reveiled that postmaster indeed
> had LANG set to "en_US" in its environment.  I disabled the system script
> that makes this setting, restarted postgres/postmaster and reran my tests.

> The problem query returns the *right* answer now !
> Turning LANG=en_US back on gives the old buggy behaviour.

Caution: you can't just change the locale willy-nilly, because doing so
invalidates the sort ordering of btree indexes.  An index built under
one sort order is effectively corrupt under another.  I recommend that
you dumpall, then initdb under the desired LANG setting, then reload,
and be careful always to start the postmaster under that same setting
henceforth.

(BTW, 7.1 prevents this type of index screwup by locking down the
database's locale at initdb time --- the ONLY way to change sort order
in 7.1 is to initdb with the right locale environment variables.  But in
7.0 you gotta be careful about keeping the locale consistent.)

> I know very little about this LANG, LOCALE etc. stuff.
> But for our application it is very important to support "weird" characters
> like "���� ..." etc. for names.  Basically we need all
> letter symbols in ISO-8859-1 (Latin 1).

As long as you are not expecting things to sort in any particular order,
it really doesn't matter what locale you run Postgres in.  If you do
care about sort order of characters that aren't bog-standard USASCII,
then you may have a problem.  But you can store 'em in any case.

			regards, tom lane

Thu, 18 Jan 2001 20:57:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: 7.0.3 reproduceable serious select error 
I meant to ask this the last time this came up on the list, but now is a
good time.  Given what Tom describes below as the behavior in 7.1
(initdb stores the locale info), how do you determine what locale a
database is running in in 7.1 after initdb?  Is there some file to look
at?  Is there some sql statement that can be used to select the setting
from the DB?

thanks,
--Barry


Tom Lane wrote:
> 
> Rob van Nieuwkerk <robn@verdi.et.tudelft.nl> writes:
> > Checking whith ps and looking in /proc reveiled that postmaster indeed
> > had LANG set to "en_US" in its environment.  I disabled the system script
> > that makes this setting, restarted postgres/postmaster and reran my tests.
> 
> > The problem query returns the *right* answer now !
> > Turning LANG=en_US back on gives the old buggy behaviour.
> 
> Caution: you can't just change the locale willy-nilly, because doing so
> invalidates the sort ordering of btree indexes.  An index built under
> one sort order is effectively corrupt under another.  I recommend that
> you dumpall, then initdb under the desired LANG setting, then reload,
> and be careful always to start the postmaster under that same setting
> henceforth.
> 
> (BTW, 7.1 prevents this type of index screwup by locking down the
> database's locale at initdb time --- the ONLY way to change sort order
> in 7.1 is to initdb with the right locale environment variables.  But in
> 7.0 you gotta be careful about keeping the locale consistent.)
> 
> > I know very little about this LANG, LOCALE etc. stuff.
> > But for our application it is very important to support "weird" characters
> > like "éõåÊ ..." etc. for names.  Basically we need all
> > letter symbols in ISO-8859-1 (Latin 1).
> 
> As long as you are not expecting things to sort in any particular order,
> it really doesn't matter what locale you run Postgres in.  If you do
> care about sort order of characters that aren't bog-standard USASCII,
> then you may have a problem.  But you can store 'em in any case.
> 
>                         regards, tom lane

Thu, 18 Jan 2001 18:30:25 -0800
Barry Lind <barry@xythos.com>
Re: 7.0.3 reproduceable serious select error
Barry Lind <barry@xythos.com> writes:
> I meant to ask this the last time this came up on the list, but now is a
> good time.  Given what Tom describes below as the behavior in 7.1
> (initdb stores the locale info), how do you determine what locale a
> database is running in in 7.1 after initdb?

Hm.  There probably ought to be an inquiry function or SHOW variable
for that, but at the moment there's not.  Offhand I can't think of any
direct way except to paw through the pg_control file looking for the
locale name (at least it's stored there in ASCII ;-)).

			regards, tom lane

Thu, 18 Jan 2001 21:38:13 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: 7.0.3 reproduceable serious select error 
Added to TODO:

	* Add SHOW command to see locale

> Barry Lind <barry@xythos.com> writes:
> > I meant to ask this the last time this came up on the list, but now is a
> > good time.  Given what Tom describes below as the behavior in 7.1
> > (initdb stores the locale info), how do you determine what locale a
> > database is running in in 7.1 after initdb?
> 
> Hm.  There probably ought to be an inquiry function or SHOW variable
> for that, but at the moment there's not.  Offhand I can't think of any
> direct way except to paw through the pg_control file looking for the
> locale name (at least it's stored there in ASCII ;-)).
> 
> 			regards, tom lane
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 18 Jan 2001 21:53:37 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: 7.0.3 reproduceable serious select error
Does this bug still exist?

[ Charset ISO-8859-1 unsupported, converting... ]
> Louis-David Mitterrand writes:
> 
> > When creating a child (through CREATE TABLE ... INHERIT (parent)) it
> > seems the child gets all of the parent's contraints _except_ its PRIMARY
> > KEY. Is this normal?
> 
> It's kind of a bug.
> 
> 
> -- 
> Peter Eisentraut                  Sernanders v?g 10:115
> peter_e@gmx.net                   75262 Uppsala
> http://yi.org/peter-e/            Sweden
> 
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 18 Jan 2001 23:57:07 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: child table doesn't inherit PRIMARY KEY?
Rob van Nieuwkerk wrote:
> 
> 
> The problem query returns the *right* answer now !
> Turning LANG=en_US back on gives the old buggy behaviour.
> 
> I know very little about this LANG, LOCALE etc. stuff.
> But for our application it is very important to support "weird" characters
> like "ďż˝ďż˝ďż˝ďż˝ ..." etc. for names.  Basically we need all letter symbols
> in ISO-8859-1 (Latin 1).  A quick experiment shows that without the
> LANG setting I can still insert & select strings containing these
> symbols.
> 
> Do I lose any postgresql functionality by just getting rid of the LANG
> environment variable ?  Will I be able to use full ISO-8859-1 in table
> fields without problems ?

You should, except that upper() and lower() will not give you right
answers 
for char>128 and order by orders in ASCII (i.e. character code value)
order.

I would suggest that instead you keep the en_US locale (or some nl
locale 
if you need the rigt ordering from DB), but do _not_ create a b-tree 
(the default) index on your text fields. If you need the index for 
exact lookup (field=const) an hash idex will do fine and i'm pretty sure 
that LIKE optimisations will not use them to spoil searches ;).

-------------------
Hannu

Fri, 19 Jan 2001 10:51:31 +0000
Hannu Krosing <hannu@tm.ee>
Re: 7.0.3 reproduceable serious select error
Bruce Momjian wrote:
> 
> Added to TODO:
> 
>         * Add SHOW command to see locale

I'd rather like it to be a function, as version() is, because SHOW
commands may not 
play nice with other interfaces than psql. 
(and it can first be included in ./contrib if it's too late for a
"feature" <grin>)

Just make sure we will not confict with SQL standard in naming the
function.

-------------------
Hannu

Fri, 19 Jan 2001 10:56:25 +0000
Hannu Krosing <hannu@tm.ee>
Re: 7.0.3 reproduceable serious select error
Rob van Nieuwkerk wrote:

> Hi Mark,
>
> I just checked: the "demo.dump" file does not contain any characters
> above 0x7F; it's just plain ASCII.  So that can't be the reason.
>
>         greetings,
>         Rob van Nieuwkerk
>
> > Rob van Nieuwkerk wrote:

I think I was close. ;-)

If I have followed the thread correctly, it is becauase of a language setting.




Fri, 19 Jan 2001 11:33:01 -0500
mlw <markw@mohawksoft.com>
Re: 7.0.3 reproduceable serious select error

Probably, since I see it in near recent sources (and it affects
UNIQUE as well.  As I remember it, the last discussion on this couldn't
determine what the correct behavior for unique/primary key constraints
was in the inheritance case (is it a single unique hierarchy through
all the tables [would be needed for fk to inheritance trees] or
separate unique constraints for each table [which would be similar
to how many people seem to currently use postgres inheritance as a 
shortcut]). 

On Thu, 18 Jan 2001, Bruce Momjian wrote:

> Does this bug still exist?
> 
> [ Charset ISO-8859-1 unsupported, converting... ]
> > Louis-David Mitterrand writes:
> > 
> > > When creating a child (through CREATE TABLE ... INHERIT (parent)) it
> > > seems the child gets all of the parent's contraints _except_ its PRIMARY
> > > KEY. Is this normal?


Fri, 19 Jan 2001 09:37:02 -0800 (PST)
Stephan Szabo <sszabo@megazone23.bigpanda.com>
Re: child table doesn't inherit PRIMARY KEY?

OK, what do people want to do with this item?  Add to TODO list?

Seems making a separat unique constraint would be easy to do and be of
value to most users.


> 
> Probably, since I see it in near recent sources (and it affects
> UNIQUE as well.  As I remember it, the last discussion on this couldn't
> determine what the correct behavior for unique/primary key constraints
> was in the inheritance case (is it a single unique hierarchy through
> all the tables [would be needed for fk to inheritance trees] or
> separate unique constraints for each table [which would be similar
> to how many people seem to currently use postgres inheritance as a 
> shortcut]). 
> 
> On Thu, 18 Jan 2001, Bruce Momjian wrote:
> 
> > Does this bug still exist?
> > 
> > [ Charset ISO-8859-1 unsupported, converting... ]
> > > Louis-David Mitterrand writes:
> > > 
> > > > When creating a child (through CREATE TABLE ... INHERIT (parent)) it
> > > > seems the child gets all of the parent's contraints _except_ its PRIMARY
> > > > KEY. Is this normal?
> 
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 24 Jan 2001 08:44:16 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [GENERAL] child table doesn't inherit PRIMARY KEY?
On Wed, 24 Jan 2001, Bruce Momjian wrote:

> 
> OK, what do people want to do with this item?  Add to TODO list?
> 
> Seems making a separat unique constraint would be easy to do and be of
> value to most users.

The problem is that doing that will pretty much guarantee that we won't
be doing foreign keys to inheritance trees without changing that behavior
and we've seen people asking about adding that too.  I think that this
falls into the general category of "Make inheritance make sense" (Now 
there's a todo item :) )  Seriously, I think the work on how inheritance
is going to work will decide this, maybe we end up with a real inheritance
tree system and something that works like the current stuff in which case
I'd say it's probably one unique for the former and one per for the
latter.




Wed, 24 Jan 2001 11:25:35 -0800 (PST)
Stephan Szabo <sszabo@megazone23.bigpanda.com>
Re: [GENERAL] child table doesn't inherit PRIMARY KEY?
> On Wed, 24 Jan 2001, Bruce Momjian wrote:
> 
> > 
> > OK, what do people want to do with this item?  Add to TODO list?
> > 
> > Seems making a separat unique constraint would be easy to do and be of
> > value to most users.
> 
> The problem is that doing that will pretty much guarantee that we won't
> be doing foreign keys to inheritance trees without changing that behavior
> and we've seen people asking about adding that too.  I think that this
> falls into the general category of "Make inheritance make sense" (Now 
> there's a todo item :) )  Seriously, I think the work on how inheritance
> is going to work will decide this, maybe we end up with a real inheritance
> tree system and something that works like the current stuff in which case
> I'd say it's probably one unique for the former and one per for the
> latter.

I smell TODO item.  In fact, I now see a TODO item:

* Unique index on base column not honored on inserts from inherited table
  INSERT INTO inherit_table (unique_index_col) VALUES (dup) should fail
  [inherit]

So it seems the fact the UNIQUE doesn't apply to the new table is just a
manifestion of the fact that people expect UNIQUE to span the entire
inheritance tree.  I will add the emails to [inherit] and mark it as
resolved.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 24 Jan 2001 14:31:29 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [GENERAL] child table doesn't inherit PRIMARY KEY?
Bruce Momjian wrote:
  >> On Wed, 24 Jan 2001, Bruce Momjian wrote:

  >I smell TODO item.  In fact, I now see a TODO item:
  >
  >* Unique index on base column not honored on inserts from inherited table
  >  INSERT INTO inherit_table (unique_index_col) VALUES (dup) should fail
  >  [inherit]
  >
  >So it seems the fact the UNIQUE doesn't apply to the new table is just a
  >manifestion of the fact that people expect UNIQUE to span the entire
  >inheritance tree.  I will add the emails to [inherit] and mark it as
  >resolved.

Bruce, could you add this text to TODO.detail on the subject of 
inherited constraints.  I first sent it on Christmas Eve, and I 
think most people were too busy holidaying to comment.

=================================================================
Tom Lane wrote:
  >Hm.  The short-term answer seems to be to modify the queries generated
  >by the RI triggers to say "ONLY foo".  I am not sure whether we
  >understand the semantics involved in allowing a REFERENCES target to be
  >taken as an inheritance tree rather than just one table, but certainly
  >the current implementation won't handle that correctly.

May I propose these semantics as a basis for future development:

1. An inheritance hierarchy (starting at any point in a tree) should be
equivalent to an updatable view of all the tables at the point of
reference and below.  By default, all descendant tables are combined
with the ancestor for all purposes.  The keyword ONLY must be used to
alter this behaviour.  Only inherited columns of descendant tables are
visible from higher in the tree.  Columns may not be dropped in descendants.
If columns are added to ancestors, they must be inserted correctly in
descendants so as to preserve column ordering and inheritance.  If
a column is dropped in an ancestor, it is dropped in all descendants.

2. Insertion into a hierarchy means insertion into the table named in
the INSERT statement; updating or deletion affects whichever table(s)
the affected rows are found in.  Updating cannot move a row from one
table to another.

3. Inheritance of a table implies inheriting all its constraints unless
ONLY is used or the constraints are subsequently dropped; again, dropping
operates through all descendant tables.  A primary key, foreign key or
unique constraint cannot be dropped or modified for a descendant.  A
unique index on a column is shared by all tables below the table for
which it is declared.  It cannot be dropped for any descendant.

In other words, only NOT NULL and CHECK constraints can be dropped in
descendants.

In multiple inheritance, a column may inherit multiple unique indices
from its several ancestors.  All inherited constraints must be satisfied
together (though check constraints may be dropped).

4. RI to a table implies the inclusion of all its descendants in the
check.  Since a referenced column may be uniquely indexed further up
the hierarchy than in the table named, the check must ensure that
the referenced value occurs in the right segment of the hierarchy.  RI
to one particular level of the hierarchy, excluding descendants, requires
the use of ONLY in the constraint.

5. Dropping a table implies dropping all its descendants.

6. Changes of permissions on a table propagate to all its descendants.
Permissions on descendants may be looser than those on ancestors; they
may not be more restrictive.


This scheme is a lot more restrictive than C++'s or Eiffel's definition
of inheritance, but it seems to me to make the concept truly useful,
without introducing excessive complexity.

============================================================

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "If anyone has material possessions and sees his
      brother in need but has no pity on him, how can the
      love of God be in him?"
                                    I John 3:17 



Wed, 24 Jan 2001 21:41:39 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: Re: [GENERAL] child table doesn't inherit PRIMARY KEY? 
Contrary to what the submitted documentation claims, there is no
permission checking done on the CHECKPOINT command.  Should there be?

Btw., is there any normal usage application of this command?  This relates
to the previous paragraph somewhat.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Wed, 24 Jan 2001 23:02:09 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Permissions on CHECKPOINT
Peter Eisentraut wrote:
  >Contrary to what the submitted documentation claims, there is no
  >permission checking done on the CHECKPOINT command.  Should there be?
  
Vadim seemed to indicate that he was going to make that restriction.
Perhaps I misunderstood.

If it's too late to make the change for 7.1, the fact should be
documented in a Bug section of the man page.

  >Btw., is there any normal usage application of this command?  This relates
  >to the previous paragraph somewhat.
  >
  >-- 
  >Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/
  >

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "If anyone has material possessions and sees his
      brother in need but has no pity on him, how can the
      love of God be in him?"
                                    I John 3:17 



Wed, 24 Jan 2001 22:14:53 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: Permissions on CHECKPOINT 

Thanks.  Done.

> Bruce Momjian wrote:
>   >> On Wed, 24 Jan 2001, Bruce Momjian wrote:
> 
>   >I smell TODO item.  In fact, I now see a TODO item:
>   >
>   >* Unique index on base column not honored on inserts from inherited table
>   >  INSERT INTO inherit_table (unique_index_col) VALUES (dup) should fail
>   >  [inherit]
>   >
>   >So it seems the fact the UNIQUE doesn't apply to the new table is just a
>   >manifestion of the fact that people expect UNIQUE to span the entire
>   >inheritance tree.  I will add the emails to [inherit] and mark it as
>   >resolved.
> 
> Bruce, could you add this text to TODO.detail on the subject of 
> inherited constraints.  I first sent it on Christmas Eve, and I 
> think most people were too busy holidaying to comment.
> 
> =================================================================
> Tom Lane wrote:
>   >Hm.  The short-term answer seems to be to modify the queries generated
>   >by the RI triggers to say "ONLY foo".  I am not sure whether we
>   >understand the semantics involved in allowing a REFERENCES target to be
>   >taken as an inheritance tree rather than just one table, but certainly
>   >the current implementation won't handle that correctly.
> 
> May I propose these semantics as a basis for future development:
> 
> 1. An inheritance hierarchy (starting at any point in a tree) should be
> equivalent to an updatable view of all the tables at the point of
> reference and below.  By default, all descendant tables are combined
> with the ancestor for all purposes.  The keyword ONLY must be used to
> alter this behaviour.  Only inherited columns of descendant tables are
> visible from higher in the tree.  Columns may not be dropped in descendants.
> If columns are added to ancestors, they must be inserted correctly in
> descendants so as to preserve column ordering and inheritance.  If
> a column is dropped in an ancestor, it is dropped in all descendants.
> 
> 2. Insertion into a hierarchy means insertion into the table named in
> the INSERT statement; updating or deletion affects whichever table(s)
> the affected rows are found in.  Updating cannot move a row from one
> table to another.
> 
> 3. Inheritance of a table implies inheriting all its constraints unless
> ONLY is used or the constraints are subsequently dropped; again, dropping
> operates through all descendant tables.  A primary key, foreign key or
> unique constraint cannot be dropped or modified for a descendant.  A
> unique index on a column is shared by all tables below the table for
> which it is declared.  It cannot be dropped for any descendant.
> 
> In other words, only NOT NULL and CHECK constraints can be dropped in
> descendants.
> 
> In multiple inheritance, a column may inherit multiple unique indices
> from its several ancestors.  All inherited constraints must be satisfied
> together (though check constraints may be dropped).
> 
> 4. RI to a table implies the inclusion of all its descendants in the
> check.  Since a referenced column may be uniquely indexed further up
> the hierarchy than in the table named, the check must ensure that
> the referenced value occurs in the right segment of the hierarchy.  RI
> to one particular level of the hierarchy, excluding descendants, requires
> the use of ONLY in the constraint.
> 
> 5. Dropping a table implies dropping all its descendants.
> 
> 6. Changes of permissions on a table propagate to all its descendants.
> Permissions on descendants may be looser than those on ancestors; they
> may not be more restrictive.
> 
> 
> This scheme is a lot more restrictive than C++'s or Eiffel's definition
> of inheritance, but it seems to me to make the concept truly useful,
> without introducing excessive complexity.
> 
> ============================================================
> 
> -- 
> Oliver Elphick                                Oliver.Elphick@lfix.co.uk
> Isle of Wight                              http://www.lfix.co.uk/oliver
> PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
> GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
>                  ========================================
>      "If anyone has material possessions and sees his
>       brother in need but has no pity on him, how can the
>       love of God be in him?"
>                                     I John 3:17 
> 
> 
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 24 Jan 2001 18:56:06 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: [GENERAL] child table doesn't inherit PRIMARY KEY?
>   >Contrary to what the submitted documentation claims, there is no
>   >permission checking done on the CHECKPOINT command.  
> Should there be?
>   
> Vadim seemed to indicate that he was going to make that restriction.
> Perhaps I misunderstood.

Yes, there should be permission checking - I'll add it later (in 7.1)
if no one else.

Vadim

Wed, 24 Jan 2001 16:03:23 -0800
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
RE: Permissions on CHECKPOINT 
Mikheev, Vadim writes:

> Yes, there should be permission checking - I'll add it later (in 7.1)
> if no one else.

Should be simple enough.  Is this okay:

Index: utility.c
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/backend/tcop/utility.c,v
retrieving revision 1.105
diff -c -r1.105 utility.c
*** utility.c   2001/01/05 06:34:20     1.105
--- utility.c   2001/01/25 16:40:40
***************
*** 18,23 ****
--- 18,24 ----

  #include "access/heapam.h"
  #include "catalog/catalog.h"
+ #include "catalog/pg_shadow.h"
  #include "commands/async.h"
  #include "commands/cluster.h"
  #include "commands/command.h"
***************
*** 851,856 ****
--- 852,859 ----
                        {
                                set_ps_display(commandTag = "CHECKPOINT");

+                               if (!superuser())
+                                       elog(ERROR, "permission denied");
                                CreateCheckPoint(false);
                        }
                        break;


-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 25 Jan 2001 17:49:51 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
RE: Permissions on CHECKPOINT 
Peter Eisentraut <peter_e@gmx.net> writes:
> Mikheev, Vadim writes:
>> Yes, there should be permission checking - I'll add it later (in 7.1)
>> if no one else.

> Should be simple enough.  Is this okay:

Actually, I think a more interesting question is "should CHECKPOINT
have permission restrictions?  If so, what should they be?"

A quite relevant precedent is that Unix systems (at least the ones
I've used) do not restrict who can call sync().

			regards, tom lane

Fri, 26 Jan 2001 00:08:25 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Permissions on CHECKPOINT 

 Hi all,

 Re-posting this  to -hackers. Will PQprint()  remain/disappear/be replaced
in the future?

 Thx

 Ed


--

 ������������������������������������


Fri, 26 Jan 2001 05:20:01 +0000
KuroiNeko <evpopkov@carrier.kiev.ua>
PQprint
Tom Lane wrote:
  >Peter Eisentraut <peter_e@gmx.net> writes:
  >> Mikheev, Vadim writes:
  >>> Yes, there should be permission checking - I'll add it later (in 7.1)
  >>> if no one else.
  >
  >> Should be simple enough.  Is this okay:
  >
  >Actually, I think a more interesting question is "should CHECKPOINT
  >have permission restrictions?  If so, what should they be?"
  >
  >A quite relevant precedent is that Unix systems (at least the ones
  >I've used) do not restrict who can call sync().

What about DoS attacks?  What would be the effect of someone's setting
off an infinite loop of CHECKPOINTs?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Wash me thoroughly from mine iniquity, and cleanse me 
      from my sin. For I acknowledge my transgressions; and 
      my sin is ever before me. Against thee, thee only, 
      have I sinned, and done this evil in thy sight..."
                                   Psalms 51:2-4 



Fri, 26 Jan 2001 05:41:41 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: Permissions on CHECKPOINT 
The 'tmpwatch' program on Red Hat will remove the /tmp/.s.PGSQL.5432.lock
file after the server has run 6 days.  This will be a problem.

We could touch (open) the file once every time the ServerLoop() runs
around.  It's not perfect but it should work in practice.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Fri, 26 Jan 2001 21:17:32 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Sure enough, the lock file is gone
> The 'tmpwatch' program on Red Hat will remove the /tmp/.s.PGSQL.5432.lock
> file after the server has run 6 days.  This will be a problem.
> 
> We could touch (open) the file once every time the ServerLoop() runs
> around.  It's not perfect but it should work in practice.

If we have to do it, let's make it an #ifdef __linux__ option.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 26 Jan 2001 15:18:13 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Sure enough, the lock file is gone
On Fri, Jan 26, 2001 at 03:18:13PM -0500, Bruce Momjian wrote:
> > The 'tmpwatch' program on Red Hat will remove the /tmp/.s.PGSQL.5432.lock
> > file after the server has run 6 days.  This will be a problem.
> > 
> > We could touch (open) the file once every time the ServerLoop() runs
> > around.  It's not perfect but it should work in practice.
> 
> If we have to do it, let's make it an #ifdef __linux__ option.

#ifdef BRAINDAMAGED_TMP_CLEANER ?

ISTR mention of non-linux platforms that do this.

Ross

Fri, 26 Jan 2001 14:24:34 -0600
"Ross J. Reedstrom" <reedstrm@rice.edu>
Re: Sure enough, the lock file is gone
* Peter Eisentraut <peter_e@gmx.net> [010126 12:11] wrote:
> The 'tmpwatch' program on Red Hat will remove the /tmp/.s.PGSQL.5432.lock
> file after the server has run 6 days.  This will be a problem.
> 
> We could touch (open) the file once every time the ServerLoop() runs
> around.  It's not perfect but it should work in practice.

Why not have the RPM/configure scripts stick it in where ever redhat
says it's safe to?

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."

Fri, 26 Jan 2001 12:49:30 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: Sure enough, the lock file is gone
Ross J. Reedstrom wrote:
> On Fri, Jan 26, 2001 at 03:18:13PM -0500, Bruce Momjian wrote:
> > > The 'tmpwatch' program on Red Hat will remove the /tmp/.s.PGSQL.5432.lock
> > > file after the server has run 6 days.  This will be a problem.
> > >
> > > We could touch (open) the file once every time the ServerLoop() runs
> > > around.  It's not perfect but it should work in practice.
> >
> > If we have to do it, let's make it an #ifdef __linux__ option.
>
> #ifdef BRAINDAMAGED_TMP_CLEANER ?
>
> ISTR mention of non-linux platforms that do this.

    Exactly  the way you want it to do (open(2) and close(2) of a
    UNIX domain socket) was what I  had  to  do  to  get  an  old
    Mach3-4.3BSD combo into a kernel-panic.

    Better use utime(2) or the like for it.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com


Fri, 26 Jan 2001 15:50:39 -0500 (EST)
Jan Wieck <janwieck@Yahoo.com>
Re: Sure enough, the lock file is gone
Bruce Momjian writes:

> If we have to do it, let's make it an #ifdef __linux__ option.

What does Linux have to do with it?  FreeBSD does the same thing, only
every three days.  I dont' know whether it's not enabled on a fresh
install, but it's there, you only need to flip the switch.  I doubt /tmp
cleaning is such an unusual thing, especially on large sites.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Fri, 26 Jan 2001 22:04:42 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Sure enough, the lock file is gone
Jan Wieck writes:

>     Exactly  the way you want it to do (open(2) and close(2) of a
>     UNIX domain socket) was what I  had  to  do  to  get  an  old
>     Mach3-4.3BSD combo into a kernel-panic.

The lock file is an ordinary file.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Fri, 26 Jan 2001 22:05:11 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Sure enough, the lock file is gone
Peter Eisentraut <peter_e@gmx.net> writes:
> Bruce Momjian writes:
>> If we have to do it, let's make it an #ifdef __linux__ option.

> What does Linux have to do with it?  FreeBSD does the same thing, only
> every three days.  I dont' know whether it's not enabled on a fresh
> install, but it's there, you only need to flip the switch.  I doubt /tmp
> cleaning is such an unusual thing, especially on large sites.

Yes, there are lots of systems that will clean /tmp --- and since the
lock file is an ordinary file (not a socket) pretty much any tmp-cleaner
is going to decide to remove it.  I think that I had intended to insert
a periodic touch of the lockfile and forgot to.

Touching it every time through ServerLoop is an overreaction though.
I'd suggest touching it in the checkpoint-process-firing code, which
runs every five minutes (or so?) by default.

			regards, tom lane

Fri, 26 Jan 2001 16:49:12 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Sure enough, the lock file is gone 
Peter Eisentraut wrote:
> Jan Wieck writes:
>
> >     Exactly  the way you want it to do (open(2) and close(2) of a
> >     UNIX domain socket) was what I  had  to  do  to  get  an  old
> >     Mach3-4.3BSD combo into a kernel-panic.
>
> The lock file is an ordinary file.

    So  the  crazy-temp-vacuum-cleaner on linux doesn't touch the
    sockets?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com


Fri, 26 Jan 2001 17:06:24 -0500 (EST)
Jan Wieck <janwieck@Yahoo.com>
Re: Sure enough, the lock file is gone
> Tom Lane wrote:
>   >Peter Eisentraut <peter_e@gmx.net> writes:
>   >> Mikheev, Vadim writes:
>   >>> Yes, there should be permission checking - I'll add it later (in 7.1)
>   >>> if no one else.
>   >
>   >> Should be simple enough.  Is this okay:
>   >
>   >Actually, I think a more interesting question is "should CHECKPOINT
>   >have permission restrictions?  If so, what should they be?"
>   >
>   >A quite relevant precedent is that Unix systems (at least the ones
>   >I've used) do not restrict who can call sync().
> 
> What about DoS attacks?  What would be the effect of someone's setting
> off an infinite loop of CHECKPOINTs?

Don't we have bigger DoS attacks?  Certainly SELECT cash_out(1) is a
much bigger one.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 26 Jan 2001 18:17:28 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Permissions on CHECKPOINT
I said:
> Yes, there are lots of systems that will clean /tmp --- and since the
> lock file is an ordinary file (not a socket) pretty much any tmp-cleaner
> is going to decide to remove it.  I think that I had intended to insert
> a periodic touch of the lockfile and forgot to.

Done now.

			regards, tom lane

Fri, 26 Jan 2001 19:06:29 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Sure enough, the lock file is gone 
> 
>  Hi all,
> 
>  Re-posting this  to -hackers. Will PQprint()  remain/disappear/be replaced
> in the future?

No idea.  We are not sure who uses it.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 26 Jan 2001 23:28:01 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: PQprint
> On Fri, Jan 26, 2001 at 03:18:13PM -0500, Bruce Momjian wrote:
> > > The 'tmpwatch' program on Red Hat will remove the /tmp/.s.PGSQL.5432.lock
> > > file after the server has run 6 days.  This will be a problem.
> > > 
> > > We could touch (open) the file once every time the ServerLoop() runs
> > > around.  It's not perfect but it should work in practice.
> > 
> > If we have to do it, let's make it an #ifdef __linux__ option.
> 
> #ifdef BRAINDAMAGED_TMP_CLEANER ?
> 
> ISTR mention of non-linux platforms that do this.

Yes, thank you.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 26 Jan 2001 23:52:51 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Sure enough, the lock file is gone
> I said:
> > Yes, there are lots of systems that will clean /tmp --- and since the
> > lock file is an ordinary file (not a socket) pretty much any tmp-cleaner
> > is going to decide to remove it.  I think that I had intended to insert
> > a periodic touch of the lockfile and forgot to.
> 
> Done now.

Yes, checkpoint is a good place to put it.  Thanks.  I still liked the
BRAINDAMAGED_TMP_CLEANER though.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 26 Jan 2001 23:55:16 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Sure enough, the lock file is gone
On Fri, Jan 26, 2001 at 05:06:24PM -0500, Jan Wieck wrote:
>     So  the  crazy-temp-vacuum-cleaner on linux doesn't touch the
>     sockets?

The tmpwatch program that comes with many Linux distributions will only
unlink regular files and empty directories by default.
-- 
Bruce Guenter <bruceg@em.ca>                       http://em.ca/~bruceg/
Fri, 26 Jan 2001 23:45:08 -0600
Bruce Guenter <bruceg@em.ca>
Re: Sure enough, the lock file is gone
On Fri, 26 Jan 2001, Peter Eisentraut wrote:
>Bruce Momjian writes:
>
>> If we have to do it, let's make it an #ifdef __linux__ option.
>
>What does Linux have to do with it?  FreeBSD does the same thing, only
>every three days.  I dont' know whether it's not enabled on a fresh
>install, but it's there, you only need to flip the switch.  I doubt /tmp
>cleaning is such an unusual thing, especially on large sites.
>

Only on a poorly configured FreeBSD box.  You do have to turn it on first.
FreeBSD (This is a 4.2-Stable box) will only delete files that have not been
modified within the set number of days.  This amount is variable.  You can also
tell clean_tmp to ignore any files you wish.  This is all configurable via
rc.conf and friends.

GB

--  
GB Clark II            | Roaming FreeBSD Admin
gclarkii@GeekTech.COM  | General Geek 
           CTHULU for President - Why choose the lesser of two evils?

Sat, 27 Jan 2001 05:07:49 -0600
GB Clark II <gclarkii@geektech.com>
Re: Sure enough, the lock file is gone
Jan Wieck wrote:
  >Peter Eisentraut wrote:
  >> Jan Wieck writes:
  >>
  >> >     Exactly  the way you want it to do (open(2) and close(2) of a
  >> >     UNIX domain socket) was what I  had  to  do  to  get  an  old
  >> >     Mach3-4.3BSD combo into a kernel-panic.
  >>
  >> The lock file is an ordinary file.
  >
  >    So  the  crazy-temp-vacuum-cleaner on linux doesn't touch the
  >    sockets?
 
tmpreaper does - that's why I moved the socket in Debian.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Come now, and let us reason together, saith the LORD; 
      though your sins be as scarlet, they shall be as white
      as snow; though they be red like crimson, they shall 
      be as wool."                     Isaiah 1:18 



Sat, 27 Jan 2001 12:29:28 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: Sure enough, the lock file is gone 
/tmp is for *temporary* files. Such a lock is not a temporary file, it
should go somewhere in /var, why not in /var/lib/pgsql/data ?


> The 'tmpwatch' program on Red Hat will remove the /tmp/.s.PGSQL.5432.lock
> file after the server has run 6 days.  This will be a problem.
> 
> We could touch (open) the file once every time the ServerLoop() runs
> around.  It's not perfect but it should work in practice.

Sat, 27 Jan 2001 18:34:26 +0100
Florent Guillaume <efgeor@noos.fr>
Re: Sure enough, the lock file is gone
Florent Guillaume:

> /tmp is for *temporary* files. Such a lock is not a temporary file,
> it should go somewhere in /var, why not in /var/lib/pgsql/data ?

/var/run ?

-- 
Alessio F. Bragadini		alessio@sevenseas.org

Sat, 27 Jan 2001 19:51:54 +0200
Alessio Bragadini <alessio@sevenseas.org>
Re: Re: Sure enough, the lock file is gone
> Jan Wieck wrote:
>   >Peter Eisentraut wrote:
>   >> Jan Wieck writes:
>   >>
>   >> >     Exactly  the way you want it to do (open(2) and close(2) of a
>   >> >     UNIX domain socket) was what I  had  to  do  to  get  an  old
>   >> >     Mach3-4.3BSD combo into a kernel-panic.
>   >>
>   >> The lock file is an ordinary file.
>   >
>   >    So  the  crazy-temp-vacuum-cleaner on linux doesn't touch the
>   >    sockets?
>  
> tmpreaper does - that's why I moved the socket in Debian.

But you have complete control over the OS, while we don't.  The problem
I see of moving it is that only Debian-compiled clients will work on
Debian systems.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 27 Jan 2001 13:17:01 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Sure enough, the lock file is gone
Florent Guillaume wrote:
> 
> /tmp is for *temporary* files. Such a lock is not a temporary file, it
> should go somewhere in /var, why not in /var/lib/pgsql/data ?

/var/lib is also not for locks, per FHS.

/var/lock/pgsql (or /var/lock/postgresql....) would be the FHS-mandated
place for such a file.

Comments?  _Why_ is the lock in /tmp?  Won't the lock always be put into
place by the uid used to run postmaster?  Is a _world_ writeable
temporary directory the right place?

7.2 discussion, however, IMHO.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 27 Jan 2001 14:09:11 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone

first off, the lock file is put in by an unprivileged user, so /tmp works
on all systems ...

second, /tmp on a large portion of systems gets cleaned out after a
reboot, so there are no 'stray locks' to generally worry about...


On Sat, 27 Jan 2001, Lamar Owen wrote:

> Florent Guillaume wrote:
> >
> > /tmp is for *temporary* files. Such a lock is not a temporary file, it
> > should go somewhere in /var, why not in /var/lib/pgsql/data ?
>
> /var/lib is also not for locks, per FHS.
>
> /var/lock/pgsql (or /var/lock/postgresql....) would be the FHS-mandated
> place for such a file.
>
> Comments?  _Why_ is the lock in /tmp?  Won't the lock always be put into
> place by the uid used to run postmaster?  Is a _world_ writeable
> temporary directory the right place?
>
> 7.2 discussion, however, IMHO.
> --
> Lamar Owen
> WGCR Internet Radio
> 1 Peter 4:11
>

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org


Sat, 27 Jan 2001 15:14:13 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re: Sure enough, the lock file is gone
The Hermit Hacker wrote:
> On Sat, 27 Jan 2001, Lamar Owen wrote:
> > Comments?  _Why_ is the lock in /tmp?  Won't the lock always be put into
> > place by the uid used to run postmaster?  Is a _world_ writeable
> > temporary directory the right place?
 
> first off, the lock file is put in by an unprivileged user, so /tmp works
> on all systems ...

If /usr/local/pgsql (to use the default) is owned by the user running
postmaster, then the postmaster has privileges to put the lockfile in,
say, /usr/local/pgsql/lock/....., right? Or am I missing something basic
here?  Is this lock placed by postmaster, or by something else?  My
7.1beta3 installation shows two files in /tmp:
srwxrwxrwx    1 postgres postgres        0 Jan 27 14:25 .s.PGSQL.5432
-rw-------    1 postgres postgres       25 Jan 27 14:25
.s.PGSQL.5432.lock

I understand why the socket needs to be in /tmp, but why the lockfile? 
What or who is using the lockfile (which contains the pid of postmaster
and the path to PGDATA for the postmaster)?
 
> second, /tmp on a large portion of systems gets cleaned out after a
> reboot, so there are no 'stray locks' to generally worry about...

Ironic that RedHat, which can clean /tmp out on a cron basis would be
one that doesn't clean it out by default on reboot.

Lock file cleanup should be the responsibility of the script that starts
postmaster -- or the responsibility of the DBA who manually starts and
restarts postmasters, after crashes or at other times.

Not a big issue, by any means. Just attempting to understand.  
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 27 Jan 2001 14:28:32 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
Peter Eisentraut wrote:
> Lamar Owen writes:
> > I understand why the socket needs to be in /tmp, but why the lockfile?
 
> The lock file protects the Unix domain socket.  Consequently, the name of
> the lock file needs to be derivable from the name of the socket file, and
> vice versa.  Also, the name of the socket file must not vary with other
> parameters such as installation layout.

I see.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 27 Jan 2001 14:56:36 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
Lamar Owen writes:

> I understand why the socket needs to be in /tmp, but why the lockfile?

The lock file protects the Unix domain socket.  Consequently, the name of
the lock file needs to be derivable from the name of the socket file, and
vice versa.  Also, the name of the socket file must not vary with other
parameters such as installation layout.

> Lock file cleanup should be the responsibility of the script that starts
> postmaster

The postmaster does that itself.  That's why the pid is in there.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 27 Jan 2001 20:58:59 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Sure enough, the lock file is gone
> I understand why the socket needs to be in /tmp, but why the lockfile?

It would probably be better if the socket files weren't in /tmp but in
a postgres-owned directory.  However, at this point we have a huge
backwards compatibility problem to overcome if we want to move the
socket files.  The location of the socket files is essentially a core
part of the frontend-backend protocol, because both client and server
must know it ab initio.  Move the socket, break your clients.

There is an option in 7.1 to support defining a different directory
for the socket files, but I doubt very many people will use it.

I see no real good reason to keep the lockfiles in a different place
from the sockets themselves, however.  Doing so would just complicate
things even more, without adding any real safety or security.

			regards, tom lane

Sat, 27 Jan 2001 15:25:31 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
Bruce Momjian wrote:
  >> Note: programs that run as non-root users may be unable to create files un
      >der 
  >> /var/run and therefore need a subdirectory owned by the appropriate user.
  >
  >This is the killer.  We can't require root.  Seems we are stuck with
  >/tmp.
 
I'd be surprised to learn that non-admin users are allowed to write
in /usr/local, either.  If, on some machine, PostgreSQL is such an
unoffical project that the admin won't agree to create /var/run/postgresql,
the user can define his own temporary directory using the method we have
already included in 7.1; if, on the other hand, he is able to create
/usr/local/pgsql, he will also be able to create /var/run/postgresql.

Really, how many users do we have who can't get their admin to do
this for them?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Come now, and let us reason together, saith the LORD; 
      though your sins be as scarlet, they shall be as white
      as snow; though they be red like crimson, they shall 
      be as wool."                     Isaiah 1:18 



Sat, 27 Jan 2001 22:05:05 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: Sure enough, the lock file is gone 
> It would probably be better if the socket files weren't in /tmp but in
> a postgres-owned directory.  However, at this point we have a huge
> backwards compatibility problem to overcome if we want to move the
> socket files.  The location of the socket files is essentially a core
> part of the frontend-backend protocol, because both client and server
> must know it ab initio.  Move the socket, break your clients.

Ok, fair enough.

But sometimes unix sucks, don't you think, having to use /tmp as a
central place for inter-process communication... blech.

Florent

-- 
<florent.guillaume@mail.com>

Sun, 28 Jan 2001 00:30:15 +0100
Florent Guillaume <efgeor@noos.fr>
Re: Sure enough, the lock file is gone
"Oliver Elphick" <olly@lfix.co.uk> writes:
> /var/run/postgresql

There's another reason why the standard socket directory is /tmp,
and that's that it exists everywhere.  Not all Unix systems even *have*
a /var hierarchy, let alone one that the admin will let you have a
playpen in.

			regards, tom lane

Sat, 27 Jan 2001 19:05:35 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Sure enough, the lock file is gone 
Tom Lane <tgl@sss.pgh.pa.us> writes:

> It would probably be better if the socket files weren't in /tmp but in
> a postgres-owned directory.  However, at this point we have a huge
> backwards compatibility problem to overcome if we want to move the
> socket files.

Not to sound scheptical, but since when did postgresql care about
backwards compatiblity? Upgrading is already demanding a lot of
knowledge from the user (including needing such information, which
almost no other package do), this is just a minor change (the files
are mostly used by bundled tools - any exceptions?)

> There is an option in 7.1 to support defining a different directory
> for the socket files, but I doubt very many people will use it.

I intend to, for the RPMs we ship.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

28 Jan 2001 12:49:20 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: Re: Sure enough, the lock file is gone
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:
> Not to sound scheptical, but since when did postgresql care about
> backwards compatiblity? Upgrading is already demanding a lot of
> knowledge from the user (including needing such information, which
> almost no other package do), this is just a minor change (the files
> are mostly used by bundled tools - any exceptions?)

It is not a minor change.  Pay attention now: moving the socket files
BREAKS THE CLIENT/SERVER PROTOCOL.  Got that?  If you do this, then
no old client program will work against your new server until you
recompile/relink it.

Don't forget, also, that clients built using the standard distribution
won't talk to your server either.  Unless you have *sole* control of
Postgres and Postgres clients on your machine, moving the socket files
is going to create problems.

We have seen enough complaints in the past due to changes in the
protocol (all of which were backwards-compatible, BTW; this isn't)
that I'm pretty hesitant to do another one on such minor grounds as
getting the socket file out of /tmp.

>> There is an option in 7.1 to support defining a different directory
>> for the socket files, but I doubt very many people will use it.

> I intend to, for the RPMs we ship.

Expect complaints.

I wouldn't be so annoyed at this, if I didn't know very well that the
resulting questions/complaints will come to the Postgres team, and not
to the perpetrator of the incompatibility.  Shall I forward all future
"can't connect to server" bug reports to you to answer?

			regards, tom lane

Sun, 28 Jan 2001 13:22:26 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
Trond Eivind Glomsrï¿½d wrote:
> 
> Tom Lane <tgl@sss.pgh.pa.us> writes:
> 
> > It would probably be better if the socket files weren't in /tmp but in
> > a postgres-owned directory.  However, at this point we have a huge
> > backwards compatibility problem to overcome if we want to move the
> > socket files.
> 
> Not to sound scheptical, but since when did postgresql care about
> backwards compatiblity? Upgrading is already demanding a lot of
> knowledge from the user (including needing such information, which
> almost no other package do), this is just a minor change (the files
> are mostly used by bundled tools - any exceptions?)

Upgrading is only one facet of backwards compatibility.  When the fe-be
protocol was changed for 6.4.2 is a good example.  The SQL itself is
kept very backwards-compatible, on purpose. Things for
backwards-compatibility are not as bad as the upgrading situation would
seem to imply.

> > There is an option in 7.1 to support defining a different directory
> > for the socket files, but I doubt very many people will use it.
 
> I intend to, for the RPMs we ship.

Ok, why not fix tmpwatch instead?  Only the lock files break FHS -- the
sockets can go there by FHS, right?  Of course, our requirement that
they be in the same location sort of forces the issue.  But, 7.1 now
touches the locks enought to keep tmpwatch from blowing them out.

To where do you intend to move them to?  /var/lock/pgsql? 
/var/run/pgsql? (Or postgresql... I'm still not happy with that change
-- the configuration is much nicer, but now the 'postgresql' suffix is
fixed -- I'm probably going to have to patch that to pgsql, as I'm
already changing many things that I'd prefer to leave closer to what 7.0
had).

The change in question is the use of '/usr/share/postgresql' and
'/usr/include/postgresql' as part of the installation, rather than
allowing '/usr/share/pgsql' and '/usr/include/pgsql' .

O well -- I'm just going to have to see how it distills. I've not
received any complaints yet, but I expect many after final. :-(
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sun, 28 Jan 2001 15:54:00 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
Lamar Owen <lamar.owen@wgcr.org> writes:

> Trond Eivind Glomsrï¿½d wrote:
> > 
> > Tom Lane <tgl@sss.pgh.pa.us> writes:
> > 
> > > It would probably be better if the socket files weren't in /tmp but in
> > > a postgres-owned directory.  However, at this point we have a huge
> > > backwards compatibility problem to overcome if we want to move the
> > > socket files.
> > 
> > Not to sound scheptical, but since when did postgresql care about
> > backwards compatiblity? Upgrading is already demanding a lot of
> > knowledge from the user (including needing such information, which
> > almost no other package do), this is just a minor change (the files
> > are mostly used by bundled tools - any exceptions?)
> 
> Upgrading is only one facet of backwards compatibility.

I know. I just mentioned one consistently painful aspect of it.

> > > There is an option in 7.1 to support defining a different directory
> > > for the socket files, but I doubt very many people will use it.
>  
> > I intend to, for the RPMs we ship.
> 
> Ok, why not fix tmpwatch instead?

Because it wouldn't be a fix, it would be a "lets workaround one
specific app which does things in a bad way"-hack. /tmp isn't supposed
be more than that... storing things there for more than than 10 days?
Ouch.  

> Only the lock files break FHS 

Explictly, yes. However, FHS says /tmp is for temporary files. Also,
it says programs shouldn't count on data to be stored there between
invocations. 10+ days isn't temporary...
 
> To where do you intend to move them to?  /var/lock/pgsql? 
> /var/run/pgsql? 

Ideally, the locks should be in /var/lock/pgsql and the socket
somewhere else - like /var/lib/pgsql (our mysql packages do this, and
both of them are specified in /etc/my.cnf).

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

28 Jan 2001 16:07:09 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: Re: Sure enough, the lock file is gone
Peter Eisentraut wrote [re using rules to guard against unprivileged
table creation]:
  >It couldn't, because the CREATE TABLE code does not go through the rule
  >system.

Could it not be done by enforcing access control on system tables?  At
present this is partially supported.  Perversely, I can deny select
privilege to pg_class but cannot deny insert privilege:


junk=# revoke all on pg_class from public;
CHANGE
junk=# \d            
          List of relations
       Name       |   Type   | Owner 
------------------+----------+-------
 a                | table    | olly
...
(14 rows)
junk=# \c - ruth
You are now connected as new user ruth.
junk=> \d
ERROR:  pg_class: Permission denied.
junk=> create table xx (id int);
CREATE
junk=> \c - olly
You are now connected as new user olly.
junk=# \d
          List of relations
       Name       |   Type   | Owner 
------------------+----------+-------
 a                | table    | olly
...
 xx               | table    | ruth
(15 rows)


If the denial of write privilege were enforced, it would not be possible
for an unprivileged user to create tables.  When a database is created,
all the system tables should be made read only for PUBLIC.  As a corollary,
when a write privilege is granted on a table, it may be necessary to
give concomitant privilege on tables needed to update sequences and other
such items (I can't think of any others, at the moment), or else by-pass
privilege checking on these.


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Many are the afflictions of the righteous; but the 
      LORD delivereth him out of them all."             
                                        Psalm 34:19 



Sun, 28 Jan 2001 21:25:53 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [ADMIN] Controlling user table creation 
Lamar Owen wrote:
  >Ok, why not fix tmpwatch instead?  Only the lock files break FHS -- the
  >sockets can go there by FHS, right? 

No, UNIX sockets are specifically mentioned as belonging under /var/run.
In section 5.10 "/var/run : Run-time variable data", it says: "Programs
that maintain transient UNIX-domain sockets should place them in this 
directory."

So what ever the outcome for the wider PostgreSQL community, I must make
the change to conform to Debian policy.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Many are the afflictions of the righteous; but the 
      LORD delivereth him out of them all."             
                                        Psalm 34:19 



Sun, 28 Jan 2001 21:38:38 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: Re: Sure enough, the lock file is gone 
Trond Eivind Glomsrï¿½d wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > Trond Eivind Glomsrï¿½d wrote:
> > > Not to sound scheptical, but since when did postgresql care about
> > > backwards compatiblity? Upgrading is already demanding a lot of

> > Upgrading is only one facet of backwards compatibility.
 
> I know. I just mentioned one consistently painful aspect of it.

Your pain is evident from the hyperbolic statement.  I share your pain.

> However, FHS says /tmp is for temporary files. Also,
> it says programs shouldn't count on data to be stored there between
> invocations. 10+ days isn't temporary...

But postmaster _doesn't_ expect the files to stay _between_
invocations.... :-)  But your point is understood.

What about the X sockets, then?  X might stay up 10+ days.  X doesn't
just put one file there, either -- there's a whole directory there in
/tmp.
 
> > To where do you intend to move them to?  /var/lock/pgsql?
> > /var/run/pgsql?
 
> Ideally, the locks should be in /var/lock/pgsql and the socket
> somewhere else - like /var/lib/pgsql (our mysql packages do this, and
> both of them are specified in /etc/my.cnf).

According to what Peter said, that could be difficult.  

But, let me ask this: is it a good thing for PostgreSQL clients to have
hard-coded socket locations?  (Good thing or not, it exists already, and
I know it does....)

I have another question of Peter, Tom, Bruce, or anyone -- is the
hard-coded socket location in libpq?  If so, wouldn't a dynamically
loaded libpq.so bring this location in for _any_ precompiled, not
statically-linked, client?  Or am I missing something else?
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sun, 28 Jan 2001 16:39:55 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
Oliver Elphick writes:

> Could it not be done by enforcing access control on system tables?

No, because CREATE TABLE does not go through access control either.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sun, 28 Jan 2001 23:11:03 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [ADMIN] Controlling user table creation 
Lamar Owen writes:

> What about the X sockets, then?

Sockets are not the problem, regular files are.  (At least for tmpwatch.)

> But, let me ask this: is it a good thing for PostgreSQL clients to have
> hard-coded socket locations?  (Good thing or not, it exists already, and
> I know it does....)

Perhaps there could be some sort of /etc/postgresql.conf file that is read
by both client and server that can control these sort of aspects.  But I
don't see much use in it besides port number and socket location.
Because those are, by definition, the only parameters in common to client
and server.

> I have another question of Peter, Tom, Bruce, or anyone -- is the
> hard-coded socket location in libpq?  If so, wouldn't a dynamically
> loaded libpq.so bring this location in for _any_ precompiled, not
> statically-linked, client?

Yes.  Good point.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sun, 28 Jan 2001 23:21:04 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Sure enough, the lock file is gone
Peter Eisentraut wrote:
> Lamar Owen writes:
> > But, let me ask this: is it a good thing for PostgreSQL clients to have
> > hard-coded socket locations?  (Good thing or not, it exists already, and
> > I know it does....)
 
> Perhaps there could be some sort of /etc/postgresql.conf file that is read
> by both client and server that can control these sort of aspects.  But I
> don't see much use in it besides port number and socket location.
> Because those are, by definition, the only parameters in common to client
> and server.

Of course, -i and TCP/IP to localhost obviate all of this.

How about an environment variable?  PGSOCKLOC?  Use the hard-coded
default if the envvar not set? This way multiple postmasters running on
multiple sockets can be smoothly supported.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sun, 28 Jan 2001 17:24:40 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:
> Ideally, the locks should be in /var/lock/pgsql and the socket
> somewhere else - like /var/lib/pgsql (our mysql packages do this, and
> both of them are specified in /etc/my.cnf).

That is not "ideal", in fact it would break one of the specific features
that UUNET asked us for.  Namely, to be able to have noninterfering
sets of socket files in different explicitly-specified directories.
If the lock files don't live where the sockets do, then this doesn't work.

> Explictly, yes. However, FHS says /tmp is for temporary files. Also,
> it says programs shouldn't count on data to be stored there between
> invocations. 10+ days isn't temporary...

We aren't counting on data to be stored in /tmp "between invocations".
The socket and lock file live only as long as the postmaster runs.

			regards, tom lane

Sun, 28 Jan 2001 17:47:23 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
Lamar Owen <lamar.owen@wgcr.org> writes:
> I have another question of Peter, Tom, Bruce, or anyone -- is the
> hard-coded socket location in libpq?  If so, wouldn't a dynamically
> loaded libpq.so bring this location in for _any_ precompiled, not
> statically-linked, client?  Or am I missing something else?

As the 7.1 code presently stands, libpq contains a compiled-in default
socketfile location, which the client code hopefully doesn't know about.
So, yes, if an old client has a dynamically linked libpq.so then
replacing the .so would bring that client into sync with a nonstandard
server.  However, the pitfalls should be obvious: independently built
clients, statically linked libraries, differing .so version numbers
to name three risk areas.

			regards, tom lane

Sun, 28 Jan 2001 17:53:10 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
Peter Eisentraut <peter_e@gmx.net> writes:
> Perhaps there could be some sort of /etc/postgresql.conf file that is read
> by both client and server that can control these sort of aspects.

Maybe ... but it seems to me that still leaves us with the issue of
a single pathname that must be known a-priori to both client and server.
You've just changed that path from "/tmp/..." to "/etc/...".

Moreover, such a setup would make it substantially more painful to run
multiple versions of Postgres on a single machine.  Right now, as long
as each version has a different default port number, it works great.
Try to put the default port number in /etc/postgresql.conf, and you've
got a problem.

			regards, tom lane

Sun, 28 Jan 2001 17:57:54 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
Tom Lane <tgl@sss.pgh.pa.us> writes:

> teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:
> > Ideally, the locks should be in /var/lock/pgsql and the socket
> > somewhere else - like /var/lib/pgsql (our mysql packages do this, and
> > both of them are specified in /etc/my.cnf).

(note that you AFAIK (I don't use mysql much, I prefer postgresql) can
have multiple sections if you want want to have multiple backends
running. 

 
> That is not "ideal", in fact it would break one of the specific features
> that UUNET asked us for.  Namely, to be able to have noninterfering
> sets of socket files in different explicitly-specified directories.
> If the lock files don't live where the sockets do, then this doesn't
> work.

I don't see why this must be so...
 
> > Explictly, yes. However, FHS says /tmp is for temporary files. Also,
> > it says programs shouldn't count on data to be stored there between
> > invocations. 10+ days isn't temporary...
> 
> We aren't counting on data to be stored in /tmp "between invocations".

Between invocations of client programs. You're using /tmp as a shared
of stored data.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

28 Jan 2001 17:59:08 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: Re: Sure enough, the lock file is gone
Lamar Owen <lamar.owen@wgcr.org> writes:
> How about an environment variable?  PGSOCKLOC?  Use the hard-coded
> default if the envvar not set? This way multiple postmasters running on
> multiple sockets can be smoothly supported.

It's spelled PGHOST as of 7.1 ... but the discussion here is about what
the default behavior of an installation will be, not what you can
override it to do. 

			regards, tom lane

Sun, 28 Jan 2001 17:59:42 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
Tom Lane wrote:
> So, yes, if an old client has a dynamically linked libpq.so then
> replacing the .so would bring that client into sync with a nonstandard
> server.

Of course, with the server and client on the same machine, the server
and the client dynamic libs are very likely to follow the same
'non-standard' as the libpq.so is likely to be from the same build or
package as the server is.

> However, the pitfalls should be obvious: independently built
> clients, statically linked libraries, differing .so version numbers
> to name three risk areas.

These are real risks, of course.  I have personal experience with the
statically linked client and differing so version number cases.

And, yes, to echo your previous sentiment, if it breaks, the
distributor/packager is not the one that gets the compliants -- the
PostgreSQL community does.

So, for future discussion, a compromise will have to be arranged -- but
this really isn't a 7.1 issue, as this isn't a 'bugfix' per se -- you
have fixed the immediate problem.  But this is something to consider for
7.2 or later, as priorities are shuffled.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sun, 28 Jan 2001 18:00:35 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:
> Explictly, yes. However, FHS says /tmp is for temporary files. Also,
> it says programs shouldn't count on data to be stored there between
> invocations. 10+ days isn't temporary...
>> 
>> We aren't counting on data to be stored in /tmp "between invocations".

> Between invocations of client programs. You're using /tmp as a shared
> of stored data.

Huh?  The socket and lockfile are created and held open by the
postmaster for the duration of its run.  Client programs don't even know
that the lockfile is there, in fact.  How can you argue that client
program lifespan has anything to do with it?

			regards, tom lane

Sun, 28 Jan 2001 18:06:39 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > How about an environment variable?  PGSOCKLOC?  Use the hard-coded
> > default if the envvar not set? This way multiple postmasters running on
> > multiple sockets can be smoothly supported.
 
> It's spelled PGHOST as of 7.1 ... but the discussion here is about what
> the default behavior of an installation will be, not what you can
> override it to do.

I'm talking about Unix domain socket location, not TCP/IP hostname,
which PGHOST is, right?

But you are very right -- this doesn't help the default. The
FHS-mandated place for such a configuration file detailing such settings
is in /etc -- but, of course, we support installations that have been
installed by a non-root user.  ISTM a 'pg_config --default-socket'
command could be used to find the location, assuming pg_config is on the
PATH.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sun, 28 Jan 2001 18:13:30 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
Oliver Elphick wrote:
> No, UNIX sockets are specifically mentioned as belonging under /var/run.
> In section 5.10 "/var/run : Run-time variable data", it says: "Programs
> that maintain transient UNIX-domain sockets should place them in this
> directory."
> 
> So what ever the outcome for the wider PostgreSQL community, I must make
> the change to conform to Debian policy.

So, if PostgreSQL is a part of Debian, then there will be problems if
the client-server situation isn't somehow fixed to allow robust
location-independent socket finding.

Looks like the same thing is going to happen with RedHat's
distribution.  So, if this is going to occur, let's get a consensus as
to where that alternate location (barring some other solution) is going
to be, so that there are the fewest variants out there.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sun, 28 Jan 2001 18:19:15 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
Lamar Owen wrote:
> Tom Lane wrote:
> > It's spelled PGHOST as of 7.1 ... but the discussion here is about what
> > the default behavior of an installation will be, not what you can
> > override it to do.
 
> I'm talking about Unix domain socket location, not TCP/IP hostname,
> which PGHOST is, right?

Found the code in fe-connect.c that changes that assumption.....sorry
for my high density :-).
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sun, 28 Jan 2001 18:28:44 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
Lamar Owen <lamar.owen@wgcr.org> writes:

> Oliver Elphick wrote:
> > No, UNIX sockets are specifically mentioned as belonging under /var/run.
> > In section 5.10 "/var/run : Run-time variable data", it says: "Programs
> > that maintain transient UNIX-domain sockets should place them in this
> > directory."
> > 
> > So what ever the outcome for the wider PostgreSQL community, I must make
> > the change to conform to Debian policy.
> 
> So, if PostgreSQL is a part of Debian, then there will be problems if
> the client-server situation isn't somehow fixed to allow robust
> location-independent socket finding.
> 
> Looks like the same thing is going to happen with RedHat's
> distribution.  So, if this is going to occur, let's get a consensus as
> to where that alternate location (barring some other solution) is going
> to be, so that there are the fewest variants out there.

FHS is a good starting (and end-) point.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

28 Jan 2001 18:37:39 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: Re: Sure enough, the lock file is gone
Tom Lane <tgl@sss.pgh.pa.us> writes:

> teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:
> > Explictly, yes. However, FHS says /tmp is for temporary files. Also,
> > it says programs shouldn't count on data to be stored there between
> > invocations. 10+ days isn't temporary...
> >> 
> >> We aren't counting on data to be stored in /tmp "between invocations".
> 
> > Between invocations of client programs. You're using /tmp as a shared
> > of stored data.
> 
> Huh?  The socket and lockfile are created and held open by the
> postmaster for the duration of its run.  Client programs don't even know
> that the lockfile is there, in fact.  How can you argue that client
> program lifespan has anything to do with it?

Nothing but the postmaster uses it? If so, there shouldn't be a
problem moving it.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

28 Jan 2001 18:43:51 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: Re: Sure enough, the lock file is gone
> Peter Eisentraut <peter_e@gmx.net> writes:
> > Perhaps there could be some sort of /etc/postgresql.conf file that is read
> > by both client and server that can control these sort of aspects.
> 
> Maybe ... but it seems to me that still leaves us with the issue of
> a single pathname that must be known a-priori to both client and server.
> You've just changed that path from "/tmp/..." to "/etc/...".

And writing to /etc requires root permissions.  That is the restriction
that got us into /tmp in the first place.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 28 Jan 2001 18:54:18 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: Sure enough, the lock file is gone
Lamar Owen <lamar.owen@wgcr.org> writes:
> Tom Lane wrote:
>> Lamar Owen <lamar.owen@wgcr.org> writes:
> How about an environment variable?  PGSOCKLOC?
 
>> It's spelled PGHOST as of 7.1 ...

> I'm talking about Unix domain socket location, not TCP/IP hostname,
> which PGHOST is, right?

No, in 7.1 PGHOST serves a dual purpose.  If a hostname beginning with
"/" is given, it's taken to specify Unix-socket communication using a
socketfile in the directory whose absolute path is PGHOST.  A tad crocky
but it avoided having to add an additional parameter to the PQconnect
family of functions ...

Also, on the postmaster side, there is a postmaster commandline
parameter to set the directory containing the socket files (and
lockfiles).  So it's possible for a given installation to configure
the socketfiles anywhere without modifying the binaries at all.
But you do need to set PGHOST on the client side to make this work.
It all comes back to what the default is.

Basically, what's bothering me is the idea that the RPM distribution
will have a different default socket location than the regular source
distribution.  I think that will cause a lot more problems than it
solves.

			regards, tom lane

Sun, 28 Jan 2001 20:00:32 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
"Oliver Elphick" <olly@lfix.co.uk> writes:
> No, UNIX sockets are specifically mentioned as belonging under /var/run.
> In section 5.10 "/var/run : Run-time variable data", it says: "Programs
> that maintain transient UNIX-domain sockets should place them in this 
> directory."

> So what ever the outcome for the wider PostgreSQL community, I must make
> the change to conform to Debian policy.

Just out of curiosity, does Debian enforce a nonstandard location for
X sockets as well?

			regards, tom lane

Sun, 28 Jan 2001 20:11:34 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
On Sun, 28 Jan 2001, Tom Lane wrote:

> "Oliver Elphick" <olly@lfix.co.uk> writes:
> > No, UNIX sockets are specifically mentioned as belonging under /var/run.
> > In section 5.10 "/var/run : Run-time variable data", it says: "Programs
> > that maintain transient UNIX-domain sockets should place them in this
> > directory."
>
> > So what ever the outcome for the wider PostgreSQL community, I must make
> > the change to conform to Debian policy.
>
> Just out of curiosity, does Debian enforce a nonstandard location for
> X sockets as well?

Just curious here ... there seems to have been *alot* of energy expended
on this ... is there any reason why we don't just have a configuration
option like other software has, that defaults to /tmp like we have it now,
but that makes it easier for others to change it for their installs, as
required?



Sun, 28 Jan 2001 21:37:56 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re: Sure enough, the lock file is gone 
The Hermit Hacker wrote:
> On Sun, 28 Jan 2001, Tom Lane wrote:
> > "Oliver Elphick" <olly@lfix.co.uk> writes:
> > > So what ever the outcome for the wider PostgreSQL community, I must make
> > > the change to conform to Debian policy.

> > Just out of curiosity, does Debian enforce a nonstandard location for
> > X sockets as well?
 
> Just curious here ... there seems to have been *alot* of energy expended
> on this ... is there any reason why we don't just have a configuration
> option like other software has, that defaults to /tmp like we have it now,
> but that makes it easier for others to change it for their installs, as
> required?

It has touched a nerve, hasn't it?  I like your solution -- but, to
reiterate, this IMHO is 7.2 material, unless we want to go the feature
patch route, or someone considers this a 'bugfix' (I don't).  Unless it
is a trivial change, that is.

Tom fixed the bug with a slight kludge -- by touching the lock
periodically, the problem is ameliorated for now.  But as long as we
have a persistent file in /tmp we will run into OS-dependent problems. 
I can see now a bug report that PostgreSQL is unreliable because it
keeps crashing every x days (due to a tmpreaper-like program the hapless
user doesn't know is running in cron....).

Since pg_config already reports what configure options were provided, if
this is a configure option then the end user can easily find it with
pg_config, if a static linkage or binary-only custom client that
directly accesses the fe-be protocol (are there any that we know
about?).

But we don't need to spend a great deal of time on it, regardless.
Speaking of time to spend, are we a 'go' for beta4 yet?  ETA? (so I can
budget time to rebuild RPM's).
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sun, 28 Jan 2001 21:48:47 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
> It has touched a nerve, hasn't it?  I like your solution -- but, to
> reiterate, this IMHO is 7.2 material, unless we want to go the feature
> patch route, or someone considers this a 'bugfix' (I don't).  Unless it
> is a trivial change, that is.
> 
> Tom fixed the bug with a slight kludge -- by touching the lock
> periodically, the problem is ameliorated for now.  But as long as we
> have a persistent file in /tmp we will run into OS-dependent problems. 
> I can see now a bug report that PostgreSQL is unreliable because it
> keeps crashing every x days (due to a tmpreaper-like program the hapless
> user doesn't know is running in cron....).
> 
> Since pg_config already reports what configure options were provided, if
> this is a configure option then the end user can easily find it with
> pg_config, if a static linkage or binary-only custom client that
> directly accesses the fe-be protocol (are there any that we know
> about?).

No one has suggested a location non-root people can put the socket/lock
file, except /tmp, and IMHO, until we find one, the default stays in
/tmp.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 28 Jan 2001 22:04:00 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: Sure enough, the lock file is gone
The Hermit Hacker <scrappy@hub.org> writes:
> Just curious here ... there seems to have been *alot* of energy expended
> on this ... is there any reason why we don't just have a configuration
> option like other software has, that defaults to /tmp like we have it now,
> but that makes it easier for others to change it for their installs, as
> required?

We *have* such a configuration option, both compile-time and run-time,
as of 7.1.  The argument is about what the default should be, and in
particular whether it's a good idea for certain binary distributions to
have a different default than other distributions ...

			regards, tom lane

Sun, 28 Jan 2001 22:46:08 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
Lamar Owen <lamar.owen@wgcr.org> writes:
> Tom fixed the bug with a slight kludge -- by touching the lock
> periodically, the problem is ameliorated for now.  But as long as we
> have a persistent file in /tmp we will run into OS-dependent problems. 
> I can see now a bug report that PostgreSQL is unreliable because it
> keeps crashing every x days

Kindly don't exaggerate the importance of this problem.  We've been
running systems with the socketfiles in /tmp for years now, and we
know quite well what the downsides of that are.  They're not that
drastic (unless you run a tmp-sweeper too stupid to distinguish socket
files from plain files, and even then you only see failure to connect).

Addition of the socket lockfile to the mix isn't increasing the risks
measurably that I can see.  Even if a tmp-sweeper is enthusiastic enough
to remove a lockfile that the postmaster touches every few minutes, so
what?  Client connections don't depend on the lockfile.  The lockfile
only exists to protect against admin error, ie, starting a second
postmaster on the same socket --- which in routine operation won't
happen anyway.

The bottom line: yes, /tmp was a poor choice of place to put the
socket files.  But no, it is not so poor as to be worth creating a
compatibility problem to fix it.  Perhaps someday we will switch to
a whole new interface protocol (CORBA or whatever floats your boat),
and then we can let the whole mess drift off into the sunset.  But
right now, there is almost nothing about our FE/BE protocol that
*isn't* a legacy decision --- and fixing it piecemeal at the cost of
a flag day for each fix is not worthwhile.  IMHO anyway.

			regards, tom lane

Sun, 28 Jan 2001 23:11:19 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
> The bottom line: yes, /tmp was a poor choice of place to put the
> socket files.  But no, it is not so poor as to be worth creating a

Was it really a poor choice.  Where else can we put it as non-root?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 28 Jan 2001 23:24:03 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: Sure enough, the lock file is gone
On Sun, 28 Jan 2001, Bruce Momjian wrote:

> > The bottom line: yes, /tmp was a poor choice of place to put the
> > socket files.  But no, it is not so poor as to be worth creating a
>
> Was it really a poor choice.  Where else can we put it as non-root?

~pgsql/var/run?  everything else was under that directory structure ...



Mon, 29 Jan 2001 00:34:10 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re: Sure enough, the lock file is gone
> On Sun, 28 Jan 2001, Bruce Momjian wrote:
> 
> > > The bottom line: yes, /tmp was a poor choice of place to put the
> > > socket files.  But no, it is not so poor as to be worth creating a
> >
> > Was it really a poor choice.  Where else can we put it as non-root?
> 
> ~pgsql/var/run?  everything else was under that directory structure ...

Sure, but how do we hard-code where ~pgsql is located, unless we somehow
check the home directory of pgsql, but then again, they really don't
have to use the home directory of pgsql for the pgsql directory.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 28 Jan 2001 23:39:56 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: Sure enough, the lock file is gone
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> The bottom line: yes, /tmp was a poor choice of place to put the
>> socket files.  But no, it is not so poor as to be worth creating a

> Was it really a poor choice.  Where else can we put it as non-root?

I would've favored something like /tmp/.postgres/socketfile,
which is comparable to typical X11 setups.  But it's moot, since
we can't change the default path even that much without creating a
compatibility headache ...

			regards, tom lane

Sun, 28 Jan 2001 23:43:11 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> The bottom line: yes, /tmp was a poor choice of place to put the
> >> socket files.  But no, it is not so poor as to be worth creating a
> 
> > Was it really a poor choice.  Where else can we put it as non-root?
> 
> I would've favored something like /tmp/.postgres/socketfile,
> which is comparable to typical X11 setups.  But it's moot, since
> we can't change the default path even that much without creating a
> compatibility headache ...

Agreed, that would have been better, but I don't see that buys us
anything against the grim reaper.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 28 Jan 2001 23:43:55 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: Sure enough, the lock file is gone
Bruce Momjian wrote:
> No one has suggested a location non-root people can put the socket/lock
> file, except /tmp, and IMHO, until we find one, the default stays in
> /tmp.

Since RPM's _must_ be installed by root, that doesn't affect them.  The
debian packages are the same way.  As long as the RPM contains the
structures set as owned by 'postgres' (the default run user for
postmaster), and the appropriate permissions are set, the directory
could be anywhere, such as /var/run/pgsql. But there are of course
problems with that approach, because libpq makes some hard-coded
assumptions about where to look.

I have no problem with the default being in /tmp, just like I have no
problem with the default source installation being in /usr/local.  But I
do think that the code should be smart enough to handle non-default
settings without major problems.

And I'll kindly not exaggerate the importance -- but I would have seen
reports had the simple fix not been applied.

But I'm not going to spend any more time arguing about it, that much is
certain. I've got other fish to fry, like beta4 RPM's.....
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sun, 28 Jan 2001 23:44:50 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
> Bruce Momjian wrote:
> > No one has suggested a location non-root people can put the socket/lock
> > file, except /tmp, and IMHO, until we find one, the default stays in
> > /tmp.
> 
> Since RPM's _must_ be installed by root, that doesn't affect them.  The
> debian packages are the same way.  As long as the RPM contains the
> structures set as owned by 'postgres' (the default run user for
> postmaster), and the appropriate permissions are set, the directory
> could be anywhere, such as /var/run/pgsql. But there are of course
> problems with that approach, because libpq makes some hard-coded
> assumptions about where to look.

The issue we have is that we don't assume root installs.  Any root
requirement is going to be RPM-specific.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 29 Jan 2001 00:57:02 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: Sure enough, the lock file is gone
Tom Lane wrote:
  >Just out of curiosity, does Debian enforce a nonstandard location for
  >X sockets as well?

X has a big exception made for it in the FHS for /usr/X11R6; I don't see
any mention of how /tmp/.X11/ is to be treated, but X isn't my problem!
I suspect no-one has ever thought to ask the question!  You may be
pleased to know that I have now stirred the coals and asked that
question, which applies not only to X but also to several other programs
that I have running.

It seems to me that the main reason for the problem is the need to
cater for non-root installs.  I would really like to know what
proportion of total PostgreSQL installs they now are.  We are
a long way now from the days when Postgres was a research database with
a poor reputation for reliability.  It is now becoming a serious
competitor to major commercial databases and will in many cases be
used as a major application on the machines where it is installed.

Are there still people who can only install PostgreSQL as a private 
application?


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "My son, if sinners entice thee, consent thou not."    
                            Proverbs 1:10 



Mon, 29 Jan 2001 06:29:17 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: Re: Sure enough, the lock file is gone 
Bruce Momjian wrote:
> Lamar Owen wrote: 
> > Bruce Momjian wrote:
> > > No one has suggested a location non-root people can put the socket/lock
> > Since RPM's _must_ be installed by root, that doesn't affect them.  The
> The issue we have is that we don't assume root installs.  Any root
> requirement is going to be RPM-specific.

[waiting on another RPM build cycle to finish]

I understand that issue.  The RPMset is just not affected by that issue,
as, for an RPM to be installed, you must be root.  No ifs ands or buts
-- an RPM installation assumes root, and can do anything along those
lines it needs to do.

Of course, that means I have to be extra careful -- people installing
RPM's I build are going to be running my %pre, %post, %preun, and
%postun scripts _as_root_.  RPM's are quite capable of royally hosing a
system, if the packager hasn't done his homework.  But that also means
malicious RPMs are possible (horrors.) -- one header in an RPM, hidden
from view, could render your system totally useless.  Yes, it's true.
No, I won't package an RPM 'bomb' -- but it could be done, easily
enough.

Now, you can _build_ an RPM (which will be installed as root) as a
non-root user, and that invokes the 'make install' as part of its
process, but I digress (badly, at that).

But my issue is that libpq or any other client should be smart enough to
not have to assume the location.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 29 Jan 2001 01:30:35 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
On Mon, 29 Jan 2001, Oliver Elphick wrote:

> It seems to me that the main reason for the problem is the need to
> cater for non-root installs.  I would really like to know what

PostgreSQL will _not_ run as root. Just try... :)

morannon:~>/usr/local/pgsql/bin/postmaster

"root" execution of the PostgreSQL backend is not permitted.

The backend must be started under its own userid to prevent
a possible system security compromise. See the INSTALL file for
more information on how to properly start the postmaster.

morannon:~>

> proportion of total PostgreSQL installs they now are.  We are

100%

> Are there still people who can only install PostgreSQL as a private 
> application?

Private application - or non-root application?


-- 
Dominic J. Eidson
                                        "Baruk Khazad! Khazad ai-menu!" - Gimli
-------------------------------------------------------------------------------
http://www.the-infinite.org/              http://www.the-infinite.org/~dominic/


Mon, 29 Jan 2001 00:34:29 -0600 (CST)
"Dominic J. Eidson" <sauron@the-infinite.org>
Re: Re: Sure enough, the lock file is gone 
Lamar Owen <lamar.owen@wgcr.org> writes:
> But my issue is that libpq or any other client should be smart enough to
> not have to assume the location.

Er, how do you propose to do that?  The client cannot learn the correct
location from the postmaster --- it must figure out *on its own* where
the socket file is.  AFAICS you can't avoid having hardwired knowledge
about how to do that in the client.

You or somebody else previously suggested hardwiring the location of
a configuration file, rather than the socketfile itself, but I can't
see that that really improves matters in this context.  In particular,
changing to such a method would still break backwards compatibility.

			regards, tom lane

Mon, 29 Jan 2001 01:44:41 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
"Dominic J. Eidson" wrote:
  >On Mon, 29 Jan 2001, Oliver Elphick wrote:
  >
  >> It seems to me that the main reason for the problem is the need to
  >> cater for non-root installs.  I would really like to know what
  >
  >PostgreSQL will _not_ run as root. Just try... :)
 
I'm talking about the installation, not execution.

By default, PostgreSQL will install in /usr/local/pgsql; most people
will need root privilege to create that directory.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "My son, if sinners entice thee, consent thou not."    
                            Proverbs 1:10 



Mon, 29 Jan 2001 06:50:03 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: Re: Sure enough, the lock file is gone 
Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > But my issue is that libpq or any other client should be smart enough to
> > not have to assume the location.
> Er, how do you propose to do that?  The client cannot learn the correct
> location from the postmaster --- it must figure out *on its own* where
> the socket file is.  AFAICS you can't avoid having hardwired knowledge
> about how to do that in the client.

How does netstat find out?  A simple 
netstat -a --unix|grep \.s\.PGSQL
will get the full list of all postmaster sockets.  A little 'cut' or
'awk' work is simple enough.

I realize, of course, that netstat (and the underlying, on Linux,
/proc/net/unix file) is not portable. On Linux, simply grep
/proc/net/unix for the .s.PGSQL pattern and get the last column (or the
column before that, with the inode information).

Is there a portable way of listing the open unix domain sockets in this
manner, then deducing from the socket path what you need to know?

> You or somebody else previously suggested hardwiring the location of
> a configuration file, rather than the socketfile itself, but I can't
> see that that really improves matters in this context.  In particular,
> changing to such a method would still break backwards compatibility.

Not me.  The less hardwiring, the better, IMHO. And I'm glad you pointed
me to the new (undocumented that I could find) usage of PGHOST.  A
dynamic socket finder (assuming no specific socket path has been passed)
would not break backwards compatibility, as it would find the default
/tmp case.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 29 Jan 2001 02:08:01 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
Lamar Owen <lamar.owen@wgcr.org> writes:
> How does netstat find out?

netstat burrows around in kernel datastructures, is how.

I don't see invoking netstat as a solution anyway.  For one thing,
it's drastically nonstandard; even if available, it varies in parameters
and output format (your "simple example" draws a usage complaint on my
box).  Furthermore, a moderately security-conscious admin would probably
choose to make netstat unavailable to unprivileged users, since it
reveals an uncomfortably large amount about the activity of other users.
A final complaint is that netstat would actually reveal *too much*
information, since a netstat-based client would have no way to choose
among multiple postmasters.  (Please recall that one of the motivations
for the UUNET patch was to allow multiple postmasters running with the
same port number in different subdirectories.  Hmm, I wonder how netstat
shows socketfiles that are in chroot'd subtrees, or outside your own
chroot ...)

			regards, tom lane

Mon, 29 Jan 2001 02:34:29 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Sure enough, the lock file is gone 
Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > How does netstat find out?
 
> netstat burrows around in kernel datastructures, is how.
 
> I don't see invoking netstat as a solution anyway.  For one thing,
> it's drastically nonstandard; even if available, it varies in parameters

I said as much as it wasn't portable.  But I asked also if a portable
way was available -- I do not currently know that answer to that, but I
will be investigating such.

> (Please recall that one of the motivations
> for the UUNET patch was to allow multiple postmasters running with the
> same port number in different subdirectories.  Hmm, I wonder how netstat
> shows socketfiles that are in chroot'd subtrees, or outside your own
> chroot ...)

When were these 'UUNET' patches issued?  I like the idea, but just
curious. I don't recall them, in fact -- nor do I recall the
discussion.  I'll look it up in the archives later. Going to bed after a
night of RPM'ing.

As to the chroot vs netstat question, that is a good one.  I have no
chroot's in effect, so I can't test that one.

So, if multiple postmasters are running on the same port in different
dirs, it would be somewhat difficult to determine which should be the
'default' in the list.  However, one would think an admin who has set up
a multiple postmaster system of that sort wouldn't be relying on a
default anyway -- but that is a dangerous assumption.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 29 Jan 2001 02:43:55 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Sure enough, the lock file is gone
On Fri, Jan 26, 2001 at 11:55:16PM -0500, Bruce Momjian wrote:
> > I said:
> > > Yes, there are lots of systems that will clean /tmp --- and since the
> > > lock file is an ordinary file (not a socket) pretty much any tmp-cleaner
> > > is going to decide to remove it.  I think that I had intended to insert
> > > a periodic touch of the lockfile and forgot to.
> > 
> > Done now.
> 
> Yes, checkpoint is a good place to put it.  Thanks.  I still liked the
> BRAINDAMAGED_TMP_CLEANER though.
> 

Well, from reading the rest of the thread, it seems that, while tmpreaper
_is_, in fact, brain damaged (it'll eat socket files) that wasn't what we
were talking about: removal of apparently stale ordinary files, like our
lockfile, is what tmp cleaners are all about, even if it is problematic.

Personally, I've always thought tmp cleaners are a bad idea, a bandaid
approach to cleaning up after poorly written software: i.e. broken by
design. Well, it's resolved now, for pgsql. Can't fix all the software
out there, I guess.

Ross

Mon, 29 Jan 2001 10:43:52 -0600
"Ross J. Reedstrom" <reedstrm@rice.edu>
Re: Sure enough, the lock file is gone
I finally percated that when data contains 'ï¿½' or 'ï¿½' it's impossible to
parse trought:

COPY products FROM '/var/lib/postgres/dadesi.txt' USING DELIMITERS '|' \g

it causes:

SELECT edicion FROM products;
     edicion     
-----------------
 Espaï¿½a|Nacional <-------puts on the same cell either there's an '|' in
the middle!!!


but changing 'ï¿½' for n

SELECT edicion FROM products;
     edicion     
-----------------
 Espana <---------------it separates cells ok


so what's my solution for a text to COPY containing such characters? 


best regards,
jaume

Mon, 26 Feb 2001 14:28:58 +0100
Jaume Teixi <teixi@6tems.com>
COPY doesn't works when containing '���' or '���' characters on db

Morning all ...

	Are there any major outstandings that ppl have on their plates,
that should prevent a release?  I'd like to put out an RC1 by Friday this
week, with a full release schedualed for March 15th ... this would give
Thomas his two weeks for the docs freeze ...

	Basically, RC1 would say to ppl that we're ready to release, there
will be no more core changes that will require an initdb ... feel
comfortable using this version in production, with the only major changes
between now and release being docs related ...

	Does this work?  Or is there something earth-shattering that still
has to be done?

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org


Mon, 26 Feb 2001 11:52:33 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Release in 2 weeks ... 
The Hermit Hacker wrote:
> 
> Morning all ...
> 
>         Are there any major outstandings that ppl have on their plates,
> that should prevent a release?  I'd like to put out an RC1 by Friday this
> week, with a full release schedualed for March 15th ... this would give
> Thomas his two weeks for the docs freeze ...
> 
>         Basically, RC1 would say to ppl that we're ready to release, there
> will be no more core changes that will require an initdb ... feel
> comfortable using this version in production, with the only major changes
> between now and release being docs related ...
> 
>         Does this work?  Or is there something earth-shattering that still
> has to be done?


Yep ! As of beta4, the ODBC driver is still seriously broken (the
original libpsqlodbc.so.0.26 doesn't even connect. A version patched by
Nick Gorham allows some connectivity (you can query the DB), but still
has some serious breakage (i. e. no "obvious" ways to see views from
StarOffice or MS-Access)).

And I have not yet had any opportunity to test the JDBC driver.

[ Explanation : I follow the Debian packages prepared by Oliver Elphick,
I'm not versed enough in Debian to recreate those packages myself, and I
do *not* want to break Debian dependencies by installing Postgres "The
Wrong Way (TM)". Hence, I'm stuck with beta4, a broken ODBC and no JDBC.
Unless some kind soul can send me a JD. 1.1 .jar file ...

Furthermore, I've had some serious hardware troubles (a dying IDE disk).
I wasn't even able to fulfill Tom Lane's suggestion to try to add -d2 to
my postmaster to debug the ODBC connection. I'll try to do that Real
Soon Now (TM, again), but not for now : my day-work backlog is ...
impressive. ]

These issues might seem small change to you die-hard plpgsql hackers. To
a lmot of people using Postgres for everyday office work through "nice"
interface, it's bread-and-butter, and these issues *should* be fixed
*before* release ...

[ crawling back under my rock ... ]

					Emmanuel Charpentier

Mon, 26 Feb 2001 20:24:01 +0100
Emmanuel Charpentier <charpent@bacbuc.dyndns.org>
Re: Release in 2 weeks ...
Jaume Teixi <teixi@6tems.com> writes:
> I finally percated that when data contains '�' or '�' it's impossible to
> parse trought:

> COPY products FROM '/var/lib/postgres/dadesi.txt' USING DELIMITERS '|' \g

> it causes:

> SELECT edicion FROM products;
>      edicion     
> -----------------
>  Espa�a|Nacional <-------puts on the same cell either there's an '|' in
> the middle!!!

Very odd.  What LOCALE and multibyte encodings are you using, if any?
This seems like it must be a multibyte issue, but I can't guess what.

Also, which Postgres version are you running?  If you said, I missed it.

			regards, tom lane

Mon, 26 Feb 2001 22:16:35 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: COPY doesn't works when containing ' ' or ' ' characters on db 
On Mon, 26 Feb 2001 22:16:35 -0500 Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Jaume Teixi <teixi@6tems.com> writes:
> > I finally percated that when data contains 'ï¿½' or 'ï¿½' it's impossible
to
> > parse trought:
> 
> > COPY products FROM '/var/lib/postgres/dadesi.txt' USING DELIMITERS '|'
\g
> 
> > it causes:
> 
> > SELECT edicion FROM products;
> >      edicion     
> > -----------------
> >  Espaï¿½a|Nacional <-------puts on the same cell either there's an '|'
in
> > the middle!!!


I finally, thanks to Oliver Elphick,

managed to create database with:
	CREATE DATABASE "demo" WITH ENCODING = 'SQL_ASCII'

and data was imported OK, great, thanks!


Tue, 27 Feb 2001 10:19:12 +0100
Jaume Teixi <teixi@6tems.com>
Re: SOLVED: COPY doesn't works when containing ' ' or ' ' characters
	on db
Tom Lane wrote:
  >Jaume Teixi <teixi@6tems.com> writes:
  >> I finally percated that when data contains '' or '' it's impossible to
  >> parse trought:
  >
  >> COPY products FROM '/var/lib/postgres/dadesi.txt' USING DELIMITERS '|' \g
  >
  >> it causes:
  >
  >> SELECT edicion FROM products;
  >>      edicion     
  >> -----------------
  >>  Espaa|Nacional <-------puts on the same cell either there's an '|' in
  >> the middle!!!
  >
  >Very odd.  What LOCALE and multibyte encodings are you using, if any?
  >This seems like it must be a multibyte issue, but I can't guess what.
  >
  >Also, which Postgres version are you running?  If you said, I missed it.

I think this happens when the front-end encoding is SQL_ASCII and the
database is using UNICODE.  Then, there are misunderstandings between
front-end and back-end, so that a single character with the eighth bit
set may be sent by the front-end and interpreted by the back-end as the
first half of a UNICODE two-byte character.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "If we confess our sins, he is faithful and just to 
      forgive us our sins, and to cleanse us from all  
      unrighteousness."       I John 1:9 



Tue, 27 Feb 2001 15:38:15 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: COPY doesn't works when containing ' ' or ' ' characters
 on db
The Hermit Hacker writes:

> 	Are there any major outstandings that ppl have on their plates,
> that should prevent a release?  I'd like to put out an RC1 by Friday this
> week, with a full release schedualed for March 15th ... this would give
> Thomas his two weeks for the docs freeze ...

I'm interested to know what exactly takes two weeks with the docs and what
could be done to speed it up.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Tue, 27 Feb 2001 17:54:18 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Release in 2 weeks ... 
"Oliver Elphick" <olly@lfix.co.uk> writes:
> I think this happens when the front-end encoding is SQL_ASCII and the
> database is using UNICODE.  Then, there are misunderstandings between
> front-end and back-end, so that a single character with the eighth bit
> set may be sent by the front-end and interpreted by the back-end as the
> first half of a UNICODE two-byte character.

I wondered about that, but his examples had one or more characters
between the eighth-bit-set character and the '|', so this doesn't seem
to explain the problem.

Still, if it went away after moving to ASCII encoding, it clearly is
a multibyte issue of some sort.

			regards, tom lane

Tue, 27 Feb 2001 12:19:01 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: COPY doesn't works when containing ' ' or ' ' characters on db 
Emmanuel Charpentier <charpent@bacbuc.dyndns.org> writes:
> Yep ! As of beta4, the ODBC driver is still seriously broken (the
> original libpsqlodbc.so.0.26 doesn't even connect. A version patched by
> Nick Gorham allows some connectivity (you can query the DB), but still
> has some serious breakage (i. e. no "obvious" ways to see views from
> StarOffice or MS-Access)).

I'd be willing to work harder on ODBC if I had any way to test it ;-).

I have a copy of OpenOffice for LinuxPPC but have not figured out how to
tell it to connect to Postgres.  If someone can slip me a clue on how to
configure it and do simple database stuff with it, I'll try to clean up
the most pressing ODBC problems before we release.

			regards, tom lane

Tue, 27 Feb 2001 14:21:22 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Release in 2 weeks ... 
Tom Lane wrote:
> 
> Emmanuel Charpentier <charpent@bacbuc.dyndns.org> writes:
> > Yep ! As of beta4, the ODBC driver is still seriously broken (the
> > original libpsqlodbc.so.0.26 doesn't even connect. A version patched by
> > Nick Gorham allows some connectivity (you can query the DB), but still
> > has some serious breakage (i. e. no "obvious" ways to see views from
> > StarOffice or MS-Access)).
> 

I think I've fixed this bug at least for MS-Access.
You could get the latest win32 driver from
  ftp://ftp.greatbridge.org/pub/pgadmin/stable/psqlodbc.zip .
Please try it.

However I'm not sure about unixODBC.

Regards,
Hiroshi Inoue

Wed, 28 Feb 2001 08:53:31 +0900
Hiroshi Inoue <Inoue@tpf.co.jp>
Re: [ODBC] Re: Release in 2 weeks ...
> "Oliver Elphick" <olly@lfix.co.uk> writes:
> > I think this happens when the front-end encoding is SQL_ASCII and the
> > database is using UNICODE.  Then, there are misunderstandings between
> > front-end and back-end, so that a single character with the eighth bit
> > set may be sent by the front-end and interpreted by the back-end as the
> > first half of a UNICODE two-byte character.
> 
> I wondered about that, but his examples had one or more characters
> between the eighth-bit-set character and the '|', so this doesn't seem
> to explain the problem.

No.

 From Jaume's example:

> SELECT edicion FROM products;
>      edicion     
> -----------------
>  Espa���a|Nacional <-------puts on the same cell either there's an '|' in
> the middle!!!

\361 == 0xf1. UTF-8 assumes that:

     if (the first byte) & 0xe0 == 0xe0, then the letter consists of 3
     bytes.

So PostgreSQL believes that "���a|" is one UTF-8 letter and eat up
'|'.

My guess is Jaume made an UNICODE database but provided it ISO 8859-1
or that kind of single-byte latin encoding data.

I'm wondering why so many people are using UTF-8 database even he does
not understand what UTF-8 is:-) I hope 7.1 would solve this kind of
confusion by enabling an automatic encoding conversion between UTF-8
and others.
--
Tatsuo Ishii

Wed, 28 Feb 2001 10:01:34 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] Re: COPY doesn't works when containing ' '
	or ' ' characters on db
> >       Are there any major outstandings that ppl have on their plates,
> > that should prevent a release?  I'd like to put out an RC1 by Friday this
> > week, with a full release schedualed for March 15th ... this would give
> > Thomas his two weeks for the docs freeze ...
> I'm interested to know what exactly takes two weeks with the docs and what
> could be done to speed it up.

The "official" version of the story is that it takes ~10-20 hours for me
to work through the docs to format them for hardcopy with ApplixWare,
primarily because something in the jade RTF tickles a bug in the page
formatting with Applix. (This round, I'll resort even to M$Word to avoid
that time sink, since I just don't have the time.)

The reality is that it is a two week quiet time for us to get the last
bugs out and to get the last platform-specific reports. At this moment
we have not started the "report now or risk having a broken platform"
threats that help iron out the last problems.

Scrappy has proposed that we start that period now. Were the concerns
about WAL etc enough to hold off on that, or are we counting down from
now?

                    - Thomas

Wed, 28 Feb 2001 03:07:06 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Release in 2 weeks ...
> I have a copy of OpenOffice for LinuxPPC but have not figured out how to
> tell it to connect to Postgres.  If someone can slip me a clue on how to
> configure it and do simple database stuff with it, I'll try to clean up
> the most pressing ODBC problems before we release.

I've got a clue for ApplixWare, if you happen to have that package
(US$90).

                    - Thomas

Wed, 28 Feb 2001 03:21:16 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [ODBC] Re: Release in 2 weeks ...
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> I'm interested to know what exactly takes two weeks with the docs and what
>> could be done to speed it up.

> The "official" version of the story is that it takes ~10-20 hours for me
> to work through the docs to format them for hardcopy with ApplixWare,
> primarily because something in the jade RTF tickles a bug in the page
> formatting with Applix.

I'm sure anything that could be done to eliminate this formatting
make-work would be just fine with Thomas ;-).  However, it probably
wouldn't really change the release scheduling much, since as he points
out it's partially an excuse for clamping down:

> The reality is that it is a two week quiet time for us to get the last
> bugs out and to get the last platform-specific reports.

In short, now is our "okay people, let's get *serious*" phase.
No features, no trivial stuff, just get the critical bugs out.

> Scrappy has proposed that we start that period now. Were the concerns
> about WAL etc enough to hold off on that, or are we counting down from
> now?

I'm pretty concerned about WAL, but have no good reason not to start
the release countdown.

			regards, tom lane

Tue, 27 Feb 2001 23:19:00 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Release in 2 weeks ... 
On Tue, 27 Feb 2001, Tom Lane wrote:

> > Scrappy has proposed that we start that period now. Were the concerns
> > about WAL etc enough to hold off on that, or are we counting down from
> > now?
>
> I'm pretty concerned about WAL, but have no good reason not to start
> the release countdown.

Figuring a 15th of March release right now, Vadim is back on the 6th (or
so), so that would essentially be the last 'critical bug' ...

Just curious ... Vadim posted yesterday about 'fixes' for WAL related
stuff ... stuff he wanted to ppl to try out ... has anyone?  I didn't see
anyone respond to his post, so am wondering if nobody but myself saw it
...



Wed, 28 Feb 2001 00:30:39 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re: Release in 2 weeks ... 
Thomas Lockhart wrote:
  >> I have a copy of OpenOffice for LinuxPPC but have not figured out how to
  >> tell it to connect to Postgres.  If someone can slip me a clue on how to
  >> configure it and do simple database stuff with it, I'll try to clean up
  >> the most pressing ODBC problems before we release.
  >
  >I've got a clue for ApplixWare, if you happen to have that package
  >(US$90).
 
Please post it, Thomas.

I got nowhere following their instructions.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "These things have I written unto you that believe on 
      the name of the Son of God; that ye may know that ye 
      have eternal life, and that ye may believe on the name
      of the Son of God."        I John 5:13 



Wed, 28 Feb 2001 06:57:50 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [ODBC] Re: Release in 2 weeks ... 
>   >I've got a clue for ApplixWare, if you happen to have that package
>   >(US$90).
> Please post it, Thomas.
> I got nowhere following their instructions.

Uh, who's instructions? We have a writeup on Applix and ODBC in the
docs. Have you found those, or are those falling short of helpful?

                       - Thomas

Wed, 28 Feb 2001 07:26:43 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [ODBC] Re: Release in 2 weeks ...
Thomas Lockhart writes:

> The "official" version of the story is that it takes ~10-20 hours for me
> to work through the docs to format them for hardcopy with ApplixWare,

Okay, I just kept hearing the "give Thomas 2 weeks for the docs" theme...

> primarily because something in the jade RTF tickles a bug in the page
> formatting with Applix. (This round, I'll resort even to M$Word to avoid
> that time sink, since I just don't have the time.)

Is that the same MS Word that generates Postscript files as a big bitmap?

I suppose by the time we release the 10th anniversary edition, the XML/XSL
architecture will be mature enough to produce printable files that way,
but until then -- whatever works.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Wed, 28 Feb 2001 21:13:40 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Release in 2 weeks ...
> > primarily because something in the jade RTF tickles a bug in the page
> > formatting with Applix. (This round, I'll resort even to M$Word to avoid
> > that time sink, since I just don't have the time.)
> Is that the same MS Word that generates Postscript files as a big bitmap?
> I suppose by the time we release the 10th anniversary edition, the XML/XSL
> architecture will be mature enough to produce printable files that way,
> but until then -- whatever works.

I'm not counting on it even then. Some "last minute markup" will always
be required imho. But I dream about it ;)

                    - Thomas

Wed, 28 Feb 2001 22:07:31 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Release in 2 weeks ...
I haven't been following this thread very carefully but I just remembered a
similar problem we had that is probably related. We did a dump from a UTF-8
db containind English, Japanese, and Korean data. When the dump was done in
the default mode (e.g., via COPY statements) then we could no restore it. It
would die on certain characters. We then tried dumping in with -nd flags.
This fixed the problem for us although the restore is a lot slower.

--Rainer

> -----Original Message-----
> From: pgsql-admin-owner@postgresql.org
> [mailto:pgsql-admin-owner@postgresql.org]On Behalf Of Tom Lane
> Sent: Tuesday, February 27, 2001 12:17 PM
> To: Jaume Teixi
> Cc: pgsql-hackers@postgresql.org; pgsql-admin@postgresql.org; Richard T.
> Robino; Stefan Huber
> Subject: Re: [ADMIN] COPY doesn't works when containing ' ' or ' '
> characters on db
>
>
> Jaume Teixi <teixi@6tems.com> writes:
> > I finally percated that when data contains '・ or '・ it's impossible to
> > parse trought:
>
> > COPY products FROM '/var/lib/postgres/dadesi.txt' USING
> DELIMITERS '|' \g
>
> > it causes:
>
> > SELECT edicion FROM products;
> >      edicion
> > -----------------
> >  Espa������痺蜿釶�辞���頤�����黶辣�繻�繪�纈�蒹鱚�����蜴
����迚粐跂　���帙鴒�粐��葹�模蛋姪�鈔���磔��釿閼蜴苴�鱚�阨�皷鈑�蜀�銷��壽蜩�繞逑�蜍�蜚���矼����磔��齠��碯��竅逾�苺纉��癆���糟齒��蜒�倆齡苒纉�纈皷闔�鱚�阨��鉗鈑�����黶蜆��迚齠繖��
��			鱚艨鰾鵺�闕�瘤�

Thu, 1 Mar 2001 08:01:32 +0900
"Rainer Mager" <rmager@vgkk.com>
RE: COPY doesn't works when containing ' ' or ' ' characters on db 
On Wed, Feb 28, 2001 at 08:53:31AM +0900, Hiroshi Inoue wrote:
... 
> I think I've fixed this bug at least for MS-Access.
> You could get the latest win32 driver from
>   ftp://ftp.greatbridge.org/pub/pgadmin/stable/psqlodbc.zip .
> Please try it.

How can I just install that file? (ie., M$ Access -> psqlodbc.dll -> real OS)

===== aside:

I just tried installing pgAdmin - the installer says:

This setup requires at least version 2.5 of the Microsoft Data Access
Components (MDAC) to be installed first. If the MDAC installer
(mdac_typ.exe) is not provided with this setup, you can find it on the
Microsoft web site (www.microsoft.com)

And after searching said website,
http://www.microsoft.com/data/download2.htm
shows:

Microsoft Data Access Components MDAC 2.1.1.3711.11      < 2.5...

Cheers,

Patrick

Thu, 1 Mar 2001 00:55:29 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: Re: [HACKERS] Release in 2 weeks ...
Patrick Welche wrote:
> 
> On Wed, Feb 28, 2001 at 08:53:31AM +0900, Hiroshi Inoue wrote:
> ...
> > I think I've fixed this bug at least for MS-Access.
> > You could get the latest win32 driver from
> >   ftp://ftp.greatbridge.org/pub/pgadmin/stable/psqlodbc.zip .
> > Please try it.
> 
> How can I just install that file? (ie., M$ Access -> psqlodbc.dll -> real OS)
> 

I don't know if M$-access requires MDAC now(it didn't require
MDAC before). I use ADO and don't use M$-access other than
testing. ADO requires MDAC and pgAdmin uses ADO AFAIK.

> ===== aside:
> 
> I just tried installing pgAdmin - the installer says:
> 
> This setup requires at least version 2.5 of the Microsoft Data Access
> Components (MDAC) to be installed first. If the MDAC installer
> (mdac_typ.exe) is not provided with this setup, you can find it on the
> Microsoft web site (www.microsoft.com)
> 
> And after searching said website,
> http://www.microsoft.com/data/download2.htm
> shows:
> 
> Microsoft Data Access Components MDAC 2.1.1.3711.11      < 2.5...
> 

I can see the following at http://www.microsoft.com/data/download.htm

Data Access Components (MDAC) redistribution releases.
Five releases of MDAC are available here: The new MDAC
2.6, two of MDAC 2.5, and two of MDAC 2.1. You can

Regards,
Hiroshi Inoue

Thu, 01 Mar 2001 11:05:02 +0900
Hiroshi Inoue <Inoue@tpf.co.jp>
Re: [ODBC] Re: Release in 2 weeks ...
At 11:52 26/02/01 -0400, The Hermit Hacker wrote:

>Morning all ...
>
>         Are there any major outstandings that ppl have on their plates,
>that should prevent a release?  I'd like to put out an RC1 by Friday this
>week, with a full release schedualed for March 15th ... this would give
>Thomas his two weeks for the docs freeze ...

It will also give me a little extra time. This week has been a tad busy 
work wise for me I've not been able to do anything at all (only now have I 
been able to find time to download the 800 emails that were waiting for me 
:-( )


>         Basically, RC1 would say to ppl that we're ready to release, there
>will be no more core changes that will require an initdb ... feel
>comfortable using this version in production, with the only major changes
>between now and release being docs related ...
>
>         Does this work?  Or is there something earth-shattering that still
>has to be done?

Not on my front except:

JDBC1.2 driver needs testing (still can't get JDK1.1.8 to install here).
The JDBC 2.1 Enterprise Edition driver also needs some testing.

The JDBC2.1 Standard Edition driver is ready. Some new patches to look at.

PS: Did you know we are only 1 thing short of being JDBC compliant with the 
JDBC2.1 SE driver?

The other not implemented bits are extras not technically (according to the 
spec) needed for compliance. But then there's not many of them either 
(about 11 at last count excluding CallableStatement - which isn't required 
which I was surprised about when I check it last weekend).

Peter


Thu, 01 Mar 2001 20:13:47 +0000
Peter Mount <peter@retep.org.uk>
Re: Release in 2 weeks ... 
At 17:54 27/02/01 +0100, Peter Eisentraut wrote:
>The Hermit Hacker writes:
>
> >       Are there any major outstandings that ppl have on their plates,
> > that should prevent a release?  I'd like to put out an RC1 by Friday this
> > week, with a full release schedualed for March 15th ... this would give
> > Thomas his two weeks for the docs freeze ...
>
>I'm interested to know what exactly takes two weeks with the docs and what
>could be done to speed it up.


Isn't it the typsetting for the postscript/pdf docs? docbook doesn't handle 
tables too well in those cases and its easier to do them by hand?

Peter


>--
>Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 01 Mar 2001 20:18:37 +0000
Peter Mount <peter@retep.org.uk>
Re: Release in 2 weeks ... 
>   >I've got a clue for ApplixWare, if you happen to have that package
>   >(US$90).
> Please post it, Thomas.
> I got nowhere following their instructions.

Have you looked at *our* instructions in the chapter on ODBC? I haven't
done much with it in quite a while, but afaik it all should still work.

I would have expected Cary O'Brien (sp? name?? Done from memory: sorry
"aka Cary" :/ to have spoken up if things have broken, so the
instructions should still be good.

                       - Thomas

Fri, 02 Mar 2001 16:28:39 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: [HACKERS] Release in 2 weeks ...
First, a warning is in order.  This will modify your registry.  I have no
idea how it might behave on win2000, but should work for win9x.  As
always, it's wise to backup your registry first.  Use at your own risk.

If you already have some version of the PG odbc driver installed, you can
just copy psqlodbc.dll over the old one.  If not, you have to "install"
it.  There is an installer if you want to use it, but I think it only has
the old version so you will have to copy the new psqlodbc.dll over this
old one.  Or, if you want to, you can use the attached .reg file to modify
your registry.  I'm told that this is all the installer does anyway.  
Give it a try.

-Cedar


On Thu, 1 Mar 2001, Patrick Welche wrote:

> On Wed, Feb 28, 2001 at 08:53:31AM +0900, Hiroshi Inoue wrote:
> ... 
> > I think I've fixed this bug at least for MS-Access.
> > You could get the latest win32 driver from
> >   ftp://ftp.greatbridge.org/pub/pgadmin/stable/psqlodbc.zip .
> > Please try it.
> 
> How can I just install that file? (ie., M$ Access -> psqlodbc.dll -> real OS)
Fri, 2 Mar 2001 21:37:35 +0200 (IST)
Cedar Cox <cedarc@visionforisrael.com>
Re: Re: [HACKERS] Release in 2 weeks ...
On Thu, Mar 01, 2001 at 11:05:02AM +0900, Hiroshi Inoue wrote:
> Patrick Welche wrote:
> > 
> > On Wed, Feb 28, 2001 at 08:53:31AM +0900, Hiroshi Inoue wrote:
> > ...
> > > I think I've fixed this bug at least for MS-Access.
> > > You could get the latest win32 driver from
> > >   ftp://ftp.greatbridge.org/pub/pgadmin/stable/psqlodbc.zip .
> > > Please try it.
> > 
> > How can I just install that file? (ie., M$ Access -> psqlodbc.dll -> real OS)
> > 
> 
> I don't know if M$-access requires MDAC now(it didn't require
> MDAC before). I use ADO and don't use M$-access other than
> testing. ADO requires MDAC and pgAdmin uses ADO AFAIK.

Indeed M$-access doesn't need it. Thanks to Emmanuel and Cedar for the
explanation (also my fault for having searched for psqlodbc with "partial
match" to find it in c:\winnt\system32)

...
> > And after searching said website,
> > http://www.microsoft.com/data/download2.htm
> > shows:
> > 
> > Microsoft Data Access Components MDAC 2.1.1.3711.11      < 2.5...
> > 
> 
> I can see the following at http://www.microsoft.com/data/download.htm

Now how come you found download.htm and I got download2.htm?! Thanks a lot!

Cheers,

Patrick

Mon, 5 Mar 2001 11:41:40 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: Re: [HACKERS] Release in 2 weeks ...
I really feel that translated error messages need to happen soon.
Managing translated message catalogs can be done easily with available
APIs.  However, translatable messages really require an error code
mechanism (otherwise it's completely impossible for programs to interpret
error messages reliably).  I've been thinking about this for much too long
now and today I finally settled to the simplest possible solution.

Let the actual method of allocating error codes be irrelevant for now,
although the ones in the SQL standard are certainly to be considered for a
start.  Essentially, instead of writing

    elog(ERROR, "disaster struck");

you'd write

    elog(ERROR, "XYZ01", "disaster struck");

Now you'll notice that this approach doesn't make the error message text
functionally dependend on the error code.  The alternative would have been
to write

    elog(ERROR, "XYZ01");

which makes the code much less clear.  Additonally, most of the elog()
calls use printf style variable argument lists.  So maybe

    elog(ERROR, "XYZ01", (arg + 1), foo);

This is not only totally obscure, but also incredibly cumbersome to
maintain and very error prone.  One earlier idea was to make the "XYZ01"
thing a macro instead that expands to a string with % arguments, that GCC
can check as it does now.  But I don't consider this a lot better, because
the initial coding is still obscured, and additonally the list of those
macros needs to be maintained.  (The actual error codes might still be
provided as readable macro names similar to the errno codes, but I'm not
sure if we should share these between server and client.)

Finally, there might also be legitimate reasons to have different error
message texts for the same error code.  For example, "type errors" (don't
know if this is an official code) can occur in a number of places that
might warrant different explanations.  Indeed, this approach would
preserve "artistic freedom" to some extent while still maintaining some
structure alongside.  And it would be rather straightforward to implement,
too.  Those who are too bored to assign error codes to new code can simply
pick some "zero" code as default.

On the protocol front, this could be pretty easy to do.  Instead of
"message text" we'd send a string "XYZ01: message text".  Worst case, we
pass this unfiltered to the client and provide an extra function that
returns only the first five characters.  Alternatively we could strip off
the prefix when returning the message text only.

At the end, the i18n part would actually be pretty easy, e.g.,

    elog(ERROR, "XYZ01", gettext("stuff happened"));


Comments?  Better ideas?

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 8 Mar 2001 23:49:50 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Internationalized error messages
Peter Eisentraut <peter_e@gmx.net> writes:

> Let the actual method of allocating error codes be irrelevant for now,
> although the ones in the SQL standard are certainly to be considered for a
> start.  Essentially, instead of writing
> 
>     elog(ERROR, "disaster struck");
> 
> you'd write
> 
>     elog(ERROR, "XYZ01", "disaster struck");

I like this approach.  One of the nice things about Oracle is that
they have an error manual.  All Oracle errors have an associated
number.  You can look up that number in the error manual to find a
paragraph giving details and workarounds.  Admittedly, sometimes the
further details are not helpful, but sometimes they are.  The basic
idea of being able to look up an error lets programmers balance the
need for a terse error message with the need for a fuller explanation.

Ian

---------------------------(end of broadcast)---------------------------
TIP 32: I just know I'm a better manager when I have Joe DiMaggio in center field.
		-- Casey Stengel

08 Mar 2001 16:16:17 -0800
Ian Lance Taylor <ian@airs.com>
Re: Internationalized error messages
On Thu, Mar 08, 2001 at 11:49:50PM +0100, Peter Eisentraut wrote:
> I really feel that translated error messages need to happen soon.
> Managing translated message catalogs can be done easily with available
> APIs.  However, translatable messages really require an error code
> mechanism (otherwise it's completely impossible for programs to interpret
> error messages reliably).  I've been thinking about this for much too long
> now and today I finally settled to the simplest possible solution.
> 
> Let the actual method of allocating error codes be irrelevant for now,
> although the ones in the SQL standard are certainly to be considered for a
> start.  Essentially, instead of writing
> 
>     elog(ERROR, "disaster struck");
> 
> you'd write
> 
>     elog(ERROR, "XYZ01", "disaster struck");
> 
> Now you'll notice that this approach doesn't make the error message text
> functionally dependend on the error code.  The alternative would have been
> to write
> 
>     elog(ERROR, "XYZ01");
> 
> which makes the code much less clear.  Additonally, most of the elog()
> calls use printf style variable argument lists.  So maybe
> 
>     elog(ERROR, "XYZ01", (arg + 1), foo);
> 
> This is not only totally obscure, but also incredibly cumbersome to
> maintain and very error prone.  One earlier idea was to make the "XYZ01"
> thing a macro instead that expands to a string with % arguments, that GCC
> can check as it does now.  But I don't consider this a lot better, because
> the initial coding is still obscured, and additonally the list of those
> macros needs to be maintained.  (The actual error codes might still be
> provided as readable macro names similar to the errno codes, but I'm not
> sure if we should share these between server and client.)
> 
> Finally, there might also be legitimate reasons to have different error
> message texts for the same error code.  For example, "type errors" (don't
> know if this is an official code) can occur in a number of places that
> might warrant different explanations.  Indeed, this approach would
> preserve "artistic freedom" to some extent while still maintaining some
> structure alongside.  And it would be rather straightforward to implement,
> too.  Those who are too bored to assign error codes to new code can simply
> pick some "zero" code as default.
> 
> On the protocol front, this could be pretty easy to do.  Instead of
> "message text" we'd send a string "XYZ01: message text".  Worst case, we
> pass this unfiltered to the client and provide an extra function that
> returns only the first five characters.  Alternatively we could strip off
> the prefix when returning the message text only.
> 
> At the end, the i18n part would actually be pretty easy, e.g.,
> 
>     elog(ERROR, "XYZ01", gettext("stuff happened"));

Similar approaches have been tried frequently, and even enshrined 
in standards (e.g. POSIX catgets), but have almost always proven too
cumbersome.  The problem is that keeping programs that interpret the 
numeric code in sync with the program they monitor is hard, and trying 
to avoid breaking all those secondary programs hinders development on 
the primary program.  Furthermore, assigning code numbers is a nuisance,
and they add uninformative clutter.  

It's better to scan the program for elog() arguments, and generate
a catalog by using the string itself as the index code.  Those 
maintaining the secondary programs can compare catalogs to see what 
has been broken by changes and what new messages to expect.  elog()
itself can (optionally) invent tokens (e.g. catalog indices) to help 
out those programs.

Nathan Myers
ncm@zembu.com

Thu, 8 Mar 2001 16:42:22 -0800
ncm@zembu.com (Nathan Myers)
Re: Internationalized error messages
> On Thu, Mar 08, 2001 at 11:49:50PM +0100, Peter Eisentraut wrote:
>> I really feel that translated error messages need to happen soon.

Agreed.

ncm@zembu.com (Nathan Myers) writes:
> Similar approaches have been tried frequently, and even enshrined 
> in standards (e.g. POSIX catgets), but have almost always proven too
> cumbersome.  The problem is that keeping programs that interpret the 
> numeric code in sync with the program they monitor is hard, and trying 
> to avoid breaking all those secondary programs hinders development on 
> the primary program.  Furthermore, assigning code numbers is a nuisance,
> and they add uninformative clutter.  

There's a difficult tradeoff to make here, but I think we do want to
distinguish between the "official error code" --- the thing that has
translations into various languages --- and what the backend is actually
allowed to print out.  It seems to me that a fairly large fraction of
the unique messages found in the backend can all be lumped under the
category of "internal error", and that we need to have only one official
error code and one user-level translated message for the lot of them.
But we do want to be able to print out different detail messages for
each of those internal errors.  There are other categories that might be
lumped together, but that one alone is sufficiently large to force us
to recognize it.  This suggests a distinction between a "primary" or
"user-level" error message, which we catalog and provide translations
for, and a "secondary", "detail", or "wizard-level" error message that
exists only in the backend source code, and only in English, and so
can be made up on the spur of the moment.

Another thing that's bothered me for a long time is our inconsistent
approach to determining where in the code a message comes from.  A lot
of the messages currently embed the name of the generating routine right
into the error text.  Again, we ought to separate the functionality:
the source-code location is valuable but ought not form part of the
primary error message.  I would like to see elog() become a macro that
invokes __FILE__ and __LINE__ to automatically make the *exact* source
code location become part of the secondary error information, and then
drop the convention of using the routine name in the message text.

Something else we have talked about off-and-on is providing locator
information for errors that can be associated with a particular point in
the query string (lexical and syntactic errors).  This would probably be
best returned as a character index.

Another thing that I missed in Peter's proposal is how we are going to
cope with messages that include parameters.  Surely we do not expect
gettext to start with 'Attribute "foo" not found' and distinguish fixed
from variable parts of that string?

So it's clear that we need to devise a way of breaking an "error
message" into multiple portions, including:

	Primary error message (localizable)
	Parameters to insert into error message (user identifiers, etc)
	Secondary (wizard) error message (optional)
	Source code location
	Query text location (optional)

and perhaps others that I have forgotten about.  One of the key things
to think about is whether we can, or should try to, transmit all this
stuff in a backwards-compatible protocol.  That would mean we'd have
to dump all the info into a single string, which is doable but would
perhaps look pretty ugly:

	ERROR: Attribute "foo" not found  -- basic message for dumb frontends
	ERRORCODE: UNREC_IDENT		-- key for finding localized message
	PARAM1: foo	-- something to embed in the localized message
	MESSAGE: Attribute or table name not known within context of query
	CODELOC: src/backend/parser/parse_clause.c line 345
	QUERYLOC: 22

Alternatively we could suppress most of this stuff unless the frontend
specifically asks for it (and presumably is willing to digest it for
the user).

Bottom line for me is that if we are going to go to the trouble of
examining and changing every single elog() in the system, we should
try to get all of these issues cleaned up at once.  Let's not have to
go back and do it again later.

			regards, tom lane

Thu, 08 Mar 2001 21:00:09 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Internationalized error messages 
On Thu, Mar 08, 2001 at 09:00:09PM -0500, Tom Lane wrote:
> ncm@zembu.com (Nathan Myers) writes:
> > Similar approaches have been tried frequently, and even enshrined 
> > in standards (e.g. POSIX catgets), but have almost always proven too
> > cumbersome.  The problem is that keeping programs that interpret the 
> > numeric code in sync with the program they monitor is hard, and trying 
> > to avoid breaking all those secondary programs hinders development on 
> > the primary program.  Furthermore, assigning code numbers is a nuisance,
> > and they add uninformative clutter.  
> 
> There's a difficult tradeoff to make here, but I think we do want to
> distinguish between the "official error code" --- the thing that has
> translations into various languages --- and what the backend is actually
> allowed to print out.  It seems to me that a fairly large fraction of
> the unique messages found in the backend can all be lumped under the
> category of "internal error", and that we need to have only one official
> error code and one user-level translated message for the lot of them.
> But we do want to be able to print out different detail messages for
> each of those internal errors.  There are other categories that might be
> lumped together, but that one alone is sufficiently large to force us
> to recognize it.  This suggests a distinction between a "primary" or
> "user-level" error message, which we catalog and provide translations
> for, and a "secondary", "detail", or "wizard-level" error message that
> exists only in the backend source code, and only in English, and so
> can be made up on the spur of the moment.

I suggest using different named functions/macros for different 
categories of message, rather than arguments to a common function.  
(I.e. "elog(ERROR, ...)" Considered Harmful.)  

You might even have more than one call at a site, one for the official
message and another for unofficial or unstable informative details.

> Another thing that I missed in Peter's proposal is how we are going to
> cope with messages that include parameters.  Surely we do not expect
> gettext to start with 'Attribute "foo" not found' and distinguish fixed
> from variable parts of that string?

The common way to deal with this is to catalog the format string itself,
with its embedded % directives.  The tricky bit, and what the printf 
family has had to be extended to handle, is that the order of the formal 
arguments varies with the target language.  The original string is an 
ordinary printf string, but the translations may have to refer to the 
substitution arguments by numeric position (as well as type).

There is probably Free code to implement that.

As much as possible, any compile-time annotations should be extracted 
into the catalog and filtered out of the source, to be reunited only
when you retrieve the catalog entry.  


> So it's clear that we need to devise a way of breaking an "error
> message" into multiple portions, including:
> 
> 	Primary error message (localizable)
> 	Parameters to insert into error message (user identifiers, etc)
> 	Secondary (wizard) error message (optional)
> 	Source code location
> 	Query text location (optional)
> 
> and perhaps others that I have forgotten about.  One of the key things
> to think about is whether we can, or should try to, transmit all this
> stuff in a backwards-compatible protocol.  That would mean we'd have
> to dump all the info into a single string, which is doable but would
> perhaps look pretty ugly:
> 
> 	ERROR: Attribute "foo" not found  -- basic message for dumb frontends
> 	ERRORCODE: UNREC_IDENT		-- key for finding localized message
> 	PARAM1: foo	-- something to embed in the localized message
> 	MESSAGE: Attribute or table name not known within context of query
> 	CODELOC: src/backend/parser/parse_clause.c line 345
> 	QUERYLOC: 22

Whitespace can be used effectively.  E.g. only primary messages appear
in column 0.  PG might emit this, which is easily filtered:

   Attribute "foo" not found
    severity: cannot proceed
    explain: An attribute or table was name not known within
    explain: the context of the query.
    index: 237 Attribute \"%s\" not found
    location: src/backend/parser/parse_clause.c line 345
    query_position: 22

Here the first line is the localized replacement of what appears in the 
code, with arguments substituted in.   The other stuff comes from the
catalog

The call looks like

  elog_query("Attribute \"%s\" not found", foo);
  elog_explain("An attribute or table was name not known within"
               "the context of the query.");
  elog_severity(ERROR);

which might gets expanded (squeezed) by the preprocessor to

  _elog(current_query_position, "Attribute \"%s\" not found", foo);

while a separate tool scans the sources and builds the catalog,
annotating it with severity, line number, etc.  Human translators
may edit copies of the resulting catalog.  The call to _elog looks up
the string in the catalog, substitutes arguments into the translation,
and emits it along with the catalog index number and whatever else
has been requested in the config file.  Alternatively, any other program 
can use the number to pull the annotations out of the catalog given
just the index.

> Alternatively we could suppress most of this stuff unless the frontend
> specifically asks for it (and presumably is willing to digest it for
> the user).
> 
> Bottom line for me is that if we are going to go to the trouble of
> examining and changing every single elog() in the system, we should
> try to get all of these issues cleaned up at once.  Let's not have to
> go back and do it again later.

The more complex it is, the more likely that will need to be redone.
The simpler the calls look, the more likely that you can automate
(or implement invisibly) any later improvements.  

Nathan Myers
ncm@zembu.com

Thu, 8 Mar 2001 19:30:41 -0800
ncm@zembu.com (Nathan Myers)
Re: Internationalized error messages
> I like this approach.  One of the nice things about Oracle is that
> they have an error manual.  All Oracle errors have an associated
> number.  You can look up that number in the error manual to find a
> paragraph giving details and workarounds.  Admittedly, sometimes the
> further details are not helpful, but sometimes they are.  The basic
> idea of being able to look up an error lets programmers balance the
> need for a terse error message with the need for a fuller explanation.

One of the examples when you need exact error message code is when you want 
to separate unique index violations from other errors. This often needed when 
you want just do insert, and leave all constraint checking to database...

-- 
Sincerely Yours,
Denis Perchine

----------------------------------
E-Mail: dyp@perchine.com
HomePage: http://www.perchine.com/dyp/
FidoNet: 2:5000/120.5
----------------------------------

Fri, 9 Mar 2001 11:34:42 +0600
Denis Perchine <dyp@perchine.com>
Re: Internationalized error messages
On Thu, Mar 08, 2001 at 09:00:09PM -0500, Tom Lane wrote:
> > On Thu, Mar 08, 2001 at 11:49:50PM +0100, Peter Eisentraut wrote:
> >> I really feel that translated error messages need to happen soon.
> 
> Agreed.

 Yes, error codes is *very* wanted feature.

> 
> 	ERROR: Attribute "foo" not found  -- basic message for dumb frontends
> 	ERRORCODE: UNREC_IDENT		-- key for finding localized message
> 	PARAM1: foo	-- something to embed in the localized message
> 	MESSAGE: Attribute or table name not known within context of query
> 	CODELOC: src/backend/parser/parse_clause.c line 345
> 	QUERYLOC: 22

 Great idea! I agree that we need some powerful Error protocol instead 
currect string based messages.
 
 For transaltion to other languages I not sure with gettext() stuff on
backend -- IMHO better (faster) solution will postgres system catalog
with it.

 May be add new command too: SET MESSAGE_LANGUAGE TO <xxx>, because
wanted language not must be always same as locale setting.

 Something like elog(ERROR, gettext(...)); is usable, but not sounds good 
for me.

			Karel

-- 
 Karel Zak  <zakkr@zf.jcu.cz>
 http://home.zf.jcu.cz/~zakkr/
 
 C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz

Fri, 9 Mar 2001 08:53:20 +0100
Karel Zak <zakkr@zf.jcu.cz>
Re: Internationalized error messages
>  For transaltion to other languages I not sure with gettext() stuff on
> backend -- IMHO better (faster) solution will postgres system catalog
> with it.
> 
>  May be add new command too: SET MESSAGE_LANGUAGE TO <xxx>, because
> wanted language not must be always same as locale setting.

In the multibyte enabled environment, that kind of command would not
be necessary except UNICODE and MULE_INTERNAL, since they are
multi-lingual encoding. For them, we might need something like:

SET LANGUAGE_PREFERENCE TO 'Japanese';

For the long term solutuon, this kind of problem should be solved in
the implemetaion of SQL-92/99 i18n features.
--
Tatsuo Ishii

Fri, 09 Mar 2001 20:42:26 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Internationalized error messages
Nathan Myers writes:

> >     elog(ERROR, "XYZ01", gettext("stuff happened"));
>
> Similar approaches have been tried frequently, and even enshrined
> in standards (e.g. POSIX catgets), but have almost always proven too
> cumbersome.  The problem is that keeping programs that interpret the
> numeric code in sync with the program they monitor is hard, and trying
> to avoid breaking all those secondary programs hinders development on
> the primary program.

That's why no one uses catgets and everyone uses gettext.

> Furthermore, assigning code numbers is a nuisance, and they add
> uninformative clutter.

The error codes are exactly what we want, to allow client programs (as
opposed to humans) to identify the errors.  The code in my example has
nothing to do with the message id in the catgets interface.

> It's better to scan the program for elog() arguments, and generate
> a catalog by using the string itself as the index code.  Those
> maintaining the secondary programs can compare catalogs to see what
> has been broken by changes and what new messages to expect.  elog()
> itself can (optionally) invent tokens (e.g. catalog indices) to help
> out those programs.

That's what gettext does for you.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Fri, 9 Mar 2001 16:45:54 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Internationalized error messages
Tom Lane writes:

> There's a difficult tradeoff to make here, but I think we do want to
> distinguish between the "official error code" --- the thing that has
> translations into various languages --- and what the backend is actually
> allowed to print out.  It seems to me that a fairly large fraction of
> the unique messages found in the backend can all be lumped under the
> category of "internal error", and that we need to have only one official
> error code and one user-level translated message for the lot of them.

That's exactly what I was trying to avoid.  You'd still be allowed to
choose the error message text freely, but client programs will be able to
make sense of them by looking at the code only, as opposed to parsing the
message text.  I'm trying to avoid making the message text to be computed
from the error code, because that obscures the source code.

> Another thing that's bothered me for a long time is our inconsistent
> approach to determining where in the code a message comes from.  A lot
> of the messages currently embed the name of the generating routine right
> into the error text.  Again, we ought to separate the functionality:
> the source-code location is valuable but ought not form part of the
> primary error message.  I would like to see elog() become a macro that
> invokes __FILE__ and __LINE__ to automatically make the *exact* source
> code location become part of the secondary error information, and then
> drop the convention of using the routine name in the message text.

These sort of things have been on my mind as well, but they're really
independent of my issue.  We can easily have runtime options to append or
not additional things to the error string.  I don't see this as part of my
proposal.

> Another thing that I missed in Peter's proposal is how we are going to
> cope with messages that include parameters.  Surely we do not expect
> gettext to start with 'Attribute "foo" not found' and distinguish fixed
> >from variable parts of that string?

Sure we do.

> That would mean we'd have to dump all the info into a single string,
> which is doable but would perhaps look pretty ugly:
>
> 	ERROR: Attribute "foo" not found  -- basic message for dumb frontends
> 	ERRORCODE: UNREC_IDENT		-- key for finding localized message

There should not be a "key" to look up localized messages.  Remember that
the localization will also have to be done in all the front-end programs.
Surely we do not wish to make a list of messages that pg_dump or psql
print out.  Gettext takes care of this stuff.  The only reason why we need
error codes is for the sake of ease of interpreting by programs.

> 	PARAM1: foo	-- something to embed in the localized message

Not necessary.

> 	MESSAGE: Attribute or table name not known within context of query

How's that different from ERROR:?

> 	CODELOC: src/backend/parser/parse_clause.c line 345

Can be appended to ERROR (or MESSAGE) depending on configuration setting.

> 	QUERYLOC: 22

Not all errors are related to a query.

The general problem here is also that this would introduce a client
incompatibility.  Older clients that do not expect this amount of detail
will print all this garbage to the screen?

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Fri, 9 Mar 2001 16:57:18 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Internationalized error messages 
Peter Eisentraut <peter_e@gmx.net> writes:
> That's exactly what I was trying to avoid.  You'd still be allowed to
> choose the error message text freely, but client programs will be able to
> make sense of them by looking at the code only, as opposed to parsing the
> message text.  I'm trying to avoid making the message text to be computed
> from the error code, because that obscures the source code.

I guess I don't understand what you have in mind, because this seems
self-contradictory.  If "client programs can look at the code only",
then how can the error message text be chosen independently of the code?

>> Surely we do not expect gettext to start with 'Attribute "foo" not
>> found' and distinguish fixed from variable parts of that string?

> Sure we do.

How does that work exactly?  You're assuming an extremely intelligent
localization mechanism, I guess, which I was not.  I think it makes more
sense to work a little harder in the backend to avoid requiring AI
software in every frontend.

>> MESSAGE: Attribute or table name not known within context of query

> How's that different from ERROR:?

Sorry, I meant that as an example of the "secondary message string", but
it's a pretty lame example...

> The general problem here is also that this would introduce a client
> incompatibility.  Older clients that do not expect this amount of detail
> will print all this garbage to the screen?

Yes, if we send it to them.  It would make sense to control the amount
of detail presented via some option (a GUC variable, probably).  For
backwards compatibility reasons we'd want the default to correspond to
roughly the existing amount of detail.

			regards, tom lane

Fri, 09 Mar 2001 11:00:53 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Internationalized error messages 
Tom Lane writes:

> I guess I don't understand what you have in mind, because this seems
> self-contradictory.  If "client programs can look at the code only",
> then how can the error message text be chosen independently of the code?

Let's say "type mismatch error", code 2200G acc. to SQL.  At one place in
the source you write

elog(ERROR, "2200G", "type mismatch in CASE expression (%s vs %s)", ...);

Elsewhere you'd write

elog(ERROR, "2200G", "type mismatch in argument %d of function %s,
     expected %s, got %s", ...);

Humans can look at this and have a fairly good idea what they'd need to
fix.  However, a client program currently only has the option of failing
or not failing.  In this example case it would probably better for it to
fail, but someone else already put forth the example of constraint
violation.  In this case the program might want to do something else.

> >> Surely we do not expect gettext to start with 'Attribute "foo" not
> >> found' and distinguish fixed from variable parts of that string?
>
> > Sure we do.
>
> How does that work exactly?  You're assuming an extremely intelligent
> localization mechanism, I guess, which I was not.  I think it makes more
> sense to work a little harder in the backend to avoid requiring AI
> software in every frontend.

Gettext takes care of this.  In the source you'd write

elog(ERROR, "2200G", gettext("type mismatch in CASE expression (%s vs %s)"),
            string, string);

When you run the xgettext utility program it scans the source for cases of
gettext(...) and creates message catalogs for the translators.  When it
finds printf arguments it automatically includes marks in the message,
such as

"type mismatch in CASE expression (%1$s vs %2$s)"

which the translator better keep in his version.  This also handles the
case where the arguments might have to appear in a different order in a
different language.

> Sorry, I meant that as an example of the "secondary message string", but
> it's a pretty lame example...

I guess I'm not sold on the concept of primary and secondary message
strings.  If the primary message isn't good enough you better fix that.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Fri, 9 Mar 2001 17:50:28 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Internationalized error messages 
Karel Zak writes:

>  For transaltion to other languages I not sure with gettext() stuff on
> backend -- IMHO better (faster) solution will postgres system catalog
> with it.

elog(ERROR, "cannot open message catalog table");

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Fri, 9 Mar 2001 17:57:13 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Internationalized error messages
Peter Eisentraut <peter_e@gmx.net> writes:
> Let's say "type mismatch error", code 2200G acc. to SQL.  At one place in
> the source you write
> elog(ERROR, "2200G", "type mismatch in CASE expression (%s vs %s)", ...);
> Elsewhere you'd write
> elog(ERROR, "2200G", "type mismatch in argument %d of function %s,
>      expected %s, got %s", ...);

Okay, so your notion of an error code is not a localizable entity at
all, it's something for client programs to look at.  Now I get it.

I object to writing "2200G" however, because that has no mnemonic value
whatever, and is much too easy to get wrong.  How about

elog(ERROR, ERR_TYPE_MISMATCH, "type mismatch in argument %d of function %s,
     expected %s, got %s", ...);

where ERR_TYPE_MISMATCH is #defined as "2200G" someplace?  Or for that
matter #defined as "TYPE_MISMATCH"?  Content-free numeric codes are no
fun to use on the client side either...

> Gettext takes care of this.  In the source you'd write

> elog(ERROR, "2200G", gettext("type mismatch in CASE expression (%s vs %s)"),
>             string, string);

Duh.  For some reason I was envisioning the localization substitution as
occurring on the client side, but of course we'd want to do it on the
server side, and before parameters are substituted into the message.
Sorry for the noise.

I am not sure we can/should use gettext (possible license problems?),
but certainly something like this could be cooked up.

>> Sorry, I meant that as an example of the "secondary message string", but
>> it's a pretty lame example...

> I guess I'm not sold on the concept of primary and secondary message
> strings.  If the primary message isn't good enough you better fix that.

The motivation isn't so much to improve on the primary message as to
reduce the number of distinct strings that really need to be translated.
Remember all those internal "can't happen" errors.  If we have only one
message component then the translator is faced with a huge pile of
internal messages and not a lot of gain from translating them.  If
there's a primary and secondary component then all the internal messages
can share the same primary component ("Internal error, please file a bug
report").  Now the translator translates that one message, and can
ignore the many secondary-component messages with a clear conscience.
(Of course, he can translate those too if he really wants to, but the
point is that he doesn't *have* to do it to attain reasonably friendly
behavior.)

Perhaps another way to look at it is that we have a bunch of errors that
are user-oriented (ie, relate pretty directly to something the user did
wrong) and another bunch that are system-oriented (relate to internal
problems, such as consistency check failures or violations of internal
APIs).  We want to provide localized translations of the first set, for
sure.  I don't think we need localized translations of the second set,
so long as we have some sort of "covering message" that can be localized
for them.  Maybe instead of "primary" and "secondary" strings for a
single error, we ought to distinguish these two categories of error and
plan different localization strategies for them.

			regards, tom lane

Fri, 09 Mar 2001 12:05:22 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Internationalized error messages 
> Peter Eisentraut <peter_e@gmx.net> writes:
> > Let's say "type mismatch error", code 2200G acc. to SQL.  At one place in
> > the source you write
> > elog(ERROR, "2200G", "type mismatch in CASE expression (%s vs %s)", ...);

Tom Lane <tgl@sss.pgh.pa.us> spake:
> I object to writing "2200G" however, because that has no mnemonic value
> whatever, and is much too easy to get wrong.  How about
> 
> elog(ERROR, ERR_TYPE_MISMATCH, "type mismatch in argument %d of function %s,
>      expected %s, got %s", ...);
> 
> where ERR_TYPE_MISMATCH is #defined as "2200G" someplace?  Or for that
> matter #defined as "TYPE_MISMATCH"?  Content-free numeric codes are no
> fun to use on the client side either...

This is one thing I think VMS does well.  All error messages are a
composite of the subsystem where they originated, the severity of the
error, and the actual error itself.  Internally this is stored in a
32-bit word.  It's been a long time, so I don't recall how many bits
they allocated for each component.  The human-readable representation
looks like "<subsystem>-<severity>-<error>".

--
Andrew Evans

Fri, 9 Mar 2001 11:43:20 -0800
Andrew Evans <andrew@zembu.com>
Re: Internationalized error messages
On Fri, Mar 09, 2001 at 12:05:22PM -0500, Tom Lane wrote:
> > Gettext takes care of this.  In the source you'd write
> 
> > elog(ERROR, "2200G", gettext("type mismatch in CASE expression (%s vs %s)"),
> >             string, string);
> 
> Duh.  For some reason I was envisioning the localization substitution as
> occurring on the client side, but of course we'd want to do it on the
> server side, and before parameters are substituted into the message.
> Sorry for the noise.
> 
> I am not sure we can/should use gettext (possible license problems?),
> but certainly something like this could be cooked up.

I've been assuming that PG's needs are specialized enough that the
project wouldn't use gettext directly, but instead something inspired 
by it.  

If you look at my last posting on the subject, by the way, you will see 
that it could work without a catalog underneath; integrating a catalog 
would just require changes in a header file (and the programs to generate 
the catalog, of course).  That quality seems to me essential to allow the 
changeover to be phased in gradually, and to allow different underlying 
catalog implementations to be tried out.

Nathan
ncm

Fri, 9 Mar 2001 11:49:20 -0800
ncm@zembu.com (Nathan Myers)
Re: Internationalized error messages
Tom Lane writes:

> I object to writing "2200G" however, because that has no mnemonic value
> whatever, and is much too easy to get wrong.  How about
>
> elog(ERROR, ERR_TYPE_MISMATCH, "type mismatch in argument %d of function %s,
>      expected %s, got %s", ...);
>
> where ERR_TYPE_MISMATCH is #defined as "2200G" someplace?  Or for that
> matter #defined as "TYPE_MISMATCH"?  Content-free numeric codes are no
> fun to use on the client side either...

Well, SQL defines these.  Do we want to make our own list?  However,
numeric codes also have the advantage that some hierarchy is possible.
E.g., the "22" in "2200G" is actually the category code "data exception".
Personally, I would stick to the SQL codes but make some readable macro
name for backend internal use.

> I am not sure we can/should use gettext (possible license problems?),

Gettext is an open standard, invented at Sun IIRC.  There is also an
independent implementation for BSDs in the works.  On GNU/Linux system
it's in the C library.  I don't see any license problems that way.  Is has
been used widely for free software and so far I haven't seen any real
alternative.

> but certainly something like this could be cooked up.

Well, I'm trying to avoid having to do the cooking. ;-)

> Perhaps another way to look at it is that we have a bunch of errors that
> are user-oriented (ie, relate pretty directly to something the user did
> wrong) and another bunch that are system-oriented (relate to internal
> problems, such as consistency check failures or violations of internal
> APIs).  We want to provide localized translations of the first set, for
> sure.  I don't think we need localized translations of the second set,
> so long as we have some sort of "covering message" that can be localized
> for them.

I'm sure this can be covered in some macro way.  A random idea:

elog(ERROR, INTERNAL_ERROR("text"), ...)

expands to

elog(ERROR, gettext("Internal error: %s"), ...)

OTOH, we should not yet make presumptions about what dedicated translators
can be capable of.  :-)

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Fri, 9 Mar 2001 21:45:14 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Internationalized error messages 
Peter Eisentraut <peter_e@gmx.net> writes:
> Well, SQL defines these.  Do we want to make our own list?  However,
> numeric codes also have the advantage that some hierarchy is possible.
> E.g., the "22" in "2200G" is actually the category code "data exception".
> Personally, I would stick to the SQL codes but make some readable macro
> name for backend internal use.

We will probably find cases where we need codes not defined by SQL
(since we have non-SQL features).  If there is room to invent our
own codes then I have no objection to this.

>> I am not sure we can/should use gettext (possible license problems?),

> Gettext is an open standard, invented at Sun IIRC.  There is also an
> independent implementation for BSDs in the works.  On GNU/Linux system
> it's in the C library.  I don't see any license problems that way.

Unless that BSD implementation is ready to go, I think we'd be talking
about relying on GPL'd (not LGPL'd) code for an essential component of
the system functionality.  Given RMS' recent antics I am much less
comfortable with that than I might once have been.

			regards, tom lane

Fri, 09 Mar 2001 15:48:33 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Internationalized error messages 

Tom Lane wrote:

> I am not sure we can/should use gettext (possible license problems?),
> but certainly something like this could be cooked up.

http://citrus.bsdclub.org/index-en.html

I'm not sure of the current status of the code.

Regards,

Giles


Sun, 11 Mar 2001 09:02:16 +1100
Giles Lean <giles@nemeton.com.au>
Re: Internationalized error messages 
On Fri, Mar 09, 2001 at 03:48:33PM -0500, Tom Lane wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > Well, SQL defines these.  Do we want to make our own list?  However,
> > numeric codes also have the advantage that some hierarchy is possible.
> > E.g., the "22" in "2200G" is actually the category code "data exception".
> > Personally, I would stick to the SQL codes but make some readable macro
> > name for backend internal use.
> 
> We will probably find cases where we need codes not defined by SQL
> (since we have non-SQL features).  If there is room to invent our
> own codes then I have no objection to this.
> 
> >> I am not sure we can/should use gettext (possible license problems?),
> 
> > Gettext is an open standard, invented at Sun IIRC.  There is also an
> > independent implementation for BSDs in the works.  On GNU/Linux system
> > it's in the C library.  I don't see any license problems that way.
> 
> Unless that BSD implementation is ready to go, I think we'd be talking
> about relying on GPL'd (not LGPL'd) code for an essential component of
> the system functionality.  Given RMS' recent antics I am much less
> comfortable with that than I might once have been.

cf. http://citrus.bsdclub.org/

and the libintl in NetBSD, at least NetBSD-current, works. The hard part
was eg convincing gmake's configure to use it as there are bits like

#if __USE_GNU_GETTEXT

rather than just checking for the existence of the functions (as well as
the internal symbol _nl_msg_cat_cntr).

So yes it's ready to go, but please don't use the same m4 in configure.in as
for GNU gettext.

Cheers,

Patrick

Sun, 11 Mar 2001 18:11:02 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: Internationalized error messages
On Fri, Mar 09, 2001 at 05:57:13PM +0100, Peter Eisentraut wrote:
> Karel Zak writes:
> 
> >  For transaltion to other languages I not sure with gettext() stuff on
> > backend -- IMHO better (faster) solution will postgres system catalog
> > with it.
> 
> elog(ERROR, "cannot open message catalog table");

 Sure, and what:

elog(ERROR, gettext("can't set LC_MESSAGES"));

 We can generate our system catalog for this by simular way as gettext, it's 
means all messages can be in sources in English too.

 But this is reflexion, performance test show more.

			Karel

-- 
 Karel Zak  <zakkr@zf.jcu.cz>
 http://home.zf.jcu.cz/~zakkr/
 
 C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz

Mon, 12 Mar 2001 10:32:33 +0100
Karel Zak <zakkr@zf.jcu.cz>
Re: Internationalized error messages
At 23:49 08/03/01 +0100, Peter Eisentraut wrote:
>I really feel that translated error messages need to happen soon.
>Managing translated message catalogs can be done easily with available
>APIs.  However, translatable messages really require an error code
>mechanism (otherwise it's completely impossible for programs to interpret
>error messages reliably).  I've been thinking about this for much too long
>now and today I finally settled to the simplest possible solution.
>
>Let the actual method of allocating error codes be irrelevant for now,
>although the ones in the SQL standard are certainly to be considered for a
>start.  Essentially, instead of writing

snip

>On the protocol front, this could be pretty easy to do.  Instead of
>"message text" we'd send a string "XYZ01: message text".  Worst case, we
>pass this unfiltered to the client and provide an extra function that
>returns only the first five characters.  Alternatively we could strip off
>the prefix when returning the message text only.

Most other DB's (I'm thinking of Oracle here) pass the code unfiltered to 
the client anyhow. Saying that, it's not impossible to get psql and other 
interactive clients to strip the error code anyhow.


>At the end, the i18n part would actually be pretty easy, e.g.,
>
>     elog(ERROR, "XYZ01", gettext("stuff happened"));
>
>
>Comments?  Better ideas?

A couple of ideas. One, if we have a master list of error codes, we need to 
have this in an independent format (ie not a .h file). However the other 
idea is to expand on the JDBC's errors.properties files. Being 
ascii/unicode, the format will work with just some extra code to implement 
them in C.

Brief description:
------------------------

The ResourceBundle's handle one language per file. From a base filename, 
each different language has a file based on:

         filename_la_ct.properties

where la is the ISO 2 character language, and ct is the ISO 2 character 
country code.

For example:

messages_en_GB.properties
messages_en_US.properties
messages_en.properties
messages_fr.properties
messages.properties

Now, here for the english locale for England it checks in this order: 
messages_en_GB.properties messages_en.properties messages.properties.

In each file, a message is of the format:

key=message, and each parameter passed into the message written like {1} 
{2} etc, so for example:

fathom=Unable to fathom update count {0}

Now apart from the base file (messages.properties in this case), the other 
files are optional, and an entry only needs to be in there if they are 
present in that language.

So, in french, fathom may be translated, but then again it may not (in JDBC 
it isn't). Then it's not included in the file. Any new messages can be 
added to the base language, but only included as and when they are translated.

Peter


Mon, 12 Mar 2001 15:09:53 +0000
Peter Mount <peter@retep.org.uk>
Re: Internationalized error messages
Karel Zak writes:

> > >  For transaltion to other languages I not sure with gettext() stuff on
> > > backend -- IMHO better (faster) solution will postgres system catalog
> > > with it.
> >
> > elog(ERROR, "cannot open message catalog table");
>
>  Sure, and what:
>
> elog(ERROR, gettext("can't set LC_MESSAGES"));
>
>  We can generate our system catalog for this by simular way as gettext, it's
> means all messages can be in sources in English too.

When there is an error condition in the backend, the last thing you want
to do (and are allowed to do) is accessing tables.  Also keep in mind that
we want to internationalize other parts of the system as well, such as
pg_dump and psql.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Mon, 12 Mar 2001 20:15:02 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Internationalized error messages
> > to re-write smgr. I don't know how useful is second sync() call, but
> > on Solaris (and I believe on many other *NIXes) rc0 calls it
> > three times, -:) Why?
> 
> The idea is, that by the time the last sync has run, the 
> first sync will be done flushing the buffers to disk. - this is what
> we were told by the IBM engineers when I worked tier-2/3 AIX support
> at IBM.

I was told the same a long ago about FreeBSD. How much can we count on
this undocumented sync() feature?

Vadim

Mon, 12 Mar 2001 21:13:44 -0800
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
RE: RE: xlog checkpoint depends on sync() ... seems uns
	afe
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes:
>> The idea is, that by the time the last sync has run, the 
>> first sync will be done flushing the buffers to disk. - this is what
>> we were told by the IBM engineers when I worked tier-2/3 AIX support
>> at IBM.

> I was told the same a long ago about FreeBSD. How much can we count on
> this undocumented sync() feature?

Sounds quite unreliable to me.  Unless there's some interlock ... like,
say, the second sync not being able to advance past a buffer page that's
as yet unwritten by the first sync.  But would all Unixen share such a
strange detail of implementation?

			regards, tom lane

Tue, 13 Mar 2001 00:22:27 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: RE: xlog checkpoint depends on sync() ... seems uns afe 
Tom Lane <tgl@sss.pgh.pa.us> writes:

> "Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes:
> >> The idea is, that by the time the last sync has run, the 
> >> first sync will be done flushing the buffers to disk. - this is what
> >> we were told by the IBM engineers when I worked tier-2/3 AIX support
> >> at IBM.
> 
> > I was told the same a long ago about FreeBSD. How much can we count on
> > this undocumented sync() feature?
> 
> Sounds quite unreliable to me.  Unless there's some interlock ... like,
> say, the second sync not being able to advance past a buffer page that's
> as yet unwritten by the first sync.  But would all Unixen share such a
> strange detail of implementation?

I'm pretty sure it has no basis in fact, it's just one of these habits 
that gives sysadmins a warm fuzzy feeling.  ;)  It's apparently been
around a long time, though I don't remember where I read about it--it
was quite a few years ago.

-Doug


13 Mar 2001 00:42:58 -0500
Doug McNaught <doug@wireboard.com>
Re: RE: xlog checkpoint depends on sync() ... seems uns afe

> Sounds quite unreliable to me.  Unless there's some interlock ... like,
> say, the second sync not being able to advance past a buffer page that's
> as yet unwritten by the first sync.  But would all Unixen share such a
> strange detail of implementation?

I heard Kirk McKusick tell this story in a 4.4BSD internals class.
His explanation was that having an *operator* type 'sync' three times
provided enough time for the first sync to do the work before the
operator powered the system down or reset it or whatever.

I've not heard of any filesystem implementation where the number of
sync() system calls issued makes a difference, and imagine that any
programmer who has written code to call sync three times has only
heard part of the story. :-)

Regards,

Giles


Tue, 13 Mar 2001 17:47:33 +1100
Giles Lean <giles@nemeton.com.au>
Re: RE: xlog checkpoint depends on sync() ... seems uns afe 
On Mon, Mar 12, 2001 at 08:15:02PM +0100, Peter Eisentraut wrote:
> Karel Zak writes:
> 
> > > >  For transaltion to other languages I not sure with gettext() stuff on
> > > > backend -- IMHO better (faster) solution will postgres system catalog
> > > > with it.
> > >
> > > elog(ERROR, "cannot open message catalog table");
> >
> >  Sure, and what:
> >
> > elog(ERROR, gettext("can't set LC_MESSAGES"));
> >
> >  We can generate our system catalog for this by simular way as gettext, it's
> > means all messages can be in sources in English too.
> 
> When there is an error condition in the backend, the last thing you want
> to do (and are allowed to do) is accessing tables.  Also keep in mind that
> we want to internationalize other parts of the system as well, such as
> pg_dump and psql.

 Agree, the pg_xxxx application are good adepts for POSIX locales, all my
previous notes are about backend error/notice messages, but forget it --
after implementation we will more judicious.

-- 
 Karel Zak  <zakkr@zf.jcu.cz>
 http://home.zf.jcu.cz/~zakkr/
 
 C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz

Tue, 13 Mar 2001 08:30:59 +0100
Karel Zak <zakkr@zf.jcu.cz>
Re: Internationalized error messages
On Tue, 13 Mar 2001, Tom Lane wrote:

> > I was told the same a long ago about FreeBSD. How much can we count on
> > this undocumented sync() feature?
> 
> Sounds quite unreliable to me.  Unless there's some interlock ...
> like, say, the second sync not being able to advance past a buffer
> page that's as yet unwritten by the first sync.  But would all Unixen
> share such a strange detail of implementation?

The Linux manpage says:

NAME
       sync - commit buffer cache to disk.
[..]

DESCRIPTION
       sync  first commits inodes to buffers, and then buffers to
       disk.
[..]

CONFORMING TO
       SVr4, SVID, X/OPEN, BSD 4.3

BUGS
       According to  the  standard  specification  (e.g.,  SVID),
       sync()  schedules  the  writes,  but may return before the
       actual writing is done.   However,  since  version  1.3.20
       Linux  does actually wait.  (This still does not guarantee
       data integrity: modern disks have large caches.)


And it's still true.  On a fast system, if you do:

$ cp /dev/zero /tmp & sleep 1; sync

the sync will often never finish.  (Of course, that's
just an implementation detail really.)

Matthew.


Tue, 13 Mar 2001 10:49:00 +0000 (GMT)
Matthew Kirkwood <matthew@hairy.beasts.org>
Re: RE: xlog checkpoint depends on sync() ... seems uns
 afe
Just a quick delurk to pass along this tidbit from linux-kernel on
Linux *sync() behavior, since we've been talking about it a lot...

-Doug


Hi,

On Wed, Mar 14, 2001 at 10:26:42PM -0500, Tom Vier wrote:
> fdatasync() is the same as fsync(), in linux.

No, in 2.4 fdatasync does the right thing and skips the inode flush if
only the timestamps have changed.

> until fdatasync() is
> implimented (ie, syncs the data only)

fdatasync is required to sync more than just the data: it has to sync
the inode too if any fields other than the timestamps have changed.
So, for appending to files or writing new files from scratch, fsync ==
fdatasync (because each write also changes the inode size).  Only for
updating existing files in place does fdatasync behave differently.

> #ifndef O_DSYNC
> # define O_DSYNC O_SYNC
> #endif

2.4's O_SYNC actually does a fdatasync internally.  This is also the
default behaviour of HPUX, which requires you to set a sysctl variable
if you want O_SYNC to flush timestamp changes to disk.

Cheers,
 Stephen
16 Mar 2001 10:15:28 -0500
Doug McNaught <doug@wireboard.com>
["Stephen C. Tweedie" <sct@redhat.com>] Re: O_DSYNC flag for open
Doug McNaught <doug@wireboard.com> forwards:
>> 2.4's O_SYNC actually does a fdatasync internally.  This is also the
>> default behaviour of HPUX, which requires you to set a sysctl variable
>> if you want O_SYNC to flush timestamp changes to disk.

Well, that guy might know all about Linux, but he doesn't know anything
about HPUX (at least not any version I've ever run).  O_SYNC is
distinctly different from O_DSYNC around here.

			regards, tom lane

Fri, 16 Mar 2001 10:55:53 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: ["Stephen C. Tweedie" <sct@redhat.com>] Re: O_DSYNC flag for open
Tom Lane <tgl@sss.pgh.pa.us> writes:

> Doug McNaught <doug@wireboard.com> forwards:
> >> 2.4's O_SYNC actually does a fdatasync internally.  This is also the
> >> default behaviour of HPUX, which requires you to set a sysctl variable
> >> if you want O_SYNC to flush timestamp changes to disk.
> 
> Well, that guy might know all about Linux, but he doesn't know anything
> about HPUX (at least not any version I've ever run).  O_SYNC is
> distinctly different from O_DSYNC around here.

Y'know, I figured that might be the case.  ;)  He's a well-respected
Linux filesystem hacker, so I trust him on the Linux stuff.  

So are we still thinking about preallocating log files as a
performance hack?  It does seem that using preallocated files along
with O_DATASYNC will eliminate pretty much all metadata writes under
Linux in future...

[NOT suggesting we try to add anything to 7.1, I'm eagerly awaiting RC1]

-Doug

16 Mar 2001 12:03:46 -0500
Doug McNaught <doug@wireboard.com>
Re: ["Stephen C. Tweedie" <sct@redhat.com>] Re: O_DSYNC flag for open
Doug McNaught <doug@wireboard.com> writes:
> So are we still thinking about preallocating log files as a
> performance hack?

We're not just thinking about it, we're doing it in current sources ...

			regards, tom lane

Fri, 16 Mar 2001 12:29:59 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: ["Stephen C. Tweedie" <sct@redhat.com>] Re: O_DSYNC flag for open
> So are we still thinking about preallocating log files as a
> performance hack?  It does seem that using preallocated files along
> with O_DATASYNC will eliminate pretty much all metadata writes under
> Linux in future...
> 
> [NOT suggesting we try to add anything to 7.1, I'm eagerly awaiting RC1]

I am pretty sure that is done.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 16 Mar 2001 12:50:41 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ["Stephen C. Tweedie" <sct@redhat.com>] Re: O_DSYNC flag for open

[ Drifting off topic ... ]

> Well, that guy might know all about Linux, but he doesn't know anything
> about HPUX (at least not any version I've ever run).  O_SYNC is
> distinctly different from O_DSYNC around here.

There is a HP_UX kernel flag 'o_sync_is_o_dsync' which will cause
O_DSYNC to be treated as O_SYNC.  It defaults to being off -- it
is/was a backward compatibility "feature" since HP-UX 9.X (which is
history now) had implemented O_SYNC as O_DSYNC.

http://docs.hp.com/cgi-bin/otsearch/getfile?id=/hpux/onlinedocs/os/KCparam.OsyncIsOdsync.html

Regards,

Giles




Sat, 17 Mar 2001 08:50:58 +1100
Giles Lean <giles@nemeton.com.au>
Re: ["Stephen C. Tweedie" <sct@redhat.com>] Re: O_DSYNC flag for open

> There is a HP_UX kernel flag 'o_sync_is_o_dsync' which will cause
> O_DSYNC to be treated as O_SYNC.  It defaults to being off -- it

... other way around there, of course.  Trying to clarify and
adding confusion instead. :-(

> is/was a backward compatibility "feature" since HP-UX 9.X (which is
> history now) had implemented O_SYNC as O_DSYNC.

Muttering,

Giles

Sat, 17 Mar 2001 09:33:05 +1100
Giles Lean <giles@nemeton.com.au>
Re: ["Stephen C. Tweedie" <sct@redhat.com>] Re: O_DSYNC flag for open

Okay folks ...

	We'd like to wrap up an RC1 and get this release happening this
year sometime :)  Tom mentioned to me that he has no outstandings left on
his plate ... does anyone else have any *show stoppers* left that need to
be addressed, or can I package things up?

	Speak now, or forever hold your piece (where forever is the time
between now and RC1 is packaged) ...



Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org


Tue, 20 Mar 2001 14:11:18 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Final Call: RC1 about to go out the door ...
The Hermit Hacker <scrappy@hub.org> writes:
> 	Speak now, or forever hold your piece (where forever is the time
> between now and RC1 is packaged) ...

I rather hope it's *NOT* ....

			regards, tom lane

Tue, 20 Mar 2001 13:17:16 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Final Call: RC1 about to go out the door ... 
* Tom Lane <tgl@sss.pgh.pa.us> [010320 10:21] wrote:
> The Hermit Hacker <scrappy@hub.org> writes:
> > 	Speak now, or forever hold your piece (where forever is the time
> > between now and RC1 is packaged) ...
> 
> I rather hope it's *NOT* ....

And still no LAZY vacuum.  *sigh*

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]


Tue, 20 Mar 2001 10:35:38 -0800
Alfred Perlstein <bright@wintelcom.net>
Re: Final Call: RC1 about to go out the door ...
The Hermit Hacker writes:

> 	We'd like to wrap up an RC1 and get this release happening this
> year sometime :)  Tom mentioned to me that he has no outstandings left on
> his plate ... does anyone else have any *show stoppers* left that need to
> be addressed, or can I package things up?

I just uploaded new man pages.  I'll probably do them once more in a few
days to catch all the changes.

We need a supported platform list.  Let's hear it.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Tue, 20 Mar 2001 20:11:21 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Final Call: RC1 about to go out the door ...
UnixWare 7, Rel 7.1.1, using UDK FS Compiler
FreeBSD 4.[23]

LER

>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<

On 3/20/01, 1:11:21 PM, Peter Eisentraut <peter_e@gmx.net> wrote regarding 
Re: [HACKERS] Final Call: RC1 about to go out the door ...:


> The Hermit Hacker writes:

> >     We'd like to wrap up an RC1 and get this release happening this
> > year sometime :)  Tom mentioned to me that he has no outstandings left on
> > his plate ... does anyone else have any *show stoppers* left that need to
> > be addressed, or can I package things up?

> I just uploaded new man pages.  I'll probably do them once more in a few
> days to catch all the changes.

> We need a supported platform list.  Let's hear it.

> --
> Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

Tue, 20 Mar 2001 19:30:03 GMT
Larry Rosenman <ler@lerctr.org>
Re: Final Call: RC1 about to go out the door ...
BSDI 4.01.

[ Charset ISO-8859-1 unsupported, converting... ]
> UnixWare 7, Rel 7.1.1, using UDK FS Compiler
> FreeBSD 4.[23]
> 
> LER
> 
> >>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<
> 
> On 3/20/01, 1:11:21 PM, Peter Eisentraut <peter_e@gmx.net> wrote regarding 
> Re: [HACKERS] Final Call: RC1 about to go out the door ...:
> 
> 
> > The Hermit Hacker writes:
> 
> > >     We'd like to wrap up an RC1 and get this release happening this
> > > year sometime :)  Tom mentioned to me that he has no outstandings left on
> > > his plate ... does anyone else have any *show stoppers* left that need to
> > > be addressed, or can I package things up?
> 
> > I just uploaded new man pages.  I'll probably do them once more in a few
> > days to catch all the changes.
> 
> > We need a supported platform list.  Let's hear it.
> 
> > --
> > Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/
> 
> 
> > ---------------------------(end of broadcast)---------------------------
> > TIP 2: you can get off all lists at once with the unregister command
> >     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
> http://www.postgresql.org/search.mpl
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 20 Mar 2001 14:36:55 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Final Call: RC1 about to go out the door ...
Peter Eisentraut <peter_e@gmx.net> writes:
> We need a supported platform list.  Let's hear it.

HPUX 10.20	(HP-PA architecture)
Linux/PPC	(LinuxPPC 2000 Q4 distro tested here; 2.2.18 kernel I think)

			regards, tom lane

Tue, 20 Mar 2001 16:10:21 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Final Call: RC1 about to go out the door ... 

Go here to report or to see the list.

http://www.postgresql.org/~vev/regress/

Vince.



On Tue, 20 Mar 2001, Larry Rosenman wrote:

> UnixWare 7, Rel 7.1.1, using UDK FS Compiler
> FreeBSD 4.[23]
>
> LER
>
> >>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<
>
> On 3/20/01, 1:11:21 PM, Peter Eisentraut <peter_e@gmx.net> wrote regarding
> Re: [HACKERS] Final Call: RC1 about to go out the door ...:
>
>
> > The Hermit Hacker writes:
>
> > >     We'd like to wrap up an RC1 and get this release happening this
> > > year sometime :)  Tom mentioned to me that he has no outstandings left on
> > > his plate ... does anyone else have any *show stoppers* left that need to
> > > be addressed, or can I package things up?
>
> > I just uploaded new man pages.  I'll probably do them once more in a few
> > days to catch all the changes.
>
> > We need a supported platform list.  Let's hear it.
>
> > --
> > Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/
>
>
> > ---------------------------(end of broadcast)---------------------------
> > TIP 2: you can get off all lists at once with the unregister command
> >     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://www.postgresql.org/search.mpl
>

-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
         56K Nationwide Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Tue, 20 Mar 2001 19:04:12 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: Final Call: RC1 about to go out the door ...
> HPUX 10.20      (HP-PA architecture)

Time to drop 9.2 from the list?

> Linux/PPC       (LinuxPPC 2000 Q4 distro tested here; 2.2.18 kernel I think)

What processor? Tatsuo had tested on a 603...

                    - Thomas

Wed, 21 Mar 2001 01:00:32 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Final Call: RC1 about to go out the door ...
OK, here is my current platform list taken from the -hackers list and
from Vince's web page. I'm sure I've missed at least a few reports, but
please confirm that platforms are actually running and passing
regression tests with recent betas or the latest release candidate.

If a platform you are running on is not listed, make sure it gets
included! Platforms with reports for 7.0 risk being demoted to the "used
to be supported list", and platforms with reports for only 6.5 are on a
deathwatch, so be sure to speak up! Also, I've included names below to
remind us who helped last time, but feel free to report even if your
name is not already listed.

I've separated out recent reports and put them at the end of the list.
Thanks in advance.

                   - Thomas

AIX 4.3.2  RS6000  7.0 2000-04-05, Andreas Zeugswetter
Compaq Tru64 5.0 Alpha 7.0 2000-04-11, Andrew McMurry
IRIX 6.5.6f MIPS   6.5.3 2000-02-18, Kevin Wheatley
Linux 2.2.x armv4l 7.0 2000-04-17, Mark Knox
Linux 2.0.x MIPS   7.0 2000-04-13, Tatsuo Ishii
mklinux PPC750     7.0 2000-04-13, Tatsuo Ishii
NetBSD 1.4 arm32   7.0 2000-04-08, Patrick Welche
NetBSD 1.4U x86    7.0 2000-03-26, Patrick Welche
NetBSD m68k        7.0 2000-04-10, Henry B. Hotz
NetBSD Sparc       7.0 2000-04-13, Tom I. Helbekkmo
QNX 4.25 x86       7.0 2000-04-01, Dr. Andreas Kardos
SCO OpenServer 5 x86 6.5 1999-05-25, Andrew Merrill
Solaris x86        7.0 2000-04-12, Marc Fournier
Solaris 2.5.1-2.7 Sparc 7.0 2000-04-12, Peter Eisentraut
SunOS 4.1.4 Sparc  7.0 2000-04-13, Tatsuo Ishii
Windows/Win32 x86  7.0 2000-04-02, Magnus Hagander (clients only)
WinNT/Cygwin x86   7.0 2000-03-30, Daniel Horak

BeOS 5.0.3 x86     7.1 2000-12-18, Cyril Velter
BSDI 4.01  x86     7.1 2001-03-19, Bruce Momjian
FreeBSD 4.2 x86    7.1 2001-03-19, Vince Vielhaber
HPUX 10.20 PA-RISC 7.1 2001-03-19, Tom Lane
IBM        S/390   7.1 2000-11-17, Neale Ferguson
Linux 2.2.x Alpha  7.1 2001-01-23, Ryan Kirkpatrick
Linux 2.2.16 x86   7.1 2001-03-19, Thomas Lockhart
Linux 2.2.15 Sparc 7.1 2001-01-30, Ryan Kirkpatrick
LinuxPPC G3        7.1 2001-03-19, Tom Lane
SCO UnixWare 7.1.1 x86 7.1 2001-03-19, Larry Rosenman
MacOS-X Darwin PowerPC 7.1 2000-12-11, Peter Bierman

Wed, 21 Mar 2001 01:05:01 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Call for platforms
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> HPUX 10.20      (HP-PA architecture)

> Time to drop 9.2 from the list?

I don't have it running here anymore.  Is there anyone on the list
who can test on HPUX 9?

>> Linux/PPC       (LinuxPPC 2000 Q4 distro tested here; 2.2.18 kernel I think)

> What processor? Tatsuo had tested on a 603...

It's a Powerbook G3 (FireWire model), but I'm not sure which chip is
inside (and Apple's spec sheet isn't too helpful)...

			regards, tom lane

Tue, 20 Mar 2001 20:13:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Final Call: RC1 about to go out the door ... 
> >> Linux/PPC       (LinuxPPC 2000 Q4 distro tested here; 2.2.18 kernel I think)
> 
> > What processor? Tatsuo had tested on a 603...
> It's a Powerbook G3 (FireWire model), but I'm not sure which chip is
> inside (and Apple's spec sheet isn't too helpful)...

 From what I can tell (which isn't much ;) Apple at least calls the
processor a "G3". Which accounts for why we can't find another
designation.

Not sure where it fits into the lineup I *used* to know about (for
embedded systems) but I don't care. Will refer to it as a "G3" for
now...

                 - Thomas

Wed, 21 Mar 2001 01:20:21 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Final Call: RC1 about to go out the door ...
> > >> Linux/PPC       (LinuxPPC 2000 Q4 distro tested here; 2.2.18 kernel I think)
> > > What processor? Tatsuo had tested on a 603...
> > It's a Powerbook G3 (FireWire model), but I'm not sure which chip is
> > inside (and Apple's spec sheet isn't too helpful)...
> From what I can tell (which isn't much ;) Apple at least calls the
> processor a "G3". Which accounts for why we can't find another
> designation.

Tatsuo, I have a separate listing for "mklinux" for the 7.0 release. Is
that distro still valid and unique? Or is there a better way to
represent the PPC options under Linux?

                    - Thomas

Wed, 21 Mar 2001 01:25:17 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Final Call: RC1 about to go out the door ...
On Wed, 21 Mar 2001, Thomas Lockhart wrote:

> > > >> Linux/PPC       (LinuxPPC 2000 Q4 distro tested here; 2.2.18 kernel I think)
> > > > What processor? Tatsuo had tested on a 603...
> > > It's a Powerbook G3 (FireWire model), but I'm not sure which chip is
> > > inside (and Apple's spec sheet isn't too helpful)...
> > From what I can tell (which isn't much ;) Apple at least calls the
> > processor a "G3". Which accounts for why we can't find another
> > designation.
> 
> Tatsuo, I have a separate listing for "mklinux" for the 7.0 release. Is
> that distro still valid and unique? Or is there a better way to
> represent the PPC options under Linux?

mklinux is older Motorola 68k-based systems
LinuxPPC is the newer powerPC-based systems


-- 
Dominic J. Eidson
                                        "Baruk Khazad! Khazad ai-menu!" - Gimli
-------------------------------------------------------------------------------
http://www.the-infinite.org/              http://www.the-infinite.org/~dominic/


Tue, 20 Mar 2001 19:42:22 -0600 (CST)
"Dominic J. Eidson" <sauron@the-infinite.org>
Re: Re: Final Call: RC1 about to go out the door ...
> mklinux is older Motorola 68k-based systems
> LinuxPPC is the newer powerPC-based systems

Hmm. I have mklinux listed as being on the 750. My vague recollection is
that the distinction is between NuBus and PCI machines (not necessarily
in that order), but...

I also vaguely recalled that the distros had merged (or something :/

                    - Thomas

Wed, 21 Mar 2001 01:51:26 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Final Call: RC1 about to go out the door ...
* Thomas Lockhart <lockhart@alumni.caltech.edu> [010320 20:04]:
> OK, here is my current platform list taken from the -hackers list and
> from Vince's web page. I'm sure I've missed at least a few reports, but
> please confirm that platforms are actually running and passing
> regression tests with recent betas or the latest release candidate.
> 
> If a platform you are running on is not listed, make sure it gets
> included! Platforms with reports for 7.0 risk being demoted to the "used
> to be supported list", and platforms with reports for only 6.5 are on a
> deathwatch, so be sure to speak up! Also, I've included names below to
> remind us who helped last time, but feel free to report even if your
> name is not already listed.
FreeBSD 4.3-BETA (will be -RELEASE by the time we release) works too.

I reported FreeBSD 4.[23]. 

LER

> 
> I've separated out recent reports and put them at the end of the list.
> Thanks in advance.
> 
>                    - Thomas
> 
> AIX 4.3.2  RS6000  7.0 2000-04-05, Andreas Zeugswetter
> Compaq Tru64 5.0 Alpha 7.0 2000-04-11, Andrew McMurry
> IRIX 6.5.6f MIPS   6.5.3 2000-02-18, Kevin Wheatley
> Linux 2.2.x armv4l 7.0 2000-04-17, Mark Knox
> Linux 2.0.x MIPS   7.0 2000-04-13, Tatsuo Ishii
> mklinux PPC750     7.0 2000-04-13, Tatsuo Ishii
> NetBSD 1.4 arm32   7.0 2000-04-08, Patrick Welche
> NetBSD 1.4U x86    7.0 2000-03-26, Patrick Welche
> NetBSD m68k        7.0 2000-04-10, Henry B. Hotz
> NetBSD Sparc       7.0 2000-04-13, Tom I. Helbekkmo
> QNX 4.25 x86       7.0 2000-04-01, Dr. Andreas Kardos
> SCO OpenServer 5 x86 6.5 1999-05-25, Andrew Merrill
> Solaris x86        7.0 2000-04-12, Marc Fournier
> Solaris 2.5.1-2.7 Sparc 7.0 2000-04-12, Peter Eisentraut
> SunOS 4.1.4 Sparc  7.0 2000-04-13, Tatsuo Ishii
> Windows/Win32 x86  7.0 2000-04-02, Magnus Hagander (clients only)
> WinNT/Cygwin x86   7.0 2000-03-30, Daniel Horak
> 
> BeOS 5.0.3 x86     7.1 2000-12-18, Cyril Velter
> BSDI 4.01  x86     7.1 2001-03-19, Bruce Momjian
> FreeBSD 4.2 x86    7.1 2001-03-19, Vince Vielhaber
> HPUX 10.20 PA-RISC 7.1 2001-03-19, Tom Lane
> IBM        S/390   7.1 2000-11-17, Neale Ferguson
> Linux 2.2.x Alpha  7.1 2001-01-23, Ryan Kirkpatrick
> Linux 2.2.16 x86   7.1 2001-03-19, Thomas Lockhart
> Linux 2.2.15 Sparc 7.1 2001-01-30, Ryan Kirkpatrick
> LinuxPPC G3        7.1 2001-03-19, Tom Lane
> SCO UnixWare 7.1.1 x86 7.1 2001-03-19, Larry Rosenman
> MacOS-X Darwin PowerPC 7.1 2000-12-11, Peter Bierman
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html
-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Tue, 20 Mar 2001 20:09:11 -0600
Larry Rosenman <ler@lerctr.org>
Re: Call for platforms
> SCO OpenServer 5 x86...

OK, I see that Billy Allie recently updated FAQ_SCO to indicate
demonstrated (?) support for OpenServer. I will reflect that in the
platform support info.

                     - Thomas

Wed, 21 Mar 2001 02:39:49 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Call for platforms
> Tatsuo, I have a separate listing for "mklinux" for the 7.0 release. Is
> that distro still valid and unique? Or is there a better way to
> represent the PPC options under Linux?

I think MkLinux is completely different from Linux/PPC. Will test RC1
on my MkLiux box soon...
--
Tatsuo Ishii

Wed, 21 Mar 2001 12:20:50 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Final Call: RC1 about to go out the door ...
> > Tatsuo, I have a separate listing for "mklinux" for the 7.0 release. Is
> > that distro still valid and unique? Or is there a better way to
> > represent the PPC options under Linux?
> 
> mklinux is older Motorola 68k-based systems

No. MkLinux runs on Power PC based system also. I believe there is a
x86 based MkLinux exists somewhere.
--
Tatsuo Ishii

Wed, 21 Mar 2001 12:24:48 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Re: Final Call: RC1 about to go out the door ...
On Wed, 21 Mar 2001, Thomas Lockhart wrote:

> > > >> Linux/PPC       (LinuxPPC 2000 Q4 distro tested here; 2.2.18 kernel I think)
> > > > What processor? Tatsuo had tested on a 603...
> > > It's a Powerbook G3 (FireWire model), but I'm not sure which chip is
> > > inside (and Apple's spec sheet isn't too helpful)...
> > From what I can tell (which isn't much ;) Apple at least calls the
> > processor a "G3". Which accounts for why we can't find another
> > designation.

  The G3s  are the MPC7XX family and the G4s are the MPC7XXX family.
  All of the ones Mac is currently releasing are MPC7450s. See

  http://e-www.motorola.com/webapp/sps/prod_cat/taxonomy.jsp?catId=M934309493763

  for more details.

FWIW.

-- 
Dominic J. Eidson
                                        "Baruk Khazad! Khazad ai-menu!" - Gimli
-------------------------------------------------------------------------------
http://www.the-infinite.org/              http://www.the-infinite.org/~dominic/



Tue, 20 Mar 2001 21:59:36 -0600 (CST)
"Dominic J. Eidson" <sauron@the-infinite.org>
Re: Re: Final Call: RC1 about to go out the door ...
> mklinux PPC750     7.0 2000-04-13, Tatsuo Ishii

I got core dump while running the parallel regression test of beta6.
Will look at...
--
Tatsuo Ishii

Wed, 21 Mar 2001 15:36:07 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Call for platforms
> Compaq Tru64 5.0 Alpha 7.0 2000-04-11, Andrew McMurry

We've got 7.0.3 and 7.1b4 running on 

Compaq Tru64 4.0G Alpha

Will do the regression test once RC1 is out.

Adriaan

Wed, 21 Mar 2001 08:46:48 +0200
Adriaan Joubert <a.joubert@albourne.com>
Re: Call for platforms
> > mklinux PPC750     7.0 2000-04-13, Tatsuo Ishii
> 
> I got core dump while running the parallel regression test of beta6.
> Will look at...
> --
> Tatsuo Ishii

  VACUUM;
! FATAL 2:  ZeroFill(logfile 0 seg 1) failed: No such file or directory
! pqReadData() -- backend closed the channel unexpectedly.

maybe a bug related to Tom recently fixed?
If so, I will try RC1...
--
Tatsuo Ishii

Wed, 21 Mar 2001 16:30:33 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Call for platforms

> Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> >> HPUX 10.20      (HP-PA architecture)
> 
> > Time to drop 9.2 from the list?
> 
> I don't have it running here anymore.  Is there anyone on the list
> who can test on HPUX 9?

HP haven't supported 9.X since the end of 1999 on servers, and since
earlier than that on workstations. I doubt anyone will expect to see
it listed on the PostgreSQL list of supported platforms for 7.1.

Regards,

Giles


Wed, 21 Mar 2001 19:27:53 +1100
Giles Lean <giles@nemeton.com.au>
Re: Re: Final Call: RC1 about to go out the door ... 
Quoting The Hermit Hacker <scrappy@hub.org>:

> 
> Okay folks ...
> 
> 	We'd like to wrap up an RC1 and get this release happening this
> year sometime :)  Tom mentioned to me that he has no outstandings left
> on his plate ... does anyone else have any *show stoppers* left that need
> to be addressed, or can I package things up?

Nothing that would stop RC1 (I've still got some testing which I'm doing later 
tonight).

> 	Speak now, or forever hold your piece (where forever is the time
> between now and RC1 is packaged) ...

I'm surprised it hasn't been out already - being worked to death the last two 
weeks, I'm still catching up with developments :( ...

Peter

-- 
Peter Mount peter@retep.org.uk
PostgreSQL JDBC Driver: http://www.retep.org.uk/postgres/
RetepPDF PDF library for Java: http://www.retep.org.uk/pdf/

Wed, 21 Mar 2001 04:16:26 -0500 (EST)
Peter T Mount <peter@retep.org.uk>
Re: Final Call: RC1 about to go out the door ...
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> ! FATAL 2:  ZeroFill(logfile 0 seg 1) failed: No such file or directory
> ! pqReadData() -- backend closed the channel unexpectedly.

Is it possible you ran out of disk space?

			regards, tom lane

Wed, 21 Mar 2001 06:13:46 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Call for platforms 
> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> > ! FATAL 2:  ZeroFill(logfile 0 seg 1) failed: No such file or directory
> > ! pqReadData() -- backend closed the channel unexpectedly.
> 
> Is it possible you ran out of disk space?

Probably not.
--
Tatsuo Ishii

Wed, 21 Mar 2001 23:04:12 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Call for platforms 
Thomas Lockhart writes:

> > HPUX 10.20      (HP-PA architecture)
>
> Time to drop 9.2 from the list?
>
> > Linux/PPC       (LinuxPPC 2000 Q4 distro tested here; 2.2.18 kernel I think)
>
> What processor? Tatsuo had tested on a 603...

Given that we list "x86", I think we wouldn't care.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Wed, 21 Mar 2001 19:19:02 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Final Call: RC1 about to go out the door ...
Hi,

I reported Linux RedHat 6.2 - 2.2.14-5.0smp #1 SMP Tue Mar 7 21:01:40 EST
2000 i686
2 cpu - 1Go RAM

Gilles DAROLD


Wed, 21 Mar 2001 19:20:06 +0100
Gilles DAROLD <gilles@darold.net>
Re: Call for platforms
Tatsuo Ishii writes:

> > > Tatsuo, I have a separate listing for "mklinux" for the 7.0 release. Is
> > > that distro still valid and unique? Or is there a better way to
> > > represent the PPC options under Linux?
> >
> > mklinux is older Motorola 68k-based systems
>
> No. MkLinux runs on Power PC based system also. I believe there is a
> x86 based MkLinux exists somewhere.

mkLinux is "micro-kernel" Linux, on top of Mach.  Consequentially, it is
non-trivially different from any other Linux.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Wed, 21 Mar 2001 19:21:15 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Final Call: RC1 about to go out the door ...
Thomas Lockhart writes:

> > SCO OpenServer 5 x86...
>
> OK, I see that Billy Allie recently updated FAQ_SCO to indicate
> demonstrated (?) support for OpenServer. I will reflect that in the
> platform support info.

The last FAQ_SCO update was by me, and it was rather the consequence of
some implementational developments and not a good indicator of any
actually working platform.  (I do have access to a Unixware box, but that
was already reported.)

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Wed, 21 Mar 2001 19:24:17 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Call for platforms

Hi,

I am currently testing beta6 on AIX 4.3.3 on a RS6000 H80 with 4 cpu and 4
Go RAM
I use :

./configure
        --with-CC=/usr/local/bin/gcc
        --with-includes=/usr/local/include
        --with-libraries=/usr/local/lib

All seem to be ok, There just the geometry failure in regression test
(following the AIX FAQ
it's normal ?)

But when I configure with --with-perl I have the following error :

make[4]: cc : Command not found

Any idea ?


Gilles DAROLD

> Hi,
>
> I reported Linux RedHat 6.2 - 2.2.14-5.0smp #1 SMP Tue Mar 7 21:01:40 EST
> 2000 i686
> 2 cpu - 1Go RAM
>
> Gilles DAROLD
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://www.postgresql.org/search.mpl


Wed, 21 Mar 2001 20:45:03 +0100
Gilles DAROLD <gilles@darold.net>
Re: Call for platforms
Gilles DAROLD wrote:

> Hi,
>
> I am currently testing beta6 on AIX 4.3.3 on a RS6000 H80 with 4 cpu and 4
> Go RAM
> I use :
>
> ./configure
>         --with-CC=/usr/local/bin/gcc
>         --with-includes=/usr/local/include
>         --with-libraries=/usr/local/lib
>
> All seem to be ok, There just the geometry failure in regression test
>
> But when I configure with --with-perl I have the following error :

Ok symbolic link between cc and gcc seem to be the better fix.

I have now tested the --with-CXX option to compile libpq++ and it
really don't work, here are the output :

ld: 0711-319 WARNING: Exported symbol not defined:
PgConnection::CloseConnection
ld: 0711-319 WARNING: Exported symbol not defined: PgConnection::Connect
ld: 0711-319 WARNING: Exported symbol not defined: PgConnection::ConnectionBad

ld: 0711-319 WARNING: Exported symbol not defined: PgConnection::DBName
ld: 0711-319 WARNING: Exported symbol not defined: PgConnection::ErrorMessage
ld: 0711-319 WARNING: Exported symbol not defined: PgConnection::Exec
ld: 0711-319 WARNING: Exported symbol not defined: PgConnection::ExecCommandOk

ld: 0711-319 WARNING: Exported symbol not defined: PgConnection::ExecTuplesOk
ld: 0711-319 WARNING: Exported symbol not defined: PgConnection::IntToString

....

ld: 0711-317 ERROR: Undefined symbol: basic_string<char,
string_char_traits<char>, __default_alloc_template<false, 0> >::nilRep
ld: 0711-317 ERROR: Undefined symbol:
__malloc_alloc_template<0>::__malloc_alloc_oom_handler
ld: 0711-317 ERROR: Undefined symbol: endl(ostream &)
ld: 0711-317 ERROR: Undefined symbol: cerr
ld: 0711-317 ERROR: Undefined symbol: .ostream::operator<<(char const *)
ld: 0711-317 ERROR: Undefined symbol: __default_alloc_template<false,
0>::_S_end_free
ld: 0711-317 ERROR: Undefined symbol: __default_alloc_template<false,
0>::_S_start_free
ld: 0711-317 ERROR: Undefined symbol: __default_alloc_template<false,
0>::_S_heap_size
ld: 0711-317 ERROR: Undefined symbol: __default_alloc_template<false,
0>::_S_free_list
ld: 0711-317 ERROR: Undefined symbol: .__out_of_range(char const *)
ld: 0711-317 ERROR: Undefined symbol: .__length_error(char const *)
collect2: ld returned 8 exit status
make[3]: *** [libpq++.so] Error 1
make[3]: Leaving directory
`/home/darold/postgresql-7.1beta6/src/interfaces/libpq++'
make[2]: *** [all] Error 2

I have change the Makefile.global and replace c++ by g++ but it the same
output.

Could you tell me what going wrong ? Is my GNU install not fully functionnal ?

I use :
gcc version 2.95.2.1 19991024 (release) libs for powerpc-ibm-aix4.3.2.0

Regards

Gilles DAROLD


Wed, 21 Mar 2001 21:53:46 +0100
Gilles DAROLD <gilles@darold.net>
Re: [HACKERS] Call for platforms AIX 4.3.3 Failed
Gilles DAROLD writes:

> I have now tested the --with-CXX option to compile libpq++ and it
> really don't work, here are the output :

> ld: 0711-317 ERROR: Undefined symbol: __default_alloc_template<false,
> 0>::_S_start_free
> ld: 0711-317 ERROR: Undefined symbol: __default_alloc_template<false,
> 0>::_S_heap_size
> ld: 0711-317 ERROR: Undefined symbol: __default_alloc_template<false,
> 0>::_S_free_list
> ld: 0711-317 ERROR: Undefined symbol: .__out_of_range(char const *)
> ld: 0711-317 ERROR: Undefined symbol: .__length_error(char const *)
> collect2: ld returned 8 exit status
> make[3]: *** [libpq++.so] Error 1
> make[3]: Leaving directory
> `/home/darold/postgresql-7.1beta6/src/interfaces/libpq++'
> make[2]: *** [all] Error 2

This could be a name mangling problem.  Maybe the linker needs to be
invoked specially when building C++ libraries.  Maybe the C++ compiler
driver needs to be invoked directly.  This could especially be a problem
if you're using the GNU compiler with system libraries, since those are
usually compiled by the system compiler.

> I have change the Makefile.global and replace c++ by g++ but it the same
> output.

I think the good C++ compiler on AIX is called xlC.  In any case, make
sure that you don't mix different C++ compilers.  You need to do 'gmake
clean' at least in the libpq++ directory if you're switching.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Wed, 21 Mar 2001 22:25:55 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: [HACKERS] Call for platforms AIX 4.3.3 Failed
Peter Eisentraut wrote:

> This could be a name mangling problem.  Maybe the linker needs to be
> invoked specially when building C++ libraries.  Maybe the C++ compiler
> driver needs to be invoked directly.  This could especially be a problem
> if you're using the GNU compiler with system libraries, since those are
> usually compiled by the system compiler.
>
> > I have change the Makefile.global and replace c++ by g++ but it the same
> > output.
>
> I think the good C++ compiler on AIX is called xlC.  In any case, make
> sure that you don't mix different C++ compilers.  You need to do 'gmake
> clean' at least in the libpq++ directory if you're switching.

AIX faq said that xlC compiler don't work with libpq++ but with g++ it
may works :

> libpq++ does not work because xlC does not have the string and bool
classes.
> compiling the few files, that fail, with g++ does work.

Humm, I have no xlC compiler installed.Here are the compilation lines :

make[3]: Entering directory
`/home/darold/postgresql-7.1beta6/src/interfaces/libpq++'
g++ -O2 -Wall  -I../../../src/interfaces/libpq -I../../../src/include
-I/usr/local/include  -c -o pgconnection.o p
gconnection.cc
g++ -O2 -Wall  -I../../../src/interfaces/libpq -I../../../src/include
-I/usr/local/include  -c -o pgdatabase.o pgd
atabase.cc
g++ -O2 -Wall  -I../../../src/interfaces/libpq -I../../../src/include
-I/usr/local/include  -c -o pgtransdb.o pgtr
ansdb.cc
g++ -O2 -Wall  -I../../../src/interfaces/libpq -I../../../src/include
-I/usr/local/include  -c -o pgcursordb.o pgc
ursordb.cc
g++ -O2 -Wall  -I../../../src/interfaces/libpq -I../../../src/include
-I/usr/local/include  -c -o pglobject.o pglo
bject.cc
ar crs libpq++.a pgconnection.o pgdatabase.o pgtransdb.o pgcursordb.o
pglobject.o
touch libpq++.a
../../../src/backend/port/aix/mkldexport.sh libpq++.a > libpq++.exp
/usr/local/bin/gcc -Wl,-H512 -Wl,-bM:SRE
-Wl,-bI:../../../src/backend/postgres.imp -Wl,-bE:libpq++.exp -o libpq++.
so libpq++.a -L/usr/local/lib  -L../../../src/interfaces/libpq -lpq -lc
ld: 0711-224 WARNING: Duplicate symbol: __start

All work until it want to link with libraries, it seems to want the libc
(-lc) and I don't
have it installed. Is it normal or this realease is not portable with AIX
4.3.3 ?


Wed, 21 Mar 2001 22:37:39 +0100
Gilles DAROLD <gilles@darold.net>
Re:Call for platforms AIX 4.3.3 Failed

$ uname -a
OpenBSD mizer 2.8 a#0 i386

P3, default 2.8 install.  Problems w/ TCL,  but I think it's a local
problem.

System needs kernel changes as noted at www.crimelabs.net.  (shared mem
stuff).

OBSD-sparc comming soon.

- b


b. palmer,  bpalmer@crimelabs.net
pgp:  www.crimelabs.net/bpalmer.pgp5


Wed, 21 Mar 2001 17:21:30 -0500 (EST)
bpalmer <bpalmer@crimelabs.net>
Re: Re: Final Call: RC1 about to go out the door ...
I see nobody did a test of 7.1 on Linux 2.4.x ?

Would be nice to certify it is running on kernel 2.4.x as they claim this
is entreprise strength kernel...

Cheers.

Thomas Lockhart wrote:

>
>
> AIX 4.3.2  RS6000  7.0 2000-04-05, Andreas Zeugswetter
> Compaq Tru64 5.0 Alpha 7.0 2000-04-11, Andrew McMurry
> IRIX 6.5.6f MIPS   6.5.3 2000-02-18, Kevin Wheatley
> Linux 2.2.x armv4l 7.0 2000-04-17, Mark Knox
> Linux 2.0.x MIPS   7.0 2000-04-13, Tatsuo Ishii
> mklinux PPC750     7.0 2000-04-13, Tatsuo Ishii
> NetBSD 1.4 arm32   7.0 2000-04-08, Patrick Welche
> NetBSD 1.4U x86    7.0 2000-03-26, Patrick Welche
> NetBSD m68k        7.0 2000-04-10, Henry B. Hotz
> NetBSD Sparc       7.0 2000-04-13, Tom I. Helbekkmo
> QNX 4.25 x86       7.0 2000-04-01, Dr. Andreas Kardos
> SCO OpenServer 5 x86 6.5 1999-05-25, Andrew Merrill
> Solaris x86        7.0 2000-04-12, Marc Fournier
> Solaris 2.5.1-2.7 Sparc 7.0 2000-04-12, Peter Eisentraut
> SunOS 4.1.4 Sparc  7.0 2000-04-13, Tatsuo Ishii
> Windows/Win32 x86  7.0 2000-04-02, Magnus Hagander (clients only)
> WinNT/Cygwin x86   7.0 2000-03-30, Daniel Horak
>
> BeOS 5.0.3 x86     7.1 2000-12-18, Cyril Velter
> BSDI 4.01  x86     7.1 2001-03-19, Bruce Momjian
> FreeBSD 4.2 x86    7.1 2001-03-19, Vince Vielhaber
> HPUX 10.20 PA-RISC 7.1 2001-03-19, Tom Lane
> IBM        S/390   7.1 2000-11-17, Neale Ferguson
> Linux 2.2.x Alpha  7.1 2001-01-23, Ryan Kirkpatrick
> Linux 2.2.16 x86   7.1 2001-03-19, Thomas Lockhart
> Linux 2.2.15 Sparc 7.1 2001-01-30, Ryan Kirkpatrick
> LinuxPPC G3        7.1 2001-03-19, Tom Lane
> SCO UnixWare 7.1.1 x86 7.1 2001-03-19, Larry Rosenman
> MacOS-X Darwin PowerPC 7.1 2000-12-11, Peter Bierman
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html


Thu, 22 Mar 2001 12:31:03 +1200
Franck Martin <franck@sopac.org>
Re: Call for platforms (linux 2.4.x ?)
Franck Martin <franck@sopac.org> writes:

> Would be nice to certify it is running on kernel 2.4.x as they claim this
> is entreprise strength kernel...

Lamar, if you send me your SRPM I can do that...

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

21 Mar 2001 19:31:13 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: Call for platforms (linux 2.4.x ?)
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
>> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> ! FATAL 2:  ZeroFill(logfile 0 seg 1) failed: No such file or directory
> ! pqReadData() -- backend closed the channel unexpectedly.
>> 
>> Is it possible you ran out of disk space?

> Probably not.

The reason I was speculating that was that it seems pretty unlikely
that a write() call could return ENOENT, as the above appears to
suggest.  I think that the errno = ENOENT value was not set by write(),
but is leftover from the expected failure of BasicOpenFile earlier in
XLogFileInit.  Probably write() returned some value less than BLCKSZ
but more than zero, and so did not set errno.

Offhand the only reason I can think of for a write to a disk file
to terminate after a partial transfer is a full disk.  What do you
think?

			regards, tom lane

Wed, 21 Mar 2001 22:29:05 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Call for platforms 
* Tom Lane <tgl@sss.pgh.pa.us> [010321 21:29]:
> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> >> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> > ! FATAL 2:  ZeroFill(logfile 0 seg 1) failed: No such file or directory
> > ! pqReadData() -- backend closed the channel unexpectedly.
> >> 
> >> Is it possible you ran out of disk space?
> 
> > Probably not.
> 
> The reason I was speculating that was that it seems pretty unlikely
> that a write() call could return ENOENT, as the above appears to
> suggest.  I think that the errno = ENOENT value was not set by write(),
> but is leftover from the expected failure of BasicOpenFile earlier in
> XLogFileInit.  Probably write() returned some value less than BLCKSZ
> but more than zero, and so did not set errno.
> 
> Offhand the only reason I can think of for a write to a disk file
> to terminate after a partial transfer is a full disk.  What do you
> think?
What about hitting a quota?

LER

> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Wed, 21 Mar 2001 21:31:09 -0600
Larry Rosenman <ler@lerctr.org>
Re: Call for platforms
On Thu, Mar 22, 2001 at 12:31:03PM +1200, Franck Martin wrote:
> I see nobody did a test of 7.1 on Linux 2.4.x ?
> 
> Would be nice to certify it is running on kernel 2.4.x as they claim this
> is entreprise strength kernel...

	I've been running the 7.1 betas on 2.4 for weeks without any problems.
I replied to the "call for platforms" e-mail, but it looks like it got
lost in the avalanche.
	I'll run the regression tests with the latest CVS snapshot and submit
a report to the list.

	-Roberto
-- 
+----| http://fslc.usu.edu USU Free Software & GNU/Linux Club|------+
  Roberto Mello - Computer Science, USU - http://www.brasileiro.net 
      http://www.sdl.usu.edu - Space Dynamics Lab, Web Developer    

Wed, 21 Mar 2001 22:48:36 -0700
Roberto Mello <rmello@cc.usu.edu>
Re: Call for platforms (linux 2.4.x ?)

Results of 'make check':

NetBSD-1.5/i386	    one spurious floating point test failure
		    (mail sent to postgresql-bugs with details)

NetBSD_1.5/alpha    all tests passed

NetBSD-1.4.2/i386   four tests fail
		         timestamp            ... FAILED
			 abstime              ... FAILED
			 tinterval            ... FAILED
		    test horology             ... FAILED

I'll look into the 1.4.2 failures when/if I get time.  If anyone wants
the test output to examine please ask.

Regards,

Giles


Thu, 22 Mar 2001 20:08:37 +1100
Giles Lean <giles@nemeton.com.au>
Call for platforms
OK:  Linux 2.4.2 i686 / gcc 2.95.2 / Debian testing/unstable

no problems.

OK?: NetBSD 1.5 i586 / egcs 2.91.66 / (netbsd-1-5 from Jan)

netbsd FAILED the geometry test, diff attached, dunno if its
critical or not.

-- 
marko
Thu, 22 Mar 2001 12:02:09 +0200
Marko Kreen <marko@l-t.ee>
Re: Call for platforms
> NetBSD-1.5/i386     one spurious floating point test failure
>                     (mail sent to postgresql-bugs with details)
> NetBSD_1.5/alpha    all tests passed
> NetBSD-1.4.2/i386   four tests fail

Thanks! I'm not too worried about 1.4.2, but be sure to let us know what
the problem was; it may help out someone else...

                      - Thomas

Thu, 22 Mar 2001 13:44:07 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Call for platforms
Here is the current scorecard. We have a couple of new platforms
reported (yeaaa!):

NetBSD 2.8 alpha   7.1 2001-03-22, Giles Lean
OpenBSD 2.8 x86    7.1 2001-03-22, B. Palmer (first name?)

Any more OpenBSD architectures out there running PostgreSQL? Here are
the remaining (unreported, or unnoted) platforms:

IRIX 6.5.6f MIPS   6.5.3 2000-02-18, Kevin Wheatley

Anyone running IRIX? It may be on the unsupported list for 7.1 :(

Linux 2.2.x armv4l 7.0 2000-04-17, Mark Knox
Linux 2.0.x MIPS   7.0 2000-04-13, Tatsuo Ishii
mklinux PPC750     7.0 2000-04-13, Tatsuo Ishii

Tatsuo, do you still have access to the MIPS box?

NetBSD m68k        7.0 2000-04-10, Henry B. Hotz
NetBSD Sparc       7.0 2000-04-13, Tom I. Helbekkmo
QNX 4.25 x86       7.0 2000-04-01, Dr. Andreas Kardos

cvs shows that there were patches from Maurizio in February, and he said
that ecpg worked for him. Bruce applied the patches, but I'm not certain
that this was done on the 7.1 code tree? Bruce, do you recall anything?

Solaris x86        7.0 2000-04-12, Marc Fournier

scrappy, do you still have this machine?

Solaris 2.5.1-2.7 Sparc 7.0 2000-04-12, Peter Eisentraut

I'll bet that someone already has Solaris covered. Report?

SunOS 4.1.4 Sparc  7.0 2000-04-13, Tatsuo Ishii

Tatsuo, I vaguely recall that you reported trouble recently. Is this
worth continuing as a supported platform?

Windows/Win32 x86  7.0 2000-04-02, Magnus Hagander (clients only)

Anyone compiled for Win32 recently?

And here are the up-to-date platforms; thanks for the reports:

AIX 4.3.3 RS6000   7.1 2001-03-21, Gilles Darold
BeOS 5.0.3 x86     7.1 2000-12-18, Cyril Velter
BSDI 4.01  x86     7.1 2001-03-19, Bruce Momjian
Compaq Tru64 5.0 Alpha 7.0 2000-04-11, Andrew McMurry
FreeBSD 4.2 x86    7.1 2001-03-19, Vince Vielhaber
HPUX 10.20 PA-RISC 7.1 2001-03-19, Tom Lane
IBM        S/390   7.1 2000-11-17, Neale Ferguson
Linux 2.2.x Alpha  7.1 2001-01-23, Ryan Kirkpatrick
Linux 2.2.16 x86   7.1 2001-03-19, Thomas Lockhart
Linux 2.2.15 Sparc 7.1 2001-01-30, Ryan Kirkpatrick
LinuxPPC G3        7.1 2001-03-19, Tom Lane
NetBSD 1.5E arm32  7.1 2001-03-21, Patrick Welche
NetBSD 1.5S x86    7.1 2001-03-21, Patrick Welche
SCO OpenServer 5 x86   7.1 2001-03-13, Billy Allie
SCO UnixWare 7.1.1 x86 7.1 2001-03-19, Larry Rosenman
MacOS-X Darwin PowerPC 7.1 2000-12-11, Peter Bierman
WinNT/Cygwin x86   7.1 2001-03-16, Jason Tishler

Thu, 22 Mar 2001 14:29:59 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Call for platforms
On Thu, 22 Mar 2001, Thomas Lockhart wrote:

> Solaris x86        7.0 2000-04-12, Marc Fournier
>
> scrappy, do you still have this machine?

Doing tests on Solaris x86/7 right now, will report as soon as they are
done ...

> Solaris 2.5.1-2.7 Sparc 7.0 2000-04-12, Peter Eisentraut
>
> I'll bet that someone already has Solaris covered. Report?

Will do up Sparc/7 also this morning ...

> AIX 4.3.3 RS6000   7.1 2001-03-21, Gilles Darold
> BeOS 5.0.3 x86     7.1 2000-12-18, Cyril Velter
> BSDI 4.01  x86     7.1 2001-03-19, Bruce Momjian
> Compaq Tru64 5.0 Alpha 7.0 2000-04-11, Andrew McMurry
> FreeBSD 4.2 x86    7.1 2001-03-19, Vince Vielhaber

FreeBSD 4.3-BETA is good to go also ...



Thu, 22 Mar 2001 10:39:33 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Call for platforms
> > FreeBSD 4.2 x86    7.1 2001-03-19, Vince Vielhaber
> FreeBSD 4.3-BETA is good to go also ...

Yeah, I'm not sure how to list that, or whether to bother. It is beta,
4.2 works fine (and nothing had to change for 4.3, right?) so maybe we
just list it when 4.3 goes stable? Or is 4.3 sufficiently different that
it would be good to list in the comments for the platform?

                          - Thomas

Thu, 22 Mar 2001 14:50:26 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Call for platforms

How much 'diviation' are we allowing for?

Solaris x86/7 results, for example, in geometry.out, show a difference of:

1.53102359078377e-11,3 (expected)
1.53102359017709e-11,3 (results)

or

3,-3.06204718156754e-11 (expected)
3,-3.06204718035418e-11 (results)

acceptable diviation?

On Thu, 22 Mar 2001, The Hermit Hacker wrote:

> On Thu, 22 Mar 2001, Thomas Lockhart wrote:
>
> > Solaris x86        7.0 2000-04-12, Marc Fournier
> >
> > scrappy, do you still have this machine?
>
> Doing tests on Solaris x86/7 right now, will report as soon as they are
> done ...
>
> > Solaris 2.5.1-2.7 Sparc 7.0 2000-04-12, Peter Eisentraut
> >
> > I'll bet that someone already has Solaris covered. Report?
>
> Will do up Sparc/7 also this morning ...
>
> > AIX 4.3.3 RS6000   7.1 2001-03-21, Gilles Darold
> > BeOS 5.0.3 x86     7.1 2000-12-18, Cyril Velter
> > BSDI 4.01  x86     7.1 2001-03-19, Bruce Momjian
> > Compaq Tru64 5.0 Alpha 7.0 2000-04-11, Andrew McMurry
> > FreeBSD 4.2 x86    7.1 2001-03-19, Vince Vielhaber
>
> FreeBSD 4.3-BETA is good to go also ...
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
>

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org


Thu, 22 Mar 2001 11:06:31 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re: Call for platforms
> Solaris x86/7 results, for example, in geometry.out, show a difference of:
> 3,-3.06204718156754e-11 (expected)
> 3,-3.06204718035418e-11 (results)
> acceptable diviation?

That sort of deviation is well within bounds, particularly for geometry
tests which might have some transcendental math involved.

Is Solaris-x86 ready to go then?

                     - Thomas

Thu, 22 Mar 2001 15:15:40 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Call for platforms

4.3 is in RELEASE CANDIDATE right now.  By the time we release, it should 
be -RELEASE or -STABLE. 

I'd include it as just 4.3. 

It will be the -RELEASE at the time we are.

LER

>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<

On 3/22/01, 8:50:26 AM, Thomas Lockhart <lockhart@alumni.caltech.edu> wrote 
regarding [HACKERS] Re: Call for platforms:


> > > FreeBSD 4.2 x86    7.1 2001-03-19, Vince Vielhaber
> > FreeBSD 4.3-BETA is good to go also ...

> Yeah, I'm not sure how to list that, or whether to bother. It is beta,
> 4.2 works fine (and nothing had to change for 4.3, right?) so maybe we
> just list it when 4.3 goes stable? Or is 4.3 sufficiently different that
> it would be good to list in the comments for the platform?

>                           - Thomas

> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster

Thu, 22 Mar 2001 15:23:04 GMT
Larry Rosenman <ler@lerctr.org>
Re: Re: Call for platforms
On Thu, 22 Mar 2001, Thomas Lockhart wrote:

> > Solaris x86/7 results, for example, in geometry.out, show a difference of:
> > 3,-3.06204718156754e-11 (expected)
> > 3,-3.06204718035418e-11 (results)
> > acceptable diviation?
>
> That sort of deviation is well within bounds, particularly for geometry
> tests which might have some transcendental math involved.
>
> Is Solaris-x86 ready to go then?

Nope, still working through some things ... the select_implicit test
failed completely:

dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> more results/select_implicit.out
psql: connectDBStart() -- connect() failed: Connection refused
        Is the postmaster running locally
        and accepting connections on Unix socket '/tmp/.s.PGSQL.65432'?

I'm going to re-run the test(s) and see if its an isolated thing or not ...


Thu, 22 Mar 2001 11:31:07 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re: Call for platforms
The Hermit Hacker <scrappy@hub.org> writes:
> Nope, still working through some things ... the select_implicit test
> failed completely:

> dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> more results/select_implicit.out
> psql: connectDBStart() -- connect() failed: Connection refused
>         Is the postmaster running locally
>         and accepting connections on Unix socket '/tmp/.s.PGSQL.65432'?

> I'm going to re-run the test(s) and see if its an isolated thing or not ...

Transient overflow of the postmaster socket's accept queue, maybe?  How
big is SOMAXCONN on your box?

			regards, tom lane

Thu, 22 Mar 2001 10:52:10 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
On Thu, 22 Mar 2001, Thomas Lockhart wrote:

> OpenBSD 2.8 x86    7.1 2001-03-22, B. Palmer (first name?)

Though it does work,  like FBSD,  there are some changes that need to be
made to the system.  Need max proc / files changes and a kernel recompile
with SEMMNI and SEMMNS changes.  Anywhere special to note this?


b. palmer,  bpalmer@crimelabs.net
pgp:  www.crimelabs.net/bpalmer.pgp5


Thu, 22 Mar 2001 10:53:16 -0500 (EST)
bpalmer <bpalmer@crimelabs.net>
Re: Re: Call for platforms
> > OpenBSD 2.8 x86    7.1 2001-03-22, B. Palmer (first name?)
> Though it does work,  like FBSD,  there are some changes that need to be
> made to the system.  Need max proc / files changes and a kernel recompile
> with SEMMNI and SEMMNS changes.  Anywhere special to note this?

So more-or-less the *same* configuration as is required for FBSD? If so,
I could note that in the comments part of the platform support table.

I'm not sure if either one (OBSD, FBSD) is actually explicitly
documented for PostgreSQL (I don't see a FAQ, and am not sure if there
is something in the sgml docs). Does anyone know if and where these
things are noted?

                       - Thomas

Thu, 22 Mar 2001 15:54:56 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Call for platforms
On Thu, 22 Mar 2001, Tom Lane wrote:

> The Hermit Hacker <scrappy@hub.org> writes:
> > Nope, still working through some things ... the select_implicit test
> > failed completely:
>
> > dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> more results/select_implicit.out
> > psql: connectDBStart() -- connect() failed: Connection refused
> >         Is the postmaster running locally
> >         and accepting connections on Unix socket '/tmp/.s.PGSQL.65432'?
>
> > I'm going to re-run the test(s) and see if its an isolated thing or not ...
>
> Transient overflow of the postmaster socket's accept queue, maybe?  How
> big is SOMAXCONN on your box?

Okay, for me, solaris has always been a nemesis as I can never find
anything on this box :(  But, looking through the header files, I find:

/usr/include/sys/socket.h:#define       SOMAXCONN       5

I reran the tests two more times since the above ... first time went
through clean as could be, with the geometry test failing (forgot to
update my expected/resultmaps file(s) in my compile tree), the second time
failed *totally* different tests then the first run:

dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> grep
FAILED regression.out
     opr_sanity           ... FAILED
     join                 ... FAILED
     aggregates           ... FAILED
     arrays               ... FAILED

I


Thu, 22 Mar 2001 11:56:34 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re: Call for platforms 
The Hermit Hacker <scrappy@hub.org> writes:
> the second time
> failed *totally* different tests then the first run:

> dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> grep
> FAILED regression.out
>      opr_sanity           ... FAILED
>      join                 ... FAILED
>      aggregates           ... FAILED
>      arrays               ... FAILED

These are parallel tests right?  What's the failure diffs?

			regards, tom lane

Thu, 22 Mar 2001 11:08:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
On Thu, 22 Mar 2001, Thomas Lockhart wrote:

> > > OpenBSD 2.8 x86    7.1 2001-03-22, B. Palmer (first name?)
> > Though it does work,  like FBSD,  there are some changes that need to be
> > made to the system.  Need max proc / files changes and a kernel recompile
> > with SEMMNI and SEMMNS changes.  Anywhere special to note this?
>
> So more-or-less the *same* configuration as is required for FBSD? If so,
> I could note that in the comments part of the platform support table.

The kernel changes are the same,  but OBSD needs the max proc, max open
file settings changes (no reboot required).

>
> I'm not sure if either one (OBSD, FBSD) is actually explicitly
> documented for PostgreSQL (I don't see a FAQ, and am not sure if there
> is something in the sgml docs). Does anyone know if and where these
> things are noted?

http://www.postgresql.org/devel-corner/docs/postgres/kernel-resources.html

This is the closest thing to docs.  kernel-resources for specific OSs.

- b


b. palmer,  bpalmer@crimelabs.net
pgp:  www.crimelabs.net/bpalmer.pgp5


Thu, 22 Mar 2001 11:08:11 -0500 (EST)
bpalmer <bpalmer@crimelabs.net>
Re: Re: Call for platforms
The Hermit Hacker writes:

> > Is Solaris-x86 ready to go then?
>
> Nope, still working through some things ... the select_implicit test
> failed completely:
>
> dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> more results/select_implicit.out
> psql: connectDBStart() -- connect() failed: Connection refused
>         Is the postmaster running locally
>         and accepting connections on Unix socket '/tmp/.s.PGSQL.65432'?
>
> I'm going to re-run the test(s) and see if its an isolated thing or not ...

Solaris is known to have trouble with Unix domain sockets.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 22 Mar 2001 17:20:03 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Call for platforms
Marko Kreen writes:

> OK:  Linux 2.4.2 i686 / gcc 2.95.2 / Debian testing/unstable
>
> no problems.
>
> OK?: NetBSD 1.5 i586 / egcs 2.91.66 / (netbsd-1-5 from Jan)
>
> netbsd FAILED the geometry test, diff attached, dunno if its
> critical or not.

Can you check whether it matches any of the other possible geometry
results?  See

http://www.postgresql.org/devel-corner/docs/postgres/regress-platform.html

about the mechanisms.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 22 Mar 2001 17:25:01 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Call for platforms
Thomas Lockhart writes:

> Here is the current scorecard. We have a couple of new platforms
> reported (yeaaa!):

> QNX 4.25 x86       7.0 2000-04-01, Dr. Andreas Kardos

This one is getting a "no good", as of latest reports.  There are some
issues to be worked out in the dreaded spin lock area, which will probably
not happen between now and next week.

> And here are the up-to-date platforms; thanks for the reports:

> IBM        S/390   7.1 2000-11-17, Neale Ferguson
  ^^^
 should be "Linux"

> LinuxPPC G3        7.1 2001-03-19, Tom Lane

The kernel is called "Linux", the processor is called "PowerPC G3".  But
"PowerPC" is probably enough, given that we list "x86".  Compare to...

> MacOS-X Darwin PowerPC 7.1 2000-12-11, Peter Bierman

...this.  There's a space, no dash, before the "X".

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 22 Mar 2001 17:29:27 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Call for platforms
The Hermit Hacker writes:

> How much 'diviation' are we allowing for?
>
> Solaris x86/7 results, for example, in geometry.out, show a difference of:
>
> 1.53102359078377e-11,3 (expected)
> 1.53102359017709e-11,3 (results)
>
> or
>
> 3,-3.06204718156754e-11 (expected)
> 3,-3.06204718035418e-11 (results)
>
> acceptable diviation?

Practically yes, technically not.  Check if the geometry results match any
of the other "expected" files so we can update the "resultmap".  See

http://www.postgresql.org/devel-corner/docs/postgres/regress-platform.html

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 22 Mar 2001 17:30:46 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Call for platforms
Thomas Lockhart writes:

> > > OpenBSD 2.8 x86    7.1 2001-03-22, B. Palmer (first name?)
> > Though it does work,  like FBSD,  there are some changes that need to be
> > made to the system.  Need max proc / files changes and a kernel recompile
> > with SEMMNI and SEMMNS changes.  Anywhere special to note this?
>
> So more-or-less the *same* configuration as is required for FBSD? If so,
> I could note that in the comments part of the platform support table.

Quite a few platforms will need some tuning in that area before production
use.  This is all documented.

> I'm not sure if either one (OBSD, FBSD) is actually explicitly
> documented for PostgreSQL (I don't see a FAQ, and am not sure if there
> is something in the sgml docs). Does anyone know if and where these
> things are noted?
>
>                        - Thomas
>

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 22 Mar 2001 17:33:38 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Call for platforms
On Thu, Mar 22, 2001 at 05:25:01PM +0100, Peter Eisentraut wrote:
> Marko Kreen writes:
> >
> > OK?: NetBSD 1.5 i586 / egcs 2.91.66 / (netbsd-1-5 from Jan)
> >
> > netbsd FAILED the geometry test, diff attached, dunno if its
> > critical or not.
> 
> Can you check whether it matches any of the other possible geometry
> results?  See

Yes, it matches geometry-positive-zeros-bsd.out.  There is
another report about NetBSD 1.5/i386 which has comment:

> one spurious floating point test failure
> (mail sent to postgresql-bugs with details)

But I could not find it in archive page.  (reporter Giles Lean
<giles@nemeton.com.au>)  Perhaps same thing?

-- 
marko


Thu, 22 Mar 2001 19:12:39 +0200
Marko Kreen <marko@l-t.ee>
Re: Call for platforms
On Thu, 22 Mar 2001, Tom Lane wrote:

> The Hermit Hacker <scrappy@hub.org> writes:
> > the second time
> > failed *totally* different tests then the first run:
>
> > dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> grep
> > FAILED regression.out
> >      opr_sanity           ... FAILED
> >      join                 ... FAILED
> >      aggregates           ... FAILED
> >      arrays               ... FAILED
>
> These are parallel tests right?  What's the failure diffs?

same as last time:

dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> more
results/opr_sanity.out
psql: connectDBStart() -- connect() failed: Connection refused
        Is the postmaster running locally
        and accepting connections on Unix socket '/tmp/.s.PGSQL.65432'?

and yet another run (and different results):

============== shutting down postmaster               ==============

=================================================
 1 of 76 tests failed, 1 failed test(s) ignored.
=================================================

The differences that caused some tests to fail can be viewed in the
file `./regression.diffs'.  A copy of the test summary that you see
above is saved in the file `./regression.out'.

make: *** [check] Error 1
dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> grep
FAILED regression.out
test misc                 ... FAILED
dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress>




Thu, 22 Mar 2001 13:29:47 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re: Call for platforms 
The Hermit Hacker <scrappy@hub.org> writes:
>> These are parallel tests right?  What's the failure diffs?

> same as last time:

> dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> more
> results/opr_sanity.out
> psql: connectDBStart() -- connect() failed: Connection refused
>         Is the postmaster running locally
>         and accepting connections on Unix socket '/tmp/.s.PGSQL.65432'?

See Peter's comment elsewhere that he doesn't think Solaris handles
Unix socket connections very well.  Try patching pg_regress to force
unix_sockets=no.


> and yet another run (and different results):

> =================================================
>  1 of 76 tests failed, 1 failed test(s) ignored.
> =================================================

That's just ye olde random "random" failure ...

			regards, tom lane

Thu, 22 Mar 2001 12:49:39 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Just a data point on the geometry test under NetBSD/i386 issue:

/etc/ld.so.conf by default now contains:
libm.so.0       machdep.fpu_present     1:libm387.so.0,libm.so.0

which means that if the sysctl machdep.fpu_present returns 1, load the
shared library libm387 to make use of the fpu.

If you remove /etc/ld.so.conf, so that ldd `which psql` does not show

         -lm.0 => /usr/lib/libm387.so.0
         -lm.0 => /usr/lib/libm.so.0

but only the libm.so.0 line

======================
 All 76 tests passed. 
======================

If you replace the /etc/ld.so.conf file and have an fpu, then the geometry
test will fail with slightly different rounding.

Do we want a specific geometry-netbsd-i386-with-fpu.out where you must
also test

% sysctl machdep.fpu_present
machdep.fpu_present = 1

?

Cheers,

Patrick

PS: AFAIK geometry-positive-zeros-bsd works for all NetBSD platforms - the
above difference is only for i386 + fpu.

On Thu, Mar 22, 2001 at 07:12:39PM +0200, Marko Kreen wrote:
> On Thu, Mar 22, 2001 at 05:25:01PM +0100, Peter Eisentraut wrote:
> > Marko Kreen writes:
> > >
> > > OK?: NetBSD 1.5 i586 / egcs 2.91.66 / (netbsd-1-5 from Jan)
> > >
> > > netbsd FAILED the geometry test, diff attached, dunno if its
> > > critical or not.
> > 
> > Can you check whether it matches any of the other possible geometry
> > results?  See
> 
> Yes, it matches geometry-positive-zeros-bsd.out.  There is
> another report about NetBSD 1.5/i386 which has comment:
> 
> > one spurious floating point test failure
> > (mail sent to postgresql-bugs with details)
> 
> But I could not find it in archive page.  (reporter Giles Lean
> <giles@nemeton.com.au>)  Perhaps same thing?
> 
> -- 
> marko
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

Thu, 22 Mar 2001 17:54:18 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: Call for platforms

> PS: AFAIK geometry-positive-zeros-bsd works for all NetBSD platforms - the
> above difference is only for i386 + fpu.

It doesn't on NetBSD-1.5/alpha -- there geometry-positive-zeros is
correct.

Regards,

Giles

Fri, 23 Mar 2001 06:25:50 +1100
Giles Lean <giles@nemeton.com.au>
Re: Call for platforms 
The Hermit Hacker wrote:
> 
> On Thu, 22 Mar 2001, Thomas Lockhart wrote:
> 
> > Solaris x86        7.0 2000-04-12, Marc Fournier
> >
> > scrappy, do you still have this machine?
> 
> Doing tests on Solaris x86/7 right now, will report as soon as they are
> done ...
> 
> > Solaris 2.5.1-2.7 Sparc 7.0 2000-04-12, Peter Eisentraut
> >
> > I'll bet that someone already has Solaris covered. Report?
> 
> Will do up Sparc/7 also this morning ...

In my tests on sparc/7 my compile died at line 3088 of
postgresql-7.1beta6/src/interfaces/python/pgmodule.c:

./pgmodule.c:3088: parse error before `init_pg'

That's line 3137 of today's (22Mar) snapshot, which reads:

/* Initialization function for the module */
DL_EXPORT(void)
init_pg(void)
{

I'm not a C expert by any means, but I can't figure how that is valid. 

Given my ignorance, I don't want to call it a bug. Plus we don't use the
python module in production, nor the sparc platform. But it seemed worth
pointing out.

-- 
Karl

Thu, 22 Mar 2001 14:43:27 -0500
Karl DeBisschop <kdebisschop@alert.infoplease.com>
Re: Re: Call for platforms

> NetBSD 2.8 alpha   7.1 2001-03-22, Giles Lean

Correction: NetBSD-1.5/alpha.

Ciao,

Giles

Fri, 23 Mar 2001 06:45:02 +1100
Giles Lean <giles@nemeton.com.au>
Re: Re: Call for platforms 
> > NetBSD 2.8 alpha   7.1 2001-03-22, Giles Lean
> Correction: NetBSD-1.5/alpha.

Right. That was a typo in transcribing my online copy of the sgml docs
to the email. I was hoping no one caught it, and didn't bother sending a
correction ;)

                  - Thomas

Thu, 22 Mar 2001 19:57:05 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Call for platforms
On Fri, Mar 23, 2001 at 06:25:50AM +1100, Giles Lean wrote:
> 
> > PS: AFAIK geometry-positive-zeros-bsd works for all NetBSD platforms - the
> > above difference is only for i386 + fpu.
> 
> It doesn't on NetBSD-1.5/alpha -- there geometry-positive-zeros is
> correct.

Sorry, that should have read:

AFAIK geometry-positive-zeros works for all NetBSD platforms - the
above difference is only for i386 + fpu.

(-bsd is for bsdi)

Thanks for the correction,

Patrick

Thu, 22 Mar 2001 19:58:04 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: Call for platforms
On Thu, Mar 22, 2001 at 07:58:04PM +0000, Patrick Welche wrote:
> On Fri, Mar 23, 2001 at 06:25:50AM +1100, Giles Lean wrote:
> > 
> > > PS: AFAIK geometry-positive-zeros-bsd works for all NetBSD platforms - the
> > > above difference is only for i386 + fpu.
> > 
> > It doesn't on NetBSD-1.5/alpha -- there geometry-positive-zeros is
> > correct.
> 
> Sorry, that should have read:
> 
> AFAIK geometry-positive-zeros works for all NetBSD platforms - the
> above difference is only for i386 + fpu.

Seems that following patch is needed.  Now It Works For Me (tm).
Giles, does the regress test now succed for you?

-- 
marko


Index: src/test/regress/resultmap
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/test/regress/resultmap,v
retrieving revision 1.45
diff -u -r1.45 resultmap
--- src/test/regress/resultmap	2001/03/22 15:13:18	1.45
+++ src/test/regress/resultmap	2001/03/22 17:29:49
@@ -17,6 +17,7 @@
 geometry/.*-openbsd=geometry-positive-zeros-bsd
 geometry/.*-irix6=geometry-irix
 geometry/.*-netbsd=geometry-positive-zeros
+geometry/i.86-.*-netbsdelf1.5=geometry-positive-zeros-bsd
 geometry/.*-sysv5uw7.*:cc=geometry-uw7-cc
 geometry/.*-sysv5uw7.*:gcc=geometry-uw7-gcc
 geometry/alpha.*-dec-osf=geometry-alpha-precision

Thu, 22 Mar 2001 22:27:44 +0200
Marko Kreen <marko@l-t.ee>
Re: [HACKERS] Call for platforms
On Thu, Mar 22, 2001 at 10:27:44PM +0200, Marko Kreen wrote:
> On Thu, Mar 22, 2001 at 07:58:04PM +0000, Patrick Welche wrote:
> > 
> > AFAIK geometry-positive-zeros works for all NetBSD platforms - the
> > above difference is only for i386 + fpu.
> 
> Seems that following patch is needed.  Now It Works For Me (tm).
> Giles, does the regress test now succed for you?

Your patch works for me (i386) - I'd just like to point out that it's
because we are both running on peecees with fpus and thus with libm387
loaded (else works without patch)

BTW
NetBSD 2.8 alpha   7.1 2001-03-22, Giles Lean

Shouldn't that be 1.5?

Cheers,

Patrick

Thu, 22 Mar 2001 20:30:36 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: [HACKERS] Call for platforms
On Thu, Mar 22, 2001 at 12:49:39PM -0500, Tom Lane wrote:
> The Hermit Hacker <scrappy@hub.org> writes:
> >> These are parallel tests right?  What's the failure diffs?
> 
> > same as last time:
> 
> > dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> more
> > results/opr_sanity.out
> > psql: connectDBStart() -- connect() failed: Connection refused
> >         Is the postmaster running locally
> >         and accepting connections on Unix socket '/tmp/.s.PGSQL.65432'?
> 
> See Peter's comment elsewhere that he doesn't think Solaris handles
> Unix socket connections very well.  Try patching pg_regress to force
> unix_sockets=no.
> 
> 
> > and yet another run (and different results):
> 
> > =================================================
> >  1 of 76 tests failed, 1 failed test(s) ignored.
> > =================================================
> 
> That's just ye olde random "random" failure ...

Funny, I get the more optimistic:

==================================================
 75 of 76 tests passed, 1 failed test(s) ignored. 
==================================================

Different version? PostgreSQL 7.1RC1

Thu, 22 Mar 2001 20:33:55 +0000
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: Re: Call for platforms

> Seems that following patch is needed.  Now It Works For Me (tm).
> Giles, does the regress test now succed for you?

Yes, but I don't like that it is 1.5 specific.  I expect that later
NetBSD/i386 releases will also have the "new" floating point behaviour
by default, subject to /etc/ld.so.conf setting as Patrick Welche
discovered.

BTW NetBSD just uses "i386" for any x86.  It's not necessary to allow
for i486, i586 etc.

Perhaps the resultmap format could be enhanced to allow wildcarding of
the result files, and just accept either match?

geometry/.*-netbsd=geometry-positive-zeros*

Regards,

Giles

> Index: src/test/regress/resultmap
> ===================================================================
> RCS file: /home/projects/pgsql/cvsroot/pgsql/src/test/regress/resultmap,v
> retrieving revision 1.45
> diff -u -r1.45 resultmap
> --- src/test/regress/resultmap	2001/03/22 15:13:18	1.45
> +++ src/test/regress/resultmap	2001/03/22 17:29:49
> @@ -17,6 +17,7 @@
>  geometry/.*-openbsd=geometry-positive-zeros-bsd
>  geometry/.*-irix6=geometry-irix
>  geometry/.*-netbsd=geometry-positive-zeros
> +geometry/i.86-.*-netbsdelf1.5=geometry-positive-zeros-bsd
>  geometry/.*-sysv5uw7.*:cc=geometry-uw7-cc
>  geometry/.*-sysv5uw7.*:gcc=geometry-uw7-gcc
>  geometry/alpha.*-dec-osf=geometry-alpha-precision


Fri, 23 Mar 2001 07:48:08 +1100
Giles Lean <giles@nemeton.com.au>
Re: [HACKERS] Call for platforms 
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:

> If a platform you are running on is not listed, make sure it gets
> included! 

Red Hat Linux, Wolverine Beta (and some updates) - glibc 2.2.2,
2.4.2ish kernel (read: lots of fixes), gcc 2.96RH: All 76 tests passed
with 7.1beta6 (parallel_schedule).

I'll update this info when we do our next release. 

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

22 Mar 2001 16:02:11 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: Call for platforms
On Thu, 22 Mar 2001, Patrick Welche wrote:

> On Thu, Mar 22, 2001 at 12:49:39PM -0500, Tom Lane wrote:
> > The Hermit Hacker <scrappy@hub.org> writes:
> > >> These are parallel tests right?  What's the failure diffs?
> >
> > > same as last time:
> >
> > > dragon:/home/centre/marc/src/postgresql-7.1RC1/src/test/regress> more
> > > results/opr_sanity.out
> > > psql: connectDBStart() -- connect() failed: Connection refused
> > >         Is the postmaster running locally
> > >         and accepting connections on Unix socket '/tmp/.s.PGSQL.65432'?
> >
> > See Peter's comment elsewhere that he doesn't think Solaris handles
> > Unix socket connections very well.  Try patching pg_regress to force
> > unix_sockets=no.
> >
> >
> > > and yet another run (and different results):
> >
> > > =================================================
> > >  1 of 76 tests failed, 1 failed test(s) ignored.
> > > =================================================
> >
> > That's just ye olde random "random" failure ...
>
> Funny, I get the more optimistic:
>
> ==================================================
>  75 of 76 tests passed, 1 failed test(s) ignored.
> ==================================================
>
> Different version? PostgreSQL 7.1RC1

7.1RC1 (the not released yet version) :)



Thu, 22 Mar 2001 17:44:47 -0400 (AST)
The Hermit Hacker <scrappy@hub.org>
Re: Re: Call for platforms
teg@redhat.com (Trond Eivind Glomsrï¿½d) writes:

> Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> 
> > If a platform you are running on is not listed, make sure it gets
> > included! 
> 
> Red Hat Linux, Wolverine Beta (and some updates) - glibc 2.2.2,
> 2.4.2ish kernel (read: lots of fixes), gcc 2.96RH: All 76 tests passed
> with 7.1beta6 (parallel_schedule).

Forgot to mention: This is x86.


-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

22 Mar 2001 17:46:18 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: Call for platforms
On 22 Mar 2001, Trond Eivind [iso-8859-1] Glomsr�d wrote:

> teg@redhat.com (Trond Eivind Glomsr�d) writes:
>
> > Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> >
> > > If a platform you are running on is not listed, make sure it gets
> > > included!
> >
> > Red Hat Linux, Wolverine Beta (and some updates) - glibc 2.2.2,
> > 2.4.2ish kernel (read: lots of fixes), gcc 2.96RH: All 76 tests passed
> > with 7.1beta6 (parallel_schedule).
>
> Forgot to mention: This is x86.

Forget to enter it into the regresstest database?

   http://www.postgresql.org/~vev/regress/

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
         56K Nationwide Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Thu, 22 Mar 2001 20:29:20 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: Call for platforms
-----BEGIN PGP SIGNED MESSAGE-----

On 22 Mar 2001, at 14:29, Thomas Lockhart wrote:

> Linux 2.2.x armv4l 7.0 2000-04-17, Mark Knox

Compiled and tested 7.1beta6 tonight. All the regression tests passed 
except two - the usual minor differences in geometry (rounding on the 
final digit) and this rather troubling output from type_sanity. I'm 
not altogether sure what impact this has. Everything seems to run 
just fine.


*** ./expected/type_sanity.out  Tue Sep 12 00:49:16 2000
- --- ./results/type_sanity.out   Thu Mar 22 21:42:49 2001
***************
*** 172,177 ****
       p1.attalign != p2.typalign OR
       p1.attbyval != p2.typbyval);
   oid | attname | oid | typname 
! -----+---------+-----+---------
! (0 rows)
  
- --- 172,239 ----
       p1.attalign != p2.typalign OR
       p1.attbyval != p2.typbyval);
    oid  | attname | oid | typname 
! -------+---------+-----+---------
!  16572 | ctid    |  27 | tid
!  16593 | ctid    |  27 | tid
!  16610 | ctid    |  27 | tid
!  16635 | ctid    |  27 | tid
!  16646 | ctid    |  27 | tid
!  16678 | ctid    |  27 | tid
!  16691 | ctid    |  27 | tid
!  16873 | ctid    |  27 | tid
!  16941 | ctid    |  27 | tid
!  16953 | ctid    |  27 | tid
!  16970 | ctid    |  27 | tid
!  17038 | ctid    |  27 | tid
!  17051 | ctid    |  27 | tid
!  17067 | ctid    |  27 | tid
!  17079 | ctid    |  27 | tid
!  17090 | ctid    |  27 | tid
!  17206 | ctid    |  27 | tid
!  17221 | ctid    |  27 | tid
!  17236 | ctid    |  27 | tid
!  17251 | ctid    |  27 | tid
!  17266 | ctid    |  27 | tid
!  17281 | ctid    |  27 | tid
!  17301 | ctid    |  27 | tid
!  17314 | ctid    |  27 | tid
!  17327 | ctid    |  27 | tid
!  17342 | ctid    |  27 | tid
!  17355 | ctid    |  27 | tid
!  18792 | ctid    |  27 | tid
!  18820 | ctid    |  27 | tid
!  18832 | ctid    |  27 | tid
!  18845 | ctid    |  27 | tid
!  18857 | ctid    |  27 | tid
!  18869 | ctid    |  27 | tid
!  18888 | ctid    |  27 | tid
!  18922 | ctid    |  27 | tid
!  18937 | ctid    |  27 | tid
!  18967 | ctid    |  27 | tid
!  18990 | ctid    |  27 | tid
!  19005 | ctid    |  27 | tid
!  19019 | ctid    |  27 | tid
!  19031 | ctid    |  27 | tid
!  19042 | ctid    |  27 | tid
!  19053 | ctid    |  27 | tid
!  19069 | ctid    |  27 | tid
!  19080 | ctid    |  27 | tid
!  19103 | ctid    |  27 | tid
!  20617 | ctid    |  27 | tid
!  20633 | ctid    |  27 | tid
!  20643 | ctid    |  27 | tid
!  20655 | ctid    |  27 | tid
!  20677 | ctid    |  27 | tid
!  20689 | ctid    |  27 | tid
!  20702 | ctid    |  27 | tid
!  20716 | ctid    |  27 | tid
!  20726 | ctid    |  27 | tid
!  20766 | ctid    |  27 | tid
!  20784 | ctid    |  27 | tid
!  20794 | ctid    |  27 | tid
!  20804 | ctid    |  27 | tid
!  20836 | ctid    |  27 | tid
!  20860 | ctid    |  27 | tid
!  20879 | ctid    |  27 | tid
! (62 rows)


-----BEGIN PGP SIGNATURE-----
Version: N/A

iQCVAwUBOrrHrv+IdJuhyV9xAQGemgQApLVZS9xWQAtIzfgw3ILQThtEdftUBH20
FCoNqod++HunTazDwQZo6Msbunlvb8cJmSXg/kRkUmN6FQ39RtK9XEWsvFUy1+Nx
eJCHiHyIMZBmmXNK1eiK0AyxFSqD8MKtgSuKvprXWNzTD4+NVZzWy9h1cONhZviN
KEj9thVXQDc=
=TG7n
-----END PGP SIGNATURE-----

Thu, 22 Mar 2001 22:49:02 -0500
"Mark Knox" <segfault@hardline.org>
Re: Call for platforms
I have tested today's snap shot on SunOS4.

% uname -a
SunOS srashd 4.1.4-JL 1 sun4m

There's a minor portability problem in
src/bin/pg_encoding/Makefile.

*** Makefile    Fri Mar 23 11:53:49 2001
--- Makefile.orig       Wed Feb 21 18:05:21 2001
***************
*** 16,28 ****
  
  all: submake pg_encoding
  
- ifdef STRTOUL
- OBJS+=$(top_builddir)/src/backend/port/strtoul.o
- 
- $(top_builddir)/src/backend/port/strtoul.o:
-       $(MAKE) -C $(top_builddir)/src/backend/port strtoul.o
- endif
- 
  pg_encoding: $(OBJS)
        $(CC) $(CFLAGS) $^ $(libpq) $(LDFLAGS) $(LIBS) -o $@
  
I'm going to check in this correction soon.

For the regression test, I got 7 failures, most of them seem harmless,
the only concern I have is bit test though.
--
Tatsuo Ishii

P.S.   I'm going to test Linux/MIPS (Cobalt RaQ2) soon...
--
Tatsuo Ishii
Fri, 23 Mar 2001 14:31:27 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Call for platforms
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> For the regression test, I got 7 failures, most of them seem harmless,
> the only concern I have is bit test though.

Most of the diffs derive from what I recall to be a known SunOS problem,
that strtol fails to notice overflow.  A value that should be rejected
is getting inserted into int4_tbl (mod 2^32 of course).

The bit test diffs seem to indicate that bit_cmp is messed up.  That
depends on memcmp.  I seem to recall something about memcmp not being
8-bit-clean on SunOS ... does that ring a bell with anyone?

			regards, tom lane

Fri, 23 Mar 2001 01:08:06 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Call for platforms 
> I have tested today's snap shot on SunOS4.
> For the regression test, I got 7 failures, most of them seem harmless,
> the only concern I have is bit test though.
> P.S.   I'm going to test Linux/MIPS (Cobalt RaQ2) soon...

Great! I'll update info for SunOS4 (individual problems will be fixed or
"known features" ;) and look forward to seeing the Linux/MIPS results.

                        - Thomas

Fri, 23 Mar 2001 06:31:43 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Call for platforms
> > I have tested today's snap shot on SunOS4.
> > For the regression test, I got 7 failures, most of them seem harmless,
> > the only concern I have is bit test though.
> > P.S.   I'm going to test Linux/MIPS (Cobalt RaQ2) soon...
> 
> Great! I'll update info for SunOS4 (individual problems will be fixed or
> "known features" ;) and look forward to seeing the Linux/MIPS results.

Sorry, after moving of my office, this is the first time to boot RaQ2
but it won't boot anymore. Seems there are some severe hardware
troubles with it. Can anyone else do the testing instead of me?
--
Tatsuo Ishii

Fri, 23 Mar 2001 16:11:18 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Re: Call for platforms
> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> > For the regression test, I got 7 failures, most of them seem harmless,
> > the only concern I have is bit test though.
> 
> Most of the diffs derive from what I recall to be a known SunOS problem,
> that strtol fails to notice overflow.  A value that should be rejected
> is getting inserted into int4_tbl (mod 2^32 of course).
> 
> The bit test diffs seem to indicate that bit_cmp is messed up.  That
> depends on memcmp.  I seem to recall something about memcmp not being
> 8-bit-clean on SunOS ... does that ring a bell with anyone?

Good point. From the man page of memcmp(3) on this machine:

BUGS
     memcmp() uses native character comparison, which  is  signed
     on  some  machines and unsigned on other machines.  Thus the
     sign of the value returned when one of  the  characters  has
     its high-order bit set is implementation-dependent.
--
Tatsuo Ishii

Fri, 23 Mar 2001 17:19:17 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Call for platforms 
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
>> The bit test diffs seem to indicate that bit_cmp is messed up.  That
>> depends on memcmp.  I seem to recall something about memcmp not being
>> 8-bit-clean on SunOS ... does that ring a bell with anyone?

> Good point. From the man page of memcmp(3) on this machine:

> BUGS
>      memcmp() uses native character comparison, which  is  signed
>      on  some  machines and unsigned on other machines.  Thus the
>      sign of the value returned when one of  the  characters  has
>      its high-order bit set is implementation-dependent.

Eeek.

The C spec documents I have at hand all agree that memcmp, strcmp,
etc shall interpret their arguments as unsigned char.  I hope Sun
were the only ones who took the above more liberal interpretation...

			regards, tom lane

Fri, 23 Mar 2001 03:33:03 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Call for platforms 
> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> >> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> > ! FATAL 2:  ZeroFill(logfile 0 seg 1) failed: No such file or directory
> > ! pqReadData() -- backend closed the channel unexpectedly.
> >> 
> >> Is it possible you ran out of disk space?
> 
> > Probably not.
> 
> The reason I was speculating that was that it seems pretty unlikely
> that a write() call could return ENOENT, as the above appears to
> suggest.  I think that the errno = ENOENT value was not set by write(),
> but is leftover from the expected failure of BasicOpenFile earlier in
> XLogFileInit.  Probably write() returned some value less than BLCKSZ
> but more than zero, and so did not set errno.
> 
> Offhand the only reason I can think of for a write to a disk file
> to terminate after a partial transfer is a full disk.  What do you
> think?

Sorry, I was wrong. I accidentaly ran out the disk space.

BTW, I got segfault when I first try beta6 on this platform. To
investigae it, I recompiled with -g (without -O2) and now the problem
has gone. It sems there's something wrong with the compiler (gcc
version egcs-2.90.25 980302 (egcs-1.0.2 prerelease)) or potential bug
in 7.1, I don't know.

Anyway, the platform is too old now, and I would like to try it
another day with newer MkLinux version installed. I don't want to make
this as a show stopper for 7.1...
-- 
Tatsuo Ishii

*** ./expected/oid.out	Tue Nov 21 12:23:20 2000
--- ./results/oid.out	Thu Mar 22 15:58:56 2001
***************
*** 6,11 ****
--- 6,12 ----
  INSERT INTO OID_TBL(f1) VALUES ('1235');
  INSERT INTO OID_TBL(f1) VALUES ('987');
  INSERT INTO OID_TBL(f1) VALUES ('-1040');
+ ERROR:  oidin: error in "-1040": can't parse "-1040"
  INSERT INTO OID_TBL(f1) VALUES ('99999999');
  INSERT INTO OID_TBL(f1) VALUES ('');
  -- bad inputs 
***************
*** 15,28 ****
  ERROR:  oidin: error in "99asdfasd": can't parse "asdfasd"
  SELECT '' AS six, OID_TBL.*;
   six |     f1     
! -----+------------
       |       1234
       |       1235
       |        987
-      | 4294966256
       |   99999999
       |          0
! (6 rows)
  
  SELECT '' AS one, o.* FROM OID_TBL o WHERE o.f1 = 1234;
   one |  f1  
--- 16,28 ----
  ERROR:  oidin: error in "99asdfasd": can't parse "asdfasd"
  SELECT '' AS six, OID_TBL.*;
   six |    f1    
! -----+----------
       |     1234
       |     1235
       |      987
       | 99999999
       |        0
! (5 rows)
  
  SELECT '' AS one, o.* FROM OID_TBL o WHERE o.f1 = 1234;
   one |  f1  
***************
*** 32,44 ****
  
  SELECT '' AS five, o.* FROM OID_TBL o WHERE o.f1 <> '1234';
   five |     f1     
! ------+------------
        |       1235
        |        987
-       | 4294966256
        |   99999999
        |          0
! (5 rows)
  
  SELECT '' AS three, o.* FROM OID_TBL o WHERE o.f1 <= '1234';
   three |  f1  
--- 32,43 ----
  
  SELECT '' AS five, o.* FROM OID_TBL o WHERE o.f1 <> '1234';
   five |    f1    
! ------+----------
        |     1235
        |      987
        | 99999999
        |        0
! (4 rows)
  
  SELECT '' AS three, o.* FROM OID_TBL o WHERE o.f1 <= '1234';
   three |  f1  
***************
*** 57,75 ****
  
  SELECT '' AS four, o.* FROM OID_TBL o WHERE o.f1 >= '1234';
   four |     f1     
! ------+------------
        |       1234
        |       1235
-       | 4294966256
        |   99999999
! (4 rows)
  
  SELECT '' AS three, o.* FROM OID_TBL o WHERE o.f1 > '1234';
   three |     f1     
! -------+------------
         |       1235
-        | 4294966256
         |   99999999
! (3 rows)
  
  DROP TABLE OID_TBL;
--- 56,72 ----
  
  SELECT '' AS four, o.* FROM OID_TBL o WHERE o.f1 >= '1234';
   four |    f1    
! ------+----------
        |     1234
        |     1235
        | 99999999
! (3 rows)
  
  SELECT '' AS three, o.* FROM OID_TBL o WHERE o.f1 > '1234';
   three |    f1    
! -------+----------
         |     1235
         | 99999999
! (2 rows)
  
  DROP TABLE OID_TBL;

======================================================================

*** ./expected/geometry-powerpc-linux-gnulibc1.out	Wed Sep 13 06:07:16 2000
--- ./results/geometry.out	Thu Mar 22 16:01:20 2001
***************
*** 445,451 ****
  -----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
       | ((-3,0),(-2.59807621135076,1.50000000000442),(-1.49999999999116,2.59807621135842),(1.53102359017709e-11,3),(1.50000000001768,2.59807621134311),(2.59807621136607,1.4999999999779),(3,-3.06204718035418e-11),(2.59807621133545,-1.50000000003094),(1.49999999996464,-2.59807621137373),(-4.59307077053127e-11,-3),(-1.5000000000442,-2.5980762113278),(-2.59807621138138,-1.49999999995138))
       | ((-99,2),(-85.6025403783588,52.0000000001473),(-48.9999999997054,88.602540378614),(1.00000000051034,102),(51.0000000005893,88.6025403781036),(87.6025403788692,51.9999999992634),(101,1.99999999897932),(87.6025403778485,-48.0000000010313),(50.9999999988214,-84.6025403791243),(0.999999998468976,-98),(-49.0000000014732,-84.6025403775933),(-85.6025403793795,-47.9999999983795))
!      | ((-4,3),(-3.33012701891794,5.50000000000737),(-1.49999999998527,7.3301270189307),(1.00000000002552,8),(3.50000000002946,7.33012701890518),(5.33012701894346,5.49999999996317),(6,2.99999999994897),(5.33012701889242,0.499999999948437),(3.49999999994107,-1.33012701895622),(0.999999999923449,-2),(-1.50000000007366,-1.33012701887966),(-3.33012701896897,0.500000000081027))
       | ((-2,2),(-1.59807621135076,3.50000000000442),(-0.499999999991161,4.59807621135842),(1.00000000001531,5),(2.50000000001768,4.59807621134311),(3.59807621136607,3.4999999999779),(4,1.99999999996938),(3.59807621133545,0.499999999969062),(2.49999999996464,-0.598076211373729),(0.999999999954069,-1),(-0.500000000044197,-0.598076211327799),(-1.59807621138138,0.500000000048616))
       | ((90,200),(91.3397459621641,205.000000000015),(95.0000000000295,208.660254037861),(100.000000000051,210),(105.000000000059,208.66025403781),(108.660254037887,204.999999999926),(110,199.999999999898),(108.660254037785,194.999999999897),(104.999999999882,191.339745962088),(99.9999999998469,190),(94.9999999998527,191.339745962241),(91.3397459620621,195.000000000162))
       | ((0,0),(13.3974596216412,50.0000000001473),(50.0000000002946,86.602540378614),(100.00000000051,100),(150.000000000589,86.6025403781036),(186.602540378869,49.9999999992634),(200,-1.02068239345139e-09),(186.602540377848,-50.0000000010313),(149.999999998821,-86.6025403791243),(99.999999998469,-100),(49.9999999985268,-86.6025403775933),(13.3974596206205,-49.9999999983795))
--- 445,451 ----
  -----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
       | ((-3,0),(-2.59807621135076,1.50000000000442),(-1.49999999999116,2.59807621135842),(1.53102359017709e-11,3),(1.50000000001768,2.59807621134311),(2.59807621136607,1.4999999999779),(3,-3.06204718035418e-11),(2.59807621133545,-1.50000000003094),(1.49999999996464,-2.59807621137373),(-4.59307077053127e-11,-3),(-1.5000000000442,-2.5980762113278),(-2.59807621138138,-1.49999999995138))
       | ((-99,2),(-85.6025403783588,52.0000000001473),(-48.9999999997054,88.602540378614),(1.00000000051034,102),(51.0000000005893,88.6025403781036),(87.6025403788692,51.9999999992634),(101,1.99999999897932),(87.6025403778485,-48.0000000010313),(50.9999999988214,-84.6025403791243),(0.999999998468976,-98),(-49.0000000014732,-84.6025403775933),(-85.6025403793795,-47.9999999983795))
!      | ((-4,3),(-3.33012701891794,5.50000000000737),(-1.49999999998527,7.3301270189307),(1.00000000002552,8),(3.50000000002946,7.33012701890518),(5.33012701894346,5.49999999996317),(6,2.99999999994897),(5.33012701889242,0.499999999948437),(3.49999999994107,-1.33012701895622),(0.999999999923449,-2),(-1.50000000007366,-1.33012701887967),(-3.33012701896897,0.500000000081028))
       | ((-2,2),(-1.59807621135076,3.50000000000442),(-0.499999999991161,4.59807621135842),(1.00000000001531,5),(2.50000000001768,4.59807621134311),(3.59807621136607,3.4999999999779),(4,1.99999999996938),(3.59807621133545,0.499999999969062),(2.49999999996464,-0.598076211373729),(0.999999999954069,-1),(-0.500000000044197,-0.598076211327799),(-1.59807621138138,0.500000000048616))
       | ((90,200),(91.3397459621641,205.000000000015),(95.0000000000295,208.660254037861),(100.000000000051,210),(105.000000000059,208.66025403781),(108.660254037887,204.999999999926),(110,199.999999999898),(108.660254037785,194.999999999897),(104.999999999882,191.339745962088),(99.9999999998469,190),(94.9999999998527,191.339745962241),(91.3397459620621,195.000000000162))
       | ((0,0),(13.3974596216412,50.0000000001473),(50.0000000002946,86.602540378614),(100.00000000051,100),(150.000000000589,86.6025403781036),(186.602540378869,49.9999999992634),(200,-1.02068239345139e-09),(186.602540377848,-50.0000000010313),(149.999999998821,-86.6025403791243),(99.999999998469,-100),(49.9999999985268,-86.6025403775933),(13.3974596206205,-49.9999999983795))

======================================================================


Fri, 23 Mar 2001 17:45:28 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Call for platforms 
Vince Vielhaber <vev@michvhf.com> writes:

> On 22 Mar 2001, Trond Eivind [iso-8859-1] Glomsrï¿½d wrote:
> 
> > teg@redhat.com (Trond Eivind Glomsrï¿½d) writes:
> >
> > > Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> > >
> > > > If a platform you are running on is not listed, make sure it gets
> > > > included!
> > >
> > > Red Hat Linux, Wolverine Beta (and some updates) - glibc 2.2.2,
> > > 2.4.2ish kernel (read: lots of fixes), gcc 2.96RH: All 76 tests passed
> > > with 7.1beta6 (parallel_schedule).
> >
> > Forgot to mention: This is x86.
> 
> Forget to enter it into the regresstest database?
> 
>    http://www.postgresql.org/~vev/regress/

I was planning on waiting with that until I test it on an official release.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

23 Mar 2001 10:35:34 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: Call for platforms
On 23 Mar 2001, Trond Eivind [iso-8859-1] Glomsr�d wrote:

> Vince Vielhaber <vev@michvhf.com> writes:
>
> > On 22 Mar 2001, Trond Eivind [iso-8859-1] Glomsr�d wrote:
> >
> > > teg@redhat.com (Trond Eivind Glomsr�d) writes:
> > >
> > > > Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> > > >
> > > > > If a platform you are running on is not listed, make sure it gets
> > > > > included!
> > > >
> > > > Red Hat Linux, Wolverine Beta (and some updates) - glibc 2.2.2,
> > > > 2.4.2ish kernel (read: lots of fixes), gcc 2.96RH: All 76 tests passed
> > > > with 7.1beta6 (parallel_schedule).
> > >
> > > Forgot to mention: This is x86.
> >
> > Forget to enter it into the regresstest database?
> >
> >    http://www.postgresql.org/~vev/regress/
>
> I was planning on waiting with that until I test it on an official release.

I figured that, it was my just smartass way of reminding EVERYONE to put
their data in the database.  I saw a few reports of things working yet
there was nothing in the database saying so, it was only posted here.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
         56K Nationwide Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Fri, 23 Mar 2001 10:48:24 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: Call for platforms
> OpenBSD 2.8 x86    7.1 2001-03-22, Brandon. Palmer

OBSD checks out for sparc and i386.  We did need to make a change to the
resultmap file to make the regression tests clean for the sparc.  I have
attached the diff.





Also,  on the sparc that i'm using (sparc4/110),  make check takes 1950
seconds.  Most of the time is spent in this test:

parallel group (13 tests):  float4 int2 int4 text name varchar oid boolean
char float8 int8 bit numeric

There is a long pause between 'bit' and 'numeric'.  Same with on i386.  Is
this a problem that is local to obsd?  Is it an expected delay?  It works,
but seems like a real perf problem.








Anyway:

++++++++++++++++

Sparc 4/110, 64M, SCSI disk, OBSD 2.8 virgin

======================
 All 76 tests passed.
======================

 1941.34s real   130.23s user    93.77s system

$ uname -a
OpenBSD azreal 2.8 GENERIC#0 sparc

++++++++++++++++

P2 300, 96M, IDE Disk,  OBSD 2.8 virgin

======================
 All 76 tests passed.
======================

  262.67s real    21.84s user    13.56s system

$ uname -a
OpenBSD orion 2.8 GENERIC#0 i386

++++++++++++++++

I can't get tcl/tk working to same my life,  but that's not too important
for a release,  just a config pain in the rear for obsd I guess.

- brandon

b. palmer,  bpalmer@crimelabs.net
pgp:  www.crimelabs.net/bpalmer.pgp5
Fri, 23 Mar 2001 12:38:16 -0500 (EST)
bpalmer <bpalmer@crimelabs.net>
Re: Re: Call for platforms
bpalmer <bpalmer@crimelabs.net> writes:
> seconds.  Most of the time is spent in this test:

> parallel group (13 tests):  float4 int2 int4 text name varchar oid boolean
> char float8 int8 bit numeric

> There is a long pause between 'bit' and 'numeric'.  Same with on i386.  Is
> this a problem that is local to obsd?  Is it an expected delay?

Yes, that's the expected behavior.  The 'numeric' test runs considerably
longer than most of the others.  (It used to be even slower, but I made
Jan trim it down ;-))

			regards, tom lane

Fri, 23 Mar 2001 15:08:27 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Hi all,

I've built 7.1beta6 on a number of different HP-UX platforms (11.00 32
bit, 11.00 64 bit, 11i 32 bit).

1. On all these platforms 'make check' hung.  Since that's not
   critical to whether PostgreSQL works or not I worked around it by
   using a different shell:

   gmake SHELL=$HOME/bin/pdksh check

   I'll look at this next week.  If someone can confirm that
   /usr/bin/sh works for make check on HP-UX 10.20 that would be
   useful.

2. I saw two different sets of output for geometry.out.  These seem to
   relate to the processor level:

   (a) on PA-RISC 1.1 some of the zero values are negative

   (b) on PA-RISC 2.0 the negative zeros were produced as on PA-RISC
       1.1, plus about three results varied in the least significant
       digit.

       The PA-RISC 2.0 values were identical on two platforms:

       (i)  PA8000 running 32 bit 11i
       (ii) PA8500 running 64 bit 11.00

   If these results are OK (I assumed they were for the purposes
   of Vince's database, so I hope they are :-) then perhaps the
   attached outputs can be added to the expected results and
   resultmap updated for HP-UX 11?

Regards,

Giles
Sat, 24 Mar 2001 14:45:54 +1100
Giles Lean <giles@nemeton.com.au>
Call for platforms (HP-UX)
Giles Lean <giles@nemeton.com.au> writes:
>    I'll look at this next week.  If someone can confirm that
>    /usr/bin/sh works for make check on HP-UX 10.20 that would be
>    useful.

It does not work.  See FAQ_HPUX.

> 2. I saw two different sets of output for geometry.out.  These seem to
>    relate to the processor level:

I think it depends more on what software you use.  The existing HPUX
resultmap (geometry-positive-zeros) works on my usual platform (C180,
PA8000 chip I think) when using gcc.  Compile with cc and you get one
different lowest-order digit in two lines, IIRC.  I have not tried it
lately on a 1.1 chip.

>    (a) on PA-RISC 1.1 some of the zero values are negative

Hmm, so does it match any of the existing geometry files?  That would
suggest that HPUX 11 has started adhering more closely to the IEEE rules
about negative zeroes ...

			regards, tom lane

Fri, 23 Mar 2001 23:14:37 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Call for platforms (HP-UX) 

> >    I'll look at this next week.  If someone can confirm that
> >    /usr/bin/sh works for make check on HP-UX 10.20 that would be
> >    useful.
> 
> It does not work.  See FAQ_HPUX.

I'm confused: I don't see anything about shells or make check hanging
in doc/FAQ_HPUX.  There is clear instruction to use GNU make, which I
am doing.

I'll look into the problem anyway.

> >    (a) on PA-RISC 1.1 some of the zero values are negative
> 
> Hmm, so does it match any of the existing geometry files?

No ... I was hoping for that, but not.

Regards,

Giles


Sat, 24 Mar 2001 15:32:29 +1100
Giles Lean <giles@nemeton.com.au>
Re: Call for platforms (HP-UX) 
Giles Lean <giles@nemeton.com.au> writes:
>> It does not work.  See FAQ_HPUX.

> I'm confused: I don't see anything about shells or make check hanging
> in doc/FAQ_HPUX.  There is clear instruction to use GNU make, which I
> am doing.

Hm, I thought I had updated that before beta6.  What it has now is

The parallel regression test script (gmake check) is known to lock up
when run under HP's default Bourne shell, at least in HPUX 10.20.  This
appears to be a shell bug, not the fault of the script.  If you see that
the tests have stopped making progress and only a shell process is
consuming CPU, kill the shell and start over with
	gmake SHELL=/bin/ksh check
to use ksh instead.


			regards, tom lane

Fri, 23 Mar 2001 23:36:06 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Call for platforms (HP-UX) 

> Hm, I thought I had updated that before beta6.  What it has now is

<grin>

> The parallel regression test script (gmake check) is known to lock up
> when run under HP's default Bourne shell, at least in HPUX 10.20.  This
> appears to be a shell bug, not the fault of the script.  If you see that
> the tests have stopped making progress and only a shell process is
> consuming CPU, kill the shell and start over with
> 	gmake SHELL=/bin/ksh check
> to use ksh instead.

Interestingly, ksh didn't work for me either, but didn't try it on all
the platforms I built on.  I'll make a note to check ksh when I'm
investigating the problem next week though, and let you know what I
find.

Regards,

Giles


Sat, 24 Mar 2001 15:41:55 +1100
Giles Lean <giles@nemeton.com.au>
Re: Call for platforms (HP-UX) 
Giles Lean <giles@nemeton.com.au> writes:
> 2. I saw two different sets of output for geometry.out.  These seem to
>    relate to the processor level:

Okay, here are my results:

Box 1: C180 (2.0 PA8000), HPUX 10.20

Compile with gcc: all tests pass
Compile with cc: two lines of diffs in geometry (attached)

Box 2: 715/75 (1.1 PA7100LC), HPUX 10.20

Compile with gcc: all tests pass
Compile with cc: all tests pass


Box 1 is more up-to-date on HP patches than box 2, so I wouldn't
necessarily attribute the difference to the processor.

			regards, tom lane

*** ./expected/geometry-positive-zeros.out	Mon Sep 11 23:21:06 2000
--- ./results/geometry.out	Sat Mar 24 02:45:35 2001
***************
*** 127,133 ****
          | (-5,-12)   | [(10,-10),(-3,-4)]            | (-1.60487804878049,-4.64390243902439)
          | (10,10)    | [(10,-10),(-3,-4)]            | (2.39024390243902,-6.48780487804878)
          | (0,0)      | [(-1000000,200),(300000,-40)] | (0.0028402365895872,15.384614860264)
!         | (-10,0)    | [(-1000000,200),(300000,-40)] | (-9.99715942258202,15.3864610140472)
          | (-3,4)     | [(-1000000,200),(300000,-40)] | (-2.99789812267519,15.3851688427303)
          | (5.1,34.5) | [(-1000000,200),(300000,-40)] | (5.09647083221496,15.3836744976925)
          | (-5,-12)   | [(-1000000,200),(300000,-40)] | (-4.99494420845634,15.3855375281616)
--- 127,133 ----
          | (-5,-12)   | [(10,-10),(-3,-4)]            | (-1.60487804878049,-4.64390243902439)
          | (10,10)    | [(10,-10),(-3,-4)]            | (2.39024390243902,-6.48780487804878)
          | (0,0)      | [(-1000000,200),(300000,-40)] | (0.0028402365895872,15.384614860264)
!         | (-10,0)    | [(-1000000,200),(300000,-40)] | (-9.99715942258202,15.3864610140473)
          | (-3,4)     | [(-1000000,200),(300000,-40)] | (-2.99789812267519,15.3851688427303)
          | (5.1,34.5) | [(-1000000,200),(300000,-40)] | (5.09647083221496,15.3836744976925)
          | (-5,-12)   | [(-1000000,200),(300000,-40)] | (-4.99494420845634,15.3855375281616)
***************
*** 445,451 ****
  -----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
       | ((-3,0),(-2.59807621135076,1.50000000000442),(-1.49999999999116,2.59807621135842),(1.53102359017709e-11,3),(1.50000000001768,2.59807621134311),(2.59807621136607,1.4999999999779),(3,-3.06204718035418e-11),(2.59807621133545,-1.50000000003094),(1.49999999996464,-2.59807621137373),(-4.59307077053127e-11,-3),(-1.5000000000442,-2.5980762113278),(-2.59807621138138,-1.49999999995138))
       | ((-99,2),(-85.6025403783588,52.0000000001473),(-48.9999999997054,88.602540378614),(1.00000000051034,102),(51.0000000005893,88.6025403781036),(87.6025403788692,51.9999999992634),(101,1.99999999897932),(87.6025403778485,-48.0000000010313),(50.9999999988214,-84.6025403791243),(0.999999998468976,-98),(-49.0000000014732,-84.6025403775933),(-85.6025403793795,-47.9999999983795))
!      | ((-4,3),(-3.33012701891794,5.50000000000737),(-1.49999999998527,7.3301270189307),(1.00000000002552,8),(3.50000000002946,7.33012701890518),(5.33012701894346,5.49999999996317),(6,2.99999999994897),(5.33012701889242,0.499999999948437),(3.49999999994107,-1.33012701895622),(0.999999999923449,-2),(-1.50000000007366,-1.33012701887967),(-3.33012701896897,0.500000000081028))
       | ((-2,2),(-1.59807621135076,3.50000000000442),(-0.499999999991161,4.59807621135842),(1.00000000001531,5),(2.50000000001768,4.59807621134311),(3.59807621136607,3.4999999999779),(4,1.99999999996938),(3.59807621133545,0.499999999969062),(2.49999999996464,-0.598076211373729),(0.999999999954069,-1),(-0.500000000044197,-0.598076211327799),(-1.59807621138138,0.500000000048616))
       | ((90,200),(91.3397459621641,205.000000000015),(95.0000000000295,208.660254037861),(100.000000000051,210),(105.000000000059,208.66025403781),(108.660254037887,204.999999999926),(110,199.999999999898),(108.660254037785,194.999999999897),(104.999999999882,191.339745962088),(99.9999999998469,190),(94.9999999998527,191.339745962241),(91.3397459620621,195.000000000162))
       | ((0,0),(13.3974596216412,50.0000000001473),(50.0000000002946,86.602540378614),(100.00000000051,100),(150.000000000589,86.6025403781036),(186.602540378869,49.9999999992634),(200,-1.02068239345139e-09),(186.602540377848,-50.0000000010313),(149.999999998821,-86.6025403791243),(99.999999998469,-100),(49.9999999985268,-86.6025403775933),(13.3974596206205,-49.9999999983795))
--- 445,451 ----
  -----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
       | ((-3,0),(-2.59807621135076,1.50000000000442),(-1.49999999999116,2.59807621135842),(1.53102359017709e-11,3),(1.50000000001768,2.59807621134311),(2.59807621136607,1.4999999999779),(3,-3.06204718035418e-11),(2.59807621133545,-1.50000000003094),(1.49999999996464,-2.59807621137373),(-4.59307077053127e-11,-3),(-1.5000000000442,-2.5980762113278),(-2.59807621138138,-1.49999999995138))
       | ((-99,2),(-85.6025403783588,52.0000000001473),(-48.9999999997054,88.602540378614),(1.00000000051034,102),(51.0000000005893,88.6025403781036),(87.6025403788692,51.9999999992634),(101,1.99999999897932),(87.6025403778485,-48.0000000010313),(50.9999999988214,-84.6025403791243),(0.999999998468976,-98),(-49.0000000014732,-84.6025403775933),(-85.6025403793795,-47.9999999983795))
!      | ((-4,3),(-3.33012701891794,5.50000000000737),(-1.49999999998527,7.3301270189307),(1.00000000002552,8),(3.50000000002946,7.33012701890518),(5.33012701894346,5.49999999996317),(6,2.99999999994897),(5.33012701889242,0.499999999948437),(3.49999999994107,-1.33012701895622),(0.999999999923449,-2),(-1.50000000007366,-1.33012701887966),(-3.33012701896897,0.500000000081027))
       | ((-2,2),(-1.59807621135076,3.50000000000442),(-0.499999999991161,4.59807621135842),(1.00000000001531,5),(2.50000000001768,4.59807621134311),(3.59807621136607,3.4999999999779),(4,1.99999999996938),(3.59807621133545,0.499999999969062),(2.49999999996464,-0.598076211373729),(0.999999999954069,-1),(-0.500000000044197,-0.598076211327799),(-1.59807621138138,0.500000000048616))
       | ((90,200),(91.3397459621641,205.000000000015),(95.0000000000295,208.660254037861),(100.000000000051,210),(105.000000000059,208.66025403781),(108.660254037887,204.999999999926),(110,199.999999999898),(108.660254037785,194.999999999897),(104.999999999882,191.339745962088),(99.9999999998469,190),(94.9999999998527,191.339745962241),(91.3397459620621,195.000000000162))
       | ((0,0),(13.3974596216412,50.0000000001473),(50.0000000002946,86.602540378614),(100.00000000051,100),(150.000000000589,86.6025403781036),(186.602540378869,49.9999999992634),(200,-1.02068239345139e-09),(186.602540377848,-50.0000000010313),(149.999999998821,-86.6025403791243),(99.999999998469,-100),(49.9999999985268,-86.6025403775933),(13.3974596206205,-49.9999999983795))

======================================================================


Sat, 24 Mar 2001 02:59:15 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Call for platforms (HP-UX) 
Tom Lane writes:

> The bit test diffs seem to indicate that bit_cmp is messed up.  That
> depends on memcmp.  I seem to recall something about memcmp not being
> 8-bit-clean on SunOS ... does that ring a bell with anyone?

Sure enough:

 - Macro: AC_FUNC_MEMCMP
     If the `memcmp' function is not available, or does not work on
     8-bit data (like the one on SunOS 4.1.3), add `memcmp.o' to output
     variable `LIBOBJS'.

We could try to mangle this into doing the right thing for us.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 24 Mar 2001 10:38:04 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Call for platforms 
Tom Lane writes:

> > and yet another run (and different results):
>
> > =================================================
> >  1 of 76 tests failed, 1 failed test(s) ignored.
> > =================================================
>
> That's just ye olde random "random" failure ...

Actually, this is one real failed test plus the "random" failure.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 24 Mar 2001 11:18:11 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Call for platforms 
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
> =================================================
> 1 of 76 tests failed, 1 failed test(s) ignored.
> =================================================
>> 
>> That's just ye olde random "random" failure ...

> Actually, this is one real failed test plus the "random" failure.

(Checks code...)  Hm, you're right.  May I suggest that this is a rather
confusing wording?  Perhaps

 1 of 76 tests failed, plus 1 failed test(s) ignored.

would be less likely to mislead people.

			regards, tom lane

Sat, 24 Mar 2001 11:18:57 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> The bit test diffs seem to indicate that bit_cmp is messed up.  That
>> depends on memcmp.  I seem to recall something about memcmp not being
>> 8-bit-clean on SunOS ... does that ring a bell with anyone?

> Sure enough:
>  - Macro: AC_FUNC_MEMCMP
>      If the `memcmp' function is not available, or does not work on
>      8-bit data (like the one on SunOS 4.1.3), add `memcmp.o' to output
>      variable `LIBOBJS'.
> We could try to mangle this into doing the right thing for us.

Not sure if it's worth the trouble.  That would be an AC_TRY_RUN test,
which you've been trying to move away from, no?  It doesn't seem like
anyone still cares about SunOS 4.1.*, so ...

			regards, tom lane

Sat, 24 Mar 2001 13:31:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Call for platforms 
Hi all.

Suddenly I obtain access to 
ULTRIX black 4.3 1 RISC

I don't shure is it supported, but I see /src/include/port/ultrix4.h file
so my guess is `yes, at least was'. I got last version from CVS and try
configure && gmake
it results in

gcc  -Wall -Wmissing-prototypes -Wmissing-declarations
-I../../../../src/include   -c xlog.c -o xlog.o
In file included from xlog.c:36:
../../../../src/include/storage/s_lock.h:88: warning: type defaults to
`int' in declaration of `slock_t'
../../../../src/include/storage/s_lock.h:88: parse error before `*'
../../../../src/include/storage/s_lock.h:91: warning: type defaults to
`int' in declaration of `slock_t'
../../../../src/include/storage/s_lock.h:91: parse error before `*'
gmake[4]: *** [xlog.o] Error 1

grep of .h files shows that slock_t usualy defined in
/src/include/port/*.h, but it is not defined in ultrix4.h
and I can't find it in system includes.

Regards,
ASK


Sun, 25 Mar 2001 18:57:24 +0200 (IST)
Alexander Klimov <ask@wisdom.weizmann.ac.il>
Re: Call for platforms
Alexander Klimov <ask@wisdom.weizmann.ac.il> writes:
> Suddenly I obtain access to 
> ULTRIX black 4.3 1 RISC

Uh ... what kind of processor is that?  Offhand I don't see any
indication that any of the entries in s_lock.h are supposed to work
for Ultrix.

			regards, tom lane

Sun, 25 Mar 2001 12:24:10 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Tom Lane <tgl@sss.pgh.pa.us> writes:

> Alexander Klimov <ask@wisdom.weizmann.ac.il> writes:
> > Suddenly I obtain access to 
> > ULTRIX black 4.3 1 RISC
> 
> Uh ... what kind of processor is that?  Offhand I don't see any
> indication that any of the entries in s_lock.h are supposed to work
> for Ultrix.

The RISC/Ultrix machines ran (older) MIPS chips.  Ultrix also ran
(amazingly slowly) on the VAX architecture.

[Fond memories of my first sysadmin job...]

-Doug

25 Mar 2001 13:25:00 -0500
Doug McNaught <doug@wireboard.com>
Re: Re: Call for platforms
> Alexander Klimov <ask@wisdom.weizmann.ac.il> writes:
>> Suddenly I obtain access to 
>> ULTRIX black 4.3 1 RISC

> Uh ... what kind of processor is that?  Offhand I don't see any
> indication that any of the entries in s_lock.h are supposed to work
> for Ultrix.

On closer look I notice that the putative support for machines without
a TEST_AND_SET implementation got broken by careless rearrangement of
the declarations in s_lock.h :-(.  I have repaired this, and if you
update from CVS you should find that things compile.

However, you don't really want to run without TEST_AND_SET support;
it'll be dog-slow.  Furthermore, the support for machines without
TEST_AND_SET is fairly new.  I doubt it existed when the Ultrix port
was last reported to work.  So the question above still stands.

I suspect that some one of the implementations in s_lock.h was intended
to be usable on Ultrix, and we've somehow dropped the declarations
needed to make it go.  You might want to pull down an old tarball (6.3
or before) and look at how it compiles the s_lock support on Ultrix.

Please send in a patch if you find that one is necessary for s_lock
support on Ultrix.

			regards, tom lane

Sun, 25 Mar 2001 13:26:47 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
"Mark Knox" <segfault@hardline.org> writes:
>> Linux 2.2.x armv4l 7.0 2000-04-17, Mark Knox

> Compiled and tested 7.1beta6 tonight. All the regression tests passed 
> except two - the usual minor differences in geometry (rounding on the 
> final digit) and this rather troubling output from type_sanity.

Most bizarre --- and definitely indicative of trouble.  Would you send
along the output of this query in that database:

select p1.oid,attrelid,relname,attname,attlen,attalign,attbyval
from pg_attribute p1, pg_class p2
where atttypid = 27 and p2.oid = attrelid
order by 1;

			regards, tom lane

Sun, 25 Mar 2001 15:02:42 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
-----BEGIN PGP SIGNED MESSAGE-----

On 25 Mar 2001, at 15:02, Tom Lane wrote:

> > (rounding on the final digit) and this rather troubling output from
> > type_sanity.
> 
> Most bizarre --- and definitely indicative of trouble.  Would you send
> along the output of this query in that database:
> 
> select p1.oid,attrelid,relname,attname,attlen,attalign,attbyval
> from pg_attribute p1, pg_class p2
> where atttypid = 27 and p2.oid = attrelid
> order by 1;

I was afraid of that ;) Here's the output:

[PostgreSQL 7.1beta6 on armv4l-unknown-linux-gnuoldld, compiled by 
GCC 2.95.1]

   type \? for help on slash commands
   type \q to quit
   type \g or terminate with semicolon to execute query
 You are currently connected to the database: postgres

postgres=> select 
p1.oid,attrelid,relname,attname,attlen,attalign,attbyval from 
pg_attribute p1, pg_class p2 where atttypid = 27 and p2.oid = 
attrelid order by 1;
  oid|attrelid|relname       |attname|attlen|attalign|attbyval
- -----+--------+--------------+-------+------+--------+--------
16401|    1247|pg_type       |ctid   |     6|i       |f       
16415|    1262|pg_database   |ctid   |     6|i       |f       
16439|    1255|pg_proc       |ctid   |     6|i       |f       
16454|    1260|pg_shadow     |ctid   |     6|i       |f       
16464|    1261|pg_group      |ctid   |     6|i       |f       
16486|    1249|pg_attribute  |ctid   |     6|i       |f       
16515|    1259|pg_class      |ctid   |     6|i       |f       
16526|    1215|pg_attrdef    |ctid   |     6|i       |f       
16537|    1216|pg_relcheck   |ctid   |     6|i       |f       
16557|    1219|pg_trigger    |ctid   |     6|i       |f       
16572|   16567|pg_inherits   |ctid   |     8|i       |f       
16593|   16579|pg_index      |ctid   |     8|i       |f       
16610|   16600|pg_statistic  |ctid   |     8|i       |f       
16635|   16617|pg_operator   |ctid   |     8|i       |f       
16646|   16642|pg_opclass    |ctid   |     8|i       |f       
16678|   16653|pg_am         |ctid   |     8|i       |f       
16691|   16685|pg_amop       |ctid   |     8|i       |f       
16873|   16867|pg_amproc     |ctid   |     8|i       |f       
16941|   16934|pg_language   |ctid   |     8|i       |f       
16953|   16948|pg_largeobject|ctid   |     8|i       |f       
16970|   16960|pg_aggregate  |ctid   |     8|i       |f       
17038|   17033|pg_ipl        |ctid   |     8|i       |f       
17051|   17045|pg_inheritproc|ctid   |     8|i       |f       
17067|   17058|pg_rewrite    |ctid   |     8|i       |f       
17079|   17074|pg_listener   |ctid   |     8|i       |f       
17090|   17086|pg_description|ctid   |     8|i       |f       
17206|   17201|pg_toast_1215 |ctid   |     8|i       |f       
17221|   17216|pg_toast_17086|ctid   |     8|i       |f       
17236|   17231|pg_toast_1255 |ctid   |     8|i       |f       
17251|   17246|pg_toast_1216 |ctid   |     8|i       |f       
17266|   17261|pg_toast_17058|ctid   |     8|i       |f       
17281|   17276|pg_toast_16600|ctid   |     8|i       |f       
17301|   17291|pg_user       |ctid   |     8|i       |f       
17314|   17309|pg_rules      |ctid   |     8|i       |f       
17327|   17322|pg_views      |ctid   |     8|i       |f       
17342|   17335|pg_tables     |ctid   |     8|i       |f       
17355|   17350|pg_indexes    |ctid   |     8|i       |f       
(37 rows)


-----BEGIN PGP SIGNATURE-----
Version: N/A

iQCVAwUBOr5XA/+IdJuhyV9xAQGfOgP6ApV6ia44bxCo/KyIE20knn/1FTysECW9
Rq9mLDhpYKHYtTWz1cgGtxzCEiRAMN+ZuO7u5nydy6TB8dp8iCd9eLAro4GAzqYM
aD9V9S3nK3YwV9RaKBWJqHXNPI5enp19YS74GxN0f9VIw/4PXlYVm2tQJLVWNGs+
lFfQnraYEZQ=
=Cj2l
-----END PGP SIGNATURE-----

Sun, 25 Mar 2001 15:37:23 -0500
"Mark Knox" <segfault@hardline.org>
Re: Re: Call for platforms 
Does that database have any user-created relations in it, or is it
just a virgin database?  It seems that the wrong attlen is being
computed for ctid fields during bootstrap, but the regression test
output (if it was complete) implies that the value inserted for
user-created fields was OK.  This doesn't make a lot of sense since
it's the same code...

			regards, tom lane

Sun, 25 Mar 2001 16:07:13 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
On Wed, 21 Mar 2001, Thomas Lockhart wrote:

> OK, here is my current platform list taken from the -hackers list and
> from Vince's web page. I'm sure I've missed at least a few reports, but
> please confirm that platforms are actually running and passing
> regression tests with recent betas or the latest release candidate.

	Updates...

> Linux 2.2.x Alpha  7.1 2001-01-23, Ryan Kirkpatrick

Tested RC1 with 2.2.17 on my XLT366 Alpha, all regression tests passed.

> Linux 2.2.15 Sparc 7.1 2001-01-30, Ryan Kirkpatrick

Tested RC1 with 2.2.18 on my Sparc 20 (SM51), all regression tests passed. 

	Both have been entered into the regression database on the website
as well. TTYL.

---------------------------------------------------------------------------
|   "For to me to live is Christ, and to die is gain."                    |
|                                            --- Philippians 1:21 (KJV)   |
---------------------------------------------------------------------------
|   Ryan Kirkpatrick  |  Boulder, Colorado  |  http://www.rkirkpat.net/   |
---------------------------------------------------------------------------


Sun, 25 Mar 2001 20:49:56 -0700 (MST)
Ryan Kirkpatrick <pgsql@rkirkpat.net>
Re: Call for platforms
Two more for the list (not a single regression test failing, which is a
first on Alpha!)

Tru64 4.0G Alpha cc-v6.3-129  7.1 2001-03-28 
Tru64 4.0G Alpha gcc-2.95.1   7.1 2001-03-28

I updated the regression test database as well.

Adriaan

Mon, 26 Mar 2001 09:24:36 +0300
Adriaan Joubert <a.joubert@albourne.com>
Re: Call for platforms
> >> Suddenly I obtain access to
> >> ULTRIX black 4.3 1 RISC
> > Uh ... what kind of processor is that?  Offhand I don't see any
> > indication that any of the entries in s_lock.h are supposed to work
> > for Ultrix.

As mentioned earlier, Ultrix on RISC means that it is a MIPS processor.
DEC implemented OSF-1 for their Alpha processors.

> I suspect that some one of the implementations in s_lock.h was intended
> to be usable on Ultrix, and we've somehow dropped the declarations
> needed to make it go.  You might want to pull down an old tarball (6.3
> or before) and look at how it compiles the s_lock support on Ultrix.

Any hints for Alexander on how to do it *if* it is a MIPS processor?

                      - Thomas

Mon, 26 Mar 2001 16:18:33 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Call for platforms
The list of unreported or "in progress" platforms has gotten much
shorter. If anyone can help on the remaining problems, we'll be able to
move closer to release status, which is A Good Thing (tm) ;)

btw, if we get most of these qualified, we will be on around 30
platforms!!!!

                  - Thomas

Unreported or problem platforms:

Linux 2.0.x MIPS   7.0 2000-04-13, Tatsuo Ishii

Tatsuo's machine has died. Anyone else with a Cobalt?

mklinux PPC750     7.0 2000-04-13, Tatsuo Ishii

Any luck with RC1?

NetBSD m68k        7.0 2000-04-10, Henry B. Hotz
NetBSD Sparc       7.0 2000-04-13, Tom I. Helbekkmo

We need some NetBSD folks to speak up! Also, there are several flavors
of OpenBSD which are not represented in our list, but which probably are
already running PostgreSQL. Anyone?

QNX 4.25 x86       7.0 2000-04-01, Dr. Andreas Kardos

Does QNX get demoted to the "unsupported list"? It is known to have
problems with 7.1, right?

Solaris x86        7.0 2000-04-12, Marc Fournier

scrappy, did you work through the tests yet?

Ultrix MIPS        7.1 2001-??-??, Alexander Klimov

Any possibilities here?


And here are the up-to-date platforms; thanks for the reports:

AIX 4.3.3 RS6000   7.1 2001-03-21, Gilles Darold
BeOS 5.0.3 x86     7.1 2000-12-18, Cyril Velter
BSDI 4.01  x86     7.1 2001-03-19, Bruce Momjian
Compaq Tru64 4.0g Alpha 7.1 2001-03-19, Brent Verner
FreeBSD 4.3 x86    7.1 2001-03-19, Vince Vielhaber
HPUX PA-RISC       7.1 2001-03-19, 10.20 Tom Lane, 11.00 Giles Lean
IRIX 6.5.11 MIPS   7.1 2001-03-22, Robert Bruccoleri
Linux 2.2.x Alpha  7.1 2001-01-23, Ryan Kirkpatrick
Linux 2.2.x armv4l 7.1 2001-03-22, Mark Knox
Linux 2.2.18 PPC750 7.1 2001-03-19, Tom Lane
Linux 2.2.x S/390  7.1 2000-11-17, Neale Ferguson
Linux 2.2.15 Sparc 7.1 2001-01-30, Ryan Kirkpatrick
Linux 2.2.16 x86   7.1 2001-03-19, Thomas Lockhart
MacOS X Darwin PPC 7.1 2000-12-11, Peter Bierman
NetBSD 1.5 alpha   7.1 2001-03-22, Giles Lean
NetBSD 1.5E arm32  7.1 2001-03-21, Patrick Welche
NetBSD 1.5S x86    7.1 2001-03-21, Patrick Welche
OpenBSD 2.8 x86    7.1 2001-03-22, Brandon Palmer
SCO OpenServer 5 x86   7.1 2001-03-13, Billy Allie
SCO UnixWare 7.1.1 x86 7.1 2001-03-19, Larry Rosenman
Solaris 2.7 Sparc  7.1 2001-03-22, Marc Fournier
SunOS 4.1.4 Sparc  7.1 2001-03-23, Tatsuo Ishii
Windows/Win32 x86  7.1 2001-03-26, Magnus Hagander (clients only)
WinNT/Cygwin x86   7.1 2001-03-16, Jason Tishler

Mon, 26 Mar 2001 17:14:17 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Call for platforms
Thomas Lockhart writes:

> SCO OpenServer 5 x86   7.1 2001-03-13, Billy Allie

Where did you see this?  I don't find it in the archives or in Vince's
database.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Mon, 26 Mar 2001 20:01:47 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Call for platforms
> > SCO OpenServer 5 x86   7.1 2001-03-13, Billy Allie
> Where did you see this?  I don't find it in the archives or in Vince's
> database.

In FAQ_SCO. I was looking to try to figure out what the differences were
between the SCO products :)

                   - Thomas

Mon, 26 Mar 2001 18:05:55 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Call for platforms

Since the SCO UDK works on both UnixWare and OpenServer, I think we are 
pretty safe.  Also, there was a post to -HACKERS about the accept bug and 
we changed the workaround to include OSR5. 

I'd leave it until disproved.  I don't have a OSR5 installation to check 
it with, however. 

LER

>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<

On 3/26/01, 12:05:55 PM, Thomas Lockhart <lockhart@alumni.caltech.edu> 
wrote regarding Re: [HACKERS] Re: Call for platforms:


> > > SCO OpenServer 5 x86   7.1 2001-03-13, Billy Allie
> > Where did you see this?  I don't find it in the archives or in Vince's
> > database.

> In FAQ_SCO. I was looking to try to figure out what the differences were
> between the SCO products :)

>                    - Thomas

> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

Mon, 26 Mar 2001 18:09:05 GMT
Larry Rosenman <ler@lerctr.org>
Re: Re: Call for platforms
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> Linux 2.2.18 PPC750 7.1 2001-03-19, Tom Lane

"PPC750"?  What's that?  "PPC G3" might be more likely to mean something
to onlookers ...

			regards, tom lane

Mon, 26 Mar 2001 13:22:56 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> As mentioned earlier, Ultrix on RISC means that it is a MIPS processor.

>> I suspect that some one of the implementations in s_lock.h was intended
>> to be usable on Ultrix, and we've somehow dropped the declarations
>> needed to make it go.  You might want to pull down an old tarball (6.3
>> or before) and look at how it compiles the s_lock support on Ultrix.

> Any hints for Alexander on how to do it *if* it is a MIPS processor?

Not sure.  The only info I see in s_lock.h is in the "SGI" section:

 * This stuff may be supplemented in the future with Masato Kataoka's MIPS-II
 * assembly from his NECEWS SVR4 port, but we probably ought to retain this
 * for the R3000 chips out there.

That name doesn't ring a bell with me --- anyone remember what code is
being referred to here, or where we might find Masato Kataoka?

MIPS-II code may or may not be compatible with Alexander's machine
anyway, but it's the only starting point I see.

Anyway, the last CVS update to port/ultrix.h that appears to have come
from someone actually using Ultrix was rev 1.2 on 7-May-97, which
predates the very existence of s_lock.h as a separate file.  So I'd
definitely advise Alexander to find a tarball from that era and look at
how Ultrix was handled then.

I dunno if we even have tarballs from that far back on-line ... I
suppose another possibility is a date-based pull from the CVS server.

			regards, tom lane

Mon, 26 Mar 2001 13:30:44 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Thomas Lockhart writes:

> > > SCO OpenServer 5 x86   7.1 2001-03-13, Billy Allie
> > Where did you see this?  I don't find it in the archives or in Vince's
> > database.
>
> In FAQ_SCO. I was looking to try to figure out what the differences were
> between the SCO products :)

I wouldn't necessarily count something dated Oct 9, 2000.  That was half a
year ago, and even two months before beta.  And the message doesn't
actually say it worked.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Mon, 26 Mar 2001 21:03:19 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Call for platforms
> "PPC750"?  What's that?  "PPC G3" might be more likely to mean something
> to onlookers ...

Actually "G3" means nothing outside of Apple afaict. The 750 series is a
follow-on to the 60x series, and there is a 7xxx series also. From my
pov, using an accepted label, rather than a marketing (re)label, better
indicates *what* this actually can run on. I'm not sure that I have it
labeled correctly yet, but "G3" is not a step in the right direction.

As we both found, it is difficult to wade through Apple's own docs to
decipher which processor is actually built into the system.

Should I put "Mac G3" in the comment section?

                       - Thomas

Mon, 26 Mar 2001 20:36:19 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Call for platforms
> > As mentioned earlier, Ultrix on RISC means that it is a MIPS processor.
> > Any hints for Alexander on how to do it *if* it is a MIPS processor?
> Not sure.  The only info I see in s_lock.h is in the "SGI" section:
>  * This stuff may be supplemented in the future with Masato Kataoka's MIPS-II
>  * assembly from his NECEWS SVR4 port, but we probably ought to retain this
>  * for the R3000 chips out there.
> That name doesn't ring a bell with me --- anyone remember what code is
> being referred to here, or where we might find Masato Kataoka?

I'm not remembering either...

> MIPS-II code may or may not be compatible with Alexander's machine
> anyway, but it's the only starting point I see.

The Ultrix machine is more likely to be a 2000- or 3000-series (older)
processor.

> Anyway, the last CVS update to port/ultrix.h that appears to have come
> from someone actually using Ultrix was rev 1.2 on 7-May-97, which
> predates the very existence of s_lock.h as a separate file.  So I'd
> definitely advise Alexander to find a tarball from that era and look at
> how Ultrix was handled then.
> I dunno if we even have tarballs from that far back on-line ... I
> suppose another possibility is a date-based pull from the CVS server.

What can we help with Alex?

                          - Thomas

Mon, 26 Mar 2001 20:39:10 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Call for platforms
> > > > SCO OpenServer 5 x86   7.1 2001-03-13, Billy Allie
> > > Where did you see this?  I don't find it in the archives or in Vince's
> > > database.
> > In FAQ_SCO. I was looking to try to figure out what the differences were
> > between the SCO products :)
> I wouldn't necessarily count something dated Oct 9, 2000.  That was half a
> year ago, and even two months before beta.  And the message doesn't
> actually say it worked.

?? I can see I was thrown off by the "last updated:" line near the top
of the file. It actually comes from a CVS commit, not from an explicit
update of the info in the file.

Very bad :(

                        - Thomas

Mon, 26 Mar 2001 20:43:11 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Call for platforms
> Did you get the message from Trond about Linux 2.4 x86?  I can also
> verify all tests passed on a RedHat Public Beta installation with kernel
> 2.4.

Trond had indicated that it was a 2.4.2 kernel with lots 'o patches, so
I figured I'd show the released stuff for now. I mentioned 2.4.2 in the
comments section.

                    - Thomas

Mon, 26 Mar 2001 21:39:38 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Call for platforms
I would..... 

LER

-- 
Larry Rosenman                                                            
         http://www.lerctr.org/~ler/
Phone: +1 972 414 9812                                                    
         E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749 US

>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<

On 3/26/01, 2:36:19 PM, Thomas Lockhart <lockhart@alumni.caltech.edu> wrote 
regarding Re: [HACKERS] Re: Call for platforms:


> > "PPC750"?  What's that?  "PPC G3" might be more likely to mean something
> > to onlookers ...

> Actually "G3" means nothing outside of Apple afaict. The 750 series is a
> follow-on to the 60x series, and there is a 7xxx series also. From my
> pov, using an accepted label, rather than a marketing (re)label, better
> indicates *what* this actually can run on. I'm not sure that I have it
> labeled correctly yet, but "G3" is not a step in the right direction.

> As we both found, it is difficult to wade through Apple's own docs to
> decipher which processor is actually built into the system.

> Should I put "Mac G3" in the comment section?

>                        - Thomas

> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly

Mon, 26 Mar 2001 21:41:28 GMT
Larry Rosenman <ler@lerctr.org>
Re: Re: Call for platforms
Mathijs Brands wrote:
> 
> Hi
> 
> Is there a list somewhere listing the platforms 7.1 is being
> tested on right now? I'd be able to run regression tests on
> the following platforms, if necessary:

http://www.postgresql.org/devel-corner/docs/admin/supported-platforms.html

is close to up to date (I made a few changes this morning).

http://www.postgresql.org/~vev/regress/

is an on-line display and data entry page, which I have not managed yet
to use in my development workflow. So I hope to look at it occasionally,
but the -hackers mailing list is where I get most of my info.

>   FreeBSD 3.3 (x86)
>   FreeBSD 4.2 (x86)

4.3 (and I think 4.2) is covered already.

>   Linux (x86 - 2.2 & 2.4 kernels, Redhat & Debian distro's)

Linux on x86 is pretty well covered, but we welcome additional tests and
tests on as many variants as possible.

>   Solaris 7 (SPARC)
>   Solaris 8 (x86)
>   Solaris 8 (SPARC)

Tests on Solaris 8, both Sparc and x86, would be very helpful. No
reports so far, afaik.

>   IRIX 6.2
>   IRIX 6.5

Irix 6.5.11 has been reported recently. But additional tests and testers
would be a good thing, since there aren't that many in the -hacker
community at the moment.

> If I can get the box back to working order, Alpha Linux is
> also an option. I'd be willing to build binary packages for
> Solaris and IRIX.

Alpha Linux is covered at the moment. Binary packages would be great.

Thanks for the help! Regards.

                     - Thomas

Mon, 26 Mar 2001 21:50:17 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Call for platforms
Karl DeBisschop <kdebisschop@range.infoplease.com> writes:
> In my tests on sparc/7 my compile died at line 3088 of
> postgresql-7.1beta6/src/interfaces/python/pgmodule.c:

> ./pgmodule.c:3088: parse error before `init_pg'

> That's line 3137 of today's (22Mar) snapshot, which reads:

> /* Initialization function for the module */
> DL_EXPORT(void)
> init_pg(void)
> {

What version of Python are you using?  In Python 1.5, I find this
in Python.h:

#ifndef DL_EXPORT	/* declarations for DLL import/export */
#define DL_EXPORT(RTYPE) RTYPE
#endif

which should make the above work.

			regards, tom lane

Mon, 26 Mar 2001 17:15:12 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Hi

Is there a list somewhere listing the platforms 7.1 is being
tested on right now? I'd be able to run regression tests on
the following platforms, if necessary:

  FreeBSD 3.3 (x86)
  FreeBSD 4.2 (x86)
  Linux (x86 - 2.2 & 2.4 kernels, Redhat & Debian distro's)
  Solaris 7 (SPARC)
  Solaris 8 (x86)
  Solaris 8 (SPARC)
  IRIX 6.2
  IRIX 6.5

If I can get the box back to working order, Alpha Linux is
also an option. I'd be willing to build binary packages for
Solaris and IRIX.

Regards,

Mathijs
-- 
"Books constitute capital." 
     Thomas Jefferson 

Tue, 27 Mar 2001 00:26:13 +0200
Mathijs Brands <mathijs@ilse.nl>
Re: Re: Call for platforms
Thomas Lockhart wrote:
> Linux 2.2.x Alpha  7.1 2001-01-23, Ryan Kirkpatrick
> Linux 2.2.x armv4l 7.1 2001-03-22, Mark Knox
> Linux 2.2.18 PPC750 7.1 2001-03-19, Tom Lane
> Linux 2.2.x S/390  7.1 2000-11-17, Neale Ferguson
> Linux 2.2.15 Sparc 7.1 2001-01-30, Ryan Kirkpatrick
> Linux 2.2.16 x86   7.1 2001-03-19, Thomas Lockhart

Did you get the message from Trond about Linux 2.4 x86?  I can also
verify all tests passed on a RedHat Public Beta installation with kernel
2.4.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 26 Mar 2001 17:27:04 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Call for platforms
Lamar Owen <lamar.owen@wgcr.org> writes:

> Thomas Lockhart wrote:
> > Linux 2.2.x Alpha  7.1 2001-01-23, Ryan Kirkpatrick
> > Linux 2.2.x armv4l 7.1 2001-03-22, Mark Knox
> > Linux 2.2.18 PPC750 7.1 2001-03-19, Tom Lane
> > Linux 2.2.x S/390  7.1 2000-11-17, Neale Ferguson
> > Linux 2.2.15 Sparc 7.1 2001-01-30, Ryan Kirkpatrick
> > Linux 2.2.16 x86   7.1 2001-03-19, Thomas Lockhart
> 
> Did you get the message from Trond about Linux 2.4 x86?  I can also
> verify all tests passed on a RedHat Public Beta installation with kernel
> 2.4.

I haven't put those in the list yet... I'll wait until we release a
product, and test it on that.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

26 Mar 2001 17:31:42 -0500
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: Re: Call for platforms

On Tue, 27 Mar 2001, Mathijs Brands wrote:

> Hi
>
> Is there a list somewhere listing the platforms 7.1 is being
> tested on right now? I'd be able to run regression tests on
> the following platforms, if necessary:
>
>   FreeBSD 3.3 (x86)
>   FreeBSD 4.2 (x86)
>   Linux (x86 - 2.2 & 2.4 kernels, Redhat & Debian distro's)
>   Solaris 7 (SPARC)
>   Solaris 8 (x86)
>   Solaris 8 (SPARC)
>   IRIX 6.2
>   IRIX 6.5
>
> If I can get the box back to working order, Alpha Linux is
> also an option. I'd be willing to build binary packages for
> Solaris and IRIX.

Check out the Developer's Corner on the website.  It's at the
top of the page.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
         56K Nationwide Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Mon, 26 Mar 2001 17:41:31 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: Re: Call for platforms
On Mon, Mar 26, 2001 at 05:41:31PM -0500, Vince Vielhaber allegedly wrote:
> On Tue, 27 Mar 2001, Mathijs Brands wrote:
> > Hi
> >
> > Is there a list somewhere listing the platforms 7.1 is being
> > tested on right now? I'd be able to run regression tests on
> > the following platforms, if necessary:
> >
> >   FreeBSD 3.3 (x86)
> >   FreeBSD 4.2 (x86)
> >   Linux (x86 - 2.2 & 2.4 kernels, Redhat & Debian distro's)
> >   Solaris 7 (SPARC)
> >   Solaris 8 (x86)
> >   Solaris 8 (SPARC)
> >   IRIX 6.2
> >   IRIX 6.5
> >
> > If I can get the box back to working order, Alpha Linux is
> > also an option. I'd be willing to build binary packages for
> > Solaris and IRIX.
> 
> Check out the Developer's Corner on the website.  It's at the
> top of the page.
> 
> Vince.

I had a look at it, but that surely can't be the complete list.
There are only 20 results listed...

Mathijs
-- 
"Where is human nature so weak as in a bookstore!" 
        Henry Ward Beecher  (1813-1887) 

Tue, 27 Mar 2001 00:46:15 +0200
Mathijs Brands <mathijs@ilse.nl>
Re: Re: Call for platforms
Trond Eivind Glomsrï¿½d wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > Did you get the message from Trond about Linux 2.4 x86?  I can also
> > verify all tests passed on a RedHat Public Beta installation with kernel
> > 2.4.
 
> I haven't put those in the list yet... I'll wait until we release a
> product, and test it on that.

Ah.  Ok.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 26 Mar 2001 17:48:21 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: Call for platforms
At 5:14 PM +0000 3/26/01, Thomas Lockhart wrote:
>NetBSD m68k        7.0 2000-04-10, Henry B. Hotz

I no longer have a 68k machine that's fast enough to reasonably test 
PG on.  I have a IIcx that sometimes serves as a router, but I'm 
using some second-generation powermac's  mostly now.  (You still have 
that Centris in your closet Tom?)

I *did* just get MacOS X this weekend though and if I get it working 
on my work G4 maybe I could give it a try there.


Signature held pending an ISO 9000 compliant
signature design and approval process.
h.b.hotz@jpl.nasa.gov, or hbhotz@oxy.edu

Mon, 26 Mar 2001 15:00:27 -0800
"Henry B. Hotz" <hotz@jpl.nasa.gov>
Re: Call for platforms
> >NetBSD m68k        7.0 2000-04-10, Henry B. Hotz
> I no longer have a 68k machine that's fast enough to reasonably test
> PG on.  I have a IIcx that sometimes serves as a router, but I'm
> using some second-generation powermac's  mostly now.  (You still have
> that Centris in your closet Tom?)

Oof. With its giant 250MB hard disk. I'm not likely to ever get that
going ;)

> I *did* just get MacOS X this weekend though and if I get it working
> on my work G4 maybe I could give it a try there.

It will require at least the second Darwin beta release to work.

                   - Thomas

Mon, 26 Mar 2001 23:06:18 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Call for platforms
> The non-test-and-set case should work again in current CVS, and I'd
> appreciate it if Alexander would verify that.  But as far as getting
> some test-and-set support for MIPS goes, it looks like the only way
> is for someone to sit down with a MIPS assembly manual.  I haven't
> got one, nor access to a machine to test on...

That is not already available from the Irix support code?

                      - Thomas

Mon, 26 Mar 2001 23:08:27 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: Call for platforms
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> Anyway, the last CVS update to port/ultrix.h that appears to have come
>> from someone actually using Ultrix was rev 1.2 on 7-May-97, which
>> predates the very existence of s_lock.h as a separate file.  So I'd
>> definitely advise Alexander to find a tarball from that era and look at
>> how Ultrix was handled then.
>> I dunno if we even have tarballs from that far back on-line ... I
>> suppose another possibility is a date-based pull from the CVS server.

> What can we help with Alex?

After digging around in the old code I have to retract my opinion that
a test-and-set implementation used to exist for MIPS.  The code did
have SysV-semaphore-based support for machines without test-and-set,
and undoubtedly that's what was used on the old Ultrix port.  (The
non-test-and-set code was broken for awhile, but I'd forgotten that
it formerly worked.)

The non-test-and-set case should work again in current CVS, and I'd
appreciate it if Alexander would verify that.  But as far as getting
some test-and-set support for MIPS goes, it looks like the only way
is for someone to sit down with a MIPS assembly manual.  I haven't
got one, nor access to a machine to test on...

			regards, tom lane

Mon, 26 Mar 2001 18:35:59 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
On Mon, Mar 26, 2001 at 06:35:59PM -0500, Tom Lane allegedly wrote:
> Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> >> Anyway, the last CVS update to port/ultrix.h that appears to have come
> >> from someone actually using Ultrix was rev 1.2 on 7-May-97, which
> >> predates the very existence of s_lock.h as a separate file.  So I'd
> >> definitely advise Alexander to find a tarball from that era and look at
> >> how Ultrix was handled then.
> >> I dunno if we even have tarballs from that far back on-line ... I
> >> suppose another possibility is a date-based pull from the CVS server.
> 
> > What can we help with Alex?
> 
> After digging around in the old code I have to retract my opinion that
> a test-and-set implementation used to exist for MIPS.  The code did
> have SysV-semaphore-based support for machines without test-and-set,
> and undoubtedly that's what was used on the old Ultrix port.  (The
> non-test-and-set code was broken for awhile, but I'd forgotten that
> it formerly worked.)
> 
> The non-test-and-set case should work again in current CVS, and I'd
> appreciate it if Alexander would verify that.  But as far as getting
> some test-and-set support for MIPS goes, it looks like the only way
> is for someone to sit down with a MIPS assembly manual.  I haven't
> got one, nor access to a machine to test on...

I've got access to an Indigoï¿½ (IRIX 6.5, MIPS R10000), another Indigoï¿½
(IRIX 6.2, MIPS R4400) and a DECStation (NetBSD 1.?, MIPS R3000). The
DECStation (also known as PMAX) originally ran Ultrix. If anybody has
some code that needs testing, I'd be more than willing. However, if
test-and-set works anything like I imagine, we really need to test it
on a multi-cpu MIPS machine. A good starting point might be the
test-and-set code in the NetBSD and Linux MIPS kernels.

Btw. Everything you never wanted to know about the MIPS architecture:
  http://www.mips.com/Documentation/

Cheers,

Mathijs
-- 
"A book is a fragile creature.  It suffers the wear of time,
 it fears rodents, the elements, clumsy hands." 
        Umberto Eco 

Tue, 27 Mar 2001 02:05:34 +0200
Mathijs Brands <mathijs@ilse.nl>
Re: Re: Call for platforms
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> That is not already available from the Irix support code?

What we have for IRIX is

#if defined(__sgi)
/*
 * SGI IRIX 5
 * slock_t is defined as a unsigned long. We use the standard SGI
 * mutex API.
 *
 * The following comment is left for historical reasons, but is probably
 * not a good idea since the mutex ABI is supported.
 *
 * This stuff may be supplemented in the future with Masato Kataoka's MIPS-II
 * assembly from his NECEWS SVR4 port, but we probably ought to retain this
 * for the R3000 chips out there.
 */
#include "mutex.h"
#define TAS(lock)	(test_and_set(lock,1))
#define S_UNLOCK(lock)	(test_then_and(lock,0))
#define S_INIT_LOCK(lock)	(test_then_and(lock,0))
#define S_LOCK_FREE(lock)	(test_then_add(lock,0) == 0)
#endif	 /* __sgi */

Doesn't look to me like it's likely to work on anything but IRIX ...

			regards, tom lane

Mon, 26 Mar 2001 19:09:38 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> "PPC750"?  What's that?  "PPC G3" might be more likely to mean something
>> to onlookers ...

> Actually "G3" means nothing outside of Apple afaict. The 750 series is a
> follow-on to the 60x series, and there is a 7xxx series also. From my
> pov, using an accepted label, rather than a marketing (re)label, better
> indicates *what* this actually can run on. I'm not sure that I have it
> labeled correctly yet, but "G3" is not a step in the right direction.

I found an apparently current "PowerPC CPU Summary" at
http://e-www.motorola.com/webapp/sps/technology/tech_tutorial.jsp?catId=M943030621280

If accurate, the chip in this PowerBook is *not* a 750, since that tops
out at 400 MHz.  Apple offered this model in 400 and 500 MHz speeds,
which makes it either a 7400 or 7410 chip ...

> Should I put "Mac G3" in the comment section?

Yes, if you won't put it where it should be ;-).  I'm still of the
opinion that "G3" will mean something to a vastly larger population
than "750" or "7400" would.  The latter are "marketing relabels" too
you know; Motorola's internal designation would probably be something
else entirely.

			regards, tom lane

Mon, 26 Mar 2001 19:53:44 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
On Mon, Mar 26, 2001 at 07:09:38PM -0500, Tom Lane allegedly wrote:
> Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> > That is not already available from the Irix support code?
> 
> What we have for IRIX is
> 
> #if defined(__sgi)
> /*
>  * SGI IRIX 5
>  * slock_t is defined as a unsigned long. We use the standard SGI
>  * mutex API.
>  *
>  * The following comment is left for historical reasons, but is probably
>  * not a good idea since the mutex ABI is supported.
>  *
>  * This stuff may be supplemented in the future with Masato Kataoka's MIPS-II
>  * assembly from his NECEWS SVR4 port, but we probably ought to retain this
>  * for the R3000 chips out there.
>  */
> #include "mutex.h"
> #define TAS(lock)	(test_and_set(lock,1))
> #define S_UNLOCK(lock)	(test_then_and(lock,0))
> #define S_INIT_LOCK(lock)	(test_then_and(lock,0))
> #define S_LOCK_FREE(lock)	(test_then_add(lock,0) == 0)
> #endif	 /* __sgi */
> 
> Doesn't look to me like it's likely to work on anything but IRIX ...
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

I just tried to compile 7.1RC1 on my IRIX 6.5 box using gcc 2.95.2.
Appearently gcc chokes on some assembly in src/backend/storage/buffer/s_lock.c
(tas_dummy on line 235 to be precise).

Here's the offending code:

#if defined(__mips__)
static void
tas_dummy()
{
        __asm__         _volatile__(
                                                                                "\
.global tas                                             \n\
tas:                                                    \n\
                        .frame  $sp, 0, $31     \n\
                        ll              $14, 0($4)      \n\
                        or              $15, $14, 1     \n\
                        sc              $15, 0($4)      \n\
                        beq             $15, 0, fail\n\
                        bne             $14, 0, fail\n\
                        li              $2, 0           \n\
                        .livereg 0x2000FF0E,0x00000FFF  \n\
                        j               $31                     \n\
fail:                                                   \n\
                        li              $2, 1           \n\
                        j       $31                     \n\
");
}

Notice the single underscore before volatile. I just checked the CVS
version of s_lock.c and this is still not fixed. Fixing this causes
as (the SGI version, not GNU as) to choke on the '.global tas' statement.

s_lock.c: At top level:
s_lock.c:234: warning: `tas_dummy' defined but not used
as: Error: /var/tmp/ccoUdrOb.s, line 421: undefined assembler operation: .global
      .global tas
gmake[4]: *** [s_lock.o] Error 1

Commenting out the .global statements does produce a binary, but it can't
complete the regression test due to other problems.

IpcSemaphoreCreate: semctl(id=0, 0, SETALL, ...) failed: Bad address

I'll see if I can come up with a solution for the .global and the
semaphore problem. I'll check wether pgsql 7.0 does run on this box too.
One wonders how Robert Bruccoleri did get 7.1RC1 to work properly. I'll
check the archive for clues.

On my FreeBSD 4.2 box 7.1RC1 runs flawlessly. I've also tested the CVS
version a few days ago on a 4.1.1 box without any problems.

FreeBSD 3.3 however does have some problems.

*** ./expected/float8-small-is-zero.out Fri Mar 31 07:30:31 2000
--- ./results/float8.out  Tue Mar 27 02:28:07 2001
***************
*** 214,220 ****
     SET f1 = FLOAT8_TBL.f1 * '-1'
     WHERE FLOAT8_TBL.f1 > '0.0';
  SELECT '' AS bad, f.f1 * '1e200' from FLOAT8_TBL f;
! ERROR:  Bad float8 input format -- overflow
  SELECT '' AS bad, f.f1 ^ '1e200' from FLOAT8_TBL f;
  ERROR:  pow() result is out of range
  SELECT '' AS bad, ln(f.f1) from FLOAT8_TBL f where f.f1 = '0.0' ;
--- 214,220 ----
     SET f1 = FLOAT8_TBL.f1 * '-1'
     WHERE FLOAT8_TBL.f1 > '0.0';
  SELECT '' AS bad, f.f1 * '1e200' from FLOAT8_TBL f;
! ERROR:  floating point exception! The last floating point operation either exceeded legal ranges or was a divide by zero
  SELECT '' AS bad, f.f1 ^ '1e200' from FLOAT8_TBL f;
  ERROR:  pow() result is out of range
  SELECT '' AS bad, ln(f.f1) from FLOAT8_TBL f where f.f1 = '0.0' ;

Some geometry tests also fail. I'll check those tomorrow, erm, today. The
same goes for 7.1RC1 on Solaris 8 (Intel and Sparc).

Cheers,

Mathijs
-- 
It's not that perl programmers are idiots, it's that the language
rewards idiotic behavior in a way that no other language or tool has
ever done.
                                                    Erik Naggum

Tue, 27 Mar 2001 03:04:50 +0200
Mathijs Brands <mathijs@ilse.nl>
Regression test on FBSD 3.3 & 4.2,
	IRIX 6.5 (was Re: Re: Call for platforms)
On Tue, Mar 27, 2001 at 12:01:24PM +1000, Justin Clift allegedly wrote:
> I know that Sourceforge has been adding all sorts of machines to their
> compile farm.
> 
> Maybe it would be worthwhile taking a look if they have platforms we
> don't?
> 
> Regards and best wishes,
> 
> Justin Clift

Compaq also still hands out free test accounts on Digital servers
running Linux, Tru64 and FreeBSD... I think it's called the Testdrive
program.

Cheers,

Mathijs
-- 
It's not that perl programmers are idiots, it's that the language
rewards idiotic behavior in a way that no other language or tool has
ever done.
                                                    Erik Naggum

Tue, 27 Mar 2001 03:08:11 +0200
Mathijs Brands <mathijs@ilse.nl>
Re: Re: Call for platforms
At 7:53 PM -0500 3/26/01, Tom Lane wrote:
>Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>>> "PPC750"?  What's that?  "PPC G3" might be more likely to mean something
>>> to onlookers ...
>
>> Actually "G3" means nothing outside of Apple afaict. The 750 series is a
>> follow-on to the 60x series, and there is a 7xxx series also. From my
>> pov, using an accepted label, rather than a marketing (re)label, better
>> indicates *what* this actually can run on. I'm not sure that I have it
>> labeled correctly yet, but "G3" is not a step in the right direction.
>
>I found an apparently current "PowerPC CPU Summary" at
>http://e-www.motorola.com/webapp/sps/technology/tech_tutorial.jsp?catId=M943030621280
>
>If accurate, the chip in this PowerBook is *not* a 750, since that tops
>out at 400 MHz.  Apple offered this model in 400 and 500 MHz speeds,
>which makes it either a 7400 or 7410 chip ...
>
>> Should I put "Mac G3" in the comment section?
>
>Yes, if you won't put it where it should be ;-).  I'm still of the
>opinion that "G3" will mean something to a vastly larger population
>than "750" or "7400" would.  The latter are "marketing relabels" too
>you know; Motorola's internal designation would probably be something
>else entirely.


A "Me Too" from the peanut gallery.

There are probably 1000x as many users that will recognize that they have a PowerPC G3 than will know they have a PPC750 or PPC7400.

-pmb



Mon, 26 Mar 2001 17:08:38 -0800
Peter Bierman <bierman@apple.com>
Re: Re: Call for platforms
Mathijs Brands <mathijs@ilse.nl> writes:
> Notice the single underscore before volatile.

That's definitely wrong --- fixed.

> Fixing this causes
> as (the SGI version, not GNU as) to choke on the '.global tas' statement.

> s_lock.c: At top level:
> s_lock.c:234: warning: `tas_dummy' defined but not used
> as: Error: /var/tmp/ccoUdrOb.s, line 421: undefined assembler operation: .global
>       .global tas
> gmake[4]: *** [s_lock.o] Error 1

Perhaps it should be ".globl"?  That's another common spelling.

Do you know whether anyone uses the GNU assembler on this platform,
or is it always SGI's?  I'm wondering if we need two versions of the
assembly code ...


I had missed the fact that s_lock.c contains some MIPS code.  Anyone
have any idea what versions of the MIPS series this code runs on?

			regards, tom lane

Mon, 26 Mar 2001 20:20:08 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Regression test on FBSD 3.3 & 4.2,
	IRIX 6.5 (was Re: Re: Call for platforms)
> Do you know whether anyone uses the GNU assembler on this platform,
> or is it always SGI's?  I'm wondering if we need two versions of the
> assembly code ...

Sure. Both compilers are available, with SGI's, uh, unique approach, and
with GNU's well understood assembler.

> I had missed the fact that s_lock.c contains some MIPS code.  Anyone
> have any idea what versions of the MIPS series this code runs on?

There is a chance it is from the Ultrix days (very pre-1998 afaicr). Or
is it the *only* MIPS code in our tree? If so, then it probably supports
Tatsuo's dead Cobalt server box, which is fairly recent vintage.

                    - Thomas

Tue, 27 Mar 2001 01:35:01 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Regression test on FBSD 3.3 & 4.2,
 IRIX 6.5 (was Re: Re: Call for
	platforms)
On Mon, Mar 26, 2001 at 07:09:38PM -0500, Tom Lane wrote:
> Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> > That is not already available from the Irix support code?
> 
> What we have for IRIX is
> ... 
> Doesn't look to me like it's likely to work on anything but IRIX ...

I have attached linuxthreads/sysdeps/mips/pt-machine.h from glibc-2.2.2
below.  (Glibc linuxthreads has alpha, arm, hppa, i386, ia64, m68k, mips,
powerpc, s390, SH, and SPARC support, at least in some degree.)

Since the actual instruction sequence is probably lifted from the 
MIPS manual, it's probably much freer than GPL.  For the paranoid,
the actual instructions, extracted, are just

   1:
     ll   %0,%3
     bnez %0,2f
      li  %1,1
     sc   %1,%2
     beqz %1,1b
   2:

Nathan Myers
ncm@zembu.com

-----------------------------------
/* Machine-dependent pthreads configuration and inline functions.

   Copyright (C) 1996, 1997, 1998, 2000 Free Software Foundation, Inc.
   This file is part of the GNU C Library.
   Contributed by Ralf Baechle <ralf@gnu.org>.
   Based on the Alpha version by Richard Henderson <rth@tamu.edu>.

   The GNU C Library is free software; you can redistribute it and/or
   modify it under the terms of the GNU Library General Public License as
   published by the Free Software Foundation; either version 2 of the
   License, or (at your option) any later version.

   The GNU C Library is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
   Library General Public License for more details.

   You should have received a copy of the GNU Library General Public
   License along with the GNU C Library; see the file COPYING.LIB.  If
   not, write to the Free Software Foundation, Inc.,
   59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.  */

#include <sgidefs.h>
#include <sys/tas.h>

#ifndef PT_EI
# define PT_EI extern inline
#endif

/* Memory barrier.  */
#define MEMORY_BARRIER() __asm__ ("" : : : "memory")


/* Spinlock implementation; required.  */

#if (_MIPS_ISA >= _MIPS_ISA_MIPS2)

PT_EI long int
testandset (int *spinlock)
{
  long int ret, temp;

  __asm__ __volatile__
    ("/* Inline spinlock test & set */\n\t"
     "1:\n\t"
     "ll	%0,%3\n\t"
     ".set	push\n\t"
     ".set	noreorder\n\t"
     "bnez	%0,2f\n\t"
     " li	%1,1\n\t"
     ".set	pop\n\t"
     "sc	%1,%2\n\t"
     "beqz	%1,1b\n"
     "2:\n\t"
     "/* End spinlock test & set */"
     : "=&r" (ret), "=&r" (temp), "=m" (*spinlock)
     : "m" (*spinlock)
     : "memory");

  return ret;
}

#else /* !(_MIPS_ISA >= _MIPS_ISA_MIPS2) */

PT_EI long int
testandset (int *spinlock)
{
  return _test_and_set (spinlock, 1);
}
#endif /* !(_MIPS_ISA >= _MIPS_ISA_MIPS2) */


/* Get some notion of the current stack.  Need not be exactly the top
   of the stack, just something somewhere in the current frame.  */
#define CURRENT_STACK_FRAME  stack_pointer
register char * stack_pointer __asm__ ("$29");


/* Compare-and-swap for semaphores. */

#if (_MIPS_ISA >= _MIPS_ISA_MIPS2)

#define HAS_COMPARE_AND_SWAP
PT_EI int
__compare_and_swap (long int *p, long int oldval, long int newval)
{
  long int ret;

  __asm__ __volatile__
    ("/* Inline compare & swap */\n\t"
     "1:\n\t"
     "ll	%0,%4\n\t"
     ".set	push\n"
     ".set	noreorder\n\t"
     "bne	%0,%2,2f\n\t"
     " move	%0,%3\n\t"
     ".set	pop\n\t"
     "sc	%0,%1\n\t"
     "beqz	%0,1b\n"
     "2:\n\t"
     "/* End compare & swap */"
     : "=&r" (ret), "=m" (*p)
     : "r" (oldval), "r" (newval), "m" (*p)
     : "memory");

  return ret;
}

#endif /* (_MIPS_ISA >= _MIPS_ISA_MIPS2) */

Mon, 26 Mar 2001 17:41:34 -0800
ncm@zembu.com (Nathan Myers)
Re: MIPS test-and-set
Hi,

I tested Solaris 8 SPARC (32 bit) over the weekend, and can test Solaris
8 INTEL this week/weekend.

The results of Solaris 8 SPARC were in Vince's database last time I
checked.  ???

+ Justin

Mathijs Brands wrote:
> 
> Hi
> 
> Is there a list somewhere listing the platforms 7.1 is being
> tested on right now? I'd be able to run regression tests on
> the following platforms, if necessary:
> 
>   FreeBSD 3.3 (x86)
>   FreeBSD 4.2 (x86)
>   Linux (x86 - 2.2 & 2.4 kernels, Redhat & Debian distro's)
>   Solaris 7 (SPARC)
>   Solaris 8 (x86)
>   Solaris 8 (SPARC)
>   IRIX 6.2
>   IRIX 6.5
> 
> If I can get the box back to working order, Alpha Linux is
> also an option. I'd be willing to build binary packages for
> Solaris and IRIX.
> 
> Regards,
> 
> Mathijs
> --
> "Books constitute capital."
>      Thomas Jefferson
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly

Tue, 27 Mar 2001 11:59:36 +1000
Justin Clift <aa2@bigpond.net.au>
Re: Re: Call for platforms
I know that Sourceforge has been adding all sorts of machines to their
compile farm.

Maybe it would be worthwhile taking a look if they have platforms we
don't?

Regards and best wishes,

Justin Clift

Thomas Lockhart wrote:
> 
> > The non-test-and-set case should work again in current CVS, and I'd
> > appreciate it if Alexander would verify that.  But as far as getting
> > some test-and-set support for MIPS goes, it looks like the only way
> > is for someone to sit down with a MIPS assembly manual.  I haven't
> > got one, nor access to a machine to test on...
> 
> That is not already available from the Irix support code?
> 
>                       - Thomas
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

Tue, 27 Mar 2001 12:01:24 +1000
Justin Clift <aa2@bigpond.net.au>
Re: Re: Call for platforms
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:

> NetBSD Sparc       7.0 2000-04-13, Tom I. Helbekkmo

Fetching the latest source kit now -- hope to have regression tests
run and a report back to you within a day or two.

> We need some NetBSD folks to speak up!

I've once again got a VAX that should be able to run PostgreSQL on
NetBSD/vax, so I hope to be able to help revitalize that port soon...

-tih
-- 
The basic difference is this: hackers build things, crackers break them.

27 Mar 2001 04:38:50 +0200
Tom Ivar Helbekkmo <tih@kpnQwest.no>
Re: Call for platforms
One that didn't compilei RC1:

 BIGBOY 71# uname -a
IRIX BIGBOY 6.5 05190003 IP22

On an Indigo2 (R4000), gcc 2.95.2 , with the following error:

gcc  -Wall -Wmissing-prototypes -Wmissing-declarations
-I../../../../src/include  -U_NO_XOPEN4  -c s_lock.c -o s_lock.o
s_lock.c: In function `s_lock':
s_lock.c:134: warning: passing arg 1 of pointer to function discards
qualifiers from pointer target type
s_lock.c: In function `tas_dummy':
s_lock.c:235: parse error before `_volatile__'
s_lock.c: At top level:
s_lock.c:234: warning: `tas_dummy' defined but not used
gmake[4]: *** [s_lock.o] Error 1
gmake[4]: Leaving directory
`/usr/people/telmnstr/pg/postgresql-7.1RC1/src/backend/storage/buffer'
gmake[3]: *** [buffer-recursive] Error 2
gmake[3]: Leaving directory
`/usr/people/telmnstr/pg/postgresql-7.1RC1/src/backend/storage'
gmake[2]: *** [storage-recursive] Error 2
gmake[2]: Leaving directory
`/usr/people/telmnstr/pg/postgresql-7.1RC1/src/backend'
gmake[1]: *** [all] Error 2
gmake[1]: Leaving directory
`/usr/people/telmnstr/pg/postgresql-7.1RC1/src'
gmake: *** [all] Error 2
*** Error code 2 (bu21)

Jeff



Mon, 26 Mar 2001 22:01:19 -0500 (EST)
Jeff Duffy <jduffy@greatbridge.com>
Re: Call for platforms
-----BEGIN PGP SIGNED MESSAGE-----

On 25 Mar 2001, at 16:07, Tom Lane wrote:

> Does that database have any user-created relations in it, or is it
> just a virgin database?  It seems that the wrong attlen is being
> computed for ctid fields during bootstrap, but the regression test
> output (if it was complete) implies that the value inserted for
> user-created fields was OK.  This doesn't make a lot of sense since
> it's the same code...

Totally virgin. I created it just for that select you wanted. The 
7.1beta6 I built was installed in /usr/pgsql so as to be entirely 
separate from any other running parts of the system. Like I said, the 
test failed, but it seems to *work* just fine...

If you want the complete regress output, I'll send it as well. The 
only failures were the type_sanity and geometry though, and the 
geometry was just fluctuations on the final digit of a few numbers.

I suspect it might be an alignment problem (ARM needs word or dword 
alignment on data access.. our kernel has an alignment trap handler 
that does fixups in 'broken' code) or something related to signedness 
(ARM has default signed char) but I don't know enough about postgres 
internals to really debug it. However, I'm certainly willing to 
learn.. :) 


-----BEGIN PGP SIGNATURE-----
Version: N/A

iQCVAwUBOsAFXf+IdJuhyV9xAQFpZgP8C7g9dqlh9Qd/wVwJn2jquVh+X3gBWBZ5
UMHx43tPfYE7xJvHl3XH/z+mg/POyzgFMCF+5USO2jzbPMDiS2OtJbp+1NvP2FHA
uuY1ra5o8WKWW/7ZrfaO5edC5e1OsKbhGsXugRIyBwFkzz28blt6gongUdio0nC3
Td8Fm3GUKNk=
=+//W
-----END PGP SIGNATURE-----

Mon, 26 Mar 2001 22:13:33 -0500
"Mark Knox" <segfault@hardline.org>
Re: Re: Call for platforms 
"Mark Knox" <segfault@hardline.org> writes:
> On 25 Mar 2001, at 16:07, Tom Lane wrote:
>> Does that database have any user-created relations in it, or is it
>> just a virgin database?

> Totally virgin. I created it just for that select you wanted.

Okay.   Would you create a couple of random tables in it and do the
select again?  I want to see what ctid looks like in a user-created
table.

> I suspect it might be an alignment problem

Sort of.  I am suspicious that sizeof(ItemPointerData) is returning 8
rather than 6 as one might expect.

			regards, tom lane

Mon, 26 Mar 2001 23:14:42 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
ncm@zembu.com (Nathan Myers) writes:

> Since the actual instruction sequence is probably lifted from the 
> MIPS manual, it's probably much freer than GPL.  For the paranoid,
> the actual instructions, extracted, are just
> 
>    1:
>      ll   %0,%3
>      bnez %0,2f
>       li  %1,1
>      sc   %1,%2
>      beqz %1,1b
>    2:

But note that the ll instruction is MIPS ISA II, which means that it
is not supported by the R3000, which means that it will not work on
most DECstations.

I don't think there is any way to do a reliable test-and-set sequence
in user mode on an R3000.

Ian

26 Mar 2001 21:18:43 -0800
Ian Lance Taylor <ian@airs.com>
Re: MIPS test-and-set
> mklinux PPC750     7.0 2000-04-13, Tatsuo Ishii
> 
> Any luck with RC1?

I will try today or tomorrow...
--
Tatsuo Ishii

Tue, 27 Mar 2001 17:57:24 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Call for platforms

On Tue, 27 Mar 2001 09:57:45 -0500 (EST), Bruce Momjian alluded:

> 
>  We just fixed that yesterday.  Can you grab the most recent CVS and give
>  it a try?

 Yep.  We have many other MIPS (ONYX Crimson, , ONYX, Challenge, Indy w/ IRIX
6.2, 6.5, etc.), Alpha and Sparc platforms if there are some others that need
testing (How about NetBSD on NeXT?).

Jeff

-- 
 Jeff Duffy
 jeff@alanne.com


27 Mar 2001 10:42:59 EST
"Local" <jeff@alanne.com>
Re: Re: Call for platforms
Thus spake Tom Ivar Helbekkmo
> > We need some NetBSD folks to speak up!

I have successfully compiled it from CVS sources on my NetBSD -current but
I can't find the tar file for RC1 to try it with the package system.  Can
someone point me to it please.

-- 
D'Arcy J.M. Cain <darcy@{druid|vex}.net>   |  Democracy is three wolves
http://www.druid.net/darcy/                |  and a sheep voting on
+1 416 425 1212     (DoD#0082)    (eNTP)   |  what's for dinner.

Tue, 27 Mar 2001 06:36:37 -0500 (EST)
darcy@druid.net (D'Arcy J.M. Cain)
Re: Re: Call for platforms
On Tue, Mar 27, 2001 at 06:36:37AM -0500, D'Arcy J.M. Cain allegedly wrote:
> Thus spake Tom Ivar Helbekkmo
> > > We need some NetBSD folks to speak up!
> 
> I have successfully compiled it from CVS sources on my NetBSD -current but
> I can't find the tar file for RC1 to try it with the package system.  Can
> someone point me to it please.

It's probably in /pub/dev (or something similar) on the ftp
server...

Mathijs
-- 
It's not that perl programmers are idiots, it's that the language
rewards idiotic behavior in a way that no other language or tool has
ever done.
                                                    Erik Naggum

Tue, 27 Mar 2001 13:56:02 +0200
Mathijs Brands <mathijs@ilse.nl>
Re: Re: Call for platforms

We just fixed that yesterday.  Can you grab the most recent CVS and give
it a try?

> One that didn't compilei RC1:
> 
>  BIGBOY 71# uname -a
> IRIX BIGBOY 6.5 05190003 IP22
> 
> On an Indigo2 (R4000), gcc 2.95.2 , with the following error:
> 
> gcc  -Wall -Wmissing-prototypes -Wmissing-declarations
> -I../../../../src/include  -U_NO_XOPEN4  -c s_lock.c -o s_lock.o
> s_lock.c: In function `s_lock':
> s_lock.c:134: warning: passing arg 1 of pointer to function discards
> qualifiers from pointer target type
> s_lock.c: In function `tas_dummy':
> s_lock.c:235: parse error before `_volatile__'
> s_lock.c: At top level:
> s_lock.c:234: warning: `tas_dummy' defined but not used
> gmake[4]: *** [s_lock.o] Error 1
> gmake[4]: Leaving directory
> `/usr/people/telmnstr/pg/postgresql-7.1RC1/src/backend/storage/buffer'
> gmake[3]: *** [buffer-recursive] Error 2
> gmake[3]: Leaving directory
> `/usr/people/telmnstr/pg/postgresql-7.1RC1/src/backend/storage'
> gmake[2]: *** [storage-recursive] Error 2
> gmake[2]: Leaving directory
> `/usr/people/telmnstr/pg/postgresql-7.1RC1/src/backend'
> gmake[1]: *** [all] Error 2
> gmake[1]: Leaving directory
> `/usr/people/telmnstr/pg/postgresql-7.1RC1/src'
> gmake: *** [all] Error 2
> *** Error code 2 (bu21)
> 
> Jeff
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 27 Mar 2001 09:57:45 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: Call for platforms
Jeff Duffy <jduffy@greatbridge.com> writes:
> s_lock.c:235: parse error before `_volatile__'

That typo is fixed in current sources (should be OK in last night's
snapshot) but there's still some doubt as to how well the MIPS assembly
code works ...

			regards, tom lane

Tue, 27 Mar 2001 10:36:26 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Mathijs Brands writes:

> I just tried to compile 7.1RC1 on my IRIX 6.5 box using gcc 2.95.2.

According to the information at
http://freeware.sgi.com/shared/howto.html#b1 it probably won't work to
compile PostgreSQL with GCC on Irix.  Or it might work and crash when run.
Be warned.  (I think it is not accidental that no one ever successfully
used a PostgreSQL/GCC/Irix combo.)

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Tue, 27 Mar 2001 17:42:03 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Regression test on FBSD 3.3 & 4.2, IRIX 6.5 (was Re:
	Re: Call for platforms)
On Tue, Mar 27, 2001 at 09:57:45AM -0500, Bruce Momjian allegedly wrote:
> We just fixed that yesterday.  Can you grab the most recent CVS and give
> it a try?

Even if you fix this it won't work (I tried it). Robert mailed why. Check the URL below
for more information. It crashes on semctl :(

http://freeware.sgi.com/shared/howto.html#b1

Cheers,

Mathijs
-- 
It's not that perl programmers are idiots, it's that the language
rewards idiotic behavior in a way that no other language or tool has
ever done.
                                                    Erik Naggum

Tue, 27 Mar 2001 18:06:17 +0200
Mathijs Brands <mathijs@ilse.nl>
Re: Re: Call for platforms
I wrote:

> > NetBSD Sparc       7.0 2000-04-13, Tom I. Helbekkmo
> 
> Fetching the latest source kit now -- hope to have regression tests
> run and a report back to you within a day or two.

Hmm.  No go here: everything looks peachy until I've started the
postmaster, and attempt to connect to it:

barsoom:postgres> psql template1
/usr/local/pgsql/lib/libpq.so.2: Undefined symbol "" (reloc type = 12, symnum = 4)
barsoom:postgres> _

I've never seen this happen before...  (For what it's worth, I use
Kerberos IV authentication here, so that's what I've configured on
this box as well.  I notice that psql does not get as far as aquiring
a service key for the database access.)

Any quick hints?

-tih
-- 
The basic difference is this: hackers build things, crackers break them.

27 Mar 2001 22:27:15 +0200
Tom Ivar Helbekkmo <tih@kpnQwest.no>
Re: Call for platforms
Mathijs Brands <mathijs@ilse.nl> writes:
> Even if you fix this it won't work (I tried it). Robert mailed
> why. Check the URL below for more information. It crashes on semctl :(

> http://freeware.sgi.com/shared/howto.html#b1

Ugh.  Given the semctl compatibility problem, I suspect we'd better note
in the platform list that IRIX is only supported for cc, not gcc.

The other uncomfy-looking thing on that page is the very first item,
about configure scripts picking up libraries that they'd best not.
(I have seen similar issues on HPUX, although they were relatively
easy to get around.)  We might need to do some more hacking on our
configure script to make it play nice on IRIX.

			regards, tom lane

Tue, 27 Mar 2001 15:44:08 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Tom Ivar Helbekkmo <tih@kpnqwest.no> wrote:
> Thomas Lockhart <lockhart@alumni.caltech.edu> writes:

>> NetBSD Sparc       7.0 2000-04-13, Tom I. Helbekkmo

> Fetching the latest source kit now -- hope to have regression tests
> run and a report back to you within a day or two.

>> We need some NetBSD folks to speak up!

> I've once again got a VAX that should be able to run PostgreSQL on
> NetBSD/vax, so I hope to be able to help revitalize that port soon...

it might also be a good idea to ask on the NetBSD ports lists - i
think there will most probably some people trying things out - the
name of the list is

  port-arch@NetBSD.org

where arch is the corresponding NetBSD port name (pmax, macppc, sparc,
i386, arm32, ...)

this might also be a good idea for the mips test-and-set thing (on
the port-pmax list - there are a lot of people knowing all that
stuff very well)

also it might be worth to eventually ask on the alpha@FreeBSD.org
list for someone willing to play with PostgreSQL on FreeBSD/alpha

just some ideas ...

t

-- 
thomas graichen <tgr@spoiled.org> ... perfection is reached, not
when there is no longer anything to add, but when there is no
longer anything to take away. --- antoine de saint-exupery

Tue, 27 Mar 2001 23:00:40 +0200
thomas graichen <list-pgsql.hackers@spoiled.org>
Re: Re: Call for platforms
> > mklinux PPC750     7.0 2000-04-13, Tatsuo Ishii
> > 
> > Any luck with RC1?
> 
> I will try today or tomorrow...

In summary no, improvemnets seen.

If compiled with -O2 or -O2 -g, I got 10 tests FAILED. misc test
failed due to a backend crash. The SQL caused the crash was:

select i, length(t), octet_length(t), oldstyle_length(i,t) from
oldstyle_test;

#0  ExecReplace (slot=0x1a4a7d0, tupleid=0x0, estate=0x1a4a708)
    at execMain.c:1408
1408            resultRelationDesc = resultRelInfo->ri_RelationDesc;
(gdb) where
#0  ExecReplace (slot=0x1a4a7d0, tupleid=0x0, estate=0x1a4a708)
    at execMain.c:1408
#1  0x188471c in ExecutePlan (estate=0x0, plan=0x1a4a410, 
    operation=CMD_SELECT, numberTuples=0, direction=27567836, 
    destfunc=0x1a4adf8) at execMain.c:1127
#2  0x188471c in ExecutePlan (estate=0x0, plan=0x1a4a410, 
    operation=CMD_SELECT, numberTuples=0, direction=27567836, 
    destfunc=0x1a4adf8) at execMain.c:1127
#3  0x18838b8 in ExecutorRun (queryDesc=0x1a4a7d0, estate=0x1a4a708, 
    feature=27567784, count=0) at execMain.c:233
#4  0x18e7784 in ProcessQuery (parsetree=0x1a4a708, plan=0x1a4a6a8, dest=None)
    at pquery.c:295
#5  0x18e5c38 in pg_exec_query_string (query_string=0x1a4a410 "", dest=None, 
    parse_context=0x1) at postgres.c:806
#6  0x18e70b8 in PostgresMain (argc=1, argv=0x0, real_argc=4, real_argv=0x0, 
    username=0x0) at postgres.c:1902
#7  0x18c92ec in DoBackend (port=0x1a4a6a8) at postmaster.c:2111
#8  0x18c8e10 in BackendStartup (port=0x1a4a708) at postmaster.c:1894
#9  0x18c7c08 in ServerLoop () at postmaster.c:992
#10 0x18c74f4 in PostmasterMain (argc=0, argv=0x1a4a6a8) at postmaster.c:682
#11 0x1899a5c in main (argc=27567784, argv=0x1a4a708) at main.c:147
#12 0x181c400 in _start ()
(gdb) 

Wed, 28 Mar 2001 10:39:51 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Re: Call for platforms
-----BEGIN PGP SIGNED MESSAGE-----

On 26 Mar 2001, at 23:14, Tom Lane wrote:

> "Mark Knox" <segfault@hardline.org> writes:
> > On 25 Mar 2001, at 16:07, Tom Lane wrote:
> >> Does that database have any user-created relations in it, or is it
> >> just a virgin database?
> 
> > Totally virgin. I created it just for that select you wanted.
> 
> Okay.   Would you create a couple of random tables in it and do the
> select again?  I want to see what ctid looks like in a user-created
> table.

Sure. I created two tables called 'test1' and 'test2'. Test1 has a 
single field 'field1' of type int4. Test2 has two fields 'field1' and 
'field2' of types char(200) and int4 respectively. 

Here are the results:

postgres=> select 
p1.oid,attrelid,relname,attname,attlen,attalign,attbyval from 
pg_attribute p1, pg_class p2 where atttypid = 27 and p2.oid = 
attrelid order by 1;
  oid|attrelid|relname       |attname|attlen|attalign|attbyval
- -----+--------+--------------+-------+------+--------+--------
16401|    1247|pg_type       |ctid   |     6|i       |f       
16415|    1262|pg_database   |ctid   |     6|i       |f       
16439|    1255|pg_proc       |ctid   |     6|i       |f       
16454|    1260|pg_shadow     |ctid   |     6|i       |f       
16464|    1261|pg_group      |ctid   |     6|i       |f       
16486|    1249|pg_attribute  |ctid   |     6|i       |f       
16515|    1259|pg_class      |ctid   |     6|i       |f       
16526|    1215|pg_attrdef    |ctid   |     6|i       |f       
16537|    1216|pg_relcheck   |ctid   |     6|i       |f       
16557|    1219|pg_trigger    |ctid   |     6|i       |f       
16572|   16567|pg_inherits   |ctid   |     8|i       |f       
16593|   16579|pg_index      |ctid   |     8|i       |f       
16610|   16600|pg_statistic  |ctid   |     8|i       |f       
16635|   16617|pg_operator   |ctid   |     8|i       |f       
16646|   16642|pg_opclass    |ctid   |     8|i       |f       
16678|   16653|pg_am         |ctid   |     8|i       |f       
16691|   16685|pg_amop       |ctid   |     8|i       |f       
16873|   16867|pg_amproc     |ctid   |     8|i       |f       
16941|   16934|pg_language   |ctid   |     8|i       |f       
16953|   16948|pg_largeobject|ctid   |     8|i       |f       
16970|   16960|pg_aggregate  |ctid   |     8|i       |f       
17038|   17033|pg_ipl        |ctid   |     8|i       |f       
17051|   17045|pg_inheritproc|ctid   |     8|i       |f       
17067|   17058|pg_rewrite    |ctid   |     8|i       |f       
17079|   17074|pg_listener   |ctid   |     8|i       |f       
17090|   17086|pg_description|ctid   |     8|i       |f       
17206|   17201|pg_toast_1215 |ctid   |     8|i       |f       
17221|   17216|pg_toast_17086|ctid   |     8|i       |f       
17236|   17231|pg_toast_1255 |ctid   |     8|i       |f       
17251|   17246|pg_toast_1216 |ctid   |     8|i       |f       
17266|   17261|pg_toast_17058|ctid   |     8|i       |f       
17281|   17276|pg_toast_16600|ctid   |     8|i       |f       
17301|   17291|pg_user       |ctid   |     8|i       |f       
17314|   17309|pg_rules      |ctid   |     8|i       |f       
17327|   17322|pg_views      |ctid   |     8|i       |f       
17342|   17335|pg_tables     |ctid   |     8|i       |f       
17355|   17350|pg_indexes    |ctid   |     8|i       |f       
18724|   18721|test1         |ctid   |     8|i       |f       
18735|   18731|test2         |ctid   |     8|i       |f       
(39 rows)

> > I suspect it might be an alignment problem
> 
> Sort of.  I am suspicious that sizeof(ItemPointerData) is returning 8
> rather than 6 as one might expect.

Maybe it's padding the structure to a dword boundary? ARM is 
notorious for such things.. I will rebuild it with 
__attribute__((packed)) on the struct and see if the size changes..


-----BEGIN PGP SIGNATURE-----
Version: N/A

iQCVAwUBOsFDJP+IdJuhyV9xAQEKxQP/YJXTxZppLd7ECk4BSwDZaStP4+bE6acc
StT//i/drdPC53DDWqiXLGA0bS384EXxyjvvaO1bTXzVFU/3+X/pY6YN/G3HMoah
dbCXRli2Y57yansf1WaVmK1lhiAqLy3iGYFp2nZvO1Sl1u+ba89HtV+G+iaKZSTr
U+HWTU3nnOM=
=vkY+
-----END PGP SIGNATURE-----

Tue, 27 Mar 2001 20:49:24 -0500
"Mark Knox" <segfault@hardline.org>
Re: Re: Call for platforms 
-----BEGIN PGP SIGNED MESSAGE-----

On 27 Mar 2001, at 20:49, Mark Knox wrote:

> > > I suspect it might be an alignment problem
> > 
> > Sort of.  I am suspicious that sizeof(ItemPointerData) is returning
> > 8 rather than 6 as one might expect.
> 
> Maybe it's padding the structure to a dword boundary? ARM is 
> notorious for such things.. I will rebuild it with 
> __attribute__((packed)) on the struct and see if the size changes..

Aha, progress! The packed directive gives attlen of 6 across the 
board! Type_sanity test passes now too, so the only failing 
regression test is geometry and that is easily dismissed. The 
variation is in the last decimal place and probably due to emulated 
floating point (ARM has no FPU).

The patch is attached.. it's tiny but seems to be effective.



-----BEGIN PGP SIGNATURE-----
Version: N/A

iQCVAwUBOsFeUv+IdJuhyV9xAQE2XAP9FF93ew+6Ml5iZ1jWjcGrs+3zaXIeWef6
SytNtIfyJqmcnyWnMaxBTlChIvBO5A2HVnBkCydM5BjUXdW1eWsEynrd+U79Yc+e
yVDGo30CK3lAkTLH3Fo6jR3YZe/TsIyr80WlDeqJiWvDmHTfqvo50jRiDq2h1OL/
LmI4YIQM0rQ=
=Vwwp
-----END PGP SIGNATURE-----

Tue, 27 Mar 2001 22:45:22 -0500
"Mark Knox" <segfault@hardline.org>
Re: Re: Call for platforms 
The following section of this message contains a file attachment
prepared for transmission using the Internet MIME message format.
If you are using Pegasus Mail, or any another MIME-compliant system,
you should be able to save it or view it from within your mailer.
If you cannot, please ask your system administrator for assistance.

   ---- File information -----------
     File:  arm-alignment.patch
     Date:  27 Mar 2001, 21:26
     Size:  533 bytes.
     Type:  Unknown
Tue, 27 Mar 2001 22:45:22 -0500
"Mark Knox" <segfault@hardline.org>
Re: Re: Call for platforms 
"Mark Knox" <segfault@hardline.org> writes:
>  {
>  	BlockIdData ip_blkid;
>  	OffsetNumber ip_posid;
> +#ifdef __arm__
> +} __attribute__((packed)) ItemPointerData;
> +#else
>  }
> +#endif

That would fix it for ARM but not for anyplace else with similar
alignment behavior.  Would you try this patch instead to see what
happens?

			regards, tom lane


*** src/backend/catalog/heap.c.orig	Thu Mar 22 09:50:36 2001
--- src/backend/catalog/heap.c	Wed Mar 28 00:24:45 2001
***************
*** 103,109 ****
   */
  
  static FormData_pg_attribute a1 = {
! 	0xffffffff, {"ctid"}, TIDOID, 0, sizeof(ItemPointerData),
  	SelfItemPointerAttributeNumber, 0, -1, -1, '\0', 'p', '\0', 'i', '\0', '\0'
  };
  
--- 103,109 ----
   */
  
  static FormData_pg_attribute a1 = {
! 	0xffffffff, {"ctid"}, TIDOID, 0, SizeOfIptrData,
  	SelfItemPointerAttributeNumber, 0, -1, -1, '\0', 'p', '\0', 'i', '\0', '\0'
  };
  

Wed, 28 Mar 2001 00:27:51 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> > mklinux PPC750     7.0 2000-04-13, Tatsuo Ishii

> If compiled with -O2 or -O2 -g, I got 10 tests FAILED. misc test
> failed due to a backend crash. The SQL caused the crash was:

> select i, length(t), octet_length(t), oldstyle_length(i,t) from
> oldstyle_test;

> #0  ExecReplace (slot=0x1a4a7d0, tupleid=0x0, estate=0x1a4a708)
>     at execMain.c:1408
> 1408            resultRelationDesc = resultRelInfo->ri_RelationDesc;
> (gdb) where
> #0  ExecReplace (slot=0x1a4a7d0, tupleid=0x0, estate=0x1a4a708)
>     at execMain.c:1408
> #1  0x188471c in ExecutePlan (estate=0x0, plan=0x1a4a410, 
>     operation=CMD_SELECT, numberTuples=0, direction=27567836, 
>     destfunc=0x1a4adf8) at execMain.c:1127
> #2  0x188471c in ExecutePlan (estate=0x0, plan=0x1a4a410, 
>     operation=CMD_SELECT, numberTuples=0, direction=27567836, 
>     destfunc=0x1a4adf8) at execMain.c:1127
> #3  0x18838b8 in ExecutorRun (queryDesc=0x1a4a7d0, estate=0x1a4a708, 
>     feature=27567784, count=0) at execMain.c:233

I think you've got a badly broken compiler there.  There's no way that
ExecReplace should be entered for a SELECT.  The backtrace is wrong on
its face anyway --- ExecutePlan does not call itself.

What gcc version does that platform have?

			regards, tom lane

Wed, 28 Mar 2001 00:51:55 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
> I think you've got a badly broken compiler there.  There's no way that
> ExecReplace should be entered for a SELECT.  The backtrace is wrong on
> its face anyway --- ExecutePlan does not call itself.

Yes, I have suspected that.

> What gcc version does that platform have?

gcc version egcs-2.90.25 980302 (egcs-1.0.2 prerelease)
--
Tatsuo Ishii

Wed, 28 Mar 2001 14:53:35 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Re: Call for platforms 
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
>> What gcc version does that platform have?

> gcc version egcs-2.90.25 980302 (egcs-1.0.2 prerelease)

Can you try a known-stable gcc version?  2.95.2 say?

			regards, tom lane

Wed, 28 Mar 2001 01:08:45 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> >> What gcc version does that platform have?
> 
> > gcc version egcs-2.90.25 980302 (egcs-1.0.2 prerelease)
> 
> Can you try a known-stable gcc version?  2.95.2 say?

I don't have time right know. Will do maybe for 7.1.1 or 7.2..
--
Tatsuo Ishii

Wed, 28 Mar 2001 15:11:59 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Re: Call for platforms 
I just had a discussion with a user who doesn't want to update from
6.4.something to 7.0.* because 7.0 broke a feature he likes, namely
the ability to change the default value of a column inherited from
a parent table.  It seems that in pre-7.0 Postgres, this works:

create table one(id int default 1, descr text);
create table two(id int default 2, tag text) inherits (one);

with the net effect that table "two" has just one "id" column with
default value 2.

I can recall a number of requests from users to be able to change
the default value when inheriting a column, but I had not realized
that it was actually possible to do this in older Postgres releases.

After digging into the CVS logs and mail archives, I find that Peter E.
changed the behavior in January 2000, apparently without realizing that
he was disabling a feature that some considered useful.  Here's his
comment in pghackers, 26 Jan 2000 19:35:14 +0100 (CET):

> ... I just looked into item 'Disallow inherited columns
> with the same name as new columns' and it seems that someone actually made
> provisions for this to be allowed, meaning that
> create table test1 (x int);
> create table test2 (x int) inherits (test1);
> would result in test2 looking exactly like test1. No one knows what the
> motivation was. (I removed it anyway.)

Given that Peter was responding to a TODO item, evidently someone had
complained about the lack of any complaint for this construction, but
I wonder whether the someone really understood all the implications.
Allowing this construction allows one to change the default, or add
(but not remove) column constraints, and in general it seems kinda
useful.

The question of the day: should we put this back the way it was?
If so, should we try to squeeze it into 7.1, or wait another release
cycle?  (I can see about equally good arguments for considering this
a feature addition or a bug fix...)  Should there be a NOTICE about
the duplicated column name, or is the old silent treatment okay?

			regards, tom lane

Wed, 28 Mar 2001 13:30:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Changing the default value of an inherited column
Tom Lane writes:

> It seems that in pre-7.0 Postgres, this works:
>
> create table one(id int default 1, descr text);
> create table two(id int default 2, tag text) inherits (one);
>
> with the net effect that table "two" has just one "id" column with
> default value 2.

Although the liberty to do anything you want seems appealing at first, I
would think that allowing this is not correct from an OO point of view.
But given that our inheritance system actually has conceivably little
resemblance to real OO, I don't really care.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Wed, 28 Mar 2001 23:15:12 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Changing the default value of an inherited column
At 12:27 AM 3/28/01 -0500, Tom Lane wrote:

>That would fix it for ARM but not for anyplace else with similar
>alignment behavior.  Would you try this patch instead to see what
>happens?

I don't think this solution would be valid on many other platforms. It forces the structure to not be padded, and assumes that the cpu will be able to fetch from unaligned boundaries. The only reason this works is that the arm linux kernel contains an alignment trap handler that catches the fault and does a fixup on the access. Otherwise it would crash with SIGBUS.

>  static FormData_pg_attribute a1 = {
>!       0xffffffff, {"ctid"}, TIDOID, 0, SizeOfIptrData,
>        SelfItemPointerAttributeNumber, 0, -1, -1, '\0', 'p', '\0', 'i', '\0', '\0'
>  }; 

Well, this patch seems to produce attlens of 6 as desired, but it causes many (13) of the regression tests to fail. Do you want to see the regression.diffs? 


Wed, 28 Mar 2001 21:16:32 -0500
Mark Knox <segfault@hardline.org>
Re: Re: Call for platforms 
Mark Knox <segfault@hardline.org> writes:
> I don't think this solution would be valid on many other platforms.

Au contraire --- the ARM is the first platform I've heard of that does
not think sizeof(ItemPointerData) is 6.  Else we'd have seen this
regress test fail before.

> Well, this patch seems to produce attlens of 6 as desired, but it
> causes many (13) of the regression tests to fail. Do you want to see
> the regression.diffs?

Please.

			regards, tom lane

Wed, 28 Mar 2001 23:06:42 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
Tom Lane wrote:
  >I just had a discussion with a user who doesn't want to update from
  >6.4.something to 7.0.* because 7.0 broke a feature he likes, namely
  >the ability to change the default value of a column inherited from
  >a parent table.  It seems that in pre-7.0 Postgres, this works:
  >
  >create table one(id int default 1, descr text);
  >create table two(id int default 2, tag text) inherits (one);
  >
  >with the net effect that table "two" has just one "id" column with
  >default value 2.
 ...
  >The question of the day: should we put this back the way it was?
  >If so, should we try to squeeze it into 7.1, or wait another release
  >cycle?  (I can see about equally good arguments for considering this
  >a feature addition or a bug fix...)  Should there be a NOTICE about
  >the duplicated column name, or is the old silent treatment okay?
 
I would very much like to have this feature restored; I think there should
be a NOTICE, just in case the duplication is caused by mistyping.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Trust in the Lord with all your heart and lean not on 
      your own understanding; in all your ways acknowledge  
      him, and he will direct your paths."  Proverbs 3:5,6  



Thu, 29 Mar 2001 14:27:03 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Changing the default value of an inherited column 
Peter Eisentraut wrote:
  >Tom Lane writes:
  >
  >> It seems that in pre-7.0 Postgres, this works:
  >>
  >> create table one(id int default 1, descr text);
  >> create table two(id int default 2, tag text) inherits (one);
  >>
  >> with the net effect that table "two" has just one "id" column with
  >> default value 2.
  >
  >Although the liberty to do anything you want seems appealing at first, I
  >would think that allowing this is not correct from an OO point of view.

I don't agree; this is equivalent to redefinition of a feature (=method) in
a descendant class, which is perfectly acceptable so long as the feature's
signature (equivalent to column type) remains unchanged.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Trust in the Lord with all your heart and lean not on 
      your own understanding; in all your ways acknowledge  
      him, and he will direct your paths."  Proverbs 3:5,6  



Thu, 29 Mar 2001 14:29:38 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Re: Changing the default value of an inherited column 
Oliver Elphick writes:

> Peter Eisentraut wrote:
>   >Tom Lane writes:
>   >
>   >> It seems that in pre-7.0 Postgres, this works:
>   >>
>   >> create table one(id int default 1, descr text);
>   >> create table two(id int default 2, tag text) inherits (one);
>   >>
>   >> with the net effect that table "two" has just one "id" column with
>   >> default value 2.
>   >
>   >Although the liberty to do anything you want seems appealing at first, I
>   >would think that allowing this is not correct from an OO point of view.
>
> I don't agree; this is equivalent to redefinition of a feature (=method) in
> a descendant class, which is perfectly acceptable so long as the feature's
> signature (equivalent to column type) remains unchanged.

The SQL equivalent of redefining a method would the redefinition of a
method [sic].  But since we don't have anything close to that, feel
free...

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 29 Mar 2001 18:53:15 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Changing the default value of an inherited column
"Oliver Elphick" <olly@lfix.co.uk> writes:
>>>> Although the liberty to do anything you want seems appealing at first, I
>>>> would think that allowing this is not correct from an OO point of view.

> I don't agree; this is equivalent to redefinition of a feature (=method) in
> a descendant class, which is perfectly acceptable so long as the feature's
> signature (equivalent to column type) remains unchanged.

Well, that does bring up the question of exactly what is signature and
exactly what is implementation.  Clearly we cannot allow the column type
to be redefined.  But what about typmod?  Is it OK to replace char(32)
with char(64)?  How about vice versa?  How about replacing numeric(9,0)
with numeric(7,2)?

The pre-7.0 code only checked that the type ID is the same, but I wonder
whether it wouldn't be a good idea to demand typmod the same as well.
For the existing types that use typmod I don't think this is absolutely
necessary (ie, I don't think the system might crash if typmods are
inconsistent in inherited tables) ... but I'm not comfortable about it
either.

			regards, tom lane

Thu, 29 Mar 2001 11:57:20 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
On Thu, Mar 29, 2001 at 02:29:38PM +0100, Oliver Elphick wrote:
> Peter Eisentraut wrote:
>   >Tom Lane writes:
>   >
>   >> It seems that in pre-7.0 Postgres, this works:
>   >>
>   >> create table one(id int default 1, descr text);
>   >> create table two(id int default 2, tag text) inherits (one);
>   >>
>   >> with the net effect that table "two" has just one "id" column with
>   >> default value 2.
>   >
>   >Although the liberty to do anything you want seems appealing at first, I
>   >would think that allowing this is not correct from an OO point of view.
> 
> I don't agree; this is equivalent to redefinition of a feature (=method) in
> a descendant class, which is perfectly acceptable so long as the feature's
> signature (equivalent to column type) remains unchanged.

The O-O principle involved here is Liskov Substitution: if the derived
table is used in the context of code that thinks it's looking at the
base table, does anything break?

Changing the default value of a column should not break anything, 
because the different default value could as well have been entered 
in the column manually.

Nathan Myers
ncm@zembu.com

Thu, 29 Mar 2001 12:46:47 -0800
ncm@zembu.com (Nathan Myers)
Re: Re: Changing the default value of an inherited column
At 11:06 PM 3/28/01 -0500, Tom Lane wrote:
>Mark Knox <segfault@hardline.org> writes:
>> I don't think this solution would be valid on many other platforms.
>
>Au contraire --- the ARM is the first platform I've heard of that does
>not think sizeof(ItemPointerData) is 6.  Else we'd have seen this
>regress test fail before.

I meant I don't think *my* solution (ie packing the struct) would be valid anywhere else. It seems to be an arm-specific problem so maybe it needs an arm-specific patch? I've had to do this type of thing many times to get packages working properly in arm linux. It's a quirky platform.

>> Well, this patch seems to produce attlens of 6 as desired, but it
>> causes many (13) of the regression tests to fail. Do you want to see
>> the regression.diffs?
>
>Please.

See attached.
Thu, 29 Mar 2001 22:33:42 -0500
Mark Knox <segfault@hardline.org>
Re: Re: Call for platforms 
Mark Knox <segfault@hardline.org> writes:
> Well, this patch seems to produce attlens of 6 as desired, but it
> causes many (13) of the regression tests to fail. Do you want to see
> the regression.diffs?
>> 
>> Please.

> See attached.

Does look pretty broken, but I don't see how my idea would have led to
all this other stuff failing.  Anyway, I guess the path of least
resistance is to install your ARM-specific packing patch.  It's
important to make sure that sizeof(ItemPointerData) is 6 if at all
possible, since it will cost you four or so wasted bytes in every
tuple header if it's not.  Will take care of it for RC2.

			regards, tom lane

Thu, 29 Mar 2001 23:58:25 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
>  Yep.  We have many other MIPS (ONYX Crimson, , ONYX, Challenge, Indy w/ IRIX
> 6.2, 6.5, etc.), Alpha and Sparc platforms if there are some others that need
> testing (How about NetBSD on NeXT?).

All of these are interesting to help others decide whether their
particular machine is supported. For my narrow purposes of documenting
which kinds of platforms are supported for the upcoming release, I'm
focused on processor/OS combinations. So the following already seem to
be covered:

MIPS/IRIX (32 bit compilation only- try 64 bit compilation?)
Alpha/Linux
Alpha/Tru64
Sparc/Solaris
Sparc/Linux
x86/NetBSD (need all other NetBSD architectures!)
x86/OpenBSD (need all other archs!)

If you have other combinations (I've forgotten what NeXT is; we need 68k
and 88k architectures tested; our NetBSD/68k guy no longer has that
machine) they would be particularly helpful.

TIA

                      - Thomas

Fri, 30 Mar 2001 15:04:25 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Call for platforms
ncm@zembu.com (Nathan Myers) writes:
> The O-O principle involved here is Liskov Substitution: if the derived
> table is used in the context of code that thinks it's looking at the
> base table, does anything break?

Good point.  That answers my concern about how to handle typmod: an
application *could* be broken by a change in typmod (eg, suppose it's
allocated a buffer just big enough for a char(N) attribute, using the N
of the parent table).  Therefore we must disallow changes in typmod in
child tables.

Further study of creatinh.c shows that we have inconsistent behavior at
the moment, as it will allow columns of the same name to be inherited
from multiple parents and (silently) combined --- how is that really
different from combining with an explicit specification?


I propose the following behavior:

1. A table can have only one column of a given name.  If the same
column name occurs in multiple parent tables and/or in the explicitly
specified column list, these column specifications are combined to
produce a single column specification.  A NOTICE will be emitted to
warn the user that this has happened.  The ordinal position of the
resulting column is determined by its first appearance.

2. An error will be reported if columns to be combined do not all have
the same datatype and typmod value.

3. The new column will have a default value if any of the combined
column specifications have one.  The last-specified default (the one
in the explicitly given column list, or the rightmost parent table
that gives a default) will be used.

4. All relevant constraints from all the column specifications will
be applied.  In particular, if any of the specifications includes NOT
NULL, the resulting column will be NOT NULL.  (But the current
implementation does not support inheritance of UNIQUE or PRIMARY KEY
constraints, and I do not have time to add that now.)

This behavior differs from prior versions as follows:

1. We return to the pre-7.0 behavior of allowing an explicit
specification of a column name that is also inherited (7.0 rejects this,
thereby preventing the default from being changed in the child).
However, we will now issue a warning NOTICE, to answer the concern that
prompted this change of behavior.

2. We will now enforce uniformity of typmod as well as type OID when
combining columns.

3. In both 7.0 and prior versions, if a column appeared in multiple
parents but not in the explicit column list, the first parent's default
value (if any) and NOT NULL state would be used, ignoring those of later
parents.  Failing to "or" together the NOT NULL flags is clearly wrong,
and I believe it's inconsistent to use an earlier rather than later
parent's default value when we want an explicitly-specified default to
win out over all of them.  The explicit column specifications are
treated as coming after the last parent for other purposes, so we should
define the default to use as the last one reading left-to-right.

Comments?  I'm going to implement and commit this today unless I hear
loud squawks ...

			regards, tom lane

Fri, 30 Mar 2001 12:10:59 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
On Fri, Mar 30, 2001 at 12:10:59PM -0500, Tom Lane wrote:
> Comments?  I'm going to implement and commit this today unless I hear
> loud squawks ...

I like it in general and I think it opens some interesting
possibilities.  I don't know much about how the inheritance system is
implemented, so I will put out this scenario in case it makes a
difference.

We recently decided to refactor our schema a bit, using inheritance.
All of our tables have a primary key called "seq" along with some
other common fields such as entry time, etc.  We realized that moving
them into a "base" table allowed us to create functions on "base"
that would work on every derived table.  The main problem was that
we needed fields like "seq" to have distinct sequences, which was
not possible without the ability to override the default value in
each derived table.  It seems like this would be easily doable with
this change.

Another thing that seems kind of interesting would be to have:

CREATE TABLE base (table_id CHAR(8) NOT NULL [, etc.]);
CREATE TABLE foo  (table_id CHAR(8) NOT NULL DEFAULT 'foo');
CREATE TABLE bar  (table_id CHAR(8) NOT NULL DEFAULT 'foo');

Then a function on "base" could look at table_id and know which
table it's working on.  A waste of space, but I can think of
uses for it.
-- 
Christopher Masto         Senior Network Monkey      NetMonger Communications
chris@netmonger.net        info@netmonger.net        http://www.netmonger.net

Free yourself, free your machine, free the daemon -- http://www.freebsd.org/

Fri, 30 Mar 2001 13:07:39 -0500
Christopher Masto <chris@netmonger.net>
Re: Re: Changing the default value of an inherited column
Tom Lane writes:

> 3. The new column will have a default value if any of the combined
> column specifications have one.  The last-specified default (the one
> in the explicitly given column list, or the rightmost parent table
> that gives a default) will be used.

This seems pretty random.  It would be more reasonable if multiple
(default) inheritance weren't allowed unless you explicitly specify a new
default for the new column, but we don't have a syntax for this.

> 4. All relevant constraints from all the column specifications will
> be applied.  In particular, if any of the specifications includes NOT
> NULL, the resulting column will be NOT NULL.  (But the current
> implementation does not support inheritance of UNIQUE or PRIMARY KEY
> constraints, and I do not have time to add that now.)

This is definitely a violation of that Liskov Substitution.  If a context
expects a certain table and gets a more restricted table, it will
certainly notice.

> Comments?  I'm going to implement and commit this today unless I hear
> loud squawks ...

If we're going to make changes to the inheritance logic, we could
certainly use some more thought than a few hours.  If you want to revert
the patch that was installed in 7.0 then ok, but the rest is not
appropriate right now, IMHO.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Fri, 30 Mar 2001 23:05:53 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Changing the default value of an inherited column
Peter Eisentraut <peter_e@gmx.net> writes:
>> 4. All relevant constraints from all the column specifications will
>> be applied.  In particular, if any of the specifications includes NOT
>> NULL, the resulting column will be NOT NULL.  (But the current
>> implementation does not support inheritance of UNIQUE or PRIMARY KEY
>> constraints, and I do not have time to add that now.)

> This is definitely a violation of that Liskov Substitution.  If a context
> expects a certain table and gets a more restricted table, it will
> certainly notice.

Au contraire --- I'd say that if the child table fails to adhere to the
constraints set for the parent table, *that* is a violation of
inheritance.  In particular, a table containing NULLs that is a child of
a table in which the same column is marked NOT NULL is likely to blow up
an application that is not expecting to get any nulls back.

In any case, we have already been inheriting general constraints from
parent tables.  Relaxing that would be a change of behavior.  The
failure to inherit NOT NULL constraints some of the time (in some cases
they were inherited, in some cases not) cannot be construed as anything
but a bug.

> If we're going to make changes to the inheritance logic, we could
> certainly use some more thought than a few hours.

The primary issue here is to revert the 7.0 behavior to what it had been
for many years before that, and secondarily to make NOT NULL inheritance
behave consistently with itself and with other constraints.  It doesn't
take hours of thought to justify either.

I will agree that left-to-right vs. right-to-left precedence of
inherited default values is pretty much a random choice, but it's
doubtful that anyone is really depending on that.  The existing behavior
was not self-consistent anyway, since it was actually not "the first
specified default" but "the default or lack of same attached to the
first parent containing such a field".  For example, if we do not change
this behavior then

	create table p1 (f1 int);
	create table p2 (f1 int default 1) inherits(p1);

results in p2.f1 having a default, while

	create table p1 (f1 int);
	create table p2 (f1 int default 1, f2 int);
	create table p3 (f3 int) inherits(p1, p2);

results in p3.f1 not having a default.  I don't think that can be argued
to be anything but a bug either (consider what happens if p2 also says
NOT NULL for f1).

			regards, tom lane

Fri, 30 Mar 2001 16:15:36 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
On Fri, Mar 30, 2001 at 12:10:59PM -0500, Tom Lane wrote:
> ncm@zembu.com (Nathan Myers) writes:
> > The O-O principle involved here is Liskov Substitution: if the derived
> > table is used in the context of code that thinks it's looking at the
> > base table, does anything break?
> 
> I propose the following behavior:
> 
> 1. A table can have only one column of a given name.  If the same
> column name occurs in multiple parent tables and/or in the explicitly
> specified column list, these column specifications are combined to
> produce a single column specification.  A NOTICE will be emitted to
> warn the user that this has happened.  The ordinal position of the
> resulting column is determined by its first appearance.

Treatment of like-named members of multiple base types is not done
consistently in the various O-O languages.  It's really a snakepit, and 
anything you do automatically will cause terrible problems for somebody.  
Nonetheless, for any given circumstances some possible approaches are 
clearly better than others.

In C++, as in most O-O languages, the like-named members are kept 
distinct.  When referred to in the context of a base type, the member 
chosen is the "right one".  Used in the context of the multiply-derived 
type, the compiler reports an ambiguity, and you are obliged to qualify 
the name explicitly to identify which among the like-named inherited 
members you meant.  You can declare which one is "really inherited".  
Some other languages presume to choose automatically which one they 
think you meant.  The real danger is from members inherited from way
back up the trees, which you might not know one are there.

Of course PG is different from any O-O language.  I don't know if PG 
has an equivalent to the "base-class context".  I suppose PG has a long 
history of merging like-named members, and that the issue is just of 
the details of how the merge happens.  

> 4. All relevant constraints from all the column specifications will
> be applied.  In particular, if any of the specifications includes NOT
> NULL, the resulting column will be NOT NULL.  (But the current
> implementation does not support inheritance of UNIQUE or PRIMARY KEY
> constraints, and I do not have time to add that now.)

Sounds like a TODO item...

Do all the triggers of the base tables get applied, to be run one after 
another?

--
Nathan Myers
ncm@zembu.com

Fri, 30 Mar 2001 13:30:35 -0800
ncm@zembu.com (Nathan Myers)
Re: Re: Changing the default value of an inherited column
On Fri, Mar 30, 2001 at 11:05:53PM +0200, Peter Eisentraut wrote:
> Tom Lane writes:
> 
> > 3. The new column will have a default value if any of the combined
> > column specifications have one.  The last-specified default (the one
> > in the explicitly given column list, or the rightmost parent table
> > that gives a default) will be used.
> 
> This seems pretty random.  It would be more reasonable if multiple
> (default) inheritance weren't allowed unless you explicitly specify a new
> default for the new column, but we don't have a syntax for this.

I agree, but I thought the original issue was that PG _does_ now have 
syntax for it.  Any conflict in default values should result in either 
a failure, or "no default".  Choosing a default randomly, or according 
to an arbitrary and complicated rule (same thing), is a source of bugs.

> > 4. All relevant constraints from all the column specifications will
> > be applied.  In particular, if any of the specifications includes NOT
> > NULL, the resulting column will be NOT NULL.  (But the current
> > implementation does not support inheritance of UNIQUE or PRIMARY KEY
> > constraints, and I do not have time to add that now.)
> 
> This is definitely a violation of that Liskov Substitution.  If a context
> expects a certain table and gets a more restricted table, it will
> certainly notice.

Not so.  The rule is that the base-table code only has to understand
the derived table.  The derived table need not be able to represent
all values possible in the base table. 

Nathan Myers
ncm@zembu.com

Fri, 30 Mar 2001 13:40:25 -0800
ncm@zembu.com (Nathan Myers)
Re: Re: Changing the default value of an inherited column
At 12:10 30/03/01 -0500, Tom Lane wrote:
>
>Comments?  I'm going to implement and commit this today unless I hear
>loud squawks ...
>

Not a squawk as such, but does this have implications for pg_dump?


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sat, 31 Mar 2001 15:06:33 +1000
Philip Warner <pjw@rhyme.com.au>
Re: Re: Changing the default value of an inherited
  column
Philip Warner <pjw@rhyme.com.au> writes:
> Not a squawk as such, but does this have implications for pg_dump?

Good point.  With recently-committed changes, try:

regression=# create table p1 (f1 int default 42 not null, f2 int);
CREATE
regression=# create table c1 (f1 int, f2 int default 7) inherits (p1);
NOTICE:  CREATE TABLE: merging attribute "f1" with inherited definition
NOTICE:  CREATE TABLE: merging attribute "f2" with inherited definition
CREATE
regression=# create table c2 (f1 int default 43, f2 int not null) inherits (p1);
NOTICE:  CREATE TABLE: merging attribute "f1" with inherited definition
NOTICE:  CREATE TABLE: merging attribute "f2" with inherited definition
CREATE

pg_dump dumps both c1 and c2 like this:

CREATE TABLE "c2" (

)
inherits ("p1");

which is OK as far as the field set goes, but it loses the additional
DEFAULT and NOT NULL information for the child table.  Any thoughts on
the best way to fix this?

			regards, tom lane

Sat, 31 Mar 2001 01:36:22 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
At 01:36 31/03/01 -0500, Tom Lane wrote:
>
>which is OK as far as the field set goes, but it loses the additional
>DEFAULT and NOT NULL information for the child table.  Any thoughts on
>the best way to fix this?
>

Can pg_dump easily detect overridden attrs? If so, we just treat them as
table attrs and let the backend do it's stuff.




----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sat, 31 Mar 2001 16:41:24 +1000
Philip Warner <pjw@rhyme.com.au>
Re: Re: Changing the default value of an inherited
  column
ncm@zembu.com (Nathan Myers) writes:
> Of course PG is different from any O-O language.  I don't know if PG 
> has an equivalent to the "base-class context".  I suppose PG has a long 
> history of merging like-named members, and that the issue is just of 
> the details of how the merge happens.  

Yes; AFAICT that behavior goes back to PostQUEL.  It was partially
disabled (without adequate discussion I guess) in 7.0, but it's been
around for a long time.

>> 4. All relevant constraints from all the column specifications will
>> be applied.  In particular, if any of the specifications includes NOT
>> NULL, the resulting column will be NOT NULL.  (But the current
>> implementation does not support inheritance of UNIQUE or PRIMARY KEY
>> constraints, and I do not have time to add that now.)

> Sounds like a TODO item...

There's something about it in TODO already.  There are some definitional
issues though (should uniqueness be across ALL tables of the inheritance
hierarchy, or per-table?  If the former, how would we implement it?).
I believe you can find past discussions about this in the archives.

> Do all the triggers of the base tables get applied, to be run one after 
> another?

Triggers aren't inherited either.  Possibly they should be, but again
I think some forethought is needed...

			regards, tom lane

Sat, 31 Mar 2001 19:36:09 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
ncm@zembu.com (Nathan Myers) writes:
>> This seems pretty random.  It would be more reasonable if multiple
>> (default) inheritance weren't allowed unless you explicitly specify a new
>> default for the new column, but we don't have a syntax for this.

> I agree, but I thought the original issue was that PG _does_ now have 
> syntax for it.  Any conflict in default values should result in either 
> a failure, or "no default".  Choosing a default randomly, or according 
> to an arbitrary and complicated rule (same thing), is a source of
> bugs.

Well, we *do* have a syntax for specifying a new default (the same one
that worked pre-7.0 and does now again).  I guess what you are proposing
is the rule "If conflicting default values are inherited from multiple
parents that each define the same column name, then an error is reported
unless the child table redeclares the column and specifies a new default
to override the inherited ones".

That is:

	create table p1 (f1 int default 1);
	create table p2 (f1 int default 2);
	create table c1 (f2 float) inherits(p1, p2);

would draw an error about conflicting defaults for c1.f1, but

	create table c1 (f1 int default 3, f2 float) inherits(p1, p2);

would be accepted (and 3 would become the default for c1.f1).

This would take a few more lines of code, but I'm willing to do it if
people think it's a safer behavior than picking one of the inherited
default values.

			regards, tom lane

Sat, 31 Mar 2001 19:44:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
Philip Warner <pjw@rhyme.com.au> writes:
> At 01:36 31/03/01 -0500, Tom Lane wrote:
>> which is OK as far as the field set goes, but it loses the additional
>> DEFAULT and NOT NULL information for the child table.  Any thoughts on
>> the best way to fix this?

> Can pg_dump easily detect overridden attrs? If so, we just treat them as
> table attrs and let the backend do it's stuff.

Well, it's already detecting inherited attrs so it can suppress them
from the explicit column list.  Perhaps we should just hack that code
to not suppress inherited attrs when they have default values and/or
NOT NULL that's not in the parent.

			regards, tom lane

Sat, 31 Mar 2001 20:02:25 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
At 20:02 31/03/01 -0500, Tom Lane wrote:
>
>Perhaps we should just hack that code
>to not suppress inherited attrs when they have default values and/or
>NOT NULL that's not in the parent.

That's what I meant; can we easily do the 'not in the parent' part, since
we may have to go up a long hierarchy to find the parent?

 
----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sun, 01 Apr 2001 11:21:59 +1000
Philip Warner <pjw@rhyme.com.au>
Re: Re: Changing the default value of an inherited
  column
Philip Warner <pjw@rhyme.com.au> writes:
> At 20:02 31/03/01 -0500, Tom Lane wrote:
>> Perhaps we should just hack that code
>> to not suppress inherited attrs when they have default values and/or
>> NOT NULL that's not in the parent.

> That's what I meant; can we easily do the 'not in the parent' part, since
> we may have to go up a long hierarchy to find the parent?

pg_dump must already contain code to traverse the inheritance hierarchy
(I haven't looked to see where).  Couldn't we just extend it to believe
that it's found a match only if the default value and NOT NULL state
match, as well as the column name?

			regards, tom lane

Sat, 31 Mar 2001 20:25:44 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
At 20:25 31/03/01 -0500, Tom Lane wrote:
>
>> That's what I meant; can we easily do the 'not in the parent' part, since
>> we may have to go up a long hierarchy to find the parent?
>
>pg_dump must already contain code to traverse the inheritance hierarchy
>(I haven't looked to see where).  Couldn't we just extend it to believe
>that it's found a match only if the default value and NOT NULL state
>match, as well as the column name?
>

You are correct; flagInhAttrs in common.c does the work, and it should be
easy to change. At the moment it extracts all tables attrs then looks for
an attr with the same name in any parent table. We can extend this to check
NOT NULL and DEFAULT. Should I also check TYPEDEFN - can that be changed?


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sun, 01 Apr 2001 11:37:54 +1000
Philip Warner <pjw@rhyme.com.au>
Re: Re: Changing the default value of an inherited
  column
Philip Warner <pjw@rhyme.com.au> writes:
> You are correct; flagInhAttrs in common.c does the work, and it should be
> easy to change. At the moment it extracts all tables attrs then looks for
> an attr with the same name in any parent table. We can extend this to check
> NOT NULL and DEFAULT. Should I also check TYPEDEFN - can that be changed?

We presently disallow change of type in child tables, but you might as
well check that too, if it's just one more strcmp ...

			regards, tom lane

Sat, 31 Mar 2001 20:40:22 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
At 20:40 31/03/01 -0500, Tom Lane wrote:
>Philip Warner <pjw@rhyme.com.au> writes:
>> You are correct; flagInhAttrs in common.c does the work, and it should be
>> easy to change. At the moment it extracts all tables attrs then looks for
>> an attr with the same name in any parent table. We can extend this to check
>> NOT NULL and DEFAULT. Should I also check TYPEDEFN - can that be changed?
>
>We presently disallow change of type in child tables, but you might as
>well check that too, if it's just one more strcmp ...

Looks like it; and just to confirm, based on previous messages, I assume I
should look at the parents from right to left?


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sun, 01 Apr 2001 11:50:06 +1000
Philip Warner <pjw@rhyme.com.au>
Re: Re: Changing the default value of an inherited
  column
Philip Warner <pjw@rhyme.com.au> writes:
> Looks like it; and just to confirm, based on previous messages, I assume I
> should look at the parents from right to left?

At the moment that would be the right thing to do.

If we change the code again based on the latest discussion, then pg_dump
would have to detect whether there are conflicting defaults, which would
mean looking at all the parents not just the rightmost one.  Ugh.  That
might be a good reason not to change...

			regards, tom lane

Sat, 31 Mar 2001 21:00:20 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
At 21:00 31/03/01 -0500, Tom Lane wrote:
>
>If we change the code again based on the latest discussion, then pg_dump
>would have to detect whether there are conflicting defaults, which would
>mean looking at all the parents not just the rightmost one.  Ugh.  That
>might be a good reason not to change...
>

Shall I hold off on this for a day or two to let the other discussion
settle down? It seems whatever happens, we should check NOT NULL.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Sun, 01 Apr 2001 12:09:36 +1000
Philip Warner <pjw@rhyme.com.au>
Re: Re: Changing the default value of an inherited
  column
Tom Ivar Helbekkmo <tih@kpnQwest.no> writes:

> > We need some NetBSD folks to speak up!
> 
> I've once again got a VAX that should be able to run PostgreSQL on
> NetBSD/vax, so I hope to be able to help revitalize that port soon...

It still works.  RC1 configures, compiles and runs on my VAX 4000/500
with NetBSD-current -- but the regression tests give a lot of failures
because the VAX doesn't have IEEE math, leading to different rounding
and erroneous assumptions about the limits of floating point values.
I'll be looking at this more closely.

Also, dynamic loading now works on NetBSD/vax, so my old #ifdef for
that in the backend/port/bsd.c file, which has since propagated into
the new *bsd.c files, can go away (actually, I'm suspicious of the
MIPS part of those, too, but I didn't put that in, and I don't have 
any MIPS-based machines):

Index: src/backend/port/dynloader/freebsd.c
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/backend/port/dynloader/freebsd.c,v
retrieving revision 1.9
diff -c -r1.9 freebsd.c
*** src/backend/port/dynloader/freebsd.c	2001/02/10 02:31:26	1.9
--- src/backend/port/dynloader/freebsd.c	2001/04/01 08:01:20
***************
*** 63,69 ****
  void *
  BSD44_derived_dlopen(const char *file, int num)
  {
! #if defined(__mips__) || (defined(__NetBSD__) && defined(__vax__))
  	sprintf(error_message, "dlopen (%s) not supported", file);
  	return NULL;
  #else
--- 63,69 ----
  void *
  BSD44_derived_dlopen(const char *file, int num)
  {
! #if defined(__mips__)
  	sprintf(error_message, "dlopen (%s) not supported", file);
  	return NULL;
  #else
***************
*** 78,84 ****
  void *
  BSD44_derived_dlsym(void *handle, const char *name)
  {
! #if defined(__mips__) || (defined(__NetBSD__) && defined(__vax__))
  	sprintf(error_message, "dlsym (%s) failed", name);
  	return NULL;
  #else
--- 78,84 ----
  void *
  BSD44_derived_dlsym(void *handle, const char *name)
  {
! #if defined(__mips__)
  	sprintf(error_message, "dlsym (%s) failed", name);
  	return NULL;
  #else
***************
*** 101,107 ****
  void
  BSD44_derived_dlclose(void *handle)
  {
! #if defined(__mips__) || (defined(__NetBSD__) && defined(__vax__))
  #else
  	dlclose(handle);
  #endif
--- 101,107 ----
  void
  BSD44_derived_dlclose(void *handle)
  {
! #if defined(__mips__)
  #else
  	dlclose(handle);
  #endif
Index: src/backend/port/dynloader/netbsd.c
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/backend/port/dynloader/netbsd.c,v
retrieving revision 1.3
diff -c -r1.3 netbsd.c
*** src/backend/port/dynloader/netbsd.c	2001/02/10 02:31:26	1.3
--- src/backend/port/dynloader/netbsd.c	2001/04/01 08:01:20
***************
*** 63,69 ****
  void *
  BSD44_derived_dlopen(const char *file, int num)
  {
! #if defined(__mips__) || (defined(__NetBSD__) && defined(__vax__))
  	sprintf(error_message, "dlopen (%s) not supported", file);
  	return NULL;
  #else
--- 63,69 ----
  void *
  BSD44_derived_dlopen(const char *file, int num)
  {
! #if defined(__mips__)
  	sprintf(error_message, "dlopen (%s) not supported", file);
  	return NULL;
  #else
***************
*** 78,84 ****
  void *
  BSD44_derived_dlsym(void *handle, const char *name)
  {
! #if defined(__mips__) || (defined(__NetBSD__) && defined(__vax__))
  	sprintf(error_message, "dlsym (%s) failed", name);
  	return NULL;
  #elif defined(__ELF__)
--- 78,84 ----
  void *
  BSD44_derived_dlsym(void *handle, const char *name)
  {
! #if defined(__mips__)
  	sprintf(error_message, "dlsym (%s) failed", name);
  	return NULL;
  #elif defined(__ELF__)
***************
*** 101,107 ****
  void
  BSD44_derived_dlclose(void *handle)
  {
! #if defined(__mips__) || (defined(__NetBSD__) && defined(__vax__))
  #else
  	dlclose(handle);
  #endif
--- 101,107 ----
  void
  BSD44_derived_dlclose(void *handle)
  {
! #if defined(__mips__)
  #else
  	dlclose(handle);
  #endif
Index: src/backend/port/dynloader/openbsd.c
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/backend/port/dynloader/openbsd.c,v
retrieving revision 1.3
diff -c -r1.3 openbsd.c
*** src/backend/port/dynloader/openbsd.c	2001/02/10 02:31:26	1.3
--- src/backend/port/dynloader/openbsd.c	2001/04/01 08:01:20
***************
*** 63,69 ****
  void *
  BSD44_derived_dlopen(const char *file, int num)
  {
! #if defined(__mips__) || (defined(__NetBSD__) && defined(__vax__))
  	sprintf(error_message, "dlopen (%s) not supported", file);
  	return NULL;
  #else
--- 63,69 ----
  void *
  BSD44_derived_dlopen(const char *file, int num)
  {
! #if defined(__mips__)
  	sprintf(error_message, "dlopen (%s) not supported", file);
  	return NULL;
  #else
***************
*** 78,84 ****
  void *
  BSD44_derived_dlsym(void *handle, const char *name)
  {
! #if defined(__mips__) || (defined(__NetBSD__) && defined(__vax__))
  	sprintf(error_message, "dlsym (%s) failed", name);
  	return NULL;
  #elif defined(__ELF__)
--- 78,84 ----
  void *
  BSD44_derived_dlsym(void *handle, const char *name)
  {
! #if defined(__mips__)
  	sprintf(error_message, "dlsym (%s) failed", name);
  	return NULL;
  #elif defined(__ELF__)
***************
*** 101,107 ****
  void
  BSD44_derived_dlclose(void *handle)
  {
! #if defined(__mips__) || (defined(__NetBSD__) && defined(__vax__))
  #else
  	dlclose(handle);
  #endif
--- 101,107 ----
  void
  BSD44_derived_dlclose(void *handle)
  {
! #if defined(__mips__)
  #else
  	dlclose(handle);
  #endif


-tih
-- 
The basic difference is this: hackers build things, crackers break them.

01 Apr 2001 10:16:56 +0200
Tom Ivar Helbekkmo <tih@kpnQwest.no>
Re: Call for platforms
Christopher Masto <chris@netmonger.net> writes:
> Another thing that seems kind of interesting would be to have:
> CREATE TABLE base (table_id CHAR(8) NOT NULL [, etc.]);
> CREATE TABLE foo  (table_id CHAR(8) NOT NULL DEFAULT 'foo');
> CREATE TABLE bar  (table_id CHAR(8) NOT NULL DEFAULT 'bar');
> Then a function on "base" could look at table_id and know which
> table it's working on.  A waste of space, but I can think of
> uses for it.

This particular need is superseded in 7.1 by the 'tableoid'
pseudo-column.  However you can certainly imagine variants of this
that tableoid doesn't handle, for example columns where the subtable
creator can provide a useful-but-not-always-correct default value.

			regards, tom lane

Sun, 01 Apr 2001 15:15:56 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
Tom Ivar Helbekkmo <tih@kpnQwest.no> writes:
> Also, dynamic loading now works on NetBSD/vax, so my old #ifdef for
> that in the backend/port/bsd.c file, which has since propagated into
> the new *bsd.c files, can go away.

Patch applied, thanks.

			regards, tom lane

Sun, 01 Apr 2001 23:09:45 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Call for platforms 
> > I've once again got a VAX that should be able to run PostgreSQL on
> > NetBSD/vax, so I hope to be able to help revitalize that port soon...
> It still works.  RC1 configures, compiles and runs on my VAX 4000/500
> with NetBSD-current -- but the regression tests give a lot of failures
> because the VAX doesn't have IEEE math, leading to different rounding
> and erroneous assumptions about the limits of floating point values.
> I'll be looking at this more closely.

Great! Will put it on the list :)

                     - Thomas

Mon, 02 Apr 2001 06:49:53 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Call for platforms
Tom Lane writes:

> Well, we *do* have a syntax for specifying a new default (the same one
> that worked pre-7.0 and does now again).  I guess what you are proposing
> is the rule "If conflicting default values are inherited from multiple
> parents that each define the same column name, then an error is reported
> unless the child table redeclares the column and specifies a new default
> to override the inherited ones".

This was the idea.  If it's to complicated to do now, let's at least keep
it in mind.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Mon, 2 Apr 2001 18:34:26 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Changing the default value of an inherited column
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> Well, we *do* have a syntax for specifying a new default (the same one
>> that worked pre-7.0 and does now again).  I guess what you are proposing
>> is the rule "If conflicting default values are inherited from multiple
>> parents that each define the same column name, then an error is reported
>> unless the child table redeclares the column and specifies a new default
>> to override the inherited ones".

> This was the idea.  If it's to complicated to do now, let's at least keep
> it in mind.

You and Nathan appear to like it, and no one else has objected.
I shall make it so.

Philip: the rule that pg_dump needs to apply w.r.t. defaults for
inherited fields is that if an inherited field has a default and
either (a) no parent table supplies a default, or (b) any parent
table supplies a default different from the child's, then pg_dump
had better emit the child field explicitly.

			regards, tom lane

Mon, 02 Apr 2001 13:27:06 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
On Sat, Mar 31, 2001 at 07:44:30PM -0500, Tom Lane wrote:
> ncm@zembu.com (Nathan Myers) writes:
> >> This seems pretty random.  It would be more reasonable if multiple
> >> (default) inheritance weren't allowed unless you explicitly specify a new
> >> default for the new column, but we don't have a syntax for this.
> 
> > I agree, but I thought the original issue was that PG _does_ now have 
> > syntax for it.  Any conflict in default values should result in either 
> > a failure, or "no default".  Choosing a default randomly, or according 
> > to an arbitrary and complicated rule (same thing), is a source of
> > bugs.
> 
> Well, we *do* have a syntax for specifying a new default (the same one
> that worked pre-7.0 and does now again).  I guess what you are proposing
> is the rule "If conflicting default values are inherited from multiple
> parents that each define the same column name, then an error is reported
> unless the child table redeclares the column and specifies a new default
> to override the inherited ones".
> 
> That is:
> 
> 	create table p1 (f1 int default 1);
> 	create table p2 (f1 int default 2);
> 	create table c1 (f2 float) inherits(p1, p2);   # XXX
> 
> would draw an error about conflicting defaults for c1.f1, but
> 
> 	create table c1 (f1 int default 3, f2 float) inherits(p1, p2);
> 
> would be accepted (and 3 would become the default for c1.f1).
> 
> This would take a few more lines of code, but I'm willing to do it if
> people think it's a safer behavior than picking one of the inherited
> default values.

I do.  

Allowing the line marked XXX above, but asserting no default for 
c1.f1 in that case, would be equally safe.  (A warning would be 
polite, anyhow.) User code that doesn't rely on the default wouldn't 
notice.  You only need to choose a default if somebody adding rows to 
c1 uses it.

Nathan Myers
ncm@zembu.com

Mon, 2 Apr 2001 14:22:05 -0700
ncm@zembu.com (Nathan Myers)
Re: Re: Changing the default value of an inherited column
On Sun, Apr 01, 2001 at 03:15:56PM -0400, Tom Lane wrote:
> Christopher Masto <chris@netmonger.net> writes:
> > Another thing that seems kind of interesting would be to have:
> > CREATE TABLE base (table_id CHAR(8) NOT NULL [, etc.]);
> > CREATE TABLE foo  (table_id CHAR(8) NOT NULL DEFAULT 'foo');
> > CREATE TABLE bar  (table_id CHAR(8) NOT NULL DEFAULT 'bar');
> > Then a function on "base" could look at table_id and know which
> > table it's working on.  A waste of space, but I can think of
> > uses for it.
> 
> This particular need is superseded in 7.1 by the 'tableoid'
> pseudo-column.  However you can certainly imagine variants of this
> that tableoid doesn't handle, for example columns where the subtable
> creator can provide a useful-but-not-always-correct default value.

A bit of O-O doctrine... when you find yourself tempted to do something 
like the above, it usually means you're trying to do the wrong thing.  
You may not have a choice, in some cases, but you should know you are 
on the way to architecture meltdown.  "She'll blow, Cap'n!"

Nathan Myers
ncm@zembu.com

Mon, 2 Apr 2001 14:37:13 -0700
ncm@zembu.com (Nathan Myers)
Re: Re: Changing the default value of an inherited column
On Mon, Apr 02, 2001 at 01:27:06PM -0400, Tom Lane wrote:
> Philip: the rule that pg_dump needs to apply w.r.t. defaults for
> inherited fields is that if an inherited field has a default and
> either (a) no parent table supplies a default, or (b) any parent
> table supplies a default different from the child's, then pg_dump
> had better emit the child field explicitly.

The rule above appears to work even if inherited-default conflicts 
are not taken as an error, but just result in a derived-table column 
with no default.

Nathan Myers
ncm@zembu.com

Mon, 2 Apr 2001 14:46:16 -0700
ncm@zembu.com (Nathan Myers)
Re: Re: Changing the default value of an inherited column
ncm@zembu.com (Nathan Myers) writes:
> On Sat, Mar 31, 2001 at 07:44:30PM -0500, Tom Lane wrote:
>> That is:
>> 
>> create table p1 (f1 int default 1);
>> create table p2 (f1 int default 2);
>> create table c1 (f2 float) inherits(p1, p2);   # XXX
>> 
>> would draw an error about conflicting defaults for c1.f1, but
>> 
>> create table c1 (f1 int default 3, f2 float) inherits(p1, p2);
>> 
>> would be accepted (and 3 would become the default for c1.f1).
>> 
>> This would take a few more lines of code, but I'm willing to do it if
>> people think it's a safer behavior than picking one of the inherited
>> default values.

> Allowing the line marked XXX above, but asserting no default for 
> c1.f1 in that case, would be equally safe.  (A warning would be 
> polite, anyhow.)

The trouble with that is that we don't have such a concept as "no
default", if by that you mean "INSERTs *must* specify a value".
What would really happen would be that the effective default would
be NULL, which I think would be fairly surprising behavior, since
none of the three tables involved asked for that.

I have committed code that raises an error in cases such as XXX above.
Let's try it like that for awhile and see if anyone complains ...

			regards, tom lane

Mon, 02 Apr 2001 18:26:41 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
At 13:27 2/04/01 -0400, Tom Lane wrote:
>
>Philip: the rule that pg_dump needs to apply w.r.t. defaults for
>inherited fields is that if an inherited field has a default and
>either (a) no parent table supplies a default, or (b) any parent
>table supplies a default different from the child's, then pg_dump
>had better emit the child field explicitly.
>

What is happening with IS NULL constraints (and type names)? I presume the
above rule should be applied to each of these fields?


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Tue, 03 Apr 2001 11:57:45 +1000
Philip Warner <pjw@rhyme.com.au>
Re: Re: Changing the default value of an inherited
  column
Philip Warner <pjw@rhyme.com.au> writes:
> At 13:27 2/04/01 -0400, Tom Lane wrote:
>> Philip: the rule that pg_dump needs to apply w.r.t. defaults for
>> inherited fields is that if an inherited field has a default and
>> either (a) no parent table supplies a default, or (b) any parent
>> table supplies a default different from the child's, then pg_dump
>> had better emit the child field explicitly.

> What is happening with IS NULL constraints (and type names)?

NOT NULL on a child field would only force it to be dumped if none
of the parents say NOT NULL.  Type name really is not an issue since
it will have to be the same in all the tables anyway; I wouldn't bother
expending any code there.

			regards, tom lane

Mon, 02 Apr 2001 23:57:30 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 
At 23:57 2/04/01 -0400, Tom Lane wrote:
>
>NOT NULL on a child field would only force it to be dumped if none
>of the parents say NOT NULL.  Type name really is not an issue since
>it will have to be the same in all the tables anyway; I wouldn't bother
>expending any code there.
>

I've made tha changes and it all seems to work, bu there is a minor
inconsistency:

    create table p3_def1(f1 int default 1, f2 int);
    create table c5(f1 int not null, f3 int) inherits(p3_def1);

c5 gets dumped as:

    CREATE TABLE "c5" (
        "f1" integer DEFAULT 1 NOT NULL,
        "f3" integer
    )
    inherits ("p3_def1");

since the NOT NULL forces the field to dump, and it is dumps as though it
were a real field. 

Similarly,

    create table p2_nn(f1 int not null, f2 int not null);
    create table c6(f1 int default 2, ,f3 int) inherits(p2_nn);

results in C6 being dumped as:

    CREATE TABLE "c6" (
        "f1" integer DEFAULT 2 NOT NULL,
        "f3" integer
    )
    inherits ("p2_nn");

I think it needs to dump ONLY the overridden settigns, since a change to
the overriding behaviour of a child seems like a bad thing.

What do you think?


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                 |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/

Tue, 03 Apr 2001 16:01:31 +1000
Philip Warner <pjw@rhyme.com.au>
Re: Re: Changing the default value of an inherited
  column
Philip Warner <pjw@rhyme.com.au> writes:
> I think it needs to dump ONLY the overridden settigns, since a change to
> the overriding behaviour of a child seems like a bad thing.

I was about to say it's not worth the trouble, but I see you already
did it ...

			regards, tom lane

Tue, 03 Apr 2001 10:11:50 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: Changing the default value of an inherited column 

For those that want to get in before the rush, I'm going to do an announce
this evenin to -general and -announce ...

Vince, can you make appropriate changes to the WebSite as far as linking
to it is concerned, so that the mirrors pick up the new links also?

Thanks ..

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org


Fri, 6 Apr 2001 11:55:21 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
RC3 ...
It looks like you wrapped the intermediate (broken) state of
interfaces/odbc/convert.c that Hiroshi had in there for a few hours.
Dunno if this is important enough to re-wrap RC3 for; it might affect
a few ODBC users ...

			regards, tom lane

Fri, 06 Apr 2001 12:21:24 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: RC3 ... 
Tom Lane wrote:
> 
> It looks like you wrapped the intermediate (broken) state of
> interfaces/odbc/convert.c that Hiroshi had in there for a few hours.
> Dunno if this is important enough to re-wrap RC3 for; it might affect
> a few ODBC users ...

Just as I was getting ready to upload a quickie RC3 RPMset... :-)
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Fri, 06 Apr 2001 12:28:17 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: RC3 ...
On Fri, 6 Apr 2001, The Hermit Hacker wrote:

>
> For those that want to get in before the rush, I'm going to do an announce
> this evenin to -general and -announce ...
>
> Vince, can you make appropriate changes to the WebSite as far as linking
> to it is concerned, so that the mirrors pick up the new links also?

Does it look like this it gonna be the one?  It's stable and all that?
Once it's on the website no matter what moniker it's got (up to and
including "DANGER THIS IS BROKEN SO DON'T USE IT") it will be viewed
as the golden apple so I want to avoid another full mailbox.  I'll
probably wait till tomorrow evening (if you announce tonite) just to
be sure.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
         56K Nationwide Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Fri, 6 Apr 2001 12:58:29 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: RC3 ...
On Fri, 6 Apr 2001, Vince Vielhaber wrote:

> On Fri, 6 Apr 2001, The Hermit Hacker wrote:
>
> >
> > For those that want to get in before the rush, I'm going to do an announce
> > this evenin to -general and -announce ...
> >
> > Vince, can you make appropriate changes to the WebSite as far as linking
> > to it is concerned, so that the mirrors pick up the new links also?
>
> Does it look like this it gonna be the one?  It's stable and all that?
> Once it's on the website no matter what moniker it's got (up to and
> including "DANGER THIS IS BROKEN SO DON'T USE IT") it will be viewed
> as the golden apple so I want to avoid another full mailbox.  I'll
> probably wait till tomorrow evening (if you announce tonite) just to
> be sure.

baring any major blow ups, the only thing we are waiting on is docs ...



Fri, 6 Apr 2001 14:19:23 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: RC3 ...
The Hermit Hacker writes:

> baring any major blow ups, the only thing we are waiting on is docs ...

The docs are ready for shipment.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Fri, 6 Apr 2001 19:38:05 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: RC3 ...
On Fri, 6 Apr 2001, Peter Eisentraut wrote:

> The Hermit Hacker writes:
>
> > baring any major blow ups, the only thing we are waiting on is docs ...
>
> The docs are ready for shipment.

Even better ...

Okay, let's let this sit as RC3 for the next week, as soon as someone
makes a change, I'll do up a new RC# that night ... if we can get a nice
quiet period where nobody pops up with "just one more thing", let's try
for a release for next Friday ... *cross fingers* *grin*


Fri, 6 Apr 2001 14:45:56 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: RC3 ...
On Fri, 6 Apr 2001, The Hermit Hacker wrote:

> On Fri, 6 Apr 2001, Peter Eisentraut wrote:
>
> > The Hermit Hacker writes:
> >
> > > baring any major blow ups, the only thing we are waiting on is docs ...
> >
> > The docs are ready for shipment.
>
> Even better ...
>
> Okay, let's let this sit as RC3 for the next week, as soon as someone
> makes a change, I'll do up a new RC# that night ... if we can get a nice
> quiet period where nobody pops up with "just one more thing", let's try
> for a release for next Friday ... *cross fingers* *grin*

So does RC3 have the docs and the odbc thing mentioned earlier rolled
in?

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
         56K Nationwide Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Fri, 6 Apr 2001 14:09:54 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: RC3 ...
On Fri, 6 Apr 2001, Vince Vielhaber wrote:

> On Fri, 6 Apr 2001, The Hermit Hacker wrote:
>
> > On Fri, 6 Apr 2001, Peter Eisentraut wrote:
> >
> > > The Hermit Hacker writes:
> > >
> > > > baring any major blow ups, the only thing we are waiting on is docs ...
> > >
> > > The docs are ready for shipment.
> >
> > Even better ...
> >
> > Okay, let's let this sit as RC3 for the next week, as soon as someone
> > makes a change, I'll do up a new RC# that night ... if we can get a nice
> > quiet period where nobody pops up with "just one more thing", let's try
> > for a release for next Friday ... *cross fingers* *grin*
>
> So does RC3 have the docs and the odbc thing mentioned earlier rolled
> in?

Yes, just re-bundled it ...



Fri, 6 Apr 2001 15:41:57 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: RC3 ...
On Fri, 6 Apr 2001, The Hermit Hacker wrote:

> On Fri, 6 Apr 2001, Vince Vielhaber wrote:
>
> > On Fri, 6 Apr 2001, The Hermit Hacker wrote:
> >
> > > On Fri, 6 Apr 2001, Peter Eisentraut wrote:
> > >
> > > > The Hermit Hacker writes:
> > > >
> > > > > baring any major blow ups, the only thing we are waiting on is docs ...
> > > >
> > > > The docs are ready for shipment.
> > >
> > > Even better ...
> > >
> > > Okay, let's let this sit as RC3 for the next week, as soon as someone
> > > makes a change, I'll do up a new RC# that night ... if we can get a nice
> > > quiet period where nobody pops up with "just one more thing", let's try
> > > for a release for next Friday ... *cross fingers* *grin*
> >
> > So does RC3 have the docs and the odbc thing mentioned earlier rolled
> > in?
>
> Yes, just re-bundled it ...

Cool!

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
         56K Nationwide Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Fri, 6 Apr 2001 15:40:05 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: RC3 ...
> > The docs are ready for shipment.
> Even better ...
> Okay, let's let this sit as RC3 for the next week...

I'll go ahead and start generating hardcopy, though I understand that it
is no longer allowed into the shipping tarball :(

Lamar, do you plan to continue to package the hardcopy somewhere in the
RPMs? If so, I'll have them ready soon.

                    - Thomas

Fri, 06 Apr 2001 20:45:45 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: RC3 ...
Thomas Lockhart wrote:
> I'll go ahead and start generating hardcopy, though I understand that it
> is no longer allowed into the shipping tarball :(
 
> Lamar, do you plan to continue to package the hardcopy somewhere in the
> RPMs? If so, I'll have them ready soon.

I didn't for 7.0, IIRC.  Or maybe I did for 7.0, but then didn't for
7.0.2? I'll have to go back to the changelog.... 

I am open to suggestion -- should it be part of the main postgresql RPM
with the source and html docs, or should it be a separate package, such
as postgresql-hardcopy-docs? Ideas?  Comments?

The 'Internals' document is still in the main package, FWIW.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Fri, 06 Apr 2001 17:51:21 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: RC3 ...
> On Fri, 6 Apr 2001, The Hermit Hacker wrote:
> 
> >
> > For those that want to get in before the rush, I'm going to do an announce
> > this evenin to -general and -announce ...
> >
> > Vince, can you make appropriate changes to the WebSite as far as linking
> > to it is concerned, so that the mirrors pick up the new links also?
> 
> Does it look like this it gonna be the one?  It's stable and all that?
> Once it's on the website no matter what moniker it's got (up to and
> including "DANGER THIS IS BROKEN SO DON'T USE IT") it will be viewed
> as the golden apple so I want to avoid another full mailbox.  I'll
> probably wait till tomorrow evening (if you announce tonite) just to
> be sure.

Smart man, that Vince.  :-)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 6 Apr 2001 17:56:20 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: RC3 ...
Thomas Lockhart wrote:
  >> > The docs are ready for shipment.
  >> Even better ...
  >> Okay, let's let this sit as RC3 for the next week...
  >
  >I'll go ahead and start generating hardcopy, though I understand that it
  >is no longer allowed into the shipping tarball :(
  >
  >Lamar, do you plan to continue to package the hardcopy somewhere in the
  >RPMs? If so, I'll have them ready soon.
 
Thomas, will you be doing .pdf files? I have had requests to put that
in the Debian documentation package.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "A gentle answer turns away wrath, but a harsh word  
      stirs up anger."        Proverbs 15:1  



Fri, 06 Apr 2001 23:16:54 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Re: RC3 ... 
On Fri, 6 Apr 2001, Thomas Lockhart wrote:

> > > The docs are ready for shipment.
> > Even better ...
> > Okay, let's let this sit as RC3 for the next week...
>
> I'll go ahead and start generating hardcopy, though I understand that it
> is no longer allowed into the shipping tarball :(

At 2Meg, is there a reason why we include any of the docs as part of the
standard tar ball?  It shouldn't be required to compile, so should be able
to be left out of the main tar ball and downloaded seperately as required
.. thereby shrinking the distribution to <6Meg from its current 8 ...


Fri, 6 Apr 2001 20:19:00 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: RC3 ...
On Fri, 6 Apr 2001, The Hermit Hacker wrote:

> On Fri, 6 Apr 2001, Thomas Lockhart wrote:
>
> > > > The docs are ready for shipment.
> > > Even better ...
> > > Okay, let's let this sit as RC3 for the next week...
> >
> > I'll go ahead and start generating hardcopy, though I understand that it
> > is no longer allowed into the shipping tarball :(
>
> At 2Meg, is there a reason why we include any of the docs as part of the
> standard tar ball?  It shouldn't be required to compile, so should be able
> to be left out of the main tar ball and downloaded seperately as required
> .. thereby shrinking the distribution to <6Meg from its current 8 ...

We may already do so, but include the upgrade instructions, first time
install instructions and basic startup.  Then bundle the docs as they
normally come in the tarball as a separate set (alone).   A number of
other packages are done this way.  If we want to include a be-all-end-all
we can do that too.  Of course other doc formats/sets will also be
available.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
         56K Nationwide Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Fri, 6 Apr 2001 19:41:25 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: RC3 ...
> On Fri, 6 Apr 2001, Thomas Lockhart wrote:
> 
> > > > The docs are ready for shipment.
> > > Even better ...
> > > Okay, let's let this sit as RC3 for the next week...
> >
> > I'll go ahead and start generating hardcopy, though I understand that it
> > is no longer allowed into the shipping tarball :(
> 
> At 2Meg, is there a reason why we include any of the docs as part of the
> standard tar ball?  It shouldn't be required to compile, so should be able
> to be left out of the main tar ball and downloaded seperately as required
> .. thereby shrinking the distribution to <6Meg from its current 8 ...

Can we drop TODO.detail from the tarball too?  No need to include that,
I think.  The web site has nice links to it now.  Uncompressed it is
1.314 megs.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 6 Apr 2001 19:43:32 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: RC3 ...
> > OTOH, if Marc was only thinking of removing the pre-built docs from the
> > tarball, I don't object to that.  I'm not sure why those weren't
> > distributed as separate tarballs from the get-go.  I just say that the
> > doc sources are part of the source distribution...

 From the get-go, the docs were not, uh, useful docs. They have grown
quite a bit from 1996 (with sources and formatting, probably by orders
of magnitude).

                      - Thomas

Fri, 06 Apr 2001 23:49:59 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: RC3 ...
> Thomas, will you be doing .pdf files? I have had requests to put that
> in the Debian documentation package.

afaik, I don't have the means to generate pdf directly. Pointers would
be appreciated, if there are mechanisms available on Linux boxes. 

We have had lots of offers of help for these conversions, so when the
hardcopy is ready we can ask someone to convert from there. OK?

                        - Thomas

Fri, 06 Apr 2001 23:52:38 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Re: RC3 ...
On Fri, 6 Apr 2001, Bruce Momjian wrote:

> > On Fri, 6 Apr 2001, Thomas Lockhart wrote:
> >
> > > > > The docs are ready for shipment.
> > > > Even better ...
> > > > Okay, let's let this sit as RC3 for the next week...
> > >
> > > I'll go ahead and start generating hardcopy, though I understand that it
> > > is no longer allowed into the shipping tarball :(
> >
> > At 2Meg, is there a reason why we include any of the docs as part of the
> > standard tar ball?  It shouldn't be required to compile, so should be able
> > to be left out of the main tar ball and downloaded seperately as required
> > .. thereby shrinking the distribution to <6Meg from its current 8 ...
>
> Can we drop TODO.detail from the tarball too?  No need to include that,
> I think.  The web site has nice links to it now.  Uncompressed it is
> 1.314 megs.

Definitely, I think TODO.detail should be refer'd to by the TODO file, but
not included in the distribution itself ...



Fri, 6 Apr 2001 21:01:30 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Re: RC3 ...
>> At 2Meg, is there a reason why we include any of the docs as part of the
>> standard tar ball?  It shouldn't be required to compile, so should be able
>> to be left out of the main tar ball and downloaded seperately as required
>> .. thereby shrinking the distribution to <6Meg from its current 8 ...

> Can we drop TODO.detail from the tarball too?  No need to include that,
> I think.  The web site has nice links to it now.  Uncompressed it is
> 1.314 megs.

That strikes me as an awfully web-centric view of things.  Not everyone
has an always-on high-speed Internet link.

If you want to make the docs and TODO.detail be a separate chunk of the
split distribution, that's fine with me.  But I don't agree with
removing them from the full tarball.

OTOH, if Marc was only thinking of removing the pre-built docs from the
tarball, I don't object to that.  I'm not sure why those weren't
distributed as separate tarballs from the get-go.  I just say that the
doc sources are part of the source distribution...

			regards, tom lane

Fri, 06 Apr 2001 20:09:05 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: RC3 ... 
> > Can we drop TODO.detail from the tarball too?  No need to include that,
> > I think.  The web site has nice links to it now.  Uncompressed it is
> > 1.314 megs.
> 
> That strikes me as an awfully web-centric view of things.  Not everyone
> has an always-on high-speed Internet link.
> 
> If you want to make the docs and TODO.detail be a separate chunk of the
> split distribution, that's fine with me.  But I don't agree with
> removing them from the full tarball.

But isn't TODO.detail mostly of interest to people who use CVS?
I see your point, though.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 6 Apr 2001 20:19:47 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: RC3 ...
On Fri, 6 Apr 2001, Tom Lane wrote:

> >> At 2Meg, is there a reason why we include any of the docs as part of the
> >> standard tar ball?  It shouldn't be required to compile, so should be able
> >> to be left out of the main tar ball and downloaded seperately as required
> >> .. thereby shrinking the distribution to <6Meg from its current 8 ...
>
> > Can we drop TODO.detail from the tarball too?  No need to include that,
> > I think.  The web site has nice links to it now.  Uncompressed it is
> > 1.314 megs.
>
> That strikes me as an awfully web-centric view of things.  Not everyone
> has an always-on high-speed Internet link.
>
> If you want to make the docs and TODO.detail be a separate chunk of the
> split distribution, that's fine with me.  But I don't agree with
> removing them from the full tarball.
>
> OTOH, if Marc was only thinking of removing the pre-built docs from the
> tarball, I don't object to that.  I'm not sure why those weren't
> distributed as separate tarballs from the get-go.  I just say that the
> doc sources are part of the source distribution...

But, why?  That sounds like a highly DSL-centric view of things *grin*  If
someone really wants docs, what hurts a second GET ftp call?



Fri, 6 Apr 2001 21:40:50 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Re: RC3 ... 
> > Thomas, will you be doing .pdf files? I have had requests to put that
> > in the Debian documentation package.
> 
> afaik, I don't have the means to generate pdf directly. Pointers would
> be appreciated, if there are mechanisms available on Linux boxes. 
> 
> We have had lots of offers of help for these conversions, so when the
> hardcopy is ready we can ask someone to convert from there. OK?

Can you use ps2pdf to generate PDF?  It is a utility that comes with
ghostscript.  I know versions >= 6.0 are fine.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 6 Apr 2001 21:23:35 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: RC3 ...
> > That strikes me as an awfully web-centric view of things.  Not everyone
> > has an always-on high-speed Internet link.
> >
> > If you want to make the docs and TODO.detail be a separate chunk of the
> > split distribution, that's fine with me.  But I don't agree with
> > removing them from the full tarball.
> >
> > OTOH, if Marc was only thinking of removing the pre-built docs from the
> > tarball, I don't object to that.  I'm not sure why those weren't
> > distributed as separate tarballs from the get-go.  I just say that the
> > doc sources are part of the source distribution...
> 
> But, why?  That sounds like a highly DSL-centric view of things *grin*  If
> someone really wants docs, what hurts a second GET ftp call?

A major issue is that we don't regenerate docs for 7.1.1 or later, so
the 7.1 docs carry for all the 7.1.X releases.  That would seem to argue
for a separate tarball for docs so people don't redownload the docs
again for 7.1.1.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 6 Apr 2001 21:25:15 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: RC3 ...
On Fri, 6 Apr 2001, Bruce Momjian wrote:

> > > That strikes me as an awfully web-centric view of things.  Not everyone
> > > has an always-on high-speed Internet link.
> > >
> > > If you want to make the docs and TODO.detail be a separate chunk of the
> > > split distribution, that's fine with me.  But I don't agree with
> > > removing them from the full tarball.
> > >
> > > OTOH, if Marc was only thinking of removing the pre-built docs from the
> > > tarball, I don't object to that.  I'm not sure why those weren't
> > > distributed as separate tarballs from the get-go.  I just say that the
> > > doc sources are part of the source distribution...
> >
> > But, why?  That sounds like a highly DSL-centric view of things *grin*  If
> > someone really wants docs, what hurts a second GET ftp call?
>
> A major issue is that we don't regenerate docs for 7.1.1 or later, so
> the 7.1 docs carry for all the 7.1.X releases.  That would seem to argue
> for a separate tarball for docs so people don't redownload the docs
> again for 7.1.1.

Okay, unless someone can come up with a really good argument *for* why
docs has to be included as part of the main tar file, I'm going to change
the distributin generating script so that it generates a .src.tar.gz file
seperate from the .doc.tar.gz file, which will make .src.tar.gz ~6Meg
instead of the 8meg we are currently forcing ppl to download ...

Peter E, is there anything part of the configure/make procedure that
*requires* pgsql/doc to be there else it will break?  If so, can you
possibly put it as a test "if docs exists, deal with it, else ignore"?



Fri, 6 Apr 2001 23:08:24 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Re: RC3 ...
On Fri, Apr 06, 2001 at 09:23:35PM -0400, Bruce Momjian allegedly wrote:
> > > Thomas, will you be doing .pdf files? I have had requests to put that
> > > in the Debian documentation package.
> > 
> > afaik, I don't have the means to generate pdf directly. Pointers would
> > be appreciated, if there are mechanisms available on Linux boxes. 
> > 
> > We have had lots of offers of help for these conversions, so when the
> > hardcopy is ready we can ask someone to convert from there. OK?
> 
> Can you use ps2pdf to generate PDF?  It is a utility that comes with
> ghostscript.  I know versions >= 6.0 are fine.

PDF files generated from postscript with Adobe Acrobat are usually of
much higher quality than those generated by ghostscript. It seems that
ghostscript encodes rendered (bitmaps) documents, while Acrobat generates
PDF files of a quality similar to the original postscript documents.

You would definately have much hihger quality PDF files if someone with
access to Acrobat would step forward. Too bad Acrobat is soo expensive :(

Regards,

Mathijs
-- 
It's not that perl programmers are idiots, it's that the language
rewards idiotic behavior in a way that no other language or tool has
ever done.
                                                    Erik Naggum

Sat, 7 Apr 2001 04:26:46 +0200
Mathijs Brands <mathijs@ilse.nl>
Re: Re: RC3 ...
> > Can you use ps2pdf to generate PDF?  It is a utility that comes with
> > ghostscript.  I know versions >= 6.0 are fine.
> 
> PDF files generated from postscript with Adobe Acrobat are usually of
> much higher quality than those generated by ghostscript. It seems that
> ghostscript encodes rendered (bitmaps) documents, while Acrobat generates
> PDF files of a quality similar to the original postscript documents.
> 
> You would definately have much hihger quality PDF files if someone with
> access to Acrobat would step forward. Too bad Acrobat is soo expensive :(

This is only true of ghostscript version <6.0.  Pre-6.0 could only
encode non-bitmapped fonts if they were the standard Adobe 35.  6.0 and
later do full curve rendering for all fonts, at least they should.  My
book PDF's that were used to print certainly were not bitmapped fonts. 
I have tons of PDF's on my web site, and none use bitmapped fonts.  All
used ghoscript 6.01.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 6 Apr 2001 22:46:07 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: RC3 ...
I have no idea if what I say is true about the PG distribution by PG people, but
I have noticed than in the rpms of other distros the postgresql-devel rpms do not
include all the .h files necessary to build PG extensions. For instance the
rtree.h and itup.h and gist.h headers are missing. Could you please ensure that
all the headers are taken into account when you write your spec file.

May be also in the tar.gz or tar.bz2 distribution (bz2 is more effective than gz
and available on all platforms) you add a developer file that list all the
required headers, so that package builders know which files to include.

It seems that the rpm distributions will go as:
postgresql
postgresql-docs (user and manager docs)
postgresql-devel (header files and developper docs)

Cheers.
Franck@sopac.org


Sat, 07 Apr 2001 14:47:27 +1200
Franck Martin <franck@sopac.org>
Re: Re: RC3 ... and rpms...

> Thanks! I'm not too worried about 1.4.2, but be sure to let us know what
> the problem was; it may help out someone else...

NetBSD-1.4.2/i386 passes all tests with 7.1RC3.

My previous test failure on this platform was due to the timezone
information on the test system not being standard; once that was
corrected all tests pass.

It is still necessary to add -ltermcap after -ledit in
src/Makefile.global to have functional history editing in psql.

Regards,

Giles



Sat, 07 Apr 2001 14:43:39 +1000
Giles Lean <giles@nemeton.com.au>
Re: Call for platforms 

> Okay, here are my results:
> 
> Box 1: C180 (2.0 PA8000), HPUX 10.20
> 
> Compile with gcc: all tests pass
> Compile with cc: two lines of diffs in geometry (attached)
> 
> Box 2: 715/75 (1.1 PA7100LC), HPUX 10.20
> 
> Compile with gcc: all tests pass
> Compile with cc: all tests pass

I haven't had time to look at this further yet, except to build 7.1RC3
a couple of times with the HP ANSI C compiler today:

PA-RISC 1.1 code (-Ae +O2 +DAportable):   all tests pass
PA-RISC 2.0 code (-Ae +O2 +DA2.0 +DS2.0): geometry failures

I'm not sure how interesting these differences are anymore -- is there
anyone familiar enough with floating point to determine if the results
are acceptable (although currently unexpected :-) or not?

Regards,

Giles






Sat, 07 Apr 2001 15:24:05 +1000
Giles Lean <giles@nemeton.com.au>
Re: Call for platforms (HP-UX) 
Giles Lean <giles@nemeton.com.au> writes:
> I'm not sure how interesting these differences are anymore -- is there
> anyone familiar enough with floating point to determine if the results
> are acceptable (although currently unexpected :-) or not?

Differences in the last couple of decimal places in the geometry test
are definitely not a cause for worry.  Although we've tried to create
exact-match reference files for the most popular platforms, I think
that's largely an exercise in time-wasting.  Eventually we will figure
out a way to make the geometry output round off a few digits, and then
the cross-platform differences should mostly vanish.

			regards, tom lane

Sat, 07 Apr 2001 01:32:15 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Call for platforms (HP-UX) 
Hi

I've been running RC3 regression tests, starting with a FreeBSD 4.2-STABLE
and a Solaris 7 Sparc box. Both tests ran without any problems. I tried
Solaris 8 Sparc next: it still suffered from the same unix socket problems.
I had a look at the code and it seems to me that the use of unix sockets
in RC3 is still enabled, even though it (appearently) doesn't work reliably
on Solaris.

Since it was rather strange that RC3 did work correctly on Solaris 7 but
not 8, I also ran regression tests on another Solaris 7 and another 8 box,
with the same results. Since I still didn't trust it, I also ran RC1 again
on both Solaris 7 and 8; same result. And now things start getting weird.
A little more than a week ago the RC1 regression tests ran with on average
10-15 tests randomly failing. Now, however, I can run the regression several
times without any test failing. But if I run the regression test enough
times (4-6 times), I do have tests that fail (about 2-5). The configuration
of these servers hasn't changed in the last months and I used the same RC1
source and binaries.

Can somebody confirm whether pgsql Solaris does or does not work correctly
out-of-the-box? Disabling unix sockets will probably fix all these problems,
so I'm naturally wondering whether unix socket will or will not be disabled
in pgsql 7.1...

Regards,

Mathijs

Ps. Vince, could you remove test results 46 and 47? I don't trust them
anymore.
-- 
"A book is a fragile creature.  It suffers the wear of time,
 it fears rodents, the elements, clumsy hands." 
        Umberto Eco 

Sat, 7 Apr 2001 07:54:16 +0200
Mathijs Brands <mathijs@ilse.nl>
Call for platforms (Solaris)
Franck Martin wrote:
> I have no idea if what I say is true about the PG distribution by PG people, but
> I have noticed than in the rpms of other distros the postgresql-devel rpms do not
> include all the .h files necessary to build PG extensions. For instance the
> rtree.h and itup.h and gist.h headers are missing. Could you please ensure that
> all the headers are taken into account when you write your spec file.

The RPMs now (as of 7.1beta4) use the 'make install-all-headers'
incantation to generate the development headers.  If this doesn't get
the headers you need, then install-all-headers needsto be modified to
really install ALL headers.

With my latest RC3 RPM's (which I am preparing to upload to the
PostgreSQL ftp server sometime this morning, once I get some other
reorganization done and some contrib stuff built), I get the following
results:

[root@utility i386]# rpm -ql postgresql-devel|grep gist
/usr/include/pgsql/access/gist.h
/usr/include/pgsql/access/gistscan.h
/usr/include/pgsql/access/giststrat.h
[root@utility i386]# rpm -ql postgresql-devel|grep rtree
/usr/include/pgsql/access/rtree.h
[root@utility i386]# rpm -ql postgresql-devel|grep itup
/usr/include/pgsql/access/itup.h
[root@utility i386]#

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 07 Apr 2001 07:59:39 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: RC3 ... and rpms...
Thomas Lockhart wrote:
> 
> > > OTOH, if Marc was only thinking of removing the pre-built docs from the
> > > tarball, I don't object to that.  I'm not sure why those weren't
> > > distributed as separate tarballs from the get-go.  I just say that the
> > > doc sources are part of the source distribution...
> 
> >From the get-go, the docs were not, uh, useful docs. They have grown
> quite a bit from 1996 (with sources and formatting, probably by orders
> of magnitude).

Today's docs make even the docs of version 6.1.1 look pretty puny.

I'm looking at the package reorganization for the RPM's this morning --
we'll see what I find in a few hours.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 07 Apr 2001 08:04:30 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: RC3 ...
The Hermit Hacker wrote:
> Okay, unless someone can come up with a really good argument *for* why
> docs has to be included as part of the main tar file, I'm going to change
> the distributin generating script so that it generates a .src.tar.gz file
> seperate from the .doc.tar.gz file, which will make .src.tar.gz ~6Meg
> instead of the 8meg we are currently forcing ppl to download ...

> Peter E, is there anything part of the configure/make procedure that
> *requires* pgsql/doc to be there else it will break?  If so, can you
> possibly put it as a test "if docs exists, deal with it, else ignore"?

We're going to do this at this point in the release cycle?  IOW, is
there going to be an RC4 with this new packaging, or is the first-off
tarball with new packaging going to be the *final* 7.1 release *raised
eyebrow*?

I am certainly NOT opposed to doing this -- just questioning the timing.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 07 Apr 2001 08:09:55 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: RC3 ...
On Sat, 7 Apr 2001, Lamar Owen wrote:

> The Hermit Hacker wrote:
> > Okay, unless someone can come up with a really good argument *for* why
> > docs has to be included as part of the main tar file, I'm going to change
> > the distributin generating script so that it generates a .src.tar.gz file
> > seperate from the .doc.tar.gz file, which will make .src.tar.gz ~6Meg
> > instead of the 8meg we are currently forcing ppl to download ...
>
> > Peter E, is there anything part of the configure/make procedure that
> > *requires* pgsql/doc to be there else it will break?  If so, can you
> > possibly put it as a test "if docs exists, deal with it, else ignore"?
>
> We're going to do this at this point in the release cycle?  IOW, is
> there going to be an RC4 with this new packaging, or is the first-off
> tarball with new packaging going to be the *final* 7.1 release *raised
> eyebrow*?

there will be an RC4, I'm just waiting to hear back from Peter E as to
whether there is anything in the build process we even risk breaking ...
we've been doing the whole split thing for the past release or two as it
is (the FreeBSD ports collection using the individual packages instead of
the great big one) so from a packaging perspective, its well tested ...



Sat, 7 Apr 2001 10:39:46 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Re: RC3 ...
Franck Martin wrote:
> 
> I have no idea if what I say is true about the PG distribution by PG people, but
> I have noticed than in the rpms of other distros the postgresql-devel rpms do not
> include all the .h files necessary to build PG extensions. For instance the
> rtree.h and itup.h and gist.h headers are missing. Could you please ensure that
> all the headers are taken into account when you write your spec file.
> 
> May be also in the tar.gz or tar.bz2 distribution (bz2 is more effective than gz
> and available on all platforms) you add a developer file that list all the
> required headers, so that package builders know which files to include.

In my experience so far, it is also noticably slower than gzip. It does
work, and it is available. I have not yet been convinced that the space
savings is worth the time lost. But ISTM this is a minor point.

> It seems that the rpm distributions will go as:
> postgresql
> postgresql-docs (user and manager docs)
> postgresql-devel (header files and developper docs)

Actually, since you can suppress installation of the docs with --nodocs,
I would very much prefer to keep the html and text docs in the main RPM.
Otherwise I have two directories in /usr/doc for one software suite.

The 'hard copy' docs can go whereever they want as far as I'm concerned,
since I typically have little use for paper these days.

Of course, these are only my preferences, but it seems unlikely that the
assertions above are universally accepted  either.

-- 
Karl DeBisschop

Sat, 07 Apr 2001 09:49:03 -0400
Karl DeBisschop <karl@debisschop.net>
Re: Re: RC3 ... and rpms...
The Hermit Hacker wrote:
> there will be an RC4, I'm just waiting to hear back from Peter E as to

Good.

> whether there is anything in the build process we even risk breaking ...
> we've been doing the whole split thing for the past release or two as it
> is (the FreeBSD ports collection using the individual packages instead of
> the great big one) so from a packaging perspective, its well tested ...

Just not well-tested for the RPM build environment :-).
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 07 Apr 2001 09:56:04 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: RC3 ...
Karl DeBisschop wrote:
> In my experience so far, it is also noticably slower than gzip. It does
> work, and it is available. I have not yet been convinced that the space
> savings is worth the time lost. But ISTM this is a minor point.

The official tarball is gzipped -- the RPM will use that until bzipped
tarballs are official.
 
> Actually, since you can suppress installation of the docs with --nodocs,
> I would very much prefer to keep the html and text docs in the main RPM.
> Otherwise I have two directories in /usr/doc for one software suite.

The html docs at the very least will remain in the main package.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 07 Apr 2001 10:03:39 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: RC3 ... and rpms...
On Sat, 7 Apr 2001, Lamar Owen wrote:

> The Hermit Hacker wrote:
> > there will be an RC4, I'm just waiting to hear back from Peter E as to
>
> Good.
>
> > whether there is anything in the build process we even risk breaking ...
> > we've been doing the whole split thing for the past release or two as it
> > is (the FreeBSD ports collection using the individual packages instead of
> > the great big one) so from a packaging perspective, its well tested ...
>
> Just not well-tested for the RPM build environment :-).

Ya, but you could concievably test that now, without us doign an RC4 ..
the files are all there :)



Sat, 7 Apr 2001 11:11:17 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Re: RC3 ...
The Hermit Hacker wrote:
> On Sat, 7 Apr 2001, Lamar Owen wrote:
> > Just not well-tested for the RPM build environment :-).
 
> Ya, but you could concievably test that now, without us doign an RC4 ..
> the files are all there :)

So the structure isn't going to change -- just there's not going to be a
'whole thing' tarball anymore?  I am now confyoozzled.... :-)  If
there's still going to be a 'whole thing' tarball, I don't need to
change much.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 07 Apr 2001 10:44:53 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: RC3 ...
"Oliver Elphick" wrote:
  >Debian packages of 7.1RC3 have been uploaded to the Debian experimental
  >distribution and are also available at http://www.debian.org/~elphick/postgr
      >esql
  >
  >These packages are built for sid (Debian unstable); I am currently
  >trying to build a set for potato (stable).
 

Packages for potato are now available from the same URL

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "I sought the Lord, and he answered me; he delivered me
      from all my fears."       Psalm 34:4  



Sat, 07 Apr 2001 17:18:18 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Debian packages of 7.1RC3 
Thomas Lockhart writes:

> > > The docs are ready for shipment.
> > Even better ...
> > Okay, let's let this sit as RC3 for the next week...
>
> I'll go ahead and start generating hardcopy, though I understand that it
> is no longer allowed into the shipping tarball :(

I'm not speaking about "allowed", I'm merely talking about the state of
affairs since 7.0.  If people think that the postscript format should be
in the main tarball, then why not, but IIRC this question was raised last
time around and the decision went the other way.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 7 Apr 2001 19:18:25 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: RC3 ...
The Hermit Hacker writes:

> At 2Meg, is there a reason why we include any of the docs as part of the
> standard tar ball?  It shouldn't be required to compile, so should be able
> to be left out of the main tar ball and downloaded seperately as required
> .. thereby shrinking the distribution to <6Meg from its current 8 ...

For that purpose you introduced the split distribution.  If there is any
good reason for it, it's this.  Currently, the .docs sub-tarball contains
the entire doc/ subtree, the consequence of which is that this tarball is
required for a functioning installation.  If we were to change this split
so that doc/src/ is a separate sub-tarball, then that one could be purely
optional and you could tell people that they don't need it unless they
want to write documentation.

However, removing any part of the documentation, built or source, from the
full tarball seems like a really bad idea.  It breaks the fundamental
principle behind a "full tarball".  The resulting confusion would be
enormous.  Especially now that we seems to start getting some outside
documentation contributors.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 7 Apr 2001 19:24:10 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: RC3 ...
Bruce Momjian writes:

> Can we drop TODO.detail from the tarball too?  No need to include that,
> I think.  The web site has nice links to it now.  Uncompressed it is
> 1.314 megs.

You see where this discussion goes?  Do we want to go through each file
and argue whether it needs to be distributed?  If you're in that kind of
mood then you should use a binary package.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 7 Apr 2001 19:25:22 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: RC3 ...
Tom Lane writes:

> OTOH, if Marc was only thinking of removing the pre-built docs from the
> tarball, I don't object to that.  I'm not sure why those weren't
> distributed as separate tarballs from the get-go.  I just say that the
> doc sources are part of the source distribution...

Why would you want to remove the pre-built docs from the tarball and ship
the sources?

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 7 Apr 2001 19:26:37 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: RC3 ... 
The Hermit Hacker writes:

> Okay, unless someone can come up with a really good argument *for* why
> docs has to be included as part of the main tar file,

Because people want to read the documentation.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 7 Apr 2001 19:29:24 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: RC3 ...
Bruce Momjian writes:

> A major issue is that we don't regenerate docs for 7.1.1 or later, so

Sure we do.

> the 7.1 docs carry for all the 7.1.X releases.  That would seem to argue
> for a separate tarball for docs so people don't redownload the docs
> again for 7.1.1.
>
>

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 7 Apr 2001 19:30:37 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: RC3 ...
Lamar Owen writes:

> We're going to do this at this point in the release cycle?  IOW, is
> there going to be an RC4 with this new packaging, or is the first-off
> tarball with new packaging going to be the *final* 7.1 release *raised
> eyebrow*?
>
> I am certainly NOT opposed to doing this -- just questioning the timing.

I'm also questioning the timing and I am undoubtedly opposed to this.
We're talking about butchering up the released distribution less than a
week before publication.  We had a year to discuss this and various
questions about how to handle documentation building, distributions, and
installation were raised during that time.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 7 Apr 2001 19:33:38 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: RC3 ...
On Sat, 7 Apr 2001, Peter Eisentraut wrote:

> The Hermit Hacker writes:
>
> > Okay, unless someone can come up with a really good argument *for* why
> > docs has to be included as part of the main tar file,
>
> Because people want to read the documentation.

get postgresql.src.tar.gz
get postgresql.docs.tar.gz

instead of just

get postgresql.tar.gz

for those that want to download the docs, same amount of time ... for
those that don't want it, it sames them 2meg of download time ...

I'm curious as to how many ppl would actually download those docs ... I
know that I'd never do so, as I'm never on the same machine that the
server is running from, so just hit the web site ...

so, for those that do, we are giving them one extra step, and for those
that don't, saving them time and bandwidth ...


Sat, 7 Apr 2001 14:34:28 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: Re: RC3 ...
Giles Lean <giles@nemeton.com.au> writes:

> It is still necessary to add -ltermcap after -ledit in
> src/Makefile.global to have functional history editing in psql.

This is a weakness in the configure script: it goes through a loop
where it tries to link a program that calls readline() with, in order,
"-lreadline", "-lreadline -ltermcap", "-lreadline -lncurses",
"-lreadline -lcurses", "-ledit", "-ledit -ltermcap", "-ledit
-lncurses" and "-ledit -lcurses".  The first link that succeeds wil
determine which libraries are used.  However, on some platforms with
dynamic libraries, the link will succeed as soon as readline() is
present -- but the shared library that contains it doesn't contain a
complete specification of all other libraries it needs at run-time
(the executable is expected to hold this information), and the program
fails at run-time even though it linked without any error message.

I don't know how the situation could best be improved, though...

-tih
-- 
The basic difference is this: hackers build things, crackers break them.

07 Apr 2001 20:09:57 +0200
Tom Ivar Helbekkmo <tih@kpnQwest.no>
Re: Re: Call for platforms
The Hermit Hacker writes:

> On Sat, 7 Apr 2001, Peter Eisentraut wrote:
>
> > The Hermit Hacker writes:
> >
> > > Okay, unless someone can come up with a really good argument *for* why
> > > docs has to be included as part of the main tar file,
> >
> > Because people want to read the documentation.
>
> get postgresql.src.tar.gz
> get postgresql.docs.tar.gz
>
> instead of just
>
> get postgresql.tar.gz

But we already have a set of split distributions.  If you want to split it
in a different way, why not, but abolishing the full distribution is going
to seriously alienate use from the conventions used in open source land.

> I'm curious as to how many ppl would actually download those docs ...

This is the wrong question to ask.  The real question to me is:  If we do
this, how many people won't download the documentation, don't read it,
don't find it, spread the word that PostgreSQL is poorly documented, don't
use it correctly, spread the word that PostgreSQL isn't easy to use, and
take our time with avoidable mailing list traffic?  Also, how many people
will consequently not even get a chance to contribute to the
documentation?  People won't do two downloads for marginal benefit.

But let me ask you this:  If we split out the documentation, why stop
there?  Why not leave out pgtclsh, how many people need that?  Or what
about the JDBC driver sources?  People can download the pre-compiled jar
file.  These are in fact valid concerns, and they are adressed by the
split tarballs that we offer.  But there *must* be a full source tarball.
I cannot count on two hands the occasion where I had a source package
where people left out "non-essential" source files.  They lost me as a
contributor.  I'm not going to set up a CVS pull to fix a sentence.

> I
> know that I'd never do so, as I'm never on the same machine that the
> server is running from, so just hit the web site ...

I think you would be glad if more people read them locally and less people
hit the web. ;-)

> so, for those that do, we are giving them one extra step, and for those
> that don't, saving them time and bandwidth ...

People that want to save time and bandwidth use binary packages or the
split tarballs that we already have.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 7 Apr 2001 20:10:05 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: RC3 ...
The Hermit Hacker writes:

> those that don't want it, it sames them 2meg of download time ...

Another way to save at least 1 MB of download time would be bzip2'ed
tarballs.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 7 Apr 2001 20:19:32 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: RC3 ...
> Bruce Momjian writes:
> 
> > A major issue is that we don't regenerate docs for 7.1.1 or later, so
> 
> Sure we do.
> 
> > the 7.1 docs carry for all the 7.1.X releases.  That would seem to argue
> > for a separate tarball for docs so people don't redownload the docs
> > again for 7.1.1.

I didn't know that. I thought we genarated postscript only major
releases.  Do we regenerate HTML for subreleases?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 7 Apr 2001 14:25:29 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: RC3 ...
On Sat, 7 Apr 2001, Peter Eisentraut wrote:

> Thomas Lockhart writes:
> 
> > > > The docs are ready for shipment.
> > > Even better ...
> > > Okay, let's let this sit as RC3 for the next week...
> >
> > I'll go ahead and start generating hardcopy, though I understand that it
> > is no longer allowed into the shipping tarball :(
> 
> I'm not speaking about "allowed", I'm merely talking about the state of
> affairs since 7.0.  If people think that the postscript format should be
> in the main tarball, then why not, but IIRC this question was raised last
> time around and the decision went the other way.

Having had to d/l PG many times on many different machines, I'd be
delighted if it came w/o .ps docs, and w/o the doc sources (the number of
people who seem to be able to turn docbook into useful stuff seems to be
<< than people who can successful compile PG!).

It sounds like the separate-tgz for docs and for Postscript makes perfect
sense. Just make sure that it's *very* obvious where/how to get these, so
that the mailing lists are deluged w/ 'where are the docs'?

Just my .02,
-- 
Joel Burton   <jburton@scw.org>
Director of Information Systems, Support Center of Washington


Sat, 7 Apr 2001 14:30:36 -0400 (EDT)
Joel Burton <jburton@scw.org>
Re: RC3 ...
Tom Ivar Helbekkmo writes:

> Giles Lean <giles@nemeton.com.au> writes:
>
> > It is still necessary to add -ltermcap after -ledit in
> > src/Makefile.global to have functional history editing in psql.
>
> This is a weakness in the configure script: it goes through a loop
> where it tries to link a program that calls readline() with, in order,
> "-lreadline", "-lreadline -ltermcap", "-lreadline -lncurses",
> "-lreadline -lcurses", "-ledit", "-ledit -ltermcap", "-ledit
> -lncurses" and "-ledit -lcurses".  The first link that succeeds wil
> determine which libraries are used.  However, on some platforms with
> dynamic libraries, the link will succeed as soon as readline() is
> present -- but the shared library that contains it doesn't contain a
> complete specification of all other libraries it needs at run-time
> (the executable is expected to hold this information), and the program
> fails at run-time even though it linked without any error message.

On such a platform it would hardly be possible to detect anything with any
reliably.  A linker that links a program "succesfully" while the program
really needs more libraries to be runnable isn't very useful.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sat, 7 Apr 2001 21:01:37 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: Call for platforms
Peter Eisentraut <peter_e@gmx.net> writes:

> On such a platform it would hardly be possible to detect anything with any
> reliably.  A linker that links a program "succesfully" while the program
> really needs more libraries to be runnable isn't very useful.

You're right, of course -- it's a bug in the linkage loader on the
platform in question.  NetBSD/vax has it:

$ uname -a
NetBSD varg.i.eunet.no 1.5T NetBSD 1.5T (VARG) #4: Thu Apr 5 23:38:04 CEST 2001
     root@varg.i.eunet.no:/usr/src/sys/arch/vax/compile/VARG vax
$ cat > foo.c
int main (int argc, char **argv) { readline(); }
$ cc -o foo foo.c
/tmp/ccFTO4Mu.o: Undefined symbol `_readline'referenced from text segment
collect2: ld returned 1 exit status
$ cc -o foo foo.c -ledit
$ echo $?
0
$ ./foo
/usr/libexec/ld.so: Undefined symbol "_tputs"in foo:/usr/lib/libedit.so.2.5
$ echo $?
1
$ ldd foo
foo:
        -ledit.2 => /usr/lib/libedit.so.2.5 (0x181b000)
        -lc.12 => /usr/lib/libc.so.12.74 (0x182d000)
$

-tih
-- 
The basic difference is this: hackers build things, crackers break them.

07 Apr 2001 21:41:04 +0200
Tom Ivar Helbekkmo <tih@kpnQwest.no>
Re: Re: Call for platforms
One quick note -- since 'R' < 'b', the RC RPM's must be forced to
install with --oldpackage, as RPM does a simple strcmp of version
numbers -- 7.1RC3 < 7.1beta1, for instance.  Just force it with
--oldpackage if you have a 7.1beta RPM already installed.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 07 Apr 2001 23:30:07 -0400
Lamar Owen <lamar.owen@wgcr.org>
RPM upgrade caveats going from a beta version to RC
Karl DeBisschop wrote:
> Actually, since you can suppress installation of the docs with --nodocs,
> I would very much prefer to keep the html and text docs in the main RPM.
> Otherwise I have two directories in /usr/doc for one software suite.

I'm researching how to get a subpackage to place docs in the main
package %doc.

HTML docs and man pages are with the main package; SGML source and any
hardcopy docs will go into the docs subpackage. The contrib tree is
getting its own subpackage.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 07 Apr 2001 23:43:00 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Re: RC3 ... and rpms...
On Sat, 7 Apr 2001, Lamar Owen wrote:

> One quick note -- since 'R' < 'b', the RC RPM's must be forced to
> install with --oldpackage, as RPM does a simple strcmp of version
> numbers -- 7.1RC3 < 7.1beta1, for instance.  Just force it with
> --oldpackage if you have a 7.1beta RPM already installed.

Huh?  I always thought that ASCII R was greater then b ... *confused*  in
the future, would it help to have 7.2Beta?  Or am I missing something? :)



Sun, 8 Apr 2001 03:08:32 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: RPM upgrade caveats going from a beta version to RC
The Hermit Hacker wrote:
  >On Sat, 7 Apr 2001, Lamar Owen wrote:
  >
  >> One quick note -- since 'R' < 'b', the RC RPM's must be forced to
  >> install with --oldpackage, as RPM does a simple strcmp of version
  >> numbers -- 7.1RC3 < 7.1beta1, for instance.  Just force it with
  >> --oldpackage if you have a 7.1beta RPM already installed.
  >
  >Huh?  I always thought that ASCII R was greater then b ... *confused*  in
  >the future, would it help to have 7.2Beta?  Or am I missing something? :)

R = 82
b = 98

so b comes after R, and `blank' comes before either!

Therefore 7.1 < 7.1RC < 7.1beta !

As I suggested in another mail, let us switch to using even minor
numbers for releases and odd ones for development:

That means the final release of 7.1 will be called 7.2.  Bugfix releases
will then be 7.2.x.  Meanwhile new development versions will be 7.3.x
which will finally be released as 7.4, and so on...

For current 7.1, the Debian releases are 7.1beta, 7.1cRC, 7.1final,
which is both cumbersome and confusing to those who are looking for
an exact match.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Do not be anxious about anything, but in everything,  
      by prayer and supplication, with thanksgiving, present
      your requests to God. And the peace of God, which  
      transcends all understanding, will guard your hearts  
      and your minds in Christ Jesus."   Philippians 4:6,7  



Sun, 08 Apr 2001 08:43:17 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: RPM upgrade caveats going from a beta version to RC 
The Hermit Hacker writes:

> On Sat, 7 Apr 2001, Lamar Owen wrote:
>
> > One quick note -- since 'R' < 'b', the RC RPM's must be forced to
> > install with --oldpackage, as RPM does a simple strcmp of version
> > numbers -- 7.1RC3 < 7.1beta1, for instance.  Just force it with
> > --oldpackage if you have a 7.1beta RPM already installed.
>
> Huh?  I always thought that ASCII R was greater then b ... *confused*  in
> the future, would it help to have 7.2Beta?  Or am I missing something? :)

How about 7.2rc1, which is greater than 7.2beta1.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sun, 8 Apr 2001 12:21:23 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: RPM upgrade caveats going from a beta version to RC
On Sun, 8 Apr 2001, Oliver Elphick wrote:

> The Hermit Hacker wrote:
>   >On Sat, 7 Apr 2001, Lamar Owen wrote:
>   >
>   >> One quick note -- since 'R' < 'b', the RC RPM's must be forced to
>   >> install with --oldpackage, as RPM does a simple strcmp of version
>   >> numbers -- 7.1RC3 < 7.1beta1, for instance.  Just force it with
>   >> --oldpackage if you have a 7.1beta RPM already installed.
>   >
>   >Huh?  I always thought that ASCII R was greater then b ... *confused*  in
>   >the future, would it help to have 7.2Beta?  Or am I missing something? :)
>
> R = 82
> b = 98
>
> so b comes after R, and `blank' comes before either!
>
> Therefore 7.1 < 7.1RC < 7.1beta !
>
> As I suggested in another mail, let us switch to using even minor
> numbers for releases and odd ones for development:
>
> That means the final release of 7.1 will be called 7.2.  Bugfix releases
> will then be 7.2.x.  Meanwhile new development versions will be 7.3.x
> which will finally be released as 7.4, and so on...

Not in this life time ... we are not going to move to a Linux-like
development cycle ... *groan*


Sun, 8 Apr 2001 09:27:13 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: RPM upgrade caveats going from a beta version to RC 
The Hermit Hacker wrote:or development:
  >>
  >> That means the final release of 7.1 will be called 7.2.  Bugfix releases
  >> will then be 7.2.x.  Meanwhile new development versions will be 7.3.x
  >> which will finally be released as 7.4, and so on...
  >
  >Not in this life time ... we are not going to move to a Linux-like
  >development cycle ... *groan*
  >

Harrumph!!

Well, pick some scheme that gives a rational set of numbers for
distributions.  The current one is only good for installation
by hand!

(Mind you, my other major package is progressing from -1.00 to 0,
so that -0.76 is followed by -0.75.  Not that I recommend you to 
follow _that_ example.)

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Do not be anxious about anything, but in everything,  
      by prayer and supplication, with thanksgiving, present
      your requests to God. And the peace of God, which  
      transcends all understanding, will guard your hearts  
      and your minds in Christ Jesus."   Philippians 4:6,7  



Sun, 08 Apr 2001 15:14:49 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: RPM upgrade caveats going from a beta version to RC 
Bruce Momjian writes:

> I didn't know that. I thought we genarated postscript only major
> releases.  Do we regenerate HTML for subreleases?

The HTML is generated every 12 hours, and whenever a distribution is
wrapped up it picks up the latest bundle.  This will probably have to be
sorted out again when a branch is made so the paths are set correctly, but
in principle it is trivial to arrange.  Not that the documentation ever
changes for minor releases.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Sun, 8 Apr 2001 18:58:57 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Re: RC3 ...
On Sun, 8 Apr 2001, Oliver Elphick wrote:

> The Hermit Hacker wrote:or development:
>   >>
>   >> That means the final release of 7.1 will be called 7.2.  Bugfix releases
>   >> will then be 7.2.x.  Meanwhile new development versions will be 7.3.x
>   >> which will finally be released as 7.4, and so on...
>   >
>   >Not in this life time ... we are not going to move to a Linux-like
>   >development cycle ... *groan*
>   >
>
> Harrumph!!
>
> Well, pick some scheme that gives a rational set of numbers for
> distributions.  The current one is only good for installation by hand!

We do, we follow the scheme as used by ... the BSD camp :)  Be thankful we
don't go all the way and use 7.2-RELEASE too :)



Sun, 8 Apr 2001 14:41:05 -0300 (ADT)
The Hermit Hacker <scrappy@hub.org>
Re: RPM upgrade caveats going from a beta version to RC 
The Hermit Hacker wrote:
> We do, we follow the scheme as used by ... the BSD camp :)  Be thankful we
> don't go all the way and use 7.2-RELEASE too :)

If we had 7.1-CURRENT, 7.1-RELEASE, and 7.1-STABLE, the versioning
comparision would be just fine -- better than now.  As it stands, an
upgrade from 7.1beta6 to 7.1RC4 and from 7.1RC4 to 7.1 is in the eyes of
at least two packaging systems a downgrade.

However, 7.1beta6 to 7.1rc4 to 7.1.0 would be an ok progression, as 7.1
< 7.1.0, I think (saying that without having tested it could be
dangerous.... :-)).

Although I must observe that if RPM used the system's locale in
determining version collation, 7.1RC4 would be greater than 7.1beta6 --
which collation breaks our indexing and our LIKE optimizations, and
breaks our regression tests. :-)  But 7.1 would still be a downgrade
based on that.

Red Hat uses a different system for their betas -- which I'm not
necessarily advocating, just presenting:

The public RedHat beta, IIRC, for what may or may not become Red Hat 7.1
carried a version of 7.0.91.

But then again the Linux kernel did that as well, going from 0.13 or so
to 0.97 (and various pl numbers) before hitting 1.0.

And just _why_ are you so adversarial to the Linux version numbering? 
After all, it's just another system..... (Rhetorical question -- I
already know the answer) :-)

Personally, I think the Linux versioning is overkill, and prefer the BSD
way of labeling versions.  But that is just my personal opinion.

But even at that, the Linux and BSD versioning is designed more for
carrying concurrent STABLE and CURRENT versions -- we don't really have
_that_ much version overlap to deal with, do we?  Debian does as much --
but it is again a matter of version concurrency -- we're not likely to
release a 7.1.0 then a 7.0.4 that fixes bugs in the STABLE branch,
whereas at one point Linux 2.0.39, a 2.2.x, and 2.4.0 were being
released concurrently.  The same happens with FreeBSDand others -- but
not us.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sun, 08 Apr 2001 20:55:06 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: RPM upgrade caveats going from a beta version to RC
Oliver Elphick wrote:

> R = 82
> b = 98

This is a very small problem of having capital R and lowercase b that I
believe can be taken into account in the development of 7.2.

> As I suggested in another mail, let us switch to using even minor
> numbers for releases and odd ones for development:

It's a Linux-ism that personally I don't like. You have to be familiar
with the project to understand that 8.3.3 is not better for general use
than 8.2.4 because 8.2 is "stable" and 8.3 is "development".

-- 
Alessio F. Bragadini		alessio@albourne.com
APL Financial Services		http://village.albourne.com
Nicosia, Cyprus		 	phone: +357-2-755750

"It is more complicated than you think"
		-- The Eighth Networking Truth from RFC 1925

Mon, 09 Apr 2001 09:56:57 +0300
Alessio Bragadini <alessio@albourne.com>
Re: RPM upgrade caveats going from a beta version to RC
Lamar Owen writes:

> One quick note -- since 'R' < 'b', the RC RPM's must be forced to
> install with --oldpackage, as RPM does a simple strcmp of version
> numbers -- 7.1RC3 < 7.1beta1, for instance.  Just force it with
> --oldpackage if you have a 7.1beta RPM already installed.

Btw., are you aware of the 'serial' tag to override the version guessing
mechanism?

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Wed, 11 Apr 2001 19:43:44 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: RPM upgrade caveats going from a beta version to RC
Peter Eisentraut wrote:
> 
> Lamar Owen writes:
> 
> > One quick note -- since 'R' < 'b', the RC RPM's must be forced to
> > install with --oldpackage, as RPM does a simple strcmp of version
> > numbers -- 7.1RC3 < 7.1beta1, for instance.  Just force it with
> > --oldpackage if you have a 7.1beta RPM already installed.
> 
> Btw., are you aware of the 'serial' tag to override the version guessing
> mechanism?

Yes, I am, actually.  But it seems a broken way of dealing with it. 
Although I do have another idea, thanks to Trond. Rather than package
'7.1RC4-1' I could package '7.1-0.1RC4' -- giving a straight
versioning.  I could progress from '7.1-0.1beta1.1' through
'7.1-0.1beta6.2' through '7.1-0.2RC1.1' to '7.1-1'.

Last time I looked at the documentation for the serial tag, its use was
strongly discouraged.  But that _has_ been awhile -- maybe it could be
useful. But I would prefer the whole version numbering thingtobe fixed,
as the Debian packages have the same issue -- and I don't know if .deb
has an analog to Serial:.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Wed, 11 Apr 2001 16:30:02 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: RPM upgrade caveats going from a beta version to RC
Lamar Owen wrote:
  >Last time I looked at the documentation for the serial tag, its use was
  >strongly discouraged.  But that _has_ been awhile -- maybe it could be
  >useful. But I would prefer the whole version numbering thingtobe fixed,
  >as the Debian packages have the same issue -- and I don't know if .deb
  >has an analog to Serial:.

We have epochs, that is, the package version is preceded by an integer
and a colon, which overrides every other part of the version and release
number.  However, if I ever use an epoch, I will be stuck with epochs for ever;
so I don't want to start.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Is any one of you in trouble? He should pray. Is
      anyone happy? Let him sing songs of praise. Is any one
      of you sick? He should call the elders of the church
      to pray over him...The prayer of a righteous man is
      powerful and effective."         James 5:13,14,16 



Wed, 11 Apr 2001 21:55:46 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: RPM upgrade caveats going from a beta version to RC 
Oliver Elphick wrote:
> Lamar Owen wrote:
>   >as the Debian packages have the same issue -- and I don't know if .deb
>   >has an analog to Serial:.
 
> We have epochs, that is, the package version is preceded by an integer
> and a colon, which overrides every other part of the version and release
> number.  However, if I ever use an epoch, I will be stuck with epochs for ever;
> so I don't want to start.

RPM also has the epoch mechanism -- and it sounds just like what you
have just described. Not something I want to start using, either. It's
more like a 'super-major' version number than the RPM serial mechanism,
which works in a more broken fashion.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Wed, 11 Apr 2001 17:01:46 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: RPM upgrade caveats going from a beta version to RC
Lamar Owen writes:

> Yes, I am, actually.  But it seems a broken way of dealing with it.
> Although I do have another idea, thanks to Trond. Rather than package
> '7.1RC4-1' I could package '7.1-0.1RC4' -- giving a straight
> versioning.  I could progress from '7.1-0.1beta1.1' through
> '7.1-0.1beta6.2' through '7.1-0.2RC1.1' to '7.1-1'.

Just name them

7.1betax
7.1rcx
7.1.0
7.1.1
etc.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Thu, 12 Apr 2001 00:18:56 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: RPM upgrade caveats going from a beta version to RC
Peter Eisentraut wrote:
> Lamar Owen writes:
> > Yes, I am, actually.  But it seems a broken way of dealing with it.
> > Although I do have another idea, thanks to Trond. Rather than package
> > '7.1RC4-1' I could package '7.1-0.1RC4' -- giving a straight
> > versioning.  I could progress from '7.1-0.1beta1.1' through
> > '7.1-0.1beta6.2' through '7.1-0.2RC1.1' to '7.1-1'.

> Just name them

> 7.1betax
> 7.1rcx
> 7.1.0
> 7.1.1

And I like that -- but that would be Marc's (and the core group)
decision to make not mine. If the current schema is continued, I can
work around it --but it would be nice if the version numbering could be
more packager-friendly.  I have a real aversion to naming the RPM
version number differently from the main package version.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Wed, 11 Apr 2001 20:34:44 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: RPM upgrade caveats going from a beta version to RC
Did we decide that "most NetBSD/i386 users have fpus" in which case Marko's
patch should be applied?

Cheers,

Patrick
(just checked, it isn't in today's cvs)


On Thu, Mar 22, 2001 at 10:27:44PM +0200, Marko Kreen wrote:
> On Thu, Mar 22, 2001 at 07:58:04PM +0000, Patrick Welche wrote:
> > On Fri, Mar 23, 2001 at 06:25:50AM +1100, Giles Lean wrote:
> > > 
> > > > PS: AFAIK geometry-positive-zeros-bsd works for all NetBSD platforms - the
> > > > above difference is only for i386 + fpu.
> > > 
> > > It doesn't on NetBSD-1.5/alpha -- there geometry-positive-zeros is
> > > correct.
> > 
> > Sorry, that should have read:
> > 
> > AFAIK geometry-positive-zeros works for all NetBSD platforms - the
> > above difference is only for i386 + fpu.
> 
> Seems that following patch is needed.  Now It Works For Me (tm).
> Giles, does the regress test now succed for you?
> 
> -- 
> marko
> 
> 
> Index: src/test/regress/resultmap
> ===================================================================
> RCS file: /home/projects/pgsql/cvsroot/pgsql/src/test/regress/resultmap,v
> retrieving revision 1.45
> diff -u -r1.45 resultmap
> --- src/test/regress/resultmap	2001/03/22 15:13:18	1.45
> +++ src/test/regress/resultmap	2001/03/22 17:29:49
> @@ -17,6 +17,7 @@
>  geometry/.*-openbsd=geometry-positive-zeros-bsd
>  geometry/.*-irix6=geometry-irix
>  geometry/.*-netbsd=geometry-positive-zeros
> +geometry/i.86-.*-netbsdelf1.5=geometry-positive-zeros-bsd
>  geometry/.*-sysv5uw7.*:cc=geometry-uw7-cc
>  geometry/.*-sysv5uw7.*:gcc=geometry-uw7-gcc
>  geometry/alpha.*-dec-osf=geometry-alpha-precision
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster

Thu, 12 Apr 2001 17:15:52 +0100
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: Call for platforms
> Did we decide that "most NetBSD/i386 users have fpus" in which case Marko's
> patch should be applied?

I'm unclear on what y'all mean by "i386 + fpu", especially since NetBSD
seems to insist on calling every Intel processor a "i386". In this case,
are you suggesting that this patch covers all NetBSD installations on
every Intel processor from i386 + fpu forward to i486, i586, etc etc? Or
is this specifically for the i386 with the 80387 coprocessor which is
how any reasonable person would interpret "i386+fpu"? ;)

                           - Thomas

> > Index: src/test/regress/resultmap
> > ===================================================================
> > RCS file: /home/projects/pgsql/cvsroot/pgsql/src/test/regress/resultmap,v
> > retrieving revision 1.45
> > diff -u -r1.45 resultmap
> > --- src/test/regress/resultmap        2001/03/22 15:13:18     1.45
> > +++ src/test/regress/resultmap        2001/03/22 17:29:49
> > @@ -17,6 +17,7 @@
> >  geometry/.*-openbsd=geometry-positive-zeros-bsd
> >  geometry/.*-irix6=geometry-irix
> >  geometry/.*-netbsd=geometry-positive-zeros
> > +geometry/i.86-.*-netbsdelf1.5=geometry-positive-zeros-bsd
> >  geometry/.*-sysv5uw7.*:cc=geometry-uw7-cc
> >  geometry/.*-sysv5uw7.*:gcc=geometry-uw7-gcc
> >  geometry/alpha.*-dec-osf=geometry-alpha-precision

Fri, 13 Apr 2001 13:25:45 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Call for platforms
On Fri, Apr 13, 2001 at 01:25:45PM +0000, Thomas Lockhart wrote:
> > Did we decide that "most NetBSD/i386 users have fpus" in which case Marko's
> > patch should be applied?
> 
> I'm unclear on what y'all mean by "i386 + fpu", especially since NetBSD
> seems to insist on calling every Intel processor a "i386".

History ;-)

> In this case,
> are you suggesting that this patch covers all NetBSD installations on
> every Intel processor from i386 + fpu forward to i486, i586, etc etc?

Yes! It's simply, if the peecee type thing has a fpu (as in the sysctl
machdep.fpu_present returns 1), then libm387.so is used, and you get
differences in the (from memory 44th insignificant figure?) otherwise it
just uses libm.so and you get what is currently correct in resultmap.

Cheers,

Patrick

Fri, 13 Apr 2001 17:48:51 +0100
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: Call for platforms
my original CIDR type implementation used BIND's inet_ntop() and inet_pton()
which therefore included latent support for ipv6.  it wouldn't take a huge
amount of effort to bring this back, would it?

(the user below is using VARCHAR for his ip addresses for this reason.)

------- Forwarded Message

Date: Fri, 20 Apr 2001 08:37:22 -0700 (PDT)
Message-Id: <200104201537.IAA26178@gulag.araneus.fi>
To: Paul A Vixie <vixie@mfnx.net>
Subject: Re: Appliance caching server configuration database schema 
In-Reply-To: <200104200314.UAA73417@redpaul.mfnx.net>
From: gson@nominum.com (Andreas Gustafsson)

Paul A. Vixie writes:
> you can use INET or CIDR for your addresses since this is postgres.

I would if it supported IPv6 addresses.

------- End of Forwarded Message


Sat, 21 Apr 2001 10:27:19 -0700
Paul A Vixie <vixie@mfnx.net>
well, now i wish we hadn't gutted the ipv6 support
Paul A Vixie <vixie@mfnx.net> writes:
> my original CIDR type implementation used BIND's inet_ntop() and inet_pton()
> which therefore included latent support for ipv6.  it wouldn't take a huge
> amount of effort to bring this back, would it?

AFAIK we never actually *had* IPV6 support in those datatypes, only
stubs for it.  A patch to bring it up to full speed would be gladly
accepted...

			regards, tom lane

Sat, 21 Apr 2001 23:28:02 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: well, now i wish we hadn't gutted the ipv6 support 
> AFAIK we never actually *had* IPV6 support in those datatypes, only
> stubs for it.

the inet_net_pton implementation that was brought in from BIND had its
IPv6 portions scrubbed.  micro-over-optimization of the contributed
"bitncmp" caused the "ipv4 as int" assumption to reoccur.  i'm going to
have to put it back to BIND-standard as much as possible.  presumably
as long as the on-disk format is compatible (such that old databases can
be both read and written by the new code) none of that will be objectionable.

> A patch to bring it up to full speed would be gladly accepted...

thanks for the invitation, i'll start work on it right now.


Sat, 21 Apr 2001 20:48:13 -0700
Paul A Vixie <vixie@mfnx.net>
Re: well, now i wish we hadn't gutted the ipv6 support 
[apologies if this appears twice; I thought I had sent it but it hasn't
appeared anywhere]
The attached patch implements a method of connection authentication for
Unix sockets that support SCM_CREDENTIALS.  This includes Linux kernels
2.2 and 2.4 at least; I don't know what other implementations support
it.

Since it is not universally supported, I have included a configure test. 
autoconf needs to be run after installing the patch.

This patch provides a new authentication method "peer" for use with
"local" connections; otherwise it works exactly like the "ident" method.

Please consider including this in PostgreSQL.



Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Rejoice with them that do rejoice, and weep with them 
      that weep."            Romans 12:15
Thu, 03 May 2001 10:28:39 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Unix sockets connection authentication - patch
Oliver Elphick writes:

> Since it is not universally supported, I have included a configure test.
> autoconf needs to be run after installing the patch.

You don't need Autoconf tests for cpp symbols.  You can just write #ifdef
WEIRD_SYMBOL in the code.

Btw., never ever use AC_EGREP_*.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Thu, 3 May 2001 16:04:59 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Unix sockets connection authentication - patch
> [apologies if this appears twice; I thought I had sent it but it hasn't
> appeared anywhere]
> The attached patch implements a method of connection authentication for
> Unix sockets that support SCM_CREDENTIALS.  This includes Linux kernels
> 2.2 and 2.4 at least; I don't know what other implementations support
> it.

Are SCM_CREDENTIALS supported by some standard?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 3 May 2001 11:19:29 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Unix sockets connection authentication - patch
Bruce Momjian wrote:
  >> The attached patch implements a method of connection authentication for
  >> Unix sockets that support SCM_CREDENTIALS.  This includes Linux kernels
  >> 2.2 and 2.4 at least; I don't know what other implementations support
  >> it.
  >
  >Are SCM_CREDENTIALS supported by some standard?

I don't know if there is a standard.  I've done a search on Google - it
seems to have been invented by Sun and implemented in newer BSD as well
as Linux.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Rejoice with them that do rejoice, and weep with them 
      that weep."            Romans 12:15 



Thu, 03 May 2001 17:22:08 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Unix sockets connection authentication - patch 
Hi all,

	Is there anyway to get the debug (-d2) log files to mark each transaction
with a unique ID. We're trying to debug dead locks and the transactions seem
to be mixed together somewhat.

Thanks,

--Rainer


Fri, 4 May 2001 14:16:39 +0900
"Rainer Mager" <rmager@vgkk.com>
log files
"Rainer Mager" <rmager@vgkk.com> writes:
> 	Is there anyway to get the debug (-d2) log files to mark each transaction
> with a unique ID.

Not per-transaction, but there's an option to include the backend PID,
which should help.

			regards, tom lane

Fri, 04 May 2001 09:51:07 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: log files 

Not sure what to do with this.  Our authentication options are already
pretty complicated, and I hate to add a new one that no one is really
sure about its portability or usefulness.


> [apologies if this appears twice; I thought I had sent it but it hasn't
> appeared anywhere]
> The attached patch implements a method of connection authentication for
> Unix sockets that support SCM_CREDENTIALS.  This includes Linux kernels
> 2.2 and 2.4 at least; I don't know what other implementations support
> it.
> 
> Since it is not universally supported, I have included a configure test. 
> autoconf needs to be run after installing the patch.
> 
> This patch provides a new authentication method "peer" for use with
> "local" connections; otherwise it works exactly like the "ident" method.
> 
> Please consider including this in PostgreSQL.
> 

Content-Description: p.diff

[ Attachment, skipping... ]

> Oliver Elphick                                Oliver.Elphick@lfix.co.uk
> Isle of Wight                              http://www.lfix.co.uk/oliver
> PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
> GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
>                  ========================================
>      "Rejoice with them that do rejoice, and weep with them 
>       that weep."            Romans 12:15 

> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 8 May 2001 14:10:12 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Unix sockets connection authentication - patch
Hi all,

	We have an application that runs on both Postgres and Oracle. One problem
we've been facing as maintaining the the installed/default database for the
application. Once it is up and running, things are fine, but since we
primarily develop on Postgres we sometimes hit problems when it is time to
convert all of our work to Oracle. I was wondering if anyone knows of any
tools that take a Postgres dump and convert it to something Oracle can
accept?

	Thanks,

--Rainer


Mon, 14 May 2001 07:55:38 +0900
"Rainer Mager" <rmager@vgkk.com>
Postgres <-> Oracle
I can't see any reason in the code why this should be happening.

------- Forwarded Message

Date:    Thu, 24 May 2001 00:25:32 -0400
From:    Marc Sherman <msherman@projectile.ca>
To:      Debian Bug Tracking System <submit@bugs.debian.org>
Subject: Bug#98565: postgresql logs notices with GMT timestamps in syslog

Package: postgresql
Version: 7.1.1-3
Severity: normal

Notices are being timestamped in GMT in the syslog, instead of local
time like all other log entries.  Here's a fragment from my syslog:

May 23 23:17:30 projectile postgres[1035]: [1] DEBUG:  connection: host=[local]
user=www-data database=inwopc
May 24 03:17:30 projectile postgres[1035]: [2] NOTICE:  Adding missing FROM-cla
u
se entry for table "games"
May 23 23:17:30 projectile apache: NOTICE:  Adding missing FROM-clause entry fo
r
 table "games"

All three entries are from the same connection.  The second entry is
the notice being logged by postgres, while the third entry (with the
local timestamp) is that same notice being logged by apache (php).

- - Marc

- -- System Information
Debian Release: testing/unstable
Architecture: i386
Kernel: Linux projectile 2.2.19 #1 Mon May 7 09:24:37 EDT 2001 i586

Versions of packages postgresql depends on:
ii  debconf                       0.9.41     Debian configuration management sy
ii  debianutils                   1.15       Miscellaneous utilities specific t
ii  libc6                         2.2.3-1    GNU C Library: Shared libraries an
ii  libpgsql2.1                   7.1.1-3    Shared library libpq.so.2.1 for Po
ii  libreadline4                  4.2-3      GNU readline and history libraries
ii  libssl0.9.6                   0.9.6-2    SSL shared libraries              
ii  postgresql-client             7.1.1-3    Front-end programs for PostgreSQL 
ii  procps                        1:2.0.7-4  The /proc file system utilities.  
ii  zlib1g                        1:1.1.3-15 compression library - runtime     


------- End of Forwarded Message


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "I will praise thee; for I am fearfully and wonderfully 
      made..."       Psalms 139:14 



Thu, 24 May 2001 09:30:05 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Bug#98565: postgresql logs notices with GMT timestamps in syslog
 (fwd)
Since there is no plan yet how to do a wholesale overhaul of the ACL
system, I'd like to stick a few improvements into the current
implementation:

* Make DELETE distinct from UPDATE privilege

* rename the internal representation: s = select, i = insert, u = update,
  d = delete, R = rules

* LOCK > AccessShare will require UPDATE or DELETE.  This is not a change
  in effect.

* Sequence nextval and setval will require UPDATE; DELETE won't do any
  longer.

* COPY FROM will require INSERT privilege.  It used to require
  UPDATE/DELETE, it think that is not correct..

* INSERT (the command) will require INSERT privilege.  UPDATE/DELETE won't
  do any longer.  (Why was this there?)

* Implement SQL REFERENCES privilege:  grant references on A to B will
  allow user B to create a foreign key referencing table A as primary key.

I'd also like to create a regression test.  That will require creating
some global users and groups in the installation where the test runs.  I
think as long as we name them "regressuser1", "regressgroup2", etc. this
won't harm anyone.

Comments?

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Thu, 24 May 2001 13:07:24 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Smaller access privilege changes
Peter Eisentraut <peter_e@gmx.net> writes:
> * Make DELETE distinct from UPDATE privilege

Okay.

> * rename the internal representation: s = select, i = insert, u = update,
>   d = delete, R = rules

Since the internal representation is visible to users, I fear that a
wholesale renaming will break existing applications.  Can we make this
part of the change less intrusive?

> * COPY FROM will require INSERT privilege.  It used to require
>   UPDATE/DELETE, it think that is not correct..
> * INSERT (the command) will require INSERT privilege.  UPDATE/DELETE won't
>   do any longer.  (Why was this there?)

Both of these are basically there because the underlying model is "read
and write", with "append" as a limited form of "write"; so "write"
allows everything that "append" does.  But if we are switching to a full
"insert/update/delete" model then this behavior should go away.

> * Implement SQL REFERENCES privilege:  grant references on A to B will
>   allow user B to create a foreign key referencing table A as primary key.

Which privilege will SELECT FOR UPDATE require, and how do you plan to
get the system to distinguish users' SELECT FOR UPDATE from the commands
issued by the foreign key triggers?

> I'd also like to create a regression test.  That will require creating
> some global users and groups in the installation where the test runs.  I
> think as long as we name them "regressuser1", "regressgroup2", etc. this
> won't harm anyone.

Seems reasonable, but be careful to cope with the case where these
objects already exist from a prior regression run.

			regards, tom lane

Thu, 24 May 2001 07:49:03 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Smaller access privilege changes 
"Oliver Elphick" <olly@lfix.co.uk> writes:
> Notices are being timestamped in GMT in the syslog, instead of local
> time like all other log entries.  Here's a fragment from my syslog:

Curious.  I always assumed that syslog timestamps were supplied by the
syslog daemon, but to make this happen they'd have to be supplied in the
syslog client process (viz. the Postgres process).  What timezone is the
Postgres backend being run in, and is it different from all the other
syslog clients on the system?

			regards, tom lane

Thu, 24 May 2001 08:03:30 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Bug#98565: postgresql logs notices with GMT timestamps in syslog
	(fwd)
Tom Lane writes:

> > * rename the internal representation: s = select, i = insert, u = update,
> >   d = delete, R = rules
>
> Since the internal representation is visible to users, I fear that a
> wholesale renaming will break existing applications.  Can we make this
> part of the change less intrusive?

I guess so.  I could make r=select, a=insert, w=update, d=delete, R=rules,
x=reference.  Of course we will have to break this eventually, but we
might as well put it off until then.

> > * Implement SQL REFERENCES privilege:  grant references on A to B will
> >   allow user B to create a foreign key referencing table A as primary key.
>
> Which privilege will SELECT FOR UPDATE require, and how do you plan to
> get the system to distinguish users' SELECT FOR UPDATE from the commands
> issued by the foreign key triggers?

The REFERENCES privilege will be checked by CREATE TABLE and ALTER TABLE
(where it currently says "must be owner").  SELECT FOR UPDATE is not
affected by this and will stay the way it is.

> > I'd also like to create a regression test.  That will require creating
> > some global users and groups in the installation where the test runs.  I
> > think as long as we name them "regressuser1", "regressgroup2", etc. this
> > won't harm anyone.
>
> Seems reasonable, but be careful to cope with the case where these
> objects already exist from a prior regression run.

I drop them at the end of the test.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Thu, 24 May 2001 14:37:09 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Smaller access privilege changes 
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>>> * rename the internal representation: s = select, i = insert, u = update,
>>> d = delete, R = rules
>> 
>> Since the internal representation is visible to users, I fear that a
>> wholesale renaming will break existing applications.  Can we make this
>> part of the change less intrusive?

> I guess so.  I could make r=select, a=insert, w=update, d=delete, R=rules,
> x=reference.  Of course we will have to break this eventually, but we
> might as well put it off until then.

My thought exactly.  If we were going straight to full SQL compliance
then I wouldn't worry, but I don't like the idea of breaking apps now
and then breaking them some more later.

A different tack is to go ahead and make the change now, but try to
ensure we won't have to change the coding again when we do the rest of
the SQL protection model.  Do you know what is still missing given this
change?

>> Seems reasonable, but be careful to cope with the case where these
>> objects already exist from a prior regression run.

> I drop them at the end of the test.

What if the prior test crashed or was aborted by the user midway
through?  Cleaning up at the end of the test is good, but I think
it'd be wise for pg_regress to also drop these users/groups before
it starts the run.

			regards, tom lane

Thu, 24 May 2001 08:42:35 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Smaller access privilege changes 
> Peter Eisentraut <peter_e@gmx.net> writes:
> > * Make DELETE distinct from UPDATE privilege
> 
> Okay.
> 
> > * rename the internal representation: s = select, i = insert, u = update,
> >   d = delete, R = rules
> 
> Since the internal representation is visible to users, I fear that a
> wholesale renaming will break existing applications.  Can we make this
> part of the change less intrusive?

If we are voting, I kind of like the newer letters.  The old ones made
little sense except to QUEL users.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 24 May 2001 10:38:18 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Smaller access privilege changes
Tom Lane wrote:
  >"Oliver Elphick" <olly@lfix.co.uk> writes:
  >> Notices are being timestamped in GMT in the syslog, instead of local
  >> time like all other log entries.  Here's a fragment from my syslog:
  >
  >Curious.  I always assumed that syslog timestamps were supplied by the
  >syslog daemon, but to make this happen they'd have to be supplied in the
  >syslog client process (viz. the Postgres process).  What timezone is the
  >Postgres backend being run in, and is it different from all the other
  >syslog clients on the system?

I just got this extra information from the reporter.

------- Forwarded Message

Date:    Thu, 24 May 2001 08:57:49 -0400
From:    "Marc Sherman" <msherman@projectile.ca>
To:      <98565@bugs.debian.org>
Subject: Bug#98565: More info on this bug

The connection that is creating this NOTICE is executing
"set time zone 'GMT';" immediately after connection; I
suspect that what's happening is that postmaster is setting the
libc timezone (by changing the environment and calling tzset)
when is executes that query, causing subsequent system calls
(including syslog) to use the new timezone.

- - Marc



------- End of Forwarded Message


-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "I will praise thee; for I am fearfully and wonderfully 
      made..."       Psalms 139:14 



Thu, 24 May 2001 18:31:14 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Bug#98565: postgresql logs notices with GMT timestamps
	in syslog (fwd)
Peter Eisentraut wrote:
  >Since there is no plan yet how to do a wholesale overhaul of the ACL
  >system, I'd like to stick a few improvements into the current
  >implementation:
 
  >* COPY FROM will require INSERT privilege.  It used to require
  >  UPDATE/DELETE, it think that is not correct..

COPY FROM should require either INSERT or UPDATE or both according to
what rows are being copied.  If a copied primary key already exists,
that will be an update.  I don't see any reason to give COPY FROM any
special treatment - surely it should succeed or fail according to 
whether what it is trying to do is within the user's privileges.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "I will praise thee; for I am fearfully and wonderfully 
      made..."       Psalms 139:14 



Thu, 24 May 2001 18:57:50 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Smaller access privilege changes 
Tom Lane <tgl@sss.pgh.pa.us> writes:

> "Oliver Elphick" <olly@lfix.co.uk> writes:
> > Notices are being timestamped in GMT in the syslog, instead of local
> > time like all other log entries.  Here's a fragment from my syslog:
> 
> Curious.  I always assumed that syslog timestamps were supplied by the
> syslog daemon, but to make this happen they'd have to be supplied in the
> syslog client process (viz. the Postgres process).

That is correct.  The syslog(3) function puts a timestamp in front of
the message, and writes it to the syslog daemon.  The string written
to the daemon starts with <N>, where N is the priority and facility
or'ed together.

Ian

24 May 2001 11:07:49 -0700
Ian Lance Taylor <ian@airs.com>
Re: Bug#98565: postgresql logs notices with GMT timestamps in syslog
	(fwd)
Oliver Elphick writes:

> COPY FROM should require either INSERT or UPDATE or both according to
> what rows are being copied.  If a copied primary key already exists,
> that will be an update.

No, it will be an error.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Thu, 24 May 2001 20:41:16 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Smaller access privilege changes 
"Oliver Elphick" <olly@lfix.co.uk> writes:
> I just got this extra information from the reporter.

> The connection that is creating this NOTICE is executing
> "set time zone 'GMT';" immediately after connection; I
> suspect that what's happening is that postmaster is setting the
> libc timezone (by changing the environment and calling tzset)
> when is executes that query, causing subsequent system calls
> (including syslog) to use the new timezone.

Uh-huh.  This is not a bug, or at least it's not our bug.

			regards, tom lane

Thu, 24 May 2001 14:58:01 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Bug#98565: postgresql logs notices with GMT timestamps in syslog
	(fwd)
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Peter Eisentraut <peter_e@gmx.net> writes:
> * rename the internal representation: s = select, i = insert, u = update,
> d = delete, R = rules

> If we are voting, I kind of like the newer letters.

Actually, I like 'em too ... *if* they're the final set.  I'm just
concerned about the idea of breaking user applications for an
intermediate stop on the way to full SQL92 privilege specs.  It'd be
especially nasty if we found ourselves using non-intuitive coding for
the full SQL specs so as to stay compatible with an intermediate stop.

			regards, tom lane

Thu, 24 May 2001 15:02:14 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Smaller access privilege changes 
Tom Lane writes:

> A different tack is to go ahead and make the change now, but try to
> ensure we won't have to change the coding again when we do the rest of
> the SQL protection model.  Do you know what is still missing given this
> change?

I don't think this will end up being the user-visible interface.  We
should have a view, perhaps as part of the standard information_schema,
that users can query for privilege information.  So for now I'll leave the
old letters the way they are and just add a few new ones.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Thu, 24 May 2001 22:05:44 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Smaller access privilege changes 

I hate the Australian configure option because it means that you can't use the
pre-built postgres
that comes with RedHat or whatever. Surely the correct solution is to have a
config file somewhere
that gets read on startup? That way us Australians don't have to be the only
ones in the world
that need a custom built postgres.




Tue, 12 Jun 2001 15:00:07 +1000
chris.bitmead@health.gov.au
Re: Australian timezone configure option
I'd just like to ask, will making USE_AUSTRALIAN_RULES as a GUC option mean 
the regression tests work?

And DST changes will work fine too (although I think that's more Linux system 
related).

etc.

As in, I'm in favour of a GUC option, but if it breaks regressions tests or 
other stuff, then I'd have second thoughts.

Regards and best wishes,

Justin Clift

On Tuesday 12 June 2001 15:00, chris.bitmead@health.gov.au wrote:
> I hate the Australian configure option because it means that you can't use
> the pre-built postgres
> that comes with RedHat or whatever. Surely the correct solution is to have
> a config file somewhere
> that gets read on startup? That way us Australians don't have to be the
> only ones in the world
> that need a custom built postgres.
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://www.postgresql.org/search.mpl

Wed, 13 Jun 2001 11:18:09 +1000
Justin Clift <aa2@bigpond.net.au>
Re: Australian timezone configure option

My patch allows the regression tests to pass too.

> I'd just like to ask, will making USE_AUSTRALIAN_RULES as a GUC option mean 
> the regression tests work?
> 
> And DST changes will work fine too (although I think that's more Linux system 
> related).
> 
> etc.
> 
> As in, I'm in favour of a GUC option, but if it breaks regressions tests or 
> other stuff, then I'd have second thoughts.
> 
> Regards and best wishes,
> 
> Justin Clift
> 
> On Tuesday 12 June 2001 15:00, chris.bitmead@health.gov.au wrote:
> > I hate the Australian configure option because it means that you can't use
> > the pre-built postgres
> > that comes with RedHat or whatever. Surely the correct solution is to have
> > a config file somewhere
> > that gets read on startup? That way us Australians don't have to be the
> > only ones in the world
> > that need a custom built postgres.
> >
> >
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 6: Have you searched our list archives?
> >
> > http://www.postgresql.org/search.mpl
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 12 Jun 2001 21:30:32 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Australian timezone configure option
> Surely the correct solution is to have a config file somewhere
> that gets read on startup? That way us Australians don't have to be the only
> ones in the world that need a custom built postgres.

I will point out that "you Australians", and, well, "us 'mericans", are
the only countries without the sense to choose unique conventions for
time zone names.

It sounds like having a second lookup table for the Australian rules is
a possibility, and this sounds fairly reasonable to me. Btw, is there an
Australian convention for referring to North American time zones for
those zones with naming conflicts?

                       - Thomas

Thu, 14 Jun 2001 00:23:22 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Australian timezone configure option
On Thu, Jun 14, 2001 at 12:23:22AM +0000, Thomas Lockhart wrote:
> > Surely the correct solution is to have a config file somewhere
> > that gets read on startup? That way us Australians don't have to be the only
> > ones in the world that need a custom built postgres.
> 
> I will point out that "you Australians", and, well, "us 'mericans", are
> the only countries without the sense to choose unique conventions for
> time zone names.
> 
> It sounds like having a second lookup table for the Australian rules is
> a possibility, and this sounds fairly reasonable to me. Btw, is there an
> Australian convention for referring to North American time zones for
> those zones with naming conflicts?

For years I've been on the TZ list, the announcement list for a 
community-maintained database of time zones.  One point they have 
firmly established is that there is no reasonable hope of making 
anything like a standard system of time zone name abbreviations work.  
Legislators and dictators compete for arbitrariness in their time
zone manipulations.

Even if you assign, for your own use, an abbreviation to a particular
administrative region, you still need a history of legislation for that 
region to know what any particular time record (particularly and April 
or September) really means.

The "best practice" for annotating times is to tag them with the numeric
offset from UTC at the time the sample is formed.  If the time sample is
the present time, you don't have to know very much make or use it.  If 
it's in the past, you have to know the legislative history of the place 
to form a proper time record, but not to use it.  If the time is in the 
future, you cannot know what offset will be in popular use at that time, 
but at least you can be precise about what actual time you really mean,
even if you can't be sure about what the wall clock says.  (Actual wall 
clock times are not reliably predictable, a fact that occasionally makes 
things tough on airline passengers.)

Things are a little more stable in some places (e.g. in Europe it is
improving) but worldwide all is chaos.

Assigning some country's current abbreviations at compile time is madness.

Nathan Myers
ncm@zembu.com

Wed, 13 Jun 2001 18:05:42 -0700
ncm@zembu.com (Nathan Myers)
Re: Australian timezone configure option

> It sounds like having a second lookup table for the Australian rules is
> a possibility, and this sounds fairly reasonable to me. Btw, is there an
> Australian convention for referring to North American time zones for
> those zones with naming conflicts?

No.

Named timezones just lose.  For those of us who use machines on both
the Australian and USA East coasts it's nearly useless to see "EST":

    Thu Jun 14 19:54:01 EST 2001

Regards,

Giles

Thu, 14 Jun 2001 19:55:22 +1000
Giles Lean <giles@nemeton.com.au>
Re: Re: [PATCHES] Australian timezone configure option 
Bruce Momjian wrote:
  >> > I think our current idea is to have people run local ident servers to
  >> > handle this.  We don't have any OS-specific stuff in pg_hba.conf and I
  >> > am not sure if we want to add that complexity.  What do others think?
  >> 
  >> This is not any less "specific" than SSL or Kerberos.  Note that opening a
  >> TCP/IP socket already opens a theoretical hole to the world.  Unix domain
  >> is much safer.
  >
  >You can install SSL/Kerberos on any Unix, and many come pre-installed. 
  >You can't add unix-domain socket user authentication to any OS.
  >
  >I assume most OS's have 127.0.0.1 set as loopback so there shouldn't be
  >a hole:
  >
  >127                       127.0.0.1                UGRS    4352 lo0
  >127.0.0.1                 127.0.0.1                UH      4352 lo0
  >
  >However, the security issue may make it worthwhile.  Which OS's support
  >user authentication again, and can we test via configure?  Maybe we can
  >strip out the mention in the pg_hba.conf file if it is not supported on
  >that OS.
 
The security issue is why I developed it.  There were complaints from people 
who did not want to have identd running at all.

I think the feature is available in Linux, Solaris and some BSD.  It can be
tested for by whether SO_PEERCRED is defined in sys/socket.h.

I don't see the need to strip mention from the comments in pg_hba.conf.  The
situation is no different from those systems which do not have Kerberos or
SSL available.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "I waited patiently for the LORD; and he inclined unto 
      me, and heard my cry. He brought me up also out of an 
      horrible pit, out of the miry clay, and set my feet 
      upon a rock, and established my goings. And he hath 
      put a new song in my mouth, even praise unto our God.
      Many shall see it, and fear, and shall trust in the 
      LORD."                 Psalms 40:1-3 



Thu, 12 Jul 2001 03:37:43 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: Re: Debian's PostgreSQL packages 
> The security issue is why I developed it.  There were complaints from people 
> who did not want to have identd running at all.
> 
> I think the feature is available in Linux, Solaris and some BSD.  It can be
> tested for by whether SO_PEERCRED is defined in sys/socket.h.

Yes, I see something similar in BSD/OS.  Manual page attached.

> 
> I don't see the need to strip mention from the comments in pg_hba.conf.  The
> situation is no different from those systems which do not have Kerberos or
> SSL available.

Yea, I guess.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

RECV(2)			BSD Programmer's Manual		    RECV(2)

NAME
     recv, recvfrom, recvmsg - receive a message from a socket

SYNOPSIS
     #include <sys/types.h>
     #include <sys/socket.h>

     ssize_t
     recv(int s, void *buf, size_t len, int flags);

     ssize_t
     recvfrom(int s, void *buf, size_t len, int flags, struct sockaddr *from,
	     socklen_t *fromlen);

     ssize_t
     recvmsg(int s, struct msghdr *msg, int flags);

DESCRIPTION
     The recvfrom() and recvmsg() calls are used to receive messages from a
     socket, and may be used to receive data on a socket whether or not it is
     connection-oriented.

     If from is non-null, and the socket is not connection-oriented, the
     source address of the message is filled in.  The fromlen pointer refers
     to a value-result parameter; it should initially contain the amount of
     space pointed to by from; on return that location will contain the actual
     length (in bytes) of the address returned.	If the buffer provided is too
     small, the name is truncated and the full size is returned in the loca-
     tion to which fromlen points.  If from is null, the value pointed to by
     fromlen is not modified.  Otherwise, if the socket is connection-orient-
     ed, the address buffer will not be modified, and the value pointed to by
     fromlen will be set to zero.

     The recv() call is normally used only on a connected socket (see
     connect(2))  and is identical to recvfrom() with a nil from parameter.
     As it is redundant, it may not be supported in future releases.

     All three routines return the length of the message on successful comple-
     tion.  If a message is too long to fit in the supplied buffer, excess
     bytes may be discarded depending on the type of socket the message is re-
     ceived from (see socket(2)).

     If no messages are available at the socket, the receive call waits for a
     message to arrive, unless the socket is nonblocking (see fcntl(2))	in
     which case the value -1 is returned and the external variable errno set
     to EAGAIN. The receive calls normally return any data available, up to
     the requested amount, rather than waiting for receipt of the full amount
     requested; this behavior is affected by the socket-level options
     SO_RCVLOWAT and SO_RCVTIMEO described in getsockopt(2).

     The select(2) call may be used to determine when more data arrive.

     The flags argument to a recv call is formed by or'ing one or more of the
     values:

	   MSG_OOB	process out-of-band data
	   MSG_PEEK	peek at incoming message
	   MSG_WAITALL	wait for full request or error

     The MSG_OOB flag requests receipt of out-of-band data that would not be
     received in the normal data stream.  Some protocols place expedited data
     at the head of the normal data queue, and thus this flag cannot be used
     with such protocols.  The MSG_PEEK flag causes the receive operation to
     return data from the beginning of the receive queue without removing that
     data from the queue.  Thus, a subsequent receive call will return the
     same data.	The MSG_WAITALL flag requests that the operation block until
     the full request is satisfied.  However, the call may still return less
     data than requested if a signal is caught, an error or disconnect occurs,
     or the next data to be received is of a different type than that re-
     turned.

     The recvmsg() call uses a msghdr structure to minimize the number of di-
     rectly supplied parameters.  This structure has the following form, as
     defined in <sys/socket.h>:

     struct msghdr {
	     caddr_t msg_name;	/* optional address */
	     u_int   msg_namelen;    /* size of address */
	     struct  iovec *msg_iov; /* scatter/gather array */
	     u_int   msg_iovlen;     /* # elements in msg_iov */
	     caddr_t msg_control;    /* ancillary data, see below */
	     u_int   msg_controllen; /* ancillary data buffer len */
	     int     msg_flags;	/* flags on received message */
     };

     If msg_name is non-null, and the socket is not connection-oriented, the
     source address of the message is filled in.  The amount of space avail-
     able for the address is provided by msg_namelen, which is modified on re-
     turn to reflect the length of the stored address.	If the buffer is too
     small, the address is truncated; this is indicated when msg_namelen is
     less than the length embedded in the address (sa_len). If msg_name is
     null, msg_namelen is not modified.	Otherwise, if the socket is connec-
     tion-oriented, the address buffer will not be modified, and msg_namelen
     will be set to zero.

     Msg_iov and msg_iovlen describe scatter gather locations, as discussed in
     read(2).  Msg_control, which has length msg_controllen, points to a
     buffer for other protocol control related messages or other miscellaneous
     ancillary data.  The messages are of the form:

     struct cmsghdr {
	     u_int   cmsg_len;	/* data byte count, including hdr */
	     int     cmsg_level;     /* originating protocol */
	     int     cmsg_type;	/* protocol-specific type */
     /* followed by
	     u_char  cmsg_data[]; */
     };

     As an example, one could use this to learn of changes in the data-stream
     in XNS/SPP, or in ISO, to obtain user-connection-request data by request-
     ing a recvmsg with no data buffer provided immediately after an accept()
     call.

     Open file descriptors are now passed as ancillary data for AF_LOCAL do-
     main sockets, with cmsg_level set to SOL_SOCKET and cmsg_type set to
     SCM_RIGHTS.

     The msg_flags field is set on return according to the message received.
     MSG_EOR indicates end-of-record; the data returned completed a record
     (generally used with sockets of type SOCK_SEQPACKET). MSG_TRUNC indicates
     that the trailing portion of a datagram was discarded because the data-
     gram was larger than the buffer supplied.	MSG_CTRUNC indicates that some
     control data were discarded due to lack of space in the buffer for ancil-
     lary data.	MSG_OOB is returned to indicate that expedited or out-of-band
     data were received.

RETURN VALUES
     These calls return the number of bytes received, or -1 if an error oc-
     curred.

EXAMPLES
     The following code is an example of parsing the control information re-
     turned in the msg_control field.  This example shows how to parse the
     control messages for a localdomain(4) socket to obtain passed file de-
     scriptors and the sender's credentials.

     #include <sys/param.h>
     #include <sys/socket.h>
     #include <sys/ucred.h>

     struct msghdr msghdr;
     struct cmsghdr *cm;
     struct fcred *fc;	/* Pointer to the credentials */
     int fdcnt;		/* The number of file descriptors passed */
     int *fds;		/* The passed array of file descriptors */

     #define ENOUGH_CMSG(p, size)    ((p)->cmsg_len >= ((size) + sizeof(*(p))))

     fc = NULL;
     fdcnt = 0;
     fds = NULL;

     if (msghdr.msg_controllen >= sizeof (struct cmsghdr) &&
	 (msghdr.msg_flags & MSG_CTRUNC) == 0) {

	     for (cm = CMSG_FIRSTHDR(&msghdr);
		 cm != NULL && cm->cmsg_len >= sizeof(*cm);
		 cm = CMSG_NXTHDR(&msghdr, cm)) {

		     if (cm->cmsg_level != SOL_SOCKET)
			     continue;

		     switch (cm->cmsg_type) {
		     case SCM_RIGHTS:
			     fdcnt = (cm->cmsg_len - sizeof(*cm)) / sizeof(int);
			     fds = (int *)CMSG_DATA(cm);
			     break;

		     case SCM_CREDS:
			     if (ENOUGH_CMSG(cm, sizeof(*fc)))
				     fc = (struct fcred *)CMSG_DATA(cm);
			     break;
		     }
	     }
     }

ERRORS
     The calls fail if:

     [EBADF]	The argument s is an invalid descriptor.

     [ENOTCONN]	The socket is associated with a connection-oriented protocol
		 and has not been connected (see connect(2) and accept(2)).

     [ENOTSOCK]	The argument s does not refer to a socket.

     [EAGAIN]	The socket is marked non-blocking, and the receive operation
		 would block, or a receive timeout had been set, and the time-
		 out expired before data were received.

     [EINTR]	The receive was interrupted by delivery of a signal before
		 any data were available.

     [EFAULT]	The receive buffer pointer(s) point outside the process's ad-
		 dress space.

SEE ALSO
     fcntl(2),	read(2),  select(2),  getsockopt(2),  socket(2),  ip(4),  lo-
     cal(4)

HISTORY
     The recv function call appeared in 4.2BSD.

4.3-Reno Berkeley Distribution February 21, 1994			     4
Wed, 11 Jul 2001 23:48:42 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [GENERAL] Re: Debian's PostgreSQL packages
I was wondering if some of you Postgres hackers could advise me on the
safety of the following.  I have written a postgres C function that
uses a popen linux system call. Orginally when I first tried it I kept
getting an ECHILD.  I read a little bit more on the pclose function
and the wait system calls and discoverd that on LINUX if the signal
handler for  SIGCHLD is set to SIG_IGN you will get the ECHILD error
on pclose(or wait4 for that matter).  So I did some snooping around in
the postgres backend code and found that in the traffic cop that the
SIGCHLD signal handler is set to SIG_IGN.  So in my C function right
before the popen call I set the signal handler for SIGCHLD to SIG_DFL
and right after the pclose I set it back to SIG_IGN.  I tested this
and it seems to solve my problem.  Not knowing much about the
internals of the postgres backend I would like to know...  Is setting
the signal handler to SIG_IGN temorarily going to do anything funky
with by database or the backend?

Thanks in advance for your insights,
Scott Shealy

11 Jul 2001 21:40:55 -0700
spshealy@yahoo.com
SIGCHLD  handler in Postgres C function.
spshealy@yahoo.com writes:
> I have written a postgres C function that
> uses a popen linux system call. Orginally when I first tried it I kept
> getting an ECHILD.  I read a little bit more on the pclose function
> and the wait system calls and discoverd that on LINUX if the signal
> handler for  SIGCHLD is set to SIG_IGN you will get the ECHILD error
> on pclose(or wait4 for that matter).  So I did some snooping around in
> the postgres backend code and found that in the traffic cop that the
> SIGCHLD signal handler is set to SIG_IGN.  So in my C function right
> before the popen call I set the signal handler for SIGCHLD to SIG_DFL
> and right after the pclose I set it back to SIG_IGN.  I tested this
> and it seems to solve my problem.

Hmm.  A possibly related bit of ugliness can be found in
src/backend/commands/dbcommands.c, where we ignore ECHILD after
a system() call:

    ret = system(buf);
    /* Some versions of SunOS seem to return ECHILD after a system() call */
    if (ret != 0 && errno != ECHILD)
    {

Interesting, no?  I wonder whether we could get rid of that kluge
if the signal handler was SIG_DFL rather than SIG_IGN.  Can anyone
try this on one of the affected versions of SunOS?  (Tatsuo, you
seem to have added the ECHILD exception on May 25 2000; the commit
message mentions Solaris but not which version.  Could you try it?)

What I'd be inclined to do, rather than swapping the handlers around
while running, is to just have backend startup (tcop/postgres.c) set
the handler to SIG_DFL not SIG_IGN in the first place.  That *should*
produce the identical results according to my man pages, but evidently
it's not quite the same thing on some systems.

Changing this might be a zero-cost solution to a portability glitch.
Comments anyone?

			regards, tom lane

Tue, 17 Jul 2001 13:21:34 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: SIGCHLD handler in Postgres C function. 
> spshealy@yahoo.com writes:
> > I have written a postgres C function that
> > uses a popen linux system call. Orginally when I first tried it I kept
> > getting an ECHILD.  I read a little bit more on the pclose function
> > and the wait system calls and discoverd that on LINUX if the signal
> > handler for  SIGCHLD is set to SIG_IGN you will get the ECHILD error
> > on pclose(or wait4 for that matter).  So I did some snooping around in
> > the postgres backend code and found that in the traffic cop that the
> > SIGCHLD signal handler is set to SIG_IGN.  So in my C function right
> > before the popen call I set the signal handler for SIGCHLD to SIG_DFL
> > and right after the pclose I set it back to SIG_IGN.  I tested this
> > and it seems to solve my problem.
> 
> Hmm.  A possibly related bit of ugliness can be found in
> src/backend/commands/dbcommands.c, where we ignore ECHILD after
> a system() call:
> 
>     ret = system(buf);
>     /* Some versions of SunOS seem to return ECHILD after a system() call */
>     if (ret != 0 && errno != ECHILD)
>     {
> 
> Interesting, no?  I wonder whether we could get rid of that kluge
> if the signal handler was SIG_DFL rather than SIG_IGN.  Can anyone
> try this on one of the affected versions of SunOS?  (Tatsuo, you
> seem to have added the ECHILD exception on May 25 2000; the commit
> message mentions Solaris but not which version.  Could you try it?)

It was Solaris 2.6.

>Subject: [HACKERS] Solaris 2.6 problems
>From: Tatsuo Ishii <t-ishii@sra.co.jp>
>To: pgsql-hackers@postgresql.org
>Cc: t-ishii@sra.co.jp
>Date: Wed, 24 May 2000 18:28:25 +0900
>X-Mailer: Mew version 1.93 on Emacs 19.34 / Mule 2.3 (SUETSUMUHANA)
>
>Hi, I have encountered a really strange problem with PostgreSQL 7.0 on
>Solaris 2.6/Sparc. The problem is that createdb command or create
>database SQL always fails. Inspecting the output of truss shows that
>system() call in createdb() (commands/dbcomand.c) fails because
>waitid() system call in system() returns error no. 10 (ECHILD).
>
>This problem was not in 6.5.3, so I checked the source of it. The
>reason why 6.5.3's createdb worked was that it just ignored the return
>code of system()!
>
>It seems that we need to ignore an error from system() if the error is
>ECHILD on Solaris.
>
>Any idea?
>
>BTW, I have compiled PostgreSQL with egcs 2.95 with/without
>optimization.
>--
>Tatsuo Ishii
>

Sun, 22 Jul 2001 09:28:03 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: SIGCHLD handler in Postgres C function. 
On Sun, 22 Jul 2001, Tatsuo Ishii wrote:

> > spshealy@yahoo.com writes:
> > > I have written a postgres C function that
> > > uses a popen linux system call. Orginally when I first tried it I kept
> > > getting an ECHILD.  I read a little bit more on the pclose function
> > > and the wait system calls and discoverd that on LINUX if the signal
> > > handler for  SIGCHLD is set to SIG_IGN you will get the ECHILD error
> > > on pclose(or wait4 for that matter).  So I did some snooping around in
> > > the postgres backend code and found that in the traffic cop that the
> > > SIGCHLD signal handler is set to SIG_IGN.  So in my C function right
> > > before the popen call I set the signal handler for SIGCHLD to SIG_DFL
> > > and right after the pclose I set it back to SIG_IGN.  I tested this
> > > and it seems to solve my problem.

Just ignore ECHILD. It's not messy at all. :-) It sounds like your kernel
is using SIG_IGN to do the same thing as the SA_NOCLDWAIT flag in *BSD
(well NetBSD at least). When a child dies, it gets re-parrented to init
(which is wait()ing). init does the child-died cleanup, rather than the
parent needing to. That way when the parent runs wait(), there is no
child, so you get an ECHILD.

All ECHILD is doing is saying there was no child. Since we aren't really
waiting for the child, I don't see how that's a problem.

Take care,

Bill


Mon, 30 Jul 2001 10:37:16 -0700 (PDT)
Bill Studenmund <wrstuden@zembu.com>
Re: SIGCHLD handler in Postgres C function. 
Bill Studenmund <wrstuden@zembu.com> writes:
> All ECHILD is doing is saying there was no child. Since we aren't really
> waiting for the child, I don't see how that's a problem.

You're missing the point: on some platforms the system() call is
returning a failure indication because of ECHILD.  It's system() that's
broken, not us, and the issue is how to work around its brokenness
without sacrificing more error detection than we have to.

			regards, tom lane

Mon, 30 Jul 2001 15:00:22 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: SIGCHLD handler in Postgres C function. 
On Mon, 30 Jul 2001, Tom Lane wrote:

> Bill Studenmund <wrstuden@zembu.com> writes:
> > All ECHILD is doing is saying there was no child. Since we aren't really
> > waiting for the child, I don't see how that's a problem.
>
> You're missing the point: on some platforms the system() call is
> returning a failure indication because of ECHILD.  It's system() that's
> broken, not us, and the issue is how to work around its brokenness
> without sacrificing more error detection than we have to.

I think I do get the point. But perhaps I didn't make my point well. :-)

I think the problem is that on some OSs, setting SIGCHLD to SIG_IGN
actually triggers automatic child reaping. So the problem is that we are:
1) setting SIGCHLD to SIG_IGN, 2) Calling system(), and 3) thinking ECHILD
means something was really wrong.

I think 4.4BSD systems will do what we expect (as the NO_CHLDWAIT flag
requests child reaping), but linux systems will give us the ECHILD.
Looking at source on the web, I found:

kernel/signal.c:1042

* Note the silly behaviour of SIGCHLD: SIG_IGN means that the
* signal isn't actually ignored, but does automatic child
* reaping, while SIG_DFL is explicitly said by POSIX to force
* the signal to be ignored.

So we get automatic reaping on Linux systems (which isn't bad).

If automatic reaping happens, system will give us an ECHILD as the waitpid
(or equivalent) will not have found a child. :-)

My suggestion is just leave the ifs as "if ((error == 0) || (error ==
ECHLD))" (or the inverse).

Take care,

Bill


Mon, 30 Jul 2001 15:15:16 -0700 (PDT)
Bill Studenmund <wrstuden@zembu.com>
Re: SIGCHLD handler in Postgres C function. 
Bill Studenmund <wrstuden@zembu.com> writes:
> Looking at source on the web, I found:

> kernel/signal.c:1042

> * Note the silly behaviour of SIGCHLD: SIG_IGN means that the
> * signal isn't actually ignored, but does automatic child
> * reaping, while SIG_DFL is explicitly said by POSIX to force
> * the signal to be ignored.

Hmm, interesting.  If you'll recall, the start of this thread was a
proposal to change our backends' handling of SIGCHLD from SIG_IGN to
SIG_DFL (and get rid of explicit tests for ECHILD).  I didn't quite see
why changing the handler should make a difference, but above we seem to
have the smoking gun.

Which kernel, and which version, is the above quote from?

			regards, tom lane

Mon, 30 Jul 2001 18:40:00 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: SIGCHLD handler in Postgres C function. 
On Mon, 30 Jul 2001, Tom Lane wrote:

> Bill Studenmund <wrstuden@zembu.com> writes:
> > Looking at source on the web, I found:
>
> > kernel/signal.c:1042
>
> > * Note the silly behaviour of SIGCHLD: SIG_IGN means that the
> > * signal isn't actually ignored, but does automatic child
> > * reaping, while SIG_DFL is explicitly said by POSIX to force
> > * the signal to be ignored.
>
> Hmm, interesting.  If you'll recall, the start of this thread was a
> proposal to change our backends' handling of SIGCHLD from SIG_IGN to
> SIG_DFL (and get rid of explicit tests for ECHILD).  I didn't quite see
> why changing the handler should make a difference, but above we seem to
> have the smoking gun.
>
> Which kernel, and which version, is the above quote from?

Linux kernel source, 2.4.3, I think i386 version (though it should be the
same for this bit, it's supposed to be MI). Check out
http://lxr.linux.no/source/

I do recall the reason for the thread. :-) I see three choices:

1) Change back to SIG_DFL for normal behavior. I think this will be fine
	as we run w/o problem on systems that lack this behavior. If
	turning off automatic child reaping would cause a problem, we'd
	have seen it already on the OSs which don't automatically reap
	children. Will a backend ever fork after it's started?

2) Change to DFL around system() and then change back.

3) Realize that ECHILD means that the child was auto-reaped (which is an
	ok think and, I think, will only happen if the child exited w/o
	error).

Take care,

Bill


Mon, 30 Jul 2001 16:09:03 -0700 (PDT)
Bill Studenmund <wrstuden@zembu.com>
Re: SIGCHLD handler in Postgres C function. 
Bill Studenmund <wrstuden@zembu.com> writes:
> I see three choices:

> 1) Change back to SIG_DFL for normal behavior. I think this will be fine
> 	as we run w/o problem on systems that lack this behavior. If
> 	turning off automatic child reaping would cause a problem, we'd
> 	have seen it already on the OSs which don't automatically reap
> 	children. Will a backend ever fork after it's started?

Backends never fork more backends --- but there are some places that
launch transient children and wait for them to finish.  A non-transient
subprocess should always be launched by the postmaster, never by a
backend, IMHO.

> 2) Change to DFL around system() and then change back.

I think this is pretty ugly, and unnecessary.

> 3) Realize that ECHILD means that the child was auto-reaped (which is an
> 	ok think and, I think, will only happen if the child exited w/o
> 	error).

That's the behavior that's in place now, but I do not like it.  We
should not need to code an assumption that "this error isn't really
an error" --- especially when it only happens on some platforms.
On a non-Linux kernel, an ECHILD failure really would be a failure,
and the existing code would fail to detect that there was a problem.

Bottom line: I like solution #1.  Does anyone have an objection to it?

			regards, tom lane

Mon, 30 Jul 2001 19:14:22 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: SIGCHLD handler in Postgres C function. 
> Bill Studenmund <wrstuden@zembu.com> writes:
> > Looking at source on the web, I found:
> 
> > kernel/signal.c:1042
> 
> > * Note the silly behaviour of SIGCHLD: SIG_IGN means that the
> > * signal isn't actually ignored, but does automatic child
> > * reaping, while SIG_DFL is explicitly said by POSIX to force
> > * the signal to be ignored.
> 
> Hmm, interesting.  If you'll recall, the start of this thread was a
> proposal to change our backends' handling of SIGCHLD from SIG_IGN to
> SIG_DFL (and get rid of explicit tests for ECHILD).  I didn't quite see
> why changing the handler should make a difference, but above we seem to
> have the smoking gun.
> 
> Which kernel, and which version, is the above quote from?

The auto-reaping is standard SysV behavior, while BSD is really ignore. 
See the Steven's Unix Programming book for more info.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 30 Jul 2001 19:41:06 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: SIGCHLD handler in Postgres C function.
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> The auto-reaping is standard SysV behavior, while BSD is really ignore. 

You'll recall the ECHILD exception was installed by Tatsuo after seeing
problems on Solaris.  Evidently Solaris uses the auto-reap behavior too.

I'm somewhat surprised that HPUX does not --- it tends to follow its
SysV heritage when there's a conflict between that and BSD practice.
Guess they went BSD on this one.

			regards, tom lane

Mon, 30 Jul 2001 19:45:12 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: SIGCHLD handler in Postgres C function. 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > The auto-reaping is standard SysV behavior, while BSD is really ignore. 
> 
> You'll recall the ECHILD exception was installed by Tatsuo after seeing
> problems on Solaris.  Evidently Solaris uses the auto-reap behavior too.

SVr4/Solaris took the SysV behavior.  Steven's didn't like it.  :-)


> I'm somewhat surprised that HPUX does not --- it tends to follow its
> SysV heritage when there's a conflict between that and BSD practice.
> Guess they went BSD on this one.

I thought HPUX was mostly SysV tools on BSD kernel.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 30 Jul 2001 19:47:41 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: SIGCHLD handler in Postgres C function.
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> I'm somewhat surprised that HPUX does not --- it tends to follow its
>> SysV heritage when there's a conflict between that and BSD practice.
>> Guess they went BSD on this one.

> I thought HPUX was mostly SysV tools on BSD kernel.

No, it was all SysV (or maybe even older) to start with, and later on
they adopted BSD features wholesale.  But where there's a conflict, it's
still mostly SysV.

			regards, tom lane

Mon, 30 Jul 2001 20:53:21 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: SIGCHLD handler in Postgres C function. 
[ redirected to pgsql-hackers for comment ]

Helge Bahmann <bahmann@math.tu-freiberg.de> writes:
> On Tue, 31 Jul 2001, Tom Lane wrote:
>> There is a more complete version of this capability in the Debian patch
>> set.  I think we've been waiting for Oliver to pull it out and submit it
>> as a patch...

> Ok found it; uses "peer" as a keyword instead of "ident" but basically
> does the same thing. I think you can discard my patch then.

Well, we need to talk about that.  I like your idea of making ident auth
"just work" on local connections better than Oliver's approach of
inventing a separate auth-type keyword.  So some kind of merger of the
two patches seems attractive to me.  But Oliver may feel that he has to
continue to support the "peer" keyword on Debian anyway, for backwards
compatibility.  If so, do we want different ways of doing the same thing
on different distros, or should we just follow the Debian precedent to
keep things ugly-but-consistent?

			regards, tom lane

Tue, 31 Jul 2001 11:44:50 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
> [ redirected to pgsql-hackers for comment ]
> 
> Helge Bahmann <bahmann@math.tu-freiberg.de> writes:
> > On Tue, 31 Jul 2001, Tom Lane wrote:
> >> There is a more complete version of this capability in the Debian patch
> >> set.  I think we've been waiting for Oliver to pull it out and submit it
> >> as a patch...
> 
> > Ok found it; uses "peer" as a keyword instead of "ident" but basically
> > does the same thing. I think you can discard my patch then.
> 
> Well, we need to talk about that.  I like your idea of making ident auth
> "just work" on local connections better than Oliver's approach of
> inventing a separate auth-type keyword.  So some kind of merger of the
> two patches seems attractive to me.  But Oliver may feel that he has to
> continue to support the "peer" keyword on Debian anyway, for backwards
> compatibility.  If so, do we want different ways of doing the same thing
> on different distros, or should we just follow the Debian precedent to
> keep things ugly-but-consistent?

We could easily just accept peer as a synonym for ident for a few
releases, because it fact our ident will become something that is used
beyond the identd server.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 31 Jul 2001 12:00:32 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> ... But Oliver may feel that he has to
>> continue to support the "peer" keyword on Debian anyway, for backwards
>> compatibility.  If so, do we want different ways of doing the same thing
>> on different distros, or should we just follow the Debian precedent to
>> keep things ugly-but-consistent?

> We could easily just accept peer as a synonym for ident for a few
> releases,

Or let Oliver patch the Debian package to accept peer as a synonym for
ident.  I don't see any real need to encourage the use of that keyword
by non-Debianers...

			regards, tom lane

Tue, 31 Jul 2001 12:22:33 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> ... But Oliver may feel that he has to
> >> continue to support the "peer" keyword on Debian anyway, for backwards
> >> compatibility.  If so, do we want different ways of doing the same thing
> >> on different distros, or should we just follow the Debian precedent to
> >> keep things ugly-but-consistent?
> 
> > We could easily just accept peer as a synonym for ident for a few
> > releases,
> 
> Or let Oliver patch the Debian package to accept peer as a synonym for
> ident.  I don't see any real need to encourage the use of that keyword
> by non-Debianers...

Good idea.  I was hoping to reduce his patching but this way he can
control how long he keeps it active.  However, the text is only one line
in hba.c.  Either way is fine.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 31 Jul 2001 12:24:30 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
Tom Lane wrote:
  >[ redirected to pgsql-hackers for comment ]
  >
  >Helge Bahmann <bahmann@math.tu-freiberg.de> writes:
  >> On Tue, 31 Jul 2001, Tom Lane wrote:
  >>> There is a more complete version of this capability in the Debian patch
  >>> set.  I think we've been waiting for Oliver to pull it out and submit it
  >>> as a patch...
  >
  >> Ok found it; uses "peer" as a keyword instead of "ident" but basically
  >> does the same thing. I think you can discard my patch then.
  >
  >Well, we need to talk about that.  I like your idea of making ident auth
  >"just work" on local connections better than Oliver's approach of
  >inventing a separate auth-type keyword.  So some kind of merger of the
  >two patches seems attractive to me.  But Oliver may feel that he has to
  >continue to support the "peer" keyword on Debian anyway, for backwards
  >compatibility.  If so, do we want different ways of doing the same thing
  >on different distros, or should we just follow the Debian precedent to
  >keep things ugly-but-consistent?

This change has only been made in the unstable release; so I don't mind
if peer and ident are folded together.  Anyone running unstable knows
the world may turn upside down beneath him!

So if you have a patch to do that, go ahead.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Have not I commanded thee? Be strong and of a good 
      courage; be not afraid, neither be thou dismayed; for 
      the LORD thy God is with thee whithersoever thou 
      goest."                        Joshua 1:9 



Tue, 31 Jul 2001 20:45:58 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
"Oliver Elphick" <olly@lfix.co.uk> writes:
> This change has only been made in the unstable release; so I don't mind
> if peer and ident are folded together.  Anyone running unstable knows
> the world may turn upside down beneath him!

> So if you have a patch to do that, go ahead.

Sounds great.  Helge, the main things your patch was missing were
autoconf support and documentation fixes.  Do you want to add those
(possibly stealing liberally from the Debian patches) and resubmit?

BTW, Bruce has recently committed some wholesale changes in hba.c, so a
patch against 7.1.2 likely won't apply cleanly.  If you could do your
patch as a diff against CVS tip, it'd ease applying it.

			regards, tom lane

Tue, 31 Jul 2001 16:00:28 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
> "Oliver Elphick" <olly@lfix.co.uk> writes:
> > This change has only been made in the unstable release; so I don't mind
> > if peer and ident are folded together.  Anyone running unstable knows
> > the world may turn upside down beneath him!
> 
> > So if you have a patch to do that, go ahead.
> 
> Sounds great.  Helge, the main things your patch was missing were
> autoconf support and documentation fixes.  Do you want to add those
> (possibly stealing liberally from the Debian patches) and resubmit?
> 
> BTW, Bruce has recently committed some wholesale changes in hba.c, so a
> patch against 7.1.2 likely won't apply cleanly.  If you could do your
> patch as a diff against CVS tip, it'd ease applying it.

I can merge into hba.conf if needed.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 31 Jul 2001 17:53:05 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)

Can you send over your version for review.  We can edit the 'peer' part.


> Tom Lane wrote:
>   >[ redirected to pgsql-hackers for comment ]
>   >
>   >Helge Bahmann <bahmann@math.tu-freiberg.de> writes:
>   >> On Tue, 31 Jul 2001, Tom Lane wrote:
>   >>> There is a more complete version of this capability in the Debian patch
>   >>> set.  I think we've been waiting for Oliver to pull it out and submit it
>   >>> as a patch...
>   >
>   >> Ok found it; uses "peer" as a keyword instead of "ident" but basically
>   >> does the same thing. I think you can discard my patch then.
>   >
>   >Well, we need to talk about that.  I like your idea of making ident auth
>   >"just work" on local connections better than Oliver's approach of
>   >inventing a separate auth-type keyword.  So some kind of merger of the
>   >two patches seems attractive to me.  But Oliver may feel that he has to
>   >continue to support the "peer" keyword on Debian anyway, for backwards
>   >compatibility.  If so, do we want different ways of doing the same thing
>   >on different distros, or should we just follow the Debian precedent to
>   >keep things ugly-but-consistent?
> 
> This change has only been made in the unstable release; so I don't mind
> if peer and ident are folded together.  Anyone running unstable knows
> the world may turn upside down beneath him!
> 
> So if you have a patch to do that, go ahead.
> 
> -- 
> Oliver Elphick                                Oliver.Elphick@lfix.co.uk
> Isle of Wight                              http://www.lfix.co.uk/oliver
> PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
> GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
>                  ========================================
>      "Have not I commanded thee? Be strong and of a good 
>       courage; be not afraid, neither be thou dismayed; for 
>       the LORD thy God is with thee whithersoever thou 
>       goest."                        Joshua 1:9 
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 31 Jul 2001 19:11:34 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: [PATCHES] Allow IDENT authentication on local connections
	(Linux  only)
BTW, while digging through my mail archives I discovered that Oliver
*did* already extract his "peer" auth patch and submit it as a proposed
patch --- see the pghackers archives for 3-May-2001.  At the time I
think we were concerned about portability issues, but as long as it's
appropriately autoconf'd and documented, I see no real objection to
supporting SO_PEERCRED authentication.

I do still like Helge's API (use "ident") better than adding another
auth keyword, though.

			regards, tom lane

Tue, 31 Jul 2001 19:39:05 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
> BTW, while digging through my mail archives I discovered that Oliver
> *did* already extract his "peer" auth patch and submit it as a proposed
> patch --- see the pghackers archives for 3-May-2001.  At the time I
> think we were concerned about portability issues, but as long as it's
> appropriately autoconf'd and documented, I see no real objection to
> supporting SO_PEERCRED authentication.
> 
> I do still like Helge's API (use "ident") better than adding another
> auth keyword, though.

There is a Solaris API someone submitted a a month ago that was sort of
rejected too.  I will have to dig that one up.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 31 Jul 2001 19:42:37 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
> BTW, while digging through my mail archives I discovered that Oliver
> *did* already extract his "peer" auth patch and submit it as a proposed
> patch --- see the pghackers archives for 3-May-2001.  At the time I
> think we were concerned about portability issues, but as long as it's
> appropriately autoconf'd and documented, I see no real objection to
> supporting SO_PEERCRED authentication.
> 
> I do still like Helge's API (use "ident") better than adding another
> auth keyword, though.

Can someone find the Solaris patch submitted a few months ago that did a
similar thing?  I can't seem to find it.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 1 Aug 2001 22:44:54 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Can someone find the Solaris patch submitted a few months ago that did a
> similar thing?  I can't seem to find it.

I couldn't find one either.  I found a couple of unsupported assertions
that Solaris and *BSD had SO_PEERCRED, so the Linux patch might work
for them.  We'll find out soon enough, I suppose.

			regards, tom lane

Thu, 02 Aug 2001 00:24:37 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Can someone find the Solaris patch submitted a few months ago that did a
> > similar thing?  I can't seem to find it.
> 
> I couldn't find one either.  I found a couple of unsupported assertions
> that Solaris and *BSD had SO_PEERCRED, so the Linux patch might work
> for them.  We'll find out soon enough, I suppose.

Not here on BSD/OS.  I know I saw a Solaris patch that did exactly this
and I questioned it because it was only for Solaris.  Now that I
research and I see different OS's doing this different ways, and I have
mucked up hba.c already, it seemed like a good patch.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 2 Aug 2001 06:22:20 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: [PATCHES] Allow IDENT authentication on local connections
	(Linux only)
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Not here on BSD/OS.  I know I saw a Solaris patch that did exactly this
> and I questioned it because it was only for Solaris.  Now that I
> research and I see different OS's doing this different ways, and I have
> mucked up hba.c already, it seemed like a good patch.

Well, if someone can come up with a way to do the same thing on other
platforms, we can easily fold it in.

Now that I think about it, it's silly to #ifdef SO_PEERCRED in three
places.  We can reduce that to one place: make ident_unix always exist,
and have it do the test for supported-or-not:

	#ifdef SO_PEERCRED
		do it the Linux way
	#else
		report error "IDENT not supported on local connections"
	#endif

Then adding variants for other platforms is just a matter of more ifdefs
in the one place.  I'll take care of doing this in a little bit...

BTW, a question for Linuxers: Oliver's older patch did
setsockopt(SO_PASSCRED) before getsockopt(SO_PEERCRED), whereas Helge's
version did not.  I included the PASSCRED step in what I committed,
because the Linux docs I had at hand implied it was needed.  But
evidently it worked without it for Helge.  Is there some variation among
Linux versions as to whether PASSCRED is enabled by default?

			regards, tom lane

Thu, 02 Aug 2001 09:00:14 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Re: [PATCHES] Allow IDENT authentication on local connections
	(Linux only)

> You'll recall the ECHILD exception was installed by Tatsuo after seeing
> problems on Solaris.  Evidently Solaris uses the auto-reap behavior too.
> 
> I'm somewhat surprised that HPUX does not --- it tends to follow its
> SysV heritage when there's a conflict between that and BSD practice.
> Guess they went BSD on this one.

If the SysV behaviour of automatically reaping child processes is
required on HP-UX the handler for SIGCHLD can be set to SIG_IGN.
When the handler is SIG_DFL the signal will be ignored but child
processes won't be reaped automatically.  This is the same behaviour
that Stevens describes for SysVr4.  ("Advanced Programming in the Unix
Environment", section 10.7.)

What different implementations of system(3) with different settings of
SIGCHLD is another can of worms, and one I've not investigated. :-)

Regards,

Giles








Sat, 04 Aug 2001 10:56:57 +1000
Giles Lean <giles@nemeton.com.au>
Re: SIGCHLD handler in Postgres C function. 
Tom Lane writes:

> Well, we need to talk about that.  I like your idea of making ident auth
> "just work" on local connections better than Oliver's approach of
> inventing a separate auth-type keyword.

This is exactly what I would not like to see.  "ident" defines a specific
protocol, with an ident server.  ident over something not TCP/IP doesn't
make sense, it could confuse admins.  Just because it works similar
doesn't mean it is the same.  In particular, the security issues are
completely different.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Sun, 5 Aug 2001 22:00:53 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
Peter Eisentraut <peter_e@gmx.net> writes:
>> Well, we need to talk about that.  I like your idea of making ident auth
>> "just work" on local connections better than Oliver's approach of
>> inventing a separate auth-type keyword.

> This is exactly what I would not like to see.  "ident" defines a specific
> protocol, with an ident server.  ident over something not TCP/IP doesn't
> make sense, it could confuse admins.  Just because it works similar
> doesn't mean it is the same.  In particular, the security issues are
> completely different.

Well, ISTM this is a documentation issue.  We've already committed the
patch using "ident" as the keyword, so I'd prefer to leave it that way
and improve the docs as necessary.

			regards, tom lane

PS: welcome back!  Hope you had a pleasant vacation.

Sun, 05 Aug 2001 16:09:50 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] Allow IDENT authentication on local connections (Linux
	only)
I have had an idea how the LIKE optimization problem could be solved.
The problem was that given the expression

    A LIKE (P || '%')

where A is a column reference and P is a constant not containing
wildcards, we wanted to find X and Y such that

    X <= A and A <= Y

where X and Y are calculated from P and the bound is usefully tight.

Note that X <= A is really strcoll(X, A) <= 0 and strcoll(X, A) is really
strcmp(strxfrm(X), strxfrm(A)) (the prototype of strxfrm() is different in
C, of course).

Let $<=$ be a non-locale based comparison (i.e., plain strcmp()).

The we can say that

    strxfrm(P) $<=$ strxfrm(A) and
    strxfrm(A) $<=$ (strxfrm(P) + 1)

where "+ 1" means adding 1 to the last character and accounting for
overflow (like considering the string a base-256 number).  Basically, this
is the funny-collation-safe equivalent to A LIKE 'foo%' yielding 'foo' <=
A <= 'fop'.

We'd need to implement the strxfrm() function in SQL and the $<=$
operator, both of which are trivial.  The index would have to be in terms
of strxfrm().  There might be other issues, but they could be solved
algorithmically, I suppose.

What do you think?

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Mon, 6 Aug 2001 00:28:20 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Possible solution for LIKE optimization
Peter Eisentraut <peter_e@gmx.net> writes:
> I have had an idea how the LIKE optimization problem could be solved.

Hmm ... so in a non-ASCII locale, we'd have to look for an index on
strxfrm(A) rather than directly on A.  And the index would need to
use a nonstandard operator set --- ie, *non* locale aware comparison
operators (which might be useful for other purposes anyway).

Interesting thought.  I'm not entirely sure how we'd teach the planner
to do this, but that's probably solvable.

A more significant problem is that I'm still not convinced this gets the
job done, because of the problem of multi-character collation elements.
If "A LIKE 'FOOS%'" should match FOOSS, but SS is treated specially by
the collation rules, does this scheme work?

			regards, tom lane

Sun, 05 Aug 2001 19:01:49 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Possible solution for LIKE optimization 
Tom Lane writes:

> Peter Eisentraut <peter_e@gmx.net> writes:
> > I have had an idea how the LIKE optimization problem could be solved.
>
> Hmm ... so in a non-ASCII locale, we'd have to look for an index on
> strxfrm(A) rather than directly on A.  And the index would need to
> use a nonstandard operator set --- ie, *non* locale aware comparison
> operators (which might be useful for other purposes anyway).

Wait, why isn't that the solution in the first place.  Let's build the
index with an opclass that uses plain strcmp comparison.  Then you can
compute the bounds using the method 'foo' <= 'foo%' <= 'fop'.  We don't
need to trick the locale facilities, we just avoid using them.  LIKE is
defined in terms of character elements, not collation elements, so that's
okay.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Mon, 6 Aug 2001 03:45:14 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Possible solution for LIKE optimization 
Peter Eisentraut <peter_e@gmx.net> writes:
> Wait, why isn't that the solution in the first place.  Let's build the
> index with an opclass that uses plain strcmp comparison.

By George, I think you've got it!  All we need is comparison ops and
an opclass that use strcmp, even when USE_LOCALE is defined.  Then we
document "here's how you make a LIKE-compatible index in non-ASCII
locales", and away we go.

			regards, tom lane

Sun, 05 Aug 2001 21:52:51 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Possible solution for LIKE optimization 
> Tom Lane writes:
> 
> > Well, we need to talk about that.  I like your idea of making ident auth
> > "just work" on local connections better than Oliver's approach of
> > inventing a separate auth-type keyword.
> 
> This is exactly what I would not like to see.  "ident" defines a specific
> protocol, with an ident server.  ident over something not TCP/IP doesn't
> make sense, it could confuse admins.  Just because it works similar
> doesn't mean it is the same.  In particular, the security issues are
> completely different.

Peter has a point here.  The only way to save the 'ident' keyword is to
make it mean 'auto-identify' rather than identd.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sun, 5 Aug 2001 22:55:48 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Re: [PATCHES] Allow IDENT authentication on local connections
	(Linux only)
Tom Lane wrote:
> 
> Peter Eisentraut <peter_e@gmx.net> writes:
> > Wait, why isn't that the solution in the first place.  Let's build the
> > index with an opclass that uses plain strcmp comparison.
> 
> By George, I think you've got it!  All we need is comparison ops and
> an opclass that use strcmp, even when USE_LOCALE is defined.  Then we
> document "here's how you make a LIKE-compatible index in non-ASCII
> locales", and away we go.
> 

Do we have to make 2 indexes for non_ASCII text field ?

regards,
Hiroshi Inoue

Mon, 06 Aug 2001 16:19:19 +0900
Hiroshi Inoue <Inoue@tpf.co.jp>
Re: Possible solution for LIKE optimization
Hiroshi Inoue <Inoue@tpf.co.jp> writes:
>> Peter Eisentraut <peter_e@gmx.net> writes:
> Wait, why isn't that the solution in the first place.  Let's build the
> index with an opclass that uses plain strcmp comparison.

> Do we have to make 2 indexes for non_ASCII text field ?

You would if you want to use indexscans for both LIKE and "x < 'FOO'"
(ie, locale-aware comparisons).  Which is not great, but I think we've
finally seen the light: a locale-sorted index is just plain not useful
for LIKE.

This discussion has restored my faith in index opclasses; finally we
have a real-world application of 'em that we can point to ...

			regards, tom lane

Mon, 06 Aug 2001 10:16:00 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Possible solution for LIKE optimization 
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> Hiroshi Inoue <Inoue@tpf.co.jp> writes:
> >> Peter Eisentraut <peter_e@gmx.net> writes:
> > Wait, why isn't that the solution in the first place.  Let's build the
> > index with an opclass that uses plain strcmp comparison.
> 
> > Do we have to make 2 indexes for non_ASCII text field ?
> 
> You would if you want to use indexscans for both LIKE and "x < 'FOO'"
> (ie, locale-aware comparisons). 

And ORDER BY ?

> Which is not great, but I think we've
> finally seen the light: a locale-sorted index is just plain not useful
> for LIKE.

I'm not familiar with non_ASCII locale.
Is 'ss'  always guaranteed to be LIKE 's%'  for example ?

regards,
Hiroshi Inoue

Tue, 7 Aug 2001 02:47:10 +0900
"Hiroshi Inoue" <Inoue@tpf.co.jp>
RE: Possible solution for LIKE optimization 
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
>>> Do we have to make 2 indexes for non_ASCII text field ?
>> 
>> You would if you want to use indexscans for both LIKE and "x < 'FOO'"
>> (ie, locale-aware comparisons). 

> And ORDER BY ?

Well, the assumption is that we'd make a second set of string comparison
operators that are defined as not-locale-aware.  Names still to be
chosen, but let's suppose they're $=$, $<$, $<=$, etc.  Then

	SELECT ... ORDER BY foo;

would want to use a plain index (defined using locale-aware comparison),
whereas

	SELECT ... ORDER BY foo USING $<$;

would want to use a non-locale-aware index.  So you could use such an
index for sorting, as long as you were content to have non-locale-aware
output ordering.

> I'm not familiar with non_ASCII locale.
> Is 'ss'  always guaranteed to be LIKE 's%'  for example ?

I'd assume so, but I'd be interested to hear whether native speakers
of German think that that's appropriate in their locale ...

			regards, tom lane

Mon, 06 Aug 2001 14:31:27 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Possible solution for LIKE optimization 
Hiroshi Inoue writes:

> I'm not familiar with non_ASCII locale.
> Is 'ss'  always guaranteed to be LIKE 's%'  for example ?

Yes.  LIKE doesn't use any collation rules, since it doesn't do any
collating.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Mon, 6 Aug 2001 21:07:09 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
RE: Possible solution for LIKE optimization 
Peter Eisentraut <peter_e@gmx.net> writes:
> Hiroshi Inoue writes:
>> I'm not familiar with non_ASCII locale.
>> Is 'ss'  always guaranteed to be LIKE 's%'  for example ?

> Yes.  LIKE doesn't use any collation rules, since it doesn't do any
> collating.

On the other hand, LIKE *is* multibyte aware.  So the hypothetical
non-locale-aware comparison operators would need to be aware of
multibyte character sets even though not aware of locale.  And the
"add one" operator that we postulated for the LIKE index optimization
needs to be able to increment a multibyte character.

This seems doable, but the sort order of such a comparison function
might not be very pleasant, depending on what character set you are
using.

			regards, tom lane

Mon, 06 Aug 2001 15:15:25 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Possible solution for LIKE optimization 
Tom Lane writes:

> On the other hand, LIKE *is* multibyte aware.  So the hypothetical
> non-locale-aware comparison operators would need to be aware of
> multibyte character sets even though not aware of locale.  And the
> "add one" operator that we postulated for the LIKE index optimization
> needs to be able to increment a multibyte character.

Both of these are not hard if you know how strcmp operates.  It would also
be irrelevant whether "add one" to a multibyte character yields another
valid multibyte character.  strcmp doesn't care.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Mon, 6 Aug 2001 21:47:01 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Possible solution for LIKE optimization 
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> On the other hand, LIKE *is* multibyte aware.  So the hypothetical
>> non-locale-aware comparison operators would need to be aware of
>> multibyte character sets even though not aware of locale.  And the
>> "add one" operator that we postulated for the LIKE index optimization
>> needs to be able to increment a multibyte character.

> Both of these are not hard if you know how strcmp operates.  It would also
> be irrelevant whether "add one" to a multibyte character yields another
> valid multibyte character.  strcmp doesn't care.

I imagine we can come up with a definition that makes LIKE optimization
work.  I was wondering more about Hiroshi's implied question: would such
an index have any use for sorting?  Or would the induced sort order (on
multibyte character sets) look too weird to be of any use?

			regards, tom lane

Mon, 06 Aug 2001 15:49:37 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Possible solution for LIKE optimization 

[ I realise the discussion has left strxfrm(), but for the archives
  if nothing else ... ]

Peter Eisentraut <peter_e@gmx.net> wrote:

> We'd need to implement the strxfrm() function in SQL and the $<=$
> operator, both of which are trivial.  The index would have to be in terms
> of strxfrm().  There might be other issues, but they could be solved
> algorithmically, I suppose.

Implementations of strxfrm() that I've looked at have had result data
that is three or four times larger than then input string -- quite a
penalty in some situations.

Regards,

Giles



Tue, 07 Aug 2001 07:46:48 +1000
Giles Lean <giles@nemeton.com.au>
Re: Possible solution for LIKE optimization 
Giles Lean <giles@nemeton.com.au> writes:
> Implementations of strxfrm() that I've looked at have had result data
> that is three or four times larger than then input string -- quite a
> penalty in some situations.

Especially so given that we don't have TOAST for indexes, so the indexed
value can't exceed about 2700 bytes (for btree and an 8K block size).
You are allowed to compress first, so that's not a hard limit, but it
could still be a problem.

I like the non-locale-aware-opclass idea much better than the original.

			regards, tom lane

Mon, 06 Aug 2001 18:21:48 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Possible solution for LIKE optimization 
Peter Eisentraut wrote:
> 
> Hiroshi Inoue writes:
> 
> > I'm not familiar with non_ASCII locale.
> > Is 'ss'  always guaranteed to be LIKE 's%'  for example ?
> 
> Yes.  LIKE doesn't use any collation rules, since it doesn't do any
> collating.
> 

Hmm I see the description like the following in SQL99 though I
don't understand the meaning.

i) If <escape character> is not specified, then the collating
   sequence used for the <like predicate> is determined by Table 3,
  ‘‘Collating sequence usage for comparisons’’, taking <character
   match value> as comparand 1 (one) and <character pattern> as
   comparand 2.

regards,
Hiroshi Inoue

Tue, 07 Aug 2001 08:40:46 +0900
Hiroshi Inoue <Inoue@tpf.co.jp>
Re: Possible solution for LIKE optimization
On Mon, 6 Aug 2001, Tom Lane wrote:

> Giles Lean <giles@nemeton.com.au> writes:
> > Implementations of strxfrm() that I've looked at have had result data
> > that is three or four times larger than then input string -- quite a
> > penalty in some situations.
>
> Especially so given that we don't have TOAST for indexes, so the indexed
> value can't exceed about 2700 bytes (for btree and an 8K block size).
> You are allowed to compress first, so that's not a hard limit, but it
> could still be a problem.
>
> I like the non-locale-aware-opclass idea much better than the original.

Does this means that if implemented we could create indexes for
different columns with/without locale support ? It's pain currently
that I had to enable locale support while actually I need
it only for several columns. Gnu sort on Linux 10 times slow if
I use LC_ALL other than C !

	Regards,

		Oleg

>
> 			regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>

	Regards,
		Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83


Tue, 7 Aug 2001 22:39:33 +0300 (GMT)
Oleg Bartunov <oleg@sai.msu.su>
Re: Possible solution for LIKE optimization 


On 30 Jul 2001 Tom Lane wrote:

> pgsql-bugs@postgresql.org writes:
> > gcc2.95; hppa2.0-hp-hpux10.20, postgresql 7.1.2
> > make check checks the first group all right (tests check out ok and
> > a process "postgres:" shows up in the process list.
> > When "make check" displays "parallel group (18 tests):  point lseg"
> > the "postgres:" process disappears from the process list and
> > the test does not complete. Or, rather, not until I lose
> > patience (15min). Incidentally the shell process that make started
> > keeps running and eats up all the CPU time.
> 
> Known bug in HPUX's Bourne shell --- evidently it can't cope with so
> many children in parallel.  It works if you do
> 
> 	gmake SHELL=/bin/ksh check
> 
> or if you run the non-parallel "installcheck".  See FAQ_HPUX.
> 
> If I still had a support contract in force with HP, I'd file a bug
> report...

I reduced the pg_regress script to a test case and submitted a CR:

    JAGad84609 /usr/bin/sh hang on PostgreSQL regression test

The defect is filed against HP-UX 11i, but I noted that the defect is
present on 11.00 and 10.20 as well.

Regards,

Giles

Wed, 15 Aug 2001 08:24:33 +1000
Giles Lean <giles@nemeton.com.au>
Re: [BUGS] pg_regress fails at "point" test 
Well, ability to lock only unlocked rows in select for update is useful,
of course. But uniq features of user'locks are:

1. They don't interfere with normal locks hold by session/transaction.
2. Share lock is available.
3. User can lock *and unlock objects* inside transaction, which is not
    (and will not be) available with locks held by transactions.

They are interesting too and proposed implementation will not impact lock
manager (just additional 4 bytes in LOCKTAG => same size of LOCKTAG
on machines with 8 bytes alignment).

> An interesting method would be to allow users to simply avoid locked
> rows:
>
> SELECT * FROM queue FOR UPDATE LIMIT 1 UNLOCKED;
>
> Unlocked, return immediately, whatever could be used as a keyword to
> avoid rows that are locked (skipping over them).
>
> For update locks the row of course.  Currently for the above type of
> thing I issue an ORDER BY random() which avoids common rows enough,
> the queue agent dies if queries start taking too long (showing it's
> waiting for other things) and tosses up new copies if it goes a while
> without waiting at all (showing increased load).
>
> --
> Rod Taylor
>
> This message represents the official view of the voices in my head
>
> ----- Original Message -----
> From: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
> To: <pgsql-hackers@postgresql.org>
> Sent: Friday, August 17, 2001 2:48 PM
> Subject: [HACKERS] User locks code
>
>
> > 1. Just noted this in contrib/userlock/README.user_locks:
> >
> > > User locks, by Massimo Dal Zotto <dz@cs.unitn.it>
> > > Copyright (C) 1999, Massimo Dal Zotto <dz@cs.unitn.it>
> > >
> > > This software is distributed under the GNU General Public License
> > > either version 2, or (at your option) any later version.
> >
> > Well, anyone can put code into contrib with whatever license
> > he/she want but "user locks" package includes interface
> > functions in contrib *and* changes in our lock manager, ie
> > changes in backend code. I wonder if backend' part of package
> > is covered by the same license above? And is it good if yes?
> >
> > 2. Not good implementation, imho.
> >
> > It's too complex (separate lock method table, etc). Much cleaner
> > would be implement this feature the same way as transactions
> > wait other transaction commit/abort: by locking objects in
> > pseudo table. We could get rid of offnum and lockmethod from
> > LOCKTAG and add
> >
> > struct
> >

> > Oid RelId;
> > Oid ObjId;
> > } userObjId;
> >
> > to objId union of LOCKTAG.
> >
> > This way user could lock whatever object he/she want in specified
> > table and note that we would be able to use table access rights to
> > control if user allowed to lock objects in table - missed in 1.
> >
> > One could object that 1. is good because user locks never wait.
> > I argue that "never waiting" for lock is same bad as "always
> waiting".
> > Someday we'll have time-wait etc features for general lock method
> > and everybody will be happy -:)
> >
> > Comments?
> >
> > Vadim
> > P.S. I could add 2. very fast, no matter if we'll keep 1. or not.
> >
> > ---------------------------(end of
> broadcast)---------------------------
> > TIP 3: if posting/reading through Usenet, please send an appropriate
> > subscribe-nomail command to majordomo@postgresql.org so that your
> > message can get through to the mailing list cleanly
> >
>



Sun, 19 Aug 2001 11:20:12 -0700
"Vadim Mikheev" <vmikheev@sectorbase.com>
RE: User locks code
> Well, ability to lock only unlocked rows in select for update is useful,
> of course. But uniq features of user'locks are:
> 
> 1. They don't interfere with normal locks hold by session/transaction.
> 2. Share lock is available.
> 3. User can lock *and unlock objects* inside transaction, which is not
>     (and will not be) available with locks held by transactions.
> 
> They are interesting too and proposed implementation will not impact lock
> manager (just additional 4 bytes in LOCKTAG => same size of LOCKTAG
> on machines with 8 bytes alignment).
> 
> > An interesting method would be to allow users to simply avoid locked
> > rows:
> >
> > SELECT * FROM queue FOR UPDATE LIMIT 1 UNLOCKED;
> >
> > Unlocked, return immediately, whatever could be used as a keyword to
> > avoid rows that are locked (skipping over them).
> >
> > For update locks the row of course.  Currently for the above type of
> > thing I issue an ORDER BY random() which avoids common rows enough,
> > the queue agent dies if queries start taking too long (showing it's
> > waiting for other things) and tosses up new copies if it goes a while
> > without waiting at all (showing increased load).
> >
> > --
> > Rod Taylor
> >
> > This message represents the official view of the voices in my head
> >
> > ----- Original Message -----
> > From: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
> > To: <pgsql-hackers@postgresql.org>
> > Sent: Friday, August 17, 2001 2:48 PM
> > Subject: [HACKERS] User locks code
> >
> >
> > > 1. Just noted this in contrib/userlock/README.user_locks:
> > >
> > > > User locks, by Massimo Dal Zotto <dz@cs.unitn.it>
> > > > Copyright (C) 1999, Massimo Dal Zotto <dz@cs.unitn.it>
> > > >
> > > > This software is distributed under the GNU General Public License
> > > > either version 2, or (at your option) any later version.
> > >
> > > Well, anyone can put code into contrib with whatever license
> > > he/she want but "user locks" package includes interface
> > > functions in contrib *and* changes in our lock manager, ie
> > > changes in backend code. I wonder if backend' part of package
> > > is covered by the same license above? And is it good if yes?
> > >
> > > 2. Not good implementation, imho.
> > >
> > > It's too complex (separate lock method table, etc). Much cleaner
> > > would be implement this feature the same way as transactions
> > > wait other transaction commit/abort: by locking objects in
> > > pseudo table. We could get rid of offnum and lockmethod from
> > > LOCKTAG and add
> > >
> > > struct
> > >
> 
> > > Oid RelId;
> > > Oid ObjId;
> > > } userObjId;
> > >
> > > to objId union of LOCKTAG.
> > >
> > > This way user could lock whatever object he/she want in specified
> > > table and note that we would be able to use table access rights to
> > > control if user allowed to lock objects in table - missed in 1.
> > >
> > > One could object that 1. is good because user locks never wait.
> > > I argue that "never waiting" for lock is same bad as "always
> > waiting".
> > > Someday we'll have time-wait etc features for general lock method
> > > and everybody will be happy -:)
> > >
> > > Comments?
> > >
> > > Vadim
> > > P.S. I could add 2. very fast, no matter if we'll keep 1. or not.
> > >

4. Most important: user locks are retained across transaction, which is
   not possible with ordinary locks.

5. User locks semantic is defined entirely by the application and is not
   related to rows in the database.

I wrote the user locks code because I needed a method to mark items as
`busy' for very long time to avoid more users modifying the same object
and overwriting each one's changes. This requires two features:

    1.	they must survive transaction boundary. The typical use of user
	locks is:

		transaction 1:	select object,user_lock(object);

		... work on object for long time

		transaction 2: update object,user_unlock(object);

    2.	they must not block if the object is already locked, so that the
	program doesn't freeze and the user simply knows it can't use that
	object.

When I wrote the code the only way to do this was to add a separate lock
table and use the same machinery of ordinary locks. I agree that the code
is complex and should probably be rewritten.

If you think there is a better way to implement this feature go ahead,
better code is always welcome.

The only problem I have found with user locks is that if a backend crashes
without releasing a lock there is no way to relase it except restarting
the whole postgres (I don't remember exactly why, I forgot the details).

Regarding the licencing of the code, I always release my code under GPL,
which is the licence I prefer, but my code in the backend is obviously
released under the original postgres licence. Since the module is loaded
dynamically and not linked into the backend I don't see a problem here.
If the licence becomes a problem I can easily change it, but I prefer the
GPL if possible.

-- 
Massimo Dal Zotto

+----------------------------------------------------------------------+
|  Massimo Dal Zotto               email: dz@cs.unitn.it               |
|  Via Marconi, 141                phone: ++39-0461534251              |
|  38057 Pergine Valsugana (TN)      www: http://www.cs.unitn.it/~dz/  |
|  Italy                             pgp: see my www home page         |
+----------------------------------------------------------------------+

Sun, 19 Aug 2001 23:15:54 +0200 (MEST)
Massimo Dal Zotto <dz@cs.unitn.it>
Re: User locks code
At 11:20 AM 8/19/01 -0700, Vadim Mikheev wrote:
>Well, ability to lock only unlocked rows in select for update is useful,
>of course. But uniq features of user'locks are:
>
>1. They don't interfere with normal locks hold by session/transaction.
>2. Share lock is available.
>3. User can lock *and unlock objects* inside transaction, which is not
>    (and will not be) available with locks held by transactions.

Would your suggested implementation allow locking on an arbitrary string?

If it does then one of the things I'd use it for is to insert unique data
without having to lock the table or rollback on failed insert (unique index
still kept as a guarantee).

Cheerio,
Link.





Mon, 20 Aug 2001 23:52:03 +0800
Lincoln Yeoh <lyeoh@pop.jaring.my>
RE: User locks code
> Regarding the licencing of the code, I always release my code under GPL,
> which is the licence I prefer, but my code in the backend is obviously
> released under the original postgres licence. Since the module is loaded
> dynamically and not linked into the backend I don't see a problem here.
> If the licence becomes a problem I can easily change it, but I prefer the
> GPL if possible.

We just wanted to make sure the backend changes were not under the GPL.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 23 Aug 2001 16:45:09 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: User locks code
> > Application would explicitly call user_lock() functions in
> > queries, so issue is still not clear for me. And once again -
> 
> Well, yes, it calls user_lock(), but the communication is not
> OS-linked, it is linked over a network socket, so I don't think
> the GPL spreads over a socket. Just as telnet'ing somewhere an
> typing 'bash' doesn't make your telnet GPL'ed, so I think the
> client code is safe. To the client, the backend is just
> returning information. You don't really link to the query
> results.

Ah, ok.

> > compare complexities of contrib/userlock and backend' userlock
> > code: what's reason to cover contrib/userlock by GPL?
> 
> Only because Massimo prefers it. I can think of no other reason.
> It clearly GPL-stamps any backend that links it in.

Ok, let's do one step back - you wrote:

> My assumption is that once you link that code into the backend,
> the entire backend is GPL'ed and any other application code
> you link into it is also (stored procedures, triggers, etc.)

So, one would have to open-source and GPL all procedures/triggers
used by application just because of application uses user_lock()
in queries?! Is it good?

Vadim

Thu, 23 Aug 2001 16:55:29 -0700
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
RE: User locks code
> > Well, yes, it calls user_lock(), but the communication is not
> > OS-linked, it is linked over a network socket, so I don't think
> > the GPL spreads over a socket. Just as telnet'ing somewhere an
> > typing 'bash' doesn't make your telnet GPL'ed, so I think the
> > client code is safe. To the client, the backend is just
> > returning information. You don't really link to the query
> > results.
> 
> Ah, ok.

Yes, kind of tricky.  I am no expert in this but I have had the usual
discussions.

> > > compare complexities of contrib/userlock and backend' userlock
> > > code: what's reason to cover contrib/userlock by GPL?
> > 
> > Only because Massimo prefers it. I can think of no other reason.
> > It clearly GPL-stamps any backend that links it in.
> 
> Ok, let's do one step back - you wrote:
> 
> > My assumption is that once you link that code into the backend,
> > the entire backend is GPL'ed and any other application code
> > you link into it is also (stored procedures, triggers, etc.)
> 
> So, one would have to open-source and GPL all procedures/triggers
> used by application just because of application uses user_lock()
> in queries?! Is it good?

Yep.  Is it good?  Well, if you like the GPL, I guess so.  If you don't,
then it isn't good.  

Of course, if you want to try and make money selling your program, it
isn't good whether you like the GPL or not.  :-)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 23 Aug 2001 20:11:38 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: User locks code
I definitely agree with Vadim here: it's fairly silly that the
contrib userlock code is GPL'd, when it consists only of a few dozen
lines of wrapper for the real functionality that's in the main backend.
The only thing this licensing setup can accomplish is to discourage
people from using the userlock code; what's the value of that?

Besides, anyone who actually wanted to use the userlock code would need
only to write their own wrapper functions to get around the GPL license.

			regards, tom lane

Thu, 23 Aug 2001 20:14:59 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: User locks code 
> I definitely agree with Vadim here: it's fairly silly that the
> contrib userlock code is GPL'd, when it consists only of a few dozen
> lines of wrapper for the real functionality that's in the main backend.
> The only thing this licensing setup can accomplish is to discourage
> people from using the userlock code; what's the value of that?
> 
> Besides, anyone who actually wanted to use the userlock code would need
> only to write their own wrapper functions to get around the GPL license.

Hey, I agree with Vadim too.  The GPL license is just a roadblock, but I
can't tell Massimo what to do with his code if it is not in the backend
proper.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 23 Aug 2001 20:31:56 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: User locks code
Tom Lane wrote:
> 
> I definitely agree with Vadim here: it's fairly silly that the
> contrib userlock code is GPL'd, when it consists only of a few dozen
> lines of wrapper for the real functionality that's in the main backend.

As it seems a generally useful feature, it could at least be LGPL'd so 
that linking to it won't force the whole backend under GPL.

> The only thing this licensing setup can accomplish is to discourage
> people from using the userlock code; what's the value of that?

Maybe it makes Massimo feel good ? It seems a worhty reason to me, as 
he has contributed a lot of useful stuff over the time.

I really think that mixing licences inside one program is bad, if not
for 
any other reason then for confusing people and making them have
discussions 
like this.

> Besides, anyone who actually wanted to use the userlock code would need
> only to write their own wrapper functions to get around the GPL license.

This is a part of copyright law that eludes me - can i write a
replacement
function for something so simple that it can essentially be done in one 
way only (like incrementing a value by one) ?

-----------------
Hannu

Fri, 24 Aug 2001 07:25:06 +0500
Hannu Krosing <hannu@tm.ee>
Re: User locks code
> Tom Lane wrote:
> > 
> > I definitely agree with Vadim here: it's fairly silly that the
> > contrib userlock code is GPL'd, when it consists only of a few dozen
> > lines of wrapper for the real functionality that's in the main backend.
> 

I was incorrect in something I said to Vadim.  I said stored procedures
would have to be released if linked against a GPL'ed backend.  They have
to be released only if they are in C or another object file linked into
the backend.  PlpgSQL or SQL functions don't have to be released because
their code is "loaded" into the backend as a script, not existing in the
backend binary or required for the backend to run.

> Maybe it makes Massimo feel good ? It seems a worhty reason to me, as 
> he has contributed a lot of useful stuff over the time.

Yes, that is probably it.  The GPL doesn't give anything to users, it
takes some control away from users and gives it to the author of the
code.

> I really think that mixing licences inside one program is bad, if not
> for 
> any other reason then for confusing people and making them have
> discussions 
> like this.

Yes, the weird part is that the BSD license is so lax (don't sue us)
that it is the addition of the GPL that changes the affect of the
license.  If you added a BSD license to a GPL'ed piece of code, the
effect would be near zero.

> > Besides, anyone who actually wanted to use the userlock code would need
> > only to write their own wrapper functions to get around the GPL license.
> 
> This is a part of copyright law that eludes me - can i write a
> replacement
> function for something so simple that it can essentially be done in one 
> way only (like incrementing a value by one) ?

Sure, if you don't cut and paste the code line by line, or retype the
code while staring at the previous version.  That is how Berkeley got
unix-free version of the BSD operating system.  However, the few places
where they lazily copied got them in trouble.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 24 Aug 2001 10:42:48 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: User locks code
Bruce Momjian wrote:
> 
> >
> > This is a part of copyright law that eludes me - can i write a
> > replacement
> > function for something so simple that it can essentially be done in one
> > way only (like incrementing a value by one) ?
> 
> Sure, if you don't cut and paste the code line by line, or retype the
> code while staring at the previous version.  That is how Berkeley got
> unix-free version of the BSD operating system.  However, the few places
> where they lazily copied got them in trouble.
>

I can imagine that when writing a trivial code for performing a trivial
and 
well-known function it is quite possible to arrive at a result that is 
virtually indistinguishable from the original.

I know that Compaq was forced to do a clean-room re-engineering of PC
BIOS 
(two teams - the dirti one with access to real bios athat does
description 
and testin and the clean team to write the actual code so that they can 
prove they did not "steal" even if the result is byte-by-byte simila)
for 
similar reasons

I guess we dont have enough provably clean developers to do it ;)

BTW, teher seems to be some problem with mailing list - I get very few 
messages from the list that are not CC:d to me too

------------------
Hannu

Fri, 24 Aug 2001 17:33:43 +0200
Hannu Krosing <hannu@tm.ee>
Re: User locks code
Bruce Momjian <pgman@candle.pha.pa.us> writes:

> > Tom Lane wrote:
> > > 
> > > I definitely agree with Vadim here: it's fairly silly that the
> > > contrib userlock code is GPL'd, when it consists only of a few dozen
> > > lines of wrapper for the real functionality that's in the main backend.
> > 
> 
> I was incorrect in something I said to Vadim.  I said stored procedures
> would have to be released if linked against a GPL'ed backend. 

Only to those you actually distribute this product to. If you're using
it internally, you have no obligations to release it to anyone, to
give one example.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

24 Aug 2001 12:02:54 -0400
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: User locks code
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> 
> > > Tom Lane wrote:
> > > > 
> > > > I definitely agree with Vadim here: it's fairly silly that the
> > > > contrib userlock code is GPL'd, when it consists only of a few dozen
> > > > lines of wrapper for the real functionality that's in the main backend.
> > > 
> > 
> > I was incorrect in something I said to Vadim.  I said stored procedures
> > would have to be released if linked against a GPL'ed backend. 
> 
> Only to those you actually distribute this product to. If you're using
> it internally, you have no obligations to release it to anyone, to
> give one example.

Yes, I was speaking only of selling the software.  Good point.
 
-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 24 Aug 2001 12:03:34 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: User locks code
Uh, guys? The last thing I can find that Massimo says about the license
is this, from Sunday:

On Sun, Aug 19, 2001 at 11:15:54PM +0200, Massimo Dal Zotto wrote:
> 
> Regarding the licencing of the code, I always release my code under GPL,
> which is the licence I prefer, but my code in the backend is obviously
> released under the original postgres licence. Since the module is loaded
> dynamically and not linked into the backend I don't see a problem here.
> If the licence becomes a problem I can easily change it, but I prefer the
> GPL if possible.
> 

So, rather than going over everone's IANAL opinons about mixing
licenses, let's just let Massimo know that it'd just be a lot easier to
PostgreSQL/BSD license the whole thing, if he doesn't mind too much.

Ross

On Fri, Aug 24, 2001 at 10:42:48AM -0400, Bruce Momjian wrote:
<I'm not sure who wrote this, Bruce trimmed the attribution>
> > Maybe it makes Massimo feel good ? It seems a worhty reason to me, as 
> > he has contributed a lot of useful stuff over the time.
> 
> Yes, that is probably it.  The GPL doesn't give anything to users, it
> takes some control away from users and gives it to the author of the
> code.
> 

Fri, 24 Aug 2001 11:06:56 -0500
"Ross J. Reedstrom" <reedstrm@rice.edu>
Re: User locks code
----- Original Message ----- 
From: Bruce Momjian <pgman@candle.pha.pa.us>
Sent: Friday, August 24, 2001 10:42 AM


> > I really think that mixing licences inside one program is bad, if not
> > for 
> > any other reason then for confusing people and making them have
> > discussions 
> > like this.
> 
> Yes, the weird part is that the BSD license is so lax (don't sue us)
> that it is the addition of the GPL that changes the affect of the
> license.  If you added a BSD license to a GPL'ed piece of code, the
> effect would be near zero.

Sorry for asking this off-topic question, but I'm not sure I completely
understand this license issue... How GPL, LGPL, and BSD are conflicting
and or overlap, so that it causes such problems? AFAIK with the GPL
one has to ship the source code along with the product every time, but
under BSD it can be shipped without the source (that's why M$ doesn't attack
BSD as it does for GPL), and why the PostgreSQL project originally is being
released under the BSD-like license? Just curious...

Serguei



Fri, 24 Aug 2001 12:07:04 -0400
"Serguei Mokhov" <sa_mokho@alcor.concordia.ca>
[OT] Re: User locks code
----- Original Message ----- 
From: Bruce Momjian <pgman@candle.pha.pa.us>
Sent: Friday, August 24, 2001 10:42 AM


> > I really think that mixing licences inside one program is bad, if not
> > for 
> > any other reason then for confusing people and making them have
> > discussions 
> > like this.
> 
> Yes, the weird part is that the BSD license is so lax (don't sue us)
> that it is the addition of the GPL that changes the affect of the
> license.  If you added a BSD license to a GPL'ed piece of code, the
> effect would be near zero.

Sorry for asking this off-topic question, but I'm not sure I completely
understand this license issue... How GPL, LGPL, and BSD are conflicting
and or overlap, so that it causes such problems? AFAIK with the GPL
one has to ship the source code along with the product every time, but
under BSD it can be shipped without the source (that's why M$ doesn't attack
BSD as it does for GPL), and why the PostgreSQL project originally is being
released under the BSD-like license? Just curious...

Serguei



Fri, 24 Aug 2001 12:07:04 -0400
"Serguei Mokhov" <sa_mokho@alcor.concordia.ca>
[OT] Re: User locks code
> > Yes, the weird part is that the BSD license is so lax (don't sue us)
> > that it is the addition of the GPL that changes the affect of the
> > license.  If you added a BSD license to a GPL'ed piece of code, the
> > effect would be near zero.
> 
> Sorry for asking this off-topic question, but I'm not sure I completely
> understand this license issue... How GPL, LGPL, and BSD are conflicting
> and or overlap, so that it causes such problems? AFAIK with the GPL
> one has to ship the source code along with the product every time, but
> under BSD it can be shipped without the source (that's why M$ doesn't attack
> BSD as it does for GPL), and why the PostgreSQL project originally is being
> released under the BSD-like license? Just curious...

Because the code we got from Berkeley was BSD licensed, we can't change
it, and because many of us like the BSD license better because we don't
want to require them to release the source code, we just want them to
use PostgreSQL.  And we think they will release the source code
eventually anyway.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 24 Aug 2001 12:16:18 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [OT] Re: User locks code
Bruce Momjian wrote:
...
  >Yes, that is probably it.  The GPL doesn't give anything to users, it
  >takes some control away from users and gives it to the author of the
  >code.

Correction - it takes away from the *distributor* of binaries the right to
give users fewer rights than he has.  If he doesn't distribute, he can do what 
he likes.

I'm sorry to be pedantic!  We need to be clear about that because Microsoft
are trying to spread FUD about it. 


 From the project's point of view, it is probably a bad idea to accept code
under any licence other than BSD.  That can only lead to confusion among
users and distributors alike and may led to inadvertent violation of the
GPL by those who don't notice that it is has been used.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "I saw in the night visions, and, behold, one like the 
      Son of man came with the clouds of heaven, and came to
      the Ancient of days, and they brought him near before 
      him. And there was given him dominion, and glory, and 
      a kingdom, that all people, nations, and languages, 
      should serve him; his dominion is an everlasting 
      dominion, which shall not pass away, and his kingdom 
      that which shall not be destroyed."     
                                    Daniel 7:13,14



Fri, 24 Aug 2001 17:33:17 +0100
"Oliver Elphick" <olly@lfix.co.uk>
Re: User locks code 
> Bruce Momjian wrote:
> ...
> >   >Yes, that is probably it.  The GPL doesn't give anything to
> >   users, it takes some control away from users and gives it to
> >   the author of the code.
> 
> Correction - it takes away from the *distributor* of binaries
> the right to give users fewer rights than he has.  If he doesn't
> distribute, he can do what he likes.

That is not totally true.  While it prevents him from distributing code
with fewer rights than the GPL code he received, he loses distribution
rights on the code he writes as well.  That is the _viral_ nature of the
GPL that MS was talking about.

I know it is not fun to hear these GPL issues, but it is a valid
concern.

> I'm sorry to be pedantic!  We need to be clear about that because
> Microsoft are trying to spread FUD about it.

I was even more pedantic.  :-)

> >From the project's point of view, it is probably a bad idea to accept code
> under any license other than BSD.  That can only lead to confusion
> among users and distributors alike and may led to inadvertent
> violation of the GPL by those who don't notice that it is has
> been used.

Considering that we link the backend against libreadline if it exists
(even though it isn't needed by the backend), we already have quite a
bit of confusion.  Only libedit can be used freely for line editing,
i.e. psql.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 24 Aug 2001 15:19:28 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: User locks code
> > Tom Lane wrote:
> > > 
> > > I definitely agree with Vadim here: it's fairly silly that the
> > > contrib userlock code is GPL'd, when it consists only of a few dozen
> > > lines of wrapper for the real functionality that's in the main backend.
> > 
> 
> I was incorrect in something I said to Vadim.  I said stored procedures
> would have to be released if linked against a GPL'ed backend.  They have
> to be released only if they are in C or another object file linked into
> the backend.  PlpgSQL or SQL functions don't have to be released because
> their code is "loaded" into the backend as a script, not existing in the
> backend binary or required for the backend to run.
> 
> > Maybe it makes Massimo feel good ? It seems a worhty reason to me, as 
> > he has contributed a lot of useful stuff over the time.
> 
> Yes, that is probably it.  The GPL doesn't give anything to users, it
> takes some control away from users and gives it to the author of the
> code.

I have to disagree here. GPL gives users the assurance that code can't be
modified and distributed under a more restrictive licence, for example with
no source code available or restrictions on commercial use.

On the contrary BSD doesn't guarantee almost anything on this sense.
Think about what happened with the original Kerberos and the microsoft
version which was deliberately modified to prevent compatibilty with
non microsoft implementations. This has been possible only because it
was released under BSD licence. This why I prefer GPL over BSD.

I choosed to release my code under GPL because I'm more concerned with
freedom of software than with commercial issues of it.

Anyway, please stop this thread. I will change the licence of my contrib
code and make it compatible with postgres licence.

After all, as someone poited out, it is a very trivial code and I don't
really care about what people is doing with it.

> > I really think that mixing licences inside one program is bad, if not
> > for 
> > any other reason then for confusing people and making them have
> > discussions 
> > like this.
> 
> Yes, the weird part is that the BSD license is so lax (don't sue us)
> that it is the addition of the GPL that changes the affect of the
> license.  If you added a BSD license to a GPL'ed piece of code, the
> effect would be near zero.
> 
> > > Besides, anyone who actually wanted to use the userlock code would need
> > > only to write their own wrapper functions to get around the GPL license.
> > 
> > This is a part of copyright law that eludes me - can i write a
> > replacement
> > function for something so simple that it can essentially be done in one 
> > way only (like incrementing a value by one) ?
> 
> Sure, if you don't cut and paste the code line by line, or retype the
> code while staring at the previous version.  That is how Berkeley got
> unix-free version of the BSD operating system.  However, the few places
> where they lazily copied got them in trouble.
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html
> 

-- 
Massimo Dal Zotto

+----------------------------------------------------------------------+
|  Massimo Dal Zotto               email: dz@cs.unitn.it               |
|  Via Marconi, 141                phone: ++39-0461534251              |
|  38057 Pergine Valsugana (TN)      www: http://www.cs.unitn.it/~dz/  |
|  Italy                             pgp: see my www home page         |
+----------------------------------------------------------------------+

Fri, 24 Aug 2001 21:33:53 +0200 (MEST)
Massimo Dal Zotto <dz@cs.unitn.it>
Re: User locks code
Hannu Krosing <hannu@tm.ee> writes:

> I know that Compaq was forced to do a clean-room re-engineering of PC
> BIOS 
> (two teams - the dirti one with access to real bios athat does
> description 
> and testin and the clean team to write the actual code so that they can 
> prove they did not "steal" even if the result is byte-by-byte simila)
> for 
> similar reasons

Compaq wasn't forced to do this.  They did it on the basis that
following this complex procedure would guarantee that they would win
if it came to a court case (which it did not).  But there is no way to
tell whether a simpler procedure would not win in court.

The GPL is a copyright license.  Copyrights, unlike patents, only put
limitations on derived works.  If you independently write the same
novel, and you can prove in court that you never saw the original
novel, you are not guily of violating copyright.  That's why Compaq
followed the procedure they did (and it's why Pierre Menard was not
guilty of copyright infringement).

But that's novels.  As far as I know, there is no U.S. law, and there
are no U.S. court decisions, determining when one program is a
derivative of another.  If you read a novel, and write a novel with
similar themes or characters, you've infringed.  However, there is
less protection for functional specifications, in which there may be
only one way to do something.  When does a computer program infringe?
Nobody knows.

Ian

24 Aug 2001 14:38:53 -0700
Ian Lance Taylor <ian@airs.com>
Re: User locks code
Serguei Mokhov wrote:
> 
> and why the PostgreSQL project originally is being
> released under the BSD-like license? Just curious...

Berkeley usually releases their free projects under BSD licence ;)

There have been some discussion about changing it, but it has never got 
enough support.

--------------
Hannu

Fri, 31 Aug 2001 12:15:47 +0200
Hannu Krosing <hannu@tm.ee>
Re: [OT] Re: User locks code
I'm running on Red Hat Linux 7.1 with all official updates.

I have a problem or two with PostgreSQL and thought to test against the 
CVS version so as to avoid the suggestion (as happened recently) to 
"try this later version."

I particularly want to install the package in some out-of-the-way place 
so
a) It doesn't replace the existing production version
b) I don't need root privilege.

To my dismay some components don't honour the "--prefix=/tmp/postgresql"
 specification and try to install in some other location.

I'd much prefer for the perl and python components to install into the 
location I specified, and to leave me to discuss with Perl and Python 
the question of how to make sure I get the right versions (or even 
better, offer a handy hint).



-- 
Cheers
John Summerfield

Microsoft's most solid OS: http://www.geocities.com/rcwoolley/

Note: mail delivered to me is deemed to be intended for me, for my 
disposition.




Sun, 02 Sep 2001 10:27:52 +0800
John Summerfield <summer@os2.ami.com.au>
Build problem with CVS version 
John Summerfield writes:

> To my dismay some components don't honour the "--prefix=/tmp/postgresql"
>  specification and try to install in some other location.
>
> I'd much prefer for the perl and python components to install into the
> location I specified, and to leave me to discuss with Perl and Python
> the question of how to make sure I get the right versions (or even
> better, offer a handy hint).

This is a very valid concern, and it's been bugging us, too.  The problem
is that by default, the majority of users would probably want the Perl and
Python modules to be put in the default place where they're easy to find
for the interpreter.  (This is pure speculation.  Personally, I certainly
wouldn't do this, in the same way as I don't install libraries in /usr/lib
because it makes it easier for the linker to find.)

What we probably want is some configure switch that switches between the
current behaviour and the behaviour you want.

Incidentally, some work toward this goal has already been done in the CVS
tip.  Basically, all I was missing is a good name for the option.

For you to proceed you could try the following (completely untested):

# for local Python install
make install python_moduledir='$(pkglibdir)' python_moduleexecdir='$(pkglibdir)'
# (yes, single quotes)

# for local Perl install
make install mysterious_feature=yes
# (seriously)

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Tue, 4 Sep 2001 14:58:16 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Build problem with CVS version 
Peter Eisentraut <peter_e@gmx.net> writes:
> This is a very valid concern, and it's been bugging us, too.  The problem
> is that by default, the majority of users would probably want the Perl and
> Python modules to be put in the default place where they're easy to find
> for the interpreter.  (This is pure speculation.  Personally, I certainly
> wouldn't do this, in the same way as I don't install libraries in /usr/lib
> because it makes it easier for the linker to find.)

I agree that that's the right place to put the perl & python modules
when doing a pure-default configure: it's reasonable to assume we are
installing a production system, and so we should install these modules
in the default places.  But it's a lot harder to make that argument when
doing a configure with a non-default --prefix: we may well be building a
playpen installation.  In any case there should be a way to suppress
automatic installation of these modules.

> What we probably want is some configure switch that switches between the
> current behaviour and the behaviour you want.

I'd suggest --prefix-like options to determine installation locations
for the perl and python modules, plus options on the order of
--no-install-perl (ie, build it, but don't install it).

			regards, tom lane

Tue, 04 Sep 2001 10:51:22 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Build problem with CVS version 
With all the great work put into allowing true 24/7 operation, as
distributed we're still unable to rotate the log file.  While the log file
tends to be smaller than the database system as a whole, this is still
going to annoy people because they can't control disk usage without taking
the server down.

I've been playing with a little program I wrote whose sole purpose is to
write its stdin to a file and close and reopen that file when it receives
a signal.  I figured this could work well when integrated transparently
into pg_ctl.

So, is log rotation a concern?  Is this a reasonable solution?  Other
ideas?

(No Grand Unified Logging Solutions please.  And no, "use syslog" doesn't
count.)

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Wed, 5 Sep 2001 03:12:28 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Log rotation?

Funny, I found this going through my mailbox.  Seems I was going to
return to this SO_PEERCRED anyway.

> Bruce Momjian wrote:
>   >> > I think our current idea is to have people run local ident servers to
>   >> > handle this.  We don't have any OS-specific stuff in pg_hba.conf and I
>   >> > am not sure if we want to add that complexity.  What do others think?
>   >> 
>   >> This is not any less "specific" than SSL or Kerberos.  Note that opening a
>   >> TCP/IP socket already opens a theoretical hole to the world.  Unix domain
>   >> is much safer.
>   >
>   >You can install SSL/Kerberos on any Unix, and many come pre-installed. 
>   >You can't add unix-domain socket user authentication to any OS.
>   >
>   >I assume most OS's have 127.0.0.1 set as loopback so there shouldn't be
>   >a hole:
>   >
>   >127                       127.0.0.1                UGRS    4352 lo0
>   >127.0.0.1                 127.0.0.1                UH      4352 lo0
>   >
>   >However, the security issue may make it worthwhile.  Which OS's support
>   >user authentication again, and can we test via configure?  Maybe we can
>   >strip out the mention in the pg_hba.conf file if it is not supported on
>   >that OS.
>  
> The security issue is why I developed it.  There were complaints from people 
> who did not want to have identd running at all.
> 
> I think the feature is available in Linux, Solaris and some BSD.  It can be
> tested for by whether SO_PEERCRED is defined in sys/socket.h.
> 
> I don't see the need to strip mention from the comments in pg_hba.conf.  The
> situation is no different from those systems which do not have Kerberos or
> SSL available.
> 
> -- 
> Oliver Elphick                                Oliver.Elphick@lfix.co.uk
> Isle of Wight                              http://www.lfix.co.uk/oliver
> PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
> GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
>                  ========================================
>      "I waited patiently for the LORD; and he inclined unto 
>       me, and heard my cry. He brought me up also out of an 
>       horrible pit, out of the miry clay, and set my feet 
>       upon a rock, and established my goings. And he hath 
>       put a new song in my mouth, even praise unto our God.
>       Many shall see it, and fear, and shall trust in the 
>       LORD."                 Psalms 40:1-3 
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 5 Sep 2001 00:48:35 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [GENERAL] Re: Debian's PostgreSQL packages
> So, is log rotation a concern?  Is this a reasonable solution?  Other
> ideas?
>
> (No Grand Unified Logging Solutions please.  And no, "use syslog" doesn't
> count.)

What's the problem with using newsyslog or logrotate at the moment?  (ie.
use the system log rotator)

Chris


Thu, 6 Sep 2001 10:39:56 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: Log rotation?
Peter Eisentraut <peter_e@gmx.net> writes:
> With all the great work put into allowing true 24/7 operation, as
> distributed we're still unable to rotate the log file.  While the log file
> tends to be smaller than the database system as a whole, this is still
> going to annoy people because they can't control disk usage without taking
> the server down.

> I've been playing with a little program I wrote whose sole purpose is to
> write its stdin to a file and close and reopen that file when it receives
> a signal.  I figured this could work well when integrated transparently
> into pg_ctl.

Aren't there log-rotation utilities out there already?  (I seem to
recall mention that Apache has one, for instance.)  Seems like this
is a wheel we shouldn't have to reinvent.

Also, I kinda thought the long-range solution was to encourage everyone
to migrate to syslog logging ...

> And no, "use syslog" doesn't count.

Why not?

			regards, tom lane

Wed, 05 Sep 2001 23:19:45 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Log rotation? 
Tom Lane <tgl@sss.pgh.pa.us> writes:

> > And no, "use syslog" doesn't count.
> 
> Why not?

The standard implementations of syslog lose log entries under heavy
load, because they rely on a daemon which reads from a named pipe with
a limited buffer space.  This is not acceptable in a production
system, since heavy load is often just the time you need to see the
log entries.

It would be possible to implement the syslog(3) interface in a
different way, of course, which did not use syslogd.  I don't know of
any such implementation.

(My personal preference these days is an approach like DJB's
daemontools, which separates the handling of log entries from the
program doing the logging.)

Ian

05 Sep 2001 20:54:44 -0700
Ian Lance Taylor <ian@airs.com>
Re: Log rotation?
Ian Lance Taylor <ian@airs.com> writes:
> Tom Lane <tgl@sss.pgh.pa.us> writes:
>>> And no, "use syslog" doesn't count.
>> 
>> Why not?

> The standard implementations of syslog lose log entries under heavy
> load,

Okay, that's a sufficient answer for that point.

> (My personal preference these days is an approach like DJB's
> daemontools, which separates the handling of log entries from the
> program doing the logging.)

That still leads back to my first question, which is whether we can't
rely on someone else's logrotation code.

			regards, tom lane

Thu, 06 Sep 2001 00:04:14 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Log rotation? 
Tom Lane writes:

> Aren't there log-rotation utilities out there already?  (I seem to
> recall mention that Apache has one, for instance.)  Seems like this
> is a wheel we shouldn't have to reinvent.

I'm aware of the Apache rotatelogs utility, but I'm not completely
satisfied with it.

1.  It tries to do the rotating itself.  I'd rather rely on the OS'
rotating and archiving facilities.

2.  Only offers a time-based rotate, no manual intervention possible (via
signal).

3.  We don't want to have to tell people to install Apache and patch their
pg_ctl.

4.  We don't want to include it in our distribution because the license
contains an advertisement clause.

It's not like what I wrote is going to look wildly different than theirs.
There's only so much variation you can put into 100 lines of code.

> > And no, "use syslog" doesn't count.
>
> Why not?

1. Might not be available (?)

2. Might not be reliable

3. Might not have root access

4. Not all messages will go through elog.  This is a bug, but not trivial
to fix.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Thu, 6 Sep 2001 12:02:55 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Log rotation? 
Christopher Kings-Lynne writes:

> What's the problem with using newsyslog or logrotate at the moment?  (ie.
> use the system log rotator)

The postmaster will never close the output file, so you can rotate all you
want, the original file will never be abandoned.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Thu, 6 Sep 2001 12:04:03 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Log rotation?
Thus spake Tom Lane
> Also, I kinda thought the long-range solution was to encourage everyone
> to migrate to syslog logging ...
> 
> > And no, "use syslog" doesn't count.
> 
> Why not?

Well, one "why not" might be that syslog is not a guaranteed delivery
logging system.  It might be good enough for some applications but I
don't think that it should be forced on everyone.

-- 
D'Arcy J.M. Cain <darcy@{druid|vex}.net>   |  Democracy is three wolves
http://www.druid.net/darcy/                |  and a sheep voting on
+1 416 425 1212     (DoD#0082)    (eNTP)   |  what's for dinner.

Thu, 6 Sep 2001 08:33:22 -0400 (EDT)
darcy@druid.net (D'Arcy J.M. Cain)
Re: Log rotation?
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> Aren't there log-rotation utilities out there already?  (I seem to
>> recall mention that Apache has one, for instance.)  Seems like this
>> is a wheel we shouldn't have to reinvent.

> I'm aware of the Apache rotatelogs utility, but I'm not completely
> satisfied with it.

Okay, those are reasonable points.  Given that it's only ~100 lines of
code, I'll withdraw my objection to rolling our own.  Let's just do it.

			regards, tom lane

Thu, 06 Sep 2001 09:51:54 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Log rotation? 
At 08:54 PM 9/5/2001 -0700, Ian Lance Taylor wrote:
>Tom Lane <tgl@sss.pgh.pa.us> writes:
>
> > > And no, "use syslog" doesn't count.
> >
> > Why not?
>
>The standard implementations of syslog lose log entries under heavy
>load, because they rely on a daemon which reads from a named pipe with
>a limited buffer space.  This is not acceptable in a production
>system, since heavy load is often just the time you need to see the
>log entries.
>
>It would be possible to implement the syslog(3) interface in a
>different way, of course, which did not use syslogd.  I don't know of
>any such implementation.
>
>(My personal preference these days is an approach like DJB's
>daemontools, which separates the handling of log entries from the
>program doing the logging.)
>
>Ian

Greetings,

Kind of ironic, I have been working on a similar logging system for Apache 
that works with PostgreSQL, and I just released 2.0-beta last night.  My 
post to announcements was delayed, but you can check it out here: 
http://www.digitalstratum.com/pglogd/

If pgLOGd looks like something similar to what you are looking for, I could 
probably modify it to log for PostgreSQL.  Two of its requirements during 
development were fast and robust, and similar to what you described above 
it does not "process" the entries, that is done later.  You also got me 
thinking that maybe syslogd needs an overhaul too...

Matthew


Thu, 06 Sep 2001 10:08:21 -0400
Matthew Hagerty <mhagerty@voyager.net>
Re: Log rotation?
You may be interested in
http://www.ietf.org/internet-drafts/draft-ietf-syslog-reliable-12.txt which
builds a reliable syslog protocol on top of BEEP.  There are free
implementations of BEEP in C and Java at http://beepcore.org

----- Original Message -----
From: "Matthew Hagerty" <mhagerty@voyager.net>
To: "Ian Lance Taylor" <ian@airs.com>; "Tom Lane" <tgl@sss.pgh.pa.us>
Cc: "Peter Eisentraut" <peter_e@gmx.net>; "PostgreSQL Development"
<pgsql-hackers@postgresql.org>
Sent: Thursday, September 06, 2001 10:08 AM
Subject: Re: [HACKERS] Log rotation?


> At 08:54 PM 9/5/2001 -0700, Ian Lance Taylor wrote:
> >Tom Lane <tgl@sss.pgh.pa.us> writes:
> >
> > > > And no, "use syslog" doesn't count.
> > >
> > > Why not?
> >
> >The standard implementations of syslog lose log entries under heavy
> >load, because they rely on a daemon which reads from a named pipe with
> >a limited buffer space.  This is not acceptable in a production
> >system, since heavy load is often just the time you need to see the
> >log entries.
> >
> >It would be possible to implement the syslog(3) interface in a
> >different way, of course, which did not use syslogd.  I don't know of
> >any such implementation.
> >
> >(My personal preference these days is an approach like DJB's
> >daemontools, which separates the handling of log entries from the
> >program doing the logging.)
> >
> >Ian
>
> Greetings,
>
> Kind of ironic, I have been working on a similar logging system for Apache
> that works with PostgreSQL, and I just released 2.0-beta last night.  My
> post to announcements was delayed, but you can check it out here:
> http://www.digitalstratum.com/pglogd/
>
> If pgLOGd looks like something similar to what you are looking for, I
could
> probably modify it to log for PostgreSQL.  Two of its requirements during
> development were fast and robust, and similar to what you described above
> it does not "process" the entries, that is done later.  You also got me
> thinking that maybe syslogd needs an overhaul too...
>
> Matthew
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org


Thu, 6 Sep 2001 10:48:52 -0400
"Ken Hirsch" <kenhirsch@myself.com>
Re: Log rotation?
Tom Lane writes:

> > What we probably want is some configure switch that switches between the
> > current behaviour and the behaviour you want.
>
> I'd suggest --prefix-like options to determine installation locations
> for the perl and python modules,

Basically, I was envisioning some option like

--enable-local-installation-layout
--enable-playpen-installation

just something, um, better.  (If we name them --with-perldir, then 51% of
the users will think that's the location where Perl itself is installed.)

Actually, if you do opt for the "playpen" version then the actual choice
of installation directory shouldn't be so interesting.  The only
reasonable place is ${libdir}/postgresql, unless you want to make up your
own file system standard.

> plus options on the order of --no-install-perl (ie, build it, but
> don't install it).

This is currently the default behaviour, if you recall. ;-)

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Thu, 6 Sep 2001 17:45:25 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Build problem with CVS version 
> Tom Lane writes:
> 
> > > What we probably want is some configure switch that switches between the
> > > current behaviour and the behaviour you want.
> >
> > I'd suggest --prefix-like options to determine installation locations
> > for the perl and python modules,
> 
> Basically, I was envisioning some option like
> 
> --enable-local-installation-layout
> --enable-playpen-installation


I'd point out this from the INSTALL document:
     --prefix=PREFIX

          Install all files under the directory PREFIX instead of
          /usr/local/pgsql. The actual files will be installed into 
various
          subdirectories; no files will ever be installed directly into 
the
          PREFIX directory.

          If you have special needs, you can also customize the 
individual
          subdirectories with the following options.

This is entirely consistent with the way other software that uses the 
same configuration procedure.

I contend that if a user wants different behaviour the onus is on the 
user to specify that.

I've no argument with those who'd make it easy to specify that 
different behaviour with, perhaps, --disable-perl-install as a 
configure option.


Installing everything under --prefix (as the document says) would also 
help package builders; the current rpm looks pretty horrible (and 
that's why I didn't pursue THAT path).


> 
> just something, um, better.  (If we name them --with-perldir, then 51% of
> the users will think that's the location where Perl itself is installed.)
> 
> Actually, if you do opt for the "playpen" version then the actual choice
> of installation directory shouldn't be so interesting.  The only
> reasonable place is ${libdir}/postgresql, unless you want to make up your
> own file system standard.
> 
> > plus options on the order of --no-install-perl (ie, build it, but
> > don't install it).
> 
> This is currently the default behaviour, if you recall. ;-)

Actually the reason it gave for not installing the perl bits is that I 
didn't have the authority. It would have been completely happy if I'd 
been root.

And I wouldn't have.



-- 
Cheers
John Summerfield

Microsoft's most solid OS: http://www.geocities.com/rcwoolley/

Note: mail delivered to me is deemed to be intended for me, for my 
disposition.




Fri, 07 Sep 2001 07:46:30 +0800
John Summerfield <summer@os2.ami.com.au>
Re: Build problem with CVS version 
Yeah, I use FreeBSD's wonderful newsyslog utility, and I do my logging like
this:

su -l pgsql -c '[ -d ${PGDATA} ] && exec /usr/local/bin/pg_ctl
start -s -w -o "-i" -l /var/log/pgsql.log'

And my /etc/newsyslog.conf entry:

/var/log/pgsql.log      pgsql:pgsql     600  3     4096 *     Z

Chris

> -----Original Message-----
> From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Peter Eisentraut
> Sent: Thursday, 6 September 2001 6:04 PM
> To: Christopher Kings-Lynne
> Cc: PostgreSQL Development
> Subject: Re: [HACKERS] Log rotation?
>
>
> Christopher Kings-Lynne writes:
>
> > What's the problem with using newsyslog or logrotate at the
> moment?  (ie.
> > use the system log rotator)
>
> The postmaster will never close the output file, so you can rotate all you
> want, the original file will never be abandoned.
>
> --
> Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://www.postgresql.org/search.mpl
>


Fri, 7 Sep 2001 10:06:19 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: Log rotation?
Christopher Kings-Lynne writes:

> Yeah, I use FreeBSD's wonderful newsyslog utility, and I do my logging like
> this:
>
> su -l pgsql -c '[ -d ${PGDATA} ] && exec /usr/local/bin/pg_ctl
> start -s -w -o "-i" -l /var/log/pgsql.log'
>
> And my /etc/newsyslog.conf entry:
>
> /var/log/pgsql.log      pgsql:pgsql     600  3     4096 *     Z

Sorry, this does not convey any information to me.  Does this newsyslog
thing do anything that's so smart that we should know about it?

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Fri, 7 Sep 2001 16:04:24 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Log rotation?

Hi Peter,

> I've been playing with a little program I wrote whose sole purpose is to
> write its stdin to a file and close and reopen that file when it receives
> a signal.  I figured this could work well when integrated transparently
> into pg_ctl.
> 
> So, is log rotation a concern?  Is this a reasonable solution?  Other
> ideas?

There was a discussion of this over a year ago.  After I contributed
to the discussion Tom Lane suggested I write something, and in the
tradition of software development I did so but never quite finished it
... I got to the point that it works for me, then got distracted
before completing a test suite.

You may well prefer your own code, but this one supports rotation on
size and/or time basis as well as on receipt of SIGHUP, and places a
timestamp on each line written. It's also pretty careful about errors,
which is one of the things that was disliked about the Apache program
last time it was discussed.

I am happy to contribute the code using the standard PostgreSQL
license if it's wanted.  (If anyone wants it under a BSD license or
GPL for another purpose that's fine too.)

I use the code on HP-UX, and developed it on NetBSD.  There shouldn't
be too many portability problems lurking, other than the usual hassles
of what % escape to use in printf() for off_t.  I doubt anyone wants
log files larger than a couple of GB anyway? :-)

ftp://ftp.nemeton.com.au/pub/src/logwrite-1.0alpha.tar.gz

> (No Grand Unified Logging Solutions please.  And no, "use syslog" doesn't
> count.)

<grin>

One improvement I suggest is that the postmaster be taught to start
(and restart if necessary) the log program.  This avoids fragile
startup scripts and also avoids taking down PostgreSQL if someone
sends the wrong signal to the log program.

Cheers,

Giles


Sat, 08 Sep 2001 07:21:19 +1000
Giles Lean <giles@nemeton.com.au>
Re: Log rotation? 
On Thursday 06 September 2001 07:46 pm, John Summerfield wrote:
> I'd point out this from the INSTALL document:
>      --prefix=PREFIX
[snip]
> Installing everything under --prefix (as the document says) would also
> help package builders; the current rpm looks pretty horrible (and
> that's why I didn't pursue THAT path).

Blame the Linux Filesystem Hierarchy Standard and the Linux Standards Base 
for the directory structure of the RPMset.  Things have to be in specified 
locations to be FHS compliant -- and the current RPMset is, to the best of my 
knowledge, FHS compliant.

Horrible looking is in the eye of the beholder, BTW.  Some think that a 
separate prefix for each software package looks horrible and that the 
intermingled nature of the FHS looks cleaner (which it does from a PATH point 
of view, for sure).  

It just comes down to the OS philosophy you deal with.  BSD ports, for 
instance, has a very different package philosophy from Solaris, which is 
different yet from Domain/OS, which is different yet from the LSB.  
PostgreSQL is very BSDish -- and, while I have to cut cartwheels to finagle 
it into an LSB mold, being BSDish is not a bad thing.  Neither is being 
LSBish -- both have their place. The is no One True Filesystem spec...

As to the can't install perl client issue, the install can fail, even if done 
as root, if you are building for installation at some other location in the 
filesystem.

The perl client install phase fails every time I build an RPMset -- the RPM 
spec file has to go through some steps to make it work.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 10 Sep 2001 11:17:28 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] Build problem with CVS version
John Summerfield writes:

> I'd point out this from the INSTALL document:
>      --prefix=PREFIX
>
>           Install all files under the directory PREFIX instead of
>           /usr/local/pgsql. The actual files will be installed into various
>           subdirectories; no files will ever be installed directly into the
>           PREFIX directory.
>
>           If you have special needs, you can also customize the individual
>           subdirectories with the following options.

But there are also exceptions listed at --with-perl and --with-python.

> This is entirely consistent with the way other software that uses the
> same configuration procedure.

I am not aware of a package that installs general-purpose Perl/Python
modules as well as items from outside of those environments.

> I contend that if a user wants different behaviour the onus is on the
> user to specify that.

You're probably right, but I suspect that the majority of users rightly
expect the Perl module to go where Perl modules usually go.  This wouldn't
be such an interesting question if Perl provided a way to add to the
module search path (cf. LD_LIBRARY_PATH and such), but to my knowledge
there isn't a generally accepted one, so this issue would introduce an
annoyance for users.

Surely the current behaviour is an annoyance too, making the whole issue a
rather unpleasant subject. ;-)

Well, as soon as I come up with a name for the install-where-I-tell-you-to
option, I'll implement it.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Tue, 11 Sep 2001 02:25:06 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: [BUGS] Build problem with CVS version 
There was a discussion about --enable-syslog by default. What was the
consensus? I think this is a good one.
--
Tatsuo Ishii

Tue, 11 Sep 2001 13:29:58 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
syslog by default?
> There was a discussion about --enable-syslog by default. What was the
> consensus? I think this is a good one.

Yes, I thought we decided it should be the default too.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 11 Sep 2001 01:07:33 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: syslog by default?
On Tue, 11 Sep 2001, Bruce Momjian wrote:

> > There was a discussion about --enable-syslog by default. What was the
> > consensus? I think this is a good one.
>
> Yes, I thought we decided it should be the default too.

I know it can have an adverse effect on a mail server, is syslog going
to give us any performance hits?

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
         56K Nationwide Dialup from $16.00/mo at Pop4 Networking
        Online Campground Directory    http://www.camping-usa.com
       Online Giftshop Superstore    http://www.cloudninegifts.com
==========================================================================




Tue, 11 Sep 2001 06:18:43 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: syslog by default?
> John Summerfield writes:
> 
> > I'd point out this from the INSTALL document:
> >      --prefix=PREFIX
> >
> >           Install all files under the directory PREFIX instead of
> >           /usr/local/pgsql. The actual files will be installed into various
> >           subdirectories; no files will ever be installed directly into the
> >           PREFIX directory.
> >
> >           If you have special needs, you can also customize the individual
> >           subdirectories with the following options.
> 
> But there are also exceptions listed at --with-perl and --with-python.
> 
> > This is entirely consistent with the way other software that uses the
> > same configuration procedure.
> 
> I am not aware of a package that installs general-purpose Perl/Python
> modules as well as items from outside of those environments.
> 
> > I contend that if a user wants different behaviour the onus is on the
> > user to specify that.
> 
> You're probably right, but I suspect that the majority of users rightly
> expect the Perl module to go where Perl modules usually go.  This wouldn't

that isn't reasonable if the installer's not root, and I wasn't, for 
the precise reason I didn't want to update the system.

> be such an interesting question if Perl provided a way to add to the
> module search path (cf. LD_LIBRARY_PATH and such), but to my knowledge
> there isn't a generally accepted one, so this issue would introduce an
> annoyance for users.

from 'man perlrun'
    PERL5LIB    A colon-separated list of directories in which to look 
for Perl library files
                   before looking in the standard library and the 
current directory.  Any architec-
                   ture-specific directories under the specified 
locations are automatically
                   included if they exist.  If PERL5LIB is not defined, 
PERLLIB is used.
 
                   When running taint checks (either because the 
program was running setuid or set-
                   gid, or the -T switch was used), neither variable is 
used.  The program should
                   instead say:

There are several other environment variables.




> 
> Surely the current behaviour is an annoyance too, making the whole issue a
> rather unpleasant subject. ;-)
> 
> Well, as soon as I come up with a name for the install-where-I-tell-you-to
> option, I'll implement it.



The current behaviour makes it difficult to have two versions on one 
computer.
-- 
Cheers
John Summerfield

Microsoft's most solid OS: http://www.geocities.com/rcwoolley/

Note: mail delivered to me is deemed to be intended for me, for my 
disposition.




Tue, 11 Sep 2001 21:08:36 +0800
John Summerfield <summer@os2.ami.com.au>
Re: [BUGS] Build problem with CVS version 
Tatsuo Ishii writes:

> There was a discussion about --enable-syslog by default. What was the
> consensus? I think this is a good one.

It would be a good one if we make the blind assumption that syslog()
exists on all platforms.  That is possible, but not guaranteed.  (BeOS,
QNX, Cygwin?)

The alternative suggestion to turn it off if syslog() is not found makes
me wary.  These schemes have invariably lead to problems in the past.
Recall libpq++ being missed because of false test results, readline
support mysteriously disappearing and nobody noticing until Mandrake had
shipped their CDs.  There are possible scenarios where syslog support
could be missed by configure, such as when you need some compat or bsd
library.

An alternative scheme I wanted to implement for readline is

--enable-foo => force feature to be used
--disable-foo => force feature not to be used
<nothing> => use feature if available

but I'm afraid that this would create more confusion than it's worth
because a prudent user would specify --enable-foo anyway.

I'd rather type a few more things and get predictable behavior from
configure rather than relying on it to pick the features for me.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Tue, 11 Sep 2001 15:10:42 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: syslog by default?

> I know it can have an adverse effect on a mail server, is syslog going
> to give us any performance hits?

Yes.  On some platforms (HP-UX at least) applications can stall ~2s
retrying if syslogd is not reading the messages written to its pipe.

syslogd also has a reputation for using too much CPU under load, but
this is annecdotal only -- I've not seen such a situation myself.

Regards,

Giles


Wed, 12 Sep 2001 07:56:11 +1000
Giles Lean <giles@nemeton.com.au>
Re: syslog by default? 
On Mar 11 Sep 2001 02:07, Bruce Momjian wrote:
> > There was a discussion about --enable-syslog by default. What was the
> > consensus? I think this is a good one.
>
> Yes, I thought we decided it should be the default too.

There was a discusion about log rotation last week, so, where are we going? 
Pipe the output of postmaster to a log rotator like apaches logrotate, or are 
we going to use syslog and have the syslog log rotator do the rotation?

Just a dought I had.

Saludos... :-)

-- 
Porquï¿½ usar una base de datos relacional cualquiera,
si podï¿½s usar PostgreSQL?
-----------------------------------------------------------------
Martï¿½n Marquï¿½s                  |        mmarques@unl.edu.ar
Programador, Administrador, DBA |       Centro de Telematica
                       Universidad Nacional
                            del Litoral
-----------------------------------------------------------------

Tue, 11 Sep 2001 19:14:54 -0300
=?iso-8859-1?q?Mart=EDn=20Marqu=E9s?= <martin@bugs.unl.edu.ar>
Re: syslog by default?
On Tuesday 11 September 2001 06:14 pm, Martï¿½n Marquï¿½s wrote:
> There was a discusion about log rotation last week, so, where are we going?
> Pipe the output of postmaster to a log rotator like apaches logrotate, or
> are we going to use syslog and have the syslog log rotator do the rotation?

Both have their place.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 11 Sep 2001 19:11:52 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: syslog by default?
> Tatsuo Ishii writes:
> 
> > There was a discussion about --enable-syslog by default. What was the
> > consensus? I think this is a good one.
> 
> It would be a good one if we make the blind assumption that syslog()
> exists on all platforms.  That is possible, but not guaranteed.  (BeOS,
> QNX, Cygwin?)
> 
> The alternative suggestion to turn it off if syslog() is not found makes
> me wary.  These schemes have invariably lead to problems in the past.
> Recall libpq++ being missed because of false test results, readline
> support mysteriously disappearing and nobody noticing until Mandrake had
> shipped their CDs.  There are possible scenarios where syslog support
> could be missed by configure, such as when you need some compat or bsd
> library.

Why are you so worrying about finding syslog() in configure? We have
already done lots of function testings. Is there anything special with
syslog()?
--
Tatsuo Ishii


Wed, 12 Sep 2001 09:59:50 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: syslog by default?
Tatsuo Ishii writes:

> Why are you so worrying about finding syslog() in configure? We have
> already done lots of function testings. Is there anything special with
> syslog()?

All the other functions we test for come with a replacement plan.  Either
we choose between several similar alternatives (atexit/on_exit), or we
link in our own function.o file.  But we don't shut off a whole piece of
functionality when one goes missing (except in the cases I mentioned).

I'm probably being paranoid.  But you did ask what became of the idea, and
that's what did, as far as I'm concerned...

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter


Wed, 12 Sep 2001 14:07:40 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: syslog by default?
> Tatsuo Ishii writes:
> 
> > Why are you so worrying about finding syslog() in configure? We have
> > already done lots of function testings. Is there anything special with
> > syslog()?
> 
> All the other functions we test for come with a replacement plan.  Either
> we choose between several similar alternatives (atexit/on_exit), or we
> link in our own function.o file.  But we don't shut off a whole piece of
> functionality when one goes missing (except in the cases I mentioned).
> 
> I'm probably being paranoid.  But you did ask what became of the idea, and
> that's what did, as far as I'm concerned...

OK, that makes sense.  My only question is how many platforms _don't_
have syslog.  If it is only NT and QNX, I think we can live with using
it by default if it exists.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 12 Sep 2001 10:20:38 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: syslog by default?
On Sep 12, pgman@candle.pha.pa.us contorted a few electrons to say...
Bruce> OK, that makes sense.  My only question is how many platforms _don't_
Bruce> have syslog.  If it is only NT and QNX, I think we can live with using
Bruce> it by default if it exists.

perhaps you could take some code from

	http://freshmeat.net/projects/cpslapi/

which implements a syslog-api that writes to NT's eventlog.

i'd be glad to change the license if it is useful.

jr

-- 
------------------------------------------------------------------------
Joel W. Reed                                                412-257-3881
--------------All the simple programs have been written.----------------
Fri, 14 Sep 2001 09:25:27 -0400
"Joel W. Reed" <jreed@support.ddiworld.com>
Re: syslog by default?
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> OK, that makes sense.  My only question is how many platforms _don't_
> have syslog.  If it is only NT and QNX, I think we can live with using
> it by default if it exists.

There seems to be a certain amount of confusion here.  The proposal at
hand was to make configure set up to *compile* the syslog support
whenever possible.  Not to *use* syslog by default.  Unless we change
the default postgresql.conf --- which I would be against --- we will
still log to stderr by default.

Given that, I'm not sure that Peter's argument about losing
functionality is right; the analogy to readline support isn't exact.
Perhaps what we should do is (a) always build syslog support if
possible, and (b) at runtime, complain if syslog logging is requested
but we don't have it available.

			regards, tom lane

Fri, 14 Sep 2001 18:05:05 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: syslog by default? 
At the just-past OSDN database conference, Bruce and I were annoyed by
some benchmark results showing that Postgres performed poorly on an
8-way SMP machine.  Based on past discussion, it seems likely that the
culprit is the known inefficiency in our spinlock implementation.
After chewing on it for awhile, we came up with an idea for a solution.

The following proposal should improve performance substantially when
there is contention for a lock, but it creates no portability risks
because it uses the same system facilities (TAS and SysV semaphores)
that we have always relied on.  Also, I think it'd be fairly easy to
implement --- I could probably get it done in a day.

Comments anyone?

			regards, tom lane


Plan:

Replace most uses of spinlocks with "lightweight locks" (LW locks)
implemented by a new lock manager.  The principal remaining use of true
spinlocks (TAS locks) will be to provide mutual exclusion of access to
LW lock structures.  Therefore, we can assume that spinlocks are never
held for more than a few dozen instructions --- and never across a kernel
call.

It's pretty easy to rejigger the spinlock code to work well when the lock
is never held for long.  We just need to change the spinlock retry code
so that it does a tight spin (continuous retry) for a few dozen cycles ---
ideally, the total delay should be some small multiple of the max expected
lock hold time.  If lock still not acquired, yield the CPU via a select()
call (10 msec minimum delay) and repeat.  Although this looks inefficient,
it doesn't matter on a uniprocessor because we expect that backends will
only rarely be interrupted while holding the lock, so in practice a held
lock will seldom be encountered.  On SMP machines the tight spin will win
since the lock will normally become available before we give up and yield
the CPU.

Desired properties of the LW lock manager include:
	* very fast fall-through when no contention for lock
	* waiting proc does not spin
	* support both exclusive and shared (read-only) lock modes
	* grant lock to waiters in arrival order (no starvation)
	* small lock structure to allow many LW locks to exist.

Proposed contents of LW lock structure:

	spinlock mutex (protects LW lock state and PROC queue links)
	count of exclusive holders (always 0 or 1)
	count of shared holders (0 .. MaxBackends)
	queue head pointer (NULL or ptr to PROC object)
	queue tail pointer (could do without this to save space)

If a backend sees it must wait to acquire the lock, it adds its PROC
struct to the end of the queue, releases the spinlock mutex, and then
sleeps by P'ing its per-backend wait semaphore.  A backend releasing the
lock will check to see if any waiter should be granted the lock.  If so,
it will update the lock state, release the spinlock mutex, and finally V
the wait semaphores of any backends that it decided should be released
(which it removed from the lock's queue while holding the sema).  Notice
that no kernel calls need be done while holding the spinlock.  Since the
wait semaphore will remember a V occurring before P, there's no problem
if the releaser is fast enough to release the waiter before the waiter
reaches its P operation.

We will need to add a few fields to PROC structures:
	* Flag to show whether PROC is waiting for an LW lock, and if so
	  whether it waits for read or write access
	* Additional PROC queue link field.
We can't reuse the existing queue link field because it is possible for a
PROC to be waiting for both a heavyweight lock and a lightweight one ---
this will occur when HandleDeadLock or LockWaitCancel tries to acquire
the LockMgr module's lightweight lock (formerly spinlock).

It might seem that we also need to create a second wait semaphore per
backend, one to wait on HW locks and one to wait on LW locks.  But I
believe we can get away with just one, by recognizing that a wait for an
LW lock can never be interrupted by a wait for a HW lock, only vice versa.
After being awoken (V'd), the LW lock manager must check to see if it was
actually granted the lock (easiest way: look at own PROC struct to see if
LW lock wait flag has been cleared).  If not, the V must have been to
grant us a HW lock --- but we still have to sleep to get the LW lock.  So
remember this happened, then loop back and P again.  When we finally get
the LW lock, if there was an extra P operation then V the semaphore once
before returning.  This will allow ProcSleep to exit the wait for the HW
lock when we return to it.

Fine points:

While waiting for an LW lock, we need to show in our PROC struct whether
we are waiting for read or write access.  But we don't need to remember
this after getting the lock; if we know we have the lock, it's easy to
see by inspecting the lock whether we hold read or write access.

ProcStructLock cannot be replaced by an LW lock, since a backend cannot
use an LW lock until it has obtained a PROC struct and a semaphore,
both of which are protected by this lock.  It seems okay to use a plain
spinlock for this purpose.  NOTE: it's okay for SInvalLock to be an LW
lock, as long as the LW mgr does not depend on accessing the SI array
of PROC objects, but only chains through the PROCs themselves.

Another tricky point is that some of the setup code executed by the
postmaster may try to to grab/release LW locks.  Here, we can probably
allow a special case for MyProc=NULL.  It's likely that we should never
see a block under these circumstances anyway, so finding MyProc=NULL when
we need to block may just be a fatal error condition.

A nastier case is checkpoint processes; these expect to grab BufMgr and
WAL locks.  Perhaps okay for them to do plain sleeps in between attempts
to grab the locks?  This says that the MyProc=NULL case should release
the spinlock mutex, sleep 10 msec, try again, rather than any sort of error
or expectation of no conflict.  Are there any cases where this represents
a horrid performance loss?  Checkpoint itself seems noncritical.

Alternative is for checkpoint to be allowed to create a PROC struct (but
not to enter it in SI list) so's it can participate normally in LW lock
operations.  That seems a good idea anyway, actually, so that the PROC
struct's facility for releasing held LW locks at elog time will work
inside the checkpointer.  (But that means we need an extra sema too?
Okay, but don't want an extra would-be backend to obtain the extra sema
and perhaps cause a checkpoint proc to fail.  So must allocate the PROC
and sema for checkpoint process separately from those reserved for
backends.)

Wed, 26 Sep 2001 12:10:00 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Spinlock performance improvement proposal

Sounds cool to me ... definitely something to fix before v7.2, if its as
"easy" as you make it sound ... I'm expecting the new drive to be
installed today (if all goes well ... Thomas still has his date/time stuff
to finish off, now that CVSup is fixed ...

Let''s try and target Monday for Beta then?  I think the only two
outstaandings are you and Thomas right now?

Bruce, that latest rtree patch looks intriguing also ... can anyone
comment positive/negative about it, so that we can try and get that in
before Beta?

On Wed, 26 Sep 2001, Tom Lane wrote:

> At the just-past OSDN database conference, Bruce and I were annoyed by
> some benchmark results showing that Postgres performed poorly on an
> 8-way SMP machine.  Based on past discussion, it seems likely that the
> culprit is the known inefficiency in our spinlock implementation.
> After chewing on it for awhile, we came up with an idea for a solution.
>
> The following proposal should improve performance substantially when
> there is contention for a lock, but it creates no portability risks
> because it uses the same system facilities (TAS and SysV semaphores)
> that we have always relied on.  Also, I think it'd be fairly easy to
> implement --- I could probably get it done in a day.
>
> Comments anyone?
>
> 			regards, tom lane
>
>
> Plan:
>
> Replace most uses of spinlocks with "lightweight locks" (LW locks)
> implemented by a new lock manager.  The principal remaining use of true
> spinlocks (TAS locks) will be to provide mutual exclusion of access to
> LW lock structures.  Therefore, we can assume that spinlocks are never
> held for more than a few dozen instructions --- and never across a kernel
> call.
>
> It's pretty easy to rejigger the spinlock code to work well when the lock
> is never held for long.  We just need to change the spinlock retry code
> so that it does a tight spin (continuous retry) for a few dozen cycles ---
> ideally, the total delay should be some small multiple of the max expected
> lock hold time.  If lock still not acquired, yield the CPU via a select()
> call (10 msec minimum delay) and repeat.  Although this looks inefficient,
> it doesn't matter on a uniprocessor because we expect that backends will
> only rarely be interrupted while holding the lock, so in practice a held
> lock will seldom be encountered.  On SMP machines the tight spin will win
> since the lock will normally become available before we give up and yield
> the CPU.
>
> Desired properties of the LW lock manager include:
> 	* very fast fall-through when no contention for lock
> 	* waiting proc does not spin
> 	* support both exclusive and shared (read-only) lock modes
> 	* grant lock to waiters in arrival order (no starvation)
> 	* small lock structure to allow many LW locks to exist.
>
> Proposed contents of LW lock structure:
>
> 	spinlock mutex (protects LW lock state and PROC queue links)
> 	count of exclusive holders (always 0 or 1)
> 	count of shared holders (0 .. MaxBackends)
> 	queue head pointer (NULL or ptr to PROC object)
> 	queue tail pointer (could do without this to save space)
>
> If a backend sees it must wait to acquire the lock, it adds its PROC
> struct to the end of the queue, releases the spinlock mutex, and then
> sleeps by P'ing its per-backend wait semaphore.  A backend releasing the
> lock will check to see if any waiter should be granted the lock.  If so,
> it will update the lock state, release the spinlock mutex, and finally V
> the wait semaphores of any backends that it decided should be released
> (which it removed from the lock's queue while holding the sema).  Notice
> that no kernel calls need be done while holding the spinlock.  Since the
> wait semaphore will remember a V occurring before P, there's no problem
> if the releaser is fast enough to release the waiter before the waiter
> reaches its P operation.
>
> We will need to add a few fields to PROC structures:
> 	* Flag to show whether PROC is waiting for an LW lock, and if so
> 	  whether it waits for read or write access
> 	* Additional PROC queue link field.
> We can't reuse the existing queue link field because it is possible for a
> PROC to be waiting for both a heavyweight lock and a lightweight one ---
> this will occur when HandleDeadLock or LockWaitCancel tries to acquire
> the LockMgr module's lightweight lock (formerly spinlock).
>
> It might seem that we also need to create a second wait semaphore per
> backend, one to wait on HW locks and one to wait on LW locks.  But I
> believe we can get away with just one, by recognizing that a wait for an
> LW lock can never be interrupted by a wait for a HW lock, only vice versa.
> After being awoken (V'd), the LW lock manager must check to see if it was
> actually granted the lock (easiest way: look at own PROC struct to see if
> LW lock wait flag has been cleared).  If not, the V must have been to
> grant us a HW lock --- but we still have to sleep to get the LW lock.  So
> remember this happened, then loop back and P again.  When we finally get
> the LW lock, if there was an extra P operation then V the semaphore once
> before returning.  This will allow ProcSleep to exit the wait for the HW
> lock when we return to it.
>
> Fine points:
>
> While waiting for an LW lock, we need to show in our PROC struct whether
> we are waiting for read or write access.  But we don't need to remember
> this after getting the lock; if we know we have the lock, it's easy to
> see by inspecting the lock whether we hold read or write access.
>
> ProcStructLock cannot be replaced by an LW lock, since a backend cannot
> use an LW lock until it has obtained a PROC struct and a semaphore,
> both of which are protected by this lock.  It seems okay to use a plain
> spinlock for this purpose.  NOTE: it's okay for SInvalLock to be an LW
> lock, as long as the LW mgr does not depend on accessing the SI array
> of PROC objects, but only chains through the PROCs themselves.
>
> Another tricky point is that some of the setup code executed by the
> postmaster may try to to grab/release LW locks.  Here, we can probably
> allow a special case for MyProc=NULL.  It's likely that we should never
> see a block under these circumstances anyway, so finding MyProc=NULL when
> we need to block may just be a fatal error condition.
>
> A nastier case is checkpoint processes; these expect to grab BufMgr and
> WAL locks.  Perhaps okay for them to do plain sleeps in between attempts
> to grab the locks?  This says that the MyProc=NULL case should release
> the spinlock mutex, sleep 10 msec, try again, rather than any sort of error
> or expectation of no conflict.  Are there any cases where this represents
> a horrid performance loss?  Checkpoint itself seems noncritical.
>
> Alternative is for checkpoint to be allowed to create a PROC struct (but
> not to enter it in SI list) so's it can participate normally in LW lock
> operations.  That seems a good idea anyway, actually, so that the PROC
> struct's facility for releasing held LW locks at elog time will work
> inside the checkpointer.  (But that means we need an extra sema too?
> Okay, but don't want an extra would-be backend to obtain the extra sema
> and perhaps cause a checkpoint proc to fail.  So must allocate the PROC
> and sema for checkpoint process separately from those reserved for
> backends.)
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>


Wed, 26 Sep 2001 13:18:38 -0400 (EDT)
"Marc G. Fournier" <scrappy@hub.org>
Re: Spinlock performance improvement proposal
"Marc G. Fournier" <scrappy@hub.org> writes:
> Let''s try and target Monday for Beta then?

Sounds like a plan.

			regards, tom lane

Wed, 26 Sep 2001 13:22:48 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Spinlock performance improvement proposal 

The plan for the new spinlocks does look like it has some potential.  My 
only comment in regards to permformance when we start looking at SMP 
machines is ... it is my belief that getting a true threaded backend may 
be the only way to get the full potential out of SMP machines.  I see that 
is one of the things to experiment with on the TODO list and I have seen 
some people have messed around already with this using Solaris threads.  
It should probably be attempted with pthreads if PostgreSQL is going to 
keep some resemblance of cross-platform compatibility.  At that time, it 
would probably be easier to go in and clean up some stuff for the 
implementation of other TODO items (put in the base framework for more 
complex future items) as threading the backend would take a little bit of 
ideology shift.

Of course, it is much easier to stand back and talk about this then 
actually do it  - especially comming from someone who has only tried to 
contribute a few pieces of code.  Keep up the good work.


On Wed, 26 Sep 2001, Tom Lane wrote:

> At the just-past OSDN database conference, Bruce and I were annoyed by
> some benchmark results showing that Postgres performed poorly on an
> 8-way SMP machine.  Based on past discussion, it seems likely that the
> culprit is the known inefficiency in our spinlock implementation.
> After chewing on it for awhile, we came up with an idea for a solution.
> 
> The following proposal should improve performance substantially when
> there is contention for a lock, but it creates no portability risks
> because it uses the same system facilities (TAS and SysV semaphores)
> that we have always relied on.  Also, I think it'd be fairly easy to
> implement --- I could probably get it done in a day.
> 
> Comments anyone?
> 
> 			regards, tom lane

-- 
//========================================================\\
||  D. Hageman                    <dhageman@dracken.com>  ||
\\========================================================//




Wed, 26 Sep 2001 13:40:46 -0500 (CDT)
"D. Hageman" <dhageman@dracken.com>
Re: Spinlock performance improvement proposal
Tom Lane wrote:
> 
> At the just-past OSDN database conference, Bruce and I were annoyed by
> some benchmark results showing that Postgres performed poorly on an
> 8-way SMP machine.  Based on past discussion, it seems likely that the
> culprit is the known inefficiency in our spinlock implementation.
> After chewing on it for awhile, we came up with an idea for a solution.
> 
> The following proposal should improve performance substantially when
> there is contention for a lock, but it creates no portability risks
> because it uses the same system facilities (TAS and SysV semaphores)
> that we have always relied on.  Also, I think it'd be fairly easy to
> implement --- I could probably get it done in a day.
> 
> Comments anyone?


We have been doing some scalability testing just recently here at Red
Hat. The machine I was using was a 4-way 550 MHz Xeon SMP machine, I
also ran the machine in uniprocessor mode to make some comparisons. All
runs were made on Red Hat Linux running 2.4.x series kernels. I've
examined a number of potentially interesting cases -- I'm still
analyzing the results, but some of the initial results might be
interesting:

- We have tried benchmarking the following: TAS spinlocks (existing
implementation), SysV semaphores (existing implementation), Pthread
Mutexes. Pgbench runs were conducted for 1 to 512 simultaneous backends.

  For these three cases we found:
  - TAS spinlocks fared the best of all three lock types, however above
100 clients the Pthread mutexes were lock step in performance. I expect
this is due to the cost of any system calls being      negligible
relative to lock wait time.
  - SysV semaphore implementation faired terribly as expected. However,
it is worse, relative to the TAS spinlocks on SMP than on uniprocessor.

- Since the above seemed to indicate that the lock implementation may
not be the problem (Pthread mutexes are supposed to be implemented to be
less bang-bang than the Postgres TAS spinlocks, IIRC), I decided to
profile Postgres. After much trouble, I got results for it using
oprofile, a kernel profiler for Linux. Unfortunately, I can only profile
for uniprocessor right now using oprofile, as it doesn't support SMP
boxes yet. (soon, I hope.)

Initial results (top five -- if you would like a complete profile, let
me know):
Each sample counts as 1 samples.
  %   cumulative   self              self     total           
 time   samples   samples    calls  T1/call  T1/call  name    
 26.57  42255.02 42255.02                            
FindLockCycleRecurse
  5.55  51081.02  8826.00                             s_lock_sleep
  5.07  59145.03  8064.00                             heapgettup
  4.48  66274.03  7129.00                             hash_search
  4.48  73397.03  7123.00                             s_lock
  2.85  77926.03  4529.00                            
HeapTupleSatisfiesSnapshot
  2.07  81217.04  3291.00                             SHMQueueNext
  1.85  84154.04  2937.00                             AllocSetAlloc
  1.84  87085.04  2931.00                             fmgr_isbuiltin
  1.64  89696.04  2611.00                             set_ps_display
  1.51  92101.04  2405.00                             FunctionCall2
  1.47  94442.04  2341.00                             XLogInsert
  1.39  96649.04  2207.00                             _bt_compare
  1.22  98597.04  1948.00                             SpinAcquire
  1.22 100544.04  1947.00                             LockBuffer
  1.21 102469.04  1925.00                             tag_hash
  1.01 104078.05  1609.00                             LockAcquire
.
.
.

(The samples are proportional to execution time.)

This would seem to point to the deadlock detector. (Which some have
fingered as a possible culprit before, IIRC.)

However, this seems to be a red herring. Removing the deadlock detector
had no effect. In fact, benchmarking showed removing it yielded no
improvement in transaction processing rate on uniprocessor or SMP
systems. Instead, it seems that the deadlock detector simply amounts to
"something to do" for the blocked backend while it waits for lock
acquisition. 

Profiling bears this out:

Flat profile:

Each sample counts as 1 samples.
  %   cumulative   self              self     total           
 time   samples   samples    calls  T1/call  T1/call  name    
 12.38  14112.01 14112.01                             s_lock_sleep
 10.18  25710.01 11598.01                             s_lock
  6.47  33079.01  7369.00                             hash_search
  5.88  39784.02  6705.00                             heapgettup
  5.32  45843.02  6059.00                            
HeapTupleSatisfiesSnapshot 
  2.62  48830.02  2987.00                             AllocSetAlloc
  2.48  51654.02  2824.00                             fmgr_isbuiltin
  1.89  53813.02  2159.00                             XLogInsert
  1.86  55938.02  2125.00                             _bt_compare
  1.72  57893.03  1955.00                             SpinAcquire
  1.61  59733.03  1840.00                             LockBuffer
  1.60  61560.03  1827.00                             FunctionCall2
  1.56  63339.03  1779.00                             tag_hash
  1.46  65007.03  1668.00                             set_ps_display
  1.20  66372.03  1365.00                             SearchCatCache
  1.14  67666.03  1294.00                             LockAcquire
. 
.
.

Our current suspicion isn't that the lock implementation is the only
problem (though there is certainly room for improvement), or perhaps
isn't even the main problem. For example, there has been some suggestion
that perhaps some component of the database is causing large lock
contention. My opinion is that rather than guessing and taking stabs in
the dark, we need to take a more reasoned approach to these things.
IMHO, the next step should be to apply instrumentation (likely via some
neat macros) to all lock acquires / releases. Then, it will be possible
to determine what components are the greatest consumers of locks, and to
determine whether it is a component problem or a systemic problem. (i.e.
some component vs. simply just the lock implementation.)

Neil

-- 
Neil Padgett
Red Hat Canada Ltd.                       E-Mail:  npadgett@redhat.com
2323 Yonge Street, Suite #300, 
Toronto, ON  M4P 2C9

Wed, 26 Sep 2001 14:46:16 -0400
Neil Padgett <npadgett@redhat.com>
Re: Spinlock performance improvement proposal
"D. Hageman" <dhageman@dracken.com> writes:

> The plan for the new spinlocks does look like it has some potential.  My 
> only comment in regards to permformance when we start looking at SMP 
> machines is ... it is my belief that getting a true threaded backend may 
> be the only way to get the full potential out of SMP machines.

Depends on what you mean.  For scaling well with many connections and
simultaneous queries, there's no reason IMHO that the current
process-per-backend model won't do, assuming the locking issues are
addressed. 

If you're talking about making a single query use multiple CPUs, then
yes, we're probably talking about a fundamental rewrite to use threads 
or some other mechanism.

-Doug
-- 
In a world of steel-eyed death, and men who are fighting to be warm,
Come in, she said, I'll give you shelter from the storm.    -Dylan

26 Sep 2001 15:18:18 -0400
Doug McNaught <doug@wireboard.com>
Re: Spinlock performance improvement proposal
On 26 Sep 2001, Doug McNaught wrote:

> "D. Hageman" <dhageman@dracken.com> writes:
> 
> > The plan for the new spinlocks does look like it has some potential.  My 
> > only comment in regards to permformance when we start looking at SMP 
> > machines is ... it is my belief that getting a true threaded backend may 
> > be the only way to get the full potential out of SMP machines.
> 
> Depends on what you mean.  For scaling well with many connections and
> simultaneous queries, there's no reason IMHO that the current
> process-per-backend model won't do, assuming the locking issues are
> addressed. 

Well, I know the current process-per-backend model does quite well.  My 
argument is not that it fails to do as intended.  My original argument is 
that it is belief (at the momment with the knowledge I have) to get the 
full potential out of SMP machines - threads might be the way to go.  The 
data from RedHat is quite interesting, so my feelings on this might 
change or could be re-inforced.  I watch anxiously ;-)

> If you're talking about making a single query use multiple CPUs, then
> yes, we're probably talking about a fundamental rewrite to use threads 
> or some other mechanism.

Well, we have several thread model ideologies that we could chose from.  
Only experimentation would let us determine the proper path to follow and 
then it wouldn't be ideal for everyone.  You kinda just have to take the 
best scenerio and run with it.  My first inclination would be something 
like a thread per connection (to reduce connection overhead), but then we 
could run into limits on different platforms (threads per process).  I 
kinda like the idea of using a thread for replication purposes ... lots 
of interesting possibilities exist and I will be first to admit that I 
don't have all the answers.  

-- 
//========================================================\\
||  D. Hageman                    <dhageman@dracken.com>  ||
\\========================================================//


Wed, 26 Sep 2001 15:03:11 -0500 (CDT)
"D. Hageman" <dhageman@dracken.com>
Re: Spinlock performance improvement proposal
Neil Padgett <npadgett@redhat.com> writes:
> Initial results (top five -- if you would like a complete profile, let
> me know):
> Each sample counts as 1 samples.
>   %   cumulative   self              self     total           
>  time   samples   samples    calls  T1/call  T1/call  name    
>  26.57  42255.02 42255.02                             FindLockCycleRecurse

Yipes.  It would be interesting to know more about the locking pattern
of your benchmark --- are there long waits-for chains, or not?  The
present deadlock detector was certainly written with an eye to "get it
right" rather than "make it fast", but I wonder whether this shows a
performance problem in the detector, or just too many executions because
you're waiting too long to get locks.

> However, this seems to be a red herring. Removing the deadlock detector
> had no effect. In fact, benchmarking showed removing it yielded no
> improvement in transaction processing rate on uniprocessor or SMP
> systems. Instead, it seems that the deadlock detector simply amounts to
> "something to do" for the blocked backend while it waits for lock
> acquisition. 

Do you have any idea about the typical lock-acquisition delay in this
benchmark?  Our docs advise trying to set DEADLOCK_TIMEOUT higher than
the typical acquisition delay, so that the deadlock detector does not
run unnecessarily.

> For example, there has been some suggestion
> that perhaps some component of the database is causing large lock
> contention.

My thought as well.  I would certainly recommend that you use more than
one test case while looking at these things.

			regards, tom lane

Wed, 26 Sep 2001 16:05:44 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Spinlock performance improvement proposal 
"D. Hageman" wrote:

> The plan for the new spinlocks does look like it has some potential.  My
> only comment in regards to permformance when we start looking at SMP
> machines is ... it is my belief that getting a true threaded backend may
> be the only way to get the full potential out of SMP machines.  I see that
> is one of the things to experiment with on the TODO list and I have seen
> some people have messed around already with this using Solaris threads.
> It should probably be attempted with pthreads if PostgreSQL is going to
> keep some resemblance of cross-platform compatibility.  At that time, it
> would probably be easier to go in and clean up some stuff for the
> implementation of other TODO items (put in the base framework for more
> complex future items) as threading the backend would take a little bit of
> ideology shift.

I can only think of two objectives for threading. (1) running the various
connections in their own thread instead of their own process. (2) running
complex queries across multiple threads.

For  item (1) I see no value to this. It is a lot of work with no tangible
benefit. If you have an old fashion pthreads implementation, it will hurt
performance because are scheduled within the single process's time slice.. If
you have a newer kernel scheduled implementation, then you will have the same
scheduling as separate processes. The only thing you will need to do is
switch your brain from figuring out how to share data, to trying to figure
out how to isolate data. A multithreaded implementation lacks many of the
benefits and robustness of a multiprocess implementation.

For item (2) I can see how that could speed up queries in a low utilization
system, and that would be cool, but in a server that is under load, threading
the queries probably be less efficient.


Wed, 26 Sep 2001 16:43:02 -0400
mlw <markw@mohawksoft.com>
Re: Spinlock performance improvement proposal
Tom Lane wrote:
> 
> Neil Padgett <npadgett@redhat.com> writes:
> > Initial results (top five -- if you would like a complete profile, let
> > me know):
> > Each sample counts as 1 samples.
> >   %   cumulative   self              self     total
> >  time   samples   samples    calls  T1/call  T1/call  name
> >  26.57  42255.02 42255.02                             FindLockCycleRecurse
> 
> Yipes.  It would be interesting to know more about the locking pattern
> of your benchmark --- are there long waits-for chains, or not?  The
> present deadlock detector was certainly written with an eye to "get it
> right" rather than "make it fast", but I wonder whether this shows a
> performance problem in the detector, or just too many executions because
> you're waiting too long to get locks.
> 
> > However, this seems to be a red herring. Removing the deadlock detector
> > had no effect. In fact, benchmarking showed removing it yielded no
> > improvement in transaction processing rate on uniprocessor or SMP
> > systems. Instead, it seems that the deadlock detector simply amounts to
> > "something to do" for the blocked backend while it waits for lock
> > acquisition.
> 
> Do you have any idea about the typical lock-acquisition delay in this
> benchmark?  Our docs advise trying to set DEADLOCK_TIMEOUT higher than
> the typical acquisition delay, so that the deadlock detector does not
> run unnecessarily.

Well. Currently the runs are the typical pg_bench runs. This was useful
since it was a handy benchmark that was already done, and I was hoping
it might be useful for comparison since it seems to be popular. More
benchmarks of different types would of course be useful though. 

I think the large time consumed by the deadlock detector in the profile
is simply due to too many executions while waiting to acquire to
contended locks. But, I agree that it seems DEADLOCK_TIMEOUT was set too
low, since it appears from the profile output that the deadlock detector
was running unnecessarily. But the deadlock detector isn't causing the
SMP performance hit right now, since the throughput is the same with it
in place or with it removed completely. I therefore didn't make any
attempt to tune DEADLOCK_TIMEOUT. As I mentioned before, it apparently
just gives the backend "something" to do while it waits for a lock. 

I'm thinking that the deadlock detector unnecessarily has no effect on
performance since the shared memory is causing some level of
serialization. So, one CPU (or two, or three, but not all) is doing
useful work, while the others are idle (that is to say, doing no useful
work). If they are idle spinning, or idle running the deadlock detector
the net throughput is still the same. (This might also indicate that
improving the lock design won't help here.) Of course, another
possibility is that you spend so long spinning simply because you do
spin (rather than sleep), and this is wasting much CPU time so the
useful work backends take longer to get things done. Either is just
speculation right now without any data to back things up.

> 
> > For example, there has been some suggestion
> > that perhaps some component of the database is causing large lock
> > contention.
> 
> My thought as well.  I would certainly recommend that you use more than
> one test case while looking at these things.

Yes. That is another suggestion for a next step. Several cases might
serve to better expose the path causing the slowdown. I think that
several test cases of varying usage patterns, coupled with hold time
instrumentation (which can tell what routine acquired the lock and how
long it held it, and yield wait-for data in the analysis), are the right
way to go about attacking SMP performance. Any other thoughts?

Neil

-- 
Neil Padgett
Red Hat Canada Ltd.                       E-Mail:  npadgett@redhat.com
2323 Yonge Street, Suite #300, 
Toronto, ON  M4P 2C9

Wed, 26 Sep 2001 16:53:08 -0400
Neil Padgett <npadgett@redhat.com>
Re: Spinlock performance improvement proposal
On Wed, 26 Sep 2001, mlw wrote:
> 
> I can only think of two objectives for threading. (1) running the various
> connections in their own thread instead of their own process. (2) running
> complex queries across multiple threads.
> 
> For  item (1) I see no value to this. It is a lot of work with no tangible
> benefit. If you have an old fashion pthreads implementation, it will hurt
> performance because are scheduled within the single process's time slice..

Old fashion ... as in a userland library that implements POSIX threads?  
Well, I would agree.  However, most *modern* implementations are done in 
the kernel or kernel and userland coop model and don't have this 
limitation (as you mention later in your e-mail).  You have kinda hit on 
one of my gripes about computers in general.  At what point in time does 
one say something is obsolete or too old to support anymore - that it 
hinders progress instead of adding a "feature"?

> you have a newer kernel scheduled implementation, then you will have the same
> scheduling as separate processes. The only thing you will need to do is
> switch your brain from figuring out how to share data, to trying to figure
> out how to isolate data. A multithreaded implementation lacks many of the
> benefits and robustness of a multiprocess implementation.

Save for the fact that the kernel can switch between threads faster then 
it can switch processes considering threads share the same address space, 
stack, code, etc.  If need be sharing the data between threads is much 
easier then sharing between processes. 

I can't comment on the "isolate data" line.  I am still trying to figure 
that one out.

That last line is a troll if I every saw it ;-)  I will agree that threads 
isn't for everything and that it has costs just like everything else.  Let 
me stress that last part - like everything else.  Certain costs exist in 
the present model, nothing is - how should we say ... perfect.

> For item (2) I can see how that could speed up queries in a low utilization
> system, and that would be cool, but in a server that is under load, threading
> the queries probably be less efficient.

Well, I don't follow your logic and you didn't give any substance to back 
up your claim.  I am willing to listen.

Another thought ... Oracle uses threads doesn't it or at least it has a 
single processor and multi-processor version last time I knew ... which do 
they claim is better?  (Not saying that Oracle's proclimation of what is 
good and what is not matters, but it is good for another view point).

-- 
//========================================================\\
||  D. Hageman                    <dhageman@dracken.com>  ||
\\========================================================//


Wed, 26 Sep 2001 16:14:22 -0500 (CDT)
"D. Hageman" <dhageman@dracken.com>
Re: Spinlock performance improvement proposal
Neil Padgett <npadgett@redhat.com> writes:
> Well. Currently the runs are the typical pg_bench runs.

With what parameters?  If you don't initialize the pg_bench database
with "scale" proportional to the number of clients you intend to use,
then you'll naturally get huge lock contention.  For example, if you
use scale=1, there's only one "branch" in the database.  Since every
transaction wants to update the branch's balance, every transaction
has to write-lock that single row, and so everybody serializes on that
one lock.  Under these conditions it's not surprising to see lots of
lock waits and lots of useless runs of the deadlock detector ...

			regards, tom lane

Wed, 26 Sep 2001 17:17:12 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Spinlock performance improvement proposal 


On Wed, 26 Sep 2001, mlw wrote:

> I can only think of two objectives for threading. (1) running the various
> connections in their own thread instead of their own process. (2) running
> complex queries across multiple threads.
> 

I did a multi-threaded version of 7.0.2 using Solaris threads about a year
ago in order to try
and get multiple backend connections working under one java process using
jni.  I used the thread per connection model.

I eventually got it working, but it was/is very messy ( there were global
variables everywhere! ).  Anyway, I was able to get a pretty good speed up
on inserts by scheduling buffer writes from multiple connections on one
common writing thread.  

I also got some other features that were important to me at the time.

1.  True prepared statements under java with bound input and output
variables
2.  Better system utilization 
	a.  fewer Solaris lightweight processes mapped to threads.
	b.  Fewer open files per postgres installation 
3.  Automatic vacuums when system activity is low by a daemon thread.

but there were some drawbacks...  One rogue thread or bad user 
function could take down all connections for that process.  This
was and seems to still be the major drawback to using threads.


Myron Scott
mscott@sacadia.com


Wed, 26 Sep 2001 15:03:00 -0700 (PDT)
Myron Scott <mscott@sacadia.com>
Re: Spinlock performance improvement proposal
"D. Hageman" <dhageman@dracken.com> writes:

> > you have a newer kernel scheduled implementation, then you will have the same
> > scheduling as separate processes. The only thing you will need to do is
> > switch your brain from figuring out how to share data, to trying to figure
> > out how to isolate data. A multithreaded implementation lacks many of the
> > benefits and robustness of a multiprocess implementation.
> 
> Save for the fact that the kernel can switch between threads faster then 
> it can switch processes considering threads share the same address space, 
> stack, code, etc.  If need be sharing the data between threads is much 
> easier then sharing between processes. 

When using a kernel threading model, it's not obvious to me that the
kernel will switch between threads much faster than it will switch
between processes.  As far as I can see, the only potential savings is
not reloading the pointers to the page tables.  That is not nothing,
but it is also not a lot.

> I can't comment on the "isolate data" line.  I am still trying to figure 
> that one out.

Sometimes you need data which is specific to a particular thread.
Basically, you have to look at every global variable in the Postgres
backend, and determine whether to share it among all threads or to
make it thread-specific.  In other words, you have to take extra steps
to isolate the data within the thread.  This is the reverse of the
current situation, in which you have to take extra steps to share data
among all backend processes.

> That last line is a troll if I every saw it ;-)  I will agree that threads 
> isn't for everything and that it has costs just like everything else.  Let 
> me stress that last part - like everything else.  Certain costs exist in 
> the present model, nothing is - how should we say ... perfect.

When writing in C, threading inevitably loses robustness.  Erratic
behaviour by one thread, perhaps in a user defined function, can
subtly corrupt the entire system, rather than just that thread.  Part
of defensive programming is building barriers between different parts
of a system.  Process boundaries are a powerful barrier.

(Actually, though, Postgres is already vulnerable to erratic behaviour
because any backend process can corrupt the shared buffer pool.)

Ian

26 Sep 2001 15:04:41 -0700
Ian Lance Taylor <ian@airs.com>
Re: Spinlock performance improvement proposal
"D. Hageman" <dhageman@dracken.com> writes:

> Save for the fact that the kernel can switch between threads faster then 
> it can switch processes considering threads share the same address space, 
> stack, code, etc.  If need be sharing the data between threads is much 
> easier then sharing between processes. 

This depends on your system.  Solaris has a huge difference between
thread and process context switch times, whereas Linux has very little 
difference (and in fact a Linux process context switch is about as
fast as a Solaris thread switch on the same hardware--Solaris is just
a pig when it comes to process context switching). 

> I can't comment on the "isolate data" line.  I am still trying to figure 
> that one out.

I think his point is one of clarity and maintainability.  When a
task's data is explicitly shared (via shared memory of some sort) it's
fairly clear when you're accessing shared data and need to worry about
locking.  Whereas when all data is shared by default (as with threads)
it's very easy to miss places where threads can step on each other.

-Doug
-- 
In a world of steel-eyed death, and men who are fighting to be warm,
Come in, she said, I'll give you shelter from the storm.    -Dylan

26 Sep 2001 18:39:44 -0400
Doug McNaught <doug@wireboard.com>
Re: Spinlock performance improvement proposal
On 26 Sep 2001, Ian Lance Taylor wrote:
>
> > Save for the fact that the kernel can switch between threads faster then 
> > it can switch processes considering threads share the same address space, 
> > stack, code, etc.  If need be sharing the data between threads is much 
> > easier then sharing between processes. 
> 
> When using a kernel threading model, it's not obvious to me that the
> kernel will switch between threads much faster than it will switch
> between processes.  As far as I can see, the only potential savings is
> not reloading the pointers to the page tables.  That is not nothing,
> but it is also not a lot.

It is my understanding that avoiding a full context switch of the 
processor can be of a significant advantage.  This is especially important 
on processor architectures that can be kinda slow at doing it (x86). I 
will admit that most modern kernels have features that assist software 
packages utilizing the forking model (copy on write for instance).  It is 
also my impression that these do a good job.  I am the kind of guy that 
looks towards the future (as in a year, year and half or so) and say that 
processors will hopefully get faster at context switching and more and 
more kernels will implement these algorithms to speed up the forking 
model.  At the same time, I see more and more processors being shoved into 
a single box and it appears that the threads model works better on these 
type of systems.   

> > I can't comment on the "isolate data" line.  I am still trying to figure 
> > that one out.
> 
> Sometimes you need data which is specific to a particular thread.

When you need data that is specific to a thread you use a TSD (Thread 
Specific Data).  

> Basically, you have to look at every global variable in the Postgres
> backend, and determine whether to share it among all threads or to
> make it thread-specific.

Yes, if one was to implement threads into PostgreSQL I would think that 
some re-writing would be in order of several areas.  Like I said before, 
give a person a chance to restructure things so future TODO items wouldn't 
be so hard to implement.  Personally, I like to stay away from global 
variables as much as possible.  They just get you into trouble.

> > That last line is a troll if I every saw it ;-)  I will agree that threads 
> > isn't for everything and that it has costs just like everything else.  Let 
> > me stress that last part - like everything else.  Certain costs exist in 
> > the present model, nothing is - how should we say ... perfect.
> 
> When writing in C, threading inevitably loses robustness.  Erratic
> behaviour by one thread, perhaps in a user defined function, can
> subtly corrupt the entire system, rather than just that thread.  Part
> of defensive programming is building barriers between different parts
> of a system.  Process boundaries are a powerful barrier.

I agree with everything you wrote above except for the first line.  My 
only comment is that process boundaries are only *truely* a powerful 
barrier if the processes are different pieces of code and are not 
dependent on each other in crippling ways.  Forking the same code with the 
bug in it - and only 1 in 5 die - is still 4 copies of buggy code running 
on your system ;-)  

> (Actually, though, Postgres is already vulnerable to erratic behaviour
> because any backend process can corrupt the shared buffer pool.)

I appreciate your total honest view of the situation.  

-- 
//========================================================\\
||  D. Hageman                    <dhageman@dracken.com>  ||
\\========================================================//



Wed, 26 Sep 2001 18:18:08 -0500 (CDT)
"D. Hageman" <dhageman@dracken.com>
Re: Spinlock performance improvement proposal
On 26 Sep 2001, Doug McNaught wrote:

> This depends on your system.  Solaris has a huge difference between
> thread and process context switch times, whereas Linux has very little 
> difference (and in fact a Linux process context switch is about as
> fast as a Solaris thread switch on the same hardware--Solaris is just
> a pig when it comes to process context switching). 

Yeah, I kinda commented on this in another e-mail.  Linux has some nice 
tweaks for software using the forking model, but I am sure a couple of 
Solaris admins out there like to run PostgreSQL.  ;-)  You are right in 
that it is very system dependent.  I should have prefaced it with "In 
general ..."

> > I can't comment on the "isolate data" line.  I am still trying to figure 
> > that one out.
> 
> I think his point is one of clarity and maintainability.  When a
> task's data is explicitly shared (via shared memory of some sort) it's
> fairly clear when you're accessing shared data and need to worry about
> locking.  Whereas when all data is shared by default (as with threads)
> it's very easy to miss places where threads can step on each other.

Well, I understand what you are saying and you are correct.  The situation 
is that when you implement anything using pthreads you lock your 
variables (which is where the major performance penalty comes into play 
with threads).  Now, the kicker is how you lock them.  Depending on how 
you do it (as per discussion earlier on this list concerning threads) it 
can be faster or slower.  It all depends on what model you use.  

Data is not explicitely shared between threads unless you make it so.  The 
threads just share the same stack and all of that, but you can't 
(shouldn't is probably a better word) really access anything you don't have 
an address for.  Threads just makes it easier to share if you want to.  
Also, see my other e-mail to the list concerning TSDs.

-- 
//========================================================\\
||  D. Hageman                    <dhageman@dracken.com>  ||
\\========================================================//


Wed, 26 Sep 2001 18:32:32 -0500 (CDT)
"D. Hageman" <dhageman@dracken.com>
Re: Spinlock performance improvement proposal
Ian Lance Taylor <ian@airs.com> writes:
> (Actually, though, Postgres is already vulnerable to erratic behaviour
> because any backend process can corrupt the shared buffer pool.)

Not to mention the other parts of shared memory.

Nonetheless, our experience has been that cross-backend failures due to
memory clobbers in shared memory are very infrequent --- certainly far
less often than we see localized-to-a-backend crashes.  Probably this is
because the shared memory is (a) small compared to the rest of the
address space and (b) only accessed by certain specific modules within
Postgres.

I'm convinced that switching to a thread model would result in a
significant degradation in our ability to recover from coredump-type
failures, even given the (implausible) assumption that we introduce no
new bugs during the conversion.  I'm also *un*convinced that such a
conversion will yield significant performance benefits, unless we
introduce additional cross-thread dependencies (and more fragility
and lock contention) by tactics such as sharing catalog caches across
threads.

			regards, tom lane

Wed, 26 Sep 2001 20:46:51 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Spinlock performance improvement proposal 
> ... Thomas still has his date/time stuff
> to finish off, now that CVSup is fixed ...

I'm now getting clean runs through the regression tests on a freshly
merged cvs tree. I'd like to look at it a little more to adjust
pg_proc.h attributes before I commit the changes.

There was a bit of a hiccup when merging since there was some bytea
stuff added to the catalogs over the last couple of weeks. Could folks
hold off on claiming new OIDs until I get this stuff committed? TIA

I expect to be able to merge this stuff by Friday at the latest, more
likely tomorrow.

                         - Thomas

Thu, 27 Sep 2001 02:45:06 +0000
Thomas Lockhart <lockhart@fourpalms.org>
Re: Spinlock performance improvement proposal
On Wed, 26 Sep 2001, D. Hageman wrote:

> > > Save for the fact that the kernel can switch between threads faster then 
> > > it can switch processes considering threads share the same address space, 
> > > stack, code, etc.  If need be sharing the data between threads is much 
> > > easier then sharing between processes. 
> > 
> > When using a kernel threading model, it's not obvious to me that the
> > kernel will switch between threads much faster than it will switch
> > between processes.  As far as I can see, the only potential savings is
> > not reloading the pointers to the page tables.  That is not nothing,
> > but it is also
<major snippage>
> > > I can't comment on the "isolate data" line.  I am still trying to figure 
> > > that one out.
> > 
> > Sometimes you need data which is specific to a particular thread.
> 
> When you need data that is specific to a thread you use a TSD (Thread 
> Specific Data).  
Which Linux does not support with a vengeance, to my knowledge.

As a matter of fact, quote from Linus on the matter was something like
"Solution to slow process switching is fast process switching, not another
kernel abstraction [referring to threads and TSD]". TSDs make
implementation of thread switching complex, and fork() complex.

The question about threads boils down to: Is there far more data that is
shared than unshared? If yes, threads are better, if not, you'll be
abusing TSD and slowing things down. 

I believe right now, postgresql' model of sharing only things that need to
be shared is pretty damn good. The only slight problem is overhead of
forking another backend, but its still _fast_.

IMHO, threads would not bring large improvement to postgresql.

 Actually, if I remember, there was someone who ported postgresql (I think
it was 6.5) to be multithreaded with major pain, because the requirement
was to integrate with CORBA. I believe that person posted some benchmarks
which were essentially identical to non-threaded postgres...

-alex


Wed, 26 Sep 2001 22:58:41 -0400 (EDT)
Alex Pilosov <alex@pilosoft.com>
Re: Spinlock performance improvement proposal
On Wed, 26 Sep 2001, Alex Pilosov wrote:

> On Wed, 26 Sep 2001, D. Hageman wrote:
> 
> > When you need data that is specific to a thread you use a TSD (Thread 
> > Specific Data).  

> Which Linux does not support with a vengeance, to my knowledge.

I am not sure what that means.  If it works it works. 

> As a matter of fact, quote from Linus on the matter was something like
> "Solution to slow process switching is fast process switching, not another
> kernel abstraction [referring to threads and TSD]". TSDs make
> implementation of thread switching complex, and fork() complex.

Linus does have some interesting ideas.  I always like to hear his 
perspective on matters, but just like the government - I don't always 
agree with him.  I don't see why TSDs would make the implementation of 
thread switching complex - seems to me that would be something that is 
implemented in the userland side part of the pthreads implemenation and 
not the kernel side.  I don't really like to talk specifics, but both the 
lightweight process and the system call fork() are implemented using the 
__clone kernel function with the parameters slightly different (This is 
in the Linux kernel, btw since you wanted to use that as an example).  The 
speed improvements the kernel has given the fork() command (like copy on 
write) only lasts until the process writes to memmory.  The next time it 
comes around - it is for all intents and purposes a full context switch 
again.  With threads ... the cost is relatively consistant.

> The question about threads boils down to: Is there far more data that is
> shared than unshared? If yes, threads are better, if not, you'll be
> abusing TSD and slowing things down. 

I think the question about threads boils down to if the core members of 
the PostgreSQL team want to try it or not.  At this time, I would have to 
say they pretty much agree they like things the way they are now, which is 
completely fine.  They are the ones that spend most of the time on it and 
want to support it.

> I believe right now, postgresql' model of sharing only things that need to
> be shared is pretty damn good. The only slight problem is overhead of
> forking another backend, but its still _fast_.

Oh, man ... am I reading stuff into what you are writing or are you 
reading stuff into what I am writing?  Maybe a little bit of both?  My 
original contention is that I think that the best way to get the full 
potential out of SMP machines is to use a threads model.  I didn't say the 
present way wasn't fast.  

>  Actually, if I remember, there was someone who ported postgresql (I think
> it was 6.5) to be multithreaded with major pain, because the requirement
> was to integrate with CORBA. I believe that person posted some benchmarks
> which were essentially identical to non-threaded postgres...

Actually, it was 7.0.2 and the performance gain was interesting.  The 
posting can be found at:

http://candle.pha.pa.us/mhonarc/todo.detail/thread/msg00007.html

The results are:

20 clients, 900 inserts per client, 1 insert per transaction, 4 different
tables.

7.0.2    About    10:52 average completion
multi-threaded    2:42 average completion
7.1beta3          1:13 average completion

If the multi-threaded version was 7.0.2 and threads increased performance 
that much - I would have to say that was a bonus.  However, the 
performance increases that the PostgreSQL team implemented later ... 
pushed the regular version ahead again.  That kinda says to me that 
potential is there.

If you look at Myron Scott's post today you will see that it had other 
advantages going for it (like auto-vacuum!) and disadvantages ... rogue 
thread corruption (already debated today).

-- 
//========================================================\\
||  D. Hageman                    <dhageman@dracken.com>  ||
\\========================================================//




Wed, 26 Sep 2001 22:41:39 -0500 (CDT)
"D. Hageman" <dhageman@dracken.com>
Re: Spinlock performance improvement proposal
On Wed, 26 Sep 2001, D. Hageman wrote:

> Oh, man ... am I reading stuff into what you are writing or are you 
> reading stuff into what I am writing?  Maybe a little bit of both?  My 
> original contention is that I think that the best way to get the full 
> potential out of SMP machines is to use a threads model.  I didn't say the 
> present way wasn't fast.  
Or alternatively, that the current inter-process locking is a bit
inefficient. Its possible to have inter-process locks that are as fast as
inter-thread locks.

> >  Actually, if I remember, there was someone who ported postgresql (I think
> > it was 6.5) to be multithreaded with major pain, because the requirement
> > was to integrate with CORBA. I believe that person posted some benchmarks
> > which were essentially identical to non-threaded postgres...
> 
> Actually, it was 7.0.2 and the performance gain was interesting.  The 
> posting can be found at:
> 
> 7.0.2    About    10:52 average completion
> multi-threaded    2:42 average completion
> 7.1beta3          1:13 average completion
> 
> If the multi-threaded version was 7.0.2 and threads increased performance 
> that much - I would have to say that was a bonus.  However, the 
> performance increases that the PostgreSQL team implemented later ... 
> pushed the regular version ahead again.  That kinda says to me that 
> potential is there.
Alternatively, you could read that 7.1 took the wind out of threaded
sails. :) But I guess we won't know until the current version is ported to
threads...

-alex


Thu, 27 Sep 2001 00:08:51 -0400 (EDT)
Alex Pilosov <alex@pilosoft.com>
Re: Spinlock performance improvement proposal
"D. Hageman" <dhageman@dracken.com> writes:
> If you look at Myron Scott's post today you will see that it had other 
> advantages going for it (like auto-vacuum!) and disadvantages ... rogue 
> thread corruption (already debated today).

But note that Myron did a number of things that are (IMHO) orthogonal
to process-to-thread conversion, such as adding prepared statements,
a separate thread/process/whateveryoucallit for buffer writing, ditto
for vacuuming, etc.  I think his results cannot be taken as indicative
of the benefits of threads per se --- these other things could be
implemented in a pure process model too, and we have no data with which
to estimate which change bought how much.

Threading certainly should reduce the context switch time, but this
comes at the price of increased overhead within each context (since
access to thread-local variables is not free).  It's by no means
obvious that there's a net win there.

			regards, tom lane

Thu, 27 Sep 2001 00:19:20 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Spinlock performance improvement proposal 

 
> But note that Myron did a number of things that are (IMHO) orthogonal

yes, I did :)

> to process-to-thread conversion, such as adding prepared statements,
> a separate thread/process/whateveryoucallit for buffer writing, ditto
> for vacuuming, etc.  I think his results cannot be taken as indicative
> of the benefits of threads per se --- these other things could be
> implemented in a pure process model too, and we have no data with which
> to estimate which change bought how much.
> 

If you are comparing just process vs. thread, I really don't think I
gained much for performance and ended up with some pretty unmanageable
code.

The one thing that led to most of the gains was scheduling all the writes
to one thread which, as noted by Tom,  you could do on the process model.
Besides, Most of the advantage in doing this was taken away with the
addition of WAL in 7.1.

The other real gain that I saw with threading was limiting the number of
open files but
that led me to alter much of the file manager in order to synchronize
access to the files which probably slowed things a bit.

To be honest, I don't think I, personally,
would try this again. I went pretty far off
the beaten path with this thing.  It works well for what I am doing 
( a limited number of SQL statements run many times over ) but there
probably was a better way.  I'm thinking now that I should have tried to 
add a CORBA interface for connections. I would have been able to 
accomplish my original goals without creating a deadend for myself.


Thanks all for a great project,

Myron
mscott@sacadia.com


Wed, 26 Sep 2001 22:24:29 -0700 (PDT)
Myron Scott <mscott@sacadia.com>
Re: Spinlock performance improvement proposal 
"D. Hageman" wrote:

> On 26 Sep 2001, Ian Lance Taylor wrote:
> >
> > > Save for the fact that the kernel can switch between threads faster then
> > > it can switch processes considering threads share the same address space,
> > > stack, code, etc.  If need be sharing the data between threads is much
> > > easier then sharing between processes.
> >
> > When using a kernel threading model, it's not obvious to me that the
> > kernel will switch between threads much faster than it will switch
> > between processes.  As far as I can see, the only potential savings is
> > not reloading the pointers to the page tables.  That is not nothing,
> > but it is also not a lot.
>
> It is my understanding that avoiding a full context switch of the
> processor can be of a significant advantage.  This is especially important
> on processor architectures that can be kinda slow at doing it (x86). I
> will admit that most modern kernels have features that assist software
> packages utilizing the forking model (copy on write for instance).  It is
> also my impression that these do a good job.  I am the kind of guy that
> looks towards the future (as in a year, year and half or so) and say that
> processors will hopefully get faster at context switching and more and
> more kernels will implement these algorithms to speed up the forking
> model.  At the same time, I see more and more processors being shoved into
> a single box and it appears that the threads model works better on these
> type of systems.

"context" switching happens all the time on a multitasking system. On the x86
processor, a context switch happens when you call into the kernel. You have to go
through a call-gate to get to a lower privilege ring. "context" switching is very
fast. The operating system dictates how heavy or light a process switch is. Under
Linux (and I believe FreeBSD with Linux threads, or version 4.x ) threads and
processes are virtually identical. The only difference is that the virtual memory
pages are not "copy on write." Process vs thread scheduling is also virtually
identical.

If you look to the future, then you should accept that process switching should
become more efficient as the operating systems improve.

>
> > > I can't comment on the "isolate data" line.  I am still trying to figure
> > > that one out.
> >
> > Sometimes you need data which is specific to a particular thread.
>
> When you need data that is specific to a thread you use a TSD (Thread
> Specific Data).

Yes, but Postgres has many global variables. The assumption has always been that
it is a stand-alone process with an explicitly shared paradigm, not implicitly.

>
> > Basically, you have to look at every global variable in the Postgres
> > backend, and determine whether to share it among all threads or to
> > make it thread-specific.
>
> Yes, if one was to implement threads into PostgreSQL I would think that
> some re-writing would be in order of several areas.  Like I said before,
> give a person a chance to restructure things so future TODO items wouldn't
> be so hard to implement.  Personally, I like to stay away from global
> variables as much as possible.  They just get you into trouble.

In real live software, software which lives from year to year with active
development, things do get messy. There are always global variables involved in a
program. Efforts, of course, should be made to keep them to a minimum, but the
reality is that they always happen.

Also, the very structure of function calls may need to change when going from a
process model to a threaded model. Functions never before reentrant are now be
reentrant, think about that. That is a huge undertaking. Every single function
may need to be examined for thread safety, with little benefit.

>
> > > That last line is a troll if I every saw it ;-)  I will agree that threads
> > > isn't for everything and that it has costs just like everything else.  Let
> > > me stress that last part - like everything else.  Certain costs exist in
> > > the present model, nothing is - how should we say ... perfect.
> >
> > When writing in C, threading inevitably loses robustness.  Erratic
> > behaviour by one thread, perhaps in a user defined function, can
> > subtly corrupt the entire system, rather than just that thread.  Part
> > of defensive programming is building barriers between different parts
> > of a system.  Process boundaries are a powerful barrier.
>
> I agree with everything you wrote above except for the first line.  My
> only comment is that process boundaries are only *truely* a powerful
> barrier if the processes are different pieces of code and are not
> dependent on each other in crippling ways.  Forking the same code with the
> bug in it - and only 1 in 5 die - is still 4 copies of buggy code running
> on your system ;-)

This is simply not true. All software has bugs, it is an undeniable fact. Some
bugs are more likely to be hit than others. 5 processes , when one process hits a
bug, that does not mean the other 4 will hit the same bug. Obscure bugs kill
software all the time, the trick is to minimize the impact. Software is not
perfect, assuming it can be is a mistake.





Thu, 27 Sep 2001 10:02:05 -0400
mlw <markw@mohawksoft.com>
Re: Spinlock performance improvement proposal
Tom Lane wrote:
> 
> Neil Padgett <npadgett@redhat.com> writes:
> > Well. Currently the runs are the typical pg_bench runs.
> 
> With what parameters?  If you don't initialize the pg_bench database
> with "scale" proportional to the number of clients you intend to use,
> then you'll naturally get huge lock contention.  For example, if you
> use scale=1, there's only one "branch" in the database.  Since every
> transaction wants to update the branch's balance, every transaction
> has to write-lock that single row, and so everybody serializes on that
> one lock.  Under these conditions it's not surprising to see lots of
> lock waits and lots of useless runs of the deadlock detector ...

The results you saw with the large number of useless runs of the
deadlock detector had a scale factor of 2. With a scale factor 2, the
performance fall-off began at about 100 clients. So, I reran the 512
client profiling run with a scale factor of 12. (2:100 as 10:500 -- so
12 might be an appropriate scale factor with some cushion?) This does,
of course, reduce the contention. However, the throughput is still only
about twice as much, which sounds good, but is still a small fraction of
the throughput realized on the same machine with a small number of
clients. (This is the uniprocessor machine.)

The new profile looks like this (uniprocessor machine):
Flat profile:

Each sample counts as 1 samples.
  %   cumulative   self              self     total           
 time   samples   samples    calls  T1/call  T1/call  name    
  9.44  10753.00 10753.00                             pg_fsync (I'd
attribute this to the slow disk in the machine -- scale 12 yields a lot
of tuples.)
  6.63  18303.01  7550.00                             s_lock_sleep
  6.56  25773.01  7470.00                             s_lock
  5.88  32473.01  6700.00                             heapgettup
  5.28  38487.02  6014.00                            
HeapTupleSatisfiesSnapshot
  4.83  43995.02  5508.00                             hash_destroy
  2.77  47156.02  3161.00                             load_file
  1.90  49322.02  2166.00                             XLogInsert
  1.86  51436.02  2114.00                             _bt_compare
  1.82  53514.02  2078.00                             AllocSetAlloc
  1.72  55473.02  1959.00                             LockBuffer
  1.50  57180.02  1707.00                             init_ps_display
  1.40  58775.03  1595.00                            
DirectFunctionCall9
  1.26  60211.03  1436.00                             hash_search
  1.14  61511.03  1300.00                             GetSnapshotData
  1.11  62780.03  1269.00                             SpinAcquire
  1.10  64028.03  1248.00                             LockAcquire
  1.04  70148.03  1190.00                             heap_fetch
  0.91  71182.03  1034.00                             _bt_orderkeys
  0.89  72201.03  1019.00                             LockRelease
  0.75  73058.03   857.00                            
InitBufferPoolAccess
.
.
.

I reran the benchmarks on the SMP machine with a scale of 12 instead of
2. The numbers still show a clear performance drop off at approximately
100 clients, albeit not as sharp. (But still quite pronounced.) In terms
of raw performance, the numbers are comparable. The scale factor
certainly helped -- but it still seems that we might have a problem
here.

Thoughts?

Neil

-- 
Neil Padgett
Red Hat Canada Ltd.                       E-Mail:  npadgett@redhat.com
2323 Yonge Street, Suite #300, 
Toronto, ON  M4P 2C9

Thu, 27 Sep 2001 14:42:42 -0400
Neil Padgett <npadgett@redhat.com>
Re: Spinlock performance improvement proposal
* Doug McNaught <doug@wireboard.com> wrote:
|
| Depends on what you mean.  For scaling well with many connections and
| simultaneous queries, there's no reason IMHO that the current
| process-per-backend model won't do, assuming the locking issues are
| addressed. 

Wouldn't a threading model allow you to share more data across different
connections ? I'm thinking in terms of introducing more cache functionality
to improve performance. What is shared memory used for today ?

-- 
Gunnar Rï¿½nning - gunnar@polygnosis.com
Senior Consultant, Polygnosis AS, http://www.polygnosis.com/

28 Sep 2001 05:03:00 +0200
Gunnar =?iso-8859-1?q?R=F8nning?= <gunnar@polygnosis.com>
Re: Spinlock performance improvement proposal
> 
> Sounds cool to me ... definitely something to fix before v7.2, if its as
> "easy" as you make it sound ... I'm expecting the new drive to be
> installed today (if all goes well ... Thomas still has his date/time stuff
> to finish off, now that CVSup is fixed ...
> 
> Let''s try and target Monday for Beta then?  I think the only two
> outstaandings are you and Thomas right now?
> 
> Bruce, that latest rtree patch looks intriguing also ... can anyone
> comment positive/negative about it, so that we can try and get that in
> before Beta?

I put it in the queue and will apply in a day or two.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 28 Sep 2001 00:13:48 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Spinlock performance improvement proposal
At 10:02 AM 9/27/01 -0400, mlw wrote:
>"D. Hageman" wrote:
>> I agree with everything you wrote above except for the first line.  My
>> only comment is that process boundaries are only *truely* a powerful
>> barrier if the processes are different pieces of code and are not
>> dependent on each other in crippling ways.  Forking the same code with the
>> bug in it - and only 1 in 5 die - is still 4 copies of buggy code running
>> on your system ;-)
>
>This is simply not true. All software has bugs, it is an undeniable fact.
Some
>bugs are more likely to be hit than others. 5 processes , when one process
hits a
>bug, that does not mean the other 4 will hit the same bug. Obscure bugs kill
>software all the time, the trick is to minimize the impact. Software is not
>perfect, assuming it can be is a mistake.

A bit off topic, but that really reminded me of how Microsoft does their
forking in hardware.

Basically they "fork" (cluster) FIVE windows machines to run the same buggy
code all on the same IP. That way if one process (machine) goes down, the
other 4 stay running, thus minimizing the impact ;).

They have many of these clusters put together.

See: http://www.microsoft.com/backstage/column_T2_1.htm
>From Microsoft.com Backstage [1]

OK so it's old (1998), but from their recent articles I believe they're
still using the same method of achieving "100% availability". And they brag
about it like it's a good thing...

When I first read it I didn't know whether to laugh or get disgusted or
whatever.

Cheerio,
Link.

[1]
http://www.microsoft.com/backstage/
http://www.microsoft.com/backstage/archives.htm



Fri, 28 Sep 2001 23:32:32 +0800
Lincoln Yeoh <lyeoh@pop.jaring.my>
Re: Spinlock performance improvement proposal

Good summary.  I agree checkpoint should look like as normal a Proc as
possible.


> At the just-past OSDN database conference, Bruce and I were annoyed by
> some benchmark results showing that Postgres performed poorly on an
> 8-way SMP machine.  Based on past discussion, it seems likely that the
> culprit is the known inefficiency in our spinlock implementation.
> After chewing on it for awhile, we came up with an idea for a solution.
> 
> The following proposal should improve performance substantially when
> there is contention for a lock, but it creates no portability risks
> because it uses the same system facilities (TAS and SysV semaphores)
> that we have always relied on.  Also, I think it'd be fairly easy to
> implement --- I could probably get it done in a day.
> 
> Comments anyone?
> 
> 			regards, tom lane
> 
> 
> Plan:
> 
> Replace most uses of spinlocks with "lightweight locks" (LW locks)
> implemented by a new lock manager.  The principal remaining use of true
> spinlocks (TAS locks) will be to provide mutual exclusion of access to
> LW lock structures.  Therefore, we can assume that spinlocks are never
> held for more than a few dozen instructions --- and never across a kernel
> call.
> 
> It's pretty easy to rejigger the spinlock code to work well when the lock
> is never held for long.  We just need to change the spinlock retry code
> so that it does a tight spin (continuous retry) for a few dozen cycles ---
> ideally, the total delay should be some small multiple of the max expected
> lock hold time.  If lock still not acquired, yield the CPU via a select()
> call (10 msec minimum delay) and repeat.  Although this looks inefficient,
> it doesn't matter on a uniprocessor because we expect that backends will
> only rarely be interrupted while holding the lock, so in practice a held
> lock will seldom be encountered.  On SMP machines the tight spin will win
> since the lock will normally become available before we give up and yield
> the CPU.
> 
> Desired properties of the LW lock manager include:
> 	* very fast fall-through when no contention for lock
> 	* waiting proc does not spin
> 	* support both exclusive and shared (read-only) lock modes
> 	* grant lock to waiters in arrival order (no starvation)
> 	* small lock structure to allow many LW locks to exist.
> 
> Proposed contents of LW lock structure:
> 
> 	spinlock mutex (protects LW lock state and PROC queue links)
> 	count of exclusive holders (always 0 or 1)
> 	count of shared holders (0 .. MaxBackends)
> 	queue head pointer (NULL or ptr to PROC object)
> 	queue tail pointer (could do without this to save space)
> 
> If a backend sees it must wait to acquire the lock, it adds its PROC
> struct to the end of the queue, releases the spinlock mutex, and then
> sleeps by P'ing its per-backend wait semaphore.  A backend releasing the
> lock will check to see if any waiter should be granted the lock.  If so,
> it will update the lock state, release the spinlock mutex, and finally V
> the wait semaphores of any backends that it decided should be released
> (which it removed from the lock's queue while holding the sema).  Notice
> that no kernel calls need be done while holding the spinlock.  Since the
> wait semaphore will remember a V occurring before P, there's no problem
> if the releaser is fast enough to release the waiter before the waiter
> reaches its P operation.
> 
> We will need to add a few fields to PROC structures:
> 	* Flag to show whether PROC is waiting for an LW lock, and if so
> 	  whether it waits for read or write access
> 	* Additional PROC queue link field.
> We can't reuse the existing queue link field because it is possible for a
> PROC to be waiting for both a heavyweight lock and a lightweight one ---
> this will occur when HandleDeadLock or LockWaitCancel tries to acquire
> the LockMgr module's lightweight lock (formerly spinlock).
> 
> It might seem that we also need to create a second wait semaphore per
> backend, one to wait on HW locks and one to wait on LW locks.  But I
> believe we can get away with just one, by recognizing that a wait for an
> LW lock can never be interrupted by a wait for a HW lock, only vice versa.
> After being awoken (V'd), the LW lock manager must check to see if it was
> actually granted the lock (easiest way: look at own PROC struct to see if
> LW lock wait flag has been cleared).  If not, the V must have been to
> grant us a HW lock --- but we still have to sleep to get the LW lock.  So
> remember this happened, then loop back and P again.  When we finally get
> the LW lock, if there was an extra P operation then V the semaphore once
> before returning.  This will allow ProcSleep to exit the wait for the HW
> lock when we return to it.
> 
> Fine points:
> 
> While waiting for an LW lock, we need to show in our PROC struct whether
> we are waiting for read or write access.  But we don't need to remember
> this after getting the lock; if we know we have the lock, it's easy to
> see by inspecting the lock whether we hold read or write access.
> 
> ProcStructLock cannot be replaced by an LW lock, since a backend cannot
> use an LW lock until it has obtained a PROC struct and a semaphore,
> both of which are protected by this lock.  It seems okay to use a plain
> spinlock for this purpose.  NOTE: it's okay for SInvalLock to be an LW
> lock, as long as the LW mgr does not depend on accessing the SI array
> of PROC objects, but only chains through the PROCs themselves.
> 
> Another tricky point is that some of the setup code executed by the
> postmaster may try to to grab/release LW locks.  Here, we can probably
> allow a special case for MyProc=NULL.  It's likely that we should never
> see a block under these circumstances anyway, so finding MyProc=NULL when
> we need to block may just be a fatal error condition.
> 
> A nastier case is checkpoint processes; these expect to grab BufMgr and
> WAL locks.  Perhaps okay for them to do plain sleeps in between attempts
> to grab the locks?  This says that the MyProc=NULL case should release
> the spinlock mutex, sleep 10 msec, try again, rather than any sort of error
> or expectation of no conflict.  Are there any cases where this represents
> a horrid performance loss?  Checkpoint itself seems noncritical.
> 
> Alternative is for checkpoint to be allowed to create a PROC struct (but
> not to enter it in SI list) so's it can participate normally in LW lock
> operations.  That seems a good idea anyway, actually, so that the PROC
> struct's facility for releasing held LW locks at elog time will work
> inside the checkpointer.  (But that means we need an extra sema too?
> Okay, but don't want an extra would-be backend to obtain the extra sema
> and perhaps cause a checkpoint proc to fail.  So must allocate the PROC
> and sema for checkpoint process separately from those reserved for
> backends.)
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 28 Sep 2001 13:11:29 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Spinlock performance improvement proposal
> Save for the fact that the kernel can switch between threads faster then 
> it can switch processes considering threads share the same address space, 
> stack, code, etc.  If need be sharing the data between threads is much 
> easier then sharing between processes. 

Just a clarification but because we fork each backend, don't they share
the same code space?  Data/stack is still separate.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 28 Sep 2001 14:52:59 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Spinlock performance improvement proposal

FYI, I have added a number of these emails to the 'thread' TODO.detail list.

> On Wed, 26 Sep 2001, D. Hageman wrote:
> 
> > > > Save for the fact that the kernel can switch between threads faster then 
> > > > it can switch processes considering threads share the same address space, 
> > > > stack, code, etc.  If need be sharing the data between threads is much 
> > > > easier then sharing between processes. 
> > > 
> > > When using a kernel threading model, it's not obvious to me that the
> > > kernel will switch between threads much faster than it will switch
> > > between processes.  As far as I can see, the only potential savings is
> > > not reloading the pointers to the page tables.  That is not nothing,
> > > but it is also
> <major snippage>
> > > > I can't comment on the "isolate data" line.  I am still trying to figure 
> > > > that one out.
> > > 
> > > Sometimes you need data which is specific to a particular thread.
> > 
> > When you need data that is specific to a thread you use a TSD (Thread 
> > Specific Data).  
> Which Linux does not support with a vengeance, to my knowledge.
> 
> As a matter of fact, quote from Linus on the matter was something like
> "Solution to slow process switching is fast process switching, not another
> kernel abstraction [referring to threads and TSD]". TSDs make
> implementation of thread switching complex, and fork() complex.
> 
> The question about threads boils down to: Is there far more data that is
> shared than unshared? If yes, threads are better, if not, you'll be
> abusing TSD and slowing things down. 
> 
> I believe right now, postgresql' model of sharing only things that need to
> be shared is pretty damn good. The only slight problem is overhead of
> forking another backend, but its still _fast_.
> 
> IMHO, threads would not bring large improvement to postgresql.
> 
>  Actually, if I remember, there was someone who ported postgresql (I think
> it was 6.5) to be multithreaded with major pain, because the requirement
> was to integrate with CORBA. I believe that person posted some benchmarks
> which were essentially identical to non-threaded postgres...
> 
> -alex
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
> http://archives.postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 28 Sep 2001 15:07:12 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Spinlock performance improvement proposal
> We have been doing some scalability testing just recently here at Red
> Hat. The machine I was using was a 4-way 550 MHz Xeon SMP machine, I
> also ran the machine in uniprocessor mode to make some comparisons. All
> runs were made on Red Hat Linux running 2.4.x series kernels. I've
> examined a number of potentially interesting cases -- I'm still
> analyzing the results, but some of the initial results might be
> interesting:

Let me add a little historical information here.  I think the first
report of bad performance on SMP machines was from Tatsuo, where he had
1000 backends running in pgbench.  He was seeing poor
transactions/second with little CPU or I/O usage.  It was clear
something was wrong.

Looking at the code, it was easy to see that on SMP machines, the
spinlock select() was a problem.  Later tests on various OS's found that
no matter how small your select interval was, select() couldn't sleep
for less than one cpu tick, which is tyically 100Hz or 10ms.  At that
point we knew that the spinlock backoff code was a serious problem.  On
multi-processor machines that could hit the backoff code on lock
failure, there where hudreds of threads sleeping for 10ms, then all
waking up, one gets the lock, and the others sleep again.

On single-cpu machines, the backoff code doesn't get hit too much, but
it is still a problem.  Tom's implementation changes backoffs in all
cases by placing them in a semaphore queue and reducing the amount of
code protected by the spinlock.

We have these TODO items out of this:

* Improve spinlock code [performance]
        o use SysV semaphores or queue of backends waiting on the lock
        o wakeup sleeper or sleep for less than one clock tick
        o spin for lock on multi-cpu machines, yield on single cpu machines
        o read/write locks




-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 28 Sep 2001 15:21:24 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Spinlock performance improvement proposal
Lincoln Yeoh wrote:
> 
> At 10:02 AM 9/27/01 -0400, mlw wrote:
> >"D. Hageman" wrote:
> >> I agree with everything you wrote above except for the first line.  My
> >> only comment is that process boundaries are only *truely* a powerful
> >> barrier if the processes are different pieces of code and are not
> >> dependent on each other in crippling ways.  Forking the same code with the
> >> bug in it - and only 1 in 5 die - is still 4 copies of buggy code running
> >> on your system ;-)
> >
> >This is simply not true. All software has bugs, it is an undeniable fact.
> Some
> >bugs are more likely to be hit than others. 5 processes , when one process
> hits a
> >bug, that does not mean the other 4 will hit the same bug. Obscure bugs kill
> >software all the time, the trick is to minimize the impact. Software is not
> >perfect, assuming it can be is a mistake.
> 
> A bit off topic, but that really reminded me of how Microsoft does their
> forking in hardware.
> 
> Basically they "fork" (cluster) FIVE windows machines to run the same buggy
> code all on the same IP. That way if one process (machine) goes down, the
> other 4 stay running, thus minimizing the impact ;).
> 
> They have many of these clusters put together.
> 
> See: http://www.microsoft.com/backstage/column_T2_1.htm
> >From Microsoft.com Backstage [1]
> 
> OK so it's old (1998), but from their recent articles I believe they're
> still using the same method of achieving "100% availability". And they brag
> about it like it's a good thing...
> 
> When I first read it I didn't know whether to laugh or get disgusted or
> whatever.

Believe me don't think anyone should be shipping software with serious bugs in
it, and I deplore Microsoft's complete lack of accountability when it comes to
quality, but come on now, lets not lie to ourselves. No matter which god you
may pray to, you have to accept that people are not perfect and mistakes will
be made.

At issue is how well programs are isolated from one another (one of the
purposes of operating systems) and how to deal with programmatic errors. I am
not advocating releasing bad software, I am just saying that you must code
defensively, assume a caller may pass the wrong parameters, don't trust that
malloc worked, etc. Stuff happens in the real world. Code to deal with it. 

In the end, no matter what you do, you will have a crash at some point. (The
tao of programming) accept it. Just try to make the damage as minimal as
possible.

Fri, 28 Sep 2001 23:13:59 -0400
mlw <markw@mohawksoft.com>
Re: Spinlock performance improvement proposal
Bruce Momjian wrote:
> 
> > Save for the fact that the kernel can switch between threads faster then
> > it can switch processes considering threads share the same address space,
> > stack, code, etc.  If need be sharing the data between threads is much
> > easier then sharing between processes.
> 
> Just a clarification but because we fork each backend, don't they share
> the same code space?  Data/stack is still separate.

In Linux and many modern UNIX programs, you share everything at fork time. The
data and stack pages are marked "copy on write" which means that if you touch
it, the processor traps and drops into the memory manager code. A new page is
created and replaced into your address space where the page, to which you were
going to write, was.

Fri, 28 Sep 2001 23:26:32 -0400
mlw <markw@mohawksoft.com>
Re: Spinlock performance improvement proposal
> Bruce Momjian wrote:
> > 
> > > Save for the fact that the kernel can switch between threads faster then
> > > it can switch processes considering threads share the same address space,
> > > stack, code, etc.  If need be sharing the data between threads is much
> > > easier then sharing between processes.
> > 
> > Just a clarification but because we fork each backend, don't they share
> > the same code space?  Data/stack is still separate.
> 
> In Linux and many modern UNIX programs, you share everything at fork time. The
> data and stack pages are marked "copy on write" which means that if you touch
> it, the processor traps and drops into the memory manager code. A new page is
> created and replaced into your address space where the page, to which you were
> going to write, was.

Yes, very true.  My point was that backends already share code space and
non-modified data space.  It is just modified data and stack that is
non-shared, but then again, they would have to be non-shared in a
threaded backend too.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 28 Sep 2001 23:41:18 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Spinlock performance improvement proposal
Bruce Momjian wrote:
> 
> > Bruce Momjian wrote:
> > >
> > > > Save for the fact that the kernel can switch between threads faster then
> > > > it can switch processes considering threads share the same address space,
> > > > stack, code, etc.  If need be sharing the data between threads is much
> > > > easier then sharing between processes.
> > >
> > > Just a clarification but because we fork each backend, don't they share
> > > the same code space?  Data/stack is still separate.
> >
> > In Linux and many modern UNIX programs, you share everything at fork time. The
> > data and stack pages are marked "copy on write" which means that if you touch
> > it, the processor traps and drops into the memory manager code. A new page is
> > created and replaced into your address space where the page, to which you were
> > going to write, was.
> 
> Yes, very true.  My point was that backends already share code space and
> non-modified data space.  It is just modified data and stack that is
> non-shared, but then again, they would have to be non-shared in a
> threaded backend too.

In a threaded system everything would be shared, depending on the OS, even the
stacks. The stacks could be allocated out of the same global pool.

You would need something like thread local storage to deal with isolating
aviables from one thread to another. That always seemed more trouble that it
was worth. Either that or go through each and every global variable in
PostgreSQL and make it a member of a structure, and create an instance of this
structure for each new thread.

IMHO once you go down the road of using Thread local memory, you are getting to
the same level of difficulty (for the OS) in task switching as just switching
processes. The exception to this is Windows where tasks are such a big hit.

I think threaded software is quite usefull, and I have a number of thread based
servers in production. However, my experience tells me that the work trying to
move PostgreSQL to a threaded ebvironment would be extensive and have little or
no tangable benefit.

I would rather see stuff like 64bit OIDs, three options for function definition
(short cache, nocache, long cache), etc. than to waste time making PostgreSQL
threaded. That's just my opinion.

Sat, 29 Sep 2001 00:28:43 -0400
mlw <markw@mohawksoft.com>
Re: Spinlock performance improvement proposal
I wrote:
> The following proposal should improve performance substantially when
> there is contention for a lock, but it creates no portability risks
> ...

I have committed changes to implement this proposal.  I'm not seeing
any significant performance difference on pgbench on my single-CPU
system ... but pgbench is I/O bound anyway on this hardware, so that's
not very surprising.  I'll be interested to see what other people
observe.  (Tatsuo, care to rerun that 1000-client test?)

			regards, tom lane

Sat, 29 Sep 2001 01:37:08 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Spinlock performance improvement proposal 
> I have committed changes to implement this proposal.  I'm not seeing
> any significant performance difference on pgbench on my single-CPU
> system ... but pgbench is I/O bound anyway on this hardware, so that's
> not very surprising.  I'll be interested to see what other people
> observe.  (Tatsuo, care to rerun that 1000-client test?)

What is your system? CPU, memory, IDE/SCSI, OS?
Scaling factor and # of clients?

BTW1 - shouldn't we rewrite pgbench to use threads instead of
"libpq async queries"? At least as option. I'd say that with 1000
clients current pgbench implementation is very poor.

BTW2 - shouldn't we learn if there are really portability/performance
issues in using POSIX mutex-es (and cond. variables) in place of
TAS (and SysV semaphores)?

Vadim



Sat, 29 Sep 2001 01:45:52 -0700
"Vadim Mikheev" <vmikheev@sectorbase.com>
Re: Spinlock performance improvement proposal 
On Thursday 27 September 2001 04:09, you wrote:
> This depends on your system.  Solaris has a huge difference between
> thread and process context switch times, whereas Linux has very little
> difference (and in fact a Linux process context switch is about as
> fast as a Solaris thread switch on the same hardware--Solaris is just
> a pig when it comes to process context switching).

I have never worked on any big systems but from what (little) I have seen, I 
think there should be a hybrid model.

This whole discussion started off, from poor performance on SMP machines. If 
I am getting this correctly, threads can be spread on multiple CPUs if 
available but process can not.

So I would suggest to have threaded approach for intensive tasks such as 
sorting/searching etc. IMHO converting entire paradigm to thread based is a 
huge task and may not be required in all cases. 

I think of an approach.  Threads are created when they are needed but they 
are kept dormant when not needed. So that there is no recreation overhead(if 
that's a concern). So at any given point of time, one back end connection has 
as many threads as number of CPUs. More than that may not yield much of 
performance improvement. Say a big task like sorting is split and given to 
different threads so that it can use them all.

It should be easy to switch the threading function and arguments on the fly, 
restricting number of threads and there will not be much of thread switching 
as each thread handles different parts of task and later the results are 
merged.

Number of threads should be equal to or twice that of number of CPUs. I don't 
think more than those many threads would yield any performance improvement.

And with this approach we can migrate one functionality at a time to threaded 
one, thus avoiding big effort at any given time.

Just a suggestion.

 Shridhar

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com


Sat, 29 Sep 2001 18:48:56 +0530
Chamanya <chamanya@yahoo.com>
Re: Spinlock performance improvement proposal
"Vadim Mikheev" <vmikheev@sectorbase.com> writes:
>> I have committed changes to implement this proposal.  I'm not seeing
>> any significant performance difference on pgbench on my single-CPU
>> system ... but pgbench is I/O bound anyway on this hardware, so that's
>> not very surprising.  I'll be interested to see what other people
>> observe.  (Tatsuo, care to rerun that 1000-client test?)

> What is your system? CPU, memory, IDE/SCSI, OS?
> Scaling factor and # of clients?

HP C180, SCSI-2 disks, HPUX 10.20.  I used scale factor 10 and between
1 and 10 clients.  Now that I think about it, I was running with the
default NBuffers (64), which probably constrained performance too.

> BTW1 - shouldn't we rewrite pgbench to use threads instead of
> "libpq async queries"? At least as option. I'd say that with 1000
> clients current pgbench implementation is very poor.

Well, it uses select() to wait for activity, so as long as all query
responses arrive as single packets I don't see the problem.  Certainly
rewriting pgbench without making libpq thread-friendly won't help a bit.

> BTW2 - shouldn't we learn if there are really portability/performance
> issues in using POSIX mutex-es (and cond. variables) in place of
> TAS (and SysV semaphores)?

Sure, that'd be worth looking into on a long-term basis.

			regards, tom lane

Sat, 29 Sep 2001 10:25:18 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Spinlock performance improvement proposal 
Chamanya wrote:
> 
> On Thursday 27 September 2001 04:09, you wrote:
> > This depends on your system.  Solaris has a huge difference between
> > thread and process context switch times, whereas Linux has very little
> > difference (and in fact a Linux process context switch is about as
> > fast as a Solaris thread switch on the same hardware--Solaris is just
> > a pig when it comes to process context switching).
> 
> I have never worked on any big systems but from what (little) I have seen, I
> think there should be a hybrid model.
> 
> This whole discussion started off, from poor performance on SMP machines. If
> I am getting this correctly, threads can be spread on multiple CPUs if
> available but process can not.

Different processes will be on handled evenly across all CPUs in an SMP
machine, unless you set process affinity for a process and a CPU.
> 
> So I would suggest to have threaded approach for intensive tasks such as
> sorting/searching etc. IMHO converting entire paradigm to thread based is a
> huge task and may not be required in all cases.

Dividing a query into multiple threads is an amazing task. I wish I had a
couple years and someone willing to pay me to try it.

> 
> I think of an approach.  Threads are created when they are needed but they
> are kept dormant when not needed. So that there is no recreation overhead(if
> that's a concern). So at any given point of time, one back end connection has
> as many threads as number of CPUs. More than that may not yield much of
> performance improvement. Say a big task like sorting is split and given to
> different threads so that it can use them all.

This is a huge undertaking, and quite frankly, if I understand PostgreSQL, a
complete redesign of the entire system.
> 
> It should be easy to switch the threading function and arguments on the fly,
> restricting number of threads and there will not be much of thread switching
> as each thread handles different parts of task and later the results are
> merged.

That is not what I would consider easy.

> 
> Number of threads should be equal to or twice that of number of CPUs. I don't
> think more than those many threads would yield any performance improvement.

That isn't true at all.

One of the problems I see when when people discuss performance on an SMP
machine, is that they usually think from the perspective of a single task. If
you are doing data mining, one sql query may take a very long time. Which may
be a problem, but in the grander scheme of things there are usually multiple
concurrent performance issues to be considered. Threading the back end for
parallel query processing will probably not help this. More often than not a
database has much more to do than one thing at a time.

Also, if you are threading query processing, you have to analyze what your
query needs to do with the threads.  If your query is CPU bound, then you will
want to use fewer threads, if your query is I/O bound, you should have as many
threads as you have I/O requests, and have each thread block on the I/O.

> 
> And with this approach we can migrate one functionality at a time to threaded
> one, thus avoiding big effort at any given time.

Perhaps I am being over dramatic, but I have moved a number of systems from
fork() to threaded (for ports to Windows NT from UNIX), and if my opinion means
anything on this mailing list, I STRONGLY urge against it. PostgreSQL is a huge
system, over a decade old. The original developers are no longer working on it,
and in fact, probably wouldn't recognize it. There are nooks and crannys that
no one knows about.

It has also been my experience going from separate processes to separate
threads does not do much for performance, simply because the operation of your
system does not change, only the methods by which you share memory. If you want
to multithread a single query, that's a different story and a good R&D project
in itself.

Sat, 29 Sep 2001 11:00:06 -0400
mlw <markw@mohawksoft.com>
Re: Spinlock performance improvement proposal
> I wrote:
> > The following proposal should improve performance substantially when
> > there is contention for a lock, but it creates no portability risks
> > ...
> 
> I have committed changes to implement this proposal.  I'm not seeing
> any significant performance difference on pgbench on my single-CPU
> system ... but pgbench is I/O bound anyway on this hardware, so that's
> not very surprising.  I'll be interested to see what other people
> observe.  (Tatsuo, care to rerun that 1000-client test?)

I ran with 20 clients:

	$ pgbench -i test
	$ pgbench -c 20 -t 100 test

and see no difference in tps performance between the two lock
implementations.  I have a Dual PIII 550MHz i386 BSD/OS machine with
SCSI disks.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 29 Sep 2001 14:37:49 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Spinlock performance improvement proposal
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I ran with 20 clients:

What scale factor?  How many buffers?

			regards, tom lane

Sat, 29 Sep 2001 14:39:34 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Spinlock performance improvement proposal 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I ran with 20 clients:
> 
> What scale factor?  How many buffers?

No scale factor, as I illustrated from the initialization command I
used.  Standard buffers too.  Let me know what values I should use for
testing.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 29 Sep 2001 14:43:58 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Spinlock performance improvement proposal
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> No scale factor, as I illustrated from the initialization command I
> used.  Standard buffers too.  Let me know what values I should use for
> testing.

Scale factor has to be >= max number of clients you use, else you're
just measuring serialization on the "branch" rows.

I think the default NBuffers (64) is too low to give meaningful
performance numbers, too.  I've been thinking that maybe we should
raise it to 1000 or so by default.  This would trigger startup failures
on platforms with small SHMMAX, but we could tell people to use -B until
they get around to fixing their kernel settings.  It's been a long time
since we fit into a 1-MB shared memory segment at the default settings
anyway, so maybe it's time to select somewhat-realistic defaults.
What we have now is neither very useful nor the lowest common
denominator...

			regards, tom lane

Sat, 29 Sep 2001 14:59:42 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Spinlock performance improvement proposal 

OK, testing now with 1000 backends and 2000 buffers.  Will report.

> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > No scale factor, as I illustrated from the initialization command I
> > used.  Standard buffers too.  Let me know what values I should use for
> > testing.
> 
> Scale factor has to be >= max number of clients you use, else you're
> just measuring serialization on the "branch" rows.
> 
> I think the default NBuffers (64) is too low to give meaningful
> performance numbers, too.  I've been thinking that maybe we should
> raise it to 1000 or so by default.  This would trigger startup failures
> on platforms with small SHMMAX, but we could tell people to use -B until
> they get around to fixing their kernel settings.  It's been a long time
> since we fit into a 1-MB shared memory segment at the default settings
> anyway, so maybe it's time to select somewhat-realistic defaults.
> What we have now is neither very useful nor the lowest common
> denominator...
> 
> 			regards, tom lane
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 29 Sep 2001 15:32:49 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Spinlock performance improvement proposal
Vadim Mikheev wrote:
> 
> > I have committed changes to implement this proposal.  I'm not seeing
> > any significant performance difference on pgbench on my single-CPU
> > system ... but pgbench is I/O bound anyway on this hardware, so that's
> > not very surprising.  I'll be interested to see what other people
> > observe.  (Tatsuo, care to rerun that 1000-client test?)
> 
> What is your system? CPU, memory, IDE/SCSI, OS?
> Scaling factor and # of clients?
> 
> BTW1 - shouldn't we rewrite pgbench to use threads instead of
> "libpq async queries"? At least as option. I'd say that with 1000
> clients current pgbench implementation is very poor.

Would it be useful to run a test like the AS3AP benchmark on this to
look for performance measurements?

On linux the Open Source Database Benchmark (osdb.sf.net) does this, and
it's multi-threaded to simulate multiple clients hitting the database at
once.  The only inconvenience is having to download a separate program
to generate the test data, as OSDB doesn't generate this itself yet.  I
can supply the test program (needs to be run through Wine) and a script
if anyone wants.

???

> 
> BTW2 - shouldn't we learn if there are really portability/performance
> issues in using POSIX mutex-es (and cond. variables) in place of
> TAS (and SysV semaphores)?
> 
> Vadim
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
     - Indira Gandhi

Sun, 30 Sep 2001 12:31:20 +1000
Justin Clift <justin@postgresql.org>
Re: Spinlock performance improvement proposal
On Sat, Sep 29, 2001 at 06:48:56PM +0530, Chamanya wrote:
> 
> Number of threads should be equal to or twice that of number of CPUs. I don't 
> think more than those many threads would yield any performance improvement.
> 
 
 This expects that thread still runnig, but each process (thread) sometime
waiting for disk, net etc. During this time can runs some other thread.
 Performance of program not directly depends on number of CPU, but on 
type of a work that execute thread. The important thing is how you can 
split a work to small and independent parts. 

	Karel

-- 
 Karel Zak  <zakkr@zf.jcu.cz>
 http://home.zf.jcu.cz/~zakkr/
 
 C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz

Mon, 1 Oct 2001 10:23:13 +0200
Karel Zak <zakkr@zf.jcu.cz>
Re: Spinlock performance improvement proposal
Tom Lane wrote:
> 
<snip>
> I think the default NBuffers (64) is too low to give meaningful
> performance numbers, too.  I've been thinking that maybe we should
> raise it to 1000 or so by default.  This would trigger startup failures
> on platforms with small SHMMAX, but we could tell people to use -B until
> they get around to fixing their kernel settings.  It's been a long time
> since we fit into a 1-MB shared memory segment at the default settings
> anyway, so maybe it's time to select somewhat-realistic defaults.
> What we have now is neither very useful nor the lowest common
> denominator...

How about a startup error message which gets displayed when used with
untuned settings (i.e. the default settings), maybe unless an option
like -q (quiet) is given?

My thought is the server should operate, but let the new/novice admin
know they need to configure PostgreSQL properly.  Would probably be a
good reminder for experienced admins if they forget too.

Maybe something simple like pg_ctl shell script message, or something
proper like a postmaster start-up check.

This wouldn't break anything would it?

Regards and best wishes,

Justin Clift

> 
>                         regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
     - Indira Gandhi

Mon, 01 Oct 2001 23:48:19 +1000
Justin Clift <justin@postgresql.org>
Re: Spinlock performance improvement proposal
> Tom Lane wrote:
> > 
> <snip>
> > I think the default NBuffers (64) is too low to give meaningful
> > performance numbers, too.  I've been thinking that maybe we should
> > raise it to 1000 or so by default.  This would trigger startup failures
> > on platforms with small SHMMAX, but we could tell people to use -B until
> > they get around to fixing their kernel settings.  It's been a long time
> > since we fit into a 1-MB shared memory segment at the default settings
> > anyway, so maybe it's time to select somewhat-realistic defaults.
> > What we have now is neither very useful nor the lowest common
> > denominator...
> 
> How about a startup error message which gets displayed when used with
> untuned settings (i.e. the default settings), maybe unless an option
> like -q (quiet) is given?
> 
> My thought is the server should operate, but let the new/novice admin
> know they need to configure PostgreSQL properly.  Would probably be a
> good reminder for experienced admins if they forget too.
> 
> Maybe something simple like pg_ctl shell script message, or something
> proper like a postmaster start-up check.

Yes, this seems like the way to go, probably something in the postmaster
log file.  For single-user developers, we want it to start but we want
production machines to tune it. In fact, picking a higher number for
these values may be almost as far off as our defaults.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 1 Oct 2001 11:41:44 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Spinlock performance improvement proposal
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Tom Lane wrote:
> I think the default NBuffers (64) is too low to give meaningful
> performance numbers, too.  I've been thinking that maybe we should
> raise it to 1000 or so by default.

>> Maybe something simple like pg_ctl shell script message, or something
>> proper like a postmaster start-up check.

> Yes, this seems like the way to go, probably something in the postmaster
> log file.

Except that a lot of people send postmaster stderr to /dev/null.
I think bleating about untuned parameters in the postmaster log will be
next to useless, because it won't do a thing except for people who are
clueful enough to (a) direct the log someplace useful and (b) look at it
carefully.  Those folks are not the ones who need help about tuning.

We already have quite detailed error messages for shmget/semget
failures, eg

$ postmaster -B 200000
IpcMemoryCreate: shmget(key=5440001, size=1668366336, 03600) failed: Invalid argument

This error can be caused by one of three things:

1. The maximum size for shared memory segments on your system was
   exceeded.  You need to raise the SHMMAX parameter in your kernel
   to be at least 4042162176 bytes.

2. The requested shared memory segment was too small for your system.
   You need to lower the SHMMIN parameter in your kernel.

3. The requested shared memory segment already exists but is of the
   wrong size.  This can occur if some other application on your system
   is also using shared memory.

The PostgreSQL Administrator's Guide contains more information about
shared memory configuration.


This is still missing a bet since it fails to mention the option of
adjusting -B and -N instead of changing kernel parameters, but that's
easily fixed.  I propose that we reword this message and the semget
one to mention first the option of changing -B/-N and second the option
of changing kernel parameters.  Then we could consider raising the
default -B setting to something more realistic.

			regards, tom lane

Mon, 01 Oct 2001 12:19:06 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Spinlock performance improvement proposal 
> This is still missing a bet since it fails to mention the option of
> adjusting -B and -N instead of changing kernel parameters, but that's
> easily fixed.  I propose that we reword this message and the semget
> one to mention first the option of changing -B/-N and second the option
> of changing kernel parameters.  Then we could consider raising the
> default -B setting to something more realistic.

Yes, we could do that but it makes things harder for newbies and really
isn't the right numbers for production use anyway.  I think anyone using
default values should see a message asking them to tune it.  Can we
throw a message during initdb?  Of course, we don't have a running
backend at that point so you would always throw a message.

 From postmaster startup, by default, could we try larger amounts of
buffer memory until it fails then back off and allocate that?  Seems
like a nice default to me.


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 1 Oct 2001 14:49:45 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Spinlock performance improvement proposal
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> From postmaster startup, by default, could we try larger amounts of
> buffer memory until it fails then back off and allocate that?  Seems
> like a nice default to me.

Chewing all available memory is the very opposite of a nice default,
I'd think.

The real problem here is that some platforms will let us have huge shmem
segments, and some will only let us have tiny ones, and neither of those
is a reasonable default behavior.  Allowing the platform to determine
our sizing is the wrong way round IMHO; the dbadmin should have a clear
idea of what he's getting, and silent adjustment of the B/N parameters
will not give him that.

			regards, tom lane

Mon, 01 Oct 2001 14:55:25 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Spinlock performance improvement proposal 

Bruce Momjian <pgman@candle.pha.pa.us> wrote:

> From postmaster startup, by default, could we try larger amounts of
> buffer memory until it fails then back off and allocate that?  Seems
> like a nice default to me.

So performance would vary depending on the amount of shared memory
that could be allocated at startup?  Not a good idea IMHO.

Regards,

Giles


Tue, 02 Oct 2001 07:57:10 +1000
Giles Lean <giles@nemeton.com.au>
Re: Spinlock performance improvement proposal 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > OK, that makes sense.  My only question is how many platforms _don't_
> > have syslog.  If it is only NT and QNX, I think we can live with using
> > it by default if it exists.
> 
> There seems to be a certain amount of confusion here.  The proposal at
> hand was to make configure set up to *compile* the syslog support
> whenever possible.  Not to *use* syslog by default.  Unless we change
> the default postgresql.conf --- which I would be against --- we will
> still log to stderr by default.
> 
> Given that, I'm not sure that Peter's argument about losing
> functionality is right; the analogy to readline support isn't exact.
> Perhaps what we should do is (a) always build syslog support if
> possible, and (b) at runtime, complain if syslog logging is requested
> but we don't have it available.

Did we decide to compile in syslog support by default?  I thought so.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 11 Oct 2001 12:39:01 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: syslog by default?
"Thomas Yackel" <yackelt@ohsu.edu> writes:
> I got the error: "Bad abstime external representation ''" when attempted to start psql as a particular user and the postmaster shutdown.

> The problem, we discovered, is that this user had a carriage return contained within his password.  Changing the password to remove the CR avoided the system shutdown.

Hmm.  I can see how a linefeed in a password would create a problem (it
breaks the line-oriented formatting of the pg_pwd file).  However, I
can't reproduce a postmaster crash here.  Either I'm not testing the
right combination of circumstances, or current sources are more robust
about this than 7.1.  That's not unlikely given that Bruce rewrote the
password-file-parsing code a couple months ago.

In any case it seems like it'd be a good idea to forbid nonprinting
characters in passwords.  Comments anyone?

			regards, tom lane

Thu, 01 Nov 2001 00:28:25 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: user authentication crash by Erik Luke (20-08-2001; 1.3kb) 

    when doing some works with views, I faced the following problem :

    consider the following schema :

    create table A (v1 int4,v2 int4);
    create table B (v1 int4,v2 int4);
    create view C as select v1,v2 from A union all select v1,v2 from B;

    populate A and B with several thousands records

    select v1 from c where v2=1000; give the following plan :

Subquery Scan c  (cost=0.00..4544.12 rows=294912 width=8)
  ->  Append  (cost=0.00..4544.12 rows=294912 width=8)
        ->  Subquery Scan *SELECT* 1  (cost=0.00..252.84 rows=16384 width=8)
              ->  Seq Scan on a  (cost=0.00..252.84 rows=16384 width=8)
        ->  Subquery Scan *SELECT* 2  (cost=0.00..4291.28 rows=278528
width=8)
              ->  Seq Scan on b  (cost=0.00..4291.28 rows=278528 width=8)


    select v1 from a where v2=5 union all select v1 from b where v2=1000;
give the following plan :

Append  (cost=0.00..217.88 rows=83 width=4)
  ->  Subquery Scan *SELECT* 1  (cost=0.00..2.02 rows=1 width=4)
        ->  Index Scan using idx1 on a  (cost=0.00..2.02 rows=1 width=4)
  ->  Subquery Scan *SELECT* 2  (cost=0.00..215.86 rows=82 width=4)
        ->  Index Scan using idx2 on b  (cost=0.00..215.86 rows=82 width=4)

    Is there a way for the optimizer to move the view "where" clause in the
elementary union queries in order to use an index scan instead of the Seq
scan ?

    I'm using 7.1.3


            cyril


Thu, 1 Nov 2001 17:40:46 +0100
"Cyril VELTER" <cyril.velter@libertysurf.fr>
Union View Optimization
Tom Lane wrote:
  >Hmm.  I can see how a linefeed in a password would create a problem (it
  >breaks the line-oriented formatting of the pg_pwd file). 
...
  >In any case it seems like it'd be a good idea to forbid nonprinting
  >characters in passwords.  Comments anyone?

That sounds too restrictive; allowing non-printing characters should
improve password security.  Why not simply exclude linefeed and
carriage return?  (And possibly ctrl-Q and ctrl-S as well, in case there
is still anyone running a terminal with XON/XOFF flow control.)

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C

     "But they that wait upon the LORD shall renew their 
      strength; they shall mount up with wings as eagles; 
      they shall run, and not be weary; and they shall walk,
      and not faint."            Isaiah 40:31 



Thu, 01 Nov 2001 17:34:01 +0000
"Oliver Elphick" <olly@lfix.co.uk>
Re: [BUGS] user authentication crash by Erik Luke 
"Oliver Elphick" <olly@lfix.co.uk> writes:
> Tom Lane wrote:
>>>> Hmm.  I can see how a linefeed in a password would create a problem (it
>>>> breaks the line-oriented formatting of the pg_pwd file). 
> ...
>>>> In any case it seems like it'd be a good idea to forbid nonprinting
>>>> characters in passwords.  Comments anyone?

> That sounds too restrictive; allowing non-printing characters should
> improve password security.  Why not simply exclude linefeed and
> carriage return?

Actually it seems that linefeed and tab are the minimum set of
characters that must be excluded to avoid breaking pg_pwd.
Working on it now ...

			regards, tom lane

Thu, 01 Nov 2001 12:52:28 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [BUGS] user authentication crash by Erik Luke (20-08-2001;
 1.3kb) 
"Cyril VELTER" <cyril.velter@libertysurf.fr> writes:
>     Is there a way for the optimizer to move the view "where" clause in the
> elementary union queries in order to use an index scan instead of the Seq
> scan ?

This is on the "to look at" list.  It's not immediately clear to me
whether there are any restrictions on when the system can safely make
such a transformation, nor whether there are cases where it would be
a pessimization rather than an optimization.

			regards, tom lane

Thu, 01 Nov 2001 13:45:05 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Union View Optimization 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > OK, that makes sense.  My only question is how many platforms _don't_
> > have syslog.  If it is only NT and QNX, I think we can live with using
> > it by default if it exists.
> 
> There seems to be a certain amount of confusion here.  The proposal at
> hand was to make configure set up to *compile* the syslog support
> whenever possible.  Not to *use* syslog by default.  Unless we change
> the default postgresql.conf --- which I would be against --- we will
> still log to stderr by default.
> 
> Given that, I'm not sure that Peter's argument about losing
> functionality is right; the analogy to readline support isn't exact.
> Perhaps what we should do is (a) always build syslog support if
> possible, and (b) at runtime, complain if syslog logging is requested
> but we don't have it available.

Is this idea dead for 7.2?  Should I add it to the TODO list?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 5 Nov 2001 11:01:51 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: syslog by default?
Are we doing beta3 soon?  I thought we were going to do it yesterday,
though it seems like we have a few packaging problems to iron out first.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Sat, 17 Nov 2001 13:19:33 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
beta3
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Are we doing beta3 soon?  I thought we were going to do it yesterday,
> though it seems like we have a few packaging problems to iron out first.

The packaging problems need to be solved before we can build a release
candidate --- but I don't see why they should hold up beta3.  beta2 has
the same problems, so beta3 wouldn't be a regression.

I don't have any pending work that ought to go in beta3, but I think
Thomas is sitting on some datetime patches ... probably should wait
for him.

			regards, tom lane

Sat, 17 Nov 2001 14:07:18 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta3 
Tom Lane writes:

> The packaging problems need to be solved before we can build a release
> candidate --- but I don't see why they should hold up beta3.  beta2 has
> the same problems, so beta3 wouldn't be a regression.

Considering the history of how-do-we-get-the-docs-in-the-tarball problem,
I'd like to see at least one beta where this works before shipping a
release candidate.  Otherwise I'll have no confidence at all that we're
not going to ship the final release with last year's docs and no man
pages.

-- 
Peter Eisentraut   peter_e@gmx.net


Sun, 18 Nov 2001 18:17:41 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: beta3 

Okay, how do we want to address/fix/test this?

On Sun, 18 Nov 2001, Peter Eisentraut wrote:

> Tom Lane writes:
>
> > The packaging problems need to be solved before we can build a release
> > candidate --- but I don't see why they should hold up beta3.  beta2 has
> > the same problems, so beta3 wouldn't be a regression.
>
> Considering the history of how-do-we-get-the-docs-in-the-tarball problem,
> I'd like to see at least one beta where this works before shipping a
> release candidate.  Otherwise I'll have no confidence at all that we're
> not going to ship the final release with last year's docs and no man
> pages.
>
> --
> Peter Eisentraut   peter_e@gmx.net
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html
>


Sun, 18 Nov 2001 22:42:43 -0500 (EST)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta3 
Marc G. Fournier writes:

> Okay, how do we want to address/fix/test this?

Assuming that we don't want to make sweeping changes to the overall
procedure, I suggest this:

1.  The postgres.tar.gz and man.tar.gz documentation tarballs required for
a release used to be at ftp://ftp.postgresql.org/pub/dev/doc/.  The new
file system location of this directory needs to be determined.

2.  Some postgres.tar.gz and man.tar.gz files need to be put there.

3.  The currently commented out portion in the mk-release script needs
to be altered to pick up these files.

(I also suggest 'set -e' in this script so it doesn't ignore errors.)

4.  The HTML-doc-build cron job needs to be reinstated on postgresql.org
and put its output file at the new location.

5.  When I'm done with the new manpages I'll upload them to the
appropriate location.

I think the answer to #1 is /var/spool/ftp or something.  For #2, use the
postgres.tar.gz files that Bruce currently builds, and use the old
man.tar.gz file from 7.1.

I suggest when #3 is done we can ship a beta3.

For #4, I don't know who wants to do it.  If it's me then it might take a
few days.  #5 is well under way.  When #4 and #5 are done we can ship rc1
or beta4, depending on the other issues.

-- 
Peter Eisentraut   peter_e@gmx.net


Mon, 19 Nov 2001 15:39:52 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: beta3 
On Mon, 19 Nov 2001, Peter Eisentraut wrote:

> Marc G. Fournier writes:
>
> > Okay, how do we want to address/fix/test this?
>
> Assuming that we don't want to make sweeping changes to the overall
> procedure, I suggest this:
>
> 1.  The postgres.tar.gz and man.tar.gz documentation tarballs required for
> a release used to be at ftp://ftp.postgresql.org/pub/dev/doc/.  The new
> file system location of this directory needs to be determined.

Should simply be ~ftp/pub/dev/doc

> 2.  Some postgres.tar.gz and man.tar.gz files need to be put there.
>
> 3.  The currently commented out portion in the mk-release script needs
> to be altered to pick up these files.
>
> (I also suggest 'set -e' in this script so it doesn't ignore errors.)

Okay, just added 'set -e', commented out, to the script ... actually, just
uncommented it, for, even without the docs, it probably shouldn't be
ignored, eh? :)

> 4.  The HTML-doc-build cron job needs to be reinstated on
> postgresql.org and put its output file at the new location.
>
> 5.  When I'm done with the new manpages I'll upload them to the
> appropriate location.
>
> I think the answer to #1 is /var/spool/ftp or something.  For #2, use the
> postgres.tar.gz files that Bruce currently builds, and use the old
> man.tar.gz file from 7.1.
>
> I suggest when #3 is done we can ship a beta3.

Okay, will bundle up Beta3 in the morning then ...

> For #4, I don't know who wants to do it.  If it's me then it might
> take a few days.  #5 is well under way.  When #4 and #5 are done we
> can ship rc1 or beta4, depending on the other issues.

Would #4 be these, from thomas old cron jobs:

# Run at 3:05 local time (EST5EDT?) every day
 5  3 * * *       $HOME/CURRENT/docbuild |& tee $HOME/CURRENT/docbuild.log
# Twice a day during beta 2000-03-28
25 11 * * *       $HOME/CURRENT/docbuild |& tee $HOME/CURRENT/docbuild.log


Mon, 19 Nov 2001 19:32:04 -0500 (EST)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta3 


> -----Original Message-----
> From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Peter Eisentraut
> Sent: Monday, 19 November 2001 10:40 PM
> To: Marc G. Fournier
> Cc: Tom Lane; Bruce Momjian; PostgreSQL-development; Thomas Lockhart
> Subject: Re: [HACKERS] beta3
>
>
> Marc G. Fournier writes:
>
> > Okay, how do we want to address/fix/test this?
>
> Assuming that we don't want to make sweeping changes to the overall
> procedure, I suggest this:
>
> 1.  The postgres.tar.gz and man.tar.gz documentation tarballs required for
> a release used to be at ftp://ftp.postgresql.org/pub/dev/doc/.  The new
> file system location of this directory needs to be determined.
>
> 2.  Some postgres.tar.gz and man.tar.gz files need to be put there.

Just a thought:  Doesn't everyone have bzip2 these days?  Many large
projects come bz2-only even these days.  Is there a good reason to stick
with the extra-large tar.gz format?

Chris


Tue, 20 Nov 2001 09:42:01 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: beta3 
On Tue, 20 Nov 2001, Christopher Kings-Lynne wrote:

>
>
> > -----Original Message-----
> > From: pgsql-hackers-owner@postgresql.org
> > [mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Peter Eisentraut
> > Sent: Monday, 19 November 2001 10:40 PM
> > To: Marc G. Fournier
> > Cc: Tom Lane; Bruce Momjian; PostgreSQL-development; Thomas Lockhart
> > Subject: Re: [HACKERS] beta3
> >
> >
> > Marc G. Fournier writes:
> >
> > > Okay, how do we want to address/fix/test this?
> >
> > Assuming that we don't want to make sweeping changes to the overall
> > procedure, I suggest this:
> >
> > 1.  The postgres.tar.gz and man.tar.gz documentation tarballs required for
> > a release used to be at ftp://ftp.postgresql.org/pub/dev/doc/.  The new
> > file system location of this directory needs to be determined.
> >
> > 2.  Some postgres.tar.gz and man.tar.gz files need to be put there.
>
> Just a thought:  Doesn't everyone have bzip2 these days?  Many large
> projects come bz2-only even these days.  Is there a good reason to stick
> with the extra-large tar.gz format?

Cause not everyone has bzip2 ... Solaris comes with gzip, doesn't come
with bzip ... not sure how many other OS, but even FreeBSD requiresyou to
compile bzip from ports, it isn't part of hte operating system ...



Mon, 19 Nov 2001 20:49:20 -0500 (EST)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta3 
> Cause not everyone has bzip2 ... Solaris comes with gzip, doesn't come
> with bzip ... not sure how many other OS, but even FreeBSD requiresyou to
> compile bzip from ports, it isn't part of hte operating system ...

FreeBSD 4.4 has it as part of the OS...

For example, the entire KDE project is bz2, and people use it on Solaris.

But anyway, fair enough - I guess it's an extra step that would inhibit the
use of Postgres.

Chris


Tue, 20 Nov 2001 10:39:48 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: beta3 
On Lun 19 Nov 2001 22:49, you wrote:
> On Tue, 20 Nov 2001, Christopher Kings-Lynne wrote:
> > > -----Original Message-----
> > > From: pgsql-hackers-owner@postgresql.org
> > > [mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Peter
> > > Eisentraut Sent: Monday, 19 November 2001 10:40 PM
> > > To: Marc G. Fournier
> > > Cc: Tom Lane; Bruce Momjian; PostgreSQL-development; Thomas Lockhart
> > > Subject: Re: [HACKERS] beta3
> > >
> > > Marc G. Fournier writes:
> > > > Okay, how do we want to address/fix/test this?
> > >
> > > Assuming that we don't want to make sweeping changes to the overall
> > > procedure, I suggest this:
> > >
> > > 1.  The postgres.tar.gz and man.tar.gz documentation tarballs required
> > > for a release used to be at ftp://ftp.postgresql.org/pub/dev/doc/.  The
> > > new file system location of this directory needs to be determined.
> > >
> > > 2.  Some postgres.tar.gz and man.tar.gz files need to be put there.
> >
> > Just a thought:  Doesn't everyone have bzip2 these days?  Many large
> > projects come bz2-only even these days.  Is there a good reason to stick
> > with the extra-large tar.gz format?
>
> Cause not everyone has bzip2 ... Solaris comes with gzip, doesn't come
> with bzip ... not sure how many other OS, but even FreeBSD requiresyou to
> compile bzip from ports, it isn't part of hte operating system ...

Go to www.sunfreeware.com and get bzip2 for Solaris.
I even recompiled the new tar with support for bzip2! :-)

Saludos... :-)

P.D.: bzip2 is slow, but you can get a real small package with it, even 
though PostgreSQL isn't that big, if we compare it with KDE or Mozilla.

-- 
Porquï¿½ usar una base de datos relacional cualquiera,
si podï¿½s usar PostgreSQL?
-----------------------------------------------------------------
Martï¿½n Marquï¿½s                  |        mmarques@unl.edu.ar
Programador, Administrador, DBA |       Centro de Telematica
                       Universidad Nacional
                            del Litoral
-----------------------------------------------------------------

Tue, 20 Nov 2001 08:51:13 -0300
=?iso-8859-1?q?Mart=EDn=20Marqu=E9s?= <martin@bugs.unl.edu.ar>
Re: beta3
=?iso-8859-1?q?Mart=EDn=20Marqu=E9s?= <martin@bugs.unl.edu.ar> writes:
> P.D.: bzip2 is slow, but you can get a real small package with it, even 
> though PostgreSQL isn't that big, if we compare it with KDE or Mozilla.

As an experiment, I zipped my current PG source tree with both.  (This
isn't an exact test of the distribution size, because I didn't bother
to get rid of the CVS control files, but it's pretty close.)

Original tar file:      37089280 bytes
gzip -9:		 8183182 bytes
bzip2:			 6762638 bytes

or slightly less than a 20% savings for bzip over gzip.  That's useful,
but not exactly compelling.  A comparison of unzip runtime also seems
relevant:

$ time gunzip pgsql.tar.gz

real    0m5.48s
user    0m4.46s
sys     0m0.62s

$ time bunzip2 pgsql.tar.bz2

real    0m27.77s
user    0m26.50s
sys     0m0.92s

If I'd downloaded this thing over a decent DSL or cable modem line,
bzip2 would actually be a net loss in total download + uncompress time.

<editorial>
The reason bzip is still an also-ran is that it's not enough better
than gzip to have persuaded people to switch over.  My bet is that
bzip will always be an also-ran, and that gzip will remain the de
facto standard until something comes along that's really significantly
better, like a factor of 2 better.  I've watched this sort of game
play out before, and I know you don't take over the world with a 20%
improvement over the existing standard.  At least not without other
compelling reasons, like speed (oops) or patent freedom (no win there
either).
</editorial>

			regards, tom lane

Tue, 20 Nov 2001 10:16:57 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta3 
On Mar 20 Nov 2001 12:16, Tom Lane wrote:
>
> As an experiment, I zipped my current PG source tree with both.  (This
> isn't an exact test of the distribution size, because I didn't bother
> to get rid of the CVS control files, but it's pretty close.)
>
> Original tar file:      37089280 bytes
> gzip -9:		 8183182 bytes
> bzip2:			 6762638 bytes
>
> or slightly less than a 20% savings for bzip over gzip.  That's useful,
> but not exactly compelling.  A comparison of unzip runtime also seems
> relevant:
>
> $ time gunzip pgsql.tar.gz
>
> real    0m5.48s
> user    0m4.46s
> sys     0m0.62s
>
> $ time bunzip2 pgsql.tar.bz2
>
> real    0m27.77s
> user    0m26.50s
> sys     0m0.92s
>
> If I'd downloaded this thing over a decent DSL or cable modem line,
> bzip2 would actually be a net loss in total download + uncompress time.

That would be if I have a decent DSL or cable modem. We have a dedicated line 
of 756 kb, but we also have 3000 users. Maybe postgreSQL isn't that big, but 
Mozilla or KDE are, and waiting for la large download isn't what I recommend 
for an internet experience. :-)

> <editorial>
> The reason bzip is still an also-ran is that it's not enough better
> than gzip to have persuaded people to switch over.  My bet is that
> bzip will always be an also-ran, and that gzip will remain the de
> facto standard until something comes along that's really significantly
> better, like a factor of 2 better.  I've watched this sort of game
> play out before, and I know you don't take over the world with a 20%
> improvement over the existing standard.  At least not without other
> compelling reasons, like speed (oops) or patent freedom (no win there
> either).
> </editorial>

I think it comes to what CPU I'm using. If I'm on an old Pentium, I may not 
be happy seeing unbzip2 grabbing all my CPU, but if I'm on a last generation 
PIV of, lets say 1000Mhz, I may not feel it.

Saludos... :-)

-- 
Porquï¿½ usar una base de datos relacional cualquiera,
si podï¿½s usar PostgreSQL?
-----------------------------------------------------------------
Martï¿½n Marquï¿½s                  |        mmarques@unl.edu.ar
Programador, Administrador, DBA |       Centro de Telematica
                       Universidad Nacional
                            del Litoral
-----------------------------------------------------------------

Tue, 20 Nov 2001 19:51:14 -0300
=?iso-8859-1?q?Mart=EDn=20Marqu=E9s?= <martin@bugs.unl.edu.ar>
Re: beta3

Okay, tried today what you suggested, about set -e and all that ... take a
look at ~pgsql/bin/mk-beta, to see if maybe I missed something ... but,
trying to do the packaging, I'm getting an error about 'build.xml'?

/usr/bin/tar cf postgresql-opt-7.2b3.tar postgresql-7.2b3/src/backend/utils/mb postgresql-7.2b3/contrib/retep postgresql-7.2b3/build.xml postgresql-7.2b3/src/tools postgresql-7.2b3/src/corba postgresql-7.2b3/src/data postgresql-7.2b3/src/tutorial postgresql-7.2b3/src/bin/pgaccess postgresql-7.2b3/src/bin/pgtclsh postgresql-7.2b3/src/bin/pg_encoding postgresql-7.2b3/src/interfaces/odbc postgresql-7.2b3/src/interfaces/libpq++ postgresql-7.2b3/src/interfaces/libpgtcl postgresql-7.2b3/src/interfaces/perl5 postgresql-7.2b3/src/interfaces/python postgresql-7.2b3/src/interfaces/jdbc postgresql-7.2b3/src/pl/plperl postgresql-7.2b3/src/pl/tcl
/usr/bin/tar: can't add file postgresql-7.2b3/build.xml : No such file or directory
gmake: *** [postgresql-opt-7.2b3.tar] Error 1
gmake: *** Deleting file `postgresql-opt-7.2b3.tar'
%

Looking in the wrong directory?

%find . -name build.xml -print
./contrib/retep/build.xml
./src/interfaces/jdbc/build.xml
%


On Mon, 19 Nov 2001, Peter Eisentraut wrote:

> Marc G. Fournier writes:
>
> > Okay, how do we want to address/fix/test this?
>
> Assuming that we don't want to make sweeping changes to the overall
> procedure, I suggest this:
>
> 1.  The postgres.tar.gz and man.tar.gz documentation tarballs required for
> a release used to be at ftp://ftp.postgresql.org/pub/dev/doc/.  The new
> file system location of this directory needs to be determined.
>
> 2.  Some postgres.tar.gz and man.tar.gz files need to be put there.
>
> 3.  The currently commented out portion in the mk-release script needs
> to be altered to pick up these files.
>
> (I also suggest 'set -e' in this script so it doesn't ignore errors.)
>
> 4.  The HTML-doc-build cron job needs to be reinstated on postgresql.org
> and put its output file at the new location.
>
> 5.  When I'm done with the new manpages I'll upload them to the
> appropriate location.
>
> I think the answer to #1 is /var/spool/ftp or something.  For #2, use the
> postgres.tar.gz files that Bruce currently builds, and use the old
> man.tar.gz file from 7.1.
>
> I suggest when #3 is done we can ship a beta3.
>
> For #4, I don't know who wants to do it.  If it's me then it might take a
> few days.  #5 is well under way.  When #4 and #5 are done we can ship rc1
> or beta4, depending on the other issues.
>
> --
> Peter Eisentraut   peter_e@gmx.net
>
>



Tue, 20 Nov 2001 23:09:11 -0500 (EST)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta3 
> 
> Okay, tried today what you suggested, about set -e and all that ... take a
> look at ~pgsql/bin/mk-beta, to see if maybe I missed something ... but,
> trying to do the packaging, I'm getting an error about 'build.xml'?
> 
> /usr/bin/tar cf postgresql-opt-7.2b3.tar postgresql-7.2b3/src/backend/utils/mb postgresql-7.2b3/contrib/retep postgresql-7.2b3/build.xml postgresql-7.2b3/src/tools postgresql-7.2b3/src/corba postgresql-7.2b3/src/data postgresql-7.2b3/src/tutorial postgresql-7.2b3/src/bin/pgaccess postgresql-7.2b3/src/bin/pgtclsh postgresql-7.2b3/src/bin/pg_encoding postgresql-7.2b3/src/interfaces/odbc postgresql-7.2b3/src/interfaces/libpq++ postgresql-7.2b3/src/interfaces/libpgtcl postgresql-7.2b3/src/interfaces/perl5 postgresql-7.2b3/src/interfaces/python postgresql-7.2b3/src/interfaces/jdbc postgresql-7.2b3/src/pl/plperl postgresql-7.2b3/src/pl/tcl
> /usr/bin/tar: can't add file postgresql-7.2b3/build.xml : No such file or directory
> gmake: *** [postgresql-opt-7.2b3.tar] Error 1
> gmake: *** Deleting file `postgresql-opt-7.2b3.tar'
> %
> 
> Looking in the wrong directory?
> 
> %find . -name build.xml -print
> ./contrib/retep/build.xml
> ./src/interfaces/jdbc/build.xml
> %

OK, patch applied.  There was a space where there should have been a
slash.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Index: GNUmakefile.in
===================================================================
RCS file: /cvsroot/pgsql/GNUmakefile.in,v
retrieving revision 1.19
diff -c -r1.19 GNUmakefile.in
*** GNUmakefile.in	2001/09/17 23:00:27	1.19
--- GNUmakefile.in	2001/11/21 05:40:51
***************
*** 65,71 ****
  $(distdir).tar: distdir
  	$(TAR) chf $@ $(distdir)
  
! opt_files := src/backend/utils/mb contrib/retep build.xml \
  	src/tools src/corba src/data src/tutorial \
  	$(addprefix src/bin/, pgaccess pgtclsh pg_encoding) \
  	$(addprefix src/interfaces/, odbc libpq++ libpgtcl perl5 python jdbc) \
--- 65,71 ----
  $(distdir).tar: distdir
  	$(TAR) chf $@ $(distdir)
  
! opt_files := src/backend/utils/mb contrib/retep/build.xml \
  	src/tools src/corba src/data src/tutorial \
  	$(addprefix src/bin/, pgaccess pgtclsh pg_encoding) \
  	$(addprefix src/interfaces/, odbc libpq++ libpgtcl perl5 python jdbc) \
Wed, 21 Nov 2001 00:41:49 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
> > 
> > Okay, tried today what you suggested, about set -e and all that ... take a
> > look at ~pgsql/bin/mk-beta, to see if maybe I missed something ... but,
> > trying to do the packaging, I'm getting an error about 'build.xml'?
> > 
> > /usr/bin/tar cf postgresql-opt-7.2b3.tar postgresql-7.2b3/src/backend/utils/mb postgresql-7.2b3/contrib/retep postgresql-7.2b3/build.xml postgresql-7.2b3/src/tools postgresql-7.2b3/src/corba postgresql-7.2b3/src/data postgresql-7.2b3/src/tutorial postgresql-7.2b3/src/bin/pgaccess postgresql-7.2b3/src/bin/pgtclsh postgresql-7.2b3/src/bin/pg_encoding postgresql-7.2b3/src/interfaces/odbc postgresql-7.2b3/src/interfaces/libpq++ postgresql-7.2b3/src/interfaces/libpgtcl postgresql-7.2b3/src/interfaces/perl5 postgresql-7.2b3/src/interfaces/python postgresql-7.2b3/src/interfaces/jdbc postgresql-7.2b3/src/pl/plperl postgresql-7.2b3/src/pl/tcl
> > /usr/bin/tar: can't add file postgresql-7.2b3/build.xml : No such file or directory
> > gmake: *** [postgresql-opt-7.2b3.tar] Error 1
> > gmake: *** Deleting file `postgresql-opt-7.2b3.tar'
> > %
> > 
> > Looking in the wrong directory?
> > 
> > %find . -name build.xml -print
> > ./contrib/retep/build.xml
> > ./src/interfaces/jdbc/build.xml
> > %
> 
> OK, patch applied.  There was a space where there should have been a
> slash.

OK, I found another problem:

  gzip -f --best postgresql-base-7.2b3.tar
  /usr/bin/tar cf postgresql-docs-7.2b3.tar
  postgresql-7.2b3/doc/postgres.tar.gz postgresql-7.2b3/doc/src

                   ^^^^^^^^^^^^^^^^^^^
  postgresql-7.2b3/doc/TODO.detail postgresql-7.2b3/doc/internals.ps
  /usr/bin/tar: can't add file postgresql-7.2b3/doc/postgres.tar.gz : No

                                                ^^^^^^^^^^^^^^^^^^^

  such file or directory

The problem is this line in pgsql/GNUmakefile.in:

  docs_files := doc/postgres.tar.gz doc/src doc/TODO.detail doc/internals.ps

Why is doc/postgres.tar.gz used.  It normally is in doc/src, and why
only that doc tarball.

And again, there is that intenrals.ps file that doesn't really belong
there, I think.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 01:21:52 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
On Tue, Nov 20, 2001 at 10:16:57AM -0500, Tom Lane wrote:
> Original tar file:      37089280 bytes
> gzip -9:		 8183182 bytes
> bzip2:			 6762638 bytes

maybe 1.420.544 it's not a big difference for you, but here in italy
there's still a lot of people using 33.6/55Kb telephone line modem.
so, this difference means, downloading at 3kb/s, seven minutes...
also:

a) cpu utilization has no cost... band cost a lot;
b) if i have to download 5 files, and i can avoid 1 MB of download for each
   one...

thanks for your time,
andrea gelmini

Wed, 21 Nov 2001 12:02:11 +0100
andrea gelmini <andrea.gelmini@linux.it>
Re: beta3
> OK, I found another problem:
> 
>   gzip -f --best postgresql-base-7.2b3.tar
>   /usr/bin/tar cf postgresql-docs-7.2b3.tar
>   postgresql-7.2b3/doc/postgres.tar.gz postgresql-7.2b3/doc/src
> 
>                    ^^^^^^^^^^^^^^^^^^^
>   postgresql-7.2b3/doc/TODO.detail postgresql-7.2b3/doc/internals.ps
>   /usr/bin/tar: can't add file postgresql-7.2b3/doc/postgres.tar.gz : No
> 
>                                                 ^^^^^^^^^^^^^^^^^^^
> 
>   such file or directory
> 
> The problem is this line in pgsql/GNUmakefile.in:
> 
>   docs_files := doc/postgres.tar.gz doc/src doc/TODO.detail doc/internals.ps
> 
> Why is doc/postgres.tar.gz used.  It normally is in doc/src, and why
> only that doc tarball.

OK, I updated this to point to doc/src.  Marc, I have a
doc/postgres.tar.gz here that you can use to make the tarball.  It is
now at ftp://candle.pha.pa.us/pub/postgresql.

> And again, there is that intenrals.ps file that doesn't really belong
> there, I think.

Tom agrees so I will move it to the web site.  Shame we don't have the
source, which was originally TeX.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 10:30:54 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
On Wed, Nov 21, 2001 at 10:30:54AM -0500, Bruce Momjian wrote:
> 
> > And again, there is that intenrals.ps file that doesn't really belong
> > there, I think.
> 
> Tom agrees so I will move it to the web site.  Shame we don't have the
> source, which was originally TeX.
> 

Digging in the archives, I find that the author of this doc offered
the source:

> Date: Wed, 13 Jan 1999 06:04:15 +0000
> From: Clark Evans <clark.evans@manhattanproject.com>
> Subject: Re: [HACKERS] Re: EXCEPT/INTERSECT for v6.4
> 
> Thomas,
> 
> The documentation Stefan wrote for his Master's thesis
> is great stuff.  Perhaps some of could be incorporated?
> 
> Stefan Simkovics wrote:
> > > I also included the text of my Master's Thesis. (a postscript
> > > version). I hope that you find something of it useful and would be
> > > happy if parts of it find their way into the PostgreSQL documentation
> > > project (If so, tell me, then I send the sources of the document!)


If we can track down his current email address, perhaps he still has it?
I'm copying this to his @ag.or.at address, since the mailer there still
recognizes it. Let's see if Stefan's around. ;-)

Ross (musing if he could find the source docs to his Ph.D. thesis ... and
if so, if they'd be readable)

-- 
Ross Reedstrom, Ph.D.                                 reedstrm@rice.edu
Executive Director                                  phone: 713-348-6166
Gulf Coast Consortium for Bioinformatics              fax: 713-348-6182
Rice University MS-39
Houston, TX 77005

Wed, 21 Nov 2001 10:14:39 -0600
"Ross J. Reedstrom" <reedstrm@rice.edu>
Re: beta3
> > Stefan Simkovics wrote:
> > > > I also included the text of my Master's Thesis. (a postscript
> > > > version). I hope that you find something of it useful and would be
> > > > happy if parts of it find their way into the PostgreSQL documentation
> > > > project (If so, tell me, then I send the sources of the document!)
> 
> 
> If we can track down his current email address, perhaps he still has it?
> I'm copying this to his @ag.or.at address, since the mailer there still
> recognizes it. Let's see if Stefan's around. ;-)
> 
> Ross (musing if he could find the source docs to his Ph.D. thesis ... and
> if so, if they'd be readable)

The thesis was in LaTeX.  I think Thomas may have copy, but I am not sure.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 11:47:40 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
Hi!

As you can see, the address is still valid :-)
Could you please let me know exactly , who is looking for what?

Then I can send you a LaTeX/PDF/PS Version of my thesis.

best regards

  Stefan Simkovics

> > > Stefan Simkovics wrote:
> > > > > I also included the text of my Master's Thesis. (a postscript
> > > > > version). I hope that you find something of it useful and would be
> > > > > happy if parts of it find their way into the PostgreSQL documentation
> > > > > project (If so, tell me, then I send the sources of the document!)
> > 
> > 
> > If we can track down his current email address, perhaps he still has it?
> > I'm copying this to his @ag.or.at address, since the mailer there still
> > recognizes it. Let's see if Stefan's around. ;-)
> > 
> > Ross (musing if he could find the source docs to his Ph.D. thesis ... and
> > if so, if they'd be readable)
> 
> The thesis was in LaTeX.  I think Thomas may have copy, but I am not sure.
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
> 

-- 
=====================================================================
DI Stefan Simkovics
KPNQwest Austria GmbH
A-1150 Wien, Diefenbachgasse 35
Tel.: +43 (1) 899 33 - 162, Fax: +43 (1) 899 33 - 10 162
Mob.: +43 (0) 664 818 91 41, e-Mail: stefan.simkovics@kpnqwest.com
http://www.kpnqwest.at, http://www.kpnqwest.com



Wed, 21 Nov 2001 19:13:05 +0100
Stefan Simkovics <ssi@Austria.EU.net>
Re: beta3 
> Hi!
> 
> As you can see, the address is still valid :-)
> Could you please let me know exactly , who is looking for what?
> 
> Then I can send you a LaTeX/PDF/PS Version of my thesis.

I think we wanted the Latex source.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 13:14:56 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
Hi!

Could you please let me know the reason you're looking for it?
(I'm just curious :-))

best regards

   Stefan


> > Hi!
> > 
> > As you can see, the address is still valid :-)
> > Could you please let me know exactly , who is looking for what?
> > 
> > Then I can send you a LaTeX/PDF/PS Version of my thesis.
> 
> I think we wanted the Latex source.
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
> 

-- 
=====================================================================
DI Stefan Simkovics
KPNQwest Austria GmbH
A-1150 Wien, Diefenbachgasse 35
Tel.: +43 (1) 899 33 - 162, Fax: +43 (1) 899 33 - 10 162
Mob.: +43 (0) 664 818 91 41, e-Mail: stefan.simkovics@kpnqwest.com
http://www.kpnqwest.at, http://www.kpnqwest.com



Wed, 21 Nov 2001 19:18:35 +0100
Stefan Simkovics <ssi@Austria.EU.net>
Re: beta3 
> Hi!
> 
> Could you please let me know the reason you're looking for it?
> (I'm just curious :-))

That's an excellent question.  We moved the Postscript file out of the
main CVS into the web page CVS because it wasn't really source code. 

Of course, if we had the LaTeX, it would be documentation source code. 
However, we have no intention of modifying it, so am not sure what value
the LaTeX would be to us.

Guys, why did we want the source, exactly?  I think just putting the
Postscript on the web site is enough.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 13:21:36 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Guys, why did we want the source, exactly?

I just said that source for the document would be appropriate to keep
in CVS whereas the PS output was not.  I'm not sure that we really want
the document, though, since we're not using it as actual documentation
(ie, it's not being updated to reflect changes in the system).

			regards, tom lane

Wed, 21 Nov 2001 13:36:15 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta3 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Guys, why did we want the source, exactly?
> 
> I just said that source for the document would be appropriate to keep
> in CVS whereas the PS output was not.  I'm not sure that we really want
> the document, though, since we're not using it as actual documentation
> (ie, it's not being updated to reflect changes in the system).

Yea, that was my reading too.  I will convert the PS to PDF and add it
to the web site.  I think that's the way to go.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 13:47:25 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
On Wed, Nov 21, 2001 at 01:36:15PM -0500, Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Guys, why did we want the source, exactly?
> 
> I just said that source for the document would be appropriate to keep
> in CVS whereas the PS output was not.  I'm not sure that we really want
> the document, though, since we're not using it as actual documentation
> (ie, it's not being updated to reflect changes in the system).

Perhaps because it _can't_ be updated, without the source? A number of
people have commented on how it's a good write up of internals of query
processing. Unfortunately, it's circa v. 6.3, and starting to show it's
age. If the source were available, pulling out sections and updating them
is at least a possibility. Otherwise, it's just a historical document:
interesting, but fading.

Stefan, could you see your way to putting it under an Open Content
license of some sort, so it could be used in this way?

Ross


Wed, 21 Nov 2001 13:29:10 -0600
"Ross J. Reedstrom" <reedstrm@rice.edu>
Re: beta3
OK, here is where I think we are on beta3 packaging.

Last night, I fixed the error Marc was seeing in the build.  I have also
put my HTML version of the docs at:

	ftp://candle.pha.pa.us/pub/postgresql/postgres-docs.tar.gz.

If that is copied to doc/src and mk-beta is run, it should work fine.

Also, can the beta directory be removed from the snapshot tar file?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 15:22:31 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
On Wed, 21 Nov 2001, Bruce Momjian wrote:

> OK, here is where I think we are on beta3 packaging.
>
> Last night, I fixed the error Marc was seeing in the build.  I have also
> put my HTML version of the docs at:
>
> 	ftp://candle.pha.pa.us/pub/postgresql/postgres-docs.tar.gz.
>
> If that is copied to doc/src and mk-beta is run, it should work fine.

Okay, we have a ~ftp/pub/doc/current, but no src ... what is the
difference between current and src?

Also, its reporting that internals.ps is missing ...


Wed, 21 Nov 2001 16:14:01 -0500 (EST)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta3
> On Wed, 21 Nov 2001, Bruce Momjian wrote:
> 
> > OK, here is where I think we are on beta3 packaging.
> >
> > Last night, I fixed the error Marc was seeing in the build.  I have also
> > put my HTML version of the docs at:
> >
> > 	ftp://candle.pha.pa.us/pub/postgresql/postgres-docs.tar.gz.
> >
> > If that is copied to doc/src and mk-beta is run, it should work fine.
> 
> Okay, we have a ~ftp/pub/doc/current, but no src ... what is the
> difference between current and src?

I meant copy into pgsql/doc/src or wherever directory tree you are using
to make the beta.  Does that make sense?

> Also, its reporting that internals.ps is missing ...

OK, sorry, fixed.  Please rerun configure to get the new code.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 16:18:20 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
"Marc G. Fournier" <scrappy@hub.org> writes:
> Also, its reporting that internals.ps is missing ...

Looks like Bruce forgot to remove the reference in the toplevel
GNUmakefile.in.

			regards, tom lane

Wed, 21 Nov 2001 16:19:37 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta3 
> "Marc G. Fournier" <scrappy@hub.org> writes:
> > Also, its reporting that internals.ps is missing ...
> 
> Looks like Bruce forgot to remove the reference in the toplevel
> GNUmakefile.in.

Yea, I removed it only from GNUmakefile the first time.  Got it now.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 16:25:47 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
> 
> Okay, how do we want to address/fix/test this?
> 
> On Sun, 18 Nov 2001, Peter Eisentraut wrote:
> 
> > Tom Lane writes:
> >
> > > The packaging problems need to be solved before we can build a release
> > > candidate --- but I don't see why they should hold up beta3.  beta2 has
> > > the same problems, so beta3 wouldn't be a regression.
> >
> > Considering the history of how-do-we-get-the-docs-in-the-tarball problem,
> > I'd like to see at least one beta where this works before shipping a
> > release candidate.  Otherwise I'll have no confidence at all that we're
> > not going to ship the final release with last year's docs and no man
> > pages.

The best we can do at this point is to package my HTML files for beta3
and see if we can test this for RC1.  At least the tar.gz file itself
will be added by the script, just not built on the same machine.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 16:39:48 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
Bruce Momjian writes:

> > The problem is this line in pgsql/GNUmakefile.in:
> >
> >   docs_files := doc/postgres.tar.gz doc/src doc/TODO.detail doc/internals.ps
> >
> > Why is doc/postgres.tar.gz used.  It normally is in doc/src, and why
> > only that doc tarball.
>
> OK, I updated this to point to doc/src.

Nooooooooooooooooooooooooooooo!

-- 
Peter Eisentraut   peter_e@gmx.net


Wed, 21 Nov 2001 23:14:20 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: beta3
Bruce Momjian writes:

> OK, I found another problem:
>
>   gzip -f --best postgresql-base-7.2b3.tar
>   /usr/bin/tar cf postgresql-docs-7.2b3.tar
>   postgresql-7.2b3/doc/postgres.tar.gz postgresql-7.2b3/doc/src
>
>                    ^^^^^^^^^^^^^^^^^^^
>   postgresql-7.2b3/doc/TODO.detail postgresql-7.2b3/doc/internals.ps
>   /usr/bin/tar: can't add file postgresql-7.2b3/doc/postgres.tar.gz : No
>
>                                                 ^^^^^^^^^^^^^^^^^^^
>
>   such file or directory

Well, I sent you the list with items 1 through 5 and if you process them
in a different order then all bets are off.

>
> The problem is this line in pgsql/GNUmakefile.in:
>
>   docs_files := doc/postgres.tar.gz doc/src doc/TODO.detail doc/internals.ps
>
> Why is doc/postgres.tar.gz used.  It normally is in doc/src, and why
> only that doc tarball.

No, this is correct.

>
> And again, there is that intenrals.ps file that doesn't really belong
> there, I think.
>
>

-- 
Peter Eisentraut   peter_e@gmx.net


Wed, 21 Nov 2001 23:14:35 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: beta3
Peter Eisentraut <peter_e@gmx.net> writes:
>> OK, I updated this to point to doc/src.

> Nooooooooooooooooooooooooooooo!

If it's wrong, please fix it.

*Somebody's* got to take responsibility for making the doc build process
work again on postgresql.org.  I don't have the foggiest idea what's wrong
or where to look.  You and Marc (and Thomas, except he's gone for awhile)
are the only candidate fixers in sight.

			regards, tom lane

Wed, 21 Nov 2001 17:22:28 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta3 
> Bruce Momjian writes:
> 
> > > The problem is this line in pgsql/GNUmakefile.in:
> > >
> > >   docs_files := doc/postgres.tar.gz doc/src doc/TODO.detail doc/internals.ps
> > >
> > > Why is doc/postgres.tar.gz used.  It normally is in doc/src, and why
> > > only that doc tarball.
> >
> > OK, I updated this to point to doc/src.
> 
> Nooooooooooooooooooooooooooooo!

OK, I put it back.  I don't know how this is supposed to work.  Marc, if
you put the postgresql.tar.gz in /doc instead of doc/src as I said
before, it should work.

Again, I am just trying to jump start this thing so we can get on to beta3.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 18:20:26 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
On Wed, 21 Nov 2001, Bruce Momjian wrote:

> >
> > Okay, how do we want to address/fix/test this?
> >
> > On Sun, 18 Nov 2001, Peter Eisentraut wrote:
> >
> > > Tom Lane writes:
> > >
> > > > The packaging problems need to be solved before we can build a release
> > > > candidate --- but I don't see why they should hold up beta3.  beta2 has
> > > > the same problems, so beta3 wouldn't be a regression.
> > >
> > > Considering the history of how-do-we-get-the-docs-in-the-tarball problem,
> > > I'd like to see at least one beta where this works before shipping a
> > > release candidate.  Otherwise I'll have no confidence at all that we're
> > > not going to ship the final release with last year's docs and no man
> > > pages.
>
> The best we can do at this point is to package my HTML files for beta3
> and see if we can test this for RC1.  At least the tar.gz file itself
> will be added by the script, just not built on the same machine.

Is there a reason why it can't be built on the same machine, by the script
that builds the package itself?  Is there something remaining missing on
the server that prevents this from  happening?


Wed, 21 Nov 2001 22:18:20 -0500 (EST)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta3
On Wed, 21 Nov 2001, Tom Lane wrote:

> Peter Eisentraut <peter_e@gmx.net> writes:
> >> OK, I updated this to point to doc/src.
>
> > Nooooooooooooooooooooooooooooo!
>
> If it's wrong, please fix it.
>
> *Somebody's* got to take responsibility for making the doc build process
> work again on postgresql.org.  I don't have the foggiest idea what's wrong
> or where to look.  You and Marc (and Thomas, except he's gone for awhile)
> are the only candidate fixers in sight.

and, right now, I don't know what is "broken" that needs to be fixed ...
Peter, my fingers are at y our disposal, what haven't I installed yet? :(


Wed, 21 Nov 2001 22:19:42 -0500 (EST)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta3 
> > The best we can do at this point is to package my HTML files for beta3
> > and see if we can test this for RC1.  At least the tar.gz file itself
> > will be added by the script, just not built on the same machine.
> 
> Is there a reason why it can't be built on the same machine, by the script
> that builds the package itself?  Is there something remaining missing on
> the server that prevents this from  happening?

Uh, I don't know.  It needs the SGML tools stuff.  Is that working?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 22:34:38 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3

Peter and I are discussing this off list and will let y'all know once we
get it working again ... once we do, beta3 will come out ...

On Wed, 21 Nov 2001, Bruce Momjian wrote:

> > > The best we can do at this point is to package my HTML files for beta3
> > > and see if we can test this for RC1.  At least the tar.gz file itself
> > > will be added by the script, just not built on the same machine.
> >
> > Is there a reason why it can't be built on the same machine, by the script
> > that builds the package itself?  Is there something remaining missing on
> > the server that prevents this from  happening?
>
> Uh, I don't know.  It needs the SGML tools stuff.  Is that working?
>
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
>


Wed, 21 Nov 2001 22:49:30 -0500 (EST)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta3

Is he awake?  It is Thu Nov 22 03:50:21 GMT 2001.

---------------------------------------------------------------------------

> 
> Peter and I are discussing this off list and will let y'all know once we
> get it working again ... once we do, beta3 will come out ...
> 
> On Wed, 21 Nov 2001, Bruce Momjian wrote:
> 
> > > > The best we can do at this point is to package my HTML files for beta3
> > > > and see if we can test this for RC1.  At least the tar.gz file itself
> > > > will be added by the script, just not built on the same machine.
> > >
> > > Is there a reason why it can't be built on the same machine, by the script
> > > that builds the package itself?  Is there something remaining missing on
> > > the server that prevents this from  happening?
> >
> > Uh, I don't know.  It needs the SGML tools stuff.  Is that working?
> >
> > --
> >   Bruce Momjian                        |  http://candle.pha.pa.us
> >   pgman@candle.pha.pa.us               |  (610) 853-3000
> >   +  If your life is a hard drive,     |  830 Blythe Avenue
> >   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
> >
> 
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 21 Nov 2001 22:50:44 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta3
Tom Lane wrote:
> <editorial>
> The reason bzip is still an also-ran is that it's not enough better
> than gzip to have persuaded people to switch over.  My bet is that
> bzip will always be an also-ran, and that gzip will remain the de
> facto standard until something comes along that's really significantly
> better, like a factor of 2 better.  I've watched this sort of game
> play out before, and I know you don't take over the world with a 20%
> improvement over the existing standard.  At least not without other
> compelling reasons, like speed (oops) or patent freedom (no win there
> either).
> </editorial>

While agree in principle with your view on bzip2, I think there is a strong
reason why you should use it, 20%

That 20% is quite valuable. Just by switching to bzip2, the hosting companies
can deliver 20% more downloads with the same equipment and bandwidth cost. The
people with slow connections can get it 20% faster.

Will bzip2 become the standard? Probably not in general use, but for
downloadable tarballs it is rapidly becoming the standard. Those who pay for
bandwidth (server or client) welcome any improvement possible. 

I would switch the argument around, time how long it takes to do:

ncftpget postgresql-xxxx.tar.gz
tar xpzvf postgresql-xxxx.tar.gz
cd postgresql-xxxx
./configure --option
make
make install

vs

ncftpget postgresql-xxxx.tar.bz2
bunzip2 -c postgresql-xxxx.tar.bz2 | tar xpzv
cd postgresql-xxxx
./configure --option
make
make install


The total time involved is almost identical, plus or minus a few seconds, but
on slow connections: users get postgresql faster and have to tie up a phone
line for a shorter amount of time.

The hosting company can serve 20% less bandwidth.
The archive takes up 20% less space on the mirrors.
The archine takes up 20% less space on server backups.

I don't think there is a compelling reason to keep a gzipped version.

Fri, 23 Nov 2001 11:10:33 -0500
mlw <markw@mohawksoft.com>
Re: beta3
On Vie 23 Nov 2001 13:10, mlw wrote:
>
> While agree in principle with your view on bzip2, I think there is a strong
> reason why you should use it, 20%
>
> That 20% is quite valuable. Just by switching to bzip2, the hosting
> companies can deliver 20% more downloads with the same equipment and
> bandwidth cost. The people with slow connections can get it 20% faster.
>
> Will bzip2 become the standard? Probably not in general use, but for
> downloadable tarballs it is rapidly becoming the standard. Those who pay
> for bandwidth (server or client) welcome any improvement possible.
>
> I would switch the argument around, time how long it takes to do:
>
> ncftpget postgresql-xxxx.tar.gz
> tar xpzvf postgresql-xxxx.tar.gz
> cd postgresql-xxxx
> ./configure --option
> make
> make install
>
> vs
>
> ncftpget postgresql-xxxx.tar.bz2
> bunzip2 -c postgresql-xxxx.tar.bz2 | tar xpzv

New versions of tar would do this:

tar xpjvf postgresql-xxxx.tar.gz

It comes with bzip2 support.

Saludos... :-)

-- 
Porquï¿½ usar una base de datos relacional cualquiera,
si podï¿½s usar PostgreSQL?
-----------------------------------------------------------------
Martï¿½n Marquï¿½s                  |        mmarques@unl.edu.ar
Programador, Administrador, DBA |       Centro de Telematica
                       Universidad Nacional
                            del Litoral
-----------------------------------------------------------------

Fri, 23 Nov 2001 16:06:54 -0300
=?iso-8859-1?q?Mart=EDn=20Marqu=E9s?= <martin@bugs.unl.edu.ar>
Re: beta3
...
> Perhaps because it _can't_ be updated, without the source? A number of
> people have commented on how it's a good write up of internals of query
> processing. Unfortunately, it's circa v. 6.3, and starting to show it's
> age. If the source were available, pulling out sections and updating them
> is at least a possibility. Otherwise, it's just a historical document:
> interesting, but fading.
> Stefan, could you see your way to putting it under an Open Content
> license of some sort, so it could be used in this way?

Stefan already authorized us to use it anyway that we see fit, but we
*do* need to preserve the attribution for himself and his academic
advisor and institution.

Some/much of the source is commented out in cvs source code (latex
markup and all) in the repository afaicr (don't have my laptop fired up
at the moment) because I was worried about losing it otherwise. Since
then, I've of course lost the original sources afaict.

                 - Thomas

Mon, 26 Nov 2001 17:29:08 +0000
Thomas Lockhart <lockhart@fourpalms.org>
Re: beta3
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


I'm trying to write C functions to handle some of the number crunching
that I have been doing via backend processing.  Specifically, I want
to be able to construct a function such that a query like:

	select crunch_number(foo) from bar where [some condition];

...where `foo' is the name of some column and `bar' is some table
name.  The function needs to create some ornate data structures (i.e.,
doubly linked lists), and outputs some summary statistic.

If my data types were simpler, I could simply use an AGGREGATE function.
Unfortunately, I don't know of any way to schlep something as complex
as a doubly-linked list of arrays of arbitrary precision numbers.

I suppose ideally I'd like some way of either:

	-Being able to call a function on each row (like most user-defined
	 functions) which only returns a result on the last row;  or
	-Being able to pass the table name, column name, and selection
	 conditions to the function, and walk through the matching rows
	 inside the function, returning a single result upon completion

In terms of logical structure, this looks similar to functions to do
things like compute means or standard deviations.  The complication (as
far as I can tell) is because I can't get by with a simple accumulator
variable/transform function.

Is there any clean way to accomplish this in Postgres?  Any pointers
or suggestions would be appreciated.






- -Steve


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.3 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE8SLyyG3kIaxeRZl8RAsNNAKCy8YDnMMZCIGrMYT6pt2IxqxtCJwCgxFp2
HFlA8B9X5BJRfnMDmQSh8Ss=
=Kx46
-----END PGP SIGNATURE-----

Fri, 18 Jan 2002 16:24:33 -0800
"Stephen P. Berry" <spb@meshuggeneh.net>
Functions in C with Ornate Data Structures
"Stephen P. Berry" <spb@meshuggeneh.net> writes:
> If my data types were simpler, I could simply use an AGGREGATE function.
> Unfortunately, I don't know of any way to schlep something as complex
> as a doubly-linked list of arrays of arbitrary precision numbers.

You could, but the amount of data copying needed would be annoying.
However, there's no law that says you can't cheat.  I'd suggest that
you build this as an aggregate function whose nominal state value is
simply a pointer to data structures that are off somewhere else.

For example, assuming that you are willing to cheat to the extent of
assuming sizeof(pointer) = sizeof(integer), try something like this:

CREATE AGGREGATE crunch_number (
 basetype = float8,  -- or whatever the input column type is
 sfunc = crunch_func,
 stype = integer,
 ffunc = crunch_finish,
 initcond = 0);

where crunch_func(integer) returns integer is your data accumulation
function, and it looks like

	datstruct *ptr = (datstruct *) PG_GET_INT32(0);
	double newdataval = PG_GET_FLOAT8(1);

	if (ptr == NULL)
	{
		/* first call of query; initialize datastructures */
	}

	/* update datastructures using newdataval */

	PG_RETURN_INT32((int32) ptr);

Finally, crunch_finish(integer) returns float8 (or whatever is needed)
contains your code to compute the final result and release the working
datastructure.

Now, the important detail: you can't allocate your working
datastructures with a simple palloc(), because these functions will be
called in a short-lived memory context.  What I'd suggest is that in
your setup step, you create a private memory context that is a child
of TransactionCommandContext; then allocate all your datastructures in
that.  Then in the crunch_finish step, you needn't bother with retail
releasing of the data structures, just destroy the private context
and you're done.

			regards, tom lane

PS: this is not a novice-level question ;-).  You should be asking this
kind of stuff on pgsql-hackers, methinks.  There really isn't any other
list that discusses C coding inside the backend.

Fri, 18 Jan 2002 20:30:56 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Functions in C with Ornate Data Structures 
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


In message <11292.1011403856@sss.pgh.pa.us>, Tom Lane writes:

>You could, but the amount of data copying needed would be annoying.
>However, there's no law that says you can't cheat.  I'd suggest that
>you build this as an aggregate function whose nominal state value is
>simply a pointer to data structures that are off somewhere else.
>For example, assuming that you are willing to cheat to the extent of
>assuming sizeof(pointer) = sizeof(integer), try something like this:

I'd actually thought of doing something like this, but couldn't find
an actual explicit argument type for pointers[0], and I can't make
the assumption you describe for portability reasons (my three main
test platforms are alpha, sparc64, and x86).

I was also hoping that I could get away with not passing the problem
data structures internally at all...i.e., have a crunch_input() function
that initialises the linked list I need and populates it,
then a crunch_result() function that spits out the result.  The
accumulator is just a dummy variable, and the interesting data structure(s)
aren't in the argument list for any of the functions.  I tried
doing such a thing with an aggregate, but it didn't work---although,
interestingly, invoking the input and result functions manually
in a single session did.  I took this to mean that I really didn't
understand aggregates, so I was assuming it was a novice-level question.
I was sorta hoping this would turn out to be a standard question (although
I couldn't find any useful references in the mailing list archives or
via web searches).

>Now, the important detail: you can't allocate your working
>datastructures with a simple palloc(), because these functions will be
>called in a short-lived memory context.  What I'd suggest is that in
>your setup step, you create a private memory context that is a child
>of TransactionCommandContext; then allocate all your datastructures in
>that.  Then in the crunch_finish step, you needn't bother with retail
>releasing of the data structures, just destroy the private context
>and you're done.

Is there any way to keep `intermediate' data used by user-defined
functions around indefinitely?  I.e., have some sort of crunch_init()
function that creates a bunch of in-memory data structures, which
can then be used by subsequent (and independent) queries?  I'm
assuming not...and if I want to do that sort of thing I should populate
a temporary table with the data from these `intermediate' results.
Or keep all this fancy stuff in standalone applications rather than
in user-defined functions.

It seems like the general class of thing I'm trying to accomplish
isn't that esoteric.  Imagine trying to write a function to compute
the standard deviation of arbitrary precision numbers using the GMP
library or some such.  Note that I'm not saying that that's what I'm
trying to do...I'm just offering it as a simple sample problem in
which one can't pass everything as an argument in an aggregate.  How
does one set about doing such a thing in Postgres?







- -Steve

- -----
0	I was hoping that there would be, since the macro widgetry in
	the version 1 function semantics clearly includes the concept
	of pointers as a distinct type.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.3 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE8SN5tG3kIaxeRZl8RAmPmAJ4ilTeyoC//MRG5JHf7AmNuR7oW/QCdHHqw
RoE/GplKts1rxNO85ADEebk=
=Oedz
-----END PGP SIGNATURE-----

Fri, 18 Jan 2002 19:01:10 -0800
"Stephen P. Berry" <spb@meshuggeneh.net>
Re: Functions in C with Ornate Data Structures 
"Stephen P. Berry" <spb@meshuggeneh.net> writes:
>> For example, assuming that you are willing to cheat to the extent of
>> assuming sizeof(pointer) = sizeof(integer), try something like this:

> I'd actually thought of doing something like this, but couldn't find
> an actual explicit argument type for pointers[0], and I can't make
> the assumption you describe for portability reasons (my three main
> test platforms are alpha, sparc64, and x86).

Fair enough.  I had actually thought better of that shortly after writing,
so here's how I'd really do it:

Still make the declaration of the state datatype be "integer" at the SQL
level, and say initcond = 0.  (If you don't do this, you have to fight
nodeAgg.c's ideas about what to do with a pass-by-reference datatype,
and it ain't worth the trouble.)  But in the C code, write acquisition
and return of the state value as

	datstruct *ptr = (datstruct *) PG_GETARG_POINTER(0);

	...

	PG_RETURN_POINTER(ptr);

This relies on the fact that what you are *really* passing and returning
is not an int but a Datum, and Datum is by definition large enough for
pointers.  The only part of the above that's even slightly dubious is
the assumption that a Datum created from an int32 zero will read as a
pointer NULL --- but I am not aware of any platform where a zero bit
pattern doesn't read as a pointer NULL (and lots of pieces of Postgres
would break on such a platform).  You could get around that too by
making the initial state condition be a SQL NULL instead of a zero, but
I don't see the point.  Unless you need to treat NULL input values as
something other than "ignores", you really want to declare the sfunc as
strict, and that gets in the way of using a NULL initcond.

> Is there any way to keep `intermediate' data used by user-defined
> functions around indefinitely?  I.e., have some sort of crunch_init()
> function that creates a bunch of in-memory data structures, which
> can then be used by subsequent (and independent) queries?

You can if you can figure out how to find them again.  However, the
only obvious answer to that is to use static variables, which falls
down miserably if someone tries to run two independent instances of
your aggregate in one query.  I'd suggest hewing closely to the external
behavior of standard aggregates --- ie, each one is an independent
calculation.  You can use the above techniques to build an efficient
implementation.  If you instead build something that has an API
involving state that persists across queries, I'm pretty sure you'll
regret it in the long run.

> It seems like the general class of thing I'm trying to accomplish
> isn't that esoteric.  Imagine trying to write a function to compute
> the standard deviation of arbitrary precision numbers using the GMP
> library or some such.  Note that I'm not saying that that's what I'm
> trying to do...I'm just offering it as a simple sample problem in
> which one can't pass everything as an argument in an aggregate.  How
> does one set about doing such a thing in Postgres?

I blink not an eye to say that I'd do it exactly as described above.
Stick all the intermediate state into a data structure that's referenced
by a single master pointer, and pass the pointer as the "state value"
of the aggregate.

BTW, mlw posted some contrib code on pghackers just a day or two back
that does something similar to this.  He did some details differently
than I would've, notably this INT32-vs-POINTER business; but it's a
working example.

			regards, tom lane

Fri, 18 Jan 2002 22:28:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Functions in C with Ornate Data Structures 
Tom Lane wrote:
> 
> "Stephen P. Berry" <spb@meshuggeneh.net> writes:
> > Is there any way to keep `intermediate' data used by user-defined
> > functions around indefinitely?  I.e., have some sort of crunch_init()
> > function that creates a bunch of in-memory data structures, which
> > can then be used by subsequent (and independent) queries?

I have had to deal with this problem. I implemented a small version of Oracle's
"contains()" API call. The API is something like this:

select score(1), score(2) from table where contains(cola, 'bla bla bal', 1) >0
and contains(colb, 'fubar', 2) > 1;

On the first call I parse the search string and store it in a hash table based
on the number passed to both contains() and score(). The number passed is an
arbitrary bookmark which separates the various result sets for a single query.

The hash table is static data allocated with malloc(), although I have been
thinking I should use MemoryContextAlloc with the right context, but malloc
seems to work.

On subsequent queries, if the bookmark numer is is found, but the string for
the contains function differes, then I delete the old entry and reparse and
store the new one.

> 
> > It seems like the general class of thing I'm trying to accomplish
> > isn't that esoteric.  Imagine trying to write a function to compute
> > the standard deviation of arbitrary precision numbers using the GMP
> > library or some such.  Note that I'm not saying that that's what I'm
> > trying to do...I'm just offering it as a simple sample problem in
> > which one can't pass everything as an argument in an aggregate.  How
> > does one set about doing such a thing in Postgres?
> 
> I blink not an eye to say that I'd do it exactly as described above.
> Stick all the intermediate state into a data structure that's referenced
> by a single master pointer, and pass the pointer as the "state value"
> of the aggregate.
> 
> BTW, mlw posted some contrib code on pghackers just a day or two back
> that does something similar to this.  He did some details differently
> than I would've, notably this INT32-vs-POINTER business; but it's a
> working example.

The sizeof(int32) == sizeof(void *) is a problem, and I am not happy with it,
although I will look into your (Tom) recommendations.

Sat, 19 Jan 2002 09:02:43 -0500
mlw <markw@mohawksoft.com>
Re: [NOVICE] Functions in C with Ornate Data Structures
I'd like to implement *something* to help us collect information on what
platforms actually have what features. This would be useful, for
example, for figuring out whether any platforms are lacking 8 byte
integers or are missing timezone infrastructure.

I was thinking about something like "make report" which would mail the
results of ./configure to, say, the ports mailing list. We could mention
it in the text message printed at the end of the make cycle.

Comments? Suggestions?

                        - Thomas

Mon, 22 Apr 2002 22:20:15 -0700
Thomas Lockhart <lockhart@fourpalms.org>
"make report"
> I'd like to implement *something* to help us collect information on what
> platforms actually have what features. This would be useful, for
> example, for figuring out whether any platforms are lacking 8 byte
> integers or are missing timezone infrastructure.
> 
> I was thinking about something like "make report" which would mail the
> results of ./configure to, say, the ports mailing list. We could mention
> it in the text message printed at the end of the make cycle.
> 
> Comments? Suggestions?

Suggestion:  Why not embed this information into the binary, and provide some 
way of extracting it.

(There's a Linux kernel option that allows something similar, so it wouldn't 
be something unprecedented.)

If all the config information is embedded in the binary, automatically, at 
compile time, then this allows the ability to be _certain_ that:

- "Oh, that was compiled with a really stupid set of compiler options; you'll 
have to recompile!"

- "That was compiled without support for FOO, but with support for BAR."

- "Announcement, people:  Look out for whether or not your distribution 
compiled PostgreSQL with proper support for 64 bit integers.  Several 
distributions got this wrong with the 7.4.17 release, and you can see if it's 
OK by looking for LONG_LONG_REVISED in the embedded configuration information."

[Downside:  "Announcement, script kiddies:  If you find option 
UPDATE_DESCR_TABS=1 in the configuration information, then there's a very easy 
root exploit..."]
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://www3.sympatico.ca/cbbrowne/x.html
Rules of  the Evil  Overlord #176.  "I will add  indelible dye  to the
moat. It won't  stop anyone from swimming across,  but even dim-witted
guards should be  able to figure out when someone  has entered in this
fashion." <http://www.eviloverlord.com/>

-- 
(concatenate 'string "cbbrowne" "@acm.org")
http://www3.sympatico.ca/cbbrowne/spiritual.html
Including a destination in the CC list that will cause the recipients'
mailer to blow out is a good way to stifle dissent.
-- from the Symbolics Guidelines for Sending Mail
Tue, 23 Apr 2002 01:39:25 -0400
cbbrowne@cbbrowne.com
Re: "make report" 
On Tue, 23 Apr 2002 cbbrowne@cbbrowne.com wrote:

> Suggestion: Why not embed this information into the binary, and
> provide some way of extracting it.

I like this!

> [Downside: "Announcement, script kiddies: If you find option
> UPDATE_DESCR_TABS=1 in the configuration information, then there's a
> very easy root exploit..."]

That's not a downside at all. If an exploit exists, you need only
try it, and it works or it doesn't.

In fact, it's an upside becuase it allows someone who doesn't have
exploit code more easily to determine whether or not he might be
vulnerable.

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC


Tue, 23 Apr 2002 17:18:14 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: "make report" 
Thomas Lockhart <lockhart@fourpalms.org> writes:
> I was thinking about something like "make report" which would mail the
> results of ./configure to, say, the ports mailing list. We could mention
> it in the text message printed at the end of the make cycle.

I think it'd be a bad idea to encourage people to send mail to the ports
list.  For one thing, a pile of nonsubscriber mail would make more work
for Marc, who has quite enough already.  If you want to do this, set up
a dedicated mail alias to receive such reports.

(Possibly my thoughts on soliciting mass email are a tad colored by the
amount of virus traffic I've been seeing lately :-( ... if the Klez
epidemic gets any worse I will be forced to shut down jpeg-info, which
is currently seeing upwards of 1000 virus mails/day...)

I do like Joe's idea of embedding a complete configuration report
right into the backend executable, where it can be retrieved long
after config.log has been thrown away.

			regards, tom lane

Tue, 23 Apr 2002 12:02:04 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: "make report" 
Thomas Lockhart writes:

> I'd like to implement *something* to help us collect information on what
> platforms actually have what features. This would be useful, for
> example, for figuring out whether any platforms are lacking 8 byte
> integers or are missing timezone infrastructure.

I don't think that's very useful.  Most configuration checks are there
because some platform needed them at one point.  (Some are not -- that's a
different story.)  Those platforms do not go away, and receiving thousands
of reports about "I have feature X" won't prove that there are no systems
without feature X.

If you want to collect information about what features are portable you
can check other software packages, product manuals, ports trees, etc.
Most issues are documented someplace.

Btw., yes, 8 byte integers are missing on some platforms.

-- 
Peter Eisentraut   peter_e@gmx.net


Tue, 23 Apr 2002 19:30:06 -0400 (EDT)
Peter Eisentraut <peter_e@gmx.net>
Re: "make report"
> If you want to collect information about what features are portable you
> can check other software packages, product manuals, ports trees, etc.
> Most issues are documented someplace.

Oh goodness. Thanks for offering me a new hobby ;)

> Btw., yes, 8 byte integers are missing on some platforms.

Right. The two areas which come to mind are integer availability and the
timezone support (as you might know we support *three* different time
zone models). At the moment, none of the developers know the features
supported on the platforms we claim to support. Which platforms do not
have int8 support still? Which do not have time zone interfaces fitting
into the two "zonefull" styles? I'd like to know, but istm that the
people *with* the platforms could do this much more easily than those
without. What am I missing here??

                      - Thomas

Wed, 24 Apr 2002 06:55:20 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: "make report"
Thomas Lockhart writes:

> Right. The two areas which come to mind are integer availability and the
> timezone support (as you might know we support *three* different time
> zone models). At the moment, none of the developers know the features
> supported on the platforms we claim to support. Which platforms do not
> have int8 support still?

"Still" is the wrong word.  There used to be platforms with certain areas
of trouble, and those platforms don't go away.

But since you asked:  QNX 4 and SCO OpenServer are known to lack 8 byte
integers.

> Which do not have time zone interfaces fitting
> into the two "zonefull" styles? I'd like to know, but istm that the
> people *with* the platforms could do this much more easily than those
> without. What am I missing here??

I don't think polling users this way will yield reliable results.  If you
really want to find out, break something and see if someone complains.

-- 
Peter Eisentraut   peter_e@gmx.net


Wed, 24 Apr 2002 14:14:29 -0400 (EDT)
Peter Eisentraut <peter_e@gmx.net>
Re: "make report"
It depends. QNX4 may be used with GCC, in which case it does have long long.
I am not sure if that combination will play along with Postgres, but it
should not be assumed impossible.

----- Original Message -----
From: "Peter Eisentraut" <peter_e@gmx.net>
To: "Thomas Lockhart" <lockhart@fourpalms.org>
Cc: "PostgreSQL Hackers" <pgsql-hackers@postgresql.org>
Sent: Wednesday, April 24, 2002 1:14 PM
Subject: Re: [HACKERS] "make report"


> Thomas Lockhart writes:
>
> > Right. The two areas which come to mind are integer availability and the
> > timezone support (as you might know we support *three* different time
> > zone models). At the moment, none of the developers know the features
> > supported on the platforms we claim to support. Which platforms do not
> > have int8 support still?
>
> "Still" is the wrong word.  There used to be platforms with certain areas
> of trouble, and those platforms don't go away.
>
> But since you asked:  QNX 4 and SCO OpenServer are known to lack 8 byte
> integers.
>
> > Which do not have time zone interfaces fitting
> > into the two "zonefull" styles? I'd like to know, but istm that the
> > people *with* the platforms could do this much more easily than those
> > without. What am I missing here??
>
> I don't think polling users this way will yield reliable results.  If you
> really want to find out, break something and see if someone complains.
>
> --
> Peter Eisentraut   peter_e@gmx.net
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
>


Wed, 24 Apr 2002 13:19:22 -0500
"Igor Kovalenko" <Igor.Kovalenko@motorola.com>
Re: "make report"
Igor Kovalenko writes:

> It depends. QNX4 may be used with GCC, in which case it does have long long.
> I am not sure if that combination will play along with Postgres, but it
> should not be assumed impossible.

The point is, it should not be assumed possible.

-- 
Peter Eisentraut   peter_e@gmx.net


Wed, 24 Apr 2002 14:34:00 -0400 (EDT)
Peter Eisentraut <peter_e@gmx.net>
Re: "make report"
Just out of curiosity, does PostgreSQL have a mission statement?

If so, where could I find it?

If not, does anyone see a need?

(No, I am not some rabid MBA, but it may be useful to have for those rabid MBAs
with whom I must deal.)

Wed, 01 May 2002 14:24:30 -0400
mlw <markw@mohawksoft.com>
PostgreSQL mission statement?
Dear Team,

I have been monitoring this list for quite some time now and have been studying PostGreSQL for a while.  I also did some internet research on the subject of "multi valued" database theory.  I know that this is the basis for the "Pick" database system, FileMaker Pro, "D3", and a few other database systems.  After carefully reviewing the theoretical arguments in favor of this type of database, I am thoroughly convinced that there are certain advantages to it that will never be matched by a traditional "relational database".

I won't waste your time in reviewing the technical advantages here, because you can do your own research.  However, I will say that it is obvious to me that an mV database will be an integral part of any truly practical AI robotics system.  It will probably be necessary to "marry" the technologies of both relational databases and mV databases in such a system.

IMHO, this is something that you, as the leaders in the most advanced database system ever developed, should carefully consider.  The Linux community needs to be aware of the special advantages that an mV database offers, the way to interface an mV system with a traditional RDBMS, and the potential application theory as it relates to AI systems.

We, as a community of leaders in GPL'd software need to make sure that this technology is part of the "knowledge base" of our community.  Thanks for listening.

Arthur







Dear Team,
 
I have been monitoring this list for quite some 
time now and have been studying PostGreSQL for a while.  I also did some 
internet research on the subject of "multi valued" database theory.  I know 
that this is the basis for the "Pick" database system, FileMaker Pro, "D3", and 
a few other database systems.  After carefully reviewing the theoretical 
arguments in favor of this type of database, I am thoroughly convinced that 
there are certain advantages to it that will never be matched by a traditional 
"relational database".
 
I won't waste your time in reviewing the technical 
advantages here, because you can do your own research.  However, I will say 
that it is obvious to me that an mV database will be an integral part of any 
truly practical AI robotics system.  It will probably be necessary to 
"marry" the technologies of both relational databases and mV databases in such a 
system.
 
IMHO, this is something that you, as the leaders in 
the most advanced database system ever developed, should carefully 
consider.  The Linux community needs to be aware of the special advantages 
that an mV database offers, the way to interface an mV system with a traditional 
RDBMS, and the potential application theory as it relates to AI 
systems.
 
We, as a community of leaders in GPL'd software 
need to make sure that this technology is part of the "knowledge base" of our 
community.  Thanks for listening.
 
Arthur
Wed, 1 May 2002 11:37:26 -0700
"Arthur@LinkLine.com" <arthur@linkline.com>
mV database tools
On Wed, 2002-05-01 at 19:37, Arthur@LinkLine.com wrote:

> I also did some internet research on the subject of "multi valued"
database theory.  I know that this is the basis for the "Pick" database
system

For those who aren't familiar with PICK, it is an untyped database
(apart from weak types provided by a separate dictionary - advisory but
not enforced).  All records must have a single key, by which the record
is retrieved. Database records are divided into "attributes" by
CHAR(254), fields are subdivided into "values" by CHAR(253) and values
can be further sub-divided into "subvalues" by CHAR(252).  When records
are listed, second and subsequent values are presented on separate lines
within their columns. 

In a PICK application, it would be common to have a set up such as:

CUSTOMERS file:
  key=id
  record=name|address1^address2^...|...|ordernumber1^ordernumber2^...
(using | for CHAR(254) and ^ for CHAR(253))

where the whole address is in one field, with each address line in a
separate value, and there is another field listing the order numbers
(record keys in CUST_ORDERS) of all outstanding orders, again as
separate values.  

Then you could use the following commands (this is not SQL, of course):

 SELECT CUSTOMERS WITH ID = "C23" SAVING ORDNOS
 SORT CUST_ORDERS

to list all the outstanding orders for custoemr C23.

The advantages of Pick are that it is very easy to program; the
corresponding disadvantage is that it is a very undisciplined
environment.  It is necessary for the programmer to remember to update
that list of order keys whenever an order is created or deleted. (Some
recent implementations now support triggers, I think.)

I suppose arrays are PostgreSQL's equivalent of multi-valued data (is it
possible to have arrays of arrays?)  So it could be argued that
PostgreSQL already provides part of what Arthur wants.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C

     "For if ye forgive men their trespasses, your heavenly 
      Father will also forgive you; But if ye forgive not 
      men their trespasses, neither will your Father forgive
      your trespasses."         Matthew 6:14,15
01 May 2002 21:46:21 +0100
Oliver Elphick <olly@lfix.co.uk>
Re: mV database tools
mlw <markw@mohawksoft.com> writes:
> Just out of curiosity, does PostgreSQL have a mission statement?

Nope.  Given the wide variety of views among the developer community,
I think we'd have a tough time agreeing on a mission statement, unless
it was so generic as to be meaningless ...

			regards, tom lane

Wed, 01 May 2002 19:05:16 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: PostgreSQL mission statement? 
> I suppose arrays are PostgreSQL's equivalent of multi-valued data (is it
> possible to have arrays of arrays?)  So it could be argued that
> PostgreSQL already provides part of what Arthur wants.

It seems to me that there would be a whopping lot of value to the exercise of 
figuring out some way of "layering" MVD on top of a relational database, even 
if only to provide something sufficiently analytical to cope with the 
perpetual claims of:

  "MultiValued Databases are Vastly, Spectacularly, the Bestest Kind of
   Database ever imagined in the universe!  No, really they are!"

It might not be necessary to go all the way to fully layering such a thing atop PostgreSQL, although it would be a nice riposte to be able to respond with:

  "Been there, done that.  Of _COURSE_ PostgreSQL supports MultiValue."
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://www.cbbrowne.com/info/lisp.html
Including a destination in the CC list that will cause the recipients'
mailer to blow out is a good way to stifle dissent.
-- from the Symbolics Guidelines for Sending Mail

-- 
(reverse (concatenate 'string "ac.notelrac.teneerf@" "454aa"))
http://www.cbbrowne.com/info/spreadsheets.html
"Starting a project in C/C++ is a premature optimization."
-- Peter Jensen
Wed, 01 May 2002 19:08:28 -0400
cbbrowne@cbbrowne.com
Re: mV database tools 
> mlw <markw@mohawksoft.com> writes:
> > Just out of curiosity, does PostgreSQL have a mission statement?

> Nope.  Given the wide variety of views among the developer community,
> I think we'd have a tough time agreeing on a mission statement, unless
> it was so generic as to be meaningless ...

Well, I think one of the things that has been agreed on that _isn't_
that generic is:
  "We use a Berkeley style license, and prefer it that way."

:-)
--
(concatenate 'string "cbbrowne" "@ntlug.org")
http://www.ntlug.org/~cbbrowne/rdbms.html
To err is human, to moo bovine. 

Wed, 01 May 2002 19:25:12 -0400
cbbrowne@cbbrowne.com
Re: PostgreSQL mission statement? 
On Wed, May 01, 2002 at 02:24:30PM -0400, mlw wrote:
> Just out of curiosity, does PostgreSQL have a mission statement?
> 
> If so, where could I find it?
> 
> If not, does anyone see a need?

"Provide a really good database and have fun doing it"

-- 
David Terrell             | "War is peace, 
Prime Minister, Nebcorp   | freedom is slavery, 
dbt@meat.net              | ignorance is strength 
http://wwn.nebcorp.com/   | Dishes are clean." - Chris Fester

Wed, 1 May 2002 16:59:34 -0700
David Terrell <dbt@meat.net>
Re: PostgreSQL mission statement?
cbbrowne@cbbrowne.com wrote:
> > mlw <markw@mohawksoft.com> writes:
> > > Just out of curiosity, does PostgreSQL have a mission statement?
>
> > Nope.  Given the wide variety of views among the developer community,
> > I think we'd have a tough time agreeing on a mission statement, unless
> > it was so generic as to be meaningless ...
>
> Well, I think one of the things that has been agreed on that _isn't_
> that generic is:
>   "We use a Berkeley style license, and prefer it that way."

    Does that now count as a mission? Whow, didn't know that I am
    on a mission!  What in our mission is mission critical?  What
    are our defined mission goals?


Jan :-)

>
> :-)
> --
> (concatenate 'string "cbbrowne" "@ntlug.org")
> http://www.ntlug.org/~cbbrowne/rdbms.html
> To err is human, to moo bovine.
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org
>


--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Wed, 1 May 2002 20:31:05 -0400 (EDT)
Jan Wieck <janwieck@yahoo.com>
Re: PostgreSQL mission statement?
cbbrowne@cbbrowne.com wrote:
> 
> > mlw <markw@mohawksoft.com> writes:
> > > Just out of curiosity, does PostgreSQL have a mission statement?
> 
> > Nope.  Given the wide variety of views among the developer community,
> > I think we'd have a tough time agreeing on a mission statement, unless
> > it was so generic as to be meaningless ...
> 
> Well, I think one of the things that has been agreed on that _isn't_
> that generic is:
>   "We use a Berkeley style license, and prefer it that way."

No! no! no! Don't even kid like that. EVERY time that debate is even mentioned,
it goes on for days.

Wed, 01 May 2002 21:05:31 -0400
mlw <markw@mohawksoft.com>
Re: PostgreSQL mission statement?
Le Jeudi 2 Mai 2002 01:59, David Terrell a écrit :
> "Provide a really good database and have fun doing it"

PostgreSQL Community is commited to providing Humanity with the best 
multi-purpose, reliable, open-source and free database system.

Thu, 2 May 2002 11:20:58 +0200
Jean-Michel POURE <jm.poure@freesurf.fr>
Re: PostgreSQL mission statement?
> Le Jeudi 2 Mai 2002 01:59, David Terrell a écrit :
> > "Provide a really good database and have fun doing it"
> 
> PostgreSQL Community is commited to providing Humanity with the best
> multi-purpose, reliable, open-source and free database system.
> 

How about "We can store your data" ?

Just a late night thought...

dali


Thu, 2 May 2002 21:42:39 +1200
"Dalibor Andzakovic" <dali@dali.net.nz>
Re: PostgreSQL mission statement?
> "Arthur@LinkLine.com" wrote:
> 
> Dear Team,
> 
> I have been monitoring this list for quite some time now and have been
> studying PostGreSQL for a while.  I also did some internet research on the
> subject of "multi valued" database theory.  I know that this is the basis for
> the "Pick" database system, FileMaker Pro, "D3", and a few other database
> systems.  After carefully reviewing the theoretical arguments in favor of
> this type of database, I am thoroughly convinced that there are certain
> advantages to it that will never be matched by a traditional "relational
> database".
> 
> I won't waste your time in reviewing the technical advantages here, because
> you can do your own research.  However, I will say that it is obvious to me
> that an mV database will be an integral part of any truly practical AI
> robotics system.  It will probably be necessary to "marry" the technologies
> of both relational databases and mV databases in such a system.

The idea of a multi-view database is interesting and all, but hardly a next
leap in database theory. It is nothing more than using an addressable array
within PostgreSQL, and PostgreSQL has contributed functions for just these
sorts of operations.

The syntax may be different than you would wish, but with the same result. It
seems that mu

> 
> IMHO, this is something that you, as the leaders in the most advanced
> database system ever developed, should carefully consider.  The Linux
> community needs to be aware of the special advantages that an mV database
> offers, the way to interface an mV system with a traditional RDBMS, and the
> potential application theory as it relates to AI systems.
> 
> We, as a community of leaders in GPL'd software need to make sure that this
> technology is part of the "knowledge base" of our community.  Thanks for
> listening.
> 
> Arthur

Thu, 02 May 2002 08:01:34 -0400
mlw <markw@mohawksoft.com>
Re: mV database tools
mlw wrote:
> 
> > "Arthur@LinkLine.com" wrote:
> >
> > Dear Team,
> >
> > I have been monitoring this list for quite some time now and have been
> > studying PostGreSQL for a while.  I also did some internet research on the
> > subject of "multi valued" database theory.  I know that this is the basis for
> > the "Pick" database system, FileMaker Pro, "D3", and a few other database
> > systems.  After carefully reviewing the theoretical arguments in favor of
> > this type of database, I am thoroughly convinced that there are certain
> > advantages to it that will never be matched by a traditional "relational
> > database".
> >
> > I won't waste your time in reviewing the technical advantages here, because
> > you can do your own research.  However, I will say that it is obvious to me
> > that an mV database will be an integral part of any truly practical AI
> > robotics system.  It will probably be necessary to "marry" the technologies
> > of both relational databases and mV databases in such a system.
> 
> The idea of a multi-view database is interesting and all, but hardly a next
> leap in database theory. It is nothing more than using an addressable array
> within PostgreSQL, and PostgreSQL has contributed functions for just these
> sorts of operations.
> 
> The syntax may be different than you would wish, but with the same result. It
> seems that mu
 Doh!! pressed send!!

It seems that a multivalue database can be implemented on top of PostgreSQL,
where as a full relational database can not be implemented on top of an MVDB.

Thu, 02 May 2002 08:06:28 -0400
mlw <markw@mohawksoft.com>
Re: mV database tools
Jean-Michel POURE wrote:
> 
> Le Jeudi 2 Mai 2002 01:59, David Terrell a ï¿½crit :
> > "Provide a really good database and have fun doing it"
> 
> PostgreSQL Community is commited to providing Humanity with the best
> multi-purpose, reliable, open-source and free database system.


The PostgreSQL community is committed to creating and maintaining the best,
most reliable, open-source multi-purpose standards based database, and with it,
promote free and open source software world wide.

Who's that? Anyone disagree?

Thu, 02 May 2002 08:15:15 -0400
mlw <markw@mohawksoft.com>
Re: PostgreSQL mission statement?
On Thu, May 02, 2002 at 08:15:15AM -0400, mlw wrote:
> Jean-Michel POURE wrote:
> > Le Jeudi 2 Mai 2002 01:59, David Terrell a ï¿½crit :
> > > "Provide a really good database and have fun doing it"
> > 
> > PostgreSQL Community is commited to providing Humanity with the best
> > multi-purpose, reliable, open-source and free database system.
> 
> 
> The PostgreSQL community is committed to creating and maintaining the best,
> most reliable, open-source multi-purpose standards based database, and with
> it, promote free and open source software world wide.
> 
> Who's that? Anyone disagree?

why does it have to be THE BEST ?  that is insulting to the other projects
like MySQL which while "competitors" are also a valid and useful benchmark
for features, performance and keeping the postgresql community on its
collective toes.

postgresql is not THE BEST in all applications, so calling it that is inviting
derision and pointless arguments.

i'd go with:

The PostgreSQL community is committed to creating and maintaining a very
reliable, open-source, multi-purpose, standards-based database, and
encouraging participation in open-source usage and development world wide.

-- 
[ Jim Mercer        jim@reptiles.org         +1 416 410-5633 ]
[          I want to live forever, or die trying.            ]

Thu, 2 May 2002 08:37:06 -0400
Jim Mercer <jim@reptiles.org>
Re: PostgreSQL mission statement?
Jim Mercer wrote:
> 
> On Thu, May 02, 2002 at 08:15:15AM -0400, mlw wrote:
> > Jean-Michel POURE wrote:
> > > Le Jeudi 2 Mai 2002 01:59, David Terrell a ï¿½crit :
> > > > "Provide a really good database and have fun doing it"
> > >
> > > PostgreSQL Community is commited to providing Humanity with the best
> > > multi-purpose, reliable, open-source and free database system.
> >
> >
> > The PostgreSQL community is committed to creating and maintaining the best,
> > most reliable, open-source multi-purpose standards based database, and with
> > it, promote free and open source software world wide.
> >
> > Who's that? Anyone disagree?
> 
> why does it have to be THE BEST ?  that is insulting to the other projects
> like MySQL which while "competitors" are also a valid and useful benchmark
> for features, performance and keeping the postgresql community on its
> collective toes.

This is interesting, a mission statement isn't necessarily about "what is" but
about what we "want to do," what it is that we "intend to do," i.e. "our
mission." It is vital that a mission statement contain the superlatives.
Mediocrity has no place here.

I don't know about you, but I want PostgreSQL to be the best, be THE most
reliable. Omitting "best" or "most" from the statement means that we should all
just give up now, because PostgreSQL is pretty damn good already.

Thu, 02 May 2002 08:43:04 -0400
mlw <markw@mohawksoft.com>
Re: PostgreSQL mission statement?


Is this an indication of a need for pgsql-marketing@postgresql.org? :)

I like:

We'll store your data; if we think it'll be an interesting enough
diversion for us.


--
Nigel Andrews


On Thu, 2 May 2002, mlw wrote:

> Jean-Michel POURE wrote:
> > 
> > Le Jeudi 2 Mai 2002 01:59, David Terrell a écrit :
> > > "Provide a really good database and have fun doing it"
> > 
> > PostgreSQL Community is commited to providing Humanity with the best
> > multi-purpose, reliable, open-source and free database system.
> 
> 
> The PostgreSQL community is committed to creating and maintaining the best,
> most reliable, open-source multi-purpose standards based database, and with it,
> promote free and open source software world wide.
> 
> Who's that? Anyone disagree?


Thu, 2 May 2002 13:43:39 +0100 (BST)
"Nigel J. Andrews" <nandrews@investsystems.co.uk>
Re: PostgreSQL mission statement?
On Thu, May 02, 2002 at 08:43:04AM -0400, mlw wrote:
> Jim Mercer wrote:
> > why does it have to be THE BEST ?  that is insulting to the other projects
> > like MySQL which while "competitors" are also a valid and useful benchmark
> > for features, performance and keeping the postgresql community on its
> > collective toes.
> 
> This is interesting, a mission statement isn't necessarily about "what is" but
> about what we "want to do," what it is that we "intend to do," i.e. "our
> mission." It is vital that a mission statement contain the superlatives.
> Mediocrity has no place here.
> 
> I don't know about you, but I want PostgreSQL to be the best, be THE most
> reliable. Omitting "best" or "most" from the statement means that we should
> all just give up now, because PostgreSQL is pretty damn good already.

i think a mission statement full of boastfulness is just a sound bite, and
will be dismissed as such.

if you want the mission statement to have an impact, then it needs to be
acceptable not only to those who fully embrace it, but also acceptable to
those who will respect the project from a distance.

otherwise its not a mission statement, its akin to a corporate cheer.
( i'm picturing Steve Balmer's superlative exhaltations to the converted
  http://www.ntk.net/ballmer/mirrors.html )

-- 
[ Jim Mercer        jim@reptiles.org         +1 416 410-5633 ]
[          I want to live forever, or die trying.            ]

Thu, 2 May 2002 08:56:44 -0400
Jim Mercer <jim@reptiles.org>
Re: PostgreSQL mission statement?
On Thu, May 02, 2002 at 01:43:39PM +0100, Nigel J. Andrews wrote:
> I like:
> 
> We'll store your data; if we think it'll be an interesting enough
> diversion for us.

gets my vote.

8^)

-- 
[ Jim Mercer        jim@reptiles.org         +1 416 410-5633 ]
[          I want to live forever, or die trying.            ]

Thu, 2 May 2002 08:59:19 -0400
Jim Mercer <jim@reptiles.org>
Re: PostgreSQL mission statement?
> > I don't know about you, but I want PostgreSQL to be the best, be
THE most
> > reliable. Omitting "best" or "most" from the statement means that
we should
> > all just give up now, because PostgreSQL is pretty damn good
already.
>
> i think a mission statement full of boastfulness is just a sound
bite, and
> will be dismissed as such.

Theres no reason Postgresql can't be the best in a good majority if
not all of the fields. Yeah, a few things are needed to accomplish
this -- but theres no reason it can't happen.

Anyway, most companies do something like  'Postgresql will become the
choice database'.  That is, majority market share.

That said, they're dumb.  They need to be changed once you meet the
goal.

A good mission statement should last the lifetime of the company /
department / project.  WalMart actually has one of the better ones,
where their mission is to beat last years sales by x%.  3M will
innovate.  HP would not release a product unless it offered the market
something new or better (they don't [ didn't for years anyway ] clone
others stuff and undercut them in price).

Perhaps Postgresql should have the mission of handling twice the
amount of data this year than last.   That is, all collective
installations will maintain more stuff.  Difficult to measure, but
would ensure we create / maintain the features required by big and
small databases.  Currently those features appear to be relational
design, reliability, etc.  Should the data storage requirements
change, Postgresql would have to follow in order to maintain it's
mission statement which is a good thing.

Goals should change, (being most reliable, or ANSI compliant) but
purpose should be consistent.


That said, skip the whole thing.  I don't think we need something to
rally behind as it's kinda self explanatory why you'd donate time or
money to the project.  It needs to meet your needs first, which
averaged out between all of the developers will meet those of most
others.  DBs work that way, desktops often don't ;)






Thu, 2 May 2002 09:44:53 -0400
"Rod Taylor" <rbt@zort.ca>
Re: PostgreSQL mission statement?
On Thu, 2002-05-02 at 14:37, Jim Mercer wrote:
> On Thu, May 02, 2002 at 08:15:15AM -0400, mlw wrote:
> > Who's that? Anyone disagree?
> 
> why does it have to be THE BEST ?  that is insulting to the other projects
> like MySQL which while "competitors" are also a valid and useful benchmark
> for features, performance and keeping the postgresql community on its
> collective toes.
> 
> postgresql is not THE BEST in all applications, so calling it that is inviting
> derision and pointless arguments.
> 

The Politically Correct mission statement follows:

The PostgreSQL community is committed to creating and maintaining a good 
but not the best, mostly reliable, open-source multi-purpose standards 
based database, and with it, promote free and open source software and 
other worthy causes world wide and to not hurting anyones feelings in doing so. 
We are also committed to not cheating our SOs, not charging too much for
our services nor eating too much and to recommending products of our
commercial competitors before ours in order to help them fullfil their
obligations to their stockholders. 

I was hoping to fit in something about being a good
Christian/Muslim/Atheist but was unable to do it in an universally
acceptable way. 
There may be other points that are not valid everywhere though.

BTW, I think PostgreSQL does _not_ need any mission statement.

-------------
Hannu




02 May 2002 16:07:54 +0200
Hannu Krosing <hannu@tm.ee>
Re: PostgreSQL mission statement?
The PostgreSQL community is committed to creating and maintaining the best,
most reliable, open-source multi-purpose standards based database, and with
it, promote free(dom) and open source software world wide.

I hope you don't mind writing "free(dom)" with the idea of fighting patent 
abuses.

Cheers,
Jean-Michel

Thu, 2 May 2002 16:29:34 +0200
Jean-Michel POURE <jm.poure@freesurf.fr>
Re: PostgreSQL mission statement?
On Thursday 02 May 2002 08:56 am, Jim Mercer wrote:
> i think a mission statement full of boastfulness is just a sound bite, and
> will be dismissed as such.

> if you want the mission statement to have an impact, then it needs to be
> acceptable not only to those who fully embrace it, but also acceptable to
> those who will respect the project from a distance.

> otherwise its not a mission statement, its akin to a corporate cheer.
> ( i'm picturing Steve Balmer's superlative exhaltations to the converted
>   http://www.ntk.net/ballmer/mirrors.html )

In the corporate world a mission statement is often the 'sound bite' and a 
'corporate cheer'.

I personally think
"To have fun making and improving the most extensible, robust, ACID-compliant 
Free database system on the planet"  wraps up at least why I think we're all 
here.  s/Free/Open Source/g if you'd rather not invoke a stallmanism.  Or 
even s/Free/BSD-licensed/g if you want to really state the obvious. :-)

If other projects' members are insulted by that, then they're just too 
sensitive.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Thu, 2 May 2002 10:44:45 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: PostgreSQL mission statement?
Jean-Michel POURE wrote:
> 
> The PostgreSQL community is committed to creating and maintaining the best,
> most reliable, open-source multi-purpose standards based database, and with
> it, promote free(dom) and open source software world wide.
> 
> I hope you don't mind writing "free(dom)" with the idea of fighting patent
> abuses.

No, the mission statement is about what the postgresql group, as a whole, is
all about. 

I know it seems silly to have such a thing, but really, the more I read on this
discussion, the more it seems like it is a useful "call to arms" for developers
and users alike.

Now, I do not wish to have a manifesto, but a short and sweet "this is who we
are, and this is what we do" could be a positive thing.

P.S. I think every software engineer worth anything should fight software
patents. If Donald Knuth didn't patent his algorithms, practically none of us
deserve patents. I mean seriously, most of the software patents are trivial and
obvious. Knuth did something, most of us only build on his work.

Thu, 02 May 2002 11:44:34 -0400
mlw <markw@mohawksoft.com>
Re: PostgreSQL mission statement?
On Thu, 2 May 2002, mlw wrote:

> Jim Mercer wrote:
> >
> > On Thu, May 02, 2002 at 08:15:15AM -0400, mlw wrote:
> > > Jean-Michel POURE wrote:
> > > > Le Jeudi 2 Mai 2002 01:59, David Terrell a �crit :
> > > > > "Provide a really good database and have fun doing it"
> > > >
> > > > PostgreSQL Community is commited to providing Humanity with the best
> > > > multi-purpose, reliable, open-source and free database system.
> > >
> > >
> > > The PostgreSQL community is committed to creating and maintaining the best,
> > > most reliable, open-source multi-purpose standards based database, and with
> > > it, promote free and open source software world wide.
> > >
> > > Who's that? Anyone disagree?
> >
> > why does it have to be THE BEST ?  that is insulting to the other projects
> > like MySQL which while "competitors" are also a valid and useful benchmark
> > for features, performance and keeping the postgresql community on its
> > collective toes.
>
> This is interesting, a mission statement isn't necessarily about "what is" but
> about what we "want to do," what it is that we "intend to do," i.e. "our
> mission." It is vital that a mission statement contain the superlatives.
> Mediocrity has no place here.
>
> I don't know about you, but I want PostgreSQL to be the best, be THE most
> reliable. Omitting "best" or "most" from the statement means that we should all
> just give up now, because PostgreSQL is pretty damn good already.

altho in most contexts, I would agree with Jim as to the use of 'The
Best', for any mission statement to say anything other then that,  IMHO,
shows a lack of commitment ... I agree with mlw on this one, the mission
statement is what we are *striving* for ... where we eventually want to
get to ... if we aren't "The Best", then there is someone better then us
that we have to work that much harder to become better then ...

I personally like mlw's wording ...



Thu, 2 May 2002 13:11:41 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: PostgreSQL mission statement?
On 2 May 2002, Hannu Krosing wrote:

> On Thu, 2002-05-02 at 14:37, Jim Mercer wrote:
> > On Thu, May 02, 2002 at 08:15:15AM -0400, mlw wrote:
> > > Who's that? Anyone disagree?
> >
> > why does it have to be THE BEST ?  that is insulting to the other projects
> > like MySQL which while "competitors" are also a valid and useful benchmark
> > for features, performance and keeping the postgresql community on its
> > collective toes.
> >
> > postgresql is not THE BEST in all applications, so calling it that is inviting
> > derision and pointless arguments.
> >
>
> The Politically Correct mission statement follows:
>
> The PostgreSQL community is committed to creating and maintaining a good
> but not the best, mostly reliable, open-source multi-purpose standards

Okay, so there now has to be someone always better then us, since we don't
want to be the best? *confused look*

> BTW, I think PostgreSQL does _not_ need any mission statement.

Nope, it doesn't ... never did before, don't know why it does suddenly ...
do any other open source projects have one?  Its kinda fun to see what ppl
banter around, but I can't see it being useful to adopt any single one,
considering I can't see *everyone* agreeing with it ...


Thu, 2 May 2002 13:14:34 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: PostgreSQL mission statement?
On Thu, 2 May 2002, Nigel J. Andrews wrote:

>
>
> Is this an indication of a need for pgsql-marketing@postgresql.org? :)

already exists as pgsql-advocacy :)



Thu, 2 May 2002 13:14:58 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: PostgreSQL mission statement?
"Marc G. Fournier" wrote:
 
> > BTW, I think PostgreSQL does _not_ need any mission statement.
> 
> Nope, it doesn't ... never did before, don't know why it does suddenly ...
> do any other open source projects have one?  Its kinda fun to see what ppl
> banter around, but I can't see it being useful to adopt any single one,
> considering I can't see *everyone* agreeing with it ...

We as developers do not need mission statements, per se' but it is often useful
as something to point to.

I am writing a business plan for my company and I was looking for PostgreSQL's
mission statement. Of course I did not see one. It is very interesting seeing
what people are coming up with.

IMHO, if we can come up with a strong, positive statement, it would help MBA
trained CIOs and CTOs choose PostgreSQL. To them, it will show a professional
minded development group, it will be recognizable to them.

Thu, 02 May 2002 12:25:50 -0400
mlw <markw@mohawksoft.com>
Re: PostgreSQL mission statement?

[ please try not to take this too seriously ]

On Thu, May 02, 2002 at 01:11:41PM -0300, Marc G. Fournier wrote:
> altho in most contexts, I would agree with Jim as to the use of 'The
> Best', for any mission statement to say anything other then that,  IMHO,
> shows a lack of commitment ... I agree with mlw on this one, the mission
> statement is what we are *striving* for ... where we eventually want to
> get to ... if we aren't "The Best", then there is someone better then us
> that we have to work that much harder to become better then ...

that's right, we have to work harder to STOMP OUT ALL COMPETITION!

in fact, we should stop giving free access to the source, as our competitors
might use the code to make their product better than ours.

if we aren't THE BEST, then all those stock options are worthless!

we could increase our chances of being the best by infiltrating the CVS
tree of MySQL and the other projects stealing our thunder and injecting
bugs into their code.

i mean, if we want to be THE BEST, why should we stop at mere rhetoric?

to be THE BEST, you need to dominate, good quality design and code are not
the complete recipe for being THE BEST.

THE BEST implies that no-one else compares, and we can completely demoralize
the competition, thereby eliminating the competition.

in all seriousness, i think that this attitude of being THE BEST goes against
the philosophy of Open Source.

if your source is open and available for modification/improvement/localization,
then there will always be a chance for someone to run with it and make
improvements.

what is THE BEST Unix-based system? Linux? Debian? RedHat? FreeBSD? NetBSD?
OpenBSD? Solaris? Minix? Qnix?

if any of them claimed to be THE BEST Unix, we'd all laugh.

none of them are THE BEST, but all of them strive to be as good as they
can make them, and all of them borrow from each other in order to make
their version better in the eyes of their audience.

-- 
[ Jim Mercer        jim@reptiles.org         +1 416 410-5633 ]
[          I want to live forever, or die trying.            ]

Thu, 2 May 2002 12:26:06 -0400
Jim Mercer <jim@reptiles.org>
Re: PostgreSQL mission statement?
On Wed, 1 May 2002, David Terrell wrote:

> On Wed, May 01, 2002 at 02:24:30PM -0400, mlw wrote:
> > Just out of curiosity, does PostgreSQL have a mission statement?
> > 
> > If so, where could I find it?
> > 
> > If not, does anyone see a need?
> 
> "Provide a really good database and have fun doing it"

Motto: "The best damned database money can't buy"

:-)


Thu, 2 May 2002 11:56:05 -0600 (MDT)
Scott Marlowe <scott.marlowe@ihs.com>
Re: PostgreSQL mission statement?
mlw wrote:
> "Marc G. Fournier" wrote:
>
> > > BTW, I think PostgreSQL does _not_ need any mission statement.
> >
> > Nope, it doesn't ... never did before, don't know why it does suddenly ...
> > do any other open source projects have one?  Its kinda fun to see what ppl
> > banter around, but I can't see it being useful to adopt any single one,
> > considering I can't see *everyone* agreeing with it ...
>
> We as developers do not need mission statements, per se' but it is often useful
> as something to point to.

    Yupp.  As  PostgreSQL  developers  we are simply committed to
    make commit what others need to commit to PostgreSQL.

    If we ever define something  like  a  mission  statement,  it
    should be named "The 10 commitments" anyway, no?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Thu, 2 May 2002 15:55:49 -0400 (EDT)
Jan Wieck <janwieck@yahoo.com>
Re: PostgreSQL mission statement?
On 2 May 2002, Hannu Krosing wrote:

> The Politically Correct mission statement follows:
> 
> The PostgreSQL community is committed to creating and maintaining a good 
> but not the best, mostly reliable, open-source multi-purpose standards 
> based database, and with it, promote free and open source software and 
> other worthy causes world wide and to not hurting anyones feelings in doing so. 
> We are also committed to not cheating our SOs, not charging too much for
> our services nor eating too much and to recommending products of our
> commercial competitors before ours in order to help them fullfil their
> obligations to their stockholders. 

As a practicing polyamorist, I find the part about not cheating on our SOs 
highly offensive.  :-)


Thu, 2 May 2002 14:44:45 -0600 (MDT)
Scott Marlowe <scott.marlowe@ihs.com>
Re: PostgreSQL mission statement?

> We as developers do not need mission statements, per se' but it is
often useful
> as something to point to.

It's comforting and useful to point to; in addition, developers
work on something because of personal "itches" (to coin a phrase)
 that happen to broadly overlap with the group

> IMHO, if we can come up with a strong, positive statement, it would
help MBA
> trained CIOs and CTOs choose PostgreSQL. To them, it will show a
professional
> minded development group, it will be recognizable to them.

I think this is an excellent point, especially since I'd say that one of
the
implicit goals of the PG project is for the database to be *used* ;)
- and the corporate world in some form or another represents probably
the
largest user base.

This reasoning is a bit dicey b/c playing PR games really isn't fun
after
the initial rush, and I don't think anyone really wants catering to the
corporate world to be first and foremost in their minds.

To this end, if a mission statement is adopted, it should probably be
a very dynamic document that remains capable of both engaging CIOs/CTOs
and intriguing developers as the vision and the landscape change.
A mission statement that is agonized over (and takes time away from
development),
finally adopted, and then allowed to become obsolete does PG no good.

Should you guys hold a vote to see who wants a mission statement (and
who
wants to write one or compile all the suggestions here into a nice form)
and then work from there? I'm not exactly familiar with the procedures
here.

thanks for listening to my rambling,
Michael Locasto


Thu, 2 May 2002 17:11:30 -0400
"Michael E. Locasto" <locasto2@tcnj.edu>
Re: PostgreSQL mission statement?
Scott Marlowe <scott.marlowe@ihs.com> writes:

> On Wed, 1 May 2002, David Terrell wrote:
> 
> > On Wed, May 01, 2002 at 02:24:30PM -0400, mlw wrote:
> > > Just out of curiosity, does PostgreSQL have a mission statement?
> > > 
> > > If so, where could I find it?
> > > 
> > > If not, does anyone see a need?
> > 
> > "Provide a really good database and have fun doing it"
> 
> Motto: "The best damned database money can't buy"

I don't think that any of the PostgreSQL developers would want, in any
way shape or form, to suggest that you can't pay money for PostgreSQL.
Nor are they likely to limit themselves to competing with free
(libre/gratis) databases.

Some of them might even object to the use of the word "damn" :).

Jason

02 May 2002 16:01:56 -0600
Jason Earl <jason.earl@simplot.com>
Re: PostgreSQL mission statement?
On Fri, 2002-05-03 at 04:25, mlw wrote:

> 
> IMHO, if we can come up with a strong, positive statement, it would help MBA
> trained CIOs and CTOs choose PostgreSQL. To them, it will show a professional
> minded development group, it will be recognizable to them.

I am not so sure about that -

In my experience the things that those guys use to decide on products
are :

1) reference sites
2) official support

(and they like to pay for a product 'cause they are used to doing
it...but lets not go there...)

Personally I find the lack of "business-speak" things (like mission
statements) refeshing, and I see it as part of the "values" that
differentiate open source / community based products from commercial
ones. 

just my NZ4c (=US2c)

Mark



03 May 2002 10:09:49 +1200
Mark kirkwood <markir@slingshot.co.nz>
Re: PostgreSQL mission statement?
On 2 May 2002, Jason Earl wrote:

> Scott Marlowe <scott.marlowe@ihs.com> writes:
> 
> > On Wed, 1 May 2002, David Terrell wrote:
> > 
> > > On Wed, May 01, 2002 at 02:24:30PM -0400, mlw wrote:
> > > > Just out of curiosity, does PostgreSQL have a mission statement?
> > > > 
> > > > If so, where could I find it?
> > > > 
> > > > If not, does anyone see a need?
> > > 
> > > "Provide a really good database and have fun doing it"
> > 
> > Motto: "The best damned database money can't buy"
> 
> I don't think that any of the PostgreSQL developers would want, in any
> way shape or form, to suggest that you can't pay money for PostgreSQL.
> Nor are they likely to limit themselves to competing with free
> (libre/gratis) databases.

True, but my point wasn't that you could pay for it, but that it couldn't 
be "bought" like so many other things (think politicians, OEMs, judges, 
etc...)  But I was pretty much just foolin' around. :-)

So how about:

"Postgresql:  Open Source, Open Standards, Open Development, Open Minds"


Thu, 2 May 2002 16:14:45 -0600 (MDT)
Scott Marlowe <scott.marlowe@ihs.com>
Re: PostgreSQL mission statement?
...
> Now, I do not wish to have a manifesto, but a short and sweet "this is who we
> are, and this is what we do" could be a positive thing.

"PostgreSQL is the most advanced open-source database available
anywhere"

has appeared in the docs for quite some time, and has appeared in other
mention of PostgreSQL (release announcements etc).

A short and sweet mission statement could describe what we do to make
this true, and what we must continue to do to stay on top. Or we could
leave it general; for example, "most advanced" could mean wrt standards
compliance, or wrt leading edge features, or wrt performance, or ?? Let
us decide what those specifics are as we go along...

"PostgreSQL is and will be the most advanced open-source database
available anywhere."

No different in substance from what we've said all along. Leave out the
specifics, because the next generations of developers will have specific
directions to go :)

                   - Thomas

Thu, 02 May 2002 17:22:25 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: PostgreSQL mission statement?
Mark kirkwood wrote:
> 
> On Fri, 2002-05-03 at 04:25, mlw wrote:
> 
> >
> > IMHO, if we can come up with a strong, positive statement, it would help MBA
> > trained CIOs and CTOs choose PostgreSQL. To them, it will show a professional
> > minded development group, it will be recognizable to them.
> 
> I am not so sure about that -
> 
> In my experience the things that those guys use to decide on products
> are :
> 
> 1) reference sites
> 2) official support
> 
> (and they like to pay for a product 'cause they are used to doing
> it...but lets not go there...)
> 
> Personally I find the lack of "business-speak" things (like mission
> statements) refeshing, and I see it as part of the "values" that
> differentiate open source / community based products from commercial
> ones.

A mission statement is like a tie.

Thu, 02 May 2002 20:41:30 -0400
mlw <markw@mohawksoft.com>
Re: PostgreSQL mission statement?
On Thu, May 02, 2002 at 08:41:30PM -0400, mlw wrote:
> A mission statement is like a tie.

straw vote!

who on the list wears ties?

-- 
[ Jim Mercer        jim@reptiles.org         +1 416 410-5633 ]
[          I want to live forever, or die trying.            ]

Thu, 2 May 2002 20:47:31 -0400
Jim Mercer <jim@reptiles.org>
Re: PostgreSQL mission statement?
Jim Mercer wrote:
> 
> On Thu, May 02, 2002 at 08:41:30PM -0400, mlw wrote:
> > A mission statement is like a tie.
> 
> straw vote!
> 
> who on the list wears ties?

How many people who make IT decisions wear ties?

Thu, 02 May 2002 21:14:03 -0400
mlw <markw@mohawksoft.com>
Re: PostgreSQL mission statement?
On Thu, May 02, 2002 at 09:14:03PM -0400, mlw wrote:
> Jim Mercer wrote:
> > 
> > On Thu, May 02, 2002 at 08:41:30PM -0400, mlw wrote:
> > > A mission statement is like a tie.
> > 
> > straw vote!
> > 
> > who on the list wears ties?
> 
> How many people who make IT decisions wear ties?

too many.

-- 
[ Jim Mercer        jim@reptiles.org         +1 416 410-5633 ]
[          I want to live forever, or die trying.            ]

Thu, 2 May 2002 21:37:18 -0400
Jim Mercer <jim@reptiles.org>
Re: PostgreSQL mission statement?
Jim Mercer wrote:
> 
> On Thu, May 02, 2002 at 09:14:03PM -0400, mlw wrote:
> > Jim Mercer wrote:
> > >
> > > On Thu, May 02, 2002 at 08:41:30PM -0400, mlw wrote:
> > > > A mission statement is like a tie.
> > >
> > > straw vote!
> > >
> > > who on the list wears ties?
> >
> > How many people who make IT decisions wear ties?
> 
> too many.

I'm sorry I started this thread.

Thu, 02 May 2002 21:45:45 -0400
mlw <markw@mohawksoft.com>
Re: PostgreSQL mission statement?
On Thu, May 02, 2002 at 09:45:45PM -0400, mlw wrote:
> Jim Mercer wrote:
> > On Thu, May 02, 2002 at 09:14:03PM -0400, mlw wrote:
> > > Jim Mercer wrote:
> > > > On Thu, May 02, 2002 at 08:41:30PM -0400, mlw wrote:
> > > > > A mission statement is like a tie.
> > > > who on the list wears ties?
> > > How many people who make IT decisions wear ties?
> > too many.
> I'm sorry I started this thread.

don't be sorry.

i'm not big on wearing the corporate suit, be it physically, or figuratively.

that's my opinion, and i'm stating it.

your opinion differs, and that's fine.

i've had to do the corporate "mission statement" dance, as well as a bunch
of other hokey crap that didn't matter squat to the bottom line due to the
fact that the execs read some magazine article or attended some Tony Robbins
-esque motivational session.

when i hear "mission statement" and "quality circle" and "internal customer",
i cringe.

if the corporate management doesn't want to buy into the Open Source concept,
fuck 'em.

i've had a number of installations where due to management panic to get
something working, it was implemented using Open Source.  Only to have
a perfectly good system replaced with "real software" when management
finds out 6 months later that it is using Open Source.

i have had successes in getting Open Source into corporate environments,
but only after battling mega-politics with CIO/CFO and MSCE IT managers
who only want to see Microsoft or Sun Solaris solutions.

we did a project using FreeBSD and Samba to replace a number of highly
unstable NT file/print servers.

recently, some consultants (friends of the managing partners) said that
it was a bad idea to use "Public Domain software that was full of bugs
and highly insecure".

when we pointed out that the servers hadn't rebooted in 160 days, and
that they were protected by both RFC1918 addressing and a firewall, the
consultants backed off a bit.

then they returned spouting the same "full of bugs and highly insecure" crap.

now management is going to have them re-implement the network using the
latest NT stuff.

this is a long winded way of saying that my feeling is the type of MBA
CFO/CIO that is impressed by a mission statement, is probably not going
to buy into technology that isn't listed on NASDAQ.

so, what's the harm in having one?

probably not much, but to me it smells of corporate bullshit.

-- 
[ Jim Mercer        jim@reptiles.org         +1 416 410-5633 ]
[          I want to live forever, or die trying.            ]

Thu, 2 May 2002 22:29:17 -0400
Jim Mercer <jim@reptiles.org>
Re: PostgreSQL mission statement?
Jim Mercer wrote:
> 
> On Thu, May 02, 2002 at 09:45:45PM -0400, mlw wrote:
> > Jim Mercer wrote:
> > > On Thu, May 02, 2002 at 09:14:03PM -0400, mlw wrote:
> > > > Jim Mercer wrote:
> > > > > On Thu, May 02, 2002 at 08:41:30PM -0400, mlw wrote:
> > > > > > A mission statement is like a tie.
> > > > > who on the list wears ties?
> > > > How many people who make IT decisions wear ties?
> > > too many.
> > I'm sorry I started this thread.
> 
> don't be sorry.
> 
> i'm not big on wearing the corporate suit, be it physically, or figuratively.

Trust me, nor am I. I haven't warn a suit professionally since the '80s.

> 
> that's my opinion, and i'm stating it.
> 
> your opinion differs, and that's fine.

Don't be so sure.

> 
> i've had to do the corporate "mission statement" dance, as well as a bunch
> of other hokey crap that didn't matter squat to the bottom line due to the
> fact that the execs read some magazine article or attended some Tony Robbins
> -esque motivational session.

Yes I know. Been there done that.

> 
> when i hear "mission statement" and "quality circle" and "internal customer",
> i cringe.

Ditto.

> 
> if the corporate management doesn't want to buy into the Open Source concept,
> fuck 'em.

He is where we differ. There is merit in displaying an amount of understanding
of the corporate personality. I am not a corporate type, but I understand that
there are people that are, and to promote PostgreSQL, we need to reach them.

It is not selling out to use chopsticks at a chinese dinner, it is following
custom. A mission statement is similar. These people are brainwashed to look at
the mission statement. Having one for them to look at is not a bad idea.
> 
> i've had a number of installations where due to management panic to get
> something working, it was implemented using Open Source.  Only to have
> a perfectly good system replaced with "real software" when management
> finds out 6 months later that it is using Open Source.

Presenting a corporate aware culture will help them.
 
> i have had successes in getting Open Source into corporate environments,
> but only after battling mega-politics with CIO/CFO and MSCE IT managers
> who only want to see Microsoft or Sun Solaris solutions.

Been there done that.
 
> we did a project using FreeBSD and Samba to replace a number of highly
> unstable NT file/print servers.

Again, been there, done that.
 
> recently, some consultants (friends of the managing partners) said that
> it was a bad idea to use "Public Domain software that was full of bugs
> and highly insecure".

Pure FUD, of course.
 
> when we pointed out that the servers hadn't rebooted in 160 days, and
> that they were protected by both RFC1918 addressing and a firewall, the
> consultants backed off a bit.

Most consultants (not me :-) are idiots.

> then they returned spouting the same "full of bugs and highly insecure" crap.

They are uninformed, or worse, believe what Microsoft says.
 
> now management is going to have them re-implement the network using the
> latest NT stuff.

Fight it! Is there any evidence that will help you?
 
> this is a long winded way of saying that my feeling is the type of MBA
> CFO/CIO that is impressed by a mission statement, is probably not going
> to buy into technology that isn't listed on NASDAQ.

That is where the fight lies!! We have to make believers out of them! We have
superior technology, we have superior quality. 

Microsoft and Oracle have billions of dollars in marketing, all we have is
ourselves. 
> 
> so, what's the harm in having one?
> 
> probably not much, but to me it smells of corporate bullshit.

Corporate bullshit or not, it is a fact of life and a custom that we open
source people need to accept. We write the best shit, we do the best work. We
are "more professional" and dedicated than most professionals. Our quality is
usually much better than proprietary our counterparts. Unfortunately business
types do not understand us. If we are unable to reach the people who would
decide to use our stuff, then it is our fault for failure.

Thu, 02 May 2002 22:46:21 -0400
mlw <markw@mohawksoft.com>
Re: PostgreSQL mission statement?
Le Vendredi 3 Mai 2002 04:46, mlw a écrit :
> Corporate bullshit or not, it is a fact of life and a custom that we open
> source people need to accept. We write the best shit, we do the best work.
> We are "more professional" and dedicated than most professionals. Our
> quality is usually much better than proprietary our counterparts.
> Unfortunately business types do not understand us. If we are unable to
> reach the people who would decide to use our stuff, then it is our fault
> for failure.

Therefore we need a moto.

Fri, 3 May 2002 14:34:08 +0200
Jean-Michel POURE <jm.poure@freesurf.fr>
Re: PostgreSQL mission statement?
Le Vendredi 3 Mai 2002 02:22, Thomas Lockhart a écrit :
> "PostgreSQL is and will be the most advanced open-source database
> available anywhere."

*******************************************************************************************
The PostgreSQL community is committed to creating and maintaining the best, 
most reliable, open-source multi-purpose standards based database, and with 
it, promote free and open source software world wide.

Ultimately, PostgreSQL database is a gift to Humanity serving freedom, 
knowledge and equal access to information, and as such belongs to every 
Human.
*******************************************************************************************

PostgreSQL is "transcendantal", which means it goes "beyond" the original 
concept of its creators. People began working on it for various reasons (for 
professional needs, because of open-source code, to have fun) ... and 
ultimately it becomes a gift to Humanity.

My feeling is that the PostgreSQL community is making history without even 
noticing it. You are all heroes my friends...

Cheers,
Jean-Michel POURE


Fri, 3 May 2002 15:12:46 +0200
Jean-Michel POURE <jm.poure@freesurf.fr>
Re: PostgreSQL mission statement?
On Thu, 2 May 2002, Scott Marlowe wrote:

> On 2 May 2002, Hannu Krosing wrote:
>
> > The Politically Correct mission statement follows:
> >
> > The PostgreSQL community is committed to creating and maintaining a good
> > but not the best, mostly reliable, open-source multi-purpose standards
> > based database, and with it, promote free and open source software and
> > other worthy causes world wide and to not hurting anyones feelings in doing so.
> > We are also committed to not cheating our SOs, not charging too much for
> > our services nor eating too much and to recommending products of our
> > commercial competitors before ours in order to help them fullfil their
> > obligations to their stockholders.
>
> As a practicing polyamorist, I find the part about not cheating on our SOs
> highly offensive.  :-)

"not cheating our SOs", !"not cheating on our SOs" ... you add an 'on'
there that wasn't actually there :)



Fri, 3 May 2002 10:24:01 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: PostgreSQL mission statement?
Mark kirkwood <markir@slingshot.co.nz> writes:

> On Fri, 2002-05-03 at 04:25, mlw wrote:
> 
> > 
> > IMHO, if we can come up with a strong, positive statement, it
> > would help MBA trained CIOs and CTOs choose PostgreSQL. To them,
> > it will show a professional minded development group, it will be
> > recognizable to them.
> 
> I am not so sure about that -
> 
> In my experience the things that those guys use to decide on
> products are :
> 
> 1) reference sites
> 2) official support
> 
> (and they like to pay for a product 'cause they are used to doing
> it...but lets not go there...)
> 
> Personally I find the lack of "business-speak" things (like mission
> statements) refeshing, and I see it as part of the "values" that
> differentiate open source / community based products from commercial
> ones.
> 
> just my NZ4c (=US2c)
> 
> Mark

As a developer and systems administrator my favorite thing about
PostgreSQL is the fact that I can get the straight dope on what works
and what doesn't.  The PostgreSQL developers are quite candid, and are
more than willing to tell you which bits of PostgreSQL are dicey.
That's a huge bonus to someone creating and maintaining an
application.

However, it's not exactly the sort of salesmanship that my boss is
looking for.

Jason

03 May 2002 11:01:30 -0600
Jason Earl <jason.earl@simplot.com>
Re: PostgreSQL mission statement?
On Thu, 2 May 2002, Jim Mercer wrote:

> On Thu, May 02, 2002 at 08:41:30PM -0400, mlw wrote:
> > A mission statement is like a tie.
> 
> straw vote!
> 
> who on the list wears ties?

Does a skinny black tie count if I'm only wearing it to go out to a jazz 
club? :-)  Not at work though.  I think ties are designed to slow the flow 
of blood to the head so you'll think slow enough for marketeers to 
understand you.


Fri, 3 May 2002 11:03:07 -0600 (MDT)
Scott Marlowe <scott.marlowe@ihs.com>
Re: PostgreSQL mission statement?
Jim Mercer wrote: 
> On Thu, May 02, 2002 at 09:45:45PM -0400, mlw wrote:
> > Jim Mercer wrote:
> > > On Thu, May 02, 2002 at 09:14:03PM -0400, mlw wrote:
> > > > Jim Mercer wrote:
> > > > > On Thu, May 02, 2002 at 08:41:30PM -0400, mlw wrote:
> > > > > > A mission statement is like a tie.
> > > > > who on the list wears ties?
> > > > How many people who make IT decisions wear ties?
> > > too many.
> > I'm sorry I started this thread.
> when i hear "mission statement" and "quality circle" and "internal customer",
> i cringe.
> if the corporate management doesn't want to buy into the Open Source concept,
> fuck 'em.

<trench warfare snippage>

Let's see... open source philosophy applied *into* corporate-speak should
be doable....
1. If you have an itch, scratch it.
2. If you want to know what's going on, use the source, luke!
3. More eyeballs = less bugs.
4. Software should be free (insert debates on speech, beer, use, licence
XYZ vs. ABC, etc., I'm not going to bother).

Hm...firing up my geekspeak->corporate BS translator. :-)

How about:
"PostgreSQL creates a dynamic environment to ensure that all customers
can effectly create highly customized solutions specific to their needs. We
share and collaborate on both problems and solutions by making all
information about our products available. By using this open and exciting
environment, we increase the amount of successful software releases
using advanced concepts of peer review and peer enhancement. We ensure our
ongoing enhancement and improvement through our community, because
our customers are also our creators."

Now, if you'll excuse me, I have to go wash my mouth out with soap.

-Ronabop

Fri, 03 May 2002 13:46:18 -0700
Ron Chmara <ron@Opus1.COM>
Re: PostgreSQL mission statement?
I mentioned in another thread, Windows does not support "fork()."  PostgreSQL
seems irrevocably tied to using fork(). Without a drastic rewrite of how
postmaster works, I don't see a way to make a pure Windows version.

The big trick to cygwin is its implementation of fork(). It represents a very
important and fairly mature technique. It can be written for PostgreSQL but it
would require a fair amount of development time and testing.

Then we would need to be able to trace all the native API calls made so that
things like file handles are dealt with correctly for the child process.

I see cygwin as a portability layer or subsystem, as such, it should be able to
emulate foreign operating system constructs. A native application should, on
the other hand, not attempt to do so.

There is a strategy PostgreSQL could use:

Put all global variables which need to be duplicated in a single place, perhaps
a struct, which can be copied into the child process. On systems without
fork(), the memory can be duplicated or passed around using a shared memory
block, on a system with fork(), nothing extra would need to be done. This could
be implemented using "standard" APIs, with little or no specialized OS
knowledge. 

This represents a lot of reworking of code, but should not affect much in the
way of operation, but would require a lot of typing and testing. It would also
force restrictions on module static and global variables.

I will sign up to do the Windows stuff to get this to work, but it will take a
lot of postgres internal reworking that I am not up for doing.

The other alternative, is to profile PostgreSQL running in the cygwin
environment and try to assess where any bottlenecks are, and if there are any
spot optimizations which can be applied.

Tue, 07 May 2002 13:16:01 -0400
mlw <markw@mohawksoft.com>
How much work is a native Windows application?
mlw <markw@mohawksoft.com> writes:
> There is a strategy PostgreSQL could use:

> Put all global variables which need to be duplicated in a single
> place, perhaps a struct, which can be copied into the child
> process. On systems without fork(), the memory can be duplicated or
> passed around using a shared memory block, on a system with fork(),
> nothing extra would need to be done. This could be implemented using
> "standard" APIs, with little or no specialized OS knowledge.

> This represents a lot of reworking of code, but should not affect much
> in the way of operation, but would require a lot of typing and
> testing. It would also force restrictions on module static and global
> variables.

Yeah.  The real problem with it in my eyes is that it'd be a continuing
maintenance headache: straightforward programming techniques that work
fine on all the Unix ports would fail (perhaps in nonobvious ways) when
moved to Windows, should you forget to put a variable in the right
place.

A lesser objection is that variables that can currently be "static" in
a single module would become exposed to the world, which again is a
maintenance problem.

> The other alternative, is to profile PostgreSQL running in the cygwin
> environment and try to assess where any bottlenecks are, and if there
> are any spot optimizations which can be applied.

It'd be worth trying to understand cygwin issues in detail before we
sign up to do and support a native Windows port.  I understand the
user-friendliness objection to cygwin (though one would think proper
packaging might largely hide cygwin from naive Windows users).  What
I don't understand is whether there are any serious performance lossages
from it, and if so whether we could work around them.

			regards, tom lane

Tue, 07 May 2002 13:49:20 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: How much work is a native Windows application? 

http://www.mkssoftware.com/docs/man3/fork.3.asp

http://www.computing.net/programming/wwwboard/forum/60.html

http://www.research.att.com/sw/tools/uwin (Semaphores & Fork)

On Tue, 7 May 2002, mlw wrote:

> I mentioned in another thread, Windows does not support "fork()."  PostgreSQL
> seems irrevocably tied to using fork(). Without a drastic rewrite of how
> postmaster works, I don't see a way to make a pure Windows version.
>
> The big trick to cygwin is its implementation of fork(). It represents a very
> important and fairly mature technique. It can be written for PostgreSQL but it
> would require a fair amount of development time and testing.
>
> Then we would need to be able to trace all the native API calls made so that
> things like file handles are dealt with correctly for the child process.
>
> I see cygwin as a portability layer or subsystem, as such, it should be able to
> emulate foreign operating system constructs. A native application should, on
> the other hand, not attempt to do so.
>
> There is a strategy PostgreSQL could use:
>
> Put all global variables which need to be duplicated in a single place, perhaps
> a struct, which can be copied into the child process. On systems without
> fork(), the memory can be duplicated or passed around using a shared memory
> block, on a system with fork(), nothing extra would need to be done. This could
> be implemented using "standard" APIs, with little or no specialized OS
> knowledge.
>
> This represents a lot of reworking of code, but should not affect much in the
> way of operation, but would require a lot of typing and testing. It would also
> force restrictions on module static and global variables.
>
> I will sign up to do the Windows stuff to get this to work, but it will take a
> lot of postgres internal reworking that I am not up for doing.
>
> The other alternative, is to profile PostgreSQL running in the cygwin
> environment and try to assess where any bottlenecks are, and if there are any
> spot optimizations which can be applied.
>



Tue, 7 May 2002 14:52:26 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: How much work is a native Windows application?
"Marc G. Fournier" wrote:
> 
> http://www.mkssoftware.com/docs/man3/fork.3.asp
> 
> http://www.computing.net/programming/wwwboard/forum/60.html
> 
> http://www.research.att.com/sw/tools/uwin (Semaphores & Fork)

These are pretty much what I have been saying. 

Is PostgreSQL going to implement its own fork()? If so, what's the point? Just
use cygwin.

Without trying to sound conceited, I can write a fork() call, that's not the
problem. How much time will it take to do and get right? What about all the
infrastructure? Tracking file handles and resources allocated so that they can
be properly duplicated for the child process, etc. It is a lot of work, and to
do it for a BSD license, I shouldn't reference the cygwin code to do so.

The semaphore, shared memory, file API, etc. all these are straight forward.
They can be handled with a set of macros and some thin functions.

The problems of a native PostgreSQL on Windows is fork(), and all the
subtleties that go with it like ownership of system resources allocated by the
parent and passed to the child and initialization of global and static
variables.

Adding fork() to postgres seems silly. Cygwin does it already, and it seems
like it is outside the scope of what should be supported by PostgreSQL.

Since RedHat owns cygwin and they want RedHat database to be a success, maybe
they can make an exception to the GNU license for PostgreSQL. 

Does anyone think it is a good idea for PostgreSQL to implement it's own
version of fork()?

Tue, 07 May 2002 14:32:04 -0400
mlw <markw@mohawksoft.com>
Re: How much work is a native Windows application?
On Tue, 7 May 2002, Tom Lane wrote:

> It'd be worth trying to understand cygwin issues in detail before we
> sign up to do and support a native Windows port.  I understand the
> user-friendliness objection to cygwin (though one would think proper
> packaging might largely hide cygwin from naive Windows users).  What I
> don't understand is whether there are any serious performance lossages
> from it, and if so whether we could work around them.

Actually, there are licensing issues involved ... we could never put a
'windows binary' up for anon-ftp, since to distribute it would require the
cygwin.dll to be distributed, and to do that, there is a licensing cost
... of course, I guess we could require ppl to download cygwin seperately,
install that, then install the binary over top of that ...


Wed, 8 May 2002 01:03:37 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: How much work is a native Windows application? 
"Marc G. Fournier" <scrappy@hub.org> writes:
> On Tue, 7 May 2002, Tom Lane wrote:
>> It'd be worth trying to understand cygwin issues in detail before we
>> sign up to do and support a native Windows port.

> Actually, there are licensing issues involved ... we could never put a
> 'windows binary' up for anon-ftp, since to distribute it would require the
> cygwin.dll to be distributed, and to do that, there is a licensing cost
> ... of course, I guess we could require ppl to download cygwin seperately,
> install that, then install the binary over top of that ...

<<itch>>  And how much development time are we supposed to expend to
avoid that?

Give me a technical case for avoiding Cygwin, and maybe I can get
excited about it.  I'm not planning to lift a finger on the basis
of licensing though... after all, Windows users are accustomed to
paying for software, no?

			regards, tom lane

Wed, 08 May 2002 00:49:27 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: How much work is a native Windows application? 
...
> Give me a technical case for avoiding Cygwin, and maybe I can get
> excited about it.  I'm not planning to lift a finger on the basis
> of licensing though... after all, Windows users are accustomed to
> paying for software, no?

<evil grin>

You tell us: RH sells our database and sells cygwin, so might have an
opinion on the subject. Perhaps they would like to contribute back some
no cost package licensing terms for cygwin if used for PostgreSQL?? ;)

                    - Thomas

Tue, 07 May 2002 22:08:59 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: How much work is a native Windows application?
Le Mardi 7 Mai 2002 20:32, mlw a écrit :
> Since RedHat owns cygwin and they want RedHat database to be a success,
> maybe they can make an exception to the GNU license for PostgreSQL.

Cygwin received contributions from several authors. To leave a GNU licence, 
you need to get the agreement of all authors, which is not possible for a big 
project like Cygwin.

Alternatively, you could pick-up the old static version of Cygwin and release 
it as a minimal Cygwin dll. This idea is probably ***stupid***.

Another possible solution is http://debian-cygwin.sourceforge.net/ project, 
which tries to port dpkg to Windows. If it was possible to release Cygwin 
using dpkg, we could create a comprehensive Cygwin + PostgreSQL on-line 
installer.

Why look for complicated solutions when the only real issue for users is the 
Cygwin installer?

Cheers,
Jean-Michel POURE

Wed, 8 May 2002 11:02:58 +0200
Jean-Michel POURE <jm.poure@freesurf.fr>
Re: How much work is a native Windows application?
Le Mercredi 8 Mai 2002 06:03, Marc G. Fournier a écrit :
> Actually, there are licensing issues involved ... we could never put a
> 'windows binary' up for anon-ftp, since to distribute it would require the
> cygwin.dll to be distributed, and to do that, there is a licensing cost
> ... of course, I guess we could require ppl to download cygwin seperately,
> install that, then install the binary over top of that ...

This is an installer problem. Let's port dpkg to Windows, create a minimal 
Cygwin dpkg distribution .. et voilà.

Wed, 8 May 2002 11:05:40 +0200
Jean-Michel POURE <jm.poure@freesurf.fr>
Re: How much work is a native Windows application?
On Wed, 2002-05-08 at 14:02, Jean-Michel POURE wrote:
> Le Mardi 7 Mai 2002 20:32, mlw a écrit :
> > Since RedHat owns cygwin and they want RedHat database to be a success,
> > maybe they can make an exception to the GNU license for PostgreSQL.
> 
> Cygwin received contributions from several authors. To leave a GNU licence, 
> you need to get the agreement of all authors, which is not possible for a big 
> project like Cygwin.
> 
> Alternatively, you could pick-up the old static version of Cygwin and release 
> it as a minimal Cygwin dll. This idea is probably ***stupid***.
> 
> Another possible solution is http://debian-cygwin.sourceforge.net/ project, 
> which tries to port dpkg to Windows. If it was possible to release Cygwin 
> using dpkg, we could create a comprehensive Cygwin + PostgreSQL on-line 
> installer.
> 
> Why look for complicated solutions when the only real issue for users is the 
> Cygwin installer?

IIRC the initial issue was bad performance, which was attributed to
win32/cygwin fork() behaviour.

That was long before this thread started.

BTW, does anyone know how other real databases (Oracle, DB2,
Interbase/Firebird, Infomix) do it on Windows.

-------------
Hannu



08 May 2002 15:12:28 +0500
Hannu Krosing <hannu@tm.ee>
Re: How much work is a native Windows application?
On Wed, 8 May 2002 01:03:37 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org> wrote:
> On Tue, 7 May 2002, Tom Lane wrote:
> 
> > It'd be worth trying to understand cygwin issues in detail before we
> > sign up to do and support a native Windows port.  I understand the
> > user-friendliness objection to cygwin (though one would think proper
> > packaging might largely hide cygwin from naive Windows users).  What I
> > don't understand is whether there are any serious performance lossages
> > from it, and if so whether we could work around them.
> 
> Actually, there are licensing issues involved ... we could never put a
> 'windows binary' up for anon-ftp, since to distribute it would require the
> cygwin.dll to be distributed, and to do that, there is a licensing cost

Why? Isn't Cyygwin GPL'd? From http://cygwin.com/licensing.html I don't
see anything that would require licensing fees for OSD-compliant software.

Cheers,

Neil

-- 
Neil Conway <neilconway@rogers.com>
PGP Key ID: DB3C29FC

Wed, 8 May 2002 08:15:09 -0400
Neil Conway <nconway@klamath.dyndns.org>
Re: How much work is a native Windows application?
On Wed, 8 May 2002, Neil Conway wrote:

> On Wed, 8 May 2002 01:03:37 -0300 (ADT)
> "Marc G. Fournier" <scrappy@hub.org> wrote:
> > On Tue, 7 May 2002, Tom Lane wrote:
> >
> > > It'd be worth trying to understand cygwin issues in detail before we
> > > sign up to do and support a native Windows port.  I understand the
> > > user-friendliness objection to cygwin (though one would think proper
> > > packaging might largely hide cygwin from naive Windows users).  What I
> > > don't understand is whether there are any serious performance lossages
> > > from it, and if so whether we could work around them.
> >
> > Actually, there are licensing issues involved ... we could never put a
> > 'windows binary' up for anon-ftp, since to distribute it would require the
> > cygwin.dll to be distributed, and to do that, there is a licensing cost
>
> Why? Isn't Cyygwin GPL'd? From http://cygwin.com/licensing.html I don't
> see anything that would require licensing fees for OSD-compliant software.

I may be wrong about this ... this was prior to Redhat buying it out,
which I totally forgot about ...


Wed, 8 May 2002 09:28:38 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: How much work is a native Windows application?
> -----Original Message-----
> From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Tom Lane
> Sent: Tuesday, May 07, 2002 1:49 PM
> To: mlw
> Cc: Marc G. Fournier; PostgreSQL-development
> Subject: Re: [HACKERS] How much work is a native Windows application?
>
>
> It'd be worth trying to understand cygwin issues in detail before we
> sign up to do and support a native Windows port.  I understand the
> user-friendliness objection to cygwin (though one would think proper
> packaging might largely hide cygwin from naive Windows users).  What
> I don't understand is whether there are any serious performance lossages
> from it, and if so whether we could work around them.

I've sent others to do a cygwin install of PG; it's not at all obvious to
them how much of cygwin they need & they end up installing ALL of cygwin (a
ton of devel tools, obscure unix utils, etc.) just to get PG working.

It would seem not too difficult to package up the cygwin.dll, the handful of
shell utils (sh, rm, etc.) req'd by PG, and perhaps even give it a standard
Windows installer.

Would this be a worthwhile move?

Joel BURTON | joel@joelburton.com | joelburton.com | aim: wjoelburton
Knowledge Management & Technology Consultant


Wed, 8 May 2002 10:23:05 -0400
"Joel Burton" <joel@joelburton.com>
Re: How much work is a native Windows application? 
> -----Original Message-----
> From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Marc G. Fournier
> Sent: Wednesday, May 08, 2002 12:04 AM
> To: Tom Lane
> Cc: mlw; PostgreSQL-development
> Subject: Re: [HACKERS] How much work is a native Windows application?
>
>
> On Tue, 7 May 2002, Tom Lane wrote:
>
> > It'd be worth trying to understand cygwin issues in detail before we
> > sign up to do and support a native Windows port.  I understand the
> > user-friendliness objection to cygwin (though one would think proper
> > packaging might largely hide cygwin from naive Windows users).  What I
> > don't understand is whether there are any serious performance lossages
> > from it, and if so whether we could work around them.
>
> Actually, there are licensing issues involved ... we could never put a
> 'windows binary' up for anon-ftp, since to distribute it would require the
> cygwin.dll to be distributed, and to do that, there is a licensing cost
> ... of course, I guess we could require ppl to download cygwin seperately,
> install that, then install the binary over top of that ...


>From http://cygwin.com/licensing.html:

"""
In accordance with section 10 of the GPL, Red Hat permits programs whose
sources are distributed under a license that complies with the Open Source
definition to be linked with libcygwin.a without libcygwin.a itself causing
the resulting program to be covered by the GNU GPL.

This means that you can port an Open Source(tm) application to cygwin, and
distribute that executable as if it didn't include a copy of libcygwin.a
linked into it. Note that this does not apply to the cygwin DLL itself. If
you distribute a (possibly modified) version of the DLL you must adhere to
the terms of the GPL, i.e. you must provide sources for the cygwin DLL.

See http://www.opensource.org/osd.html for the precise Open Source
Definition referenced above.
"""

Not following this exactly, but would this give PG the exception it needs
(my eyes start to glaze over on stuff like this)? Anyone from RedHat still
on this list?

In any event, if PG can't release a PG+Cygwin in one package, we could
maintain a official web page about how to get PG running under Cygwin that
walks through exactly what to install, how to install, and how to set up.

There are some notes at http://www.ca.postgresql.org/docs/faq-mswin.html,
but these are assuming that you want to build PG, rather than simply install
PG from the cygwin packages.

I'd be very willing to help with this effort, once there's some consensus on
what direction we want to head.

- J.

Joel BURTON | joel@joelburton.com | joelburton.com | aim: wjoelburton
Knowledge Management & Technology Consultant


Wed, 8 May 2002 10:42:07 -0400
"Joel Burton" <joel@joelburton.com>
Re: How much work is a native Windows application? 
"Marc G. Fournier" <scrappy@hub.org> writes:

> On Tue, 7 May 2002, Tom Lane wrote:
> 
> > It'd be worth trying to understand cygwin issues in detail before we
> > sign up to do and support a native Windows port.  I understand the
> > user-friendliness objection to cygwin (though one would think proper
> > packaging might largely hide cygwin from naive Windows users).  What I
> > don't understand is whether there are any serious performance lossages
> > from it, and if so whether we could work around them.
> 
> Actually, there are licensing issues involved ... we could never put
> a 'windows binary' up for anon-ftp, since to distribute it would
> require the cygwin.dll to be distributed, and to do that, there is a
> licensing cost ... of course, I guess we could require ppl to
> download cygwin seperately, install that, then install the binary
> over top of that ...

 From the Cygwin FAQ:

        Is it free software?

        Yes. Parts are GNU software (gcc, gas, ld, etc...), parts are
        covered by the standard X11 license, some of it is public
        domain, some of it was written by Cygnus and placed under the
        GPL. None of it is shareware. You don't have to pay anyone to
        use it but you should be sure to read the copyright section of
        the FAQ more more information on how the GNU General Public
        License may affect your use of these tools.

There is even a clause allowing you to link to the cygwin dll without
your software falling under the GPL if your software is released under
a license that complies with the Open Source Definition.

        *** NOTE ***

        In accordance with section 10 of the GPL, Red Hat permits
        programs whose sources are distributed under a license that
        complies with the Open Source definition to be linked with
        libcygwin.a without libcygwin.a itself causing the resulting
        program to be covered by the GNU GPL.

        This means that you can port an Open Source(tm) application to
        cygwin, and distribute that executable as if it didn't include
        a copy of libcygwin.a linked into it. Note that this does not
        apply to the cygwin DLL itself. If you distribute a (possibly
        modified) version of the DLL you must adhere to the terms of
        the GPL, i.e. you must provide sources for the cygwin DLL.

        See http://www.opensource.org/osd.html for the precise Open
        Source Definition referenced above.

        Red Hat sells a special Cygwin License for customers who are
        unable to provide their application in open source code
        form. For more information, please see:
        http://www.redhat.com/software/tools/cygwin/, or call
        866-2REDHAT ext. 3007

In other words, you could even distribute a BSD licensed PostgreSQL
that that ran on Windows.  Not that such a loophole is particularly
useful.  GPL projects regularly include BSD code, this doesn't make
the BSD version GPLed.  The GPL might be viral in nature, but it's not
that viral.

Now, I can understand why the PostgreSQL mirrors might be a little bit
concerned about distributing GPLed software, because of the legal
ramifications, but they could leave the distribution of Cygwin up to
RedHat, and simply distribute a BSD-licensed PostgreSQL Windows
binary.

Jason

08 May 2002 13:18:30 -0600
Jason Earl <jason.earl@simplot.com>
Re: How much work is a native Windows application?
On Tue, May 07, 2002 at 01:16:01PM -0400, mlw wrote:
> I mentioned in another thread, Windows does not support "fork()."  PostgreSQL
> seems irrevocably tied to using fork(). Without a drastic rewrite of how
> postmaster works, I don't see a way to make a pure Windows version.

 I watch this discussion and only one question is still in my head:
 how much people use Windows for server side part of stable application 
 based Oracle or DB2? Why my employer spend a lot of money with
 SGI cluster + IRIX?

 _IMHO_ if you want support Windows, please, write good tools for admins, 
 DB designers and developers (forms?). The server is really not a problem if 
 you think about real DB application. There is more important things in our
 TODO than support GUI-OS for server running... (IMHO:-)
 
    Karel

-- 
 Karel Zak  <zakkr@zf.jcu.cz>
 http://home.zf.jcu.cz/~zakkr/
 
 C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz

Thu, 9 May 2002 09:25:04 +0200
Karel Zak <zakkr@zf.jcu.cz>
Re: How much work is a native Windows application?
Le Jeudi 9 Mai 2002 09:25, Karel Zak a écrit :
>  _IMHO_ if you want support Windows, please, write good tools for admins,
>  DB designers and developers (forms?). The server is really not a problem
> if you think about real DB application. There is more important things in
> our TODO than support GUI-OS for server running... (IMHO:-)
Try pgAdmin2 (http://pgadmin.postgresql.org. Other project are in the hub. 
Mono port for example.

Thu, 9 May 2002 10:10:44 +0200
Jean-Michel POURE <jm.poure@freesurf.fr>
Re: How much work is a native Windows application?
Tom Lane wrote:
> "Marc G. Fournier" <scrappy@hub.org> writes:
> > On Tue, 7 May 2002, Tom Lane wrote:
> >> It'd be worth trying to understand cygwin issues in detail before we
> >> sign up to do and support a native Windows port.
>
> > Actually, there are licensing issues involved ... we could never put a
> > 'windows binary' up for anon-ftp, since to distribute it would require the
> > cygwin.dll to be distributed, and to do that, there is a licensing cost
> > ... of course, I guess we could require ppl to download cygwin seperately,
> > install that, then install the binary over top of that ...
>
> <<itch>>  And how much development time are we supposed to expend to
> avoid that?
>
> Give me a technical case for avoiding Cygwin, and maybe I can get
> excited about it.  I'm not planning to lift a finger on the basis
> of licensing though... after all, Windows users are accustomed to
> paying for software, no?

    Nobody  asked  you  to lift any of your fingers. A few people
    (including me) just see  value  in  a  native  Windows  port,
    kicking out the Cygwin requirement.

    I have the impression you never did use Cygwin. I did, thanks
    but no thanks.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Thu, 9 May 2002 09:44:11 -0400 (EDT)
Jan Wieck <janwieck@yahoo.com>
Re: How much work is a native Windows application?
Jan Wieck wrote:
> 
> Tom Lane wrote:
> > "Marc G. Fournier" <scrappy@hub.org> writes:
> > > On Tue, 7 May 2002, Tom Lane wrote:
> > >> It'd be worth trying to understand cygwin issues in detail before we
> > >> sign up to do and support a native Windows port.
> >
> > > Actually, there are licensing issues involved ... we could never put a
> > > 'windows binary' up for anon-ftp, since to distribute it would require the
> > > cygwin.dll to be distributed, and to do that, there is a licensing cost
> > > ... of course, I guess we could require ppl to download cygwin seperately,
> > > install that, then install the binary over top of that ...
> >
> > <<itch>>  And how much development time are we supposed to expend to
> > avoid that?
> >
> > Give me a technical case for avoiding Cygwin, and maybe I can get
> > excited about it.  I'm not planning to lift a finger on the basis
> > of licensing though... after all, Windows users are accustomed to
> > paying for software, no?
> 
>     Nobody  asked  you  to lift any of your fingers. A few people
>     (including me) just see  value  in  a  native  Windows  port,
>     kicking out the Cygwin requirement.
> 
>     I have the impression you never did use Cygwin. I did, thanks
>     but no thanks.

I have used the cygwin version too. It is a waste of time. No Windows user will
ever accept it. No windows-only user is going to use the cygwin tools. From a
production stand point, would anyone reading this trust their data to
PostgreSQL running on cygwin? Think about it, if you wouldn't, why would anyone
else.

I think, and I know people are probably sick of me spouting opinions, that if
you want a Windows presence for PostgreSQL, then we should write a real Win32
version.

If the global/static variables which are initialized by the postmaster are
moved to a structure, we can should be able to remove the fork() requirement
and port to a Win32 native system.

Thu, 09 May 2002 10:05:03 -0400
mlw <markw@mohawksoft.com>
Re: How much work is a native Windows application?
Jan Wieck writes:
 > Tom Lane wrote:
 > > Give me a technical case for avoiding Cygwin, and maybe I can get
 > > excited about it.  I'm not planning to lift a finger on the basis
 > > of licensing though... after all, Windows users are accustomed to
 > > paying for software, no?
 >     Nobody  asked  you  to lift any of your fingers. A few people
 >     (including me) just see  value  in  a  native  Windows  port,
 >     kicking out the Cygwin requirement.
 >     I have the impression you never did use Cygwin. I did, thanks
 >     but no thanks.

I think the crux of the the problem is that a native Windows port
would require a LOT of changes in the source (switching over to API
wrappers, adding compatibility layers). Obviously this has the
possibility of introducing a lot of bugs with zero gain for the folk
who are already happily running PostgreSQL on UNIX-like systems. And
what of performance?

Sure It'd be nice to have a native PostgreSQL on XP Server (I don't
see the point in consumer level Microsoft OSs) but how high is the
demand? What's the prize? What are the current limitations - fork,
semaphores, ugly interface...?

Lee.

Thu, 9 May 2002 15:13:27 +0100
Lee Kindness <lkindness@csl.co.uk>
Re: How much work is a native Windows application?
Lee Kindness wrote:

> 
> Sure It'd be nice to have a native PostgreSQL on XP Server (I don't
> see the point in consumer level Microsoft OSs) but how high is the
> demand? What's the prize? What are the current limitations - fork,
> semaphores, ugly interface...?

The demand for PostgreSQL on Windows is currently as near to zero as you can
imagine. This is probably because there is no viable PostgreSQL on Windows.

If written correctly, a Win32 version of PostgreSQL would rock the Windows
world. I see no reason why it would be limted to the "professional" version.
Hell, it could even run on Windows 98.

Right now, in the small to medium space, there is only one choice for Windows,
MSSQL. It requires the "professional" or server versions of the Microsoft
platforms. PostgreSQL could come in and run on all of them. 

PostgreSQL's feature set and price ($0), with a good installer, would do VERY
well.

Thu, 09 May 2002 10:23:41 -0400
mlw <markw@mohawksoft.com>
Re: How much work is a native Windows application?
mlw <markw@mohawksoft.com> writes:
> I have used the cygwin version too. It is a waste of time. No Windows user will
> ever accept it. No windows-only user is going to use the cygwin tools.

With decent packaging, no windows-only user would even know we have
cygwin in there.  The above argument is just plain irrelevant.  The real
point is that we need a nice clean friendly GUI for both installation
and administration --- and AFAICS that will take about the same amount of
work to write whether the server requires cygwin internally or not.

Rather than expending largely-pointless work on internal rewrites of
the server, people who care about this issue ought to be thinking about
the GUI problems.

> From a production stand point, would anyone reading this trust their
> data to PostgreSQL running on cygwin?

I wouldn't trust my data to *any* database running on a Microsoft OS.
Period.  The above argument thus doesn't impress me at all, especially
when it's being made without offering a shred of evidence that cygwin
contributes any major degree of instability.

I am especially unhappy about the prospect of major code revisions
and development time spent on chasing this rather than improving our
performance and stability on Unix-type OSes.  I agree with the comment
someone else made: that's just playing Microsoft's game.

			regards, tom lane

Thu, 09 May 2002 10:25:43 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: How much work is a native Windows application? 
Tom Lane wrote:
> With decent packaging, no windows-only user would even know we have
> cygwin in there.  The above argument is just plain irrelevant.  The real
> point is that we need a nice clean friendly GUI for both installation
> and administration --- and AFAICS that will take about the same amount of
> work to write whether the server requires cygwin internally or not.

Can a cygwin version of PostgreSQL see the native file system, like: C:\My
Database, D:\postgres?

> > From a production stand point, would anyone reading this trust their
> > data to PostgreSQL running on cygwin?
> 
> I wouldn't trust my data to *any* database running on a Microsoft OS.

That is a prejudice that is affecting your judgment. Many people do trust
Windows, do I? No, but a lot of people do. People have trusted their businesses
on Windows NT/2K/XP, many are still doing so. We want these people to use
PostgreSQL, so when they see the error in their ways, they have a way out.


> The above argument thus doesn't impress me at all, especially
> when it's being made without offering a shred of evidence that cygwin
> contributes any major degree of instability.

 From a software development standpoint, I am VERY uncomfortable with the
technique of a user space program copying its writeable memory to another
process's. It may work until Microsoft changes something with the next version
of IE. What about anti-virus software, cygwin has problems with them, and you
have to have anti-virus software on Windows.

On top of that, the time spent copying the whole process is too long, and it
forces real memory to be allocated and initialized at process startup. 

So, the cygwin fork() will cause PostgreSQL to be slower and use more memory
than a native version, and will not co-exist well with anti-virus software.

> 
> I am especially unhappy about the prospect of major code revisions
> and development time spent on chasing this rather than improving our
> performance and stability on Unix-type OSes.  I agree with the comment
> someone else made: that's just playing Microsoft's game.

Maybe is is playing "Microsoft's Game" but the end result will be a program
that can seriously compete with MSSQL on Windows, and provide a REAL migration
path to UNIX. 

Many developers use MSSQL because they "have it" in MSDN, so to them, it is
free. Once they develop something using it, they are tied to Windows. When it
comes time to deploy their pet project, the company has to cough up the price
of the server.

A native, friendly, Win32 PostgreSQL that works the same on Windows as it does
on FreeBSD, Linux, Solaris, etc. Will offer the developer real options away
from Windows.

Also: I don't think it needs to be a major rewrite, no strategy needs to
change, it is basically renaming variables, i.e. my_global_var becomes
pg_globals.my_global_var.

Once that is done, a port writer can do what ever they need to do to get that
structure to the child correctly. As an exercise, I bet if we did this, we
would find bugs which are lurking, as yet unfound.

Besides, the discipline of using a globals structure will improve the code
base. Don't you agree?

Thu, 09 May 2002 10:55:01 -0400
mlw <markw@mohawksoft.com>
Re: How much work is a native Windows application?
...
> PostgreSQL's feature set and price ($0), with a good installer, would do VERY
> well.

That may be (I'd like to think so!).

We've identified at least a couple of barriers to folks running
PostgreSQL on Windows. The installer and GUI issue needs to be solved no
matter what, and we *could* have a version running on Windows with just
those things in place.

imho if we are going down the path, we need to take the first steps. And
those do *not* require code rewrites to do so (or at least don't appear
to).

If we had a package available for Windows -- with some developers such
as yourself supporting it -- then we could talk about putting more
resources into supporting that platform better. But the perception of at
least some of the key developers (including myself) is that *if* we did
the code rewrite, and *if* we spent the effort to end up as a native on
Windows, then we *very well might* be an unreliable database on an
unreliable platform.

istm that getting a well packaged system running now, then being able to
identify *only cygwin* as the barrier to better reliability would get
more support for changes in the backend code.

And if we were working toward some ability to do threading anyway (I
don't see that in the near future, but we've talked in the past about
structuring the query engine around "tuple sources" which could then be
distributed across threads or across machines) then maybe the next step
is easier.

My 2c...

                        - Thomas

Thu, 09 May 2002 08:13:17 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: How much work is a native Windows application?
There are some issues that the whole idea of a win32 port should bring up.  
One of them is whether or not postgresql should be rewritten as a 
multi-threaded app.

If postgresql will never be rewritten as a multi-threaded app, then 
performance under Windows is likely to ALWAYS be slow, since that 
multi-thread is the preferred model for good performance on W32.  note 
that many Unixes prefer multi-threaded models as well (Solaris comes to 
mind) so there's the possibility that a multi-threaded postgresql could 
enjoy better performance on more than just windows.

If postgresql IS going to eventually be multi-threaded, then the whole 
win32 port should probably be delayed until then, since it would solve 
many of the issues of fork() versus createprocess().

Just my thoughts on it.

Scott


Thu, 9 May 2002 10:34:58 -0600 (MDT)
Scott Marlowe <scott.marlowe@ihs.com>
Issues tangential to win32 support
mlw wrote:
> I think, and I know people are probably sick of me spouting opinions, that if
> you want a Windows presence for PostgreSQL, then we should write a real Win32
> version.
>
> If the global/static variables which are initialized by the postmaster are
> moved to a structure, we can should be able to remove the fork() requirement
> and port to a Win32 native system.

    My  opinion  here is that until May 1998 Postgres did exec(),
    so it was clean and okay for CreateProcess() up to then. Just
    because  we  optimized  it  for  the copy-on-write behaviour,
    modern Unix kernels do with fork()  only,  is  NO  reason  to
    accept  sloppy  coding.  The  Postmaster and the backend have
    different responsibilities. In fact, I  still  consider  them
    beeing   different  programs  even  if  they  reside  in  one
    executable. Mixing global variables of one with the other  is
    wrong.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Thu, 9 May 2002 12:37:16 -0400 (EDT)
Jan Wieck <janwieck@yahoo.com>
Re: How much work is a native Windows application?


mlw wrote:
> Can a cygwin version of PostgreSQL see the native file system, like: C:\My
> Database, D:\postgres?

Sort of.  C:\My Database
becomes  /cygdrive/c/My Database
under cygwin.  So the path names need to be munged, but you can access 
the entire windows filesystem from within cygwin.

--Barry



Thu, 09 May 2002 09:44:14 -0700
Barry Lind <barry@xythos.com>
Re: How much work is a native Windows application?
> Lee Kindness wrote:
>> Sure It'd be nice to have a native PostgreSQL on XP Server (I don't
>> see the point in consumer level Microsoft OSs) but how high is the
>> demand? What's the prize? What are the current limitations - fork,
>> semaphores, ugly interface...?

> The demand for PostgreSQL on Windows is currently as near to zero
> as you can imagine. This is probably because there is no viable
> PostgreSQL on Windows.
>
> If written correctly, a Win32 version of PostgreSQL would rock
> the Windows world. I see no reason why it would be limted to the
> "professional" version. Hell, it could even run on Windows 98.
>
> Right now, in the small to medium space, there is only one choice for
> Windows, MSSQL. It requires the "professional" or server versions of
> the Microsoft platforms. PostgreSQL could come in and run on all of
> them.
>
> PostgreSQL's feature set and price ($0), with a good installer, would
> do VERY well.

If "fixing" PostgreSQL to "work" on Win32 caused a whole lot of breakage on the Unix side, that would _not_ be a "win."  It might do well on Win32, but breakage could lead to a LOSS of interest on Unix, as people decided to take the point of view that the developers considered it more important to toady to Win-Needs than to improve how it works on Unix.

Is that a totally "fair" point of view?  No, but in a world where New York office buildings get bombed resulting in absolutely bizarre combinations of cheering and jeering, an expectation of "fairness" is definitely too much to ask.  (You won't get anything resembling "fairness" at the airport, that's for sure...)

How about a totally different perspective:  
  Why not let someone else go after the Windows "market"?

People are actively working on SAP-DB and Firebird, and putting pretty serious efforts into the Win32 side of those database systems.  How outrageous an idea is it to say: "Let them deal with that set of headaches?"

Aside from that, there's also the "Show me the patch" option.  If someone is excited about porting PostgreSQL to Win32, nothing is stopping them from doing so, and contributing patches back.  There seem to be several such efforts out there; if one becomes mature enough, it may represent a useful basis to port in to make the main codebase "more portable."

There are at least the two clear options: 
- Other DBs;
- Volunteers porting PostgreSQL to Win32.

If a "winner" emerges, that would surely be useful to guide later PostgreSQL efforts.
--
(reverse (concatenate 'string "gro.mca@" "enworbbc"))
http://www.cbbrowne.com/info/wp.html
"What  you end  up with,  after  running an  operating system  concept
through these  many marketing coffee filters, is  something not unlike
plain hot water."  -- Matt Welsh
-- 
(reverse (concatenate 'string "moc.enworbbc@" "enworbbc"))
http://www3.sympatico.ca/cbbrowne/sap.html
"I'm not switching from slrn.   I'm quite confident that anything that
*needs* to be posted in HTML is fatuous garbage not worth my time."
-- David M. Cook <davecook@home.com>
Thu, 09 May 2002 13:00:31 -0400
Christopher Browne <cbbrowne@cbbrowne.com>
Re: How much work is a native Windows application? 
> I think, and I know people are probably sick of me spouting opinions,
> that if you want a Windows presence for PostgreSQL, then we should
> write a real Win32 version.

The crucial wrong word is the word "we."

If _you_ want a Windows presence, then _you_ should write a real Win32 
version.  That clearly attaches responsibility to someone who is interested.
--
(reverse (concatenate 'string "gro.mca@" "enworbbc"))
http://www.cbbrowne.com/info/unix.html
"I'm not switching from slrn.   I'm quite confident that anything that
*needs* to be posted in HTML is fatuous garbage not worth my time."
-- David M. Cook <davecook@home.com>

-- 
(reverse (concatenate 'string "moc.enworbbc@" "enworbbc"))
http://www.ntlug.org/~cbbrowne/spreadsheets.html
Rules of the Evil Overlord #166.  "If the rebels manage to trick me, I
will make a  note of what they did  so that I do not  keep falling for
the same trick over and over again." <http://www.eviloverlord.com/>
Thu, 09 May 2002 13:02:33 -0400
cbbrowne@cbbrowne.com
Re: How much work is a native Windows application? 
cbbrowne@cbbrowne.com wrote:
> 
> > I think, and I know people are probably sick of me spouting opinions,
> > that if you want a Windows presence for PostgreSQL, then we should
> > write a real Win32 version.
> 
> The crucial wrong word is the word "we."
> 
> If _you_ want a Windows presence, then _you_ should write a real Win32
> version.  That clearly attaches responsibility to someone who is interested.

I have already said that I am willing to write the pieces for a Windows port.
The issue is changes in PostgreSQL required to do it.

Thu, 09 May 2002 13:24:03 -0400
mlw <markw@mohawksoft.com>
Re: How much work is a native Windows application?
Christopher Browne wrote:
> 
> If "fixing" PostgreSQL to "work" on Win32 caused a whole lot of
> breakage on the Unix side, that would _not_ be a "win."  It might
> do well on Win32, but breakage could lead to a LOSS of interest
> on Unix, as people decided to take the point of view that the
> developers considered it more important to toady to Win-Needs
> than to improve how it works on Unix.
 
As a PostgreSQL user, I *wholeheartedly* agree.  I have no need nor
interest in a Win32 solution.  Period.  If I perceive that an effort
to add a Win32 postgresql is adversely impacting the ongoing
development of Unix-based PostgreSQL then I will start looking at
other solutions.

In fact, if you folks could find additional resources that would
support Win32 development, it still seems to me that perhaps those
resources could be better spent improving the Unix version.

-- 
Steve Wampler -- swampler@noao.edu
O sibile, si ergo.  Fortibus es enaro.
Nobile, demis trux.  Demis phulla causan dux.

Thu, 09 May 2002 10:29:29 -0700
Steve Wampler <swampler@noao.edu>
Re: How much work is a native Windows application?
Scott Marlowe wrote:
> 
> There are some issues that the whole idea of a win32 port should bring up.
> One of them is whether or not postgresql should be rewritten as a
> multi-threaded app.

Perhaps.

> 
> If postgresql will never be rewritten as a multi-threaded app, then
> performance under Windows is likely to ALWAYS be slow, since that
> multi-thread is the preferred model for good performance on W32.

There are methods for reducing process creation load on Windows. One way is to
make PostgreSQL one big .DLL and just spin off a small program. A windows .DLL
is different than a UNIX shared library, in some ways better, in other ways
worse, either way, it is a usefull tool.


>  note
> that many Unixes prefer multi-threaded models as well (Solaris comes to
> mind) so there's the possibility that a multi-threaded postgresql could
> enjoy better performance on more than just windows.

The isolation of a process is very important to reliable operation. Going
threaded usually means allowing a single connection to bring down the whole
server.

Thu, 09 May 2002 13:36:05 -0400
mlw <markw@mohawksoft.com>
Re: Issues tangential to win32 support
Le Jeudi 9 Mai 2002 16:55, mlw a écrit :
> Can a cygwin version of PostgreSQL see the native file system, like: C:\My
> Database, D:\postgres?

You have the choice to keep Windows or Unix paths. Both are supported.
/Jean-Michel POURE

Thu, 9 May 2002 19:42:10 +0200
Jean-Michel POURE <jm.poure@freesurf.fr>
Re: How much work is a native Windows application?
> cbbrowne@cbbrowne.com wrote:
>>> I think, and I know people are probably sick of me spouting
>>> opinions, that if you want a Windows presence for PostgreSQL, then
>>> we should write a real Win32 version.
>>
>> The crucial wrong word is the word "we."

>> If _you_ want a Windows presence, then _you_ should write a real
>> Win32 version.  That clearly attaches responsibility to someone who
>> is interested.

> I have already said that I am willing to write the pieces for a
> Windows port.  The issue is changes in PostgreSQL required to do it.

No, I don't think you understand.

If you're planning to do a port, then _all_ changes are your
responsibility.  Nobody ought to need to change PostgreSQL in order for
you to write a Windows port; that, in fact, would be a waste of time,
having several people working on something that should probably be done
by one person.
--
(concatenate 'string "aa454" "@freenet.carleton.ca")
http://www.ntlug.org/~cbbrowne/x.html
Why are they called apartments, when they're all stuck together? 

Thu, 09 May 2002 13:46:04 -0400
cbbrowne@cbbrowne.com
Re: How much work is a native Windows application? 
Scott Marlowe wrote:
> There are some issues that the whole idea of a win32 port should bring up.
> One of them is whether or not postgresql should be rewritten as a
> multi-threaded app.

    Please, don't add this one to it.

    I'm  all for the native Windows port, yes, but I've discussed
    the multi-thread questions for days  at  Great  Bridge,  then
    again with my new employer, with people on shows and whatnot.

    Anything in the whole  backend  is  designed  with  a  multi-
    process  model in mind.  You'll not do that in any reasonable
    amount of time.

> If postgresql will never be rewritten as a multi-threaded app, then
> performance under Windows is likely to ALWAYS be slow, since that
> multi-thread is the preferred model for good performance on W32.  note
> that many Unixes prefer multi-threaded models as well (Solaris comes to
> mind) so there's the possibility that a multi-threaded postgresql could
> enjoy better performance on more than just windows.

    As soon as you switch to a multi-threaded middle  tier,  that
    uses connection pools and avoids constant process creation, I
    think you don't have much of a problem  any  more.  The  high
    connect/disconnect  ratio, acceptable for threaded databases,
    is what kills PostgreSQL.

> If postgresql IS going to eventually be multi-threaded, then the whole
> win32 port should probably be delayed until then, since it would solve
> many of the issues of fork() versus createprocess().

    If multi-threading is the plan, then there is  light  at  the
    end of the tunnel ... the upcoming train...


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Thu, 9 May 2002 13:51:29 -0400 (EDT)
Jan Wieck <janwieck@yahoo.com>
Re: Issues tangential to win32 support
On Thu, 9 May 2002, Jan Wieck wrote:

> > If postgresql IS going to eventually be multi-threaded, then the whole
> > win32 port should probably be delayed until then, since it would solve
> > many of the issues of fork() versus createprocess().
> 
>     If multi-threading is the plan, then there is  light  at  the
>     end of the tunnel ... the upcoming train...

That's a bit extreme don't you think?  I'm not fan of multi-threading as 
the one true way, and since I use linux as my server for postgresql, there 
is no gain for me in a multi-threaded model.  In fact, I'd prefer 
postgresql stay multi-process for robustness.

BUT, if there are plans to go multi-thread, or could be plans, then those 
should take priority over how to port to windows, since making postgresql 
multi-threaded will change it so much as to make the current "how do we 
port to windows" thread meaningless.

One of the primary reasons given for avoiding cygwin is that postgresql 
runs so slowly under it on windows, but I submit that it probably won't 
run a heck of a lot faster if it was written as a native app as long as 
it's running as a multi-process design.  Since there's probably no great 
gain to be had from moving it out from under cygwin, why bother?

My vote would be to stay multi-process and Unix compatible.  I have no 
real use for windows as a server, and do NOT want to sacrifice the 
performance / reliability I have with postgresql under Linux for a windows 
port.




Thu, 9 May 2002 12:13:59 -0600 (MDT)
Scott Marlowe <scott.marlowe@ihs.com>
Re: Issues tangential to win32 support
cbbrowne@cbbrowne.com wrote:
> 
> > cbbrowne@cbbrowne.com wrote:
> >>> I think, and I know people are probably sick of me spouting
> >>> opinions, that if you want a Windows presence for PostgreSQL, then
> >>> we should write a real Win32 version.
> >>
> >> The crucial wrong word is the word "we."
> 
> >> If _you_ want a Windows presence, then _you_ should write a real
> >> Win32 version.  That clearly attaches responsibility to someone who
> >> is interested.
> 
> > I have already said that I am willing to write the pieces for a
> > Windows port.  The issue is changes in PostgreSQL required to do it.
> 
> No, I don't think you understand.
> 
> If you're planning to do a port, then _all_ changes are your
> responsibility.  Nobody ought to need to change PostgreSQL in order for
> you to write a Windows port; that, in fact, would be a waste of time,
> having several people working on something that should probably be done
> by one person.

Without buy-in from the group, there is no point in me wasting my time doing
all the work necessary. I'm not interested in making Mark's special version of
PostgreSQL.

If we can agree on a strategy and a course, then it is worth doing. If all the
changes made fall on the floor because the group does not like them, then I
wasted my time. Got it?

Also, doing the Windows portions of the code will represent a significant
investment of my time. I'm not interested in doing a lot of work on a shoddy
project. If you ask the core group to put out a crappy version of PostgreSQL
for a UNIX, they would fight long and hard against it. Why should we be willing
to produce a crappy version for Windows, just because the people here don't
like Windows.

I don't care about Solaris, but I understand WHY it is important to make
PostgreSQL work well on it. I don't understand why the people in this group
don't see the same purpose for a Windows port. To be honest, I think a good
Windows port will do wonders for PostgreSQL's acceptance.

Thu, 09 May 2002 14:23:02 -0400
mlw <markw@mohawksoft.com>
Re: How much work is a native Windows application?
cbbrowne@cbbrowne.com wrote:
> > cbbrowne@cbbrowne.com wrote:
> >>> I think, and I know people are probably sick of me spouting
> >>> opinions, that if you want a Windows presence for PostgreSQL, then
> >>> we should write a real Win32 version.
> >>
> >> The crucial wrong word is the word "we."
>
> >> If _you_ want a Windows presence, then _you_ should write a real
> >> Win32 version.  That clearly attaches responsibility to someone who
> >> is interested.
>
> > I have already said that I am willing to write the pieces for a
> > Windows port.  The issue is changes in PostgreSQL required to do it.
>
> No, I don't think you understand.
>
> If you're planning to do a port, then _all_ changes are your
> responsibility.  Nobody ought to need to change PostgreSQL in order for
> you to write a Windows port; that, in fact, would be a waste of time,
> having several people working on something that should probably be done
> by one person.

    Who  said PostgreSQL shall not support any other OS than *NIX
    or things that look alike?

    When I first used Postgres, it still had a VMS  port.   Well,
    we  dropped  the VMS port at some point, when we where pretty
    sure it was broken and nobody complained.

    Now we  face  the  fact  that  Microsoft  managed  to  create
    something  useful  (other than a joystick or optical mouse, I
    mean Win2K).  And as a  logical  consequence  more  and  more
    people ask for support of their OS.

    A  good  deal  of  effort  in the original Postgres was about
    portability.  I hope we've not become a UNIX-only show.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Thu, 9 May 2002 14:40:56 -0400 (EDT)
Jan Wieck <janwieck@yahoo.com>
Re: How much work is a native Windows application?
On Thu, 2002-05-09 at 22:36, mlw wrote:
> Scott Marlowe wrote:
> >  note
> > that many Unixes prefer multi-threaded models as well (Solaris comes to
> > mind) so there's the possibility that a multi-threaded postgresql could
> > enjoy better performance on more than just windows.
> 
> The isolation of a process is very important to reliable operation. Going
> threaded usually means allowing a single connection to bring down the whole
> server.

AFAIK we do that already in forked model - any time postmaster thinks
that a dying child has corrupted shared memory it kills all its
children.

--------
Hannu




10 May 2002 00:06:48 +0500
Hannu Krosing <hannu@tm.ee>
Re: Issues tangential to win32 support
On Thu, 2002-05-09 at 22:51, Jan Wieck wrote:
> Scott Marlowe wrote:
> > There are some issues that the whole idea of a win32 port should bring up.
> > One of them is whether or not postgresql should be rewritten as a
> > multi-threaded app.
> 
>     Please, don't add this one to it.
> 
>     I'm  all for the native Windows port, yes, but I've discussed
>     the multi-thread questions for days  at  Great  Bridge,  then
>     again with my new employer, with people on shows and whatnot.
> 
>     Anything in the whole  backend  is  designed  with  a  multi-
>     process  model in mind.  You'll not do that in any reasonable
>     amount of time.

IIRC you are replying to the man who _has_ actually don this ?

Perhaps using an unreasonable amount of time but still ... :)

----------
Hannu



10 May 2002 00:09:58 +0500
Hannu Krosing <hannu@tm.ee>
Re: Issues tangential to win32 support
On Fri, 2002-05-10 at 00:09, Hannu Krosing wrote:
> On Thu, 2002-05-09 at 22:51, Jan Wieck wrote:
> > Scott Marlowe wrote:
> > > There are some issues that the whole idea of a win32 port should bring up.
> > > One of them is whether or not postgresql should be rewritten as a
> > > multi-threaded app.
> > 
> >     Please, don't add this one to it.
> > 
> >     I'm  all for the native Windows port, yes, but I've discussed
> >     the multi-thread questions for days  at  Great  Bridge,  then
> >     again with my new employer, with people on shows and whatnot.
> > 
> >     Anything in the whole  backend  is  designed  with  a  multi-
> >     process  model in mind.  You'll not do that in any reasonable
> >     amount of time.
> 
> IIRC you are replying to the man who _has_ actually don this ?

Sorry, mistaken identity. I meant Myron scott ( mkscott@sacadia.com )

who has made this http://sourceforge.net/projects/mtpgsql

> Perhaps using an unreasonable amount of time but still ... :)
> 
> ----------
> Hannu
> 



10 May 2002 00:17:24 +0500
Hannu Krosing <hannu@tm.ee>
Re: Issues tangential to win32 support
On Thu, 2002-05-09 at 19:23, mlw wrote:
> Lee Kindness wrote:
> 
> > 
> > Sure It'd be nice to have a native PostgreSQL on XP Server (I don't
> > see the point in consumer level Microsoft OSs) but how high is the
> > demand? What's the prize? What are the current limitations - fork,
> > semaphores, ugly interface...?
> 
> The demand for PostgreSQL on Windows is currently as near to zero as you can
> imagine. This is probably because there is no viable PostgreSQL on Windows.
> 
> If written correctly, a Win32 version of PostgreSQL would rock the Windows
> world. I see no reason why it would be limted to the "professional" version.
> Hell, it could even run on Windows 98.

Perhaps we could simpultaneously solve another problem - creating a
singlethreaded embeddable version of postgresql "engine"

-------------
Hannu



10 May 2002 00:31:24 +0500
Hannu Krosing <hannu@tm.ee>
Re: How much work is a native Windows application?
On Thu, 2002-05-09 at 19:25, Tom Lane wrote:
> mlw <markw@mohawksoft.com> writes:
> > I have used the cygwin version too. It is a waste of time. No Windows user will
> > ever accept it. No windows-only user is going to use the cygwin tools.
> 
> With decent packaging, no windows-only user would even know we have
> cygwin in there.  The above argument is just plain irrelevant.  The real
> point is that we need a nice clean friendly GUI for both installation
> and administration --- and AFAICS that will take about the same amount of
> work to write whether the server requires cygwin internally or not.

<evil grin>
We can go the Oracle way and write a 200MB cross-platform java installer
requiring and exact version of java runtime 
</evil grin>

> Rather than expending largely-pointless work on internal rewrites of
> the server, people who care about this issue ought to be thinking about
> the GUI problems.

pgAccess is quite nice (Disclaimer: I'm not a windows weenie, I run it
inside vmware/win98 IE browser test environment on my Linux workstation
;). 

Why not just bundle what we've got ?

> > From a production stand point, would anyone reading this trust their
> > data to PostgreSQL running on cygwin?
> 
> I wouldn't trust my data to *any* database running on a Microsoft OS.
> Period. 

Do we support Xenix and SCO ?

> The above argument thus doesn't impress me at all, especially
> when it's being made without offering a shred of evidence that cygwin
> contributes any major degree of instability.

 From the comments here it seems to be either cygwin or more likely
cygipc

> I am especially unhappy about the prospect of major code revisions
> and development time spent on chasing this rather than improving our
> performance and stability on Unix-type OSes.  I agree with the comment
> someone else made: that's just playing Microsoft's game.

Not!

I think that this thread is mostly about coordinating code and interface
cleanups that are likely beneficial for both *NIX and non-*NIX platforms
mainly
 * cleaner support for semaphores
 * separating shared and per-process data
 * process creation
 * (file operations)
 * (init and service scripts)
if done properly none of these will degrade code quality nor
performance.

Also, having a clean interface for those will not only enable any
interested party to make windows/BeOS/OSX/QNX binaries with less effort,
it will most likely make it easier make use of advances in *NIX world
like AIO, multiprocessor systems, NUMA and distributed systems, and just
make things more robust and reliable by making code inspection easier.

---------------
Hannu



10 May 2002 00:53:27 +0500
Hannu Krosing <hannu@tm.ee>
Re: How much work is a native Windows application?
Hannu Krosing wrote:
> 
> On Thu, 2002-05-09 at 22:36, mlw wrote:
> > Scott Marlowe wrote:
> > >  note
> > > that many Unixes prefer multi-threaded models as well (Solaris comes to
> > > mind) so there's the possibility that a multi-threaded postgresql could
> > > enjoy better performance on more than just windows.
> >
> > The isolation of a process is very important to reliable operation. Going
> > threaded usually means allowing a single connection to bring down the whole
> > server.
> 
> AFAIK we do that already in forked model - any time postmaster thinks
> that a dying child has corrupted shared memory it kills all its
> children.

I know there are cases when postmaster will kill all its children, but take the
case of a faulty user function that gets a segfault. That process dies and the
others continue. Without a lot of OS specific crap in postgres, that sort of
behavior would be difficult to have with a threaded server.

Thu, 09 May 2002 17:09:56 -0400
mlw <markw@mohawksoft.com>
Re: Issues tangential to win32 support
Tom Lane <tgl@sss.pgh.pa.us> writes:

> mlw <markw@mohawksoft.com> writes:
> > I have used the cygwin version too. It is a waste of time. No Windows
user will
> > ever accept it. No windows-only user is going to use the cygwin tools.
>
> With decent packaging, no windows-only user would even know we have
> cygwin in there.  The above argument is just plain irrelevant.  The real
> point is that we need a nice clean friendly GUI for both installation
> and administration --- and AFAICS that will take about the same amount of
> work to write whether the server requires cygwin internally or not.

I'm afraid I agree with mlw, Tom. I don't think the problem ends at the GUI,
although for many people it would.  The issue extends at least also to
support and troubleshooting.  In a production environment, I have a better
chance of figuring out what's going wrong with an application written
natively for an operating system dealing directly with that operating
system. I would take a dim view of using PostgreSQL running on cygwin unless
I had extensive experience doing it, or if there were no other alternative.

> > From a production stand point, would anyone reading this trust their
> > data to PostgreSQL running on cygwin?
>
> I wouldn't trust my data to *any* database running on a Microsoft OS.
> Period.  The above argument thus doesn't impress me at all, especially
> when it's being made without offering a shred of evidence that cygwin
> contributes any major degree of instability.

If you could prove to me that cygwin doesn't contribute *any* instability,
I'd still be pretty worried, probably for the same reasons that you don't
trust any Microsoft OS. There are increased chances that something could go
critically wrong, particularly in an environment fundamentally different. I
think mlw's basic point is quite valid, that PG+cygwin will not ever find
favor with decision-makers who are used to Windows systems.  Suspicion of
the other environment's foibles is common, and goes both ways.

> I am especially unhappy about the prospect of major code revisions
> and development time spent on chasing this rather than improving our
> performance and stability on Unix-type OSes.  I agree with the comment
> someone else made: that's just playing Microsoft's game.

There I don't deny you may be right.

Ernie Gutierrez
Walnut Creek, CA


Thu, 9 May 2002 17:59:55 -0700
"Ernesto Gutierrez" <egutierrez@eldergriffon.org>
Re: How much work is a native Windows application? 
On Sat, 2002-05-11 at 02:25, mlw wrote:
> A binary version of PostgreSQL for Windows should not use the cygwin dll. I
> know and understand there is some disagreement with this position, but in this
> I'm sure about this.
 
...

> I believe we can use the cygwin development environment, and direct gcc not to
> link with the cygwin dll. Last time I looked it was a command line option. This
> will produce a native windows application. No emulation, just a standard C
> runtime.

It seems that mingw (http://www.mingw.org/) does exactly this and
provides needed headers/libs. And they have also non-cycwin minimal
build environment (MSYS) that supplies make,sh and other stuff we might
use for running initdb and other shell scripts.

> Some of the hits will be file path manipulation, '/' vs '\', the notion of
> drive letters, and case insensitivity in file names. 
> 
> Unicode may be an issue, I haven't looked at that yet. Is that a must for the
> initial release?

Probably not.

>> 
> A couple simple programs can be written using msvc to monitor, start and stop
> PostgreSQL. The programs will be simple using the application wizard, just make
> a small dialog box application.

dev-c++ has also wizards for easy making of trivial user interfaces

http://sourceforge.net/projects/dev-cpp/

--------------
Hannu



11 May 2002 00:48:12 +0500
Hannu Krosing <hannu@tm.ee>
Re: Native Win32, How about this?
A binary version of PostgreSQL for Windows should not use the cygwin dll. I
know and understand there is some disagreement with this position, but in this
I'm sure about this.

The tools used to create the binary need not be Microsoft, many venders have
used Borland or Watcom, the run of the mill user/developer does not care. The
developers who do care won't mind the cygwin development environment as long as
it produces a native Windows binary that does not play tricks such as fork().

Windows developers don't care too much about source code. The build environment
will not be a problem.

The issue is that the system must perform well and must be stable. I do not
believe that cygwin can meet this requirement. Having done some research for
these discussions, I think we know it has startup performance issues and
unknown operational issues.

FYI: My PHP project msession, can produce both a Windows version and a Cygwin
version. It is threaded C++, but I have measured a performance improvements
using the native Windows version over the cygwin version. I have since
abandoned the cygwin version.

I believe we can use the cygwin development environment, and direct gcc not to
link with the cygwin dll. Last time I looked it was a command line option. This
will produce a native windows application. No emulation, just a standard C
runtime.

Some of the hits will be file path manipulation, '/' vs '\', the notion of
drive letters, and case insensitivity in file names. 

Unicode may be an issue, I haven't looked at that yet. Is that a must for the
initial release?

There will be a need for some emulation/api specification of things like
semaphores, shared memory, file API (I would like to use Windows native
CreateFile routines, as these should be pretty fast.), and so on.

We will also have to breakup postgres and postmaster, and for the Windows
version use CreateProcess. There are a number of ways to attack this, globals
in a structure based in shared memory, globals in a .DLL exported to processes
and shared, and so on.

I think a huge time savings can be had by avoiding rewriting everything for the
Microsoft build environment. As far as I know, and please correct me if I'm
wrong, code produced by the cygwin gcc is freely distributable and need not be
GPL.

Once we have it working without fork() using the cygwin build environment, we
will have a native Windows application, we can then further evaluate whether or
not we want to expend the work to make a MSC version. 

Once the backend and most of the tools are built without requiring the
cygwin.dll, installation is a breeze. Just dump it somewhere and run it.

A couple simple programs can be written using msvc to monitor, start and stop
PostgreSQL. The programs will be simple using the application wizard, just make
a small dialog box application.

Pgaccess will provide all the GUI stuff, and we may even be able to wrap the
monitor code into pgaccess.

The server install can be done with install shield.

There is code that will run any program as an NT service. We can use that for
server installations. We can use the MSVC wizard application to pop-up in the
tool bar.

Have I missed anything?
Is this a realistic and attainable plan?

Fri, 10 May 2002 17:25:02 -0400
mlw <markw@mohawksoft.com>
Native Win32, How about this?
> A binary version of PostgreSQL for Windows should not use the cygwin
> dll. I know and understand there is some disagreement with this
> position, but in this I'm sure about this.

That may ultimately be desirable.

In the short term, it is likely preferable to use cygwin.

It is only necessary to point at MySQL for an example.  Cygwin is used there.
<http://www.mysql.com/downloads/mysql-3.23.html>  It is being used widely, 
"crap" or not.

Cygwin may not ultimately be the ideal thing to use; we don't yet live in 
Pangloss' "Best of All Possible Worlds," and thus have to live with some 
things not being ideal.

If having the installer install Cygwin as well as the DBMS makes it easy to 
have something usable soon, and this allows 100,000 WinFolk to try out 
PostgreSQL, then that's a Big Win.  Out of 100K users, surely two or three may 
be attracted into working on a more Panglossian solution.

It may be fair to say that none of those 100K folk would be using PostgreSQL 
to support HA applications involving hundreds of GB of data.  That's _fine_.

If there are new 100K folk using PostgreSQL/cygwin, _some_ of them will 
outgrow its capabilities, and come looking for improvements.

And as they're Windows users, accustomed to having to pay hefty amounts to 
Microsoft to get support no better than that provided by the Psychic Friends 
Network (see <http://www.bmug.org/news/articles/MSvsPF.html>), they'll 
doubtless be prepared to have to pay _something_ in order for 
"PostgreSQL/Win3K-Enterprise Edition" to become available.

That seems a not too unreasonable path towards the "Best of All Possible 
Worlds."  There may be a bit of hyperbole in the above, but any time Voltaire 
gets quoted, that's likely to happen :-).
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://www.cbbrowne.com/info/wp.html
Eagles may soar, but weasels don't get sucked into jet engines.

-- 
(concatenate 'string "cbbrowne" "@ntlug.org")
http://www.cbbrowne.com/info/multiplexor.html
It's a  little known fact  that the Dark  Ages were caused by  the Y1K
problem.
Sat, 11 May 2002 11:26:18 -0400
cbbrowne@cbbrowne.com
Re: Native Win32, How about this? 
On Thu, May 02, 2002 at 01:14:34PM -0300, Marc G. Fournier wrote:
> On 2 May 2002, Hannu Krosing wrote:
...
> > BTW, I think PostgreSQL does _not_ need any mission statement.
> 
> Nope, it doesn't ... never did before, don't know why it does suddenly ...
> do any other open source projects have one?  Its kinda fun to see what ppl
> banter around, but I can't see it being useful to adopt any single one,
> considering I can't see *everyone* agreeing with it ...

Quick - get out the Dilbert!

Patick

Sun, 12 May 2002 18:45:11 +0100
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: PostgreSQL mission statement?
mlw,

On Fri, May 10, 2002 at 05:25:02PM -0400, mlw wrote:
> A binary version of PostgreSQL for Windows should not use the cygwin
> dll. I know and understand there is some disagreement with this position,
> but in this I'm sure about this.

Sorry, but I'm not going to touch the above -- even with a ten foot pole.
Or, at least try not to... :,)

> I believe we can use the cygwin development environment, and direct gcc
> not to link with the cygwin dll. Last time I looked it was a command line
> option. This will produce a native windows application. No emulation,
> just a standard C runtime.

Yes, the above mentioned option is "-mno-cygwin".

> Some of the hits will be file path manipulation, '/' vs '\', the notion of
> drive letters, and case insensitivity in file names. 

Case insensitivity is typically "enabled" regardless.  Unless you are
referring to CYGWIN=check_case:strict, but almost no one uses this setting.

Just to be explicit, another hit will be the loss of Posix.

> [snip]
> 
> I think a huge time savings can be had by avoiding rewriting everything
> for the Microsoft build environment.

Yes, you should use Cygwin and gcc -mno-cygwin or MSYS and Mingw.

> As far as I know, and please correct me if I'm wrong, code produced by
> the cygwin gcc is freely distributable and need not be GPL.

The above is true only with gcc -mno-cygwin or Mingw code.  Any code
produced by the normal Cygwin gcc (and hence, linked against cygwin1.dll)
is effectively GPL'd or at least required to be open source.

> [snip]

Jason

Mon, 13 May 2002 09:59:04 -0400
Jason Tishler <jason@tishler.net>
Re: Native Win32, How about this?
Hi, 

Something is pretty broken in redhat 7.3 but I'm not sure what and I
don't have time to dig further

masm@test=# select cast('1967-04-18' as timestamptz);
      timestamptz
------------------------
 1967-04-17 18:00:00-06
(1 row)

masm@test=# select cast(cast('1967-04-18' as date) as timestamp);
ERROR:  Unable to convert date to tm
masm@test=#

Both cases works correctly in Redhat 7.2. Sorry if this is not the
correct forum however an alert is nice for people planning an Redhat
upgrade. I hope to see pretty soon an update since I don't want to
downgrade my server.

Regards,
Manuel.

17 May 2002 23:34:42 -0500
Manuel Sugawara <masm@fciencias.unam.mx>
Redhat 7.3 time manipulation bug
On Saturday 18 May 2002 12:34 am, Manuel Sugawara wrote:
> Something is pretty broken in redhat 7.3 but I'm not sure what and I
> don't have time to dig further

> Both cases works correctly in Redhat 7.2. Sorry if this is not the
> correct forum however an alert is nice for people planning an Redhat
> upgrade. I hope to see pretty soon an update since I don't want to
> downgrade my server.

Filed on Red Hat's Bugzilla system as bug# 65227, as I can reliably reporoduce 
this bug here, and PostgreSQL 7.2.1 on Red Hat 6.2 on SPARC does not exhibit 
the bug.

Tom (or Thomas):

Where would we go to ferret out the source of this bug?  More to the point: we 
need a test case in C that could expose this as a glibc bug.  Methinks Red 
Hat would want this bug ferretted out, as it would likely cause problems with 
RedHat Database on RH 7.3's glibc.....
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 20 May 2002 13:58:12 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Redhat 7.3 time manipulation bug
Lamar Owen <lamar.owen@wgcr.org> writes:

> Filed on Red Hat's Bugzilla system as bug# 65227, as I can reliably
> reporoduce this bug here, and PostgreSQL 7.2.1 on Red Hat 6.2 on
> SPARC does not exhibit the bug.

Thanks for filling that report. I couldn't remember what had forgotten
;-)

Regards,
Manuel.

20 May 2002 13:22:14 -0500
Manuel Sugawara <masm@fciencias.unam.mx>
Re: Redhat 7.3 time manipulation bug
Lamar Owen <lamar.owen@wgcr.org> writes:

> Where would we go to ferret out the source of this bug?  More to the
> point: we need a test case in C that could expose this as a glibc
> bug.  

Seems like mktime(3) is having problems with dates before the
epoch. Attached is the a program to test this. The glibc source is now
downloading I will try to hunt down this bug but not until the next
week.

Regards,
Manuel.
20 May 2002 19:08:07 -0500
Manuel Sugawara <masm@fciencias.unam.mx>
Re: Redhat 7.3 time manipulation bug
On Monday 20 May 2002 08:08 pm, Manuel Sugawara wrote:
> > Where would we go to ferret out the source of this bug?  More to the
> > point: we need a test case in C that could expose this as a glibc
> > bug.

> Seems like mktime(3) is having problems with dates before the
> epoch. Attached is the a program to test this. The glibc source is now
> downloading I will try to hunt down this bug but not until the next
> week.

It's not a bug.  At least not according to the ISO C standard.  See 
http://www.opengroup.org/onlinepubs/007904975/basedefs/xbd_chap04.html#tag_04_14 
for the definition of 'Seconds Since the Epoch', then cross-reference to the 
man page of mktime.

I don't like it any more than you do, but that is the letter of the standard.  

Thomas, any comments?

Our implementation is broken, then.  Thomas, is this fixable for a 7.2.x 
release, or something for 7.3?
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 20 May 2002 23:12:21 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Redhat 7.3 time manipulation bug
...
> Our implementation is broken, then.  Thomas, is this fixable for a 7.2.x
> release, or something for 7.3?

"Our implementation is broken, then" is really not a conclusion to be
reached. The de facto behavior of mktime() on all world-class
implementations is to support pre-1970 times. This has been true forever
afaik, certainly far longer than PostgreSQL (or Postgres) has been in
existance.

Any standard which chooses to ignore pre-1970 dates is fundamentally
broken imho, and I'm really ticked off that the glibc folks have so
glibly introduced a change of this nature and magnitude without further
discussion.

                         - Thomas

Tue, 21 May 2002 06:30:06 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: Redhat 7.3 time manipulation bug
Lamar Owen <lamar.owen@wgcr.org> writes:

> On Monday 20 May 2002 08:08 pm, Manuel Sugawara wrote:
> > > Where would we go to ferret out the source of this bug?  More to the
> > > point: we need a test case in C that could expose this as a glibc
> > > bug.
> 
> > Seems like mktime(3) is having problems with dates before the
> > epoch. Attached is the a program to test this. The glibc source is now
> > downloading I will try to hunt down this bug but not until the next
> > week.
> 
> It's not a bug.  At least not according to the ISO C standard.  See
> http://www.opengroup.org/onlinepubs/007904975/basedefs/xbd_chap04.html#tag_04_14
> for the definition of 'Seconds Since the Epoch', then
> cross-reference to the man page of mktime.

I see. This behavior is consistent with the fact that mktime is
supposed to return -1 on error, but then is broken in every other Unix
implementation that I know.

Any other workaround than downgrade or install FreeBSD?

Regards,
Manuel.

21 May 2002 10:04:00 -0500
Manuel Sugawara <masm@fciencias.unam.mx>
Re: Redhat 7.3 time manipulation bug
On Tuesday 21 May 2002 11:04 am, Manuel Sugawara wrote:
> I see. This behavior is consistent with the fact that mktime is
> supposed to return -1 on error, but then is broken in every other Unix
> implementation that I know.

> Any other workaround than downgrade or install FreeBSD?

Complain to Red Hat.  Loudly. However, as this is a glibc change, other 
distributors are very likely to fold in this change sooner rather than later.

Try using timestamp without timezone?
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 21 May 2002 12:29:45 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Redhat 7.3 time manipulation bug
On Tue, 21 May 2002, Lamar Owen wrote:

> On Tuesday 21 May 2002 11:04 am, Manuel Sugawara wrote:
> > I see. This behavior is consistent with the fact that mktime is
> > supposed to return -1 on error, but then is broken in every other Unix
> > implementation that I know.
> 
> > Any other workaround than downgrade or install FreeBSD?
> 
> Complain to Red Hat.  Loudly. However, as this is a glibc change, other 
> distributors are very likely to fold in this change sooner rather than 
> later. 

Relying on nonstandardized/nondocumented behaviour is a program bug, not a 
glibc bug. PostgreSQL needs fixing. Since we ship both, we're looking at 
it, but glibc is not the component with a problem.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.


Tue, 21 May 2002 12:31:14 -0400 (EDT)
=?ISO-8859-1?Q?Trond_Eivind_Glomsr=F8d?= <teg@redhat.com>
Re: Redhat 7.3 time manipulation bug
On Tuesday 21 May 2002 12:31 pm, Trond Eivind Glomsrï¿½d wrote:
> On Tue, 21 May 2002, Lamar Owen wrote:
> > However, as this is a glibc change, other
> > distributors are very likely to fold in this change sooner rather than
> > later.

> Relying on nonstandardized/nondocumented behaviour is a program bug, not a
> glibc bug. PostgreSQL needs fixing. Since we ship both, we're looking at
> it, but glibc is not the component with a problem.

In your opinion.  Not everyone agrees with the new glibc behavior.  In fact, 
some here are rather livid about it.  But that's a digression.  The matter at 
hand is making it work right again.  

It seems to me someone should have thought about this before making the glibc 
change that had the potential for breaking a large application -- regardless 
of who is at fault, it's egg in Red Hat's face for not catching it sooner 
(and egg in my face as well, as I must admit that I of all people should have 
caught this earlier).  

Is the change in glibc behavior noted in the release notes?  The man page 
isn't changed either, from Red Hat 6.2, in fact.  The only change is adhering 
to the ISO definition of 'Seconds Since the Epoch' rather than the defacto 
industry-accepted definition that has been with us a very long time.  

Like PostgreSQL's refusal to upgrade in a sane manner, this should have 
received release notes billing, IMHO.  Then I, nor anyone else, could 
reasonably complain.

But this does show the need for the regression testing packge, no? :-) And it 
also shows the danger in becoming too familiar with certain regression tests 
failing due to locale -- to the extent that a locale issue was the first 
thing thought of.  To the extent that I'm going to change my build process to 
include regression testing as a part of the build -- and any failure will 
abort the build. Maybe that will get my attention.  And anyone else's 
rebuilding from the source RPM.  SuSE already does this.  I wonder how 
they've handled this issue with 8.0? 

In any case, this isn't just a Red Hat problem, as it's going to cause 
problems with the use of timestamps on ANY glibc 2.2.5 dist.  That's more 
than Red Hat, by a large margin.

And I think that it is naive to think that PostgreSQL is the only program that 
has used mktime's behavior in the negative-time_t zone.  Other packages are 
likely impacted, to a greater or lesser extent.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 21 May 2002 13:24:41 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Redhat 7.3 time manipulation bug
On Tue, 2002-05-21 at 21:31, Trond Eivind Glomsrød wrote:
> On Tue, 21 May 2002, Lamar Owen wrote:
> 
> > On Tuesday 21 May 2002 11:04 am, Manuel Sugawara wrote:
> > > I see. This behavior is consistent with the fact that mktime is
> > > supposed to return -1 on error, but then is broken in every other Unix
> > > implementation that I know.
> > 
> > > Any other workaround than downgrade or install FreeBSD?
> > 
> > Complain to Red Hat.  Loudly. However, as this is a glibc change, other 
> > distributors are very likely to fold in this change sooner rather than 
> > later. 
> 
> Relying on nonstandardized/nondocumented behaviour is a program bug, not a 
> glibc bug. PostgreSQL needs fixing. Since we ship both, we're looking at 
> it, but glibc is not the component with a problem.

Still it seems kind of silly to have a function that works different
from all other implementations and forces people to use their own
function of the same name (lifted from BSD and also compliant).

Speaking of nonstandardized/nondocumented behaviour, I read from "The
Open Sources" book that if you implement TCP/IP strictly from the RFCs
then it won't interoperate with any other TCP/IP stack. 

I hope that Red Hat is not going to be "standards compliant" here ;)

--------------
Hannu



21 May 2002 23:54:06 +0500
Hannu Krosing <hannu@tm.ee>
Re: Redhat 7.3 time manipulation bug
Trond Eivind Glomsrød <teg@redhat.com> writes:

> Relying on nonstandardized/nondocumented behaviour is a program bug,
> not a glibc bug.

The question is: how this thing didn't show up before? ISTM that
someone is not doing his work correctly.

> PostgreSQL needs fixing.

Arguably, however, right now is *a lot easier* to fix glibc, and it's
really needed for production systems using postgreSQL and working on
RedHat. But redhat users doesn't matter, the most important thing is
*strict* conformace to standars, right?

> Since we ship both, we're looking at it, but glibc is not the
                      ^^^^^^^^^^^^^^^^^^^
The sad true is: you only answered when the 'Complain to Red Hat'
statement appeared, not a single word before and not a single word
when the bug report were closed. I'm really disappointed.

The nice thing is: glibc is free software and we don't have to wait or
relay on some of the redhat staff members (thanks god) for this to get
fixed or say: for the standard to get extended again. The patch to
glibc is pretty straightforward and attached.

Regards,
Manuel.
21 May 2002 13:59:39 -0500
Manuel Sugawara <masm@fciencias.unam.mx>
Re: Redhat 7.3 time manipulation bug
On 21 May 2002, Manuel Sugawara wrote:

> Trond Eivind Glomsrï¿½d <teg@redhat.com> writes:
> 
> > Relying on nonstandardized/nondocumented behaviour is a program bug,
> > not a glibc bug.
> 
> The question is: how this thing didn't show up before? ISTM that
> someone is not doing his work correctly.

FWIW, I ran the regressions tests some time ago(probably before that 
change to glibc) . Since the tests are known 
to be broken wrt. time issues anyway (as well as currency, math and sorting), 
it's easy to overlook.

> > PostgreSQL needs fixing.
> 
> Arguably, however, right now is *a lot easier* to fix glibc, and it's
> really needed for production systems using postgreSQL and working on
> RedHat. 

You're not "fixing" glibc, you're reintroducing non-standardized, upstream 
removed behaviour. That's typically a very bad thing. If anything, it 
demonstrates the importance of not using or relying on 
unstandardized/undocumented behaviour (and given that time_t is pretty 
restrictive anyway, you'll need something else to keep dates. It doesn't 
even cover all living people, and definitely not historical dates).

> > Since we ship both, we're looking at it, but glibc is not the
>                       ^^^^^^^^^^^^^^^^^^^
> The sad true is: you only answered when the 'Complain to Red Hat'
> statement appeared, not a single word before and not a single word
> when the bug report were closed. I'm really disappointed.

The bug wasn't open for long, and was closed by someone else.

> The nice thing is: glibc is free software 

Also, notice that this was where the fix came from: The upstream 
maintainers (some of whom work for us, others don't).

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.


Tue, 21 May 2002 15:09:34 -0400 (EDT)
=?ISO-8859-1?Q?Trond_Eivind_Glomsr=F8d?= <teg@redhat.com>
Re: Redhat 7.3 time manipulation bug
On Tuesday 21 May 2002 03:09 pm, Trond Eivind Glomsrï¿½d wrote:
> FWIW, I ran the regressions tests some time ago(probably before that
> change to glibc) . Since the tests are known
> to be broken wrt. time issues anyway (as well as currency, math and
> sorting), it's easy to overlook.

The time tests have never broken in this manner before on Red Hat.  When the 
original regression failure report was posted, I saw right away that this was 
not the run of the mill locale issue -- this was a real problem.  Regression 
testing must become a regularly scheduled activity, methinks.  In the RPM 
build process, we can control the locale to the extent that the tests will 
pass (except on DST days) reliably.  I am going to implement this for my next 
RPM set.  Along with a patch to this problem -- we _can_ patch around this, I 
believe, but it's not likely going to be an easy one.

We have gotten blind to the regular locale-induced failures -- this is not a 
good thing.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Tue, 21 May 2002 15:44:40 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Redhat 7.3 time manipulation bug
=?ISO-8859-1?Q?Trond_Eivind_Glomsr=F8d?= <teg@redhat.com> writes:
> Relying on nonstandardized/nondocumented behaviour is a program bug, not a 
> glibc bug. PostgreSQL needs fixing. Since we ship both, we're looking at 
> it, but glibc is not the component with a problem.

A library that can no longer cope with dates before 1970 is NOT my idea
of a component without a problem.  We will be looking at ways to get
around glibc's breakage at the application level, since we have little
alternative other than to declare Linux an unsupported platform;
but it's still glibc (and the ISO spec:-() that are broken.

			regards, tom lane

Tue, 21 May 2002 17:14:13 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Redhat 7.3 time manipulation bug 
On Tue, 2002-05-21 at 18:24, Lamar Owen wrote:
> In any case, this isn't just a Red Hat problem, as it's going to cause 
> problems with the use of timestamps on ANY glibc 2.2.5 dist.  That's more 
> than Red Hat, by a large margin.

I'm running glibc 2.2.5 on Debian and all regression tests pass OK (with
make check).  I don't see any note in the glibc Debian changelog about
reversing an upstream change to mktime().

I missed the first messages in this thread and I can't find them in the
archive.  What should I be looking for to see if I have the problem you
have encountered or to see why I don't have it if I ought to have?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C

     "O come, let us worship and bow down; let us kneel
      before the LORD our maker."          Psalms 95:6
21 May 2002 23:09:58 +0100
Oliver Elphick <olly@lfix.co.uk>
Re: Redhat 7.3 time manipulation bug
On Tuesday 21 May 2002 06:09 pm, Oliver Elphick wrote:
> On Tue, 2002-05-21 at 18:24, Lamar Owen wrote:
> > In any case, this isn't just a Red Hat problem, as it's going to cause
> > problems with the use of timestamps on ANY glibc 2.2.5 dist.  That's more
> > than Red Hat, by a large margin.

> I'm running glibc 2.2.5 on Debian and all regression tests pass OK (with
> make check).  I don't see any note in the glibc Debian changelog about
> reversing an upstream change to mktime().

> I missed the first messages in this thread and I can't find them in the
> archive.  What should I be looking for to see if I have the problem you
> have encountered or to see why I don't have it if I ought to have?

Hmmm.  Compile and run the attached program.  If you get -1, it's the new 
behavior.  It might be interesting to see the differences here.....
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11
Tue, 21 May 2002 18:47:05 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Redhat 7.3 time manipulation bug
Manuel Sugawara <masm@fciencias.unam.mx> writes:
 
> +#if 0
>    /* Only years after 1970 are defined.
>       If year is 69, it might still be representable due to
>       timezone differnces.  */
>    if (year < 69)
>      return -1;
> +#endif

Hm.  If that fixes it, it implies that all the other support for
pre-1970 dates is still there (notably, entries in the timezone tables).

Should we assume that future glibc releases will rip out the timezone
database entries and other support for pre-1970 dates?  Or is the
breakage going to stop with mktime?

			regards, tom lane

Tue, 21 May 2002 19:33:20 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Redhat 7.3 time manipulation bug 
Lamar Owen writes:

> SuSE already does this.  I wonder how they've handled this issue with
> 8.0?

Their glibc doesn't have that problem.

Personally, I think if you need time (zone) support before 1970, obtain
one of the various operating systems that support it.  There's little
value in hacking around it in PostgreSQL, since the rest of your system
will be broken as well.

-- 
Peter Eisentraut   peter_e@gmx.net


Wed, 22 May 2002 03:09:11 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Redhat 7.3 time manipulation bug
> > SuSE already does this.  I wonder how they've handled this issue with
> > 8.0?
> Their glibc doesn't have that problem.

My strong recollection is that a SuSE guy was the one applying the
change. So this is coming to those systems too. I may not remember that
correctly though...

> Personally, I think if you need time (zone) support before 1970, obtain
> one of the various operating systems that support it.  There's little
> value in hacking around it in PostgreSQL, since the rest of your system
> will be broken as well.

Yes, I'm afraid I agree. In practice, maybe most applications won't
notice. But after getting the Linux time zone databases set up to be
better than most (Solaris has the best I've found for fidelity to
pre-1970 year-to-year conventions) throwing that work away is just plain
silly. I consider this a major gaff on the part of the commercial Linux
houses to not see this coming and to contribute to a better solution.

                     - Thomas

Tue, 21 May 2002 19:01:55 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: Redhat 7.3 time manipulation bug
On Tue, 2002-05-21 at 23:47, Lamar Owen wrote:
> Hmmm.  Compile and run the attached program.  If you get -1, it's the new 
> behavior.  It might be interesting to see the differences here.....

$ a.out
The system thinks 11/30/1969 is a timestamp of -176400 
$ dpkg -l libc6
...
||/ Name           Version        Description
+++-==============-==============-============================================
ii  libc6          2.2.5-6        GNU C Library: Shared libraries and Timezone

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C

     "We are troubled on every side, yet not distressed; we 
      are perplexed, but not in despair; persecuted, but not
      forsaken; cast down, but not destroyed; Always bearing
      about in the body the dying of the Lord Jesus, that 
      the life also of Jesus might be made manifest in our 
      body."        II Corinthians 4:8-10
22 May 2002 05:18:43 +0100
Oliver Elphick <olly@lfix.co.uk>
Re: Redhat 7.3 time manipulation bug
On Wed, 2002-05-22 at 02:14, Tom Lane wrote:
> =?ISO-8859-1?Q?Trond_Eivind_Glomsr=F8d?= <teg@redhat.com> writes:
> > Relying on nonstandardized/nondocumented behaviour is a program bug, not a 
> > glibc bug. PostgreSQL needs fixing. Since we ship both, we're looking at 
> > it, but glibc is not the component with a problem.
> 
> A library that can no longer cope with dates before 1970 is NOT my idea
> of a component without a problem.  We will be looking at ways to get
> around glibc's breakage at the application level, since we have little
> alternative other than to declare Linux an unsupported platform;
> but it's still glibc (and the ISO spec:-() that are broken.

IIRC the spec is not _really_ broken - it still allows the correct
behaviour :)

The fact the ISO spec is broken usually means that at least one of the
big vendors involved in ISO spec creation must have had a broken
implementation at that time.

Most likely they have fixed it by now ...

Does anyone know _any_ other libc that has this behaviour ?

--------------
Hannu



22 May 2002 09:33:45 +0500
Hannu Krosing <hannu@tm.ee>
Re: Redhat 7.3 time manipulation bug
> IIRC the spec is not _really_ broken - it still allows the correct
> behaviour :)

Yes.

> The fact the ISO spec is broken usually means that at least one of the
> big vendors involved in ISO spec creation must have had a broken
> implementation at that time.

Right. IBM.

> Most likely they have fixed it by now ...

Nope, though I don't know for sure. Anyone here have a recent AIX
machine to test?

> Does anyone know _any_ other libc that has this behaviour ?

AIX and (I think) Irix.

Trond, do you have a suggestion on how to get this addressed at the
glibc level? Does someone within RH participate in glibc development? If
so, can we get them to champion changes which would comply with the
standard but remove this arbitrary breakage?

The changes would involve returning -1 from mktime() for dates before
1970, and using the tm_isdst flag to indicate whether a time zone
translation was not possible.

                   - Thomas

Wed, 22 May 2002 06:30:05 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: Redhat 7.3 time manipulation bug
Thomas Lockhart writes:
 > Right. IBM.
 > > Most likely they have fixed it by now ...
 > Nope, though I don't know for sure. Anyone here have a recent AIX
 > machine to test?

Well on AIX 4.3.3 the output from  Lamar's earlier test program is:

 The system thinks 11/30/1969 is a timestamp of -1

and tm_isdst is left at -1...

I could boot the machine into 5.0 too, but going by the AIX 5L
manpages it still returns -1:

 Note: The mktime subroutine cannot convert time values before
 00:00:00 UTC, January 1, 1970 and after 03:14:07 UTC, January 19,
 2038.

And getting an Irix 5.3 box up and running would be a chore!

Lee.

Wed, 22 May 2002 14:57:25 +0100
Lee Kindness <lkindness@csl.co.uk>
Re: Redhat 7.3 time manipulation bug
...
> > AIX and (I think) Irix.
> How do we currently support AIX/Irix ?

Dates and times prior to 1970 have no time zone (they are in GMT, as is
the case for all platforms on dates before 1903 and after 2038). We have
separate regression test results for those platforms.

                          - Tom

Wed, 22 May 2002 06:58:06 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: Redhat 7.3 time manipulation bug
On Wed, 22 May 2002, Thomas Lockhart wrote:

> > IIRC the spec is not _really_ broken - it still allows the correct
> > behaviour :)
> 
> Yes.
> 
> > The fact the ISO spec is broken usually means that at least one of the
> > big vendors involved in ISO spec creation must have had a broken
> > implementation at that time.
> 
> Right. IBM.
> 
> > Most likely they have fixed it by now ...
> 
> Nope, though I don't know for sure. Anyone here have a recent AIX
> machine to test?
> 
> > Does anyone know _any_ other libc that has this behaviour ?
> 
> AIX and (I think) Irix.
> 
> Trond, do you have a suggestion on how to get this addressed at the
> glibc level? Does someone within RH participate in glibc development?

Jakub Jelinek, Ulrich Drepper and others.

> If so, can we get them to champion changes which would comply with the
> standard but remove this arbitrary breakage?

Unlikely. They already saw (and participated, at least Ulrich) a thread on 
this with Lamar. Their take is "this is the  standard, you should do what 
the standard says and not rely on  
undocumented, non-standardized sideeffects.

> The changes would involve returning -1 from mktime() for dates before
> 1970, and using the tm_isdst flag to indicate whether a time zone
> translation was not possible.

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.


Wed, 22 May 2002 10:25:02 -0400 (EDT)
=?ISO-8859-1?Q?Trond_Eivind_Glomsr=F8d?= <teg@redhat.com>
Re: Redhat 7.3 time manipulation bug
On Wed, 2002-05-22 at 15:30, Thomas Lockhart wrote:
> > IIRC the spec is not _really_ broken - it still allows the correct
> > behaviour :)
> 
> Yes.
> 
> > The fact the ISO spec is broken usually means that at least one of the
> > big vendors involved in ISO spec creation must have had a broken
> > implementation at that time.
> 
> Right. IBM.
> 
> > Most likely they have fixed it by now ...
> 
> Nope, though I don't know for sure. Anyone here have a recent AIX
> machine to test?
> 
> > Does anyone know _any_ other libc that has this behaviour ?
> 
> AIX and (I think) Irix.

How do we currently support AIX/Irix ?

----------------
Hannu



22 May 2002 16:48:54 +0200
Hannu Krosing <hannu@tm.ee>
Re: Redhat 7.3 time manipulation bug
On Wed, 2002-05-22 at 08:00, Thomas Lockhart wrote:
> ...
> > > If so, can we get them to champion changes which would comply with the
> > > standard but remove this arbitrary breakage?
> > Unlikely. They already saw (and participated, at least Ulrich) a thread on
> > this with Lamar. Their take is "this is the  standard, you should do what
> > the standard says and not rely on undocumented, non-standardized sideeffects.
> 
> OK. They must be new guys.

:-) Very funny.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------
22 May 2002 10:12:31 -0700
Ulrich Drepper <drepper@redhat.com>
Re: Redhat 7.3 time manipulation bug
On Wednesday 22 May 2002 01:12 pm, Ulrich Drepper wrote:
> On Wed, 2002-05-22 at 08:00, Thomas Lockhart wrote:
> > > > If so, can we get them to champion changes which would comply with
> > > > the standard but remove this arbitrary breakage?

> > > Unlikely. They already saw (and participated, at least Ulrich) a thread
> > > on this with Lamar. Their take is "this is the  standard, you should do
> > > what the standard says and not rely on undocumented, non-standardized
> > > sideeffects.

> > OK. They must be new guys.

> :-) Very funny.

What isn't funny is Oliver Elphick's results on Debian, running glibc 2.2.5 
(same as Red Hat 7.3's version).  They are different.  And, IMO, those 
results are the 'expected' results on a unixoid system, ISO or no ISO.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Wed, 22 May 2002 13:51:20 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Redhat 7.3 time manipulation bug
On Wed, 2002-05-22 at 10:51, Lamar Owen wrote:

> What isn't funny is Oliver Elphick's results on Debian, running glibc 2.2.5 
> (same as Red Hat 7.3's version).

This is a completely different version.  Once Debian updates (in a few
years) they'll get the same result.

If you are misusing interfaces you get what you deserve.  At no time was
it correct to use these functions for general date manipulation.  It
always only was allowed to use them to represent system times and there
was no Unix system before the epoch.  Therefore you argumentation is
completely wrong.

If you need date manipulation write your own code which work for all the
times you want to represent.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------
22 May 2002 10:58:15 -0700
Ulrich Drepper <drepper@redhat.com>
Re: Redhat 7.3 time manipulation bug
Ulrich Drepper <drepper@redhat.com> writes:
> If you are misusing interfaces you get what you deserve.  At no time was
> it correct to use these functions for general date manipulation.  It
> always only was allowed to use them to represent system times and there
> was no Unix system before the epoch.  Therefore you argumentation is
> completely wrong.

> If you need date manipulation write your own code which work for all the
> times you want to represent.

We may indeed end up doing that, if glibc fails to provide the
functionality we need; but your argument is nonsense.  Unix systems have
*always* interpreted time_t as a signed offset from the epoch.  Do you
really think that when Unixen were first built in the early 70s, there
was no interest in working with pre-1970 dates?  Hardly likely.

glibc has just taken a large step backwards by comparison to every other
libc on the planet.  Claiming that you are okay because you conform to
a lowest-common-denominator ISO spec is beside the point.  ISO specs are
only pieces of paper.  You have removed functionality that is de facto
standard, important in practice, and *provided by most of your
competition*.  People will start going to the competition.  Or perhaps
there will be a glibc fork.

Postgres is not the only application that will be complaining loudly
about this change; anyone who has to deal with historical information
is going to be unhappy.  We just happen to be the earliest bearers of
the bad news.  But you will end up reverting this change due to pushback
from users.  Want to make a side bet?

			regards, tom lane

Wed, 22 May 2002 14:23:54 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Redhat 7.3 time manipulation bug 
On Wed, 2002-05-22 at 11:23, Tom Lane wrote:

> Unix systems have
> *always* interpreted time_t as a signed offset from the epoch.

No.  This always was an accident if it happens.

> Do you
> really think that when Unixen were first built in the early 70s, there
> was no interest in working with pre-1970 dates?  Hardly likely.

There never were files or any system events with these dates.  Yes.

And just to educate you and your likes: the majority of systems on this
planet use mktime this way.  I hate using this as an argument, but
beside major Unixes M$ systems also do this.

> But you will end up reverting this change due to pushback
> from users.  Want to make a side bet?

Sure.  Especially not everybody is that stubborn.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------
22 May 2002 11:40:58 -0700
Ulrich Drepper <drepper@redhat.com>
Re: Redhat 7.3 time manipulation bug
On Wednesday 22 May 2002 01:58 pm, Ulrich Drepper wrote:
> On Wed, 2002-05-22 at 10:51, Lamar Owen wrote:
> > What isn't funny is Oliver Elphick's results on Debian, running glibc
> > 2.2.5 (same as Red Hat 7.3's version).

> This is a completely different version.  Once Debian updates (in a few
> years) they'll get the same result.

A completely different version with the same version number?  Or is this a 
case of a Red Hat version number really meaning something different   
Shouldn't glibc 2.2.5 be the same as glibc 2.2.5 regardless of distribution?

And who's to stop them patching out the new stuff and reinstating the old 
behavior? :-)

> If you are misusing interfaces you get what you deserve.  At no time was
> it correct to use these functions for general date manipulation.  It
> always only was allowed to use them to represent system times and there
> was no Unix system before the epoch.  Therefore you argumentation is
> completely wrong.

If it is completely wrong, then tell Sun, HP, and all the rest of the Unix 
vendors, including the authors of the original AT&T code as lifted by 
Berkeley, that they're wrong and you're right.  They'll laugh you to scorn.

And just which 'major Unixen' other than AIX and Irix that follow the letter 
of the 'seconds since the epoch' definition of the ISO standard?

> If you need date manipulation write your own code which work for all the
> times you want to represent.

The mktime bug doesn't effect our representation of dates and times other than 
the timezone at this time.  What's aggravating to me is the surprise factor.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Wed, 22 May 2002 14:58:38 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Redhat 7.3 time manipulation bug
This thread is getting pretty annoying rather than constructive. By
the mean time I can see the users of many db's running under linux
loudly complaining.

As a user of both products (glibc and postgres), I would like to see a
good compromise in both sides. For instance: postgreSQL will implement
it's own time engine in one or to releases (In a few months I can do
it if no one else volunteers) and glibc will revert this change for
one or two releases (I can do it right now). That way everyone will be
happy; particularly me and many others using glibc and postgres.

Regards,
Manuel.

22 May 2002 14:09:18 -0500
Manuel Sugawara <masm@fciencias.unam.mx>
Re: Redhat 7.3 time manipulation bug
> > OK. They must be new guys.
> :-) Very funny.

Hey, it worked. Got you out of the woodwork. :))

                   - Thomas

Wed, 22 May 2002 17:20:21 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: Redhat 7.3 time manipulation bug
> On Wed, 2002-05-22 at 15:30, Thomas Lockhart wrote:
> > > IIRC the spec is not _really_ broken - it still allows the correct
> > > behaviour :)
> > 
> > Yes.
> > 
> > > The fact the ISO spec is broken usually means that at least one of the
> > > big vendors involved in ISO spec creation must have had a broken
> > > implementation at that time.
> > 
> > Right. IBM.
> > 
> > > Most likely they have fixed it by now ...
> > 
> > Nope, though I don't know for sure. Anyone here have a recent AIX
> > machine to test?
> > 
> > > Does anyone know _any_ other libc that has this behaviour ?
> > 
> > AIX and (I think) Irix.
> 
> How do we currently support AIX/Irix ?

Why should we rely on broken glibc and the standard? Why don't we make
our own mktime() and use it on all platforms.
--
Tatsuo Ishii

Thu, 23 May 2002 10:14:01 +0900 (JST)
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Redhat 7.3 time manipulation bug
...
> Why should we rely on broken glibc and the standard? Why don't we make
> our own mktime() and use it on all platforms.

The downside to doing that is that we then take over maintenance of the
code and, more importantly, the timezone database.

But it might be the best thing to do. It looks like the zic package
pointed to the other day could be used, though we may have to change
some of the variables and entry points to avoid library conflicts. But
we could also blow past the y2038 limit afaict which would be nice.

                    - Thomas

Wed, 22 May 2002 18:51:07 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: Redhat 7.3 time manipulation bug
Thomas Lockhart <lockhart@fourpalms.org> writes:
>> Why should we rely on broken glibc and the standard? Why don't we make
>> our own mktime() and use it on all platforms.

> The downside to doing that is that we then take over maintenance of the
> code and, more importantly, the timezone database.

> But it might be the best thing to do.

I've been sorta thinking the same thing.  We could get out from under
the Y2038 issue, and also eliminate a whole lot of platform
dependencies.  Not to mention sillinesses like being unable to recognize
a bad timezone name when it's fed to us.

Exactly how much work (and code bulk) would we be taking on?  I've
never looked at how big the timezone databases are...

			regards, tom lane

Wed, 22 May 2002 22:19:04 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Redhat 7.3 time manipulation bug 
Tom Lane <tgl@sss.pgh.pa.us> wrote:
[snip]
> 
> Exactly how much work (and code bulk) would we be taking on?  I've
> never looked at how big the timezone databases are...
> 

Some answers on database sizes, if this is any help...
I did "du -sh /usr/share/zoneinfo/" on them all.

OpenBSD 3.1, sparc64:
1.3M    /usr/share/zoneinfo/

Linux, i686, oldish mandrake (6.x?), glibc 2.1.3:
478k    /usr/share/zoneinfo

Linux, i686, newish redhat 7.2, glibc 2.2.4:
4.9M    /usr/share/zoneinfo

Linux, alpha EV56, oldish redhat 6.2, glibc 2.1.3
1.4M    /usr/share/zoneinfo

> regards, tom lane

Magnus

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
 Programmer/Networker [|] Magnus Naeslund
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


Thu, 23 May 2002 04:42:13 +0200
"Magnus Naeslund(f)" <mag@fbab.net>
Re: Redhat 7.3 time manipulation bug 
On Thu, May 23, 2002 at 04:42:13AM +0200, Magnus Naeslund(f) wrote:
> Some answers on database sizes, if this is any help...
> I did "du -sh /usr/share/zoneinfo/" on them all.
> 
> OpenBSD 3.1, sparc64:
> 1.3M    /usr/share/zoneinfo/
> 
> Linux, i686, oldish mandrake (6.x?), glibc 2.1.3:
> 478k    /usr/share/zoneinfo
> 
> Linux, i686, newish redhat 7.2, glibc 2.2.4:
> 4.9M    /usr/share/zoneinfo
  ^^^^

What do they do with that di?

> Linux, alpha EV56, oldish redhat 6.2, glibc 2.1.3
> 1.4M    /usr/share/zoneinfo

Here's what my Debian Woody system says:

1.5M    /usr/share/zoneinfo

And this is with glibc 2.2.5. Of course this wouldn't be the first time
that RedHat uses the same version number for a different version.

Michael
-- 
Michael Meskes
Michael@Fam-Meskes.De
Go SF 49ers! Go Rhein Fire!
Use Debian GNU/Linux! Use PostgreSQL!

Thu, 23 May 2002 10:44:04 +0200
Michael Meskes <meskes@postgresql.org>
Re: Redhat 7.3 time manipulation bug
On Wed, May 22, 2002 at 10:58:15AM -0700, Ulrich Drepper wrote:
> On Wed, 2002-05-22 at 10:51, Lamar Owen wrote:
> 
> > What isn't funny is Oliver Elphick's results on Debian, running glibc 2.2.5 
> > (same as Red Hat 7.3's version).
> 
> This is a completely different version.  Once Debian updates (in a few
> years) they'll get the same result.

Ulrich, how shall I understand this? I'm pretty sure Oliver
does not use a Debian 2.2 system with glibc-2.1.3 but a pretty
up-to-date one. The glibc version in the soon to be released Woody
release is 2.2.5. 

This seems to be the very same version that RedHat uses. So what
could/should Debian update? Besides, the "in a few years" comment looks
like FUD to me. It may be a few years since we talked the last time, but
I cannot imagine you changed that much that you spread FUD
nowadays. So I probably misunderstood this sentence, but nevertheless
would like to know what Debian should update.

Or do you mean that once Debian updates to glibc 2.3 (or whatever the
next release will be) it will show the same results? Does RedHat 7.3
already run on that new release? But then I would think they changed the
version number.

Michael

-- 
Michael Meskes
Michael@Fam-Meskes.De
Go SF 49ers! Go Rhein Fire!
Use Debian GNU/Linux! Use PostgreSQL!

Thu, 23 May 2002 16:20:22 +0200
Michael Meskes <meskes@postgresql.org>
Re: Redhat 7.3 time manipulation bug
On 22 May 2002, Ulrich Drepper wrote:

> On Wed, 2002-05-22 at 10:51, Lamar Owen wrote:
>
> > What isn't funny is Oliver Elphick's results on Debian, running glibc 2.2.5
> > (same as Red Hat 7.3's version).
>
> This is a completely different version.  Once Debian updates (in a few
> years) they'll get the same result.
>
> If you are misusing interfaces you get what you deserve.  At no time was
> it correct to use these functions for general date manipulation.  It
> always only was allowed to use them to represent system times and there
> was no Unix system before the epoch.  Therefore you argumentation is
> completely wrong.
>
> If you need date manipulation write your own code which work for all the
> times you want to represent.

We are Redhat, you will be assimilated



Thu, 23 May 2002 11:36:35 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: Redhat 7.3 time manipulation bug
On 22 May 2002, Ulrich Drepper wrote:

> On Wed, 2002-05-22 at 11:23, Tom Lane wrote:
>
> > Unix systems have
> > *always* interpreted time_t as a signed offset from the epoch.
>
> No.  This always was an accident if it happens.
>
> > Do you
> > really think that when Unixen were first built in the early 70s, there
> > was no interest in working with pre-1970 dates?  Hardly likely.
>
> There never were files or any system events with these dates.  Yes.
>
> And just to educate you and your likes: the majority of systems on this
> planet use mktime this way.  I hate using this as an argument, but
> beside major Unixes M$ systems also do this.

M$ systems crashes regularly too ... is Redhat going to adopt that too?


Thu, 23 May 2002 11:39:38 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: Redhat 7.3 time manipulation bug
> --=-Z1lifK4QZqKV8kHqHfYT
> Content-Type: text/plain
> Content-Transfer-Encoding: quoted-printable
> 
> On Wed, 2002-05-22 at 10:51, Lamar Owen wrote:
> 
> > What isn't funny is Oliver Elphick's results on Debian, running glibc 2.2=
> .5=20
> > (same as Red Hat 7.3's version).
> 
> This is a completely different version.  Once Debian updates (in a few
> years) they'll get the same result.
> 
> If you are misusing interfaces you get what you deserve.  At no time was
> it correct to use these functions for general date manipulation.  It
> always only was allowed to use them to represent system times and there
> was no Unix system before the epoch.  Therefore you argumentation is
> completely wrong.
> 
> If you need date manipulation write your own code which work for all the
> times you want to represent.

This is indeed a problem with dates on LIBC, because even if it is 
theoretically satisfactory to describe dates within some range within 2^31 
seconds of 1970, that is certainly NOT satisfactory for describing all dates 
of interest unless you're being terribly parochial about what is to be 
considered "of interest."

My father's birth falls within 2^31 seconds of 1970, but the Battle of 
Agincourt doesn't.

Any backup of any Unix system in history falls within 2^31 seconds of 1970, 
but there are people alive whose births don't.

People get away with using Unix dates as a "general" date type when they don't 
have to work outside a limited range.  If/when there is a move to 64 bit 
values, that will provide something with enough range to cover history back to 
ridiculously early times, relieving the limit.

But anybody using Unix dates as "general dates" has leaped into exactly the 
same sort of trap that caused people to get so paranoid about Y2K.
--
(concatenate 'string "cbbrowne" "@acm.org")
http://www.cbbrowne.com/info/oses.html
Do Roman paramedics refer to IV's as "4's"? 

-- 
(concatenate 'string "aa454" "@freenet.carleton.ca")
http://www.ntlug.org/~cbbrowne/linuxxian.html
"So, when you typed in the date, it exploded into a sheet of blue
flame and burned the entire admin wing to the ground? Yes, that's a
known bug. We'll be fixing it in the next release. Until then, try not
to use European date format, and keep an extinguisher handy."
-- slam@pobox.com (Tequila Rapide) 



Thu, 23 May 2002 11:02:03 -0400
cbbrowne@cbbrowne.com
Re: Redhat 7.3 time manipulation bug 
> On 22 May 2002, Ulrich Drepper wrote:
> 
> > On Wed, 2002-05-22 at 11:23, Tom Lane wrote:
> >
> > > Unix systems have
> > > *always* interpreted time_t as a signed offset from the epoch.
> >
> > No.  This always was an accident if it happens.
> >
> > > Do you
> > > really think that when Unixen were first built in the early 70s, there
> > > was no interest in working with pre-1970 dates?  Hardly likely.
> >
> > There never were files or any system events with these dates.  Yes.
> >
> > And just to educate you and your likes: the majority of systems on this
> > planet use mktime this way.  I hate using this as an argument, but
> > beside major Unixes M$ systems also do this.
> 
> M$ systems crashes regularly too ... is Redhat going to adopt that too?

Harbison and Steele indicates that:

  "Although the traditional return type of time is long, the value returned is 
usually of type unsigned long."

That is NOT a "Linux" reference; it was published before Linus had started 
working on his kernel project.

ANSI does not indicate that time_t is a signed int, signed long, or whatever.  
It is only defined to be an arithmetic type.

It would not be a bug for GLIBC to define time_t to be an unsigned int, with 
an epoch beginning of January 1, 1999.  That definition would conform to ANSI 
C.

Since that definition can conform to ANSI C, and Unix systems have more 
particularly been known to use unsigned ints with epoch of 1970, it is NOT 
reasonable to assume that time_t can be used to express dates that are at all 
ancient in the past.

Indeed, it is fairly _useful_ for different libc implementations to make 
different assumptions about things whose definitions are permitted to vary, as 
this makes it easier to call out as WRONG those systems that make up their own 
definitions.

People will no doubt get defensive about their own non-standard 
implementations of things; it is certainly far easier to cry "They're trying 
to play Microsoft!" than it is to be honest and actually look at the standards.
--
(concatenate 'string "cbbrowne" "@ntlug.org")
http://www.cbbrowne.com/info/advocacy.html
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

-- 
(reverse (concatenate 'string "gro.mca@" "enworbbc"))
http://www.cbbrowne.com/info/linuxxian.html
As of next Monday, COMSAT will be flushed in favor of a string and two tin
cans.  Please update your software.



Thu, 23 May 2002 12:17:09 -0400
cbbrowne@cbbrowne.com
Re: Redhat 7.3 time manipulation bug 
On Thu, 2002-05-23 at 07:20, Michael Meskes wrote:

> The glibc version in the soon to be released Woody
> release is 2.2.5. 

The version in RHL7.3 is 2.2.5-34.  This is not what Debian uses.  Maybe
you should read the changelog for the version.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------
23 May 2002 09:29:06 -0700
Ulrich Drepper <drepper@redhat.com>
Re: Redhat 7.3 time manipulation bug
On Thu, 23 May 2002 cbbrowne@cbbrowne.com wrote:

> > On 22 May 2002, Ulrich Drepper wrote:
> >
> > > On Wed, 2002-05-22 at 11:23, Tom Lane wrote:
> > >
> > > > Unix systems have
> > > > *always* interpreted time_t as a signed offset from the epoch.
> > >
> > > No.  This always was an accident if it happens.
> > >
> > > > Do you
> > > > really think that when Unixen were first built in the early 70s, there
> > > > was no interest in working with pre-1970 dates?  Hardly likely.
> > >
> > > There never were files or any system events with these dates.  Yes.
> > >
> > > And just to educate you and your likes: the majority of systems on this
> > > planet use mktime this way.  I hate using this as an argument, but
> > > beside major Unixes M$ systems also do this.
> >
> > M$ systems crashes regularly too ... is Redhat going to adopt that too?

< stuff deleted >

> People will no doubt get defensive about their own non-standard
> implementations of things; it is certainly far easier to cry "They're trying
> to play Microsoft!" than it is to be honest and actually look at the standards.

Just to clarify, if this was directed at my comment, I wasn't the one that
brought up the fact that "Redhat is trying to play Microsoft", Ulrich was
the one that brought it into the argument ... I was just curious as to how
far they planned on getting to what M$ systems do ...


Thu, 23 May 2002 21:12:26 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: Redhat 7.3 time manipulation bug 
On Thu, 2002-05-23 at 15:20, Michael Meskes wrote:
> On Wed, May 22, 2002 at 10:58:15AM -0700, Ulrich Drepper wrote:
> > On Wed, 2002-05-22 at 10:51, Lamar Owen wrote:
> > 
> > > What isn't funny is Oliver Elphick's results on Debian, running glibc 2.2.5 
> > > (same as Red Hat 7.3's version).
> > 
> > This is a completely different version.  Once Debian updates (in a few
> > years) they'll get the same result.
> 
> Ulrich, how shall I understand this? I'm pretty sure Oliver
> does not use a Debian 2.2 system with glibc-2.1.3 but a pretty
> up-to-date one. The glibc version in the soon to be released Woody
> release is 2.2.5. 

In fact, I run "unstable" and the version is 2.2.5-6.  I couldn't see
any reference in the Debian changelog, but I'm wondering if the change
was commented out.  I haven't had the time to download the source and go
looking.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C

     "I will praise thee; for I am fearfully and wonderfully 
      made..."       Psalms 139:14
24 May 2002 07:25:53 +0100
Oliver Elphick <olly@lfix.co.uk>
Re: Redhat 7.3 time manipulation bug
On Thu, May 23, 2002 at 09:29:06AM -0700, Ulrich Drepper wrote:
> On Thu, 2002-05-23 at 07:20, Michael Meskes wrote:
> 
> > The glibc version in the soon to be released Woody
> > release is 2.2.5. 
> 
> The version in RHL7.3 is 2.2.5-34.  This is not what Debian uses.  Maybe
> you should read the changelog for the version.

So with your arithmetics 2.2.5 != 2.2.5. Hey that's great. How about
naming the next Linux kernel release 2.0 just to confuse some people?
:-)

Seriously though, while this is offtopic on this list, I wonder what's
going on. If I have a system with glibc 2.2.5, I cannot expect this to
be the same version as on another system with glibc 2.2.5. Is this the
correct understanding?

And which changelog should I read? The RedHat one, the Debian one or the
glibc one? 

Or does the -34 mean more than just the RedHat version number? The
Debian version is correctly named 2.2.5-6 where the -6 means that this
is the 6th release of glibc 2.2.5 for Debian,

Michael

-- 
Michael Meskes
Michael@Fam-Meskes.De
Go SF 49ers! Go Rhein Fire!
Use Debian GNU/Linux! Use PostgreSQL!

Fri, 24 May 2002 09:10:44 +0200
Michael Meskes <meskes@postgresql.org>
Re: Redhat 7.3 time manipulation bug
...
> But anybody using Unix dates as "general dates" has leaped into exactly the
> same sort of trap that caused people to get so paranoid about Y2K.

Certainly true. We don't use Unix dates as "general dates", we use the
Unix time zone database and API for dates and times within the year
range of 1903 to 2038. Well, up until now anyway...

Prior to the 1900s, the concept of time zones was more localized and not
universally adopted. In the US, a first round of time zone
standardization came with the transcontinental railroads in the 1860s.
After 2038, it is a good bet that time zones will resemble those in use
today, but they are as much a political construct as a physical one so
the details are subject to change.

                      - Thomas

Fri, 24 May 2002 06:10:39 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: Redhat 7.3 time manipulation bug
On Fri, 24 May 2002 06:10:39 PDT, the world broke into rejoicing as
Thomas Lockhart <lockhart@fourpalms.org>  said:
> ...
> > But anybody using Unix dates as "general dates" has leaped into exactly the
> > same sort of trap that caused people to get so paranoid about Y2K.

> Certainly true. We don't use Unix dates as "general dates", we use the
> Unix time zone database and API for dates and times within the year
> range of 1903 to 2038. Well, up until now anyway...

I don't think going past 1970 is particularly safe; it certainly doesn't
seem to fit with ANSI...

By the way, the seemingly relevant link to look at for TZ info is 
http://www.twinsun.com/tz/tz-link.htm, linking to the data used by
various Unix implementations.

> Prior to the 1900s, the concept of time zones was more localized and
> not universally adopted. In the US, a first round of time zone
> standardization came with the transcontinental railroads in the 1860s.
> After 2038, it is a good bet that time zones will resemble those in
> use today, but they are as much a political construct as a physical
> one so the details are subject to change.

Some of the zones are quite peculiar if you head to Africa and Asia;
there are some sitting on 15 minute intervales, rather than the usual 1h
intervals.

(The classic Canadian timezone joke is "World ends at 9:00; 9:30 in
Newfoundland".  For more information, see TZ='America/St_Johns')
--
(concatenate 'string "chris" "@cbbrowne.com")
http://www.ntlug.org/~cbbrowne/spreadsheets.html
"Heuristics (from the  French heure, "hour") limit the  amount of time
spent executing something.  [When using heuristics] it shouldn't take
longer than an hour to do something."

Fri, 24 May 2002 09:47:52 -0400
cbbrowne@cbbrowne.com
Re: Redhat 7.3 time manipulation bug 
cbbrowne@cbbrowne.com writes:
> By the way, the seemingly relevant link to look at for TZ info is 
> http://www.twinsun.com/tz/tz-link.htm, linking to the data used by
> various Unix implementations.

Oh, this is interesting: it claims that

: This database (often called tz or zoneinfo) is used by several
: implementations, including GNU/Linux, FreeBSD, NetBSD, OpenBSD, DJGPP,
: HP-UX, IRIX, Open UNIX, Solaris, and Tru64.

The actual timezone database seems to consist of about half a meg of
heavily commented text data files.  The accompanying code (probably
providing far more functionality than we actually need) is under 400k.
(Both figures are for uncompressed source.)  Not large at all.

I cannot find any sign of a copyright or license in the files; I think
it is intended to be public domain, and in any case if the *BSDs are
using it then it must have a BSD-compatible license.

It seems to me that it'd be really practical to just take what we need
out of this distribution and forgo all dependency on system-provided
timezone databases.  And, since there's a mailing list maintaining it,
we could expect someone else to handle updates ;-) ... we'd just have
to be careful to use the database files unmodified, so that we could
drop in new releases from time to time.

Comments?  Anyone want to do the legwork?

			regards, tom lane

Fri, 24 May 2002 10:17:04 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Redhat 7.3 time manipulation bug 
Tom Lane wrote:
 > It seems to me that it'd be really practical to just take what we
 > need out of this distribution and forgo all dependency on
 > system-provided timezone databases.  And, since there's a mailing
 > list maintaining it, we could expect someone else to handle updates
 > ;-) ... we'd just have to be careful to use the database files
 > unmodified, so that we could drop in new releases from time to time.
 >
 > Comments?  Anyone want to do the legwork?
 >

I don't understand precisely what need to be done, but I'll give it a 
shot if you get me pointed in the right direction.

<downloads and looks at code>
I see that tzcode2002c.tar.gz includes a mktime() function. Is the idea 
to pull this out (with just whatever support it needs), incorporate it 
into PostgreSQL source (perhaps in a new src/backend/utils/tz directory) 
and use this in place of the system provided mktime()?

Joe



Fri, 24 May 2002 10:27:18 -0700
Joe Conway <mail@joeconway.com>
Re: Redhat 7.3 time manipulation bug
Joe Conway <mail@joeconway.com> writes:
> I don't understand precisely what need to be done, but I'll give it a 
> shot if you get me pointed in the right direction.
> <downloads and looks at code>
> I see that tzcode2002c.tar.gz includes a mktime() function. Is the idea 
> to pull this out (with just whatever support it needs), incorporate it 
> into PostgreSQL source (perhaps in a new src/backend/utils/tz directory) 
> and use this in place of the system provided mktime()?

Well, that's the zeroth-order approximation.  We should take the
opportunity to get out from under the mktime()/tzset() API.  The real
idea here is to make use of the timezone database info in the ways that
Postgres needs.  Some things that are not good about mktime()/tzset():

* Arbitrary restrictions on range of dates.  We certainly don't want to
be limited by a 32-bit time_t, whether you think it's signed or not.
The APIs should be recast in terms of PG's preferred internal
representations.  (Lockhart would be the man to point you in the right
direction here, not me.)

* No way to tell whether a user-provided timezone name is actually good.

* No support for concurrent access to multiple zones, short of flushing
all memory of one zone to load the next.  Although we do not really need
this now, I can foresee wanting it.  I'd be inclined to store all the
info about a particular zone in some struct that can be referenced by
a pointer; that would give us the flexibility to have multiple such
structs floating around a backend in the future (perhaps living in a
hashtable indexed by timezone name?)

My guess is that we want to borrow the parts of the tzcode library that
are associated with reading a tz database file and loading it into some
suitable internal representation; there's probably not a lot else that
we'll want to use as-is.

			regards, tom lane

Fri, 24 May 2002 13:51:01 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Redhat 7.3 time manipulation bug 
Michael Meskes writes:

> Or does the -34 mean more than just the RedHat version number? The
> Debian version is correctly named 2.2.5-6 where the -6 means that this
> is the 6th release of glibc 2.2.5 for Debian,

Just for general amusement:  I run SuSE's glibc 2.2.5-38 which contains
neither the questionable code in the original sources nor is there any
reference to it in the patch set.  Go figure.

-- 
Peter Eisentraut   peter_e@gmx.net


Fri, 24 May 2002 21:03:56 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Redhat 7.3 time manipulation bug
On Fri, 2002-05-24 at 12:03, Peter Eisentraut wrote:

> > Or does the -34 mean more than just the RedHat version number? The
> > Debian version is correctly named 2.2.5-6 where the -6 means that this
> > is the 6th release of glibc 2.2.5 for Debian,
> 
> Just for general amusement:  I run SuSE's glibc 2.2.5-38 which contains
> neither the questionable code in the original sources nor is there any
> reference to it in the patch set.  Go figure.

This is getting silly.  Does nobody here understand that the release
number is local for each distribution.  Comparing them does not lead to
anything.  If you want to find out run

   rpm -q --changelog glibc | less

on a RH system.  Don't know what other systems provide in this
direction.  You'll see that the glibc in RHL7.3 contains a lot of the
code from the glibc 2.3 branch.  It's not named 2.2.90 because major
pieces are missing.

If you still don't know that version numbers are meaningless for
determining feature lists you might want to consider going back to your
CS101 class and revisit software configuration management.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------
24 May 2002 12:15:47 -0700
Ulrich Drepper <drepper@redhat.com>
Re: Redhat 7.3 time manipulation bug
Tom Lane wrote:
> Well, that's the zeroth-order approximation.  We should take the
> opportunity to get out from under the mktime()/tzset() API.  The real
> idea here is to make use of the timezone database info in the ways that
> Postgres needs.  Some things that are not good about mktime()/tzset():
> 
> * Arbitrary restrictions on range of dates.  We certainly don't want to
> be limited by a 32-bit time_t, whether you think it's signed or not.
> The APIs should be recast in terms of PG's preferred internal
> representations.  (Lockhart would be the man to point you in the right
> direction here, not me.)
> 
> * No way to tell whether a user-provided timezone name is actually good.
> 
> * No support for concurrent access to multiple zones, short of flushing
> all memory of one zone to load the next.  Although we do not really need
> this now, I can foresee wanting it.  I'd be inclined to store all the
> info about a particular zone in some struct that can be referenced by
> a pointer; that would give us the flexibility to have multiple such
> structs floating around a backend in the future (perhaps living in a
> hashtable indexed by timezone name?)
> 
> My guess is that we want to borrow the parts of the tzcode library that
> are associated with reading a tz database file and loading it into some
> suitable internal representation; there's probably not a lot else that
> we'll want to use as-is.

Well, this does sound a bit more involved than I was envisioning. There 
are a few items wrt SRFs that I should finish first, but I'll come back 
to this afterward if no one else does first.

Joe


Fri, 24 May 2002 15:30:24 -0700
Joe Conway <mail@joeconway.com>
Re: Redhat 7.3 time manipulation bug
...
> Well, this does sound a bit more involved than I was envisioning. There
> are a few items wrt SRFs that I should finish first, but I'll come back
> to this afterward if no one else does first.

The first cut might be to reproduce the functionality we already have.
That would allow us to (optionally) use this internal code *or* the
system-provided code with a configure-time switch. We could then strip
out two of the three time zone interface styles we support. And we could
(possibly) allow folks to use their built-in time zone databases if they
want, to minimize inconsistancies between PostgreSQL and other programs
on the system. We might need to modify function and variable signatures
to avoid conflicts with system-supplied libraries.

The next step would be to see how to generalize this past Y2038 (as
mentioned previously, time zone info for pre-1900 is not likely to be
interesting). If it involves mass substitution of time_t for, say,
pg_time_t, then that might be all we need for the second phase, at which
time we've broken the y2038 limit by moving to 64-bit integer time.

The last phase could be extending the API to allow multiple simultaneous
time zones, detection of bad time zones, etc etc. This would involve API
changes or extensions, and breaks compatibility with system-supplied
infrastructure.

                    - Thomas

Fri, 24 May 2002 16:35:03 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: Redhat 7.3 time manipulation bug
Thomas Lockhart <lockhart@fourpalms.org> writes:
> The last phase could be extending the API to allow multiple simultaneous
> time zones, detection of bad time zones, etc etc. This would involve API
> changes or extensions, and breaks compatibility with system-supplied
> infrastructure.

One thing that wasn't clear to me, but could use investigation: if so
many systems are using the same underlying timezone database info, maybe
there is some commonality at a level below the ISO mktime/tzset/etc API.
If we could make use of the system-provided TZ database at a lower level
while still using our own APIs not tied to time_t, it'd answer the issue
of compatibility with the surrounding system.  (Which is a real issue,
I agree --- we should be able to accept the system's standard TZ setting
if possible.)

			regards, tom lane

Fri, 24 May 2002 19:41:24 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Redhat 7.3 time manipulation bug 
> > The last phase could be extending the API to allow multiple simultaneous
> > time zones, detection of bad time zones, etc etc. This would involve API
> > changes or extensions, and breaks compatibility with system-supplied
> > infrastructure.
> One thing that wasn't clear to me, but could use investigation: if so
> many systems are using the same underlying timezone database info, maybe
> there is some commonality at a level below the ISO mktime/tzset/etc API.
> If we could make use of the system-provided TZ database at a lower level
> while still using our own APIs not tied to time_t, it'd answer the issue
> of compatibility with the surrounding system.  (Which is a real issue,
> I agree --- we should be able to accept the system's standard TZ setting
> if possible.)

The fundamental problem (which of course can have a fundamental solution
;) is that a time zone database built with a 32-bit time_t will have
time zone info through 2038 only (it is a binary file with 32-bit time
fields -- almost certainly anyway). So if we have an extended time zone
infrastructure using something different for time_t we would need to
handle the case of reading non-extended time zones databases, which puts
us back to having limitations.

I'm guessing that a better approach might be to have our time zone stuff
inside our own API, which then could choose to call, for example,
mktime() or pg_mktime(), which could each have different signatures.
Then the heuristics for matching one to the other are isolated to our
thin API implementation, not to the underlying system- or pg-provided
libraries.

afaik there is no API provision for the "inverse time zone" problem of
matching "stringy time zones" to numeric offsets for input date/times.
The time zone databases themselves don't lend themselves to this, since
the tables have those stringy zones somewhere on the right hand side of
each row of information and the fields can change from year to year.

                   - Thomas

Fri, 24 May 2002 17:04:11 -0700
Thomas Lockhart <lockhart@fourpalms.org>
Re: Redhat 7.3 time manipulation bug
Thomas Lockhart <lockhart@fourpalms.org> writes:
> The fundamental problem (which of course can have a fundamental solution
> ;) is that a time zone database built with a 32-bit time_t will have
> time zone info through 2038 only (it is a binary file with 32-bit time
> fields -- almost certainly anyway).

I'm not sure that the time zone database tables store time_t's at all.
They certainly could be coded not to use 'em; but I do not know exactly
how various vendors have chosen to represent the data.

A random extract from the tzdata2002c files looks like:

Zone America/Chicago	-5:50:36 -	LMT	1883 Nov 18 12:00
			-6:00	US	C%sT	1920
			-6:00	Chicago	C%sT	1936 Mar  1 2:00
			-5:00	-	EST	1936 Nov 15 2:00
			-6:00	Chicago	C%sT	1942
			-6:00	US	C%sT	1946
			-6:00	Chicago	C%sT	1967
			-6:00	US	C%sT

which might well be represented with separate y/m/d/h/m fields...
certainly we'd choose some such thing if we have to implement it
ourselves.

			regards, tom lane

Fri, 24 May 2002 20:09:49 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Redhat 7.3 time manipulation bug 
> > > The last phase could be extending the API to allow multiple simultaneous
> > > time zones, detection of bad time zones, etc etc. This would involve API
> > > changes or extensions, and breaks compatibility with system-supplied
> > > infrastructure.
> > One thing that wasn't clear to me, but could use investigation: if so
> > many systems are using the same underlying timezone database info, maybe
> > there is some commonality at a level below the ISO mktime/tzset/etc API.
> > If we could make use of the system-provided TZ database at a lower level
> > while still using our own APIs not tied to time_t, it'd answer the issue
> > of compatibility with the surrounding system.  (Which is a real issue,
> > I agree --- we should be able to accept the system's standard TZ setting
> > if possible.)

> The fundamental problem (which of course can have a fundamental
> solution ;) is that a time zone database built with a 32-bit time_t
> will have time zone info through 2038 only (it is a binary file with
> 32-bit time fields -- almost certainly anyway). So if we have an
> extended time zone infrastructure using something different for time_t
> we would need to handle the case of reading non-extended time zones
> databases, which puts us back to having limitations.

Ah, but the database in question _doesn't_ consist of 32 bit time_t
values.

It consists of things like:

# @(#)zone.tab	1.26
#
# TZ zone descriptions
#
# From Paul Eggert <eggert@twinsun.com> (1996-08-05):
#
# This file contains a table with the following columns:
# 1.  ISO 3166 2-character country code.  See the file `iso3166.tab'.
# 2.  Latitude and longitude of the zone's principal location
#     in ISO 6709 sign-degrees-minutes-seconds format,
#     either +-DDMM+-DDDMM or +-DDMMSS+-DDDMMSS,
#     first latitude (+ is north), then longitude (+ is east).
# 3.  Zone name used in value of TZ environment variable.
# 4.  Comments; present if and only if the country has multiple rows.
#
# Columns are separated by a single tab.
# The table is sorted first by country, then an order within the country that
# (1) makes some geographical sense, and
# (2) puts the most populous zones first, where that does not contradict (1).
#
# Lines beginning with `#' are comments.
#
#country-
#code	coordinates	TZ			comments
AD	+4230+00131	Europe/Andorra
AE	+2518+05518	Asia/Dubai
AF	+3431+06912	Asia/Kabul
AG	+1703-06148	America/Antigua
AI	+1812-06304	America/Anguilla
AL	+4120+01950	Europe/Tirane
AM	+4011+04430	Asia/Yerevan
AN	+1211-06900	America/Curacao
AO	-0848+01314	Africa/Luanda

Then a "leapseconds" table, looking like:
# The correction (+ or -) is made at the given time, so lines
# will typically look like:
#	Leap	YEAR	MON	DAY	23:59:60	+	R/S
# or
#	Leap	YEAR	MON	DAY	23:59:59	-	R/S

# If the leapsecond is Rolling (R) the given time is local time
# If the leapsecond is Stationary (S) the given time is UTC

# Leap	YEAR	MONTH	DAY	HH:MM:SS	CORR	R/S
Leap	1972	Jun	30	23:59:60	+	S
Leap	1972	Dec	31	23:59:60	+	S
Leap	1973	Dec	31	23:59:60	+	S
Leap	1974	Dec	31	23:59:60	+	S
Leap	1975	Dec	31	23:59:60	+	S
Leap	1976	Dec	31	23:59:60	+	S

And then a set of rules about timezone adjustments for all sorts of
localities, including the following:

# Rule	NAME	FROM	TO	TYPE	IN	ON	AT	SAVE	LETTER/S
# Summer Time Act, 1916
Rule	GB-Eire	1916	only	-	May	21	2:00s	1:00	BST
Rule	GB-Eire	1916	only	-	Oct	 1	2:00s	0	GMT
# S.R.&O. 1917, No. 358
Rule	GB-Eire	1917	only	-	Apr	 8	2:00s	1:00	BST
Rule	GB-Eire	1917	only	-	Sep	17	2:00s	0	GMT


# Zone	NAME		GMTOFF	RULES	FORMAT	[UNTIL]
Zone Antarctica/Casey	0	-	zzz	1969
			8:00	-	WST	# Western (Aus) Standard Time
Zone Antarctica/Davis	0	-	zzz	1957 Jan 13
			7:00	-	DAVT	1964 Nov # Davis Time
			0	-	zzz	1969 Feb
			7:00	-	DAVT
Zone Antarctica/Mawson	0	-	zzz	1954 Feb 13
			6:00	-	MAWT	# Mawson Time

> I'm guessing that a better approach might be to have our time zone
> stuff inside our own API, which then could choose to call, for
> example, mktime() or pg_mktime(), which could each have different
> signatures.  Then the heuristics for matching one to the other are
> isolated to our thin API implementation, not to the underlying system-
> or pg-provided libraries.

> matching "stringy time zones" to numeric offsets for input date/times.
> The time zone databases themselves don't lend themselves to this,
> since the tables have those stringy zones somewhere on the right hand
> side of each row of information and the fields can change from year to
> year.

The ultimate goal would seem likely to be to store dates internally in
some form like UTC, with some reasonably huge dynamic range, that is,
not limited to 32 bit timestamps, but rather using something like a
proleptic Gregorian calendar (per _Calendrical Calculations_, page 50).

Some reasonable treatments would include:

  - 32 bits is an signed int indicating number of days since GREG_EPOCH,
    where logical epochs would include January 1, 1, January 1, 1900, or
    perhaps even something actually proleptic (proleptic indicates
    "future"), such as January 1, 2038.

  - 8 bits indicating the month; 8 bits indicating the day of month;
    16 bits providing a range of years from -32767 to 32768.

Both have merits...

Timestamps would then forcibly expand things by _at least_ 22 bits, the
minimum needed to express 1/100ths of seconds.  Might as well head on to
32 bits for the time and so have something that can easily represent
values down to well below a millisecond.

The "stringy stuff" indicates how values are to be displayed or parsed.
It does nothing about what is stored internally, or at least shouldn't.
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://www.cbbrowne.com/info/emacs.html
In the name of the Lord-High mutant, we sacrifice this suburban girl
-- `Future Schlock'

Fri, 24 May 2002 20:37:24 -0400
cbbrowne@cbbrowne.com
Re: Redhat 7.3 time manipulation bug 
On Fri, 24 May 2002, Peter Eisentraut wrote:

> Michael Meskes writes:
> 
> > Or does the -34 mean more than just the RedHat version number? The
> > Debian version is correctly named 2.2.5-6 where the -6 means that this
> > is the 6th release of glibc 2.2.5 for Debian,
> 
> Just for general amusement:  I run SuSE's glibc 2.2.5-38 which contains
> neither the questionable code in the original sources nor is there any
> reference to it in the patch set.  Go figure.

You've got to remember that you're talking about systems where, a long time ago
now, certain groups felt it necessary to supply nonstandard versions of the
core component (the kernel). Sure they helped development of the kernel but
only through bastardisation of version numbers where 2.0.1 didn't really mean 
a Linux 2.0.1 kernel. Is it really surprising the system support stuff has been
mangled beyond sense?

Anyway, I've composed several and aborted all but this message on this subject
and I'm not going to persue it. I have my own views on the right and wrongs off
the change in glibc but they wouldn't have advanced anything so I'm keeping
quiet on it, still. It seems there is a solution forming. Plus, I'd hate to
side with the baddies from the first paragraph :)


-- 
Nigel J. Andrews
Director

---
Logictree Systems Limited
Computer Consultants


Sat, 25 May 2002 02:55:58 +0100 (BST)
"Nigel J. Andrews" <nandrews@investsystems.co.uk>
Re: Redhat 7.3 time manipulation bug
On Fri, May 24, 2002 at 12:15:47PM -0700, Ulrich Drepper wrote:
> This is getting silly.  Does nobody here understand that the release

Yes, but I'm not sure on which side.

> number is local for each distribution.  Comparing them does not lead to

No, this is simply not true. The version number is what the upstream
gives its release. No more no less. What RH does is becoming as subtly
incompatible a possible. If that's the goal, it doesn't look like free
software for me. Sure all changes are published, but why forcing this
kind of difference between linux distributions? Why not calling it 2.2.6
or something if there has to be some changes that are not compatible?

Michael
-- 
Michael Meskes
Michael@Fam-Meskes.De
Go SF 49ers! Go Rhein Fire!
Use Debian GNU/Linux! Use PostgreSQL!

Sat, 25 May 2002 21:55:01 +0200
Michael Meskes <meskes@postgresql.org>
Re: Redhat 7.3 time manipulation bug
On Sat, 25 May 2002, Michael Meskes wrote:

> No, this is simply not true. The version number is what the upstream
> gives its release. No more no less. What RH does is becoming as subtly
> incompatible a possible. If that's the goal, it doesn't look like free
> software for me. Sure all changes are published, but why forcing this
> kind of difference between linux distributions? Why not calling it 2.2.6
> or something if there has to be some changes that are not compatible?

Or rlibc? :)


Sun, 26 May 2002 00:32:40 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: Redhat 7.3 time manipulation bug
On Friday 24 May 2002 03:15 pm, Ulrich Drepper wrote:
> This is getting silly.

Yes, Ulrich, it is.  Very silly.  And Red Hat's stance is one of the silliest, 
IMHO.

>You'll see that the glibc in RHL7.3 contains a lot of the
> code from the glibc 2.3 branch.  It's not named 2.2.90 because major
> pieces are missing.

> If you still don't know that version numbers are meaningless for
> determining feature lists you might want to consider going back to your
> CS101 class and revisit software configuration management.

IOW, Red Hat's glibc 2.2.5 isn't really pristine glibc 2.2.5 as found straight 
from the GNU repository.  In fact, Red Hat glibc 2.2.5 isn't really 2.2.5 -- 
how about 2.2.96? :-)  .96 was good enough for gcc....

Furthermore, Red Hat glibc 2.2.5 isn't even fully compatible with GNU glibc 
2.2.5 -- at least in the area of time_t stuff.

In the open source world, version numbers are actually supposed to mean 
something -- at least for package dependencies.  Of course, I also have read 
the kernel-2.4.18 source RPM and its 21.8MB 'ac-bits' patch.

You do realize that this sort of thing doesn't help Red Hat's PR state amongst 
the greater open source community, right?  Nor would it help Mandrake, SuSE, 
or any other Linux distributor (I specifically excluded Debian due to its 
unique community supported state).  But, if you don't care about the greater 
open source community, well...

And I say all of that while running and enjoying the greater part of Red Hat 
7.3.  For the most part it is extraordinarily stable.  And I know that that 
21.8MB kernel patch is one of the reasons it is so stable.  But I still 
question the versioning of glibc.

So, in summary, the glibc version number in any particular linux distribution 
is meaningless because the distributor is free to patch the bloody daylights 
out of it at any time.  Sweet.  And so standard.

But, if glibc 2.3 is where this bit came from, it is just a matter of time 
before all Linux distributions (that aren't willing to patch away) get this 
braindead behavior.  Oh well.  The general solution will happen.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Mon, 27 May 2002 22:18:17 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Redhat 7.3 time manipulation bug
Tom Lane wrote:
> Thomas Lockhart <lockhart@fourpalms.org> writes:
> >> Why should we rely on broken glibc and the standard? Why don't we make
> >> our own mktime() and use it on all platforms.
> 
> > The downside to doing that is that we then take over maintenance of the
> > code and, more importantly, the timezone database.
> 
> > But it might be the best thing to do.
> 
> I've been sorta thinking the same thing.  We could get out from under
> the Y2038 issue, and also eliminate a whole lot of platform
> dependencies.  Not to mention sillinesses like being unable to recognize
> a bad timezone name when it's fed to us.
> 
> Exactly how much work (and code bulk) would we be taking on?  I've
> never looked at how big the timezone databases are...

I am not really excited about distributing a timezone database as part
of PostgreSQL, and it wouldn't match the OS's timezone.  (We do need a
64-time time_t, but I think we can wait to get closer to 2038.) Can we
detect if glibc is being used for the compile (easy), and substitute a
non-broken mktime in the link path ahead of glibc's mktime?  Seems that
would be the easiest solution.

Of course, pre-1970 dates then wouldn't match the OS on glibc systems,
but that seems like a win.  :-)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 7 Jun 2002 01:04:30 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Redhat 7.3 time manipulation bug
Trond Eivind Glomsr�d wrote:
> On Tue, 21 May 2002, Lamar Owen wrote:
> 
> > On Tuesday 21 May 2002 11:04 am, Manuel Sugawara wrote:
> > > I see. This behavior is consistent with the fact that mktime is
> > > supposed to return -1 on error, but then is broken in every other Unix
> > > implementation that I know.
> > 
> > > Any other workaround than downgrade or install FreeBSD?
> > 
> > Complain to Red Hat.  Loudly. However, as this is a glibc change, other 
> > distributors are very likely to fold in this change sooner rather than 
> > later. 
> 
> Relying on nonstandardized/nondocumented behaviour is a program bug, not a 
> glibc bug. PostgreSQL needs fixing. Since we ship both, we're looking at 
> it, but glibc is not the component with a problem.

No one has really answered the question --- if the way PostgreSQL is
using mktime() for pre-1970 dates is wrong, why do timezone databases
have pre-1970 timezone information?

I assume Linux does or the old mktime() wouldn't have worked for
pre-1970 dates.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 7 Jun 2002 01:11:22 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Redhat 7.3 time manipulation bug
Anyone mind if we bump the DTD version to Docbook 4.2?

This consists on all users who wish to build docs on installing the 4.2
DTD set, and updating some depreciated tags within the sgml files.

comment -> remark
docinfo -> appendixinfo, chapterinfo, bookinfo, etc.


What it buys is a number of useful tags, SVGs and probably more
importantly for the future, xsl and fop support which will probably be
important in the future.  OpenJade hasn't had a new release in quite a
long time -- not to say work isn't needed.

Yes, after updating docs to the newer DTD I intend to make them XML
compliant to ensure they work with v5 of docbook in the future.



14 Aug 2002 23:14:03 -0400
Rod Taylor <rbt@zort.ca>
Documentation DTD
Reading about the pgmonitor thread and mention of gborg made me wonder
about replication and ready ability to uniformly monitor it.  Just as
pg_stat* tables exist to allow for statistic gathering and monitoring in
a uniform fashion, it occurred to me that a predefined set of views
and/or tables for all replication implementations may be worthwhile. 
That way, no matter what replication method/tool is being used, as long
as it conforms to the defined replication interfaces, generic monitoring
tools can be used to keep an eye on things.

Think this has any merit?

Greg Copeland
14 Aug 2002 22:15:32 -0500
Greg Copeland <greg@CopelandConsulting.Net>
Standard replication interface?
Greg Copeland <greg@CopelandConsulting.Net> writes:
> ... it occurred to me that a predefined set of views
> and/or tables for all replication implementations may be worthwhile.

Do we understand replication well enough to define such a set of views?
I sure don't ...

			regards, tom lane

Wed, 14 Aug 2002 23:47:37 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Standard replication interface? 
Rod Taylor <rbt@zort.ca> writes:
> Anyone mind if we bump the DTD version to Docbook 4.2?

Peter E. is the gatekeeper on that, I think --- he pushed us to 4.1
not long ago.

If Peter's okay with 4.2, then full speed ahead ...

			regards, tom lane

PS: pgsql-docs is probably the more appropriate forum for this
discussion.

Thu, 15 Aug 2002 00:12:42 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Documentation DTD 
Well, that's a different issue.  ;)

I initially wanted to get feedback to see if anyone else thought the
concept might hold some merit.

I take it from your answer you think it might...but are scratching your
head wondering exactly what it entails...

Greg


On Wed, 2002-08-14 at 22:47, Tom Lane wrote:
> Greg Copeland <greg@CopelandConsulting.Net> writes:
> > ... it occurred to me that a predefined set of views
> > and/or tables for all replication implementations may be worthwhile.
> 
> Do we understand replication well enough to define such a set of views?
> I sure don't ...
> 
> 			regards, tom lane
15 Aug 2002 09:05:47 -0500
Greg Copeland <greg@CopelandConsulting.Net>
Re: Standard replication interface?
On Wed, Aug 14, 2002 at 10:15:32PM -0500, Greg Copeland wrote:

> Reading about the pgmonitor thread and mention of gborg made me wonder
> about replication and ready ability to uniformly monitor it.  Just as
> pg_stat* tables exist to allow for statistic gathering and monitoring in
> a uniform fashion, it occurred to me that a predefined set of views
> and/or tables for all replication implementations may be worthwhile. 
> That way, no matter what replication method/tool is being used, as long
> as it conforms to the defined replication interfaces, generic monitoring
> tools can be used to keep an eye on things.

That sounds like the cart is before the horse.  You need to know what
sort of replication scheme you might ever have before you could
know the statistics that you might want to know.

There are different sorts of replication schemes under consideration. 
For instance, rserv uses an asynchronous master/slave approach, which
relies on slaves that are almost dumb as chickens.  (Not quite. 
There is some data about the state of replication in the slave
database; but most of it is in the master.)  Postgres-R, on the other
hand, contemplates a distributed model wherein different database
machines participate in a pool.

So for rserv-style replication, you want to know (for instance)
average slave-update times, and whether slaves are getting behind,
and by how much, and such.  Balancing of inserts, however, is not
relevant, because you can't do that.

Postgres-R will have the opposite need: you'll want to know what sort
of load balancing you're getting, but time-to-replicate is not
relevant, because a commit on one machine is necessarily a commit
everywhere (that's why it's "eager" replication).

You probably could design a set of statistics that would cover all
cases, but only after you know what the cases were.

A

-- 
----
Andrew Sullivan                               87 Mowat Avenue 
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M6K 3E3
                                         +1 416 646 3304 x110


Thu, 15 Aug 2002 10:47:56 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: Standard replication interface?
Andrew Sullivan <andrew@libertyrms.info> writes:
> On Wed, Aug 14, 2002 at 10:15:32PM -0500, Greg Copeland wrote:
> > Reading about the pgmonitor thread and mention of gborg made me wonder
> > about replication and ready ability to uniformly monitor it.  Just as
> > pg_stat* tables exist to allow for statistic gathering and monitoring in
> > a uniform fashion, it occurred to me that a predefined set of views
> > and/or tables for all replication implementations may be worthwhile. 
> > That way, no matter what replication method/tool is being used, as long
> > as it conforms to the defined replication interfaces, generic monitoring
> > tools can be used to keep an eye on things.
> 
> That sounds like the cart is before the horse.

That's exactly what I was going to say -- I'd prefer that any
interested parties concentrate on producing a *really good*
replication implementation, which might eventually be integrated into
PostgreSQL itself.

Producing a "generic API" for something that really doesn't need
genericity sounds like a waste of time, IMHO.

Cheers,

Neil

-- 
Neil Conway <neilconway@rogers.com>
PGP Key ID: DB3C29FC


15 Aug 2002 10:53:15 -0400
Neil Conway <nconway@klamath.dyndns.org>
Re: Standard replication interface?
On Thu, 2002-08-15 at 09:47, Andrew Sullivan wrote:
> On Wed, Aug 14, 2002 at 10:15:32PM -0500, Greg Copeland wrote:
> > That way, no matter what replication method/tool is being used, as long
> > as it conforms to the defined replication interfaces, generic monitoring
> > tools can be used to keep an eye on things.
> 
> That sounds like the cart is before the horse.  You need to know what
> sort of replication scheme you might ever have before you could
> know the statistics that you might want to know.

Hmmm.  Never heard of an inquiry for interest in a concept as putting
the cart before the horse.  Considering this is pretty much how things
get developed in the real world, I'm not sure what you feel is so
special about replication.

First step is always identify the need.  I'm attempting to do so.  Not
sure what you'd consider the first step to be but I can assure you,
regardless of this concept seeing the light of day, it is the first
step.  The horse is correctly positioned in front of the cart.

I also stress that I'm talking about a statistical replication
interface.  It occurred to me that you might of been confused on this
matter.  That is, a set of tables and views will allow for the
replication process to be uniformly *monitored*.  I am not talking about
a set of interfaces which all manner of replication much perform its job
through (interface with databases for replication).

> 
> There are different sorts of replication schemes under consideration. 

Yep.  Thus it would seemingly be ideal to have a specification which
different implementations would seek to implement.  Off of the top of my
head and for starters, a table and/or view which could can queried that
returns the tables that are being replicated sounds good to me.  Same
thing for the list of databases, the servers involved and their
associated role (master, slave, peer).

Without such a concept, there will be no standardized way to monitor
your replication.  As such, chances are one of two things will happen. 
One, a single replication method will be championed and fair tools will
develop to support where all others are bastards.  Two, quality tools to
monitor replication will never materialize because each method for
monitoring is specific to the different types of implementations. 
Resources will constantly be spread amongst a variety of well meaning
projects.


--Greg
15 Aug 2002 12:08:54 -0500
Greg Copeland <greg@CopelandConsulting.Net>
Re: Standard replication interface?
On Thu, 2002-08-15 at 09:53, Neil Conway wrote:
> That's exactly what I was going to say -- I'd prefer that any
> interested parties concentrate on producing a *really good*
> replication implementation, which might eventually be integrated into
> PostgreSQL itself.
> 
> Producing a "generic API" for something that really doesn't need
> genericity sounds like a waste of time, IMHO.
> 
> Cheers,
> 
> Neil


Some how I get the impression that I've been completely misunderstood. 
Somehow, people seem to of only read the subject and skipped the body
explaining the concept.

In what way would providing a generic interface to *monitor* be a "waste
of time"?  In what way would that prevent someone from "producing a
*readlly good* replication implementation"?  I utterly fail to see the
connection.

Regards,
	Greg Copeland
15 Aug 2002 12:50:59 -0500
Greg Copeland <greg@CopelandConsulting.Net>
Re: Standard replication interface?
Greg Copeland <greg@CopelandConsulting.Net> writes:
> In what way would providing a generic interface to *monitor* be a
> "waste of time"?

As I said -- I don't really see the need for a bunch of replication
implementations, and therefore I don't see the need for a generic API
to make the whole mess (slightly) more manageable.

> In what way would that prevent someone from "producing a readlly
> good* replication implementation"?

It wouldn't -- it's just that if/when such an implementation exists
and everyone who needs replication is using it, a "generic monitoring
API" would be pointless.

Cheers,

Neil

-- 
Neil Conway <neilconway@rogers.com>
PGP Key ID: DB3C29FC


15 Aug 2002 13:57:08 -0400
Neil Conway <nconway@klamath.dyndns.org>
Re: Standard replication interface?
> As I said -- I don't really see the need for a bunch of replication
> implementations, and therefore I don't see the need for a generic API
> to make the whole mess (slightly) more manageable.

I see.  So the intension of the core developers is to have one and only
one replication solution?

Greg
15 Aug 2002 13:03:53 -0500
Greg Copeland <greg@CopelandConsulting.Net>
Re: Standard replication interface?
Greg Copeland <greg@CopelandConsulting.Net> writes:
> > As I said -- I don't really see the need for a bunch of replication
> > implementations, and therefore I don't see the need for a generic API
> > to make the whole mess (slightly) more manageable.
> 
> I see.  So the intension of the core developers is to have one and only
> one replication solution?

Not being a core developer, I can't comment on their intentions.

That said, I _personally_ don't see the need for more than one or two
replication implementations. You might need more than one if you
wanted to do both lazy and eager replication, for example. But you
certainly don't need 5 or 6 or however many implementations exist at
the moment.

I think the reason there are a lot of different implementations at the
moment is that each one has some pretty serious problems. So rather
than trying to reduce the problem by making it slightly easier for the
different replication solutions to inter-operate, I think it's a
better idea to solve the problem outright by improving one of the
existing replication projects to the point at which it is ready for
widespread production usage.

Cheers,

Neil

-- 
Neil Conway <neilconway@rogers.com>
PGP Key ID: DB3C29FC


15 Aug 2002 14:18:15 -0400
Neil Conway <nconway@klamath.dyndns.org>
Re: Standard replication interface?
On Thu, 2002-08-15 at 13:18, Neil Conway wrote:
> That said, I _personally_ don't see the need for more than one or two
> replication implementations. You might need more than one if you
> wanted to do both lazy and eager replication, for example. But you
> certainly don't need 5 or 6 or however many implementations exist at
> the moment.

Fair enough.  Thank you for offering a complete explanation.

You're argument certainly made sense.  I wasn't aware of any single
serious effort underway which sought to finally put replication to bed,
let alone integrated into the core code base.

Sign,

	Greg Copeland
15 Aug 2002 13:37:59 -0500
Greg Copeland <greg@CopelandConsulting.Net>
Re: Standard replication interface?
Rod Taylor writes:

> Anyone mind if we bump the DTD version to Docbook 4.2?

Not sure if we should do this now.  We're approaching the time where
people should be writing documentation, not having to refiddle their
carefully crafted DocBook installations.  We're not going to realize any
immediate benefits anyway.

> What it buys is a number of useful tags, SVGs and probably more
> importantly for the future, xsl and fop support which will probably be
> important in the future.  OpenJade hasn't had a new release in quite a
> long time -- not to say work isn't needed.

The last release was in January.

> Yes, after updating docs to the newer DTD I intend to make them XML
> compliant to ensure they work with v5 of docbook in the future.

Ah, an XML vs. SGML debate.  I look forward to it.

-- 
Peter Eisentraut   peter_e@gmx.net


Thu, 15 Aug 2002 21:30:38 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Documentation DTD

> > Yes, after updating docs to the newer DTD I intend to make them XML
> > compliant to ensure they work with v5 of docbook in the future.
> 
> Ah, an XML vs. SGML debate.  I look forward to it.

This one is pretty simple.  It's been announced that the docbook group
isn't looking to continue with SGML.  This is shown on the oasis-open
pages as well as their discussion in the mailing lists (xsltproc and fop
rather than jade and dsssl).

I prefer working with SGML, but not enough to try hacking away at
openjade to finish it off :)


Anyway, you're right about the patch.  Lets apply it to the 7.4 tree 
after branching.


15 Aug 2002 15:33:11 -0400
Rod Taylor <rbt@zort.ca>
Re: Documentation DTD
> --=-QQHYShMlxI2BY71i6NiO
> Content-Type: text/plain
> Content-Transfer-Encoding: quoted-printable
> 
> > As I said -- I don't really see the need for a bunch of replication
> > implementations, and therefore I don't see the need for a generic API
> > to make the whole mess (slightly) more manageable.
> 
> I see.  So the intension of the core developers is to have one and only
> one replication solution?

If the various "solutions" may be folded down into a smaller set of programs, 
perhaps, ultimately, into _one_ program, that would surely be easier to 
manage, in the codebase, than having five or six such programs.

If one program can do the job that needs to be done, and it has not been 
_clearly_ established that that is _not_ possible, then I'd think it rather 
silly to have a bunch of "replication solutions" that need to be updated any 
time a relevant change goes into the database engine.

I'd be surprised if, in the end, there truly _needed_ to be more than about 
two approaches.

Should the team plan to _have_ a mess?  I'd think not.
--
(concatenate 'string "cbbrowne" "@ntlug.org")
http://cbbrowne.com/info/linuxdistributions.html
"We don't understand the  software, and sometimes we don't  understand
the hardware, but we can *see* the blinking lights!"  -- Unknown



Thu, 15 Aug 2002 16:01:04 -0400
cbbrowne@cbbrowne.com
Re: Standard replication interface? 
Neil Conway <nconway@klamath.dyndns.org> writes:
> Greg Copeland <greg@CopelandConsulting.Net> writes:
>> I see.  So the intension of the core developers is to have one and only
>> one replication solution?

> Not being a core developer, I can't comment on their intentions.

Well, I am, but I'm only speaking for myself here:

I think there's definitely a need for at least two replication
implementations: sync and async.  The space of requirements is wide
enough that there's not a one-size-fits-all solution.  You might care
to look at Darren Johnson's OSCON slides for more about this:
http://conferences.oreillynet.com/cs/os2002/view/e_sess/3280
I think there is room for several replication solutions for Postgres
(three or four, maybe).

It's difficult to say what will wind up in our core distribution.
A tightly linked implementation like Postgres-R is really impractical
as an add-on: you need enough mods of the core code that it'd be a
nightmare to try to maintain if it's not integrated into the regular
CVS tree.  So assuming that the Postgres-R project gets to the point
of usefulness, I'd vote in favor of integrating it.  On the other hand,
it's possible to do good stuff without touching the core code at all
(cf. PostgreSQL Inc's rserv) and in that case there may or may not be
any interest in integrating the code.  It's really gonna depend mostly
on the wishes of the people who develop the replication solutions,
I think.

I can foresee a time when there are one or two replication solutions
that are included in the base distribution and others are available
separately.  In fact, counting contrib/rserv that more or less describes
the state of affairs today.  What we need is more work on the available
solutions to improve their quality and general usefulness.

As for the point at hand: I'm fairly dubious that a common monitoring
API will be very useful, considering how different the possible
replication approaches are.  If Greg can prove me wrong, fine.  But
I don't want to see us artificially constraining replication solutions
by insisting that they meet some prespecified API.

			regards, tom lane

Thu, 15 Aug 2002 16:36:31 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Standard replication interface? 
> Rod Taylor writes:
> 
> > Anyone mind if we bump the DTD version to Docbook 4.2?
> 
> Not sure if we should do this now.  We're approaching the time where
> people should be writing documentation, not having to refiddle their
> carefully crafted DocBook installations.  We're not going to realize any
> immediate benefits anyway.

Indeed.

> > What it buys is a number of useful tags, SVGs and probably more
> > importantly for the future, xsl and fop support which will probably be
> > important in the future.  OpenJade hasn't had a new release in quite a
> > long time -- not to say work isn't needed.
> 
> The last release was in January.
> 
> > Yes, after updating docs to the newer DTD I intend to make them XML
> > compliant to ensure they work with v5 of docbook in the future.
> 
> Ah, an XML vs. SGML debate.  I look forward to it.

Please no!

If and when it becomes forcibly preferable to use XML, there's a
tool called sgml2xml that is part of the "sp" package (which includes nsgmls 
and sgmlnorm) that does a Perfectly Good Job of this.  Totally automated.

Possible exception: sgml2xml capitalizes all the tags, and it looks like the 
XML DTD wants MixedCaseTagging, which is a rather irritating thing about XML; 
in any case, that's something that should be fixed up in one fell swoop in a 
"normalize it all and make it into XML" process LATER.

It would make sense to fix use of any deprecated elements, but "fixing" any 
XML aspects of it now is pretty much a senseless exercise.
--
(reverse (concatenate 'string "moc.enworbbc@" "enworbbc"))
http://www.ntlug.org/~cbbrowne/emacs.html
"Computers in the future may weigh no more than 1.5 tons".  -- POPULAR
MECHANICS magazine forecasting the "relentless march of science" 1955



Thu, 15 Aug 2002 18:33:41 -0400
cbbrowne@cbbrowne.com
Re: Documentation DTD 
On Thu, 2002-08-15 at 15:36, Tom Lane wrote:
> Well, I am, but I'm only speaking for myself here:
> 

Fair enough.

> I think there is room for several replication solutions for Postgres
> (three or four, maybe).

If the ideal solution count is merely one with a maybe on two then I
tend to concur that any specification along these lines would *mostly*
be a waste.  On the other hand, if we can count three or more possible
replication solutions, IMHO, there seemingly would be merit is providing
some sort of defacto monitoring interface.

Seems the current difficulty is forecasting the future in this regard. 
Perhaps other core developers would care to chime in and share their
vision?

> CVS tree.  So assuming that the Postgres-R project gets to the point
> of usefulness, I'd vote in favor of integrating it.  On the other hand,

I guess I should ask.  Do the developers foresee immediate usability
from this project or are we looking at something that's a year+ away?  I
don't think I have a problem helping guide what could be an interim
solution if the interim window were large enough.  In theory, monitoring
tools developed between now and the closing of the window could largely
continue to function without change.  That, of course, assumes that even
the end-run solutions would implement the interface as well.

The return on such a concept is that it allows generic monitoring tools
to mature while providing value now and in the future.  The end result
should be a stronger, more powerful tool base which matures while other
technologies are still being developed.

Another question along this line is, once something rolls into a core
position, does that obsolete all other existing implementations or
merely become the defacto in a bag of solutions?  Tom seems to hint at
the later.  If the answer is the former then that seemingly argues not
to worry about this...unless the window for usefulness and/or inclusion
is rather large.

> As for the point at hand: I'm fairly dubious that a common monitoring
> API will be very useful, considering how different the possible

Well, all replication scenarios have a lot in common.  They should, 
after all, they are all doing the same thing.  Since the different
strategies for accomplishing replication are well understood, it seems
well within reason to assume that someone can put their brain around
this.

I can also imagine that the specification includes requirements as well
as optional facilities.  Certainly capability queries would further iron
out any gaps between differing solutions/implementations.

> replication approaches are.  If Greg can prove me wrong, fine.  But
> I don't want to see us artificially constraining replication solutions
> by insisting that they meet some prespecified API.

Hmmm.  I'm not sure how it would act as a constraining force.  To me,
this implies that any such specification would fail to evolve and could
not be revised based on feedback.  IMO, most specifications are regarded
as living documents.  While I can see that some specifications are set
in stone, I certainly am not so bold as to assert my crystal ball even
came with batteries.  ;)  That is, I assume some level of revision to an
initial specification would be required following real-world use.


Regards,

	Greg Copeland
16 Aug 2002 09:20:11 -0500
Greg Copeland <greg@CopelandConsulting.Net>
Re: Standard replication interface?
Greg Copeland <greg@copelandconsulting.net> writes:
> I guess I should ask.  Do the developers foresee immediate usability
> from [Postgres-R] or are we looking at something that's a year+ away?

Darren Johnson would be the man to answer that, but from what he said
at OSCON it sounded like we'd be seeing something useful by the end of
the year, with all the usual caveats about time actually being available
to work on it.

>> As for the point at hand: I'm fairly dubious that a common monitoring
>> API will be very useful, considering how different the possible

> Well, all replication scenarios have a lot in common.  They should,=20
> after all, they are all doing the same thing.

The end goal is approximately the same, but the mechanisms are totally
different, and that means that what you want to monitor is totally
different.

Perhaps the problem is that you're using the wrong word, and that what
you would like to standardize is not monitoring but administrative
functions.  For example, I'd classify selecting tables to be replicated
as an admin task.  Monitoring to me means something like "how much data
is in the queue to be pushed out to slave X?", which is a question that
already presupposes a heck of a lot about the implementation.

I could agree with a set of guidelines that say stuff like "if your
mechanism is capable of selecting individual tables to replicate,
then here's the preferred way to control that feature."  But I'm not
sure that there's enough common functionality for monitoring (in the
above sense) to be worth standardizing.

			regards, tom lane

Fri, 16 Aug 2002 10:35:26 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Standard replication interface? 
Rod Taylor writes:

> This one is pretty simple.  It's been announced that the docbook group
> isn't looking to continue with SGML.

I don't know where you got this from, but it's not true.  DocBook 5 will
support SGML.  And as long as they publish DTDs you can use them with SGML
tools anyway.

-- 
Peter Eisentraut   peter_e@gmx.net


Sat, 17 Aug 2002 17:47:59 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Documentation DTD
On Sat, 2002-08-17 at 11:47, Peter Eisentraut wrote:
> Rod Taylor writes:
> 
> > This one is pretty simple.  It's been announced that the docbook group
> > isn't looking to continue with SGML.
> 
> I don't know where you got this from, but it's not true.  DocBook 5 will
> support SGML.  And as long as they publish DTDs you can use them with SGML
> tools anyway.

Yes, jade and friends will work.  But Fop is quickly catching up to the
dsssl abilities and can already do some things much cleaner (title
pages, headers and footers).

Anyway, XML or SGML doesn't really matter.  There are a number of
enhancements I'd like to make to the doc process which won't be affected
either way.  Auto-generated example output, and others to help things
stay in sync.


17 Aug 2002 13:56:03 -0400
Rod Taylor <rbt@zort.ca>
Re: [HACKERS] Documentation DTD
Rod Taylor writes:

> Yes, jade and friends will work.  But Fop is quickly catching up to the
> dsssl abilities and can already do some things much cleaner (title
> pages, headers and footers).

The real concern is that the XSLT stylesheets aren't anywhere near the
maturity of the DSSSL releases.  I occasionally build the PostgreSQL
documentation with various combinations of XSL tools and the results are
basically too ugly to look at -- if you get anything to look at in the
first place.

-- 
Peter Eisentraut   peter_e@gmx.net


Sun, 18 Aug 2002 23:35:20 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Documentation DTD

will announce it on -announce tomorrow, if ppl want to take a quick look
at it ... man pages weren't included, but I did regenerate the docs per
Peter's suggested commands ...

Scary, even with removing a load of stuff over to gborg, its still gotten
bigger then the last release :)

%ls -lt ~ftp/pub/source/v7.3beta
total 21125
-rw-r--r--  1 pgsql  pgsql        70 Sep  4 22:28 postgresql-test-7.3b1.tar.gz.md5
-rw-r--r--  1 pgsql  pgsql        65 Sep  4 22:28 postgresql-7.3b1.tar.gz.md5
-rw-r--r--  1 pgsql  pgsql        70 Sep  4 22:28 postgresql-base-7.3b1.tar.gz.md5
-rw-r--r--  1 pgsql  pgsql        70 Sep  4 22:28 postgresql-docs-7.3b1.tar.gz.md5
-rw-r--r--  1 pgsql  pgsql        69 Sep  4 22:28 postgresql-opt-7.3b1.tar.gz.md5
-rw-r--r--  1 pgsql  pgsql   1070154 Sep  4 22:28 postgresql-test-7.3b1.tar.gz
-rw-r--r--  1 pgsql  pgsql   2629533 Sep  4 22:28 postgresql-opt-7.3b1.tar.gz
-rw-r--r--  1 pgsql  pgsql   2577818 Sep  4 22:28 postgresql-docs-7.3b1.tar.gz
-rw-r--r--  1 pgsql  pgsql   4505929 Sep  4 22:28 postgresql-base-7.3b1.tar.gz
-rw-r--r--  1 pgsql  pgsql  10783992 Sep  4 22:27 postgresql-7.3b1.tar.gz



Wed, 4 Sep 2002 23:39:33 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
beta1 packaged
On Wed, 4 Sep 2002, Marc G. Fournier wrote:

> %ls -lt ~ftp/pub/source/v7.3beta

Is this where you're putting it this time?  Last time was ~ftp/pub/beta.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
         56K Nationwide Dialup from $16.00/mo at Pop4 Networking
      http://www.camping-usa.com      http://www.cloudninegifts.com
   http://www.meanstreamradio.com       http://www.unknown-artists.com
==========================================================================





Thu, 5 Sep 2002 05:51:51 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: beta1 packaged
On Thu, 5 Sep 2002, Vince Vielhaber wrote:

> On Wed, 4 Sep 2002, Marc G. Fournier wrote:
>
> > %ls -lt ~ftp/pub/source/v7.3beta
>
> Is this where you're putting it this time?  Last time was ~ftp/pub/beta.

actually, should be a symlink, but until I know the packaging and all is
well, I'm avoiding put it in there ...

>


Thu, 5 Sep 2002 09:16:07 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta1 packaged
On Thu, 5 Sep 2002, Marc G. Fournier wrote:

> On Thu, 5 Sep 2002, Vince Vielhaber wrote:
>
> > On Wed, 4 Sep 2002, Marc G. Fournier wrote:
> >
> > > %ls -lt ~ftp/pub/source/v7.3beta
> >
> > Is this where you're putting it this time?  Last time was ~ftp/pub/beta.
>
> actually, should be a symlink, but until I know the packaging and all is
> well, I'm avoiding put it in there ...

Ok, I'll leave the script as is then.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
         56K Nationwide Dialup from $16.00/mo at Pop4 Networking
      http://www.camping-usa.com      http://www.cloudninegifts.com
   http://www.meanstreamradio.com       http://www.unknown-artists.com
==========================================================================




Thu, 5 Sep 2002 08:19:10 -0400 (EDT)
Vince Vielhaber <vev@michvhf.com>
Re: beta1 packaged
"Marc G. Fournier" <scrappy@hub.org> writes:
> actually, should be a symlink, but until I know the packaging and all is
> well, I'm avoiding put it in there ...

I pulled down the main tarball -- looks good AFAICT.

			regards, tom lane

Thu, 05 Sep 2002 10:09:13 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta1 packaged 
On Wed, 2002-09-04 at 22:39, Marc G. Fournier wrote:
> 
> will announce it on -announce tomorrow, if ppl want to take a quick look
> at it ... man pages weren't included, but I did regenerate the docs per
> Peter's suggested commands ...

'./configure && make check' passes on i386 FreeBSD.

SunOS control.shared2 5.7 Generic_106541-20 sun4u sparc SUNW,Ultra-5_10
shows an error in ALTER TABLE tests:


c> cat src/test/regress/regression.diffs 
*** ./expected/alter_table.out  Fri Aug 30 12:23:20 2002
--- ./results/alter_table.out   Thu Sep  5 07:44:18 2002
***************
*** 367,374 ****
  -- As should this
  ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
pktable(ptest1);
  NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
check(s)
  DROP TABLE pktable cascade;
- NOTICE:  Drop cascades to constraint $2 on table fktable
  NOTICE:  Drop cascades to constraint $1 on table fktable
  DROP TABLE fktable;
  CREATE TEMP TABLE PKTABLE (ptest1 int, ptest2 inet,
--- 367,374 ----
  -- As should this
  ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
pktable(ptest1);
  NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
check(s)
+ ERROR:  Relation "pg_temp_5"."" does not exist
  DROP TABLE pktable cascade;
  NOTICE:  Drop cascades to constraint $1 on table fktable
  DROP TABLE fktable;
  CREATE TEMP TABLE PKTABLE (ptest1 int, ptest2 inet,

======================================================================




05 Sep 2002 10:47:07 -0400
Rod Taylor <rbt@zort.ca>
Re: beta1 packaged
Rod Taylor <rbt@zort.ca> writes:
> SunOS control.shared2 5.7 Generic_106541-20 sun4u sparc SUNW,Ultra-5_10
> shows an error in ALTER TABLE tests:

>   ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
> pktable(ptest1);
>   NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
> check(s)
> + ERROR:  Relation "pg_temp_5"."" does not exist

That's pretty bizarre.  Is it reproducible?  Can you get in there with a
debugger and try to figure out what's going wrong?

			regards, tom lane

Thu, 05 Sep 2002 11:19:06 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta1 packaged 

Guys,

postgresql7.3b1 does not build :-(, seems like a missing multibyte
directory

'----------------------------------------
| make[4]: Entering directory `/home/masm/download/postgresql-7.3b1/src/backend/utils/time'
| gcc -O2 -Wall -Wmissing-prototypes -Wmissing-declarations -I../../../../src/include   -c -o tqual.o tqual.c
| /usr/bin/ld -r -o SUBSYS.o tqual.o
| make[4]: Leaving directory `/home/masm/download/postgresql-7.3b1/src/backend/utils/time'
| make -C mb SUBSYS.o
| make: Entering an unknown directory
| make: *** mb: No such file or directory.  Stop.
| make: Leaving an unknown directory
| make[3]: *** [mb-recursive] Error 2
| make[3]: Leaving directory `/home/masm/download/postgresql-7.3b1/src/backend/utils'
| make[2]: *** [utils-recursive] Error 2
| make[2]: Leaving directory `/home/masm/download/postgresql-7.3b1/src/backend'
| make[1]: *** [all] Error 2
| make[1]: Leaving directory `/home/masm/download/postgresql-7.3b1/src'
| make: *** [all] Error 2
`----------------------------------------

or I'm missing something?

Regards,
Manuel.

05 Sep 2002 11:08:02 -0500
Manuel Sugawara <masm@fciencias.unam.mx>
Re: beta1 packaged
On Thu, 2002-09-05 at 11:19, Tom Lane wrote:
> Rod Taylor <rbt@zort.ca> writes:
> > SunOS control.shared2 5.7 Generic_106541-20 sun4u sparc SUNW,Ultra-5_10
> > shows an error in ALTER TABLE tests:
> 
> >   ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
> > pktable(ptest1);
> >   NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
> > check(s)
> > + ERROR:  Relation "pg_temp_5"."" does not exist
> 
> That's pretty bizarre.  Is it reproducible?  Can you get in there with a
> debugger and try to figure out what's going wrong?

No, I've been unable to reproduce.



05 Sep 2002 12:22:22 -0400
Rod Taylor <rbt@zort.ca>
Re: beta1 packaged
Manuel Sugawara <masm@fciencias.unam.mx> writes:
> or I'm missing something?

So it would seem.  The utils/mb directory is certainly there in the full
tarball that I pulled from ftp.us.postgresql.org this morning.  How did
you acquire your source tree, exactly?

			regards, tom lane

Thu, 05 Sep 2002 12:33:04 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta1 packaged 
Tom Lane <tgl@sss.pgh.pa.us> writes:

> So it would seem.  The utils/mb directory is certainly there in the full
> tarball that I pulled from ftp.us.postgresql.org this morning.  How did
> you acquire your source tree, exactly?

The file is postgresql-base-7.3b1.tar.gz from
ftp://ftp.postgresql.org/pub/source/v7.3beta/

may be I need postgresql-7.3b1.tar.gz?

Regards,
Manuel.

-- 
No es que no puedan hallar la solución: es que no ven el problema.
G.K. Chesterson

05 Sep 2002 11:46:25 -0500
Manuel Sugawara <masm@fciencias.unam.mx>
Re: beta1 packaged
"Marc G. Fournier" <scrappy@hub.org> writes:

> You need either the 7.3b1.tar.gz (which is everything), or you need to get
> all the various -*- parts (which are more manageable)

Oh, well. Thanks

Regards,
Manuel.
-- 
No es que no puedan hallar la solución: es que no ven el problema.
G.K. Chesterson

05 Sep 2002 11:48:50 -0500
Manuel Sugawara <masm@fciencias.unam.mx>
Re: beta1 packaged
On 5 Sep 2002, Manuel Sugawara wrote:

> Tom Lane <tgl@sss.pgh.pa.us> writes:
>
> > So it would seem.  The utils/mb directory is certainly there in the full
> > tarball that I pulled from ftp.us.postgresql.org this morning.  How did
> > you acquire your source tree, exactly?
>
> The file is postgresql-base-7.3b1.tar.gz from
> ftp://ftp.postgresql.org/pub/source/v7.3beta/
>
> may be I need postgresql-7.3b1.tar.gz?

You need either the 7.3b1.tar.gz (which is everything), or you need to get
all the various -*- parts (which are more manageable)



Thu, 5 Sep 2002 13:51:26 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta1 packaged
Marc G. Fournier wrote:
> On 5 Sep 2002, Manuel Sugawara wrote:
> 
> > Tom Lane <tgl@sss.pgh.pa.us> writes:
> >
> > > So it would seem.  The utils/mb directory is certainly there in the full
> > > tarball that I pulled from ftp.us.postgresql.org this morning.  How did
> > > you acquire your source tree, exactly?
> >
> > The file is postgresql-base-7.3b1.tar.gz from
> > ftp://ftp.postgresql.org/pub/source/v7.3beta/
> >
> > may be I need postgresql-7.3b1.tar.gz?
> 
> You need either the 7.3b1.tar.gz (which is everything), or you need to get
> all the various -*- parts (which are more manageable)

I am confused.  Are you saying the base file isn't compilable?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 5 Sep 2002 13:24:38 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta1 packaged
Bruce Momjian <pgman@candle.pha.pa.us> writes:

> > You need either the 7.3b1.tar.gz (which is everything), or you need to get
> > all the various -*- parts (which are more manageable)
> 
> I am confused.  Are you saying the base file isn't compilable?

My idea was that it is.

Regards,
Manuel.

05 Sep 2002 12:50:25 -0500
Manuel Sugawara <masm@fciencias.unam.mx>
Re: beta1 packaged
On Thu, 5 Sep 2002, Bruce Momjian wrote:

> Marc G. Fournier wrote:
> > On 5 Sep 2002, Manuel Sugawara wrote:
> >
> > > Tom Lane <tgl@sss.pgh.pa.us> writes:
> > >
> > > > So it would seem.  The utils/mb directory is certainly there in the full
> > > > tarball that I pulled from ftp.us.postgresql.org this morning.  How did
> > > > you acquire your source tree, exactly?
> > >
> > > The file is postgresql-base-7.3b1.tar.gz from
> > > ftp://ftp.postgresql.org/pub/source/v7.3beta/
> > >
> > > may be I need postgresql-7.3b1.tar.gz?
> >
> > You need either the 7.3b1.tar.gz (which is everything), or you need to get
> > all the various -*- parts (which are more manageable)
>
> I am confused.  Are you saying the base file isn't compilable?

Hrmm ... that is odd, now that you mention it ... but the file
'distributions' between v7.2 and v7.3beta appear to be the same, so -base-
was broken in the old one too?



Thu, 5 Sep 2002 15:15:39 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta1 packaged
Bruce Momjian writes:

> I am confused.  Are you saying the base file isn't compilable?

The mb stuff is missing because it used to be optional in the old
splitting scheme.  Needs to be rethought.

-- 
Peter Eisentraut   peter_e@gmx.net


Thu, 5 Sep 2002 20:27:33 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: beta1 packaged
Marc G. Fournier writes:

> Scary, even with removing a load of stuff over to gborg, its still gotten
> bigger then the last release :)

Not hard to find the culprit:

7.2:

3.4M    src/backend/utils/mb

7.3:

9.6M    src/backend/utils/mb

-- 
Peter Eisentraut   peter_e@gmx.net


Thu, 5 Sep 2002 20:28:28 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: beta1 packaged
"Marc G. Fournier" <scrappy@hub.org> writes:
> Hrmm ... that is odd, now that you mention it ... but the file
> 'distributions' between v7.2 and v7.3beta appear to be the same, so -base-
> was broken in the old one too?

It was never intended that the "base" tarfile was alone sufficient to do
anything, was it?

			regards, tom lane

Thu, 05 Sep 2002 14:47:19 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta1 packaged 
Tom Lane wrote:
> "Marc G. Fournier" <scrappy@hub.org> writes:
> > Hrmm ... that is odd, now that you mention it ... but the file
> > 'distributions' between v7.2 and v7.3beta appear to be the same, so -base-
> > was broken in the old one too?
> 
> It was never intended that the "base" tarfile was alone sufficient to do
> anything, was it?

OK, so if base isn't compilable, then what is it good for?  I don't see
any add-on packages that would make it usable.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 5 Sep 2002 14:49:43 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta1 packaged
Peter Eisentraut wrote:
> Marc G. Fournier writes:
> 
> > Scary, even with removing a load of stuff over to gborg, its still gotten
> > bigger then the last release :)
> 
> Not hard to find the culprit:
> 
> 7.2:
> 
> 3.4M    src/backend/utils/mb
> 
> 7.3:
> 
> 9.6M    src/backend/utils/mb

Wow.  Just checking my CVS, the /mb stuff appears to be 17% of our
source tree.  Now that they are loadable, can we get enough of it in the
base to make it compilable and leave the rest for the full tarball.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 5 Sep 2002 14:56:06 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta1 packaged
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> OK, so if base isn't compilable, then what is it good for?  I don't see
> any add-on packages that would make it usable.

AFAIR, the only reason for having the split packaging is to accommodate
people who are downloading across flaky connections --- less to retry if
your connection drops.  You still have to download all the data if you
want to have a useful package, no?

			regards, tom lane

Thu, 05 Sep 2002 16:13:06 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta1 packaged 
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > OK, so if base isn't compilable, then what is it good for?  I don't see
> > any add-on packages that would make it usable.
> 
> AFAIR, the only reason for having the split packaging is to accommodate
> people who are downloading across flaky connections --- less to retry if
> your connection drops.  You still have to download all the data if you
> want to have a useful package, no?

That was not my understanding.  If that was the issue, we would release
packages numbered 1-5.  Marc splits them up saying if you don't want the
docs, don't download them.  I assume opt is /contrib.  My guess is he
had the multibyte stuff out in /opt but now they are required.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 5 Sep 2002 16:38:03 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta1 packaged
On Fri, 2002-09-06 at 03:14, Marc G. Fournier wrote:
> 0On Thu, 5 Sep 2002, Tom Lane wrote:
> 
> > Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > > OK, so if base isn't compilable, then what is it good for?  I don't see
> > > any add-on packages that would make it usable.
> >
> > AFAIR, the only reason for having the split packaging is to accommodate
> > people who are downloading across flaky connections --- less to retry if
> > your connection drops.  You still have to download all the data if you
> > want to have a useful package, no?
> 
> Correct ... even on high speed connections, I've had problems in the past
> getting large files to download, so it makes it easier if you only have to
> retry a part, instead of hte whole thing ...

Most modern ftp servers and clients support resuming aborted downloads,
no ?


[hannu@rh72 DL]$ ncftp ftp://ftp.postgresql.org/pub/source/v7.3beta/
NcFTP 3.1.3 (Mar 27, 2002) by Mike Gleason (ncftp@ncftp.com).
Connecting to
64.49.215.8...                                                                                                 
postgresql.org FTP server (lukemftpd 1.2 beta 1) ready.
Logging
in...                                                                                                                
Guest login ok, access restrictions apply.
Logged in to
ftp.postgresql.org.                                                                                             
Current remote directory is /pub/source/v7.3beta.
ncftp /pub/source/v7.3beta > get postgresql-test-7.3b1.tar.gz
postgresql-test-7.3b1.tar.gz:   ETA: 0:03  0.63/  1.02 MB  131.44 kB/s 

*** NB! I pressed ^C here

select: Interrupted system call
postgresql-test-7.3b1.tar.gz:              0.72/  1.02 MB  112.17 kB/s  
get postgresql-test-7.3b1.tar.gz: data transfer aborted by local user.
ncftp /pub/source/v7.3beta > get postgresql-test-7.3b1.tar.gz

The local file "postgresql-test-7.3b1.tar.gz" already exists.
    Local:        753768 bytes, dated Thu Sep 05 07:28:40 GMT-5 2002.
    Remote:      1070154 bytes, dated Thu Sep 05 07:28:40 GMT-5 2002.

        [O]verwrite?  [R]esume?  [A]ppend to?  [S]kip?  [N]ew Name?
        [O!]verwrite all?  [R!]esume all?  [S!]kip all?  [C]ancel  > R
postgresql-test-7.3b1.tar.gz:                            1.02 MB  112.59
kB/s  
ncftp /pub/source/v7.3beta >


-----------
Hannu




06 Sep 2002 01:59:13 +0500
Hannu Krosing <hannu@tm.ee>
Re: beta1 packaged
0On Thu, 5 Sep 2002, Tom Lane wrote:

> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > OK, so if base isn't compilable, then what is it good for?  I don't see
> > any add-on packages that would make it usable.
>
> AFAIR, the only reason for having the split packaging is to accommodate
> people who are downloading across flaky connections --- less to retry if
> your connection drops.  You still have to download all the data if you
> want to have a useful package, no?

Correct ... even on high speed connections, I've had problems in the past
getting large files to download, so it makes it easier if you only have to
retry a part, instead of hte whole thing ...



Thu, 5 Sep 2002 19:14:07 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta1 packaged 
Marc G. Fournier wrote:
> 0On Thu, 5 Sep 2002, Tom Lane wrote:
> 
> > Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > > OK, so if base isn't compilable, then what is it good for?  I don't see
> > > any add-on packages that would make it usable.
> >
> > AFAIR, the only reason for having the split packaging is to accommodate
> > people who are downloading across flaky connections --- less to retry if
> > your connection drops.  You still have to download all the data if you
> > want to have a useful package, no?
> 
> Correct ... even on high speed connections, I've had problems in the past
> getting large files to download, so it makes it easier if you only have to
> retry a part, instead of hte whole thing ...

Then why do we mark them with stuff.  Isn't it easier to just label them
1-5?  I can see the docs being split out, but I don't understand the
other splits.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 5 Sep 2002 18:15:14 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta1 packaged
On Thu, 5 Sep 2002, Bruce Momjian wrote:

> Tom Lane wrote:
> > Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > > OK, so if base isn't compilable, then what is it good for?  I don't see
> > > any add-on packages that would make it usable.
> >
> > AFAIR, the only reason for having the split packaging is to accommodate
> > people who are downloading across flaky connections --- less to retry if
> > your connection drops.  You still have to download all the data if you
> > want to have a useful package, no?
>
> That was not my understanding.  If that was the issue, we would release
> packages numbered 1-5.  Marc splits them up saying if you don't want the
> docs, don't download them.  I assume opt is /contrib.  My guess is he
> had the multibyte stuff out in /opt but now they are required.

Actually, I just asked for the split, I think it was peter that actually
did it ... :)


Thu, 5 Sep 2002 19:15:21 -0300 (ADT)
"Marc G. Fournier" <scrappy@hub.org>
Re: beta1 packaged

Tom Lane writes:
> Rod Taylor <rbt@zort.ca> writes:
> > SunOS control.shared2 5.7 Generic_106541-20 sun4u sparc SUNW,Ultra-5_10
> > shows an error in ALTER TABLE tests:
> 
> >   ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
> > pktable(ptest1);
> >   NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
> > check(s)
> > + ERROR:  Relation "pg_temp_5"."" does not exist
> 
> That's pretty bizarre.  Is it reproducible?  Can you get in there with a
> debugger and try to figure out what's going wrong?

I saw a similar error on a NetBSD-1.5.1/i386 box, but have not been
able to reproduce it. Subsequent runs of 'gmake check' have all
passed.

Until I saw Rod's message I was thinking it was more evidence of
hardware flakiness with this particular machine, but perhaps not.

*** ./expected/alter_table.out  Sat Aug 31 05:23:20 2002
--- ./results/alter_table.out   Fri Sep  6 16:54:35 2002
***************
*** 332,337 ****
--- 332,338 ----
  -- Try (and succeed)
  ALTER TABLE tmp3 add constraint tmpconstr foreign key (a) references tmp2 matc
h full;
  NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY check(s)
+ ERROR:  Relation "public"."^B^U&W<88><F0>0}" does not exist
  -- Try (and fail) to create constraint from tmp5(a) to tmp4(a) - unique constr
aint on
  -- tmp4 is a,b
  ALTER TABLE tmp5 add constraint tmpconstr foreign key(a) references tmp4(a) ma
tch full;

Regards,

Giles

Sat, 07 Sep 2002 00:02:27 +1000
Giles Lean <giles@nemeton.com.au>
Re: beta1 packaged 
Marc G. Fournier writes:

> Actually, I just asked for the split, I think it was peter that actually
> did it ... :)

I recall that you thought of the split in order to save bandwidth for
those who didn't need everything.  It was expressedly intended that the
-base tarball was usable by itself and that you only needed the others if
you wanted any of the optional features (--with-* etc.).

But now that the optional stuff has mostly either gone away or isn't
optional anymore a revised split would come out pretty skewed:

-rw-r--r--    1 peter    users    10824414 Sep  6 23:21 postgresql-7.3b1.tar.gz
-rw-r--r--    1 peter    users     6675930 Sep  6 23:25 postgresql-base-7.3b1.tar.gz
-rw-r--r--    1 peter    users     2585621 Sep  6 23:30 postgresql-docs-7.3b1.tar.gz
-rw-r--r--    1 peter    users      485095 Sep  6 23:30 postgresql-opt-7.3b1.tar.gz
-rw-r--r--    1 peter    users     1072069 Sep  6 23:30 postgresql-test-7.3b1.tar.gz

-- 
Peter Eisentraut   peter_e@gmx.net


Sat, 7 Sep 2002 00:01:10 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: beta1 packaged

Rod, are you still seeing this failure?

---------------------------------------------------------------------------

Rod Taylor wrote:
> On Wed, 2002-09-04 at 22:39, Marc G. Fournier wrote:
> > 
> > will announce it on -announce tomorrow, if ppl want to take a quick look
> > at it ... man pages weren't included, but I did regenerate the docs per
> > Peter's suggested commands ...
> 
> './configure && make check' passes on i386 FreeBSD.
> 
> SunOS control.shared2 5.7 Generic_106541-20 sun4u sparc SUNW,Ultra-5_10
> shows an error in ALTER TABLE tests:
> 
> 
> c> cat src/test/regress/regression.diffs 
> *** ./expected/alter_table.out  Fri Aug 30 12:23:20 2002
> --- ./results/alter_table.out   Thu Sep  5 07:44:18 2002
> ***************
> *** 367,374 ****
>   -- As should this
>   ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
> pktable(ptest1);
>   NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
> check(s)
>   DROP TABLE pktable cascade;
> - NOTICE:  Drop cascades to constraint $2 on table fktable
>   NOTICE:  Drop cascades to constraint $1 on table fktable
>   DROP TABLE fktable;
>   CREATE TEMP TABLE PKTABLE (ptest1 int, ptest2 inet,
> --- 367,374 ----
>   -- As should this
>   ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
> pktable(ptest1);
>   NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
> check(s)
> + ERROR:  Relation "pg_temp_5"."" does not exist
>   DROP TABLE pktable cascade;
>   NOTICE:  Drop cascades to constraint $1 on table fktable
>   DROP TABLE fktable;
>   CREATE TEMP TABLE PKTABLE (ptest1 int, ptest2 inet,
> 
> ======================================================================
> 
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 10 Sep 2002 22:50:20 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta1 packaged
I've not been able to reproduce it, so no.  But there was another report
from someone else about the same failure on another platform.

On Tue, 2002-09-10 at 22:50, Bruce Momjian wrote:
> 
> Rod, are you still seeing this failure?
> 
> ---------------------------------------------------------------------------
> 
> Rod Taylor wrote:
> > On Wed, 2002-09-04 at 22:39, Marc G. Fournier wrote:
> > > 
> > > will announce it on -announce tomorrow, if ppl want to take a quick look
> > > at it ... man pages weren't included, but I did regenerate the docs per
> > > Peter's suggested commands ...
> > 
> > './configure && make check' passes on i386 FreeBSD.
> > 
> > SunOS control.shared2 5.7 Generic_106541-20 sun4u sparc SUNW,Ultra-5_10
> > shows an error in ALTER TABLE tests:
> > 
> > 
> > c> cat src/test/regress/regression.diffs 
> > *** ./expected/alter_table.out  Fri Aug 30 12:23:20 2002
> > --- ./results/alter_table.out   Thu Sep  5 07:44:18 2002
> > ***************
> > *** 367,374 ****
> >   -- As should this
> >   ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
> > pktable(ptest1);
> >   NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
> > check(s)
> >   DROP TABLE pktable cascade;
> > - NOTICE:  Drop cascades to constraint $2 on table fktable
> >   NOTICE:  Drop cascades to constraint $1 on table fktable
> >   DROP TABLE fktable;
> >   CREATE TEMP TABLE PKTABLE (ptest1 int, ptest2 inet,
> > --- 367,374 ----
> >   -- As should this
> >   ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
> > pktable(ptest1);
> >   NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
> > check(s)
> > + ERROR:  Relation "pg_temp_5"."" does not exist
> >   DROP TABLE pktable cascade;
> >   NOTICE:  Drop cascades to constraint $1 on table fktable
> >   DROP TABLE fktable;
> >   CREATE TEMP TABLE PKTABLE (ptest1 int, ptest2 inet,
> > 
> > ======================================================================
> > 
> > 
> > 
> > 
> > ---------------------------(end of broadcast)---------------------------
> > TIP 5: Have you checked our extensive FAQ?
> > 
> > http://www.postgresql.org/users-lounge/docs/faq.html
> > 
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
> 
-- 
  Rod Taylor


10 Sep 2002 22:52:36 -0400
Rod Taylor <rbt@zort.ca>
Re: beta1 packaged

Yep, and he couldn't reproduce it either, and on a different platform. 
I think that indicates we do have a problem in there, it just doesn't
show very often.  He even got ASCII garbage in the error message.

---------------------------------------------------------------------------

Rod Taylor wrote:
> I've not been able to reproduce it, so no.  But there was another report
> from someone else about the same failure on another platform.
> 
> On Tue, 2002-09-10 at 22:50, Bruce Momjian wrote:
> > 
> > Rod, are you still seeing this failure?
> > 
> > ---------------------------------------------------------------------------
> > 
> > Rod Taylor wrote:
> > > On Wed, 2002-09-04 at 22:39, Marc G. Fournier wrote:
> > > > 
> > > > will announce it on -announce tomorrow, if ppl want to take a quick look
> > > > at it ... man pages weren't included, but I did regenerate the docs per
> > > > Peter's suggested commands ...
> > > 
> > > './configure && make check' passes on i386 FreeBSD.
> > > 
> > > SunOS control.shared2 5.7 Generic_106541-20 sun4u sparc SUNW,Ultra-5_10
> > > shows an error in ALTER TABLE tests:
> > > 
> > > 
> > > c> cat src/test/regress/regression.diffs 
> > > *** ./expected/alter_table.out  Fri Aug 30 12:23:20 2002
> > > --- ./results/alter_table.out   Thu Sep  5 07:44:18 2002
> > > ***************
> > > *** 367,374 ****
> > >   -- As should this
> > >   ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
> > > pktable(ptest1);
> > >   NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
> > > check(s)
> > >   DROP TABLE pktable cascade;
> > > - NOTICE:  Drop cascades to constraint $2 on table fktable
> > >   NOTICE:  Drop cascades to constraint $1 on table fktable
> > >   DROP TABLE fktable;
> > >   CREATE TEMP TABLE PKTABLE (ptest1 int, ptest2 inet,
> > > --- 367,374 ----
> > >   -- As should this
> > >   ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
> > > pktable(ptest1);
> > >   NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
> > > check(s)
> > > + ERROR:  Relation "pg_temp_5"."" does not exist
> > >   DROP TABLE pktable cascade;
> > >   NOTICE:  Drop cascades to constraint $1 on table fktable
> > >   DROP TABLE fktable;
> > >   CREATE TEMP TABLE PKTABLE (ptest1 int, ptest2 inet,
> > > 
> > > ======================================================================
> > > 
> > > 
> > > 
> > > 
> > > ---------------------------(end of broadcast)---------------------------
> > > TIP 5: Have you checked our extensive FAQ?
> > > 
> > > http://www.postgresql.org/users-lounge/docs/faq.html
> > > 
> > 
> > -- 
> >   Bruce Momjian                        |  http://candle.pha.pa.us
> >   pgman@candle.pha.pa.us               |  (610) 359-1001
> >   +  If your life is a hard drive,     |  13 Roberts Road
> >   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
> > 
> -- 
>   Rod Taylor
> 
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 10 Sep 2002 22:55:48 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: beta1 packaged
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Yep, and he couldn't reproduce it either, and on a different platform. 
> I think that indicates we do have a problem in there, it just doesn't
> show very often.

I agree, this looks a lot like a low-probability bug.  But how to attack
it when we can't reproduce it with even small probability?  We need the
reporters to try to figure out what environment made it happen for them.
I can chase a bug if I can make it happen one-time-in-ten, or even
one-time-in-a-hundred, but I can't do much with a bug that I've only
heard secondhand reports of.

			regards, tom lane

Tue, 10 Sep 2002 23:53:46 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta1 packaged 
Giles Lean <giles@nemeton.com.au> writes:
>> Rod Taylor <rbt@zort.ca> writes:
>>> ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references
>>> pktable(ptest1);
>>> NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY
>>> check(s)
>>> + ERROR:  Relation "pg_temp_5"."" does not exist
>> 
>> That's pretty bizarre.  Is it reproducible?  Can you get in there with a
>> debugger and try to figure out what's going wrong?

> I saw a similar error on a NetBSD-1.5.1/i386 box, but have not been
> able to reproduce it. Subsequent runs of 'gmake check' have all
> passed.

> Until I saw Rod's message I was thinking it was more evidence of
> hardware flakiness with this particular machine, but perhaps not.

>   NOTICE:  ALTER TABLE will create implicit trigger(s) for FOREIGN KEY check(s)
> + ERROR:  Relation "public"."^B^U&W<88><F0>0}" does not exist


I've applied the attached patch, which I think may cure these failures.

			regards, tom lane


*** src/backend/commands/tablecmds.c.orig	Wed Sep  4 17:30:18 2002
--- src/backend/commands/tablecmds.c	Thu Sep 12 17:06:58 2002
***************
*** 2920,2926 ****
  	 * unfortunately).
  	 */
  	myRel = makeRangeVar(get_namespace_name(RelationGetNamespace(rel)),
! 						 RelationGetRelationName(rel));
  
  	/*
  	 * Preset objectAddress fields
--- 2920,2926 ----
  	 * unfortunately).
  	 */
  	myRel = makeRangeVar(get_namespace_name(RelationGetNamespace(rel)),
! 						 pstrdup(RelationGetRelationName(rel)));
  
  	/*
  	 * Preset objectAddress fields

Thu, 12 Sep 2002 17:18:54 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: beta1 packaged 
I have the beginnings of an idea about improving our interlock logic
for postmaster startup.  The existing method is pretty good, but we
have had multiple reports that it can fail during system boot if the
old postmaster wasn't given a chance to shut down cleanly: there's
a fair-sized chance that the old postmaster PID will have been assigned
to some other process, and that fools the interlock check.

I think we can improve matters by combining the existing checks for
old-postmaster-PID and old-shared-memory-segment into one cohesive
entity.  To do this, we must abandon the existing special case for
"private memory" when running a bootstrap or standalone backend.
Even a standalone backend will be required to get a shmem segment
just like a postmaster would.  This ensures that we can use both
parts of the safety check, even when the old holder of the data
directory interlock was a standalone backend.

Here's a sketch of the improved startup procedure:

1. Try to open and read the $PGDATA/postmaster.pid file.  If we fail
because it's not there, okay to continue, because old postmaster must
have shut down cleanly; skip to step 8.  If we fail for any other reason
(eg, permissions failure), complain and abort startup.  (Because we
write the postmaster.pid file mode 600, getting past this step
guarantees we are either the same UID as the old postmaster or root;
else we'd have failed to read the old file.  This fact justifies some
assumptions below.)

2. Extract old postmaster PID and old shared memory key from file.
(Both will now always be there, per above; abort if file contents are
not as expected.)  We do not bother with trying kill(PID, 0) anymore,
because it doesn't prove anything.

3. Try to attach to the old shared memory segment using the old key.
There are three possible outcomes:
A: fail because it's not there.  Then we know the old postmaster
   (or standalone backend) is gone, and so are all its children.
   Okay to skip to step 7.
B: fail for some other reason, eg permissions violation.  Because
   we know we are the same UID (or root) as before, this must indicate
   that the "old" shmem segment actually belongs to someone else;
   so we have a chance collision with someone else's shmem key.
   Ignore the shmem segment, skip to step 7.  (In short,
   we can treat all failures alike, which is a Good Thing.)
C: attach succeeds. Continue to step 4.

4. Examine header of old shmem segment to see if it contains the right
   magic number *and* old postmaster PID.  If not, it isn't really
   a Postgres shmem segment, so ignore it; detach and skip to step 7.

5. If old shmem segment still has other processes attached to it,
   abort: these must be an old postmaster and/or old backends still
   alive.  (We can check nattach > 1 in the SysV case, or just assume
   they are there in the hugepages-segment case that Neil wants to add.)

6. Detach from and delete the old shmem segment.  (Deletion isn't
   strictly necessary, but we should do it to avoid sucking resources.)

7. Delete the old postmaster.pid file.  If this fails for any reason,
   abort.  (Either we've got permissions problems or a race condition
   with someone else trying to start up.)

8. Create a shared memory segment.

9. Create a new postmaster.pid file and record my PID and segment key.
   If we fail to do this (with O_EXCL create), abort; someone else
   must be trying to start up at the same time.  Be careful to create
   the lockfile mode 600, per notes above.


This is not quite ready for prime time yet, because it's not very
bulletproof against the scenario where two would-be postmasters are
starting concurrently.  The first one might get all the way through the
sequence before the second one arrives at step 7 --- in which case the
second one will be deleting the first one's lockfile.  Oops.  A possible
answer is to create a second lockfile that only exists for the duration
of the startup sequence, and use that to ensure that only one process is
trying this sequence at a time.  This reintroduces the same problem
we're trying to get away from (must rely on kill(PID, 0) to determine
validity of the lock file), but at least the window of vulnerability is
much smaller than before.  Does anyone see a better way?

A more general objection is that this approach will hardwire, even more
solidly than before, the assumption that we are using a shared-memory
API that provides identifiable shmem segments (ie, something we can
record a key for and later try to attach to).  I think some people
wanted to get away from that.  But so far I've not seen any proposal
for an alternative startup interlock that doesn't require attachable
shared memory.

			regards, tom lane

Sat, 28 Sep 2002 12:01:33 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Improving backend startup interlock
I have seen no discussion on whether to go ahead with a 7.2.3 to add
several serious fixes Tom has made to the code in the past few days. 

Are we too close to 7.3 for this to be worthwhile?  Certainly there will
be people distributing 7.2.X for some time as 7.3 stabilizes.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sat, 28 Sep 2002 14:36:40 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
7.2.3?
Bruce Momjian wrote:
> 
> I have seen no discussion on whether to go ahead with a 7.2.3 to add
> several serious fixes Tom has made to the code in the past few days.

This will allow production sites to run the 7.2 series and also do
VACUUM FULL won't it?

If so, then the idea is already pretty good.  :-)

Which other fixes would be included?

Regards and best wishes,

Justin Clift

 
> Are we too close to 7.3 for this to be worthwhile?  Certainly there will
> be people distributing 7.2.X for some time as 7.3 stabilizes.
> 
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
   - Indira Gandhi

Sun, 29 Sep 2002 04:45:40 +1000
Justin Clift <justin@postgresql.org>
Re: 7.2.3?
On Saturday 28 September 2002 02:36 pm, Bruce Momjian wrote:
> I have seen no discussion on whether to go ahead with a 7.2.3 to add
> several serious fixes Tom has made to the code in the past few days.

> Are we too close to 7.3 for this to be worthwhile?  Certainly there will
> be people distributing 7.2.X for some time as 7.3 stabilizes.

IMHO, I believe a 7.2.3 is worthwhile.  It isn't _that_ much effort, is it?  I 
am most certainly of the school of thought that backporting serious issues 
into the last stable release is a Good Thing.  I don't think a released 7.3 
should prevent us from a 7.2.4 down the road, either -- or even a 7.1.4 if a 
serious security issue were to be found there.  Probably not a 7.0.4, though.  
And definitely not a 6.5.4.  Some people can have great difficulty migrating 
-- if we're not going to make it easy for people to migrate, we should 
support older versions with fixes.  IMHO, of course.

If it hasn't already, a fix for the Red Hat 7.3/glibc mktime(3) issue 
(workaround really) would be nice, as I understand the 7.3 branch has one.

RPM's will take me all of an hour if I'm at work when it's released.  That is 
if my wife doesn't go into labor first (she's at 37 weeks and having 
Braxton-Hicks already). #4.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 28 Sep 2002 14:47:28 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: 7.2.3?
Justin Clift dijo: 

> Bruce Momjian wrote:
>
> > I have seen no discussion on whether to go ahead with a 7.2.3 to add
> > several serious fixes Tom has made to the code in the past few days.
> 
> This will allow production sites to run the 7.2 series and also do
> VACUUM FULL won't it?
> 
> If so, then the idea is already pretty good.  :-)
> 
> Which other fixes would be included?

At least the VACUUM code should prevent VACUUM from running inside a
function.  At least one user has been bitten by it.

Memory leaks and such in the PL modules should be backported also.

-- 
Alvaro Herrera (<alvherre[a]atentus.com>)
"El sentido de las cosas no viene de las cosas, sino de
las inteligencias que las aplican a sus problemas diarios
en busca del progreso." (Ernesto Hernï¿½ndez-Novich)


Sat, 28 Sep 2002 15:19:44 -0400 (CLT)
Alvaro Herrera <alvherre@atentus.com>
Re: 7.2.3?
Alvaro Herrera <alvherre@atentus.com> writes:
> Memory leaks and such in the PL modules should be backported also.

This is getting out of hand :-(

7.2 is in maintenance status at this point.  I'm willing to do backports
for bugs that cause data loss, like this VACUUM/CLOG issue.
Performance problems are not on the radar screen at all (especially
not when the putative fixes for them haven't received much of any
testing, and are barely worthy to be called beta status).

We do not have either the developer manpower or the testing resources
to do more than the most minimal maintenance on back versions.  Major
back-port efforts just aren't going to happen.  If they did, they would
significantly impact our ability to work on 7.3 and up; does that seem
like a good tradeoff to you?

			regards, tom lane

Sat, 28 Sep 2002 16:14:17 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: 7.2.3? 
Tom Lane dijo: 

> Alvaro Herrera <alvherre@atentus.com> writes:
> > Memory leaks and such in the PL modules should be backported also.
> 
> This is getting out of hand :-(

Yes, I agree with you.

> Major back-port efforts just aren't going to happen.  If they did,
> they would significantly impact our ability to work on 7.3 and up;
> does that seem like a good tradeoff to you?

I understand the issue.  I also understand that is very nice for
PostgreSQL to advance very quickly, and requiring backports (and
subsequent slowdown) is not nice at all.  However, for users it's very
important to have the fixes present in newer versions...  _without_ the
burden of having to upgrade!

I agree with Lamar that upgrading is a very difficult process right now.
Requiring huge amounts of disk space and database downtime to do
dump/restore is in some cases too high a price to pay.  So maybe the
upgrading process should be observed instead of wasting time on people
trying to stay behind because of the price of that process.

Maybe there is some way of making the life easier for the upgrader.
Let's see, when you upgrade there are basically two things that change:

a) system catalogs
   Going from one version to another requires a number of changes: new
   tuples, deleted tuples, new attributes, deleted attributes.  On-line
   transforming syscatalogs for the three first types seems easy.  The
   last one may be difficult, but it also may not be, I'm not sure.  It
   will require a standalone backend for shared relations and such, but
   hey, it's much cheaper than the process that's required now.

b) on-disk representation of user data
   This is not easy.  Upgrading means changing each filenode from one
   version to another; it requires a tool that understands both (and
   more than two) versions.  It also requires a backend that is able to
   detect that a page is not the version it should, and either abort or
   convert it on the fly (this last possibility seems very nice).

   Note that only tables should be converted: other objects (indexes)
   should just be rebuilt.

There are other things that change.  For example, dependencies are new
in 7.3; building them without the explicit schema construction seems
difficult, but it's certainly possible.  The implicit/explicit cast
system is also new, but it doesn't depend on user data (except for user
defined datatypes, and that should be done manually by the user), so
should just be created from scratch.

Is this at least remotely possible to do?

-- 
Alvaro Herrera (<alvherre[a]atentus.com>)
"La fuerza no estï¿½ en los medios fï¿½sicos
sino que reside en una voluntad indomable" (Gandhi)


Sat, 28 Sep 2002 16:42:43 -0400 (CLT)
Alvaro Herrera <alvherre@atentus.com>
Re: 7.2.3? 
On Sat, 28 Sep 2002, Bruce Momjian wrote:

> I have seen no discussion on whether to go ahead with a 7.2.3 to add
> several serious fixes Tom has made to the code in the past few days.
>
> Are we too close to 7.3 for this to be worthwhile?  Certainly there will
> be people distributing 7.2.X for some time as 7.3 stabilizes.

The vacuum thing is big enough that there should be since as always people
aren't going to move immediately forward with a major version change.



Sat, 28 Sep 2002 13:43:55 -0700 (PDT)
Stephan Szabo <sszabo@megazone23.bigpanda.com>
Re: 7.2.3?
Alvaro Herrera <alvherre@atentus.com> writes:
> Maybe there is some way of making the life easier for the upgrader.
> Let's see, when you upgrade there are basically two things that change:
> a) system catalogs
> b) on-disk representation of user data
> [much snipped]

Yup.  I see nothing wrong with the pg_upgrade process that we've
previously used for updating the system catalogs, however.  Trying to
do it internally in some way will be harder and more dangerous (ie,
much less reliable) than relying on schema-only dump and restore
followed by moving the physical data.

Updates that change the on-disk representation of user data are much
harder, as you say.  But I think they can be made pretty infrequent.
We've only had two such updates that I know of in Postgres' history:
adding WAL in 7.1 forced some additions to page headers, and now in
7.3 we've changed tuple headers for space-saving reasons, and fixed
some problems with alignment in array data.

pg_upgrade could have worked for the 7.2 cycle, but it wasn't done,
mostly for lack of effort.

Going forward I think we should try to maintain compatibility of on-disk
user data and ensure that pg_upgrade works.

			regards, tom lane

Sat, 28 Sep 2002 16:57:54 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Upgrade process (was Re: 7.2.3?)
On Saturday 28 September 2002 04:14 pm, Tom Lane wrote:
> 7.2 is in maintenance status at this point.  I'm willing to do backports
> for bugs that cause data loss, like this VACUUM/CLOG issue.
> Performance problems are not on the radar screen at all (especially
> not when the putative fixes for them haven't received much of any
> testing, and are barely worthy to be called beta status).

A fix that is beta-quality for a non-serious issue (serious issues being of 
the level of the VACUUM/CLOG issue) is, in my mind at least, not for 
inclusion into a _stable_ release.  Simple fixes (the localtime versus mktime 
fix) might be doable, but might not depending upon the particular fix, how 
difficult the packport, etc.  But 7.2 is considered _stable_ -- and I agree 
that this means maintenance mode only.  Only the most trivial or the most 
serious problems should be tackled here.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 28 Sep 2002 17:00:31 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: 7.2.3?
On Saturday 28 September 2002 04:57 pm, Tom Lane wrote:
> 7.3 we've changed tuple headers for space-saving reasons, and fixed
> some problems with alignment in array data.

> Going forward I think we should try to maintain compatibility of on-disk
> user data and ensure that pg_upgrade works.

This is of course a two-edged sword.

1.)	Keeping pg_upgrade working, which depends upon pg_dump working;
2.)	Maintaining security fixes for 7.2 for a good period of time to come, 
since migration from 7.2 to >7.2 isn't easy.

If pg_upgrade is going to be the cookie, then let's all try to test the 
cookie.  I'll certainly try to do my part.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 28 Sep 2002 17:05:58 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Upgrade process (was Re: 7.2.3?)
Lamar Owen <lamar.owen@wgcr.org> writes:
> This is of course a two-edged sword.

> 1.)	Keeping pg_upgrade working, which depends upon pg_dump working;

... which we have to have anyway, of course ...

> 2.)	Maintaining security fixes for 7.2 for a good period of time to come, 
> since migration from 7.2 to >7.2 isn't easy.

True, but I think we'll have to deal with that anyway.  Even if the
physical database upgrade were trivial, people are going to find
application compatibility problems due to schemas and other 7.3 changes.
So we're going to have to expend at least some work on fixing critical
7.2.* problems.  (I just want to keep a tight rein on how much.)

			regards, tom lane

Sat, 28 Sep 2002 17:22:02 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Upgrade process (was Re: 7.2.3?) 
Alvaro Herrera wrote:
<snip>
> I agree with Lamar that upgrading is a very difficult process right now.
> Requiring huge amounts of disk space and database downtime to do
> dump/restore is in some cases too high a price to pay.  So maybe the
> upgrading process should be observed instead of wasting time on people
> trying to stay behind because of the price of that process.

As a "simple for the user approach", would it be
too-difficult-to-bother-with to add to the postmaster an ability to
start up with the data files from the previous version, for it to
recognise an old data format automatically, then for it to do the
conversion process of the old data format to the new one before going
any further?

Sounds like a pain to create initially, but nifty in the end.

:-)

Regards and best wishes,

Justin Clift


<snip>
> --
> Alvaro Herrera (<alvherre[a]atentus.com>)
> "La fuerza no estï¿½ en los medios fï¿½sicos
> sino que reside en una voluntad indomable" (Gandhi)

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
   - Indira Gandhi

Sun, 29 Sep 2002 07:30:35 +1000
Justin Clift <justin@postgresql.org>
Re: 7.2.3?

Tom Lane wrote:

[ discussion of new startup interlock ]

> This is not quite ready for prime time yet, because it's not very
> bulletproof against the scenario where two would-be postmasters are
> starting concurrently.

A solution to this is to require would-be postmasters to obtain an
exclusive lock on a lock file before touching the pid file.  (The lock
file perhaps could be the pid file, but it doesn't have to be.)

Is there some reason that file locking is not acceptable?  Is there
any platform or filesystem supported for use with PostgreSQL which
doesn't have working exclusive file locking?

> A possible answer is to create a second lockfile that only exists
> for the duration of the startup sequence, and use that to ensure
> that only one process is trying this sequence at a time.
> ...
> This reintroduces the same problem
> we're trying to get away from (must rely on kill(PID, 0) to determine
> validity of the lock file), but at least the window of vulnerability is
> much smaller than before.

A lock file locked for the whole time the postmaster is running can be
responsible for preventing multiple postmasters running without
relying on pids.  All that is needed is that the OS drop exclusive
file locks on process exit and that locks not survive across reboots.

The checks of the shared memory segment (number of attachements etc)
look after orphaned back end processes, per the proposal.

Regards,

Giles

Sun, 29 Sep 2002 08:37:20 +1000
Giles Lean <giles@nemeton.com.au>
Re: Improving backend startup interlock 

Tom lane wrote:

> True, but I think we'll have to deal with that anyway.  Even if the
> physical database upgrade were trivial, people are going to find
> application compatibility problems due to schemas and other 7.3 changes.

More reasons:

a) learning curve -- I want to use 7.3 and gain some experience with
   7.2.x -> 7.3 migration before rolling out 7.3 to my users.

b) change control and configuration freezes sometimes dictate when
   upgrades may be done.  A 7.2.2 -> 7.2.3 upgrade for bug fixes is
   much less intrusive than an upgrade to 7.3.

> So we're going to have to expend at least some work on fixing critical
> 7.2.* problems.  (I just want to keep a tight rein on how much.)

No argument here.  Supporting multiple versions eats resources and
eventually destabilises the earlier releases, so critial fixes only,
please.  New features and non-critical fixes however minor are
actually unhelpful.

Since PostgreSQL is open source, anyone who "just has" to have some
minor new feature back ported can do it, or pay for it to be done.
But this doesn't have to effect all users.

Regards,

Giles

Sun, 29 Sep 2002 08:58:29 +1000
Giles Lean <giles@nemeton.com.au>
Re: Upgrade process (was Re: 7.2.3?) 
Justin Clift wrote:
> Alvaro Herrera wrote:
> <snip>
> > I agree with Lamar that upgrading is a very difficult process right now.
> > Requiring huge amounts of disk space and database downtime to do
> > dump/restore is in some cases too high a price to pay.  So maybe the
> > upgrading process should be observed instead of wasting time on people
> > trying to stay behind because of the price of that process.
> 
> As a "simple for the user approach", would it be
> too-difficult-to-bother-with to add to the postmaster an ability to
> start up with the data files from the previous version, for it to
> recognise an old data format automatically, then for it to do the
> conversion process of the old data format to the new one before going
> any further?
> 
> Sounds like a pain to create initially, but nifty in the end.

Yes, we could, but if we are going to do that, we may as well just
automate the dump/reload.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sat, 28 Sep 2002 21:23:31 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: 7.2.3?
Giles Lean <giles@nemeton.com.au> writes:
> Is there some reason that file locking is not acceptable?  Is there
> any platform or filesystem supported for use with PostgreSQL which
> doesn't have working exclusive file locking?

How would we know?  We have never tried to use such a feature.

For sure I would not trust it on an NFS filesystem.  (Although we
disparage running an NFS-mounted database, people do it anyway.)

			regards, tom lane

Sat, 28 Sep 2002 21:47:59 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Improving backend startup interlock 
On Saturday 28 September 2002 09:23 pm, Bruce Momjian wrote:
> Justin Clift wrote:
> > Alvaro Herrera wrote:
> > > I agree with Lamar that upgrading is a very difficult process right

> > As a "simple for the user approach", would it be
> > too-difficult-to-bother-with to add to the postmaster an ability to
> > start up with the data files from the previous version, for it to
> > recognise an old data format automatically, then for it to do the
> > conversion process of the old data format to the new one before going
> > any further?

> > Sounds like a pain to create initially, but nifty in the end.

> Yes, we could, but if we are going to do that, we may as well just
> automate the dump/reload.

Automating the dump/reload is fraught with pitfalls.  Been there; done that; 
got the t-shirt.  The dump from the old version many times requires 
hand-editing for cases where the complexity is above a certain threshold.  
The 7.2->7.3 threshold is just a little lower than normal.  

Our whole approach to the system catalog is wrong for what Justin (and many 
others would like to see).

With MySQL, for instance, one can migrate on a table-by-table basis from one 
table type to another.  As older table types are continuously supported, one 
can upgrade each table in turn as you need the featureset supported by that 
tabletype.

Yes, I know that doesn't fit our existing model of 'all in one' system 
catalogs.  And the solution doesn't present itself readily -- but one day 
someone will see the way to do this, and it will be good.  It _will_ involve 
refactoring the system catalog schema so that user 'system catalog' metadata 
and system 'system catalog' data aren't codependent.  A more modular data 
storage approach at a level above the existing broken storage manager 
modularity will result, and things will be different.

However, the number of messages on this subject has increased; one day it will 
become an important feature worthy of core developer attention.  That will be 
a happy day for me, as well as many others.  I have not the time to do it 
myself; but I can be a gadfly, at least.  In the meantime we have pg_upgrade 
for the future 7.3 -> 7.4 upgrade.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sat, 28 Sep 2002 22:19:29 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: 7.2.3?

Tom Lane wrote:

> Giles Lean <giles@nemeton.com.au> writes:
> > Is there some reason that file locking is not acceptable?  Is there
> > any platform or filesystem supported for use with PostgreSQL which
> > doesn't have working exclusive file locking?
> 
> How would we know?  We have never tried to use such a feature.

I asked because I've not been following this project long enough to
know if it had been tried and rejected previously.  Newcomers being
prone to making silly suggestions and all that. :-)

> For sure I would not trust it on an NFS filesystem.  (Although we
> disparage running an NFS-mounted database, people do it anyway.)

<scratches head>

I can't work out if that's an objection or not.

I'm certainly no fan of NFS locking, but if someone trusts their NFS
client and server implementations enough to put their data on, they
might as well trust it to get a single lock file for startup right
too.  IMHO.  Your mileage may vary.

Regards,

Giles

Sun, 29 Sep 2002 12:58:53 +1000
Giles Lean <giles@nemeton.com.au>
Re: Improving backend startup interlock 
Giles Lean <giles@nemeton.com.au> writes:
> I'm certainly no fan of NFS locking, but if someone trusts their NFS
> client and server implementations enough to put their data on, they
> might as well trust it to get a single lock file for startup right
> too.  IMHO.  Your mileage may vary.

Well, my local man page for lockf() sez

     The advisory record-locking capabilities of lockf() are implemented
     throughout the network by the ``network lock daemon'' (see lockd(1M)).
     If the file server crashes and is rebooted, the lock daemon attempts
     to recover all locks associated with the crashed server.  If a lock
     cannot be reclaimed, the process that held the lock is issued a
     SIGLOST signal.

and the lockd man page mentions that not only lockd but statd have to be
running locally *and* at the NFS server.

This sure sounds like file locking on NFS introduces additional
failure modes above and beyond what we have already.

Since the entire point of this locking exercise is to improve PG's
robustness, solutions that depend on other daemons not crashing
don't sound like a step forward to me.  I'm willing to trust the local
kernel, but I get antsy if I have to trust more than that.

			regards, tom lane

Sat, 28 Sep 2002 23:06:07 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Improving backend startup interlock 
Bruce Momjian dijo: 

> Justin Clift wrote:
> > Alvaro Herrera wrote:

> > As a "simple for the user approach", would it be
> > too-difficult-to-bother-with to add to the postmaster an ability to
> > start up with the data files from the previous version, for it to
> > recognise an old data format automatically, then for it to do the
> > conversion process of the old data format to the new one before going
> > any further?
> 
> Yes, we could, but if we are going to do that, we may as well just
> automate the dump/reload.

I don't think that's an acceptable solution.  It requires too much free
disk space and too much time.  On-line upgrading, meaning altering the
databases on a table-by-table basis (or even page-by-page) solves both
problems (binary conversion sure takes less than converting to text
representation and parsing it to binary again).

I think a converting postmaster would be a waste, because it's unneeded
functionality 99.999% of the time.  I'm leaning towards an external
program doing the conversion, and the backend just aborting if it finds
old or in-conversion data.  The converter should be able to detect that
it has aborted and resume conversion.

What would that converter need:
- the old system catalog  (including user defined data)
- the new system catalog  (ditto, including the schema)
- the storage manager subsystem

I think that should be enough for converting table files.  I'd like to
experiment with something like this when I have some free time.  Maybe
next year...

-- 
Alvaro Herrera (<alvherre[a]atentus.com>)
"I think my standards have lowered enough that now I think 'good design'
is when the page doesn't irritate the living fuck out of me." (JWZ)


Sun, 29 Sep 2002 00:31:38 -0400 (CLT)
Alvaro Herrera <alvherre@atentus.com>
Re: 7.2.3?
Alvaro Herrera <alvherre@atentus.com> writes:
> What would that converter need:
> [snip]
> I think that should be enough for converting table files.  I'd like to
> experiment with something like this when I have some free time.  Maybe
> next year...

It's difficult to say anything convincing on this topic without a
specific conversion requirement in mind.

Localized conversions like 7.3's tuple header change could be done on a
page-by-page basis as you suggest.  (In fact, one reason I insisted on
putting in a page header version number was to leave the door open for
such a converter, if someone wants to do one.)

But one likely future format change for user data is combining parent
and child tables into a single physical table, per recent inheritance
thread.  (I'm not yet convinced that that's feasible or desirable,
I'm just using it as an example of a possible conversion requirement.)
You can't very well do that page-by-page; it'd require a completely
different approach.

			regards, tom lane

Sun, 29 Sep 2002 00:47:44 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: 7.2.3? 
On Sun, 2002-09-29 at 07:19, Lamar Owen wrote:
> On Saturday 28 September 2002 09:23 pm, Bruce Momjian wrote:
> > Justin Clift wrote:
> > > Alvaro Herrera wrote:
> > > > I agree with Lamar that upgrading is a very difficult process right
> 
> > > As a "simple for the user approach", would it be
> > > too-difficult-to-bother-with to add to the postmaster an ability to
> > > start up with the data files from the previous version, for it to
> > > recognise an old data format automatically, then for it to do the
> > > conversion process of the old data format to the new one before going
> > > any further?
> 
> > > Sounds like a pain to create initially, but nifty in the end.
> 
> > Yes, we could, but if we are going to do that, we may as well just
> > automate the dump/reload.
> 
> Automating the dump/reload is fraught with pitfalls.  Been there; done that; 
> got the t-shirt.  The dump from the old version many times requires 
> hand-editing for cases where the complexity is above a certain threshold.  
> The 7.2->7.3 threshold is just a little lower than normal.  
> 
> Our whole approach to the system catalog is wrong for what Justin (and many 
> others would like to see).
> 
> With MySQL, for instance, one can migrate on a table-by-table basis from one 
> table type to another.  As older table types are continuously supported, one 
> can upgrade each table in turn as you need the featureset supported by that 
> tabletype.

The initial Postgres design had a notion of StorageManager's, which
should make this very easy indeed, if it had been kept working .

IIRC the black box nature of storage manager interface was broken at
latest when adding WAL (if it had really been there in the first place).

----------------------
Hannu



29 Sep 2002 12:50:47 +0500
Hannu Krosing <hannu@tm.ee>
Re: 7.2.3?
On Sun, 2002-09-29 at 09:47, Tom Lane wrote:
> Alvaro Herrera <alvherre@atentus.com> writes:
> > What would that converter need:
> > [snip]
> > I think that should be enough for converting table files.  I'd like to
> > experiment with something like this when I have some free time.  Maybe
> > next year...
> 
> It's difficult to say anything convincing on this topic without a
> specific conversion requirement in mind.
> 
> Localized conversions like 7.3's tuple header change could be done on a
> page-by-page basis as you suggest.  (In fact, one reason I insisted on
> putting in a page header version number was to leave the door open for
> such a converter, if someone wants to do one.)
> 
> But one likely future format change for user data is combining parent
> and child tables into a single physical table, per recent inheritance
> thread.  (I'm not yet convinced that that's feasible or desirable,
> I'm just using it as an example of a possible conversion requirement.)
> You can't very well do that page-by-page; it'd require a completely
> different approach.

I started to think about possible upgrade strategy for this scenario and
came up with a whole new way for the whole storage :

We could extend our current way of 1G split files for inheritance, so
that each inherited table is in its own (set of) physical files which
represent a (set of) 1G segment(s) for the logical file definition of
all parent. This would even work for both single and multiple
inheritance !

In this case the indexes (which enforce the uniquenaess and are required
for RI) would see the thing as a single file and can use plain TIDs. The
process of mapping from TID.PAGENR to actual file will happen below the
level visible to executor. It would also naturally cluster similar
tuples.

Aa an extra bonus migration can be done only by changing system catalogs
and recreating indexes.

It will limit the size of inherited structure to at most 16K different
tables (max unsigned int/pagesize), but I don't think this will be a
real limit anytime soon.

---------------------
Hannu



29 Sep 2002 13:08:35 +0500
Hannu Krosing <hannu@tm.ee>
Re: 7.2.3?
Hannu Krosing <hannu@tm.ee> writes:
> The initial Postgres design had a notion of StorageManager's, which
> should make this very easy indeed, if it had been kept working .

But the storage manager interface was never built to hide issues like
tuple representation --- storage managers just deal in raw pages.
I doubt it would have helped in the least for anything we've been
concerned about.

			regards, tom lane

Sun, 29 Sep 2002 10:28:50 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: 7.2.3? 
On Sun, 2002-09-29 at 19:28, Tom Lane wrote:
> Hannu Krosing <hannu@tm.ee> writes:
> > The initial Postgres design had a notion of StorageManager's, which
> > should make this very easy indeed, if it had been kept working .
> 
> But the storage manager interface was never built to hide issues like
> tuple representation --- storage managers just deal in raw pages.

I had an impression that SM was meant to be a little higher-level. IIRC
the original Berkeley Postgres had at one point a storage manager for
write-once storage on CDWr jukeboxes.

the README in src/backend/storage/smgr still contains mentions about
Sony jukebox drivers.

http://www.ndim.edrc.cmu.edu/postgres95/www/pglite1.html also claims
this:

Version 3 appeared in 1991 and added support for multiple storage
managers, an improved query executor and a rewritten rewrite rule
system. For the most part, releases since then have focused on
portability and reliability. 

> I doubt it would have helped in the least for anything we've been
> concerned about.

Yes, it seems that we do not have a SM in the semse I hoped.

Still, if we could use a clean SM interface over old page format, then
the tuple conversion could be done there.

That of course would need the storage manager to be aware of old/new
tuple structures ;(

-----------------
Hannu




29 Sep 2002 22:05:55 +0500
Hannu Krosing <hannu@tm.ee>
Re: 7.2.3?
Should an advisory be issued for production sites to not perform a
vacuum full with a notice that a bug fix will be coming shortly?

Greg



On Sat, 2002-09-28 at 13:45, Justin Clift wrote:
> Bruce Momjian wrote:
> > 
> > I have seen no discussion on whether to go ahead with a 7.2.3 to add
> > several serious fixes Tom has made to the code in the past few days.
> 
> This will allow production sites to run the 7.2 series and also do
> VACUUM FULL won't it?
> 
> If so, then the idea is already pretty good.  :-)
> 
> Which other fixes would be included?
> 
> Regards and best wishes,
> 
> Justin Clift
> 
>  
> > Are we too close to 7.3 for this to be worthwhile?  Certainly there will
> > be people distributing 7.2.X for some time as 7.3 stabilizes.
> > 
> > --
> >   Bruce Momjian                        |  http://candle.pha.pa.us
> >   pgman@candle.pha.pa.us               |  (610) 359-1001
> >   +  If your life is a hard drive,     |  13 Roberts Road
> >   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
> > 
> > ---------------------------(end of broadcast)---------------------------
> > TIP 5: Have you checked our extensive FAQ?
> > 
> > http://www.postgresql.org/users-lounge/docs/faq.html
> 
> -- 
> "My grandfather once told me that there are two kinds of people: those
> who work and those who take the credit. He told me to try to be in the
> first group; there was less competition there."
>    - Indira Gandhi
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
30 Sep 2002 10:39:28 -0500
Greg Copeland <greg@CopelandConsulting.Net>
Re: 7.2.3?
Greg Copeland <greg@CopelandConsulting.Net> writes:
> Should an advisory be issued for production sites to not perform a
> vacuum full with a notice that a bug fix will be coming shortly?

People seem to be misunderstanding the bug.  Whether your vacuum is FULL
or not (or VERBOSE or not, or ANALYZE or not) is not relevant.  The
dangerous thing is to execute a VACUUM that's not a single-table VACUUM
*as a non-superuser*.  The options don't matter.  If you see any notices
about "skipping tables" out of VACUUM, then you are at risk.

I'm not averse to issuing an announcement, but let's be sure we have
the details straight.

			regards, tom lane

Mon, 30 Sep 2002 14:11:42 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: 7.2.3? 
Hi everyone,

Have just put together a prototype page to show off the multi-lingual
capabilities that the Advocacy sites' infrastructure has:

http://advocacy.postgresql.org/?lang=de

The text was translated to german via Altavista's Babelfish, so it's
probably only about 80% accurate, but it conveys the concept.

Is anyone interested in translating the English version to other
languages?  All Latin based languages should be fine (German, French,
Italian, Spanish, Portuguese, Turkish, Greek, etc).

If there's strong interest, then an interface to let volunteers
translators do it easily can be constructed over the next fortnight or
so.

:-)

Regards and best wishes,

Justin Clift

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
   - Indira Gandhi

Thu, 03 Oct 2002 08:53:07 +1000
Justin Clift <justin@postgresql.org>
Anyone want to assist with the translation of the Advocacy site?
Justin Clift <justin@postgresql.org> wrote:

> Hi everyone,
>
> Have just put together a prototype page to show off the multi-lingual
> capabilities that the Advocacy sites' infrastructure has:
>
> http://advocacy.postgresql.org/?lang=de
>
> The text was translated to german via Altavista's Babelfish, so it's
> probably only about 80% accurate, but it conveys the concept.
>
> Is anyone interested in translating the English version to other
> languages?  All Latin based languages should be fine (German, French,
> Italian, Spanish, Portuguese, Turkish, Greek, etc).
>
> If there's strong interest, then an interface to let volunteers
> translators do it easily can be constructed over the next fortnight or
> so.

Hi Justin,

I am from Austria, and I would like to help. I could provide a German
translation. The Babelfish's translation is really funny. Machine
translation is readable, but it is no advocacy. ;-) I do not really nead an
interface, but just tell me in what way you want the texts.

Best Regards,
Michael Paesold


Thu, 3 Oct 2002 01:21:00 +0200
"Michael Paesold" <mpaesold@gmx.at>
Re: [HACKERS] Anyone want to assist with the translation of the
	Advocacy site?
Hi Michael,

Michael Paesold wrote:
<snip> 
> Hi Justin,
> 
> I am from Austria, and I would like to help. I could provide a German
> translation. The Babelfish's translation is really funny. Machine
> translation is readable, but it is no advocacy. ;-) I do not really nead an
> interface, but just tell me in what way you want the texts.

Cool.  Could you deal with an OpenOffice Calc or M$ Excel file having
the lines of English text in one column, and doing the German
translation into a second column?

That might be easiest, and will allow a cut-n-paste of the German
version straight into the database backend.

Sound workable to you?

:-)

Regards and best wishes,

Justin Clift

> Best Regards,
> Michael Paesold

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
   - Indira Gandhi

Thu, 03 Oct 2002 09:23:28 +1000
Justin Clift <justin@postgresql.org>
Re: [HACKERS] Anyone want to assist with the translation of the
	Advocacy
Justin Clift <justin@postgresql.org> wrote:


> Hi Michael,
>
> Michael Paesold wrote:
> <snip>
> > Hi Justin,
> >
> > I am from Austria, and I would like to help. I could provide a German
> > translation. The Babelfish's translation is really funny. Machine
> > translation is readable, but it is no advocacy. ;-) I do not really nead
an
> > interface, but just tell me in what way you want the texts.
>
> Cool.  Could you deal with an OpenOffice Calc or M$ Excel file having
> the lines of English text in one column, and doing the German
> translation into a second column?
>
> That might be easiest, and will allow a cut-n-paste of the German
> version straight into the database backend.
>
> Sound workable to you?

Spreadsheet sounds great. I use M$.
Perhaps you can group the items in categories, at least navigation and text.
So I know where the text will be put on the website. The translation could
be different depending on how a word is used. E.g. it is quite common on
German websites to use the same English word "Home" for the main page; but
you would not use "Home" in a different context. The exceptable length of a
translation depends on the context, too.

Best Regards,
Michael Paesold



Thu, 3 Oct 2002 01:36:30 +0200
"Michael Paesold" <mpaesold@gmx.at>
Re: [HACKERS] Anyone want to assist with the translation of the
	Advocacy site?
The lock on the relation owning the rule wasn't unlocked.

On Wed, 2002-10-02 at 19:36, Michael Paesold wrote:
> Justin Clift <justin@postgresql.org> wrote:
> 
> 
> > Hi Michael,
> >
> > Michael Paesold wrote:
> > <snip>
> > > Hi Justin,
> > >
> > > I am from Austria, and I would like to help. I could provide a German
> > > translation. The Babelfish's translation is really funny. Machine
> > > translation is readable, but it is no advocacy. ;-) I do not really nead
> an
> > > interface, but just tell me in what way you want the texts.
> >
> > Cool.  Could you deal with an OpenOffice Calc or M$ Excel file having
> > the lines of English text in one column, and doing the German
> > translation into a second column?
> >
> > That might be easiest, and will allow a cut-n-paste of the German
> > version straight into the database backend.
> >
> > Sound workable to you?
> 
> Spreadsheet sounds great. I use M$.
> Perhaps you can group the items in categories, at least navigation and text.
> So I know where the text will be put on the website. The translation could
> be different depending on how a word is used. E.g. it is quite common on
> German websites to use the same English word "Home" for the main page; but
> you would not use "Home" in a different context. The exceptable length of a
> translation depends on the context, too.
> 
> Best Regards,
> Michael Paesold
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html
> 
-- 
  Rod Taylor
02 Oct 2002 20:14:00 -0400
Rod Taylor <rbt@rbt.ca>
Re: [HACKERS] Anyone want to assist with the translation of the
Justin wrote:
> Hi Michael,
> Michael Paesold wrote:
> <snip> 
> > Hi Justin,
> > I am from Austria, and I would like to help. I could provide a German
> > translation. The Babelfish's translation is really funny. Machine
> > translation is readable, but it is no advocacy. ;-) I do not really nead an
> > interface, but just tell me in what way you want the texts.
> 
> Cool.  Could you deal with an OpenOffice Calc or M$ Excel file having
> the lines of English text in one column, and doing the German
> translation into a second column?

Isn't this, um, the sort of thing you might want to put into, um, a, um, 
database?
--
(concatenate 'string "aa454" "@freenet.carleton.ca")
http://cbbrowne.com/info/internet.html
"you  can   obvioulsy understand what  i'm  saying.  you're just being
pendantic." -- bazzz777@yahoo.com



Wed, 02 Oct 2002 20:58:25 -0400
cbbrowne@cbbrowne.com
Re: Anyone want to assist with the translation of the 
cbbrowne@cbbrowne.com wrote:
<snip>
> > Cool.  Could you deal with an OpenOffice Calc or M$ Excel file having
> > the lines of English text in one column, and doing the German
> > translation into a second column?
> 
> Isn't this, um, the sort of thing you might want to put into, um, a, um,
> database?

Sure is.  Are there any good options apart from?

a) Build an interface for people to translate through
b) Allow selected people to connect directly to the database

For the present b) is not an option as I don't have the needed access to
the postgresql.org database server to be able to adjust the pg_hba.conf
file, and a) Would take some decent time and effort to get up and
running.  A lot longer than cut and pasting into an Excel document then
back out again.  :-/

Regards and best wishes,

Justin Clift


> --
> (concatenate 'string "aa454" "@freenet.carleton.ca")
> http://cbbrowne.com/info/internet.html
> "you  can   obvioulsy understand what  i'm  saying.  you're just being
> pendantic." -- bazzz777@yahoo.com

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
   - Indira Gandhi

Thu, 03 Oct 2002 11:09:18 +1000
Justin Clift <justin@postgresql.org>
Re: Anyone want to assist with the translation of the Advocacy
Rod Taylor <rbt@rbt.ca> writes:
>   	/* Call CreateComments() to create/drop the comments */
>   	CreateComments(ruleoid, classoid, 0, comment);
> +
> + 	heap_close(relation, AccessShareLock);
>   }
> 
>   /*

Ooops.

I think though that this should read

+	heap_close(relation, NoLock);

In general, we hold locks on user relations we are modifying until end
of transaction.  This is different from the rule for system catalogs
(eg, it's okay to drop the AccessShareLock on pg_rewrite a few lines
above this).  The reason for the distinction is that we want to be
sure that the user relation won't get DROPped by someone else before
we've committed our changes.  (If someone else did try to drop it in
that interval, they'd not delete the pg_description row we just added,
because they couldn't see it.)  On the other hand, system catalogs such
as pg_rewrite are not going to go away, by definition, and so it's okay
to drop their locks early.  The only reason we lock system catalogs at
all is to allow VACUUM FULL to nail down exclusive access to a catalog
while it vacuums it.

			regards, tom lane

Wed, 02 Oct 2002 23:32:40 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Anyone want to assist with the translation of the 
> Cool.  Could you deal with an OpenOffice Calc or M$ Excel file having
> the lines of English text in one column, and doing the German
> translation into a second column?
> 
> That might be easiest, and will allow a cut-n-paste of the German
> version straight into the database backend.

Hi Justin,

I would gladly do the translation in French. I don't have too much time
being both a freelancer and a young daddy, but there isn't that much content
so I should be able to manage.

I don't have OpenOffice, so an Excel worksheet would do fine.

Cheers.

--------
François

Home page: http://www.monpetitcoin.com/
"A fox is a wolf who sends flowers"


Thu, 03 Oct 2002 09:42:38 +0200
Francois Suter <dba@paragraf.ch>
Re: Anyone want to assist with the translation of the Advocacy 
>
>
>Hi everyone,
>
>Have just put together a prototype page to show off the multi-lingual
>capabilities that the Advocacy sites' infrastructure has:
>
>http://advocacy.postgresql.org/?lang=de
>
>The text was translated to german via Altavista's Babelfish, so it's
>probably only about 80% accurate, but it conveys the concept.
>
>Is anyone interested in translating the English version to other
>languages?  All Latin based languages should be fine (German, French,
>Italian, Spanish, Portuguese, Turkish, Greek, etc).
>
>If there's strong interest, then an interface to let volunteers
>translators do it easily can be constructed over the next fortnight or
>so.
>
>:-)
>
>Regards and best wishes,
>
>Justin Clift
>
>  
>
Justin,

I would be glad to translate it to brazilian portuguese.
Here we have a lot of companies starting to use PostgreSQL, including 
the one where I work and some of our clients.
It would be very nice to have this site translated.

Cheers,

-- 
Diogo de Oliveira Biazus
diogo@ikono.com.br
Ikono Sistemas e Automaï¿½ï¿½o
http://www.ikono.com.br



Thu, 03 Oct 2002 09:35:19 -0300
Diogo Biazus <diogo@ikono.com.br>
Re: Anyone want to assist with the translation of the Advocacy
> Is anyone interested in translating the English version to other
> languages?

I don't have time for the translation, unfortunately, but i would 
suggest changing "worlds" to "world's" on the main page.

-tfo

Thu, 03 Oct 2002 10:48:31 -0500
Thomas O'Connell <tfo@monsterlabs.com>
Re: Anyone want to assist with the translation of the Advocacy site?
Thomas O'Connell wrote:
> 
> > Is anyone interested in translating the English version to other
> > languages?
> 
> I don't have time for the translation, unfortunately, but i would
> suggest changing "worlds" to "world's" on the main page.

Um, doesn't "world's" mean "world is" ?

That wouldn't make sense then though.  ?

Regards and best wishes,

Justin Clift

 
> -tfo
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
   - Indira Gandhi

Fri, 04 Oct 2002 01:51:31 +1000
Justin Clift <justin@postgresql.org>
Re: Anyone want to assist with the translation of the Advocacy 
> Um, doesn't "world's" mean "world is" ?

In this situation, the "'s" denotes possession, as in "the most advanced 
open source database of the world".

"worlds" here is basically saying "every world most advanced open source 
database" and does not, in any case, connote possession.

-tfo


Thu, 3 Oct 2002 10:54:57 -0500
Thomas F.O'Connell <tfo@monsterlabs.com>
Re: Anyone want to assist with the translation of the Advocacy site?
"Thomas F.O'Connell" wrote:
> 
> > Um, doesn't "world's" mean "world is" ?
> 
> In this situation, the "'s" denotes possession, as in "the most advanced
> open source database of the world".
> 
> "worlds" here is basically saying "every world most advanced open source
> database" and does not, in any case, connote possession.

Ok, updating it now.  Thanks heaps Thomas.

:-)

Regards and best wishes,

Justin Clift

 
> -tfo

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
   - Indira Gandhi

Fri, 04 Oct 2002 01:57:23 +1000
Justin Clift <justin@postgresql.org>
Re: Anyone want to assist with the translation of the 
> Um, doesn't "world's" mean "world is" ?

i forgot to provide a real-world example:

http://www.amazon.com/

"Earth's Biggest Selection"

-tfo


Thu, 3 Oct 2002 10:57:45 -0500
Thomas F.O'Connell <tfo@monsterlabs.com>
Re: Anyone want to assist with the translation of the Advocacy site?
Hi Justin,

you want probably use the language-negotiation
rather then a query variable :-)

Regards
Tino

--On Donnerstag, 3. Oktober 2002 08:53 +1000 Justin Clift 
<justin@postgresql.org> wrote:

> Hi everyone,
>
> Have just put together a prototype page to show off the multi-lingual
> capabilities that the Advocacy sites' infrastructure has:
>
> http://advocacy.postgresql.org/?lang=de
>
> The text was translated to german via Altavista's Babelfish, so it's
> probably only about 80% accurate, but it conveys the concept.
>
> Is anyone interested in translating the English version to other
> languages?  All Latin based languages should be fine (German, French,
> Italian, Spanish, Portuguese, Turkish, Greek, etc).
>
> If there's strong interest, then an interface to let volunteers
> translators do it easily can be constructed over the next fortnight or
> so.
>
> :-)
>
> Regards and best wishes,
>
> Justin Clift
>
> --
> "My grandfather once told me that there are two kinds of people: those
> who work and those who take the credit. He told me to try to be in the
> first group; there was less competition there."
>    - Indira Gandhi
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster



Thu, 03 Oct 2002 19:05:33 +0200
Tino Wildenhain <tino@wildenhain.de>
Re: Anyone want to assist with the translation of the
Hi Tino,

Tino Wildenhain wrote:
> 
> Hi Justin,
> 
> you want probably use the language-negotiation
> rather then a query variable :-)

Um, language-negotiation in good in theory, but there are real world
scenarios it doesn't take into account.  :(

However, the query variable is an override, and if one isn't present
then the backend is supposed to use other means to determine the
appropriate language, including the browsers preferred language.  It's
just that the code to do this bit hasn't been written yet.  :-)

If all else fails it falls back to a default language, English for this
site.

:-)

Regards and best wishes,

Justin Clift

 
> Regards
> Tino
<snip>

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
   - Indira Gandhi

Fri, 04 Oct 2002 03:06:53 +1000
Justin Clift <justin@postgresql.org>
Re: Anyone want to assist with the translation of theAdvocacy 
Hi Justin,

--On Donnerstag, 3. Oktober 2002 09:23 +1000 Justin Clift 
<justin@postgresql.org> wrote:

> Hi Michael,
>
> Michael Paesold wrote:
> <snip>
>> Hi Justin,
>>
>> I am from Austria, and I would like to help. I could provide a German
>> translation. The Babelfish's translation is really funny. Machine
>> translation is readable, but it is no advocacy. ;-) I do not really nead
>> an interface, but just tell me in what way you want the texts.
>
> Cool.  Could you deal with an OpenOffice Calc or M$ Excel file having
> the lines of English text in one column, and doing the German
> translation into a second column?

> That might be easiest, and will allow a cut-n-paste of the German
> version straight into the database backend.

Haha cut&paste ;-) Ever heard of csv? :-))

However, I can also have a look at it, if desired.

Regards
Tino

> Sound workable to you?
>
> :-)
>
> Regards and best wishes,
>
> Justin Clift
>
>> Best Regards,
>> Michael Paesold
>
> --
> "My grandfather once told me that there are two kinds of people: those
> who work and those who take the credit. He told me to try to be in the
> first group; there was less competition there."
>    - Indira Gandhi
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)



Thu, 03 Oct 2002 19:14:13 +0200
Tino Wildenhain <tino@wildenhain.de>
Re: [HACKERS] Anyone want to assist with the translation
Tino Wildenhain wrote:
<snip> 
> Haha cut&paste ;-) Ever heard of csv? :-))
> 
> However, I can also have a look at it, if desired.

Heh Heh Heh

Good point.  For the moment we've whipped up that MS Excel document
(created in OpenOffice of course) of all the English text strings in the
site and emailed it to the volunteers.  :)

So far community members have volunteered for German, Turkish, French,
Spanish, Brazilian Portuguese, and Polish.

Cool.  :)

Want to co-ordinate with the other two German language volunteers?

Regards and best wishes,

Justin Clift


> Regards
> Tino

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
   - Indira Gandhi

Fri, 04 Oct 2002 03:28:39 +1000
Justin Clift <justin@postgresql.org>
Re: [HACKERS] Anyone want to assist with the translationof the 
Hi Justin,

> Good point.  For the moment we've whipped up that MS Excel document
> (created in OpenOffice of course) of all the English text strings in the
> site and emailed it to the volunteers.  :)

Btw. did you ever unzip the native OpenOffice (aka StarOffice)
file?

>
> So far community members have volunteered for German, Turkish, French,
> Spanish, Brazilian Portuguese, and Polish.
>
> Cool.  :)
>
> Want to co-ordinate with the other two German language volunteers?

Sure. So I'm here :-)

Regards
Tino

Thu, 03 Oct 2002 21:34:22 +0200
Tino Wildenhain <tino@wildenhain.de>
Re: [HACKERS] Anyone want to assist with the
Tino Wildenhain <tino@wildenhain.de> wrote:

> Hi Justin,
>
> > Good point.  For the moment we've whipped up that MS Excel document
> > (created in OpenOffice of course) of all the English text strings in the
> > site and emailed it to the volunteers.  :)
>
> Btw. did you ever unzip the native OpenOffice (aka StarOffice)
> file?
>
> >
> > So far community members have volunteered for German, Turkish, French,
> > Spanish, Brazilian Portuguese, and Polish.
> >
> > Cool.  :)
> >
> > Want to co-ordinate with the other two German language volunteers?
>
> Sure. So I'm here :-)
>
> Regards
> Tino

You should have already got at least two mails, haven't you?

Michael


Thu, 3 Oct 2002 21:54:39 +0200
"Michael Paesold" <mpaesold@gmx.at>
Re: [HACKERS] Anyone want to assist with the translationof the
	Advocacy
Hi Michael,

yeah, I got :-)

I'm busy reviewing :-)

Regards
Tino

--On Donnerstag, 3. Oktober 2002 21:54 +0200 Michael Paesold 
<mpaesold@gmx.at> wrote:

> Tino Wildenhain <tino@wildenhain.de> wrote:
>
>> Hi Justin,
>>
>> > Good point.  For the moment we've whipped up that MS Excel document
>> > (created in OpenOffice of course) of all the English text strings in
>> > the site and emailed it to the volunteers.  :)
>>
>> Btw. did you ever unzip the native OpenOffice (aka StarOffice)
>> file?
>>
>> >
>> > So far community members have volunteered for German, Turkish, French,
>> > Spanish, Brazilian Portuguese, and Polish.
>> >
>> > Cool.  :)
>> >
>> > Want to co-ordinate with the other two German language volunteers?
>>
>> Sure. So I'm here :-)
>>
>> Regards
>> Tino
>
> You should have already got at least two mails, haven't you?
>
> Michael
>



Thu, 03 Oct 2002 22:35:08 +0200
Tino Wildenhain <tino@wildenhain.de>
Re: [HACKERS] Anyone want to assist with the

Have people considered flock (advisory locking) on the postmaster.pid
file for backend detection?   It has a nonblocking option.  Don't most
OS's support it?

I can't understand why we can't get an easier solution to postmaster
detection than shared memory.

---------------------------------------------------------------------------

Tom Lane wrote:
> Giles Lean <giles@nemeton.com.au> writes:
> > I'm certainly no fan of NFS locking, but if someone trusts their NFS
> > client and server implementations enough to put their data on, they
> > might as well trust it to get a single lock file for startup right
> > too.  IMHO.  Your mileage may vary.
> 
> Well, my local man page for lockf() sez
> 
>      The advisory record-locking capabilities of lockf() are implemented
>      throughout the network by the ``network lock daemon'' (see lockd(1M)).
>      If the file server crashes and is rebooted, the lock daemon attempts
>      to recover all locks associated with the crashed server.  If a lock
>      cannot be reclaimed, the process that held the lock is issued a
>      SIGLOST signal.
> 
> and the lockd man page mentions that not only lockd but statd have to be
> running locally *and* at the NFS server.
> 
> This sure sounds like file locking on NFS introduces additional
> failure modes above and beyond what we have already.
> 
> Since the entire point of this locking exercise is to improve PG's
> robustness, solutions that depend on other daemons not crashing
> don't sound like a step forward to me.  I'm willing to trust the local
> kernel, but I get antsy if I have to trust more than that.
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 3 Oct 2002 21:09:41 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Improving backend startup interlock

Tom's change added to Rod's patch:

Your patch has been added to the PostgreSQL unapplied patches list at:

	http://candle.pha.pa.us/cgi-bin/pgpatches

I will try to apply it within the next 48 hours.

---------------------------------------------------------------------------


Tom Lane wrote:
> Rod Taylor <rbt@rbt.ca> writes:
> >   	/* Call CreateComments() to create/drop the comments */
> >   	CreateComments(ruleoid, classoid, 0, comment);
> > +
> > + 	heap_close(relation, AccessShareLock);
> >   }
> > 
> >   /*
> 
> Ooops.
> 
> I think though that this should read
> 
> +	heap_close(relation, NoLock);
> 
> In general, we hold locks on user relations we are modifying until end
> of transaction.  This is different from the rule for system catalogs
> (eg, it's okay to drop the AccessShareLock on pg_rewrite a few lines
> above this).  The reason for the distinction is that we want to be
> sure that the user relation won't get DROPped by someone else before
> we've committed our changes.  (If someone else did try to drop it in
> that interval, they'd not delete the pg_description row we just added,
> because they couldn't see it.)  On the other hand, system catalogs such
> as pg_rewrite are not going to go away, by definition, and so it's okay
> to drop their locks early.  The only reason we lock system catalogs at
> all is to allow VACUUM FULL to nail down exclusive access to a catalog
> while it vacuums it.
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 3 Oct 2002 22:29:47 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Anyone want to assist with the translation of
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Have people considered flock (advisory locking) on the postmaster.pid
> file for backend detection?

$ man flock
No manual entry for flock.
$

HPUX has generally taken the position of adopting both BSD and SysV
features, so if it doesn't exist here, it's not portable to older
Unixen ...

			regards, tom lane

Thu, 03 Oct 2002 22:45:51 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Improving backend startup interlock 

Tom Lane writes:

> $ man flock
> No manual entry for flock.
> $
> 
> HPUX has generally taken the position of adopting both BSD and SysV
> features, so if it doesn't exist here, it's not portable to older
> Unixen ...

If only local locking is at issue then finding any one of fcntl()
locking, flock(), or lockf() would do.  All Unixen will have one or
more of these and autoconf machinery exists to find them.

The issue Tom raised about NFS support remains: locking over NFS
introduces new failure modes.  It also only works for NFS clients
that support NFS locking, which not all do.

Mind you NFS users are currently entirely unprotected from someone
starting a postmaster on a different NFS client using the same data
directory right now, which file locking would prevent. So there is
some win for NFS users as well as local filesystem users.  (Anyone
using NFS care to put their hand up?  Maybe nobody does?)

Is the benefit of better local filesystem behaviour plus multiple
client protection for NFS users who have file locking enough to
outweigh the drawbacks?  My two cents says it is, but my two cents are
worth approximately USD$0.01, which is to say not very much ...

Regards,

Giles

Sat, 05 Oct 2002 09:24:55 +1000
Giles Lean <giles@nemeton.com.au>
Re: Improving backend startup interlock 
Giles Lean <giles@nemeton.com.au> wrote:

> Tom Lane writes:
> 
> > $ man flock
> > No manual entry for flock.
> > $
> > 
> > HPUX has generally taken the position of adopting both BSD and SysV
> > features, so if it doesn't exist here, it's not portable to older
> > Unixen ...
> 
> If only local locking is at issue then finding any one of fcntl()
> locking, flock(), or lockf() would do.  All Unixen will have one or
> more of these and autoconf machinery exists to find them.
> 
> The issue Tom raised about NFS support remains: locking over NFS
> introduces new failure modes.  It also only works for NFS clients
> that support NFS locking, which not all do.
> 
> Mind you NFS users are currently entirely unprotected from someone
> starting a postmaster on a different NFS client using the same data
> directory right now, which file locking would prevent. So there is
> some win for NFS users as well as local filesystem users.  (Anyone
> using NFS care to put their hand up?  Maybe nobody does?)
> 
> Is the benefit of better local filesystem behaviour plus multiple
> client protection for NFS users who have file locking enough to
> outweigh the drawbacks?  My two cents says it is, but my two cents are
> worth approximately USD$0.01, which is to say not very much ...

Well, I am going to do some tests with postgresql and our netapp
filer later in October. If that setup proves to work fast and reliable
I would also be interested in such a locking. I don't care about
the feature if I find the postgresql/NFS/netapp-filer setup to be
unreliable or bad performing.

I'll see.

Regards,
Michael Paesold


Sat, 5 Oct 2002 01:42:08 +0200
"Michael Paesold" <mpaesold@gmx.at>
Re: Improving backend startup interlock 
Michael Paesold wrote:
> Giles Lean <giles@nemeton.com.au> wrote:
>>Mind you NFS users are currently entirely unprotected from someone
>>starting a postmaster on a different NFS client using the same data
>>directory right now, which file locking would prevent. So there is
>>some win for NFS users as well as local filesystem users.  (Anyone
>>using NFS care to put their hand up?  Maybe nobody does?)
>>
>>Is the benefit of better local filesystem behaviour plus multiple
>>client protection for NFS users who have file locking enough to
>>outweigh the drawbacks?  My two cents says it is, but my two cents are
>>worth approximately USD$0.01, which is to say not very much ...
> 
> 
> Well, I am going to do some tests with postgresql and our netapp
> filer later in October. If that setup proves to work fast and reliable
> I would also be interested in such a locking. I don't care about
> the feature if I find the postgresql/NFS/netapp-filer setup to be
> unreliable or bad performing.
> 

We have multiple Oracle databases running over NFS from an HPUX server to a 
netapp and have been pleased with the performance overall. It does require 
some tuning to get it right, and it hasn't been entirely without issues, but I 
don't see us going back to local storage. We also just recently set up a Linux 
box running Oracle against an NFS mounted netapp. Soon I'll be adding Postgres 
on the same machine, initially using locally attached storage, but at some 
point I may need to shift to the netapp due to data volume.

If you do try Postgres on the netapp, please post your results/experience and 
I'll do the same.

Anyway, I guess I qualify as interested in an NFS safe locking method.

Joe


Fri, 04 Oct 2002 18:14:58 -0700
Joe Conway <mail@joeconway.com>
Re: Improving backend startup interlock
Hi Adrian,

Wow.  That's pretty cool.  :)

No-one has offered to do Romanian yet, so you're very welcome to.

First things first:

 - What is the two letter language identifier most often used for
Romanian?  i.e. fr = Franch, de = German, etc.  ro?
 - What is the character set that should be used to send out Romanian
pages?  i.e. for English, French, German it's iso-8859-1, for Turkish
it's iso-8859-9, Romanian = ?

:-)

Regards and best wishes,

Justin Clift


am@fx.ro wrote:
> 
> Hello !
> 
> I'd like to translate the advocacy site to Romanian,
> as long as nobody else has already offered himself/herself
> to do it.
> 
> Just tell me, please, what should i do.
> 
> Adrian Maier
> (am@fx.ro)

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
   - Indira Gandhi

Tue, 15 Oct 2002 05:07:46 +1000
Justin Clift <justin@postgresql.org>
Re: Anyone want to assist with the translation of the Advocacy 
Hello !

I'd like to translate the advocacy site to Romanian,
as long as nobody else has already offered himself/herself
to do it.

Just tell me, please, what should i do.


Adrian Maier
(am@fx.ro)


Tue, 15 Oct 2002 00:13:58 +0300
am@fx.ro
Re: Anyone want to assist with the translation of the Advocacy site?
On Tue, Oct 15, 2002 at 05:07:46AM +1000, Justin Clift wrote:
> Hi Adrian,
> 
> Wow.  That's pretty cool.  :)
> 
> No-one has offered to do Romanian yet, so you're very welcome to.
> 
> First things first:
> 
>  - What is the two letter language identifier most often used for
> Romanian?  i.e. fr = Franch, de = German, etc.  ro?

ro = Romanian

>  - What is the character set that should be used to send out Romanian
> pages?  i.e. for English, French, German it's iso-8859-1, for Turkish
> it's iso-8859-9, Romanian = ?

iso-8859-2

Tue, 15 Oct 2002 20:38:21 +0300
am@fx.ro
Re: Anyone want to assist with the translation of the Advocacy site?

I have put the latest patch at:

     http://downloads.rhyme.com.au/postgresql/pg_dump/

along with two dump files of the regression DB, one with 4 byte
and the other with 8 byte offsets. I can read/restore each from
the other, so it looks pretty good. Once the endianness is tested,
we should be OK.

Known problems:

- will not cope with > 4GB files and size_t not 64 bit.
- when printing data position, it is assumed that off_t is UINT64
   (we could remove this entirely - it's just for display)
- if seek is not supported, then an intXX is assigned to off_t
   when file offsets are needed. This *should* not cause a problem
   since without seek, the offsets will not be written to the file.

Changes from Prior Version:

- No longer stores or outputs data length
- Assumes result of ftello is correct if it disagrees with internally
   kept tally.
- 'pg_restore -l' now shows sizes of int and offset.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Sat, 19 Oct 2002 17:08:09 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?

Your patch has been added to the PostgreSQL unapplied patches list at:

	http://momjian.postgresql.org/cgi-bin/pgpatches

I will try to apply it within the next 48 hours.

---------------------------------------------------------------------------


Philip Warner wrote:
> 
> I have put the latest patch at:
> 
>      http://downloads.rhyme.com.au/postgresql/pg_dump/
> 
> along with two dump files of the regression DB, one with 4 byte
> and the other with 8 byte offsets. I can read/restore each from
> the other, so it looks pretty good. Once the endianness is tested,
> we should be OK.
> 
> Known problems:
> 
> - will not cope with > 4GB files and size_t not 64 bit.
> - when printing data position, it is assumed that off_t is UINT64
>    (we could remove this entirely - it's just for display)
> - if seek is not supported, then an intXX is assigned to off_t
>    when file offsets are needed. This *should* not cause a problem
>    since without seek, the offsets will not be written to the file.
> 
> Changes from Prior Version:
> 
> - No longer stores or outputs data length
> - Assumes result of ftello is correct if it disagrees with internally
>    kept tally.
> - 'pg_restore -l' now shows sizes of int and offset.
> 
> 
> ----------------------------------------------------------------
> Philip Warner                    |     __---_____
> Albatross Consulting Pty. Ltd.   |----/       -  \
> (A.B.N. 75 008 659 498)          |          /(@)   ______---_
> Tel: (+61) 0500 83 82 81         |                 _________  \
> Fax: (+61) 0500 83 82 82         |                 ___________ |
> Http://www.rhyme.com.au          |                /           \|
>                                   |    --________--
> PGP key available upon request,  |  /
> and from pgp5.ai.mit.edu:11371   |/
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sun, 20 Oct 2002 21:18:49 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 09:18 PM 20/10/2002 -0400, Bruce Momjian wrote:
>I will try to apply it within the next 48 hours.

I'm happy to apply it when necessary; but I wouldn't do it until we've from 
some someone with a big-endian machine...



----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Mon, 21 Oct 2002 11:47:40 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
Philip Warner wrote:
> At 09:18 PM 20/10/2002 -0400, Bruce Momjian wrote:
> >I will try to apply it within the next 48 hours.
> 
> I'm happy to apply it when necessary; but I wouldn't do it until we've from 
> some someone with a big-endian machine...

Well, I think Tom was going to try it on his HPUX machine.  However, it
is on the open items list, so we are going to need to get it in there
soon anyway, or yank it all out.  If no big endian people want to test
it, we will have to ship and then I am sure some big-ending testing will
happen.  ;-)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sun, 20 Oct 2002 21:50:30 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 09:50 PM 20/10/2002 -0400, Bruce Momjian wrote:
>Well, I think Tom was going to try it on his HPUX machine.

It might be good if someone who knows a little more than me about
endianness etc has a look at the patch - specifically this bit of code:

#if __BYTE_ORDER == __LITTLE_ENDIAN
         for (off = 0 ; off < sizeof(off_t) ; off++) {
#else
         for (off = sizeof(off_t) -1 ; off >= 0 ; off--) {
#endif
             i = *(char*)(ptr+off);
             (*AH->WriteBytePtr) (AH, i);
         }

It is *intended* to write the data such that the least significant byte
is written first to the file, but the dump Giles put on his FTP site
is not correct - it's written msb->lsb.

There seem to be two possibilities (a) I am an idiot and there is something
wrong with the code above that I can not see, or (b) the test:

     #if __BYTE_ORDER == __LITTLE_ENDIAN

is not the right thing to do. Any insights would be appreciated.



----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Mon, 21 Oct 2002 23:15:00 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
Philip Warner <pjw@rhyme.com.au> writes:
> It might be good if someone who knows a little more than me about
> endianness etc has a look at the patch - specifically this bit of code:

> #if __BYTE_ORDER == __LITTLE_ENDIAN

Well, the main problem with that is there's no such symbol as
__BYTE_ORDER ...

I'd prefer not to introduce one, either, if we can possibly avoid it.
I know that we have BYTE_ORDER defined in the port header files, but
I think it's quite untrustworthy, since there is no other place in the
main distribution that uses it anymore (AFAICS only contrib/pgcrypto
uses it at all).

The easiest way to write and reassemble an arithmetic value in a
platform-independent order is via shifting.  For instance,

	// write, LSB first
	for (i = 0; i < sizeof(off_t); i++)
	{
		writebyte(val & 0xFF);
		val >>= 8;
	}

	// read, LSB first
	val = 0;
	shift = 0;
	for (i = 0; i < sizeof(off_t); i++)
	{
		val |= (readbyte() << shift);
		shift += 8;
	}

(This assumes readbyte delivers an unsigned byte, else you might need to
mask it with 0xFF before shifting.)

			regards, tom lane

Mon, 21 Oct 2002 09:47:10 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: pg_dump and large files - is this a problem? 
At 09:47 AM 21/10/2002 -0400, Tom Lane wrote:
>Well, the main problem with that is there's no such symbol as
>__BYTE_ORDER ...

What about just:

     int i = 256;

then checking the first byte? This should give me the endianness, and makes 
a non-destructive write (not sure it it's important). Currently the 
commonly used code does not rely on off_t arithmetic, so if possible I'd 
like to avoid shift. Does that sound reasonable? Or overly cautious?







----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Tue, 22 Oct 2002 00:10:22 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem? 
Philip Warner <pjw@rhyme.com.au> writes:
> then checking the first byte? This should give me the endianness, and makes 
> a non-destructive write (not sure it it's important). Currently the 
> commonly used code does not rely on off_t arithmetic, so if possible I'd 
> like to avoid shift. Does that sound reasonable? Or overly cautious?

I think it's pointless.  Let's assume off_t is not an arithmetic type
but some weird struct dreamed up by a crazed kernel hacker.  What are
the odds that dumping the bytes in it, in either order, will produce
something that's compatible with any other platform?  There could be
padding, or the fields might be in an order that doesn't match the
byte order within the fields, or something else.

The shift method requires *no* directly endian-dependent code,
and I think it will work on any platform where you have any hope of
portability anyway.

			regards, tom lane

Mon, 21 Oct 2002 10:16:11 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: pg_dump and large files - is this a problem? 

Here is a modified version of Philip's patch that has the changes Tom
suggested;  treating off_t as an integral type.  I did light testing on
my BSD/OS machine that has 8-byte off_t but I don't have 4 gigs of free
space to test larger files.  

	ftp://candle.pha.pa.us/pub/postgresql/mypatches/pg_dump

Can others test?

---------------------------------------------------------------------------

Tom Lane wrote:
> Philip Warner <pjw@rhyme.com.au> writes:
> > then checking the first byte? This should give me the endianness, and makes 
> > a non-destructive write (not sure it it's important). Currently the 
> > commonly used code does not rely on off_t arithmetic, so if possible I'd 
> > like to avoid shift. Does that sound reasonable? Or overly cautious?
> 
> I think it's pointless.  Let's assume off_t is not an arithmetic type
> but some weird struct dreamed up by a crazed kernel hacker.  What are
> the odds that dumping the bytes in it, in either order, will produce
> something that's compatible with any other platform?  There could be
> padding, or the fields might be in an order that doesn't match the
> byte order within the fields, or something else.
> 
> The shift method requires *no* directly endian-dependent code,
> and I think it will work on any platform where you have any hope of
> portability anyway.
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 21 Oct 2002 21:47:37 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
On Mon, 2002-10-21 at 20:47, Bruce Momjian wrote:
> 
> Here is a modified version of Philip's patch that has the changes Tom
> suggested;  treating off_t as an integral type.  I did light testing on
> my BSD/OS machine that has 8-byte off_t but I don't have 4 gigs of free
> space to test larger files.  
I can make an account for anyone that wants to play on UnixWare 7.1.3.



-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


21 Oct 2002 20:50:07 -0500
Larry Rosenman <ler@lerctr.org>
Re: pg_dump and large files - is this a problem?
Larry Rosenman wrote:
> On Mon, 2002-10-21 at 20:47, Bruce Momjian wrote:
> > 
> > Here is a modified version of Philip's patch that has the changes Tom
> > suggested;  treating off_t as an integral type.  I did light testing on
> > my BSD/OS machine that has 8-byte off_t but I don't have 4 gigs of free
> > space to test larger files.  
> I can make an account for anyone that wants to play on UnixWare 7.1.3.

If you have 7.3, you can just test this way:
	
	1) apply the patch
	2) run the regression tests
	3) pg_dump -Fc regression >/tmp/x
	4) pg_restore  -Fc  </tmp/x

That's all I did and it worked.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 21 Oct 2002 21:52:13 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
On Mon, 2002-10-21 at 20:52, Bruce Momjian wrote:
> Larry Rosenman wrote:
> > On Mon, 2002-10-21 at 20:47, Bruce Momjian wrote:
> > > 
> > > Here is a modified version of Philip's patch that has the changes Tom
> > > suggested;  treating off_t as an integral type.  I did light testing on
> > > my BSD/OS machine that has 8-byte off_t but I don't have 4 gigs of free
> > > space to test larger files.  
> > I can make an account for anyone that wants to play on UnixWare 7.1.3.
> 
> If you have 7.3, you can just test this way:
I haven't had the time to play with 7.3 (busy on a NUMBER of other
things). 

I'm more than willing to supply resources, just my time is short right
now. 


> 	
> 	1) apply the patch
> 	2) run the regression tests
> 	3) pg_dump -Fc regression >/tmp/x
> 	4) pg_restore  -Fc  </tmp/x
> 
> That's all I did and it worked.
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


21 Oct 2002 21:05:09 -0500
Larry Rosenman <ler@lerctr.org>
Re: pg_dump and large files - is this a problem?
At 09:52 PM 21/10/2002 -0400, Bruce Momjian wrote:
>         4) pg_restore  -Fc  </tmp/x

pg_restore  /tmp/x

is enough; it will determine the file type, and by avoiding the pipe, you 
allow it to do seeks which are not much use here, but are usefull when you 
only restore one table in a very large backup.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Tue, 22 Oct 2002 17:52:37 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
At 10:16 AM 21/10/2002 -0400, Tom Lane wrote:
>What are
>the odds that dumping the bytes in it, in either order, will produce
>something that's compatible with any other platform?

None, but it will be compatible with itself (the most we can hope for), and 
will work even if shifting is not supported for off_t (how likely is 
that?). I agree shift is definitely the way to go if it works on arbitrary 
data - ie. it does not rely on off_t being an integer. Can I shift a struct?


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Tue, 22 Oct 2002 21:46:25 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem? 
Philip Warner <pjw@rhyme.com.au> writes:
> None, but it will be compatible with itself (the most we can hope for), and 
> will work even if shifting is not supported for off_t (how likely is 
> that?). I agree shift is definitely the way to go if it works on arbitrary 
> data - ie. it does not rely on off_t being an integer. Can I shift a struct?

You can't.  If there are any platforms where in fact off_t isn't an
arithmetic type, then shifting code would break there.  I am not sure
there are any; can anyone provide a counterexample?

It would be simple enough to add a configure test to see whether off_t
is arithmetic (just try to compile "off_t x; x <<= 8;").  How about
	#ifdef OFF_T_IS_ARITHMETIC_TYPE
		// cross-platform compatible
		use shifting method
	#else
		// not cross-platform compatible
		read or write bytes of struct in storage order
	#endif

			regards, tom lane

Tue, 22 Oct 2002 09:29:22 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: pg_dump and large files - is this a problem? 
Tom Lane wrote:
> Philip Warner <pjw@rhyme.com.au> writes:
> > None, but it will be compatible with itself (the most we can hope for), and 
> > will work even if shifting is not supported for off_t (how likely is 
> > that?). I agree shift is definitely the way to go if it works on arbitrary 
> > data - ie. it does not rely on off_t being an integer. Can I shift a struct?
> 
> You can't.  If there are any platforms where in fact off_t isn't an
> arithmetic type, then shifting code would break there.  I am not sure
> there are any; can anyone provide a counterexample?
> 
> It would be simple enough to add a configure test to see whether off_t
> is arithmetic (just try to compile "off_t x; x <<= 8;").  How about
> 	#ifdef OFF_T_IS_ARITHMETIC_TYPE
> 		// cross-platform compatible
> 		use shifting method
> 	#else
> 		// not cross-platform compatible
> 		read or write bytes of struct in storage order
> 	#endif

It is my understanding that off_t is an integral type and fpos_t is
perhaps a struct.  My fgetpos manual page says:

     The fgetpos() and fsetpos() functions are alternate interfaces equivalent
     to ftell() and fseek() (with whence set to SEEK_SET ), setting and stor-
     ing the current value of the file offset into or from the object refer-
     enced by pos. On some (non-UNIX) systems an ``fpos_t'' object may be a
     complex object and these routines may be the only way to portably reposi-
     tion a text stream.

I poked around and found this Usenet posting:

	http://groups.google.com/groups?q=C+off_t+standard+integral&hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=E958tG.8tH%40root.co.uk&rnum=1

stating that while off_t must be arithmetic, it doesn't have to be
integral, meaning it could be float or double, which can't be shifted.

However, since we don't know if we support any non-integral off_t
platforms, and because a configure test would require us to have two
code paths for with/without integral off_t, I suggest we apply my
version of Philip's patch and let's see if everyone can compile it
cleanly.  It does have the advantage of being more portable on systems
that do have integral off_t, which I think is most/all of our supported
platforms.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 22 Oct 2002 12:00:01 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 12:00 PM 22/10/2002 -0400, Bruce Momjian wrote:
>It does have the advantage of being more portable on systems
>that do have integral off_t

I suspect it is no more portable than determining storage order by using 
'int i = 256', then writing in storage order, and has the disadvantage that 
it may break as discussed.

AFAICT, using storage order will not break under any circumstances within 
one OS/architecture (unlike using shift), and will not break any more often 
than using shift in cases where off_t is integral.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Wed, 23 Oct 2002 02:20:45 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> However, since we don't know if we support any non-integral off_t
> platforms, and because a configure test would require us to have two
> code paths for with/without integral off_t, I suggest we apply my
> version of Philip's patch and let's see if everyone can compile it
> cleanly.

Actually, it looks to me like configure will spit up if off_t is not
an integral type:

 /* Check that off_t can represent 2**63 - 1 correctly.
    We can't simply define LARGE_OFF_T to be 9223372036854775807,
    since some C++ compilers masquerading as C compilers
    incorrectly reject 9223372036854775807.  */
#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
  int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
		       && LARGE_OFF_T % 2147483647 == 1)
		      ? 1 : -1];

So I think we're wasting our time to debate whether we need to support
non-integral off_t ... let's just apply Bruce's version and wait to
see if anyone has a problem before doing more work.

			regards, tom lane

Tue, 22 Oct 2002 12:22:57 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: pg_dump and large files - is this a problem? 
Philip Warner wrote:
> At 12:00 PM 22/10/2002 -0400, Bruce Momjian wrote:
> >It does have the advantage of being more portable on systems
> >that do have integral off_t
> 
> I suspect it is no more portable than determining storage order by using 
> 'int i = 256', then writing in storage order, and has the disadvantage that 
> it may break as discussed.
> 
> AFAICT, using storage order will not break under any circumstances within 
> one OS/architecture (unlike using shift), and will not break any more often 
> than using shift in cases where off_t is integral.

Your version will break more often because we are assuming we can
determine the endian-ness of the OS, _and_ for quad off_t types,
assuming we know that is stored the same too.  While we have ending for
int's, I have no idea if quads are always stored the same.  By accessing
it as an integral type, we make certain it is output the same way every
time for every OS.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 22 Oct 2002 12:26:30 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > However, since we don't know if we support any non-integral off_t
> > platforms, and because a configure test would require us to have two
> > code paths for with/without integral off_t, I suggest we apply my
> > version of Philip's patch and let's see if everyone can compile it
> > cleanly.
> 
> Actually, it looks to me like configure will spit up if off_t is not
> an integral type:
> 
>  /* Check that off_t can represent 2**63 - 1 correctly.
>     We can't simply define LARGE_OFF_T to be 9223372036854775807,
>     since some C++ compilers masquerading as C compilers
>     incorrectly reject 9223372036854775807.  */
> #define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
>   int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
> 		       && LARGE_OFF_T % 2147483647 == 1)
> 		      ? 1 : -1];
> 
> So I think we're wasting our time to debate whether we need to support
> non-integral off_t ... let's just apply Bruce's version and wait to
> see if anyone has a problem before doing more work.

I am concerned about one more thing.  On BSD/OS, we have off_t of quad
(8 byte), but we don't have fseeko, so this call looks questionable:

	if (fseeko(AH->FH, tctx->dataPos, SEEK_SET) != 0)

In this case, dataPos is off_t (8 bytes), while fseek only accepts long
in that parameter (4 bytes).  When this code is hit, a file > 4 gigs
will seek to the wrong offset, I am afraid.  Also, I don't understand
why the compiler doesn't produce a warning.

I wonder if I should add a conditional test so this code is hit only if
HAVE_FSEEKO is defined.  There is alternative code for all the non-zero
fseeks.

Comments?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 22 Oct 2002 12:46:55 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Your version will break more often because we are assuming we can
> determine the endian-ness of the OS, _and_ for quad off_t types,
> assuming we know that is stored the same too.  While we have ending for
> int's, I have no idea if quads are always stored the same.

There is precedent for problems of that ilk, too, cf PDP_ENDIAN: years
ago someone made double-word-integer software routines and did not
think twice about which word should appear first in storage, with the
consequence that the storage order was neither little-endian nor
big-endian.  (We have exactly the same issue with our CRC routines for
compilers without int64: the two-int32 struct is defined in a way that's
compatible with little-endian storage, and on a big-endian machine it'll
produce a funny storage order.)

Unless someone can point to a supported (or potentially interesting)
platform on which off_t is indeed not integral, I think the shift-based
code is our safest bet.  (The precedent of the off_t checking code in
configure makes me really doubt that there are any platforms with
non-integral off_t.)

			regards, tom lane

Tue, 22 Oct 2002 13:15:05 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: pg_dump and large files - is this a problem? 

Patch applied with shift <</>> changes by me.  Thanks.

---------------------------------------------------------------------------


Philip Warner wrote:
> 
> I have put the latest patch at:
> 
>      http://downloads.rhyme.com.au/postgresql/pg_dump/
> 
> along with two dump files of the regression DB, one with 4 byte
> and the other with 8 byte offsets. I can read/restore each from
> the other, so it looks pretty good. Once the endianness is tested,
> we should be OK.
> 
> Known problems:
> 
> - will not cope with > 4GB files and size_t not 64 bit.
> - when printing data position, it is assumed that off_t is UINT64
>    (we could remove this entirely - it's just for display)
> - if seek is not supported, then an intXX is assigned to off_t
>    when file offsets are needed. This *should* not cause a problem
>    since without seek, the offsets will not be written to the file.
> 
> Changes from Prior Version:
> 
> - No longer stores or outputs data length
> - Assumes result of ftello is correct if it disagrees with internally
>    kept tally.
> - 'pg_restore -l' now shows sizes of int and offset.
> 
> 
> ----------------------------------------------------------------
> Philip Warner                    |     __---_____
> Albatross Consulting Pty. Ltd.   |----/       -  \
> (A.B.N. 75 008 659 498)          |          /(@)   ______---_
> Tel: (+61) 0500 83 82 81         |                 _________  \
> Fax: (+61) 0500 83 82 82         |                 ___________ |
> Http://www.rhyme.com.au          |                /           \|
>                                   |    --________--
> PGP key available upon request,  |  /
> and from pgp5.ai.mit.edu:11371   |/
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 22 Oct 2002 15:16:18 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Bruce Momjian wrote:
> > So I think we're wasting our time to debate whether we need to support
> > non-integral off_t ... let's just apply Bruce's version and wait to
> > see if anyone has a problem before doing more work.
> 
> I am concerned about one more thing.  On BSD/OS, we have off_t of quad
> (8 byte), but we don't have fseeko, so this call looks questionable:
> 
> 	if (fseeko(AH->FH, tctx->dataPos, SEEK_SET) != 0)
> 
> In this case, dataPos is off_t (8 bytes), while fseek only accepts long
> in that parameter (4 bytes).  When this code is hit, a file > 4 gigs
> will seek to the wrong offset, I am afraid.  Also, I don't understand
> why the compiler doesn't produce a warning.
> 
> I wonder if I should add a conditional test so this code is hit only if
> HAVE_FSEEKO is defined.  There is alternative code for all the non-zero
> fseeks.

Here is a patch that I think fixes the problem I outlined above.  If
there is no fseeko(), it will not call fseek with a non-zero offset
unless sizeof(off_t) <= sizeof(long).

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Index: src/bin/pg_dump/pg_backup_custom.c
===================================================================
RCS file: /cvsroot/pgsql-server/src/bin/pg_dump/pg_backup_custom.c,v
retrieving revision 1.22
diff -c -c -r1.22 pg_backup_custom.c
*** src/bin/pg_dump/pg_backup_custom.c	22 Oct 2002 19:15:23 -0000	1.22
--- src/bin/pg_dump/pg_backup_custom.c	22 Oct 2002 21:36:30 -0000
***************
*** 431,437 ****
  	if (tctx->dataState == K_OFFSET_NO_DATA)
  		return;
  
! 	if (!ctx->hasSeek || tctx->dataState == K_OFFSET_POS_NOT_SET)
  	{
  		/* Skip over unnecessary blocks until we get the one we want. */
  
--- 431,441 ----
  	if (tctx->dataState == K_OFFSET_NO_DATA)
  		return;
  
! 	if (!ctx->hasSeek || tctx->dataState == K_OFFSET_POS_NOT_SET
! #if !defined(HAVE_FSEEKO)
! 		|| sizeof(off_t) > sizeof(long)
! #endif
! 		)
  	{
  		/* Skip over unnecessary blocks until we get the one we want. */
  
***************
*** 809,815 ****
  		 * be ok to just use the existing self-consistent block
  		 * formatting.
  		 */
! 		if (ctx->hasSeek)
  		{
  			fseeko(AH->FH, tpos, SEEK_SET);
  			WriteToc(AH);
--- 813,823 ----
  		 * be ok to just use the existing self-consistent block
  		 * formatting.
  		 */
! 		if (ctx->hasSeek
! #if !defined(HAVE_FSEEKO)
! 			&& sizeof(off_t) <= sizeof(long)
! #endif
! 			)
  		{
  			fseeko(AH->FH, tpos, SEEK_SET);
  			WriteToc(AH);
Tue, 22 Oct 2002 17:37:39 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Bruce Momjian writes:

> I am concerned about one more thing.  On BSD/OS, we have off_t of quad
> (8 byte), but we don't have fseeko, so this call looks questionable:
>
> 	if (fseeko(AH->FH, tctx->dataPos, SEEK_SET) != 0)

Maybe you want to ask your OS provider how the heck this is supposed to
work.  I mean, it's great to have wide types, but what's the point if the
API can't handle them?

-- 
Peter Eisentraut   peter_e@gmx.net


Wed, 23 Oct 2002 00:20:53 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: pg_dump and large files - is this a problem?
Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > I am concerned about one more thing.  On BSD/OS, we have off_t of quad
> > (8 byte), but we don't have fseeko, so this call looks questionable:
> >
> > 	if (fseeko(AH->FH, tctx->dataPos, SEEK_SET) != 0)
> 
> Maybe you want to ask your OS provider how the heck this is supposed to
> work.  I mean, it's great to have wide types, but what's the point if the
> API can't handle them?

Excellent question.  They do have fsetpos/fgetpos, and I think they
think you are supposed to use those.  However, they don't do seek from
current position, and they don't take an off_t, so I am confused myself.

I did ask on the mailing list and everyone kind of agreed it was a
missing feature.  However, because of the way we call fseeko not knowing
if it is a quad or a long, I think we have to add the checks to prevent
such wild seeks from happening.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 22 Oct 2002 21:13:19 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 05:37 PM 22/10/2002 -0400, Bruce Momjian wrote:
>!               if (ctx->hasSeek
>! #if !defined(HAVE_FSEEKO)
>!                       && sizeof(off_t) <= sizeof(long)
>! #endif
>!                       )

Just to clarify my understanding:

- HAVE_FSEEKO is tested & defined in configure
- If it is not defined, then all calls to fseeko will magically be 
translated to fseek calls, and use the 'long' parameter type.

Is that right?

If so, why don't we:

#if defined(HAVE_FSEEKO)
#define FILE_OFFSET off_t
#define FSEEK fseeko
#else
#define FILE_OFFSET long
#define FSEEK fseek
#end if

then replace all refs to off_t with FILE_OFFSET, and fseeko with FSEEK.

Existing checks etc will then refuse to load file offsets with significant 
bytes after the 4th byte, we will still use fseek/o in broken OS 
implementations of off_t.



----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Wed, 23 Oct 2002 12:39:23 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
Philip Warner wrote:
> At 05:37 PM 22/10/2002 -0400, Bruce Momjian wrote:
> >!               if (ctx->hasSeek
> >! #if !defined(HAVE_FSEEKO)
> >!                       && sizeof(off_t) <= sizeof(long)
> >! #endif
> >!                       )
> 
> Just to clarify my understanding:
> 
> - HAVE_FSEEKO is tested & defined in configure
> - If it is not defined, then all calls to fseeko will magically be 
> translated to fseek calls, and use the 'long' parameter type.
> 
> Is that right?
> 
> If so, why don't we:
> 
> #if defined(HAVE_FSEEKO)
> #define FILE_OFFSET off_t
> #define FSEEK fseeko
> #else
> #define FILE_OFFSET long
> #define FSEEK fseek
> #end if
> 
> then replace all refs to off_t with FILE_OFFSET, and fseeko with FSEEK.
> 
> Existing checks etc will then refuse to load file offsets with significant 
> bytes after the 4th byte, we will still use fseek/o in broken OS 
> implementations of off_t.

Uh, not exactly.  I have off_t as a quad, and I don't have fseeko, so
the above conditional doesn't work. I want to use off_t, but can't use
fseek().  As it turns out, the code already has options to handle no
fseek, so it seems to work anyway.  I think what you miss may be the
table of contents in the archive, if I am reading the code correctly.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 22 Oct 2002 22:46:17 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 10:46 PM 22/10/2002 -0400, Bruce Momjian wrote:
>Uh, not exactly.  I have off_t as a quad, and I don't have fseeko, so
>the above conditional doesn't work. I want to use off_t, but can't use
>fseek().

Then when you create dumps, they will be invalid since I assume that ftello 
is also broken in the same way. You need to fix _getFilePos as well. And 
any other place that uses an off_t needs to be looked at very carefully. 
The code was written assuming that if 'hasSeek' was set, then we could 
trust it.

Given that you say you do have support for some kind of 64 bt offset, I 
would be a lot happier with these changes if you did something akin to my 
original sauggestion:

#if defined(HAVE_FSEEKO)
#define FILE_OFFSET off_t
#define FSEEK fseeko
#elseif defined(HAVE_SOME_OTHER_FSEEK)
#define FILE_OFFSET some_other_offset
#define FSEEK some_other_fseek
#else
#define FILE_OFFSET long
#define FSEEK fseek
#end if

...assuming you have a non-broken 64 bit fseek/tell pair, then this will 
work in all cases, and make the code a lot less ugly (assuming of course 
the non-broken version can be shifted).



----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Wed, 23 Oct 2002 13:06:08 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?

Sounds messy.  Let me see if I can code up an fseeko/ftello for BSD/OS
and add that to /port.  No reason to hold up beta for that, though.

I wonder if any other platforms have this limitation.  I think we need
to add some type of test for no-fseeko()/ftello() and sizeof(off_t) >
sizeof(long).  This fseeko/ftello/off_t is just too fluid, and the
failure modes too serious.

---------------------------------------------------------------------------

Philip Warner wrote:
> At 10:46 PM 22/10/2002 -0400, Bruce Momjian wrote:
> >Uh, not exactly.  I have off_t as a quad, and I don't have fseeko, so
> >the above conditional doesn't work. I want to use off_t, but can't use
> >fseek().
> 
> Then when you create dumps, they will be invalid since I assume that ftello 
> is also broken in the same way. You need to fix _getFilePos as well. And 
> any other place that uses an off_t needs to be looked at very carefully. 
> The code was written assuming that if 'hasSeek' was set, then we could 
> trust it.
> 
> Given that you say you do have support for some kind of 64 bt offset, I 
> would be a lot happier with these changes if you did something akin to my 
> original sauggestion:
> 
> #if defined(HAVE_FSEEKO)
> #define FILE_OFFSET off_t
> #define FSEEK fseeko
> #elseif defined(HAVE_SOME_OTHER_FSEEK)
> #define FILE_OFFSET some_other_offset
> #define FSEEK some_other_fseek
> #else
> #define FILE_OFFSET long
> #define FSEEK fseek
> #end if
> 
> ...assuming you have a non-broken 64 bit fseek/tell pair, then this will 
> work in all cases, and make the code a lot less ugly (assuming of course 
> the non-broken version can be shifted).
> 
> 
> 
> ----------------------------------------------------------------
> Philip Warner                    |     __---_____
> Albatross Consulting Pty. Ltd.   |----/       -  \
> (A.B.N. 75 008 659 498)          |          /(@)   ______---_
> Tel: (+61) 0500 83 82 81         |                 _________  \
> Fax: (+61) 0500 83 82 82         |                 ___________ |
> Http://www.rhyme.com.au          |                /           \|
>                                   |    --________--
> PGP key available upon request,  |  /
> and from pgp5.ai.mit.edu:11371   |/
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 00:29:39 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I wonder if any other platforms have this limitation.  I think we need
> to add some type of test for no-fseeko()/ftello() and sizeof(off_t) >
> sizeof(long).  This fseeko/ftello/off_t is just too fluid, and the
> failure modes too serious.

I am wondering why pg_dump has to depend on either fseek or ftell.

			regards, tom lane

Wed, 23 Oct 2002 00:32:56 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: pg_dump and large files - is this a problem? 
At 12:32 AM 23/10/2002 -0400, Tom Lane wrote:
>I am wondering why pg_dump has to depend on either fseek or ftell.

It doesn't - it just works better and has more features if they are 
available, much like zlib etc.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Wed, 23 Oct 2002 14:36:06 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem? 
At 12:29 AM 23/10/2002 -0400, Bruce Momjian wrote:
>This fseeko/ftello/off_t is just too fluid, and the
>failure modes too serious.

I agree. Can you think of a better solution than the one I suggested???


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Wed, 23 Oct 2002 14:38:18 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?

OK, you are saying if we don't have fseeko(), there is no reason to use
off_t, and we may as well use long.  What limitations does that impose,
and are the limitations clear to the user.

What has me confused is that I only see two places that use a non-zero
fseeko, and in those cases, there is a non-fseeko code path that does
the same thing, or the call isn't actually required.  Both cases are in
pg_dump/pg_dump_custom.c.  It appears seeking in the file is an
optimization that prevents all the blocks from being read.  That is
fine, but we shouldn't introduce failure cases to do that.

If BSD/OS is the only problem OS, I can deal with that, but I have no
idea if other OS's have the same limitation, and because of the way our
code exists now, we are not even checking to see if there is a problem.

I did some poking around, and on BSD/OS, fgetpos/fsetpos use fpos_t,
which is actually off_t, and interestingly, lseek() uses off_t too. 
Seems only fseek/ftell is limited to long.  I can easily implemnt
fseeko/ftello using fgetpos/fsetpos, but that is only one OS.

One idea would be to patch up BSD/OS in backend/port/bsdi and add a
configure tests that actually fails if fseeko doesn't exist _and_
sizeof(off_t) > sizeof(long).  That would at least catch OS's before
they make >2gig backups that can't be restored.

---------------------------------------------------------------------------

Philip Warner wrote:
> At 10:46 PM 22/10/2002 -0400, Bruce Momjian wrote:
> >Uh, not exactly.  I have off_t as a quad, and I don't have fseeko, so
> >the above conditional doesn't work. I want to use off_t, but can't use
> >fseek().
> 
> Then when you create dumps, they will be invalid since I assume that ftello 
> is also broken in the same way. You need to fix _getFilePos as well. And 
> any other place that uses an off_t needs to be looked at very carefully. 
> The code was written assuming that if 'hasSeek' was set, then we could 
> trust it.
> 
> Given that you say you do have support for some kind of 64 bt offset, I 
> would be a lot happier with these changes if you did something akin to my 
> original sauggestion:
> 
> #if defined(HAVE_FSEEKO)
> #define FILE_OFFSET off_t
> #define FSEEK fseeko
> #elseif defined(HAVE_SOME_OTHER_FSEEK)
> #define FILE_OFFSET some_other_offset
> #define FSEEK some_other_fseek
> #else
> #define FILE_OFFSET long
> #define FSEEK fseek
> #end if
> 
> ...assuming you have a non-broken 64 bit fseek/tell pair, then this will 
> work in all cases, and make the code a lot less ugly (assuming of course 
> the non-broken version can be shifted).
> 
> 
> 
> ----------------------------------------------------------------
> Philip Warner                    |     __---_____
> Albatross Consulting Pty. Ltd.   |----/       -  \
> (A.B.N. 75 008 659 498)          |          /(@)   ______---_
> Tel: (+61) 0500 83 82 81         |                 _________  \
> Fax: (+61) 0500 83 82 82         |                 ___________ |
> Http://www.rhyme.com.au          |                /           \|
>                                   |    --________--
> PGP key available upon request,  |  /
> and from pgp5.ai.mit.edu:11371   |/
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 01:02:22 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 01:02 AM 23/10/2002 -0400, Bruce Momjian wrote:

>OK, you are saying if we don't have fseeko(), there is no reason to use
>off_t, and we may as well use long.  What limitations does that impose,
>and are the limitations clear to the user.

What I'm saying is that if we have not got fseeko then we should use any 
'seek-class' function that returns a 64 bit value. We have already made the 
assumption that off_t is an integer; the same logic that came to that 
conclusion, applies just as validly to the other seek functions.

Secondly, if there is no 64 bit 'seek-class' function, then we should 
probably use a size_t, but a long would probably be fine too. I am not 
particularly attached to this part; long, int etc etc. Whatever is most 
likely to return an integer and work with whatever function we choose.

As to implications: assuming they are all integers (which as you know I 
don't like), we should have no problems.

If a system does not have any function to access 64 bit file offsets, then 
I'd say they are pretty unlikely to have files > 2GB.




----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Wed, 23 Oct 2002 15:41:57 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
Philip Warner wrote:
> At 01:02 AM 23/10/2002 -0400, Bruce Momjian wrote:
> 
> >OK, you are saying if we don't have fseeko(), there is no reason to use
> >off_t, and we may as well use long.  What limitations does that impose,
> >and are the limitations clear to the user.
> 
> What I'm saying is that if we have not got fseeko then we should use any 
> 'seek-class' function that returns a 64 bit value. We have already made the 
> assumption that off_t is an integer; the same logic that came to that 
> conclusion, applies just as validly to the other seek functions.

Oh, I see, so try to use fsetpos/fgetpos?  I can write wrappers for
those to look like fgetpos/fsetpos and put it in /port.

> Secondly, if there is no 64 bit 'seek-class' function, then we should 
> probably use a size_t, but a long would probably be fine too. I am not 
> particularly attached to this part; long, int etc etc. Whatever is most 
> likely to return an integer and work with whatever function we choose.
> 
> As to implications: assuming they are all integers (which as you know I 
> don't like), we should have no problems.
> 
> If a system does not have any function to access 64 bit file offsets, then 
> I'd say they are pretty unlikely to have files > 2GB.

OK, my OS can handle 64-bit files, but has only fgetpos/fsetpos, so I
could get that working.  The bigger question is what about OS's that
have 64-bit off_t/files but don't have any seek-type functions.  I did
research to find mine, but what about others that may have other
variants?

I think you are right that we have to not use off_t and use long if we
can't find a proper 64-bit seek function, but what are the failure modes
of doing this?  Exactly what happens for larger files?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 10:24:58 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Philip Warner wrote:
> At 01:02 AM 23/10/2002 -0400, Bruce Momjian wrote:
> 
> >OK, you are saying if we don't have fseeko(), there is no reason to use
> >off_t, and we may as well use long.  What limitations does that impose,
> >and are the limitations clear to the user.
> 
> What I'm saying is that if we have not got fseeko then we should use any 
> 'seek-class' function that returns a 64 bit value. We have already made the 
> assumption that off_t is an integer; the same logic that came to that 
> conclusion, applies just as validly to the other seek functions.
> 
> Secondly, if there is no 64 bit 'seek-class' function, then we should 
> probably use a size_t, but a long would probably be fine too. I am not 
> particularly attached to this part; long, int etc etc. Whatever is most 
> likely to return an integer and work with whatever function we choose.
> 
> As to implications: assuming they are all integers (which as you know I 
> don't like), we should have no problems.
> 
> If a system does not have any function to access 64 bit file offsets, then 
> I'd say they are pretty unlikely to have files > 2GB.

Let me see if I can be clearer.  With shifting off_t, if that fails, we
will find out right away, at compile time.  I think that is acceptable.

What I am concerned about are cases that fail at runtime, specifically
during a restore of a >2gig file.  In my reading of the code, those
failures will be silent or will produce unusual error messages.  I don't
think we can ship code that has strange failure modes for data restore.

Now, if someone knows those failure cases, I would love to hear about
it.  If not, I will dig into the code today and find out where they are.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 10:42:03 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Bruce Momjian writes:

> I think you are right that we have to not use off_t and use long if we
> can't find a proper 64-bit seek function, but what are the failure modes
> of doing this?  Exactly what happens for larger files?

First we need to decide what we want to happen and after that think about
how to implement it.  Given sizeof(off_t) > sizeof(long) and no fseeko(),
we have the following options:

1. Disable access to large files.

2. Seek in some other way.

What's it gonna be?

-- 
Peter Eisentraut   peter_e@gmx.net


Wed, 23 Oct 2002 23:50:34 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: pg_dump and large files - is this a problem?
Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > I think you are right that we have to not use off_t and use long if we
> > can't find a proper 64-bit seek function, but what are the failure modes
> > of doing this?  Exactly what happens for larger files?
> 
> First we need to decide what we want to happen and after that think about
> how to implement it.  Given sizeof(off_t) > sizeof(long) and no fseeko(),
> we have the following options:
> 
> 1. Disable access to large files.
> 
> 2. Seek in some other way.
> 
> What's it gonna be?

OK, well BSD/OS now works, but I wonder if there are any other quad
off_t OS's out there without fseeko.

How would we disable access to large files?  Do we fstat the file and
see if it is too large?   I suppose we are looking for cases where the
file system has large files, but fseeko doesn't allow us to access them.
Should we leave this issue alone and wait to find another OS with this
problem, and we can then rejigger fseeko.c to handle that OS too?

Looking at the pg_dump code, it seems the fseeks are optional in there
anyway because it already has code to read the file sequentially rather
than use fseek, and the TOC case in pg_backup_custom.c says that is
optional too.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 17:50:50 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Peter Eisentraut <peter_e@gmx.net> writes:
> First we need to decide what we want to happen and after that think about
> how to implement it.  Given sizeof(off_t) > sizeof(long) and no fseeko(),
> we have the following options:

It seems obvious to me that there are no platforms that offer
sizeof(off_t) > sizeof(long) but have no API for doing seeks with off_t.
That would be just plain silly.  IMHO it's acceptable for us to fail at
configure time if we can't figure out how to seek.

The question is *which* seek APIs we need to support.  Are there any
besides fseeko() and fgetpos()?

			regards, tom lane

Wed, 23 Oct 2002 17:52:52 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: pg_dump and large files - is this a problem? 
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> How would we disable access to large files?

I think configure should fail if it can't find a way to seek.
Workaround for anyone in that situation is configure --disable-largefile.

			regards, tom lane

Wed, 23 Oct 2002 17:55:37 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: pg_dump and large files - is this a problem? 
Tom Lane wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > First we need to decide what we want to happen and after that think about
> > how to implement it.  Given sizeof(off_t) > sizeof(long) and no fseeko(),
> > we have the following options:
> 
> It seems obvious to me that there are no platforms that offer
> sizeof(off_t) > sizeof(long) but have no API for doing seeks with off_t.
> That would be just plain silly.  IMHO it's acceptable for us to fail at
> configure time if we can't figure out how to seek.

I would certainly be happy failing at configure time, so we know at the
start what is broken, rather than failures during restore.

> The question is *which* seek APIs we need to support.  Are there any
> besides fseeko() and fgetpos()?

What I have added is BSD/OS specific because only on BSD/OS do I know
fpos_t and off_t are the same type.  If we come up with other platforms,
we will have to deal with it then.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 18:01:57 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?

Bruce Momjian <pgman@candle.pha.pa.us> writes:

> OK, well BSD/OS now works, but I wonder if there are any other quad
> off_t OS's out there without fseeko.

NetBSD prior to 1.6, released September 14, 2002. (Source: CVS logs.)

OpenBSD prior to 2.7, released June 15, 2000.  (Source: release notes.)

FreeBSD has had fseeko() for some time, but I'm not sure which release
introduced it -- perhaps 3.2.0, released May, 1999. (Source: CVS logs.)

Regards,

Giles





Thu, 24 Oct 2002 11:14:19 +1000
Giles Lean <giles@nemeton.com.au>
Re: pg_dump and large files - is this a problem? 
At 05:50 PM 23/10/2002 -0400, Bruce Momjian wrote:
>Looking at the pg_dump code, it seems the fseeks are optional in there
>anyway because it already has code to read the file sequentially rather

But there are features that are not available if it can't seek: eg. it will 
not restore in a different order to that in which it was written; it will 
not dump data offsets in the TOC so dump files can not be restored in 
alternate orders; restore times will be large for a single table (it has to 
read the entire file potentially).


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 24 Oct 2002 11:33:50 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
Philip Warner wrote:
> At 05:50 PM 23/10/2002 -0400, Bruce Momjian wrote:
> >Looking at the pg_dump code, it seems the fseeks are optional in there
> >anyway because it already has code to read the file sequentially rather
> 
> But there are features that are not available if it can't seek: eg. it will 
> not restore in a different order to that in which it was written; it will 
> not dump data offsets in the TOC so dump files can not be restored in 
> alternate orders; restore times will be large for a single table (it has to 
> read the entire file potentially).

OK, that helps.  We just got a list of 2 other OS's without fseeko and
with large file support.  Any NetBSD before Auguest 2002 has that
problem.  We are going to need to either get fseeko workarounds for
those, or disable those features in a meaningful way.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 21:36:35 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Giles Lean wrote:
> 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> 
> > OK, well BSD/OS now works, but I wonder if there are any other quad
> > off_t OS's out there without fseeko.
> 
> NetBSD prior to 1.6, released September 14, 2002. (Source: CVS logs.)

OK, does pre-1.6 NetBSD have fgetpos/fsetpos that is off_t/quad?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 21:37:08 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 10:42 AM 23/10/2002 -0400, Bruce Momjian wrote:
>What I am concerned about are cases that fail at runtime, specifically
>during a restore of a >2gig file.

Please give an example that would still apply assuming we get a working 
seek/tell pair that works with whatever we use as an offset?

If you are concerned about reading a dump file with 8 byte offsets on a 
machine with 4 byte off_t, that case and it's permutations are already covered.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 24 Oct 2002 11:37:20 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
Philip Warner wrote:
> At 10:42 AM 23/10/2002 -0400, Bruce Momjian wrote:
> >What I am concerned about are cases that fail at runtime, specifically
> >during a restore of a >2gig file.
> 
> Please give an example that would still apply assuming we get a working 
> seek/tell pair that works with whatever we use as an offset?

If we get this, everything is fine.  I have done that for BSD/OS today. 
I may need to do the same for NetBSD/OpenBSD too.

> If you are concerned about reading a dump file with 8 byte offsets on a 
> machine with 4 byte off_t, that case and it's permutations are already covered.

No, I know that is covered because it will report a proper error message
on the restore on the 4-byte off_t machine.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 21:41:45 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 11:50 PM 23/10/2002 +0200, Peter Eisentraut wrote:

>1. Disable access to large files.
>
>2. Seek in some other way.

This gets my vote, but I would like to see a clean implementation (not huge 
quantities if ifdefs every time we call fseek); either we write our own 
fseek as Bruce seems to be suggesting, or we have a single header file that 
defines the FSEEK/FTELL/OFF_T to point to the 'right' functions, where 
'right' is defined as 'most likely to generate an integer and which makes 
use of the largest number of bytes'.

The way the code is currently written it does not matter if this is a 16 or 
3 byte value - so long as it is an integer.



----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 24 Oct 2002 11:43:14 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
Philip Warner wrote:
> At 11:50 PM 23/10/2002 +0200, Peter Eisentraut wrote:
> 
> >1. Disable access to large files.
> >
> >2. Seek in some other way.
> 
> This gets my vote, but I would like to see a clean implementation (not huge 
> quantities if ifdefs every time we call fseek); either we write our own 
> fseek as Bruce seems to be suggesting, or we have a single header file that 
> defines the FSEEK/FTELL/OFF_T to point to the 'right' functions, where 
> 'right' is defined as 'most likely to generate an integer and which makes 
> use of the largest number of bytes'.

We have to write another function because fsetpos doesn't do SEEK_CUR so
you have to implement it with more complex code.  It isn't a drop in
place thing.

> The way the code is currently written it does not matter if this is a 16 or 
> 3 byte value - so long as it is an integer.

Right. What we are assuming now is that off_t can be seeked using
whatever we defined for fseeko, which is incorrect in one, and now I
hear more than one OS.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 21:45:55 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 09:36 PM 23/10/2002 -0400, Bruce Momjian wrote:
>We are going to need to either get fseeko workarounds for
>those, or disable those features in a meaningful way.

????? if we have not got a 64 bit seek function of any kind, then use a 32 
bit seek - the features don't need to be disabled. AFAICT, this is a 
non-issue: no 64 bit seek means no large files.

I'm not sure we should even worry about it, but if you are genuinely 
concerned that we have no 64 bit seek call, but we do have files > 4GB, 
then If you really want to disable seek, just modify the code that sets 
'hasSeek' - don't screw around with every seek call. But only modify clear 
it if the file is > 4GB.






----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 24 Oct 2002 11:50:47 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
At 09:41 PM 23/10/2002 -0400, Bruce Momjian wrote:
>If we get this, everything is fine.  I have done that for BSD/OS today.
>I may need to do the same for NetBSD/OpenBSD too.

What did you do to achieve this?


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 24 Oct 2002 11:51:42 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
Philip Warner wrote:
> At 09:41 PM 23/10/2002 -0400, Bruce Momjian wrote:
> >If we get this, everything is fine.  I have done that for BSD/OS today.
> >I may need to do the same for NetBSD/OpenBSD too.
> 
> What did you do to achieve this?

See src/port/fseeko.c in current CVS, with some configure.in glue.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 21:52:55 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 09:45 PM 23/10/2002 -0400, Bruce Momjian wrote:
>We have to write another function because fsetpos doesn't do SEEK_CUR so
>you have to implement it with more complex code.  It isn't a drop in
>place thing.

The only code that uses SEEK_CUR is the code to check if seek is available 
- I am ver happy to change that to SEEK_SET - I can't even recall why I 
used SEEK_CUR. The code that does the real seeks uses SEEK_SET.



----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 24 Oct 2002 11:55:17 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
At 11:55 AM 24/10/2002 +1000, Philip Warner wrote:

>The only code that uses SEEK_CUR is the code to check if seek is available 
>- I am ver happy to change that to SEEK_SET - I can't even recall why I 
>used SEEK_CUR. The code that does the real seeks uses SEEK_SET.

Come to think of it:

     ctx->hasSeek = (fseeko(AH->FH, 0, SEEK_CUR) == 0);

should be replaced by:

#ifdef HAS_FSEEK[O]
     ctx->hasSeek = TRUE;
#else
     ctx->hasSeek = FALSE;
#endif

Since we're now checking for it in configure, we should remove the checks 
from the pg_dump code.




----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 24 Oct 2002 12:02:14 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
Philip Warner wrote:
> At 09:45 PM 23/10/2002 -0400, Bruce Momjian wrote:
> >We have to write another function because fsetpos doesn't do SEEK_CUR so
> >you have to implement it with more complex code.  It isn't a drop in
> >place thing.
> 
> The only code that uses SEEK_CUR is the code to check if seek is available 
> - I am ver happy to change that to SEEK_SET - I can't even recall why I 
> used SEEK_CUR. The code that does the real seeks uses SEEK_SET.

There are other problems.  fgetpos() expects a pointer to an fpos_t,
while ftello just returns off_t, so you need a local variable in the
function to pass to fgetpos() and they return that from the function.

It is much cleaner to just duplicate the entire API so you don't have
any limitations or failure cases.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 22:03:17 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?

Well, that certainly changes the functionality of the code.  I thought
that fseeko test was done so that things that couldn't be seeked on were
detected.  Not sure what isn't seek-able, maybe named pipes.  I thought
it was testing that so I didn't touch that variable.

This was my original thought, that we have non-fseeko code in place. 
Can we just trigger the non-fseeko code on HAS_FSEEKO.  The code would
be something like:
	
	if (sizeof(long) >= sizeof(off_t))
		ctx->hasSeek = TRUE;
	else
	#ifdef HAVE_FSEEKO
	     ctx->hasSeek = TRUE;
	#else
	     ctx->hasSeek = FALSE;
	#endif

---------------------------------------------------------------------------

Philip Warner wrote:
> At 11:55 AM 24/10/2002 +1000, Philip Warner wrote:
> 
> >The only code that uses SEEK_CUR is the code to check if seek is available 
> >- I am ver happy to change that to SEEK_SET - I can't even recall why I 
> >used SEEK_CUR. The code that does the real seeks uses SEEK_SET.
> 
> Come to think of it:
> 
>      ctx->hasSeek = (fseeko(AH->FH, 0, SEEK_CUR) == 0);
> 
> should be replaced by:
> 
> #ifdef HAS_FSEEK[O]
>      ctx->hasSeek = TRUE;
> #else
>      ctx->hasSeek = FALSE;
> #endif
> 
> Since we're now checking for it in configure, we should remove the checks 
> from the pg_dump code.
> 
> 
> 
> 
> ----------------------------------------------------------------
> Philip Warner                    |     __---_____
> Albatross Consulting Pty. Ltd.   |----/       -  \
> (A.B.N. 75 008 659 498)          |          /(@)   ______---_
> Tel: (+61) 0500 83 82 81         |                 _________  \
> Fax: (+61) 0500 83 82 82         |                 ___________ |
> Http://www.rhyme.com.au          |                /           \|
>                                   |    --________--
> PGP key available upon request,  |  /
> and from pgp5.ai.mit.edu:11371   |/
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 22:08:15 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 10:08 PM 23/10/2002 -0400, Bruce Momjian wrote:
>Well, that certainly changes the functionality of the code.  I thought
>that fseeko test was done so that things that couldn't be seeked on were
>detected.

You are quite correct. It should read:

         #ifdef HAVE_FSEEKO
              ctx->hasSeek = fseeko(...,SEEK_SET);
         #else
              ctx->hasSeek = FALSE;
         #endif

pipes are the main case for which we are checking.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 24 Oct 2002 12:14:35 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
At 10:03 PM 23/10/2002 -0400, Bruce Momjian wrote:
>It is much cleaner to just duplicate the entire API so you don't have
>any limitations or failure cases.

We may still end up using macros in pg_dump to cope with cases where off_t 
& fseeko are not defined - if there are any. I presume we would then just 
revert to calling fseek/ftell etc.




----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 24 Oct 2002 12:36:20 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
Philip Warner wrote:
> At 10:03 PM 23/10/2002 -0400, Bruce Momjian wrote:
> >It is much cleaner to just duplicate the entire API so you don't have
> >any limitations or failure cases.
> 
> We may still end up using macros in pg_dump to cope with cases where off_t 
> & fseeko are not defined - if there are any. I presume we would then just 
> revert to calling fseek/ftell etc.

Well, we have fseeko falling back to fseek already, so that is working
fine.  I don't think we will find any OS's without off_t.  We just need
a little smarts.  Let me see if I can work on it now.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 22:37:49 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?

> OK, does pre-1.6 NetBSD have fgetpos/fsetpos that is off_t/quad?

Yes:

    int
    fgetpos(FILE *stream, fpos_t *pos);

    int
    fsetpos(FILE *stream, const fpos_t *pos);

Per comments in <stdio.h> fpos_t is the same format as off_t, and
off_t and fpos_t have been 64 bit since 1994.

    http://cvsweb.netbsd.org/bsdweb.cgi/basesrc/include/stdio.h

Regards,

Giles





Thu, 24 Oct 2002 12:54:50 +1000
Giles Lean <giles@nemeton.com.au>
Re: pg_dump and large files - is this a problem? 

Looks like I have some more work to do.  Thanks.

---------------------------------------------------------------------------

Giles Lean wrote:
> 
> > OK, does pre-1.6 NetBSD have fgetpos/fsetpos that is off_t/quad?
> 
> Yes:
> 
>     int
>     fgetpos(FILE *stream, fpos_t *pos);
> 
>     int
>     fsetpos(FILE *stream, const fpos_t *pos);
> 
> Per comments in <stdio.h> fpos_t is the same format as off_t, and
> off_t and fpos_t have been 64 bit since 1994.
> 
>     http://cvsweb.netbsd.org/bsdweb.cgi/basesrc/include/stdio.h
> 
> Regards,
> 
> Giles
> 
> 
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 22:56:01 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?

OK, NetBSD added.  

Any other OS's need this?  Is it safe for me to code something that
assumes fpos_t and off_t are identical?  I can't think of a good way to
test if two data types are identical.  I don't think sizeof is enough.

---------------------------------------------------------------------------

Giles Lean wrote:
> 
> > OK, does pre-1.6 NetBSD have fgetpos/fsetpos that is off_t/quad?
> 
> Yes:
> 
>     int
>     fgetpos(FILE *stream, fpos_t *pos);
> 
>     int
>     fsetpos(FILE *stream, const fpos_t *pos);
> 
> Per comments in <stdio.h> fpos_t is the same format as off_t, and
> off_t and fpos_t have been 64 bit since 1994.
> 
>     http://cvsweb.netbsd.org/bsdweb.cgi/basesrc/include/stdio.h
> 
> Regards,
> 
> Giles
> 
> 
> 
> 
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 23 Oct 2002 23:11:44 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Bruce Momjian writes:

> OK, NetBSD added.
>
> Any other OS's need this?  Is it safe for me to code something that
> assumes fpos_t and off_t are identical?  I can't think of a good way to
> test if two data types are identical.  I don't think sizeof is enough.

No, you can't assume that fpos_t and off_t are identical.

But you can simulate a long fseeko() by calling fseek() multiple times, so
it should be possible to write a replacement that works on all systems.

-- 
Peter Eisentraut   peter_e@gmx.net


Fri, 25 Oct 2002 00:12:56 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: pg_dump and large files - is this a problem?
Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > OK, NetBSD added.
> >
> > Any other OS's need this?  Is it safe for me to code something that
> > assumes fpos_t and off_t are identical?  I can't think of a good way to
> > test if two data types are identical.  I don't think sizeof is enough.
> 
> No, you can't assume that fpos_t and off_t are identical.

I was wondering --- if fpos_t and off_t are identical sizeof, and fpos_t
can do shift << or >>, that means fpos_t is also integral like off_t.
Can I then assume they are the same?

> But you can simulate a long fseeko() by calling fseek() multiple times, so
> it should be possible to write a replacement that works on all systems.

Yes, but I can't simulate ftello, so I then can't do SEEK_CUR. and if I
can't duplicate the entire API, I don't want to try.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 24 Oct 2002 19:43:00 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Philip Warner wrote:
> At 10:08 PM 23/10/2002 -0400, Bruce Momjian wrote:
> >Well, that certainly changes the functionality of the code.  I thought
> >that fseeko test was done so that things that couldn't be seeked on were
> >detected.
> 
> You are quite correct. It should read:
> 
>          #ifdef HAVE_FSEEKO
>               ctx->hasSeek = fseeko(...,SEEK_SET);
>          #else
>               ctx->hasSeek = FALSE;
>          #endif
> 
> pipes are the main case for which we are checking.

OK, I have applied the following patch to set hasSeek only if
fseek/fseeko is reliable.  This takes care of the random failure case
for large files.  Now I need to see if I can get the custom fseeko
working for more platforms.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Index: src/bin/pg_dump/common.c
===================================================================
RCS file: /cvsroot/pgsql-server/src/bin/pg_dump/common.c,v
retrieving revision 1.71
diff -c -c -r1.71 common.c
*** src/bin/pg_dump/common.c	9 Oct 2002 16:20:25 -0000	1.71
--- src/bin/pg_dump/common.c	25 Oct 2002 01:30:51 -0000
***************
*** 290,296 ****
  		 * attr with the same name, then only dump it if:
  		 *
  		 * - it is NOT NULL and zero parents are NOT NULL
! 		 *   OR 
  		 * - it has a default value AND the default value does not match
  		 *   all parent default values, or no parents specify a default.
  		 *
--- 290,296 ----
  		 * attr with the same name, then only dump it if:
  		 *
  		 * - it is NOT NULL and zero parents are NOT NULL
! 		 *   OR
  		 * - it has a default value AND the default value does not match
  		 *   all parent default values, or no parents specify a default.
  		 *
Index: src/bin/pg_dump/pg_backup_archiver.c
===================================================================
RCS file: /cvsroot/pgsql-server/src/bin/pg_dump/pg_backup_archiver.c,v
retrieving revision 1.59
diff -c -c -r1.59 pg_backup_archiver.c
*** src/bin/pg_dump/pg_backup_archiver.c	22 Oct 2002 19:15:23 -0000	1.59
--- src/bin/pg_dump/pg_backup_archiver.c	25 Oct 2002 01:30:57 -0000
***************
*** 2338,2343 ****
--- 2338,2369 ----
  }
  
  
+ /*
+  * checkSeek
+  *	  check to see if fseek can be performed.
+  */
+ 
+ bool
+ checkSeek(FILE *fp)
+ {
+ 
+ 	if (fseek(fp, 0, SEEK_CUR) != 0)
+ 		return false;
+ 	else if (sizeof(off_t) > sizeof(long))
+ 	/*
+ 	 *	At this point, off_t is too large for long, so we return
+ 	 *	based on whether an off_t version of fseek is available.
+ 	 */
+ #ifdef HAVE_FSEEKO
+ 		return true;
+ #else
+ 		return false;
+ #endif
+ 	else
+ 		return true;
+ }
+ 
+ 
  static void
  _SortToc(ArchiveHandle *AH, TocSortCompareFn fn)
  {
Index: src/bin/pg_dump/pg_backup_archiver.h
===================================================================
RCS file: /cvsroot/pgsql-server/src/bin/pg_dump/pg_backup_archiver.h,v
retrieving revision 1.48
diff -c -c -r1.48 pg_backup_archiver.h
*** src/bin/pg_dump/pg_backup_archiver.h	22 Oct 2002 19:15:23 -0000	1.48
--- src/bin/pg_dump/pg_backup_archiver.h	25 Oct 2002 01:30:58 -0000
***************
*** 27,32 ****
--- 27,33 ----
  
  #include "postgres_fe.h"
  
+ #include <stdio.h>
  #include <time.h>
  #include <errno.h>
  
***************
*** 284,289 ****
--- 285,291 ----
  extern void WriteDataChunks(ArchiveHandle *AH);
  
  extern int	TocIDRequired(ArchiveHandle *AH, int id, RestoreOptions *ropt);
+ extern bool checkSeek(FILE *fp);
  
  /*
   * Mandatory routines for each supported format
Index: src/bin/pg_dump/pg_backup_custom.c
===================================================================
RCS file: /cvsroot/pgsql-server/src/bin/pg_dump/pg_backup_custom.c,v
retrieving revision 1.22
diff -c -c -r1.22 pg_backup_custom.c
*** src/bin/pg_dump/pg_backup_custom.c	22 Oct 2002 19:15:23 -0000	1.22
--- src/bin/pg_dump/pg_backup_custom.c	25 Oct 2002 01:31:01 -0000
***************
*** 179,185 ****
  		if (!AH->FH)
  			die_horribly(AH, modulename, "could not open archive file %s: %s\n", AH->fSpec, strerror(errno));
  
! 		ctx->hasSeek = (fseeko(AH->FH, 0, SEEK_CUR) == 0);
  	}
  	else
  	{
--- 179,185 ----
  		if (!AH->FH)
  			die_horribly(AH, modulename, "could not open archive file %s: %s\n", AH->fSpec, strerror(errno));
  
! 		ctx->hasSeek = checkSeek(AH->FH);
  	}
  	else
  	{
***************
*** 190,196 ****
  		if (!AH->FH)
  			die_horribly(AH, modulename, "could not open archive file %s: %s\n", AH->fSpec, strerror(errno));
  
! 		ctx->hasSeek = (fseeko(AH->FH, 0, SEEK_CUR) == 0);
  
  		ReadHead(AH);
  		ReadToc(AH);
--- 190,196 ----
  		if (!AH->FH)
  			die_horribly(AH, modulename, "could not open archive file %s: %s\n", AH->fSpec, strerror(errno));
  
! 		ctx->hasSeek = checkSeek(AH->FH);
  
  		ReadHead(AH);
  		ReadToc(AH);
Index: src/bin/pg_dump/pg_backup_files.c
===================================================================
RCS file: /cvsroot/pgsql-server/src/bin/pg_dump/pg_backup_files.c,v
retrieving revision 1.20
diff -c -c -r1.20 pg_backup_files.c
*** src/bin/pg_dump/pg_backup_files.c	22 Oct 2002 19:15:23 -0000	1.20
--- src/bin/pg_dump/pg_backup_files.c	25 Oct 2002 01:31:01 -0000
***************
*** 129,135 ****
  		if (AH->FH == NULL)
  			die_horribly(NULL, modulename, "could not open output file: %s\n", strerror(errno));
  
! 		ctx->hasSeek = (fseeko(AH->FH, 0, SEEK_CUR) == 0);
  
  		if (AH->compression < 0 || AH->compression > 9)
  			AH->compression = Z_DEFAULT_COMPRESSION;
--- 129,135 ----
  		if (AH->FH == NULL)
  			die_horribly(NULL, modulename, "could not open output file: %s\n", strerror(errno));
  
! 		ctx->hasSeek = checkSeek(AH->FH);
  
  		if (AH->compression < 0 || AH->compression > 9)
  			AH->compression = Z_DEFAULT_COMPRESSION;
***************
*** 147,153 ****
  		if (AH->FH == NULL)
  			die_horribly(NULL, modulename, "could not open input file: %s\n", strerror(errno));
  
! 		ctx->hasSeek = (fseeko(AH->FH, 0, SEEK_CUR) == 0);
  
  		ReadHead(AH);
  		ReadToc(AH);
--- 147,153 ----
  		if (AH->FH == NULL)
  			die_horribly(NULL, modulename, "could not open input file: %s\n", strerror(errno));
  
! 		ctx->hasSeek = checkSeek(AH->FH);
  
  		ReadHead(AH);
  		ReadToc(AH);
Index: src/bin/pg_dump/pg_backup_tar.c
===================================================================
RCS file: /cvsroot/pgsql-server/src/bin/pg_dump/pg_backup_tar.c,v
retrieving revision 1.31
diff -c -c -r1.31 pg_backup_tar.c
*** src/bin/pg_dump/pg_backup_tar.c	22 Oct 2002 19:15:23 -0000	1.31
--- src/bin/pg_dump/pg_backup_tar.c	25 Oct 2002 01:31:04 -0000
***************
*** 190,196 ****
  		 */
  		/* setvbuf(ctx->tarFH, NULL, _IONBF, 0); */
  
! 		ctx->hasSeek = (fseeko(ctx->tarFH, 0, SEEK_CUR) == 0);
  
  		if (AH->compression < 0 || AH->compression > 9)
  			AH->compression = Z_DEFAULT_COMPRESSION;
--- 190,196 ----
  		 */
  		/* setvbuf(ctx->tarFH, NULL, _IONBF, 0); */
  
! 		ctx->hasSeek = checkSeek(ctx->tarFH);
  
  		if (AH->compression < 0 || AH->compression > 9)
  			AH->compression = Z_DEFAULT_COMPRESSION;
***************
*** 227,233 ****
  
  		ctx->tarFHpos = 0;
  
! 		ctx->hasSeek = (fseeko(ctx->tarFH, 0, SEEK_CUR) == 0);
  
  		/*
  		 * Forcibly unmark the header as read since we use the lookahead
--- 227,233 ----
  
  		ctx->tarFHpos = 0;
  
! 		ctx->hasSeek = checkSeek(ctx->tarFH);
  
  		/*
  		 * Forcibly unmark the header as read since we use the lookahead
Thu, 24 Oct 2002 21:32:12 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?

The patch will not work. Please reread my quoted email.

At 09:32 PM 24/10/2002 -0400, Bruce Momjian wrote:
>Philip Warner wrote:
> >
> > You are quite correct. It should read:
> >
> >          #ifdef HAVE_FSEEKO
> >               ctx->hasSeek = fseeko(...,SEEK_SET);
> >          #else
> >               ctx->hasSeek = FALSE;
> >          #endif
> >
> > pipes are the main case for which we are checking.
>
>OK, I have applied the following patch to set hasSeek only if
>fseek/fseeko is reliable.



----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Fri, 25 Oct 2002 11:51:48 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?

You are going to have to be more specific than that.

---------------------------------------------------------------------------

Philip Warner wrote:
> 
> The patch will not work. Please reread my quoted email.
> 
> At 09:32 PM 24/10/2002 -0400, Bruce Momjian wrote:
> >Philip Warner wrote:
> > >
> > > You are quite correct. It should read:
> > >
> > >          #ifdef HAVE_FSEEKO
> > >               ctx->hasSeek = fseeko(...,SEEK_SET);
> > >          #else
> > >               ctx->hasSeek = FALSE;
> > >          #endif
> > >
> > > pipes are the main case for which we are checking.
> >
> >OK, I have applied the following patch to set hasSeek only if
> >fseek/fseeko is reliable.
> 
> 
> 
> ----------------------------------------------------------------
> Philip Warner                    |     __---_____
> Albatross Consulting Pty. Ltd.   |----/       -  \
> (A.B.N. 75 008 659 498)          |          /(@)   ______---_
> Tel: (+61) 0500 83 82 81         |                 _________  \
> Fax: (+61) 0500 83 82 82         |                 ___________ |
> Http://www.rhyme.com.au          |                /           \|
>                                   |    --________--
> PGP key available upon request,  |  /
> and from pgp5.ai.mit.edu:11371   |/
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 24 Oct 2002 21:56:04 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 09:56 PM 24/10/2002 -0400, Bruce Momjian wrote:
> > > > You are quite correct. It should read:
> > > >
> > > >          #ifdef HAVE_FSEEKO
> > > >               ctx->hasSeek = fseeko(...,SEEK_SET);

                                     ^^^^^^^^^^^^^^^^^^^^^^


> > > >          #else
> > > >               ctx->hasSeek = FALSE;
> > > >          #endif

----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Fri, 25 Oct 2002 12:09:25 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?

OK, finally figured it out. I had used fseek instead of fseeko.

---------------------------------------------------------------------------

Philip Warner wrote:
> At 09:56 PM 24/10/2002 -0400, Bruce Momjian wrote:
> > > > > You are quite correct. It should read:
> > > > >
> > > > >          #ifdef HAVE_FSEEKO
> > > > >               ctx->hasSeek = fseeko(...,SEEK_SET);
> 
>                                      ^^^^^^^^^^^^^^^^^^^^^^
> 
> 
> > > > >          #else
> > > > >               ctx->hasSeek = FALSE;
> > > > >          #endif
> 
> ----------------------------------------------------------------
> Philip Warner                    |     __---_____
> Albatross Consulting Pty. Ltd.   |----/       -  \
> (A.B.N. 75 008 659 498)          |          /(@)   ______---_
> Tel: (+61) 0500 83 82 81         |                 _________  \
> Fax: (+61) 0500 83 82 82         |                 ___________ |
> Http://www.rhyme.com.au          |                /           \|
>                                   |    --________--
> PGP key available upon request,  |  /
> and from pgp5.ai.mit.edu:11371   |/
> 
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 24 Oct 2002 23:47:20 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Hi All,

This relates to the recent discussion about floating point output format.

The discussion was at a point where one parameter would be added to 
specify the number of extra digits used in fp output formatting.
The parameter would have a 0 default value, a maximum of 2 and the 
minimum remained open for discussion.

In a previous message I proposed that for double precision numbers a 
minimum value of -13 would be usefful. For single precision numbers this 
corresponds to a value of -4.

I downloaded the PG sources and added two parameters (as PGC_USERSET):

int extra_float4_digits, default 0, min -4, max 2
int extra_float8_digits, defualt 0, min -13, max 2

Compiled and tested for these functionalities. It is ok.

The afected files are:

src/backend/utils/adt/float.c
src/backend/utils/misc/guc.c
src/bin/psql/tab-complete.c
src/backend/utils/misc/postgresql.conf.sample

I used sources from Debian source package, postgresql_7.2.1-2woody2.

Diff's produced with diff -u are enclosed as attachments.
Can you comment on this (particularly the min values) ?
Also, if we concluded that there is a need of 2 more digits, should'nt 
this be the default ?

Best regards,
Pedro M. Ferreira
-- 
----------------------------------------------------------------------
Pedro Miguel Frazao Fernandes Ferreira
Universidade do Algarve
Faculdade de Ciencias e Tecnologia
Campus de Gambelas
8000-117 Faro
Portugal
Tel./Fax:  (+351) 289 800950 / 289 819403
http://w3.ualg.pt/~pfrazao
Mon, 04 Nov 2002 13:50:21 +0000
"Pedro M. Ferreira" <pfrazao@ualg.pt>
Float output formatting options
"Pedro M. Ferreira" <pfrazao@ualg.pt> writes:
> int extra_float4_digits, default 0, min -4, max 2
> int extra_float8_digits, defualt 0, min -13, max 2

I think a single setting extra_float_digits would be sufficient.

> Also, if we concluded that there is a need of 2 more digits, should'nt 
> this be the default ?

No.  pg_dump would want to bump it up on-the-fly.

			regards, tom lane

Mon, 04 Nov 2002 09:46:43 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Float output formatting options 
Tom Lane wrote:
> "Pedro M. Ferreira" <pfrazao@ualg.pt> writes:
> 
>>int extra_float4_digits, default 0, min -4, max 2
>>int extra_float8_digits, defualt 0, min -13, max 2
> 
> 
> I think a single setting extra_float_digits would be sufficient.

Ok. Assuming,

int extra_float_digits, default 0, min -13, max 2

If extra_float_digits==-13 and we are outputing a float4 this results in 
a negative value for FLT_DIG+extra_float_digits. I dont know if 
sprintf's behaviour is the same across different libraries for this 
situation.

Should I include the following to handle this case ?

if(extra_float_digits<-4)
   sprintf(ascii, "%.*g", FLT_DIG-4, num);
else
   sprintf(ascii, "%.*g", FLT_DIG+extra_float_digits, num);

> 
> 
>>Also, if we concluded that there is a need of 2 more digits, should'nt 
>>this be the default ?
> 
> 
> No.  pg_dump would want to bump it up on-the-fly.
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
> 
> 


-- 
----------------------------------------------------------------------
Pedro Miguel Frazao Fernandes Ferreira
Universidade do Algarve
Faculdade de Ciencias e Tecnologia
Campus de Gambelas
8000-117 Faro
Portugal
Tel./Fax:  (+351) 289 800950 / 289 819403
http://w3.ualg.pt/~pfrazao


Mon, 04 Nov 2002 15:27:10 +0000
"Pedro M. Ferreira" <pfrazao@ualg.pt>
Re: Float output formatting options
"Pedro M. Ferreira" <pfrazao@ualg.pt> writes:
> Tom Lane wrote:
>> I think a single setting extra_float_digits would be sufficient.

> Ok. Assuming,

> int extra_float_digits, default 0, min -13, max 2

> If extra_float_digits==-13 and we are outputing a float4 this results in 
> a negative value for FLT_DIG+extra_float_digits.

You would want to clamp the values passed to %g to not less than 1.
I'd favor code like
	int	ndig = FLT_DIG + extra_float_digits;
	if (ndig < 1)
		ndig = 1;
	sprintf(ascii, "%.*g", ndig, num);

Probably best to do it this way with float8 too; otherwise we're
essentially wiring in the assumption that we know what DBL_DIG is.
Which is exactly what we're trying to avoid doing.

			regards, tom lane

Mon, 04 Nov 2002 10:33:09 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Float output formatting options 
Tom Lane wrote:
> "Pedro M. Ferreira" <pfrazao@ualg.pt> writes:
 >>
>>If extra_float_digits==-13 and we are outputing a float4 this results in 
>>a negative value for FLT_DIG+extra_float_digits.
> 
> You would want to clamp the values passed to %g to not less than 1.
> I'd favor code like
> 	int	ndig = FLT_DIG + extra_float_digits;
> 	if (ndig < 1)
> 		ndig = 1;
> 	sprintf(ascii, "%.*g", ndig, num);
> 
> Probably best to do it this way with float8 too; otherwise we're
> essentially wiring in the assumption that we know what DBL_DIG is.
> Which is exactly what we're trying to avoid doing.

Good.
Corrected this, compiled and tested it. Works fine.

I am attaching the diff's made with diff -u. Sources were from Debian 
source package Version 7.2.1-2woody2.

Best regards,
Pedro

> 
> 			regards, tom lane
> 
> 


-- 
----------------------------------------------------------------------
Pedro Miguel Frazao Fernandes Ferreira
Universidade do Algarve
Faculdade de Ciencias e Tecnologia
Campus de Gambelas
8000-117 Faro
Portugal
Tel./Fax:  (+351) 289 800950 / 289 819403
http://w3.ualg.pt/~pfrazao
Mon, 04 Nov 2002 16:51:10 +0000
"Pedro M. Ferreira" <pfrazao@ualg.pt>
Re: Float output formatting options
"Pedro M. Ferreira" <pfrazao@ualg.pt> writes:
> I am attaching the diff's made with diff -u. Sources were from Debian 
> source package Version 7.2.1-2woody2.

Looks good, will keep to apply after we branch for 7.4 development.

BTW, did you check to see if this affects the geometric types or not?
I am not sure that they go through float8out; they may need similar
adjustments in their output routines.

			regards, tom lane

Mon, 04 Nov 2002 11:56:41 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Float output formatting options 
Tom Lane wrote:

> Looks good, will keep to apply after we branch for 7.4 development.
> 
> BTW, did you check to see if this affects the geometric types or not?
> I am not sure that they go through float8out; they may need similar
> adjustments in their output routines.

Only checked arrays.
I will check this and get you posted about it ASAP.

Best regards,
Pedro

> 
> 			regards, tom lane
> 
> 


-- 
----------------------------------------------------------------------
Pedro Miguel Frazao Fernandes Ferreira
Universidade do Algarve
Faculdade de Ciencias e Tecnologia
Campus de Gambelas
8000-117 Faro
Portugal
Tel./Fax:  (+351) 289 800950 / 289 819403
http://w3.ualg.pt/~pfrazao


Mon, 04 Nov 2002 17:02:46 +0000
"Pedro M. Ferreira" <pfrazao@ualg.pt>
Re: Float output formatting options
Tom Lane wrote:

> BTW, did you check to see if this affects the geometric types or not?
> I am not sure that they go through float8out; they may need similar
> adjustments in their output routines.

In fact they need adjustments.

The *_out routines (in src/backend/utils/adt/geo_ops.c) for the 
geometric types rely on two functions to output data:

static int      pair_encode(float8 x, float8 y, char *str);
static int      single_encode(float8 x, char *str);

These functions produce output with (for pair_encode):

sprintf(str, "%.*g,%.*g", digits8, x, digits8, y);

digits8 is defined as ,

#define P_MAXDIG DBL_DIG
static int      digits8 = P_MAXDIG;

I think it would be done the same way as for float4_out and float8_out:

extern int extra_float_digits;


int 
ndig = digits8 + extra_float_digits;

if (ndig < 1)
	ndig = 1;

sprintf(str, "%.*g,%.*g", ndig, x, ndig, y);

There a bunch of other places where output is produced. They are all 
within #ifdef GEODEBUG / #enfif blocks. Should these be corrected the 
same way ?

Regards,
Pedro


Mon, 04 Nov 2002 18:10:12 +0000
"Pedro M. Ferreira" <pfrazao@ualg.pt>
Re: Float output formatting options
"Pedro M. Ferreira" <pfrazao@ualg.pt> writes:
> These functions produce output with (for pair_encode):

> sprintf(str, "%.*g,%.*g", digits8, x, digits8, y);

> digits8 is defined as ,

> #define P_MAXDIG DBL_DIG
> static int      digits8 = P_MAXDIG;

> I think it would be done the same way as for float4_out and float8_out:

Yeah.  In fact I'd be inclined to remove the static variable and make
the code match float8out exactly (do "DBL_DIG + extra_float_digits").

> There a bunch of other places where output is produced. They are all 
> within #ifdef GEODEBUG / #enfif blocks. Should these be corrected the 
> same way ?

Up to you.  Personally I'd just leave them alone...

			regards, tom lane

Mon, 04 Nov 2002 13:14:48 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Float output formatting options 
Tom Lane wrote:
> 
> Yeah.  In fact I'd be inclined to remove the static variable and make
> the code match float8out exactly (do "DBL_DIG + extra_float_digits").

P_MAXDIG is only used in the lines below:

#define P_MAXDIG DBL_DIG
#define P_MAXLEN (2*(P_MAXDIG+7)+1)
static int      digits8 = P_MAXDIG;

Is it ok to remove #define P_MAXDIG DBL_DIG,
change P_MAXLEN to 2*(DBL_DIG+7)+1) and
remove the line 'static int      digits8 = P_MAXDIG;' ?

Would then change the two geo output functions and replace digits8 by 
DBL_DIG in the #ifdef GEODEBUG / #enfif output stuff.

>>There a bunch of other places where output is produced. They are all 
>>within #ifdef GEODEBUG / #enfif blocks. Should these be corrected the 
>>same way ?


Mon, 04 Nov 2002 18:28:28 +0000
"Pedro M. Ferreira" <pfrazao@ualg.pt>
Re: Float output formatting options
"Pedro M. Ferreira" <pfrazao@ualg.pt> writes:
> Is it ok to remove #define P_MAXDIG DBL_DIG,
> change P_MAXLEN to 2*(DBL_DIG+7)+1) and
> remove the line 'static int      digits8 = P_MAXDIG;' ?

Perhaps P_MAXLEN now needs to be (2*(DBL_DIG+2+7)+1), considering
that we'll allow extra_float_digits to be up to 2.  What's it used for?

			regards, tom lane

Mon, 04 Nov 2002 13:38:54 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Float output formatting options 
Tom Lane wrote:
> "Pedro M. Ferreira" <pfrazao@ualg.pt> writes:
> 
>>Is it ok to remove #define P_MAXDIG DBL_DIG,
>>change P_MAXLEN to 2*(DBL_DIG+7)+1) and
>>remove the line 'static int      digits8 = P_MAXDIG;' ?
> 
> Perhaps P_MAXLEN now needs to be (2*(DBL_DIG+2+7)+1), considering
> that we'll allow extra_float_digits to be up to 2.  What's it used for?

Yes. I guess so, because it is used in what I think is a memory 
allocation function. P_MAXLEN is only used twice:

* 1st use in path_encode() (allmost all the geo *_out functions use 
path_encode):

int        size = npts * (P_MAXLEN + 3) + 2;

/* Check for integer overflow */
if ((size - 2) / npts != (P_MAXLEN + 3))
                 elog(ERROR, "Too many points requested");


* 2nd use in circle_out(PG_FUNCTION_ARGS):

result = palloc(3 * (P_MAXLEN + 1) + 3);

I will do the changes tomorrow and send in the appropriate diff's.

Regards,
Pedro


Mon, 04 Nov 2002 18:51:42 +0000
"Pedro M. Ferreira" <pfrazao@ualg.pt>
Re: Float output formatting options
> > Perhaps P_MAXLEN now needs to be (2*(DBL_DIG+2+7)+1), considering
> > that we'll allow extra_float_digits to be up to 2.  What's it used for?
> 
> Yes. I guess so, because it is used in what I think is a memory 
> allocation function. P_MAXLEN is only used twice:

Curse my slowness, but what's the actual problem being fixed here?

Chris


Tue, 5 Nov 2002 09:43:26 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: Float output formatting options
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes:
> Curse my slowness, but what's the actual problem being fixed here?

Two things:

* allow pg_dump to accurately dump and restore float quantities (setting
float_extra_digits to 2 during the dump will accomplish this, at least
on systems with reasonable float I/O routines).

* allow us to get out from under the geometry regression test's platform
dependency problems (setting float_extra_digits to -2 or so during the
test should make most or all of the variations go away).

This proposal is the first one I've seen that solves both these problems
without introducing any compatibility issues of its own.

			regards, tom lane

Mon, 04 Nov 2002 23:16:40 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Float output formatting options 
Pedro M. Ferreira wrote:
> Tom Lane wrote:
>> Perhaps P_MAXLEN now needs to be (2*(DBL_DIG+2+7)+1), considering
>> that we'll allow extra_float_digits to be up to 2.  What's it used for?
> 
> Yes. I guess so, because it is used in what I think is a memory 
> allocation function. P_MAXLEN is only used twice:
<...>
> 
> I will do the changes tomorrow and send in the appropriate diff's.

Ok. Its done now.
Only one file changed: src/backend/utils/adt/geo_ops.c

All the geometric types should now account for float_extra_digits on output.

A diff -u is attached.

Best reagards,
Pedro

> Regards,
> Pedro
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>    (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> 
> 


-- 
----------------------------------------------------------------------
Pedro Miguel Frazao Fernandes Ferreira
Universidade do Algarve
Faculdade de Ciencias e Tecnologia
Campus de Gambelas
8000-117 Faro
Portugal
Tel./Fax:  (+351) 289 800950 / 289 819403
http://w3.ualg.pt/~pfrazao
Tue, 05 Nov 2002 11:11:25 +0000
"Pedro M. Ferreira" <pfrazao@ualg.pt>
Re: Float output formatting options
I am including a set of 4 small patches that enable PostgreSQL 7.3b3 to build
successfully on OpenUnix 8.0.  These same patches should also work for UnixWare
7.x.  I will confirm that tomorrow (Nov 7, 2002).

Here is an explanation of the patches:

1. An update of the FAQ_SCO file.

2. This patch removes a static declaration of a in-line function in
   src/backend/utils/sort/tuplesort.c 

3. This patch to src/makefiles/Makefile.unixware, together with the patch to
   src/Makefile.global.in allows any addition library search directories (added
   with the configure --with-libraries option) to be added to the rpath option 
   sent to the linker.  The use of a different variable to pass the addition 
   search paths was necessary to avoid a circular reference to LDFLAGS.

4. This patch creates the variable (trpath) used by the patch to 
   Makefile.unixware.  This patch would also be for other platforms that would 
   have to add the additional library search paths to  the rpath linker option.
   See Makefile.unixware for an example of how to do this.

After applying these patches, PostgreSQL successfully compiled on OpenUnix 8 
and it passed all the regression tests.



____       | Billy G. Allie    | Domain....: Bill.Allie@mug.org
|  /|      | 7436 Hartwell     | MSN.......: B_G_Allie@email.msn.com
|-/-|----- | Dearborn, MI 48126|
|/  |LLIE  | (313) 582-1540    |
Wed, 06 Nov 2002 22:57:26 -0500
"Billy G. Allie" <Bill.Allie@mug.org>
PostgreSQL supported platform report and a patch.

I am fine with this because it only touches unixware-specific stuff,
except the change to Tom's inline function:

  [static] inline Datum
  myFunctionCall2(FmgrInfo *flinfo, Datum arg1, Datum arg2)

Tom will have to comment on that.

---------------------------------------------------------------------------

Billy G. Allie wrote:
-- Start of PGP signed section.
> I am including a set of 4 small patches that enable PostgreSQL 7.3b3 to build
> successfully on OpenUnix 8.0.  These same patches should also work for UnixWare
> 7.x.  I will confirm that tomorrow (Nov 7, 2002).
> 
> Here is an explanation of the patches:
> 
> 1. An update of the FAQ_SCO file.
> 
> 2. This patch removes a static declaration of a in-line function in
>    src/backend/utils/sort/tuplesort.c 
> 
> 3. This patch to src/makefiles/Makefile.unixware, together with the patch to
>    src/Makefile.global.in allows any addition library search directories (added
>    with the configure --with-libraries option) to be added to the rpath option 
>    sent to the linker.  The use of a different variable to pass the addition 
>    search paths was necessary to avoid a circular reference to LDFLAGS.
> 
> 4. This patch creates the variable (trpath) used by the patch to 
>    Makefile.unixware.  This patch would also be for other platforms that would 
>    have to add the additional library search paths to  the rpath linker option.
>    See Makefile.unixware for an example of how to do this.
> 
> After applying these patches, PostgreSQL successfully compiled on OpenUnix 8 
> and it passed all the regression tests.
> 

Content-Description: ou8.patch.20021106

[ Attachment, skipping... ]

> ____       | Billy G. Allie    | Domain....: Bill.Allie@mug.org
> |  /|      | 7436 Hartwell     | MSN.......: B_G_Allie@email.msn.com
> |-/-|----- | Dearborn, MI 48126|
> |/  |LLIE  | (313) 582-1540    |
-- End of PGP section, PGP failed!

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 6 Nov 2002 23:03:18 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a patch.
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I am fine with this because it only touches unixware-specific stuff,
> except the change to Tom's inline function:
>   [static] inline Datum
>   myFunctionCall2(FmgrInfo *flinfo, Datum arg1, Datum arg2)
> Tom will have to comment on that.

That change would actively break some platforms (see C99 inline
specifications).  Why is it necessary for SCO?  We certainly have
plenty of other static inline functions ...

			regards, tom lane

Wed, 06 Nov 2002 23:38:28 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a patch. 
We already have success messages from Olivier Prenant for 7.3B4 on 8.0.0, 
and me for 7.1.3.

I don't believe your changes are necessary.



--On Wednesday, November 06, 2002 22:57:26 -0500 "Billy G. Allie" 
<Bill.Allie@mug.org> wrote:

> I am including a set of 4 small patches that enable PostgreSQL 7.3b3 to
> build successfully on OpenUnix 8.0.  These same patches should also work
> for UnixWare 7.x.  I will confirm that tomorrow (Nov 7, 2002).
>
> Here is an explanation of the patches:
>
> 1. An update of the FAQ_SCO file.
>
> 2. This patch removes a static declaration of a in-line function in
>    src/backend/utils/sort/tuplesort.c
>
> 3. This patch to src/makefiles/Makefile.unixware, together with the patch
> to    src/Makefile.global.in allows any addition library search
> directories (added    with the configure --with-libraries option) to be
> added to the rpath option     sent to the linker.  The use of a different
> variable to pass the addition     search paths was necessary to avoid a
> circular reference to LDFLAGS.
>
> 4. This patch creates the variable (trpath) used by the patch to
>    Makefile.unixware.  This patch would also be for other platforms that
> would     have to add the additional library search paths to  the rpath
> linker option.    See Makefile.unixware for an example of how to do this.
>
> After applying these patches, PostgreSQL successfully compiled on
> OpenUnix 8  and it passed all the regression tests.
>



-- 
Larry Rosenman, Sr. Network Engineer, Internet America, Inc.
E-Mail: ler@airmail.net
Phone: +1 214-861-2571, Fax: 214-861-2663
US Mail: 350 N. St. Paul, Suite 3000, Dallas, TX 75201

Wed, 06 Nov 2002 23:27:31 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a patch.
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I am fine with this because it only touches unixware-specific stuff,
> > except the change to Tom's inline function:
> >   [static] inline Datum
> >   myFunctionCall2(FmgrInfo *flinfo, Datum arg1, Datum arg2)
> > Tom will have to comment on that.
> 
> That change would actively break some platforms (see C99 inline
> specifications).  Why is it necessary for SCO?  We certainly have
> plenty of other static inline functions ...
> 
> 			regards, tom lane

Here is the error messages generated during the compile:

cc -K pentium_pro,host,inline,loop_unroll -I../../../../src/include 
-I/usr/local/include -I/usr/local/ssl/include  -c -o tuplesort.o tuplesort.c
UX:acomp: ERROR: "tuplesort.c", line 1854: "inline" functions cannot use 
"static" identifier: myFunctionCall2
UX:acomp: ERROR: "tuplesort.c", line 1856: "inline" functions cannot use 
"static" identifier: myFunctionCall2
UX:acomp: ERROR: "tuplesort.c", line 1870: "inline" functions cannot use 
"static" identifier: myFunctionCall2
UX:acomp: ERROR: "tuplesort.c", line 1872: "inline" functions cannot use 
"static" identifier: myFunctionCall2
UX:acomp: ERROR: "tuplesort.c", line 1885: "inline" functions cannot use 
"static" identifier: myFunctionCall2
UX:acomp: ERROR: "tuplesort.c", line 1897: "inline" functions cannot use 
"static" identifier: myFunctionCall2
gmake[4]: *** [tuplesort.o] Error 1

The problem only occurs in tuplesort.c.  It does not occur in pg_lzcompress.c 
or aset.c, which are the only other source files that contain static inline 
function definitions that get compiled.  The rest are IF DEFed out.

I think the problem is that myFunctionCall2 is called by a non-static inline 
function, ApplySortFunction.  If I make ApplySortFunction static, it compiles 
(but break the link phase).  If I remove the inline from ApplySortFunction, it 
compiles and builds.  In order for tuplesort.c to compile on OpenUNIX the code 
must be changed to either:

1.  Remove the static modifier from myFuntionCall2
    or
2.  Remove the inline from ApplySortFunction
    or
3.  Wrap the static modifier for myFunctionCall2 with an IF DEF so it's not
    there when USE_UNIVEL_CC is defined.

I think that option 2 is the best choice, but it's your call.
-- 
____       | Billy G. Allie    | Domain....: Bill.Allie@mug.org
|  /|      | 7436 Hartwell     | MSN.......: B_G_Allie@email.msn.com
|-/-|----- | Dearborn, MI 48126|
|/  |LLIE  | (313) 582-1540    |
Thu, 07 Nov 2002 01:11:13 -0500
"Billy G. Allie" <bga@mug.org>
Re: [HACKERS] PostgreSQL supported platform report and a patch. 
"Billy G. Allie" <bga@mug.org> writes:
> Here is the error messages generated during the compile:

> cc -K pentium_pro,host,inline,loop_unroll -I../../../../src/include 
> -I/usr/local/include -I/usr/local/ssl/include  -c -o tuplesort.o tuplesort.c
> UX:acomp: ERROR: "tuplesort.c", line 1854: "inline" functions cannot use 
> "static" identifier: myFunctionCall2

Uh, what version are you testing exactly?  I thought we'd resolved that
as of a week or so back (certainly in 7.3b4).

			regards, tom lane

Thu, 07 Nov 2002 01:16:08 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a patch. 
Larry Rosenman wrote:
> We already have success messages from Olivier Prenant for 7.3B4 on 8.0.0, 
> and me for 7.1.3.
> 
> I don't believe your changes are necessary.
> 

Was that using gcc or the native compiler?
Was it in linux kernel personality mode or OpenUNIX mode.

I was compiling using the native (UDK) compiler. and it failed in tuplesort.c.
It was also unable to file the readline shared libraries without the changes to the makefiles or setting LD_RUN_PATH (which is a pain and is depreciated in OpenUNIX 8 and UnixWare 7).

-- 
____       | Billy G. Allie    | Domain....: Bill.Allie@mug.org
|  /|      | 7436 Hartwell     | MSN.......: B_G_Allie@email.msn.com
|-/-|----- | Dearborn, MI 48126|
|/  |LLIE  | (313) 582-1540    |
Thu, 07 Nov 2002 01:17:55 -0500
"Billy G. Allie" <bga@mug.org>
Re: [HACKERS] PostgreSQL supported platform report and a patch. 
Tom Lane wrote:
> "Billy G. Allie" <bga@mug.org> writes:
> > Here is the error messages generated during the compile:
> 
> > cc -K pentium_pro,host,inline,loop_unroll -I../../../../src/include 
> > -I/usr/local/include -I/usr/local/ssl/include  -c -o tuplesort.o tuplesort.
> c
> > UX:acomp: ERROR: "tuplesort.c", line 1854: "inline" functions cannot use 
> > "static" identifier: myFunctionCall2
> 
> Uh, what version are you testing exactly?  I thought we'd resolved that
> as of a week or so back (certainly in 7.3b4).

It was 7.3b3.  I've just downloaded 7.3b5 and will re-test.
If that's the case then ignore the patch to tuplesort.c  The rest of the 
patches are still valid though.
-- 
____       | Billy G. Allie    | Domain....: Bill.Allie@mug.org
|  /|      | 7436 Hartwell     | MSN.......: B_G_Allie@email.msn.com
|-/-|----- | Dearborn, MI 48126|
|/  |LLIE  | (313) 582-1540    |
Thu, 07 Nov 2002 01:22:05 -0500
"Billy G. Allie" <bga@mug.org>
Re: [HACKERS] PostgreSQL supported platform report and a patch. 
Larry Rosenman <ler@lerctr.org> writes:
> I don't believe your changes are necessary.

The static-inline change was obsoleted by a recent fix, per discussion.
But the rpath changes seem possibly useful (or maybe my thoughts are
just colored by the fact that I'm currently trying to persuade OpenSSL
to build with a non-broken rpath setup on HPUX...)  Do you have an
objection to the rpath part of Billy's patch?

			regards, tom lane

Thu, 07 Nov 2002 02:42:47 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a patch. 
That's true!

But I had to export CFLAGS=-Xb to compile (this should be in port Makefile
IMHO)
Also, I think the Setting of LD_LIBRARY_PATH could be a win to. Although I
doubt anyone would run uw whith at least
LD_LIBRARY_PATH=/lib:/usr/local/lib, setting LD_LIBRARY_PATH and includes
in the port makefile could ease the configure  process as readline is not
found if you don't add --with-includes ans --with-libs on configure
command.

Reagrds
 On Wed, 6 Nov 2002, Larry Rosenman wrote:

> Date: Wed, 06 Nov 2002 23:27:31 -0600
> From: Larry Rosenman <ler@lerctr.org>
> To: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a
>     patch.
> 
> We already have success messages from Olivier Prenant for 7.3B4 on 8.0.0, 
> and me for 7.1.3.
> 
> I don't believe your changes are necessary.
> 
> 
> 
> --On Wednesday, November 06, 2002 22:57:26 -0500 "Billy G. Allie" 
> <Bill.Allie@mug.org> wrote:
> 
> > I am including a set of 4 small patches that enable PostgreSQL 7.3b3 to
> > build successfully on OpenUnix 8.0.  These same patches should also work
> > for UnixWare 7.x.  I will confirm that tomorrow (Nov 7, 2002).
> >
> > Here is an explanation of the patches:
> >
> > 1. An update of the FAQ_SCO file.
> >
> > 2. This patch removes a static declaration of a in-line function in
> >    src/backend/utils/sort/tuplesort.c
> >
> > 3. This patch to src/makefiles/Makefile.unixware, together with the patch
> > to    src/Makefile.global.in allows any addition library search
> > directories (added    with the configure --with-libraries option) to be
> > added to the rpath option     sent to the linker.  The use of a different
> > variable to pass the addition     search paths was necessary to avoid a
> > circular reference to LDFLAGS.
> >
> > 4. This patch creates the variable (trpath) used by the patch to
> >    Makefile.unixware.  This patch would also be for other platforms that
> > would     have to add the additional library search paths to  the rpath
> > linker option.    See Makefile.unixware for an example of how to do this.
> >
> > After applying these patches, PostgreSQL successfully compiled on
> > OpenUnix 8  and it passed all the regression tests.
> >
> 
> 
> 
> 

-- 
Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
FRANCE                      Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)


Thu, 7 Nov 2002 13:32:18 +0100 (MET)
Olivier PRENANT <ohp@pyrenet.fr>
Re: [HACKERS] PostgreSQL supported platform report and a


--On Thursday, November 07, 2002 01:17:55 -0500 "Billy G. Allie" 
<bga@mug.org> wrote:

> Larry Rosenman wrote:
>> We already have success messages from Olivier Prenant for 7.3B4 on
>> 8.0.0,  and me for 7.1.3.
>>
>> I don't believe your changes are necessary.
>>
>
> Was that using gcc or the native compiler?
Native.
> Was it in linux kernel personality mode or OpenUNIX mode.
>
Native.
> I was compiling using the native (UDK) compiler. and it failed in
> tuplesort.c. It was also unable to file the readline shared libraries
> without the changes to the makefiles or setting LD_RUN_PATH (which is a
> pain and is depreciated in OpenUNIX 8 and UnixWare 7).
Tom fixed the tuplesort.c issue with some help from the Caldera/SCO 
Compiler team
(I'm on the 7.1.3 BETA).


My system has always found the readline stuff (I use the skunkware 
readline).

It hasn't been an issue on my system.


>
> --
> ____       | Billy G. Allie    | Domain....: Bill.Allie@mug.org
>|  /|      | 7436 Hartwell     | MSN.......: B_G_Allie@email.msn.com
>| -/-|----- | Dearborn, MI 48126|
>| /  |LLIE  | (313) 582-1540    |
>
>



-- 
Larry Rosenman, Sr. Network Engineer, Internet America, Inc.
E-Mail: ler@airmail.net
Phone: +1 214-861-2571, Fax: 214-861-2663
US Mail: 350 N. St. Paul, Suite 3000, Dallas, TX 75201

Thu, 07 Nov 2002 06:34:18 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a patch. 


--On Thursday, November 07, 2002 02:42:47 -0500 Tom Lane 
<tgl@sss.pgh.pa.us> wrote:

> Larry Rosenman <ler@lerctr.org> writes:
>> I don't believe your changes are necessary.
>
> The static-inline change was obsoleted by a recent fix, per discussion.
> But the rpath changes seem possibly useful (or maybe my thoughts are
> just colored by the fact that I'm currently trying to persuade OpenSSL
> to build with a non-broken rpath setup on HPUX...)  Do you have an
> objection to the rpath part of Billy's patch?
Not necessarily.  I was just concerned about the tuplesort one, and the fact
that mine builds and passes without the changes.


>
> 			regards, tom lane



-- 
Larry Rosenman, Sr. Network Engineer, Internet America, Inc.
E-Mail: ler@airmail.net
Phone: +1 214-861-2571, Fax: 214-861-2663
US Mail: 350 N. St. Paul, Suite 3000, Dallas, TX 75201

Thu, 07 Nov 2002 06:35:31 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a patch. 


--On Thursday, November 07, 2002 13:32:18 +0100 Olivier PRENANT 
<ohp@pyrenet.fr> wrote:

> That's true!
>
> But I had to export CFLAGS=-Xb to compile (this should be in port Makefile
> IMHO)
Tom fixed that with a later tuplesort.c fix (per a discussion with the 
Caldera/SCO
compiler guys).

> Also, I think the Setting of LD_LIBRARY_PATH could be a win to. Although I
> doubt anyone would run uw whith at least
> LD_LIBRARY_PATH=/lib:/usr/local/lib, setting LD_LIBRARY_PATH and includes
> in the port makefile could ease the configure  process as readline is not
> found if you don't add --with-includes ans --with-libs on configure
> command.
Not a problem here. (the change that is).


>
> Reagrds
>  On Wed, 6 Nov 2002, Larry Rosenman wrote:
>
>> Date: Wed, 06 Nov 2002 23:27:31 -0600
>> From: Larry Rosenman <ler@lerctr.org>
>> To: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org
>> Cc: pgsql-hackers@postgresql.org
>> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a
>>     patch.
>>
>> We already have success messages from Olivier Prenant for 7.3B4 on
>> 8.0.0,  and me for 7.1.3.
>>
>> I don't believe your changes are necessary.
>>
>>
>>
>> --On Wednesday, November 06, 2002 22:57:26 -0500 "Billy G. Allie"
>> <Bill.Allie@mug.org> wrote:
>>
>> > I am including a set of 4 small patches that enable PostgreSQL 7.3b3 to
>> > build successfully on OpenUnix 8.0.  These same patches should also
>> > work for UnixWare 7.x.  I will confirm that tomorrow (Nov 7, 2002).
>> >
>> > Here is an explanation of the patches:
>> >
>> > 1. An update of the FAQ_SCO file.
>> >
>> > 2. This patch removes a static declaration of a in-line function in
>> >    src/backend/utils/sort/tuplesort.c
>> >
>> > 3. This patch to src/makefiles/Makefile.unixware, together with the
>> > patch to    src/Makefile.global.in allows any addition library search
>> > directories (added    with the configure --with-libraries option) to be
>> > added to the rpath option     sent to the linker.  The use of a
>> > different variable to pass the addition     search paths was necessary
>> > to avoid a circular reference to LDFLAGS.
>> >
>> > 4. This patch creates the variable (trpath) used by the patch to
>> >    Makefile.unixware.  This patch would also be for other platforms
>> >    that would     have to add the additional library search paths to
>> > the rpath linker option.    See Makefile.unixware for an example of
>> > how to do this.
>> >
>> > After applying these patches, PostgreSQL successfully compiled on
>> > OpenUnix 8  and it passed all the regression tests.
>> >
>>
>>
>>
>>
>
> --
> Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> FRANCE                      Email: ohp@pyrenet.fr
> -------------------------------------------------------------------------
> ----- Make your life a dream, make your dream a reality. (St Exupery)



-- 
Larry Rosenman, Sr. Network Engineer, Internet America, Inc.
E-Mail: ler@airmail.net
Phone: +1 214-861-2571, Fax: 214-861-2663
US Mail: 350 N. St. Paul, Suite 3000, Dallas, TX 75201

Thu, 07 Nov 2002 06:41:02 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a
On Thu, 7 Nov 2002, Larry Rosenman wrote:

> Date: Thu, 07 Nov 2002 06:41:02 -0600
> From: Larry Rosenman <ler@lerctr.org>
> To: ohp@pyrenet.fr
> Cc: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>      pgsql-hackers@postgresql.org
> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a
> 
> 
> 
> --On Thursday, November 07, 2002 13:32:18 +0100 Olivier PRENANT 
> <ohp@pyrenet.fr> wrote:
> 
> > That's true!
> >
> > But I had to export CFLAGS=-Xb to compile (this should be in port Makefile
> > IMHO)
> Tom fixed that with a later tuplesort.c fix (per a discussion with the 
> Caldera/SCO
> compiler guys).
Huh! I just tried to compile 7.3b5 without CFLAGS=-Xb, it still bugs the
compiler...
> 

-- 
Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
FRANCE                      Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)


Thu, 7 Nov 2002 14:23:43 +0100 (MET)
Olivier PRENANT <ohp@pyrenet.fr>
Re: [HACKERS] PostgreSQL supported platform report and a


--On Thursday, November 07, 2002 14:23:43 +0100 Olivier PRENANT 
<ohp@pyrenet.fr> wrote:

> On Thu, 7 Nov 2002, Larry Rosenman wrote:
>
>> Date: Thu, 07 Nov 2002 06:41:02 -0600
>> From: Larry Rosenman <ler@lerctr.org>
>> To: ohp@pyrenet.fr
>> Cc: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>>      pgsql-hackers@postgresql.org
>> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a
>>
>>
>>
>> --On Thursday, November 07, 2002 13:32:18 +0100 Olivier PRENANT
>> <ohp@pyrenet.fr> wrote:
>>
>> > That's true!
>> >
>> > But I had to export CFLAGS=-Xb to compile (this should be in port
>> > Makefile IMHO)
>> Tom fixed that with a later tuplesort.c fix (per a discussion with the
>> Caldera/SCO
>> compiler guys).
> Huh! I just tried to compile 7.3b5 without CFLAGS=-Xb, it still bugs the
> compiler...
>>
Didn't for me.... :-(

Wierd.


>
> --
> Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> FRANCE                      Email: ohp@pyrenet.fr
> -------------------------------------------------------------------------
> ----- Make your life a dream, make your dream a reality. (St Exupery)



-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Thu, 07 Nov 2002 08:41:58 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a
On Thu, 7 Nov 2002, Larry Rosenman wrote:

> Date: Thu, 07 Nov 2002 08:41:58 -0600
> From: Larry Rosenman <ler@lerctr.org>
> To: ohp@pyrenet.fr
> Cc: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>      pgsql-hackers@postgresql.org
> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a
> 
> 
> 
> >> Tom fixed that with a later tuplesort.c fix (per a discussion with the
> >> Caldera/SCO
> >> compiler guys).
> > Huh! I just tried to compile 7.3b5 without CFLAGS=-Xb, it still bugs the
> > compiler...
> >>
> Didn't for me.... :-(
> 
> Wierd.
BTW, this is on 7.1.1 not (yet) on 8.0.0
I'll let you know hopefully today.

(How did you get 713 when it's due for december?) Can I have a copy?
> 
> 
> >
> > --
> > Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> > Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> > 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> > FRANCE                      Email: ohp@pyrenet.fr
> > -------------------------------------------------------------------------
> > ----- Make your life a dream, make your dream a reality. (St Exupery)
> 
> 
> 
> 

-- 
Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
FRANCE                      Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)


Thu, 7 Nov 2002 15:44:37 +0100 (MET)
Olivier PRENANT <ohp@pyrenet.fr>
Re: [HACKERS] PostgreSQL supported platform report and a


--On Thursday, November 07, 2002 15:44:37 +0100 Olivier PRENANT 
<ohp@pyrenet.fr> wrote:

> On Thu, 7 Nov 2002, Larry Rosenman wrote:
>
>> Date: Thu, 07 Nov 2002 08:41:58 -0600
>> From: Larry Rosenman <ler@lerctr.org>
>> To: ohp@pyrenet.fr
>> Cc: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>>      pgsql-hackers@postgresql.org
>> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a
>>
>>
>>
>> >> Tom fixed that with a later tuplesort.c fix (per a discussion with the
>> >> Caldera/SCO
>> >> compiler guys).
>> > Huh! I just tried to compile 7.3b5 without CFLAGS=-Xb, it still bugs
>> > the compiler...
>> >>
>> Didn't for me.... :-(
>>
>> Wierd.
> BTW, this is on 7.1.1 not (yet) on 8.0.0
> I'll let you know hopefully today.
>
> (How did you get 713 when it's due for december?) Can I have a copy?
I'm on the Beta.  No, I can't give it to you.  You might want to sign up
on http://www.caldera.com/beta/ to get in on the next one.


>>
>>
>> >
>> > --
>> > Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
>> > Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
>> > 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
>> > FRANCE                      Email: ohp@pyrenet.fr
>> > ----------------------------------------------------------------------
>> > --- ----- Make your life a dream, make your dream a reality. (St
>> > Exupery)
>>
>>
>>
>>
>
> --
> Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> FRANCE                      Email: ohp@pyrenet.fr
> -------------------------------------------------------------------------
> ----- Make your life a dream, make your dream a reality. (St Exupery)


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Thu, 07 Nov 2002 08:47:33 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a
Olivier PRENANT <ohp@pyrenet.fr> writes:
> Huh! I just tried to compile 7.3b5 without CFLAGS=-Xb, it still bugs the
> compiler...

It won't get better if you don't show any details...

			regards, tom lane

Thu, 07 Nov 2002 10:21:25 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a 
On Thu, 7 Nov 2002, Tom Lane wrote:

> Date: Thu, 07 Nov 2002 10:21:25 -0500
> From: Tom Lane <tgl@sss.pgh.pa.us>
> To: ohp@pyrenet.fr
> Cc: Larry Rosenman <ler@lerctr.org>, Billy G. Allie <Bill.Allie@mug.org>,
>      pgsql-ports@postgresql.org, pgsql-hackers@postgresql.org
> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a 
> 
> Olivier PRENANT <ohp@pyrenet.fr> writes:
> > Huh! I just tried to compile 7.3b5 without CFLAGS=-Xb, it still bugs the
> > compiler...
> 
> It won't get better if you don't show any details...
Ok... (sorry) this is on UW 711 WITHOUT CFLAGS=-Xb:
Script started on Thu Nov  7 16:57:05 2002
$ cd postgresql*5
$ make
Using GNU make found at /usr/local/bin/gmake
/usr/local/bin/gmake -C doc all
gmake[1]: Entering directory `/home/postgres/postgresql-7.3b5/doc'
gmake[1]: Nothing to be done for `all'.
gmake[1]: Leaving directory `/home/postgres/postgresql-7.3b5/doc'
/usr/local/bin/gmake -C src all
gmake[1]: Entering directory `/home/postgres/postgresql-7.3b5/src'
/usr/local/bin/gmake -C port all
gmake[2]: Entering directory `/home/postgres/postgresql-7.3b5/src/port'
gmake[2]: Nothing to be done for `all'.
gmake[2]: Leaving directory `/home/postgres/postgresql-7.3b5/src/port'
/usr/local/bin/gmake -C backend all
gmake[2]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend'
/usr/local/bin/gmake -C ../../src/port all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/port'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/port'
/usr/local/bin/gmake -C access all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/access'
/usr/local/bin/gmake -C common SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/access/common'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/access/common'
/usr/local/bin/gmake -C gist SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/access/gist'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/access/gist'
/usr/local/bin/gmake -C hash SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/access/hash'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/access/hash'
/usr/local/bin/gmake -C heap SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/access/heap'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/access/heap'
/usr/local/bin/gmake -C index SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/access/index'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/access/index'
/usr/local/bin/gmake -C nbtree SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/access/nbtree'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/access/nbtree'
/usr/local/bin/gmake -C rtree SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/access/rtree'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/access/rtree'
/usr/local/bin/gmake -C transam SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/access/transam'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/access/transam'
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/access'
/usr/local/bin/gmake -C bootstrap all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/bootstrap'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/bootstrap'
/usr/local/bin/gmake -C catalog all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/catalog'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/catalog'
/usr/local/bin/gmake -C parser all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/parser'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/parser'
/usr/local/bin/gmake -C commands all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/commands'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/commands'
/usr/local/bin/gmake -C executor all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/executor'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/executor'
/usr/local/bin/gmake -C lib all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/lib'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/lib'
/usr/local/bin/gmake -C libpq all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/libpq'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/libpq'
/usr/local/bin/gmake -C main all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/main'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/main'
/usr/local/bin/gmake -C nodes all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/nodes'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/nodes'
/usr/local/bin/gmake -C optimizer all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer'
/usr/local/bin/gmake -C geqo SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer/geqo'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer/geqo'
/usr/local/bin/gmake -C path SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer/path'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer/path'
/usr/local/bin/gmake -C plan SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer/plan'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer/plan'
/usr/local/bin/gmake -C prep SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer/prep'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer/prep'
/usr/local/bin/gmake -C util SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer/util'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer/util'
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer'
/usr/local/bin/gmake -C port all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/port'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/port'
/usr/local/bin/gmake -C postmaster all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/postmaster'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/postmaster'
/usr/local/bin/gmake -C regex all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/regex'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/regex'
/usr/local/bin/gmake -C rewrite all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/rewrite'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/rewrite'
/usr/local/bin/gmake -C storage all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/storage'
/usr/local/bin/gmake -C buffer SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/storage/buffer'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/storage/buffer'
/usr/local/bin/gmake -C file SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/storage/file'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/storage/file'
/usr/local/bin/gmake -C freespace SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/storage/freespace'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/storage/freespace'
/usr/local/bin/gmake -C ipc SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/storage/ipc'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/storage/ipc'
/usr/local/bin/gmake -C large_object SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/storage/large_object'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/storage/large_object'
/usr/local/bin/gmake -C lmgr SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/storage/lmgr'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/storage/lmgr'
/usr/local/bin/gmake -C page SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/storage/page'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/storage/page'
/usr/local/bin/gmake -C smgr SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/storage/smgr'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/storage/smgr'
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/storage'
/usr/local/bin/gmake -C tcop all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/tcop'
gmake[3]: Nothing to be done for `all'.
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/tcop'
/usr/local/bin/gmake -C utils all
gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/utils'
/usr/local/bin/gmake -C adt SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/utils/adt'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/utils/adt'
/usr/local/bin/gmake -C cache SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/utils/cache'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/utils/cache'
/usr/local/bin/gmake -C error SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/utils/error'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/utils/error'
/usr/local/bin/gmake -C fmgr SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/utils/fmgr'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/utils/fmgr'
/usr/local/bin/gmake -C hash SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/utils/hash'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/utils/hash'
/usr/local/bin/gmake -C init SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/utils/init'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/utils/init'
/usr/local/bin/gmake -C misc SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/utils/misc'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/utils/misc'
/usr/local/bin/gmake -C mmgr SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/utils/mmgr'
gmake[4]: `SUBSYS.o' is up to date.
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/utils/mmgr'
/usr/local/bin/gmake -C sort SUBSYS.o
gmake[4]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend/utils/sort'
cc -O -K inline -I../../../../src/include -I/usr/local/include  -c tuplesort.c -o tuplesort.o
UX:acomp: ERREUR: "tuplesort.c", ligne 1854: "inline" functions cannot use "static" identifier: myFunctionCall2
UX:acomp: ERREUR: "tuplesort.c", ligne 1856: "inline" functions cannot use "static" identifier: myFunctionCall2
UX:acomp: ERREUR: "tuplesort.c", ligne 1870: "inline" functions cannot use "static" identifier: myFunctionCall2
UX:acomp: ERREUR: "tuplesort.c", ligne 1872: "inline" functions cannot use "static" identifier: myFunctionCall2
UX:acomp: ERREUR: "tuplesort.c", ligne 1885: "inline" functions cannot use "static" identifier: myFunctionCall2
UX:acomp: ERREUR: "tuplesort.c", ligne 1897: "inline" functions cannot use "static" identifier: myFunctionCall2
gmake[4]: *** [tuplesort.o] Error 1
gmake[4]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/utils/sort'
gmake[3]: *** [sort-recursive] Error 2
gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/utils'
gmake[2]: *** [utils-recursive] Error 2
gmake[2]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend'
gmake[1]: *** [all] Error 2
gmake[1]: Leaving directory `/home/postgres/postgresql-7.3b5/src'
gmake: *** [all] Error 2
*** Code d'erreur 2 (bu21)
UX:make: ERREUR: erreur irrï¿½mï¿½diable.

script done on Thu Nov  7 16:57:29 2002
It works OK with -Xb...

Regards
> 
> 			regards, tom lane
> 

-- 
Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
FRANCE                      Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)


Thu, 7 Nov 2002 17:00:21 +0100 (MET)
Olivier PRENANT <ohp@pyrenet.fr>
Re: [HACKERS] PostgreSQL supported platform report and a 
It looks like you do **NOT** have B4 or B5....

LER


--On Thursday, November 07, 2002 17:00:21 +0100 Olivier PRENANT 
<ohp@pyrenet.fr> wrote:

> On Thu, 7 Nov 2002, Tom Lane wrote:
>
>> Date: Thu, 07 Nov 2002 10:21:25 -0500
>> From: Tom Lane <tgl@sss.pgh.pa.us>
>> To: ohp@pyrenet.fr
>> Cc: Larry Rosenman <ler@lerctr.org>, Billy G. Allie <Bill.Allie@mug.org>,
>>      pgsql-ports@postgresql.org, pgsql-hackers@postgresql.org
>> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and
>> a
>>
>> Olivier PRENANT <ohp@pyrenet.fr> writes:
>> > Huh! I just tried to compile 7.3b5 without CFLAGS=-Xb, it still bugs
>> > the compiler...
>>
>> It won't get better if you don't show any details...
> Ok... (sorry) this is on UW 711 WITHOUT CFLAGS=-Xb:
> Script started on Thu Nov  7 16:57:05 2002
> $ cd postgresql*5
> $ make
> Using GNU make found at /usr/local/bin/gmake
> /usr/local/bin/gmake -C doc all
> gmake[1]: Entering directory `/home/postgres/postgresql-7.3b5/doc'
> gmake[1]: Nothing to be done for `all'.
> gmake[1]: Leaving directory `/home/postgres/postgresql-7.3b5/doc'
> /usr/local/bin/gmake -C src all
> gmake[1]: Entering directory `/home/postgres/postgresql-7.3b5/src'
> /usr/local/bin/gmake -C port all
> gmake[2]: Entering directory `/home/postgres/postgresql-7.3b5/src/port'
> gmake[2]: Nothing to be done for `all'.
> gmake[2]: Leaving directory `/home/postgres/postgresql-7.3b5/src/port'
> /usr/local/bin/gmake -C backend all
> gmake[2]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend'
> /usr/local/bin/gmake -C ../../src/port all
> gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/port'
> gmake[3]: Nothing to be done for `all'.
> gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/port'
> /usr/local/bin/gmake -C access all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/access' /usr/local/bin/gmake
> -C common SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/common' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/common'
> /usr/local/bin/gmake -C gist SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/gist' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/gist'
> /usr/local/bin/gmake -C hash SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/hash' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/hash'
> /usr/local/bin/gmake -C heap SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/heap' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/heap'
> /usr/local/bin/gmake -C index SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/index' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/index'
> /usr/local/bin/gmake -C nbtree SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/nbtree' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/nbtree'
> /usr/local/bin/gmake -C rtree SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/rtree' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/rtree'
> /usr/local/bin/gmake -C transam SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/transam' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/access/transam' gmake[3]:
> Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/access'
> /usr/local/bin/gmake -C bootstrap all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/bootstrap' gmake[3]: Nothing
> to be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/bootstrap'
> /usr/local/bin/gmake -C catalog all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/catalog' gmake[3]: Nothing
> to be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/catalog'
> /usr/local/bin/gmake -C parser all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/parser' gmake[3]: Nothing to
> be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/parser' /usr/local/bin/gmake
> -C commands all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/commands' gmake[3]: Nothing
> to be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/commands'
> /usr/local/bin/gmake -C executor all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/executor' gmake[3]: Nothing
> to be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/executor'
> /usr/local/bin/gmake -C lib all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/lib' gmake[3]: Nothing to be
> done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/lib' /usr/local/bin/gmake -C
> libpq all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/libpq' gmake[3]: Nothing to
> be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/libpq' /usr/local/bin/gmake
> -C main all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/main' gmake[3]: Nothing to
> be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/main' /usr/local/bin/gmake
> -C nodes all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/nodes' gmake[3]: Nothing to
> be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/nodes' /usr/local/bin/gmake
> -C optimizer all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/optimizer'
> /usr/local/bin/gmake -C geqo SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/optimizer/geqo' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/optimizer/geqo'
> /usr/local/bin/gmake -C path SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/optimizer/path' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/optimizer/path'
> /usr/local/bin/gmake -C plan SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/optimizer/plan' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/optimizer/plan'
> /usr/local/bin/gmake -C prep SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/optimizer/prep' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/optimizer/prep'
> /usr/local/bin/gmake -C util SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/optimizer/util' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/optimizer/util' gmake[3]:
> Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer'
> /usr/local/bin/gmake -C port all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/port' gmake[3]: Nothing to
> be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/port' /usr/local/bin/gmake
> -C postmaster all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/postmaster' gmake[3]:
> Nothing to be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/postmaster'
> /usr/local/bin/gmake -C regex all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/regex' gmake[3]: Nothing to
> be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/regex' /usr/local/bin/gmake
> -C rewrite all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/rewrite' gmake[3]: Nothing
> to be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/rewrite'
> /usr/local/bin/gmake -C storage all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage'
> /usr/local/bin/gmake -C buffer SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/buffer' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/buffer'
> /usr/local/bin/gmake -C file SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/file' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/file'
> /usr/local/bin/gmake -C freespace SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/freespace' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/freespace'
> /usr/local/bin/gmake -C ipc SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/ipc' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/ipc'
> /usr/local/bin/gmake -C large_object SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/large_object'
> gmake[4]: `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/large_object'
> /usr/local/bin/gmake -C lmgr SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/lmgr' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/lmgr'
> /usr/local/bin/gmake -C page SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/page' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/page'
> /usr/local/bin/gmake -C smgr SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/smgr' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/storage/smgr' gmake[3]:
> Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/storage'
> /usr/local/bin/gmake -C tcop all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/tcop' gmake[3]: Nothing to
> be done for `all'.
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/tcop' /usr/local/bin/gmake
> -C utils all
> gmake[3]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils' /usr/local/bin/gmake
> -C adt SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/adt' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/adt'
> /usr/local/bin/gmake -C cache SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/cache' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/cache'
> /usr/local/bin/gmake -C error SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/error' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/error'
> /usr/local/bin/gmake -C fmgr SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/fmgr' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/fmgr'
> /usr/local/bin/gmake -C hash SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/hash' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/hash'
> /usr/local/bin/gmake -C init SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/init' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/init'
> /usr/local/bin/gmake -C misc SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/misc' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/misc'
> /usr/local/bin/gmake -C mmgr SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/mmgr' gmake[4]:
> `SUBSYS.o' is up to date.
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/mmgr'
> /usr/local/bin/gmake -C sort SUBSYS.o
> gmake[4]: Entering directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/sort' cc -O -K inline
> -I../../../../src/include -I/usr/local/include  -c tuplesort.c -o
> tuplesort.o UX:acomp: ERREUR: "tuplesort.c", ligne 1854: "inline"
> functions cannot use "static" identifier: myFunctionCall2 UX:acomp:
> ERREUR: "tuplesort.c", ligne 1856: "inline" functions cannot use "static"
> identifier: myFunctionCall2 UX:acomp: ERREUR: "tuplesort.c", ligne 1870:
> "inline" functions cannot use "static" identifier: myFunctionCall2
> UX:acomp: ERREUR: "tuplesort.c", ligne 1872: "inline" functions cannot
> use "static" identifier: myFunctionCall2 UX:acomp: ERREUR: "tuplesort.c",
> ligne 1885: "inline" functions cannot use "static" identifier:
> myFunctionCall2 UX:acomp: ERREUR: "tuplesort.c", ligne 1897: "inline"
> functions cannot use "static" identifier: myFunctionCall2 gmake[4]: ***
> [tuplesort.o] Error 1
> gmake[4]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils/sort' gmake[3]: ***
> [sort-recursive] Error 2
> gmake[3]: Leaving directory
> `/home/postgres/postgresql-7.3b5/src/backend/utils' gmake[2]: ***
> [utils-recursive] Error 2
> gmake[2]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend'
> gmake[1]: *** [all] Error 2
> gmake[1]: Leaving directory `/home/postgres/postgresql-7.3b5/src'
> gmake: *** [all] Error 2
> *** Code d'erreur 2 (bu21)
> UX:make: ERREUR: erreur irrémédiable.
>
> script done on Thu Nov  7 16:57:29 2002
> It works OK with -Xb...
>
> Regards
>>
>> 			regards, tom lane
>>
>
> --
> Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> FRANCE                      Email: ohp@pyrenet.fr
> -------------------------------------------------------------------------
> ----- Make your life a dream, make your dream a reality. (St Exupery)


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Thu, 07 Nov 2002 10:07:37 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a 
*WHAT??**

this directory has hust been created by de-taring postgresql-7.3b5.tar.gz
from my own mirror dated nov 6 20:04...


On Thu, 7 Nov 2002, Larry Rosenman wrote:

> Date: Thu, 07 Nov 2002 10:07:37 -0600
> From: Larry Rosenman <ler@lerctr.org>
> To: ohp@pyrenet.fr, Tom Lane <tgl@sss.pgh.pa.us>
> Cc: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>      pgsql-hackers@postgresql.org
> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a 
> 
> It looks like you do **NOT** have B4 or B5....
> 
> LER
> 
> 
> --On Thursday, November 07, 2002 17:00:21 +0100 Olivier PRENANT 
> <ohp@pyrenet.fr> wrote:
> 
> > On Thu, 7 Nov 2002, Tom Lane wrote:
> >
> >> Date: Thu, 07 Nov 2002 10:21:25 -0500
> >> From: Tom Lane <tgl@sss.pgh.pa.us>
> >> To: ohp@pyrenet.fr
> >> Cc: Larry Rosenman <ler@lerctr.org>, Billy G. Allie <Bill.Allie@mug.org>,
> >>      pgsql-ports@postgresql.org, pgsql-hackers@postgresql.org
> >> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and
> >> a
> >>
> >> Olivier PRENANT <ohp@pyrenet.fr> writes:
> >> > Huh! I just tried to compile 7.3b5 without CFLAGS=-Xb, it still bugs
> >> > the compiler...
> >>
> >> It won't get better if you don't show any details...
> > Ok... (sorry) this is on UW 711 WITHOUT CFLAGS=-Xb:
> > Script started on Thu Nov  7 16:57:05 2002
> > $ cd postgresql*5
> > $ make
> > Using GNU make found at /usr/local/bin/gmake
> > /usr/local/bin/gmake -C doc all
> > gmake[1]: Entering directory `/home/postgres/postgresql-7.3b5/doc'
> > gmake[1]: Nothing to be done for `all'.
> > gmake[1]: Leaving directory `/home/postgres/postgresql-7.3b5/doc'
> > /usr/local/bin/gmake -C src all
> > gmake[1]: Entering directory `/home/postgres/postgresql-7.3b5/src'
> > /usr/local/bin/gmake -C port all
> > gmake[2]: Entering directory `/home/postgres/postgresql-7.3b5/src/port'
> > gmake[2]: Nothing to be done for `all'.
> > gmake[2]: Leaving directory `/home/postgres/postgresql-7.3b5/src/port'
> > /usr/local/bin/gmake -C backend all
> > gmake[2]: Entering directory `/home/postgres/postgresql-7.3b5/src/backend'
> > /usr/local/bin/gmake -C ../../src/port all
> > gmake[3]: Entering directory `/home/postgres/postgresql-7.3b5/src/port'
> > gmake[3]: Nothing to be done for `all'.
> > gmake[3]: Leaving directory `/home/postgres/postgresql-7.3b5/src/port'
> > /usr/local/bin/gmake -C access all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access' /usr/local/bin/gmake
> > -C common SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/common' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/common'
> > /usr/local/bin/gmake -C gist SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/gist' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/gist'
> > /usr/local/bin/gmake -C hash SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/hash' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/hash'
> > /usr/local/bin/gmake -C heap SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/heap' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/heap'
> > /usr/local/bin/gmake -C index SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/index' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/index'
> > /usr/local/bin/gmake -C nbtree SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/nbtree' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/nbtree'
> > /usr/local/bin/gmake -C rtree SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/rtree' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/rtree'
> > /usr/local/bin/gmake -C transam SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/transam' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/access/transam' gmake[3]:
> > Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/access'
> > /usr/local/bin/gmake -C bootstrap all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/bootstrap' gmake[3]: Nothing
> > to be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/bootstrap'
> > /usr/local/bin/gmake -C catalog all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/catalog' gmake[3]: Nothing
> > to be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/catalog'
> > /usr/local/bin/gmake -C parser all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/parser' gmake[3]: Nothing to
> > be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/parser' /usr/local/bin/gmake
> > -C commands all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/commands' gmake[3]: Nothing
> > to be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/commands'
> > /usr/local/bin/gmake -C executor all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/executor' gmake[3]: Nothing
> > to be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/executor'
> > /usr/local/bin/gmake -C lib all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/lib' gmake[3]: Nothing to be
> > done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/lib' /usr/local/bin/gmake -C
> > libpq all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/libpq' gmake[3]: Nothing to
> > be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/libpq' /usr/local/bin/gmake
> > -C main all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/main' gmake[3]: Nothing to
> > be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/main' /usr/local/bin/gmake
> > -C nodes all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/nodes' gmake[3]: Nothing to
> > be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/nodes' /usr/local/bin/gmake
> > -C optimizer all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/optimizer'
> > /usr/local/bin/gmake -C geqo SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/optimizer/geqo' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/optimizer/geqo'
> > /usr/local/bin/gmake -C path SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/optimizer/path' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/optimizer/path'
> > /usr/local/bin/gmake -C plan SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/optimizer/plan' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/optimizer/plan'
> > /usr/local/bin/gmake -C prep SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/optimizer/prep' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/optimizer/prep'
> > /usr/local/bin/gmake -C util SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/optimizer/util' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/optimizer/util' gmake[3]:
> > Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/optimizer'
> > /usr/local/bin/gmake -C port all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/port' gmake[3]: Nothing to
> > be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/port' /usr/local/bin/gmake
> > -C postmaster all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/postmaster' gmake[3]:
> > Nothing to be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/postmaster'
> > /usr/local/bin/gmake -C regex all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/regex' gmake[3]: Nothing to
> > be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/regex' /usr/local/bin/gmake
> > -C rewrite all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/rewrite' gmake[3]: Nothing
> > to be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/rewrite'
> > /usr/local/bin/gmake -C storage all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage'
> > /usr/local/bin/gmake -C buffer SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/buffer' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/buffer'
> > /usr/local/bin/gmake -C file SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/file' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/file'
> > /usr/local/bin/gmake -C freespace SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/freespace' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/freespace'
> > /usr/local/bin/gmake -C ipc SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/ipc' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/ipc'
> > /usr/local/bin/gmake -C large_object SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/large_object'
> > gmake[4]: `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/large_object'
> > /usr/local/bin/gmake -C lmgr SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/lmgr' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/lmgr'
> > /usr/local/bin/gmake -C page SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/page' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/page'
> > /usr/local/bin/gmake -C smgr SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/smgr' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/storage/smgr' gmake[3]:
> > Leaving directory `/home/postgres/postgresql-7.3b5/src/backend/storage'
> > /usr/local/bin/gmake -C tcop all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/tcop' gmake[3]: Nothing to
> > be done for `all'.
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/tcop' /usr/local/bin/gmake
> > -C utils all
> > gmake[3]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils' /usr/local/bin/gmake
> > -C adt SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/adt' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/adt'
> > /usr/local/bin/gmake -C cache SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/cache' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/cache'
> > /usr/local/bin/gmake -C error SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/error' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/error'
> > /usr/local/bin/gmake -C fmgr SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/fmgr' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/fmgr'
> > /usr/local/bin/gmake -C hash SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/hash' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/hash'
> > /usr/local/bin/gmake -C init SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/init' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/init'
> > /usr/local/bin/gmake -C misc SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/misc' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/misc'
> > /usr/local/bin/gmake -C mmgr SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/mmgr' gmake[4]:
> > `SUBSYS.o' is up to date.
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/mmgr'
> > /usr/local/bin/gmake -C sort SUBSYS.o
> > gmake[4]: Entering directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/sort' cc -O -K inline
> > -I../../../../src/include -I/usr/local/include  -c tuplesort.c -o
> > tuplesort.o UX:acomp: ERREUR: "tuplesort.c", ligne 1854: "inline"
> > functions cannot use "static" identifier: myFunctionCall2 UX:acomp:
> > ERREUR: "tuplesort.c", ligne 1856: "inline" functions cannot use "static"
> > identifier: myFunctionCall2 UX:acomp: ERREUR: "tuplesort.c", ligne 1870:
> > "inline" functions cannot use "static" identifier: myFunctionCall2
> > UX:acomp: ERREUR: "tuplesort.c", ligne 1872: "inline" functions cannot
> > use "static" identifier: myFunctionCall2 UX:acomp: ERREUR: "tuplesort.c",
> > ligne 1885: "inline" functions cannot use "static" identifier:
> > myFunctionCall2 UX:acomp: ERREUR: "tuplesort.c", ligne 1897: "inline"
> > functions cannot use "static" identifier: myFunctionCall2 gmake[4]: ***
> > [tuplesort.o] Error 1
> > gmake[4]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils/sort' gmake[3]: ***
> > [sort-recursive] Error 2
> > gmake[3]: Leaving directory
> > `/home/postgres/postgresql-7.3b5/src/backend/utils' gmake[2]: ***
> > [utils-recursive] Error 2
> > gmake[2]: Leaving directory `/home/postgres/postgresql-7.3b5/src/backend'
> > gmake[1]: *** [all] Error 2
> > gmake[1]: Leaving directory `/home/postgres/postgresql-7.3b5/src'
> > gmake: *** [all] Error 2
> > *** Code d'erreur 2 (bu21)
> > UX:make: ERREUR: erreur irrï¿½mï¿½diable.
> >
> > script done on Thu Nov  7 16:57:29 2002
> > It works OK with -Xb...
> >
> > Regards
> >>
> >> 			regards, tom lane
> >>
> >
> > --
> > Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> > Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> > 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> > FRANCE                      Email: ohp@pyrenet.fr
> > -------------------------------------------------------------------------
> > ----- Make your life a dream, make your dream a reality. (St Exupery)
> 
> 
> 

-- 
Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
FRANCE                      Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)


Thu, 7 Nov 2002 17:15:08 +0100 (MET)
Olivier PRENANT <ohp@pyrenet.fr>
Re: [HACKERS] PostgreSQL supported platform report and a 
Olivier PRENANT <ohp@pyrenet.fr> writes:
> *WHAT??**
> this directory has hust been created by de-taring postgresql-7.3b5.tar.gz
> from my own mirror dated nov 6 20:04...

Well, there's something darn weird here.  Is
src/backend/utils/sort/tuplesort.c version 1.28 or 1.29?  At line 1838,
do you see
	inline int32
	ApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,
or
	static inline int32
	inlineApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,

			regards, tom lane

Thu, 07 Nov 2002 11:26:42 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a 
I see the latter in both b4 and b5!

I've just relaunched my mirroring procedure and it did'nt pick another b4
or b5!

What happens??

Regards,
On Thu, 7 Nov 2002, Tom Lane wrote:

> Date: Thu, 07 Nov 2002 11:26:42 -0500
> From: Tom Lane <tgl@sss.pgh.pa.us>
> To: ohp@pyrenet.fr
> Cc: Larry Rosenman <ler@lerctr.org>, Billy G. Allie <Bill.Allie@mug.org>,
>      pgsql-ports@postgresql.org, pgsql-hackers@postgresql.org
> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a 
> 
> Olivier PRENANT <ohp@pyrenet.fr> writes:
> > *WHAT??**
> > this directory has hust been created by de-taring postgresql-7.3b5.tar.gz
> > from my own mirror dated nov 6 20:04...
> 
> Well, there's something darn weird here.  Is
> src/backend/utils/sort/tuplesort.c version 1.28 or 1.29?  At line 1838,
> do you see
> 	inline int32
> 	ApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,
> or
> 	static inline int32
> 	inlineApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,
> 
> 			regards, tom lane
> 

-- 
Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
FRANCE                      Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)


Thu, 7 Nov 2002 17:34:49 +0100 (MET)
Olivier PRENANT <ohp@pyrenet.fr>
Re: [HACKERS] PostgreSQL supported platform report and a 
It **LOOKS** right.  I'm about to double check it on 7.1.3.

Olivier, is this the 7.1.1b FS Compiler?

I wonder if a bug fix made it in...

Wierd.



--On Thursday, November 07, 2002 17:34:49 +0100 Olivier PRENANT 
<ohp@pyrenet.fr> wrote:

> I see the latter in both b4 and b5!
>
> I've just relaunched my mirroring procedure and it did'nt pick another b4
> or b5!
>
> What happens??
>
> Regards,
> On Thu, 7 Nov 2002, Tom Lane wrote:
>
>> Date: Thu, 07 Nov 2002 11:26:42 -0500
>> From: Tom Lane <tgl@sss.pgh.pa.us>
>> To: ohp@pyrenet.fr
>> Cc: Larry Rosenman <ler@lerctr.org>, Billy G. Allie <Bill.Allie@mug.org>,
>>      pgsql-ports@postgresql.org, pgsql-hackers@postgresql.org
>> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and
>> a
>>
>> Olivier PRENANT <ohp@pyrenet.fr> writes:
>> > *WHAT??**
>> > this directory has hust been created by de-taring
>> > postgresql-7.3b5.tar.gz from my own mirror dated nov 6 20:04...
>>
>> Well, there's something darn weird here.  Is
>> src/backend/utils/sort/tuplesort.c version 1.28 or 1.29?  At line 1838,
>> do you see
>> 	inline int32
>> 	ApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,
>> or
>> 	static inline int32
>> 	inlineApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,
>>
>> 			regards, tom lane
>>
>
> --
> Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> FRANCE                      Email: ohp@pyrenet.fr
> -------------------------------------------------------------------------
> ----- Make your life a dream, make your dream a reality. (St Exupery)


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Thu, 07 Nov 2002 10:37:36 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a 
What's FS,  it(s the 7.1.1b compiler yes.

I don't mind having CFLAGS=-Xb  though, done it for php already...
By everyone says this should go off so..

Regards
On Thu, 7 Nov 2002, Larry Rosenman wrote:

> Date: Thu, 07 Nov 2002 10:37:36 -0600
> From: Larry Rosenman <ler@lerctr.org>
> To: ohp@pyrenet.fr, Tom Lane <tgl@sss.pgh.pa.us>
> Cc: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>      pgsql-hackers@postgresql.org
> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a 
> 
> It **LOOKS** right.  I'm about to double check it on 7.1.3.
> 
> Olivier, is this the 7.1.1b FS Compiler?
> 
> I wonder if a bug fix made it in...
> 
> Wierd.
> 
> 
> 
> --On Thursday, November 07, 2002 17:34:49 +0100 Olivier PRENANT 
> <ohp@pyrenet.fr> wrote:
> 
> > I see the latter in both b4 and b5!
> >
> > I've just relaunched my mirroring procedure and it did'nt pick another b4
> > or b5!
> >
> > What happens??
> >
> > Regards,
> > On Thu, 7 Nov 2002, Tom Lane wrote:
> >
> >> Date: Thu, 07 Nov 2002 11:26:42 -0500
> >> From: Tom Lane <tgl@sss.pgh.pa.us>
> >> To: ohp@pyrenet.fr
> >> Cc: Larry Rosenman <ler@lerctr.org>, Billy G. Allie <Bill.Allie@mug.org>,
> >>      pgsql-ports@postgresql.org, pgsql-hackers@postgresql.org
> >> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and
> >> a
> >>
> >> Olivier PRENANT <ohp@pyrenet.fr> writes:
> >> > *WHAT??**
> >> > this directory has hust been created by de-taring
> >> > postgresql-7.3b5.tar.gz from my own mirror dated nov 6 20:04...
> >>
> >> Well, there's something darn weird here.  Is
> >> src/backend/utils/sort/tuplesort.c version 1.28 or 1.29?  At line 1838,
> >> do you see
> >> 	inline int32
> >> 	ApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,
> >> or
> >> 	static inline int32
> >> 	inlineApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,
> >>
> >> 			regards, tom lane
> >>
> >
> > --
> > Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> > Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> > 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> > FRANCE                      Email: ohp@pyrenet.fr
> > -------------------------------------------------------------------------
> > ----- Make your life a dream, make your dream a reality. (St Exupery)
> 
> 
> 

-- 
Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
FRANCE                      Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)


Thu, 7 Nov 2002 17:40:24 +0100 (MET)
Olivier PRENANT <ohp@pyrenet.fr>
Re: [HACKERS] PostgreSQL supported platform report and a 
FS== Feature Supplement.

I've got a compile running with the 7.3b5 tarball on my 7.1.3 system (with 
the newer
compiler).

We'll see. :-)



--On Thursday, November 07, 2002 17:40:24 +0100 Olivier PRENANT 
<ohp@pyrenet.fr> wrote:

> What's FS,  it(s the 7.1.1b compiler yes.
>
> I don't mind having CFLAGS=-Xb  though, done it for php already...
> By everyone says this should go off so..
>
> Regards
> On Thu, 7 Nov 2002, Larry Rosenman wrote:
>
>> Date: Thu, 07 Nov 2002 10:37:36 -0600
>> From: Larry Rosenman <ler@lerctr.org>
>> To: ohp@pyrenet.fr, Tom Lane <tgl@sss.pgh.pa.us>
>> Cc: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>>      pgsql-hackers@postgresql.org
>> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and
>> a
>>
>> It **LOOKS** right.  I'm about to double check it on 7.1.3.
>>
>> Olivier, is this the 7.1.1b FS Compiler?
>>
>> I wonder if a bug fix made it in...
>>
>> Wierd.
>>
>>
>>
>> --On Thursday, November 07, 2002 17:34:49 +0100 Olivier PRENANT
>> <ohp@pyrenet.fr> wrote:
>>
>> > I see the latter in both b4 and b5!
>> >
>> > I've just relaunched my mirroring procedure and it did'nt pick another
>> > b4 or b5!
>> >
>> > What happens??
>> >
>> > Regards,
>> > On Thu, 7 Nov 2002, Tom Lane wrote:
>> >
>> >> Date: Thu, 07 Nov 2002 11:26:42 -0500
>> >> From: Tom Lane <tgl@sss.pgh.pa.us>
>> >> To: ohp@pyrenet.fr
>> >> Cc: Larry Rosenman <ler@lerctr.org>, Billy G. Allie
>> >> <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>> >>      pgsql-hackers@postgresql.org
>> >> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report
>> >> and a
>> >>
>> >> Olivier PRENANT <ohp@pyrenet.fr> writes:
>> >> > *WHAT??**
>> >> > this directory has hust been created by de-taring
>> >> > postgresql-7.3b5.tar.gz from my own mirror dated nov 6 20:04...
>> >>
>> >> Well, there's something darn weird here.  Is
>> >> src/backend/utils/sort/tuplesort.c version 1.28 or 1.29?  At line
>> >> 1838, do you see
>> >> 	inline int32
>> >> 	ApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,
>> >> or
>> >> 	static inline int32
>> >> 	inlineApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind
>> >> 	kind,
>> >>
>> >> 			regards, tom lane
>> >>
>> >
>> > --
>> > Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
>> > Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
>> > 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
>> > FRANCE                      Email: ohp@pyrenet.fr
>> > ----------------------------------------------------------------------
>> > --- ----- Make your life a dream, make your dream a reality. (St
>> > Exupery)
>>
>>
>>
>
> --
> Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> FRANCE                      Email: ohp@pyrenet.fr
> -------------------------------------------------------------------------
> ----- Make your life a dream, make your dream a reality. (St Exupery)


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Thu, 07 Nov 2002 10:42:53 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a 
cc -O -g -I../../../../src/include -I/usr/local/include  -c -o tuplesort.o 
tuplesort.c
UX:cc: WARNING: debugging and optimization mutually exclusive; -O disabled

It still passes here.  I really wonder if they fixed something in the 4.1 
compiler. IIRC
the 7.1.1b compiler is 4.0.

Truly wierd.

LER


--On Thursday, November 07, 2002 17:40:24 +0100 Olivier PRENANT 
<ohp@pyrenet.fr> wrote:

> What's FS,  it(s the 7.1.1b compiler yes.
>
> I don't mind having CFLAGS=-Xb  though, done it for php already...
> By everyone says this should go off so..
>
> Regards
> On Thu, 7 Nov 2002, Larry Rosenman wrote:
>
>> Date: Thu, 07 Nov 2002 10:37:36 -0600
>> From: Larry Rosenman <ler@lerctr.org>
>> To: ohp@pyrenet.fr, Tom Lane <tgl@sss.pgh.pa.us>
>> Cc: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>>      pgsql-hackers@postgresql.org
>> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and
>> a
>>
>> It **LOOKS** right.  I'm about to double check it on 7.1.3.
>>
>> Olivier, is this the 7.1.1b FS Compiler?
>>
>> I wonder if a bug fix made it in...
>>
>> Wierd.
>>
>>
>>
>> --On Thursday, November 07, 2002 17:34:49 +0100 Olivier PRENANT
>> <ohp@pyrenet.fr> wrote:
>>
>> > I see the latter in both b4 and b5!
>> >
>> > I've just relaunched my mirroring procedure and it did'nt pick another
>> > b4 or b5!
>> >
>> > What happens??
>> >
>> > Regards,
>> > On Thu, 7 Nov 2002, Tom Lane wrote:
>> >
>> >> Date: Thu, 07 Nov 2002 11:26:42 -0500
>> >> From: Tom Lane <tgl@sss.pgh.pa.us>
>> >> To: ohp@pyrenet.fr
>> >> Cc: Larry Rosenman <ler@lerctr.org>, Billy G. Allie
>> >> <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>> >>      pgsql-hackers@postgresql.org
>> >> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report
>> >> and a
>> >>
>> >> Olivier PRENANT <ohp@pyrenet.fr> writes:
>> >> > *WHAT??**
>> >> > this directory has hust been created by de-taring
>> >> > postgresql-7.3b5.tar.gz from my own mirror dated nov 6 20:04...
>> >>
>> >> Well, there's something darn weird here.  Is
>> >> src/backend/utils/sort/tuplesort.c version 1.28 or 1.29?  At line
>> >> 1838, do you see
>> >> 	inline int32
>> >> 	ApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,
>> >> or
>> >> 	static inline int32
>> >> 	inlineApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind
>> >> 	kind,
>> >>
>> >> 			regards, tom lane
>> >>
>> >
>> > --
>> > Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
>> > Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
>> > 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
>> > FRANCE                      Email: ohp@pyrenet.fr
>> > ----------------------------------------------------------------------
>> > --- ----- Make your life a dream, make your dream a reality. (St
>> > Exupery)
>>
>>
>>
>
> --
> Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> FRANCE                      Email: ohp@pyrenet.fr
> -------------------------------------------------------------------------
> ----- Make your life a dream, make your dream a reality. (St Exupery)


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Thu, 07 Nov 2002 10:45:13 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a 
Haha!!!

It passes (b4) on 800 and not on uw 711..

Larry, should I install 800 SDK on 711?

On Thu, 7 Nov 2002, Larry Rosenman wrote:

> Date: Thu, 07 Nov 2002 10:45:13 -0600
> From: Larry Rosenman <ler@lerctr.org>
> To: ohp@pyrenet.fr
> Cc: Tom Lane <tgl@sss.pgh.pa.us>, Billy G. Allie <Bill.Allie@mug.org>,
>      pgsql-ports@postgresql.org, pgsql-hackers@postgresql.org
> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a 
> 
> cc -O -g -I../../../../src/include -I/usr/local/include  -c -o tuplesort.o 
> tuplesort.c
> UX:cc: WARNING: debugging and optimization mutually exclusive; -O disabled
> 
> It still passes here.  I really wonder if they fixed something in the 4.1 
> compiler. IIRC
> the 7.1.1b compiler is 4.0.
> 
> Truly wierd.
> 
> LER
> 
> 
> --On Thursday, November 07, 2002 17:40:24 +0100 Olivier PRENANT 
> <ohp@pyrenet.fr> wrote:
> 
> > What's FS,  it(s the 7.1.1b compiler yes.
> >
> > I don't mind having CFLAGS=-Xb  though, done it for php already...
> > By everyone says this should go off so..
> >
> > Regards
> > On Thu, 7 Nov 2002, Larry Rosenman wrote:
> >
> >> Date: Thu, 07 Nov 2002 10:37:36 -0600
> >> From: Larry Rosenman <ler@lerctr.org>
> >> To: ohp@pyrenet.fr, Tom Lane <tgl@sss.pgh.pa.us>
> >> Cc: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
> >>      pgsql-hackers@postgresql.org
> >> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and
> >> a
> >>
> >> It **LOOKS** right.  I'm about to double check it on 7.1.3.
> >>
> >> Olivier, is this the 7.1.1b FS Compiler?
> >>
> >> I wonder if a bug fix made it in...
> >>
> >> Wierd.
> >>
> >>
> >>
> >> --On Thursday, November 07, 2002 17:34:49 +0100 Olivier PRENANT
> >> <ohp@pyrenet.fr> wrote:
> >>
> >> > I see the latter in both b4 and b5!
> >> >
> >> > I've just relaunched my mirroring procedure and it did'nt pick another
> >> > b4 or b5!
> >> >
> >> > What happens??
> >> >
> >> > Regards,
> >> > On Thu, 7 Nov 2002, Tom Lane wrote:
> >> >
> >> >> Date: Thu, 07 Nov 2002 11:26:42 -0500
> >> >> From: Tom Lane <tgl@sss.pgh.pa.us>
> >> >> To: ohp@pyrenet.fr
> >> >> Cc: Larry Rosenman <ler@lerctr.org>, Billy G. Allie
> >> >> <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
> >> >>      pgsql-hackers@postgresql.org
> >> >> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report
> >> >> and a
> >> >>
> >> >> Olivier PRENANT <ohp@pyrenet.fr> writes:
> >> >> > *WHAT??**
> >> >> > this directory has hust been created by de-taring
> >> >> > postgresql-7.3b5.tar.gz from my own mirror dated nov 6 20:04...
> >> >>
> >> >> Well, there's something darn weird here.  Is
> >> >> src/backend/utils/sort/tuplesort.c version 1.28 or 1.29?  At line
> >> >> 1838, do you see
> >> >> 	inline int32
> >> >> 	ApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,
> >> >> or
> >> >> 	static inline int32
> >> >> 	inlineApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind
> >> >> 	kind,
> >> >>
> >> >> 			regards, tom lane
> >> >>
> >> >
> >> > --
> >> > Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> >> > Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> >> > 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> >> > FRANCE                      Email: ohp@pyrenet.fr
> >> > ----------------------------------------------------------------------
> >> > --- ----- Make your life a dream, make your dream a reality. (St
> >> > Exupery)
> >>
> >>
> >>
> >
> > --
> > Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> > Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> > 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> > FRANCE                      Email: ohp@pyrenet.fr
> > -------------------------------------------------------------------------
> > ----- Make your life a dream, make your dream a reality. (St Exupery)
> 
> 
> 

-- 
Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
FRANCE                      Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)


Thu, 7 Nov 2002 18:02:51 +0100 (MET)
Olivier PRENANT <ohp@pyrenet.fr>
Re: [HACKERS] PostgreSQL supported platform report and a 


--On Thursday, November 07, 2002 18:02:51 +0100 Olivier PRENANT 
<ohp@pyrenet.fr> wrote:

> Haha!!!
>
> It passes (b4) on 800 and not on uw 711..
>
> Larry, should I install 800 SDK on 711?
Yes.


>
> On Thu, 7 Nov 2002, Larry Rosenman wrote:
>
>> Date: Thu, 07 Nov 2002 10:45:13 -0600
>> From: Larry Rosenman <ler@lerctr.org>
>> To: ohp@pyrenet.fr
>> Cc: Tom Lane <tgl@sss.pgh.pa.us>, Billy G. Allie <Bill.Allie@mug.org>,
>>      pgsql-ports@postgresql.org, pgsql-hackers@postgresql.org
>> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and
>> a
>>
>> cc -O -g -I../../../../src/include -I/usr/local/include  -c -o
>> tuplesort.o  tuplesort.c
>> UX:cc: WARNING: debugging and optimization mutually exclusive; -O
>> disabled
>>
>> It still passes here.  I really wonder if they fixed something in the
>> 4.1  compiler. IIRC
>> the 7.1.1b compiler is 4.0.
>>
>> Truly wierd.
>>
>> LER
>>
>>
>> --On Thursday, November 07, 2002 17:40:24 +0100 Olivier PRENANT
>> <ohp@pyrenet.fr> wrote:
>>
>> > What's FS,  it(s the 7.1.1b compiler yes.
>> >
>> > I don't mind having CFLAGS=-Xb  though, done it for php already...
>> > By everyone says this should go off so..
>> >
>> > Regards
>> > On Thu, 7 Nov 2002, Larry Rosenman wrote:
>> >
>> >> Date: Thu, 07 Nov 2002 10:37:36 -0600
>> >> From: Larry Rosenman <ler@lerctr.org>
>> >> To: ohp@pyrenet.fr, Tom Lane <tgl@sss.pgh.pa.us>
>> >> Cc: Billy G. Allie <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>> >>      pgsql-hackers@postgresql.org
>> >> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report
>> >> and a
>> >>
>> >> It **LOOKS** right.  I'm about to double check it on 7.1.3.
>> >>
>> >> Olivier, is this the 7.1.1b FS Compiler?
>> >>
>> >> I wonder if a bug fix made it in...
>> >>
>> >> Wierd.
>> >>
>> >>
>> >>
>> >> --On Thursday, November 07, 2002 17:34:49 +0100 Olivier PRENANT
>> >> <ohp@pyrenet.fr> wrote:
>> >>
>> >> > I see the latter in both b4 and b5!
>> >> >
>> >> > I've just relaunched my mirroring procedure and it did'nt pick
>> >> > another b4 or b5!
>> >> >
>> >> > What happens??
>> >> >
>> >> > Regards,
>> >> > On Thu, 7 Nov 2002, Tom Lane wrote:
>> >> >
>> >> >> Date: Thu, 07 Nov 2002 11:26:42 -0500
>> >> >> From: Tom Lane <tgl@sss.pgh.pa.us>
>> >> >> To: ohp@pyrenet.fr
>> >> >> Cc: Larry Rosenman <ler@lerctr.org>, Billy G. Allie
>> >> >> <Bill.Allie@mug.org>, pgsql-ports@postgresql.org,
>> >> >>      pgsql-hackers@postgresql.org
>> >> >> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report
>> >> >> and a
>> >> >>
>> >> >> Olivier PRENANT <ohp@pyrenet.fr> writes:
>> >> >> > *WHAT??**
>> >> >> > this directory has hust been created by de-taring
>> >> >> > postgresql-7.3b5.tar.gz from my own mirror dated nov 6 20:04...
>> >> >>
>> >> >> Well, there's something darn weird here.  Is
>> >> >> src/backend/utils/sort/tuplesort.c version 1.28 or 1.29?  At line
>> >> >> 1838, do you see
>> >> >> 	inline int32
>> >> >> 	ApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind kind,
>> >> >> or
>> >> >> 	static inline int32
>> >> >> 	inlineApplySortFunction(FmgrInfo *sortFunction, SortFunctionKind
>> >> >> 	kind,
>> >> >>
>> >> >> 			regards, tom lane
>> >> >>
>> >> >
>> >> > --
>> >> > Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
>> >> > Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
>> >> > 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
>> >> > FRANCE                      Email: ohp@pyrenet.fr
>> >> > -------------------------------------------------------------------
>> >> > --- --- ----- Make your life a dream, make your dream a reality. (St
>> >> > Exupery)
>> >>
>> >>
>> >>
>> >
>> > --
>> > Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
>> > Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
>> > 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
>> > FRANCE                      Email: ohp@pyrenet.fr
>> > ----------------------------------------------------------------------
>> > --- ----- Make your life a dream, make your dream a reality. (St
>> > Exupery)
>>
>>
>>
>
> --
> Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
> Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
> 31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
> FRANCE                      Email: ohp@pyrenet.fr
> -------------------------------------------------------------------------
> ----- Make your life a dream, make your dream a reality. (St Exupery)
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
>


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Thu, 07 Nov 2002 11:23:13 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a 


--On Thursday, November 07, 2002 22:21:56 +0100 Peter Eisentraut 
<peter_e@gmx.net> wrote:

> Olivier PRENANT writes:
>
>> I don't mind having CFLAGS=-Xb  though, done it for php already...
>> By everyone says this should go off so..
>
> The idea of the platform testing is not to determine whether you can
> compile PostgreSQL after performing a secret dance.  If it doesn't compile
> with the default options, please don't report it as supported.
It DOES compile out of the box now on 8.0.0(7.1.2) and 7.1.3.  Apparently a 
compiler
fix between the 7.1.1b FS and 7.1.2.

The -Xb switch is NOT a secret dance.  It's needed for LOTS of open source 
stuff.

See the discussion from the Caldera folks last week.

Tom's fix fixed the defaults for 7.1.2+


>
> --
> Peter Eisentraut   peter_e@gmx.net


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Thu, 07 Nov 2002 15:17:28 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a 
Olivier PRENANT writes:

> I don't mind having CFLAGS=-Xb  though, done it for php already...
> By everyone says this should go off so..

The idea of the platform testing is not to determine whether you can
compile PostgreSQL after performing a secret dance.  If it doesn't compile
with the default options, please don't report it as supported.

-- 
Peter Eisentraut   peter_e@gmx.net


Thu, 7 Nov 2002 22:21:56 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] PostgreSQL supported platform report and a 
Bruce Momjian writes:

> I am fine with this because it only touches unixware-specific stuff,

This is an entirely new feature, so it's inappropriate to do now.  And if
we do it, we should do it for all platforms.

-- 
Peter Eisentraut   peter_e@gmx.net


Thu, 7 Nov 2002 22:22:26 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] PostgreSQL supported platform report and a
Larry Rosenman <ler@lerctr.org> writes:
> It DOES compile out of the box now on 8.0.0(7.1.2) and 7.1.3.  Apparently a 
> compiler
> fix between the 7.1.1b FS and 7.1.2.

Well, this is what the REMARKS column is for in the supported-platform
list.  Seems we need a comment like "for older compiler versions, you
may need to add -Xb to CFLAGS".  Can anyone provide a short and accurate
description of when to do this?

			regards, tom lane

Thu, 07 Nov 2002 16:34:12 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a 
For compilers earlier than the one released with OpenUNIX 8.0.0(UnixWare 
7.1.2), Including
the 7.1.1b Feature Supplement, you may need to specify -Xb in CFLAGS or the 
CC environment
variable.



--On Thursday, November 07, 2002 16:34:12 -0500 Tom Lane 
<tgl@sss.pgh.pa.us> wrote:

> Larry Rosenman <ler@lerctr.org> writes:
>> It DOES compile out of the box now on 8.0.0(7.1.2) and 7.1.3.
>> Apparently a  compiler
>> fix between the 7.1.1b FS and 7.1.2.
>
> Well, this is what the REMARKS column is for in the supported-platform
> list.  Seems we need a comment like "for older compiler versions, you
> may need to add -Xb to CFLAGS".  Can anyone provide a short and accurate
> description of when to do this?
>
> 			regards, tom lane


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Thu, 07 Nov 2002 15:41:53 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a 

We do point to the FAQ_SCO file for specifics.  Would you send a diff
for that file?

---------------------------------------------------------------------------

Larry Rosenman wrote:
> For compilers earlier than the one released with OpenUNIX 8.0.0(UnixWare 
> 7.1.2), Including
> the 7.1.1b Feature Supplement, you may need to specify -Xb in CFLAGS or the 
> CC environment
> variable.
> 
> 
> 
> --On Thursday, November 07, 2002 16:34:12 -0500 Tom Lane 
> <tgl@sss.pgh.pa.us> wrote:
> 
> > Larry Rosenman <ler@lerctr.org> writes:
> >> It DOES compile out of the box now on 8.0.0(7.1.2) and 7.1.3.
> >> Apparently a  compiler
> >> fix between the 7.1.1b FS and 7.1.2.
> >
> > Well, this is what the REMARKS column is for in the supported-platform
> > list.  Seems we need a comment like "for older compiler versions, you
> > may need to add -Xb to CFLAGS".  Can anyone provide a short and accurate
> > description of when to do this?
> >
> > 			regards, tom lane
> 
> 
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 7 Nov 2002 17:08:04 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a
With or withou Billie's update(s)?

(I haven't looked at them)....

LER


--On Thursday, November 07, 2002 17:08:04 -0500 Bruce Momjian 
<pgman@candle.pha.pa.us> wrote:

>
> We do point to the FAQ_SCO file for specifics.  Would you send a diff
> for that file?
>
> -------------------------------------------------------------------------
> --
>
> Larry Rosenman wrote:
>> For compilers earlier than the one released with OpenUNIX 8.0.0(UnixWare
>> 7.1.2), Including
>> the 7.1.1b Feature Supplement, you may need to specify -Xb in CFLAGS or
>> the  CC environment
>> variable.
>>
>>
>>
>> --On Thursday, November 07, 2002 16:34:12 -0500 Tom Lane
>> <tgl@sss.pgh.pa.us> wrote:
>>
>> > Larry Rosenman <ler@lerctr.org> writes:
>> >> It DOES compile out of the box now on 8.0.0(7.1.2) and 7.1.3.
>> >> Apparently a  compiler
>> >> fix between the 7.1.1b FS and 7.1.2.
>> >
>> > Well, this is what the REMARKS column is for in the supported-platform
>> > list.  Seems we need a comment like "for older compiler versions, you
>> > may need to add -Xb to CFLAGS".  Can anyone provide a short and
>> > accurate description of when to do this?
>> >
>> > 			regards, tom lane
>>
>>
>> --
>> Larry Rosenman                     http://www.lerctr.org/~ler
>> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
>> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 3: if posting/reading through Usenet, please send an appropriate
>> subscribe-nomail command to majordomo@postgresql.org so that your
>> message can get through to the mailing list cleanly
>>
>
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania
> 19073


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Thu, 07 Nov 2002 16:10:44 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a

His updates deal only with the LDLIBRARY path issue, which I think we
are keeping for 7.4:

*** ./doc/FAQ_SCO.orig  Wed Nov  6 21:35:46 2002
--- ./doc/FAQ_SCO       Wed Nov  6 21:40:44 2002
***************
*** 71,76 ****
--- 71,79 ----
  
  configure --with-libs=/usr/local/lib --with-includes=/usr/local/include
  
+ You will also need to set LD_LIBRARY_PATH to '/usr/local/lib' (or add it to
+ LD_LIBRARY_PATH, if LD_LIBRARY_PATH already exists) before running config-
+ ure or the test for readline will fail.
    

---------------------------------------------------------------------------

Larry Rosenman wrote:
> With or withou Billie's update(s)?
> 
> (I haven't looked at them)....
> 
> LER
> 
> 
> --On Thursday, November 07, 2002 17:08:04 -0500 Bruce Momjian 
> <pgman@candle.pha.pa.us> wrote:
> 
> >
> > We do point to the FAQ_SCO file for specifics.  Would you send a diff
> > for that file?
> >
> > -------------------------------------------------------------------------
> > --
> >
> > Larry Rosenman wrote:
> >> For compilers earlier than the one released with OpenUNIX 8.0.0(UnixWare
> >> 7.1.2), Including
> >> the 7.1.1b Feature Supplement, you may need to specify -Xb in CFLAGS or
> >> the  CC environment
> >> variable.
> >>
> >>
> >>
> >> --On Thursday, November 07, 2002 16:34:12 -0500 Tom Lane
> >> <tgl@sss.pgh.pa.us> wrote:
> >>
> >> > Larry Rosenman <ler@lerctr.org> writes:
> >> >> It DOES compile out of the box now on 8.0.0(7.1.2) and 7.1.3.
> >> >> Apparently a  compiler
> >> >> fix between the 7.1.1b FS and 7.1.2.
> >> >
> >> > Well, this is what the REMARKS column is for in the supported-platform
> >> > list.  Seems we need a comment like "for older compiler versions, you
> >> > may need to add -Xb to CFLAGS".  Can anyone provide a short and
> >> > accurate description of when to do this?
> >> >
> >> > 			regards, tom lane
> >>
> >>
> >> --
> >> Larry Rosenman                     http://www.lerctr.org/~ler
> >> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
> >> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
> >>
> >>
> >> ---------------------------(end of broadcast)---------------------------
> >> TIP 3: if posting/reading through Usenet, please send an appropriate
> >> subscribe-nomail command to majordomo@postgresql.org so that your
> >> message can get through to the mailing list cleanly
> >>
> >
> > --
> >   Bruce Momjian                        |  http://candle.pha.pa.us
> >   pgman@candle.pha.pa.us               |  (610) 359-1001
> >   +  If your life is a hard drive,     |  13 Roberts Road
> >   +  Christ can be your backup.        |  Newtown Square, Pennsylvania
> > 19073
> 
> 
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 7 Nov 2002 17:44:37 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a
OK, I'll try and do up a diff to B5's tonite (probably late, my Daughter's 
elementary school honors choir has a performance tonite).

LER


--On Thursday, November 07, 2002 17:44:37 -0500 Bruce Momjian 
<pgman@candle.pha.pa.us> wrote:

>
> His updates deal only with the LDLIBRARY path issue, which I think we
> are keeping for 7.4:
>
> *** ./doc/FAQ_SCO.orig  Wed Nov  6 21:35:46 2002
> --- ./doc/FAQ_SCO       Wed Nov  6 21:40:44 2002
> ***************
> *** 71,76 ****
> --- 71,79 ----
>
>   configure --with-libs=/usr/local/lib --with-includes=/usr/local/include
>
> + You will also need to set LD_LIBRARY_PATH to '/usr/local/lib' (or add
> it to + LD_LIBRARY_PATH, if LD_LIBRARY_PATH already exists) before
> running config- + ure or the test for readline will fail.
>
>
> -------------------------------------------------------------------------
> --
>
> Larry Rosenman wrote:
>> With or withou Billie's update(s)?
>>
>> (I haven't looked at them)....
>>
>> LER
>>
>>
>> --On Thursday, November 07, 2002 17:08:04 -0500 Bruce Momjian
>> <pgman@candle.pha.pa.us> wrote:
>>
>> >
>> > We do point to the FAQ_SCO file for specifics.  Would you send a diff
>> > for that file?
>> >
>> > ----------------------------------------------------------------------
>> > --- --
>> >
>> > Larry Rosenman wrote:
>> >> For compilers earlier than the one released with OpenUNIX
>> >> 8.0.0(UnixWare 7.1.2), Including
>> >> the 7.1.1b Feature Supplement, you may need to specify -Xb in CFLAGS
>> >> or the  CC environment
>> >> variable.
>> >>
>> >>
>> >>
>> >> --On Thursday, November 07, 2002 16:34:12 -0500 Tom Lane
>> >> <tgl@sss.pgh.pa.us> wrote:
>> >>
>> >> > Larry Rosenman <ler@lerctr.org> writes:
>> >> >> It DOES compile out of the box now on 8.0.0(7.1.2) and 7.1.3.
>> >> >> Apparently a  compiler
>> >> >> fix between the 7.1.1b FS and 7.1.2.
>> >> >
>> >> > Well, this is what the REMARKS column is for in the
>> >> > supported-platform list.  Seems we need a comment like "for older
>> >> > compiler versions, you may need to add -Xb to CFLAGS".  Can anyone
>> >> > provide a short and accurate description of when to do this?
>> >> >
>> >> > 			regards, tom lane
>> >>
>> >>
>> >> --
>> >> Larry Rosenman                     http://www.lerctr.org/~ler
>> >> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
>> >> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
>> >>
>> >>
>> >> ---------------------------(end of
>> >> broadcast)--------------------------- TIP 3: if posting/reading
>> >> through Usenet, please send an appropriate subscribe-nomail command
>> >> to majordomo@postgresql.org so that your message can get through to
>> >> the mailing list cleanly
>> >>
>> >
>> > --
>> >   Bruce Momjian                        |  http://candle.pha.pa.us
>> >   pgman@candle.pha.pa.us               |  (610) 359-1001
>> >   +  If your life is a hard drive,     |  13 Roberts Road
>> >   +  Christ can be your backup.        |  Newtown Square, Pennsylvania
>> > 19073
>>
>>
>> --
>> Larry Rosenman                     http://www.lerctr.org/~ler
>> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
>> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 4: Don't 'kill -9' the postmaster
>>
>
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania
> 19073


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Thu, 07 Nov 2002 16:50:26 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a

I am confused about this patch.  I don't see extra_float_digits defined
anywhere.  Am I missing a patch?

---------------------------------------------------------------------------

Pedro M. Ferreira wrote:
> Pedro M. Ferreira wrote:
> > Tom Lane wrote:
> >> Perhaps P_MAXLEN now needs to be (2*(DBL_DIG+2+7)+1), considering
> >> that we'll allow extra_float_digits to be up to 2.  What's it used for?
> > 
> > Yes. I guess so, because it is used in what I think is a memory 
> > allocation function. P_MAXLEN is only used twice:
> <...>
> > 
> > I will do the changes tomorrow and send in the appropriate diff's.
> 
> Ok. Its done now.
> Only one file changed: src/backend/utils/adt/geo_ops.c
> 
> All the geometric types should now account for float_extra_digits on output.
> 
> A diff -u is attached.
> 
> Best reagards,
> Pedro
> 
> > Regards,
> > Pedro
> > 
> > 
> > ---------------------------(end of broadcast)---------------------------
> > TIP 2: you can get off all lists at once with the unregister command
> >    (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> > 
> > 
> 
> 
> -- 
> ----------------------------------------------------------------------
> Pedro Miguel Frazao Fernandes Ferreira
> Universidade do Algarve
> Faculdade de Ciencias e Tecnologia
> Campus de Gambelas
> 8000-117 Faro
> Portugal
> Tel./Fax:  (+351) 289 800950 / 289 819403
> http://w3.ualg.pt/~pfrazao

> --- old/postgresql-7.2.1/src/backend/utils/adt/geo_ops.c	Mon Nov  4 12:01:39 2002
> +++ postgresql-7.2.1/src/backend/utils/adt/geo_ops.c	Tue Nov  5 10:47:56 2002
> @@ -80,11 +80,11 @@
>  #define RDELIM_C		'>'
>  
>  /* Maximum number of output digits printed */
> -#define P_MAXDIG DBL_DIG
> -#define P_MAXLEN (2*(P_MAXDIG+7)+1)
> -
> -static int	digits8 = P_MAXDIG;
> +/* ...+2+7 : 2 accounts for extra_float_digits max value */
> +#define P_MAXLEN (2*(DBL_DIG+2+7)+1)
>  
> +/* Extra digits in float output formatting (in float.c) */
> +extern int extra_float_digits;
>  
>  /*
>   * Geometric data types are composed of points.
> @@ -139,7 +139,12 @@
>  static int
>  single_encode(float8 x, char *str)
>  {
> -	sprintf(str, "%.*g", digits8, x);
> +	int	ndig = DBL_DIG + extra_float_digits;
> +
> +	if (ndig < 1)
> +		ndig=1;
> +
> +	sprintf(str, "%.*g", ndig, x);
>  	return TRUE;
>  }	/* single_encode() */
>  
> @@ -190,7 +195,12 @@
>  static int
>  pair_encode(float8 x, float8 y, char *str)
>  {
> -	sprintf(str, "%.*g,%.*g", digits8, x, digits8, y);
> +	int	ndig = DBL_DIG + extra_float_digits;
> +
> +	if (ndig < 1)
> +		ndig=1;
> +
> +	sprintf(str, "%.*g,%.*g", ndig, x, ndig, y);
>  	return TRUE;
>  }
>  
> @@ -974,7 +984,7 @@
>  #endif
>  #ifdef GEODEBUG
>  		printf("line_construct_pts- line is neither vertical nor horizontal (diffs x=%.*g, y=%.*g\n",
> -			   digits8, (pt2->x - pt1->x), digits8, (pt2->y - pt1->y));
> +			   DBL_DIG, (pt2->x - pt1->x), DBL_DIG, (pt2->y - pt1->y));
>  #endif
>  	}
>  }
> @@ -1180,8 +1190,8 @@
>  
>  #ifdef GEODEBUG
>  	printf("line_interpt- lines are A=%.*g, B=%.*g, C=%.*g, A=%.*g, B=%.*g, C=%.*g\n",
> -		   digits8, l1->A, digits8, l1->B, digits8, l1->C, digits8, l2->A, digits8, l2->B, digits8, l2->C);
> -	printf("line_interpt- lines intersect at (%.*g,%.*g)\n", digits8, x, digits8, y);
> +		   DBL_DIG, l1->A, DBL_DIG, l1->B, DBL_DIG, l1->C, DBL_DIG, l2->A, DBL_DIG, l2->B, DBL_DIG, l2->C);
> +	printf("line_interpt- lines intersect at (%.*g,%.*g)\n", DBL_DIG, x, DBL_DIG, y);
>  #endif
>  
>  	return result;
> @@ -2390,14 +2400,14 @@
>  	p = line_interpt_internal(&tmp, line);
>  #ifdef GEODEBUG
>  	printf("interpt_sl- segment is (%.*g %.*g) (%.*g %.*g)\n",
> -		   digits8, lseg->p[0].x, digits8, lseg->p[0].y, digits8, lseg->p[1].x, digits8, lseg->p[1].y);
> +		   DBL_DIG, lseg->p[0].x, DBL_DIG, lseg->p[0].y, DBL_DIG, lseg->p[1].x, DBL_DIG, lseg->p[1].y);
>  	printf("interpt_sl- segment becomes line A=%.*g B=%.*g C=%.*g\n",
> -		   digits8, tmp.A, digits8, tmp.B, digits8, tmp.C);
> +		   DBL_DIG, tmp.A, DBL_DIG, tmp.B, DBL_DIG, tmp.C);
>  #endif
>  	if (PointerIsValid(p))
>  	{
>  #ifdef GEODEBUG
> -		printf("interpt_sl- intersection point is (%.*g %.*g)\n", digits8, p->x, digits8, p->y);
> +		printf("interpt_sl- intersection point is (%.*g %.*g)\n", DBL_DIG, p->x, DBL_DIG, p->y);
>  #endif
>  		if (on_ps_internal(p, lseg))
>  		{

> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
> http://archives.postgresql.org

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 7 Nov 2002 23:50:44 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Float output formatting options

This has been saved for the 7.4 release:

	http:/momjian.postgresql.org/cgi-bin/pgpatches2

I will not apply the _inline_ part.

---------------------------------------------------------------------------

Billy G. Allie wrote:
-- Start of PGP signed section.
> I am including a set of 4 small patches that enable PostgreSQL 7.3b3 to build
> successfully on OpenUnix 8.0.  These same patches should also work for UnixWare
> 7.x.  I will confirm that tomorrow (Nov 7, 2002).
> 
> Here is an explanation of the patches:
> 
> 1. An update of the FAQ_SCO file.
> 
> 2. This patch removes a static declaration of a in-line function in
>    src/backend/utils/sort/tuplesort.c 
> 
> 3. This patch to src/makefiles/Makefile.unixware, together with the patch to
>    src/Makefile.global.in allows any addition library search directories (added
>    with the configure --with-libraries option) to be added to the rpath option 
>    sent to the linker.  The use of a different variable to pass the addition 
>    search paths was necessary to avoid a circular reference to LDFLAGS.
> 
> 4. This patch creates the variable (trpath) used by the patch to 
>    Makefile.unixware.  This patch would also be for other platforms that would 
>    have to add the additional library search paths to  the rpath linker option.
>    See Makefile.unixware for an example of how to do this.
> 
> After applying these patches, PostgreSQL successfully compiled on OpenUnix 8 
> and it passed all the regression tests.
> 

Content-Description: ou8.patch.20021106

[ Attachment, skipping... ]

> ____       | Billy G. Allie    | Domain....: Bill.Allie@mug.org
> |  /|      | 7436 Hartwell     | MSN.......: B_G_Allie@email.msn.com
> |-/-|----- | Dearborn, MI 48126|
> |/  |LLIE  | (313) 582-1540    |
-- End of PGP section, PGP failed!

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 7 Nov 2002 23:57:31 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a patch.
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I am confused about this patch.  I don't see extra_float_digits defined
> anywhere.  Am I missing a patch?

Evidently.  I have the patch and was planning to apply it myself as soon
as Pedro does something with the geometry types...

			regards, tom lane

Fri, 08 Nov 2002 00:20:51 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Float output formatting options 
HI All,

Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> 
>>I am confused about this patch.  I don't see extra_float_digits defined
>>anywhere.  Am I missing a patch  ?

Probably!  :)

The definition of extra_float_digits was made in float.c and the diff was sent 
to the list and Tom before. I think its the one Tom refers below.
There were diff's to four files:

src/backend/utils/adt/float.c
src/backend/utils/misc/guc.c
src/bin/psql/tab-complete.c
src/backend/utils/misc/postgresql.conf.sample

> 
> 
> Evidently.  I have the patch and was planning to apply it myself as soon
> as Pedro does something with the geometry types...

Lots of mail in the list!   ;)
Its done. The diff file to src/backend/utils/adt/geo_ops.c (to handle the 
gemetry types) was sent some days ago. Its the one in the message where Bruce 
says that extra_float_digits is not defined anywhere.

If you want I can send in the diff's again.

Best regards,
Pedro
> 
> 			regards, tom lane


-- 
----------------------------------------------------------------------
Pedro Miguel Frazao Fernandes Ferreira
Universidade do Algarve
Faculdade de Ciencias e Tecnologia
Campus de Gambelas
8000-117 Faro
Portugal
Tel./Fax:  (+351) 289 800950 / 289 819403
http://w3.ualg.pt/~pfrazao


Fri, 08 Nov 2002 10:20:39 +0000
"Pedro M. Frazao F. Ferreira" <pfrazao@ualg.pt>
Re: Float output formatting options
Many thanks Larry,

Sorry I had omited the -Xb (no secret dance) in my report...

On Thu, 7 Nov 2002, Larry Rosenman wrote:

> Date: Thu, 07 Nov 2002 15:17:28 -0600
> From: Larry Rosenman <ler@lerctr.org>
> To: Peter Eisentraut <peter_e@gmx.net>, Olivier PRENANT <ohp@pyrenet.fr>
> Cc: Tom Lane <tgl@sss.pgh.pa.us>, Billy G. Allie <Bill.Allie@mug.org>,
>      pgsql-ports@postgresql.org, pgsql-hackers@postgresql.org
> Subject: Re: [PORTS] [HACKERS] PostgreSQL supported platform report and a 
> 
> 
> 
> --On Thursday, November 07, 2002 22:21:56 +0100 Peter Eisentraut 
> <peter_e@gmx.net> wrote:
> 
> > Olivier PRENANT writes:
> >
> >> I don't mind having CFLAGS=-Xb  though, done it for php already...
> >> By everyone says this should go off so..
> >
> > The idea of the platform testing is not to determine whether you can
> > compile PostgreSQL after performing a secret dance.  If it doesn't compile
> > with the default options, please don't report it as supported.
> It DOES compile out of the box now on 8.0.0(7.1.2) and 7.1.3.  Apparently a 
> compiler
> fix between the 7.1.1b FS and 7.1.2.
> 
> The -Xb switch is NOT a secret dance.  It's needed for LOTS of open source 
> stuff.
> 
> See the discussion from the Caldera folks last week.
> 
> Tom's fix fixed the defaults for 7.1.2+
> 
> 
> >
> > --
> > Peter Eisentraut   peter_e@gmx.net
> 
> 
> 

-- 
Olivier PRENANT         	Tel:	+33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou           +33-5-61-50-97-01 (Fax)
31190 AUTERIVE                      +33-6-07-63-80-64 (GSM)
FRANCE                      Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)


Fri, 8 Nov 2002 12:57:28 +0100 (MET)
Olivier PRENANT <ohp@pyrenet.fr>
Re: [HACKERS] PostgreSQL supported platform report and a 
Here is diff.  Please let me know if you need better wording.



--On Thursday, November 07, 2002 16:50:26 -0600 Larry Rosenman 
<ler@lerctr.org> wrote:

> OK, I'll try and do up a diff to B5's tonite (probably late, my
> Daughter's elementary school honors choir has a performance tonite).
>
> LER
>
>
> --On Thursday, November 07, 2002 17:44:37 -0500 Bruce Momjian
> <pgman@candle.pha.pa.us> wrote:
>
>>
>> His updates deal only with the LDLIBRARY path issue, which I think we
>> are keeping for 7.4:
>>
>> *** ./doc/FAQ_SCO.orig  Wed Nov  6 21:35:46 2002
>> --- ./doc/FAQ_SCO       Wed Nov  6 21:40:44 2002
>> ***************
>> *** 71,76 ****
>> --- 71,79 ----
>>
>>   configure --with-libs=/usr/local/lib --with-includes=/usr/local/include
>>
>> + You will also need to set LD_LIBRARY_PATH to '/usr/local/lib' (or add
>> it to + LD_LIBRARY_PATH, if LD_LIBRARY_PATH already exists) before
>> running config- + ure or the test for readline will fail.
>>
>>
>> -------------------------------------------------------------------------
>> --
>>
>> Larry Rosenman wrote:
>>> With or withou Billie's update(s)?
>>>
>>> (I haven't looked at them)....
>>>
>>> LER
>>>
>>>
>>> --On Thursday, November 07, 2002 17:08:04 -0500 Bruce Momjian
>>> <pgman@candle.pha.pa.us> wrote:
>>>
>>> >
>>> > We do point to the FAQ_SCO file for specifics.  Would you send a diff
>>> > for that file?
>>> >
>>> > ----------------------------------------------------------------------
>>> > --- --
>>> >
>>> > Larry Rosenman wrote:
>>> >> For compilers earlier than the one released with OpenUNIX
>>> >> 8.0.0(UnixWare 7.1.2), Including
>>> >> the 7.1.1b Feature Supplement, you may need to specify -Xb in CFLAGS
>>> >> or the  CC environment
>>> >> variable.
>>> >>
>>> >>
>>> >>
>>> >> --On Thursday, November 07, 2002 16:34:12 -0500 Tom Lane
>>> >> <tgl@sss.pgh.pa.us> wrote:
>>> >>
>>> >> > Larry Rosenman <ler@lerctr.org> writes:
>>> >> >> It DOES compile out of the box now on 8.0.0(7.1.2) and 7.1.3.
>>> >> >> Apparently a  compiler
>>> >> >> fix between the 7.1.1b FS and 7.1.2.
>>> >> >
>>> >> > Well, this is what the REMARKS column is for in the
>>> >> > supported-platform list.  Seems we need a comment like "for older
>>> >> > compiler versions, you may need to add -Xb to CFLAGS".  Can anyone
>>> >> > provide a short and accurate description of when to do this?
>>> >> >
>>> >> > 			regards, tom lane
>>> >>
>>> >>
>>> >> --
>>> >> Larry Rosenman                     http://www.lerctr.org/~ler
>>> >> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
>>> >> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
>>> >>
>>> >>
>>> >> ---------------------------(end of
>>> >> broadcast)--------------------------- TIP 3: if posting/reading
>>> >> through Usenet, please send an appropriate subscribe-nomail command
>>> >> to majordomo@postgresql.org so that your message can get through to
>>> >> the mailing list cleanly
>>> >>
>>> >
>>> > --
>>> >   Bruce Momjian                        |  http://candle.pha.pa.us
>>> >   pgman@candle.pha.pa.us               |  (610) 359-1001
>>> >   +  If your life is a hard drive,     |  13 Roberts Road
>>> >   +  Christ can be your backup.        |  Newtown Square, Pennsylvania
>>> > 19073
>>>
>>>
>>> --
>>> Larry Rosenman                     http://www.lerctr.org/~ler
>>> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
>>> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
>>>
>>>
>>> ---------------------------(end of broadcast)---------------------------
>>> TIP 4: Don't 'kill -9' the postmaster
>>>
>>
>> --
>>   Bruce Momjian                        |  http://candle.pha.pa.us
>>   pgman@candle.pha.pa.us               |  (610) 359-1001
>>   +  If your life is a hard drive,     |  13 Roberts Road
>>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania
>> 19073
>
>
> --
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
>


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
Fri, 08 Nov 2002 06:43:15 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a
"Pedro M. Frazao F. Ferreira" <pfrazao@ualg.pt> writes:
> Tom Lane wrote:
>> Evidently.  I have the patch and was planning to apply it myself as soon
>> as Pedro does something with the geometry types...

> Its done.

So it is, dunno what I was thinking.  Will work on getting it applied.

			regards, tom lane

Fri, 08 Nov 2002 09:43:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Float output formatting options 
Larry Rosenman <ler@lerctr.org> writes:
> +For compilers earlier than the one released with OpenUNIX 8.0.0(UnixWare
> +7.1.2), Including the 7.1.1b Feature Supplement, you may need to specify -Xb
> + in CFLAGS or the  CC environment variable.  The indication of this is an
> +error in compiling tuplesort.c referencing inline parameters. 

s/inline parameters/inline functions/.  Otherwise looks good.

			regards, tom lane

Fri, 08 Nov 2002 09:45:44 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a 
Do y'all want a new diff, or can you deal with it when you do the patch?

LER


--On Friday, November 08, 2002 09:45:44 -0500 Tom Lane <tgl@sss.pgh.pa.us> 
wrote:

> Larry Rosenman <ler@lerctr.org> writes:
>> +For compilers earlier than the one released with OpenUNIX 8.0.0(UnixWare
>> +7.1.2), Including the 7.1.1b Feature Supplement, you may need to
>> specify -Xb + in CFLAGS or the  CC environment variable.  The indication
>> of this is an +error in compiling tuplesort.c referencing inline
>> parameters.
>
> s/inline parameters/inline functions/.  Otherwise looks good.
>
> 			regards, tom lane


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Fri, 08 Nov 2002 09:07:51 -0600
Larry Rosenman <ler@lerctr.org>
Re: [HACKERS] PostgreSQL supported platform report and a 

I will deal with it when I apply.

---------------------------------------------------------------------------

Larry Rosenman wrote:
> Do y'all want a new diff, or can you deal with it when you do the patch?
> 
> LER
> 
> 
> --On Friday, November 08, 2002 09:45:44 -0500 Tom Lane <tgl@sss.pgh.pa.us> 
> wrote:
> 
> > Larry Rosenman <ler@lerctr.org> writes:
> >> +For compilers earlier than the one released with OpenUNIX 8.0.0(UnixWare
> >> +7.1.2), Including the 7.1.1b Feature Supplement, you may need to
> >> specify -Xb + in CFLAGS or the  CC environment variable.  The indication
> >> of this is an +error in compiling tuplesort.c referencing inline
> >> parameters.
> >
> > s/inline parameters/inline functions/.  Otherwise looks good.
> >
> > 			regards, tom lane
> 
> 
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
> 
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 8 Nov 2002 11:40:58 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a

Patch applied to 7.3 and CVS, with Tom's correction.

---------------------------------------------------------------------------

Larry Rosenman wrote:
> Here is diff.  Please let me know if you need better wording.
> 
> 
> 
> --On Thursday, November 07, 2002 16:50:26 -0600 Larry Rosenman 
> <ler@lerctr.org> wrote:
> 
> > OK, I'll try and do up a diff to B5's tonite (probably late, my
> > Daughter's elementary school honors choir has a performance tonite).
> >
> > LER
> >
> >
> > --On Thursday, November 07, 2002 17:44:37 -0500 Bruce Momjian
> > <pgman@candle.pha.pa.us> wrote:
> >
> >>
> >> His updates deal only with the LDLIBRARY path issue, which I think we
> >> are keeping for 7.4:
> >>
> >> *** ./doc/FAQ_SCO.orig  Wed Nov  6 21:35:46 2002
> >> --- ./doc/FAQ_SCO       Wed Nov  6 21:40:44 2002
> >> ***************
> >> *** 71,76 ****
> >> --- 71,79 ----
> >>
> >>   configure --with-libs=/usr/local/lib --with-includes=/usr/local/include
> >>
> >> + You will also need to set LD_LIBRARY_PATH to '/usr/local/lib' (or add
> >> it to + LD_LIBRARY_PATH, if LD_LIBRARY_PATH already exists) before
> >> running config- + ure or the test for readline will fail.
> >>
> >>
> >> -------------------------------------------------------------------------
> >> --
> >>
> >> Larry Rosenman wrote:
> >>> With or withou Billie's update(s)?
> >>>
> >>> (I haven't looked at them)....
> >>>
> >>> LER
> >>>
> >>>
> >>> --On Thursday, November 07, 2002 17:08:04 -0500 Bruce Momjian
> >>> <pgman@candle.pha.pa.us> wrote:
> >>>
> >>> >
> >>> > We do point to the FAQ_SCO file for specifics.  Would you send a diff
> >>> > for that file?
> >>> >
> >>> > ----------------------------------------------------------------------
> >>> > --- --
> >>> >
> >>> > Larry Rosenman wrote:
> >>> >> For compilers earlier than the one released with OpenUNIX
> >>> >> 8.0.0(UnixWare 7.1.2), Including
> >>> >> the 7.1.1b Feature Supplement, you may need to specify -Xb in CFLAGS
> >>> >> or the  CC environment
> >>> >> variable.
> >>> >>
> >>> >>
> >>> >>
> >>> >> --On Thursday, November 07, 2002 16:34:12 -0500 Tom Lane
> >>> >> <tgl@sss.pgh.pa.us> wrote:
> >>> >>
> >>> >> > Larry Rosenman <ler@lerctr.org> writes:
> >>> >> >> It DOES compile out of the box now on 8.0.0(7.1.2) and 7.1.3.
> >>> >> >> Apparently a  compiler
> >>> >> >> fix between the 7.1.1b FS and 7.1.2.
> >>> >> >
> >>> >> > Well, this is what the REMARKS column is for in the
> >>> >> > supported-platform list.  Seems we need a comment like "for older
> >>> >> > compiler versions, you may need to add -Xb to CFLAGS".  Can anyone
> >>> >> > provide a short and accurate description of when to do this?
> >>> >> >
> >>> >> > 			regards, tom lane
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Larry Rosenman                     http://www.lerctr.org/~ler
> >>> >> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
> >>> >> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
> >>> >>
> >>> >>
> >>> >> ---------------------------(end of
> >>> >> broadcast)--------------------------- TIP 3: if posting/reading
> >>> >> through Usenet, please send an appropriate subscribe-nomail command
> >>> >> to majordomo@postgresql.org so that your message can get through to
> >>> >> the mailing list cleanly
> >>> >>
> >>> >
> >>> > --
> >>> >   Bruce Momjian                        |  http://candle.pha.pa.us
> >>> >   pgman@candle.pha.pa.us               |  (610) 359-1001
> >>> >   +  If your life is a hard drive,     |  13 Roberts Road
> >>> >   +  Christ can be your backup.        |  Newtown Square, Pennsylvania
> >>> > 19073
> >>>
> >>>
> >>> --
> >>> Larry Rosenman                     http://www.lerctr.org/~ler
> >>> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
> >>> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
> >>>
> >>>
> >>> ---------------------------(end of broadcast)---------------------------
> >>> TIP 4: Don't 'kill -9' the postmaster
> >>>
> >>
> >> --
> >>   Bruce Momjian                        |  http://candle.pha.pa.us
> >>   pgman@candle.pha.pa.us               |  (610) 359-1001
> >>   +  If your life is a hard drive,     |  13 Roberts Road
> >>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania
> >> 19073
> >
> >
> > --
> > Larry Rosenman                     http://www.lerctr.org/~ler
> > Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
> > US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 3: if posting/reading through Usenet, please send an appropriate
> > subscribe-nomail command to majordomo@postgresql.org so that your
> > message can get through to the mailing list cleanly
> >
> 
> 
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

[ Attachment, skipping... ]

> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 8 Nov 2002 11:49:13 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] PostgreSQL supported platform report and a
"Pedro M. Ferreira" <pfrazao@ualg.pt> writes:
> [ patch for extra_float_digits ]

I've applied this patch along with followup changes to pg_dump (it sets
extra_float_digits to 2 to allow accurate dump/reload) and the geometry
regression test (it sets extra_float_digits to -3).

I find that two geometry 'expected' files are now sufficient to cover
all the platforms I have available to test.  (We'd only need one, if
everyone displayed minus zero as '-0', but some platforms print '0'.)
I tested on HPUX 10.20 (HPPA), Red Hat Linux 8.0 (Intel), Mac OS X 10.2.1
and LinuxPPC (PPC).

I'd be interested to hear results of testing CVS tip (now 7.4devel)
on other platforms.  Does geometry pass cleanly for you?

			regards, tom lane

Fri, 08 Nov 2002 15:33:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Geometry regression tests (was Re: Float output formatting options)

> I'd be interested to hear results of testing CVS tip (now 7.4devel)
> on other platforms.  Does geometry pass cleanly for you?

Yes for NetBSD-1.5.1/i386, where it previously didn't due to processor
specific math libraries on this platform.

Giles



Sat, 09 Nov 2002 13:10:45 +1100
Giles Lean <giles@nemeton.com.au>
Re: Geometry regression tests (was Re: Float output formatting
	options)
Tom Lane wrote:
> "Pedro M. Ferreira" <pfrazao@ualg.pt> writes:
> 
>>[ patch for extra_float_digits ]
> 
> 
> I've applied this patch along with followup changes to pg_dump (it sets
> extra_float_digits to 2 to allow accurate dump/reload) and the geometry
> regression test (it sets extra_float_digits to -3).
> 
> I find that two geometry 'expected' files are now sufficient to cover
> all the platforms I have available to test.  (We'd only need one, if
> everyone displayed minus zero as '-0', but some platforms print '0'.)
> I tested on HPUX 10.20 (HPPA), Red Hat Linux 8.0 (Intel), Mac OS X 10.2.1
> and LinuxPPC (PPC).
> 
> I'd be interested to hear results of testing CVS tip (now 7.4devel)
> on other platforms.  Does geometry pass cleanly for you?

Yes!   :)
All tests passed on a dual AMD Athlon MP with Debian GNU/Linux 3.0 (Woody), 
kernel 2.4.18-5.

Tested with a snapshot downloaded yesterday.

Best regards,
Pedro M. Ferreira


> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster


Tue, 12 Nov 2002 10:35:03 +0000
"Pedro M. Frazao F. Ferreira" <pfrazao@ualg.pt>
Re: Geometry regression tests (was Re: Float output formatting
Tom Lane writes:

> I find that two geometry 'expected' files are now sufficient to cover
> all the platforms I have available to test.  (We'd only need one, if
> everyone displayed minus zero as '-0', but some platforms print '0'.)

Judging from the platforms affected by this, I would suspect that this is
a (mis-)feature of the snprintf() implementation rather than compiler or
processor.  Would it make sense to provide a fixed version of snprintf()
and get rid of these differences?

-- 
Peter Eisentraut   peter_e@gmx.net


Fri, 15 Nov 2002 18:35:40 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: Geometry regression tests (was Re: Float output
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> I find that two geometry 'expected' files are now sufficient to cover
>> all the platforms I have available to test.  (We'd only need one, if
>> everyone displayed minus zero as '-0', but some platforms print '0'.)

> Judging from the platforms affected by this, I would suspect that this is
> a (mis-)feature of the snprintf() implementation rather than compiler or
> processor.  Would it make sense to provide a fixed version of snprintf()
> and get rid of these differences?

Certainly it's a library issue on most of these platforms --- AFAIK,
all these machines have IEEE-compliant float hardware, so it must be
sprintf's fault and not a matter of not getting the minus zero in the
first place.

I wouldn't want to write a float converter from scratch, but maybe we
could add a few lines in src/port/snprintf.c to patch up a wrong result?

			regards, tom lane

Fri, 15 Nov 2002 12:46:12 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Geometry regression tests (was Re: Float output formatting
	options)


> -----Original Message-----
> From: Marc G. Fournier [mailto:scrappy@hub.org] 
> Sent: 03 December 2002 19:12
> To: Bruce Momjian
> Cc: PostgreSQL-development
> Subject: Re: [HACKERS] [GENERAL] PostgreSQL Global 
> Development Group Announces
> 
> 
> On Thu, 28 Nov 2002, Bruce Momjian wrote:
> 
> > Wow, this sounds great.
> >
> > Where can I get a copy?  Why would anyone use anything else?  ;-)
> 
> Well, if you read the announcement in its entirety, you would have
> noticed:
> 
> "Source for this release is available at:
>     http://advocacy.postgresql.org/download/
>

I could have sworn we used to have a bunch of ftp mirrors for downloads.
Come to think of it I rewrote/stole a load of Vince's PHP code to allow
you to select one from the portal recently. Are we not using them
anymore?

:-)

Regards, Dave.

Tue, 3 Dec 2002 20:29:48 -0000
"Dave Page" <dpage@vale-housing.co.uk>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Tue, 3 Dec 2002, Dave Page wrote:

>
>
> > -----Original Message-----
> > From: Marc G. Fournier [mailto:scrappy@hub.org]
> > Sent: 03 December 2002 19:12
> > To: Bruce Momjian
> > Cc: PostgreSQL-development
> > Subject: Re: [HACKERS] [GENERAL] PostgreSQL Global
> > Development Group Announces
> >
> >
> > On Thu, 28 Nov 2002, Bruce Momjian wrote:
> >
> > > Wow, this sounds great.
> > >
> > > Where can I get a copy?  Why would anyone use anything else?  ;-)
> >
> > Well, if you read the announcement in its entirety, you would have
> > noticed:
> >
> > "Source for this release is available at:
> >     http://advocacy.postgresql.org/download/
> >
>
> I could have sworn we used to have a bunch of ftp mirrors for downloads.
> Come to think of it I rewrote/stole a load of Vince's PHP code to allow
> you to select one from the portal recently. Are we not using them
> anymore?

Haven't you been paying attention?  There's this new advocacy and suit
marketing thing going on that makes all of that irrelevant.  It's just
there for show now.

:)

Vince.
-- 
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Tue, 3 Dec 2002 15:48:16 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Tue, 3 Dec 2002, Dave Page wrote:

> I could have sworn we used to have a bunch of ftp mirrors for downloads.
> Come to think of it I rewrote/stole a load of Vince's PHP code to allow
> you to select one from the portal recently. Are we not using them
> anymore?

Yup, as with doing anything for the firs ttime, the press release itself
had its 'bugs' ... considering how many times Josh asked for comments on
it, I'm surprised that nobody picked up on it *shrug*

We are looking at some improvements to the download stuff ... Greg(?)
suggested a layout that I really liked for a web based version that would
have to tie into the main mirror database ... one that provided a wee bit
more information then just the directory listings ... but, with that
thought, isn't there a file you can put into an ftp directory that, when
you web into that directory, i gives  you the listings with various
comments, or is that just using the .messages file?



Tue, 3 Dec 2002 17:17:15 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Tue, 3 Dec 2002, Vince Vielhaber wrote:

> Haven't you been paying attention?  There's this new advocacy and suit
> marketing thing going on that makes all of that irrelevant.  It's just
> there for show now.



Tue, 3 Dec 2002 17:18:38 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Tue, 3 Dec 2002, Marc G. Fournier wrote:

> On Tue, 3 Dec 2002, Dave Page wrote:
>
> > I could have sworn we used to have a bunch of ftp mirrors for downloads.
> > Come to think of it I rewrote/stole a load of Vince's PHP code to allow
> > you to select one from the portal recently. Are we not using them
> > anymore?
>
> Yup, as with doing anything for the firs ttime, the press release itself
> had its 'bugs' ... considering how many times Josh asked for comments on
> it, I'm surprised that nobody picked up on it *shrug*

I understood it was intentional so comments wouldn't have done any good.

> We are looking at some improvements to the download stuff ... Greg(?)
> suggested a layout that I really liked for a web based version that would
> have to tie into the main mirror database ... one that provided a wee bit
> more information then just the directory listings ... but, with that
> thought, isn't there a file you can put into an ftp directory that, when
> you web into that directory, i gives  you the listings with various
> comments, or is that just using the .messages file?

All of them I've seen had an index.html in it.

Vince.
-- 
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Tue, 3 Dec 2002 16:40:15 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Dave Page wrote:
<snip>
> I could have sworn we used to have a bunch of ftp mirrors for downloads.
> Come to think of it I rewrote/stole a load of Vince's PHP code to allow
> you to select one from the portal recently. Are we not using them
> anymore?

Of course we are, it's just that we're also trying to direct people to 
the Advocacy site where there is a lot more info, in a lot more languages.

The only reason for the download page not having a list of mirrors is 
due to not having done it yet.

:-)

Regards and best wishes,

Justin Clift


> :-)
> 
> Regards, Dave.
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org


-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Wed, 04 Dec 2002 10:14:47 +1100
Justin Clift <justin@postgresql.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Marc G. Fournier writes:

> Yup, as with doing anything for the firs ttime, the press release itself
> had its 'bugs' ... considering how many times Josh asked for comments on
> it, I'm surprised that nobody picked up on it *shrug*

And how should we have guessed that release management is now done by the
"advocacy" group?  While you're out advocating, don't forget the existing
users.

-- 
Peter Eisentraut   peter_e@gmx.net


Wed, 4 Dec 2002 00:29:23 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Peter Eisentraut wrote:
> Marc G. Fournier writes:
> 
> 
>>Yup, as with doing anything for the firs ttime, the press release itself
>>had its 'bugs' ... considering how many times Josh asked for comments on
>>it, I'm surprised that nobody picked up on it *shrug*
> 
> 
> And how should we have guessed that release management is now done by the
> "advocacy" group?  While you're out advocating, don't forget the existing
> users.

Sorry Peter.

Regards and best wishes,

Justin Clift

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Wed, 04 Dec 2002 10:29:34 +1100
Justin Clift <justin@postgresql.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Peter Eisentraut wrote:
> Justin Clift writes:
> 
> 
>>Of course we are, it's just that we're also trying to direct people to
>>the Advocacy site where there is a lot more info, in a lot more languages.
> 
> 
> Why don't we just shut down the regular web site.  Clearly it's not
> considered adequate anymore.

Well, qe're trying to move the new "portal" side of things into place 
(presently at wwwdevel.postgresql.org), so that all of the different 
PostgreSQL pieces are more easily accessible.

Regards and best wishes,

Justin Clift


-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Wed, 04 Dec 2002 10:31:51 +1100
Justin Clift <justin@postgresql.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Justin Clift writes:

> Of course we are, it's just that we're also trying to direct people to
> the Advocacy site where there is a lot more info, in a lot more languages.

Why don't we just shut down the regular web site.  Clearly it's not
considered adequate anymore.

-- 
Peter Eisentraut   peter_e@gmx.net


Wed, 4 Dec 2002 00:33:44 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Tue, 3 Dec 2002, Vince Vielhaber wrote:

> > Yup, as with doing anything for the firs ttime, the press release itself
> > had its 'bugs' ... considering how many times Josh asked for comments on
> > it, I'm surprised that nobody picked up on it *shrug*
>
> I understood it was intentional so comments wouldn't have done any good.

Anything is only as intentional as nobody making constructive critisms of
it ... ewwww, that was major bad english ... not part of solution, you are
part of problem sort of thing...



Wed, 4 Dec 2002 09:32:27 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Wed, 4 Dec 2002, Peter Eisentraut wrote:

> Marc G. Fournier writes:
>
> > Yup, as with doing anything for the firs ttime, the press release itself
> > had its 'bugs' ... considering how many times Josh asked for comments on
> > it, I'm surprised that nobody picked up on it *shrug*
>
> And how should we have guessed that release management is now done by the
> "advocacy" group?  While you're out advocating, don't forget the existing
> users.

It isn't, but those working on -advocacy were asked to help come up with a
stronger release *announcement* then we've had in the past ...



Wed, 4 Dec 2002 09:33:24 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Wed, 4 Dec 2002, Justin Clift wrote:

> Dave Page wrote:
> <snip>
> > I could have sworn we used to have a bunch of ftp mirrors for downloads.
> > Come to think of it I rewrote/stole a load of Vince's PHP code to allow
> > you to select one from the portal recently. Are we not using them
> > anymore?
>
> Of course we are, it's just that we're also trying to direct people to
> the Advocacy site where there is a lot more info, in a lot more languages.
>
> The only reason for the download page not having a list of mirrors is
> due to not having done it yet.

So as to not recreate the wheel, or, at least, get the wheel properly
rolling, can we get that download page redirected to the one that does
list the mirrors? :)

I liked Greg(?)'s ideas, but I don't see it as being implemented overnight
:)



Wed, 4 Dec 2002 09:35:29 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Wed, 4 Dec 2002, Peter Eisentraut wrote:

> Justin Clift writes:
>
> > Of course we are, it's just that we're also trying to direct people to
> > the Advocacy site where there is a lot more info, in a lot more languages.
>
> Why don't we just shut down the regular web site.  Clearly it's not
> considered adequate anymore.

As of yet, the new portal isn't ready yet ... and the adequacy of the
existing site isn't so much a problem, but maintainability of it ...
according to Vince, trying to add anything to it is virtually impossible
:(




Wed, 4 Dec 2002 09:37:33 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Wed, 4 Dec 2002, Marc G. Fournier wrote:

> On Tue, 3 Dec 2002, Vince Vielhaber wrote:
>
> > > Yup, as with doing anything for the firs ttime, the press release itself
> > > had its 'bugs' ... considering how many times Josh asked for comments on
> > > it, I'm surprised that nobody picked up on it *shrug*
> >
> > I understood it was intentional so comments wouldn't have done any good.
>
> Anything is only as intentional as nobody making constructive critisms of
> it ... ewwww, that was major bad english ... not part of solution, you are
> part of problem sort of thing...

That may be how you understood it, but not how I understood it.  There
appears to be an incremental takeover occurring.

Vince.
-- 
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Wed, 4 Dec 2002 08:41:43 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Marc G. Fournier wrote:
<snip>
> So as to not recreate the wheel, or, at least, get the wheel properly
> rolling, can we get that download page redirected to the one that does
> list the mirrors? :)

Yep.

Would the best way to do this be changing the wording to say something like:

"PostgreSQL can be downloaded as source code from any of the many mirror 
sites:"

With a link after it directing to somewhere that gives the list.  The 
present "www.postgresql.org" with the list of mirrors would probably be 
adequate, but it'll need to be a different url than the straight 
"www.postgresql.org" as that's going to change as soon as the new portal 
is in place.

Does this sound like a workable approach for now?

Regards and best wishes,

Justin Clift


> I liked Greg(?)'s ideas, but I don't see it as being implemented overnight
> :)
> 
> 


-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Thu, 05 Dec 2002 00:42:11 +1100
Justin Clift <justin@postgresql.org>
Re: [HACKERS] [GENERAL] PostgreSQL Global Development Group Announces
On Wed, 4 Dec 2002, Marc G. Fournier wrote:

> On Wed, 4 Dec 2002, Peter Eisentraut wrote:
>
> > Marc G. Fournier writes:
> >
> > > Yup, as with doing anything for the firs ttime, the press release itself
> > > had its 'bugs' ... considering how many times Josh asked for comments on
> > > it, I'm surprised that nobody picked up on it *shrug*
> >
> > And how should we have guessed that release management is now done by the
> > "advocacy" group?  While you're out advocating, don't forget the existing
> > users.
>
> It isn't, but those working on -advocacy were asked to help come up with a
> stronger release *announcement* then we've had in the past ...

That wasn't stronger, it was fluffier.  It was full of buzzwords that were
masking the actual content.  Are you trying to hide the accomplishments or
promote them?  If you're trying to hide them like in this announcement you
may want to try using this tool:  http://www.dack.com/web/bullshit.html
The stored phrases are much more refined and better paired.

Vince.
-- 
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Wed, 4 Dec 2002 08:47:53 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Wed, 4 Dec 2002, Marc G. Fournier wrote:

> On Wed, 4 Dec 2002, Peter Eisentraut wrote:
>
> > Justin Clift writes:
> >
> > > Of course we are, it's just that we're also trying to direct people to
> > > the Advocacy site where there is a lot more info, in a lot more languages.
> >
> > Why don't we just shut down the regular web site.  Clearly it's not
> > considered adequate anymore.
>
> As of yet, the new portal isn't ready yet ... and the adequacy of the
> existing site isn't so much a problem, but maintainability of it ...
> according to Vince, trying to add anything to it is virtually impossible
> :(

I have a new design for it, now it's just getting the time to implement
it.  It's easy to add to and looks alot nicer.

Vince.
-- 
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Wed, 4 Dec 2002 08:58:23 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Wed, 4 Dec 2002, Vince Vielhaber wrote:

> That wasn't stronger, it was fluffier.  It was full of buzzwords that
> were masking the actual content.  Are you trying to hide the
> accomplishments or promote them?  If you're trying to hide them like in
> this announcement you may want to try using this tool:
> http://www.dack.com/web/bullshit.html The stored phrases are much more
> refined and better paired.

Bookmark'd for the next release ... thanks for the suggestion ...



Wed, 4 Dec 2002 13:22:20 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Wed, 4 Dec 2002, Vince Vielhaber wrote:

> I have a new design for it, now it's just getting the time to implement
> it.  It's easy to add to and looks alot nicer.

Cool, I think the only beef I ever had with it was the way the results
were presented, but loved teh whole annotated aspects ...



Wed, 4 Dec 2002 13:23:32 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Marc G. Fournier wrote:
> On Wed, 4 Dec 2002, Vince Vielhaber wrote:
> 
> > That wasn't stronger, it was fluffier.  It was full of buzzwords that
> > were masking the actual content.  Are you trying to hide the
> > accomplishments or promote them?  If you're trying to hide them like in
> > this announcement you may want to try using this tool:
> > http://www.dack.com/web/bullshit.html The stored phrases are much more
> > refined and better paired.
> 
> Bookmark'd for the next release ... thanks for the suggestion ...

I was hoping for something that would take existing text and *Bullshit*
it.  Bummer.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 4 Dec 2002 12:57:05 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Wed, 4 Dec 2002, Bruce Momjian wrote:

> Marc G. Fournier wrote:
> > On Wed, 4 Dec 2002, Vince Vielhaber wrote:
> >
> > > That wasn't stronger, it was fluffier.  It was full of buzzwords that
> > > were masking the actual content.  Are you trying to hide the
> > > accomplishments or promote them?  If you're trying to hide them like in
> > > this announcement you may want to try using this tool:
> > > http://www.dack.com/web/bullshit.html The stored phrases are much more
> > > refined and better paired.
> >
> > Bookmark'd for the next release ... thanks for the suggestion ...
>
> I was hoping for something that would take existing text and *Bullshit*
> it.  Bummer.

Click on it a few times.  You'll get the text you need.  I've actually
used it for real things with excellent results (I'm not going to
elaborate).

Vince.
-- 
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Wed, 4 Dec 2002 13:01:19 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Wed, 4 Dec 2002, Bruce Momjian wrote:

> Marc G. Fournier wrote:
> > On Wed, 4 Dec 2002, Vince Vielhaber wrote:
> >
> > > That wasn't stronger, it was fluffier.  It was full of buzzwords that
> > > were masking the actual content.  Are you trying to hide the
> > > accomplishments or promote them?  If you're trying to hide them like in
> > > this announcement you may want to try using this tool:
> > > http://www.dack.com/web/bullshit.html The stored phrases are much more
> > > refined and better paired.
> >
> > Bookmark'd for the next release ... thanks for the suggestion ...
>
> I was hoping for something that would take existing text and *Bullshit*
> it.  Bummer.

No, but I figure that at least it will give me a good site to give me BS
fodder from ... man, just wait for the next release announcement :)



Wed, 4 Dec 2002 16:03:59 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Marc G. Fournier writes:

> It isn't, but those working on -advocacy were asked to help come up with a
> stronger release *announcement* then we've had in the past ...

Consider that a failed experiment.  PostgreSQL is driven by the
development group and, to some extent, by the existing user base.  The
last thing we need is a marketing department in that mix.

-- 
Peter Eisentraut   peter_e@gmx.net


Thu, 5 Dec 2002 01:01:16 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Peter Eisentraut wrote:
> Marc G. Fournier writes:
> 
> > It isn't, but those working on -advocacy were asked to help come up with a
> > stronger release *announcement* then we've had in the past ...
> 
> Consider that a failed experiment.  PostgreSQL is driven by the
> development group and, to some extent, by the existing user base.  The
> last thing we need is a marketing department in that mix.

Peter, I understand your perspective, but I think you are in the
minority on this one.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 4 Dec 2002 19:23:52 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [GENERAL] PostgreSQL Global Development Group Announces
> > It isn't, but those working on -advocacy were asked to help come up with
a
> > stronger release *announcement* then we've had in the past ...
>
> Consider that a failed experiment.  PostgreSQL is driven by the
> development group and, to some extent, by the existing user base.  The
> last thing we need is a marketing department in that mix.

Ummm...I disagree.  Lack of marketing is one of Postgres's major problems.
Particularly when you compare against similar efforts from MySQL, Oracle,
etc.

Chris


Wed, 4 Dec 2002 17:48:37 -0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: [GENERAL] PostgreSQL Global Development Group Announces
> > > It isn't, but those working on -advocacy were asked to help come up with
> a
> > > stronger release *announcement* then we've had in the past ...
> >
> > Consider that a failed experiment.  PostgreSQL is driven by the
> > development group and, to some extent, by the existing user base.  The
> > last thing we need is a marketing department in that mix.
> 
> Ummm...I disagree.  Lack of marketing is one of Postgres's major problems.
> Particularly when you compare against similar efforts from MySQL, Oracle,
> etc.

Yes, indeed.

The _prime_ reason for the fact that MySQL is the "M" in "LAMP" is that there 
is a steady, intent set of efforts going into marketing the "M."  People think 
that MySQL is faster, easier to use and "more standard" than its alternatives, 
and that is certainly the result of marketing.

The /real/ technical merit of MySQL has been that there are some integrated 
tools for ISPs like CPANEL that make it easy for ISPs that don't know 
/anything/ about DBMSes to provide MySQL for their customers.  CPANEL doesn't 
support PostgreSQL, and historically, it has been somewhat more difficult to 
support large numbers of PostgreSQL instances on a web server.  Some of that 
has changed, though CPANEL /still/ doesn't support PostgreSQL.

If any of you consider these "technical" issues to be small and petty, I'm 
afraid I don't /care/.  More importantly, the hundreds of ISPs licensing 
CPANEL don't care.  /They/ are the ones that would need convincing, and I 
don't think there's any real route to convince them that they should be 
pounding down CPANEL's door asking for a PostgreSQL front end and to convince 
them that they have to tell their customers:

  "We sold you MySQL, telling you it was good for you to use.  We were
   wrong, and our new story is that you should convert your databases over
   to use PostgreSQL."

Anyone consider that a likely scenario?  Anyone?

It's fair to say that PostgreSQL doesn't need the likes of the "Database 
HOWTO" that gives a sales job that's so blindly enthusiastic as to be, well, 
blind.

But an organization that has /no/ "marketing department" is at a severe 
disadvantage, like it or not.

It is unfortunate that it is almost impossible to have a marketing group 
without there being some wilful blinders involved; it's vital for there to be 
some technical involvement in the marketing group to pop whatever bubbles they 
grow that are woefully wrong.  But even if it operates with some occasional 
lack of /real/ vision, it's necessary to have a marketing group...
--
(reverse (concatenate 'string "moc.enworbbc@" "sirhc"))
http://cbbrowne.com/info/advocacy.html
Rules of the  Evil Overlord #106. "If my  supreme command center comes
under attack, I will immediately  flee to safety in my prepared escape
pod and  direct the  defenses from  there. I will  not wait  until the
troops break into my inner sanctum to attempt this."
<http://www.eviloverlord.com/>



Wed, 04 Dec 2002 22:43:40 -0500
cbbrowne@cbbrowne.com
Re: [GENERAL] PostgreSQL Global Development Group Announces 
At 05:48 PM 4/12/2002 -0800, Christopher Kings-Lynne wrote:
>Lack of marketing is one of Postgres's major problems.

What are the consequences of the problem?


>Particularly when you compare against similar efforts from MySQL, Oracle,
>etc.

You could even include Microsoft here - they do a lot of database 
marketing. I am not at all sure the fact that a lot of large companies with 
dubious products engage in extensive marketing is a reason for *us* to 
engage in extensive marketing.

We already have a substantial following, and our clients have direct access 
to the developers, so any marketing group is pretty irrelevant for existing 
clients. So the only place I can see for a marketing group is in building 
our market share by bringing in new clients.

If that is what we want, then fine. But I don't want to see any part of the 
development effort distorted or the existing user base inconvenienced in an 
effort to purely gain that market share. I usually associate increased 
marketing with decreased quality, and I think the causality works *both* ways.



----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 03 5330 3172          |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 05 Dec 2002 14:52:56 +1100
Philip Warner <pjw@rhyme.com.au>
Re: [GENERAL] PostgreSQL Global Development Group
[cc: list trimmed]

On Wednesday 04 December 2002 22:52, Philip Warner wrote:
> At 05:48 PM 4/12/2002 -0800, Christopher Kings-Lynne wrote:
> >Lack of marketing is one of Postgres's major problems.

> What are the consequences of the problem?

Actually, lack of easy upgrading is one of PostgreSQL's major problems....

But lack of focused marketing -- truthful, not, as has been said, like the 
'Database HOWTO' -- is a real problem.  It would be nice to increase our 
usage.

> If that is what we want, then fine. But I don't want to see any part of the
> development effort distorted or the existing user base inconvenienced in an
> effort to purely gain that market share. I usually associate increased
> marketing with decreased quality, and I think the causality works *both*
> ways.

ISTM there's a separate, non-code-developer group doing this.  It doesn't seem 
to take away _any_ developer resources to do an advocacy site.

However, I seriously question the need in the long term for our sites to be as 
fractured as they are.  Good grief!  We've got advocacy.postgresql.org, 
techdocs.postgresql.org, odbc.postgresql.org, gborg.postgresql.org, 
developer.postgresql.org, jdbc.postgresql.org, etc.  Oh, and we also have 
www.postgresql.org on the side?  I think not.  Oh, and they are fractured in 
their styles -- really, guys, we need a unified style here.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Wed, 4 Dec 2002 23:22:41 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [GENERAL] PostgreSQL Global Development Group
On Wed, 04 Dec 2002 22:54:37 -0500, Philip Warner wrote:
> At 05:48 PM 4/12/2002 -0800, Christopher Kings-Lynne wrote:
>>Lack of marketing is one of Postgres's major problems.
> 
> What are the consequences of the problem?
> 

One consequence that probably hits home for everyone here is it makes it
extremely hard to make a living working with postgresql.  A quick search on
monster.com gives me 17 jobs mentioning postgresql, with none listed in the
last week. A search on mysql gives me 100 jobs, with 3 filed just today. 
I won't even go into the numbers for Oracle, DB2, and M$. We all have to 
pay the bills and I think we'd like to do it working with postgresql.

>>Particularly when you compare against similar efforts from MySQL,
>>Oracle, etc.
> 
> You could even include Microsoft here - they do a lot of database
> marketing. I am not at all sure the fact that a lot of large companies
> with dubious products engage in extensive marketing is a reason for *us*
> to engage in extensive marketing.
> 

You can't win marketshare on technology alone, so unless you think we
don't need to increase our market share, that is reason enough to do more
marketing.

> We already have a substantial following, and our clients have direct
> access to the developers, so any marketing group is pretty irrelevant
> for existing clients. So the only place I can see for a marketing group
> is in building our market share by bringing in new clients.
> 

Well, my previous employer uses postgresql, but they were under constant
assault from their clients to use oracle or db2.  Technically there was no
reason to switch, but if your choice is switch databases or go out of 
business, there really isn't much choice. 

In the company I work for now we use at least 4 different
database systems.  We could probably switch all of these to postgresql,
but it probably be one heck of a battle to convince people of that. A
simple argument that could be raised is that several of the database
developers use ERWin from computer associates. ERWin's postgresql support
is spotty compared to its support of oracle, and unless there is a
groundswell of demand for better postgresql support, that's not going to
change. If postgresql can gain a larger market share, computer associates
might improve their postgresql support, and we, existing clients that we
are, will be able to use postgresql in more areas. 

Marketing is very relevant to existing customers.

> If that is what we want, then fine. But I don't want to see any part of
> the development effort distorted or the existing user base
> inconvenienced in an effort to purely gain that market share. I usually
> associate increased marketing with decreased quality, and I think the
> causality works *both* ways.
> 

Aren't most development efforts made simply to gain market share? After
all, I don't think we added schema support to get *less* people to use
postgresql.

Robert Treat

Thu, 05 Dec 2002 00:12:04 -0500
"Robert Treat" <xzilla@users.sourceforge.net>
Re: [GENERAL] PostgreSQL Global Development Group
Robert Treat wrote:
> On Wed, 04 Dec 2002 22:54:37 -0500, Philip Warner wrote:
> > At 05:48 PM 4/12/2002 -0800, Christopher Kings-Lynne wrote:
> >>Lack of marketing is one of Postgres's major problems.
> > 
> > What are the consequences of the problem?
> > 
> 
> One consequence that probably hits home for everyone here is it makes it
> extremely hard to make a living working with postgresql.  A quick search on
> monster.com gives me 17 jobs mentioning postgresql, with none listed in the
> last week. A search on mysql gives me 100 jobs, with 3 filed just today. 
> I won't even go into the numbers for Oracle, DB2, and M$. We all have to 
> pay the bills and I think we'd like to do it working with postgresql.

One other thing marketing does is attracting developers, including
_paid_ developers, to work on PostgreSQL.  Fortunately PostgreSQL is a
big hit in Japan, so SRA can pay me to work on PostgreSQL.  If we can
increase PostgreSQL's popularity, we will get more people working to
improve PostgreSQL, both paid and volunteers.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 5 Dec 2002 00:20:20 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [GENERAL] PostgreSQL Global Development Group

On Thu, 5 Dec 2002, Philip Warner wrote:

> What are the consequences of the problem?

Speaking from the perspective of a long time postgresql user, who
currently has several very mission critical applications using postgresql
on the back end, at a very large company...

I can say the one consequence of the problem that I have run into
personally, is convincing management to allow me to use postgresql for my
projects to begin with. Fortunately, where I am currently employed, I was
able to bash my head against the brick wall until they got tired of
hearing from me, and allowed me to go with postgresql instead of sybase
(which was their first choice, as the corporation already has a sybase
site license).

The lack of name recognition was a factor that contributed to the
difficulty of getting postgresql accepted. The last thing a non technical
middle manager wants to tell his or her manager is that some mission
critical application that just crashed was running on some database he had
never heard of before that he gave the go ahead to use.

Anyway, this probably doesn't belong on this mailing list, but I saw the
question and figured I'd answer :)

By the way, I'm happy to report that after a year of absolutely flawless
performance ( except the day the raid array imploded, which was hardly
postgres's fault ), postgresql has a very good reputation in my
department.

Brian Knox
Systems Programmer

Thu, 5 Dec 2002 00:34:19 -0500 (EST)
Brian Knox <brian@mail.pantalaimon.net>
Re: [GENERAL] PostgreSQL Global Development Group
Lamar Owen wrote:
> However, I seriously question the need in the long term for our sites to be as 
> fractured as they are.  Good grief!  We've got advocacy.postgresql.org, 
> techdocs.postgresql.org, odbc.postgresql.org, gborg.postgresql.org, 
> developer.postgresql.org, jdbc.postgresql.org, etc.  Oh, and we also have 
> www.postgresql.org on the side?  I think not.  Oh, and they are fractured in 
> their styles -- really, guys, we need a unified style here.

I'd love to see this happen. From reading the messages here, it sounds 
like the perception is that marketing == spouting bullshit. I don't 
believe that's true. I think having an informative, up-to-date, 
stylistically consistent website would do a tremendous amount of good.

The JDBC one is a particularly bad example right now - it doesn't fit in 
with any of the rest of the site and its most prominent link is to a 
completely out-of-date list of compliance tests the driver fails. The 
driver may have its flaws but it's a lot better than presented there.

IMHO these things make a difference to technical people as well as 
suits. If that site and the MySQL JDBC driver's site were my first 
impressions, I would be using MySQL.

The JDBC site is certainly not the only one with flaws. The main website 
has this paragraph in <http://www15.us.postgresql.org/related.html>:

     For encrypted postgresql connections, Brett McCormick
     (brett-public@speakeasy.org) has made a patch for PostgreSQL
     version 6.3.2 using SSL. Visit his info page for more information.

That's horribly obselete. In fact, I think a lot of the related projects 
are. That's only two clicks away from the main page.

I'm volunteering to do work here. I could at the very least go through 
the sites and make a longer list of things like this that I notice. If 
they are public CVS somewhere, I can send patches. I saw that there's a 
<http://wwwdevel.postgresql.org/>. What's going on with that? Is there 
anything I can do to speed up its adoption? How will it affect the rest 
of the sites?

Is this list the appropriate place to discuss the websites? or should I 
take it to -advocacy? My impression here is that the two sites are 
maintained separately and the people involved haven't interacted very 
much. Is that accurate or no?

Thanks,
Scott


Thu, 05 Dec 2002 00:37:09 -0600
Scott Lamb <slamb@slamb.org>
Re: [GENERAL] PostgreSQL Global Development Group
On Wed, 4 Dec 2002 cbbrowne@cbbrowne.com wrote:

> It is unfortunate that it is almost impossible to have a marketing group
> without there being some wilful blinders involved; it's vital for there to be
> some technical involvement in the marketing group to pop whatever bubbles they
> grow that are woefully wrong.  But even if it operates with some occasional
> lack of /real/ vision, it's necessary to have a marketing group...

And, for the most part, those that are -advocacy are techies that wish to
contribute as they can, but don't have the knowledge/time to dedicate to
actual code ...

Bruce is kinda quiet, but both he and I are on that list, and I read (and
imagine Bruce does to) pretty much everything that goes through ...
but, again, these aren't 'marketing droids' we have over there, but
techies that are using the software and have an idea of her limitations
and benefits ...


Thu, 5 Dec 2002 10:31:11 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Thu, 5 Dec 2002, Philip Warner wrote:

> At 05:48 PM 4/12/2002 -0800, Christopher Kings-Lynne wrote:
> >Lack of marketing is one of Postgres's major problems.
>
> What are the consequences of the problem?

Well, I'd have to say the major one is a difficult in increasing our user
base, as ppl like MySQL are making sure they are heard whenever they
add something new that we've had for years ...

> If that is what we want, then fine. But I don't want to see any part of
> the development effort distorted or the existing user base
> inconvenienced in an effort to purely gain that market share. I usually
> associate increased marketing with decreased quality, and I think the
> causality works *both* ways.

That is what we want, and the efforts in no way are meant to
undermine/distort anything ... go to archives.postgresql.org and read
through the threads to get a feel ... its not a closed/hidden list by any
means ...



Thu, 5 Dec 2002 10:34:05 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group  Announces
On Wed, 4 Dec 2002, Lamar Owen wrote:

> However, I seriously question the need in the long term for our sites to be as
> fractured as they are.  Good grief!  We've got advocacy.postgresql.org,
> techdocs.postgresql.org, odbc.postgresql.org, gborg.postgresql.org,
> developer.postgresql.org, jdbc.postgresql.org, etc.  Oh, and we also have
> www.postgresql.org on the side?  I think not.  Oh, and they are fractured in
> their styles -- really, guys, we need a unified style here.

Ummm, actually, we have:

advocacy, techdocs, gborg, developer, archives, jobs

note that altho they are seperate URLs, the end result is going to be that
http://www.postgresql.org we become the "town square" of sorts, which
should be "real soon now" ...

jdbc/odbc are 'project sites' off of gborg, similar to what sourceforge
provides ...



Thu, 5 Dec 2002 10:37:48 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group
On Thu, 5 Dec 2002, Scott Lamb wrote:

> Is this list the appropriate place to discuss the websites? or should I
> take it to -advocacy? My impression here is that the two sites are
> maintained separately and the people involved haven't interacted very
> much. Is that accurate or no?

Expect some major changes coming down the pipe ...
http://www.postgresql.org is in its final stages of a major face lift ...
the informatoin that iscurrently on that site, Vince is in the process of
doing a major face lift on, but as it is now, I guess its been a veritible
nightmare for him to really add anyting to it ...

Once we announce the new http://www.postgresql.org (hopefully this coming
week *cross fingers*), then start bombarding us with problems :)

Note that for the web site development effort itself, there is a
closed list with about a dozen or so of us on it ... the -advocacy list is
meant to be open, with its focus reflected on the advocacy web site itself
...




Thu, 5 Dec 2002 10:42:25 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group
On Thursday 05 December 2002 09:37, Marc G. Fournier wrote:
> On Wed, 4 Dec 2002, Lamar Owen wrote:
> > However, I seriously question the need in the long term for our sites to
> > be as fractured as they are.  Good grief!  We've got

> note that altho they are seperate URLs, the end result is going to be that
> http://www.postgresql.org we become the "town square" of sorts, which
> should be "real soon now" ...

> jdbc/odbc are 'project sites' off of gborg, similar to what sourceforge
> provides ...

Glad to hear this.

One question: is there any particular reason the www list is closed?  Just 
curious -- reading archives of this list, or getting a digest or this list, 
even in a read-only manner, might alleviate some misconceptions.  Those who 
care can at least read what's planned for the web site.

As far as advocacy is concerned, I made a conscious decision to not read that 
list -- I don't need to be convinced to use PostgreSQL. :-).  Nor am I 
necessarily a good 'advocacy' person......my 'convincing' many times comes 
across much different from what I meant.  So I don't read that list.

Can you (or Vince) distill a roadmap for the website and post here, on 
hackers?
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Thu, 5 Dec 2002 12:01:43 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [GENERAL] PostgreSQL Global Development Group
Marc G. Fournier wrote:
> On Wed, 4 Dec 2002 cbbrowne@cbbrowne.com wrote:
> 
> > It is unfortunate that it is almost impossible to have a marketing group
> > without there being some wilful blinders involved; it's vital for there to be
> > some technical involvement in the marketing group to pop whatever bubbles they
> > grow that are woefully wrong.  But even if it operates with some occasional
> > lack of /real/ vision, it's necessary to have a marketing group...
> 
> And, for the most part, those that are -advocacy are techies that wish to
> contribute as they can, but don't have the knowledge/time to dedicate to
> actual code ...
> 
> Bruce is kinda quiet, but both he and I are on that list, and I read (and
> imagine Bruce does to) pretty much everything that goes through ...
> but, again, these aren't 'marketing droids' we have over there, but
> techies that are using the software and have an idea of her limitations
> and benefits ...

Yes, I have been way too quiet.  I am trying to carve out time before
starting on 7.4 work, but it seems stuff keeps coming up.  I have
updated the developers page with company names, and Vince is going to
integrate that.  My next step is to split out my advocacy mailbox and
start shooting out content for the advocacy site.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 5 Dec 2002 13:54:46 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Peter Eisentraut wrote:
>> Marc G. Fournier writes:
>>> It isn't, but those working on -advocacy were asked to help come up with a
>>> stronger release *announcement* then we've had in the past ...
>> 
>> Consider that a failed experiment.  PostgreSQL is driven by the
>> development group and, to some extent, by the existing user base.  The
>> last thing we need is a marketing department in that mix.

> Peter, I understand your perspective, but I think you are in the
> minority on this one.

I tend to agree with Peter.  Not that we don't need a marketing
presence; we do (I think Great Bridge's marketing efforts are sorely
missed).  But the point he is making is that the pgsql mailing lists
go to people who are generally unimpressed by marketing fluff.  And
they're already "sold" on PG anyway.

The right way to handle this next time is to generate a PR-style
press release to send to outside contacts, but to do our more
traditional, technically-oriented announcement on the mailing lists.

			regards, tom lane

Thu, 05 Dec 2002 14:17:08 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [GENERAL] PostgreSQL Global Development Group Announces 
At 12:12 AM 5/12/2002 -0500, Robert Treat wrote:
> >
> > What are the consequences of the problem?
> >
>
>One consequence that probably hits home for everyone here is it makes it
>extremely hard to make a living working with postgresql.
...
>You can't win marketshare on technology alone

I am happy with increasing market share so long a development is not 
distorted or current users inconvenienced. We have seen the latter with the 
misplaced announcements. And the former because I am writing this on 
-hackers, rather than implementing dependency-tracking in pg_dump ;-).


>...lots of stuff deleted...
>Marketing is very relevant to existing customers.

Good point. Market Share -> Influence ->Corprate Support -> more features 
-> market share.

Gaining market share *is* a natural consequence of improving the product; 
marketing is about convincing people a product has improved, even if it 
hasn't. Advocacy is about telling people about the product as it is - and I 
have no problem with that, with the above proviso.


>Aren't most development efforts made simply to gain market share?

<diatribe>
I seriously hope not - in fact I would find that very depressing.

In my opinion, anyone who devotes their personal free time to an open 
source development project probably has a slew of complex motivations that 
have little to do with market share. Perhaps the closest they would come 
would be to say "I want to make it better", and in some peoples minds, 
"better" is measured by market share.

In my case, development I did on other open source projects (libgd) was 
driven by a philosophical objection to application of patents to software 
in the US, and to a need for particular features (gd2 format, & gif 
support). My work on PG is driven by a desire to make the product more 
useful (to me), more usable (for me), and by a philosophical belief in the 
importance of free & open software. The fact that other people (& I) profit 
from this work is great. In any case, market share, for me, is at best a 
third order influence - and I assume that's true for most people who 
contribute to OS software. Although I do admit that there is a natural 
tendency to want "your team" to win.
</diatribe>


>After
>all, I don't think we added schema support to get *less* people to use
>postgresql.

I am not sure why it was added, and it's sufficiently esoteric and large 
that I doubt market share was an issue. If we wanted market share, then 
online-vacuum and online-upgrade would have been the big-hitters.

My guess is that it was done because we did not support it, it is in the 
SQL standard, and it solved a number of issues that caused existing users & 
developers problems. It was probably also an interesting project. Maybe I'm 
wrong...




----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 03 5330 3172          |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Fri, 06 Dec 2002 12:31:05 +1100
Philip Warner <pjw@rhyme.com.au>
Re: [GENERAL] PostgreSQL Global Development Group
On Thu, 05 Dec 2002 21:26:13 -0500, Philip Warner wrote:
> At 12:12 AM 5/12/2002 -0500, Robert Treat wrote:
> I am happy with increasing market share so long a development is not
> distorted or current users inconvenienced. We have seen the latter with
> the misplaced announcements.

It seems to me that people were inconvenienced solely because Mark forgot
to CC the right groups and he didn't put the word "7.3" in the right
place in his subject line. Oh, and guess it was disruptive for people who
killfile any piece of email that has quoted text in it...

>  And the former because I am writing this on
> -hackers, rather than implementing dependency-tracking in pg_dump ;-).
> 

so get back to coding already...

>>...lots of stuff deleted...
>>Marketing is very relevant to existing customers.
> 
> Good point. Market Share -> Influence ->Corprate Support -> more
> features -> market share.
> 
> Gaining market share *is* a natural consequence of improving the
> product; 

really? postgresql has been improving by leaps and bounds of the last few
years, but I guarantee you it's been losing market share, and it's losing
that market share to databases without half the features.

> marketing is about convincing people a product has improved,
> even if it hasn't. Advocacy is about telling people about the product as
> it is - and I have no problem with that, with the above proviso.
> 

<snip lots more stuff that basically says marketing isn't all bad, it's
irrelevant too>

well, i think any more discussion at this point becomes a semantical
argument or a flame war, and I've time for neither. 

Robert Treat

Thu, 05 Dec 2002 23:13:39 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: [GENERAL] PostgreSQL Global Development Group
As someone who exists mainly as an active user (and part-time 
advocate/documentation tweaker), I have found the release of PostgreSQL 
7.3 to be disappointing. The ensuing pseudo-flamewar on the various 
lists has been similarly disappointing.

I was surprised, for instance, to receive a non-list email announcing 
the release of the software but then to have to wait for days actually 
to see it show up on the official (or even the advocacy) website in a 
news item. Even now it is not listed at PostgreSQL, Inc.

Consider the pieces of the puzzle here:

1) an official website (http://www.postgresql.org/)
2) an advocacy website (http://advocacy.postgresql.org/)
3) official mailing lists
4) a separate email database
5) a developers' website (http://developers.postgresql.org/)
6) an official ftp site (ftp://ftp.postgresql.org/)
7) mirror websites
8) mirror ftp sites
9) a corporate website (http://www.pgsql.com/)

While I have remained impressed with the software itself, the 
organization of these pieces has left much to be desired for the 
duration of my involvement as an end user.

As someone who works in a small startup company, I am a frequent witness 
to both the advantages and disadvantages of the lack of a strong 
benevolent dictatorship in the form of management. I think one of the 
core problems with the advocacy and presentation of the PostgreSQL 
project is the fact that it has been a developer-centric project for 
quite some time, and that process, while there are drivers, does not 
tend to affect much other than the code. There does not seem to be a 
single, driving vision (or even a Board or consensus-based vision) 
behind the public face of PostgreSQL. Granted, when a project is 
entirely volunteer-based, the management and development are loose. I've 
noticed that in many such projects, web design and maintenance become 
very low priority, especially when left to groups of hackers. Witness 
GNU, Debian, and, I would say PostgreSQL: extremely spare official 
websites often intimidating and/or difficult for the newbie.

I've wanted to see a bit more structure given to the PostgreSQL website, 
the release process, and various other portions of the project for quite 
some time, but often it seems as though such a structure would not even 
be welcome. As someone who has not had time to be a true developer on 
the project, I'm content to wait for the missing features I'd like to 
see.

Still, I'm hoping that developers and advocates alike realize that the 
release process and these lists are in the public domain, and the way 
business is conducted affects the perceptions of users as much as the 
quality of the software or any amount of marketing.

In any case, thanks for all the hard work. I actually thought the text 
of the email release I received was good and am working on the upgrade 
process now in my own environment.

-tfo

In article <29852.1039115828@sss.pgh.pa.us>,
 tgl@sss.pgh.pa.us (Tom Lane) wrote:

> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Peter Eisentraut wrote:
> >> Marc G. Fournier writes:
> >>> It isn't, but those working on -advocacy were asked to help come up with a
> >>> stronger release *announcement* then we've had in the past ...
> >> 
> >> Consider that a failed experiment.  PostgreSQL is driven by the
> >> development group and, to some extent, by the existing user base.  The
> >> last thing we need is a marketing department in that mix.
> 
> > Peter, I understand your perspective, but I think you are in the
> > minority on this one.
> 
> I tend to agree with Peter.  Not that we don't need a marketing
> presence; we do (I think Great Bridge's marketing efforts are sorely
> missed).  But the point he is making is that the pgsql mailing lists
> go to people who are generally unimpressed by marketing fluff.  And
> they're already "sold" on PG anyway.
> 
> The right way to handle this next time is to generate a PR-style
> press release to send to outside contacts, but to do our more
> traditional, technically-oriented announcement on the mailing lists.

Fri, 06 Dec 2002 19:14:12 -0600
Thomas O'Connell <tfo@monsterlabs.com>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Wed, 4 Dec 2002, Bruce Momjian wrote:

> Peter Eisentraut wrote:
> > Marc G. Fournier writes:
> >
> > > It isn't, but those working on -advocacy were asked to help come up with a
> > > stronger release *announcement* then we've had in the past ...
> >
> > Consider that a failed experiment.  PostgreSQL is driven by the
> > development group and, to some extent, by the existing user base.  The
> > last thing we need is a marketing department in that mix.
>
> Peter, I understand your perspective, but I think you are in the
> minority on this one.

Kinda depends who you're asking now, doesn't it?  I happen to agree with
him, but as long as you're only going to involve a selected few in the
opinion gathering you can pretty much get the answer you want to get.  I
can survey 100 people and get the opposite result putting you in the
minority.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Sat, 7 Dec 2002 19:20:19 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Thu, 5 Dec 2002, Robert Treat wrote:

> Well, my previous employer uses postgresql, but they were under constant
> assault from their clients to use oracle or db2.  Technically there was no
> reason to switch, but if your choice is switch databases or go out of
> business, there really isn't much choice.

That tells me their clients wanted a commercial database, not one that's
open source.  All the marketing in the world won't change that.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Sat, 7 Dec 2002 20:53:20 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group
Vince Vielhaber wrote:
> On Thu, 5 Dec 2002, Robert Treat wrote:
> 
> 
>>Well, my previous employer uses postgresql, but they were under constant
>>assault from their clients to use oracle or db2.  Technically there was no
>>reason to switch, but if your choice is switch databases or go out of
>>business, there really isn't much choice.
> 
> 
> That tells me their clients wanted a commercial database, not one that's
> open source.  All the marketing in the world won't change that.

Really?

Why do you say that?

:-)

Regards and best wishes,

Justin Clift


> Vince.


-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Sun, 08 Dec 2002 12:58:41 +1100
Justin Clift <justin@postgresql.org>
Re: [GENERAL] PostgreSQL Global Development Group
On Thu, 5 Dec 2002, Brian Knox wrote:

> Speaking from the perspective of a long time postgresql user, who
> currently has several very mission critical applications using postgresql
> on the back end, at a very large company...
>
> I can say the one consequence of the problem that I have run into
> personally, is convincing management to allow me to use postgresql for my
> projects to begin with. Fortunately, where I am currently employed, I was
> able to bash my head against the brick wall until they got tired of
> hearing from me, and allowed me to go with postgresql instead of sybase
> (which was their first choice, as the corporation already has a sybase
> site license).
>
> The lack of name recognition was a factor that contributed to the
> difficulty of getting postgresql accepted. The last thing a non technical
> middle manager wants to tell his or her manager is that some mission
> critical application that just crashed was running on some database he had
> never heard of before that he gave the go ahead to use.

Not name recognition, but it'd be nice to think that's the reason.
Mysql has alot of name recognition but you didn't mention them.  You
mentioned sybase and having a sybase site license.  Marketing wouldn't
help here, they want a commercial database used that they've already
paid for.

What too many people fail to realize is that in a commercial environment
many companies want another company to point the finger at in case of
disaster.  Sybase failed, or HP failed, or IBM failed, or Microsoft
failed.  They feel they can do something about that.  If they lose a
few million they have someone they can go after, who are they going to
go after if PostgreSQL fails them?  Marc?  Bruce?

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Sat, 7 Dec 2002 21:13:12 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group
> What too many people fail to realize is that in a commercial environment
> many companies want another company to point the finger at in case of
> disaster.  Sybase failed, or HP failed, or IBM failed, or Microsoft
> failed.  They feel they can do something about that.  If they lose a
> few million they have someone they can go after, who are they going to
> go after if PostgreSQL fails them?  Marc?  Bruce?

This is when you start to shout that RedHat offers commercial support,
licencing, etc. INCLUDING a free, non-restrictive source licence to the
core components of RHDB.

-- 
Rod Taylor <rbt@rbt.ca>

PGP Key: http://www.rbt.ca/rbtpub.asc
07 Dec 2002 22:11:18 -0500
Rod Taylor <rbt@rbt.ca>
Re: [GENERAL] PostgreSQL Global Development Group
On Sun, 8 Dec 2002, Justin Clift wrote:

> Vince Vielhaber wrote:
> > On Thu, 5 Dec 2002, Robert Treat wrote:
> >
> >
> >>Well, my previous employer uses postgresql, but they were under constant
> >>assault from their clients to use oracle or db2.  Technically there was no
> >>reason to switch, but if your choice is switch databases or go out of
> >>business, there really isn't much choice.
> >
> >
> > That tells me their clients wanted a commercial database, not one that's
> > open source.  All the marketing in the world won't change that.
>
> Really?
>
> Why do you say that?

Because of this taken from the above quoted text:

"they were under constant assault from their clients to use oracle or db2"

Last I looked neither Oracle or DB2 were open source, but they both just
happen to be commercial and I don't see mysql mentioned.

Anything else you don't understand about that?

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Sun, 8 Dec 2002 15:52:10 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group
On 7 Dec 2002, Rod Taylor wrote:

>
> > What too many people fail to realize is that in a commercial environment
> > many companies want another company to point the finger at in case of
> > disaster.  Sybase failed, or HP failed, or IBM failed, or Microsoft
> > failed.  They feel they can do something about that.  If they lose a
> > few million they have someone they can go after, who are they going to
> > go after if PostgreSQL fails them?  Marc?  Bruce?
>
> This is when you start to shout that RedHat offers commercial support,
> licencing, etc. INCLUDING a free, non-restrictive source licence to the
> core components of RHDB.

I had considered mentioning redhat but didn't want to blur things.  Red
hat markets PostgreSQL under a different name and they're offering a
complete package (including support as you note).  The PGDG isn't doing
that and they shouldn't be.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Sun, 8 Dec 2002 15:57:03 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group
Vince Vielhaber wrote:
> Because of this taken from the above quoted text:
> 
> "they were under constant assault from their clients to use oracle or db2"
> 
> Last I looked neither Oracle or DB2 were open source, but they both just
> happen to be commercial and I don't see mysql mentioned.

And.... ?

Regards and best wishes,

Justin Clift


> Anything else you don't understand about that?
> 
> Vince.


-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Mon, 09 Dec 2002 08:19:13 +1100
Justin Clift <justin@postgresql.org>
Re: [GENERAL] PostgreSQL Global Development Group
On Sun, 2002-12-08 at 20:52, Vince Vielhaber wrote:
> > Why do you say that?
> 
> Because of this taken from the above quoted text:
> 
> "they were under constant assault from their clients to use oracle or db2"
> 
> Last I looked neither Oracle or DB2 were open source, but they both just
> happen to be commercial and I don't see mysql mentioned.

This is a reason to increase marketing effort.  I know the word has
pejorative overtones in our community, but it means talking about
PostgreSQL so that the PHBs hear about it and therefore begin to feel
comfortable about using it.

If something is familiar, it feels safe.  We need to make PostgreSQL
familiar.  That's why we need marketing.

-- 
Oliver Elphick <olly@lfix.co.uk>
LFIX Limited


08 Dec 2002 22:01:00 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: [GENERAL] PostgreSQL Global Development Group
On Mon, 9 Dec 2002, Justin Clift wrote:

> Vince Vielhaber wrote:
> > Because of this taken from the above quoted text:
> >
> > "they were under constant assault from their clients to use oracle or db2"
> >
> > Last I looked neither Oracle or DB2 were open source, but they both just
> > happen to be commercial and I don't see mysql mentioned.
>
> And.... ?

And what?  If you can't understand the above you're in the wrong business.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Sun, 8 Dec 2002 17:07:55 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group
Vince Vielhaber wrote:
> On Mon, 9 Dec 2002, Justin Clift wrote:
> 
> 
>>Vince Vielhaber wrote:
>>
>>>Because of this taken from the above quoted text:
>>>
>>>"they were under constant assault from their clients to use oracle or db2"
>>>
>>>Last I looked neither Oracle or DB2 were open source, but they both just
>>>happen to be commercial and I don't see mysql mentioned.
>>
>>And.... ?
> 
> 
> And what?  If you can't understand the above you're in the wrong business.

And.... ?


Regards and best wishes,

Justin Clift


> Vince.


-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Mon, 09 Dec 2002 09:18:27 +1100
Justin Clift <justin@postgresql.org>
Re: [GENERAL] PostgreSQL Global Development Group
On 8 Dec 2002, Oliver Elphick wrote:

> On Sun, 2002-12-08 at 20:52, Vince Vielhaber wrote:
> > > Why do you say that?
> >
> > Because of this taken from the above quoted text:
> >
> > "they were under constant assault from their clients to use oracle or db2"
> >
> > Last I looked neither Oracle or DB2 were open source, but they both just
> > happen to be commercial and I don't see mysql mentioned.
>
> This is a reason to increase marketing effort.  I know the word has
> pejorative overtones in our community, but it means talking about
> PostgreSQL so that the PHBs hear about it and therefore begin to feel
> comfortable about using it.
>
> If something is familiar, it feels safe.  We need to make PostgreSQL
> familiar.  That's why we need marketing.

Then why wasn't mysql in the list?  It's familiar.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Sun, 8 Dec 2002 17:27:40 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group
On Mon, 9 Dec 2002, Justin Clift wrote:

> Vince Vielhaber wrote:
> > On Mon, 9 Dec 2002, Justin Clift wrote:
> >
> >
> >>Vince Vielhaber wrote:
> >>
> >>>Because of this taken from the above quoted text:
> >>>
> >>>"they were under constant assault from their clients to use oracle or db2"
> >>>
> >>>Last I looked neither Oracle or DB2 were open source, but they both just
> >>>happen to be commercial and I don't see mysql mentioned.
> >>
> >>And.... ?
> >
> >
> > And what?  If you can't understand the above you're in the wrong business.
>
> And.... ?

That's what I thought.  You have no argument so your just typing.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Sun, 8 Dec 2002 17:30:08 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group
On Sun, 2002-12-08 at 22:27, Vince Vielhaber wrote:
> On 8 Dec 2002, Oliver Elphick wrote:

> > If something is familiar, it feels safe.  We need to make PostgreSQL
> > familiar.  That's why we need marketing.
> 
> Then why wasn't mysql in the list?  It's familiar.

To PHBs?

MySQL doesn't have anything like the marketing clout of Oracle and IBM. 
Be thankful it isn't in the list; it would make it a hell of a lot more
difficult to dislodge it.

If we want people to use PostgreSQL in preference to anything else, we
have to make it known.  That is marketing.  If we believe we have a good
product we need to say so and say why and how it's better, cheaper and
purer than anything else.  If there's no good marketing, bad marketing
will rule the world for sure.

If we don't care, we can retreat into a pure technological huddle and
disappear up our own navels.  The rest of the world won't even notice. 
Such purity will eventually destroy the project because it will lose the
momentum for growth through a lack of new input.  You can grow or you
can decline; a steady state is almost impossible to achieve.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "For I am the LORD your God; ye shall therefore  
      sanctify yourselves, and ye shall be holy; for I am 
      holy."          Leviticus 11:44 


08 Dec 2002 22:59:09 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: [GENERAL] PostgreSQL Global Development Group
On 8 Dec 2002, Oliver Elphick wrote:

> On Sun, 2002-12-08 at 22:27, Vince Vielhaber wrote:
> > On 8 Dec 2002, Oliver Elphick wrote:
>
> > > If something is familiar, it feels safe.  We need to make PostgreSQL
> > > familiar.  That's why we need marketing.
> >
> > Then why wasn't mysql in the list?  It's familiar.
>
> To PHBs?

I would argue yes.  Everywhere you turn you see "Powered by MySQL".
If years of working on it isn't getting them the familiarity to overcome
the PHBs then the PHBs are either not considering open source or the
marketing attempts aren't strong or capable enough to penetrate.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Sun, 8 Dec 2002 18:14:53 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group
Vince Vielhaber wrote:
 >
> That's what I thought.  You have no argument so your just typing.

Hi Vince,

Was more hoping you'd care to share your basis for stating Robert's 
employers clients wanted a "commercial database", after he mentioned 
specifically DB2 and Oracle.  Knowing one of the obvious common factors 
they have and then stating it was definitely the reason - not having 
sought clarification nor confirmation from Robert - and then further 
stating that the PG Advocacy and Marketing group wouldn't be able to 
assist even if that were the case, is extremely bad form coming from 
anyone, let alone you.

Please consider the statements you make by a more accurate approach in 
the future.

Regards and best wishes,

Justin Clift


> Vince.

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Mon, 09 Dec 2002 10:52:56 +1100
Justin Clift <justin@postgresql.org>
Re: [GENERAL] PostgreSQL Global Development Group
On Mon, 9 Dec 2002, Justin Clift wrote:

> Vince Vielhaber wrote:
>  >
> > That's what I thought.  You have no argument so your just typing.
>
> Hi Vince,
>
> Was more hoping you'd care to share your basis for stating Robert's
> employers clients wanted a "commercial database", after he mentioned
> specifically DB2 and Oracle.  Knowing one of the obvious common factors
> they have and then stating it was definitely the reason - not having
> sought clarification nor confirmation from Robert - and then further
> stating that the PG Advocacy and Marketing group wouldn't be able to
> assist even if that were the case, is extremely bad form coming from
> anyone, let alone you.

Then they come with the insults.   Justin, I'm finished discussing this
with you.  You're obviously not capable of understanding it and you're
simply wasting my time - like usual.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Sun, 8 Dec 2002 19:14:12 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [GENERAL] PostgreSQL Global Development Group
Oliver Elphick wrote:

> If we want people to use PostgreSQL in preference to anything else, we
> have to make it known.  That is marketing.  If we believe we have a good
> product we need to say so and say why and how it's better, cheaper and
> purer than anything else.  If there's no good marketing, bad marketing
> will rule the world for sure.
> 
> If we don't care, we can retreat into a pure technological huddle and
> disappear up our own navels.  The rest of the world won't even notice. 
> Such purity will eventually destroy the project because it will lose the
> momentum for growth through a lack of new input.  You can grow or you
> can decline; a steady state is almost impossible to achieve.

Couldn't agree more with that last point.

I've had the perspective of working in big companies using various database software, a company specifically focused on PostgreSQL (Great Bridge), and now a new ISV with PostgreSQL underneath a vertical application (OpenMFG).  I can tell you that even though the pgsql-hacker community is as strong as it's ever been, I think there's a serious danger of the larger world passing PostgreSQL by.

Oracle and DB2 continue to get better and - significantly - cheaper, and SQL Server ... well, Oracle and DB2 are getting better.  MySQL, even though it's an inferior product for most real database work, has always had a significantly larger installed base than PostgreSQL- and it's less controversial for people like Sun (who have deep relationships with Oracle) to get involved with.  And despite the productizing of RHDB, Red Hat doesn't seem interested in making a real push for PostgreSQL either.  While there are a number of smaller companies trying to help out, I think it's clear that the burden for helping PostgreSQL to find wider acceptance in the marketplace will be on the pgsql-hacker community for some time to come.

I applaud the efforts of the advocacy group, and encourage others here not to look at the marketing as somehow dirty or beneath the dignity of the project.

Keep up the good work,
Ned


Sun, 8 Dec 2002 20:53:17 -0500
"Ned Lilly" <ned@nedscape.com>
Re: [GENERAL] PostgreSQL Global Development Group
On Sunday 08 December 2002 06:14 pm, Vince Vielhaber wrote:
> On 8 Dec 2002, Oliver Elphick wrote:
> > On Sun, 2002-12-08 at 22:27, Vince Vielhaber wrote:
> > > On 8 Dec 2002, Oliver Elphick wrote:
> > > > If something is familiar, it feels safe.  We need to make PostgreSQL
> > > > familiar.  That's why we need marketing.
> > >
> > > Then why wasn't mysql in the list?  It's familiar.
> >
> > To PHBs?
>
> I would argue yes.  Everywhere you turn you see "Powered by MySQL".
> If years of working on it isn't getting them the familiarity to overcome
> the PHBs then the PHBs are either not considering open source or the
> marketing attempts aren't strong or capable enough to penetrate.
>

I don't think mysql has penetrated the "enterprise class/ mission critical" 
mindest, which is the level our service had to be provided that.  To be 
honest, it was tough to argue PostgreSQL belonged in that group, though we 
had a good 2 years worth of history in actually running the business on 
PostgreSQL which couldn't be dismissed.  Of course, some of these companies 
weren't too happy things were running on linux, and not aix or solaris; are 
we seeing a pointy haired trend here?  Personally I never understood why our 
sales guys didn't just tell them "ok we'll port the service to oracle/solaris 
for you, but it's going to cost you at least twice what it does now, if not 
three times. Oh, and you won't see any better performance."

Robert Treat


Sun, 8 Dec 2002 22:48:54 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: [GENERAL] PostgreSQL Global Development Group
Am Donnerstag, 5. Dezember 2002 05:22 schrieb Lamar Owen:
> [cc: list trimmed]
>
> On Wednesday 04 December 2002 22:52, Philip Warner wrote:
> > At 05:48 PM 4/12/2002 -0800, Christopher Kings-Lynne wrote:
> > >Lack of marketing is one of Postgres's major problems.
> >
> > What are the consequences of the problem?
>
> Actually, lack of easy upgrading is one of PostgreSQL's major problems....
>
> But lack of focused marketing -- truthful, not, as has been said, like the
> 'Database HOWTO' -- is a real problem.  It would be nice to increase our
> usage.
>
> > If that is what we want, then fine. But I don't want to see any part of
> > the development effort distorted or the existing user base inconvenienced
> > in an effort to purely gain that market share. I usually associate
> > increased marketing with decreased quality, and I think the causality
> > works *both* ways.
>
> ISTM there's a separate, non-code-developer group doing this.  It doesn't
> seem to take away _any_ developer resources to do an advocacy site.
>
> However, I seriously question the need in the long term for our sites to be
> as fractured as they are.  Good grief!  We've got advocacy.postgresql.org,
> techdocs.postgresql.org, odbc.postgresql.org, gborg.postgresql.org,
> developer.postgresql.org, jdbc.postgresql.org, etc.  Oh, and we also have
> www.postgresql.org on the side?  I think not.  Oh, and they are fractured
> in their styles -- really, guys, we need a unified style here.

Hi,

there are lots of sites talking about postgresql. But if someone hear about 
postgresql he sure tries www.postgresql.org. There he just get a list of 
mirrors. Not really a good start. But worse: there is no links to gborg, 
advocacy, techdocs, ... Advocacy should be found at www.postgresql.org and 
have links to the other pages. I found gborg when reading the mailinglistst. 
It is something like a insidertip.

www.apache.org has a much better structure. You go to www.apache.org and get a 
welcome-message and links to subprojects as the webserver.

Another point that comes to my mind is design. I'm not a designer, but I like 
the design of www.postgresql.org but not advocacy.postrgresql.org.


Tommi

-- 
Dr. Eckhardt + Partner GmbH
http://www.epgmbh.de

Mon, 9 Dec 2002 09:03:44 +0100
Tommi Maekitalo <t.maekitalo@epgmbh.de>
Re: [GENERAL] PostgreSQL Global Development Group
Hi Tommi,

Tommi Maekitalo wrote:
<snip>
> Hi,
> 
> there are lots of sites talking about postgresql. But if someone hear about 
> postgresql he sure tries www.postgresql.org. There he just get a list of 
> mirrors. Not really a good start. But worse: there is no links to gborg, 
> advocacy, techdocs, ... Advocacy should be found at www.postgresql.org and 
> have links to the other pages. I found gborg when reading the mailinglistst. 
> It is something like a insidertip.

There is a new front page for the www.postgresql.org site that was 
recently finished, and will be moved into the correct place soon.  You 
can view it for now at wwwdevel.postgresql.org.

The new front page has links to the other main websites, so it should 
help people find the information they need in a much easier way.  :-)

Hope that's helpful to know.

:-)

Regards and best wishes,

Justin Clift


<snip>
> Tommi
> 


-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Mon, 09 Dec 2002 19:16:01 +1100
Justin Clift <justin@postgresql.org>
Re: [GENERAL] PostgreSQL Global Development Group
Vince Vielhaber wrote:
> On Sun, 8 Dec 2002, Justin Clift wrote:
> 
> > Vince Vielhaber wrote:
> > > On Thu, 5 Dec 2002, Robert Treat wrote:
> > >
> > >
> > >>Well, my previous employer uses postgresql, but they were under constant
> > >>assault from their clients to use oracle or db2.  Technically there was no
> > >>reason to switch, but if your choice is switch databases or go out of
> > >>business, there really isn't much choice.
> > >
> > >
> > > That tells me their clients wanted a commercial database, not one that's
> > > open source.  All the marketing in the world won't change that.
> >
> > Really?
> >
> > Why do you say that?
> 
> Because of this taken from the above quoted text:
> 
> "they were under constant assault from their clients to use oracle or db2"
> 
> Last I looked neither Oracle or DB2 were open source, but they both just
> happen to be commercial and I don't see mysql mentioned.
> 
> Anything else you don't understand about that?

There are a number of reasons their clients could have been clamoring
for DB2 or Oracle, only some of which are related to the fact that
they're commercial, closed-source databases:

1.  They already have significant in-house expertise with one or the
    other product.

2.  They need 24x7 support, and are convinced that they'll get better
    support for Oracle or DB2 than anything else.

3.  They want a company to blame in case things go wrong.

4.  They require certain capabilities that they believe only DB2 or
    Oracle can provide.

5.  They have an established partnership with IBM or Oracle.

6.  Some combination of the above.


Some of those reasons are such that it might be possible (depending on
the specifics of the situation) to successfully market PostgreSQL (or
even MySQL) to them, and some of them aren't.  It just depends.

And that's why it's a bad idea to simply discard that situation as one
in which it would be impossible to market PostgreSQL.


Marketing is the art of convincing someone that they want your
product.  Since the keyword here is "want", it's an art that combines
reason and emotion.  Even if the situation seems logically hopeless
(that is, there's no logical reason for the customer to prefer your
product over another), you may still manage to successfully market
your product to them by appealing to their emotions.  Happens all the
time.

My personal feeling is that in the case of PostgreSQL, it should be
marketed primarily using reason.  More precisely, it should *not* be
marketed to someone for whom a different product would better suit
them.  That, to me, would be shady at best and would eventually become
a blemish on the reputation of the PostgreSQL community.  But it
doesn't mean giving up just because the client thinks he wants a
commercial database: he may well want something else that a commercial
database just happens to provide.

If you're trying to sell someone on PostgreSQL, it behooves you to
figure out what their real needs are first.  Their actual needs may be
significantly different from what they tell you they want.



-- 
Kevin Brown					      kevin@sysexperts.com

Mon, 9 Dec 2002 01:20:45 -0800
Kevin Brown <kevin@sysexperts.com>
Re: [GENERAL] PostgreSQL Global Development Group
On 9 Dec 2002 at 1:20, Kevin Brown wrote:

> 2.  They need 24x7 support, and are convinced that they'll get better
>     support for Oracle or DB2 than anything else.

I have experienced what oracle support means for 24x7. I wouldn't even wish 
that penalty for my worst enemy.

I can tell a story about it but I digress. Details aren't important though 
true. 

What really matters is how kindly and dearly you stand by your product. That is 
where all support originates.. Rest is marketing..

Bye
 Shridhar

--
I have never understood the female capacity to avoid a direct answer toany 
question.		-- Spock, "This Side of Paradise", stardate 3417.3


Mon, 09 Dec 2002 15:13:12 +0530
"Shridhar Daithankar" <shridhar_daithankar@persistent.co.in>
Re: [GENERAL] PostgreSQL Global Development Group
On Thu, 5 Dec 2002, Tom Lane wrote:

> I tend to agree with Peter.  Not that we don't need a marketing
> presence; we do (I think Great Bridge's marketing efforts are sorely
> missed).  But the point he is making is that the pgsql mailing lists go
> to people who are generally unimpressed by marketing fluff.  And they're
> already "sold" on PG anyway.
>
> The right way to handle this next time is to generate a PR-style
> press release to send to outside contacts, but to do our more
> traditional, technically-oriented announcement on the mailing lists.

Agreed ... we tried to do 'two-in-one' on this one and it didn't quite
work out as well as it could have ... next time, we'll go with both
methods ...


Mon, 9 Dec 2002 18:15:21 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
On Fri, 6 Dec 2002, Thomas O'Connell wrote:

> I was surprised, for instance, to receive a non-list email announcing
> the release of the software but then to have to wait for days actually
> to see it show up on the official (or even the advocacy) website in a
> news item. Even now it is not listed at PostgreSQL, Inc.

ack, an oversight, I can assure you ... I have proded the apporpriate ppl
for this one :(


Mon, 9 Dec 2002 18:18:16 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [HACKERS] [GENERAL] PostgreSQL Global Development Group Announces
On Sat, 7 Dec 2002, Vince Vielhaber wrote:

> On Wed, 4 Dec 2002, Bruce Momjian wrote:
>
> > Peter Eisentraut wrote:
> > > Marc G. Fournier writes:
> > >
> > > > It isn't, but those working on -advocacy were asked to help come up with a
> > > > stronger release *announcement* then we've had in the past ...
> > >
> > > Consider that a failed experiment.  PostgreSQL is driven by the
> > > development group and, to some extent, by the existing user base.  The
> > > last thing we need is a marketing department in that mix.
> >
> > Peter, I understand your perspective, but I think you are in the
> > minority on this one.
>
> Kinda depends who you're asking now, doesn't it?  I happen to agree with
> him, but as long as you're only going to involve a selected few in the
> opinion gathering you can pretty much get the answer you want to get.  I
> can survey 100 people and get the opposite result putting you in the
> minority.

Me, I think Peter went to the 'far left', while the press release went to
the 'far right' (or vice versa) ... i think Tom sum'd it up best that we
should have had one for each 'market' we were trying to address ...
definitely something to keep in mind and strive for for the next release
...


Mon, 9 Dec 2002 18:21:29 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
>
>
>>>Peter Eisentraut wrote:
>>>      
>>>
>>>>Marc G. Fournier writes:
>>>>
>>>>        
>>>>
>>>>>It isn't, but those working on -advocacy were asked to help come up with a
>>>>>stronger release *announcement* then we've had in the past ...
>>>>>          
>>>>>
>>>>Consider that a failed experiment.  PostgreSQL is driven by the
>>>>development group and, to some extent, by the existing user base.  The
>>>>last thing we need is a marketing department in that mix.
>>>>
Then you will have what you want. You will be used by a limited number 
of developers who understand the idea. And you will have ugly dialogues 
like that. This sounds a bit like 'what would happen if all population 
of the world were male'. Or all were developers. You should accept the 
fact that you never have developers on the front line. Even if you take 
Microsoft - I even do not know the name of the chief software engineer 
(do not tell me this is Mr. Gates, he is not - there is a guy with a 
beard, the third richest man in the world or so). Or if you take Oracle 
- you have Larry. Larry is not a developer. Or even with MySQL - you see 
the marketing machine. Even with Linux - I have not seen Linus in the 
press for ages. Or Alan. All 'gurus' are hidden. You take the hype - the 
hype of Bill or the hype of Linus. Or the charming and successfully 
arogant Lary. And make a product out of it and a market. As long as the 
developers of PostgreSQL want to be on the front line - it will be what 
it is - a fine database used by the people who have the clue to talk to 
and understand developers. An uncut diamond.

I actually do not understand why is the whole cry - why not somebody who 
has REALLY the marketing in his/her heart - does not make an open source 
amazingly beautiful and powerful web site. You do not have to ask Bruce 
for that. You get BRICOLAGE - it is free, and it is good - salon.com 
runs on it. You inspire some great designer to do the desing (do not ask 
a developer to do that, otherwise a designer might want to do some code 
and PostgreSQL is lost). Call Mario Garcia (www.mariogarcia.com) - he 
will be proud to help. And you take ten fanatic advocacy people to fill 
in success stories and case studies. News. Whatever.

It does not take that much. It take strong individuals that lead. 
However, some people on HACKERS find special pleasure to kill all 
initiative. I do not see this for first time...

Iavor

www.pgaccess.org



Sat, 14 Dec 2002 01:44:57 +0100
Iavor Raytchev <pobox@verysmall.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Iavor Raytchev wrote:
> I actually do not understand why is the whole cry - why not somebody who 
> has REALLY the marketing in his/her heart - does not make an open source 
> amazingly beautiful and powerful web site. You do not have to ask Bruce 
> for that. You get BRICOLAGE - it is free, and it is good - salon.com 
> runs on it. You inspire some great designer to do the desing (do not ask 
> a developer to do that, otherwise a designer might want to do some code 
> and PostgreSQL is lost). Call Mario Garcia (www.mariogarcia.com) - he 
> will be proud to help. And you take ten fanatic advocacy people to fill 
> in success stories and case studies. News. Whatever.
> 
> It does not take that much. It take strong individuals that lead. 
> However, some people on HACKERS find special pleasure to kill all 
> initiative. I do not see this for first time...

I think we have gotten over that hurdle and _most_ agree marketing is a
priority.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 13 Dec 2002 19:50:25 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Bruce Momjian wrote:

>Iavor Raytchev wrote:
>  
>
>>I actually do not understand why is the whole cry - why not somebody who 
>>has REALLY the marketing in his/her heart - does not make an open source 
>>amazingly beautiful and powerful web site. You do not have to ask Bruce 
>>for that. You get BRICOLAGE - it is free, and it is good - salon.com 
>>runs on it. You inspire some great designer to do the desing (do not ask 
>>a developer to do that, otherwise a designer might want to do some code 
>>and PostgreSQL is lost). Call Mario Garcia (www.mariogarcia.com) - he 
>>will be proud to help. And you take ten fanatic advocacy people to fill 
>>in success stories and case studies. News. Whatever.
>>
>>It does not take that much. It take strong individuals that lead. 
>>However, some people on HACKERS find special pleasure to kill all 
>>initiative. I do not see this for first time...
>>    
>>
>
>I think we have gotten over that hurdle and _most_ agree marketing is a
>priority.
>
I am sorry. Seems I came too late. I did it out of my good feelings.

Iavor




Sat, 14 Dec 2002 02:16:18 +0100
Iavor Raytchev <pobox@verysmall.org>
Re: [GENERAL] PostgreSQL Global Development Group Announces
Peter Eisentraut wrote:

>Marc G. Fournier writes:
>
>  
>
>>It isn't, but those working on -advocacy were asked to help come up with a
>>stronger release *announcement* then we've had in the past ...
>>    
>>
>
>Consider that a failed experiment.  PostgreSQL is driven by the
>development group and, to some extent, by the existing user base.  The
>last thing we need is a marketing department in that mix.
>

I am a long term user of PostgreSQL and I think it suffers from a lack 
of a marketing department.

If you have the best restaurant in town, but no one eats there, what's 
the point?

We all correspond and work on PostgreSQL to make it the best we can. To 
create something "good" that people can use. One of the prime parts of 
that sentence is "people can use." Like it or not, that means getting 
the word out.

MySQL is an appalling database, but people use it, a lot! Why? Because 
they really market it. They push it. They craft deceptive benchmarks 
which show it is better. PostgreSQL doesn't even need to be deceptive.

My company is working on a Suite of applications and PostgreSQL is a key 
component. We will be doing our own local marketing, but it it would 
help if the PostgreSQL core understood that a clean professional looking 
website, geared toward end users would make a big difference.

Furthermore, I think it would be very rewarding for everyone involved if 
we could get some of the "street cred" that MySQL has. PostgreSQL *is* a 
better database in almost every way. If MySQL virtually owns the open 
source mind share for SQL databases, it is our fault.

Peter, Tom, Bruce, et al. you guys do a great job, IMHO PostgreSQL isn't 
lacking in anything technical, as of 7.2, with non-locking vacuum, I 
would consider it a viable database with no caveats. 7.3 is superior.  A 
pure Win32 version would be awesome.

I just think that if we could get people equally talented at spreading 
the word and making the noise, it would make a big difference in the 
number of users. More users eventually translates to more funding or 
development.

Wouldn't you like to say to someone: "I contribute the PostgreSQL 
project" and have them say "Cool" instead of "What's that?"






Sat, 14 Dec 2002 08:26:51 -0500
mlw <pgsql@mohawksoft.com>
Re: [GENERAL] PostgreSQL Global Development Group Announces

----- Original Message -----
From: "Devrim GО©╫NDО©╫Z" <devrim@tr.net>
To: "PostgreSQL-development" <pgsql-hackers@postgresql.org>
Sent: Saturday, December 14, 2002 4:58 PM
Subject: Re: [HACKERS] [GENERAL] PostgreSQL Global Development Group
> Also, I have something to say about win32 port.
>
> I'm a Linux user. I'm happy that PostgreSQL does not have win32 version.
> If someone wants to use a real database server, then they should install
> Linux (or *bsd,etc). This is what Oracle offers,too. Native Windows
> support will cause some problems; such as some dummy windows users will
> begin using it. I do not believe that PostgreSQL needs native windowz
> support.

Ooops.
I'm a Linux user too, but i have a SCO Openserver, UnixWare, Netware and lot
of windows boxes in my office.
Also I have Informix, Sybase ... etc.
This isn't for my entertainment.
Our customers need to "use a real database server".
But what about small business?
A lot of our small customers can't spent money for dedicated linux box :(((

I spent  2 month in trying open source databases (PostgreSQL, SAP DB,
Interbase/Firebird)
finaly i choose PostgreSQL. Now we port one of our products from Sybase SQL
Anywhere to PostgreSQL.

We have more than 100 customers with small networks (2-10). Most of them
cant't aford dedicated linux box.
Another situation DHL Bulgaria and TNT Worldwide Express Bulgaria are our
customers too.
In HQ they choose windows nt (i don't comment how "smart" is this decision),
pay a lot of money to mr.Gates and now what - we say PostgreSQL is great ,
but ......
( and i have personal contacts with their sysadmins i don't believe they are
"dummy windows users")

So if you don't want windows support just don't use it!!!!!





Sat, 14 Dec 2002 16:31:58 +0100
"Igor Georgiev" <gory@alphasoft-bg.com>
Re: [GENERAL] PostgreSQL Global Development Group
Hi,

On Sat, 2002-12-14 at 13:26, mlw wrote:

> MySQL is an appalling database, but people use it, a lot! Why? Because 
> they really market it. They push it. They craft deceptive benchmarks 
> which show it is better. PostgreSQL doesn't even need to be deceptive.
> 
<snip>
> Furthermore, I think it would be very rewarding for everyone involved if 
> we could get some of the "street cred" that MySQL has. PostgreSQL *is* a 
> better database in almost every way. If MySQL virtually owns the open 
> source mind share for SQL databases, it is our fault.

I do NOT like hearing about MySQL in this (these) list(s).

PostgreSQL is not in the same category with MySQL. MySQL is for
*dummies*, not database admins. I do not even call  it a database. I
have never forgotten my data loss 2,5 years ago; when I used MySQL for
just 2 months!!! 

If we want to "sell" PostgreSQL, we should talk about, maybe, Oracle.
I have never took care of MySQL said. I just know that I'm running
PostgreSQL since 2,5 years and I only stopped it "JUST" before upgrades
of PostgreSQL. It's just *working*; which is unfamiliar to MySQL users. 

I've presented about 28 seminars in last 12 months on PostgreSQL... In
all of them, I always tried to avoid talking about MySQL. But always
"hit" Oracle. I'm sick of hearing such sentences : "We paid $$$$ to
Oracle, we hold 1 GB of data!". Even MySQL can hold that amount of data
:-) 

Also, I have something to say about win32 port.

I'm a Linux user. I'm happy that PostgreSQL does not have win32 version.
If someone wants to use a real database server, then they should install
Linux (or *bsd,etc). This is what Oracle offers,too. Native Windows
support will cause some problems; such as some dummy windows users will
begin using it. I do not believe that PostgreSQL needs native windowz
support. 

So, hackers (I'm not a hacker) should decide whether PostgreSQL should
be used widely in real database apps, or it should be used even by dummy
users?

I prefer the first one, if we want to compete with Oracle; not MySQL.

Best regards,

--
Devrim GUNDUZ
TR.NET System Support Specialist
devrim@tr.net


14 Dec 2002 15:58:29 +0000
Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= <devrim@tr.net>
Re: [GENERAL] PostgreSQL Global Development Group
Hi,

On Sat, 2002-12-14 at 15:31, Igor Georgiev wrote:

<snip>
> In HQ they choose windows nt (i don't comment how "smart" is this decision),
> pay a lot of money to mr.Gates and now what - we say PostgreSQL is great ,
> but ......
> ( and i have personal contacts with their sysadmins i don't believe they are
> "dummy windows users")

Hey, I did not say that "any windowz user is dummy". If you read my
previous post from the beginning; you'll see that my target is MySQL
users on Windows...

What I've been trying to say that is: If we have a chance to choose, I'd
prefer using PostgreSQL in *nix systems. This is what I've been doing
since 2,5 years. 

> So if you don't want windows support just don't use it!!!!!

I can't, even if I want it; since I do not have a windows installed
computer. ;-)

Anyway, this will be a "windows-linux" discussion; which is offtopic for
this list.

Best regards,
--
Devrim GUNDUZ
TR.NET System Support Specialist
devrim@tr.net


14 Dec 2002 16:56:24 +0000
Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= <devrim@tr.net>
Re: [GENERAL] PostgreSQL Global Development Group
Devrim G?ND?Z wrote:
> I do NOT like hearing about MySQL in this (these) list(s).
> 
> PostgreSQL is not in the same category with MySQL. MySQL is for
> *dummies*, not database admins. I do not even call  it a database. I
> have never forgotten my data loss 2,5 years ago; when I used MySQL for
> just 2 months!!! 

I think you're on to something here, but it's obscured by the way you
said it.

There's no question in my mind that PostgreSQL is superior in almost
every way to MySQL.  For those of us who are technically minded, it
boggles the mind that people would choose MySQL over PostgreSQL.  Yet
they do.  And it's important to understand why.

Simply saying "MySQL has better marketing" isn't enough.  It's too
simple an answer and obscures some issues that should probably be
addressed.

People use MySQL because it's very easy to set up, relatively easy to
maintain (when something doesn't go wrong, that is), is very well
documented and supported, and is initially adequate for the task they
have in mind (that the task may change significantly such that MySQL
is no longer adequate is something only those with experience will
consider).

PostgreSQL has come a long way and, with the exception of a few minor
things (the need to VACUUM, for instance.  The current version makes
the VACUUM requirement almost a non-issue as regards performance and
availability, but it really should be something that the database
takes care of itself), is equivalent to MySQL in the above things
except for documentation and support.

MySQL's documentation is very, very good.  My experience with it is
that it's possible, and relatively easy, to find information about
almost anything you might need to know.

PostgreSQL's documentation is good, but not quite as good as MySQL's.
It's not quite as complete.  For instance, I didn't find any
documentation at all in the User's Guide or Administrator's Guide on
creating tables (if I missed it, then that might illustrate that the
documentation needs to be organized slightly differently).  I did find
a little in the tutorial (about the amount that you'd want in a
tutorial), but to find out more I had to go to the SQL statement
reference (in my case I was looking for the means by which one could
create a constraint on a column during table creation time).

The reason this is important is that the documentation is *the* way
people are going to learn the database.  If it's too sparse or too
disorganized, people who don't have a lot of time to spend searching
through the documentation for something may well decide that a
different product (such as MySQL) would suit their needs better.

The documentation for PostgreSQL improves all the time, largely in
response to comments such as this one, and that's a very good thing.
My purpose in bringing this up is to show you what PostgreSQL is up
against in terms of widespread adoption.

> If we want to "sell" PostgreSQL, we should talk about, maybe, Oracle.
> I have never took care of MySQL said. I just know that I'm running
> PostgreSQL since 2,5 years and I only stopped it "JUST" before upgrades
> of PostgreSQL. It's just *working*; which is unfamiliar to MySQL
> users. 

The experience people have with MySQL varies a lot, and much of it has
to do with the load people put on it.  If MySQL were consistently bad
and unreliable it would have a much smaller following (since it's not
in a monopoly position the way Microsoft is).

But you're mistaken if you believe that MySQL isn't competition for
PostgreSQL.  It is, because it serves the same purpose: a means of
storing information in an easily retrievable way.

Selling potential MySQL users on PostgreSQL should be easier than
doing the same for Oracle users because potential MySQL users have at
least already decided that a free database is worthy of consideration.
As their needs grow beyond what MySQL offers, they'll look for a more
capable database engine.  It's a target market that we'd be idiots to
ignore, and we do so at our peril (the more people out there using
MySQL, the fewer there are using PostgreSQL).

> I'm a Linux user. I'm happy that PostgreSQL does not have win32 version.
> If someone wants to use a real database server, then they should install
> Linux (or *bsd,etc). This is what Oracle offers,too. Native Windows
> support will cause some problems; such as some dummy windows users will
> begin using it. I do not believe that PostgreSQL needs native windowz
> support. 

I hate to break it to you (assuming that I didn't misunderstand what
you said), but Oracle offers a native Windows port of their database
engine, and has done so for some time.  It's *stupid* to ignore the
native Windows market.  There are a lot of people who need a database
engine to store their data and who would benefit from a native Windows
implementation of PostgreSQL, but aren't interested in the additional
burden of setting up a Linux server because they lack the money, time,
or expertise.

> So, hackers (I'm not a hacker) should decide whether PostgreSQL should
> be used widely in real database apps, or it should be used even by dummy
> users?

What makes you think we can't meet the needs of both groups?  The
capabilities of PostgreSQL are (with very few exceptions) a superset
of MySQL's, which means that wherever someone deploys a MySQL server,
they could probably have deployed a PostgreSQL server in its place.
It should be an easy sell: they get a database engine that is
significantly more capable than MySQL for the same low price!

Selling to the Oracle market is going to be harder.  The capabilities
of Oracle are a superset of those of PostgreSQL.  Shops which plan to
deploy a database server and who need the capabilities of PostgreSQL
at a minimum are going to look at Oracle for the same reason that
shops which at a minimum need the capabilities of MySQL would be smart
to look at PostgreSQL: their needs may grow over time and changing the
database mid-project is difficult and time-consuming.  The difference
is that the prices of MySQL and PostgreSQL are the same, while the
prices of PostgreSQL and Oracle are vastly different.

That's not to say that going after the Oracle market shouldn't be done
(quite the opposite, provided it's done honestly), only that *not*
going after the MySQL market is folly.


-- 
Kevin Brown					      kevin@sysexperts.com

Sat, 14 Dec 2002 22:02:47 -0800
Kevin Brown <kevin@sysexperts.com>
Re: [HACKERS] PostgreSQL Global Development Group
Kevin Brown wrote:
> Devrim G?ND?Z wrote:
> > I do NOT like hearing about MySQL in this (these) list(s).
> > 
> > PostgreSQL is not in the same category with MySQL. MySQL is for
> > *dummies*, not database admins. I do not even call  it a database. I
> > have never forgotten my data loss 2,5 years ago; when I used MySQL for
> > just 2 months!!! 
> 
> I think you're on to something here, but it's obscured by the way you
> said it.
> 
> There's no question in my mind that PostgreSQL is superior in almost
> every way to MySQL.  For those of us who are technically minded, it
> boggles the mind that people would choose MySQL over PostgreSQL.  Yet
> they do.  And it's important to understand why.
> 
> Simply saying "MySQL has better marketing" isn't enough.  It's too
> simple an answer and obscures some issues that should probably be
> addressed.

I think it /is/ a significant factor, the point being that the MySQL company 
has been quite activist in pressing MySQL as "the answer," to the point to 
which there's a development strategy called "LAMP" (Linux + Apache + MySQL + 
(Perl|Python|PHP)).

> People use MySQL because it's very easy to set up, relatively easy to
> maintain (when something doesn't go wrong, that is), is very well
> documented and supported, and is initially adequate for the task they
> have in mind (that the task may change significantly such that MySQL
> is no longer adequate is something only those with experience will
> consider).

... And the consistent marketing pressure that in essence claims:

 - It's easier to use than any alternative;
 - It's much faster than any other DBMS;
 - It's plenty powerful and robust enough.

As near as I can tell, /none/ of these things are true outside of very 
carefully selected application domains.  But the claims have been presented 
enough times that people actually believe them to be true.

> PostgreSQL has come a long way and, with the exception of a few minor
> things (the need to VACUUM, for instance.  The current version makes
> the VACUUM requirement almost a non-issue as regards performance and
> availability, but it really should be something that the database
> takes care of itself), is equivalent to MySQL in the above things
> except for documentation and support.

I would point to a third thing:  Tools to support "hands-off administration."  
My web hosting provider has a set of tools to let me administer various 
aspects of my site complete with "pretty GUI" that covers:
 - Configuring email accounts, including mailing lists, Spam Assassin, and 
such;
 - Configuring subdomains;
 - Managing files/directories, doing backups;
 - Apache configuration;
 - Cron jobs;
 - A couple of "shopping cart" systems;
 - A "chat room system;"
 - Last, but certainly not least, the ability to manage MySQL databases.

There is no "canned" equivalent for PostgreSQL, which means that ISPs that 
don't have people with DBMS expertise will be inclined to prefer MySQL.  It's 
a better choice for them.

> MySQL's documentation is very, very good.  My experience with it is
> that it's possible, and relatively easy, to find information about
> almost anything you might need to know.
> 
> PostgreSQL's documentation is good, but not quite as good as MySQL's.
> It's not quite as complete.  For instance, I didn't find any
> documentation at all in the User's Guide or Administrator's Guide on
> creating tables (if I missed it, then that might illustrate that the
> documentation needs to be organized slightly differently).  I did find
> a little in the tutorial (about the amount that you'd want in a
> tutorial), but to find out more I had to go to the SQL statement
> reference (in my case I was looking for the means by which one could
> create a constraint on a column during table creation time).
> 
> The reason this is important is that the documentation is *the* way
> people are going to learn the database.  If it's too sparse or too
> disorganized, people who don't have a lot of time to spend searching
> through the documentation for something may well decide that a
> different product (such as MySQL) would suit their needs better.
> 
> The documentation for PostgreSQL improves all the time, largely in
> response to comments such as this one, and that's a very good thing.
> My purpose in bringing this up is to show you what PostgreSQL is up
> against in terms of widespread adoption.

That's probably pretty fair.  I'm using the word "fair" advisedly, too.

If someone objects, saying that PostgreSQL docs /are/ good, keep in mind that 
new users are not mandated to be "fair" about this.  If they have trouble 
finding what they were looking for, they couldn't care less that you think the 
docs are pretty good: /they/ didn't find what /they/ were looking for, and 
that's all they care about.

> > If we want to "sell" PostgreSQL, we should talk about, maybe, Oracle.
> > I have never took care of MySQL said. I just know that I'm running
> > PostgreSQL since 2,5 years and I only stopped it "JUST" before upgrades
> > of PostgreSQL. It's just *working*; which is unfamiliar to MySQL
> > users. 
> 
> The experience people have with MySQL varies a lot, and much of it has
> to do with the load people put on it.  If MySQL were consistently bad
> and unreliable it would have a much smaller following (since it's not
> in a monopoly position the way Microsoft is).
> 
> But you're mistaken if you believe that MySQL isn't competition for
> PostgreSQL.  It is, because it serves the same purpose: a means of
> storing information in an easily retrievable way.

Indeed.  People with modest data storage requirements that came in with /no/ 
comprehension of what a "relational" database is may find the limited 
functionality of MySQL perfectly reasonable for their purposes.

And I'll pull in a quote I saw on comp.databases this week that I think is 
quite fabulous:

-------------------------------------------------------------

>>if you mean by "ideal" that it runs on Unix and crashes all the time
>>and needs a bazillion DBA's to keep them running and you want to
>>constantly recover your database and your data files, then you can
>>have ideal.

> A little background on my original comment might be in order. I
> don't tend to use the term "ideal" myself, much. I was referring to
> a comment made fairly frequently in this forum, to the effect that
> "A commercial Relational Databse system has never been built." These
> people exclude Oracle, SQL Server, DB2, Informix, Interbase, yada
> yada, because all of them fail, in one way or another to live up to
> the "ideal" of a truly relational system. I have a hard time with such
> terminological rigidity, myself. One can say that all those products
> aren't perfect relational products, but one shouldn't, in my view, say
> that they "aren't even relational".

Why do you think that they are "relational" ? Do they operate on relations ? I 
don't think so. If their primary business is not to operate on relations but 
on bags of rows, calling them relational is misleading.

Just like ODBMS are often database construction kits or persistence libraries, 
SQL DBMSes are a real DBMS (they do provide transactions, recovery, 
concurrency control, some data integrity) + a *relational construction kit*. 
Meaning that by a skillful use of SQL one can come somewhere close to a 
relational database.

But the complexity is left on the user to shoulder, and it is very difficult 
to stretch SQL so that you are still in the realm of relational model. And 
guess what: most users don't and most users suffer as a consequence.

It's even worse than that : very often product documentation and books 
sponsored by the vendors (Oracle press: anyone there ?) simply lie to the 
users by defining relational model in the most ridiculous terms. Actually they 
screwed up their products, they built a multi-billion dollars industry by 
taking agressive shortcuts on the implementation side and transfering the 
complexity to the user and now they try to lie and cheat by proclaiming their 
version of "relational" (not long ago the auto industry maintained seat belts 
and airbags were unnecessarily expensive and not needed).

Best regards,
Costin Cozianu
-------------------------------------------------------------

The interesting argument that Costin makes is that SQL databases are /not/ 
"relational databases," but rather that they are tools that can be used to 
construct relational database systems.

PostgreSQL has enough decent constructs, what with mature implementations of 
foreign keys, views, and constraints that it is fairly easy to build 
relational systems using PostgreSQL.  In contrast, the paucity of supportive 
constructs in MySQL means that neither the database nor the resulting 
applications are likely to be terribly "relational" in the senses intended by 
Codd and Date.

> Selling potential MySQL users on PostgreSQL should be easier than
> doing the same for Oracle users because potential MySQL users have at
> least already decided that a free database is worthy of consideration.
> As their needs grow beyond what MySQL offers, they'll look for a more
> capable database engine.  It's a target market that we'd be idiots to
> ignore, and we do so at our peril (the more people out there using
> MySQL, the fewer there are using PostgreSQL).

The unfortunate part is that those that outgrow MySQL are likely to have /two/ 
misconceptions:

1.  That the only /real/ reliability improvement will come in moving to 
something like Oracle;

2.  That PostgreSQL will be a huge step backwards into performance problems 
because it is "so much slower."

That these are misconceptions does not prevent people from believing them.  
(The third deceptive misconception I see is that MySQL is somehow "more 
standard" than some of its competitors.)

> > I'm a Linux user. I'm happy that PostgreSQL does not have win32 version.
> > If someone wants to use a real database server, then they should install
> > Linux (or *bsd,etc). This is what Oracle offers,too. Native Windows
> > support will cause some problems; such as some dummy windows users will
> > begin using it. I do not believe that PostgreSQL needs native windowz
> > support. 
> 
> I hate to break it to you (assuming that I didn't misunderstand what
> you said), but Oracle offers a native Windows port of their database
> engine, and has done so for some time.  It's *stupid* to ignore the
> native Windows market.  There are a lot of people who need a database
> engine to store their data and who would benefit from a native Windows
> implementation of PostgreSQL, but aren't interested in the additional
> burden of setting up a Linux server because they lack the money, time,
> or expertise.

I think it would be a Bad Thing if making PostgreSQL support Windows better 
were to compromise how well it works on Unix, but I haven't seen evidence of 
anyone actually proposing patches that would have that result.

> > So, hackers (I'm not a hacker) should decide whether PostgreSQL should
> > be used widely in real database apps, or it should be used even by dummy
> > users?
> 
> What makes you think we can't meet the needs of both groups?  The
> capabilities of PostgreSQL are (with very few exceptions) a superset
> of MySQL's, which means that wherever someone deploys a MySQL server,
> they could probably have deployed a PostgreSQL server in its place.
> It should be an easy sell: they get a database engine that is
> significantly more capable than MySQL for the same low price!

You can't sell into the "ISP appliance market" until there's something as 
ubiquitous as "PHPMyAdmin" for PostgreSQL.  And note that the "ISP appliance 
market" only cares about this in a very indirect way.  They don't actually use 
the database; their /customers/ do.  And their customers are likely to be 
fairly unsophisticated souls who will use whatever database is given to them.

> Selling to the Oracle market is going to be harder.  The capabilities
> of Oracle are a superset of those of PostgreSQL.  Shops which plan to
> deploy a database server and who need the capabilities of PostgreSQL
> at a minimum are going to look at Oracle for the same reason that
> shops which at a minimum need the capabilities of MySQL would be smart
> to look at PostgreSQL: their needs may grow over time and changing the
> database mid-project is difficult and time-consuming.  The difference
> is that the prices of MySQL and PostgreSQL are the same, while the
> prices of PostgreSQL and Oracle are vastly different.

There are Oracle markets /not/ worth going after, at this point.  You /don't/ 
go after the "ERP" markets or the data center markets where license budgets 
are in millions of dollars, and where it's going to be tough to take 
PostgreSQL seriously when Oracle is entirely prepared to send in a group of 10 
technical marketing people to swamp the customer with marketing information.

What /is/ worth going after is the "small server" market, for departmental 
applications.  It's not "big bucks;" in the Oracle realm, it might lead to a 
licensing fee of $20K.  For $20K, they aren't going to send in a swarm of 
marketers to fight for the account.

> That's not to say that going after the Oracle market shouldn't be done
> (quite the opposite, provided it's done honestly), only that *not*
> going after the MySQL market is folly.

Indeed.

It is almost a "necessary defense" to counter the deceptive claims that are 
made.  If nobody says anything, people may actually /believe/ that PostgreSQL 
is vastly slower.
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://cbbrowne.com/info/nonrdbms.html
"Power tends  to corrupt and absolute power  corrupts absolutely."  
-- First Baron Acton, 1834 - 1902



Sun, 15 Dec 2002 16:40:33 -0500
cbbrowne@cbbrowne.com
Re: [GENERAL] PostgreSQL Global Development Group 
> You can't sell into the "ISP appliance market" until there's something as
> ubiquitous as "PHPMyAdmin" for PostgreSQL.  And note that the "ISP
appliance
> market" only cares about this in a very indirect way.  They don't actually
use
> the database; their /customers/ do.  And their customers are likely to be
> fairly unsophisticated souls who will use whatever database is given to
them.

Hey!  What about phpPgAdmin?

We're actually working on a next generation version atm which is a total
rewrite that:

1. modern php
2. register_globals off, full error checking
3. themable
4. Easily supports all versions
5. etc.

However, even with repeated calls for developers, it's just me and Rob
Treat!

phpPgAdmin does not work with 7.3 so this in an increasingly important
project.

Anyone wanna help? :)

http://phppgdamin.sourceforge.net/

Maybe we should move to gborg?

Chris



Mon, 16 Dec 2002 10:23:27 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: [GENERAL] PostgreSQL Global Development Group 
cbbrowne@cbbrowne.com wrote:
> Kevin Brown wrote:
> > Simply saying "MySQL has better marketing" isn't enough.  It's too
> > simple an answer and obscures some issues that should probably be
> > addressed.
> 
> I think it /is/ a significant factor, the point being that the MySQL company 
> has been quite activist in pressing MySQL as "the answer," to the point to 
> which there's a development strategy called "LAMP" (Linux + Apache + MySQL + 
> (Perl|Python|PHP)).

Oh, I'll certainly not dispute that marketing has had a significant
effect, but I don't think it's the only reason for MySQL's success.

History has a lot to do with it, because it's through history that
momentum gets built up, as it has with MySQL.

> > People use MySQL because it's very easy to set up, relatively easy to
> > maintain (when something doesn't go wrong, that is), is very well
> > documented and supported, and is initially adequate for the task they
> > have in mind (that the task may change significantly such that MySQL
> > is no longer adequate is something only those with experience will
> > consider).
> 
> ... And the consistent marketing pressure that in essence claims:
> 
>  - It's easier to use than any alternative;
>  - It's much faster than any other DBMS;
>  - It's plenty powerful and robust enough.
> 
> As near as I can tell, /none/ of these things are true outside of very 
> carefully selected application domains.  But the claims have been presented 
> enough times that people actually believe them to be true.

I agree with you -- now.  But the situation as it is now has not
always been.  Consider where PostgreSQL was 4 years ago.  I believe it
was at version 6 at that time, if I remember correctly.  And as I
recall, many people had very significant issues with it in the key
areas of performance and reliability.  Now, I didn't experience these
things firsthand because I wasn't using it at the time, but it is the
general impression I got when reading the accounts of people who
*were* using it.

MySQL at the time wasn't necessarily any more reliable, but it had one
thing going for it that PostgreSQL didn't: myisamchk.  Even if the
database crashed, you stood a very good chance of being able to
recover your data without having to restore from backups.  PostgreSQL
didn't have this at all: either you had to be a guru with the
PostgreSQL database format or you had to restore from backups.  That
meant that *in practice* MySQL was easier to maintain, even it crashed
more often as PostgreSQL, because the amount of administrative effort
to deal with a MySQL crash was so much less.

> > PostgreSQL has come a long way and, with the exception of a few minor
> > things (the need to VACUUM, for instance.  The current version makes
> > the VACUUM requirement almost a non-issue as regards performance and
> > availability, but it really should be something that the database
> > takes care of itself), is equivalent to MySQL in the above things
> > except for documentation and support.
> 
> I would point to a third thing: Tools to support "hands-off
> administration."  My web hosting provider has a set of tools to let
> me administer various aspects of my site complete with "pretty GUI"
> that covers:
>
>  - Configuring email accounts, including mailing lists, Spam
>    Assassin, and such;
>  - Configuring subdomains;
>  - Managing files/directories, doing backups;
>  - Apache configuration;
>  - Cron jobs;
>  - A couple of "shopping cart" systems;
>  - A "chat room system;"
>  - Last, but certainly not least, the ability to manage MySQL
>    databases.
> 
> There is no "canned" equivalent for PostgreSQL, which means that
> ISPs that don't have people with DBMS expertise will be inclined to
> prefer MySQL.  It's a better choice for them.

This is true, but the only way to combat that is to get PostgreSQL
more widely deployed.  Network effects such as that are common in the
computing world, so it doesn't come as much surprise that the most
popular database engine in the webhosting world is the best supported
one for that role.

It's only because of the relative popularity of MySQL that it has so
much support.  The only way to grow PostgreSQL's popularity is to get
it deployed in situations where the tools available for it are
sufficient.

> > But you're mistaken if you believe that MySQL isn't competition for
> > PostgreSQL.  It is, because it serves the same purpose: a means of
> > storing information in an easily retrievable way.
> 
> Indeed.  People with modest data storage requirements that came in
> with /no/ comprehension of what a "relational" database is may find
> the limited functionality of MySQL perfectly reasonable for their
> purposes.

This is true, but the biggest problem is that the requirements of a
project often balloon over time, and the demands on the database
backend will also tend to increase.  Because MySQL is rather limited
in its functionality, it doesn't take much until you'll be forced to
use a different database backend.

This is why I view PostgreSQL as a much wiser choice in almost all
cases where you need a database engine.  Your needs will have to be
quite considerable before PostgreSQL's capabilities are no longer
enough.

> PostgreSQL has enough decent constructs, what with mature
> implementations of foreign keys, views, and constraints that it is
> fairly easy to build relational systems using PostgreSQL.  In
> contrast, the paucity of supportive constructs in MySQL means that
> neither the database nor the resulting applications are likely to be
> terribly "relational" in the senses intended by Codd and Date.

This is true, but what everyone fails to ask is whether or not any
particular customer really *cares* about that.  The customer isn't
interested in whether or not an application is "relational", they care
whether or not the application does the job it's supposed to.  How
"relational" it is is an implementation detail to them.

The reason that PostgreSQL wins over MySQL is not so much that it's
easier to build relational systems with it, but that it's easier to
build *reliable* systems with it.  That building the system in a
relational way is one way to achieve that is, again, an implementation
detail.

> > Selling potential MySQL users on PostgreSQL should be easier than
> > doing the same for Oracle users because potential MySQL users have at
> > least already decided that a free database is worthy of consideration.
> > As their needs grow beyond what MySQL offers, they'll look for a more
> > capable database engine.  It's a target market that we'd be idiots to
> > ignore, and we do so at our peril (the more people out there using
> > MySQL, the fewer there are using PostgreSQL).
> 
> The unfortunate part is that those that outgrow MySQL are likely to
> have /two/ misconceptions:
> 
> 1.  That the only /real/ reliability improvement will come in moving to 
> something like Oracle;
>
> 2.  That PostgreSQL will be a huge step backwards into performance problems 
> because it is "so much slower."

This is because people lack familiarity with PostgreSQL.  That's where
marketing PostgreSQL well comes in.

The performance misconception is the result of history.  At one time
PostgreSQL *was* much slower than MySQL.  People need to be informed
of the current state of affairs.

> That these are misconceptions does not prevent people from believing them.  
> (The third deceptive misconception I see is that MySQL is somehow "more 
> standard" than some of its competitors.)

The third misconception happens because most people equate "standard"
with "popular".  And in the real world, they're not entirely wrong to
do so, unfortunately.

> I think it would be a Bad Thing if making PostgreSQL support Windows
> better were to compromise how well it works on Unix, but I haven't
> seen evidence of anyone actually proposing patches that would have
> that result.

I agree, and I also believe that the maintainers would not accept a
patch that compromised the performance under Unix for the sake of
supporting Windows.  And rightly so: such a patch would indicate that
the people doing the Windows port haven't solved the problem properly.

> You can't sell into the "ISP appliance market" until there's
> something as ubiquitous as "PHPMyAdmin" for PostgreSQL.

But there is: PHPPgAdmin (or whatever it's called these days.  I seem
to remember that they changed the name of it).  Unfortunately it's not
as well known, largely because PostgreSQL itself isn't as well known.

> And note that the "ISP appliance market" only cares about this in a
> very indirect way.  They don't actually use the database; their
> /customers/ do.  And their customers are likely to be fairly
> unsophisticated souls who will use whatever database is given to
> them.

And if that's *really* true, then providers will do just as well to
provide PostgreSQL as they would MySQL (since their customers will
just use whatever database they're given).  So it's really a question
of selling the providers on it, which (as you mentioned earlier) is in
part a matter of giving them the tools they need to make managing a
PostgreSQL installation easy.

> There are Oracle markets /not/ worth going after, at this point.
> You /don't/ go after the "ERP" markets or the data center markets
> where license budgets are in millions of dollars, and where it's
> going to be tough to take PostgreSQL seriously when Oracle is
> entirely prepared to send in a group of 10 technical marketing
> people to swamp the customer with marketing information.

This is why marketing PostgreSQL *honestly* is so important.  If it
won't do the ERP job well, then it behooves those who are promoting it
to realize that and restrain themselves appropriately.

> What /is/ worth going after is the "small server" market, for
> departmental applications.  It's not "big bucks;" in the Oracle
> realm, it might lead to a licensing fee of $20K.  For $20K, they
> aren't going to send in a swarm of marketers to fight for the
> account.

And this is exactly one of the markets that MySQL is currently
targeting.  Of course, MS-SQL is *also* targeting this market, with a
reasonable amount of success.  PostgreSQL is a *perfect* fit for this
kind of operation, and it's one of the reasons that it really *is*
important to have a native Windows port.

> > That's not to say that going after the Oracle market shouldn't be done
> > (quite the opposite, provided it's done honestly), only that *not*
> > going after the MySQL market is folly.
> 
> Indeed.
> 
> It is almost a "necessary defense" to counter the deceptive claims
> that are made.  If nobody says anything, people may actually
> /believe/ that PostgreSQL is vastly slower.

The way you counter such deceptive claims is to provide proof that
those claims are wrong.  Point them at the head-to-head comparison on
the PHPBuilder site.  Prove to them that PostgreSQL is in the same
league (if not better) as MySQL in the performance arena.  And for
deity's sake, show them how much *less* work they'd have to do under
PostgreSQL because of its referential integrity features.  I really
think most people would be willing to sacrifice a small bit of speed
if it meant doing a whole lot less work.


Copied to the advocacy group because of the relevance.


-- 
Kevin Brown					      kevin@sysexperts.com

Sun, 15 Dec 2002 23:15:49 -0800
Kevin Brown <kevin@sysexperts.com>
Re: [HACKERS] [GENERAL] PostgreSQL Global Development Group
Something I have done at little cost was to submit a request for a few 
books on
PostgreSQL to my local library and I check them out once in while and 
see that
others are also checking them out.....

I agree with poor level of documentation....the skeleton is 
there....perhpas we could
have some volunteers write up some parts (or add more....)...

Kevin Brown wrote:

>Devrim G?ND?Z wrote:
>  
>
>>I do NOT like hearing about MySQL in this (these) list(s).
>>
>>PostgreSQL is not in the same category with MySQL. MySQL is for
>>*dummies*, not database admins. I do not even call  it a database. I
>>have never forgotten my data loss 2,5 years ago; when I used MySQL for
>>just 2 months!!! 
>>    
>>
>
>I think you're on to something here, but it's obscured by the way you
>said it.
>
>There's no question in my mind that PostgreSQL is superior in almost
>every way to MySQL.  For those of us who are technically minded, it
>boggles the mind that people would choose MySQL over PostgreSQL.  Yet
>they do.  And it's important to understand why.
>
>Simply saying "MySQL has better marketing" isn't enough.  It's too
>simple an answer and obscures some issues that should probably be
>addressed.
>
>People use MySQL because it's very easy to set up, relatively easy to
>maintain (when something doesn't go wrong, that is), is very well
>documented and supported, and is initially adequate for the task they
>have in mind (that the task may change significantly such that MySQL
>is no longer adequate is something only those with experience will
>consider).
>
>PostgreSQL has come a long way and, with the exception of a few minor
>things (the need to VACUUM, for instance.  The current version makes
>the VACUUM requirement almost a non-issue as regards performance and
>availability, but it really should be something that the database
>takes care of itself), is equivalent to MySQL in the above things
>except for documentation and support.
>
>MySQL's documentation is very, very good.  My experience with it is
>that it's possible, and relatively easy, to find information about
>almost anything you might need to know.
>
>PostgreSQL's documentation is good, but not quite as good as MySQL's.
>It's not quite as complete.  For instance, I didn't find any
>documentation at all in the User's Guide or Administrator's Guide on
>creating tables (if I missed it, then that might illustrate that the
>documentation needs to be organized slightly differently).  I did find
>a little in the tutorial (about the amount that you'd want in a
>tutorial), but to find out more I had to go to the SQL statement
>reference (in my case I was looking for the means by which one could
>create a constraint on a column during table creation time).
>
>The reason this is important is that the documentation is *the* way
>people are going to learn the database.  If it's too sparse or too
>disorganized, people who don't have a lot of time to spend searching
>through the documentation for something may well decide that a
>different product (such as MySQL) would suit their needs better.
>
>The documentation for PostgreSQL improves all the time, largely in
>response to comments such as this one, and that's a very good thing.
>My purpose in bringing this up is to show you what PostgreSQL is up
>against in terms of widespread adoption.
>
>  
>
>>If we want to "sell" PostgreSQL, we should talk about, maybe, Oracle.
>>I have never took care of MySQL said. I just know that I'm running
>>PostgreSQL since 2,5 years and I only stopped it "JUST" before upgrades
>>of PostgreSQL. It's just *working*; which is unfamiliar to MySQL
>>users. 
>>    
>>
>
>The experience people have with MySQL varies a lot, and much of it has
>to do with the load people put on it.  If MySQL were consistently bad
>and unreliable it would have a much smaller following (since it's not
>in a monopoly position the way Microsoft is).
>
>But you're mistaken if you believe that MySQL isn't competition for
>PostgreSQL.  It is, because it serves the same purpose: a means of
>storing information in an easily retrievable way.
>
>Selling potential MySQL users on PostgreSQL should be easier than
>doing the same for Oracle users because potential MySQL users have at
>least already decided that a free database is worthy of consideration.
>As their needs grow beyond what MySQL offers, they'll look for a more
>capable database engine.  It's a target market that we'd be idiots to
>ignore, and we do so at our peril (the more people out there using
>MySQL, the fewer there are using PostgreSQL).
>
>  
>
>>I'm a Linux user. I'm happy that PostgreSQL does not have win32 version.
>>If someone wants to use a real database server, then they should install
>>Linux (or *bsd,etc). This is what Oracle offers,too. Native Windows
>>support will cause some problems; such as some dummy windows users will
>>begin using it. I do not believe that PostgreSQL needs native windowz
>>support. 
>>    
>>
>
>I hate to break it to you (assuming that I didn't misunderstand what
>you said), but Oracle offers a native Windows port of their database
>engine, and has done so for some time.  It's *stupid* to ignore the
>native Windows market.  There are a lot of people who need a database
>engine to store their data and who would benefit from a native Windows
>implementation of PostgreSQL, but aren't interested in the additional
>burden of setting up a Linux server because they lack the money, time,
>or expertise.
>
>  
>
>>So, hackers (I'm not a hacker) should decide whether PostgreSQL should
>>be used widely in real database apps, or it should be used even by dummy
>>users?
>>    
>>
>
>What makes you think we can't meet the needs of both groups?  The
>capabilities of PostgreSQL are (with very few exceptions) a superset
>of MySQL's, which means that wherever someone deploys a MySQL server,
>they could probably have deployed a PostgreSQL server in its place.
>It should be an easy sell: they get a database engine that is
>significantly more capable than MySQL for the same low price!
>
>Selling to the Oracle market is going to be harder.  The capabilities
>of Oracle are a superset of those of PostgreSQL.  Shops which plan to
>deploy a database server and who need the capabilities of PostgreSQL
>at a minimum are going to look at Oracle for the same reason that
>shops which at a minimum need the capabilities of MySQL would be smart
>to look at PostgreSQL: their needs may grow over time and changing the
>database mid-project is difficult and time-consuming.  The difference
>is that the prices of MySQL and PostgreSQL are the same, while the
>prices of PostgreSQL and Oracle are vastly different.
>
>That's not to say that going after the Oracle market shouldn't be done
>(quite the opposite, provided it's done honestly), only that *not*
>going after the MySQL market is folly.
>
>
>  
>




Mon, 16 Dec 2002 13:15:46 -0800
Medi Montaseri <medi.montaseri@intransa.com>
Re: [HACKERS] PostgreSQL Global Development Group

Last night, we packaged up v7.3.1 of PostgreSQL, our latest stable
release.

Purely meant to be a bug fix release, this one does have one major change,
in that the major number of the libpq library was increased, which means
that everyone is encouraged to recompile their clients along with this
upgrade.

This release can be found on all the mirrors, and on the root ftp server,
under:

		ftp://ftp.postgresql.org/pub/source/v7.3.1

Please report all bugs to pgsql-bugs@postgresql.org.




Sun, 22 Dec 2002 15:12:05 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
v7.3.1 Bundled and Released ...
On Sun, 2002-12-22 at 13:12, Marc G. Fournier wrote:
> Last night, we packaged up v7.3.1 of PostgreSQL, our latest stable
> release.
> 
> Purely meant to be a bug fix release, this one does have one major change,
> in that the major number of the libpq library was increased, which means
> that everyone is encouraged to recompile their clients along with this
> upgrade.
> 
> This release can be found on all the mirrors, and on the root ftp server,
> under:
> 
> 		ftp://ftp.postgresql.org/pub/source/v7.3.1
> 
> Please report all bugs to pgsql-bugs@postgresql.org.
> 


Hmm.  For some reason I'm not seeing a 7.3.1 tag in CVS.  Do you guys do
something else for sub-releases?  Case in point:
cvs [server aborted]: no such tag REL7_3_1_STABLE
It's still early here so I may be suffering from early morning brain
rot.  ;)


Regards,

	Greg



-- 
Greg Copeland <greg@copelandconsulting.net>
Copeland Computer Consulting


23 Dec 2002 07:07:07 -0600
Greg Copeland <greg@CopelandConsulting.Net>
Re: [HACKERS] v7.3.1 Bundled and Released ...
Greg Copeland wrote:
> On Sun, 2002-12-22 at 13:12, Marc G. Fournier wrote:
> > Last night, we packaged up v7.3.1 of PostgreSQL, our latest stable
> > release.
> > 
> > Purely meant to be a bug fix release, this one does have one major change,
> > in that the major number of the libpq library was increased, which means
> > that everyone is encouraged to recompile their clients along with this
> > upgrade.
> > 
> > This release can be found on all the mirrors, and on the root ftp server,
> > under:
> > 
> > 		ftp://ftp.postgresql.org/pub/source/v7.3.1
> > 
> > Please report all bugs to pgsql-bugs@postgresql.org.
> > 
> 
> 
> Hmm.  For some reason I'm not seeing a 7.3.1 tag in CVS.  Do you guys do
> something else for sub-releases?  Case in point:
> cvs [server aborted]: no such tag REL7_3_1_STABLE
> It's still early here so I may be suffering from early morning brain
> rot.  ;)

There should be a 7.3.1 tag, but you can use the 7_3 branch to pull
7.3.1.  Of course, that will shift as we patch for 7.3.2.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 23 Dec 2002 10:57:52 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] v7.3.1 Bundled and Released ...
On Sunday 22 December 2002 14:12, Marc G. Fournier wrote:
> Last night, we packaged up v7.3.1 of PostgreSQL, our latest stable
> release.

RPMs are available for RedHat8 + tcl8.4.1 at 
ftp://ftp.postgresql.org/pub/binary/v7.3.1/RPMS (once mirrors propagate it 
will be at the mirrors)

Yes, Red Hat 8.0 _+_ tcl8.4.1.  If you wish to use the pl subpackage without 
pl/tcl, you need to add --nodeps to the rpm command line when installing the 
pl subpackage.  The postgresql-tcl subpackage was built against tcl 8.4.1 -- 
Red Hat 8 ships with an old tcl (8.3.3) which is not even installed by 
default.  If this is too much of a problem for people, please let me know, 
and I'l rebuild for Red Hat 8 without any tcl support, or with the older tcl 
support.  Otherwise obtain tcl 8.4.1.  I have to have it installed here for 
work purposes.  This is the only exception to the 'pristine Red Hat install' 
I have made in a very long time.  Rebuilding from source RPM will require an 
installed tcl of some version.

Source RPMS in SRPMS, Red Hat 8 binaries in redhat-8.0

If installing the contrib subpackage, you must use --nodeps due to a broken 
dependency upon the now removed postgresql-perl subpackage.  There will be 
soon a postgresql-perl RPM -- I have to get the new gborg code and build it, 
I guess.

Changelog since 7.3-2:
* Mon Dec 23 2002 Lamar Owen <lamar.owen@ramifordistat.net>
- 7.3.1-1PGDG
- Fix dependency order for test and pl subpackages.
- Fixed a bug in the initscript for echo_success
- Fixed the bug in .bashprofile (argh).

(quick note: lamar.owen@ramifordistat.net does work.  I typically post from 
lamar.owen@wgcr.org; both addresses come to me....)
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Mon, 23 Dec 2002 14:12:16 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [HACKERS] v7.3.1 Bundled and Released ...
Just a reminder, there still doesn't appear to be a 7.3.1 tag.

This is from the "HISTORY" file.

symbolic names:
        REL7_3_STABLE: 1.182.0.2
        REL7_2_3: 1.153.2.8
        REL7_2_STABLE: 1.153.0.2
        REL7_2: 1.153


Notice 7.3 stable but nothing about 7.3.x!  I also see a 7.2.3, etc.,
just as one would expect but nothing about 7.3 dot releases.

I'm still getting, "cvs [server aborted]: no such tag REL7_3_1_STABLE". 
Something overlooked here?


Regards,

	Greg Copeland


On Mon, 2002-12-23 at 09:57, Bruce Momjian wrote:
> Greg Copeland wrote:
> > On Sun, 2002-12-22 at 13:12, Marc G. Fournier wrote:
> > > Last night, we packaged up v7.3.1 of PostgreSQL, our latest stable
> > > release.
> > > 
> > > Purely meant to be a bug fix release, this one does have one major change,
> > > in that the major number of the libpq library was increased, which means
> > > that everyone is encouraged to recompile their clients along with this
> > > upgrade.
> > > 
> > > This release can be found on all the mirrors, and on the root ftp server,
> > > under:
> > > 
> > > 		ftp://ftp.postgresql.org/pub/source/v7.3.1
> > > 
> > > Please report all bugs to pgsql-bugs@postgresql.org.
> > > 
> > 
> > 
> > Hmm.  For some reason I'm not seeing a 7.3.1 tag in CVS.  Do you guys do
> > something else for sub-releases?  Case in point:
> > cvs [server aborted]: no such tag REL7_3_1_STABLE
> > It's still early here so I may be suffering from early morning brain
> > rot.  ;)
> 
> There should be a 7.3.1 tag, but you can use the 7_3 branch to pull
> 7.3.1.  Of course, that will shift as we patch for 7.3.2.
-- 
Greg Copeland <greg@copelandconsulting.net>
Copeland Computer Consulting


29 Dec 2002 21:51:46 -0600
Greg Copeland <greg@CopelandConsulting.Net>
Re: [HACKERS] v7.3.1 Bundled and Released ...
Greg Copeland writes:

> Just a reminder, there still doesn't appear to be a 7.3.1 tag.

There is a long tradition of systematically failing to tag releases in
this project.  Don't expect it to improve.

-- 
Peter Eisentraut   peter_e@gmx.net


Sat, 4 Jan 2003 11:27:53 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [GENERAL] v7.3.1 Bundled and Released ...
On Sat, 4 Jan 2003, Peter Eisentraut wrote:

> Greg Copeland writes:
>
> > Just a reminder, there still doesn't appear to be a 7.3.1 tag.
>
> There is a long tradition of systematically failing to tag releases in
> this project.  Don't expect it to improve.

It was I who suggested that a release team would be a good idea.  I think
that was soundly rejected.  I still think it's a good idea.  If only to
ensure that things are properly tagged, the right annoucements go out at
the right times, that a code freeze goes into effect, etc. These concepts
are not new.  A release is an important step in the life cycle.

I volunteered to document the release procedure as it resides only within
lore and a couple of heads.  I have yet to start.


Sat, 4 Jan 2003 06:44:18 -0500 (EST)
Dan Langille <dan@langille.org>
Re: [GENERAL] v7.3.1 Bundled and Released ...
Dan Langille <dan@langille.org> writes:
> On Sat, 4 Jan 2003, Peter Eisentraut wrote:
>> There is a long tradition of systematically failing to tag releases in
>> this project.  Don't expect it to improve.

> It was I who suggested that a release team would be a good idea.

We *have* a release team.  Your problem is that Marc, who is the man who
would need to do this, doesn't appear to consider it an important thing
to do.  Try to convince him to put it on his checklist.

			regards, tom lane

Sat, 04 Jan 2003 11:08:05 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [GENERAL] v7.3.1 Bundled and Released ... 
On Sat, 2003-01-04 at 04:27, Peter Eisentraut wrote:
> Greg Copeland writes:
> 
> > Just a reminder, there still doesn't appear to be a 7.3.1 tag.
> 
> There is a long tradition of systematically failing to tag releases in
> this project.  Don't expect it to improve.

Well, I thought I remembered from the "release team" thread that it was
said there was a "punch list" of things that are done prior to actually
releasing.  If not, it certainly seems like we need one.  If there is
one, tagging absolutely needs to be on it.  If we have one and this is
already on the list, seems we need to be eating our own food.  ;)


-- 
Greg Copeland <greg@copelandconsulting.net>
Copeland Computer Consulting


04 Jan 2003 12:13:17 -0600
Greg Copeland <greg@CopelandConsulting.Net>
Re: [GENERAL] v7.3.1 Bundled and Released ...
msg resent because I incorrectly copied/pasted some addresses.  
Sorry.

On 4 Jan 2003 at 11:08, Tom Lane wrote:

> Dan Langille <dan@langille.org> writes:
> > On Sat, 4 Jan 2003, Peter Eisentraut wrote:
> >> There is a long tradition of systematically failing to tag releases
> >> in this project.  Don't expect it to improve.
> 
> > It was I who suggested that a release team would be a good idea.
> 
> We *have* a release team.

I have a suggestion.  Let us document who is the release team and who 
is responsible for each step of the release.  Perhaps that is the 
problem: a lack of process.

I'll add that to my list of things I've promised to do.
-- 
Dan Langille : http://www.langille.org/


Sat, 04 Jan 2003 15:56:41 -0500
"Dan Langille" <dan@langille.org>
Re: [GENERAL] v7.3.1 Bundled and Released ... 
msg resent because I incorrectly copied/pasted some addresses.  Sorry.

On 4 Jan 2003 at 11:08, Tom Lane wrote:

> Dan Langille <dan@langille.org> writes:
> > On Sat, 4 Jan 2003, Peter Eisentraut wrote:
> >> There is a long tradition of systematically failing to tag releases
> >> in this project.  Don't expect it to improve.
> 
> > It was I who suggested that a release team would be a good idea.
> 
> We *have* a release team.  Your problem is that Marc, who is the man
> who would need to do this, doesn't appear to consider it an important
> thing to do.  Try to convince him to put it on his checklist.

Marc?  Is this true?  You don't consider it important to tag the 
release?  I'm quite sure that's not the case and that Marc does 
consider it important.  It's just something which he forgot to do.

A recent post by Greg Copeland implies this item is on his checklist.

IMHO, it is vital that the tree is properly tagged for each release.  
AFAIK, a tag can be laid with with respect to timestamp value.  So 
why don't we just do it?
-- 
Dan Langille : http://www.langille.org/


Sat, 04 Jan 2003 15:56:56 -0500
"Dan Langille" <dan@langille.org>
Re: [GENERAL] v7.3.1 Bundled and Released ... 
On Sat, 4 Jan 2003, Tom Lane wrote:

> Dan Langille <dan@langille.org> writes:
> > On Sat, 4 Jan 2003, Peter Eisentraut wrote:
> >> There is a long tradition of systematically failing to tag releases in
> >> this project.  Don't expect it to improve.
>
> > It was I who suggested that a release team would be a good idea.
>
> We *have* a release team.  Your problem is that Marc, who is the man who
> would need to do this, doesn't appear to consider it an important thing
> to do.  Try to convince him to put it on his checklist.

I never considered tag'ng for minor releases as having any importance,
since the tarball's themselves provide the 'tag' ... branches give us the
ability to back-patch, but tag's don't provide us anything ... do they?

That said, I can back-tag the whole source tree for past releases if ppl
do think it is important, its just a matter of knowing the 'timestamp' to
base it on, which I can do based on the dates of the tar files ...

Its not like tag'ng is hard to do ...

Sat, 4 Jan 2003 21:04:32 -0400 (AST)
"Marc G. Fournier" <scrappy@hub.org>
Re: [GENERAL] v7.3.1 Bundled and Released ... 


--On Saturday, January 04, 2003 21:04:32 -0400 "Marc G. Fournier" 
<scrappy@hub.org> wrote:

> On Sat, 4 Jan 2003, Tom Lane wrote:
>
>> Dan Langille <dan@langille.org> writes:
>> > On Sat, 4 Jan 2003, Peter Eisentraut wrote:
>> >> There is a long tradition of systematically failing to tag releases in
>> >> this project.  Don't expect it to improve.
>>
>> > It was I who suggested that a release team would be a good idea.
>>
>> We *have* a release team.  Your problem is that Marc, who is the man who
>> would need to do this, doesn't appear to consider it an important thing
>> to do.  Try to convince him to put it on his checklist.
>
> I never considered tag'ng for minor releases as having any importance,
> since the tarball's themselves provide the 'tag' ... branches give us the
> ability to back-patch, but tag's don't provide us anything ... do they?
>
> That said, I can back-tag the whole source tree for past releases if ppl
> do think it is important, its just a matter of knowing the 'timestamp' to
> base it on, which I can do based on the dates of the tar files ...
It's useful for those using the CVS files to RECREATE a version based on 
the TAG
to checkout something (without pulling the whole tarball).

LER

>
> Its not like tag'ng is hard to do ...
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>



-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749




Sat, 04 Jan 2003 19:08:20 -0600
Larry Rosenman <ler@lerctr.org>
Re: [GENERAL] v7.3.1 Bundled and Released ... 
"Marc G. Fournier" <scrappy@hub.org> writes:
> I never considered tag'ng for minor releases as having any importance,
> since the tarball's themselves provide the 'tag' ... branches give us the
> ability to back-patch, but tag's don't provide us anything ... do they?

Well, a tag makes it feasible for someone else to recreate the tarball,
given access to the CVS server.  Dunno how important that is in the real
world --- but I have seen requests before for us to tag release points.

Any other arguments out there?

			regards, tom lane

Sat, 04 Jan 2003 20:10:19 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [GENERAL] v7.3.1 Bundled and Released ... 

Tom Lane wrote:

> Any other arguments out there?

Per-release tags make it easier to see quickly if some code has
changed in -current or not.  As the CVS tree is available via anoymous
CVS (I think?), CVSup, and via the web so there are many potential
users who are not active developers and who probably run releases
rather than -current.

Regards,

Giles



Sun, 05 Jan 2003 14:05:45 +1100
Giles Lean <giles@nemeton.com.au>
Re: [GENERAL] v7.3.1 Bundled and Released ... 
On Sat, 4 Jan 2003, Tom Lane wrote:

> "Marc G. Fournier" <scrappy@hub.org> writes:
> > I never considered tag'ng for minor releases as having any importance,
> > since the tarball's themselves provide the 'tag' ... branches give us the
> > ability to back-patch, but tag's don't provide us anything ... do they?
>
> Well, a tag makes it feasible for someone else to recreate the tarball,
> given access to the CVS server.  Dunno how important that is in the real
> world --- but I have seen requests before for us to tag release points.

FWIW, in the real world, a release doesn't happen if it's not taqged.


Sun, 5 Jan 2003 07:41:37 -0500 (EST)
Dan Langille <dan@langille.org>
Re: [GENERAL] v7.3.1 Bundled and Released ... 
On Sun, 2003-01-05 at 06:41, Dan Langille wrote:
> On Sat, 4 Jan 2003, Tom Lane wrote:
> 
> > "Marc G. Fournier" <scrappy@hub.org> writes:
> > > I never considered tag'ng for minor releases as having any importance,
> > > since the tarball's themselves provide the 'tag' ... branches give us the
> > > ability to back-patch, but tag's don't provide us anything ... do they?
> >
> > Well, a tag makes it feasible for someone else to recreate the tarball,
> > given access to the CVS server.  Dunno how important that is in the real
> > world --- but I have seen requests before for us to tag release points.
> 
> FWIW, in the real world, a release doesn't happen if it's not taqged.

Agreed!  Any tarballs, rpms, etc., should be made from the tagged
source.  Period.  If rpm's are made from a tarball that is made from
tagged source, that's fine.  Nonetheless, any official release (major or
minor) should always be made from the resulting tagged source.  This
does two things.  First, it validates that everything has been properly
tagged.  Two, it ensures that there are not any localized files or
changes which might become part of a tarball/release which are not
officially part of the repository.

I can't stress enough that a release should never happen unless source
has been tagged.  Releases should ALWAYS be made from a checkout based
on tags.


-- 
Greg Copeland <greg@copelandconsulting.net>
Copeland Computer Consulting


05 Jan 2003 08:53:25 -0600
Greg Copeland <greg@CopelandConsulting.Net>
Re: [GENERAL] v7.3.1 Bundled and Released ...

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> Well, a tag makes it feasible for someone else to recreate the tarball,
> given access to the CVS server.  Dunno how important that is in the real
> world --- but I have seen requests before for us to tag release points.
>
> Any other arguments out there?

FWIW, I use the tags often in some scripts that rely on the output 
of 'cvs status -v'. Seeing REL7_3_STABLE at the top of the 
"Existing Tags" list is a bit disconcerting when you know that 
it's not true. My scripts assume that the latest release should 
always be tagged.

Greg Sabino Mullane  greg@turnstep.com
PGP Key: 0x14964AC8 200208

-----BEGIN PGP SIGNATURE-----
Comment: http://www.turnstep.com/pgp.html

iD8DBQE+GE1NvJuQZxSWSsgRAh/1AKCPEKQeQ3OnKzbeSl5DXstnwwiFPQCfQ2mn
KplkOouzodJqZvQNN2tk8Fk=
=OaZY
-----END PGP SIGNATURE-----




Sun,  5 Jan 2003 15:10:15 -0000
greg@turnstep.com
Re: [GENERAL] v7.3.1 Bundled and Released ... 
I noticed sync() is used in PostgreSQL.

CHECKPOINT -> FlushBufferPool() -> smgrsync() -> mdsync() -> sync()

Can someone tell me why we need sync() here?
--
Tatsuo Ishii

Wed, 08 Jan 2003 15:08:49 +0900 (JST)
Tatsuo Ishii <t-ishii@sra.co.jp>
sync()
Tatsuo Ishii wrote:
> I noticed sync() is used in PostgreSQL.
> 
> CHECKPOINT -> FlushBufferPool() -> smgrsync() -> mdsync() -> sync()
> 
> Can someone tell me why we need sync() here?

As part of checkpoint, we discard some WAL files.  To do that, we must
first be sure that all the dirty buffers we have written to the kernel
are actually on the disk.  That is why the sync() is required.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 8 Jan 2003 01:18:35 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: sync()
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Tatsuo Ishii wrote:
>> Can someone tell me why we need sync() here?

> As part of checkpoint, we discard some WAL files.  To do that, we must
> first be sure that all the dirty buffers we have written to the kernel
> are actually on the disk.  That is why the sync() is required.

What we really need is something better than sync(), viz flush all dirty
buffers to disk *and* wait till they're written.  But sync() and sleep
for awhile is the closest portable approximation.

			regards, tom lane

Wed, 08 Jan 2003 01:34:49 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: sync() 
> Tatsuo Ishii wrote:
> > I noticed sync() is used in PostgreSQL.
> > 
> > CHECKPOINT -> FlushBufferPool() -> smgrsync() -> mdsync() -> sync()
> > 
> > Can someone tell me why we need sync() here?
> 
> As part of checkpoint, we discard some WAL files.  To do that, we must
> first be sure that all the dirty buffers we have written to the kernel
> are actually on the disk.  That is why the sync() is required.

?? I thought WAL files are synced by pg_fsync() (if needed).
--
Tatsuo Ishii

Wed, 08 Jan 2003 15:36:44 +0900 (JST)
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: sync()
> > As part of checkpoint, we discard some WAL files.  To do that, we must
> > first be sure that all the dirty buffers we have written to the kernel
> > are actually on the disk.  That is why the sync() is required.
> 
> What we really need is something better than sync(), viz flush all dirty
> buffers to disk *and* wait till they're written.  But sync() and sleep
> for awhile is the closest portable approximation.

Are you saying that fsync() might not wait untill the IO completes?
--
Tatsuo Ishii

Wed, 08 Jan 2003 15:39:17 +0900 (JST)
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: sync() 
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> Can someone tell me why we need sync() here?

> ?? I thought WAL files are synced by pg_fsync() (if needed).

They are.  But to write a checkpoint record --- which implies that the
WAL records before it need no longer be replayed --- we have to ensure
that all the changes-so-far in the regular database files are written
down to disk.  That is what we need sync() for.

			regards, tom lane

Wed, 08 Jan 2003 01:42:42 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: sync() 
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
>> What we really need is something better than sync(), viz flush all dirty
>> buffers to disk *and* wait till they're written.  But sync() and sleep
>> for awhile is the closest portable approximation.

> Are you saying that fsync() might not wait untill the IO completes?

No, I said that sync() might not.  Read the man pages.  HPUX's man
page for sync(2) says

     sync() causes all information in memory that should be on disk to be
     written out.
     ...
     The writing, although scheduled, is not necessarily complete upon
     return from sync.

			regards, tom lane

Wed, 08 Jan 2003 01:46:57 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: sync() 
> > Are you saying that fsync() might not wait untill the IO completes?
> 
> No, I said that sync() might not.  Read the man pages.  HPUX's man
> page for sync(2) says
> 
>      sync() causes all information in memory that should be on disk to be
>      written out.
>      ...
>      The writing, although scheduled, is not necessarily complete upon
>      return from sync.

I'm just wondering why we do not use fsync() to flush data/index
pages.
--
Tatsuo Ishii

Wed, 08 Jan 2003 15:51:52 +0900 (JST)
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: sync() 
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> I'm just wondering why we do not use fsync() to flush data/index
> pages.

There isn't any efficient way to do that AFAICS.  The process that wants
to do the checkpoint hasn't got any way to know just which files need to
be sync'd.  Even if it did know, it's not clear to me that we can
portably assume that process A issuing an fsync on a file descriptor F
it's opened for file X will force to disk previous writes issued against
the same physical file X by a different process B using a different file
descriptor G.

sync() is surely overkill, in that it writes out dirty kernel buffers
that might have nothing at all to do with Postgres.  But I don't see how
to do better.

			regards, tom lane

Wed, 08 Jan 2003 02:02:46 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: sync() 
> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> > I'm just wondering why we do not use fsync() to flush data/index
> > pages.
> 
> There isn't any efficient way to do that AFAICS.  The process that wants
> to do the checkpoint hasn't got any way to know just which files need to
> be sync'd.  Even if it did know, it's not clear to me that we can
> portably assume that process A issuing an fsync on a file descriptor F
> it's opened for file X will force to disk previous writes issued against
> the same physical file X by a different process B using a different file
> descriptor G.
> 
> sync() is surely overkill, in that it writes out dirty kernel buffers
> that might have nothing at all to do with Postgres.  But I don't see how
> to do better.

Thanks for a good summary. Maybe this is yet another reason to have
a separate IO process like Oracle...
--
Tatsuo Ishii

Wed, 08 Jan 2003 16:17:09 +0900 (JST)
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: sync() 
Tom Lane wrote:
> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> >> What we really need is something better than sync(), viz flush all dirty
> >> buffers to disk *and* wait till they're written.  But sync() and sleep
> >> for awhile is the closest portable approximation.
> 
> > Are you saying that fsync() might not wait untill the IO completes?
> 
> No, I said that sync() might not.  Read the man pages.  HPUX's man
> page for sync(2) says
> 
>      sync() causes all information in memory that should be on disk to be
>      written out.
>      ...
>      The writing, although scheduled, is not necessarily complete upon
>      return from sync.

Yep, BSD/OS says:

	BUGS
	     Sync() may return before the buffers are completely flushed.

At least they classify it as a bug.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 8 Jan 2003 09:51:01 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: sync()
Tom Lane wrote:
> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> > I'm just wondering why we do not use fsync() to flush data/index
> > pages.
> 
> There isn't any efficient way to do that AFAICS.  The process that wants
> to do the checkpoint hasn't got any way to know just which files need to
> be sync'd.

So the backends have to keep a common list of all the files they
touch.  Admittedly, that could be a problem if it means using a bunch
of shared memory, and it may have additional performance implications
depending on the implementation ...

>  Even if it did know, it's not clear to me that we can
> portably assume that process A issuing an fsync on a file descriptor F
> it's opened for file X will force to disk previous writes issued against
> the same physical file X by a different process B using a different file
> descriptor G.

If the manpages are to be believed, then under FreeBSD, Linux, and
HP-UX, calling fsync() will force to disk *all* unwritten buffers
associated with the file pointed to by the filedescriptor.

Sadly, however, the Solaris and IRIX manpages suggest that only
buffers associated with the specific file descriptor itself are
written, not necessarily all buffers associated with the file pointed
at by the file descriptor (and interestingly, the Solaris version
appears to be implemented as a library function and not a system call,
if the manpage's section is any indication).

> sync() is surely overkill, in that it writes out dirty kernel buffers
> that might have nothing at all to do with Postgres.  But I don't see how
> to do better.

It's obvious to me that sync() can have some very significant
performance implications on a system that is acting as more than just
a database server.  So it should probably be used only when there's no
good alternative.

So: this is probably one of those cases where it's important to
distinguish between operating systems and use the sync() approach only
when it's uncertain that fsync() will do the job.  So FreeBSD (and
probably all the other BSD derivatives) definitely should use fsync()
since they have known-good implementations.  Linux and HP-UX 11 (if
the manpage's wording can be trusted.  Not sure about earlier
versions) should use fsync() as well.  Solaris and IRIX should use
sync() since their manpages indicate that only data associated with
the filedescriptor will be written to disk.

Under Linux (and perhaps HP-UX), it may be necessary to fsync() the
directories leading to the file as well, so that the state of the
filesystem on disk is consistent and safe in the event that the files
in question are newly-created.  Whether that's truly necessary or not
appears to be filesystem-dependent.  A quick perusal of the Linux
source shows that ext2 appears to only sync the data and metadata
associated with the inode of the specific file and not any parent
directories, so it's probably a safe bet to fsync() any ancestor
directories that matter as well as the file even if the system is
running on top of a journalled filesystem.  Since all the files in
question probably reside in the same set of directories, the directory
fsync()s can be deferred until the very end.


-- 
Kevin Brown					      kevin@sysexperts.com

Sun, 12 Jan 2003 21:31:02 -0800
Kevin Brown <kevin@sysexperts.com>
Re: sync()
Kevin Brown <kevin@sysexperts.com> writes:
> So the backends have to keep a common list of all the files they
> touch.  Admittedly, that could be a problem if it means using a bunch
> of shared memory, and it may have additional performance implications
> depending on the implementation ...

It would have to be a list of all files that have been touched since the
last checkpoint.  That's a serious problem for storage in shared memory,
which is by definition fixed-size.

>> Even if it did know, it's not clear to me that we can
>> portably assume that process A issuing an fsync on a file descriptor F
>> it's opened for file X will force to disk previous writes issued against
>> the same physical file X by a different process B using a different file
>> descriptor G.

> If the manpages are to be believed, then under FreeBSD, Linux, and
> HP-UX, calling fsync() will force to disk *all* unwritten buffers
> associated with the file pointed to by the filedescriptor.

> Sadly, however, the Solaris and IRIX manpages suggest that only
> buffers associated with the specific file descriptor itself are
> written, not necessarily all buffers associated with the file pointed
> at by the file descriptor (and interestingly, the Solaris version
> appears to be implemented as a library function and not a system call,
> if the manpage's section is any indication).

Right.  "Portably" was the key word in my comment (sorry for not
emphasizing this more clearly).  The real problem here is how to know
what is the actual behavior of each platform?  I'm certainly not
prepared to trust reading-between-the-lines-of-some-man-pages.  And I
can't think of a simple yet reliable direct test.  You'd really have to
invest detailed study of the kernel source code to know for sure ...
and many of our platforms don't have open-source kernels.

> Under Linux (and perhaps HP-UX), it may be necessary to fsync() the
> directories leading to the file as well, so that the state of the
> filesystem on disk is consistent and safe in the event that the files
> in question are newly-created.

AFAIK, all Unix implementations are paranoid about consistency of
filesystem metadata, including directory contents.  So fsync'ing
directories from a user process strikes me as a waste of time, even
assuming that it were portable, which I doubt.  What we need to worry
about is whether fsync'ing a bunch of our own data files is a practical
substitute for a global sync() call.

			regards, tom lane

Mon, 13 Jan 2003 00:52:43 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: sync() 
Tom Lane wrote:
> Kevin Brown <kevin@sysexperts.com> writes:
> > So the backends have to keep a common list of all the files they
> > touch.  Admittedly, that could be a problem if it means using a bunch
> > of shared memory, and it may have additional performance implications
> > depending on the implementation ...
> 
> It would have to be a list of all files that have been touched since the
> last checkpoint.  That's a serious problem for storage in shared memory,
> which is by definition fixed-size.

Of course, the file list needn't be stored in SysV shared memory.  It
could be stored in a file that's later read by the checkpointing
process.  The backends could serialize their writes via fcntl() or
ioctl() style locks, whichever is appropriate.  Locking might even be
avoided entirely if the individual writes are small enough.

> Right.  "Portably" was the key word in my comment (sorry for not
> emphasizing this more clearly).  The real problem here is how to know
> what is the actual behavior of each platform?  I'm certainly not
> prepared to trust reading-between-the-lines-of-some-man-pages.  

Reading between the lines isn't necessarily required, just literal
interpretation.  :-)

> And I can't think of a simple yet reliable direct test.  You'd
> really have to invest detailed study of the kernel source code to
> know for sure ...  and many of our platforms don't have open-source
> kernels.

Linux appears to do the right thing with the file data itself, even if
it doesn't handle the directory entry simultaneously.  Others claim,
in messages written to pgsql-general and elsewhere (via Google
search), that FreeBSD does the right thing for sure.

I certainly agree that non-open-source kernels are uncertain.  That's
why it wouldn't be a bad idea to control this via a GUC variable.

> > Under Linux (and perhaps HP-UX), it may be necessary to fsync() the
> > directories leading to the file as well, so that the state of the
> > filesystem on disk is consistent and safe in the event that the files
> > in question are newly-created.
> 
> AFAIK, all Unix implementations are paranoid about consistency of
> filesystem metadata, including directory contents.  

Not ext2 under Linux!  By default, it writes everything
asynchronously.  I don't know how many people use ext2 to do serious
tasks under Linux, so this may not be that much of an issue.

> So fsync'ing directories from a user process strikes me as a waste
> of time, even assuming that it were portable, which I doubt.  What
> we need to worry about is whether fsync'ing a bunch of our own data
> files is a practical substitute for a global sync() call.

I'm positive that under certain operating systems, fsyncing the data
is a better option than a global sync(), especially since sync() isn't
guaranteed to wait until the buffers are flushed.  Right now the state
of the data on disk immediately after a checkpoint is just a guess
because of that.  I don't see that using fsync() would introduce
significantly more uncertainty on systems where the manpage explicitly
says that the buffers associated with the file referenced by the file
descriptor are the ones written to disk.  For instance, the FreeBSD
manpage says:

    Fsync() causes all modified data and attributes of fd to be moved
    to a permanent storage device.  This normally results in all
    in-core modified copies of buffers for the associated file to be
    written to a disk.

    Fsync() should be used by programs that require a file to be in a
    known state, for example, in building a simple transaction
    facility.

and the Linux manpage says:

    fsync copies all in-core parts of a file to disk, and waits until
    the device reports that all parts are on stable storage.  It also
    updates metadata stat information.  It does not necessarily ensure
    that the entry in the directory containing the file has also
    reached disk.  For that an explicit fsync on the file descriptor
    of the directory is also needed.

Both are rather unambiguous, and a cursory review of the Linux source
confirms what its manpage says, at least.  The FreeBSD manpage might
be ambiguous, but the fact that they also have an fsync command line
utility essentially proves that FreeBSD's fsync() flushes all buffers
associated with the file.

Conversely, the Solaris manpage says:

    The fsync() function moves all modified data and attributes of the
    file descriptor fildes to a storage device. When fsync() returns,
    all in-memory modified copies of buffers associated with fildes
    have been written to the physical medium.

It's pretty clear from the Solaris description that its fsync()
concerns itself only with the buffers associated with a file
descriptor and not with the file itself.  The fact that it's
implemented as a library call (the manpage is in section 3 instead of
section 2) convinces me further that its fsync() implementation is as
described.


The PostgreSQL default for checkpoints should probably be sync(), but
I think fsync() should be an available option, just as it's possible
to control whether or not synchronous writes are used for the
transaction log as well as the type of synchronization mechanism used
for it.  Yes, it's another parameter for the administrator to concern
himself with, but it seems to me that a significant amount of speed
could be gained under certain (perhaps quite common) circumstances
with such a mechanism.




-- 
Kevin Brown					      kevin@sysexperts.com

Sun, 12 Jan 2003 23:32:07 -0800
Kevin Brown <kevin@sysexperts.com>
Re: sync()

Tom Lane writes:

> Right.  "Portably" was the key word in my comment (sorry for not
> emphasizing this more clearly).  The real problem here is how to know
> what is the actual behavior of each platform?  I'm certainly not
> prepared to trust reading-between-the-lines-of-some-man-pages.  And I
> can't think of a simple yet reliable direct test.

Is the "Single Unix Standard, version 2" (aka UNIX98) any better?
It says for fsync():

    "The fsync() function forces all currently queued I/O operations
    associated with the file indicated by file descriptor fildes to
    the synchronised I/O completion state. All I/O operations are
    completed as defined for synchronised I/O file integrity
    completion."

This to me clearly says that changes to the file must be written,
not just changes made via this file descriptor.

I did have to test this behaviour once (for a customer, strange
situation) but I couldn't find a portable way to do it, either.

What I did was read the appropriate disk block from the raw device to
bypass the buffer cache.  As this required low level knowledge of the
on-disk filesystem layout it was not very portable.  For anyone
interested Tom Christiansen's "icat" program can be ported to UFS
derived filesystems fairly easily:

    http://www.rosat.mpe-garching.mpg.de/mailing-lists/perl5-porters/1997-04/msg00487.html

> AFAIK, all Unix implementations are paranoid about consistency of
> filesystem metadata, including directory contents.  So fsync'ing
> directories from a user process strikes me as a waste of time, ...

There is one variant where this is not the case: Linux using ext2fs
and possibly other filesystems.

There was a flame fest of great entertainment value a few years ago
between Linus Torvalds and Dan Bernstein.  Of course, neither was able
to influence the opinion of the other to any noticible degree, but it
made fun reading. I think this might be a starting point:

    http://www.ornl.gov/cts/archives/mailing-lists/qmail/1998/05/msg00667.html

A more recent posting from Linus where he continues to recommend
fsync() is this:

    http://www.cs.helsinki.fi/linux/linux-kernel/2001-29/0659.html

I've not heard that any other Unix-like OS has abandoned the
traditional and POSIX semantic.

> assuming that it were portable, which I doubt.  What we need to worry
> about is whether fsync'ing a bunch of our own data files is a practical
> substitute for a global sync() call.

I wish that it were.  There are situations (serveral GB buffer caches,
for example) where I mistrust the current use of sync() to have all
writes completed before the sleep() returns.  My concern is
theoretical at the moment -- I never get to play with machines that
large!

Regards,

Giles



Mon, 13 Jan 2003 19:31:08 +1100
Giles Lean <giles@nemeton.com.au>
Re: sync() 
tom lane wrote:
> <flame on>
> In all honesty, I do not *want* Windows people to think that
> they're not running on the "poor stepchild" platform.

We should distinguish between "poor stepchild" from a client support
perspective and a production environment perspective.

What is the downside to supporting development of client products
better? That is what I am really suggesting.

If people are deciding what open-source database server they want to
use, Linux or FreeBSD is the obvious choice for the server OS. The kind
of people who are inclined to use PostgreSQL or MySQL will mostly NOT be
considering Windows servers.


> I have no objection to there being a Windows port that people
> can use to do SQL-client development on their laptops.  But 
> let us please not confuse this with an industrial-strength 
> solution; nor give any level of support that might lead 
> others to make such confusion.

All we can do is simply to make it clear that Windows
is not recommended for production server use and outline
all the reasons why Windows sucks for that purpose.

Beyond that, if people want to shoot themselves in the head, they will
do so and I don't see much point in trying to stop them.


> The MySQL guys made the right choice here: they don't want to
> buy into making Windows a grade-A platform, either. <flame off>

<flame retardent on>
How does providing a native Windows executable that doesn't require
Cygwin accomplish your objective. It seems to me that you are going to
have the problem if you release a native version irrespective of the
issue at hand (Visual C++ project support).

I don't see how making it easier to build adds to this problem.

I also don't see how making it harder for Windows client developer to
adopt PostgreSQL helps anyone. <flame retardent off>

I hate Microsoft and I don't like Windows, but I am forced to use it
because the software we need to run our business runs only on 
Windows. I use Unix whenever possible and whenever reliability is
required.

- Curtis

P.S. The lack of a real C++ client library that supports the most common
development environment out there is another problem that seriously
impedes Windows client developers.

I like libpqxx, Jeroen did a find job. However, one needs to 
jump through hoops to get it to run on Visual C++ 6.0 at
the moment.



Wed, 29 Jan 2003 12:54:51 -0400
"Curtis Faith" <curtis@galtcapital.com>
Re: [mail] Re: Windows Build System 
Curtis Faith wrote:
<snip>
 > If people are deciding what open-source database server they want to
> use, Linux or FreeBSD is the obvious choice for the server OS. The kind
> of people who are inclined to use PostgreSQL or MySQL will mostly NOT be
> considering Windows servers.

For another perspective, we've been getting a few requests per day 
through the PostgreSQL Advocacy and Marketing site's request form along 
the lines of:

"Is there a license fee for using PostgreSQL?  We'd like to distribute 
it with our XYZ product that needs a database."

Probably about 4 or so per day like this at present.  A lot of the 
people sending these emails appear to have windows based products that 
need a database, and have heard of PostgreSQL being a database that they 
don't need to pay license fee's for.  They've kind of missed the point 
of Open Source from the purist point of view, but it's still working for 
them.  ;-)

Regards and best wishes,

Justin Clift

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Thu, 30 Jan 2003 03:56:52 +1030
Justin Clift <justin@postgresql.org>
Re: [mail] Re: Windows Build System
*sigh*

Often there isn't a choice of OS. If I am selling to a large enterprise
whose corporate standards say they will only run Windows in their data
center, my chances of getting them to make an exception are none. But my
chances of getting them to install Pg just for my application are far
greater. Would I prefer *nix? You betcha. Would I break a deal over it? No.
Would I prefer to be able to recommend Pg over, say, Oracle, or MS-SQL?
Absolutely. I'm not alone.

I don't care how it's built. I have a lot of sympathy for the folks saying
make the build process universal, rather than having a special one for
Windows. Requiring cygwin shouldn't be a big deal. You aren't going to get a
sudden flood of *nix-ignorant windows developers rushing in, no matter what
you do.

I've been mildly surprised and disappointed by the venom I detect in this
thread. I want to be able to recommend a single Db to my customers no matter
what OS they run. MySQL just doesn't do it, SAPdB is a nightmare, Pg is my
last hope other than a proprietary system. If you are an OpenSource zealot,
think of this as an opportunity to get some into places where it is often
anaethema.

cheers

andrew

----- Original Message -----
From: "Curtis Faith" <curtis@galtcapital.com>
To: <pgsql-hackers@postgresql.org>
Sent: Wednesday, January 29, 2003 11:54 AM
Subject: Re: [mail] Re: [HACKERS] Windows Build System


> tom lane wrote:
> > <flame on>
> > In all honesty, I do not *want* Windows people to think that
> > they're not running on the "poor stepchild" platform.
>
> We should distinguish between "poor stepchild" from a client support
> perspective and a production environment perspective.
>
> What is the downside to supporting development of client products
> better? That is what I am really suggesting.
>
> If people are deciding what open-source database server they want to
> use, Linux or FreeBSD is the obvious choice for the server OS. The kind
> of people who are inclined to use PostgreSQL or MySQL will mostly NOT be
> considering Windows servers.
>
>
> > I have no objection to there being a Windows port that people
> > can use to do SQL-client development on their laptops.  But
> > let us please not confuse this with an industrial-strength
> > solution; nor give any level of support that might lead
> > others to make such confusion.
>
> All we can do is simply to make it clear that Windows
> is not recommended for production server use and outline
> all the reasons why Windows sucks for that purpose.
>
> Beyond that, if people want to shoot themselves in the head, they will
> do so and I don't see much point in trying to stop them.
>
>
> > The MySQL guys made the right choice here: they don't want to
> > buy into making Windows a grade-A platform, either. <flame off>
>
> <flame retardent on>
> How does providing a native Windows executable that doesn't require
> Cygwin accomplish your objective. It seems to me that you are going to
> have the problem if you release a native version irrespective of the
> issue at hand (Visual C++ project support).
>
> I don't see how making it easier to build adds to this problem.
>
> I also don't see how making it harder for Windows client developer to
> adopt PostgreSQL helps anyone. <flame retardent off>
>
> I hate Microsoft and I don't like Windows, but I am forced to use it
> because the software we need to run our business runs only on
> Windows. I use Unix whenever possible and whenever reliability is
> required.
>
> - Curtis
>
> P.S. The lack of a real C++ client library that supports the most common
> development environment out there is another problem that seriously
> impedes Windows client developers.
>
> I like libpqxx, Jeroen did a find job. However, one needs to
> jump through hoops to get it to run on Visual C++ 6.0 at
> the moment.
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster


Wed, 29 Jan 2003 12:54:09 -0500
"Andrew Dunstan" <andrew@dunslane.net>
Re: [mail] Re: Windows Build System 
Justin Clift wrote:
> For another perspective, we've been getting a few requests per day 
> through the PostgreSQL Advocacy and Marketing site's request form along 
> the lines of:
> 
> "Is there a license fee for using PostgreSQL?  We'd like to distribute 
> it with our XYZ product that needs a database."
> 
> Probably about 4 or so per day like this at present.  A lot of the 
> people sending these emails appear to have windows based products that 
> need a database, and have heard of PostgreSQL being a database that they 
> don't need to pay license fee's for.  They've kind of missed the point 
> of Open Source from the purist point of view, but it's still working for 
> them.  ;-)

If they are:
 a) not clueful enough to actually look at the license, and
 b) looking at it from the purely selfish perspective of "not having to
    pay license fees,"
then are they /truly/ people where it is useful to put effort into being
helpful?

Furthermore, if their lawyers are incapable of reading the license and
explaining to them "You don't have to pay," I'd suggest the thought that
maybe they have bigger problems than you can possibly solve for them.

The great security quote of recent days is thus:
  "If you spend more on coffee than on IT security, then you will be
  hacked." -- Richard Clarke

The analagous thing might be:

"If you spend more on coffee than you do on getting proper legal advice
about software licenses, then it's just possible that you might do
something DOWNRIGHT STUPID and get yourself in a whole barrel of legal
hot water."

If these people are incapable of reading software licenses, and haven't
any competent legal counsel to to do it for them, you've got to wonder
if they are competent to sell licenses to their own software.  I
seriously doubt that they are.

Furthermore, I'm not at all sure that it is wise for you to even /try/
to give them any guidance in this, beyond giving them a URL to the
license, and saying "Have your lawyer read this."  If you start giving
them interpretations of the license, that smacks of "giving legal
advice," and bar associations tend to frown on that.
--
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne> rate me
http://cbbrowne.com/info/
"Interfaces keep things tidy, but don't accelerate growth: functions
do." -- Alan Perlis

Wed, 29 Jan 2003 18:15:19 -0500
cbbrowne@cbbrowne.com
Re: [mail] Re: Windows Build System 


> -----Original Message-----
> From: Vince Vielhaber [mailto:vev@michvhf.com] 
> Sent: 30 January 2003 09:17
> To: Ron Mayer
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [mail] Re: [HACKERS] Windows Build System
> 
> 
> On Wed, 29 Jan 2003, Ron Mayer wrote:
> 
> >
> > Cool irony in the automated .sig on the mailinglist software...
> >
> > On Wed, 29 Jan 2003, Vince Vielhaber wrote:
> > > ...
> > > hammering the betas is a far cry from an "industrial-strength 
> > > solution". ... TIP 4: Don't 'kill -9' the postmaster
> >
> > Sounds like you're basically saying is
> >
> >    _do_ 'kill -9' the postmaster...
> >
> > and make sure it recovers gracefully when testing for an 
> "industrial- 
> > strength solution".
> 
> Not what I said at all.

It's not far off, but it's quite amusing none the less.

What I read from your postings it that you are demanding more rigourous
testing for a new major feature *prior* to it being comitted to CVS in a
dev cycle than I think we ever gave any previous new feature even in the
beta test phase. I don't object to testing, and have been thinking about
coding something to address Tom's concerns, but let's demand heavy
testing for the right reasons, not just to try to justify not doing a
Win32 port.

I would also point out that we already list the Cygwin port of
PostgreSQL as supported. Who ever gave that the kind of testing people
are demanding now? I think the worst case scenario will be that our
Win32 port is far better than the existing 'supported' solution.

Regards, Dave.

Thu, 30 Jan 2003 13:08:58 -0000
"Dave Page" <dpage@vale-housing.co.uk>
Re: [mail] Re: Windows Build System
On Thu, 30 Jan 2003, Dave Page wrote:

> > On Wed, 29 Jan 2003, Ron Mayer wrote:
> >
> > >
> > > Cool irony in the automated .sig on the mailinglist software...
> > >
> > > On Wed, 29 Jan 2003, Vince Vielhaber wrote:
> > > > ...
> > > > hammering the betas is a far cry from an "industrial-strength
> > > > solution". ... TIP 4: Don't 'kill -9' the postmaster
> > >
> > > Sounds like you're basically saying is
> > >
> > >    _do_ 'kill -9' the postmaster...
> > >
> > > and make sure it recovers gracefully when testing for an
> > "industrial-
> > > strength solution".
> >
> > Not what I said at all.
>
> It's not far off, but it's quite amusing none the less.

I agree with Tom on yanking the plug while it's operating.  Do you
know the difference between kill -9 and yanking the plug?

> What I read from your postings it that you are demanding more rigourous
> testing for a new major feature *prior* to it being comitted to CVS in a
> dev cycle than I think we ever gave any previous new feature even in the
> beta test phase. I don't object to testing, and have been thinking about
> coding something to address Tom's concerns, but let's demand heavy
> testing for the right reasons, not just to try to justify not doing a
> Win32 port.

Nice try.  I've demanded nothing, quit twisting my words to fit your
argument.  If you're going to test and call it conclusive, do some
conclusive testing or call it something else.  But I suspect that since
you don't know the difference between yanking the plug and kill -9 this
conversation is a waste of time.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Thu, 30 Jan 2003 08:24:50 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [mail] Re: Windows Build System
On Thu, 2003-01-30 at 13:24, Vince Vielhaber wrote:
> On Thu, 30 Jan 2003, Dave Page wrote:
> 
> > > On Wed, 29 Jan 2003, Ron Mayer wrote:
> > >
> > > >
> > > > Cool irony in the automated .sig on the mailinglist software...
> > > >
> > > > On Wed, 29 Jan 2003, Vince Vielhaber wrote:
> > > > > ...
> > > > > hammering the betas is a far cry from an "industrial-strength
> > > > > solution". ... TIP 4: Don't 'kill -9' the postmaster
> > > >
> > > > Sounds like you're basically saying is
> > > >
> > > >    _do_ 'kill -9' the postmaster...
> > > >
> > > > and make sure it recovers gracefully when testing for an
> > > "industrial-
> > > > strength solution".
> > >
> > > Not what I said at all.
> >
> > It's not far off, but it's quite amusing none the less.
> 
> I agree with Tom on yanking the plug while it's operating.  Do you
> know the difference between kill -9 and yanking the plug?

Kill -9 seems to me _less_ severe than yanking the plug but much easier
to automate, so that could be the first thing to test. You have no hope
of passing the pull-the-plug test if you can't survive even kill -9.

Perhaps we could have a special "reliability-regression" test that does
"kill -9 postmaster", repeatedly, at random intervals, and checks for
consistency ?

Maybe we will find even some options for some OS'es to "force-unmount"
disks. I guess that setting IDE disk's to read-only with hdparm could
possibly achieve something like that on Linux. 

> > What I read from your postings it that you are demanding more rigourous
> > testing for a new major feature *prior* to it being comitted to CVS in a
> > dev cycle than I think we ever gave any previous new feature even in the
> > beta test phase. I don't object to testing, and have been thinking about
> > coding something to address Tom's concerns, but let's demand heavy
> > testing for the right reasons, not just to try to justify not doing a
> > Win32 port.
> 
> Nice try.  I've demanded nothing, quit twisting my words to fit your
> argument.  If you're going to test and call it conclusive, do some
> conclusive testing or call it something else. 

So we have no conclusive testing done that /proves/ postgres to be
reliable ? I guess that such thing (positive conclusive reliability
test) is impossible even in theory. 

But Dave has done some testing that could not prove the opposite and
concluded that it is good enough for him. So I guess that his test were
if fact "conclusive", if only just for him ;)

Sometimes it is very hard to do the pull-the-plug test - I've seen
people pondering over a HP server they could not switch off after
accidentally powering it up. Pulling the plug just made it beep, but did
not switch it off ;)

> But I suspect that since
> you don't know the difference between yanking the plug and kill -9 this
> conversation is a waste of time.

I assume you realize that U can't "kill -9" the plug ;)

-- 
Hannu Krosing <hannu@tm.ee>

30 Jan 2003 15:44:19 +0000
Hannu Krosing <hannu@tm.ee>
Re: [mail] Re: Windows Build System
Hannu Krosing <hannu@tm.ee> writes:
> Kill -9 seems to me _less_ severe than yanking the plug but much easier
> to automate, so that could be the first thing to test. You have no hope
> of passing the pull-the-plug test if you can't survive even kill -9.

Actually, they're two orthogonal issues.

In the pull-the-plug case you have to worry about what is on disk at any
given instant and whether you can make all the bits on disk consistent
again.  (And also about whether your filesystem can perform the
equivalent exercise for its own metadata; which is why we are
questioning Windows here.  Oracle's Windows port may have an advantage,
if they bypass the OS to do raw disk I/O as they do on other platforms.)

In the kill -9 case there is no risk of losing data consistency on disk,
because the OS isn't crashing; whatever we last wrote we can expect to
read.  The issue for kill -9 is whether we can deal with leftover
dynamic state, like pre-existing shared memory segments, pre-existing
SysV semaphores, TCP port numbers that the kernel won't reassign until
some timeout expires, that kind of fun stuff.  The reason the TIP is
still there is that there are platforms on which that stuff doesn't work
very nicely.  It's better to let the postmaster exit cleanly so that
that state gets cleaned up.  I have no idea what the comparable issues
are for a native Windows port, but I bet there are some...

			regards, tom lane

Thu, 30 Jan 2003 10:56:24 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [mail] Re: Windows Build System 
"Dave Page" <dpage@vale-housing.co.uk> writes:
> I would also point out that we already list the Cygwin port of
> PostgreSQL as supported. Who ever gave that the kind of testing people
> are demanding now? I think the worst case scenario will be that our
> Win32 port is far better than the existing 'supported' solution.

A good point --- but what this is really about is expectations.  If we
support a native Windows port then people will probably think that it's
okay to run production databases on that setup; whereas I doubt many
people would think that about the Cygwin-based port.  So what we need to
know is whether the platform is actually stable enough that that's a
reasonable thing to do; so that we can plaster the docs with appropriate
disclaimers if necessary.  Windows, unlike the other OSes mentioned in
this thread, has a long enough and sorry enough track record that it
seems appropriate to run such tests ...

			regards, tom lane

Thu, 30 Jan 2003 11:12:10 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [mail] Re: Windows Build System 
On Thursday 30 January 2003 11:12, Tom Lane wrote:
> A good point --- but what this is really about is expectations.  If we
> support a native Windows port then people will probably think that it's
> okay to run production databases on that setup; whereas I doubt many
> people would think that about the Cygwin-based port.  So what we need to
> know is whether the platform is actually stable enough that that's a
> reasonable thing to do; so that we can plaster the docs with appropriate
> disclaimers if necessary.  Windows, unlike the other OSes mentioned in
> this thread, has a long enough and sorry enough track record that it
> seems appropriate to run such tests ...

I think it's just developer backlash to Win32.  I am on record (see the 
archives) as not wanting the Win32 port -- but the vitriol I've seen in this 
thread from several people is entirely uncalled for and is sickening.

Dave appears to have tested this Win32 beta at least as much as a regular 
PostgreSQL release would be tested.  These tests are being held to 
artificially high standards, simply because it's native Win32.  That is 
disgusting.  And poor Katie just got _slammed_ -- and she's the lead 
developer.

Vince, I would say that we, the developers of PostgreSQL, are then not 
qualified to test our own releases for the reasons you mentioned that Katie 
should not test her own releases.   Of course that's ridiculous -- often the 
developers can do a better job of testing because they know better than the 
regular user would about what conditions can cause crashes.

I don't like the thoughts of native Win32 either.  I think Win32 should die a 
long horrible death.  But that doesn't give me the right to publicly ridicule 
the folks that want to use PostgreSQL, even if it's in an 'industrial 
strength setting,' on Win32.  The BSD license indemnifies us anyway.  So 
what's the problem.

The developers don't like Win32.  That's the problem.

But as to 'industrial strength testing' -- do ANY of our releases get this 
sort of testing on ANY platform? No, typically it's 'regression passed'  'Ok, 
it's supported on that platform.'
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 30 Jan 2003 13:03:00 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [mail] Re: Windows Build System
Hi,

On Thursday 30 January 2003 17:12, you wrote:
> "Dave Page" <dpage@vale-housing.co.uk> writes:
> > I would also point out that we already list the Cygwin port of
> > PostgreSQL as supported. Who ever gave that the kind of testing people
> > are demanding now? I think the worst case scenario will be that our
> > Win32 port is far better than the existing 'supported' solution.
>
> A good point --- but what this is really about is expectations.  If we
> support a native Windows port then people will probably think that it's
> okay to run production databases on that setup; whereas I doubt many
> people would think that about the Cygwin-based port.  So what we need to
> know is whether the platform is actually stable enough that that's a
> reasonable thing to do; so that we can plaster the docs with appropriate
> disclaimers if necessary.  Windows, unlike the other OSes mentioned in
> this thread, has a long enough and sorry enough track record that it
> seems appropriate to run such tests ...
>
> 			regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster

Ah, well - I wanted to hold off but could not.

First, a disclaimer: I don't like Windows at all. There, you got it.

But: it's actually quite stable if you configure it well, and don't run the 3 
million available 'dang, this looks nice' tools on it. Place it in the 
corner, let it run only server apps, and it serves well and stable. In my 
experience (and I have quite some experience in letting Win machines run in 
heavy-duty 24/7 production floors) they will happily run and not eat data 
until the some hardware breaks or disks overflow, just like any OS.

So, please, don't let a 'I don't like it' kind of flamewar hinder a native 
port. And please no more 'not for production use' warnings - see above.
Make this 'not for production use on workstations'.

Greetings,
	Joerg
-- 
Leading SW developer  - S.E.A GmbH
Mail: joerg.hessdoerfer@sea-gmbh.com
WWW:  http://www.sea-gmbh.com

Thu, 30 Jan 2003 19:11:04 +0100
Joerg Hessdoerfer <Joerg.Hessdoerfer@sea-gmbh.com>
Re: [mail] Re: Windows Build System
On Thu, 30 Jan 2003, Lamar Owen wrote:

> Vince, I would say that we, the developers of PostgreSQL, are then not
> qualified to test our own releases for the reasons you mentioned that Katie
> should not test her own releases.   Of course that's ridiculous -- often the
> developers can do a better job of testing because they know better than the
> regular user would about what conditions can cause crashes.

Don't twist what I said.  My statement about Katie was that she has a
knowledge of the port and the OS to the point where there are things
that she knows are wrong to do and would avoid doing it.  In the case
of this port the idea is to make sure that those things that may cause
the backend to close are something that SHOULD be tested.  By their own
admission they haven't been doing that.  All they've done is loaded it
down and made sure it continued to work.  The other ports have a long
history, the windows port has ZERO history.  If you're being sickened
now, how sick would you be if something went wrong and you started seeing
things all over /. and other sites going on about how PG crashed and
blew away some corporation's data and half the OS away on something
that at worse should have only caused the backend to close?  It won't
matter that it was running on windows, it would have been a native
port that was blessed by the PGDG.

If anything, the resistance to this testing should sicken you.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Thu, 30 Jan 2003 13:17:34 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [mail] Re: Windows Build System
Lamar Owen <lamar.owen@wgcr.org> writes:
> And poor Katie just got _slammed_ -- and she's the lead developer.

We could definitely do without the vitriol.  I'd like to apologize if
anyone took anything I said as a personal attack.  It wasn't meant that
way.

> The developers don't like Win32.  That's the problem.

Sure, we're on record as not liking Windows.  But:

> But as to 'industrial strength testing' -- do ANY of our releases get this 
> sort of testing on ANY platform? No, typically it's 'regression passed'  'Ok, 
> it's supported on that platform.'

Most variants of Unix are known to be pretty stable.  Most variants of
Unix are known to follow the Unix standard semantics for sync() and
fsync().  I think we are entirely justified in doubting whether Windows
is a suitable platform for PG, and in wanting to run tests to find out.
Yes, we are holding Windows to a higher standard than we would for a
Unix variant.

Partly this is a matter of wanting to protect Postgres' reputation.
Just on sheer numbers, if there is a native Windows port then there are
likely to be huge numbers of people using Postgres on Windows.  If
that's not going to be a reliable combination, we need to know it and
tell them so up-front.  Otherwise, people will be blaming Postgres, not
Windows, when they lose data.  It's an entirely different situation from
whether Postgres-on-Joe-Blow's-Unix-Variant loses data, first because of
visibility, and second because of the different user base.  Am I being
paranoid to suspect that the average Postgres-on-Windows user will be
less clueful than the average Postgres-on-Unix user?  I don't think so.

Between the population factors and Windows' hard-earned reputation for
unreliability, we would be irresponsible not to be asking tough
questions here.  If the Windows partisans don't think Windows should be
held to a higher standard than the platforms we already deal with,
why not?  Are they afraid that their platform won't pass the scrutiny?

			regards, tom lane

Thu, 30 Jan 2003 13:34:32 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [mail] Re: Windows Build System 
On Thursday 30 January 2003 13:17, Vince Vielhaber wrote:
> On Thu, 30 Jan 2003, Lamar Owen wrote:
> > Vince, I would say that we, the developers of PostgreSQL, are then not
> > qualified to test our own releases for the reasons you mentioned that
> > Katie should not test her own releases.

> Don't twist what I said.  My statement about Katie was that she has a
> knowledge of the port and the OS to the point where there are things
> that she knows are wrong to do and would avoid doing it.

Then she would not be honestly testing, would she?

> admission they haven't been doing that.  All they've done is loaded it
> down and made sure it continued to work.  The other ports have a long
> history, the windows port has ZERO history.

Do we do powerfail testing on a unix-type port now?  That's not testing the 
port, incidentally, it's testing the OS, sync semantics aside.  Do we hold 
the other ports to the same standards?  Yes, the Win32 port is a substantial 
change from the Unix ports.  Yes, it needs robust testing.  But all the ports 
need that same grade of testing, not just Win32.  And that type of testing is 
not being rigorously done on any port now, unless it is being done by a few 
that aren't announcing that they are doing it.

And thanks to hardware write-back caching on many hard drives, powerfail 
testing may be moot regardless of OS or filesystem type.

>  If you're being sickened
> now, how sick would you be if something went wrong and you started seeing
> things all over /. and other sites going on about how PG crashed and
> blew away some corporation's data and half the OS away on something
> that at worse should have only caused the backend to close?

Sick enough.  But that applies to all our supported platforms, not just Win32.  
>From what I've seen and heard the 'supported' Cygwin port will barf all over 
itself under high load.  So, the first thing I personally would test for a 
Win32 native port is 'how well is it performing under load?' -- after it 
passes that I would then throw the more pathological cases at it.

>  It won't
> matter that it was running on windows, it would have been a native
> port that was blessed by the PGDG.

So?  How many users out there actually know about the PGDG?  How many users 
have gotten PostgreSQL from their distributor of choice (whether a Linux 
distribution, the Cygwin distribution, FreeBSD ports, or wherever) and know 
nothing of PGDG or even postgresql.org?  We make ourselves too important.

I know enough to take all those sites with a shakerful of salt.  But then 
again I know enough to know that the batboy didn't help Clinton or Bush do 
anything, 'Weekly World News' aside.  We can't prevent the tabloid mentality 
regardless of what we do.  Or don't do.  

The point being that if any release of anything labeled 'PostgreSQL', 
regardless of its status as blessed or not blessed (or even cursed) by the 
PGDG, does what you've said, PostgreSQL as a whole will suffer.  Our blessing 
or cursing is meaningless to most users.  Or, in slightly different words, if 
they can't be bothered to care that it's on Windows then they aren't going to 
care whether we gave it the Royal Seal of PGDG either.

However, I'm sure the folks that are wanting to sell this Win32 native port 
care a whole lot about how much return business they get -- so I'm sure they 
care more about whether it is robustly tested than you give them credit.

> If anything, the resistance to this testing should sicken you.

There isn't any resistance to this testing that I've seen.  ISTM that the 
resistance is to the idea of a 'supported' WIn32 native port.  So, let's test 
the Win32 native beta using your scheme, and see what falls down.  And let's 
test Linux, *BSD, HP-UX, and AIX using the same scheme and see if it falls 
down.  Let's just be fair about the testing.  The Win32 stuff is being 
proclaimed as beta already -- so none are being misled into thinking it's 
production grade right now.  But it is passing those tests that hitherto have 
been thrown at it -- and it seems to be passing them well.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 30 Jan 2003 13:41:50 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [mail] Re: Windows Build System
On Thu, 30 Jan 2003, Lamar Owen wrote:

> On Thursday 30 January 2003 13:17, Vince Vielhaber wrote:
> > On Thu, 30 Jan 2003, Lamar Owen wrote:
> > > Vince, I would say that we, the developers of PostgreSQL, are then not
> > > qualified to test our own releases for the reasons you mentioned that
> > > Katie should not test her own releases.
>
> > Don't twist what I said.  My statement about Katie was that she has a
> > knowledge of the port and the OS to the point where there are things
> > that she knows are wrong to do and would avoid doing it.
>
> Then she would not be honestly testing, would she?

She consider herself testing to her own standards as a windows user/
developer.  Is that enough?  IMO, No.  I've been on both sides know
that the windows user/developer doesn't hold things to the same standards
as the unix user/developer.

> > admission they haven't been doing that.  All they've done is loaded it
> > down and made sure it continued to work.  The other ports have a long
> > history, the windows port has ZERO history.
>
> Do we do powerfail testing on a unix-type port now?  That's not testing the
> port, incidentally, it's testing the OS, sync semantics aside.  Do we hold
> the other ports to the same standards?  Yes, the Win32 port is a substantial
> change from the Unix ports.  Yes, it needs robust testing.  But all the ports
> need that same grade of testing, not just Win32.  And that type of testing is
> not being rigorously done on any port now, unless it is being done by a few
> that aren't announcing that they are doing it.

Since you're pretty much ignoring my reasoning, I'll give you the same
consideration.  The history of windows as a platform has shown itself
to be rather fragile compared to unix.

Before you respond to this, read Tom Lane's response and reply to that.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Thu, 30 Jan 2003 14:20:26 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: [mail] Re: Windows Build System
On Thursday 30 January 2003 13:34, Tom Lane wrote:
> anyone took anything I said as a personal attack.  It wasn't meant that
> way.

With a <flame on> tag?  Flames are by long tradition personal.  But I 
understand that that wasn't the intent -- the <flame on> was more of a 
<emphasis> tag.

> Sure, we're on record as not liking Windows.  But:
> > But as to 'industrial strength testing' -- do ANY of our releases get
> > this sort of testing on ANY platform? No, typically it's 'regression
> > passed'  'Ok, it's supported on that platform.'

> Most variants of Unix are known to be pretty stable.  Most variants of
> Unix are known to follow the Unix standard semantics for sync() and
> fsync().  I think we are entirely justified in doubting whether Windows
> is a suitable platform for PG, and in wanting to run tests to find out.

Testing is being done.  Those who are testing it are comfortable so far in its 
capabilities.  We will hear about it, loadly, when that changes, I'm sure.

> Yes, we are holding Windows to a higher standard than we would for a
> Unix variant.

Which is pretty ironic, given Win's reputation, right?

> Partly this is a matter of wanting to protect Postgres' reputation.

And here's where the rubber meets the road.  We, like many developers of 
software (open source and otherwise) have worked on this for so long and so 
hard that we have personified the program and it has become our child, so to 
speak.  As a father of four, I know what that can do.  We will protect our 
child at any cost, vehemently so.  I for one can recognize this, and further 
recognize that _it's_just_a_program_ (!!!!!) and not my child.  This is hard 
to do.  We're seeing our child experiment with what we consider to be bad 
company, and the defense mechanism is kicking in.

> Just on sheer numbers, if there is a native Windows port then there are
> likely to be huge numbers of people using Postgres on Windows.  If

While I understand (and agree with) your (and Vince's) reasoning on why 
Windows should be considered less reliable, neither of you have provided a 
sound technical basis for why we should not hold the other ports to the same 
standards.  I believe we should test every release as pathologically as Vince 
has stated for Win32.  The more reliable we become, the worse our test cases 
should become.  Across the board, and not just on Win32.  

Do we want to encourage Win32? (some obviously do, but I don't)  Well, telling 
people that we have tested PostgreSQL on Win32 much more thoroughly than on 
Unix is in a way telling them that we think it is _better_ than the 
time-tested Unix ports ('It passed a harder test on Win32.  Are we afraid the 
Unix ports won't pass those same tests?').  I for one don't want that to be a 
conclusion -- but the 'suits' will see it that way, rest assured.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 30 Jan 2003 14:48:50 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [mail] Re: Windows Build System

From: "Tom Lane" <tgl@sss.pgh.pa.us>
>
> Most variants of Unix are known to be pretty stable.  Most variants of
> Unix are known to follow the Unix standard semantics for sync() and
> fsync().  I think we are entirely justified in doubting whether Windows
> is a suitable platform for PG, and in wanting to run tests to find out.
> Yes, we are holding Windows to a higher standard than we would for a
> Unix variant.

The patches that were released implement fsync() by a call to _commit(),
which is what I expected to see after a brief tour of the M$ support site.
Is there any reason to think this won't have the desired effect? IANAWD, but
my reading suggests these should be pretty much equivalent.

andrew


Thu, 30 Jan 2003 14:55:02 -0500
"Andrew Dunstan" <andrew@dunslane.net>
Re: [mail] Re: Windows Build System 
Lamar Owen <lamar.owen@wgcr.org> writes:
> While I understand (and agree with) your (and Vince's) reasoning on why 
> Windows should be considered less reliable, neither of you have provided a 
> sound technical basis for why we should not hold the other ports to the same 
> standards.

The point here is that Windows is virgin territory for us.  We know
about Unix.  When we port to a new Unix variant, we are dealing with the
same system APIs, and in many cases large chunks of the same system
code, that we've dealt with before.  It's reasonable for us to have
confidence that Postgres will work the same on such a platform as it
does on other Unix variants.  And the track record of reliability that
we have built up across a bunch of Unix variants gives us
cross-pollinating confidence in all of them.

Windows shares none of that heritage.  It is the first truly new port,
onto a system without any Unix background, that we have ever done AFAIK.
Claiming that it doesn't require an increased level of testing is
somewhere between ridiculous and irresponsible.

> I believe we should test every release as pathologically as Vince 
> has stated for Win32.

Great, go to it.  That does not alter the fact that today, with our
existing port history, Windows has to be treated with extra suspicion.

I do not buy the argument you are making that we should treat all
platforms alike.  If we had a ten-year-old Windows port, we could
consider it as stable as all our other ten-year-old Unix ports.
We don't.  Given that we don't have infinite resources for testing,
it's simple rationality to put more testing emphasis on the places
that we suspect there will be problems.  And if you don't suspect
there will be problems on Windows, you are being way too naive :-(

> Do we want to encourage Win32? (some obviously do, but I don't)  Well, telling 
> people that we have tested PostgreSQL on Win32 much more thoroughly than on 
> Unix is in a way telling them that we think it is _better_ than the 
> time-tested Unix ports ('It passed a harder test on Win32.  Are we afraid the 
> Unix ports won't pass those same tests?').

If it passes the tests, good for it.  I honestly do not expect that it
will.  My take on this is that we want to be able to document the
problems in advance, rather than be blindsided.

			regards, tom lane

Thu, 30 Jan 2003 15:29:50 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [mail] Re: Windows Build System 

Dave, Lamar and Katie can cheer now 'cuze this is the last comment
I'm going to make on this.  All others will be ignored, probably.

The one thing I haven't seen from Dave, Lamar or Katie on this is
reputation.  You're all for the PostgreSQL name going on it but I
have yet to see any of you so sure of yourselves that you'd put
your own name on it.  The license allows it.  Red Hat did it.  I
see no "PageSQL" or "KatieSQL" or even an "Oh-Win SQL" being offered
up.  Yet all three of you are advocating that the PostgreSQL stamp
of approval should be immediately placed on it (ok, Lamar may not
be as in favor as the Dave and Katie).

Without documented testing and sufficient warnings until enough
history is banked, I don't think a native windows port should be
given any kind of seal of approval.  After that, what about keeping
the code current?  In a year or so will it suffer from bit-rot and
be the source of complaints?  Are there going to be security concerns
surrounding it?  Is there going to be a bunch of scrambling going on
to put out a patch when the latest active-x bug hoses the data dir?

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Thu, 30 Jan 2003 16:01:22 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: Windows Build System - My final thoughts
Vince Vielhaber <vev@michvhf.com> writes:
> Without documented testing and sufficient warnings until enough
> history is banked, I don't think a native windows port should be
> given any kind of seal of approval.

That was my last point also: we have years of track record on most of
our Unix ports, and none yet on Windows.  Even several months of
intensive testing by a small number of people will hardly level the
playing field.

> After that, what about keeping the code current?

I don't think that's an issue.  We are not blessing anything based on
7.2 ;-).  The objective is to merge the changes into CVS tip and have
a first "official" Windows port as part of the 7.4 release.  After that,
it'll stay as current as any other port that's being actively used.

			regards, tom lane

Thu, 30 Jan 2003 16:12:21 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Windows Build System - My final thoughts 
On Thursday 30 January 2003 15:29, Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > While I understand (and agree with) your (and Vince's) reasoning on why
> > Windows should be considered less reliable, neither of you have provided

> Windows shares none of that heritage.  It is the first truly new port,
> onto a system without any Unix background, that we have ever done AFAIK.
> Claiming that it doesn't require an increased level of testing is
> somewhere between ridiculous and irresponsible.

I am saying that as we mature we need increased testing across the board.  And 
it is a very low percentage of code that is tied into the OS API, right?  The 
majority of the code (the vast majority) isn't touched by it. 

> that we suspect there will be problems.  And if you don't suspect
> there will be problems on Windows, you are being way too naive :-(

Reread my statement above.  I _agree_ with the rationale -- but I fear it will 
have the opposite impact.  And I am not convinced that just because we have 
good history with the unixoid ports means that we can slack on them -- Linux, 
*BSD, etc all change.  The strftime(3) breakage with RedHat of a cycle ago 
should show us that much.

I suspect there will be problems on Win32 -- it is, after all, a new port.  
But if we're going to immediately throw pathological test cases at it that 
we're not even bothering to test against now, that immediately throws up a 
flag to me.  And TESTING IS BEING DONE on the Win32 port, nobody is yet 
trying to put the PGDG blessing on it as yet, and progress is being made by 
those who wish to see it made.  It is still being touted as beta software, 
right?  The patches from Jan are very preliminary still, correct?  Katie 
hasn't issued a press release saying that it's not beta, right?

<hyperbole>
I don't see what the uproar is about, other than 'Win32 is so unstable that it 
can't possibly work as well as you are seeing it work -- you must be doing 
something wrong.  Test it harder.  Pull the plug repeatedly!! Test it until 
it breaks!  HA! Told you it would break! (yeah, firing up the old 
oxyacetlyene torch and hitting the hard drive with a 6,000 degree flame did 
the trick -- this has got to be a bad operating system!)'
</hyperbole>

And, by the way, who in their right mind tests a database server by repeated 
yanking of the AC power?  To go to that extreme for Win32 when we caution 
against something as mundane as a kill -9 of postmaster on Unix is absurd.  
And, yes, I know the difference.  I also know that the AC power pull has 
nothing to do with PostgreSQL, but it has to do with the OS under it.  
Although a kill -9, from the point of view of the running process, is 
identical to a power failure. It simply dies (unless it becomes a zombie, in 
which case it is undead) either way.  The effects of a kill -9 shouldn't be 
as severe as a power fail, since the OS can properly flush written buffers 
even after the process writing them has died.

And I also can point the finger at some Unix swervers (spelling intentional) 
that would fail that test in a miserable way.  I can also point at a few VMS 
machines that couldn't pass that test.  I've even seen machines blow up due 
to improper power cycling.  

And I've seen Win2k machines come right up after repeated power blips (I've 
also seen them not come up).  

It really depends upon what the hard disk is doing at the instant the 
regulators drop out the 5 and 12V supplies (and which supply goes out first, 
which can depend upon the respective loads -- for modern Pentium 4 systems 
the 12V will probably go down first since it is more heavily loaded than the 
5V supply in these systems).  Under certain conditions where the 12V goes 
down before the 5V does, the head might still be writing as the servo spirals 
towards park, causing all manner of damage (maybe even to servo information, 
which normally cannot be written). So the power cycle becomes a test of 
hardware, too, played Russian Roulette-style.

Talk about an unscientific test.

A database server that needs that kind of testing is going to be hardened 
hardware on a doubly redundant UPS anyway.

But, then again I've seen a Linux server survive a power cycle with no lost 
data (ext3 filesystem -- I've seen lost data with ext2).  And I've seen the 
same server barf all over itself due to a single bit error in memory.  Blew 
out the entire root filesystem, which was journaled and residing on a RAID 1 
partition (the corruption was perfectly mirrored, by the way).  Serves me 
right for not having ECC RAM installed at the time.

> If it passes the tests, good for it.  I honestly do not expect that it
> will.  My take on this is that we want to be able to document the
> problems in advance, rather than be blindsided.

I fully expect that Katie, Jan, Dave, and all the others working on this share 
your concerns and want the Win32 port to be as solid as is possible on that 
OS.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 30 Jan 2003 16:29:15 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [mail] Re: Windows Build System
On Thursday 30 January 2003 16:01, Vince Vielhaber wrote:
> Dave, Lamar and Katie can cheer now 'cuze this is the last comment
> I'm going to make on this.  All others will be ignored, probably.

> up.  Yet all three of you are advocating that the PostgreSQL stamp
> of approval should be immediately placed on it (ok, Lamar may not
> be as in favor as the Dave and Katie).

For the record, again, I am not at all in favor of a Win32 native port.  I 
have never been in favor of a Win32 native port (see the archives -- it's in 
there).  I am in favor of fair testing for all ports, and less of an 
emotional response to the idea of a Win32 port.  It's going to happen; we 
can't stop it; we might as well see how best to handle it. 

And I am definitely not in favor of putting the Royal Seal of PGDG on the code 
that is out there now.  It _isn't_ proven. And, as Tom just said, it's 7.2, 
and we're not due to make an Officially Stamped Win32 native port until 7.4.  

But it doesn't take AC power cycling to prove it, either.  And so I objected 
to the tone and to the extremity of the proposed testing, relative to the 
testing we do now for the other ports.

But I also see the futility of withholding the Official Stamp of Approval -- 
if Win32 PostgreSQL is out there (and it will be, whether we like it or not), 
then we will get flak over it if it breaks.  Logically we should do 
everything we can to make sure the port is as stable as possible for Win32 -- 
and power cycle testing ain't the right way.  ISTM that Dave, Katie, Jan, et 
all are doing this.  They even seem to know what they are talking about, 
which is better than most Win32 partisans.  There actually _can_ be 
reasonable people who use an unreasonable OS, for whatever reasons they may 
have.

Do I like it?  No.  Can I change it? No.  Can I help test the Win32 port? Yes, 
even though I don't want to do so.  Can I be reasonable and patient with 
those who are doing the work on the Win32 port?  Yes, I can.  Do I need to 
sling the napalm because I don't like it?  Not on the mailing lists (hmm, 
need to get some naptha, some palmitic acid....might be fun to sling some 
napalm in the back yard to rid the place of weeds, and get some relaxation to 
boot....).
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 30 Jan 2003 16:42:11 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Windows Build System - My final thoughts
Lamar Owen <lamar.owen@wgcr.org> writes:
> And, by the way, who in their right mind tests a database server by repeated 
> yanking of the AC power?

Anybody who would like their data to survive a power outage.

>  To go to that extreme for Win32 when we caution 
> against something as mundane as a kill -9 of postmaster on Unix is absurd.  
> And, yes, I know the difference.  I also know that the AC power pull has 
> nothing to do with PostgreSQL, but it has to do with the OS under it.  
> Although a kill -9, from the point of view of the running process, is 
> identical to a power failure.

No, it is not.  Did you not read my comments earlier today?  The reasons
why we are concerned about kill -9 have *nothing* to do with whether the
database can survive system crashes.  Rather, the issues created by kill
-9 have to do with coping with leftover state from a previous postmaster
in the same system lifecycle.  I forgot to mention one of the biggest
headaches, which is that kill -9 the postmaster doesn't kill the child
backends.  We've got an interlock that tries to prevent starting a new
postmaster when there are still old children around, but it's one of the
things that I think is most likely to break on any new port.  (And I'm
dead certain that that code doesn't work on Windows.)  It's that sort of
thing that we have painfully worked out on Unix-based systems, and are
going to have to do over again for Windows.  In many places we are
probably not even going to realize that we have to do something over
again, until someone gets bitten.

The fact that Postgres is reliable does not come (only) from the code
being "right" in some abstract sense that will carry over to a new
platform.  A big reason it's reliable is that we have painfully learned
about Unix-ish failure modes and put in defenses against them.  Windows
is going to bring a whole new set of failure modes that we don't have
defenses for.  (Yet.)  *That* is what we need extensive testing to learn
about, and claiming that we are discriminating against Windows just
because it's Windows misses the point completely.

Or, if you prefer, we can ship Postgres 7.4 for Windows with no more
testing than we need for any of the existing, long-since-well-tested
ports.  But I'll bet a great deal that our reputation will go down the
drain (along with many people's data) if we do that.

			regards, tom lane

Thu, 30 Jan 2003 16:54:39 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [mail] Re: Windows Build System 
On Thursday 30 January 2003 16:54, Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > And, by the way, who in their right mind tests a database server by
> > repeated yanking of the AC power?

> Anybody who would like their data to survive a power outage.

I don't buy that.  That's why I have $36,000 worth of lead acid in the room 
next door, with $5,000 of inverters and chargers in the server room.  Until I 
had to upgrade RAM I had 240+ days of uptime on one box.  The longest power 
interruption was 28 hours.  The battery held the whole time.  There was never 
more than 30 days between interruptions.  The last time I had the server 
actually power down was during a maintenance run on the inverter/charge 
system, and I had to transfer power to the servers onto another branch, 
necessitating two power cycles, which were clean shutdown/reboots.  I haven't 
had an unscheduled dirty powerdown in two years.

We cannot on any system guarantee the data surviving a sudden power outage. 
Until we can be certain the write-back cache on that high performance drive 
(or NAS array using iSCSI, perhaps) flushes we cannot know the data hit the 
disks.

> >  To go to that extreme for Win32 when we caution
> > against something as mundane as a kill -9 of postmaster on Unix is
> > absurd. And, yes, I know the difference.  I also know that the AC power
> > pull has nothing to do with PostgreSQL, but it has to do with the OS
> > under it. Although a kill -9, from the point of view of the running
> > process, is identical to a power failure.

> No, it is not.  Did you not read my comments earlier today?

Of course I did -- I'm not daft.  And that's why I specified 'from the point 
of view of the running process' -- that is, the process you are SIGKILLing 
cannot itself determine the difference between the power cycle and SIGKILL.  
It just simply goes down, hard.  Of course there is:

> I forgot to mention one of the biggest
> headaches, which is that kill -9 the postmaster doesn't kill the child
> backends.

This is a real difference, and one that I forgot as well. So SIGKILL is 
different to the whole backend system, but not to the singular process that 
is being SIGKILL'd.  Suppose I issue a SIGKILL to postmaster and all forked 
backends simultaneously?  Where does SIGKILL differ from a power failure from 
the point of view of the database system in that scenario?  This is also 
assuming that you clean reboot the OS after the SIGKILL to postmaster, as 
there is that dynamic state you mentioned to worry about.  I probably should 
have mentioned that before.

> Windows
> is going to bring a whole new set of failure modes that we don't have
> defenses for.  (Yet.)  *That* is what we need extensive testing to learn
> about, and claiming that we are discriminating against Windows just
> because it's Windows misses the point completely.

And ISTM that an experienced Windows developer, such as Katie or Dave, would 
know to do this, would know how to do this, and would know the best way of 
doing this.  And I wasn't singling you out, Tom.  It was the whole thread and 
the turns it took that got me rather upset. 

> Or, if you prefer, we can ship Postgres 7.4 for Windows with no more
> testing than we need for any of the existing, long-since-well-tested
> ports.  But I'll bet a great deal that our reputation will go down the
> drain (along with many people's data) if we do that.

We don't have a standard testing methodology for any of our ports.  We need 
one for all of our ports.  I fully expect the Win32 port to need a different 
methodology than the FreeBSD port or the Linux port.  And I expect we have 
enough experienced Win32 developers (which I am not) here that can provide 
insight into how the methodologies should differ.

I prefer more extensive testing for all of our ports.  You did read that when 
I wrote it, right?  (When I wrote it multiple times....)  Just saying 'it 
passed regression' shouldn't be enough -- but we should really spend some 
cycles thinking about what the test suite really should be.  For all 
platforms.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 30 Jan 2003 17:30:54 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [mail] Re: Windows Build System
Tom Lane wrote:
> "Dave Page" <dpage@vale-housing.co.uk> writes:
> > I would also point out that we already list the Cygwin port of
> > PostgreSQL as supported. Who ever gave that the kind of testing people
> > are demanding now? I think the worst case scenario will be that our
> > Win32 port is far better than the existing 'supported' solution.
> 
> A good point --- but what this is really about is expectations.  If we
> support a native Windows port then people will probably think that it's
> okay to run production databases on that setup; 

Sure.  But it's only common sense that a piece of software is only as
reliable as the platform it's running on.

People run production databases under MS-SQL all the time.  Has MS-SQL
itself gained a reputation for being an unreliable piece of junk?
Perhaps.  But if so, that obviously hasn't stopped people from putting
their production databases on it!

Is MS-SQL's reputation for unreliability, if any, because of MS-SQL
itself or the platform it's operating on?  The way to answer that is
to ask the same question of Oracle and DB/2 under Windows.  And
therefore, the answer seems to be that the platform is a minor
determinant, if any.

> whereas I doubt many
> people would think that about the Cygwin-based port.  

Why not?  Seriously, if the people in question are the simpletons that
you appear to be expecting them to be, then wouldn't they have that
same expectation of the Cygwin based port?  Why not?

> So what we need to
> know is whether the platform is actually stable enough that that's a
> reasonable thing to do; so that we can plaster the docs with appropriate
> disclaimers if necessary.  

Well, shouldn't we do that anyway, then, until we know otherwise?
Shouldn't we do that with *any* new port?

> Windows, unlike the other OSes mentioned in
> this thread, has a long enough and sorry enough track record that it
> seems appropriate to run such tests ...

With this I agree, but before you start thinking that Windows is the
only OS that qualifies, consider this: I've run the "pull the plug"
test under early Linux 2.4 kernels running with ReiserFS.  I'd start a
make of a large project, pull the power, bring the system back up, and
restart the build.  And the end result was that some of the files
files in the build directory were corrupted, such that the build could
not continue.  I haven't tried this under current versions of the
kernel, so I don't know if things have improved or not.

Doesn't that -- shouldn't that -- give you pause about declaring
*Linux* an industrial-strength solution?


My point: if you're going to hold *one* OS to a given standard, you
should hold *all* of them to that same standard.



-- 
Kevin Brown					      kevin@sysexperts.com

Thu, 30 Jan 2003 14:39:59 -0800
Kevin Brown <kevin@sysexperts.com>
Re: [mail] Re: Windows Build System
On Thu, Jan 30, 2003 at 02:39:59PM -0800, Kevin Brown wrote:
> 
> With this I agree, but before you start thinking that Windows is the
> only OS that qualifies, consider this: I've run the "pull the plug"
> test under early Linux 2.4 kernels running with ReiserFS.  I'd start a
> make of a large project, pull the power, bring the system back up, and
> restart the build.  And the end result was that some of the files
> files in the build directory were corrupted, such that the build could
> not continue.

Afaik, ReiserFS does not guarantee data consistency, only meta
data.  As in, the file system itself will be consistent, and an
fsck shouldn't find a problem.


Kurt


Fri, 31 Jan 2003 00:09:05 +0100
Kurt Roeckx <Q@ping.be>
Re: [mail] Re: Windows Build System
Kurt Roeckx wrote:
> On Thu, Jan 30, 2003 at 02:39:59PM -0800, Kevin Brown wrote:
> > 
> > With this I agree, but before you start thinking that Windows is the
> > only OS that qualifies, consider this: I've run the "pull the plug"
> > test under early Linux 2.4 kernels running with ReiserFS.  I'd start a
> > make of a large project, pull the power, bring the system back up, and
> > restart the build.  And the end result was that some of the files
> > files in the build directory were corrupted, such that the build could
> > not continue.
> 
> Afaik, ReiserFS does not guarantee data consistency, only meta
> data.  As in, the file system itself will be consistent, and an
> fsck shouldn't find a problem.

Exactly.  Does NTFS?  Not as far as I know.  Why should we hold NTFS
to a standard that ReiserFS doesn't meet?

That said, I do agree with Tom that the Windows port is basically
virgin territory and needs to be approached with caution.  But we
shouldn't be so cautious that we hesitate to release the port to the
world (sufficient disclaimers are appropriate, as with any new
port)...



-- 
Kevin Brown					      kevin@sysexperts.com

Thu, 30 Jan 2003 15:16:36 -0800
Kevin Brown <kevin@sysexperts.com>
Re: [mail] Re: Windows Build System
Lamar Owen <lamar.owen@wgcr.org> writes:
> On Thursday 30 January 2003 16:54, Tom Lane wrote:
>> Lamar Owen <lamar.owen@wgcr.org> writes:
>>> And, by the way, who in their right mind tests a database server by
>>> repeated yanking of the AC power?

>> Anybody who would like their data to survive a power outage.

> I don't buy that.  That's why I have $36,000 worth of lead acid in the room 
> next door, with $5,000 of inverters and chargers in the server room.

Well, great; you're probably proof against misfeasance of your local
power company.  But how about someone tripping over the power cord?
Or a blowout in the server's internal power supply?  Or a kernel crash?
Pulling the power plug is just a convenient way of (approximately)
modeling a whole class of unpleasant events.  I don't think the fact
that you can afford to spend that much on batteries makes it
uninteresting to test such scenarios.

But we're pretty much talking at cross-purposes here.  The real issue
IMHO is that the Windows port needs a lot of testing because it is a
new platform (for us), and one not like the platforms we've used before.
It is faulty to equate the amount of testing required to gain confidence
in that port with the amount of testing required to gain confidence that
PG 7.4 will run reliably on, say, HPUX 10.20, when we already know that
every PG back to 6.4 has run reliably on HPUX 10.20.  You're attacking a
straw man you have set up, namely the idea that only specific testing
produces confidence in a port.  In my mind past track record has a lot
more to do with confidence than whatever testing we do for an individual
release.

			regards, tom lane

Thu, 30 Jan 2003 18:39:51 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [mail] Re: Windows Build System 
On Thursday 30 January 2003 18:39, Tom Lane wrote:
> Well, great; you're probably proof against misfeasance of your local
> power company.  But how about someone tripping over the power cord?

Twistlok.

> Or a blowout in the server's internal power supply?

Redundant supplies.

>  Or a kernel crash?

Different from pulling the plug.

> It is faulty to equate the amount of testing required to gain confidence
> in that port with the amount of testing required to gain confidence that
> PG 7.4 will run reliably on, say, HPUX 10.20, when we already know that
> every PG back to 6.4 has run reliably on HPUX 10.20.

But does the fact that PG 6.4 ran reliably on HP-UX 10 mean PG 7.4 will run as 
reliably on HP-UX 11?  Does the fact that PG 6.2.1 ran well on Linux kernel 
2.0.30 with libc 5.3.12 mean PG 7.4 will run well on Linux 2.6.x with glibc 
2.4.x?  The OS is also a moving target.  Hmph.  PG 7.3 won't even build on 
Red Hat 5.2, for instance.  So much for track record.

>  You're attacking a
> straw man you have set up, namely the idea that only specific testing
> produces confidence in a port.  In my mind past track record has a lot
> more to do with confidence than whatever testing we do for an individual
> release.

Track record means nothing if sufficient items have changed in the underlying 
OS.  I remember the Linux fiasco with PostgreSQL 6.3.1.  It was so bad that 
Red Hat was considering releasing Red Hat 5.1 with a CVS checkout of 
pre-6.3.2.  That is not Red Hat's normal policy.

Also, between major versions enough may have changed to make it necessary to 
test thoroughly -- WAL, for instance.  MVCC for another instance.  PITR is 
going to be another instance requiring a different test methodology.  One 
will indeed be required to blow down the whole system to properly test PITR, 
on all platforms.

Track record indicates that all of our x.y.1 releases are typically hosed in 
some fashion.  7.3.1 proved that wrong.  Track record only requires a single 
failure to invalidate -- and we should test for those failures across the 
board, regardless of track record.  Records are meant to be broken.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 30 Jan 2003 18:57:27 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: [mail] Re: Windows Build System
Tom Lane wrote:
> Most variants of Unix are known to be pretty stable.  Most variants of
> Unix are known to follow the Unix standard semantics for sync() and
> fsync().  I think we are entirely justified in doubting whether Windows
> is a suitable platform for PG, and in wanting to run tests to find out.
> Yes, we are holding Windows to a higher standard than we would for a
> Unix variant.
> 
> Partly this is a matter of wanting to protect Postgres' reputation.
> Just on sheer numbers, if there is a native Windows port then there are
> likely to be huge numbers of people using Postgres on Windows.  If
> that's not going to be a reliable combination, we need to know it and
> tell them so up-front.  Otherwise, people will be blaming Postgres, not
> Windows, when they lose data.  It's an entirely different situation from
> whether Postgres-on-Joe-Blow's-Unix-Variant loses data, first because of
> visibility, and second because of the different user base.  Am I being
> paranoid to suspect that the average Postgres-on-Windows user will be
> less clueful than the average Postgres-on-Unix user?  I don't think so.

Assuming all your assumptions are right, why the hell is Oracle's and MS
SQL-Server's reputation that bloody good? And what about MySQL? They all
have a native Windows (sup)port for some time ... didn't harm their
reputation. I think that we got in bed with this ugly Cybill ... er ...
Cygwin thing had cost us more reputation than the sucking performance of
pre-7 releases all together.


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #

Thu, 30 Jan 2003 22:06:36 -0500
Jan Wieck <JanWieck@Yahoo.com>
Re: [mail] Re: Windows Build System

On Thu, 30 Jan 2003, Dave Page wrote:
> > On Wed, 29 Jan 2003, Ron Mayer wrote:
> > >
> > > Cool irony in the automated .sig on the mailinglist software...
> > > [...]
> > > Sounds like you're basically saying is
> > >    _do_ 'kill -9' the postmaster...
> > > and make sure it recovers gracefully...
> > ... 
> It's not far off, but it's quite amusing none the less.

Sorry it looks like I should have added "50% :-), 50% :-|".  I was
just amused by the irony of having the admonition against the 
standard linux-low-memory condition.  [ 90% :-) ]



More constructively, I think it'd be best for everyone if

   a) postgresql does have a native windows port, and

   b) it's positioned as a "well-tested beta" rather than "production
      ready", with documentation saying what part people have
      confidence in (the engine?  operation in non-failure modes),
      and what part is still in beta (OS failure modes).

IMHO this would have the advantages of

   Even if it works better than many expect, it shows...
    
      ... that postgresql has a high standard even for beta releases,
          where this community thinks about even broader system issues.

      ... corporations using it what aspects they need to focus on in
          internal testing before using it in production environments.

      ... that we're interested in reaching a broader community.

      ... that postgresql is quite portable across platforms.

Note that I don't even care about running the windows version.  I just think 
that such a release can be positioned to _strengthen_ PostgreSQL's brand image 
rather than weaken it.


   Ron


Thu, 30 Jan 2003 19:20:15 -0800 (PST)
Ron Mayer <ron@intervideo.com>
Re: [mail] Re: Windows Build System
I've been looking at the PID file creation mechanism we currently use.
It goes through a loop in an attempt to create the PID file, and if
one is there it attempts to remove it if the PID it contains no longer
exists (there are checks for shared memory usage as well).

This could be cleaned up rather dramatically if we were to use one of
the file locking primitives supplied by the OS to grab an exclusive
lock on the file, and the upside is that, when the locking code is
used, the postmaster would *know* whether or not there's another
postmaster running, but the price for that is that we'd have to eat a
file descriptor (closing the file means losing the lock), and we'd
still have to retain the old code anyway in the event that there is no
suitable file locking mechanism to use on the platform in question.

The first question for the group is: is it worth doing that?

The second question for the group is: if we do indeed decide to do
file locking in that manner, what *other* applications of the OS-level
file locking mechanism will we have?  Some of them allow you to lock
sections of a file, for instance, while others apply a lock on the
entire file.  It's not clear to me that the former will be available
on all the platforms we're interested in, so locking the entire file
is probably the only thing we can really count on (and keep in mind
that even if an API to lock sections of a file is available, it may
well be that it's implemented by locking the entire file anyway).

What I had in mind was implementation of a file locking function that
would take a file descriptor and a file range.  If the underlying OS
mechanism supported it, it would lock that range.  The interesting
case is when the underlying OS mechanism did *not* support it.  Would
it be more useful in that case to return an error indication?  Would
it be more useful to simply lock the entire file?  If no underlying
file locking mechanism is available, it seems obvious to me that the
function would have to always return an error.


Thoughts?



-- 
Kevin Brown					      kevin@sysexperts.com

Thu, 30 Jan 2003 19:23:54 -0800
Kevin Brown <kevin@sysexperts.com>
On file locking
Mmy problem is freebsd getting totally loaded at which point it sends kills
to various processes.  This sometime seems to end up with several actual
postmasters running, and none of them working.

Better existing process detection would help that greatly I'm sure.

Chris

> -----Original Message-----
> From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Kevin Brown
> Sent: Friday, 31 January 2003 11:24 AM
> To: PostgreSQL Development
> Subject: [HACKERS] On file locking
>
>
> I've been looking at the PID file creation mechanism we currently use.
> It goes through a loop in an attempt to create the PID file, and if
> one is there it attempts to remove it if the PID it contains no longer
> exists (there are checks for shared memory usage as well).
>
> This could be cleaned up rather dramatically if we were to use one of
> the file locking primitives supplied by the OS to grab an exclusive
> lock on the file, and the upside is that, when the locking code is
> used, the postmaster would *know* whether or not there's another
> postmaster running, but the price for that is that we'd have to eat a
> file descriptor (closing the file means losing the lock), and we'd
> still have to retain the old code anyway in the event that there is no
> suitable file locking mechanism to use on the platform in question.
>
> The first question for the group is: is it worth doing that?
>
> The second question for the group is: if we do indeed decide to do
> file locking in that manner, what *other* applications of the OS-level
> file locking mechanism will we have?  Some of them allow you to lock
> sections of a file, for instance, while others apply a lock on the
> entire file.  It's not clear to me that the former will be available
> on all the platforms we're interested in, so locking the entire file
> is probably the only thing we can really count on (and keep in mind
> that even if an API to lock sections of a file is available, it may
> well be that it's implemented by locking the entire file anyway).
>
> What I had in mind was implementation of a file locking function that
> would take a file descriptor and a file range.  If the underlying OS
> mechanism supported it, it would lock that range.  The interesting
> case is when the underlying OS mechanism did *not* support it.  Would
> it be more useful in that case to return an error indication?  Would
> it be more useful to simply lock the entire file?  If no underlying
> file locking mechanism is available, it seems obvious to me that the
> function would have to always return an error.
>
>
> Thoughts?
>
>
>
> --
> Kevin Brown					      kevin@sysexperts.com
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>


Fri, 31 Jan 2003 11:54:57 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: On file locking
Jan Wieck <JanWieck@Yahoo.com> writes:
> Assuming all your assumptions are right, why the hell is Oracle's and MS
> SQL-Server's reputation that bloody good?

They have marketing departments.

> And what about MySQL?

What about it?  Someone claimed in this thread that MySQL's Windows port
requires Cygwin.  Is that true or not?

			regards, tom lane

Thu, 30 Jan 2003 23:08:41 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [mail] Re: Windows Build System 
Kevin Brown <kevin@sysexperts.com> writes:
> This could be cleaned up rather dramatically if we were to use one of
> the file locking primitives supplied by the OS to grab an exclusive
> lock on the file, and the upside is that, when the locking code is
> used, the postmaster would *know* whether or not there's another
> postmaster running, but the price for that is that we'd have to eat a
> file descriptor (closing the file means losing the lock),

Yeah, I was just thinking about that this morning.  Eating one file
descriptor in the postmaster is absolutely no problem --- the postmaster
doesn't have all that many files open anyhow.  What I was wondering was
whether it was worth eating an FD for every backend process, by holding
open the file inherited from the postmaster.  If we did that, we would
have a reliable way of detecting that the old postmaster died but left
surviving child backends.  (As I mentioned in a nearby flamefest, the
existing interlock for this situation strikes me as mighty fragile.)

But this only wins if a child process inheriting an open file also
inherits copies of any locks held by the parent.  If not, then the
issue is moot.  Anybody have any idea if file locks work that way?
Is it portable??

> The second question for the group is: if we do indeed decide to do
> file locking in that manner, what *other* applications of the OS-level
> file locking mechanism will we have?

I can't see any use in partial-file locks for us, and would not want
to design an internal API that expects them to work.

			regards, tom lane

Thu, 30 Jan 2003 23:26:12 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: On file locking 
Tom Lane wrote:
> 
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > And, by the way, who in their right mind tests a database server by repeated
> > yanking of the AC power?
> 
> Anybody who would like their data to survive a power outage.

... has UPS, ECC Ram on quality boards and storage subsystems that
guarantee the data to hit "some" surface after it passed the interface
... what's your point? Are you telling me that the reliability of an
EMC2 system depends on which OS it is receiving the bits from? Is SuSE
as reliable as TurboLinux? Or do I have to buy AIX to get the best
result? 


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #

Thu, 30 Jan 2003 23:37:17 -0500
Jan Wieck <JanWieck@Yahoo.com>
Re: [mail] Re: Windows Build System
> file descriptor (closing the file means losing the lock), and we'd
> still have to retain the old code anyway in the event that there is no
> suitable file locking mechanism to use on the platform in question.

What is the gain given the above statement?  If what we currently do can
cause issues (fail), then beefing it up where available may be useful --
but otherwise it's just additional code.
-- 
Rod Taylor <rbt@rbt.ca>

PGP Key: http://www.rbt.ca/rbtpub.asc
30 Jan 2003 23:42:03 -0500
Rod Taylor <rbt@rbt.ca>
Re: On file locking
Hannu Krosing wrote:
> > I agree with Tom on yanking the plug while it's operating.  Do you
> > know the difference between kill -9 and yanking the plug?
> 
> Kill -9 seems to me _less_ severe than yanking the plug but much easier
> to automate, so that could be the first thing to test. You have no hope
> of passing the pull-the-plug test if you can't survive even kill -9.
> 
> Perhaps we could have a special "reliability-regression" test that does
> "kill -9 postmaster", repeatedly, at random intervals, and checks for
> consistency ?
> 
> Maybe we will find even some options for some OS'es to "force-unmount"
> disks. I guess that setting IDE disk's to read-only with hdparm could
> possibly achieve something like that on Linux.

Get VMWare for Linux, run whatever OS you like in it and "kill -9" the
virtual machine. That's as close as you can get to "yanking" without
wearing out your power plugs.


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #

Thu, 30 Jan 2003 23:54:17 -0500
Jan Wieck <JanWieck@Yahoo.com>
Re: [mail] Re: Windows Build System

> This could be cleaned up rather dramatically if we were to use one of
> the file locking primitives supplied by the OS to grab an exclusive
> lock on the file, ...
> ...
> The first question for the group is: is it worth doing that?

In the past it has been proposed and declined -- there is some stuff
in the archives.  While it would be beneficial to installations using
local data it would introduce new failure modes for installations
using NFS.

Regards,

Giles


Fri, 31 Jan 2003 16:03:22 +1100
Giles Lean <giles@nemeton.com.au>
Re: On file locking 
> What about it?  Someone claimed in this thread that MySQL's Windows port
> requires Cygwin.  Is that true or not?

It's been a while, but I know I've installed MySQL on windows without any 
separate step of installing Cygwin (I can't say 100% for sure that it didn't 
install some part of Cygwin transparently to me).

Regards,
	Jeff Davis

Thu, 30 Jan 2003 22:27:15 -0800
Jeff Davis <jdavis-pgsql@empires.org>
Re: [mail] Re: Windows Build System
On Friday 31 Jan 2003 9:56 am, you wrote:
> Kevin Brown <kevin@sysexperts.com> writes:
> But this only wins if a child process inheriting an open file also
> inherits copies of any locks held by the parent.  If not, then the
> issue is moot.  Anybody have any idea if file locks work that way?
> Is it portable??

In my experience of HP-UX and linux, they do differ. How much, I don't 
remember.

I have a stupid proposal. Keep file lock aside. I think shared memory can be 
kept alive even after process dies. Why not write a shared memory segment id 
to a file and let postmaster check that segment. That would be much easier.

Besides file locking is implemented using setgid  bit on most unices. And 
everybody is free to do what he/she thinks right with it.

May be stupid but just a thought..

 Shridhar


Fri, 31 Jan 2003 12:20:41 +0530
"Shridhar Daithankar<shridhar_daithankar@persistent.co.in>"
	<shridhar_daithankar@persistent.co.in>
Re: On file locking
Jeff Davis wrote:
>>What about it?  Someone claimed in this thread that MySQL's Windows port
>>requires Cygwin.  Is that true or not?
> 
> It's been a while, but I know I've installed MySQL on windows without any 
> separate step of installing Cygwin (I can't say 100% for sure that it didn't 
> install some part of Cygwin transparently to me).

 From the MySQL site's page about MySQL vs PostgreSQL:
http://www.mysql.com/doc/en/MySQL-PostgreSQL_features.html

"MySQL Server works better on Windows than PostgreSQL does. MySQL Server 
runs as a native Windows application (a service on NT/2000/XP), while 
PostgreSQL is run under the Cygwin emulation."

That seems pretty straightforward.

Regards and best wishes,

Justin Clift


> Regards,
> 	Jeff Davis

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Fri, 31 Jan 2003 18:19:23 +1030
Justin Clift <justin@postgresql.org>
Re: [mail] Re: Windows Build System
Man, I go away for one day, and look what you guys get into.  :-)

Let me shoot out some comments on this.

First, clearly the Win32 port is going to have more port-specific code
paths than any other port, so it is going to require extra testing even
if it wasn't our first non-Unix port.  You can expect it to take some
extra effort even after the port has stabalized because when we add
something that works only on Unix, we will need to code some workaround
in Win32.

Second, there are going to be new error cases on this platform that we
can't anticipate, and some of that isn't going to show until we get it
released.  Documenting those pitfalls, like only using NTFS, is a good
start.

Third, I suspect folks running Win32 aren't as particular about
stability/reliability, or they would have left MS products already.

Fourth, some say Win32 isn't an acceptable platform.  It may or may not
be for specific people, but Linux may be an unacceptable platform for
some people too.  I don't think we can second guess the users.  We will
do our best and see how it goes.

Also, I have heard from several people that XP is the first OS MS got
right.  That may or may not be true, but some feel things are getting
better.  It is all a continuim with these OS's.  Some are great, some
mediocre, some really bad, but people make decisions and choose bad ones
all the time.  PostgreSQL just needs to be there, if only to migrate
them to a better platform later.  If we aren't there, we can't show them
how good we are.

As for build environment, we have two audiences --- those using
binaries, and those compiling from source.  Clearly we are going to have
more binary users vs. source users on Win32 than on any other platform,
so at this stage I think making thing easier for the majority of our
Unix developers is the priority, meaning we should use our existing
Makefiles and cygwin to compile.  Later, if things warrant it, we can do
VC++ project files somehow.

Lastly, SRA just released _today_ their first Win32 port of PostgreSQL,
and it is _threaded_:

	http://osb.sra.co.jp/PowerGres/

Now, that's a port!

Also, when I am back home for an extended period starting in March, I
will going through Jan's patch (if no one does it first) and
submit/apply it in pieces that address specific Win32 issues, like path
names or carriage returns. Once those are in, we can look at the more
complex issues of build handling.

So, as far as I am concerned, we will have a Win32 port in 7.4.  It will
not be perfect, but it will be as good as we can do.  We are also
getting point-in-time recovery in 7.4, so that may help us with Win32
port failures too.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 31 Jan 2003 03:21:19 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Windows Build System - My final thoughts
Tom Lane wrote:
> But this only wins if a child process inheriting an open file also
> inherits copies of any locks held by the parent.  If not, then the
> issue is moot.  Anybody have any idea if file locks work that way?
> Is it portable??

An alternate way might be to use semaphores, but I can't see how to do
that using the standard PGSemaphores implementation: it appears to
depend on cooperating processes inheriting a copy of the postmaster's
heap.

And since the POSIX semaphores default to unnamed ones, it appears
this idea is also a dead end unless my impressions are dead wrong...



-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 31 Jan 2003 00:46:39 -0800
Kevin Brown <kevin@sysexperts.com>
Re: On file locking
On Friday 31 January 2003 05:08, Tom Lane wrote:
> Jan Wieck <JanWieck@Yahoo.com> writes:
>
> > And what about MySQL?
>
> What about it?  Someone claimed in this thread that MySQL's Windows port
> requires Cygwin.  Is that true or not?

For reference, from the INSTALL-SOURCE file included in 
the MySQL sources which I have lying about [*]:

[*] danged legacy applications ;-)

--QUOTE START--

Windows Source Distribution
---------------------------

You will need the following:

   * VC++ 6.0 compiler (updated with 4 or 5 SP and Pre-processor
     package) The Pre-processor package is necessary for the macro
     assembler.  More details at:
     `http://msdn.microsoft.com/vstudio/sp/vs6sp5/faq.asp'.

   * The MySQL source distribution for Windows, which can be downloaded
     from `http://www.mysql.com/downloads/'.

Building MySQL

  1. Create a work directory (e.g., workdir).

  2. Unpack the source distribution in the aforementioned directory.

  3. Start the VC++ 6.0 compiler.

  4. In the `File' menu, select `Open Workspace'.

  5. Open the `mysql.dsw' workspace you find on the work directory.

  6. From the `Build' menu, select the `Set Active Configuration' menu.

  7. Click over the screen selecting `mysqld - Win32 Debug' and click
     OK.

  8. Press `F7' to begin the build of the debug server, libs, and some
     client applications.

  9. When the compilation finishes, copy the libs and the executables
     to a separate directory.

 10. Compile the release versions that you want, in the same way.

 11. Create the directory for the MySQL stuff: e.g., `c:\mysql'

 12. From the workdir directory copy for the c:\mysql directory the
     following directories:

        * Data

        * Docs

        * Share

 13. Create the directory `c:\mysql\bin' and copy all the servers and
     clients that you compiled previously.

 14. If you want, also create the `lib' directory and copy the libs
     that you compiled previously.

 15. Do a clean using Visual Studio.

Set up and start the server in the same way as for the binary Windows
distribution. *Note Windows prepare environment::.

--QUOTE END--

Ian Barwick
barwick@gmx.net


Fri, 31 Jan 2003 09:55:09 +0100
Ian Barwick <barwick@gmx.net>
Re: [mail] Re: Windows Build System
On Thu, 2003-01-30 at 15:56, Tom Lane wrote:
> The reason the TIP is
> still there is that there are platforms on which that stuff doesn't work
> very nicely.  It's better to let the postmaster exit cleanly so that
> that state gets cleaned up.  I have no idea what the comparable issues
> are for a native Windows port, but I bet there are some...

That's why I proposed an automated test for this too. It is mostly
important when conquering new OS'es, but could also be nice to have when
testing if changes to storage manager or some other important subsystem
will break anything.

> 			regards, tom lane
-- 
Hannu Krosing <hannu@tm.ee>

31 Jan 2003 11:42:58 +0000
Hannu Krosing <hannu@tm.ee>
Re: [mail] Re: Windows Build System

> But this only wins if a child process inheriting an open file also
> inherits copies of any locks held by the parent.  If not, then the
> issue is moot.  Anybody have any idea if file locks work that way?
> Is it portable??

 From RedHat 8.0 manages fork(2):

SYNOPSIS
       #include <sys/types.h>
       #include <unistd.h>

       pid_t fork(void);

DESCRIPTION
       fork  creates a child process that differs from the parent process only
       in its PID and PPID, and in the fact that resource utilizations are set
       to 0.  File locks and pending signals are not inherited.
              ^^^^^^^^^^                     ^^^^^^^^^^^^^^^^^^

And from SunOS 5.8 flock
     Locks are on files, not file  descriptors.   That  is,  file
     descriptors  duplicated  through  dup(2)  or  fork(2) do not
     result in multiple instances of a lock, but rather  multiple
     references to a single lock.  If a process holding a lock on
     a file forks and the child explicitly unlocks the file,  the
     parent  will  lose  its  lock.  Locks are not inherited by a
     child process.

If I understand correctly it says that if parent dies, file is unlocked no
matter if there's children still running?

-- 
Antti Haapala


Fri, 31 Jan 2003 14:33:39 +0200 (EET)
Antti Haapala <antti.haapala@iki.fi>
Re: On file locking 
On Thu, 2003-01-30 at 20:29, Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > While I understand (and agree with) your (and Vince's) reasoning on why 
> > Windows should be considered less reliable, neither of you have provided a 
> > sound technical basis for why we should not hold the other ports to the same 
> > standards.
> 
> The point here is that Windows is virgin territory for us.  We know
> about Unix.  When we port to a new Unix variant, we are dealing with the
> same system APIs, and in many cases large chunks of the same system
> code, that we've dealt with before.  It's reasonable for us to have
> confidence that Postgres will work the same on such a platform as it
> does on other Unix variants.  And the track record of reliability that
> we have built up across a bunch of Unix variants gives us
> cross-pollinating confidence in all of them.
> 
> Windows shares none of that heritage.  It is the first truly new port,
> onto a system without any Unix background, that we have ever done AFAIK.

I don't know how much Unix backgroun BeOS has. It does have a better 
POSIX support than Win32, but I don't know how much of it is really from
Unix.

> Claiming that it doesn't require an increased level of testing is
> somewhere between ridiculous and irresponsible.

We should have at least _some_ platforms (besides Win32) that we could
clain to have run thorough test on. 

I suspect that RedHat does some (perhaps even severe) testing for
RHAS/RHDB, but I don't know of any other thorough testing. 

Or should reliability testing actually be something left for commercial
entities ? 

> > I believe we should test every release as pathologically as Vince 
> > has stated for Win32.
> 
> Great, go to it.  That does not alter the fact that today, with our
> existing port history, Windows has to be treated with extra suspicion.

I don't think that the pull-the-plug scenario happens enough in the wild
that even our seven-year track record can prove anything conlusive about
the reliability. I have not found instructions about providing that kind
of reliability in the docs either - things like what filesystems to use
on what OSes and with which mount options. 

We just mention -f as a way to get non-reliable system ;)

> I do not buy the argument you are making that we should treat all
> platforms alike.  If we had a ten-year-old Windows port, we could
> consider it as stable as all our other ten-year-old Unix ports.
> We don't.  Given that we don't have infinite resources for testing,
> it's simple rationality to put more testing emphasis on the places
> that we suspect there will be problems.  And if you don't suspect
> there will be problems on Windows, you are being way too naive :-(

"We" don't have that old windows port, but I guess that there are native
windows ports at least a few years old.

> > Do we want to encourage Win32? (some obviously do, but I don't)  Well, telling 
> > people that we have tested PostgreSQL on Win32 much more thoroughly than on 
> > Unix is in a way telling them that we think it is _better_ than the 
> > time-tested Unix ports ('It passed a harder test on Win32.  Are we afraid the 
> > Unix ports won't pass those same tests?').
> 
> If it passes the tests, good for it.  I honestly do not expect that it
> will.  My take on this is that we want to be able to document the
> problems in advance, rather than be blindsided.

Where can I read such documentations for *nix ports ?

What I have read in this list is that losing different voltages in wrong
order can just write over any sectors on a disk, and that power-cycling
can blow up computers. I don't expect even Unix to survive that!

-- 
Hannu Krosing <hannu@tm.ee>

31 Jan 2003 12:44:32 +0000
Hannu Krosing <hannu@tm.ee>
Re: [mail] Re: Windows Build System
> Jan Wieck <JanWieck@Yahoo.com> writes:
> > Assuming all your assumptions are right, why the hell is Oracle's and MS
> > SQL-Server's reputation that bloody good?
> 
> They have marketing departments.

... As well as sizable systems integration departments devoted to the 
platforms in question.  PostgreSQL doesn't have the latter, although the 
recent efforts make a move towards it.

> > And what about MySQL?
> 
> What about it?  Someone claimed in this thread that MySQL's Windows port
> requires Cygwin.  Is that true or not?

http://www.mysql.com/downloads/mysql-3.23.html
"Windows downloads

The Windows binaries use the Cygwin library. Source code for the version of 
Cygwin we have used is available on this page."
http://www.mysql.com/downloads/cygwin.html
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://www.ntlug.org/~cbbrowne/spiritual.html
"When you have eliminated the impossible, whatever remains, however
improbable, must be the truth." -- Sir Arthur Conan Doyle (1859-1930),
English author. Sherlock Holmes, in The Sign of Four, ch. 6 (1889).
[...but see the Holmesian Fallacy, due to Bob Frankston...
<http://www.frankston.com/public/Essays/Holmesian%20Fallacy.asp>]



Fri, 31 Jan 2003 08:12:14 -0500
cbbrowne@cbbrowne.com
Re: [mail] Re: Windows Build System 
Jeff Davis wrote:
> > What about it?  Someone claimed in this thread that MySQL's Windows port
> > requires Cygwin.  Is that true or not?
> 
> It's been a while, but I know I've installed MySQL on windows without any 
> separate step of installing Cygwin (I can't say 100% for sure that it didn't 
> install some part of Cygwin transparently to me).

That may have involved "not being sufficiently observant," because the company 
quite clearly documents Cygwin as a dependancy.
http://www.mysql.com/downloads/cygwin.html
--
output = ("aa454" "@freenet.carleton.ca")
http://www3.sympatico.ca/cbbrowne/linuxxian.html
Change is inevitable, except from a vending machine. 



Fri, 31 Jan 2003 08:16:24 -0500
cbbrowne@cbbrowne.com
Re: [mail] Re: Windows Build System 
> Jeff Davis wrote:
> >>What about it?  Someone claimed in this thread that MySQL's Windows port
> >>requires Cygwin.  Is that true or not?
> > 
> > It's been a while, but I know I've installed MySQL on windows without any 
> > separate step of installing Cygwin (I can't say 100% for sure that it didn'
t 
> > install some part of Cygwin transparently to me).
> 
>  From the MySQL site's page about MySQL vs PostgreSQL:
> http://www.mysql.com/doc/en/MySQL-PostgreSQL_features.html
> 
> "MySQL Server works better on Windows than PostgreSQL does. MySQL Server 
> runs as a native Windows application (a service on NT/2000/XP), while 
> PostgreSQL is run under the Cygwin emulation."
> 
> That seems pretty straightforward.

But it's not /nearly/ that straightforward.

If you look at the downloads that MySQL AB provides, they point you to a link 
that says "Windows binaries use the Cygwin library."

Which apparently means that this "feature" is not actually a feature.  Unlike 
PostgreSQL, which "is run under the Cygwin emulation," MySQL runs as a native 
Windows application (with Cygwin emulation).  Apparently those are not at all 
the same thing, even though they are both using Cygwin...
--
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne> rate me
http://cbbrowne.com/info/linuxdistributions.html
(1) Sigs are preceded by the "sigdashes" line, ie "\n-- \n" (dash-dash-space).
(2) Sigs contain at least the name and address of the sender in the first line.
(3) Sigs are at most four lines and at most eighty characters per line.



Fri, 31 Jan 2003 08:22:47 -0500
Christopher Browne <cbbrowne@cbbrowne.com>
Re: [mail] Re: Windows Build System 
Christopher Browne wrote:
<snip>
>> From the MySQL site's page about MySQL vs PostgreSQL:
>>http://www.mysql.com/doc/en/MySQL-PostgreSQL_features.html
>>
>>"MySQL Server works better on Windows than PostgreSQL does. MySQL Server 
>>runs as a native Windows application (a service on NT/2000/XP), while 
>>PostgreSQL is run under the Cygwin emulation."
>>
>>That seems pretty straightforward.
> 
> But it's not /nearly/ that straightforward.
> 
> If you look at the downloads that MySQL AB provides, they point you to a link 
> that says "Windows binaries use the Cygwin library."
> 
> Which apparently means that this "feature" is not actually a feature.  Unlike 
> PostgreSQL, which "is run under the Cygwin emulation," MySQL runs as a native 
> Windows application (with Cygwin emulation).  Apparently those are not at all 
> the same thing, even though they are both using Cygwin...

Hmm... wonder if they're meaning that MySQL compiles and executes as a 
True native windows application (skipping any unix compatibility calls), 
and it's just some of the support utils that use cygwin, or if they're 
trying to say that PostgreSQL has to operate entirely in the cygwin 
environment, whereas they don't?

Regards and best wishes,

Justin Clift

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Sat, 01 Feb 2003 00:22:00 +1030
Justin Clift <justin@postgresql.org>
Re: [mail] Re: Windows Build System
On Thu, 2003-01-30 at 16:01, Vince Vielhaber wrote:
> 
> Dave, Lamar and Katie can cheer now 'cuze this is the last comment
> I'm going to make on this.  All others will be ignored, probably.
> 
> The one thing I haven't seen from Dave, Lamar or Katie on this is
> reputation.  You're all for the PostgreSQL name going on it but I
> have yet to see any of you so sure of yourselves that you'd put
> your own name on it.  The license allows it.  Red Hat did it.  I
> see no "PageSQL" or "KatieSQL" or even an "Oh-Win SQL" being offered
> up.  Yet all three of you are advocating that the PostgreSQL stamp
> of approval should be immediately placed on it (ok, Lamar may not
> be as in favor as the Dave and Katie).
> 

Oh-win SQL! Man that was great :-) If only all of your posts were so
witty...

> Without documented testing and sufficient warnings until enough
> history is banked, I don't think a native windows port should be
> given any kind of seal of approval.  After that, what about keeping
> the code current?  In a year or so will it suffer from bit-rot and
> be the source of complaints?  Are there going to be security concerns
> surrounding it?  Is there going to be a bunch of scrambling going on
> to put out a patch when the latest active-x bug hoses the data dir?
> 

We already support postgresql on cygwin, and we know that's crap. Having
a native emulation can only improve that situation, so I don't see any
reason not to move in that direction. All of this "stamp of approval"
talk is really pointless at this juncture; no matter how much testing
has been done, none of it means a lick until the code is integrated into
the 7.4 branch. In the mean time, if some of the unix oriented guys want
to devise a suggested test plan that can be used to determine if we are
going to call the native windows support "production grade" or merely a
vast improvement over the cygwin developers version, well I bet the
windows folks would appreciate that.  Even more so if someone runs those
tests against a linux box so that we have actual statistics to compare
against. 

Robert Treat


31 Jan 2003 09:48:25 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: Windows Build System - My final thoughts
Antti Haapala <antti.haapala@iki.fi> writes:
> And from SunOS 5.8 flock
>      Locks are on files, not file  descriptors.   That  is,  file
>      descriptors  duplicated  through  dup(2)  or  fork(2) do not
>      result in multiple instances of a lock, but rather  multiple
>      references to a single lock.  If a process holding a lock on
>      a file forks and the child explicitly unlocks the file,  the
>      parent  will  lose  its  lock.  Locks are not inherited by a
>      child process.

That seems self-contradictory.  If the fork results in multiple
references to the open file, then I should think that if the parent
dies but the child still holds the file open, then the lock still
exists.  Seems that some experimentation is called for ...

			regards, tom lane

Fri, 31 Jan 2003 10:34:52 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: On file locking 
On Fri, 2003-01-31 at 07:22, Christopher Browne wrote:
> But it's not /nearly/ that straightforward.
> 
> If you look at the downloads that MySQL AB provides, they point you to a link 
> that says "Windows binaries use the Cygwin library."
> 
> Which apparently means that this "feature" is not actually a feature.  Unlike 
> PostgreSQL, which "is run under the Cygwin emulation," MySQL runs as a native 
> Windows application (with Cygwin emulation).  Apparently those are not at all 
> the same thing, even though they are both using Cygwin...

I'm confused as to whether you are being sarcastic or truly seem to
think there is a distinction here.  Simple question, does MySQL require
the cygwin dll's (or statically linked to) to run?

If the answer is yes, then there is little question that they are as
"emulated" as is the current PostgreSQL/Win32 effort.

Care to expand on exactly what you believe the distinction is?  ...or
did I miss the humor boat?  :(


Regards,

-- 
Greg Copeland <greg@copelandconsulting.net>
Copeland Computer Consulting


31 Jan 2003 12:20:00 -0600
Greg Copeland <greg@CopelandConsulting.Net>
Re: [mail] Re: Windows Build System

----- Original Message -----
From: "Greg Copeland" <greg@CopelandConsulting.Net>
> I'm confused as to whether you are being sarcastic or truly seem to
> think there is a distinction here.  Simple question, does MySQL require
> the cygwin dll's (or statically linked to) to run?
>
> If the answer is yes, then there is little question that they are as
> "emulated" as is the current PostgreSQL/Win32 effort.
>
> Care to expand on exactly what you believe the distinction is?  ...or
> did I miss the humor boat?  :(

I just installed it (their latest gama), to see what was there (and
uninstalled it straight away ;-). There was a cygwinb19.dll (I think that's
what it was called) installed.

In any case, if we are talking about "industrial strength", is this the
comparison we should be using? ;-)

andrew


Fri, 31 Jan 2003 14:08:04 -0500
"Andrew Dunstan" <andrew@dunslane.net>
Re: [mail] Re: Windows Build System
Christopher Browne wrote:
> <snip>
> >> From the MySQL site's page about MySQL vs PostgreSQL: 
> >>http://www.mysql.com/doc/en/MySQL-PostgreSQL_features.html
> >>
> >>"MySQL Server works better on Windows than PostgreSQL does. MySQL 
> >>Server runs as a native Windows application (a service on 
> >>NT/2000/XP), while PostgreSQL is run under the Cygwin emulation."
> >>
> >>That seems pretty straightforward.
> > 
> > But it's not /nearly/ that straightforward.
> > 
> > If you look at the downloads that MySQL AB provides, they 
> point you to a link that says "Windows binaries use the Cygwin
library."
> > 
> > Which apparently means that this "feature" is not actually 
> a feature.  
> > Unlike
> > PostgreSQL, which "is run under the Cygwin emulation," 
> MySQL runs as a native 
> > Windows application (with Cygwin emulation).  Apparently 
> those are not at all 
> > the same thing, even though they are both using Cygwin...

Justin Clift replied:
> Hmm... wonder if they're meaning that MySQL compiles and 
> executes as a True native windows application (skipping any unix 
> compatibility calls), and it's just some of the support utils that
> use cygwin, or if they're trying to say that PostgreSQL has to
> operate entirely in the cygwin environment, whereas they don't?

I just downloaded the latest productin source (3.3.55) and it appears to
me that:

1) It uses Cygwin emulation via a dll.

2) It uses Visual Studio C++ 6.0 for the primary build environment. It
compiles out of the box without having to learn Unix-style build
systems, config, make, etc. No warnings, no errors, it just builds out
of the box. If I did not have a lot of experience building databases I
certainly would have found their support for Windows compelling. This is
a big reason why they are #1.

3) The statement by the MySQL folks above that MySQL runs as a native
Windows application (a service on NT/2000/XP) is indicative of why MySQL
is kicking PostgreSQL's butt in terms of popularity. It is "marketing
speak" at its best. It is technically true, MySQL runs as a service. As
Christopher Browne points out, they still use the Cygwin Emulation
layer. The statement is misleading, however, as it implies that they
don't use any emulation but they do.

The salient points:
   
   a) Running as a service is important as this the way NT/2000
administrators manage server tasks. The fact that PostgreSQL's Cygwin
emulation doesn't do this is very indicative of inferior Windows
support.

   b) MySQL recognizes that the important issue is to appear to be a
well supported Windows application rather than to actually be one.

   c) It is probably much easier to add the support for running as an NT
service than it is to write a true native port with no Cygwin
dependency. NT Service support is basically a single funtion wrapper for
certain API calls (startup, shutdown, etc.) that enable the Windows
administration tools to deal with all servers in a similar manner. 

They have worked on that which makes them look better, makes their
prospective customers happier, and makes it easier to support. Exactly
what any good product development organization that listens to their
customers would have done.

<flame on>
IMHO, PostgreSQL will never have the same level of use in the field as
MySQL currently does as long as there is the kind "head in the sand"
attitude about Windows that I've seen here on the hackers list,
especially as evidenced by the recent outright attacks against those who
are simply trying to port PostgreSQL to the largest platform out there
today.

There have been some very legitimate points about Windows being a new
platform, one that will likely see a lot of users, and therefore one
that should be more thoroughly tested before release than the typical
port to another flavor of *nix. 

However, the way the conversation started reminds me of some of the chat
discussions I've seen between young teens.

I was a Mac developer way, way back and long ago realized that the best
often loses and that better marketing beats better engineering every
single time.
<\flame off>

DISCLAIMER: I hate Microsoft and Windows drives me nuts.

- Curtis






Fri, 31 Jan 2003 15:18:29 -0400
"Curtis Faith" <curtis@galtcapital.com>
Re: [mail] Re: Windows Build System
On Mon, Jan 13, 2003 at 07:31:08PM +1100, Giles Lean wrote:
> 
> Is the "Single Unix Standard, version 2" (aka UNIX98) any better?
> It says for fsync():
> 
>     "The fsync() function forces all currently queued I/O operations
>     associated with the file indicated by file descriptor fildes to
>     the synchronised I/O completion state. All I/O operations are
>     completed as defined for synchronised I/O file integrity
>     completion."

In version 3 it says:

     The fsync() function shall request that all data for the open file
     descriptor named by fildes is to be transferred to the storage
     device associated with the file described by fildes in an
     implementation-defined manner. The fsync() function shall not
     return until the system has completed that action or until an error
     is detected.

     [SIO] [Option Start] If _POSIX_SYNCHRONIZED_IO is defined, the
     fsync() function shall force all currently queued I/O operations
     associated with the file indicated by file descriptor fildes to the
     synchronized I/O completion state. All I/O operations shall be
     completed as defined for synchronized I/O file integrity
     completion. [Option End]


Kurt


Fri, 31 Jan 2003 22:11:47 +0100
Kurt Roeckx <Q@ping.be>
Re: sync()
> On Fri, 2003-01-31 at 07:22, Christopher Browne wrote:
> > But it's not /nearly/ that straightforward.

>> If you look at the downloads that MySQL AB provides, they point you
>> to a link that says "Windows binaries use the Cygwin library."

>> Which apparently means that this "feature" is not actually a feature.
>> Unlike PostgreSQL, which "is run under the Cygwin emulation," MySQL
>> runs as a native Windows application (with Cygwin emulation).
>> Apparently those are not at all the same thing, even though they are
>> both using Cygwin...
 
> I'm confused as to whether you are being sarcastic or truly seem to
> think there is a distinction here.  Simple question, does MySQL require
> the cygwin dll's (or statically linked to) to run?

I don't know if there's a distinction; read in whatever sarcasm is
deserved by the reality of things.

> If the answer is yes, then there is little question that they are as
> "emulated" as is the current PostgreSQL/Win32 effort.

Just so.  If the answer is yes, then the MySQL folk are claiming an
advantage that has no reality to it, in effect, "We aren't using Cygwin
emulation, so we're better...  (Whoops, we're actually /using/ Cygwin
emulation.)

> Care to expand on exactly what you believe the distinction is?  ...or
> did I miss the humor boat?  :(

I'm making the generous assumption that since /they/ claim that there is
some distinction, that there perhaps is one.
--
(concatenate 'string "cbbrowne" "@cbbrowne.com")
http://cbbrowne.com/info/oses.html
"All language designers are arrogant.  Goes with the territory..."  
-- Larry Wall

Fri, 31 Jan 2003 17:07:34 -0500
Christopher Browne <cbbrowne@cbbrowne.com>
Re: [mail] Re: Windows Build System 
On Friday 31 January 2003 03:21, Bruce Momjian wrote:
> Man, I go away for one day, and look what you guys get into.  :-)

No duh.  Whew.

> Lastly, SRA just released _today_ their first Win32 port of PostgreSQL,
> and it is _threaded_:

> 	http://osb.sra.co.jp/PowerGres/

Is there an English translation of the site so one who doesn't speak or write 
Japanese can try it out?
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Fri, 31 Jan 2003 19:26:27 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Windows Build System - My final thoughts
On Fri, 31 Jan 2003, Shridhar Daithankar<shridhar_daithankar@persistent.co.in> wrote:

> Besides file locking is implemented using setgid  bit on most unices. And
> everybody is free to do what he/she thinks right with it.

I don't believe it's implemented with the setgid bit on most Unices. As
I recall, it's certainly not on Xenix, SCO Unix, any of the BSDs, Linux,
SunOS, Solaris, and Tru64 Unix.

(I'm talking about the flock system call, here.)

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC

Sat, 1 Feb 2003 09:26:37 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: On file locking
Curtis Faith writes:

>    a) Running as a service is important as this the way NT/2000
> administrators manage server tasks. The fact that PostgreSQL's Cygwin
> emulation doesn't do this is very indicative of inferior Windows
> support.

No, it is indicative of the inability to read the documentation.
PostgreSQL on Cygwin runs as a service if and only if you ask it to.

-- 
Peter Eisentraut   peter_e@gmx.net


Sat, 1 Feb 2003 02:06:48 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [mail] Re: Windows Build System
> As for build environment, we have two audiences --- those using
> binaries, and those compiling from source.  Clearly we are going to have
> more binary users vs. source users on Win32 than on any other platform,
> so at this stage I think making thing easier for the majority of our
> Unix developers is the priority, meaning we should use our existing
> Makefiles and cygwin to compile.  Later, if things warrant it, we can do
> VC++ project files somehow.

I'm ignorant when it comes to build environments on windows, but I was under 
the impression that DJGPP was mostly a complete environment. Are there any 
plans to support it, or is it even possible?

> So, as far as I am concerned, we will have a Win32 port in 7.4.  It will
> not be perfect, but it will be as good as we can do.  We are also
> getting point-in-time recovery in 7.4, so that may help us with Win32
> port failures too.

Interesting consolation :)

Regards,
	Jeff Davis



Fri, 31 Jan 2003 17:35:54 -0800
Jeff Davis <jdavis-pgsql@empires.org>
Re: Windows Build System - My final thoughts
Bruce Momjian wrote:
<snip>
> 
> So, as far as I am concerned, we will have a Win32 port in 7.4.  It will
> not be perfect, but it will be as good as we can do.  We are also
> getting point-in-time recovery in 7.4, so that may help us with Win32
> port failures too.

If anyone's interested, the "PostgreSQL 7.3.1 Proof of Concept for 
Windows Alpha 1" (yes the warnings are even built into the name) 
easy-installer that I whipped up using Inno Setup was quietly uploaded 
to the pgsql project on Sourceforge the other night.  It's using 
PostgreSQL + cygwin, pretty much stock standard but pre-installed and 
wrapped up into a single installable.

As an indicater, having made no release annoucement, and only having put 
a one paragraph small mention with a link to it on the Techdocs 
"Installing On Windows" page (with warnings), over 1,600 people 
downloaded it in the first 24 hours (that's about 17.1 GB of bandwidth).

This was just a version so that I could practise some windows packaging 
and see what kind of things we'd need to address.  Dave has already 
pointed out that we're probably going to need to do this so it can be 
made into a "Merge Module" and other things.

A couple of bits of interest turned up whilst packaging:

  + There are unix command line tools that PostgreSQL relies on.  For 
example, when running initdb, it errors out if some tools aren't 
present.  i.e. sed, grep, ash (cygwin's "/bin/sh"), and from memory a 
few others


  + GPL licensing issues.  Am trying to get my head around the 
implications - with regards to licensing - if we released a proper 
version with some of the cygwin tools included... i.e. grep, sed, etc. 
Don't think that places could use it embedded with their products and 
not at least have source available, but still haven't totally grokked 
this all completely yet.  Not going to commit any code to the GBorg 
project that was setup the other day until this is sorted out. 
PostgreSQL 7.4 on Win32 should be properly BSD too.


  + Aside from all this, it might be nice to have a few Win32 specific 
gui pieces in place at the time that PostgreSQL 7.4 Win32 is released. 
Am sure they'll develop over time, but was thinking we should at least 
make a good impression with the first release.  Hey, if we make a really 
bad impression with the first release, then there might not be the 
quadruple-zillion Windows PG users after all.  If that sounds like a 
good idea, maybe adding the GUC variables "random_query_delay" 
(minutes), "crash_how_often" (seconds), and "reboot_plus_corrupt_please" 
(true/false)?

Regards and best wishes,

Justin Clift

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Sat, 01 Feb 2003 15:38:31 +1030
Justin Clift <justin@postgresql.org>
Re: Windows Build System - My final thoughts
On Sat, 1 Feb 2003, Peter Eisentraut wrote:

> Curtis Faith writes:
>
> >    a) Running as a service is important as this the way NT/2000
> > administrators manage server tasks. The fact that PostgreSQL's Cygwin
> > emulation doesn't do this is very indicative of inferior Windows
> > support.
>
> No, it is indicative of the inability to read the documentation.
> PostgreSQL on Cygwin runs as a service if and only if you ask it to.

I would say that not supporting those who have an inability to read
documentation would count as "inferior Windows support." :-)

What I'm hearing here is that all we really need to do to "compete" with
MySQL on Windows is to make the UI a bit slicker. So what's the problem
with someone building, for each release, a set of appropriate binaries, and
someone making a slick install program that will install postgres,
install parts of cygwin if necessary, and set up postgres as a service?

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC

Sat, 1 Feb 2003 15:06:29 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: [mail] Re: Windows Build System
On Fri, 31 Jan 2003, Tom Lane wrote:

> Antti Haapala <antti.haapala@iki.fi> writes:
> > And from SunOS 5.8 flock
> >      Locks are on files, not file  descriptors.   That  is,  file
> >      descriptors  duplicated  through  dup(2)  or  fork(2) do not
> >      result in multiple instances of a lock, but rather  multiple
> >      references to a single lock.  If a process holding a lock on
> >      a file forks and the child explicitly unlocks the file,  the
> >      parent  will  lose  its  lock.  Locks are not inherited by a
> >      child process.
>
> That seems self-contradictory.

Yes. I note that in NetBSD, that paragraph of the manual page is
identical except that the last sentence has been removed.

At any rate, it seems to me highly unlikely that, since the child has
the *same* descriptor as the parent had, that the lock would disappear.

The other option would be that the lock belongs to the process, in which
case one would think that a child doing an unlock should not affect the
parent, because it's a different process....

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC

Sat, 1 Feb 2003 15:11:28 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: On file locking 
On Fri, 2003-01-31 at 16:07, Christopher Browne wrote:

> I'm making the generous assumption that since /they/ claim that there is
> some distinction, that there perhaps is one.

I've used the cygwin environment enough to know that there isn't any. 
If it's linked against the cygwin dll, the application runs in an
"emulated unix environment."  To say it's emulated is really too strong
but to say it adds *tons* of overhead certainly won't make you a lair. 
;)


-- 
Greg Copeland <greg@copelandconsulting.net>
Copeland Computer Consulting


01 Feb 2003 00:33:58 -0600
Greg Copeland <greg@CopelandConsulting.Net>
Re: [mail] Re: Windows Build System
On Saturday 01 February 2003 01:26, Lamar Owen wrote:
> On Friday 31 January 2003 03:21, Bruce Momjian wrote:
> > Man, I go away for one day, and look what you guys get into.  :-)
>
> No duh.  Whew.
>
> > Lastly, SRA just released _today_ their first Win32 port of PostgreSQL,
> > and it is _threaded_:
> >
> > 	http://osb.sra.co.jp/PowerGres/
>
> Is there an English translation of the site so one who doesn't speak or
> write Japanese can try it out?

Can't see one, but here is a summarized translation of the relevant
parts as I understand them.

HTH

Ian Barwick
barwick@gmx.net


http://osb.sra.co.jp/PowerGres/
-------------------------------

"Announcement about Powergres"

* Release of [ Beta download ] of PowerGres (31.1.2003)
* [Press release] (27.11.2002)


http://osb.sra.co.jp/PowerGres/introduction.php
-----------------------------------------------

"PowerGres (PostgreSQL on Windows)"
"The standard open source database 'PostgreSQL' on Windows"

PowerGres is a DBMS which has been developed on the basis
of PostgreSQL and ported to Windows (2000 / XP).

PowerGres' features:
  Port of Postgres to Windows
     The popular Unix/Linux OS Database "postgresql"
     becomes more accessible

  Optimised for the Windows environment
     A thread model enabling effective processing of
     multiple transactions is used. This enables
     maximum performance in a windows environment.

  Web back end DB at low cost
     There is no limit to the number of users who
     can connect concurrently, making (PowerGres)
     suitable as a low cost web app backend DB

  GUI admin tool
     A GUI admin tool is packaged with powergres. This
     enables beginners to perform database management 
     visually / per point and click
  
  Japanese manual provided
     (translation of original Postgres manuals)

  C, Java Interface spport
      API for C and Java provided

  (pretty pictures, presumably of GUI admin tool)


http://osb.sra.co.jp/PowerGres/function.php
===========================================

"Table of PowerGres functions"

(comparision with "other DBs for windows", 
seems a bit pointless, left out)


http://osb.sra.co.jp/PowerGres/catalog.php
==========================================

"Product catalogue"
(more: overview)

Environment:
  CPU: Pentiium or compat, min 300Mhz
  OS: Windows 2000 (SP2 or later), XP
  Memory: 128MB (rec: 256MB +)
  Drive space: 100MB+

Product:
   - 1 CD ROM
   - PowerGres installer
   - PowerGres
   . PowerGres GUI admin tol
   - PostgreSQL 7.3 Japanese documentation
   - also:
      - PowerGres handbook
      - user registration
      - misc

Installation support
   Free support by email and fax for 30 days after registration

Price
   48,000 Yen + tax  (probably 5% sales tax; we're talking roughly
                      total US$ 500 or about the same in Euros)
   Available from March 2003 (scheduled)
   Beta download available from Jan 2003


http://powergres.sra.co.jp/
===========================

(Beta dowload)

Thankyou for your interest in PowerGres.
A free beta version of PowerGres is available.
Currently 1.0b s available for download.
It can be evaluated for 30 days. Please
do not hesitate to try before you buy.
We cannot offer any support for this software.
Use at own risk (blah blah). Also , be aware
the Beta version has some restrictions /
lack of features compared to the release version, see
here:
  http://osb.sra.co.jp/PowerGres/beta_restriction.php

(list of things, mainly command line tools with certain
 options not working properly)

DOWNLOAD FORM:

Name*
Email*
Company
Dept.
Postal code*
Address*

* required. (Note: Japanese postal code are like 111-2222 ).

( There follows a select box clicked by default enabling
SRA to send you emails...  The button is "Send".)

(following that, privacy info boilerplate).



http://osb.sra.co.jp/PowerGres/faq.php
======================================

(not a complete translation, only the interesting points 
from this page)

"FAQ"


- License: one license per server; client software
  is unrestricted ,though no free support available.
- it is possible to transfer data from a PostgreSQL 
  installation to PowerGres, though some
  restrictions apply;
- restrictions are among others:
   - max simultaneous connections 50 users
     (seems to contradict a previous statement...)
   - User-definable functions only in C, SQL, PL/pgSQL
   - No UNIX domain socket support
   - authentication only trust/reject/md5
   



Sat, 1 Feb 2003 11:08:53 +0100
Ian Barwick <barwick@gmx.net>
Re: Windows Build System - My final thoughts
Kurt Roeckx wrote:
>      [SIO] [Option Start] If _POSIX_SYNCHRONIZED_IO is defined, the
>      fsync() function shall force all currently queued I/O operations
>      associated with the file indicated by file descriptor fildes to the
>      synchronized I/O completion state. All I/O operations shall be
>      completed as defined for synchronized I/O file integrity
>      completion. [Option End]

Hmmm....so if I consistently want these semantics out of fsync() I
have to #define _POSIX_SYNCHRONIZED_IO?  Or does the above mean that
you'll get those semantics if and only if the OS defines the above for
you?

I certainly hope the former is the case, because the newer semantics
which you mentioned in the section I cut don't do us any good at all
and we can't rely on the OS to define something like
_POSIX_SYNCHRONIZED_IO for us...

Being able to open a file, do an fsync(), and have the kernel actually
write all the buffers associated with that file to disk could be, I
think, a significant performance win compared with the "flush
everything known to the kernel" approach we take now, at least on
systems that do something other than PostgreSQL...



-- 
Kevin Brown					      kevin@sysexperts.com

Sat, 1 Feb 2003 08:15:17 -0800
Kevin Brown <kevin@sysexperts.com>
Re: sync()
Curt Sampson wrote:
> On Fri, 31 Jan 2003, Shridhar Daithankar<shridhar_daithankar@persistent.co.in> wrote:
> 
> > Besides file locking is implemented using setgid  bit on most unices. And
> > everybody is free to do what he/she thinks right with it.
> 
> I don't believe it's implemented with the setgid bit on most Unices. As
> I recall, it's certainly not on Xenix, SCO Unix, any of the BSDs, Linux,
> SunOS, Solaris, and Tru64 Unix.
> 
> (I'm talking about the flock system call, here.)

Linux, at least, supports mandatory file locks.  The Linux kernel
documentation mentions that you're supposed to use fcntl() or lockf()
(the latter being a library wrapper around the former) to actually
lock the file but, when those operations are applied to a file that
has the setgid bit set but without the group execute bit set, the
kernel enforces it as a mandatory lock.  That means that operations
like open(), read(), and write() initiated by other processes on the
same file will block (or return EAGAIN, if O_NONBLOCK was used to open
it) if that's what the lock on the file calls for.

That same documentation mentions that locks acquired using flock()
will *not* invoke the mandatory lock semantics even if on a file
marked for it, so I guess flock() isn't implemented on top of fcntl()
in Linux.

So if we wanted to make use of mandatory locks, we'd have to refrain
from using flock().




-- 
Kevin Brown					      kevin@sysexperts.com

Sat, 1 Feb 2003 08:31:04 -0800
Kevin Brown <kevin@sysexperts.com>
Re: On file locking
Kevin Brown <kevin@sysexperts.com> writes:
> So if we wanted to make use of mandatory locks, we'd have to refrain
> from using flock().

We have no need for mandatory locks; the advisory style will do fine.
This is true because we have no desire to interoperate with any
non-Postgres code ... everyone else is supposed to stay the heck out of
$PGDATA.

			regards, tom lane

Sat, 01 Feb 2003 11:52:28 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: On file locking 
On Sat, Feb 01, 2003 at 08:15:17AM -0800, Kevin Brown wrote:
> Kurt Roeckx wrote:
> >      [SIO] [Option Start] If _POSIX_SYNCHRONIZED_IO is defined, the
> >      fsync() function shall force all currently queued I/O operations
> >      associated with the file indicated by file descriptor fildes to the
> >      synchronized I/O completion state. All I/O operations shall be
> >      completed as defined for synchronized I/O file integrity
> >      completion. [Option End]
> 
> Hmmm....so if I consistently want these semantics out of fsync() I
> have to #define _POSIX_SYNCHRONIZED_IO?  Or does the above mean that
> you'll get those semantics if and only if the OS defines the above for
> you?

It's something that will be defined in unistd.h.  Depending on
the value you know if the system supports it always, you can turn
it on per application, or it's always on.

You know that this standard is freely available on internet?
(http://www.unix-systems.org/version3/online.html)

There are other comments in about the usage of it.

Note that there also is a function call fdatasync() in the
Synchronized IO extention.


Kurt


Sat, 1 Feb 2003 17:56:21 +0100
Kurt Roeckx <Q@ping.be>
Re: sync()
Tom Lane wrote:
> Kevin Brown <kevin@sysexperts.com> writes:
> > So if we wanted to make use of mandatory locks, we'd have to refrain
> > from using flock().
> 
> We have no need for mandatory locks; the advisory style will do fine.
> This is true because we have no desire to interoperate with any
> non-Postgres code ... everyone else is supposed to stay the heck out of
> $PGDATA.

True.  But, of course, mandatory locks could be used to *make*
everyone else stay out of $PGDATA.  :-)


-- 
Kevin Brown					      kevin@sysexperts.com

Sat, 1 Feb 2003 09:11:45 -0800
Kevin Brown <kevin@sysexperts.com>
Re: On file locking
Kurt Roeckx wrote:
> On Sat, Feb 01, 2003 at 08:15:17AM -0800, Kevin Brown wrote:
> > Kurt Roeckx wrote:
> > >      [SIO] [Option Start] If _POSIX_SYNCHRONIZED_IO is defined, the
> > >      fsync() function shall force all currently queued I/O operations
> > >      associated with the file indicated by file descriptor fildes to the
> > >      synchronized I/O completion state. All I/O operations shall be
> > >      completed as defined for synchronized I/O file integrity
> > >      completion. [Option End]
> > 
> > Hmmm....so if I consistently want these semantics out of fsync() I
> > have to #define _POSIX_SYNCHRONIZED_IO?  Or does the above mean that
> > you'll get those semantics if and only if the OS defines the above for
> > you?
> 
> It's something that will be defined in unistd.h.  Depending on
> the value you know if the system supports it always, you can turn
> it on per application, or it's always on.
> 
> You know that this standard is freely available on internet?
> (http://www.unix-systems.org/version3/online.html)
> 
> There are other comments in about the usage of it.
> 
> Note that there also is a function call fdatasync() in the
> Synchronized IO extention.

Ah, excellent, thank you.  Yes, fdatasync() is *exactly* what we need,
since it's defined thusly: "The functionality shall be equivalent to
fsync() with the symbol _POSIX_SYNCHRONIZED_IO defined, with the
exception that all I/O operations shall be completed as defined for
synchronized I/O data integrity completion".

Looks to me like we have a winner.  Question is, can we bank on its
existence and, if so, is it properly implemented on all platforms that
support it?


Since we've been talking about porting to rather different platforms
(win32 in particular), it seems logical to build a PGFileSync()
function or something (perhaps a single PGSync() which synchronizes
all relevant PG files to disk, with sync() if necessary) and which
would thus use fdatasync() or its equivalent.



-- 
Kevin Brown					      kevin@sysexperts.com

Sat, 1 Feb 2003 09:20:36 -0800
Kevin Brown <kevin@sysexperts.com>
Re: sync()
Curt Sampson wrote:
<snip>
 > What I'm hearing here is that all we really need to do to "compete" with
 > MySQL on Windows is to make the UI a bit slicker. So what's the problem
 > with someone building, for each release, a set of appropriate 
binaries, and
 > someone making a slick install program that will install postgres,
 > install parts of cygwin if necessary, and set up postgres as a service?

The non-code related parts of the Win32 port of PostgreSQL that are 
being looked at:

  + Working on the packaging bits (slick install program) already.  Have 
created a project - pgsqlwin - on GBorg to hold any specific bits we need.

    http://gborg.postgresql.org/project/pgsqlwin/projdisplay.php

   First release of the *extremely alpha* "Proof of Concept" version is at:

    http://prdownloads.sourceforge.net/pgsql/PgSQL731wina1.exe?download


  + Concerned about including GPL stuff without having 100% totally 
investigated the ramifications for people including the Win32 version of 
PostgreSQL as a built-in part of their applications.  Not going to 
commit anything even slightly GPL related to that GBorg project until it 
100% safe to do so without affect our ability to release it as BSD. 
Have some preliminary information regarding this, but just need to wrap 
my head around it properly.  Not going to look at it closely for another 
week or so.

  + It would be greatly helpful to have some way for the install program 
to automatically add the "Log in as a service" Win32 priviledge to the 
"postgres" user without having to instruct the user to do so.  We can 
create the user automatically through a shell command, but no idea how 
to add that permission.  If someone could do some Win32 API stuff to do 
it behind the scenes without a shell command even, that would be great.

  + The WinMaster project is a first go at creating a Win32 GUI command 
console for controlling the PostgreSQL service.  It's still a bit too 
basic for real use though:

    http://gborg.postgresql.org/project/winmaster/projdisplay.php

Further suggestions, volunteers, etc are totally welcome.

:-)

Regards and best wishes,

Justin Clift


 > cjs


-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Sun, 02 Feb 2003 14:12:49 +1030
Justin Clift <justin@postgresql.org>
Re: [mail] Re: Windows Build System

Curt Sampson <cjs@cynic.net> wrote:

> At any rate, it seems to me highly unlikely that, since the child has
> the *same* descriptor as the parent had, that the lock would
> disappear.

It depends on the lock function.  After fork():

    o with flock() the lock continues to be held, but will be unlocked
      if any child process explicitly unlocks it

    o with fcntl() the lock is not inherited in the child

    o with lockf() the standards and manual pages don't say

Boring reference material follows.

flock
===== 

 From the NetBSD manual page:

NOTES
     Locks are on files, not file descriptors.  That is, file descriptors du-
     plicated through dup(2) or fork(2) do not result in multiple instances of
     a lock, but rather multiple references to a single lock.  If a process
     holding a lock on a file forks and the child explicitly unlocks the file,
     the parent will lose its lock.

The Red Hat Linux 8.0 manual page has similar wording.  (No standards
to check here -- flock() is not standardised in POSIX, X/Open, Single
Unix Standard, ...)

fcntl
=====

The NetBSD manual page notes that these locks are not inherited by
child processes:

     Another minor semantic problem with this interface is that locks
     are not inherited by a child process created using the fork(2)
     function.

Ditto the Single Unix Standard versions 2 and 3.

lockf()
=======

The standards and manual pages that I've checked don't discuss
fork() in relation to lockf(), which seems a peculiar ommission
and makes me suspect that behaviour has varied historically.

In practice I would expect lockf() semantics to be the same as
fcntl().

Regards,

Giles
  







Sun, 02 Feb 2003 16:17:22 +1100
Giles Lean <giles@nemeton.com.au>
Re: On file locking 
Giles Lean <giles@nemeton.com.au> writes:
> Boring reference material follows.

Couldn't help noticing that you omitted HPUX ;-)

On HPUX 10.20, flock doesn't seem to exist (hasn't got a man page nor
any mention in /usr/include).  lockf says

     All locks for a process are released upon
     the first close of the file, even if the process still has the file
     opened, and all locks held by a process are released when the process
     terminates.

and

     When a file descriptor is closed, all locks on the file from the
     calling process are deleted, even if other file descriptors for that
     file (obtained through dup() or open(), for example) still exist.

which seems to imply (but doesn't actually say) that HPUX keeps track of
exactly which process took out the lock, even if the file is held open
by multiple processes.

This all doesn't look good for using file locks in the way I had in
mind :-( ... but considering that all these man pages seem pretty vague,
maybe some direct experimentation is called for.

			regards, tom lane

Sun, 02 Feb 2003 02:31:47 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: On file locking 
Lamar Owen wrote:
> On Friday 31 January 2003 03:21, Bruce Momjian wrote:
> > Man, I go away for one day, and look what you guys get into.  :-)
> 
> No duh.  Whew.
> 
> > Lastly, SRA just released _today_ their first Win32 port of PostgreSQL,
> > and it is _threaded_:
> 
> > 	http://osb.sra.co.jp/PowerGres/
> 
> Is there an English translation of the site so one who doesn't speak or write 
> Japanese can try it out?

No, sorry.  Tatsuo mentioned that.  However, Babelfish will do the
translation:

	http://world.altavista.com/

Put in the URL, and choose translate Japanese to English.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sun, 2 Feb 2003 07:17:17 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Windows Build System - My final thoughts
Jeff Davis wrote:
> > As for build environment, we have two audiences --- those using
> > binaries, and those compiling from source.  Clearly we are going to have
> > more binary users vs. source users on Win32 than on any other platform,
> > so at this stage I think making thing easier for the majority of our
> > Unix developers is the priority, meaning we should use our existing
> > Makefiles and cygwin to compile.  Later, if things warrant it, we can do
> > VC++ project files somehow.
> 
> I'm ignorant when it comes to build environments on windows, but I was under 
> the impression that DJGPP was mostly a complete environment. Are there any 
> plans to support it, or is it even possible?

I don't think we want to throw our Unix folks into culture shock.  Let's
pick one build environment and go from there, either cygwin or something
else.  Once the patches are in, folks can test the various build options.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sun, 2 Feb 2003 07:19:24 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Windows Build System - My final thoughts
Justin Clift wrote:
>   + Aside from all this, it might be nice to have a few Win32 specific 
> gui pieces in place at the time that PostgreSQL 7.4 Win32 is released. 
> Am sure they'll develop over time, but was thinking we should at least 
> make a good impression with the first release.  Hey, if we make a really 
> bad impression with the first release, then there might not be the 
> quadruple-zillion Windows PG users after all.  If that sounds like a 
> good idea, maybe adding the GUC variables "random_query_delay" 
> (minutes), "crash_how_often" (seconds), and "reboot_plus_corrupt_please" 
> (true/false)?

What we need is for the backend to query postgresql.org to set those
parameters, so we can control how many Win32 users adopt PostgreSQL.  :-)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sun, 2 Feb 2003 07:21:58 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Windows Build System - My final thoughts
Bruce Momjian wrote:
> Justin Clift wrote:
> 
>>  + Aside from all this, it might be nice to have a few Win32 specific 
>>gui pieces in place at the time that PostgreSQL 7.4 Win32 is released. 
>>Am sure they'll develop over time, but was thinking we should at least 
>>make a good impression with the first release.  Hey, if we make a really 
>>bad impression with the first release, then there might not be the 
>>quadruple-zillion Windows PG users after all.  If that sounds like a 
>>good idea, maybe adding the GUC variables "random_query_delay" 
>>(minutes), "crash_how_often" (seconds), and "reboot_plus_corrupt_please" 
>>(true/false)?
> 
> 
> What we need is for the backend to query postgresql.org to set those
> parameters, so we can control how many Win32 users adopt PostgreSQL.  :-)

"All your [data] base belong to us" ?

;-)

Regards and best wishes,

Justin Clift


-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Mon, 03 Feb 2003 01:31:26 +1030
Justin Clift <justin@postgresql.org>
Re: Windows Build System - My final thoughts
On Sun, 2 Feb 2003, Tom Lane wrote:

> This all doesn't look good for using file locks in the way I had in
> mind :-( ... but considering that all these man pages seem pretty vague,
> maybe some direct experimentation is called for.

Definitely. I wonder about the NetBSD manpage quotes in the post you
followed up to, given that last time I checked flock() was implmented,
in the kernel, using fcntl(). Either that's changed, or the manpages
are unclear or lying.

This has been my experience in the past; locking semantics are subtle
and unclear enough that you really need to test for exactly what you
want at build time on every system, and you've got to do this testing
on the filesystem you intend to put the locks on. (So you don't, e.g.,
test a local filesystem but end up with data on an NFS filesystem with
different locking semantics.) That's what procmail does.

Given this, I'm not even sure the whole idea is worth persuing. (Though
I guess I should find out what NetBSD is really doing, and fix the
manual pages correspond to reality.)

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC

Mon, 3 Feb 2003 04:27:28 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: On file locking 

Curt Sampson <cjs@cynic.net> wrote:

> On Sun, 2 Feb 2003, Tom Lane wrote:
> 
> > This all doesn't look good for using file locks in the way I had in
> > mind :-( ... but considering that all these man pages seem pretty vague,
> > maybe some direct experimentation is called for.
> 
> Definitely. I wonder about the NetBSD manpage quotes in the post you
> followed up to, given that last time I checked flock() was implmented,
> in the kernel, using fcntl(). Either that's changed, or the manpages
> are unclear or lying.

Using the same kernel code != same semantics.

I think the NetBSD manual pages are trying to say that it's "safe" to
have lockf(), fcntl(), and flock() locking playing together.  That
needn't be the case on all operating systems and the standards don't
require it.

> This has been my experience in the past; locking semantics are subtle
> and unclear enough that you really need to test for exactly what you
> want at build time on every system, and you've got to do this testing
> on the filesystem you intend to put the locks on.

What he said ...

Giles

Mon, 03 Feb 2003 09:08:38 +1100
Giles Lean <giles@nemeton.com.au>
Re: On file locking 

Tom Lane wrote:

> On HPUX 10.20, flock doesn't seem to exist (hasn't got a man page nor
> any mention in /usr/include).

Correct.  Still isn't there in later releases.

>  lockf says
> 
>      All locks for a process are released upon
>      the first close of the file, even if the process still has the file
>      opened, and all locks held by a process are released when the process
>      terminates.
> 
> and
> 
>      When a file descriptor is closed, all locks on the file from the
>      calling process are deleted, even if other file descriptors for that
>      file (obtained through dup() or open(), for example) still exist.
> 
> which seems to imply (but doesn't actually say) that HPUX keeps track of
> exactly which process took out the lock, even if the file is held open
> by multiple processes.

Having done some testing today, I now understand what the standards
are trying to say when they talk about locks being "inherited". Or at
least I think I understand: standards are tricky, locking is subtle,
and I'm prepared to be corrected if I'm wrong!

All of these lock functions succeed when the same process asks for a
lock that it already has.  That is:

     fcntl(fd, ...);
     fcntl(fd, ...);  /* success -- no error returned */

For flock() only, the lock is inherited by a child process along
with the file descriptor so the child can re-issue the flock()
call and that will pass, too:

     flock(fd, ...);
     pid = fork();
     if (pid == 0)
         flock(fd, ...);  /* success -- no error returned */

For fcntl() and lockf() the locks are not inherited, and the
call in a child fails:

     fcntl(fd, ...);
     pid = fork();
     if (pid == 0)
         fcntl(fd, ...);  /* will fail and return -1 */

In no case does just closing the file descriptor in the child lose
the parent's lock.  I rationalise this as follows:

1. flock() is using a "last close" semantic, so closing the file
   descriptor is documented not to lose the lock

2. lockf() and fcntl() use a "first close", but because the locks
   are not inherited by the child process the child can't unlock
   them

> This all doesn't look good for using file locks in the way I had in
> mind :-( ... but considering that all these man pages seem pretty vague,
> maybe some direct experimentation is called for.

I conjecture that Tom was looking for a facility to lock a file and
have it stay locked if the postmaster or any child process was still
running.  flock() fits the bill, but it's not portable everywhere.

One additional warning: this stuff *is* potentially filesystem
dependent, per the source code I looked at, which would call
filesystem specific routines.

I tested with HP-UX 11.00 (VxFS), NetBSD (FFS) and Linux (ext3).  I've
put the rough and ready test code up for FTP, if anyone wants to check
my working:

    ftp://ftp.nemeton.com.au/pub/pgsql/

Limitations in the testing:

I only used whole file locking (no byte ranges) and didn't prove that
a lock taken by flock() is still held after a child calls close() as
it is documented to be.

Regards,

Giles

Mon, 03 Feb 2003 19:19:39 +1100
Giles Lean <giles@nemeton.com.au>
Re: On file locking 

> That same documentation mentions that locks acquired using flock()
> will *not* invoke the mandatory lock semantics even if on a file
> marked for it, so I guess flock() isn't implemented on top of fcntl()
> in Linux.

They're not. And there's another difference between fcntl and flock in
Linux: although fork(2) states that file locks are not inherited, locks
made by flock are inherited to children and they keep the lock even when
the parent process is killed with SIGKILL. Tested this.

Just see man syscall, there exists both
	flock(2)
and
	fcntl(2)



-- 
Antti Haapala
+358 50 369 3535
ICQ: #177673735


Mon, 3 Feb 2003 12:29:47 +0200 (EET)
Antti Haapala <antti.haapala@iki.fi>
Re: On file locking

----- Original Message -----
From: "Justin Clift" <justin@postgresql.org>
To: "Curt Sampson" <cjs@cynic.net>
Cc: "Peter Eisentraut" <peter_e@gmx.net>; "Curtis Faith"
<curtis@galtcapital.com>; <pgsql-hackers@postgresql.org>
Sent: Sunday, February 02, 2003 4:42 AM
Subject: Re: [mail] Re: [HACKERS] Windows Build System
>   + It would be greatly helpful to have some way for the install program
> to automatically add the "Log in as a service" Win32 priviledge to the
> "postgres" user without having to instruct the user to do so.  We can
> create the user automatically through a shell command, but no idea how
> to add that permission.  If someone could do some Win32 API stuff to do
> it behind the scenes without a shell command even, that would be great.
>
>   + The WinMaster project is a first go at creating a Win32 GUI command
> console for controlling the PostgreSQL service.  It's still a bit too
> basic for real use though:
>     http://gborg.postgresql.org/project/winmaster/projdisplay.php>
> Further suggestions, volunteers, etc are totally welcome.
> :-)
> Regards and best wishes,
> Justin Clift

 It's still a bit too  basic for real use though:
Yeah i know. I write this for my internal use.
Initial purpose of this stuff is only to avoid teaching of an old lady with
minimum computer skills to use bash and hide this ugly dos box :)
Mark L. Woodward (mlw) anounce few monts ago a self installing PostgreSQL
for Windows so
i write him about this console. He do a lof job to.
Special thanks Mark.

OK, now how to make WinMaster more usefull ?
It's open source so if any1 want use it he/she may help to
develop it.

I.     Install as a service feature for winmaster are included in my plans
for future.
II.    I'm thinking about direct link to PostgreSQL server instead usung
CreateProcess,
        but this is unclear idea at present time. Any suggestions will be
welcome.
III    Please add any feature rquests to
http://gborg.postgresql.org/project/winmaster/bugs/buglist.php?fr=yes
        and ideas to mailto:winmaster-general@gborg.postgresql.org

Justin you are right !!!
Further suggestions, volunteers, etc are totally welcome!!!
    Further suggestions, volunteers, etc are totally welcome!!!
        Further suggestions, volunteers, etc are totally welcome!!!



---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.449 / Virus Database: 251 - Release Date: 27/01/2003


Mon, 3 Feb 2003 12:29:27 +0100
"Igor Georgiev" <gory@alphasoft-bg.com>
Re: [mail] Re: Windows Build System

> All of these lock functions succeed when the same process asks for a
> lock that it already has.  That is:
>
>      fcntl(fd, ...);
>      fcntl(fd, ...);  /* success -- no error returned */
>
> For flock() only, the lock is inherited by a child process along
> with the file descriptor so the child can re-issue the flock()
> call and that will pass, too:
>
>      flock(fd, ...);
>      pid = fork();
>      if (pid == 0)
>          flock(fd, ...);  /* success -- no error returned */

True...

> For fcntl() and lockf() the locks are not inherited, and the
> call in a child fails:
>
>      fcntl(fd, ...);
>      pid = fork();
>      if (pid == 0)
>          fcntl(fd, ...);  /* will fail and return -1 */
>
> In no case does just closing the file descriptor in the child lose
> the parent's lock.  I rationalise this as follows:
>
> 1. flock() is using a "last close" semantic, so closing the file
>    descriptor is documented not to lose the lock

Yep.

> 2. lockf() and fcntl() use a "first close", but because the locks
>    are not inherited by the child process the child can't unlock
>    them

And at least old linux system call manuals seems to reflect this
(incorrectly) when they state that file locks are not inherited (should
be "record locks obtained by fcntl").

> One additional warning: this stuff *is* potentially filesystem
> dependent, per the source code I looked at, which would call
> filesystem specific routines.
>
> I only used whole file locking (no byte ranges) and didn't prove that
> a lock taken by flock() is still held after a child calls close() as
> it is documented to be.

I tested this on Linux 2.4.x ext2 fs and it seems to follow the spec
exactly. If child is forked and it closes the file, parent still has the
lock until it's killed or it has also closed the file.

What about having two different lock files: one that would indicate that
there are some child processes still running and another which would
indicate that there's postmaster's parent process running? - Using flock
and fcntl semantics respectively (or flock semantics with children
immediately closing their fds).

And of course locking is file system dependant, just think NFS on linux
where virtually no locking semantics actually work :)

-- 
Antti Haapala


Mon, 3 Feb 2003 14:10:49 +0200 (EET)
Antti Haapala <antti.haapala@iki.fi>
Re: On file locking 
Hannu Krosing wrote:
> 
> On Thu, 2003-01-30 at 20:29, Tom Lane wrote:
> > Claiming that it doesn't require an increased level of testing is
> > somewhere between ridiculous and irresponsible.
> 
> We should have at least _some_ platforms (besides Win32) that we could
> clain to have run thorough test on.
> 
> I suspect that RedHat does some (perhaps even severe) testing for
> RHAS/RHDB, but I don't know of any other thorough testing.
> 
> Or should reliability testing actually be something left for commercial
> entities ?

The testing has to be done before we make anything available as an
official release. As of now, the status of this project is at the
beginning of incorporating a 7.2.1 based patch into CVS HEAD.

Asking for exzessive tests at this stage of development and (ab)using
the absence of 100% proof of rock solid reliability as an excuse to
reject the entire aproach would be ridiculous.


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #

Mon, 03 Feb 2003 16:20:09 -0500
Jan Wieck <JanWieck@Yahoo.com>
Re: [mail] Re: Windows Build System
Lamar Owen wrote:
> 
> On Friday 31 January 2003 03:21, Bruce Momjian wrote:
> > Man, I go away for one day, and look what you guys get into.  :-)
> 
> No duh.  Whew.
> 
> > Lastly, SRA just released _today_ their first Win32 port of PostgreSQL,
> > and it is _threaded_:
> 
> >       http://osb.sra.co.jp/PowerGres/
> 
> Is there an English translation of the site so one who doesn't speak or write
> Japanese can try it out?

Bruce, better be careful!

If SRA hasn't done exzessive power-off and other crash testing, beware
of the dogs :-)


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #

Mon, 03 Feb 2003 16:59:53 -0500
Jan Wieck <JanWieck@Yahoo.com>
Re: Windows Build System - My final thoughts
The debate on the configuration file sparked a memory of an old patch I 
submitted in 7.1 days.

One of the things I do not like about PostgreSQL is, IMHO, is a 
backwards configuration process. Rather than specify a data directory, 
the administrator should specify a database configuration file. Within 
the configuration file is the location and names of the data directory 
and other information. Most admins want a central location for this 
information.

One of the things that is frustrating to me, is to have to hunt down 
where the data directory is so that I can administrate a DB. It can be 
anywhere, in any directory on any volume. If you had, say, a 
/usr/local/pgsql/admin, then you could have mydb.conf which could then 
be checked in to CVS. A standard location for configuration files is a 
more normal process as the location of the data directory is less so. I 
just don't think the PG data directory should not contain configuration 
information.

The original patch allowed a user to specify the location of the 
postgresql.conf file, rather than assuming it lived in $PGDATA
Also included in that patch, was the ability to specify the location of 
the PGDATA directory as well as the names of the pg_hba.conf and other 
configuration files.

It also allowed the user to use PG as it has always worked, The patch 
was not applied because a better solution was supposedly coming, but 
that was two major revisions ago. I would still like to see this 
functionality. Would anyone else?


Tue, 11 Feb 2003 13:44:15 -0500
mlw <pgsql@mohawksoft.com>
location of the configuration files
On Tue, 2003-02-11 at 13:44, mlw wrote:
> The debate on the configuration file sparked a memory of an old patch I 
> submitted in 7.1 days.
> 
> One of the things I do not like about PostgreSQL is, IMHO, is a 
> backwards configuration process. Rather than specify a data directory, 
> the administrator should specify a database configuration file. Within 
> the configuration file is the location and names of the data directory 
> and other information. Most admins want a central location for this 
> information.
> 
> One of the things that is frustrating to me, is to have to hunt down 
> where the data directory is so that I can administrate a DB. It can be 
> anywhere, in any directory on any volume. If you had, say, a 
> /usr/local/pgsql/admin, then you could have mydb.conf which could then 
> be checked in to CVS. A standard location for configuration files is a 
> more normal process as the location of the data directory is less so. I 
> just don't think the PG data directory should not contain configuration 
> information.
> 
> The original patch allowed a user to specify the location of the 
> postgresql.conf file, rather than assuming it lived in $PGDATA
> Also included in that patch, was the ability to specify the location of 
> the PGDATA directory as well as the names of the pg_hba.conf and other 
> configuration files.
> 
> It also allowed the user to use PG as it has always worked, The patch 
> was not applied because a better solution was supposedly coming, but 
> that was two major revisions ago. I would still like to see this 
> functionality. Would anyone else?
> 

I'm going to be lazy and ask if you can post what the better solution
that was coming was (or a link to the thread). While I'll grant you that
the "it's coming" argument is pretty weak after two releases, that fact
that it may have been a better solution could still hold up.

Robert Treat




11 Feb 2003 13:55:08 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: location of the configuration files

Robert Treat wrote:

>I'm going to be lazy and ask if you can post what the better solution
>that was coming was (or a link to the thread). While I'll grant you that
>the "it's coming" argument is pretty weak after two releases, that fact
>that it may have been a better solution could still hold up.
>
>Robert Treat
>  
>
AFAIK it wasn't actually done. It was more of a, "we should do something 
different" argument. At one point it was talked about rewriting the 
configuration system to allow "include" and other things.



Tue, 11 Feb 2003 14:10:34 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files

I, personally, also think it makes more sense to pass to the postmaster
a configuration file that contains all the rest of the information about
the database system, including the disk locations of the various data
directories and whatnot.

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC

Wed, 12 Feb 2003 13:53:17 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: location of the configuration files
mlw wrote:
> AFAIK it wasn't actually done. It was more of a, "we should do something 
> different" argument. At one point it was talked about rewriting the 
> configuration system to allow "include" and other things.

That seems like extreme overkill.  The PostgreSQL configuration
mechanism doesn't seem to me to be anywhere near complicated enough to
justify an "include" mechanism.

I agree with you: you should be able to specify all of the base
configuration information (including the location of the data
directories) in one file, and it makes perfect sense to me for the
location of the data directory to be a GUC variable.

I'd say the only thing the postmaster needs to know prior to startup
is the directory containing the postgresql.conf file.  An
administrator who wishes to set up multiple independent databases can
easily do so by using multiple config file directories.  When
consistency is required, he can easily use symlinks to point to master
config files where appropriate.


I assume $PGDATA was around long before GUC?


-- 
Kevin Brown					      kevin@sysexperts.com

Tue, 11 Feb 2003 22:43:00 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Kevin Brown <kevin@sysexperts.com> writes:
> I assume $PGDATA was around long before GUC?

Yes, it was.  But I have not yet seen an argument here that justifies
why $SOMECONFIGDIRECTORY/postgresql.conf is better than
$PGDATA/postgresql.conf.  The latter keeps all the related files
together.  The former seems only to introduce unnecessary complexity.
You can only justify it as simpler if you propose hardwiring a value for
$SOMECONFIGDIRECTORY ... which is a proposal that will not fly with any
of the core developers, because we all run multiple versions of Postgres
on our machines so that we can deal with back-version bug reports,
test installations, etc.  It is unlikely to fly with any of the RPM
packagers either, due to the wildly varying ideas out there about the
One True Place where applications should put their config files.

(This point was pretty much why mlw's previous proposal was rejected,
IIRC.)

			regards, tom lane

Wed, 12 Feb 2003 02:22:50 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Tom Lane wrote:
> Kevin Brown <kevin@sysexperts.com> writes:
> > I assume $PGDATA was around long before GUC?
> 
> Yes, it was.  But I have not yet seen an argument here that justifies
> why $SOMECONFIGDIRECTORY/postgresql.conf is better than
> $PGDATA/postgresql.conf.  

Okay, here's one: most Unix systems store all of the configuration
files in a well known directory: /etc.  These days it's a hierarchy of
directories with /etc as the root of the hierarchy.  When an
administrator is looking for configuration files, the first place he's
going to look is in /etc and its subdirectories.  After that, he's
forced to look through the startup scripts to figure out where things
are located.  And if those aren't revealing, then he has to read
manpages and hope they're actually useful.  :-)  And if that doesn't
work, then he has to resort to tricks like doing "strings" on the
binaries (he doesn't necessarily have access to the sources that the
binaries were compiled from, which is all that matters here).

> The latter keeps all the related files together.  The former seems
> only to introduce unnecessary complexity.

Well, I'd say it's "unnecessary" only when you already know where the
data files are located -- which is true when you're a developer or
someone who is already familiar with the installation you're working
with.  But if you're just getting started and installed it from a
package like an RPM file, then you have to look in the package to see
where it created the data file areas, or look at the startup scripts,
etc.

> You can only justify it as simpler if you propose hardwiring a value
> for $SOMECONFIGDIRECTORY ...

Making things simpler from the standpoint of the code isn't the point.
Making things simpler for the DBA and/or Unix sysadmin is.

I'd say $SOMECONFIGDIRECTORY should be a hardwired default with a
command line override.

I doubt you'll get a whole lot of argument from the general user
community if you say that the hard wired default should be
/etc/postgresql.

> which is a proposal that will not fly with any of the core
> developers, because we all run multiple versions of Postgres on our
> machines so that we can deal with back-version bug reports, test
> installations, etc.

I absolutely agree that the config directory to use should be
something that can be controlled with a command line option.

> It is unlikely to fly with any of the RPM packagers either, due to
> the wildly varying ideas out there about the One True Place where
> applications should put their config files.

There seems to be substantial agreement among the distribution
maintainers that config files belong somewhere in /etc.  At least,
I've seen very little disagreement with that idea except from people
who believe that each package should have its own, separate directory
hierarchy.  And the fact that the vast majority of packages put their
config files somewhere in /etc supports this.

Debian, for instance, actually *does* put the PostgreSQL config files
in /etc/postgresql and creates symlinks in the data directory that
point to them.  This works, but it's a kludge.

There are highly practical reasons for putting all the config files
under /etc, not the least of which is that it makes backing up files
that are *very* likely to change from the default, and which are also
highly critical to the operation of the system, very easy.

You'll get A LOT more disagreement about where to put data files than
config files, as standards go.  And in the case of PostgreSQL, where
you put your data files is especially important for performance
reasons, so it therefore makes even less sense to put the config files
in the same location: it means that the config files could literally
be anywhere, and any administrator who is unfamiliar with the system
will have to dig through startup scripts (or worse!) to figure it out.


Oh, here's another reason $SOMECONFIGDIRECTORY is better than $PGDATA:
it allows much more appropriate separation of concern by default.

Most installations of PostgreSQL start the database from a startup
script that's run at boot time.  With $PGDATA, changing the target
data directory requires changing the startup script, which requires
root access to the system -- if it didn't require root access then the
entire system is open to the possibility of a world of hurt because
the DBA isn't necessarily the same guy as the Unix sysadmin and
therefore doesn't necessarily know his way around shell scripts in
general, and rc scripts in particular, the way the Unix admin will.
The possibility of hurt comes from the fact that the rc script runs at
root, at a time that the system is hardest to work with in the event
of a failure (many systems haven't even put up any console login
prompts and may not have even started any remote login facilities
before the PostgreSQL startup script runs).  A sufficiently bad
screwup on the part of the DBA with that kind of setup will require
the Unix sysadmin to go to single user mode or worse to fix it.  So
unless the Unix sysadmin really trusts the DBA, he's not going to
allow the DBA that kind of access.  Instead he'll kludge something
together so that the DBA can make the appropriate kinds of changes
without compromising the system.  But every shop will do this a
different way.  Even Debian, which usually is pretty good about
dealing with issues like these, doesn't address this.

But it shouldn't even be necessary for a shop to kludge around the
problem: with $SOMECONFIGDIRECTORY, the Unix sysadmin can safely give
write permissions to the DBA on the config files (and even on the
entire directory they reside in), the DBA can point the database at
whatever directory he wants the data to reside in, and the Unix admin
only has to set up the storage areas, set the permissions, and let the
DBA loose on it -- he doesn't have to touch the startup scripts at
all, the DBA doesn't have to be much of a shell script wizard, and
everyone is (relatively) happy.  And, even better, the DBA will be
able to do this at most shops he works for because his knowledge will
be based on the standard PostgreSQL install.

If we want to get wider acceptance amongst the heavy database users
(which often *do* separate DBAs from Unix sysadmins), we may want to
think about things like this from time to time...



-- 
Kevin Brown					      kevin@sysexperts.com

Wed, 12 Feb 2003 05:24:30 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
On Wed, 2003-02-12 at 08:24, Kevin Brown wrote:
> Tom Lane wrote:
> > You can only justify it as simpler if you propose hardwiring a value
> > for $SOMECONFIGDIRECTORY ...
> 
> Making things simpler from the standpoint of the code isn't the point.
> Making things simpler for the DBA and/or Unix sysadmin is.
> 
> I'd say $SOMECONFIGDIRECTORY should be a hardwired default with a
> command line override.
> 
> I doubt you'll get a whole lot of argument from the general user
> community if you say that the hard wired default should be
> /etc/postgresql.
> 
> > which is a proposal that will not fly with any of the core
> > developers, because we all run multiple versions of Postgres on our
> > machines so that we can deal with back-version bug reports, test
> > installations, etc.
> 
> I absolutely agree that the config directory to use should be
> something that can be controlled with a command line option.
> 
> > It is unlikely to fly with any of the RPM packagers either, due to
> > the wildly varying ideas out there about the One True Place where
> > applications should put their config files.
> 
> There seems to be substantial agreement among the distribution
> maintainers that config files belong somewhere in /etc.  At least,
> I've seen very little disagreement with that idea except from people
> who believe that each package should have its own, separate directory
> hierarchy.  And the fact that the vast majority of packages put their
> config files somewhere in /etc supports this.
> 
> Debian, for instance, actually *does* put the PostgreSQL config files
> in /etc/postgresql and creates symlinks in the data directory that
> point to them.  This works, but it's a kludge.
> 

Seems like a good compromise would be to make the hard wired default
$SOMECONFIGDIRECTORY be $PGDATA; this makes each version of the software
more self contained/ less likely to interfere with another installation.
(This becomes really handy when doing major upgrades). If you really
have a strong desire to change this, you can.

As I see it, this change would (should?) need to be something that could
be changed in the configure script when building postgresql, as well
changeable via a command line option, any other places?

Robert Treat



12 Feb 2003 10:39:24 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: location of the configuration files
> Okay, here's one: most Unix systems store all of the configuration
> files in a well known directory: /etc.  These days it's a hierarchy of
> directories with /etc as the root of the hierarchy.  When an
> administrator is looking for configuration files, the first place he's
> going to look is in /etc and its subdirectories.  After that, he's
> forced to look through the startup scripts to figure out where things
> are located.  And if those aren't revealing, then he has to read
> manpages and hope they're actually useful.  :-)  And if that doesn't
> work, then he has to resort to tricks like doing "strings" on the
> binaries (he doesn't necessarily have access to the sources that the
> binaries were compiled from, which is all that matters here).

No goddammit - /usr/local/etc.  Why can't the Linux community respect
history!!!!

It is the ONE TRUE PLACE dammit!!!

Chris

(btw, there is humour + seriousness in above post...)


Thu, 13 Feb 2003 09:37:16 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: location of the configuration files
> > binaries (he doesn't necessarily have access to the sources that the
> > binaries were compiled from, which is all that matters here).
> 
> No goddammit - /usr/local/etc.  Why can't the Linux community respect
> history!!!!

History?  It's the only way to make a read-only / (enforced by
secure-level) and still be able to change the user applications.

I don't mind /usr/X11R6/etc either, but it's not exactly appropriate for
PostgreSQL ;)

-- 
Rod Taylor <rbt@rbt.ca>

PGP Key: http://www.rbt.ca/rbtpub.asc
12 Feb 2003 21:17:01 -0500
Rod Taylor <rbt@rbt.ca>
Re: location of the configuration files

"Christopher Kings-Lynne" <chriskl@familyhealth.com.au> wrote: 

> > Okay, here's one: most Unix systems store all of the configuration
> > files in a well known directory: /etc.  These days it's a hierarchy of
> > directories with /etc as the root of the hierarchy.  When an
> > administrator is looking for configuration files, the first place he's
> > going to look is in /etc and its subdirectories.  

> No goddammit - /usr/local/etc.  Why can't the Linux community respect
> history!!!!
> 
> It is the ONE TRUE PLACE dammit!!!

Well, to the extent that you're serious, you understand that 
a lot of people feel that /usr/local should be reserved for 
stuff that's installed by the local sysadmin, and your
vendor/distro isn't supposed to be messing with it. 

Which means if the the vendor installed Postgresql (say, the
Red Hat Database) you'd expect config files to be in /etc.
If the postgresql is compiled from source by local admin, 
you might look somewhere in /usr/local.

I've got the vauge feeling that this is all more than a
little silly... directory locations floating about depending
on who did what, as thought it were such a radical thing 
to do a ./configure, make & make install.  But this is a 
pretty common feeling among the unix world (more wide spread
than just in the Linux world). 



Wed, 12 Feb 2003 19:03:33 -0800
"J. M. Brenner" <doom@kzsu.stanford.edu>
Re: location of the configuration files 
On Wednesday 12 February 2003 20:37, Christopher Kings-Lynne wrote:
> > Okay, here's one: most Unix systems store all of the configuration
> > files in a well known directory: /etc.  These days it's a hierarchy of

> No [snip] - /usr/local/etc.  Why can't the Linux community respect
> history!!!!

> It is the ONE TRUE PLACE [snip]

If PostgreSQL is supported as a part of the base operating system in a Linux 
distribution, and that distribution wishes to be Linux Standards Base 
compliant (most do), then PostgreSQL cannot go in /usr/local -- period.

IDIC at work.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Wed, 12 Feb 2003 22:08:23 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
Tom Lane wrote:

>Kevin Brown <kevin@sysexperts.com> writes:
>  
>
>>I assume $PGDATA was around long before GUC?
>>    
>>
>
>Yes, it was.  But I have not yet seen an argument here that justifies
>why $SOMECONFIGDIRECTORY/postgresql.conf is better than
>$PGDATA/postgresql.conf.  The latter keeps all the related files
>together.  The former seems only to introduce unnecessary complexity.
>You can only justify it as simpler if you propose hardwiring a value for
>$SOMECONFIGDIRECTORY ... which is a proposal that will not fly with any
>of the core developers, because we all run multiple versions of Postgres
>on our machines so that we can deal with back-version bug reports,
>test installations, etc.  It is unlikely to fly with any of the RPM
>packagers either, due to the wildly varying ideas out there about the
>One True Place where applications should put their config files.
>
>(This point was pretty much why mlw's previous proposal was rejected,
>IIRC.)
>  
>
I wasn't talking about a "default directory" I was talking about 
configuring a database in a configuration file.

While I accept that the PostgreSQL group can not be playing catch-up 
with other databases, this does not preclude the notion accepting common 
practices and adopting them.

Understand, I really like PostgreSQL. I like it better than Oracle, and 
it is my DB of choice.  That being said, I see what other DBs do right. 
Putting the configuration in the data directory is "wrong," no other 
database or service under UNIX or Windows does this, Period.

Does the PostgreSQL team know better than the rest of the world?

The idea that a, more or less, arbitrary data location determines the 
database configuration is wrong. It should be obvious to any 
administrator that a configuration file location which controls the 
server is the "right" way to do it.  Regardless of where ever you choose 
to put the default configuration file, it is EASIER to configure a 
database by using a file in a standard configuration directory (/etc, 
/usr/etc, /usr/local/etc, /usr/local/pgsql/conf or what ever). The data 
directory should not contain configuration data as it is typically 
dependent on where the admin chooses to mount storage.

I am astounded that this point of view is missed by the core group.


Mark.








Tom Lane wrote:

Kevin Brown <kevin@sysexperts.com> writes:
  

I assume $PGDATA was around long before GUC?
    


Yes, it was.  But I have not yet seen an argument here that justifies
why $SOMECONFIGDIRECTORY/postgresql.conf is better than
$PGDATA/postgresql.conf.  The latter keeps all the related files
together.  The former seems only to introduce unnecessary complexity.
You can only justify it as simpler if you propose hardwiring a value for
$SOMECONFIGDIRECTORY ... which is a proposal that will not fly with any
of the core developers, because we all run multiple versions of Postgres
on our machines so that we can deal with back-version bug reports,
test installations, etc.  It is unlikely to fly with any of the RPM
packagers either, due to the wildly varying ideas out there about the
One True Place where applications should put their config files.

(This point was pretty much why mlw's previous proposal was rejected,
IIRC.)
  

I wasn't talking about a "default directory" I was talking about configuring
a database in a configuration file.

While I accept that the PostgreSQL group can not be playing catch-up with
other databases, this does not preclude the notion accepting common practices
and adopting them.

Understand, I really like PostgreSQL. I like it better than Oracle, and it
is my DB of choice.  That being said, I see what other DBs do right. Putting
the configuration in the data directory is "wrong," no other database or
service under UNIX or Windows does this, Period.

Does the PostgreSQL team know better than the rest of the world?

The idea that a, more or less, arbitrary data location determines the database
configuration is wrong. It should be obvious to any administrator that a
configuration file location which controls the server is the "right" way
to do it.  Regardless of where ever you choose to put the default configuration
file, it is EASIER to configure a database by using a file in a standard
configuration directory (/etc, /usr/etc, /usr/local/etc, /usr/local/pgsql/conf
or what ever). The data directory should not contain configuration data as
it is typically dependent on where the admin chooses to mount storage.

I am astounded that this point of view is missed by the core group.


Mark.
Thu, 13 Feb 2003 00:31:01 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
mlw <pgsql@mohawksoft.com> writes:
> The idea that a, more or less, arbitrary data location determines the 
> database configuration is wrong. It should be obvious to any 
> administrator that a configuration file location which controls the 
> server is the "right" way to do it.

I guess I'm just dense, but I entirely fail to see why this is the One
True Way To Do It.  What you seem to be proposing (ignoring
syntactic-sugar issues) is that we replace "postmaster -D
/some/data/dir" by "postmaster -config /some/config/file".  I am not
seeing the nature of the improvement.  It looks to me like the sysadmin
must now grant the Postgres DBA write access on *two* directories, viz
/some/config/ and /wherever/the/data/directory/is.  How is that better
than granting write access on one directory?  Given that we can't manage
to standardize the data directory location across multiple Unixen, how
is it that we will be more successful at standardizing a config file
location?

All I see here is an arbitrary break with our past practice.  I do not
see any net improvement.

			regards, tom lane

Thu, 13 Feb 2003 00:51:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Tom Lane wrote:

>mlw <pgsql@mohawksoft.com> writes:
>  
>
>>The idea that a, more or less, arbitrary data location determines the 
>>database configuration is wrong. It should be obvious to any 
>>administrator that a configuration file location which controls the 
>>server is the "right" way to do it.
>>    
>>
>
>I guess I'm just dense, but I entirely fail to see why this is the One
>True Way To Do It.  What you seem to be proposing (ignoring
>syntactic-sugar issues) is that we replace "postmaster -D
>/some/data/dir" by "postmaster -config /some/config/file".  I am not
>seeing the nature of the improvement.  It looks to me like the sysadmin
>must now grant the Postgres DBA write access on *two* directories, viz
>/some/config/ and /wherever/the/data/directory/is.  How is that better
>than granting write access on one directory?  Given that we can't manage
>to standardize the data directory location across multiple Unixen, how
>is it that we will be more successful at standardizing a config file
>location?
>
>All I see here is an arbitrary break with our past practice.  I do not
>see any net improvement.
>
>  
>
There is a pretty well understood convention that a configuration file 
will be located in some standard location depending on your distro. 
Would you disagree with that?

There is also a convention that most servers are configured by a 
configuration file, located in a central location. Look at sendmail, 
named,, et al.

Here is the test, configure a server, with sendmail, named, apache, and 
PostgreSQL. Tell me which of these systems doesn't configure right.








Tom Lane wrote:

mlw <pgsql@mohawksoft.com> writes:
  

The idea that a, more or less, arbitrary data location determines the 
database configuration is wrong. It should be obvious to any 
administrator that a configuration file location which controls the 
server is the "right" way to do it.
    


I guess I'm just dense, but I entirely fail to see why this is the One
True Way To Do It.  What you seem to be proposing (ignoring
syntactic-sugar issues) is that we replace "postmaster -D
/some/data/dir" by "postmaster -config /some/config/file".  I am not
seeing the nature of the improvement.  It looks to me like the sysadmin
must now grant the Postgres DBA write access on *two* directories, viz
/some/config/ and /wherever/the/data/directory/is.  How is that better
than granting write access on one directory?  Given that we can't manage
to standardize the data directory location across multiple Unixen, how
is it that we will be more successful at standardizing a config file
location?

All I see here is an arbitrary break with our past practice.  I do not
see any net improvement.

  

There is a pretty well understood convention that a configuration file will
be located in some standard location depending on your distro. Would you
disagree with that?

There is also a convention that most servers are configured by a configuration
file, located in a central location. Look at sendmail, named,, et al. 

Here is the test, configure a server, with sendmail, named, apache, and PostgreSQL.
Tell me which of these systems doesn't configure right.
Thu, 13 Feb 2003 01:10:41 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
At 12:31 AM -0500 2/13/03, mlw wrote:
>The idea that a, more or less, arbitrary data location determines 
>the database configuration is wrong. It should be obvious to any 
>administrator that a configuration file location which controls the 
>server is the "right" way to do it.


Isn't the database data itself a rather significant portion of the 
'configuration' of the database?

What do you gain by having the postmaster config and the database 
data live in different locations?

-pmb

Wed, 12 Feb 2003 22:14:50 -0800
Peter Bierman <bierman@apple.com>
Re: location of the configuration files
Peter Bierman wrote:

> At 12:31 AM -0500 2/13/03, mlw wrote:
>
>> The idea that a, more or less, arbitrary data location determines the 
>> database configuration is wrong. It should be obvious to any 
>> administrator that a configuration file location which controls the 
>> server is the "right" way to do it.
>
>
>
> Isn't the database data itself a rather significant portion of the 
> 'configuration' of the database?
>
> What do you gain by having the postmaster config and the database data 
> live in different locations? 

While I don't like to use another product as an example, I think amongst 
the number of things Oracle does right is that it has a fairly standard 
way for an admin to find everything. All one needs to do is find the 
"ORACLE_HOME" directory, and everything can be found from there.

If, assume, PostgreSQL worked like every other system. It would have 
either an entry in /etc or some other directory specified by configure.

Somene please tell me how what I'm proposing differs from things like 
sendmail, named, or anyother standards based UNIX server?


Thu, 13 Feb 2003 01:31:47 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files

Before I get started, I should note that it may be a good compromise
to have the data directory be the same as the config file directory,
when neither the config file nor the command line specify something
different.  So the changes I think may make the most sense are:

1.  We add a new GUC variable which specifies where the data is.
    The data is assumed to reside in the same place the config files
    reside unless the GUC variable is defined (either in
    postgresql.conf or on the command line, as usual for a GUC
    variable).  Both -D and $PGDATA therefore retain their current
    semantics unless overridden by the GUC variable, in which case
    they fall back to the new semantics of specifying only where the
    config files can be found.

2.  We add a configure option that specifies what the hardcoded
    fallback directory should be when neither -D nor $PGDATA are
    specified: /etc/postgresql when the option isn't specified to
    configure.

3.  We supply a different default startup script and a different
    default configuration file (but can make the older versions
    available in the distribution as well if we wish).  The former
    uses neither $PGDATA nor -D (or uses /etc/postgresql for them),
    and the latter uses the new GUC variable to specify a data
    directory location (/var/lib/postgres by default?)

This combination should work nicely for transitioning and for package
builders.  It accomplishes all of the goals mentioned in this thread
and will cause minimal pain for developers, since they can use their
current methods.  Sounds like it'll make Tom happy, at least.  :-)


Tom Lane wrote:
> mlw <pgsql@mohawksoft.com> writes:
> > The idea that a, more or less, arbitrary data location determines the 
> > database configuration is wrong. It should be obvious to any 
> > administrator that a configuration file location which controls the 
> > server is the "right" way to do it.
> 
> I guess I'm just dense, but I entirely fail to see why this is the One
> True Way To Do It.  

But we're not saying it's the One True Way, just saying that it's a
way that has very obvious benefits over the way we're using now, if
your job is to manage a system that someone else set up.

> What you seem to be proposing (ignoring syntactic-sugar issues) is
> that we replace "postmaster -D /some/data/dir" by "postmaster
> -config /some/config/file".  I am not seeing the nature of the
> improvement.

The nature of the improvement is that the configuration of a
PostgreSQL install will becomes obvious to anyone who looks in the
obvious places.  Remember, the '-D ...' is optional!  The PGDATA
environment variable can be used instead, and *is* used in what few
installations I've seen.  That's not something that shows up on the
command line when looking at the process list, which forces the
administrator to hunt down the data directory through other means.

> It looks to me like the sysadmin must now grant the Postgres DBA
> write access on *two* directories, viz /some/config/ and
> /wherever/the/data/directory/is.  How is that better than granting
> write access on one directory?

The difference in where you grant write access isn't a benefit to be
gained here.  The fact that you no longer have to give root privileges
to the DBA so that he can change the data directory as needed is the
benefit (well, one of them, at least).  A standard packaged install
can easily set the /etc/postgresql directory up with write permissions
for the postgres user by default, so the sysadmin won't even have to
touch it if he doesn't want to.

A big production database box is usually managed by one or more system
administrators and one or more DBAs.  Their roles are largely
orthogonal.  The sysadmins have the responsibility of keeping the
boxes up and making sure they don't fall over or crawl to a
standstill.  The DBAs have the responsibility of maximizing the
performance and availability of the database and *that's all*.  Giving
the DBAs root privileges means giving them the power to screw up the
system in ways that they can't recover from and might not even know
about.  The ways you can take down a system by misconfiguring the
database are bad enough.  No sane sysadmin is going to give the DBA
the power to run an arbitrary script as root at a time during the boot
cycle that the system is the most difficult to manage unless he thinks
the DBA is *really* good at system administration tasks, too.  And
that's assuming the sysadmin even *has* the authority to grant the DBA
that kind of access.  Many organizations keep a tight rein on who can
do what in an effort to minimize the damage from screwups.

The point is that the DBA isn't likely to have root access to the box.
When the DBA lacks that ability, the way we currently do things places
greater demand on the sysadmin than is necessary, because root access
is required to change the startup scripts, as it should be, and the
location of the data, as it should *not* be.

> Given that we can't manage to standardize the data directory
> location across multiple Unixen, how is it that we will be more
> successful at standardizing a config file location?

A couple of ways.

Firstly, as we mentioned before, just about every other daemon that
runs on a Unix system has its configuration file somewhere in the /etc
hierarchy.  By putting our config files in that same hierarchy we'll
be *adhering* to a standard.  We don't have to worry about
"standardizing" that config file location because it's *already* a
standard that we're currently ignoring.

Secondly, standards arise as a result of being declared standards and
by most people using them.  So simply by making /etc/postgresql the
default configuration directory, *that* will become the standard
place.  Most people won't mess with the default install if they don't
have to.

Right now they almost *have to* mess with the default install, because
there is no standard place on a Unix system for high speed, highly
reliable disk access.  And that means that, right now, there *is* no
standard place for our config files -- it's wherever the person who
configured the database decided the data should be, and he made that
decision based on performance and reliability considerations, not on
any standards.

> All I see here is an arbitrary break with our past practice.  I do not
> see any net improvement.

That's probably because you're looking at this from the point of view
of a developer.  From that standpoint there really isn't any net
improvement, because *you* still have to specify something on the
command line to get your test databases going.  As a developer you
*always* install and manage your own database installations, so *of
course* you'll always know where the config files are.  But that's not
how it works in the production world.

The break we'd be making is *not* arbitrary, and that's much of the
point: it's a break towards existing standards, and there are good
reasons for doing it, benefits to be had by adhering to those
standards.


The way we currently handle configuration files is fine for research
and development use -- the environment from which PostgreSQL sprang.
But now we're talking about getting it used in production
environments, and their requirements are very different.

Since it is *we* who are not currently adhering to the standard,
shouldn't the burden of proof (so to speak) be on those who wish to
keep things as they are?




-- 
Kevin Brown					      kevin@sysexperts.com

Wed, 12 Feb 2003 23:36:45 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files

> Well, to the extent that you're serious, you understand that 
> a lot of people feel that /usr/local should be reserved for 
> stuff that's installed by the local sysadmin, and your
> vendor/distro isn't supposed to be messing with it. 
> 
> Which means if the the vendor installed Postgresql (say, the
> Red Hat Database) you'd expect config files to be in /etc.
> If the postgresql is compiled from source by local admin, 
> you might look somewhere in /usr/local.

Indeed. For better or worse, there is a Filesystem Hierarcy Standard,
and most of the important Linux distros, BSDs and some legacy Unixen
stick to it, so so should we.

Configuration files should be in /etc/postgresql/, or at the very least
symlinked from there.

Martin


13 Feb 2003 11:20:46 +0000
Martin Coxall <coxall@cream.org>
Re: location of the configuration files
On Wed, 12 Feb 2003, J. M. Brenner wrote:

>
> "Christopher Kings-Lynne" <chriskl@familyhealth.com.au> wrote:
>
> > > Okay, here's one: most Unix systems store all of the configuration
> > > files in a well known directory: /etc.  These days it's a hierarchy of
> > > directories with /etc as the root of the hierarchy.  When an
> > > administrator is looking for configuration files, the first place he's
> > > going to look is in /etc and its subdirectories.
>
> > No goddammit - /usr/local/etc.  Why can't the Linux community respect
> > history!!!!
> >
> > It is the ONE TRUE PLACE dammit!!!
>
> Well, to the extent that you're serious, you understand that
> a lot of people feel that /usr/local should be reserved for
> stuff that's installed by the local sysadmin, and your
> vendor/distro isn't supposed to be messing with it.
>
> Which means if the the vendor installed Postgresql (say, the
> Red Hat Database) you'd expect config files to be in /etc.
> If the postgresql is compiled from source by local admin,
> you might look somewhere in /usr/local.

Then why not ~postgres/etc ??  Or substitute ~postgres with the
db admin user you (or the distro) decided on at installation time.
Gives a common location no matter who installed it or where it was
installed.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Thu, 13 Feb 2003 07:00:11 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: location of the configuration files 
On Wed, 12 Feb 2003, Peter Bierman wrote:

> What do you gain by having the postmaster config and the database
> data live in different locations?

You can then standardize a location for the configuration files.

Everybody has room in /etc for another 10K of data. Where you have
room for something that might potentially be a half terrabyte of
data, and is not infrequently several gigabytes or more, is pretty
system-depenendent.

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC

Thu, 13 Feb 2003 22:09:41 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: location of the configuration files
In the last exciting episode, cjs@cynic.net (Curt Sampson) wrote:
> On Wed, 12 Feb 2003, Peter Bierman wrote:
>
>> What do you gain by having the postmaster config and the database
>> data live in different locations?
>
> You can then standardize a location for the configuration files.
>
> Everybody has room in /etc for another 10K of data. Where you have
> room for something that might potentially be a half terrabyte of
> data, and is not infrequently several gigabytes or more, is pretty
> system-depenendent.

Ah, but this has two notable problems:

1.  It assumes that there is "a location" for "the configuration files
    for /the single database instance./"

    If I have a second database instance, that may conflict.

2.  It assumes I have write access to /etc

    If I'm a Plain Old User, as opposed to root, I may only have
    read-only access to /etc.

These conditions have both been known to occur...
-- 
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne> rate me
http://www.ntlug.org/~cbbrowne/rdbms.html
"Some people, when confronted with a Unix problem, think "I know, I'll
use sed." Now they have two problems."  -- Jamie Zawinski

Thu, 13 Feb 2003 08:32:51 -0500
Christopher Browne <cbbrowne@acm.org>
Re: location of the configuration files
On Thu, 13 Feb 2003, Christopher Browne wrote:

> 1.  It assumes that there is "a location" for "the configuration files
>     for /the single database instance./"

No; it assumes that there's a location for "the default instance." If
you have more than one, you could have one default and one elsewhere, or
just do what I often do, which is put in an empty config file except for
a comment saying "we have several instances of <xxx> on this machine; look
in <yyy> for them."

> 2.  It assumes I have write access to /etc
>
>     If I'm a Plain Old User, as opposed to root, I may only have
>     read-only access to /etc.

Right. It's dependent on the sysadmin to create /etc/postgres/ and make
it writeable, or set up proper symlinks, or whatever.

Fortunately, the files in /etc are only the defaults, to be used if
they're not overridden on the command line. If you're in a situation
like #2, you're basically stuck where we are now all the time: you have
to just put it somewhere and hope that, if someone else needs to find
it, they can.

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC

Thu, 13 Feb 2003 23:18:00 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: location of the configuration files
Christopher Browne wrote:

>In the last exciting episode, cjs@cynic.net (Curt Sampson) wrote:
>  
>
>>On Wed, 12 Feb 2003, Peter Bierman wrote:
>>
>>    
>>
>>>What do you gain by having the postmaster config and the database
>>>data live in different locations?
>>>      
>>>
>>You can then standardize a location for the configuration files.
>>
>>Everybody has room in /etc for another 10K of data. Where you have
>>room for something that might potentially be a half terrabyte of
>>data, and is not infrequently several gigabytes or more, is pretty
>>system-depenendent.
>>    
>>
>
>Ah, but this has two notable problems:
>
>1.  It assumes that there is "a location" for "the configuration files
>    for /the single database instance./"
>
>    If I have a second database instance, that may conflict.
>
>2.  It assumes I have write access to /etc
>
>    If I'm a Plain Old User, as opposed to root, I may only have
>    read-only access to /etc.
>
>These conditions have both been known to occur...
>  
>
These are not issues at all. You could put the configuration file 
anywhere, just as you can for any UNIX service.

postmaster --config=/home/myhome/mydb.conf

I deal with a number of PG databases on a number of sites, and it is a 
real pain in the ass to get to a PG box and hunt around for data 
directory so as to be able to administer the system. What's really 
annoying is when you have to find the data directory when someone else 
set up the system.

Configuring postgresql via a configuration file which specifies all the 
data, i.e. data directory, name of other configuration files, etc. is 
the right way to do it. Even if you have reasons against it, even if you 
think it is a bad idea, a bad standard is almost always a better 
solution than an arcane work of perfection.

Personally, however, I think the configuration issue is a no-brainer and 
I am amazed that people are balking. EVERY other service on a UNIX box 
is configured in this way, why not do it this way in PostgreSQL? The 
patch I submitted allowed the configuration to work as it currently 
does, but allowed for the more standard configuration file methodology.

I just don't understand what the resistance is, it makes no sense.













Christopher Browne wrote:

In the last exciting episode, cjs@cynic.net (Curt Sampson) wrote:
  

On Wed, 12 Feb 2003, Peter Bierman wrote:

    

What do you gain by having the postmaster config and the database
data live in different locations?
      

You can then standardize a location for the configuration files.

Everybody has room in /etc for another 10K of data. Where you have
room for something that might potentially be a half terrabyte of
data, and is not infrequently several gigabytes or more, is pretty
system-depenendent.
    


Ah, but this has two notable problems:

1.  It assumes that there is "a location" for "the configuration files
    for /the single database instance./"

    If I have a second database instance, that may conflict.

2.  It assumes I have write access to /etc

    If I'm a Plain Old User, as opposed to root, I may only have
    read-only access to /etc.

These conditions have both been known to occur...
  

These are not issues at all. You could put the configuration file anywhere,
just as you can for any UNIX service.

postmaster --config=/home/myhome/mydb.conf

I deal with a number of PG databases on a number of sites, and it is a real
pain in the ass to get to a PG box and hunt around for data directory so
as to be able to administer the system. What's really annoying is when you
have to find the data directory when someone else set up the system.

Configuring postgresql via a configuration file which specifies all the data,
i.e. data directory, name of other configuration files, etc. is the right
way to do it. Even if you have reasons against it, even if you think it is
a bad idea, a bad standard is almost always a better solution than an arcane
work of perfection.

Personally, however, I think the configuration issue is a no-brainer and
I am amazed that people are balking. EVERY other service on a UNIX box is
configured in this way, why not do it this way in PostgreSQL? The patch I
submitted allowed the configuration to work as it currently does, but allowed
for the more standard configuration file methodology.

I just don't understand what the resistance is, it makes no sense.
Thu, 13 Feb 2003 09:23:20 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
On Thursday 13 February 2003 08:32, Christopher Browne wrote:
> In the last exciting episode, cjs@cynic.net (Curt Sampson) wrote:
> > Everybody has room in /etc for another 10K of data. Where you have
> > room for something that might potentially be a half terrabyte of
> > data, and is not infrequently several gigabytes or more, is pretty
> > system-depenendent.

> 1.  It assumes that there is "a location" for "the configuration files
>     for /the single database instance./"

>     If I have a second database instance, that may conflict.

If you run multiple servers of any kind, the second and subsequent servers 
must have a command line switch specifying the location of the config file.  
This is the way named, sendmail, et al do it.  I have run multiple nameds on 
a single box, by using alternate config file locations.

> 2.  It assumes I have write access to /etc

>     If I'm a Plain Old User, as opposed to root, I may only have
>     read-only access to /etc.

So you start postmaster with a config file switch pointing to your config file 
in your tree.  Or you specify the default location with a configure switch at 
compile time.  Or you do it the same way you would run any other typical 
daemon as a regular user.  How does Apache, AOLserver (my favorite), 
sendmail, jabberd, named, or any other typical daemon do it?  

For example, AOLserver can easily be installed and run as a plain user (just 
not on port 80).  The command line switch '-t' specifies the tcl 
configuration script's location.  There is no default.  The configuration 
script then specifies pageroot and the like -- and a webserver is very much 
like running PostgreSQL -- you have configuration, you have logs, and you 
have the spool (database or pageroot).  All can be in different locations at 
the discretion of the admin.  And hardcoding dependencies like this stifles 
the discretion of the admin.

> These conditions have both been known to occur...

Just because the situation is known to occur doesn't mean the whole direction 
of a project should hinge on those corner cases.  They should be allowed but 
not forced.

For better or for worse, thanks to Karl DeBisschop, the latest RPMs have the 
ability to start multiple postmasters with their data trees and 
configurations in different places.  This is all done in the startup script, 
and required no new functionality from postmaster.   I personally think it is 
for the better; YMMV.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 13 Feb 2003 09:50:22 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
On Thu, 13 Feb 2003, Curt Sampson wrote:

> On Thu, 13 Feb 2003, Christopher Browne wrote:
> 
> > 1.  It assumes that there is "a location" for "the configuration files
> >     for /the single database instance./"
> 
> No; it assumes that there's a location for "the default instance." If
> you have more than one, you could have one default and one elsewhere, or
> just do what I often do, which is put in an empty config file except for
> a comment saying "we have several instances of <xxx> on this machine; look
> in <yyy> for them."
> 
> > 2.  It assumes I have write access to /etc
> >
> >     If I'm a Plain Old User, as opposed to root, I may only have
> >     read-only access to /etc.
> 
> Right. It's dependent on the sysadmin to create /etc/postgres/ and make
> it writeable, or set up proper symlinks, or whatever.
> 
> Fortunately, the files in /etc are only the defaults, to be used if
> they're not overridden on the command line. If you're in a situation
> like #2, you're basically stuck where we are now all the time: you have
> to just put it somewhere and hope that, if someone else needs to find
> it, they can.

It doesn't follow this line of argument directly but it's to do with this
thread...

Is everyone forgetting that wherever the configuration file is stored and
whether or not it needs a command line argument to specify it the database is
not going to start up automatically unless at least part of the installation is
done as root anyway?

As I like to install software as a non root user normally anyway I am happy
that the config file lives somewhere not requiring write access by the
installer. However, I think having it in an etc directory is a good thing
(tm). So, colour me an uncommited, fence sitter :)

I'm not talking distribution/package installation here but just plain system
administration. Being an untrusting soul I do _not_ want to type make install
as root and find things installed outside of where I say I want things placed.
That includes configuration files. Doing this as a normal user protects the
system from bad software which assumes things about the host system. It also
simplifies switching between versions of software, try doing that if your
config is /etc/postgresql/postgres.conf.


-- 
Nigel J. Andrews


Thu, 13 Feb 2003 15:03:09 +0000 (GMT)
"Nigel J. Andrews" <nandrews@investsystems.co.uk>
Re: location of the configuration files
mlw <pgsql@mohawksoft.com> writes:
> Here is the test, configure a server, with sendmail, named, apache, and 
> PostgreSQL. Tell me which of these systems doesn't configure right.

AFAIK, only one of those four is designed to support multiple instances
running on a single machine.  This is not unrelated.

			regards, tom lane

Thu, 13 Feb 2003 10:03:42 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Tom Lane wrote:

>mlw <pgsql@mohawksoft.com> writes:
>  
>
>>Here is the test, configure a server, with sendmail, named, apache, and 
>>PostgreSQL. Tell me which of these systems doesn't configure right.
>>    
>>
>
>AFAIK, only one of those four is designed to support multiple instances
>running on a single machine.  This is not unrelated.
>
>  
>
While I will agree with you on sendmail and named, Apache is often run 
more than once with different options.
Furthermore, I hate to keep bringing it up, Oracle does use the 
configuration file methodology.

Tom, I just don't understand why this is being resisted so vigorously. 
What is wrong with starting PostgreSQL as:

postmaster -C /etc/postgresql.conf

UNIX admins would love to have this as a methodology, I don't think you 
can deny this, can you? I, as a long term PG user, really really want 
this, because in the long run, it makes PostgreSQL easier to administer.

If a patch allows PG to function as it does, but also allows a 
configuration file methodology, why not?








Tom Lane wrote:

mlw <pgsql@mohawksoft.com> writes:
  

Here is the test, configure a server, with sendmail, named, apache, and 
PostgreSQL. Tell me which of these systems doesn't configure right.
    


AFAIK, only one of those four is designed to support multiple instances
running on a single machine.  This is not unrelated.

  

While I will agree with you on sendmail and named, Apache is often run more
than once with different options.
Furthermore, I hate to keep bringing it up, Oracle does use the configuration
file methodology.

Tom, I just don't understand why this is being resisted so vigorously. What
is wrong with starting PostgreSQL as:

postmaster -C /etc/postgresql.conf

UNIX admins would love to have this as a methodology, I don't think you can
deny this, can you? I, as a long term PG user, really really want this, because
in the long run, it makes PostgreSQL easier to administer.

If a patch allows PG to function as it does, but also allows a configuration
file methodology, why not?
Thu, 13 Feb 2003 10:19:11 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
On Thu, 13 Feb 2003, mlw wrote:

> Tom, I just don't understand why this is being resisted so vigorously. 
> What is wrong with starting PostgreSQL as:
> 
> postmaster -C /etc/postgresql.conf
> 
> UNIX admins would love to have this as a methodology, I don't think you 
> can deny this, can you? I, as a long term PG user, really really want 
> this, because in the long run, it makes PostgreSQL easier to administer.
> 
> If a patch allows PG to function as it does, but also allows a 
> configuration file methodology, why not?

I forgot to say that I don't see why this facility can't be included in
addition to the existing scheme.


-- 
Nigel J. Andrews


Thu, 13 Feb 2003 15:22:39 +0000 (GMT)
"Nigel J. Andrews" <nandrews@investsystems.co.uk>
Re: location of the configuration files
On Thursday 13 February 2003 10:03, Tom Lane wrote:
> mlw <pgsql@mohawksoft.com> writes:
> > Here is the test, configure a server, with sendmail, named, apache, and
> > PostgreSQL. Tell me which of these systems doesn't configure right.

> AFAIK, only one of those four is designed to support multiple instances
> running on a single machine.  This is not unrelated.

One can run many nameds on a single machine by specifying '-c 
alternate_named.conf' , which then points to a different set of zone files, 
listens to either a different port or address, etc.  I have personally done 
this, and it worked as if it were designed to do so.

Apache can easily have multiple instances by passing the location of 
httpd.conf on the command line.  Everything else comes from that. Although 
Apache's virtual hosting is typically use instead, it may be necessary for 
large sites to run multiple instances with degrees of separation at the 
config file level.

I use AOLserver, which in version 3 is designed from the ground up for many 
(even thousands) of instances on a single box.  You access this capability 
with the '-t' switch (it stands for 'tcl config script' -- previous versions 
of AOLserver had an 'ini' file accessed with '-i', and version 3 added the 
tcl config script and deprecated the ini file).  In fact, since there is no 
default, you MUST use -t.  The tcl config script specifies all the parameters 
that instance needs (with the exception of the user and group ID the server 
should run as, if started as root (for port 80 access) -- but that doesn't 
effect PostgreSQL since our port is above 1024).  Two instances can even 
share a tcl config script, as long as the virtual server name is specified on 
the command line, and the tcl config has multiple virtual server sections.  

I personally only lightly use this feature, running a mere half dozen 
AOLserver's on one of my production servers.  All of which share a single 
PostgreSQL instance; but that's a different story.  

AOLserver is an excellent example here, as everything that has a location is 
configurable.  During ./configure, you can specify the prefix and the other 
standard autoconf type options.  This includes the location of the 
--enable-thread built Tcl 8.4 that you have to have first.  I have two 
versions of AOLserver on that machine, and they coexist very well, because I 
_can_ be so specific in where everything lies.  I run OpenACS on two of those 
instances, and, due to the size of that install I have those two pageroots on 
a separate filesystem from the binaries and config script.  It was just a 
single tcl config entry.  No biggie.

Even sendmail has a -c switch to specify the location of sendmail.cf: so, yes, 
you can run multiple instances, although it could be argued that it wasn't 
designed in.

Next?
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 13 Feb 2003 10:32:19 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
Tom Lane wrote:

>mlw <pgsql@mohawksoft.com> writes:
>  
>
>>Here is the test, configure a server, with sendmail, named, apache, and 
>>PostgreSQL. Tell me which of these systems doesn't configure right.
>>    
>>
>
>AFAIK, only one of those four is designed to support multiple instances
>running on a single machine.  This is not unrelated.
>  
>
Also, using an explicit configuration file will also help you run 
multiple postgresql's on the same machien in a consistent manner, for 
instance:

postmaster -C /etc/postgres/common.conf -D /RAID0/postgres -p 5432
postmaster -C /etc/postgres/common.conf -D /RAID1/postgres -p 5433

Please, Tom, tell me why this is such a bad idea?

I will make the patch, I will submit it, will you guys put it in?

If not, why?

>  
>









Tom Lane wrote:

mlw <pgsql@mohawksoft.com> writes:
  

Here is the test, configure a server, with sendmail, named, apache, and 
PostgreSQL. Tell me which of these systems doesn't configure right.
    


AFAIK, only one of those four is designed to support multiple instances
running on a single machine.  This is not unrelated.
  

Also, using an explicit configuration file will also help you run multiple
postgresql's on the same machien in a consistent manner, for instance:

postmaster -C /etc/postgres/common.conf -D /RAID0/postgres -p 5432
postmaster -C /etc/postgres/common.conf -D /RAID1/postgres -p 5433

Please, Tom, tell me why this is such a bad idea?

I will make the patch, I will submit it, will you guys put it in?

If not, why?
Thu, 13 Feb 2003 10:59:11 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
On Thu, Feb 13, 2003 at 15:03:09 +0000,
  "Nigel J. Andrews" <nandrews@investsystems.co.uk> wrote:
> 
> Is everyone forgetting that wherever the configuration file is stored and
> whether or not it needs a command line argument to specify it the database is
> not going to start up automatically unless at least part of the installation is
> done as root anyway?

Users can use cron to start there own instance. Your cron script can check
if the server is running every (say) 15 minutes and start the server
if it isn't.

Thu, 13 Feb 2003 10:28:52 -0600
Bruno Wolff III <bruno@wolff.to>
Re: location of the configuration files
On Thu, 2003-02-13 at 09:23, mlw wrote: 
> I deal with a number of PG databases on a number of sites, and it is a
> real pain in the ass to get to a PG box and hunt around for data
> directory so as to be able to administer the system. What's really
> annoying is when you have to find the data directory when someone else
> set up the system.
> 

find / -name postgresql.conf -print

you now know where all of your configuration files are and where the
data for each of those servers is as well. 

(Not I'm not against the idea...)

Robert Treat


13 Feb 2003 11:32:24 -0500
Robert Treat <rtreat@webmd.net>
Re: location of the configuration files
On Thu, Feb 13, 2003 at 09:23:20 -0500,
  mlw <pgsql@mohawksoft.com> wrote:
> 
> Personally, however, I think the configuration issue is a no-brainer and 
> I am amazed that people are balking. EVERY other service on a UNIX box 
> is configured in this way, why not do it this way in PostgreSQL? The 
> patch I submitted allowed the configuration to work as it currently 
> does, but allowed for the more standard configuration file methodology.

If you are interested in reading a contrary position, you can read
Berstein's arguments for his recommended way to install services at:
http://cr.yp.to/unix.html

Thu, 13 Feb 2003 10:36:51 -0600
Bruno Wolff III <bruno@wolff.to>
Re: location of the configuration files
Robert Treat wrote:

>On Thu, 2003-02-13 at 09:23, mlw wrote: 
>  
>
>>I deal with a number of PG databases on a number of sites, and it is a
>>real pain in the ass to get to a PG box and hunt around for data
>>directory so as to be able to administer the system. What's really
>>annoying is when you have to find the data directory when someone else
>>set up the system.
>>
>>    
>>
>
>find / -name postgresql.conf -print
>

LOL, That is NOT an option.  It can take hours on some systems. 
Specifically, one of the systems is  freedb server, it has over 300,000 
files in a directory tree.









Robert Treat wrote:

On Thu, 2003-02-13 at 09:23, mlw wrote: 
  

I deal with a number of PG databases on a number of sites, and it is a
real pain in the ass to get to a PG box and hunt around for data
directory so as to be able to administer the system. What's really
annoying is when you have to find the data directory when someone else
set up the system.

    


find / -name postgresql.conf -print


LOL, That is NOT an option.  It can take hours on some systems. Specifically,
one of the systems is  freedb server, it has over 300,000 files in a directory
tree.
Thu, 13 Feb 2003 11:47:22 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
On Thu, 13 Feb 2003, mlw wrote:

>
>
> Robert Treat wrote:
>
> >On Thu, 2003-02-13 at 09:23, mlw wrote:
> >
> >
> >>I deal with a number of PG databases on a number of sites, and it is a
> >>real pain in the ass to get to a PG box and hunt around for data
> >>directory so as to be able to administer the system. What's really
> >>annoying is when you have to find the data directory when someone else
> >>set up the system.

You realize that the actual code feature doesn't necessarily help this
case, right? Putting configuration in /etc and having a configuration file
option on the command line are separate concepts.

I think the feature is worthwhile, but I have some initial condition
functionality questions that may have been answered in the previous patch,
but I don't remember at this point.

Mostly these have to deal with initial creation.  Does the user specify an
output location to initdb, do they just specify a data dir as now where
the configuration goes but then they need to move it somewhere, does
initdb now do nothing relating to configuration file and the user should
make one on his own.  Related, is the admin expected to have already made
(say) /etc/postgresql to stick the config in and set the permissions
correctly (since initdb doesn't run as root)?



Thu, 13 Feb 2003 08:49:21 -0800 (PST)
Stephan Szabo <sszabo@megazone23.bigpanda.com>
Re: location of the configuration files
Bruno Wolff III wrote:

>On Thu, Feb 13, 2003 at 09:23:20 -0500,
>  mlw <pgsql@mohawksoft.com> wrote:
>  
>
>>Personally, however, I think the configuration issue is a no-brainer and 
>>I am amazed that people are balking. EVERY other service on a UNIX box 
>>is configured in this way, why not do it this way in PostgreSQL? The 
>>patch I submitted allowed the configuration to work as it currently 
>>does, but allowed for the more standard configuration file methodology.
>>    
>>
>
>If you are interested in reading a contrary position, you can read
>Berstein's arguments for his recommended way to install services at:
>http://cr.yp.to/unix.html
>
>  
>
Where, specificaly are his arguements against a configuration file 
methodology?

>  
>









Bruno Wolff III wrote:

On Thu, Feb 13, 2003 at 09:23:20 -0500,
  mlw <pgsql@mohawksoft.com> wrote:
  

Personally, however, I think the configuration issue is a no-brainer and 
I am amazed that people are balking. EVERY other service on a UNIX box 
is configured in this way, why not do it this way in PostgreSQL? The 
patch I submitted allowed the configuration to work as it currently 
does, but allowed for the more standard configuration file methodology.
    


If you are interested in reading a contrary position, you can read
Berstein's arguments for his recommended way to install services at:
http://cr.yp.to/unix.html

  

Where, specificaly are his arguements against a configuration file methodology?
Thu, 13 Feb 2003 11:53:26 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
Stephan Szabo wrote:

>On Thu, 13 Feb 2003, mlw wrote:
>
>  
>
>>Robert Treat wrote:
>>
>>    
>>
>>>On Thu, 2003-02-13 at 09:23, mlw wrote:
>>>
>>>
>>>      
>>>
>>>>I deal with a number of PG databases on a number of sites, and it is a
>>>>real pain in the ass to get to a PG box and hunt around for data
>>>>directory so as to be able to administer the system. What's really
>>>>annoying is when you have to find the data directory when someone else
>>>>set up the system.
>>>>        
>>>>
>
>You realize that the actual code feature doesn't necessarily help this
>case, right? Putting configuration in /etc and having a configuration file
>option on the command line are separate concepts.
>
>I think the feature is worthwhile, but I have some initial condition
>functionality questions that may have been answered in the previous patch,
>but I don't remember at this point.
>
>Mostly these have to deal with initial creation.  Does the user specify an
>output location to initdb, do they just specify a data dir as now where
>the configuration goes but then they need to move it somewhere, does
>initdb now do nothing relating to configuration file and the user should
>make one on his own.  Related, is the admin expected to have already made
>(say) /etc/postgresql to stick the config in and set the permissions
>correctly (since initdb doesn't run as root)?
>
My patch only works on the PostgreSQL server code. No changes have been 
made to the initialization scripts.

The patch declares three extra configuration file parameters:
hbafile= '/etc/postgres/pg_hba.conf'
identfile='/etc/postgres/pg_ident.conf'
datadir='/RAID0/postgres'

The command line option is a capital 'C,' as in:
postmaster -C /etc/postgresql.conf

I have no problem leaving the default configuration files remaining in 
the data directory as sort of a maintenance / boot strap sort of thing, 
so I don't see any reason to alter the installation.


As for this feature helping or not, I think it will. I think it 
accomplishes two things:
(1) Separates configuration from data.
(2) Allows an administrator to create a convention across multiple 
systems regardless of the location and mount points of the database storage.
(3) Lastly, it is a familiar methodology to DBAs not familiar with 
PostgreSQL.

Again, I don't see a valid reason for not including the patch. Yes, if 
you don't want to configure PostgreSQL that way, then so be it, but why 
not add the functionality for those who do?

I can envision the configuration file methodology of managing a database 
becoming the "preferred" approach over time as it is a more familiar and 
standard way of configuring servers on UNIX.


>  
>






Stephan Szabo wrote:

On Thu, 13 Feb 2003, mlw wrote:

  


Robert Treat wrote:

    

On Thu, 2003-02-13 at 09:23, mlw wrote:


      

I deal with a number of PG databases on a number of sites, and it is a
real pain in the ass to get to a PG box and hunt around for data
directory so as to be able to administer the system. What's really
annoying is when you have to find the data directory when someone else
set up the system.
        




You realize that the actual code feature doesn't necessarily help this
case, right? Putting configuration in /etc and having a configuration file
option on the command line are separate concepts.

I think the feature is worthwhile, but I have some initial condition
functionality questions that may have been answered in the previous patch,
but I don't remember at this point.

Mostly these have to deal with initial creation.  Does the user specify an
output location to initdb, do they just specify a data dir as now where
the configuration goes but then they need to move it somewhere, does
initdb now do nothing relating to configuration file and the user should
make one on his own.  Related, is the admin expected to have already made
(say) /etc/postgresql to stick the config in and set the permissions
correctly (since initdb doesn't run as root)?

My patch only works on the PostgreSQL server code. No changes have been made
to the initialization scripts.

The patch declares three extra configuration file parameters:
hbafile= '/etc/postgres/pg_hba.conf'
identfile='/etc/postgres/pg_ident.conf'
datadir='/RAID0/postgres'

The command line option is a capital 'C,' as in:
postmaster -C /etc/postgresql.conf

I have no problem leaving the default configuration files remaining in the
data directory as sort of a maintenance / boot strap sort of thing, so I
don't see any reason to alter the installation.


As for this feature helping or not, I think it will. I think it accomplishes
two things:
(1) Separates configuration from data.
(2) Allows an administrator to create a convention across multiple systems
regardless of the location and mount points of the database storage.
(3) Lastly, it is a familiar methodology to DBAs not familiar with PostgreSQL.

Again, I don't see a valid reason for not including the patch. Yes, if you
don't want to configure PostgreSQL that way, then so be it, but why not add
the functionality for those who do?

I can envision the configuration file methodology of managing a database
becoming the "preferred" approach over time as it is a more familiar and
standard way of configuring servers on UNIX.
Thu, 13 Feb 2003 12:13:00 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
On Thu, 2003-02-13 at 13:32, Christopher Browne wrote:
> > Everybody has room in /etc for another 10K of data. Where you have
> > room for something that might potentially be a half terrabyte of
> > data, and is not infrequently several gigabytes or more, is pretty
> > system-depenendent.
> 
> Ah, but this has two notable problems:
> 
> 1.  It assumes that there is "a location" for "the configuration files
>     for /the single database instance./"
> 
>     If I have a second database instance, that may conflict.

I think that moving configuration to [/usr/local]/etc/postgresql implies
the need for sub-directories by port, possibly with a default config to
be used if there is no port-specific config file.

> 2.  It assumes I have write access to /etc
> 
>     If I'm a Plain Old User, as opposed to root, I may only have
>     read-only access to /etc.

The location should be configurable; I hope we're talking about the
default here.  For distributions it should be /etc/postgresql; for local
builds it should be /usr/local/etc/postgresql, assuming you have root
access.  If you don't, the -c configfile switch suggested elsewhere in
this debate would be needed.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "The earth is the LORD'S, and the fullness thereof; the
      world, and they that dwell therein."            
                                   Psalms 24:1 


13 Feb 2003 17:28:14 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files
On Thu, 2003-02-13 at 12:00, Vince Vielhaber wrote:
> > Which means if the the vendor installed Postgresql (say, the
> > Red Hat Database) you'd expect config files to be in /etc.
> > If the postgresql is compiled from source by local admin,
> > you might look somewhere in /usr/local.
> 
> Then why not ~postgres/etc ??  Or substitute ~postgres with the
> db admin user you (or the distro) decided on at installation time.
> Gives a common location no matter who installed it or where it was
> installed.

Because it doesn't comply with FHS.  All projects should remember that
they coexist with many others and should do their best to stick to
common standards.

The default config file location should be set as a parameter to
./configure, which should default to /usr/local/etc/postgresql.  Those
of us who build for distributions will change it to /etc/postgresql.

I suppose if we want to run different postmasters simultaneously, we
could have /etc/postgresql/5432/ and so on for each port number being
used.  Perhaps have a default set in /etc/postgresql/ which can be used
if there is no port-specific directory, but a postmaster using those
defaults would have to have PGDATA specified on the command line.



On the same lines, I have just had a request (as Debian maintainer) to
move the location of postmaster.pid to the /var/run hierarchy; firstly,
so as to make it easier for the administrator to find, and secondly so
as to make it easier to configure SE Linux policy for file access.  (SE
Linux is the highly secure version produced by the NSA.)

I'm not entirely sure why SE Linux has a problem, seeing that postgres
needs read-write access to all the files in $PGDATA, but assuming the
need is verified, I could do this by moving the pid file from
$PGDATA/postmaster.pid to /var/run/postgresql/5432.pid and similarly for
other ports.  This would also have the benefit of being more FHS
compliant  What do people think about that?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "The earth is the LORD'S, and the fullness thereof; the
      world, and they that dwell therein."            
                                   Psalms 24:1 


13 Feb 2003 17:28:35 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files
On 13 Feb 2003, Oliver Elphick wrote:

> On Thu, 2003-02-13 at 12:00, Vince Vielhaber wrote:
> > > Which means if the the vendor installed Postgresql (say, the
> > > Red Hat Database) you'd expect config files to be in /etc.
> > > If the postgresql is compiled from source by local admin,
> > > you might look somewhere in /usr/local.
> >
> > Then why not ~postgres/etc ??  Or substitute ~postgres with the
> > db admin user you (or the distro) decided on at installation time.
> > Gives a common location no matter who installed it or where it was
> > installed.
>
> Because it doesn't comply with FHS.  All projects should remember that
> they coexist with many others and should do their best to stick to
> common standards.
>
> The default config file location should be set as a parameter to
> ./configure, which should default to /usr/local/etc/postgresql.  Those
> of us who build for distributions will change it to /etc/postgresql.

Seems to me that if FHS allows such a mess, it's reason enough to avoid
compliance.  Either that or those of you who build for distributions are
making an ill advised change.  Simply because the distribution makes the
decision to add PostgreSQL, or some other package, to it's distribution
doesn't make it a requirement to change the location of the config files.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Thu, 13 Feb 2003 12:52:23 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: location of the configuration files
On Thu, 2003-02-13 at 17:52, Vince Vielhaber wrote:
> Seems to me that if FHS allows such a mess, it's reason enough to avoid
> compliance.  Either that or those of you who build for distributions are
> making an ill advised change.  Simply because the distribution makes the
> decision to add PostgreSQL, or some other package, to it's distribution
> doesn't make it a requirement to change the location of the config files.

Debian (and FHS) specifically requires that.  All configuration files
MUST be under /etc; the reason is to make the system administrator's job
easier.  Part of the raison d'etre of a distribution is to rationalise
the idiosyncrasies of individual projects.  The locations used by
locally-built packages are up to the local administrator, but they
really should not be in /etc and are recommended to be under /usr/local.

I really don't see why there is such a not-invented-here mentality about
this issue.  I say again, standards-compliance is the best way.  It
makes life easier for everyone if standards are followed.  Don't we
pride ourselves on being closer to the SQL spec than other databases? 
Any way, if PostgreSQL stays as it is, I will continue to have to ensure
that initdb creates symlinks to /etc/postgresql/, as happens now.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "The earth is the LORD'S, and the fullness thereof; the
      world, and they that dwell therein."            
                                   Psalms 24:1 


13 Feb 2003 18:18:31 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files
Oliver Elphick wrote:
> On Thu, 2003-02-13 at 17:52, Vince Vielhaber wrote:
> > Seems to me that if FHS allows such a mess, it's reason enough to avoid
> > compliance.  Either that or those of you who build for distributions are
> > making an ill advised change.  Simply because the distribution makes the
> > decision to add PostgreSQL, or some other package, to it's distribution
> > doesn't make it a requirement to change the location of the config files.
> 
> Debian (and FHS) specifically requires that.  All configuration files
> MUST be under /etc; the reason is to make the system administrator's job
> easier.  Part of the raison d'etre of a distribution is to rationalise
> the idiosyncrasies of individual projects.  The locations used by
> locally-built packages are up to the local administrator, but they
> really should not be in /etc and are recommended to be under /usr/local.
> 
> I really don't see why there is such a not-invented-here mentality about
> this issue.  I say again, standards-compliance is the best way.  It
> makes life easier for everyone if standards are followed.  Don't we
> pride ourselves on being closer to the SQL spec than other databases? 
> Any way, if PostgreSQL stays as it is, I will continue to have to ensure
> that initdb creates symlinks to /etc/postgresql/, as happens now.

It doesn't have anything to do with "not-invented-here", which is a
common refrain by people who don't like our decisions, like "Why don't
you use mmap()?  Oh, it's because I thought of it and you didn't".  Does
anyone seriously believe that is the motiviation of anyone in this
project!  I certainly don't.

Now, on to this configuration discussion.  Seems moving the config file
out of $PGDATA requies either:
	
	1) we specifiy both the config directory and the data directory on
	postmaster start
	
	2) we specify the pgdata directory inside postgresql.conf or
	other config file

Is this accurate?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 13 Feb 2003 13:45:27 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Bruce Momjian wrote:

>Oliver Elphick wrote:
>  
>
>>On Thu, 2003-02-13 at 17:52, Vince Vielhaber wrote:
>>    
>>
>>>Seems to me that if FHS allows such a mess, it's reason enough to avoid
>>>compliance.  Either that or those of you who build for distributions are
>>>making an ill advised change.  Simply because the distribution makes the
>>>decision to add PostgreSQL, or some other package, to it's distribution
>>>doesn't make it a requirement to change the location of the config files.
>>>      
>>>
>>Debian (and FHS) specifically requires that.  All configuration files
>>MUST be under /etc; the reason is to make the system administrator's job
>>easier.  Part of the raison d'etre of a distribution is to rationalise
>>the idiosyncrasies of individual projects.  The locations used by
>>locally-built packages are up to the local administrator, but they
>>really should not be in /etc and are recommended to be under /usr/local.
>>
>>I really don't see why there is such a not-invented-here mentality about
>>this issue.  I say again, standards-compliance is the best way.  It
>>makes life easier for everyone if standards are followed.  Don't we
>>pride ourselves on being closer to the SQL spec than other databases? 
>>Any way, if PostgreSQL stays as it is, I will continue to have to ensure
>>that initdb creates symlinks to /etc/postgresql/, as happens now.
>>    
>>
>
>It doesn't have anything to do with "not-invented-here", which is a
>common refrain by people who don't like our decisions, like "Why don't
>you use mmap()?  Oh, it's because I thought of it and you didn't".  Does
>anyone seriously believe that is the motiviation of anyone in this
>project!  I certainly don't.
>
>Now, on to this configuration discussion.  Seems moving the config file
>out of $PGDATA requies either:
>	
>	1) we specifiy both the config directory and the data directory on
>	postmaster start
>	
>	2) we specify the pgdata directory inside postgresql.conf or
>	other config file
>
>Is this accurate?
>  
>
The patch that I have adds three settings to postgresql.conf and one 
command line parameter.

hba_conf = 'filename'
ident_conf='filename'
data_dir='path'

The command linae parameter is  -C, used as:

postmaster -C /usr/local/etc/postgresql.conf

I think this will help administrators.

Bruce, can you shed some light as to why this is being so strongly 
rejected. I just don't see any downside. I just don't get it.

I will be resubmitting my patch for the 7.3.2 tree.


>  
>









Bruce Momjian wrote:

Oliver Elphick wrote:
  

On Thu, 2003-02-13 at 17:52, Vince Vielhaber wrote:
    

Seems to me that if FHS allows such a mess, it's reason enough to avoid
compliance.  Either that or those of you who build for distributions are
making an ill advised change.  Simply because the distribution makes the
decision to add PostgreSQL, or some other package, to it's distribution
doesn't make it a requirement to change the location of the config files.
      

Debian (and FHS) specifically requires that.  All configuration files
MUST be under /etc; the reason is to make the system administrator's job
easier.  Part of the raison d'etre of a distribution is to rationalise
the idiosyncrasies of individual projects.  The locations used by
locally-built packages are up to the local administrator, but they
really should not be in /etc and are recommended to be under /usr/local.

I really don't see why there is such a not-invented-here mentality about
this issue.  I say again, standards-compliance is the best way.  It
makes life easier for everyone if standards are followed.  Don't we
pride ourselves on being closer to the SQL spec than other databases? 
Any way, if PostgreSQL stays as it is, I will continue to have to ensure
that initdb creates symlinks to /etc/postgresql/, as happens now.
    


It doesn't have anything to do with "not-invented-here", which is a
common refrain by people who don't like our decisions, like "Why don't
you use mmap()?  Oh, it's because I thought of it and you didn't".  Does
anyone seriously believe that is the motiviation of anyone in this
project!  I certainly don't.

Now, on to this configuration discussion.  Seems moving the config file
out of $PGDATA requies either:
	
	1) we specifiy both the config directory and the data directory on
	postmaster start
	
	2) we specify the pgdata directory inside postgresql.conf or
	other config file

Is this accurate?
  

The patch that I have adds three settings to postgresql.conf and one command
line parameter.

hba_conf = 'filename'
ident_conf='filename'
data_dir='path'

The command linae parameter is  -C, used as:

postmaster -C /usr/local/etc/postgresql.conf

I think this will help administrators. 

Bruce, can you shed some light as to why this is being so strongly rejected.
I just don't see any downside. I just don't get it. 

I will be resubmitting my patch for the 7.3.2 tree.
Thu, 13 Feb 2003 14:06:23 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
On Thu, 2003-02-13 at 18:45, Bruce Momjian wrote:
> Now, on to this configuration discussion.  Seems moving the config file
> out of $PGDATA requies either:
> 	
> 	1) we specifiy both the config directory and the data directory on
> 	postmaster start
> 	
> 	2) we specify the pgdata directory inside postgresql.conf or
> 	other config file
> 
> Is this accurate?

The default start would read the config file from its predefined
location, set by ./configure.  No command line options would be
necessary for the postmaster to run, though they could be provided.

The config file should contain the pgdata location; this and any other
parameter should be overridden if a different location is specified by a
command-line option. I think the config should be able to contain all
information that can be specified on the command line (except, of
course, the location of the configuration file.)

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "The earth is the LORD'S, and the fullness thereof; the
      world, and they that dwell therein."            
                                   Psalms 24:1 


13 Feb 2003 19:13:14 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files
On Thu, 2003-02-13 at 18:45, Bruce Momjian wrote:
> Oliver Elphick wrote:
> > On Thu, 2003-02-13 at 17:52, Vince Vielhaber wrote:
> > > Seems to me that if FHS allows such a mess, it's reason enough to avoid
> > > compliance.  Either that or those of you who build for distributions are
> > > making an ill advised change.  Simply because the distribution makes the
> > > decision to add PostgreSQL, or some other package, to it's distribution
> > > doesn't make it a requirement to change the location of the config files.
> > ...
> > I really don't see why there is such a not-invented-here mentality about
> > this issue.  I say again, standards-compliance is the best way.  It
> > makes life easier for everyone if standards are followed.  Don't we
> > pride ourselves on being closer to the SQL spec than other databases? 
> > Any way, if PostgreSQL stays as it is, I will continue to have to ensure
> > that initdb creates symlinks to /etc/postgresql/, as happens now.
> 
> It doesn't have anything to do with "not-invented-here", which is a
> common refrain by people who don't like our decisions, like "Why don't
> you use mmap()?  Oh, it's because I thought of it and you didn't".  Does
> anyone seriously believe that is the motiviation of anyone in this
> project!  I certainly don't.

My apologies.  I withdraw the comment, which was provoked mostly by
Vince's response, quoted above.  I agree that it is not characteristic
of the project.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "The earth is the LORD'S, and the fullness thereof; the
      world, and they that dwell therein."            
                                   Psalms 24:1 


13 Feb 2003 19:15:36 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files
mlw wrote:
> >It doesn't have anything to do with "not-invented-here", which is a
> >common refrain by people who don't like our decisions, like "Why don't
> >you use mmap()?  Oh, it's because I thought of it and you didn't".  Does
> >anyone seriously believe that is the motiviation of anyone in this
> >project!  I certainly don't.
> >
> >Now, on to this configuration discussion.  Seems moving the config file
> >out of $PGDATA requies either:
> >	
> >	1) we specifiy both the config directory and the data directory on
> >	postmaster start
> >	
> >	2) we specify the pgdata directory inside postgresql.conf or
> >	other config file
> >
> >Is this accurate?
> >  
> >
> The patch that I have adds three settings to postgresql.conf and one 
> command line parameter.
> 
> hba_conf = 'filename'
> ident_conf='filename'
> data_dir='path'
> 
> The command linae parameter is  -C, used as:
> 
> postmaster -C /usr/local/etc/postgresql.conf
> 
> I think this will help administrators.
> 
> Bruce, can you shed some light as to why this is being so strongly 
> rejected. I just don't see any downside. I just don't get it.
> 
> I will be resubmitting my patch for the 7.3.2 tree.

Well, in a sense, it trades passing one parameter, PGDATA, for another. 
I see your point that we should specify configuration first, and let
everything pass from there.  However, it does add extra configuration
parameters, and because you still need to specify/create pgdata, it adds
an extra level of abstraction to setting up the server.

Also, there is nothing preventing someone from symlinking the
configuration files from pgdata to somewhere else.

I don't think separate params for each config file is good.  At the
most, I think we will specify the configuration _directory_ for all the
config files, perhaps pgsql/etc, and have pgdata default to ../data, or
honor $PGDATA.  That might be the cleanest.

Of course, that now gives us $PGCONFIG and $PGDATA, and possible
intraction if postgresql.conf specifies a different pgdata from $PGDATA.
As you can see, it could get messy.

And, if you specify pgdata in postgresql.conf, it prevents you from
using that file by different postmasters.

My best guess would be to not specify pgdata in postgresql.conf, and
have a new $PGCONFIG param to specify the configuration directory, but
if we do that, $PGDATA/postgresql.conf becomes meaningless, which could
also be confusing.  Maybe we don't allow those files to exist in $PGDATA
if $PGCONFIG is used, _and_ $PGCONFIG is not the same as $PGDATA.  See,
I am getting myself confused.  :-)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 13 Feb 2003 14:17:51 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
On Thu, 2003-02-13 at 14:06, mlw wrote:
> 
> I will be resubmitting my patch for the 7.3.2 tree.
> 

I'm no core developer, but surely this wont be included in the 7.3.x
branch. Any change needs to be made against CVS head.

Robert Treat



13 Feb 2003 14:23:38 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: location of the configuration files
Robert Treat wrote:
> On Thu, 2003-02-13 at 14:06, mlw wrote:
> > 
> > I will be resubmitting my patch for the 7.3.2 tree.
> > 
> 
> I'm no core developer, but surely this wont be included in the 7.3.x
> branch. Any change needs to be made against CVS head.

I assume he meant he will repost his 7.3.2-based patch and we will merge
it into CVS HEAD if it is accepted.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 13 Feb 2003 14:28:48 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
On Thu, 2003-02-13 at 12:13, mlw wrote:
> 
> My patch only works on the PostgreSQL server code. No changes have been
> made to the initialization scripts.
> 
> The patch declares three extra configuration file parameters:
> hbafile= '/etc/postgres/pg_hba.conf'
> identfile='/etc/postgres/pg_ident.conf'
> datadir='/RAID0/postgres'
> 

If we're going to do this, I think we need to account for all of the
files in the directory including PG_VERSION, postmaster.opts,
postmaster.pid. In the end if we can't build so that we are either fully
FHS compliant and/or LSB compliant, we've not done enough work on it.

Robert Treat



13 Feb 2003 14:30:43 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: location of the configuration files
Robert Treat wrote:
> On Thu, 2003-02-13 at 12:13, mlw wrote:
> > 
> > My patch only works on the PostgreSQL server code. No changes have been
> > made to the initialization scripts.
> > 
> > The patch declares three extra configuration file parameters:
> > hbafile= '/etc/postgres/pg_hba.conf'
> > identfile='/etc/postgres/pg_ident.conf'
> > datadir='/RAID0/postgres'
> > 
> 
> If we're going to do this, I think we need to account for all of the
> files in the directory including PG_VERSION, postmaster.opts,
> postmaster.pid. In the end if we can't build so that we are either fully
> FHS compliant and/or LSB compliant, we've not done enough work on it.

Woh, how do we move some of those files into /etc or /var/run if we
aren't running as root?  We certainly don't want to require that.  I
guess /etc/postgresql will work if that directory is owned by the
PostgreSQL superuser, but /var/run will be a problem.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 13 Feb 2003 14:43:27 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
On Thu, 2003-02-13 at 19:30, Robert Treat wrote:
> If we're going to do this, I think we need to account for all of the
> files in the directory including PG_VERSION, postmaster.opts,

Not PG_VERSION; that is intimately associated with the data itself and
ought to stay in the data directory.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "The earth is the LORD'S, and the fullness thereof; the
      world, and they that dwell therein."            
                                   Psalms 24:1 


13 Feb 2003 19:47:12 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files


Bruce Momjian wrote:

>Well, in a sense, it trades passing one parameter, PGDATA, for another. 
>I see your point that we should specify configuration first, and let
>everything pass from there.  However, it does add extra configuration
>parameters, and because you still need to specify/create pgdata, it adds
>an extra level of abstraction to setting up the server.
>
While this is true, it is not uncommon, and it is more or less expected 
by most UNIX admins.

>
>Also, there is nothing preventing someone from symlinking the
>configuration files from pgdata to somewhere else.
>
Stop!!!  symlinks are not sufficient. When happens when a native Win32 
version comes out? there are no symlinks. Also, most of the admins I 
know don't like to use simlinks as they are not self documenting. 
Symlinks are "bad."

>
>I don't think separate params for each config file is good.  At the
>most, I think we will specify the configuration _directory_ for all the
>config files, perhaps pgsql/etc, and have pgdata default to ../data, or
>honor $PGDATA.  That might be the cleanest.
>
The problem with that is that you are back to symlinking shared files. 
Symlinks are a kludge.

>
>Of course, that now gives us $PGCONFIG and $PGDATA, and possible
>intraction if postgresql.conf specifies a different pgdata from $PGDATA.
>As you can see, it could get messy.
>
I don't see it as very messy, for instance:

postmaster -C /etc/postgres/postgresql.conf -D /RAID0/postgres -p 5432
postmaster -C /etc/postgres/postgresql.conf -D /RAID1/postgres -p 5433

That looks like a real clean way to run multiple PostgreSQL servers on 
the same box using the same configuration files.

>And, if you specify pgdata in postgresql.conf, it prevents you from
>using that file by different postmasters.
>
Not true, command line parameters, as a rule, override configuration 
file defaults.

>
>My best guess would be to not specify pgdata in postgresql.conf, and
>have a new $PGCONFIG param to specify the configuration directory, but
>if we do that, $PGDATA/postgresql.conf becomes meaningless, which could
>also be confusing.  Maybe we don't allow those files to exist in $PGDATA
>if $PGCONFIG is used, _and_ $PGCONFIG is not the same as $PGDATA.  See,
>I am getting myself confused.  :-)
>
I think you are making it too complicated.

I wouldn't remove the default configration set, it would be useful as a 
failsafe or maintainence feature.



Thu, 13 Feb 2003 14:48:28 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files

On Thu, 13 Feb 2003, mlw wrote:

> Stephan Szabo wrote:
> >>>On Thu, 2003-02-13 at 09:23, mlw wrote:
> >>>>I deal with a number of PG databases on a number of sites, and it is a
> >>>>real pain in the ass to get to a PG box and hunt around for data
> >>>>directory so as to be able to administer the system. What's really
> >>>>annoying is when you have to find the data directory when someone else
> >>>>set up the system.
> >>>>
> >>>>
> >
> >You realize that the actual code feature doesn't necessarily help this
> >case, right? Putting configuration in /etc and having a configuration file
> >option on the command line are separate concepts.

Re-read my statement and yours about the case you were mentioning. ;)
Sure, putting the files in /etc lets you find them easily.  However, if
you're doing things like finding configuration made by someone else and
said configuration isn't in /etc (which if they wanted to they could do
now with symlinks I believe - yes symlinks aren't a complete solution, but
I think they're reasonable on most of our current ports) then you still
have to search the system for the configuration file, except now it might
not even be postgresql.conf. That's why I said the two issues aren't the
same.

> >I think the feature is worthwhile, but I have some initial condition
> >functionality questions that may have been answered in the previous patch,
> >but I don't remember at this point.
> >
> >Mostly these have to deal with initial creation.  Does the user specify an
> >output location to initdb, do they just specify a data dir as now where
> >the configuration goes but then they need to move it somewhere, does
> >initdb now do nothing relating to configuration file and the user should
> >make one on his own.  Related, is the admin expected to have already made
> >(say) /etc/postgresql to stick the config in and set the permissions
> >correctly (since initdb doesn't run as root)?
> >
> My patch only works on the PostgreSQL server code. No changes have been
> made to the initialization scripts.
>
> The patch declares three extra configuration file parameters:
> hbafile= '/etc/postgres/pg_hba.conf'
> identfile='/etc/postgres/pg_ident.conf'
> datadir='/RAID0/postgres'
>
> The command line option is a capital 'C,' as in:
> postmaster -C /etc/postgresql.conf
>
> I have no problem leaving the default configuration files remaining in
> the data directory as sort of a maintenance / boot strap sort of thing,
> so I don't see any reason to alter the installation.
>
>
> As for this feature helping or not, I think it will. I think it
> accomplishes two things:
> (1) Separates configuration from data.
> (2) Allows an administrator to create a convention across multiple
> systems regardless of the location and mount points of the database storage.
> (3) Lastly, it is a familiar methodology to DBAs not familiar with
> PostgreSQL.

I agree on all these points ("I think the feature is worthwhile, but...").
I just wonder if we were going to do this, we might as well look at all of
the various things people want and decide what we want to do, for example,
people commenting on default configuration locations through configure,
how does this interact with what we have now, etc.  I'd rather have a
month spent arguing out a behavior rather than just adding a new behavior
that we'll need to possibly revisit again in the future. :)


Thu, 13 Feb 2003 11:50:59 -0800 (PST)
Stephan Szabo <sszabo@megazone23.bigpanda.com>
Re: location of the configuration files
Robert Treat wrote:

>On Thu, 2003-02-13 at 12:13, mlw wrote:
>  
>
>>My patch only works on the PostgreSQL server code. No changes have been
>>made to the initialization scripts.
>>
>>The patch declares three extra configuration file parameters:
>>hbafile= '/etc/postgres/pg_hba.conf'
>>identfile='/etc/postgres/pg_ident.conf'
>>datadir='/RAID0/postgres'
>>
>>    
>>
>
>If we're going to do this, I think we need to account for all of the
>files in the directory including PG_VERSION, postmaster.opts,
>postmaster.pid. In the end if we can't build so that we are either fully
>FHS compliant and/or LSB compliant, we've not done enough work on it.
>
>Robert Treat
>
>
>  
>
postmaster.opts, PG_VERSION, and postmaster.pid are not configuration 
parameters.

PG_VERSION is VERY important, it is how you know the version of the 
database.
Postmaster.pid is a postgres writable value
AFAIK, postmaster.opts is also a postgres writable value.








Robert Treat wrote:

On Thu, 2003-02-13 at 12:13, mlw wrote:
  

My patch only works on the PostgreSQL server code. No changes have been
made to the initialization scripts.

The patch declares three extra configuration file parameters:
hbafile= '/etc/postgres/pg_hba.conf'
identfile='/etc/postgres/pg_ident.conf'
datadir='/RAID0/postgres'

    


If we're going to do this, I think we need to account for all of the
files in the directory including PG_VERSION, postmaster.opts,
postmaster.pid. In the end if we can't build so that we are either fully
FHS compliant and/or LSB compliant, we've not done enough work on it.

Robert Treat


  

postmaster.opts, PG_VERSION, and postmaster.pid are not configuration parameters.


PG_VERSION is VERY important, it is how you know the version of the database.
Postmaster.pid is a postgres writable value 
AFAIK, postmaster.opts is also a postgres writable value.
Thu, 13 Feb 2003 14:51:27 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
Bruce Momjian wrote:

>Robert Treat wrote:
>  
>
>>On Thu, 2003-02-13 at 12:13, mlw wrote:
>>    
>>
>>>My patch only works on the PostgreSQL server code. No changes have been
>>>made to the initialization scripts.
>>>
>>>The patch declares three extra configuration file parameters:
>>>hbafile= '/etc/postgres/pg_hba.conf'
>>>identfile='/etc/postgres/pg_ident.conf'
>>>datadir='/RAID0/postgres'
>>>
>>>      
>>>
>>If we're going to do this, I think we need to account for all of the
>>files in the directory including PG_VERSION, postmaster.opts,
>>postmaster.pid. In the end if we can't build so that we are either fully
>>FHS compliant and/or LSB compliant, we've not done enough work on it.
>>    
>>
>
>Woh, how do we move some of those files into /etc or /var/run if we
>aren't running as root?  We certainly don't want to require that.  I
>guess /etc/postgresql will work if that directory is owned by the
>PostgreSQL superuser, but /var/run will be a problem.
>
>  
>
I don't think those files need to move. As I said in another post, they 
are postgres writable and should in the PostgreSQL data directory. 
However, I suppose, that those also could be configuration parameters? No?

PG_VERSION obviously should not move.
postmaster.opts gets created when postmaster is run, correct?

The only issue would be the PID file, and I don't have strong feelings 
about it, except that using a /var/run system will make running multiple 
postmasters a pain.










Bruce Momjian wrote:

Robert Treat wrote:
  

On Thu, 2003-02-13 at 12:13, mlw wrote:
    

My patch only works on the PostgreSQL server code. No changes have been
made to the initialization scripts.

The patch declares three extra configuration file parameters:
hbafile= '/etc/postgres/pg_hba.conf'
identfile='/etc/postgres/pg_ident.conf'
datadir='/RAID0/postgres'

      

If we're going to do this, I think we need to account for all of the
files in the directory including PG_VERSION, postmaster.opts,
postmaster.pid. In the end if we can't build so that we are either fully
FHS compliant and/or LSB compliant, we've not done enough work on it.
    


Woh, how do we move some of those files into /etc or /var/run if we
aren't running as root?  We certainly don't want to require that.  I
guess /etc/postgresql will work if that directory is owned by the
PostgreSQL superuser, but /var/run will be a problem.

  

I don't think those files need to move. As I said in another post, they are 
postgres writable and should in the PostgreSQL data directory. However, I 
suppose, that those also could be configuration parameters? No?

 PG_VERSION obviously should not move.
 postmaster.opts gets created when postmaster is run, correct?

 The only issue would be the PID file, and I don't have strong feelings about
it, except that using a /var/run system will make running multiple postmasters
a pain.
Thu, 13 Feb 2003 14:58:26 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
Stephan Szabo wrote:

>On Thu, 13 Feb 2003, mlw wrote:
>
>  
>
>>Stephan Szabo wrote:
>>    
>>
>>>>>On Thu, 2003-02-13 at 09:23, mlw wrote:
>>>>>          
>>>>>
>>>>>>I deal with a number of PG databases on a number of sites, and it is a
>>>>>>real pain in the ass to get to a PG box and hunt around for data
>>>>>>directory so as to be able to administer the system. What's really
>>>>>>annoying is when you have to find the data directory when someone else
>>>>>>set up the system.
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>You realize that the actual code feature doesn't necessarily help this
>>>case, right? Putting configuration in /etc and having a configuration file
>>>option on the command line are separate concepts.
>>>      
>>>
>
>Re-read my statement and yours about the case you were mentioning. ;)
>Sure, putting the files in /etc lets you find them easily.  However, if
>you're doing things like finding configuration made by someone else and
>said configuration isn't in /etc (which if they wanted to they could do
>now with symlinks I believe - yes symlinks aren't a complete solution, but
>I think they're reasonable on most of our current ports) then you still
>have to search the system for the configuration file, except now it might
>not even be postgresql.conf. That's why I said the two issues aren't the
>same.
>
>  
>
>>>I think the feature is worthwhile, but I have some initial condition
>>>functionality questions that may have been answered in the previous patch,
>>>but I don't remember at this point.
>>>
>>>Mostly these have to deal with initial creation.  Does the user specify an
>>>output location to initdb, do they just specify a data dir as now where
>>>the configuration goes but then they need to move it somewhere, does
>>>initdb now do nothing relating to configuration file and the user should
>>>make one on his own.  Related, is the admin expected to have already made
>>>(say) /etc/postgresql to stick the config in and set the permissions
>>>correctly (since initdb doesn't run as root)?
>>>
>>>      
>>>
>>My patch only works on the PostgreSQL server code. No changes have been
>>made to the initialization scripts.
>>
>>The patch declares three extra configuration file parameters:
>>hbafile= '/etc/postgres/pg_hba.conf'
>>identfile='/etc/postgres/pg_ident.conf'
>>datadir='/RAID0/postgres'
>>
>>The command line option is a capital 'C,' as in:
>>postmaster -C /etc/postgresql.conf
>>
>>I have no problem leaving the default configuration files remaining in
>>the data directory as sort of a maintenance / boot strap sort of thing,
>>so I don't see any reason to alter the installation.
>>
>>
>>As for this feature helping or not, I think it will. I think it
>>accomplishes two things:
>>(1) Separates configuration from data.
>>(2) Allows an administrator to create a convention across multiple
>>systems regardless of the location and mount points of the database storage.
>>(3) Lastly, it is a familiar methodology to DBAs not familiar with
>>PostgreSQL.
>>    
>>
>
>I agree on all these points ("I think the feature is worthwhile, but...").
>I just wonder if we were going to do this, we might as well look at all of
>the various things people want and decide what we want to do, for example,
>people commenting on default configuration locations through configure,
>how does this interact with what we have now, etc.  I'd rather have a
>month spent arguing out a behavior rather than just adding a new behavior
>that we'll need to possibly revisit again in the future. :)
>

I have absolutely no problem debating and augmenting the feature. None 
what so ever, I am more pushing to get momentum to actually do it. In 
7.1 I proposed this, and was told that it wasn't needed because (a) 
symlinks provide all the functionality you need and (b) that they were 
going to redesign the configuration system. That was well over a year 
ago (two?). I am willing to do the work, but what's the point if the 
core group isn't even going to use it?

Most of the admins I know don't use symlinks as they can not carry 
comments. Without knowing,  you can change or delete a file that does 
not appear to be in use but which kills a working server. Symlinks are 
dangerous in production systems, it is easy to screw them up with scp 
when administering a cluster of computers.


>  
>








Stephan Szabo wrote:

On Thu, 13 Feb 2003, mlw wrote:

  

Stephan Szabo wrote:
    



On Thu, 2003-02-13 at 09:23, mlw wrote:
          

I deal with a number of PG databases on a number of sites, and it is a
real pain in the ass to get to a PG box and hunt around for data
directory so as to be able to administer the system. What's really
annoying is when you have to find the data directory when someone else
set up the system.


            



You realize that the actual code feature doesn't necessarily help this
case, right? Putting configuration in /etc and having a configuration file
option on the command line are separate concepts.
      



Re-read my statement and yours about the case you were mentioning. ;)
Sure, putting the files in /etc lets you find them easily.  However, if
you're doing things like finding configuration made by someone else and
said configuration isn't in /etc (which if they wanted to they could do
now with symlinks I believe - yes symlinks aren't a complete solution, but
I think they're reasonable on most of our current ports) then you still
have to search the system for the configuration file, except now it might
not even be postgresql.conf. That's why I said the two issues aren't the
same.

  


I think the feature is worthwhile, but I have some initial condition
functionality questions that may have been answered in the previous patch,
but I don't remember at this point.

Mostly these have to deal with initial creation.  Does the user specify an
output location to initdb, do they just specify a data dir as now where
the configuration goes but then they need to move it somewhere, does
initdb now do nothing relating to configuration file and the user should
make one on his own.  Related, is the admin expected to have already made
(say) /etc/postgresql to stick the config in and set the permissions
correctly (since initdb doesn't run as root)?

      

My patch only works on the PostgreSQL server code. No changes have been
made to the initialization scripts.

The patch declares three extra configuration file parameters:
hbafile= '/etc/postgres/pg_hba.conf'
identfile='/etc/postgres/pg_ident.conf'
datadir='/RAID0/postgres'

The command line option is a capital 'C,' as in:
postmaster -C /etc/postgresql.conf

I have no problem leaving the default configuration files remaining in
the data directory as sort of a maintenance / boot strap sort of thing,
so I don't see any reason to alter the installation.


As for this feature helping or not, I think it will. I think it
accomplishes two things:
(1) Separates configuration from data.
(2) Allows an administrator to create a convention across multiple
systems regardless of the location and mount points of the database storage.
(3) Lastly, it is a familiar methodology to DBAs not familiar with
PostgreSQL.
    


I agree on all these points ("I think the feature is worthwhile, but...").
I just wonder if we were going to do this, we might as well look at all of
the various things people want and decide what we want to do, for example,
people commenting on default configuration locations through configure,
how does this interact with what we have now, etc.  I'd rather have a
month spent arguing out a behavior rather than just adding a new behavior
that we'll need to possibly revisit again in the future. :)


I have absolutely no problem debating and augmenting the feature. None what
so ever, I am more pushing to get momentum to actually do it. In 7.1 I proposed
this, and was told that it wasn't needed because (a) symlinks provide all
the functionality you need and (b) that they were going to redesign the configuration
system. That was well over a year ago (two?). I am willing to do the work,
but what's the point if the core group isn't even going to use it?

Most of the admins I know don't use symlinks as they can not carry comments.
Without knowing,  you can change or delete a file that does not appear to
be in use but which kills a working server. Symlinks are dangerous in production
systems, it is easy to screw them up with scp when administering a cluster
of computers.
Thu, 13 Feb 2003 15:08:39 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
On 13 Feb 2003, Martin Coxall wrote:

> 
> > Well, to the extent that you're serious, you understand that 
> > a lot of people feel that /usr/local should be reserved for 
> > stuff that's installed by the local sysadmin, and your
> > vendor/distro isn't supposed to be messing with it. 
> > 
> > Which means if the the vendor installed Postgresql (say, the
> > Red Hat Database) you'd expect config files to be in /etc.
> > If the postgresql is compiled from source by local admin, 
> > you might look somewhere in /usr/local.
> 
> Indeed. For better or worse, there is a Filesystem Hierarcy Standard,
> and most of the important Linux distros, BSDs and some legacy Unixen
> stick to it, so so should we.
> 
> Configuration files should be in /etc/postgresql/, or at the very least
> symlinked from there.

So, how do we handle things like installing three or four versions at the 
same time.  This isn't the same thing as /etc/fstab.  While we only would 
likely need to have one fstab or whatever, with postgresql, it's not 
unreasonable to want to intall more than one copy or version for various 
reason.

Generally things that live in /etc are owned and operated by the OS.  
Postgresql, by it's definition is a userspace program, not an OS owned 
one.

I've found having a $PGDATA var where EVERYTHING lives to be a huge 
advantage when you need to run a half dozen instances of pgsql under 
different accounts or for different versions on the same box.

Now, if we could do it like X, where the base stuff is all in the 
/etc/X11R6 directory, but your own personal config lives in your home 
directory, then we're right as rain.  but what parts of postgresql would 
always be common to all flavors that might need to be run at the same 
time?  Not much.


Thu, 13 Feb 2003 13:11:13 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: location of the configuration files
I don't see why we can't keep everyone happy and let the users choose the 
setup they want. To wit, make the following, probably simple, changes:

1) Have postgresql default to using /etc/postgresql.conf
2) Add a setting in postgresql.conf specifying the data directory
3) Change the meaning of -D to mean "use this config file"
4) In the absence of a specified data directory in postgresql.conf, use the 
location of the postgresql.conf file as the data directory

I see several advantages:

1) Anyone who doesn't want to change doesn't have to - leaving the data 
directory spec out of postgresql.conf and starting with -D will be 
essentially identical to how things are now (except it would be -D 
/foo/bar/postgresql.conf instead of -D /foo/bar/ - even this could be 
overcome with a bit of bailing wire saying if -D specifies a directory, look 
for postgresql.conf in that directory).

2) Postgresql will be more "familiar" to those who expect or desire configs 
to be in /etc.

3) Adding a postgresql.conf line for data location sets the stage for being 
able to specify directories for all sorts of files (WAL, index, etc.) without 
the need for symlinks.

4) Multiple config files could be more easily managed for 
testing/benchmarking/etc.

Cheers,
Steve


On Wednesday 12 February 2003 10:14 pm, Peter Bierman wrote:
> At 12:31 AM -0500 2/13/03, mlw wrote:
> >The idea that a, more or less, arbitrary data location determines
> >the database configuration is wrong. It should be obvious to any
> >administrator that a configuration file location which controls the
> >server is the "right" way to do it.
>
> Isn't the database data itself a rather significant portion of the
> 'configuration' of the database?
>
> What do you gain by having the postmaster config and the database
> data live in different locations?
>
> -pmb
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

Thu, 13 Feb 2003 12:28:07 -0800
Steve Crawford <scrawford@pinpointresearch.com>
Re: location of the configuration files
On Thu, 13 Feb 2003, mlw wrote:

> 
> 
> Christopher Browne wrote:
> 
> >In the last exciting episode, cjs@cynic.net (Curt Sampson) wrote:
> >
> >>On Wed, 12 Feb 2003, Peter Bierman wrote:
> >>
> >>>What do you gain by having the postmaster config and the database
> >>>data live in different locations?
> >>>
> >>You can then standardize a location for the configuration files.
> >>
> >>Everybody has room in /etc for another 10K of data. Where you have
> >>room for something that might potentially be a half terrabyte of
> >>data, and is not infrequently several gigabytes or more, is pretty
> >>system-depenendent.
> >
> >Ah, but this has two notable problems:
> >
> >1.  It assumes that there is "a location" for "the configuration files
> >    for /the single database instance./"
> >
> >    If I have a second database instance, that may conflict.
> >
> >2.  It assumes I have write access to /etc
> >
> >    If I'm a Plain Old User, as opposed to root, I may only have
> >    read-only access to /etc.
> >
> >These conditions have both been known to occur...
> >  
> >
> These are not issues at all. You could put the configuration file 
> anywhere, just as you can for any UNIX service.
> 
> postmaster --config=/home/myhome/mydb.conf
> 
> I deal with a number of PG databases on a number of sites, and it is a 
> real pain in the ass to get to a PG box and hunt around for data 
> directory so as to be able to administer the system. What's really 
> annoying is when you have to find the data directory when someone else 
> set up the system.

Really?  I would think it's easier to do this:

su - pgsuper
cd $PGDATA
pwd

Than to try to figure out what someone entered when they ran ./configure 
--config=...

> Configuring postgresql via a configuration file which specifies all the 
> data, i.e. data directory, name of other configuration files, etc. is 
> the right way to do it. Even if you have reasons against it, even if you 
> think it is a bad idea, a bad standard is almost always a better 
> solution than an arcane work of perfection.

Wrong, I strongly disagree with this sentament.  Conformity to standards 
for simple conformity's sake is as wrong as sticking to the old way 
because it's what we're all comfy with.  

> Personally, however, I think the configuration issue is a no-brainer and 
> I am amazed that people are balking. EVERY other service on a UNIX box 
> is configured in this way, why not do it this way in PostgreSQL? The 
> patch I submitted allowed the configuration to work as it currently 
> does, but allowed for the more standard configuration file methodology.

If I do a .tar.gz install of apache, I get /usr/local/apache/conf, which 
is not the standard way you're listing.  If I install openldap from 
.tar.gz, I get a /usr/local/etc/openldap directory, close, but still not 
the same.  The fact is, it's the packagers that put things into /etc and 
whatnot, and I can see the postgresql RPMs or debs or whatever having that 
as the default, but for custom built software, NOTHING that I know of 
builds from source and uses /etc without a switch to tell it to, just like 
postgresql can do now.

> I just don't understand what the resistance is, it makes no sense.

I agree, but from the other side of the fence.


Thu, 13 Feb 2003 14:07:50 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: location of the configuration files
On 13 Feb 2003, Oliver Elphick wrote:

> On Thu, 2003-02-13 at 18:45, Bruce Momjian wrote:
> > Oliver Elphick wrote:
> > > On Thu, 2003-02-13 at 17:52, Vince Vielhaber wrote:
> > > > Seems to me that if FHS allows such a mess, it's reason enough to avoid
> > > > compliance.  Either that or those of you who build for distributions are
> > > > making an ill advised change.  Simply because the distribution makes the
> > > > decision to add PostgreSQL, or some other package, to it's distribution
> > > > doesn't make it a requirement to change the location of the config files.
> > > ...
> > > I really don't see why there is such a not-invented-here mentality about
> > > this issue.  I say again, standards-compliance is the best way.  It
> > > makes life easier for everyone if standards are followed.  Don't we
> > > pride ourselves on being closer to the SQL spec than other databases?
> > > Any way, if PostgreSQL stays as it is, I will continue to have to ensure
> > > that initdb creates symlinks to /etc/postgresql/, as happens now.
> >
> > It doesn't have anything to do with "not-invented-here", which is a
> > common refrain by people who don't like our decisions, like "Why don't
> > you use mmap()?  Oh, it's because I thought of it and you didn't".  Does
> > anyone seriously believe that is the motiviation of anyone in this
> > project!  I certainly don't.
>
> My apologies.  I withdraw the comment, which was provoked mostly by
> Vince's response, quoted above.  I agree that it is not characteristic
> of the project.

I certainly wasn't trying to provoke anything.  It just seems odd to me
that when the distribution installs a package and places it's config files
in /etc and later the admin happens to upgrade by the instructions with
the package, it's acceptable for the config files to now be in two places
and you don't find it confusing.  What happens when a new admin comes on
and tries to figure out which config file is which?   Ever try to figure
out where the hell Pine's config really is?

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Thu, 13 Feb 2003 16:21:48 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: location of the configuration files
mlw writes:

> AFAIK it wasn't actually done. It was more of a, "we should do something
> different" argument. At one point it was talked about rewriting the
> configuration system to allow "include" and other things.

The core of the problem was, and continues to be, this:  If you move
postgresql.conf somewhere else, then someone else will also want to move
pg_hba.conf and all the rest.  And that opens up a number of security and
cumbersome-to-install problems.

Just having an option that says, the configuration file is "there", is a
first step but not a complete solution.

-- 
Peter Eisentraut   peter_e@gmx.net


Thu, 13 Feb 2003 22:25:49 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: location of the configuration files
Peter Eisentraut wrote:

>mlw writes:
>
>  
>
>>AFAIK it wasn't actually done. It was more of a, "we should do something
>>different" argument. At one point it was talked about rewriting the
>>configuration system to allow "include" and other things.
>>    
>>
>
>The core of the problem was, and continues to be, this:  If you move
>postgresql.conf somewhere else, then someone else will also want to move
>pg_hba.conf and all the rest.  And that opens up a number of security and
>cumbersome-to-install problems.
>
>Just having an option that says, the configuration file is "there", is a
>first step but not a complete solution.
>
The location of pg_hba.conf and pg_ident.conf can be specified within 
the postgresql.conf file if desired.

I don't understand the security concerns, what security issues can there be?

>  
>








Peter Eisentraut wrote:

mlw writes:

  

AFAIK it wasn't actually done. It was more of a, "we should do something
different" argument. At one point it was talked about rewriting the
configuration system to allow "include" and other things.
    


The core of the problem was, and continues to be, this:  If you move
postgresql.conf somewhere else, then someone else will also want to move
pg_hba.conf and all the rest.  And that opens up a number of security and
cumbersome-to-install problems.

Just having an option that says, the configuration file is "there", is a
first step but not a complete solution.

The location of pg_hba.conf and pg_ident.conf can be specified within the
postgresql.conf file if desired.

I don't understand the security concerns, what security issues can there
be?
Thu, 13 Feb 2003 16:26:37 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
scott.marlowe wrote:

>>These are not issues at all. You could put the configuration file 
>>anywhere, just as you can for any UNIX service.
>>
>>postmaster --config=/home/myhome/mydb.conf
>>
>>I deal with a number of PG databases on a number of sites, and it is a 
>>real pain in the ass to get to a PG box and hunt around for data 
>>directory so as to be able to administer the system. What's really 
>>annoying is when you have to find the data directory when someone else 
>>set up the system.
>>    
>>
>
>Really?  I would think it's easier to do this:
>
>su - pgsuper
>cd $PGDATA
>pwd
>
>Than to try to figure out what someone entered when they ran ./configure 
>--config=...
>  
>
Why do you think PGDATA would be set for root?

>  
>
>>Configuring postgresql via a configuration file which specifies all the 
>>data, i.e. data directory, name of other configuration files, etc. is 
>>the right way to do it. Even if you have reasons against it, even if you 
>>think it is a bad idea, a bad standard is almost always a better 
>>solution than an arcane work of perfection.
>>    
>>
>
>Wrong, I strongly disagree with this sentament.  Conformity to standards 
>for simple conformity's sake is as wrong as sticking to the old way 
>because it's what we're all comfy with.  
>
It isn't conformity for conformitys sake. It is following an established 
practice, like driving on the same side of the road or stopping at red 
lights.

>
>  
>
>>Personally, however, I think the configuration issue is a no-brainer and 
>>I am amazed that people are balking. EVERY other service on a UNIX box 
>>is configured in this way, why not do it this way in PostgreSQL? The 
>>patch I submitted allowed the configuration to work as it currently 
>>does, but allowed for the more standard configuration file methodology.
>>    
>>
>
>If I do a .tar.gz install of apache, I get /usr/local/apache/conf, which 
>is not the standard way you're listing.  If I install openldap from 
>.tar.gz, I get a /usr/local/etc/openldap directory, close, but still not 
>the same.  The fact is, it's the packagers that put things into /etc and 
>whatnot, and I can see the postgresql RPMs or debs or whatever having that 
>as the default, but for custom built software, NOTHING that I know of 
>builds from source and uses /etc without a switch to tell it to, just like 
>postgresql can do now.
>
You are confusing the default location of a file with the ability to use 
the file. The default I have proposed all along was to use the existing 
practice of keeping everything in the $PGDATA directory.

The change I wish to make to the code allows this to be changed. Most 
admins want configuration and data separate. Most admins do not want to 
use symlinks because they are dangerous in a production environment.

I would rather have a simpler solution sooner than a perfect solution never.









scott.marlowe wrote:



These are not issues at all. You could put the configuration file 
anywhere, just as you can for any UNIX service.

postmaster --config=/home/myhome/mydb.conf

I deal with a number of PG databases on a number of sites, and it is a 
real pain in the ass to get to a PG box and hunt around for data 
directory so as to be able to administer the system. What's really 
annoying is when you have to find the data directory when someone else 
set up the system.
    


Really?  I would think it's easier to do this:

su - pgsuper
cd $PGDATA
pwd

Than to try to figure out what someone entered when they ran ./configure 
--config=...
  

Why do you think PGDATA would be set for root?



  

Configuring postgresql via a configuration file which specifies all the 
data, i.e. data directory, name of other configuration files, etc. is 
the right way to do it. Even if you have reasons against it, even if you 
think it is a bad idea, a bad standard is almost always a better 
solution than an arcane work of perfection.
    


Wrong, I strongly disagree with this sentament.  Conformity to standards 
for simple conformity's sake is as wrong as sticking to the old way 
because it's what we're all comfy with.  

It isn't conformity for conformitys sake. It is following an established
practice, like driving on the same side of the road or stopping at red lights.



  

Personally, however, I think the configuration issue is a no-brainer and 
I am amazed that people are balking. EVERY other service on a UNIX box 
is configured in this way, why not do it this way in PostgreSQL? The 
patch I submitted allowed the configuration to work as it currently 
does, but allowed for the more standard configuration file methodology.
    


If I do a .tar.gz install of apache, I get /usr/local/apache/conf, which 
is not the standard way you're listing.  If I install openldap from 
.tar.gz, I get a /usr/local/etc/openldap directory, close, but still not 
the same.  The fact is, it's the packagers that put things into /etc and 
whatnot, and I can see the postgresql RPMs or debs or whatever having that 
as the default, but for custom built software, NOTHING that I know of 
builds from source and uses /etc without a switch to tell it to, just like 
postgresql can do now.

You are confusing the default location of a file with the ability to use
the file. The default I have proposed all along was to use the existing practice
of keeping everything in the $PGDATA directory.

The change I wish to make to the code allows this to be changed. Most admins
want configuration and data separate. Most admins do not want to use symlinks
because they are dangerous in a production environment.

I would rather have a simpler solution sooner than a perfect solution never.
Thu, 13 Feb 2003 16:44:05 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
> All I see here is an arbitrary break with our past practice.  I do
> not see any net improvement.

<FreeBSD Port Maintainer>
Well, given that there's a trend to make PostgreSQL more usable, I can
say with absolute certainty, that an FAQ that I get about once a week
is (and granted only from new users) "where is the postgresql.conf?  I
don't see it in ${LOCALBASE}/etc/."  PostgreSQL is one of a few ports
in an extreme minority that uses a local configuration directive and
it violates the policy of least surprise for sysadmins.

PS LOCALBASE/PREFIX is /usr/local 99.999% of the time
</FreeBSD Port Maintainer>

With my DBA hat on, however, here are a few reasons that I'd like to
see the conf moved out of the data directory:

1) pg_dumpall > foo && rm -rf $PGDATA && initdb

   As a DBA I don't have to worry about backing up my config file when
   doing upgrades since the config file is located in an external
   directory.

2) Backing up config files in ${LOCALBASE}/etc is a pretty common
   practice.  Having to make a special case for postgresql's kind of a
   PITA.


Suggestions:

1) gmake install installs a default configuration file in
   ${LOCALBASE}/etc/postgresql.conf.default.  Promote that DBAs should
   diff postgresql.conf.default with postgresql.conf and make
   adjustments as they see fit, but gmake install will _not_, under
   any circumstances, touch postgresql.conf (by default, it should cp
   postgresql.conf.default to postgresql.conf that way things "just
   work" out of the box).

2) Leave the current functionality in place.  Being able to have
   multiple databases on the same machine is a _really_ nice feature
   of PostgreSQL.  If you want multiple databases, having the config
   file in $PGDATA makes some sense because with multiple
   installations, you want to keep everything together... though it
   doesn't make much sense if you have only one installation per
   server... and really, the only reason to have multiple
   installations is to handle username collisions (hint hint).

3) In the absence of a PGDATA environment variable (don't want to
   break backward compatible installations) being set, the future
   behavior allow for a default location of a config file (if no CLI
   switch is specified for an explicit location) that points to a
   config file.  The path would be ${PREFIX}/etc and would provide
   most admins with a standard launching off point for running/tuning
   their databases.  The config file would have to specify the data
   directory as well as the path to the hba.conf, which should be
   outside of the datadir as well (speaking of the hba.conf, am I the
   only one who things that hba.conf should be converted into a system
   catalog?  ::shrug::)

Just some random thoughts from someone who's had to deal with this on
all of the mentioned levels (new users, single installations, multiple
installations, and multiple copies running via daemontools). -sc


PS If there is no huge press for this, I should have the time do do
this in a few weeks if someone doesn't beat me to it.

-- 
Sean Chittenden

Thu, 13 Feb 2003 14:06:25 -0800
Sean Chittenden <sean@chittenden.org>
Re: location of the configuration files
On Thu, 2003-02-13 at 21:21, Vince Vielhaber wrote:
> I certainly wasn't trying to provoke anything.  It just seems odd to me
> that when the distribution installs a package and places it's config files
> in /etc and later the admin happens to upgrade by the instructions with
> the package, it's acceptable for the config files to now be in two places
> and you don't find it confusing.  What happens when a new admin comes on
> and tries to figure out which config file is which?   Ever try to figure
> out where the hell Pine's config really is?

I've not used pine, and there doesn't seem to be an official Debian
package, (it doesn't allow any changes to its source, I believe, which
makes it ineligible).  But if it were an official package, I know I
should look in /etc/pine.

If the admin installs a local build of something he has installed as a
package, he will presumably take care to separate the two.  If his local
build is to replace the package, he should purge the installed package,
so that there are no traces of it left.  Since he is administering a
distribution installation, it is certainly his responsibility to
understand the difference between local and distributed packages, as
well as the different places that each should put their configuration
files.  (Incidentally, Debian's changes from the upstream configuration
are documented in the package.)  In the end, though, when we package for
a distribution, we expect people to use the packages.  If they want to
build from source, the packages system lets them do it.  Anyone who is
building from the upstream source must be presumed to know what he is
doing and take responsibility for it.

What your comments strongly suggest to me is that projects like
PostgreSQL and pine, along with everything else, should comply with FHS;
then there will be no confusion because everyone will be following the
smae standards.  Messes arise when people ignore standards; we have all
seen the dreadful examples of MySQL and the Beast, haven't we?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "The earth is the LORD'S, and the fullness thereof; the
      world, and they that dwell therein."            
                                   Psalms 24:1 


13 Feb 2003 22:22:56 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files
On Thu, 2003-02-13 at 14:28, Bruce Momjian wrote:
> Robert Treat wrote:
> > On Thu, 2003-02-13 at 14:06, mlw wrote:
> > > 
> > > I will be resubmitting my patch for the 7.3.2 tree.
> > > 
> > 
> > I'm no core developer, but surely this wont be included in the 7.3.x
> > branch. Any change needs to be made against CVS head.
> 
> I assume he meant he will repost his 7.3.2-based patch and we will merge
> it into CVS HEAD if it is accepted.
> 

IIRC he originally wrote the patch for a pre 7.3 version, so it seems
like he'd be reworking it for 7.3.x with the above statement. I'm only
suggesting he rework it against CVS head if he doesn't have plans to do
so already. Course if yall are willing to merge it in for him, none of
this really matters does it? :-)

Robert Treat


13 Feb 2003 17:25:13 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: location of the configuration files
On Thu, 2003-02-13 at 14:43, Bruce Momjian wrote:
> Robert Treat wrote:
> > On Thu, 2003-02-13 at 12:13, mlw wrote:
> > > 
> > > My patch only works on the PostgreSQL server code. No changes have been
> > > made to the initialization scripts.
> > > 
> > > The patch declares three extra configuration file parameters:
> > > hbafile= '/etc/postgres/pg_hba.conf'
> > > identfile='/etc/postgres/pg_ident.conf'
> > > datadir='/RAID0/postgres'
> > > 
> > 
> > If we're going to do this, I think we need to account for all of the
> > files in the directory including PG_VERSION, postmaster.opts,
> > postmaster.pid. In the end if we can't build so that we are either fully
> > FHS compliant and/or LSB compliant, we've not done enough work on it.
> 
> Woh, how do we move some of those files into /etc or /var/run if we
> aren't running as root?  We certainly don't want to require that.  I
> guess /etc/postgresql will work if that directory is owned by the
> PostgreSQL superuser, but /var/run will be a problem.
> 

Seems like some are saying one of the problems with the current system
is it doesn't follow FHS or LSB. If those are valid reasons to change
the system, it seems like a change which doesn't actually address those
concerns would not be acceptable. (Unless those really aren't valid
concerns...)

Robert Treat


13 Feb 2003 17:29:24 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: location of the configuration files
On Thu, 2003-02-13 at 14:51, mlw wrote:
> 
> 
> Robert Treat wrote:
> 
> 
> On Thu, 2003-02-13 at 12:13, mlw wrote:
> 
>   
> 
> My patch only works on the PostgreSQL server code. No changes have been
> 
> made to the initialization scripts.
> 
> 
> 
> The patch declares three extra configuration file parameters:
> 
> hbafile= '/etc/postgres/pg_hba.conf'
> 
> identfile='/etc/postgres/pg_ident.conf'
> 
> datadir='/RAID0/postgres'
> 
> 
> If we're going to do this, I think we need to account for all of the
> 
> files in the directory including PG_VERSION, postmaster.opts,
> 
> postmaster.pid. In the end if we can't build so that we are either fully
> 
> FHS compliant and/or LSB compliant, we've not done enough work on it.
> 
> 
> postmaster.opts, PG_VERSION, and postmaster.pid are not configuration
> parameters. 
>

So? I'm not saying they all have to be moved, just they all need to be
accounted for. 
 
> PG_VERSION is VERY important, it is how you know the version of the
> database.
> Postmaster.pid is a postgres writable value 
> AFAIK, postmaster.opts is also a postgres writable value.
> 

IIRC the postmaster.pid file should be in /var/run according to FHS, I'm
not sure about postmaster.opts though...

Again, if we're going to make a change, let's make sure we think it
through.

Robert Treat


13 Feb 2003 17:44:29 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: location of the configuration files
Oliver Elphick wrote:
> What your comments strongly suggest to me is that projects like
> PostgreSQL and pine, along with everything else, should comply with FHS;
> then there will be no confusion because everyone will be following the
> smae standards.  Messes arise when people ignore standards; we have all
> seen the dreadful examples of MySQL and the Beast, haven't we?

Can the FHS handle installing PostgreSQL as non-root?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 13 Feb 2003 17:53:53 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Robert Treat wrote:
> IIRC the postmaster.pid file should be in /var/run according to FHS, I'm
> not sure about postmaster.opts though...
> 
> Again, if we're going to make a change, let's make sure we think it
> through.

Can non-root write to /var/run?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 13 Feb 2003 17:55:16 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
On Thu, 2003-02-13 at 15:08, mlw wrote:
> Stephan Szabo wrote:
> 
> Re-read my statement and yours about the case you were mentioning. ;)
> 
> Sure, putting the files in /etc lets you find them easily.  However, if
> 
> you're doing things like finding configuration made by someone else and
> 
> said configuration isn't in /etc (which if they wanted to they could do
> 
> now with symlinks I believe - yes symlinks aren't a complete solution,
> but
> 
> I think they're reasonable on most of our current ports) then you still
> 
> have to search the system for the configuration file, except now it
> might
> 
> not even be postgresql.conf. That's why I said the two issues aren't the
> 
> same.
> 

Actually, I'd almost go so far as to say it will make it worse. In the
current system, if you can figure out where $PGDATA is, you've found
everything you need for that installation. In the new system, there's no
telling where people will put things, and it certainly won't be any
easier to find it. THinking on the above Stephan, you'd almost have to
require that the config file be called postgresql.conf in order to run,
anything else leads to real scary scenario's.


> On Thu, 13 Feb 2003, mlw wrote:
> 
> I have absolutely no problem debating and augmenting the feature. None
> what so ever, I am more pushing to get momentum to actually do it. 

Stick with it, I think most of us here can see the value in the option,
but there are valid concerns that it be implemented correctly.
Personally I think a postgresql installation is much more like an apache
installation, which generally contains all of the files (data and
config) under /usr/local/apache. Maybe someone can dig more to see if
that system is more appropriate a comparison than something like bind.

Robert Treat



13 Feb 2003 17:59:17 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: location of the configuration files
Bruce Momjian wrote:

>Robert Treat wrote:
>  
>
>>IIRC the postmaster.pid file should be in /var/run according to FHS, I'm
>>not sure about postmaster.opts though...
>>
>>Again, if we're going to make a change, let's make sure we think it
>>through.
>>    
>>
>
>Can non-root write to /var/run?
>
>  
>
Shouldn't be able too








Bruce Momjian wrote:

Robert Treat wrote:
  

IIRC the postmaster.pid file should be in /var/run according to FHS, I'm
not sure about postmaster.opts though...

Again, if we're going to make a change, let's make sure we think it
through.
    


Can non-root write to /var/run?

  

Shouldn't be able too
Thu, 13 Feb 2003 18:06:15 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
On 13 Feb 2003, Oliver Elphick wrote:

> What your comments strongly suggest to me is that projects like
> PostgreSQL and pine, along with everything else, should comply with FHS;
> then there will be no confusion because everyone will be following the
> smae standards.  Messes arise when people ignore standards; we have all
> seen the dreadful examples of MySQL and the Beast, haven't we?

Actually FHS says the opposite.  If the distribution installs PostgreSQL
then the config files belong in /etc/postgresql.  If the admin does then
they belong in /usr/local/etc/postgresql.  FHS is out of their tree.  If
PostgreSQL or any other package is not critical to the basic operation of
the operating system, it's config files shouldn't be polluting /etc.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Thu, 13 Feb 2003 18:07:46 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: location of the configuration files
Robert Treat wrote:

>On Thu, 2003-02-13 at 14:51, mlw wrote:
>  
>
>>Robert Treat wrote:
>>
>>
>>On Thu, 2003-02-13 at 12:13, mlw wrote:
>>
>>  
>>
>>My patch only works on the PostgreSQL server code. No changes have been
>>
>>made to the initialization scripts.
>>
>>
>>
>>The patch declares three extra configuration file parameters:
>>
>>hbafile= '/etc/postgres/pg_hba.conf'
>>
>>identfile='/etc/postgres/pg_ident.conf'
>>
>>datadir='/RAID0/postgres'
>>
>>
>>If we're going to do this, I think we need to account for all of the
>>
>>files in the directory including PG_VERSION, postmaster.opts,
>>
>>postmaster.pid. In the end if we can't build so that we are either fully
>>
>>FHS compliant and/or LSB compliant, we've not done enough work on it.
>>
>>
>>postmaster.opts, PG_VERSION, and postmaster.pid are not configuration
>>parameters. 
>>
>>    
>>
>
>So? I'm not saying they all have to be moved, just they all need to be
>accounted for. 
>
OK, what was the point?

>>PG_VERSION is VERY important, it is how you know the version of the
>>database.
>>Postmaster.pid is a postgres writable value 
>>AFAIK, postmaster.opts is also a postgres writable value.
>>
>>    
>>
>
>IIRC the postmaster.pid file should be in /var/run according to FHS, I'm
>not sure about postmaster.opts though...
>
>Again, if we're going to make a change, let's make sure we think it
>through.
>
I'm not a big fan of the "/var/run" directory convention, especially 
when we expect multiple instances of the server to be able to run 
concurrently. I suppose it can be a parameter in both the configuration 
file and command line.









Robert Treat wrote:

On Thu, 2003-02-13 at 14:51, mlw wrote:
  


Robert Treat wrote:


On Thu, 2003-02-13 at 12:13, mlw wrote:

  

My patch only works on the PostgreSQL server code. No changes have been

made to the initialization scripts.



The patch declares three extra configuration file parameters:

hbafile= '/etc/postgres/pg_hba.conf'

identfile='/etc/postgres/pg_ident.conf'

datadir='/RAID0/postgres'


If we're going to do this, I think we need to account for all of the

files in the directory including PG_VERSION, postmaster.opts,

postmaster.pid. In the end if we can't build so that we are either fully

FHS compliant and/or LSB compliant, we've not done enough work on it.


postmaster.opts, PG_VERSION, and postmaster.pid are not configuration
parameters. 

    


So? I'm not saying they all have to be moved, just they all need to be
accounted for. 

OK, what was the point?



PG_VERSION is VERY important, it is how you know the version of the
database.
Postmaster.pid is a postgres writable value 
AFAIK, postmaster.opts is also a postgres writable value.

    


IIRC the postmaster.pid file should be in /var/run according to FHS, I'm
not sure about postmaster.opts though...

Again, if we're going to make a change, let's make sure we think it
through.

I'm not a big fan of the "/var/run" directory convention, especially when
we expect multiple instances of the server to be able to run concurrently.
I suppose it can be a parameter in both the configuration file and command
line.
Thu, 13 Feb 2003 18:25:20 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
On Thursday 13 February 2003 17:53, Bruce Momjian wrote:
> Oliver Elphick wrote:
> > What your comments strongly suggest to me is that projects like
> > PostgreSQL and pine, along with everything else, should comply with FHS;
> > then there will be no confusion because everyone will be following the
> > smae standards.  Messes arise when people ignore standards; we have all
> > seen the dreadful examples of MySQL and the Beast, haven't we?

> Can the FHS handle installing PostgreSQL as non-root?

Once again, no one is trying to make an FHS install the default 'let's force 
everyone to think our way or no way' coercion.

We just want the option.

For those who wish to do non-root installs, nothing would need to change.  You 
can still put it into /usr/local/pgsql (assuming you have permissions to put 
it there) or your home directory, or wherever.

I deal with RPMs; Oliver deals with .deb's.  Neither can be installed as 
non-root.  The daemon can of course run as non-root (and it does, which is 
exactly correct); but the installation of the files is done as root _always_ 
in an RPM or deb environment.  So I really don't care about non-root 
installs; sorry.  I wonder what percentage of our users are not the 
administrator of the machine on which they are running PostgreSQL?

I dispute the statement made earlier in the thread (not by Bruce) that 
PostgreSQL is by definition not an OS service.  This is false, and needs to 
be realized by this community.  PostgreSQL is becoming an essential OS core 
service in many cases: virtually all Linux distributions (the lion's share of 
our current distribution) include PostgreSQL as a core service.  Many of our 
new users see PostgreSQL as 'SQL server' in the Red Hat installation menu.

Now, on a Win32 server, what is PostgreSQL going to be considered?  It is 
probably going to run as a service, right? So you need to be Administrator 
there to perform the install, right?

This isn't the same environment, Bruce, that you got into back when it was 
still Postgres95.  We are in the big leagues OS-wise, and we need to act like 
it.  Assuming that we are a 'userspace' program (which is a misnomer anyway, 
as _anything_ non-kernel is 'userspace') is not going to cut it anymore.  

So we need to fit in to an OS environment, whether it is FreeBSD, OS/X, Win32, 
Solaris, or Linux.  In FreeBSD, as the ports maintainer excellently posted, 
PostgreSQL should live in LOCALBASE.  We should make that easy.  In Win32, 
configuration might be better stored in the system registry (Argh! Did I 
actually say THAT! Yuck!) -- we should make even that easy.  In OS/X we 
should use the OS/X paradigm (whatever that is).  And we should make it easy 
to make PostgreSQL LSB-compliant for our very large Linux user community.  We 
should be adaptable to the accepted administration paradigm on whatever 
system we are running -- this should be a minimum.

These concerns vastly outweigh the occasional non-root install from source, in 
my mind at least.  I am not opposed to that way even being the default; after 
all, leaving the default the same as now agrees with the principle of least 
surprise (although we really don't ascribe to that; witness the 7.2-7.3 
migration fiasco -- 7.3 should have been 8.0 to warn people of the major 
changes going on in client connections).  But I do advocate _allowing_ the 
configuration options Mark has enumerated -- although I really wish we could 
use the lowercase c instead, for consistency with other OS services.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 13 Feb 2003 18:26:00 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
On Thursday 13 February 2003 18:07, Vince Vielhaber wrote:
> Actually FHS says the opposite.  If the distribution installs PostgreSQL
> then the config files belong in /etc/postgresql.  If the admin does then
> they belong in /usr/local/etc/postgresql.  FHS is out of their tree.  If
> PostgreSQL or any other package is not critical to the basic operation of
> the operating system, it's config files shouldn't be polluting /etc.

PostgreSQL is as critical as PHP, Apache, or whatever other package is being 
backended by PostgreSQL.  If the package is provided by the distributor, 
consider it part of the OS.  If it isn't, well, it isn't.

This is so that local admin installed (from source -- not from binary package) 
files don't get clobbered by the next operating system binary upgrade.  In 
that context the FHS (LSB) mandate makes lots of sense.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 13 Feb 2003 18:30:45 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
On Thu, 13 Feb 2003, Lamar Owen wrote:

> On Thursday 13 February 2003 18:07, Vince Vielhaber wrote:
> > Actually FHS says the opposite.  If the distribution installs PostgreSQL
> > then the config files belong in /etc/postgresql.  If the admin does then
> > they belong in /usr/local/etc/postgresql.  FHS is out of their tree.  If
> > PostgreSQL or any other package is not critical to the basic operation of
> > the operating system, it's config files shouldn't be polluting /etc.
>
> PostgreSQL is as critical as PHP, Apache, or whatever other package is being
> backended by PostgreSQL.  If the package is provided by the distributor,
> consider it part of the OS.  If it isn't, well, it isn't.

You completely miss my point, but lately you've been real good at that.

Can the system boot without PHP, Apache, PostgreSQL, Mysql and/or Pine?
Can the root user log in without PHP, Apache, PostgreSQL, Mysql and/or Pine?
Can any user log in without PHP, Apache, PostgreSQL, Mysql and/or Pine?

Note, I'm not even including an MTA here.  I said BASIC OPERATION.

If a package is not critical as I just outlined, it shouldn't matter who
installed it.

After the last go around with you Lamar, this will be my last response
to you on this.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Thu, 13 Feb 2003 18:41:06 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: location of the configuration files
On Thursday 13 February 2003 18:41, Vince Vielhaber wrote:
> On Thu, 13 Feb 2003, Lamar Owen wrote:
> > PostgreSQL is as critical as PHP, Apache, or whatever other package is
> > being backended by PostgreSQL.  If the package is provided by the
> > distributor, consider it part of the OS.  If it isn't, well, it isn't.

> You completely miss my point, but lately you've been real good at that.

No, Vince, I understand your point.  But understand mine: it does matter who 
installed it.

> Note, I'm not even including an MTA here.  I said BASIC OPERATION.

> If a package is not critical as I just outlined, it shouldn't matter who
> installed it.

'Critical' is in the eye of the admin of the system in question.  For my 
servers, if, for instance, sshd doesn't come up, then there's a major 
problem, as they are all headless.  If the webserver doesn't come up, I have 
other problems, as OpenACS is mission-critical here.  So what's critical is a 
question for the individual sysadmin.

So, to continue your point, what is 'critical' to the 'basic operation' of the 
system shouldn't pollute /etc.  So, let's eliminate the /etc/mail, 
/etc/samba, /etc/xinetd.d, /etc/X11, /etc/httpd, and the other subtrees foung 
in at least Red Hat 8.  While we're at it, many other files in /etc need to 
go: named.conf for one.  It depends on what you consider 'critical'.  
PostgreSQL is at least as critical on my systems as some of the other things 
that already 'pollute' /etc.

> After the last go around with you Lamar, this will be my last response
> to you on this.

Aw Vince, I don't know what your problem is with conflicting opinions.  But 
that's your choice.  And Open Source is about _choice_.  You are free to 
admin your systems your way, and I'm free to do so my way.  And all's well.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 13 Feb 2003 19:10:04 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
On Thu, Feb 13, 2003 at 05:59:17PM -0500, Robert Treat wrote:
> On Thu, 2003-02-13 at 15:08, mlw wrote:
> > Stephan Szabo wrote:
> > 
> > On Thu, 13 Feb 2003, mlw wrote:
> > 
> > I have absolutely no problem debating and augmenting the feature. None
> > what so ever, I am more pushing to get momentum to actually do it. 
> 
> Stick with it, I think most of us here can see the value in the option,
> but there are valid concerns that it be implemented correctly.
> Personally I think a postgresql installation is much more like an apache
> installation, which generally contains all of the files (data and
> config) under /usr/local/apache. Maybe someone can dig more to see if
> that system is more appropriate a comparison than something like bind.

	I think you are making a pretty uninformed, if not just plain wrong 
generalization.  I've run exactly one system with apache configuration 
files in /usr/local/apache, and even then, the data was not there.

	A quick straw poll of the people I know who actually do run real systems
also mentioned that they use packaging systems like encap or rpm to manage
upgrades, and would almost never put datafiles into /usr/local.

RedHat (7.3 at least)'s default httpd datafiles go in /var/www/html and
config goes in /etc/httpd

One OpenBSD user I talked to puts his in /home/www and config files in
/etc/httpd.  The defaults are /var/www and /var/www/conf

Another user reports:
On systems that I set up I have /web/{apache|httpd}/ and put all 
the config info there.
And /web/sites/name/ holds site data.



What does this mean?

People will put things in different places, and there are typically
very good reasons for this.  This is ESPECIALLY true when one wants to
have configuration files, at least the base ones in a common place such
as /etc or /usr/local/etc in order to make backup of configuration easy
and clean, while leaving data somewhere else for performance or magnitude
of partition reasons.  It just makes sense to ME to have postgresql.conf
reside in /etc, yet put my data in /var/data/postgresql, yet retain the
option to put my data in /raid/data/postgresql at a later date, when the
new hardware comes in.

Yes, symlinks are an option on most systems.  No, they are not a good
one on most systems.


What _I_ would like to see:

o. a default postgresql.conf location of $PREFIX/data/postgresql.conf
o. a default PGDATA location of whatever directory postgresql.conf is in
(this should maintain backward compatibility)
o. a ./configure - time option to override the location of the postgresql.conf
o. a run-time option to override the location of the postgresql.conf
o. options in postgresql.conf to specify the location of PGDATA and PID files.

($PREFIX is already settable at ./configure - time)

This would allow:
	o. Config files in /usr/local/pgsql/data, /etc, /usr/local/etc, ~postgresql, 
	or /dev/.hidden-node, whichever you prefer, so long as you either know
	the compile-time default, or are willing to specify it at startup.

	o. Datafiles to be in /usr/local/pgsql/data, /var/data, /raid0, /nfs/bigmount
	or whichever you prefer, so long as you either know the compile-time default,
	or are willing to specify it in a config file that you specify at startup.

Does it add complexity to the system?  Sure -- a very little bit, IMHO, especially
compared to the BTREE-folding that I see being bantered about.

Is it some work?  Sure -- a very little bit, and it seems that it has already
been done.


However, this seems, to me, to be a very small addition that has some real-world
(and yes, we need to start paying attention to the real world) advantages.

And finally, don't go telling me that I'm wrong to put my data and config files
where I am.  You can offer advice, but I'm probably going to ignore it because
I like where they are and don't need to explain why.


-- 
Adam Haberlach         | "Because manholes are round."
adam@mediariffic.com   |
http://mediariffic.com |

Thu, 13 Feb 2003 16:22:40 -0800
Adam Haberlach <adam@newsnipple.com>
Re: location of the configuration files
Lamar Owen wrote:
> This isn't the same environment, Bruce, that you got into back when it was 
> still Postgres95.  We are in the big leagues OS-wise, and we need to act like 
> it.  Assuming that we are a 'userspace' program (which is a misnomer anyway, 
> as _anything_ non-kernel is 'userspace') is not going to cut it anymore.  

So you are saying this isn't my grandma's database anymore.  :-)

Anyway, I think I have _a_ proposal that we can use to work toward a
goal.

First, a few conclusions:

	We can't use /var/run because we need the postmaster to create
	those, and it isn't root.

	Right now, the fact that we mix the config stuff with the
	data isn't ideal.  Someone mentioned:

		pg_dumpall > foo && rm -rf $PGDATA && initdb

	discards all the config files.

So, I propose we change a few things.  The good news is that this is
something only administrators deal with;  client apps don't deal with
it.


OK, first, we keep postmaster.pid and postmaster.opts in /data.  We
can't put them in /var/run, and /data seems like the best spot for them.

That leaves postgresql.conf, pg_hba.conf, and pg_ident.conf.  I
recommend moving them all, by default, into pgsql/etc.  I recommend we
add these to postgresql.conf:

	data_dir = ../data
	pg_hba_dir = ./
	pg_ident_dir = ./

Those paths are relative to postgresql.conf.

We then add a PGCONFIG variable and postmaster -C flag to point to the
config _directory_.  That way, if folks want to move all of this into
/etc, then easily do that.  This also pulls those files out of /data so
they are easier to back up.

We can also firm up stuff in 7.5 by removing PGDATA and -D, and perhaps
removing the other duplicate postmaster flags that have postgresql.conf
entries.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 13 Feb 2003 20:09:34 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
On Thursday 13 February 2003 20:09, Bruce Momjian wrote:
> Lamar Owen wrote:
> > This isn't the same environment, Bruce, that you got into back when it
> > was still Postgres95.

> So you are saying this isn't my grandma's database anymore.  :-)

I actually thought of saying it that way, too. :-)

> Anyway, I think I have _a_ proposal that we can use to work toward a
> goal.

> First, a few conclusions:

> 	We can't use /var/run because we need the postmaster to create
> 	those, and it isn't root.

It isn't without precedent to have a directory under /var/run.  Maybe 
/var/run/postgresql.  Under this one could have a uniquely named pid file.  I 
say uniquely named so that multiple postmasters could run.  Naming those 
files could be fun. /var/run/postgresql would be owned by the postmaster run 
user.  This of course requires root to install -- but would be completely 
optional.

> 		pg_dumpall > foo && rm -rf $PGDATA && initdb

> 	discards all the config files.

Yes, this is a big deal.  It makes it more difficult to properly restore.  
While it's not impossible to do so now, of course, it just could be a little 
easier.

> So, I propose we change a few things.

> OK, first, we keep postmaster.pid and postmaster.opts in /data.  We
> can't put them in /var/run, and /data seems like the best spot for them.

Can we make that configurable?  The default in pgdata is fine; just having the 
option is good.

> That leaves postgresql.conf, pg_hba.conf, and pg_ident.conf.  I
> recommend moving them all, by default, into pgsql/etc.  I recommend we
> add these to postgresql.conf:

> 	data_dir = ../data
> 	pg_hba_dir = ./
> 	pg_ident_dir = ./

> Those paths are relative to postgresql.conf.

And these are all just defaults, easily changed.  Good.

> We then add a PGCONFIG variable and postmaster -C flag to point to the
> config _directory_.  That way, if folks want to move all of this into
> /etc, then easily do that.  This also pulls those files out of /data so
> they are easier to back up.

Yes.  I'm thinking along the lines of this sort of structure:
/etc
|---postgresql
                |----- name of postmaster one (unique ID of some kind)
                |----- name of postmaster two
                    .
                    .

Not difficult.

> We can also firm up stuff in 7.5 by removing PGDATA and -D, and perhaps
> removing the other duplicate postmaster flags that have postgresql.conf
> entries.

Now I really _like_ this idea.  By removing it to 7.5, and therefore 
deprecating it in 7.4, this brings best practice into effect.

However, at the same time, I wouldn't be opposed to leaving them in place, 
either, for backwards compatibility.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 13 Feb 2003 20:39:13 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
Bruce Momjian wrote:
> I don't think separate params for each config file is good.  At the
> most, I think we will specify the configuration _directory_ for all the
> config files, perhaps pgsql/etc, and have pgdata default to ../data, or
> honor $PGDATA.  That might be the cleanest.
> 
> Of course, that now gives us $PGCONFIG and $PGDATA, and possible
> intraction if postgresql.conf specifies a different pgdata from $PGDATA.
> As you can see, it could get messy.

Uh...why are we having to mess with environment variables at all?
It's one thing for shell scripts to make use of them, but another
thing entirely for an executable like the postmaster to do the same.

Seems logical to me to eliminate the use of $PGDATA in the postmaster
entirely.  It usually gets started from a shell script, so let the
shell script pass the appropriate parameter telling the postmaster
where to find the data, or the config files, or whatever.

> And, if you specify pgdata in postgresql.conf, it prevents you from
> using that file by different postmasters.

Not at all.  Don't GUC variables that are specified on the command
line override the ones in the configuration file?

> My best guess would be to not specify pgdata in postgresql.conf, and
> have a new $PGCONFIG param to specify the configuration directory, but
> if we do that, $PGDATA/postgresql.conf becomes meaningless, which could
> also be confusing.  Maybe we don't allow those files to exist in $PGDATA
> if $PGCONFIG is used, _and_ $PGCONFIG is not the same as $PGDATA.  See,
> I am getting myself confused.  :-)

I think the solution is real simple:

1.  Eliminate the use of $PGDATA in the postmaster.  It causes far
    more headaches than it's worth.  Instead, require that -D be
    passed on the command line.  It's fine if the postmaster *sets*
    $PGDATA in order to minimize any changes that need to be made
    elsewhere, but the postmaster should not use it until it sets it.
    The postmaster right now reads all the config files (including
    postgresql.conf) from the directory specified by the -D option.
    Keep it that way.

2.  Add a GUC variable that specifies where the data is.  If this
    variable is not defined either on the command line or in the
    config file, then assume that the data is in the same place as the
    config file.  Obviously files like PG_VERSION are associated with
    the data and not with the config, so they get treated
    appropriately.

The above addresses *everyone's* concerns that I've seen thus far, I
think.  Thoughts?


-- 
Kevin Brown					      kevin@sysexperts.com

Thu, 13 Feb 2003 17:47:11 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Lamar Owen wrote:
> > First, a few conclusions:
> 
> > 	We can't use /var/run because we need the postmaster to create
> > 	those, and it isn't root.
> 
> It isn't without precedent to have a directory under /var/run.  Maybe 
> /var/run/postgresql.  Under this one could have a uniquely named pid file.  I 
> say uniquely named so that multiple postmasters could run.  Naming those 
> files could be fun. /var/run/postgresql would be owned by the postmaster run 
> user.  This of course requires root to install -- but would be completely 
> optional.

But how do you handle the default then, where you have postmaster.pid in
/data?  Do we rename it to postmaster.pid.5432 so it can sit in
/var/run/postgresql alone with other backends?

Another issue is that pg_ctl looks at that file, so moving it around is
going to be tricky.  Also, this brings up a new issue that pg_ctl all of
a sudden can't just look at $PGDATA but must instead grope through
postgresql.conf to find the data directory location.  That could be
interesting.  Of course, it can still supply the /data path on the
command line, but if we use only $PGCONFIG, we would need to have it
find /data automatically from postgresql.conf.


> > OK, first, we keep postmaster.pid and postmaster.opts in /data.  We
> > can't put them in /var/run, and /data seems like the best spot for them.
> 
> Can we make that configurable?  The default in pgdata is fine; just having the 
> option is good.

Basically, I am saying that unless someone wants to use this
configurability, it is going to cause code confusion so it is best
avoided.
`
> Yes.  I'm thinking along the lines of this sort of structure:
> /etc
> |---postgresql
>                 |----- name of postmaster one (unique ID of some kind)
>                 |----- name of postmaster two
>                     .
>                     .
> 
> Not difficult.

Yes, that would work easily.

> > We can also firm up stuff in 7.5 by removing PGDATA and -D, and perhaps
> > removing the other duplicate postmaster flags that have postgresql.conf
> > entries.
> 
> Now I really _like_ this idea.  By removing it to 7.5, and therefore 
> deprecating it in 7.4, this brings best practice into effect.
> 
> However, at the same time, I wouldn't be opposed to leaving them in place, 
> either, for backwards compatibility.

The problem is that we would be having too many ways to specify the
/data directory.

I am now wondering if we even want pg_hba_dir and pg_ident_dir.  Seems
we can assume they are in the same directory as postgresql.conf.  That
leaves only data_dir as new for postgresql.conf.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 13 Feb 2003 21:13:29 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Vince Vielhaber wrote:
> On Thu, 13 Feb 2003, Lamar Owen wrote:
> 
> > On Thursday 13 February 2003 18:07, Vince Vielhaber wrote:
> > > Actually FHS says the opposite.  If the distribution installs PostgreSQL
> > > then the config files belong in /etc/postgresql.  If the admin does then
> > > they belong in /usr/local/etc/postgresql.  FHS is out of their tree.  If
> > > PostgreSQL or any other package is not critical to the basic operation of
> > > the operating system, it's config files shouldn't be polluting /etc.
> >
> > PostgreSQL is as critical as PHP, Apache, or whatever other package is being
> > backended by PostgreSQL.  If the package is provided by the distributor,
> > consider it part of the OS.  If it isn't, well, it isn't.
> 
> You completely miss my point, but lately you've been real good at that.
> 
> Can the system boot without PHP, Apache, PostgreSQL, Mysql and/or
> Pine?

Yep.

> Can the root user log in without PHP, Apache, PostgreSQL, Mysql
> and/or Pine?

Hopefully.

> Can any user log in without PHP, Apache, PostgreSQL, Mysql and/or
> Pine?

That depends, doesn't it?  There exist PAM modules that allow
authentication against a database, for instance.  If you're using them
and the database doesn't come up, the users can't log in.  So suddenly
the database config files belong in /etc?

The mission of the box is what counts.  If the mission of the box is
to be a web server then I'm probably not going to care whether
non-root users can log into it: that simply doesn't factor into the
mission profile.  The web server process is going to be as critical to
the mission of the box as almost anything else on it, as will anything
the web server process depends on -- which may well include a
database.

> Note, I'm not even including an MTA here.  I said BASIC OPERATION.

So by your reasoning sendmail.cf doesn't belong in /etc??  I dare say
that's news to most of us.  Where, then, *does* it belong?

> If a package is not critical as I just outlined, it shouldn't matter
> who installed it.

Oh, it matters a great deal, because people upgrade their OS installs
from time to time.  Many OS distributions come with a lot of packages
that aren't "critical" as you define them but which nevertheless will
cause much pain and suffering for the sysadmin if they install
themselves over what the sysadmin has previously built by hand.

The purpose for differentiating between a package that was compiled
and installed from the source by the sysadmin and a prebuilt package
that was provided to the sysadmin by the vendor is to keep them from
stepping on each other -- if the sysadmin went to the trouble of
compiling and installing a package from the source instead of using a
prebuilt version from the vendor, then he probably did so for a very
good reason, and is going to be *really* annoyed if an OS upgrade
blows away his work.


There are some good reasons for putting all the config files in /etc,
one of them being that it gives you *one* directory full of config
files to worry about backing up instead of many.  If you've got other
ideas I'm certainly interested in hearing the reasoning behind them.
But from the point of view of maintaining a widely deployed package
like PostgreSQL, the conventions the distributions and sysadmins use
matter a great deal, whether or not you happen to agree with those
conventions.



-- 
Kevin Brown					      kevin@sysexperts.com

Thu, 13 Feb 2003 18:20:05 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
> On 13 Feb 2003, Oliver Elphick wrote:
> 
> > What your comments strongly suggest to me is that projects like
> > PostgreSQL and pine, along with everything else, should comply with FHS;
> > then there will be no confusion because everyone will be following the
> > smae standards.  Messes arise when people ignore standards; we have all
> > seen the dreadful examples of MySQL and the Beast, haven't we?
> 
> Actually FHS says the opposite.  If the distribution installs PostgreSQL
> then the config files belong in /etc/postgresql.  If the admin does then
> they belong in /usr/local/etc/postgresql.  FHS is out of their tree.  If
> PostgreSQL or any other package is not critical to the basic operation of
> the operating system, it's config files shouldn't be polluting /etc.

I suspect you may be conflating BSD usage with Linux usage here...

The point isn't of being "critical to basic operation of the operating 
system;" it is of whether or not the software is being "package-managed" or 
not.

One of the operating principles in FHS is that "/usr/local" is an area that 
the distribution should never "pollute."  And so, a "package-managed" 
PostgreSQL installation should never touch that area.

Looking at FHS, for a moment: http://www.pathname.com/fhs/2.2/

3.7.1 Purpose
/etc contains configuration files and directories that are specific to the 
current system.

3.7.4  Indicates that 

"Host-specific configuration files for add-on application software packages 
must be installed within the directory /etc/opt/<package>, where <package> is 
the name of the subtree in /opt where the static data from that package is 
stored."

3.12 indicates: /opt is reserved for the installation of add-on application 
software packages.

A package to be installed in /opt must locate its static files in a separate 
/opt/<package> directory tree, where <package> is a name that describes the 
software package.

Then comes 5.1, on /var

/var contains variable data files. This includes spool directories and files, 
administrative and logging data, and transient and temporary files.

It would make most sense, based on FHS, for PostgreSQL information to 
assortedly reside in:

- /etc/opt/postgresql or /etc/postgresql, for static config information;
- Binaries could assortedly live in /usr/bin or /opt/postgresql;
- Logs should live in /var/log or /var/log/postgresql;
- Data could assortedly live in /var/lib/postgresql, /var/opt/postgresql;
- PIDs should live in /var/lock or /var/lock/postgresql.

None of these choices should come as any spectacular shock to anyone; there 
are an assortment of sets of bigotry out there surrounding the Proper Purposes 
of /opt and /usr/local, and there's probably enough wriggle room there to 
avoid overly enraging anyone that (for instance) felt calling a directory 
"/opt" would make someone deserving of carpet bombing by B-52s.

Interestingly, the Debian install of PostgreSQL somewhat resembles this, with, 
assortedly:

/etc/postgresql
/etc/postgresql/postgresql.conf
/etc/postgresql/postmaster.conf
/etc/postgresql/pg_hba.conf
/etc/postgresql/pg_ident.conf
/etc/init.d/postgresql
/usr/share/doc/postgresql
/usr/share/man/man1/pg_ctl.1.gz
/usr/lib/postgresql
/usr/lib/postgresql/bin/postgres
/usr/lib/postgresql/bin/enable_lang
/usr/lib/postgresql/bin/initdb
/usr/lib/postgresql/bin/initlocation
/usr/lib/postgresql/bin/ipcclean
/usr/lib/postgresql/bin/pg_ctl
/usr/lib/postgresql/bin/pg_dumpall
/var/run/postgresql

(This is obviously incomplete; this just gives the flavor that there are files 
in a reasonably rational but diverse assortment of places.)

Note that the server software hides in /usr/lib/postgresql/bin; it's not stuff 
you should normally run from the command line, so, quel surprise, it is 
stashed somewhere that's unlikely to be in your $PATH.

Stashing _everything_ in /var/lib/postgres would seem a tad surprising.

Mind you, if I need to manage 4 instances on one box, I might very well 
install several instances some place custom, say /opt/postgres, or similar, 
and in that case, it's probably preferable for everything to reside clearly 
underneath that, and for my custom backup scripts to work in that area.

But if I'm managing 4 instances on one box, it should be quite evident that 
I'm going well beyond what any packaging system is likely to be prepared to 
handle.  Again, quel surprise.
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://www3.sympatico.ca/cbbrowne/linuxxian.html
"Of  _course_ it's the murder weapon.   Who would frame someone with a
fake?"



Thu, 13 Feb 2003 21:45:47 -0500
cbbrowne@cbbrowne.com
Re: location of the configuration files 
On Thu, 13 Feb 2003, scott.marlowe wrote:

> If I do a .tar.gz install of apache, I get /usr/local/apache/conf, which
> is not the standard way you're listing.

I'm going to stay out of this argument from now on, but this struck a sore
point.

/usr is designed to be a filesystem that can be shared. Is the stuff in
/usr/local/apache/conf really supposed to be shared amongst all machines
of that architecture on your site that run apache?

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC

Fri, 14 Feb 2003 11:46:39 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: location of the configuration files
On Thursday 13 February 2003 21:13, Bruce Momjian wrote:
> Lamar Owen wrote:
> > It isn't without precedent to have a directory under /var/run.  Maybe
> > /var/run/postgresql.  Under this one could have a uniquely named pid
> > file.

> But how do you handle the default then, where you have postmaster.pid in
> /data?  Do we rename it to postmaster.pid.5432 so it can sit in
> /var/run/postgresql alone with other backends?

Well, you can have the default as 'postmaster.pid' if it wasn't named.  But 
more thought is needed. I'll have to admit; the wisdom of AOLserver having a 
full-fledged tcl config script is beginning to look better and better.

> Another issue is that pg_ctl looks at that file, so moving it around is
> going to be tricky.

pg_ctl could be interesting.

> I am now wondering if we even want pg_hba_dir and pg_ident_dir.  Seems
> we can assume they are in the same directory as postgresql.conf.  That
> leaves only data_dir as new for postgresql.conf.

Ok, if we're going this far already, tell me exactly why we have three config 
files.  Why not really Unify things and fulfil the full promise of Grand 
Unified Configuration by rolling hba and ident into postgresql.conf.  Is 
there a compelling reason not to do so?  The structure of that configuration 
data would have to change, for sure.  Although I seem to remember this being 
suggested once before, but my mind draws a blank trying to recall it.  Just a 
suggestion; maybe not even a good one, but something that crossed my mind.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 13 Feb 2003 21:47:19 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
Oliver Elphick <olly@lfix.co.uk> writes:
> I'm not entirely sure why SE Linux has a problem, seeing that postgres
> needs read-write access to all the files in $PGDATA, but assuming the
> need is verified, I could do this by moving the pid file from
> $PGDATA/postmaster.pid to /var/run/postgresql/5432.pid and similarly for
> other ports.  This would also have the benefit of being more FHS
> compliant  What do people think about that?

No chance at all.  Breaking the connection between the data directory
and the postmaster.pid file means we don't have an interlock against
starting two postmasters in the same data directory.

I do not see the argument for moving the pid file anyway.  Surely no
one's going to tell us that the postmaster shouldn't have write access
to the data directory?

			regards, tom lane

Thu, 13 Feb 2003 21:49:16 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
On Thursday 13 February 2003 21:49, Tom Lane wrote:
> Oliver Elphick <olly@lfix.co.uk> writes:
> > need is verified, I could do this by moving the pid file from
> > $PGDATA/postmaster.pid to /var/run/postgresql/5432.pid and similarly for
> > other ports.  This would also have the benefit of being more FHS
> > compliant  What do people think about that?

> No chance at all.  Breaking the connection between the data directory
> and the postmaster.pid file means we don't have an interlock against
> starting two postmasters in the same data directory.

It's not a pid file in the /var/run sense, really.  It's an interlock for 
PGDATA.  So it might be argued that postmaster.pid is misnamed.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Thu, 13 Feb 2003 21:59:25 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
Lamar Owen wrote:
> > I am now wondering if we even want pg_hba_dir and pg_ident_dir.  Seems
> > we can assume they are in the same directory as postgresql.conf.  That
> > leaves only data_dir as new for postgresql.conf.
> 
> Ok, if we're going this far already, tell me exactly why we have three config 
> files.  Why not really Unify things and fulfil the full promise of Grand 
> Unified Configuration by rolling hba and ident into postgresql.conf.  Is 
> there a compelling reason not to do so?  The structure of that configuration 
> data would have to change, for sure.  Although I seem to remember this being 
> suggested once before, but my mind draws a blank trying to recall it.  Just a 
> suggestion; maybe not even a good one, but something that crossed my mind.

postgresql.conf is var=val, while the others are column-based.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 13 Feb 2003 22:11:05 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Lamar Owen <lamar.owen@wgcr.org> writes:
> Ok, if we're going this far already, tell me exactly why we have three config
> files.  Why not really Unify things and fulfil the full promise of Grand 
> Unified Configuration by rolling hba and ident into postgresql.conf.  Is 
> there a compelling reason not to do so?

Lack of backwards compatibility; unnecessary complexity.  Unifying those
files would create a big headache in terms of having to unify their
syntax.  And there are some basic semantic differences too.  For
instance, order matters in pg_hba.conf, but not in postgresql.conf.

Another reason not to do it is that there are differences in the
security requirements of these files.  postgresql.conf probably doesn't
contain anything that needs to be hidden from prying eyes, but I'd be
inclined to want to keep the other two mode 600.

---

Okay, I've been laying low all day, but here are my thoughts on the
discussion:

I do see the value in being able to (as opposed to being forced to,
please) keep hand-edited config files in a separate location from
the machine-processed data files.  We have already gone some distance
in that direction over the past few releases --- there's much less in
the top $PGDATA directory than there once was.  It makes sense to let
people keep hand-edited files away from what initdb will overwrite.

I would favor a setup that allows a -C *directory* (not file) to be
specified as a postmaster parameter separately from the -D directory;
then the hand-editable config files would be sought in -C not -D.  In
the absence of -C the config files should be sought in -D, same as they
ever were (thus simplifying life for people like me who run many
postmasters and don't give a darn about FHS ;-)).

I don't see any great value in a separate postgresql.conf parameter for
each secondary config file; that just means clutter to me, especially
if we add more such files in future.  I am also distinctly not in favor
of eliminating the PGDATA environment variable; that reads to me as
"we are going to force you to do it our way rather than the way you've
always done it, even if you like the old way".

To make the RPM packagers happy, I guess that the default -C directory
has to be settable via configure.  We do not currently have a default
-D directory, and I didn't hear anyone arguing in favor of adding one.
So that leaves the following possible combinations that the postmaster
might see at startup, for which I propose the following behaviors:

1. No -C switch, no -D switch, no PGDATA found in environment: seek
postgresql.conf in the default -C directory established at configure
time.  Use the 'datadir' specified therein as -D.  Fail if postgresql.conf
doesn't define a datadir value.

2. No -C switch, no -D switch, PGDATA found in environment: use $PGDATA
as both -C and -D.  (Minor detail: if the postgresql.conf in the $PGDATA
directory specifies a different directory as datadir, do we follow that
or raise an error?  I'd be inclined to say "follow it" but maybe there
is an argument for erroring out.)

(In all the following cases, any environment PGDATA value is ignored.)

3. No -C switch, -D switch on command line: use -D value as both -C and -D,
proceed as in case 2.

4. -C switch, no -D switch on command line: seek postgresql.conf in
-C directory, use the datadir it specifies.

5. -C and -D on command line: seek postgresql.conf in -C directory,
use -D as datadir overriding what is in postgresql.conf (this is just
the usual rule that command line switches override postgresql.conf).

Cases 2 and 3 are backwards-compatible with our historical behavior,
so that anyone who likes the historical behavior will not be unhappy.
Cases 1 and 4 I think will make mlw and our packagers happy.  Case 5
is just the logical conclusion for that combination.

In all cases, pg_hba.conf and pg_ident.conf would be sought in the
same directory as postgresql.conf.  The other stuff in the toplevel
$PGDATA directory should stay where it is, IMHO.

I would venture that the configure-time-default for -C should be
${prefixdir}/etc if configure is not told differently, while the
packagers would probably set it to /etc/postgresql/ (ie, the
config files should live in a subdirectory that can be owned by
postgres user).  I'm not wedded to that though.

Another interesting question is whether the installed-by-default
postgresql.conf should specify a datadir value, and if so what.
If initdb installs it, it can and probably should insert the actual
datadir location the user gave to initdb into the file.  But should
initdb install any config files at all anymore?  I'm leaning to the
thought that initdb should store default config files into $PGDATA
same as it ever did, and then it's up to the user (or package install
scripts) to move them to the desired -C directory if appropriate.
Or I suppose we could add a -C parameter to initdb to tell it where to
put 'em.

Comments?

			regards, tom lane

Thu, 13 Feb 2003 23:00:02 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Tom Lane wrote:
> I don't see any great value in a separate postgresql.conf parameter for
> each secondary config file; that just means clutter to me, especially
> if we add more such files in future.  I am also distinctly not in favor
> of eliminating the PGDATA environment variable; that reads to me as
> "we are going to force you to do it our way rather than the way you've
> always done it, even if you like the old way".

The scripts needn't ignore PGDATA at all.  Only postmaster.  Since the
vast majority of people start the postmaster from a script, this winds
up being a minor issue, except for the fact that without PGDATA
administrators will be able to count on looking at the output of 'ps'
to determine where the postmaster is looking for either the config
file or the data directory.  In other words, they'll have somewhere to
start from without having to poke through scripts that might not even
have been used (what happens when a user defines PGDATA and starts a
postmaster?  The administrator will have to go to more extreme
lengths, like using lsof, to figure out where the data directory is.
Not all systems have such tools).

> Comments?

I agree with your assessment for the most part, except for PGDATA.
There's no good reason I can think of for the postmaster to look at
it.  It's fine if it sets it for processes it forks to inherit, but it
shouldn't pay attention to it on startup.  Some people might complain,
but there's little difference in doing a "postmaster -D $PGDATA" and
just "postmaster", and people who are starting things by hand
hopefully aren't so inflexible as to demand that PGDATA remain treated
as-is.  People who really care can create a simple little 'pm.sh'
script that simply does a "postmaster -D $PGDATA", which will save
them typing even over just doing a "postmaster" from the command line.



-- 
Kevin Brown					      kevin@sysexperts.com

Thu, 13 Feb 2003 21:59:42 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
On Thu, 2003-02-13 at 23:06, mlw wrote:
> 
> Bruce Momjian wrote:

> > Can non-root write to /var/run?
> > 
> >   
> Shouldn't be able too

But it should be able to write under /var/run/postgresql, which the
distribution will set up with the correct permissions.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "God be merciful unto us, and bless us; and cause his 
      face to shine upon us."          Psalms 67:1 


14 Feb 2003 06:17:43 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files
On Thu, 2003-02-13 at 22:53, Bruce Momjian wrote:
> Oliver Elphick wrote:
> > What your comments strongly suggest to me is that projects like
> > PostgreSQL and pine, along with everything else, should comply with FHS;
> > then there will be no confusion because everyone will be following the
> > smae standards.  Messes arise when people ignore standards; we have all
> > seen the dreadful examples of MySQL and the Beast, haven't we?
> 
> Can the FHS handle installing PostgreSQL as non-root?

Certainly.  It is only necessary to set permissions correctly in
/etc/postgresql and /var/run/postgresql.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "God be merciful unto us, and bless us; and cause his 
      face to shine upon us."          Psalms 67:1 


14 Feb 2003 06:18:22 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files
On Fri, 2003-02-14 at 02:49, Tom Lane wrote:
> Oliver Elphick <olly@lfix.co.uk> writes:
> > I'm not entirely sure why SE Linux has a problem, seeing that postgres
> > needs read-write access to all the files in $PGDATA, but assuming the
> > need is verified, I could do this by moving the pid file from
> > $PGDATA/postmaster.pid to /var/run/postgresql/5432.pid and similarly for
> > other ports.  This would also have the benefit of being more FHS
> > compliant  What do people think about that?
> 
> No chance at all.  Breaking the connection between the data directory
> and the postmaster.pid file means we don't have an interlock against
> starting two postmasters in the same data directory.

Yes; that would take a lot of effort to get round. Not worth it, I
think.

> I do not see the argument for moving the pid file anyway.  Surely no
> one's going to tell us that the postmaster shouldn't have write access
> to the data directory?

I'm waiting for a response on that one; I don't understand it either.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "God be merciful unto us, and bless us; and cause his 
      face to shine upon us."          Psalms 67:1 


14 Feb 2003 06:26:12 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files
Kevin Brown <kevin@sysexperts.com> writes:
> I agree with your assessment for the most part, except for PGDATA.
> There's no good reason I can think of for the postmaster to look at
> it.

The other side of that coin is, what's the good reason to remove it?
There's a long way between "I don't want my setup to depend on PGDATA"
and "I don't think your setup should be allowed to depend on PGDATA".
If you don't want to use it, then don't use it.  Why do you need to
tell me how I'm allowed to run my installation?

> ... people who are starting things by hand hopefully aren't so
> inflexible as to demand that PGDATA remain treated as-is.

Yes, I could reconfigure my scripts to not depend on this.  You have
not given me an adequate argument why I should have to.

			regards, tom lane

Fri, 14 Feb 2003 01:30:21 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
On Fri, 2003-02-14 at 02:45, cbbrowne@cbbrowne.com wrote:
> 3.7.1 Purpose
> /etc contains configuration files and directories that are specific to the 
> current system.
> 
> 3.7.4  Indicates that 
> 
> "Host-specific configuration files for add-on application software packages 
> must be installed within the directory /etc/opt/<package>, where <package> is 
> the name of the subtree in /opt where the static data from that package is 
> stored."
> 
> 3.12 indicates: /opt is reserved for the installation of add-on application 
> software packages.
> 
> A package to be installed in /opt must locate its static files in a separate 
> /opt/<package> directory tree, where <package> is a name that describes the 
> software package.
...
> It would make most sense, based on FHS, for PostgreSQL information to 
> assortedly reside in:
> 
> - /etc/opt/postgresql or /etc/postgresql, for static config information;

I feel that /opt (and therefore /etc/opt) are intended for the use of
vendors; so commercial packages designed to fit in with FHS should use
those.  I don't think they are for locally built stuff.

No matter; it illustrates the main point, which is that these things
should be easily configurable.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "God be merciful unto us, and bless us; and cause his 
      face to shine upon us."          Psalms 67:1 


14 Feb 2003 06:37:30 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files
Apache explicitly supports a number of different layouts for files out of
the box (and provides support for you to roll your own very easily). From
the manual:

The second, and more flexible way to configure the install path locations
for Apache is using the config.layout file. Using this method, it is
possible to separately specify the location for each type of file within the
Apache installation. The config.layout file contains several example
configurations, and you can also create your own custom configuration
following the examples. The different layouts in this file are grouped into
<Layout FOO>...</Layout> sections and referred to by name as in FOO.
  --enable-layout=LAYOUT
  Use the named layout in the config.layout file to specify the installation
paths.
Maybe pg could benefit from something similar?

cheers

andrew

----- Original Message -----
From: "scott.marlowe" <scott.marlowe@ihs.com>
Sent: Thursday, February 13, 2003 4:07 PM
[snip]
> If I do a .tar.gz install of apache, I get /usr/local/apache/conf, which
> is not the standard way you're listing.  If I install openldap from
> .tar.gz, I get a /usr/local/etc/openldap directory, close, but still not
> the same.  The fact is, it's the packagers that put things into /etc and
> whatnot, and I can see the postgresql RPMs or debs or whatever having that
> as the default, but for custom built software, NOTHING that I know of
> builds from source and uses /etc without a switch to tell it to, just like
> postgresql can do now.


Fri, 14 Feb 2003 02:35:03 -0500
"Andrew Dunstan" <andrew@dunslane.net>
Re: location of the configuration files
Tom Lane wrote:
> Kevin Brown <kevin@sysexperts.com> writes:
> > I agree with your assessment for the most part, except for PGDATA.
> > There's no good reason I can think of for the postmaster to look at
> > it.
> 
> The other side of that coin is, what's the good reason to remove it?
> There's a long way between "I don't want my setup to depend on PGDATA"
> and "I don't think your setup should be allowed to depend on PGDATA".
> If you don't want to use it, then don't use it.  Why do you need to
> tell me how I'm allowed to run my installation?

I'm not talking about getting rid of ALL dependency on PGDATA in our
entire distribution, only postmaster's.

Recall that the main purpose of making any of these changes at all is
to make life easier for the guys who have to manage the systems that
will be running PostgreSQL.  Agreed?

So: imagine you're the newly-hired DBA and your boss points you to the
system and says "administrate the database on that".  You go over to
the computer and start looking around.

You do a "ps" and see a postmaster process running.  You know that
it's the process that is listening for connections.  The "ps" listing
only says "/usr/bin/postmaster".  No arguments to clue you in,
nothing.  Where do you look to figure out where the data is?  How do
you figure out what port it's listening on?

Well, we're already agreed on how to deal with that question: you look
in /etc/postgresql, and because this is a relatively new install (and
the PostgreSQL maintainers, who are very wise and benevolent, made
that the default location for configs :-), it has a postgresql.conf
file with a line that says "data_directory = /var/lib/pgsql".  It
doesn't mention a port to listen to so you know that it's listening on
port 5432.  As a DBA, you're all set.

Now let's repeat that scenario, except that instead of seeing one
postmaster process, you see five.  And they all say
"/usr/bin/postmaster" in the "ps" listing.  No arguments to clue you
in or anything, as before.  You might be able to figure out where one
of them is going by looking at /etc/postgresql, but what about the
rest?  Now you're stuck unless you want to do a "find" (time consuming
and I/O intensive -- a good way to slow the production database down a
bit), or you're knowledgeable enough to use 'lsof' or black magic like
digging into kernel memory to figure out where the config files and
data directories are, or you have enough knowledge to pore through the
startup scripts and understand what they're doing.

Lest you think that this is an unlikely scenario, keep in mind that
most startup scripts, including pg_ctl, currently start the postmaster
without arguments and rely on PGDATA, so a shop that hasn't already
been bitten by this *will* be.  Right now shops that wish to avoid the
trap I described have to go to *extra* lengths: they have to make
exactly the same kinds of changes to the scripts that I'm talking
about us making (putting an explicit '-D "$PGDATA"' where none now
exists) or they have to resort to tricks like renaming the postmaster
executable and creating a shell script in its place that will invoke
the (renamed) postmaster with '-D "$PGDATA"'.

It's not so bad if only a few shops have to make those changes.  But
what if it's thousands?  Yeah, the distribution guys can patch the
scripts to do this, but why should they have to?  They, and the shops
that run PostgreSQL, are our customers.


All of that is made possible because the postmaster can use an
inherited PGDATA for the location of the config files and (if the
config files don't say differently in our new scheme) the data
directory, and pg_ctl takes advantage of that fact (as do most startup
scripts that I've seen, that don't just invoke pg_ctl).

I'm not arguing that we should remove the use of PGDATA *everywhere*,
only in postmaster (and then, only postmaster's use of an *inherited*
PGDATA.  It should still set PGDATA so its children can use it).  It
means changing pg_ctl and the startup scripts we ship.  The earlier we
make these changes, the less overall pain there will be in the long
run.


> > ... people who are starting things by hand hopefully aren't so
> > inflexible as to demand that PGDATA remain treated as-is.
> 
> Yes, I could reconfigure my scripts to not depend on this.  You have
> not given me an adequate argument why I should have to.

[By this I'm assuming you're referring to the scripts you use for
testing, and not the ones that ship with the distribution]

I'm not arguing that you should get rid of all the references to
PGDATA in your scripts or anything crazy like that.  The changes I'm
talking about are minor: where you see "postmaster" without any "-D"
arguments, you simply add '-D "$PGDATA"' to it, before any other
arguments that you might also be passing.  That's it.  Nothing else
should be needed.

The reason for removing postmaster's use of an inherited PGDATA is the
same as the reason for making the other changes we already agree
should be made: to make things easier for the guys in the field who
have to manage production systems.



-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 14 Feb 2003 02:58:49 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Kevin Brown wrote:
> Tom Lane wrote:
> > Kevin Brown <kevin@sysexperts.com> writes:
> > > I agree with your assessment for the most part, except for PGDATA.
> > > There's no good reason I can think of for the postmaster to look at
> > > it.
> > 
> > The other side of that coin is, what's the good reason to remove it?
> > There's a long way between "I don't want my setup to depend on PGDATA"
> > and "I don't think your setup should be allowed to depend on PGDATA".
> > If you don't want to use it, then don't use it.  Why do you need to
> > tell me how I'm allowed to run my installation?
> 
> I'm not talking about getting rid of ALL dependency on PGDATA in our
> entire distribution, only postmaster's.
> 
> Recall that the main purpose of making any of these changes at all is
> to make life easier for the guys who have to manage the systems that
> will be running PostgreSQL.  Agreed?
> 
> So: imagine you're the newly-hired DBA and your boss points you to the
> system and says "administrate the database on that".  You go over to
> the computer and start looking around.
> 
> You do a "ps" and see a postmaster process running.  You know that
> it's the process that is listening for connections.  The "ps" listing
> only says "/usr/bin/postmaster".  No arguments to clue you in,
> nothing.  Where do you look to figure out where the data is?  How do
> you figure out what port it's listening on?

If you want ps to display the data dir, you should use -D.  Remember, it
is mostly important for multiple postmaster, so if you are doing that,
just use -D, but don't prevent single-postmaster folks from using
PGDATA.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 07:17:17 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files

OK, here is an updated proposal.  I think we have decided:

	Moving postmaster.pid and postmaster.opts isn't worth it.

	We don't want per-file GUC variables, but assume it is in
	the same config directory as postgresql.conf.  I don't
	see any valid reason they would want to put them somewhere
	different than postgresql.conf.

So, we add data_dir to postgresql.conf, and add -C/PGCONFIG to
postmaster.

Regarding Tom's idea of replacing data_dir with a full path during
initdb, I think we are better having it be relative to the config
directory, that way if they move pgsql/, the system still works.
However, if the config directory is in a different lead directory path,
we should replace it with the full path, e.g. /usr/local/pgsql/data and
/usr/local/pgsql/etc use relative paths, ../data, while /etc/postgresql
and /usr/local/pgsql/data get an absolute path.

My idea is to introduce the above capabilities in 7.4, and keep the
config files in /data.  This will allow package folks to move the config
files in 7.4.

I also think we should start telling people to use PGCONFIG rather than
PGDATA.  Then, in 7.5, we move the default config file location to
pgsql/etc, and tell folks to point there rather than /data.  I think
there is major value to getting those config files out of the initdb
creation area and for backups.

I am now wondering if we should add PGCONFIG and move them out of data
all in the same release.  Not sure if delaying the split is valuable.

---------------------------------------------------------------------------

Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > Ok, if we're going this far already, tell me exactly why we have three config
> > files.  Why not really Unify things and fulfil the full promise of Grand 
> > Unified Configuration by rolling hba and ident into postgresql.conf.  Is 
> > there a compelling reason not to do so?
> 
> Lack of backwards compatibility; unnecessary complexity.  Unifying those
> files would create a big headache in terms of having to unify their
> syntax.  And there are some basic semantic differences too.  For
> instance, order matters in pg_hba.conf, but not in postgresql.conf.
> 
> Another reason not to do it is that there are differences in the
> security requirements of these files.  postgresql.conf probably doesn't
> contain anything that needs to be hidden from prying eyes, but I'd be
> inclined to want to keep the other two mode 600.
> 
> ---
> 
> Okay, I've been laying low all day, but here are my thoughts on the
> discussion:
> 
> I do see the value in being able to (as opposed to being forced to,
> please) keep hand-edited config files in a separate location from
> the machine-processed data files.  We have already gone some distance
> in that direction over the past few releases --- there's much less in
> the top $PGDATA directory than there once was.  It makes sense to let
> people keep hand-edited files away from what initdb will overwrite.
> 
> I would favor a setup that allows a -C *directory* (not file) to be
> specified as a postmaster parameter separately from the -D directory;
> then the hand-editable config files would be sought in -C not -D.  In
> the absence of -C the config files should be sought in -D, same as they
> ever were (thus simplifying life for people like me who run many
> postmasters and don't give a darn about FHS ;-)).
> 
> I don't see any great value in a separate postgresql.conf parameter for
> each secondary config file; that just means clutter to me, especially
> if we add more such files in future.  I am also distinctly not in favor
> of eliminating the PGDATA environment variable; that reads to me as
> "we are going to force you to do it our way rather than the way you've
> always done it, even if you like the old way".
> 
> To make the RPM packagers happy, I guess that the default -C directory
> has to be settable via configure.  We do not currently have a default
> -D directory, and I didn't hear anyone arguing in favor of adding one.
> So that leaves the following possible combinations that the postmaster
> might see at startup, for which I propose the following behaviors:
> 
> 1. No -C switch, no -D switch, no PGDATA found in environment: seek
> postgresql.conf in the default -C directory established at configure
> time.  Use the 'datadir' specified therein as -D.  Fail if postgresql.conf
> doesn't define a datadir value.
> 
> 2. No -C switch, no -D switch, PGDATA found in environment: use $PGDATA
> as both -C and -D.  (Minor detail: if the postgresql.conf in the $PGDATA
> directory specifies a different directory as datadir, do we follow that
> or raise an error?  I'd be inclined to say "follow it" but maybe there
> is an argument for erroring out.)
> 
> (In all the following cases, any environment PGDATA value is ignored.)
> 
> 3. No -C switch, -D switch on command line: use -D value as both -C and -D,
> proceed as in case 2.
> 
> 4. -C switch, no -D switch on command line: seek postgresql.conf in
> -C directory, use the datadir it specifies.
> 
> 5. -C and -D on command line: seek postgresql.conf in -C directory,
> use -D as datadir overriding what is in postgresql.conf (this is just
> the usual rule that command line switches override postgresql.conf).
> 
> Cases 2 and 3 are backwards-compatible with our historical behavior,
> so that anyone who likes the historical behavior will not be unhappy.
> Cases 1 and 4 I think will make mlw and our packagers happy.  Case 5
> is just the logical conclusion for that combination.
> 
> In all cases, pg_hba.conf and pg_ident.conf would be sought in the
> same directory as postgresql.conf.  The other stuff in the toplevel
> $PGDATA directory should stay where it is, IMHO.
> 
> I would venture that the configure-time-default for -C should be
> ${prefixdir}/etc if configure is not told differently, while the
> packagers would probably set it to /etc/postgresql/ (ie, the
> config files should live in a subdirectory that can be owned by
> postgres user).  I'm not wedded to that though.
> 
> Another interesting question is whether the installed-by-default
> postgresql.conf should specify a datadir value, and if so what.
> If initdb installs it, it can and probably should insert the actual
> datadir location the user gave to initdb into the file.  But should
> initdb install any config files at all anymore?  I'm leaning to the
> thought that initdb should store default config files into $PGDATA
> same as it ever did, and then it's up to the user (or package install
> scripts) to move them to the desired -C directory if appropriate.
> Or I suppose we could add a -C parameter to initdb to tell it where to
> put 'em.
> 
> Comments?
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 07:32:31 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
In reference to determining what port postgres or any program is listening on
On my Redhat Linux machines
netstat --inet -nlp
when run as root
produces a nice list of all programs listening on the network with IP and port 
number the process is listening on, the name of the process and the pid.

The environment used to start each of these postmasters can be found at
cat /proc/${POSTMASTER-PID}/environ | tr "\000" "\n"

I'm not arguing one way or the other on your issue, just hope these tips make 
the "black magic" a little easier to use.

On Friday 14 February 2003 04:58 am, Kevin Brown wrote:
> Now let's repeat that scenario, except that instead of seeing one
> postmaster process, you see five.  And they all say
> "/usr/bin/postmaster" in the "ps" listing.  No arguments to clue you
> in or anything, as before.  You might be able to figure out where one
> of them is going by looking at /etc/postgresql, but what about the
> rest?  Now you're stuck unless you want to do a "find" (time consuming
> and I/O intensive -- a good way to slow the production database down a
> bit), or you're knowledgeable enough to use 'lsof' or black magic like
> digging into kernel memory to figure out where the config files and
> data directories are, or you have enough knowledge to pore through the
> startup scripts and understand what they're doing.


Fri, 14 Feb 2003 06:32:37 -0600
David Walker <pgsql@grax.com>
Re: [MLIST]  Re: location of the configuration files
On Fri, 2003-02-14 at 12:17, Bruce Momjian wrote:
> If you want ps to display the data dir, you should use -D.  Remember, it
> is mostly important for multiple postmaster, so if you are doing that,
> just use -D, but don't prevent single-postmaster folks from using
> PGDATA.

Could not the ps line be rewritten to show this, as the backend's ps
lines are rewritten?

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "God be merciful unto us, and bless us; and cause his 
      face to shine upon us."          Psalms 67:1 


14 Feb 2003 13:11:58 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files

I am not sure if it is a good idea to be mucking with it.  For backend,
we do the entire thing, so it is clear we modified something.

---------------------------------------------------------------------------

Oliver Elphick wrote:
> On Fri, 2003-02-14 at 12:17, Bruce Momjian wrote:
> > If you want ps to display the data dir, you should use -D.  Remember, it
> > is mostly important for multiple postmaster, so if you are doing that,
> > just use -D, but don't prevent single-postmaster folks from using
> > PGDATA.
> 
> Could not the ps line be rewritten to show this, as the backend's ps
> lines are rewritten?
> 
> -- 
> Oliver Elphick                                Oliver.Elphick@lfix.co.uk
> Isle of Wight, UK                             http://www.lfix.co.uk/oliver
> GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
>                  ========================================
>      "God be merciful unto us, and bless us; and cause his 
>       face to shine upon us."          Psalms 67:1 
> 
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 08:29:29 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Bruce Momjian wrote:
> If you want ps to display the data dir, you should use -D.  Remember, it
> is mostly important for multiple postmaster, so if you are doing that,
> just use -D, but don't prevent single-postmaster folks from using
> PGDATA.

Perhaps the best compromise would be to change pg_ctl so that it uses
-D explicitly when invoking postmaster.  That's an easy change.

Could you describe how you and other developers use PGDATA?  I'm quite
interested in knowing why there seems to be so much resistance to
removing the "potential_DataDir = getenv("PGDATA");" line from
postmaster.c.



-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 14 Feb 2003 05:41:48 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Bruce Momjian wrote:
> If you want ps to display the data dir, you should use -D.  Remember, it
> is mostly important for multiple postmaster, so if you are doing that,
> just use -D, but don't prevent single-postmaster folks from using
> PGDATA.

Perhaps another reasonable approach would be to put an #ifdef/#endif
around the "potential_DataDir = getenv("PGDATA");" line in postmater.c
and create a configure option to enable it.  That way you guys get the
behavior you want for testing but production builds could disable it
if that's viewed as desirable.  You'd want to make the error message
that's produced when no data directory is specified depend on the same
#ifdef variable, of course.

Then the group would get to fight it out over whether the configure
default should be "enable" or "disable".  :-)


-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 14 Feb 2003 05:47:37 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Kevin Brown wrote:
> Bruce Momjian wrote:
> > If you want ps to display the data dir, you should use -D.  Remember, it
> > is mostly important for multiple postmaster, so if you are doing that,
> > just use -D, but don't prevent single-postmaster folks from using
> > PGDATA.
> 
> Perhaps the best compromise would be to change pg_ctl so that it uses
> -D explicitly when invoking postmaster.  That's an easy change.
> 
> Could you describe how you and other developers use PGDATA?  I'm quite
> interested in knowing why there seems to be so much resistance to
> removing the "potential_DataDir = getenv("PGDATA");" line from
> postmaster.c.

I just set PGDATA in my login and I don't have to deal with it again.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 09:00:44 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files

> > If you are interested in reading a contrary position, you can read
> > Berstein's arguments for his recommended way to install services at:
> > http://cr.yp.to/unix.html

But since DJB is a class-A monomaniac, he may not be the best person to
listen to. /var/qmail/control for qmail configuration files? Yeah, good
one, DJB.

-- 
Martin Coxall <coxall@cream.org>


14 Feb 2003 14:07:09 +0000
Martin Coxall <coxall@cream.org>
Re: location of the configuration files

> Generally things that live in /etc are owned and operated by the OS.  
> Postgresql, by it's definition is a userspace program, not an OS owned 
> one.

Partially true. The FHS specifies that the /etc top layer is for system-own3d 
stuff, but the subdirectories off it are explicitly used for user space programs
and, well, everything. (/etc/apache, /etc/postgres, /etc/tomcat3, /etc/tomcat4...)

Martin Coxall


14 Feb 2003 14:07:11 +0000
Martin Coxall <coxall@cream.org>
Re: location of the configuration files
On Thu, 2003-02-13 at 20:28, Steve Crawford wrote:
> I don't see why we can't keep everyone happy and let the users choose the 
> setup they want. To wit, make the following, probably simple, changes:
> 
> 1) Have postgresql default to using /etc/postgresql.conf

/etc/postgres/postgresql.conf, if we want to be proper FHS-bitches.

> 2) Add a setting in postgresql.conf specifying the data directory
> 3) Change the meaning of -D to mean "use this config file"
> 4) In the absence of a specified data directory in postgresql.conf, use the 
> location of the postgresql.conf file as the data directory

Shouldn't it in that case default to, say /var/lib/postgres?

-- 
Martin Coxall <coxall@cream.org>


14 Feb 2003 14:07:17 +0000
Martin Coxall <coxall@cream.org>
Re: location of the configuration files
Bruce Momjian wrote:
> I just set PGDATA in my login and I don't have to deal with it
> again.

Hmm...you don't use pg_ctl to start/stop/whatever the database?  You
invoke the postmaster directly (I can easily see that you would, just
asking if you do)?



-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 14 Feb 2003 06:12:38 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Kevin Brown wrote:
> Bruce Momjian wrote:
> > I just set PGDATA in my login and I don't have to deal with it
> > again.
> 
> Hmm...you don't use pg_ctl to start/stop/whatever the database?  You
> invoke the postmaster directly (I can easily see that you would, just
> asking if you do)?

I can use either to start/stop it.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 09:15:58 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Bruce Momjian wrote:
> I just set PGDATA in my login and I don't have to deal with it
> again.

Duh....I just realized a reason you guys might care about this so
much.

It's because you want to be able to start the postmaster from within a
debugger (or profiler, or whatever), and you don't want to have to
mess with command line options from there, right?


Sounds like fixing pg_ctl to use -D explicitly when invoking the
postmaster is the right change to make here, since that's probably how
the majority of the production shops are going to be starting the
database anyway.  Takes care of the majority of the "visibility"
problem and leaves PGDATA intact.  Thoughts?



-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 14 Feb 2003 06:19:59 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
On 14 Feb 2003, Martin Coxall wrote:

>
> > > If you are interested in reading a contrary position, you can read
> > > Berstein's arguments for his recommended way to install services at:
> > > http://cr.yp.to/unix.html
>
> But since DJB is a class-A monomaniac, he may not be the best person to
> listen to. /var/qmail/control for qmail configuration files? Yeah, good
> one, DJB.

I'm guessing that rather than reading it the above mentioned link you
chose to waste our time with this instead.  Good one, MC.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Fri, 14 Feb 2003 09:21:16 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: location of the configuration files

Give it up.  As long as we have -D, we will allow PGDATA. If you don't
want to use it, don't use it.

---------------------------------------------------------------------------

Kevin Brown wrote:
> Bruce Momjian wrote:
> > I just set PGDATA in my login and I don't have to deal with it
> > again.
> 
> Duh....I just realized a reason you guys might care about this so
> much.
> 
> It's because you want to be able to start the postmaster from within a
> debugger (or profiler, or whatever), and you don't want to have to
> mess with command line options from there, right?
> 
> 
> Sounds like fixing pg_ctl to use -D explicitly when invoking the
> postmaster is the right change to make here, since that's probably how
> the majority of the production shops are going to be starting the
> database anyway.  Takes care of the majority of the "visibility"
> problem and leaves PGDATA intact.  Thoughts?
> 
> 
> 
> -- 
> Kevin Brown					      kevin@sysexperts.com
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 09:22:26 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files

My point is that folks with multiple postmasters may not want to use
PGDATA, but for folks who have single postmasters, it makes things
easier and less error-prone.

---------------------------------------------------------------------------

Bruce Momjian wrote:
> 
> Give it up.  As long as we have -D, we will allow PGDATA. If you don't
> want to use it, don't use it.
> 
> ---------------------------------------------------------------------------
> 
> Kevin Brown wrote:
> > Bruce Momjian wrote:
> > > I just set PGDATA in my login and I don't have to deal with it
> > > again.
> > 
> > Duh....I just realized a reason you guys might care about this so
> > much.
> > 
> > It's because you want to be able to start the postmaster from within a
> > debugger (or profiler, or whatever), and you don't want to have to
> > mess with command line options from there, right?
> > 
> > 
> > Sounds like fixing pg_ctl to use -D explicitly when invoking the
> > postmaster is the right change to make here, since that's probably how
> > the majority of the production shops are going to be starting the
> > database anyway.  Takes care of the majority of the "visibility"
> > problem and leaves PGDATA intact.  Thoughts?
> > 
> > 
> > 
> > -- 
> > Kevin Brown					      kevin@sysexperts.com
> > 
> > ---------------------------(end of broadcast)---------------------------
> > TIP 2: you can get off all lists at once with the unregister command
> >     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> > 
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 09:29:34 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
On Fri, 2003-02-14 at 14:21, Vince Vielhaber wrote:
> On 14 Feb 2003, Martin Coxall wrote:
> 
> >
> > > > If you are interested in reading a contrary position, you can read
> > > > Berstein's arguments for his recommended way to install services at:
> > > > http://cr.yp.to/unix.html
> >
> > But since DJB is a class-A monomaniac, he may not be the best person to
> > listen to. /var/qmail/control for qmail configuration files? Yeah, good
> > one, DJB.
> 
> I'm guessing that rather than reading it the above mentioned link you
> chose to waste our time with this instead.  Good one, MC.

Yeah, I've read it several times, and have often linked to it as an
example of why one should be wary of DJB's software. It seems to me that
since DJB doesn't follow his own advice regarding the filesystem
hierarchy (see both qmail and djbdns), it'd be odd for him to expect
anyone else to. *Especially* seing as he's a bit mental. ("I'm not going
to take this any more. I demand cross-platform compatibility!")

-- 
Martin Coxall <coxall@cream.org>


14 Feb 2003 14:33:53 +0000
Martin Coxall <coxall@cream.org>
Re: location of the configuration files
Bruce Momjian wrote:
> OK, here is an updated proposal.  I think we have decided:
> 
> 	Moving postmaster.pid and postmaster.opts isn't worth it.
> 
> 	We don't want per-file GUC variables, but assume it is in
> 	the same config directory as postgresql.conf.  I don't
> 	see any valid reason they would want to put them somewhere
> 	different than postgresql.conf.
> 
> So, we add data_dir to postgresql.conf, and add -C/PGCONFIG to
> postmaster.

Agreed.  One additional thing: when pg_ctl invokes the postmaster, it
should explicitly specify -C on the postmaster command line, and if it
doesn't find a data_dir in $PGCONFIG/postgresql.conf then it should
explicitly specify a -D as well.  Pg_ctl is going to have to be
modified to take a -C argument anyway, so we may as well go all the
way to do the right thing here.

This way, people who start the database using the standard tools we
supply will know exactly what's going on when they get a "ps" listing.



-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 14 Feb 2003 06:35:54 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Bruce Momjian wrote:
> 
> Give it up.  As long as we have -D, we will allow PGDATA. If you don't
> want to use it, don't use it.

Agreed.

I'm not sure I see how this diminishes the argument for fixing pg_ctl
so that it passes an explicit -D option to the postmaster when
invoking it...



-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 14 Feb 2003 06:40:11 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Kevin Brown wrote:
> Bruce Momjian wrote:
> > OK, here is an updated proposal.  I think we have decided:
> > 
> > 	Moving postmaster.pid and postmaster.opts isn't worth it.
> > 
> > 	We don't want per-file GUC variables, but assume it is in
> > 	the same config directory as postgresql.conf.  I don't
> > 	see any valid reason they would want to put them somewhere
> > 	different than postgresql.conf.
> > 
> > So, we add data_dir to postgresql.conf, and add -C/PGCONFIG to
> > postmaster.
> 
> Agreed.  One additional thing: when pg_ctl invokes the postmaster, it
> should explicitly specify -C on the postmaster command line, and if it
> doesn't find a data_dir in $PGCONFIG/postgresql.conf then it should
> explicitly specify a -D as well.  Pg_ctl is going to have to be
> modified to take a -C argument anyway, so we may as well go all the
> way to do the right thing here.
> 
> This way, people who start the database using the standard tools we
> supply will know exactly what's going on when they get a "ps" listing.

No.  If you want ps to display, don't use environment variables. Many
don't care --- especially those with only one postmaster.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 09:40:27 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Bruce Momjian wrote:
> > This way, people who start the database using the standard tools we
> > supply will know exactly what's going on when they get a "ps" listing.
> 
> No.  If you want ps to display, don't use environment variables. Many
> don't care --- especially those with only one postmaster.

You know that the code in pg_ctl doesn't send an explicit -D to the
postmaster even if pg_ctl itself is invoked with a -D argument, right?
The only way to make pg_ctl do that is by using the "-o" option.

A typical vendor-supplied install is going to invoke pg_ctl to do the
dirty work.  That's why I'm focusing on pg_ctl.

I completely understand your need for keeping PGDATA in postmaster.  I
don't understand why pg_ctl *shouldn't* be changed to invoke
postmaster with an explicit -D option.  It might be desirable for ps
to not show any arguments to postmaster in some circumstances (I have
no idea what those would be), but why in the world would you want that
to be the *default*?  Why would we want the default behavior to make
things harder on administrators and not easier?


-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 14 Feb 2003 06:58:49 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Kevin Brown <kevin@sysexperts.com> writes:
> Tom Lane wrote:
>> The other side of that coin is, what's the good reason to remove it?
>> There's a long way between "I don't want my setup to depend on PGDATA"
>> and "I don't think your setup should be allowed to depend on PGDATA".
>> If you don't want to use it, then don't use it.  Why do you need to
>> tell me how I'm allowed to run my installation?

> I'm not talking about getting rid of ALL dependency on PGDATA in our
> entire distribution, only postmaster's.

We're obviously talking past each other.  You are arguing that under
circumstances X, Y, or Z, depending on a PGDATA setting is a bad idea.
You are then drawing the conclusion that I should not be allowed to
depend on PGDATA, whether or not I care about X, Y, or Z.

I am happy to design an arrangement that allows you not to depend on
PGDATA if you don't want to.  But I don't see why you need to break
my configuration procedures in order to fix yours.  As I outlined last
night, it's possible to do what you want without breaking backwards
compatibility for those that like PGDATA.

			regards, tom lane

Fri, 14 Feb 2003 10:02:02 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
On 14 Feb 2003, Martin Coxall wrote:

> On Fri, 2003-02-14 at 14:21, Vince Vielhaber wrote:
> > On 14 Feb 2003, Martin Coxall wrote:
> >
> > >
> > > > > If you are interested in reading a contrary position, you can read
> > > > > Berstein's arguments for his recommended way to install services at:
> > > > > http://cr.yp.to/unix.html
> > >
> > > But since DJB is a class-A monomaniac, he may not be the best person to
> > > listen to. /var/qmail/control for qmail configuration files? Yeah, good
> > > one, DJB.
> >
> > I'm guessing that rather than reading it the above mentioned link you
> > chose to waste our time with this instead.  Good one, MC.
>
> Yeah, I've read it several times, and have often linked to it as an
> example of why one should be wary of DJB's software. It seems to me that
> since DJB doesn't follow his own advice regarding the filesystem
> hierarchy (see both qmail and djbdns), it'd be odd for him to expect
> anyone else to. *Especially* seing as he's a bit mental. ("I'm not going
> to take this any more. I demand cross-platform compatibility!")

I seriously doubt your ability to judge anyone's mental stability.
I can also see that you prefer cross-platform INcompatibility.  Your
position and mindset are now crystal clear.

Vince.
-- 
 Fast, inexpensive internet service 56k and beyond!  http://www.pop4.net/
   http://www.meanstreamradio.com       http://www.unknown-artists.com
         Internet radio: It's not file sharing, it's just radio.


Fri, 14 Feb 2003 10:07:32 -0500 (EST)
Vince Vielhaber <vev@michvhf.com>
Re: location of the configuration files
Tom Lane wrote:
> I am happy to design an arrangement that allows you not to depend on
> PGDATA if you don't want to.  But I don't see why you need to break
> my configuration procedures in order to fix yours.  As I outlined last
> night, it's possible to do what you want without breaking backwards
> compatibility for those that like PGDATA.

Yes, I agree.  I hadn't really thought of all the possible benefits of
PGDATA.  Sorry.  :-(

Would you agree that it would be a beneficial change to have pg_ctl
pass explicit arguments to postmaster?  It would go a long way towards
eliminating most of the situations I described.

A warning in the documentation about the consequences of using PGDATA
might not be a bad idea, either...


-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 14 Feb 2003 07:10:03 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> So, we add data_dir to postgresql.conf, and add -C/PGCONFIG to
> postmaster.

Wait one second.  You are blithely throwing in a PGCONFIG variable
without any detailed proposal of exactly how it will work.  Does
that override a PGDATA environment variable?  How do they interact?

Also, please note Kevin Brown's nearby arguments against using PGDATA
at all, which surely apply with equal force to a PGCONFIG variable.
Now, I don't buy that Kevin's arguments are enough reason to break
backwards compatibility by removing PGDATA --- but I think they are
enough reason not to introduce a new environment variable.  PGCONFIG
wouldn't offer any backwards-compatibility value, and that tilts the
scales against it.

> Regarding Tom's idea of replacing data_dir with a full path during
> initdb, I think we are better having it be relative to the config
> directory, that way if they move pgsql/, the system still works.

Good thought, but you're assuming that initdb knows where the config
files will eventually live.  If we do that, then moving the config
files breaks the installation.  I think it will be fairly common to
let initdb drop its proposed config files into $PGDATA, and then
manually place them where they should go (or even more likely,
manually merge them with a prior version).  Probably better to force
datadir to be an absolute path in the config file.  (In fact, on safety
grounds I'd argue in favor of rejecting a datadir value taken from the
config file that wasn't absolute.)

> I also think we should start telling people to use PGCONFIG rather than
> PGDATA.  Then, in 7.5, we move the default config file location to
> pgsql/etc, and tell folks to point there rather than /data.

I agree with none of this.  This is not improvement, this is only change
for the sake of change.  The packagers will do what they want to do
(and are already doing, mostly) regardless.

			regards, tom lane

Fri, 14 Feb 2003 10:20:56 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Oliver Elphick <olly@lfix.co.uk> writes:
> On Fri, 2003-02-14 at 12:17, Bruce Momjian wrote:
>> If you want ps to display the data dir, you should use -D.  Remember, it
>> is mostly important for multiple postmaster, so if you are doing that,
>> just use -D, but don't prevent single-postmaster folks from using
>> PGDATA.

> Could not the ps line be rewritten to show this, as the backend's ps
> lines are rewritten?

I for one would rather it didn't do that.  I already set my postmaster
command lines the way I want 'em, and I don't want the code overriding
that.  (I prefer to use explicit -p arguments to distinguish the various
postmasters I have running --- shorter and easier to read than explicit
-D would be.  At least for me.)

			regards, tom lane

Fri, 14 Feb 2003 10:27:23 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Kevin Brown <kevin@sysexperts.com> writes:
> I'm quite interested in knowing why there seems to be so much resistance to
> removing the "potential_DataDir = getenv("PGDATA");" line from
> postmaster.c.

Backwards compatibility.  Also, you still haven't explained why
"I don't want to use PGDATA" should translate to "no one should
be allowed to use PGDATA".  If you don't set PGDATA, what problem
is there for you in that line being there?

			regards, tom lane

Fri, 14 Feb 2003 10:30:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Martin Coxall <coxall@cream.org> writes:
> Partially true. The FHS specifies that the /etc top layer is for system-own3d
> stuff, but the subdirectories off it are explicitly used for user space programs
> and, well, everything. (/etc/apache, /etc/postgres, /etc/tomcat3,
> /etc/tomcat4...)

FHS or no FHS, I would think that the preferred arrangement would be to
keep Postgres' config files in a postgres-owned subdirectory, not
directly in /etc.  That way you need not be root to edit them.  (My idea
of an editor, Emacs, always wants to write a backup file, so I dislike
having to edit files that live in directories I can't write.)

Here's a pretty topic for a flamewar: should it be /etc/postgres/ or
/etc/postgresql/ ?

			regards, tom lane

Fri, 14 Feb 2003 10:35:41 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
On Thu, 2003-02-13 at 19:22, Adam Haberlach wrote:
> On Thu, Feb 13, 2003 at 05:59:17PM -0500, Robert Treat wrote:
> > On Thu, 2003-02-13 at 15:08, mlw wrote:
> > > Stephan Szabo wrote:
> > > 
> > > On Thu, 13 Feb 2003, mlw wrote:
> > Personally I think a postgresql installation is much more like an apache
> > installation, which generally contains all of the files (data and
> > config) under /usr/local/apache. Maybe someone can dig more to see if
> > that system is more appropriate a comparison than something like bind.
> 
> 	I think you are making a pretty uninformed, if not just plain wrong 
> generalization.  I've run exactly one system with apache configuration 
> files in /usr/local/apache, and even then, the data was not there.

Uh... the last time I built apache from source, it stuck everything
under /usr/local/apache. It uses a conf directory for the config files,
and htdocs for the "data" files...  That is it's default configuration.

<snip stories of all the different ways people run apache>

You know, this is why I actually suggested looking closer at apache. By
default, everything is crammed in one directory, but if you want to, you
can configure it "six different ways to sunday". That seems to be a big
plus IMO

> 
> What does this mean?
> 
> People will put things in different places, and there are typically
> very good reasons for this.  This is ESPECIALLY true when one wants to
> have configuration files, at least the base ones in a common place such
> as /etc or /usr/local/etc in order to make backup of configuration easy
> and clean, while leaving data somewhere else for performance or magnitude
> of partition reasons.  It just makes sense to ME to have postgresql.conf
> reside in /etc, yet put my data in /var/data/postgresql, yet retain the
> option to put my data in /raid/data/postgresql at a later date, when the
> new hardware comes in.

Is anyone arguing against this? I'm certainly not. But maybe my needs
are more varied than yours. On my local development box, I run multiple
versions of apache, compiled with different versions of php. It really
helps to keep all of apache's stuff centralized, and using things like
rpms actually overly complicates this. Now sure, that's a development
machine, but on the phppgadmin demo server, which is essentially a
production system, I run three different versions of postgresql. In
fact, I need to upgrade two of those (to 7.2.4 and 7.3.2), I shudder to
think about doing that if postgresql forced me to use the /etc/
directory for all of my config files. Now sure, this probably isn't
typical use, but I would say that when it comes time to upgrade major
versions, unless you running an operation where you can have large
amounts of downtime, postgresql needs to have the ability to have
multiple versions install that don't conflict with each other, and it
needs to do this easily. The upgrade process is hard enough already.

<snip> 
> However, this seems, to me, to be a very small addition that has some real-world
> (and yes, we need to start paying attention to the real world) advantages.
> 
> And finally, don't go telling me that I'm wrong to put my data and config files
> where I am.  You can offer advice, but I'm probably going to ignore it because
> I like where they are and don't need to explain why.
> 

Have I wronged you in some former life? I've very little concern for
where you put your data files, and have no idea why you'd think I'd
criticize your setup. 

Robert Treat


14 Feb 2003 10:38:27 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: location of the configuration files
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> My point is that folks with multiple postmasters may not want to use
> PGDATA, but for folks who have single postmasters, it makes things
> easier and less error-prone.

Actually, for multi postmasters too.  I have little shell-environment
config files that switch my entire world view between different source
trees and installation trees, for example this one sets me up to mess
with the 7.2 branch:

STDPATH=${STDPATH:-$PATH}
STDMANPATH=${STDMANPATH:-$MANPATH}

PGSRCROOT=$HOME/REL7_2/pgsql
PGINSTROOT=$HOME/version72
PATH=$PGINSTROOT/bin:/opt/perl5.6.1/bin:$STDPATH
MANPATH=$PGINSTROOT/man:$STDMANPATH
PGLIB=$PGINSTROOT/lib
PGDATA=$PGINSTROOT/data
PMOPTIONS="-p 5472 -i -F"
PMLOGFILE=server72.log

export PGSRCROOT PGINSTROOT PATH MANPATH PGLIB PGDATA STDPATH STDMANPATH
export PMOPTIONS PMLOGFILE

After sourcing one of these, I can use pg_ctl as well as a half dozen
other convenient little scripts that do things like remake and reinstall
the backend:

#!/bin/sh

pg_ctl -w stop

cd $PGSRCROOT/src/backend

make install-bin

startpg

or this one that fires up gdb on a crashed backend:

#!/bin/sh

# Usage: gdbcore

cd $HOME

CORES=`find $PGDATA/base -name core -type f -print`

if [ x"$CORES" != x"" ]
then
    ls -l $CORES
fi

if [ `echo "$CORES" | wc -w` -eq 1 ]
then
    exec gdb $PGINSTROOT/bin/postgres "$CORES"
else
    exec gdb $PGINSTROOT/bin/postgres
fi

This is vastly less error-prone than keeping track of the various
related elements in my head.

Now, it's certainly true that I could still make this work if I had
to explicitly say -D $PGDATA to the postmaster.  But that would clutter
my ps display.  I am happy with -p as the ps indicator of which
postmaster is which; I don't want more stuff in there.

			regards, tom lane

Fri, 14 Feb 2003 10:51:16 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
On Friday 14 Feb 2003 9:05 pm, you wrote:
> Martin Coxall <coxall@cream.org> writes:
> Here's a pretty topic for a flamewar: should it be /etc/postgres/ or
> /etc/postgresql/ ?

I vote for /etc/pgsql. Keeping in line of unix philosophy of cryptic and short 
names. Who wants a descriptive names anyway..:-)

Seriously, the traffic on last three days ahd very high noise ratio. 
Especially the whole discussion of PGDATA stuff fails to register as 
significant IMO. Right now, I can do things the way I want to do and I guess 
it is pretty much same with everyone else. Is it last topic left to improve?

Keep it simple and on tpoic guys. This is hackers. Keep it low volume 
otherwise, two years down the lines, archives will be unsearchable..

 Shridhar

Fri, 14 Feb 2003 21:32:48 +0530
"Shridhar Daithankar<shridhar_daithankar@persistent.co.in>"
	<shridhar_daithankar@persistent.co.in>
Re: location of the configuration files
On Thu, 13 Feb 2003, mlw wrote:

> 
> 
> scott.marlowe wrote:
> 
> >>These are not issues at all. You could put the configuration file 
> >>anywhere, just as you can for any UNIX service.
> >>
> >>postmaster --config=/home/myhome/mydb.conf
> >>
> >>I deal with a number of PG databases on a number of sites, and it is a 
> >>real pain in the ass to get to a PG box and hunt around for data 
> >>directory so as to be able to administer the system. What's really 
> >>annoying is when you have to find the data directory when someone else 
> >>set up the system.
> >>    
> >>
> >
> >Really?  I would think it's easier to do this:
> >
> >su - pgsuper
> >cd $PGDATA
> >pwd
> >
> >Than to try to figure out what someone entered when they ran ./configure 
> >--config=...
> >  
> >
> Why do you think PGDATA would be set for root?

Did you not notice the "su - pgsuper" line above?  You know, the one where 
you become the account that runs that instance of the database.  Again, I 
ask you, isn't that easier than trying to find out what someone typed when 
they typed ./configure --config=?

> >>Configuring postgresql via a configuration file which specifies all the 
> >>data, i.e. data directory, name of other configuration files, etc. is 
> >>the right way to do it. Even if you have reasons against it, even if you 
> >>think it is a bad idea, a bad standard is almost always a better 
> >>solution than an arcane work of perfection.
> >>    
> >>
> >
> >Wrong, I strongly disagree with this sentament.  Conformity to standards 
> >for simple conformity's sake is as wrong as sticking to the old way 
> >because it's what we're all comfy with.  
> >
> It isn't conformity for conformitys sake. It is following an established 
> practice, like driving on the same side of the road or stopping at red 
> lights.

But this isn't the same thing at all.  Apache, when built from a tar ball, 
goes into /usr/local/apache/ and ALL it's configuration files are there.  
When installed as a package, my OS manufacturer decides where that goes.  
Those are the two "standard" ways of doing things.  I like that postgresql 
installs into the /usr/local/pgsql directory from a tar ball.  I like the 
fact that it uses $PGDATA to tell it where the cluster is, so that all my 
scripts, like pg_ctl, just know where it is without a -D switch each time.



Fri, 14 Feb 2003 09:14:52 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: location of the configuration files
On Fri, 2003-02-14 at 15:35, Tom Lane wrote:
> Here's a pretty topic for a flamewar: should it be /etc/postgres/ or
> /etc/postgresql/ ?

It should be configurable!

Debian uses /etc/postgresql, if you want to stick to what quite a lot of
people are familiar with.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "God be merciful unto us, and bless us; and cause his 
      face to shine upon us."          Psalms 67:1 


14 Feb 2003 16:44:49 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: location of the configuration files
On Fri, 14 Feb 2003, Curt Sampson wrote:

> On Thu, 13 Feb 2003, scott.marlowe wrote:
> 
> > If I do a .tar.gz install of apache, I get /usr/local/apache/conf, which
> > is not the standard way you're listing.
> 
> I'm going to stay out of this argument from now on, but this struck a sore
> point.
> 
> /usr is designed to be a filesystem that can be shared. Is the stuff in
> /usr/local/apache/conf really supposed to be shared amongst all machines
> of that architecture on your site that run apache?

Interesting.  I've always viewed usr EXCEPT for local this way.  In 
fact, on most of my boxes I create a seperate mount point for /usr/local 
so it's easier to backup and maintain, and it doesn't fill up the /usr 
directory.

Asking for everything in a directory with the name local in it to be 
shared is kind of counter intuitive to me.


Fri, 14 Feb 2003 09:46:35 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: location of the configuration files

> I seriously doubt your ability to judge anyone's mental stability.
> I can also see that you prefer cross-platform INcompatibility.  Your
> position and mindset are now crystal clear.

Come now- don't take it personally. All I said is, as someone who
wrestles daily with QMail, we should prefer the FHS over DJB's way of
doing things and that DJB is a little, ahem, egocentric at times.
Neither of these things was meant as a mortal insult to you personally,
and if I offended you I apologise.

Anyway, it looks like it's all been agreed over there, anyway.

-- 
Martin Coxall <coxall@cream.org>


14 Feb 2003 16:57:58 +0000
Martin Coxall <coxall@cream.org>
Re: location of the configuration files
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > So, we add data_dir to postgresql.conf, and add -C/PGCONFIG to
> > postmaster.
> 
> Wait one second.  You are blithely throwing in a PGCONFIG variable
> without any detailed proposal of exactly how it will work.  Does
> that override a PGDATA environment variable?  How do they interact?

I am just throwing out ideas. I don't think we are near interaction
issues yet.

I think the big question is whether we want the default to install the
configs in a separate directory, pgsql/etc, or just allow the
specification of a separate location.  Advantages of pgsql/etc are
initdb-safe, and easier backups.

I do think PGCONFIG would be helpful for the same reason that PGDATA is.
However, there is clearly a problem of how does data_dir interact with
PGDATA.

The big question is whether PGDATA is still our driving config variable,
and PGCONFIG/-C is just an additional option, or whether we are moving
in a direction where PGCONFIG/-C is going to be the driving value, and
data_dir is going to be read as part of that.


> Also, please note Kevin Brown's nearby arguments against using PGDATA
> at all, which surely apply with equal force to a PGCONFIG variable.
> Now, I don't buy that Kevin's arguments are enough reason to break
> backwards compatibility by removing PGDATA --- but I think they are
> enough reason not to introduce a new environment variable.  PGCONFIG
> wouldn't offer any backwards-compatibility value, and that tilts the
> scales against it.

Weren't you just showing how you set environment variables to easily
configure stuff.  If you use a separate configure dir, isn't PGCONFIG
part of that?

> > Regarding Tom's idea of replacing data_dir with a full path during
> > initdb, I think we are better having it be relative to the config
> > directory, that way if they move pgsql/, the system still works.
> 
> Good thought, but you're assuming that initdb knows where the config
> files will eventually live.  If we do that, then moving the config
> files breaks the installation.  I think it will be fairly common to
> let initdb drop its proposed config files into $PGDATA, and then
> manually place them where they should go (or even more likely,
> manually merge them with a prior version).  Probably better to force
> datadir to be an absolute path in the config file.  (In fact, on safety
> grounds I'd argue in favor of rejecting a datadir value taken from the
> config file that wasn't absolute.)

Maybe.  Not sure.

> > I also think we should start telling people to use PGCONFIG rather than
> > PGDATA.  Then, in 7.5, we move the default config file location to
> > pgsql/etc, and tell folks to point there rather than /data.
> 
> I agree with none of this.  This is not improvement, this is only change
> for the sake of change.  The packagers will do what they want to do
> (and are already doing, mostly) regardless.

Well, it is a step forward in terms of initdb-safe and easier backups.
Several people said they liked that.  I thought you were one of them.

This is back to the big question, who drives things in the default
install, config file or pgdata.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 12:33:42 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Bruce Momjian wrote:
> The big question is whether PGDATA is still our driving config variable,
> and PGCONFIG/-C is just an additional option, or whether we are moving
> in a direction where PGCONFIG/-C is going to be the driving value, and
> data_dir is going to be read as part of that.

I'm actually leaning towards PGCONFIG + PGDATA.

Yeah, it may be a surprise given my previous arguments, but I can't
help but think that the advantages you get with PGDATA will also exist
for PGCONFIG.

My previous arguments for removing PGDATA from postmaster can be dealt
with by fixing pg_ctl to use explicit command line directives when
invoking postmaster -- no changes to postmaster needed.  PGCONFIG
would be no different in that regard.


Sorry if I seem a big gung-ho on the administrator point of view, but
as a system administrator myself I understand and feel their pain.
:-)



-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 14 Feb 2003 10:09:37 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files

"scott.marlowe" <scott.marlowe@ihs.com> writes:

> But this isn't the same thing at all.  Apache, when built from a tar ball, 
> goes into /usr/local/apache/ and ALL it's configuration files are there.  

Two comments:

1) Even in that case the config files go into /usr/local/apache/conf and the
   other kinds of files like data logs and cache files, all go in other
   subdirectories.

2) What you describe is only true if you configure with the default
   "--with-layout=Apache". The naming should perhaps be a clue that this isn't
   a conventional layout. If you configure with --with-layout=GNU you get the
   conventional Unix layout in /usr/local, If you use --with-layout=RedHat you
   get the conventional layout in /usr directly which is mainly useful for
   distribution packagers.

Putting stuff in a subdirectory like /usr/local/apache or /usr/local/pgsql is
unfortunately a widespread practice. It does have some advantages over the
conventional layout in /usr/local/{etc,bin,...} directly. But the major
disadvantage is that users can't run programs without adding dozens of entries
to their paths, can't compile programs without dozens of -L and -I lines, etc.

GNU autoconf script makes it pretty easy to configure packages to work either
though, and /usr/local is the purview of the local admin. As long as it's easy
to configure postgres to install "properly" with --prefix=/usr/local it won't
be any more of an offender than lots of other packages like apache, kde, etc.

Though I'll mention, please make it $prefix/etc not $prefix/conf. No need to
be gratuitously non-standard on an arbitrary name, and no need to pollute
/usr/local with multiple redundant directories.

-- 
greg


14 Feb 2003 13:16:26 -0500
Greg Stark <gsstark@mit.edu>
Re: location of the configuration files
On Thu, Feb 13, 2003 at 11:53:26 -0500,
  mlw <pgsql@mohawksoft.com> wrote:
> 
> Where, specificaly are his arguements against a configuration file 
> methodology?

I don't think he is argueing against a configuration methodology, but
rather against the methodology being used in Unix distributions.
In particular he doesn't file the Linux File Standard because it
puts the same software in different places depending on whether the
vendor or using installed it.

Fri, 14 Feb 2003 12:27:49 -0600
Bruno Wolff III <bruno@wolff.to>
Re: location of the configuration files
Kevin Brown wrote:
> Bruce Momjian wrote:
> > The big question is whether PGDATA is still our driving config variable,
> > and PGCONFIG/-C is just an additional option, or whether we are moving
> > in a direction where PGCONFIG/-C is going to be the driving value, and
> > data_dir is going to be read as part of that.
> 
> I'm actually leaning towards PGCONFIG + PGDATA.
> 
> Yeah, it may be a surprise given my previous arguments, but I can't
> help but think that the advantages you get with PGDATA will also exist
> for PGCONFIG.
> 
> My previous arguments for removing PGDATA from postmaster can be dealt
> with by fixing pg_ctl to use explicit command line directives when
> invoking postmaster -- no changes to postmaster needed.  PGCONFIG
> would be no different in that regard.

I see your point --- pg_ctl does a PGDATA trick when passed -D:

        -D)
            shift
            # pass environment into new postmaster
            PGDATA="$1"
            export PGDATA

It should pass -D just like it was given.

> Sorry if I seem a big gung-ho on the administrator point of view, but
> as a system administrator myself I understand and feel their pain.

Making things easy for sysadmins is an important feature of PostgreSQL.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 13:37:22 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
On Fri, Feb 14, 2003 at 02:58:49 -0800,
  Kevin Brown <kevin@sysexperts.com> wrote:
> 
> Lest you think that this is an unlikely scenario, keep in mind that
> most startup scripts, including pg_ctl, currently start the postmaster
> without arguments and rely on PGDATA, so a shop that hasn't already
> been bitten by this *will* be.  Right now shops that wish to avoid the
> trap I described have to go to *extra* lengths: they have to make
> exactly the same kinds of changes to the scripts that I'm talking
> about us making (putting an explicit '-D "$PGDATA"' where none now
> exists) or they have to resort to tricks like renaming the postmaster
> executable and creating a shell script in its place that will invoke
> the (renamed) postmaster with '-D "$PGDATA"'.

On at least some systems ps will dump process' environment and could be
easily used to check PGDATA.

Fri, 14 Feb 2003 12:50:17 -0600
Bruno Wolff III <bruno@wolff.to>
Re: location of the configuration files
Bruce Momjian wrote:
> I see your point --- pg_ctl does a PGDATA trick when passed -D:
> 
>         -D)
>             shift
>             # pass environment into new postmaster
>             PGDATA="$1"
>             export PGDATA
> 
> It should pass -D just like it was given.

Yes, exactly.

Now, the more interesting question in my mind is: if pg_ctl isn't
passed -D but inherits PGDATA, should it nonetheless pass -D
explicitly to the postmaster?  We can make it do that, and it would
have the benefit of making transparent what would otherwise be opaque.

I'm inclined to answer "yes" to that question, but only because
someone who *really* doesn't want the postmaster to show up with a -D
argument in "ps" can start the postmaster directly without using
pg_ctl at all.  Tom made a good argument for sometimes wanting to keep
the ps output clean, but it's not clear to me that it should
necessarily apply to pg_ctl.

But you guys might have a different perspective on that.  :-)



-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 14 Feb 2003 10:56:28 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
On Friday 14 February 2003 6:07 am, Martin Coxall wrote:
> On Thu, 2003-02-13 at 20:28, Steve Crawford wrote:
> > I don't see why we can't keep everyone happy and let the users choose the
> > setup they want. To wit, make the following, probably simple, changes:
> >
> > 1) Have postgresql default to using /etc/postgresql.conf
>
> /etc/postgres/postgresql.conf, if we want to be proper FHS-bitches.
>
> > 2) Add a setting in postgresql.conf specifying the data directory
> > 3) Change the meaning of -D to mean "use this config file"
> > 4) In the absence of a specified data directory in postgresql.conf, use
> > the location of the postgresql.conf file as the data directory
>
> Shouldn't it in that case default to, say /var/lib/postgres?

Idea 4 was just a way to preserve current behaviour for those who desire. 
Moving postgresql.conf requires adding the data directory info into 
postgresql.conf or specifying it in some other way. If, in the absence of any 
specification in postgresql.conf, postgres just looks in the same directory 
as postgresql.conf then it will be almost identical to the current setup.

Cheers,
Steve

Fri, 14 Feb 2003 11:24:19 -0800
Steve Crawford <scrawford@pinpointresearch.com>
Re: location of the configuration files
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I think the big question is whether we want the default to install the
> configs in a separate directory, pgsql/etc, or just allow the
> specification of a separate location.  Advantages of pgsql/etc are
> initdb-safe, and easier backups.

I don't see why we don't just let initdb install suggested config files
into the new $PGDATA directory, same as it ever did.  Then (as long as
we don't use relative paths in the config files) people can move them
somewhere else if they like, or not if they prefer not to.  Adding more
mechanism than that just adds complexity without buying much (except the
possibility of initdb overwriting your old config files, which is
exactly what I thought we wanted to avoid).

> The big question is whether PGDATA is still our driving config variable,
> and PGCONFIG/-C is just an additional option, or whether we are moving
> in a direction where PGCONFIG/-C is going to be the driving value, and
> data_dir is going to be read as part of that.

I thought the idea was to allow both approaches.  We are not moving in
the direction of one or the other, we are giving people a choice of how
they want to drive it.

> Weren't you just showing how you set environment variables to easily
> configure stuff.  If you use a separate configure dir, isn't PGCONFIG
> part of that?

I'm just pointing out that there's no backward-compatibility argument
for PGCONFIG.  It should only be put in if the people who want to use
the -C-is-driver approach want it.  Kevin clearly doesn't ;-).

			regards, tom lane

Fri, 14 Feb 2003 15:10:13 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Tom Lane writes:

> I would favor a setup that allows a -C *directory* (not file) to be
> specified as a postmaster parameter separately from the -D directory;

A directory is not going to satisfy people.

> I don't see any great value in a separate postgresql.conf parameter for
> each secondary config file; that just means clutter to me,

Not to other people.

> 1. No -C switch, no -D switch, no PGDATA found in environment: seek
> postgresql.conf in the default -C directory established at configure
> time.  Use the 'datadir' specified therein as -D.  Fail if postgresql.conf
> doesn't define a datadir value.

OK.

> 2. No -C switch, no -D switch, PGDATA found in environment: use $PGDATA
> as both -C and -D.

This behavior would be pretty inconsistent.  But maybe it's the best we
can do.

> 3. No -C switch, -D switch on command line: use -D value as both -C and -D,
> proceed as in case 2.

Same as above.

> 4. -C switch, no -D switch on command line: seek postgresql.conf in
> -C directory, use the datadir it specifies.

OK.

> 5. -C and -D on command line: seek postgresql.conf in -C directory,
> use -D as datadir overriding what is in postgresql.conf (this is just
> the usual rule that command line switches override postgresql.conf).

But that usual rule seems to be in conflict with cases 2 and 3 above.
(The usual rule is that a command-line option overrides a postgresql.conf
parameter.  The rule in 3, for example is, that a command-line option (the
same one!) overrides where postgresql.conf is in the first place.)

> I would venture that the configure-time-default for -C should be
> ${prefixdir}/etc if configure is not told differently,

Yeah, we already have that as --sysconfdir.

-- 
Peter Eisentraut   peter_e@gmx.net


Fri, 14 Feb 2003 21:14:30 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: location of the configuration files 
Peter Eisentraut wrote:
> Tom Lane writes:
> 
> > I would favor a setup that allows a -C *directory* (not file) to be
> > specified as a postmaster parameter separately from the -D directory;
> 
> A directory is not going to satisfy people.

Who is asking to put postgresql.conf, pg_hba.conf, and pg_ident.conf in
different directories?  I haven't heard anyone ask for that.

> > I don't see any great value in a separate postgresql.conf parameter for
> > each secondary config file; that just means clutter to me,
> 
> Not to other people.
> 
> > 1. No -C switch, no -D switch, no PGDATA found in environment: seek
> > postgresql.conf in the default -C directory established at configure
> > time.  Use the 'datadir' specified therein as -D.  Fail if postgresql.conf
> > doesn't define a datadir value.
> 
> OK.
> 
> > 2. No -C switch, no -D switch, PGDATA found in environment: use $PGDATA
> > as both -C and -D.
> 
> This behavior would be pretty inconsistent.  But maybe it's the best we
> can do.

What happens if postgresql.conf then defines data_dir?  Seems we ignore it.

This brings up the same issue of whether -C/PGCONFIG is a inferior
option to -D/PGDATA, and whether we keep the config files in /data by
default.

> > 3. No -C switch, -D switch on command line: use -D value as both -C and -D,
> > proceed as in case 2.
> 
> Same as above.


> 
> > 4. -C switch, no -D switch on command line: seek postgresql.conf in
> > -C directory, use the datadir it specifies.
> 
> OK.

Here we are saying the -C doesn't override postgresql.conf as the proper
PGDATA value.  Is that what we want?  We had the question above over how
a data_dir in postgresql.conf is handled.

> 
> > 5. -C and -D on command line: seek postgresql.conf in -C directory,
> > use -D as datadir overriding what is in postgresql.conf (this is just
> > the usual rule that command line switches override postgresql.conf).
> 
> But that usual rule seems to be in conflict with cases 2 and 3 above.
> (The usual rule is that a command-line option overrides a postgresql.conf
> parameter.  The rule in 3, for example is, that a command-line option (the
> same one!) overrides where postgresql.conf is in the first place.)


Yes, the big question seems to be if we are defaulting -C to be the same
as -D, whether that is an actual specification of -D that should
override postgresql.conf.

This is part of the reason I don't like the -D assumes -C and stuff like
that.

I think we need to move the config files to pgsql/etc, for backup and
initdb safety, and move toward having PGCONFIG/-C as the driving
parameter.  I think having both function equally and defaulting if the
other is not specified is going to breed confusion.

I am willing to make thing a little difficult for backward compatibility
to do this, and I think because it is only administrators, they will
welcome the improvement.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 15:19:36 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> I would favor a setup that allows a -C *directory* (not file) to be
>> specified as a postmaster parameter separately from the -D directory;

> A directory is not going to satisfy people.

Why not?  Who won't it satisfy, and what's their objection?

AFAICS, you can either set -C to /etc if you want your PG config files
loose in /etc, or you can set it to /etc/postgresql/ if you want them
in a privately-owned directory.  Which other arrangements are needed?

			regards, tom lane

Fri, 14 Feb 2003 15:27:53 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I think the big question is whether we want the default to install the
> > configs in a separate directory, pgsql/etc, or just allow the
> > specification of a separate location.  Advantages of pgsql/etc are
> > initdb-safe, and easier backups.
> 
> I don't see why we don't just let initdb install suggested config files
> into the new $PGDATA directory, same as it ever did.  Then (as long as
> we don't use relative paths in the config files) people can move them
> somewhere else if they like, or not if they prefer not to.  Adding more
> mechanism than that just adds complexity without buying much (except the
> possibility of initdb overwriting your old config files, which is
> exactly what I thought we wanted to avoid).
> 
> > The big question is whether PGDATA is still our driving config variable,
> > and PGCONFIG/-C is just an additional option, or whether we are moving
> > in a direction where PGCONFIG/-C is going to be the driving value, and
> > data_dir is going to be read as part of that.
> 
> I thought the idea was to allow both approaches.  We are not moving in
> the direction of one or the other, we are giving people a choice of how
> they want to drive it.

That's where I am unsure.  Is the initdb-safe and backup advantages
enough to start to migrate those out to data/?  I need to hear
comments on that.

One new idea is to move the config files into data/etc.  That makes it
clear which are config files, and makes backup a little easier.  It
would make -C more logical because you are not moving a clear directory.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 15:53:45 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Kevin Brown wrote:
> Bruce Momjian wrote:
> > The big question is whether PGDATA is still our driving config variable,
> > and PGCONFIG/-C is just an additional option, or whether we are moving
> > in a direction where PGCONFIG/-C is going to be the driving value, and
> > data_dir is going to be read as part of that.
> 
> I'm actually leaning towards PGCONFIG + PGDATA.
> 
> Yeah, it may be a surprise given my previous arguments, but I can't
> help but think that the advantages you get with PGDATA will also exist
> for PGCONFIG.
> 
> My previous arguments for removing PGDATA from postmaster can be dealt
> with by fixing pg_ctl to use explicit command line directives when
> invoking postmaster -- no changes to postmaster needed.  PGCONFIG
> would be no different in that regard.

The following patch propogates pg_ctl -D to the postmaster as a -D flag.
I see no other pg_ctl flags that make sense to propogate.

Applied.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Index: src/bin/pg_ctl/pg_ctl.sh
===================================================================
RCS file: /cvsroot/pgsql-server/src/bin/pg_ctl/pg_ctl.sh,v
retrieving revision 1.30
diff -c -c -r1.30 pg_ctl.sh
*** src/bin/pg_ctl/pg_ctl.sh	18 Oct 2002 22:05:35 -0000	1.30
--- src/bin/pg_ctl/pg_ctl.sh	14 Feb 2003 22:04:56 -0000
***************
*** 115,120 ****
--- 115,122 ----
  logfile=
  silence_echo=
  shutdown_mode=smart
+ PGDATAOPTS=""
+ POSTOPTS=""
  
  while [ "$#" -gt 0 ]
  do
***************
*** 129,135 ****
  	    ;;
  	-D)
  	    shift
! 	    # pass environment into new postmaster
  	    PGDATA="$1"
  	    export PGDATA
  	    ;;
--- 131,138 ----
  	    ;;
  	-D)
  	    shift
! 	    # we need to do this so -D datadir shows in ps display
! 	    PGDATAOPTS="-D $1"
  	    PGDATA="$1"
  	    export PGDATA
  	    ;;
***************
*** 333,344 ****
      fi
  
      if [ -n "$logfile" ]; then
!         "$po_path" ${1+"$@"} </dev/null >>$logfile 2>&1 &
      else
          # when starting without log file, redirect stderr to stdout, so
          # pg_ctl can be invoked with >$logfile and still have pg_ctl's
          # stderr on the terminal.
!         "$po_path" ${1+"$@"} </dev/null 2>&1 &
      fi
  
      # if had an old lockfile, check to see if we were able to start
--- 336,347 ----
      fi
  
      if [ -n "$logfile" ]; then
!         "$po_path" ${1+"$@"} ${PGDATAOPTS+$PGDATAOPTS} </dev/null >>$logfile 2>&1 &
      else
          # when starting without log file, redirect stderr to stdout, so
          # pg_ctl can be invoked with >$logfile and still have pg_ctl's
          # stderr on the terminal.
!         "$po_path" ${1+"$@"} ${PGDATAOPTS+$PGDATAOPTS} </dev/null 2>&1 &
      fi
  
      # if had an old lockfile, check to see if we were able to start
Fri, 14 Feb 2003 17:17:22 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files

I don't want to over-engineer this.  Propogating -D into postmaster
makes sense, but grabbing PGDATA doesn't to me.

---------------------------------------------------------------------------

Kevin Brown wrote:
> Bruce Momjian wrote:
> > I see your point --- pg_ctl does a PGDATA trick when passed -D:
> > 
> >         -D)
> >             shift
> >             # pass environment into new postmaster
> >             PGDATA="$1"
> >             export PGDATA
> > 
> > It should pass -D just like it was given.
> 
> Yes, exactly.
> 
> Now, the more interesting question in my mind is: if pg_ctl isn't
> passed -D but inherits PGDATA, should it nonetheless pass -D
> explicitly to the postmaster?  We can make it do that, and it would
> have the benefit of making transparent what would otherwise be opaque.
> 
> I'm inclined to answer "yes" to that question, but only because
> someone who *really* doesn't want the postmaster to show up with a -D
> argument in "ps" can start the postmaster directly without using
> pg_ctl at all.  Tom made a good argument for sometimes wanting to keep
> the ps output clean, but it's not clear to me that it should
> necessarily apply to pg_ctl.
> 
> But you guys might have a different perspective on that.  :-)
> 
> 
> 
> -- 
> Kevin Brown					      kevin@sysexperts.com
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 14 Feb 2003 17:41:26 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
On Fri, 14 Feb 2003, scott.marlowe wrote:

> Asking for everything in a directory with the name local in it to be
> shared is kind of counter intuitive to me.

Not really. If you install a particular program that doesn't come with
the OS on one machine on your site, why would you not want to install it
separately on all of the others?

Typically, I want my favourite non-OS utilities on all machines, not
just one. (Even if I don't use them on all machines.) Thus /usr/local is
for site-local stuff.

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC

Sat, 15 Feb 2003 13:44:00 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: location of the configuration files


Robert Treat wrote:

>Seems like some are saying one of the problems with the current system
>is it doesn't follow FHS or LSB. If those are valid reasons to change
>the system, it seems like a change which doesn't actually address those
>concerns would not be acceptable. (Unless those really aren't valid
>concerns...)
>
>  
>
I did not start this thread to make PostgreSQL FHS compatible, someone 
else brought that up.

As I said somewhere else, I'm an old fashioned UNIX guy, capability 
without policy. The patch that I submitted for 7.3.2 will allow the user 
to configure PostgreSQL with a configuration file outside the $PGDATA 
directory. That's all I care about. If someone wants to get on the FHS 
bandwagon, that's fine. PostgreSQL should allow that ability but should 
not require it.


Sat, 15 Feb 2003 09:48:57 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files


Martin Coxall wrote:

>On Thu, 2003-02-13 at 20:28, Steve Crawford wrote:
>  
>
>>I don't see why we can't keep everyone happy and let the users choose the 
>>setup they want. To wit, make the following, probably simple, changes:
>>
>>1) Have postgresql default to using /etc/postgresql.conf
>>    
>>
>
>/etc/postgres/postgresql.conf, if we want to be proper FHS-bitches.
>
>  
>
>>2) Add a setting in postgresql.conf specifying the data directory
>>3) Change the meaning of -D to mean "use this config file"
>>4) In the absence of a specified data directory in postgresql.conf, use the 
>>location of the postgresql.conf file as the data directory
>>    
>>
>
>Shouldn't it in that case default to, say /var/lib/postgres?
>
I would really like to push back this whole discussion to adding the 
ability the flexibility to configure PostgreSQ as opposed to determining 
a specific configuration strategy.

Adding the ability is easy. Let the distros determine their strategy. 
Trying to enforce one way over another will make this continue on 
forever and will never be solved.

>  
>


Sat, 15 Feb 2003 09:53:23 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
mlw <pgsql@mohawksoft.com> writes:
> I would really like to push back this whole discussion to adding the 
> ability the flexibility to configure PostgreSQ as opposed to determining 
> a specific configuration strategy.

> Adding the ability is easy. Let the distros determine their strategy. 
> Trying to enforce one way over another will make this continue on 
> forever and will never be solved.

I agree that we shouldn't be in the business of dictating choices.

But it is important to examine what the plausible choices are, so that
we can be sure the solution we provide will accommodate all of them.
So I don't think this part of the thread has been useless.

			regards, tom lane

Sat, 15 Feb 2003 11:00:11 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
On Friday 14 February 2003 15:10, Tom Lane wrote:
> I don't see why we don't just let initdb install suggested config files
> into the new $PGDATA directory, same as it ever did.

Ok, let me take another tack.

Just exactly why does initdb need to drop any config files anywhere?  We 
provide templates; initdb can initialize the data structure.  If we can by 
default (as part of make install) put the config file templates in 
$SYSCONFDIR (as set by ./configure), then why does initdb need to retouch 
them?  I say that having configured PostgreSQL like this: (this is for 7.2.4, 
not 7.3.x)
--enable-locale --with-CXX --prefix=/usr --disable-rpath --with-perl 
--enable-multibyte --with-tcl --with-odbc --enable-syslog --with-python 
--with-openssl --with-pam --with-krb5=/usr/kerberos --enable-nls 
--sysconfdir=/etc/pgsql --mandir=/usr/share/man --docdir=/usr/share/doc 
--includedir=/usr/include --datadir=/usr/share/pgsql

So, in my case, it would be preferable to me for initdb to not make a default 
postgresql.conf, pg_hba.conf, and pg_ident.conf.  The make install process 
should populate sysconfdir (/etc/pgsql here) with those files.

Why does initdb even need to be involved now (I know the historical reason)?

Comments?
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Sat, 15 Feb 2003 19:36:51 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
On Friday 14 February 2003 11:02, "Shridhar Daithankar wrote:
> Especially the whole discussion of PGDATA stuff fails to register as
> significant IMO. Right now, I can do things the way I want to do and I
> guess it is pretty much same with everyone else. Is it last topic left to
> improve?

If it weren't significant to a few, then there wouldn't be the traffic.  If 
there's too much traffic, well, there are alternatives.

> Keep it simple and on tpoic guys. This is hackers. Keep it low volume
> otherwise, two years down the lines, archives will be unsearchable..

The system configuration of PostgreSQL is on topic for -hackers.  IMNSHO.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Sat, 15 Feb 2003 19:38:23 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
Lamar Owen <lamar.owen@wgcr.org> writes:
> Just exactly why does initdb need to drop any config files anywhere?

Because we'd like it to edit the correct datadir into the config file,
to take just the most obvious example.  There has also been a great deal
of discussion recently about other things initdb might automatically put
into the config file after looking at the system environment.  That's
not happened yet, but we'd really be restricting ourselves to say that
initdb can never customize the config files.

			regards, tom lane

Sat, 15 Feb 2003 20:19:59 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
On Saturday 15 February 2003 20:19, Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > Just exactly why does initdb need to drop any config files anywhere?

> Because we'd like it to edit the correct datadir into the config file,
> to take just the most obvious example.

Shouldn't we be consistent and have initdb use the datadir set in the config 
file, which could be supplied by a ./configure switch?

>  There has also been a great deal
> of discussion recently about other things initdb might automatically put
> into the config file after looking at the system environment.  That's
> not happened yet, but we'd really be restricting ourselves to say that
> initdb can never customize the config files.

Customize != writing the original.

I'm looking at a packager point of view here.  The initdb is done well after 
the package is made, and installed.  It would be ideal from this point of 
view to have things fully configured pre-initdb (and thus pre-packaging).

But I understand that this might not be ideal from a multipostmaster point of 
view.  Surely these two can be reconciled.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Sat, 15 Feb 2003 20:24:57 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
On Saturday 15 February 2003 09:48 am, mlw wrote:
> Robert Treat wrote:
> >Seems like some are saying one of the problems with the current system
> >is it doesn't follow FHS or LSB. If those are valid reasons to change
> >the system, it seems like a change which doesn't actually address those
> >concerns would not be acceptable. (Unless those really aren't valid
> >concerns...)
>
> I did not start this thread to make PostgreSQL FHS compatible, someone
> else brought that up.
>
> As I said somewhere else, I'm an old fashioned UNIX guy, capability
> without policy. The patch that I submitted for 7.3.2 will allow the user
> to configure PostgreSQL with a configuration file outside the $PGDATA
> directory. That's all I care about. If someone wants to get on the FHS
> bandwagon, that's fine. PostgreSQL should allow that ability but should
> not require it.

If we're going to go through the trouble to change the way things work, we 
might as well try to get something that will allow instalation to match real 
desired configurations out there, like FHS and LSB, or how Oliver and Lamar 
want for packaging without symlinks.  If the goal is just to get something 
that you like, apply the patch locally and be done with it. 

Robert Treat

Sat, 15 Feb 2003 20:34:16 -0500
Robert Treat <xzilla@users.sourceforge.net>
Re: location of the configuration files
Lamar Owen <lamar.owen@wgcr.org> writes:
> On Saturday 15 February 2003 20:19, Tom Lane wrote:
>> Lamar Owen <lamar.owen@wgcr.org> writes:
> Just exactly why does initdb need to drop any config files anywhere?

>> Because we'd like it to edit the correct datadir into the config file,
>> to take just the most obvious example.

> Shouldn't we be consistent and have initdb use the datadir set in the config 
> file, which could be supplied by a ./configure switch?

That'd mean there is no way to perform an initdb into a nonstandard
location without first hand-preparing a config file.  I don't much care
for that.

> I'm looking at a packager point of view here.  The initdb is done well after 
> the package is made, and installed.  It would be ideal from this point of 
> view to have things fully configured pre-initdb (and thus pre-packaging).

This point of view means that no post-configure knowledge can be
applied.  We might as well forget the separate initdb step altogether
and have "make install" do it.

I realize that from a packager's point of view, the separate initdb step
is not very useful.  But it is from my point of view.

			regards, tom lane

Sat, 15 Feb 2003 21:06:20 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
On Saturday 15 February 2003 21:06, Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > Shouldn't we be consistent and have initdb use the datadir set in the
> > config file, which could be supplied by a ./configure switch?

> That'd mean there is no way to perform an initdb into a nonstandard
> location without first hand-preparing a config file.  I don't much care
> for that.

Six of one and half-dozen of another.  And that's my real point.  We do things 
quite differently from many other standard services, even those which are 
built from the ground up for multiple instances.  Making things more 
consistent for admins, even if it's not what we are used to or what we might 
like (because it's familiar) should at least be thought about.  I'm not 
advocating changing just for the sake of change; but getting a new fresh look 
at our current setup can't hurt.

> > I'm looking at a packager point of view here.  The initdb is done well
> > after the package is made, and installed.  It would be ideal from this
> > point of view to have things fully configured pre-initdb (and thus
> > pre-packaging).

> This point of view means that no post-configure knowledge can be
> applied.  We might as well forget the separate initdb step altogether
> and have "make install" do it.

I wouldn't complain.  Although that isn't conducive to the multiple instance 
case.  The necessary post-configure knowledge would be in the instance 
postgresql.conf file.  One place for it.  But this is hypothetical; fishing 
around the waters here at this point. Realize that my own packages apply an 
initdb automatically if an install isn't found the first time the system 
initscript is started.  It is virtually automatic.  With the multiple 
postmaster support, creating a couple of files and a symlink (for the 
initscript), and starting the new initscript symlink does all the dirty work.  
But it could be easier.

> I realize that from a packager's point of view, the separate initdb step
> is not very useful.  But it is from my point of view.

Would you mind elucidating which point of view is yours?  General idea of who 
else might have the same point of view, and why you find the initdb in its 
current form to be more useful than alternatives.  Again, I'm fishing for 
knowledge -- if nothing else it gives me an answer to those users who send me 
nastygrams about the way things are right now.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Sat, 15 Feb 2003 22:03:56 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
Lamar Owen <lamar.owen@wgcr.org> writes:
>> I realize that from a packager's point of view, the separate initdb step
>> is not very useful.  But it is from my point of view.

> Would you mind elucidating which point of view is yours?

Primarily, one that wants to have multiple postmasters running, of the
same or different versions; including test and temporary installations
that mustn't conflict with the existing primary installation on a machine.

Currently, I don't need to do anything more than set PGDATA or say -D
to initdb in order to set up the data directory wherever I like.  I also
don't need to worry about whether I'm selecting the wrong config file.

You're talking about making manual installations significantly more
difficult (and error-prone, I think) in order to simplify automated
installs.  Now you've acknowledged that your script can do what
you want it to, and in fact already does.  Why is it good to make my
life more difficult to make a script's easier?  Cycles are cheap.
I like to think that my time is worth something.

Nor will I buy an argument that only a few developers have need for test
installations.  Ordinary users will want to do that anytime they are
doing preliminary tests on a new PG version before migrating their
production database to it.  To the extent that you make manual selection
of a nonstandard datadir location more difficult and error-prone, you
are hurting them too.

			regards, tom lane

Sun, 16 Feb 2003 00:16:44 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
On Sunday 16 February 2003 00:16, Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > Would you mind elucidating which point of view is yours?

> Primarily, one that wants to have multiple postmasters running, of the
> same or different versions; including test and temporary installations
> that mustn't conflict with the existing primary installation on a machine.

Well, due to our upgrading difficulty, having multiple versions running has 
its advantages.

> You're talking about making manual installations significantly more
> difficult (and error-prone, I think) in order to simplify automated
> installs.  Now you've acknowledged that your script can do what
> you want it to, and in fact already does.  Why is it good to make my
> life more difficult to make a script's easier?  Cycles are cheap.
> I like to think that my time is worth something.

The script's been out there for awhile.  It does some things well, and some 
things not so well.  The config files are still coresident with the database, 
and backup is more difficult than it can be.  Meeting all these needs (with 
configure switches, configuration file directives, etc) would be a good 
thing.  And that's what I'm after; maximum usability for the maximum 
audience.  I believe pretty strongly that the usage to which you or I would 
put PostgreSQL is probably quite different from the average user's way of 
using PostgreSQL.  Most probably the typical user has a single installation 
with multiple databases with little need to run isolated postmasters.

> Nor will I buy an argument that only a few developers have need for test
> installations.  Ordinary users will want to do that anytime they are
> doing preliminary tests on a new PG version before migrating their
> production database to it.  To the extent that you make manual selection
> of a nonstandard datadir location more difficult and error-prone, you
> are hurting them too.

While I'm not going to speak for all users, I know that I don't put a 
development database system on my production servers.  The production machine 
only runs production servers, period.  Hardware is cheap.  I have development 
machines for development databases.  One also has the error-prone PATH issues 
with multiple versions, which, if you are running a typical RPM installation 
becomes quite difficult to manage, since the RPM version's executables are in 
/usr/bin.  This could be changed; I've thought about changing it.  But I'm 
not sure of the best way to make multiple versions peacefully and seamlessly 
coexist -- particularly when older versions may not even build on the newer 
OS: but we've been over that discussion.

Care for a poll?
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Sun, 16 Feb 2003 00:31:03 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files
Tom Lane wrote:
> Currently, I don't need to do anything more than set PGDATA or say -D
> to initdb in order to set up the data directory wherever I like.  I also
> don't need to worry about whether I'm selecting the wrong config file.

So in your case, what's the advantage of having initdb write anything
to a config file, when you're probably also relying on PGDATA or -D to
start the database (if you're not, then fair enough.  But see below)?

I'd expect initdb to initialize a database.  If I were running initdb
without a lot of foreknowledge of its side effects, I think I'd
probably be a bit surprised to find that it had touched my config
file.  Initdb doesn't have prior knowledge of how you intend to
start the database so that it refers to the database initdb just
created, so it can't really know whether it's desirable to touch the
config file.

If it's desirable for initdb to be able to write to the config file,
wouldn't it be more appropriate for that to an option that has to be
explicitly enabled on initdb's command line?  I don't know how often
you'd want it to write into the config file for your purposes, but
having it do so automatically seems to violate the principle of least
surprise.



-- 
Kevin Brown					      kevin@sysexperts.com

Sat, 15 Feb 2003 22:37:07 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Tom Lane writes:

> AFAICS, you can either set -C to /etc if you want your PG config files
> loose in /etc, or you can set it to /etc/postgresql/ if you want them
> in a privately-owned directory.  Which other arrangements are needed?

People might want to share them between servers, or allow a user to select
from a few pre-configured ones that which reside in the same directory.

-- 
Peter Eisentraut   peter_e@gmx.net


Sun, 16 Feb 2003 15:10:05 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: location of the configuration files 
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> AFAICS, you can either set -C to /etc if you want your PG config files
>> loose in /etc, or you can set it to /etc/postgresql/ if you want them
>> in a privately-owned directory.  Which other arrangements are needed?

> People might want to share them between servers, or allow a user to select
> from a few pre-configured ones that which reside in the same directory.

You can accomplish that without the need to customize the .conf file
names; you just make, eg,

	/etc/postgres/myconfig/postgresql.conf
	/etc/postgres/yourconfig/postgresql.conf
	/etc/postgres/herconfig/postgresql.conf

(plus additional config files as needed in each of these directories)
and then the postmaster start command is

	postmaster -C /etc/postgres/myconfig

I see no real gain in flexibility in allowing people to choose random
names for the individual config files.  Also, it'd defeat the
ultimate-fallback approach of doing "find / -name postgresql.conf"
to figure out where the config files are hiding in an unfamiliar
installation.

			regards, tom lane

Sun, 16 Feb 2003 12:54:23 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Lamar Owen <lamar.owen@wgcr.org> writes:
> The script's been out there for awhile.  It does some things well, and some 
> things not so well.  The config files are still coresident with the database,
> and backup is more difficult than it can be.  Meeting all these needs (with 
> configure switches, configuration file directives, etc) would be a good 
> thing.

Sure.  I'm happy to change the software in a way that *allows* moving the
config files elsewhere.  But it's not apparent to me why you insist on
forcing people who are perfectly happy with their existing configuration
arrangements to change them.  I have not seen any reason in this
discussion why we can't support both a separate-config-location approach
and the traditional single-location one.

Please remember that the existing approach has been evolved over quite
a few releases.  It may not satisfy the dictates of the FHS religion,
but it does meet some people's needs perfectly well.  Let's look for a
solution that permits coexistence, rather than one that forces change
on people who don't need or want change.

			regards, tom lane

Sun, 16 Feb 2003 13:15:45 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Kevin Brown <kevin@sysexperts.com> writes:
> So in your case, what's the advantage of having initdb write anything
> to a config file, when you're probably also relying on PGDATA or -D to
> start the database (if you're not, then fair enough.  But see below)?

Keep in mind that initdb doesn't currently *need* to put the datadir
location into the config file.  It *will* need to do so if we separate
config and data dirs.  Or at least, *somebody* will need to do so.
It's not apparent to me how it simplifies life not to have initdb do it.
Especially when there are other configuration items that initdb should
or already does record: locale settings, database encoding.  And we
have already been talking about improving PG's self-tuning capability.
initdb would be the natural place to look around for information like
available RAM and adjust the config-file settings like sort_mem
accordingly.

Basically, the notion that initdb shouldn't write a config file seems
like a complete dead end to me.  It cannot possibly be more convenient
than the alternatives.  We'd be giving up a lot of current and future
functionality --- and for what?

> I'd expect initdb to initialize a database.  If I were running initdb
> without a lot of foreknowledge of its side effects, I think I'd
> probably be a bit surprised to find that it had touched my config
> file.

If we do it the way I suggested (dump into the datadir, which is
initially empty, same as always) then it cannot overwrite your existing
config files.  Think of it as providing a suggested config file to
compare against what you have.

			regards, tom lane

Sun, 16 Feb 2003 14:20:38 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
On Sunday 16 February 2003 13:15, Tom Lane wrote:
> Sure.  I'm happy to change the software in a way that *allows* moving the
> config files elsewhere.

So we agree.  Perfect.

>  But it's not apparent to me why you insist on
> forcing people who are perfectly happy with their existing configuration
> arrangements to change them.

Me? Trying to force things to change?  You misunderstand me.  No, I'm trying 
to understand the rationale for a (relative to the way other 
designed-multiple daemons do things) different, non-standard configuration 
process.  I understand better now; the exercise was a success.  Many thanks.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Sun, 16 Feb 2003 18:56:57 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: location of the configuration files


Tom Lane wrote:

>Peter Eisentraut <peter_e@gmx.net> writes:
>  
>
>>Tom Lane writes:
>>    
>>
>>>I would favor a setup that allows a -C *directory* (not file) to be
>>>specified as a postmaster parameter separately from the -D directory;
>>>      
>>>
>
>  
>
>>A directory is not going to satisfy people.
>>    
>>
>
>Why not?  Who won't it satisfy, and what's their objection?
>
>AFAICS, you can either set -C to /etc if you want your PG config files
>loose in /etc, or you can set it to /etc/postgresql/ if you want them
>in a privately-owned directory.  Which other arrangements are needed?
>
>  
>
The idea of using a "directory" puts us back to using symlinks to share 
files.

While I know the core development teams thinks that symlinks are a 
viable configuration option, most admins, myself included, do not like 
to use symlinks because they do not have the ability to carry 
documentation, i.e. comments in a configuration file, and are DANGEROUS 
in a production environment.

Any configuration strategy that depends on symlinks is inadequate and 
poorly designed.


>  
>



Sun, 16 Feb 2003 21:40:08 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
mlw <pgsql@mohawksoft.com> writes:
> The idea of using a "directory" puts us back to using symlinks to share 
> files.

So?  If you want to share files, you're probably sharing all three
config files and don't need a separate directory at all.  This is
not a sufficient argument to make me buy into the mess of letting
people choose nonstandard configuration file names --- especially
when most of the opposite camp seems to be more interested in choosing
*standard* names for things.  Why does that policy stop short at the
directory name?

			regards, tom lane

Sun, 16 Feb 2003 21:48:47 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
Tom Lane wrote:
> Keep in mind that initdb doesn't currently *need* to put the datadir
> location into the config file.  It *will* need to do so if we separate
> config and data dirs.  Or at least, *somebody* will need to do so.
> It's not apparent to me how it simplifies life not to have initdb do it.
> Especially when there are other configuration items that initdb should
> or already does record: locale settings, database encoding.  

Is it possible for the database engine to properly deal with a
database when it is told to use a different database encoding than the
one the database was initdb'd with?

If it's not, then that suggests to me that the database encoding is
something that doesn't belong in the configuration file but rather in
some other place that is intimately tied with the database itself and
which is difficult/impossible to change, like perhaps a read-only
system table that gets created at initdb time.

> And we have already been talking about improving PG's self-tuning
> capability.  initdb would be the natural place to look around for
> information like available RAM and adjust the config-file settings
> like sort_mem accordingly.

I agree here, and since you're thinking of just putting the resulting
config file in the database data directory, then as a DBA I wouldn't
be terribly surprised by it ... especially if it came back with a
message that told me what it had done.

> If we do it the way I suggested (dump into the datadir, which is
> initially empty, same as always) then it cannot overwrite your existing
> config files.  Think of it as providing a suggested config file to
> compare against what you have.

There is one minor complication: what if there's an existing config
file in the target directory?

One use for initdb would be as a quick way to completely wipe the
database and start over (e.g., if the encoding were found to be
incorrect), but the config file that's already there could easily
contain a lot of customization that the administrator would want to
retain.  Which suggests that we should consider writing to a file
using a slightly different name (e.g., postgresql.conf.initdb), at
least in the event that a config file already exists in the target
directory.

Not sure what the overall right thing to do here is...


-- 
Kevin Brown					      kevin@sysexperts.com

Sun, 16 Feb 2003 19:33:24 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files


Tom Lane wrote:

>mlw <pgsql@mohawksoft.com> writes:
>  
>
>>The idea of using a "directory" puts us back to using symlinks to share 
>>files.
>>    
>>
>
>So?  If you want to share files, you're probably sharing all three
>config files and don't need a separate directory at all.  This is
>not a sufficient argument to make me buy into the mess of letting
>people choose nonstandard configuration file names --- especially
>when most of the opposite camp seems to be more interested in choosing
>*standard* names for things.  Why does that policy stop short at the
>directory name?
>  
>
symlinks suck. Sorry Tom, but they are *BAD* in a production server. You 
can not add comments to symlinks. Most of the admins I know, myself 
included, HATE symlinks and use them as a last resort. Requiring 
symlinks is just pointless, we are talking about a few lines of code hat 
has nothing to do with performance.

The patch that I submitted allows PostgreSQL to work as it always has, 
but adds the ability for a configuration file to do what is normally 
done with fixed names in $PGDATA.

I have said before, I do not like policy, I like flexibility, forcing a 
directory is similarly restricting as requiring the files in $PGDATA.

Why is this such a problem? MANY people want to configure PostgreSQL 
this way, but the patch I submitted allows it, but does not force 
anything. Any configuration solution that requires symlinks is flawed.

>  
>


Sun, 16 Feb 2003 22:41:37 -0500
mlw <pgsql@mohawksoft.com>
Re: location of the configuration files
mlw wrote:
> symlinks suck. Sorry Tom, but they are *BAD* in a production
> server. 

Well, at least they're better than hard links.  ;-)


-- 
Kevin Brown					      kevin@sysexperts.com

Sun, 16 Feb 2003 19:55:49 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Kevin Brown <kevin@sysexperts.com> writes:
> Is it possible for the database engine to properly deal with a
> database when it is told to use a different database encoding than the
> one the database was initdb'd with?

It can't be "told to use a different database encoding".  However, the
default *client* encoding matches the database encoding, and that is
something that can be set in the config file.

>> If we do it the way I suggested (dump into the datadir, which is
>> initially empty, same as always) then it cannot overwrite your existing
>> config files.  Think of it as providing a suggested config file to
>> compare against what you have.

> There is one minor complication: what if there's an existing config
> file in the target directory?

If there's anything at all in the target directory, initdb refuses to
run.

			regards, tom lane

Mon, 17 Feb 2003 13:39:07 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
On Fri, Feb 14, 2003 at 10:35:41AM -0500, Tom Lane wrote:

> FHS or no FHS, I would think that the preferred arrangement would be to
> keep Postgres' config files in a postgres-owned subdirectory, not
> directly in /etc.  That way you need not be root to edit them.  (My idea

Besides, what are you going to do for people installing on a box
where they don't have root?  Are they going to need a whole mess of
extra directories in their private copy?

> of an editor, Emacs, always wants to write a backup file, so I dislike
> having to edit files that live in directories I can't write.)
> 
> Here's a pretty topic for a flamewar: should it be /etc/postgres/ or
> /etc/postgresql/ ?

Wow, two flamewar topics in one mail.  I'm impressed.

Andrew "ed is the one true editor" Sullivan

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Mon, 17 Feb 2003 13:51:16 -0500
Andrew Sullivan <andrew@libertyrms.info>
Re: location of the configuration files
On Sun, Feb 16, 2003 at 12:16:44AM -0500, Tom Lane wrote:
> Nor will I buy an argument that only a few developers have need for test
> installations.  Ordinary users will want to do that anytime they are
> doing preliminary tests on a new PG version before migrating their
> production database to it.  To the extent that you make manual selection
> of a nonstandard datadir location more difficult and error-prone, you
> are hurting them too.

Not only that.  For safety's sake, you may need to run multiple
postmasters on one machine (so that database user X can't DoS
database user Y, for instance).  And making that sort of
production-grade work more difficult and error-prone would also be
bad.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Mon, 17 Feb 2003 14:03:31 -0500
Andrew Sullivan <andrew@libertyrms.info>
Re: location of the configuration files
On Sat, 15 Feb 2003, Curt Sampson wrote:

> On Fri, 14 Feb 2003, scott.marlowe wrote:
> 
> > Asking for everything in a directory with the name local in it to be
> > shared is kind of counter intuitive to me.
> 
> Not really. If you install a particular program that doesn't come with
> the OS on one machine on your site, why would you not want to install it
> separately on all of the others?
> 
> Typically, I want my favourite non-OS utilities on all machines, not
> just one. (Even if I don't use them on all machines.) Thus /usr/local is
> for site-local stuff.

Good point.  Of course, in apache, it's quite easy to use the -f switch to 
pick the file you're running on.  so, with a 

httpd -f /usr/local/apache/conf/`uname -a|cut -d " " -f 2`.conf

I can pick and choose the file to run.  So, yes, I would gladly use it in 
a cluster, and all the files would be in one place, easy to backup.


Tue, 18 Feb 2003 10:00:38 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: location of the configuration files

I have a new idea.  You know how we have search_path where you can
specify multiple schema names.  What if we allow the config_dirs/-C to
specify multiple directories to search for config files.  That way, we
can use only one variable, and we can allow people to place different
config files in different directories.

---------------------------------------------------------------------------

Andrew Sullivan wrote:
> On Sun, Feb 16, 2003 at 12:16:44AM -0500, Tom Lane wrote:
> > Nor will I buy an argument that only a few developers have need for test
> > installations.  Ordinary users will want to do that anytime they are
> > doing preliminary tests on a new PG version before migrating their
> > production database to it.  To the extent that you make manual selection
> > of a nonstandard data_dir location more difficult and error-prone, you
> > are hurting them too.
> 
> Not only that.  For safety's sake, you may need to run multiple
> postmasters on one machine (so that database user X can't DoS
> database user Y, for instance).  And making that sort of
> production-grade work more difficult and error-prone would also be
> bad.
> 
> A
> 
> -- 
> ----
> Andrew Sullivan                         204-4141 Yonge Street
> Liberty RMS                           Toronto, Ontario Canada
> <andrew@libertyrms.info>                              M2P 2A8
>                                          +1 416 646 3304 x110
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 18 Feb 2003 21:55:09 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Bruce Momjian wrote:
> 
> I have a new idea.  You know how we have search_path where you can
> specify multiple schema names.  What if we allow the config_dirs/-C to
> specify multiple directories to search for config files.  That way, we
> can use only one variable, and we can allow people to place different
> config files in different directories.

That's an interesting idea.  Were you thinking, perhaps, that you
could put, say, a postgresql.conf file in multiple directories and
have the settings in the latest one override the settings in earlier
ones?  That would mean you could set up a single postgresql.conf that
has settings common to all of your instances (settings related to the
system such as shared buffers, and default settings that would apply
to any instance if not overridden), and a postgresql.conf file for
each instance that defines the instance-specific configuration
information.

I'm not sure that's quite what you had in mind, though.  :-)


-- 
Kevin Brown					      kevin@sysexperts.com

Wed, 19 Feb 2003 04:09:52 -0800
Kevin Brown <kevin@sysexperts.com>
Re: location of the configuration files
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I have a new idea.  You know how we have search_path where you can
> specify multiple schema names.  What if we allow the config_dirs/-C to
> specify multiple directories to search for config files.  That way, we
> can use only one variable, and we can allow people to place different
> config files in different directories.

Hm, a search path for config files?  I could support that if it
satisfies the folks who object to specifying config directories
rather than file names.

One thing that would have to be thought about is whether to re-search
the path on each config file reload --- if you first find pg_hba.conf
in, say, the third directory on the path, should you pay attention if
one materializes in the second directory later?  Or do you keep going
back to the same well?  I can see arguments either way.

			regards, tom lane

Wed, 19 Feb 2003 10:35:56 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 

The problem I have with Bruce's scheme is that you could put your config
file where you want it and someone else puts one somewhere higher in the
search path and you have no idea what went wrong. It sounds to me like a
recipe for an SA's nightmare. Other people have claimed to speak from the SA
perspective - do they see this too?

andrew

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I have a new idea.  You know how we have search_path where you can
> > specify multiple schema names.  What if we allow the config_dirs/-C to
> > specify multiple directories to search for config files.  That way, we
> > can use only one variable, and we can allow people to place different
> > config files in different directories.
>
> Hm, a search path for config files?  I could support that if it
> satisfies the folks who object to specifying config directories
> rather than file names.
>
> One thing that would have to be thought about is whether to re-search
> the path on each config file reload --- if you first find pg_hba.conf
> in, say, the third directory on the path, should you pay attention if
> one materializes in the second directory later?  Or do you keep going
> back to the same well?  I can see arguments either way.


Wed, 19 Feb 2003 10:58:20 -0500
"Andrew Dunstan" <andrew@dunslane.net>
Re: location of the configuration files 
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I have a new idea.  You know how we have search_path where you can
> > specify multiple schema names.  What if we allow the config_dirs/-C to
> > specify multiple directories to search for config files.  That way, we
> > can use only one variable, and we can allow people to place different
> > config files in different directories.
> 
> Hm, a search path for config files?  I could support that if it
> satisfies the folks who object to specifying config directories
> rather than file names.
> 
> One thing that would have to be thought about is whether to re-search
> the path on each config file reload --- if you first find pg_hba.conf
> in, say, the third directory on the path, should you pay attention if
> one materializes in the second directory later?  Or do you keep going
> back to the same well?  I can see arguments either way.

Oh, I hadn't thought of that.  I would vote for researching the path,
but I am not sure why.

The bigger question is whether you can modify config_dirs while the
postmaster is running.  I would think not.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 19 Feb 2003 10:58:22 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: location of the configuration files
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> The bigger question is whether you can modify config_dirs while the
> postmaster is running.  I would think not.

There would be no way to do that, because the only way to set it would
be from -C on the command line or a PGCONFIG environment variable.
But I can't see a real good reason why you'd need to.

			regards, tom lane

Wed, 19 Feb 2003 11:02:06 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
"Andrew Dunstan" <andrew@dunslane.net> writes:
> The problem I have with Bruce's scheme is that you could put your config
> file where you want it and someone else puts one somewhere higher in the
> search path and you have no idea what went wrong. It sounds to me like a
> recipe for an SA's nightmare. Other people have claimed to speak from the SA
> perspective - do they see this too?

If you have "your" file you'd put it in the directory at the front of
the search path.  End of problem.  Any additional directories would be
for config files that you *want* to share.

Offhand I find it hard to visualize needing more than two directories in
this path (private and shared), unless we grow to having many more
config files than we do now.  But we may as well build the feature with
no artificial restriction about path length.

Search path management seems well understood for $PATH --- people do
get burnt by having the wrong $PATH, but it doesn't qualify as a
nightmare...

			regards, tom lane

Wed, 19 Feb 2003 11:40:20 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: location of the configuration files 
After failing to make Itanium competitive, Intel is now downplaying
64-bit CPU's.  Of course, they didn't think that until Itanium failed. 
Here is the slashdot story:


	http://slashdot.org/article.pl?sid=03/02/23/2050237&mode=nested&tid=118

Seems AMD's hammer is going to be the popular 64-bit desktop CPU.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 24 Feb 2003 18:49:37 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Intel drops 64-bit Itanium
> After failing to make Itanium competitive, Intel is now downplaying
> 64-bit CPU's.  Of course, they didn't think that until Itanium failed.
> Here is the slashdot story:
>   http://slashdot.org/article.pl?sid=03/02/23/2050237&mode=nested&tid=118
> 
> Seems AMD's hammer is going to be the popular 64-bit desktop CPU.

It's really unsurprising; there was /no/ likelihood of Itanium getting
widely deployed on desktops when there would be an absolute dearth of
desktop software.

Think back: Alpha was presented in /exactly/ the same role, years ago,
and the challenges it had vis-a-vis:

 a) Need for emulation to run legacy software that can't get recompiled;
 b) Need to deploy varying binaries on the substantially varying
    platforms;
 c) It's real costly to be an early adoptor of new hardware, so the
    hardware is expensive stuff.

Certain sorts of "enterprise" software got deployed on Alpha, but you
never got the ordinary stuff like MS Office and such, which meant there
was no point to anyone pushing "desktop" software to Alpha.  And we
thereby had the result that Alpha became server-only.

Why should it be the slightest bit remarkable that IA-64 is revisiting
the very same marketing challenges?  

It has the very same set of technical challenges.

It may well be that by the time it /is/ time to generally deploy IA-64,
it will have become the Alpha platform.  After all, Compaq sold the
architecture to Intel, and Alpha already has a mature set of hardware
designs as well as compilers...
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://www3.sympatico.ca/cbbrowne/oses.html
"Everything should be built top-down, except the first time."
-- Alan Perlis

Mon, 24 Feb 2003 22:25:04 -0500
cbbrowne@cbbrowne.com
Re: Intel drops 64-bit Itanium 
http://www.theinquirer.net/?article=7966


On Mon, 2003-02-24 at 22:25, cbbrowne@cbbrowne.com wrote:
> > After failing to make Itanium competitive, Intel is now downplaying
> > 64-bit CPU's.  Of course, they didn't think that until Itanium failed.
> > Here is the slashdot story:
> >   http://slashdot.org/article.pl?sid=03/02/23/2050237&mode=nested&tid=118
> > 
> > Seems AMD's hammer is going to be the popular 64-bit desktop CPU.
> 
> It's really unsurprising; there was /no/ likelihood of Itanium getting
> widely deployed on desktops when there would be an absolute dearth of
> desktop software.
> 
> Think back: Alpha was presented in /exactly/ the same role, years ago,
> and the challenges it had vis-a-vis:
> 
>  a) Need for emulation to run legacy software that can't get recompiled;
>  b) Need to deploy varying binaries on the substantially varying
>     platforms;
>  c) It's real costly to be an early adoptor of new hardware, so the
>     hardware is expensive stuff.
> 
> Certain sorts of "enterprise" software got deployed on Alpha, but you
> never got the ordinary stuff like MS Office and such, which meant there
> was no point to anyone pushing "desktop" software to Alpha.  And we
> thereby had the result that Alpha became server-only.
> 
> Why should it be the slightest bit remarkable that IA-64 is revisiting
> the very same marketing challenges?  
> 
> It has the very same set of technical challenges.
> 
> It may well be that by the time it /is/ time to generally deploy IA-64,
> it will have become the Alpha platform.  After all, Compaq sold the
> architecture to Intel, and Alpha already has a mature set of hardware
> designs as well as compilers...
> --
> (reverse (concatenate 'string "gro.gultn@" "enworbbc"))
> http://www3.sympatico.ca/cbbrowne/oses.html
> "Everything should be built top-down, except the first time."
> -- Alan Perlis
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
-- 
Dave Cramer <dave@fastcrypt.com>
Cramer Consulting


25 Feb 2003 07:02:00 -0500
Dave Cramer <dave@fastcrypt.com>
Re: Intel drops 64-bit Itanium response from Linus

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Patch to add XML output to psql:

http://www.gtsm.com/xml.patch.txt

Notes and questions:

The basic output looks something like this:

<?xml version="1.0" encoding="SQL_ASCII"?>
<resultset psql_version="7.4devel" query="select * from foo;">

<columns>
  <col num="1">a</col>
  <col num="2">b</col>
  <col num="3">c</col>
  <col num="4">mucho nacho  </col>
</columns>
<row num="1">
  <a>1</a>
  <b>pizza</b>
  <c>2003-02-25 15:19:22.169797</c>
  <"mucho nacho  "></"mucho nacho  ">
</row>
<row num="2">
  <a>2</a>
  <b>mushroom</b>
  <c>2003-02-25 15:19:26.969415</c>
  <"mucho nacho  "></"mucho nacho  ">
</row>
<footer>(2 rows)</footer>
</resultset>

and with the \x option:

<?xml version="1.0" encoding="SQL_ASCII"?>
<resultset psql_version="7.4devel" query="select * from foo;">

<columns>
  <col num="1">a</col>
  <col num="2">b</col>
  <col num="3">c</col>
  <col num="4">mucho nacho  </col>
</columns>
<row num="1">
  <cell name="a">1</cell>
  <cell name="b">pizza</cell>
  <cell name="c">2003-02-25 15:19:22.169797</cell>
  <cell name="mucho nacho  "></cell>
</row>
<row num="2">
  <cell name="a">2</cell>
  <cell name="b">mushroom</cell>
  <cell name="c">2003-02-25 15:19:26.969415</cell>
  <cell name="mucho nacho  "></cell>
</row>
</resultset>


The default encoding "SQL-ASCII" is not valid for XML. 
Should it be automatically changed to something else?

The flag "-X" is already taken, unfortunately, although \X is not. 
I used "-L" and "\L" but they are not as memorable as "X". Anyone 
see a way around this? Can we still use \X inside of psql?


It would be nice to include the string representation of the column 
types in the xml output:
<col type="int8">foo</col>
....but I could not find an easy way to do this: PQftype returns the 
OID only (which is close but not quite there). Is there an 
existing way to get the name of the type of a column from a 
PQresult item?

The HTML, XML, and Latex modes should have better documentation - 
I'll submit a separate doc patch when/if this gets finalized.


- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200302261518

-----BEGIN PGP SIGNATURE-----
Comment: http://www.turnstep.com/pgp.html

iD8DBQE+XSR/vJuQZxSWSsgRAi2jAJ9IAKnMBmNcVEEI8TXQBBd/rtm4XQCg0Vjq
IO9OsCSkdnNJqnrYYutM3jw=
=9kwY
-----END PGP SIGNATURE-----



Wed, 26 Feb 2003 20:46:15 -0000
greg@turnstep.com
XML ouput for psql
greg@turnstep.com kirjutas K, 26.02.2003 kell 22:46:

> 
> and with the \x option:
> 
> <?xml version="1.0" encoding="SQL_ASCII"?>
> <resultset psql_version="7.4devel" query="select * from foo;">
> 
> <columns>
>   <col num="1">a</col>
>   <col num="2">b</col>
>   <col num="3">c</col>
>   <col num="4">mucho nacho  </col>
> </columns>
> <row num="1">
>   <cell name="a">1</cell>
>   <cell name="b">pizza</cell>
>   <cell name="c">2003-02-25 15:19:22.169797</cell>
>   <cell name="mucho nacho  "></cell>
> </row>
> <row num="2">
>   <cell name="a">2</cell>
>   <cell name="b">mushroom</cell>
>   <cell name="c">2003-02-25 15:19:26.969415</cell>
>   <cell name="mucho nacho  "></cell>
> </row>
> </resultset>
> 
> 
> The default encoding "SQL-ASCII" is not valid for XML. 
> Should it be automatically changed to something else?

I think you should force conversion to something standard, try using
automatic conversion to some known client encoding.

btw, "UNICODE" is also not any known encoding in XML, but PostgreSQL
uses it to mean utf-8

> The flag "-X" is already taken, unfortunately, although \X is not. 
> I used "-L" and "\L" but they are not as memorable as "X". Anyone 
> see a way around this? Can we still use \X inside of psql?
> 
> 
> It would be nice to include the string representation of the column 
> types in the xml output:
> <col type="int8">foo</col>
> ....but I could not find an easy way to do this: PQftype returns the 
> OID only (which is close but not quite there). Is there an 
> existing way to get the name of the type of a column from a 
> PQresult item?

Run "select  oid,typname from pg_type;" first if run in xml mode and
store the oid/columnname pairs.

you could also store the result in ~/.psql for faster access later on
and manually clear it if new types are defined

----------------
Hannu


26 Feb 2003 23:17:11 +0200
Hannu Krosing <hannu@tm.ee>
Re: XML ouput for psql
greg@turnstep.com writes:

> Patch to add XML output to psql:

This would get me more excited if you do one or both of the following:

1. Look into the SQL/XML standard draft (ftp.sqlstandards.org) to find out
whether the standard addresses this sort of thing.

2. Use an established/standardized XML (or SGML) table model rather than
rolling your own.

Incidentally, the HTML table model is such an established and standardized
XML and SGML table model, so the easiest way to get the task "add XML
output to psql" done is to update the HTML output to conform to XHTML.
That way you get both the strict XML and you can look at the formatted
result with any old (er, new) browser.

-- 
Peter Eisentraut   peter_e@gmx.net


Thu, 27 Feb 2003 14:50:48 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: XML ouput for psql

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hannu Krosing wrote:
> I think you should force conversion to something standard, try using
> automatic conversion to some known client encoding.

I've thought about this some more, and the only thing I can think 
about doing without being too heavy-handed is to change the encoding 
to "US-ASCII" whenever someone enters "XML" mode if the encoding is set 
to "SQL-ASCII". Perhaps with a warning.

"The character set most commonly use in the Internet and used especially in 
protocol standards is US-ASCII, this is strongly encouraged."
http://www.iana.org/assignments/character-sets

On the other hand, SQLX seems to lean toward a strict unicode encoding 
(see my reply to Peter Eisentraut for more on that).

> Run "select oid,typname from pg_type;" first if run in xml mode and
> store the oid/columnname pairs.

I realize that I could run a SQL query against pg_type to grab the info, 
but I was hoping there was an internal function similar to PQtype which 
would return the information.

> you could also store the result in ~/.psql for faster access 
> later on and manually clear it if new types are defined

Not only does pg_type has literally hundreds of entries, but there is no 
way to guarantee that these are correct at the time when the query is 
run, so I don't think this is viable.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200302280938
-----BEGIN PGP SIGNATURE-----
Comment: http://www.turnstep.com/pgp.html

iD8DBQE+X3hCvJuQZxSWSsgRArMTAKChouxnFF1ugI1mutXYJf14p1ICGwCfUDG9
yISxrIvqxnYWHfvD0lOWZAQ=
=M6nd
-----END PGP SIGNATURE-----




Fri, 28 Feb 2003 15:00:47 -0000
greg@turnstep.com
Re: XML ouput for psql

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Peter Eisentraut wrote:
> 1. Look into the SQL/XML standard draft (ftp.sqlstandards.org) to find out
> whether the standard addresses this sort of thing.

The URL you gave leads to a site curiously content-free and full of dead links. 
I've looked around a bit, but found nothing definitive. One good resource I 
did find was this:

http://www.wiscorp.com/sql/SQLX_Bringing_SQL_and_XML_Together.pdf

The article mentions a lot of links on the sqlstandards.org and iso.org sites, none 
of which work or are restricted. If anyone knows of some good links, please 
let me know. (especially ISO 9075). From what I've read of the SQLX stuff, the 
format in my patch should be mostly standard:

<row>
 <name>Joe Sixpack</name>
 <age>35</age>
 <state>Alabama</state>
</row>

One problem is that the recommended way to handle non-standard characters 
(including spaces) is to escape them like this:

foobar baz => <foobar_x0020_baz>

This also includes escaping things like "_x*" and "xml*". We don't have 
anything like that in the code yet (?), but we should probably think about 
heading that way. I think escaping whitespace in quotes is good enough 
for now for:

foobar baz => <"foobar baz">

The xsd and xsi standards are also interesting, but needlessly complicated 
for psql output, IMO.

> Incidentally, the HTML table model is such an established and standardized
> XML and SGML table model, so the easiest way to get the task "add XML
> output to psql" done is to update the HTML output to conform to XHTML.
> That way you get both the strict XML and you can look at the formatted
> result with any old (er, new) browser.

I don't agree with this: XML and XHTML are two different things. We could 
certainly upgrade the HTML portion, but I am pretty sure that the XML 
standard calls for this format:

<columnname>data here</columnname>

..which is not valid XHTML and won't be viewable by any browser. The other 
suggested XML formats are even further from XHTML than the above. The HTML 
format should be "html table/layout" specific and the XML should be 
"schema/data" specific.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200302280938

-----BEGIN PGP SIGNATURE-----
Comment: http://www.turnstep.com/pgp.html

iD8DBQE+X3k5vJuQZxSWSsgRAuXFAKDGO1IsjB9Lwtkcws1xJy47PibcLQCg3dx5
fsy27qguZv841lPvCjzdUic=
=4f9B
-----END PGP SIGNATURE-----



Fri, 28 Feb 2003 15:03:14 -0000
greg@turnstep.com
Re: XML ouput for psql
OSDL has just come out with a set of open-source database benchmarks:
http://www.osdl.org/projects/performance/

The bad news:
"This tool kit works with SAP DB open source database versions 7.3.0.23
or 7.3.0.25."

(In fact, they seem to think they are testing kernel performance, not
database performance, which strikes me as rather bizarre.  But anyway.)

The good news:
"We are planning to port this test kit to other databases."

Perhaps someone around here should help out...

			regards, tom lane

Mon, 03 Mar 2003 00:16:36 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Yet another open-source benchmark
Tom Lane wrote:
> OSDL has just come out with a set of open-source database benchmarks:
> http://www.osdl.org/projects/performance/
> 
> The bad news:
> "This tool kit works with SAP DB open source database versions 7.3.0.23
> or 7.3.0.25."
> 
> (In fact, they seem to think they are testing kernel performance, not
> database performance, which strikes me as rather bizarre.  But anyway.)
> 
> The good news:
> "We are planning to port this test kit to other databases."
> 
> Perhaps someone around here should help out...

Yep, this is the group that have hit a performance limit with SAPDB and 
are 100% definitely looking to move it to PostgreSQL, *if* they can get 
people to assist them.

:-)

Regards and best wishes,

Justin Clift

> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi


Mon, 03 Mar 2003 15:55:40 +1030
Justin Clift <justin@postgresql.org>
Re: Yet another open-source benchmark
> OSDL has just come out with a set of open-source database benchmarks:
> http://www.osdl.org/projects/performance/
> 
> The bad news:
> "This tool kit works with SAP DB open source database versions 7.3.0.23
> or 7.3.0.25."
> 
> (In fact, they seem to think they are testing kernel performance, not
> database performance, which strikes me as rather bizarre.  But anyway.)

That may be a terminology thing; the main SAP-DB process is called the 
"kernel," and it's more than likely that the "SAP-DB Kernel" is the sense in 
which the term is being used.

When they translate things from German, sometimes wordings change :-).
--
output = reverse("moc.enworbbc@" "enworbbc")
http://www.ntlug.org/~cbbrowne/linuxxian.html
Rules  of the  Evil Overlord  #41. "Once  my power  is secure,  I will
destroy all those pesky time-travel devices."
<http://www.eviloverlord.com/>



Mon, 03 Mar 2003 07:41:05 -0500
cbbrowne@cbbrowne.com
Re: Yet another open-source benchmark 
On the results page they list kernels like linux-2.4.18-1tier or 
linux-2.4.19-rc2 or redhat-stock-2.4.7-10cmp. This sounds really like 
linux-kernel-versions.

Am Montag, 3. März 2003 13:41 schrieb cbbrowne@cbbrowne.com:
> > OSDL has just come out with a set of open-source database benchmarks:
> > http://www.osdl.org/projects/performance/
> >
> > The bad news:
> > "This tool kit works with SAP DB open source database versions 7.3.0.23
> > or 7.3.0.25."
> >
> > (In fact, they seem to think they are testing kernel performance, not
> > database performance, which strikes me as rather bizarre.  But anyway.)
>
> That may be a terminology thing; the main SAP-DB process is called the
> "kernel," and it's more than likely that the "SAP-DB Kernel" is the sense
> in which the term is being used.
>
> When they translate things from German, sometimes wordings change :-).
> --
> output = reverse("moc.enworbbc@" "enworbbc")
> http://www.ntlug.org/~cbbrowne/linuxxian.html
> Rules  of the  Evil Overlord  #41. "Once  my power  is secure,  I will
> destroy all those pesky time-travel devices."
> <http://www.eviloverlord.com/>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org

-- 
Dr. Eckhardt + Partner GmbH
http://www.epgmbh.de

Mon, 3 Mar 2003 14:53:50 +0100
Tommi Maekitalo <t.maekitalo@epgmbh.de>
Re: Yet another open-source benchmark
greg@turnstep.com writes:

> I don't agree with this: XML and XHTML are two different things.

No one claimed anything to the contrary.

> We could certainly upgrade the HTML portion, but I am pretty sure that
> the XML standard calls for this format:
>
> <columnname>data here</columnname>

The XML standard does not call for any table format.  But a number of
table formats have been established within the XML framework.  Some of
them are formatting-oriented (e.g., the HTML model, or CALS which is used
in DocBook) and some of them are processing-oriented (e.g., SQL/XML).
Which do we need?  And which do we need from psql in particular (keeping
in mind that psql is primarily for interactive use and shell-scripting)?
In any case, it should most likely be a standard table model and not a
hand-crafted one.

(If, for whatever reason, we go the "processing-oriented" route, then I
claim that there should not be a different output with and without \x
mode.)

-- 
Peter Eisentraut   peter_e@gmx.net


Mon, 3 Mar 2003 18:55:12 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [PATCHES] XML ouput for psql
On Mon, 3 Mar 2003, Tommi Maekitalo wrote:

> On the results page they list kernels like linux-2.4.18-1tier or 
> linux-2.4.19-rc2 or redhat-stock-2.4.7-10cmp. This sounds really like 
> linux-kernel-versions.
> 
> Am Montag, 3. März 2003 13:41 schrieb cbbrowne@cbbrowne.com:
> > > OSDL has just come out with a set of open-source database benchmarks:
> > > http://www.osdl.org/projects/performance/
> > >
> > > The bad news:
> > > "This tool kit works with SAP DB open source database versions 7.3.0.23
> > > or 7.3.0.25."
> > >
> > > (In fact, they seem to think they are testing kernel performance, not
> > > database performance, which strikes me as rather bizarre.  But anyway.)
> >
> > That may be a terminology thing; the main SAP-DB process is called the
> > "kernel," and it's more than likely that the "SAP-DB Kernel" is the sense
> > in which the term is being used.
> >
> > When they translate things from German, sometimes wordings change :-).
> > --
> > output = reverse("moc.enworbbc@" "enworbbc")
> > http://www.ntlug.org/~cbbrowne/linuxxian.html
> > Rules  of the  Evil Overlord  #41. "Once  my power  is secure,  I will
> > destroy all those pesky time-travel devices."
> > <http://www.eviloverlord.com/>

I think they are testing how tuning the linux kernel impacts the database 
running on top, at least that's the feeling I got from the site.


Mon, 3 Mar 2003 11:17:01 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Yet another open-source benchmark
On Mon, 2003-03-03 at 07:41, cbbrowne@cbbrowne.com wrote:
> > (In fact, they seem to think they are testing kernel performance, not
> > database performance, which strikes me as rather bizarre.  But anyway.)
> 
> That may be a terminology thing; the main SAP-DB process is called the 
> "kernel," and it's more than likely that the "SAP-DB Kernel" is the sense in 
> which the term is being used.

Actually, I believe the reason the benchmark was developed was to
provide a workload for optimizing high-end Linux kernel performance
(with the inference being that SAP-DB is pretty close to Oracle, Oracle
performance is important for enterprise deployment of Linux, and
therefore optimizing the kernel's handling of SAP-DB running TPC
benchmarks will tend to improve the kernel's performance running
Oracle/DB2/etc.) So when they mean "kernel", I think they really mean
"kernel".

That's not to say that the benchmark wouldn't be useful for doing other
stuff, like pure database benchmarks (as long as its a valid
implementation of TPC-C (or TPC-H, etc.), it should be fine...)

A research group at the university I attend (www.queensu.ca) expressed
some interested in a TPC-C implementation for PostgreSQL, so I was
planning to port the OSDL TPC-C implementation to PostgreSQL.
Unfortunately, I got sidetracked for a couple reasons: (1) lack of time
(2) increasing awareness of just how boring writing benchmark apps is
:-) (3) distaste for ODBC. While I'd like to get some time to do the
port in the future, that shouldn't stop anyone else from doing so in the
mean time :-)

Cheers,

Neil
-- 
Neil Conway <neilc@samurai.com> || PGP Key ID: DB3C29FC




03 Mar 2003 15:29:55 -0500
Neil Conway <neilc@samurai.com>
Re: Yet another open-source benchmark
On Mon, 2003-03-03 at 12:29, Neil Conway wrote:
> On Mon, 2003-03-03 at 07:41, cbbrowne@cbbrowne.com wrote:
> > > (In fact, they seem to think they are testing kernel performance, not
> > > database performance, which strikes me as rather bizarre.  But anyway.)
> > 
> > That may be a terminology thing; the main SAP-DB process is called the 
> > "kernel," and it's more than likely that the "SAP-DB Kernel" is the sense in 
> > which the term is being used.
> 
> Actually, I believe the reason the benchmark was developed was to
> provide a workload for optimizing high-end Linux kernel performance
> (with the inference being that SAP-DB is pretty close to Oracle, Oracle
> performance is important for enterprise deployment of Linux, and
> therefore optimizing the kernel's handling of SAP-DB running TPC
> benchmarks will tend to improve the kernel's performance running
> Oracle/DB2/etc.) So when they mean "kernel", I think they really mean
> "kernel".

Yeah, Neil more-or-less hit it on the nose.  The SAP DB folks do refer
to their processes as kernel processes, but our focus is on the Linux
kernel and helping Linux gain more ground for the enterprise.
 
> That's not to say that the benchmark wouldn't be useful for doing other
> stuff, like pure database benchmarks (as long as its a valid
> implementation of TPC-C (or TPC-H, etc.), it should be fine...)
> 
> A research group at the university I attend (www.queensu.ca) expressed
> some interested in a TPC-C implementation for PostgreSQL, so I was
> planning to port the OSDL TPC-C implementation to PostgreSQL.
> Unfortunately, I got sidetracked for a couple reasons: (1) lack of time
> (2) increasing awareness of just how boring writing benchmark apps is
> :-) (3) distaste for ODBC. While I'd like to get some time to do the
> port in the future, that shouldn't stop anyone else from doing so in the
> mean time :-)

And we're prepared to aid any effort. :)
 
> Cheers,
> 
> Neil
> -- 
> Neil Conway <neilc@samurai.com> || PGP Key ID: DB3C29FC
> 
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html

-- 
Mark Wong - - markw@osdl.org
Open Source Development Lab Inc - A non-profit corporation
15275 SW Koll Parkway - Suite H - Beaverton OR, 97006
(503)-626-2455 x 32 (office)
(503)-626-2436      (fax)
http://www.osdl.org/archive/markw/


03 Mar 2003 13:20:26 -0800
Mark Wong <markw@osdl.org>
Re: Yet another open-source benchmark
> greg@turnstep.com writes:
> 
> > I don't agree with this: XML and XHTML are two different things.
> 
> No one claimed anything to the contrary.
> 
> > We could certainly upgrade the HTML portion, but I am pretty sure that
> > the XML standard calls for this format:
> >
> > <columnname>data here</columnname>
> 
> The XML standard does not call for any table format.  But a number of
> table formats have been established within the XML framework.  Some of
> them are formatting-oriented (e.g., the HTML model, or CALS which is used
> in DocBook) and some of them are processing-oriented (e.g., SQL/XML).
> Which do we need?  And which do we need from psql in particular (keeping
> in mind that psql is primarily for interactive use and shell-scripting)?
> In any case, it should most likely be a standard table model and not a
> hand-crafted one.

I would expect XML output to be based on whatever the tree of data
contained.

If the tree is to be rewritten, then this would mean having some sort of
transformation engine in PostgreSQL that you would have to program.

If I want a CALS table, then I'll push CALS table data into the
database.

If I'm storing a GnuCash chart of accounts in PostgreSQL, I am
ludicrously uninterested in seeing it rewritten for some sort of
physical layout.  Spit out the tags that are stored in the database, not
some rewriting of it.
--
(reverse (concatenate 'string "moc.enworbbc@" "enworbbc"))
http://cbbrowne.com/info/linuxdistributions.html
(1) Sigs are preceded by the "sigdashes" line, ie "\n-- \n" (dash-dash-space).
(2) Sigs contain at least the name and address of the sender in the first line.
(3) Sigs are at most four lines and at most eighty characters per line.

Mon, 03 Mar 2003 18:57:26 -0500
cbbrowne@cbbrowne.com
Re: [PATCHES] XML ouput for psql 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> The XML standard does not call for any table format.  But a number of
> table formats have been established within the XML framework.  Some of
> them are formatting-oriented (e.g., the HTML model, or CALS which is used
> in DocBook) and some of them are processing-oriented (e.g., SQL/XML).
> Which do we need?  And which do we need from psql in particular (keeping
> in mind that psql is primarily for interactive use and shell-scripting)?
> In any case, it should most likely be a standard table model and not a
> hand-crafted one.
 
I think all psql needs is a simple output, similar to the ones used by 
Oracle, Sybase, and MySQL; the calling application should then process 
it in some way as needed (obviously this is not for interactive use).
Where can one find a "standard table model?"

All of the DBs I mentioned (and the perl module DBIx:XML_RDB) all share 
a similar theme, with subtle differences (i.e. some use <row>, some 
<row num="x">, some have <rowset>). I'd be happy to write whatever 
format we can find or develop. My personal vote is the DBIx::XML_RDB 
format, perhaps with the row number that Oracle uses, producing this:

<?xml version="1.0"?>
<RESULTSET statement="select * from xmltest">
<ROW num="1">
 <scoops>3</scoops>
 <flavor>chocolate</flavor>
</ROW>
<ROW num="2">
 <scoops>2</scoops>
 <flavor>vanilla</flavor>
</ROW>
</RESULTSET>


> (If, for whatever reason, we go the "processing-oriented" route, then I
> claim that there should not be a different output with and without \x
> mode.)

I agree with this.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200303041444
-----BEGIN PGP SIGNATURE-----
Comment: http://www.turnstep.com/pgp.html

iD8DBQE+ZQJNvJuQZxSWSsgRArGEAKD4xs+4Ns3syG175T3k80B6MvNJvgCbBkvF
hCkf5SMjLzMJ84uMl1w4tMY=
=a2Uq
-----END PGP SIGNATURE-----



Tue,  4 Mar 2003 19:50:12 -0000
greg@turnstep.com
Re: XML ouput for psql
* greg@turnstep.com <greg@turnstep.com> [2003-03-04 14:21]:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> > The XML standard does not call for any table format.  But a number of
> > table formats have been established within the XML framework.  Some of
> > them are formatting-oriented (e.g., the HTML model, or CALS which is used
> > in DocBook) and some of them are processing-oriented (e.g., SQL/XML).
> > Which do we need?  And which do we need from psql in particular (keeping
> > in mind that psql is primarily for interactive use and shell-scripting)?
> > In any case, it should most likely be a standard table model and not a
> > hand-crafted one.
>  
> I think all psql needs is a simple output, similar to the ones used by 
> Oracle, Sybase, and MySQL; the calling application should then process 
> it in some way as needed (obviously this is not for interactive use).
> Where can one find a "standard table model?"
> 
> All of the DBs I mentioned (and the perl module DBIx:XML_RDB) all share 
> a similar theme, with subtle differences (i.e. some use <row>, some 
> <row num="x">, some have <rowset>). I'd be happy to write whatever 
> format we can find or develop. My personal vote is the DBIx::XML_RDB 
> format, perhaps with the row number that Oracle uses, producing this:
> 
> <?xml version="1.0"?>
> <RESULTSET statement="select * from xmltest">
> <ROW num="1">
>  <scoops>3</scoops>
>  <flavor>chocolate</flavor>
> </ROW>
> <ROW num="2">
>  <scoops>2</scoops>
>  <flavor>vanilla</flavor>
> </ROW>
> </RESULTSET>
> 
> 
> > (If, for whatever reason, we go the "processing-oriented" route, then I
> > claim that there should not be a different output with and without \x
> > mode.)
> 
> I agree with this.

I'm interested in creating XML documents that have heirarcy.
I can produce the above with Perl.

Acually, the difficult part has been getting the information back
into the database. Getting it out is a very simple query. I imagine
that every language/environment has an SQL->XML library somewhere,
but I wasn't able to find something that would go from XML to SQL.

I wrote a utility that takes an xml document, and xml configuration
file, and writes the document to a PostgerSQL data base using the
configuration file to figure out what goes where. The configuration
file makes some use of XPath to pluck the correct values out of the
xml doucment.

I suppose the same code could generate a document, but it is so easy
to do using Perl and cgi, I've not bothered.

It has some constraints, but it is a very useful utility. I've been
able to abosorb XML documents into my PostgreSQL db just by tweeking
the configuration file.

Currently, I am porting it to C++ from Perl.

-- 
Alan Gutierrez - ajglist@izzy.net
http://khtml-win32.sourceforge.net/ - KHTML on Windows

Tue, 4 Mar 2003 14:44:39 -0600
Alan Gutierrez <ajglist@izzy.net>
Re: XML ouput for psql
* greg@turnstep.com <greg@turnstep.com> [2003-03-04 14:21]:
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1

> > The XML standard does not call for any table format.  But a
> > number of table formats have been established within the XML
> > framework.  Some of them are formatting-oriented (e.g., the HTML
> > model, or CALS which is used in DocBook) and some of them are
> > processing-oriented (e.g., SQL/XML).  Which do we need?  And
> > which do we need from psql in particular (keeping in mind that
> > psql is primarily for interactive use and shell-scripting)?  In
> > any case, it should most likely be a standard table model and
> > not a hand-crafted one.
>  
> I think all psql needs is a simple output, similar to the ones used by 
> Oracle, Sybase, and MySQL; the calling application should then process 
> it in some way as needed (obviously this is not for interactive use).
> Where can one find a "standard table model?"
> 
> All of the DBs I mentioned (and the perl module DBIx:XML_RDB) all share 
> a similar theme, with subtle differences (i.e. some use <row>, some 
> <row num="x">, some have <rowset>). I'd be happy to write whatever 
> format we can find or develop. My personal vote is the DBIx::XML_RDB 
> format, perhaps with the row number that Oracle uses, producing this:
> 
> <?xml version="1.0"?>
> <RESULTSET statement="select * from xmltest">
> <ROW num="1">
>  <scoops>3</scoops>
>  <flavor>chocolate</flavor>
> </ROW>
> <ROW num="2">
>  <scoops>2</scoops>
>  <flavor>vanilla</flavor>
> </ROW>
> </RESULTSET>
> 
> 
> > (If, for whatever reason, we go the "processing-oriented" route, then I
> > claim that there should not be a different output with and without \x
> > mode.)
> 
> I agree with this.

I'm interested in creating XML documents that have heirarcy.
I can produce the above with Perl.

I wrote a utility that takes an xml document, and xml configuration
file, and writes the document to a PostgerSQL data base using the
configuration file to figure out what goes where. The configuration
file makes some use of XPath to pluck the correct values out of the
xml doucment.

I suppose the same code could generate a document, but it is so easy
to do using Perl and cgi, I've not bothered.

This util has been very helpful to me in developing a document
mangement application. Rather than writing insert/update logic every
time the db or xml schema changes, I just tweak the config file and
it will generated the inserts, updates, and deletes by comparing the
XML document with the tables to which the XML elements are mapped.

I've been able to handle tree structures tolerably well.

I am currently rewriting the code in C++ from Perl.

-- 
Alan Gutierrez - ajglist@izzy.net
http://khtml-win32.sourceforge.net/ - KHTML on Windows

Tue, 4 Mar 2003 17:27:03 -0600
Alan Gutierrez <ajglist@izzy.net>
Re: XML ouput for psql
I've done a lot with XML lately, so I'll throw in my $0.02 worth.

One thing I have noticed about the schemes that are being advanced is that
they seem to be inherently unspecifiable, formally, because column names are
being used as tags.

An alternative might look something like this:

<?xml version="1.0"?>
<RESULTSET statement="select * from xmltest">
<COLUMNS>
  <COLUMN name="scoops" type="int" />
  <COLUMN name="flavor" type="varchar(40)" />
</COLUMNS>
<ROW>
 <FIELD name="scoops" isNull="false">3</FIELD>
 <FIELD name="flavor" isNull="false">chocolate</FIELD>
</ROW>
<ROW>
 <FIELD name="scoops" isNull="false">2</FIELD>
 <FIELD name="flavor" isNull="false">vanilla</FIELD>
</ROW>
</RESULTSET>


Numbering the rows should be redundant (XPath will give it to you using
"position()", for example). OTOH, reporting out a null value as opposed to
an empty one is probably a good idea.

The formal DTD would be something like this (courtesy of the wonderful tools
at http://www.hitsw.com/xml_utilites/:

<!ELEMENT RESULTSET ( COLUMNS, ROW* ) >
<!ATTLIST RESULTSET statement CDATA #REQUIRED >
<!ELEMENT COLUMNS ( COLUMN+ ) >

<!ELEMENT COLUMN EMPTY >
<!ATTLIST COLUMN name NMTOKEN #REQUIRED >
<!ATTLIST COLUMN type CDATA #REQUIRED >

<!ELEMENT ROW ( FIELD+ ) ><!ELEMENT FIELD ( #PCDATA ) >
<!ATTLIST FIELD isNull ( false| true ) "false" >
<!ATTLIST FIELD name NMTOKEN #REQUIRED >
 or the equivalent in a schema:<?xml version="1.0" encoding="UTF-8" ?>

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="COLUMN">
    <xs:complexType>
      <xs:attribute name="type" type="xs:string" use="required" />
      <xs:attribute name="name" type="xs:NMTOKEN" use="required" />
    </xs:complexType>
  </xs:element>

  <xs:element name="COLUMNS">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="COLUMN" minOccurs="1" maxOccurs="unbounded" />
      </xs:sequence>
    </xs:complexType>
  </xs:element>

  <xs:element name="FIELD">
    <xs:complexType mixed="true">
      <xs:attribute name="isNull" use="optional" default="false">
        <xs:simpleType>
          <xs:restriction base="xs:NMTOKEN">
            <xs:enumeration value="false" />
            <xs:enumeration value="true" />
          </xs:restriction>
        </xs:simpleType>
      </xs:attribute>
      <xs:attribute name="name" type="xs:NMTOKEN" use="required" />
    </xs:complexType>
  </xs:element>

  <xs:element name="RESULTSET">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="COLUMNS" minOccurs="1" maxOccurs="1" />
        <xs:element ref="ROW" minOccurs="0" maxOccurs="unbounded" />
      </xs:sequence>
      <xs:attribute name="statement" type="xs:string" use="required" />
    </xs:complexType>
  </xs:element>

  <xs:element name="ROW">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="FIELD" minOccurs="1" maxOccurs="unbounded" />
      </xs:sequence>
    </xs:complexType>
  </xs:element>

</xs:schema>


Wed, 5 Mar 2003 10:50:03 -0500
"Andrew Dunstan" <andrew@dunslane.net>
Re: XML ouput for psql
greg@turnstep.com writes:

> I think all psql needs is a simple output, similar to the ones used by
> Oracle, Sybase, and MySQL; the calling application should then process
> it in some way as needed (obviously this is not for interactive use).
> Where can one find a "standard table model?"

I think for processing-oriented output, the system described in the
SQL/XML standard draft is the way to go.  Considering the people who wrote
it, it's probably pulled from, or bound to appear in, a major commercial
database.

I also think that psql is not the place to implement something like this.
It's most likely best put in the backend, as a function like

    xmlfoo('select * from t1;')

Then any interface and application that likes it, not just psql-based
ones, can use it.

-- 
Peter Eisentraut   peter_e@gmx.net


Wed, 5 Mar 2003 23:37:35 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: XML ouput for psql
Andrew Dunstan writes:

> One thing I have noticed about the schemes that are being advanced is that
> they seem to be inherently unspecifiable, formally, because column names are
> being used as tags.

The SQL/XML draft addresses this by specifying that a mapping from SQL
things to XML things spits out both the specification (XML Schema, IIRC)
and the data in one operation.

-- 
Peter Eisentraut   peter_e@gmx.net


Wed, 5 Mar 2003 23:38:27 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: XML ouput for psql
Peter Eisentraut <peter_e@gmx.net> writes:
> I also think that psql is not the place to implement something like this.

Agreed.

> It's most likely best put in the backend, as a function like
>     xmlfoo('select * from t1;')

That seems a little bizarre.  Wouldn't we want to have a switch that
just flips the SELECT output format from one style to the other?

This is also a good time to stop and ask whether the frontend/backend
protocol needs to change to support this.  Not having read the spec,
I have no idea what the low-level transport needs are for XML output,
but I suspect our present protocol is not it ...

			regards, tom lane

Wed, 05 Mar 2003 19:16:54 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: XML ouput for psql 
Tom Lane wrote:
> This is also a good time to stop and ask whether the frontend/backend
> protocol needs to change to support this.  Not having read the spec,
> I have no idea what the low-level transport needs are for XML output,
> but I suspect our present protocol is not it ...

It might be interesting to modify the protocol (and the backend at the 
point of projection to the front end) so that a user defined formating 
function could be applied and either accepted or rejected by the front 
end. Perhaps one flavor of XML output is a start, but I could imagine 
wanting a custom or even different "standard" output format.

Joe


Wed, 05 Mar 2003 17:32:04 -0800
Joe Conway <mail@joeconway.com>
Re: XML ouput for psql
> Peter Eisentraut <peter_e@gmx.net> writes:
> > I also think that psql is not the place to implement something like this.
> 
> Agreed.
> 
> > It's most likely best put in the backend, as a function like
> >     xmlfoo('select * from t1;')

> That seems a little bizarre.  Wouldn't we want to have a switch that
> just flips the SELECT output format from one style to the other?

Ah, but this approach has the merit that it doesn't require pushing out
a completely new set of tools.

> This is also a good time to stop and ask whether the frontend/backend
> protocol needs to change to support this.  Not having read the spec, I
> have no idea what the low-level transport needs are for XML output,
> but I suspect our present protocol is not it ...

That could be; there's enough variation in what one might want to do
with XML that it is not trivial to suggest an 'ideal' answer.

We have already seen the proposal of:
<record a="b" c="d" e="f">
<record a="c" c="e" e="g">
<record a="d" c="f" e="h">
<record a="e" c="g" e="i">

I would rather prefer something like:
<tablea>
 <record>
   <a>b</a> <c>d</c> <e>f</e>
 </record>  
 <record>
   <a>c</a> <c>d</c> <e>f</e>
 </record>  
 <record>
   <a>d</a> <c>d</c> <e>f</e>
 </record>  
<tablea>

(Note that both approaches are quite rational possibilities.)

I'd think that the "protocol" would involve passing back a row-as-string
for each row in the result set.
--
output = ("cbbrowne" "@cbbrowne.com")
http://www.ntlug.org/~cbbrowne/xml.html
"There are two major products that come out of Berkeley: LSD and Unix.
We don't believe this to be a coincidence." - Jeremy S. Anderson

Wed, 05 Mar 2003 21:10:33 -0500
cbbrowne@cbbrowne.com
Re: XML ouput for psql 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> I think for processing-oriented output, the system described in the
> SQL/XML standard draft is the way to go.  Considering the people who wrote
> it, it's probably pulled from, or bound to appear in, a major commercial
> database.

Do you have a link to the exact section? I've found conflicting versions 
of what constitutes the "standard" for xml output of SQL data.

> I also think that psql is not the place to implement something like this.
> It's most likely best put in the backend, as a function like
> 
>     xmlfoo('select * from t1;')
> 
> Then any interface and application that likes it, not just psql-based
> ones, can use it.

I think that is a good long-term solution, but I still think we need 
to address the TODO item in the short run, and allow for a simple 
reformatting of the query results from psql. If not, we should remove 
that TODO item form psql and add a different one to the backend section.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200303061020

-----BEGIN PGP SIGNATURE-----
Comment: http://www.turnstep.com/pgp.html

iD8DBQE+Z2jHvJuQZxSWSsgRAj7IAJ4hLEos9OlE67O02gVrrqxwT9n3AQCeJxto
N2LFyvXPfGY2whPUs5k+PQA=
=PYfs
-----END PGP SIGNATURE-----



Thu,  6 Mar 2003 15:31:41 -0000
greg@turnstep.com
Re: XML ouput for psql
Peter Eisentraut kirjutas N, 06.03.2003 kell 00:37:
> greg@turnstep.com writes:
> 
> > I think all psql needs is a simple output, similar to the ones used by
> > Oracle, Sybase, and MySQL; the calling application should then process
> > it in some way as needed (obviously this is not for interactive use).
> > Where can one find a "standard table model?"
> 
> I think for processing-oriented output, the system described in the
> SQL/XML standard draft is the way to go.  Considering the people who wrote
> it, it's probably pulled from, or bound to appear in, a major commercial
> database.
> 
> I also think that psql is not the place to implement something like this.
> It's most likely best put in the backend, as a function like
> 
>     xmlfoo('select * from t1;')
> 
> Then any interface and application that likes it, not just psql-based
> ones, can use it.

I have written an aggregate function in pl/python for my own needs that
returns underlying query fomatted as XML, but it has some problems:

1) both the row-to-xml-fragment and
collect-the-fragments-to-wellformed-xml-doc have to be defined for each
and every different query (the actual function text is the same).

2) it is unneccesaryly hard to define a function that takes a record as
argument - the record type is lost: for even simple things like this

select * from (select * from mytable) mtab; 

the result of inner query is _not_ of rowtype mytable, i.e.

you can do

select xmlfrag(mytable) from mytable;

but not

select xmlfrag(mytable) from (select * from mytable) mytable;


----------------
Hannu


06 Mar 2003 23:06:36 +0200
Hannu Krosing <hannu@tm.ee>
Re: XML ouput for psql
Tom Lane writes:

> This is also a good time to stop and ask whether the frontend/backend
> protocol needs to change to support this.  Not having read the spec,
> I have no idea what the low-level transport needs are for XML output,
> but I suspect our present protocol is not it ...

The spec defines "mappings" between tables, schemas, and catalogs on the
one side and each time a pair of XML documents on the other side, one of
which is an XML schema document (sort of a document type declaration) and
the other is an XML data document that follows the constraints of the
schema document and contains the actual data.  A table could of course
more or less be interpreted to mean a query result.  That means, this
functionality provides both query result retrieval via XML and a pg_dump
type mechanism with XML output.

So I imagine, if this is done fully with changes in the protocol layer,
then certain commands like "get table schema in XML" would have to exist
in the protocol, which doesn't seem right.  Also, the XML output isn't a
sibling of the current text/binary tuples, since an XML result is always
a whole document, not tuple data.

What we could perhaps consider is a family of functions like I
illustrated, but then provide a fast-path-driven layer on the client side,
like for large objects.  Initially, the development of these mapping
functions could take place totally in user-space.

-- 
Peter Eisentraut   peter_e@gmx.net


Fri, 7 Mar 2003 00:08:17 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: XML ouput for psql 
Peter Eisentraut <peter_e@gmx.net> writes:
> So I imagine, if this is done fully with changes in the protocol layer,
> then certain commands like "get table schema in XML" would have to exist
> in the protocol, which doesn't seem right.  Also, the XML output isn't a
> sibling of the current text/binary tuples, since an XML result is always
> a whole document, not tuple data.

I would envision a distinction comparable to the existing one between T
and D messages (RowDescription and AsciiRow, using the documentation's
names): you send the table schema first, then the data.  Also note that
there is no "command" to get the T message; it comes for free whenever
a SELECT result is sent to the frontend.

> What we could perhaps consider is a family of functions like I
> illustrated, but then provide a fast-path-driven layer on the client side,
> like for large objects.  Initially, the development of these mapping
> functions could take place totally in user-space.

I don't object to that as a quick-and-dirty context for prototyping work,
but I'd sure hate to see it as the production version.  The fastpath
protocol is a mess, and until/unless we get it cleaned up, we ought not
increase dependency on it.

A larger point is that this is still a protocol revision; pretending it
ain't is just willful obscurantism.  You can tell it's a protocol revision
because you will need to rewrite client-side libraries to take advantage
of it.  If we try to look the other way and pretend it isn't one, then
we'll just be incurring pain --- the most obvious pain being that it
will be hard for those client libraries to tell whether the protocol
extension is supported or not.

The way I'd prefer to see this handled is by providing alternatives to
the printtup.c DestReceiver routines.  The backend could be switched to
any desired output representation just by invoking different sets of
receiver routines.  What we seem to need first is a context for doing
that, in particular a way to understand how different output formats can
be fit into the FE/BE protocol.

			regards, tom lane

Thu, 06 Mar 2003 18:33:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: XML ouput for psql 
I tried general, but no response. Anyone here can shed some light on the
issue? Do I need to code merge sort into postgresql?

----- Forwarded message from Taral <taral@taral.net> -----

From: Taral <taral@taral.net>
To: pgsql-general@postgresql.org
Date: Wed, 12 Mar 2003 17:54:35 -0600
Subject: [GENERAL] No merge sort?
Message-ID: <20030312235435.GA3007@taral.net>

I have a table "test" that looks like this:

CREATE TABLE test (
    id BIGINT,
    time INTEGER
);

There is an index:

CREATE INDEX idx ON test(id, time);

The table has been loaded with 2M rows, where time ranges sequentially
from 0 to 1999999 and id is random values from 0 to 49999.

This query:

SELECT * FROM idx WHERE id IN (...) AND time > 198000 AND time < 199800
ORDER BY time DESC LIMIT 20;

has an EXPLAIN ANALYZE of:

Limit  (cost=3635.28..3635.28 rows=20 width=12) (actual time=22.94...22.96 rows=14 loops=1)
  ->  Sort  (cost=3635.28..3635.28 rows=23 width=12) (actual time=22.93..22.93 rows=14 loops=1)
        ->  Index Scan using idx, idx, ..., idx, idx on test  (cost=0.00...3634.77 rows=23 width=12) (actual time=1.01..22.10 rows=14 loops=1)
Total runtime: 29.12 msec

This query:

SELECT * FROM idx WHERE id IN (...) AND time < 199800 ORDER BY time DESC
LIMIT 20;

has an EXPLAIN ANALYZE of:

Limit  (cost=14516.46..14516.46 rows=20 width=12) (actual time=1448..83..1448.86 rows=20 loops=1)
  ->  Sort  (cost=14516.46..14516.46 rows=2527 width=12) (actual time=1448.82..1448.83 rows=21 loops=1)
        ->  Index Scan using idx, idx, ..., idx, idx on test  (cost=0.00...14373.67 rows=2527 width=12) (actual time=0.14..1437.33 rows=2048 loops=1)
Total runtime: 1454.62 msec

Since the index will output 'time' sorted data for each 'id', why isn't
a merge sort being used here? A merge sort would reduce the execution
time back to 30 ms.

-- 
Taral <taral@taral.net>
This message is digitally signed. Please PGP encrypt mail to me.
"Most parents have better things to do with their time than take care of
their children." -- Me
Thu, 13 Mar 2003 15:10:49 -0600
Taral <taral@taral.net>
No merge sort?
Taral <taral@taral.net> writes:
> Do I need to code merge sort into postgresql?

Seems like a waste of effort to me.  I find this example less than
compelling --- the case that could be sped up is quite narrow,
and the potential performance gain not all that large.  (A sort
is a sort however you slice it, with O(N log N) runtime...)

Also, the direction we'd likely be going in in future is to merge
multiple indexscans using bitmap techniques, so that the output
ordering of the scans couldn't be counted on anyway.

			regards, tom lane

Thu, 13 Mar 2003 16:28:34 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: No merge sort? 
On Thu, Mar 13, 2003 at 04:28:34PM -0500, Tom Lane wrote:
> Seems like a waste of effort to me.  I find this example less than
> compelling --- the case that could be sped up is quite narrow,
> and the potential performance gain not all that large.  (A sort
> is a sort however you slice it, with O(N log N) runtime...)

Actually, it's O(N) time. The index can produce "time" sorted data for
each "id" in linear time, and the merge sort can merge them in linear
time. Also, the existing system insists on loading _all_ candidate rows
whereas this method can benefit from the limit.

If you don't want to code it, I will. I need it for the livejournal
mysql->postgresql transition. (No, mysql doesn't do it right either.)
But a few pointers to the right places to look in the code would be
helpful.

> Also, the direction we'd likely be going in in future is to merge
> multiple indexscans using bitmap techniques, so that the output
> ordering of the scans couldn't be counted on anyway.

I don't understand this. What do these bitmap techniques do?

-- 
Taral <taral@taral.net>
This message is digitally signed. Please PGP encrypt mail to me.
"Most parents have better things to do with their time than take care of
their children." -- Me
Thu, 13 Mar 2003 19:58:14 -0600
Taral <taral@taral.net>
Re: No merge sort?
Taral <taral@taral.net> writes:
> On Thu, Mar 13, 2003 at 04:28:34PM -0500, Tom Lane wrote:
>> Seems like a waste of effort to me.  I find this example less than
>> compelling --- the case that could be sped up is quite narrow,
>> and the potential performance gain not all that large.  (A sort
>> is a sort however you slice it, with O(N log N) runtime...)

> Actually, it's O(N) time.

Only if you assume a fixed number of input streams.

>> Also, the direction we'd likely be going in in future is to merge
>> multiple indexscans using bitmap techniques, so that the output
>> ordering of the scans couldn't be counted on anyway.

> I don't understand this. What do these bitmap techniques do?

The idea is you look at the index to make a list of main-table tuple
positions you are interested in, which you represent compactly as a
compressed bitmap.  (There is some finagling needed because PG actually
uses block/line number rather than a pure tuple number to identify
tuples, but you can fake it with a reasonably small amount of overhead.)
Then you can combine multiple index inputs by ANDing or ORing bitmaps
(the OR case applies to your example).  Finally, you traverse the heap,
accessing the desired rows in heap-location order.  This loses in terms
of producing presorted output --- but it often saves enough in I/O costs
to more than justify doing the sort in memory.

			regards, tom lane

Thu, 13 Mar 2003 22:30:27 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: No merge sort? 
On Thu, Mar 13, 2003 at 10:30:27PM -0500, Tom Lane wrote:
> The idea is you look at the index to make a list of main-table tuple
> positions you are interested in, which you represent compactly as a
> compressed bitmap.  (There is some finagling needed because PG actually
> uses block/line number rather than a pure tuple number to identify
> tuples, but you can fake it with a reasonably small amount of overhead.)
> Then you can combine multiple index inputs by ANDing or ORing bitmaps
> (the OR case applies to your example).  Finally, you traverse the heap,
> accessing the desired rows in heap-location order.  This loses in terms
> of producing presorted output --- but it often saves enough in I/O costs
> to more than justify doing the sort in memory.

And it loses bigtime in the case of LIMIT. If the unlimited query
returns 4,000 records and I only want 20, you're retrieving 200x too
much data from disk.

-- 
Taral <taral@taral.net>
This message is digitally signed. Please PGP encrypt mail to me.
"Most parents have better things to do with their time than take care of
their children." -- Me
Thu, 13 Mar 2003 23:04:01 -0600
Taral <taral@taral.net>
Re: No merge sort?
Same setup, different query:

test=> explain select max(time) from test where id = '1';
NOTICE:  QUERY PLAN:

Aggregate  (cost=5084.67..5084.67 rows=1 width=0)
  ->  Index Scan using idx on test  (cost=0.00..5081.33 rows=1333 width=0)

Since the index is (id, time), why isn't the index being used to
retrieve the maximum value?

On Thu, Mar 13, 2003 at 03:10:49PM -0600, Taral wrote:
> I have a table "test" that looks like this:
> 
> CREATE TABLE test (
>     id BIGINT,
>     time INTEGER
> );
> 
> There is an index:
> 
> CREATE INDEX idx ON test(id, time);
> 
> The table has been loaded with 2M rows, where time ranges sequentially
> from 0 to 1999999 and id is random values from 0 to 49999.

-- 
Taral <taral@taral.net>
This message is digitally signed. Please PGP encrypt mail to me.
"Most parents have better things to do with their time than take care of
their children." -- Me
Fri, 14 Mar 2003 14:19:46 -0600
Taral <taral@taral.net>
No index maximum? (was Re: No merge sort?)
Taral <taral@taral.net> writes:
> On Thu, Mar 13, 2003 at 10:30:27PM -0500, Tom Lane wrote:
>> The idea is you look at the index to make a list of main-table tuple
>> positions you are interested in, which you represent compactly as a
>> compressed bitmap.  [snip]

> And it loses bigtime in the case of LIMIT. If the unlimited query
> returns 4,000 records and I only want 20, you're retrieving 200x too
> much data from disk.

Sure.  That's why we have a planner that distinguishes between startup
cost and total cost, and interpolates when a LIMIT is involved.  But
if this mergesort idea only helps for small-limit cases, that's another
restriction on its scope of usefulness...

			regards, tom lane

Fri, 14 Mar 2003 22:43:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: No merge sort? 
On Fri, Mar 14, 2003 at 10:43:30PM -0500, Tom Lane wrote:
> Sure.  That's why we have a planner that distinguishes between startup
> cost and total cost, and interpolates when a LIMIT is involved.  But
> if this mergesort idea only helps for small-limit cases, that's another
> restriction on its scope of usefulness...

I don't think so, since even in the non-limit case it avoids having to
do a full sort if the number of initial streams is finite and small (as
in the case I demonstrated), reducing time complexity to O(N).

-- 
Taral <taral@taral.net>
This message is digitally signed. Please PGP encrypt mail to me.
"Most parents have better things to do with their time than take care of
their children." -- Me
Fri, 14 Mar 2003 22:07:03 -0600
Taral <taral@taral.net>
Re: No merge sort?
On Fri, Mar 14, 2003 at 14:19:46 -0600,
  Taral <taral@taral.net> wrote:
> Same setup, different query:
> 
> test=> explain select max(time) from test where id = '1';
> NOTICE:  QUERY PLAN:
> 
> Aggregate  (cost=5084.67..5084.67 rows=1 width=0)
>   ->  Index Scan using idx on test  (cost=0.00..5081.33 rows=1333 width=0)
> 
> Since the index is (id, time), why isn't the index being used to
> retrieve the maximum value?

It looks like an index scan is being done.

If the index was on (time, id) instead of (id, time), then you could get
a further speed up by rewriting the query as:
select time from test where id = '1' order by time desc limit 1;

Sat, 15 Mar 2003 09:23:28 -0600
Bruno Wolff III <bruno@wolff.to>
Re: No index maximum? (was Re: No merge sort?)
On Sat, Mar 15, 2003 at 09:23:28AM -0600, Bruno Wolff III wrote:
> On Fri, Mar 14, 2003 at 14:19:46 -0600,
>   Taral <taral@taral.net> wrote:
> > Same setup, different query:
> > 
> > test=> explain select max(time) from test where id = '1';
> > NOTICE:  QUERY PLAN:
> > 
> > Aggregate  (cost=5084.67..5084.67 rows=1 width=0)
> >   ->  Index Scan using idx on test  (cost=0.00..5081.33 rows=1333 width=0)
> > 
> > Since the index is (id, time), why isn't the index being used to
> > retrieve the maximum value?
> 
> It looks like an index scan is being done.
> 
> If the index was on (time, id) instead of (id, time), then you could get
> a further speed up by rewriting the query as:
> select time from test where id = '1' order by time desc limit 1;

Yes, that's exactly it. It's an index _scan_. It should simply be able
to read the maximum straight from the btree.

-- 
Taral <taral@taral.net>
This message is digitally signed. Please PGP encrypt mail to me.
"Most parents have better things to do with their time than take care of
their children." -- Me
Mon, 17 Mar 2003 11:23:47 -0600
Taral <taral@taral.net>
Re: No index maximum? (was Re: No merge sort?)
On Mon, Mar 17, 2003 at 11:23:47 -0600,
  Taral <taral@taral.net> wrote:
> On Sat, Mar 15, 2003 at 09:23:28AM -0600, Bruno Wolff III wrote:
> > On Fri, Mar 14, 2003 at 14:19:46 -0600,
> >   Taral <taral@taral.net> wrote:
> > > Same setup, different query:
> > > 
> > > test=> explain select max(time) from test where id = '1';
> > > NOTICE:  QUERY PLAN:
> > > 
> > > Aggregate  (cost=5084.67..5084.67 rows=1 width=0)
> > >   ->  Index Scan using idx on test  (cost=0.00..5081.33 rows=1333 width=0)
> > > 
> > > Since the index is (id, time), why isn't the index being used to
> > > retrieve the maximum value?
> > 
> > It looks like an index scan is being done.
> > 
> > If the index was on (time, id) instead of (id, time), then you could get
> > a further speed up by rewriting the query as:
> > select time from test where id = '1' order by time desc limit 1;
> 
> Yes, that's exactly it. It's an index _scan_. It should simply be able
> to read the maximum straight from the btree.

max and min don't use indexes. They are generic aggregate functions and
postgres doesn't have the special knowledge to know that for those
aggregate functions and index can be used. You can get around this
by rewriting the query as I previously indicated.

For more details on why things are this way, search the archives. This
topic comes up a lot.

I was also mistaken about have to switch the index around for this case.
It should work the way you have it (if you rewrite the query).

Mon, 17 Mar 2003 11:59:23 -0600
Bruno Wolff III <bruno@wolff.to>
Re: No index maximum? (was Re: No merge sort?)

Greg, do you have a newer patch to address the feedback you received, or
is this one good?

---------------------------------------------------------------------------

greg@turnstep.com wrote:
[ There is text before PGP section. ]
> 
[ PGP not available, raw data follows ]
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> Peter Eisentraut wrote:
> > 1. Look into the SQL/XML standard draft (ftp.sqlstandards.org) to find out
> > whether the standard addresses this sort of thing.
> 
> The URL you gave leads to a site curiously content-free and full of dead links. 
> I've looked around a bit, but found nothing definitive. One good resource I 
> did find was this:
> 
> http://www.wiscorp.com/sql/SQLX_Bringing_SQL_and_XML_Together.pdf
> 
> The article mentions a lot of links on the sqlstandards.org and iso.org sites, none 
> of which work or are restricted. If anyone knows of some good links, please 
> let me know. (especially ISO 9075). From what I've read of the SQLX stuff, the 
> format in my patch should be mostly standard:
> 
> <row>
>  <name>Joe Sixpack</name>
>  <age>35</age>
>  <state>Alabama</state>
> </row>
> 
> One problem is that the recommended way to handle non-standard characters 
> (including spaces) is to escape them like this:
> 
> foobar baz => <foobar_x0020_baz>
> 
> This also includes escaping things like "_x*" and "xml*". We don't have 
> anything like that in the code yet (?), but we should probably think about 
> heading that way. I think escaping whitespace in quotes is good enough 
> for now for:
> 
> foobar baz => <"foobar baz">
> 
> The xsd and xsi standards are also interesting, but needlessly complicated 
> for psql output, IMO.
> 
> > Incidentally, the HTML table model is such an established and standardized
> > XML and SGML table model, so the easiest way to get the task "add XML
> > output to psql" done is to update the HTML output to conform to XHTML.
> > That way you get both the strict XML and you can look at the formatted
> > result with any old (er, new) browser.
> 
> I don't agree with this: XML and XHTML are two different things. We could 
> certainly upgrade the HTML portion, but I am pretty sure that the XML 
> standard calls for this format:
> 
> <columnname>data here</columnname>
> 
> ..which is not valid XHTML and won't be viewable by any browser. The other 
> suggested XML formats are even further from XHTML than the above. The HTML 
> format should be "html table/layout" specific and the XML should be 
> "schema/data" specific.
> 
> - --
> Greg Sabino Mullane greg@turnstep.com
> PGP Key: 0x14964AC8 200302280938
> 
> -----BEGIN PGP SIGNATURE-----
> Comment: http://www.turnstep.com/pgp.html
> 
> iD8DBQE+X3k5vJuQZxSWSsgRAuXFAKDGO1IsjB9Lwtkcws1xJy47PibcLQCg3dx5
> fsy27qguZv841lPvCjzdUic=
> =4f9B
> -----END PGP SIGNATURE-----
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 
[ Decrypting message... End of raw data. ]

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 17 Mar 2003 16:36:36 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: XML ouput for psql

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> Greg, do you have a newer patch to address the feedback you received, or
> is this one good?

I have a newer patch, but I am not 100% sure a consensus was reached. I recall 
the thread veering into talk of XML on the backend, but don't recall if anyone 
still had strong objections to a quick psql wrapper. If not, I will clean up 
the existing patch and resubmit tomorrow.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200303171641

-----BEGIN PGP SIGNATURE-----
Comment: http://www.turnstep.com/pgp.html

iD8DBQE+dkE5vJuQZxSWSsgRAkVSAJ9aLoLC23OoNcVEw4hQiaBrPcSqNQCfTxH3
crC4ssFKbBo60gHvJT3WsU0=
=Qsif
-----END PGP SIGNATURE-----



Mon, 17 Mar 2003 21:42:57 -0000
greg@turnstep.com
Re: XML ouput for psql

I like the idea of doing XML in psql --- it seems like a natural place
for it.

---------------------------------------------------------------------------

greg@turnstep.com wrote:
[ There is text before PGP section. ]
> 
[ PGP not available, raw data follows ]
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> > Greg, do you have a newer patch to address the feedback you received, or
> > is this one good?
> 
> I have a newer patch, but I am not 100% sure a consensus was reached. I recall 
> the thread veering into talk of XML on the backend, but don't recall if anyone 
> still had strong objections to a quick psql wrapper. If not, I will clean up 
> the existing patch and resubmit tomorrow.
> 
> - --
> Greg Sabino Mullane greg@turnstep.com
> PGP Key: 0x14964AC8 200303171641
> 
> -----BEGIN PGP SIGNATURE-----
> Comment: http://www.turnstep.com/pgp.html
> 
> iD8DBQE+dkE5vJuQZxSWSsgRAkVSAJ9aLoLC23OoNcVEw4hQiaBrPcSqNQCfTxH3
> crC4ssFKbBo60gHvJT3WsU0=
> =Qsif
> -----END PGP SIGNATURE-----
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
> 
[ Decrypting message... End of raw data. ]

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 17 Mar 2003 17:03:33 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: XML ouput for psql
On Mon, Mar 17, 2003 at 11:23:47AM -0600, Taral wrote:
> Yes, that's exactly it. It's an index _scan_. It should simply be able
> to read the maximum straight from the btree.

Still doesn't work, even with rewritten query. It sort a
Limit(Sort(Index Scan)), with 1333 rows being pulled from the index.

-- 
Taral <taral@taral.net>
This message is digitally signed. Please PGP encrypt mail to me.
"Most parents have better things to do with their time than take care of
their children." -- Me
Mon, 17 Mar 2003 16:09:10 -0600
Taral <taral@taral.net>
Re: No index maximum? (was Re: No merge sort?)
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I like the idea of doing XML in psql --- it seems like a natural place
> for it.

Not really; what of applications other than shell scripts that would
like to get XML-formatted output?

There was some talk in the FE/BE protocol thread of adding hooks to
support more than one output format from the backend.  Much of the
infrastructure already exists (see DestReceiver in the backend); we
just need an agreement on the protocol.  On the whole I'd rather see
it done that way than burying the logic in psql.

			regards, tom lane

Mon, 17 Mar 2003 17:59:15 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: XML ouput for psql 
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I like the idea of doing XML in psql --- it seems like a natural place
> > for it.
> 
> Not really; what of applications other than shell scripts that would
> like to get XML-formatted output?
> 
> There was some talk in the FE/BE protocol thread of adding hooks to
> support more than one output format from the backend.  Much of the
> infrastructure already exists (see DestReceiver in the backend); we
> just need an agreement on the protocol.  On the whole I'd rather see
> it done that way than burying the logic in psql.

Well, programs can run psql using popen.  It seems overkill to get the
protocol involved, specially since it is output-only.  I can't imagine
who would bother with the wire protocol messiness just to get xml.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 17 Mar 2003 18:08:12 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: XML ouput for psql
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Tom Lane wrote:
>> Not really; what of applications other than shell scripts that would
>> like to get XML-formatted output?

> Well, programs can run psql using popen.  It seems overkill to get the
> protocol involved, specially since it is output-only.  I can't imagine
> who would bother with the wire protocol messiness just to get xml.

Having to popen a psql isn't overkill?  This seems like a far messier
solution than the other.  Furthermore, it's just plain not an available
solution in many scenarios (think of a Java program running JDBC; it may
not have privileges to do popen, and may not have access to a copy of
psql anyway).

If we were not already opening up the protocol for changes, I'd be
resistant to the idea too.  But since we are, I think it should be fixed
where it's cleanest to fix it.

			regards, tom lane

Mon, 17 Mar 2003 18:18:49 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: XML ouput for psql 
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Tom Lane wrote:
> >> Not really; what of applications other than shell scripts that would
> >> like to get XML-formatted output?
> 
> > Well, programs can run psql using popen.  It seems overkill to get the
> > protocol involved, specially since it is output-only.  I can't imagine
> > who would bother with the wire protocol messiness just to get xml.
> 
> Having to popen a psql isn't overkill?  This seems like a far messier
> solution than the other.  Furthermore, it's just plain not an available
> solution in many scenarios (think of a Java program running JDBC; it may
> not have privileges to do popen, and may not have access to a copy of
> psql anyway).
> 
> If we were not already opening up the protocol for changes, I'd be
> resistant to the idea too.  But since we are, I think it should be fixed
> where it's cleanest to fix it.

What would be interesting would be to enable libpq to dump XML, and have
psql use that.  Why put XML capability in the backend?  Of course, that
doesn't help jdbc.  How do you propose the backend would do XML?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 17 Mar 2003 18:47:28 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: XML ouput for psql
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> What would be interesting would be to enable libpq to dump XML, and have
> psql use that.

... or in the backend so libpq could use it, and thence psql.

>  Why put XML capability in the backend?

So that non-libpq-based clients could use it.

			regards, tom lane

Tue, 18 Mar 2003 01:22:34 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: XML ouput for psql 
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > What would be interesting would be to enable libpq to dump XML, and have
> > psql use that.
> 
> ... or in the backend so libpq could use it, and thence psql.
> 
> >  Why put XML capability in the backend?
> 
> So that non-libpq-based clients could use it.

OK, I have two ideas here.  First, can we create a function that takes a
query result and returns one big XML string.  I am not sure how to pump
a result into a function.  The other downside is that we would have to
construct the entire result string in memory.

The other idea I had was a GUC variable that returned all query results
as one big XML string.  That would prevent creating the entire string in
backend memory, and might enable cursor fetches through the XML string.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 18 Mar 2003 09:34:53 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: XML ouput for psql
Since now is the time for contrib/ flamewars, this seemed a good time
to suggest this.

My colleague, Sorin Iszlai, wrote us a little program for rotating
our Postgres logs.  It reads stdout and stderr, and sends them to
different files (and rotates them as necessary).  It is currently
hand-configureable (i.e. by altering some variables at the top of the
script), and is more or less designed for use in our own environment.

Tom Lane recently mentioned to me that a common complaint is that
postgres doesn't have its own log rotator.  There are, of course,
plenty of good ones, and syslog itself works pretty well for most
people.  But there are still complaints from time to time about the
lack of a "built in" log rotator.

We'd be happy to release our rotator under the PostgreSQL BSD
license, if it would be of use to people.  I was thinking that
perhaps contrib/ would be a good place for it, since the idea is to
reduce complaints that there's no log rotator "included".  

Is anyone interested in having pglog-rotator?

A
-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Thu, 3 Apr 2003 10:21:52 -0500
Andrew Sullivan <andrew@libertyrms.info>
more contrib: log rotator
Andrew Sullivan <andrew@libertyrms.info> writes:
> Is anyone interested in having pglog-rotator?

FWIW, I saw an early version of pglog-rotator about a year and a half
ago (while consulting for LibertyRMS), and thought at the time that
it was pretty cool.  So I'm for including it ... maybe even as
mainstream instead of contrib.

			regards, tom lane


Thu, 03 Apr 2003 11:07:44 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: more contrib: log rotator 
Would the plan be to add it to pg_ctl?


> Andrew Sullivan <andrew@libertyrms.info> writes:
> > Is anyone interested in having pglog-rotator?
> 
> FWIW, I saw an early version of pglog-rotator about a year and a half
> ago (while consulting for LibertyRMS), and thought at the time that
> it was pretty cool.  So I'm for including it ... maybe even as
> mainstream instead of contrib.
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly


Thu, 3 Apr 2003 13:05:51 -0500
"Jim Buttafuoco" <jim@contactbda.com>
Re: more contrib: log rotator 
"Jim Buttafuoco" <jim@contactbda.com> writes:
> Would the plan be to add it to pg_ctl?

You would not actually have to: you could just pipe pg_ctl's output to
pglog-rotator.  But I think it'd be cool if pg_ctl had an option to use
pglog-rotator, or maybe even adopt it as standard behavior.

I think we would have to make the rotator script be mainstream rather
than contrib if we wanted pg_ctl to use it directly.  That was why I was
thinking maybe mainstream ...

Andrew, could you toss up the script on pgsql-patches just so people can
take a look?  Then we could think more about where to go with it.

			regards, tom lane


Thu, 03 Apr 2003 13:41:08 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: more contrib: log rotator 
Does this log rotator do something that apache's doesn't?

Dave
On Thu, 2003-04-03 at 13:41, Tom Lane wrote:
> "Jim Buttafuoco" <jim@contactbda.com> writes:
> > Would the plan be to add it to pg_ctl?
> 
> You would not actually have to: you could just pipe pg_ctl's output to
> pglog-rotator.  But I think it'd be cool if pg_ctl had an option to use
> pglog-rotator, or maybe even adopt it as standard behavior.
> 
> I think we would have to make the rotator script be mainstream rather
> than contrib if we wanted pg_ctl to use it directly.  That was why I was
> thinking maybe mainstream ...
> 
> Andrew, could you toss up the script on pgsql-patches just so people can
> take a look?  Then we could think more about where to go with it.
> 
> 			regards, tom lane
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
-- 
Dave Cramer <dave@fastcrypt.com>
Cramer Consulting


03 Apr 2003 14:12:03 -0500
Dave Cramer <dave@fastcrypt.com>
Re: more contrib: log rotator
On Thu, Apr 03, 2003 at 01:41:08PM -0500, Tom Lane wrote:
> You would not actually have to: you could just pipe pg_ctl's output to
> pglog-rotator.  But I think it'd be cool if pg_ctl had an option to use
> pglog-rotator, or maybe even adopt it as standard behavior.

It's currently built to call a program, and read its stdout and
stderr, rather than acting as a pipe.  I guess it shouldn't be too
hard to modify, though.  We actually call the postmaster directly
with it, so we use it as a replacement for pg_ctl at startup.

> Andrew, could you toss up the script on pgsql-patches just so people can
> take a look?  Then we could think more about where to go with it.

Ok, I sent it.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Thu, 3 Apr 2003 16:12:14 -0500
Andrew Sullivan <andrew@libertyrms.info>
Re: more contrib: log rotator
On Thu, Apr 03, 2003 at 02:12:03PM -0500, Dave Cramer wrote:
> Does this log rotator do something that apache's doesn't?

Probably not.  This was just easier for us.

A little information might be handy here: we run postgres nder a
hosted environment, and we do not have root on the relevant boxes. 
So installing anything even a little complicated means building
everything ourselves.  As a result, we end up re-creating plenty of
functionality just to make it easy to install.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Thu, 3 Apr 2003 16:14:19 -0500
Andrew Sullivan <andrew@libertyrms.info>
Re: more contrib: log rotator
On Thu, Apr 03, 2003 at 01:41:08PM -0500, Tom Lane wrote:
> Andrew, could you toss up the script on pgsql-patches just so people can
> take a look?  Then we could think more about where to go with it.

Ok, the first try failed (of course) because I wasn't subscribed. 
Should be there now, though.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Thu, 3 Apr 2003 22:00:23 -0500
Andrew Sullivan <andrew@libertyrms.info>
Re: more contrib: log rotator
Andrew Sullivan writes:

> Is anyone interested in having pglog-rotator?

What would get me a whole lot more excited is if the server could write
directly to a file and do its own rotating (or at least reopening of
files).

Considering that your rotator is tailored to a rather specific setup, it
doesn't do anything better compared to established ones, it prevents the
use of pg_ctl, it's written in Perl, and it doesn't do anything for
Windows users, I think it's not suitable for a general audience.

-- 
Peter Eisentraut   peter_e@gmx.net


Fri, 4 Apr 2003 17:13:13 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: more contrib: log rotator
On Fri, 4 Apr 2003, Peter Eisentraut wrote:

> Andrew Sullivan writes:
> 
> > Is anyone interested in having pglog-rotator?
> 
> What would get me a whole lot more excited is if the server could write
> directly to a file and do its own rotating (or at least reopening of
> files).
> 
> Considering that your rotator is tailored to a rather specific setup, it
> doesn't do anything better compared to established ones, it prevents the
> use of pg_ctl, it's written in Perl, and it doesn't do anything for
> Windows users, I think it's not suitable for a general audience.

That said, a log rotation capability built right into pg_ctl or 
thereabouts would be a very nice feature.  I.e. 'pg_ctl -r 86400 -l 
$PGDATA/logs/pgsql start'

where -r is the rotation period in seconds.  If it's an external program 
that pg_ctl calls that's fine, and it could even just be a carbon copy of 
apache's log rotater if their license is compatible (isn't it?)


Fri, 4 Apr 2003 09:16:39 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: more contrib: log rotator
On Fri, Apr 04, 2003 at 09:16:39AM -0700, scott.marlowe wrote:
> where -r is the rotation period in seconds.  If it's an external program 

Ours rotates based on size rather than time.  I can see some
advantages to the time-based approach, but if you have wide
variations in traffic, you run the risk of rotating over useful files
with more or less empty ones if you use it.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Fri, 4 Apr 2003 11:35:09 -0500
Andrew Sullivan <andrew@libertyrms.info>
Re: more contrib: log rotator
On Fri, Apr 04, 2003 at 05:13:13PM +0200, Peter Eisentraut wrote:
> use of pg_ctl, it's written in Perl, and it doesn't do anything for
> Windows users, I think it's not suitable for a general audience.

It doesn't prevent the use of pg_ctl, although it does indeed prevent
the use of pg_ctl for startup.  

I'm not sufficiently familiar with Windows to know how this does or
does not help them.  Could you elaborate?  And what's wrong with
Perl?

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Fri, 4 Apr 2003 11:37:55 -0500
Andrew Sullivan <andrew@libertyrms.info>
Re: more contrib: log rotator
On Friday April 4 2003 9:16, scott.marlowe wrote:
>
> That said, a log rotation capability built right into pg_ctl or
> thereabouts would be a very nice feature.  I.e. 'pg_ctl -r 86400 -l
> $PGDATA/logs/pgsql start'
>
> where -r is the rotation period in seconds.  If it's an external program
> that pg_ctl calls that's fine, and it could even just be a carbon copy of
> apache's log rotater if their license is compatible (isn't it?)

By way of feature ideas, one very convenient but not widely used feature of 
Apache's log rotator is the ability to specify a strftime() format string 
for the file extension.  For example, if I want to have my logs rollover 
every 24 hours and be named log.Mon, log.Tue, log.Wed, I say something like

	pg_ctl start | rotatelogs 86400 "%a"

This causes the logs to overwrite themselves every seven days, taking log 
maintenance time to very near zero.  We also customized our use of it to 
allow us to automatically move existing logs out of the way to "log.1", 
"log.2", or to simply overwrite existing logs.

Ed


Fri, 4 Apr 2003 10:04:25 -0700
"Ed L." <pgsql@bluepolka.net>
Re: more contrib: log rotator
On Friday April 4 2003 10:04, Ed L. wrote:
> By way of feature ideas, one very convenient but not widely used feature
> of Apache's log rotator is the ability to specify a strftime() format
> string for the file extension.  For example, if I want to have my logs
> rollover every 24 hours and be named log.Mon, log.Tue, log.Wed, I say
> something like
>
> 	pg_ctl start | rotatelogs 86400 "%a"

More accurately, something like this:

	pg_ctl start | rotatelogs 86400 "log.%a"

Ed


Fri, 4 Apr 2003 10:06:42 -0700
"Ed L." <pgsql@bluepolka.net>
Re: more contrib: log rotator
On Fri, 4 Apr 2003, Andrew Sullivan wrote:

> On Fri, Apr 04, 2003 at 09:16:39AM -0700, scott.marlowe wrote:
> > where -r is the rotation period in seconds.  If it's an external program 
> 
> Ours rotates based on size rather than time.  I can see some
> advantages to the time-based approach, but if you have wide
> variations in traffic, you run the risk of rotating over useful files
> with more or less empty ones if you use it.

I would want time based for sure, and I can see the use for size based 
splitting as well.  I wouldn't be hard to have it do both would it?

I just like the idea of it being one of the dozens or so options for 
pg_ctl so it's painless to use for joe six pack.

pg_ctl -r 86400 -l $PGDATA/logs/pgsql

where -r is the rotation period

OR

pg_ctl -f 10M -l $PGDATA/logs/pgsql

where -f is the max file size of a log

I'd recommend that the nameing convnention should probably 
be:

filenamespec.timestamp, like:  $PGDATA/logs/pgsql.1049414400

for time rotated logs, and 

filename.incnumber like:  $PGDATa/logs/pgsql.0000000001


Fri, 4 Apr 2003 10:10:22 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: more contrib: log rotator
Peter Eisentraut wrote:
> 
> Andrew Sullivan writes:
> 
> > Is anyone interested in having pglog-rotator?
> 
> What would get me a whole lot more excited is if the server could write
> directly to a file and do its own rotating (or at least reopening of
> files).

 From a technical point of view I don't think that is desirable. The
entire log traffic would have to be routed through the postmaster, as it
is in LibertyRMS's log rotator now through the perl script. And we
really try to keep everything outside the postmaster that does not
absolutely have to be in there for stability reasons.

We can discuss if the log rotator should be a child process of the
postmaster or the other way round, but that will not change the flow of
bytes between the processes in any way.

I would say it's better the way it is, because it does not pollute the
postmasters wait logic with another exception.

My ideal solution would be to integrate the log rotators functionality
into a C version of pg_ctl that forks and detaches from the control
terminal in the way, daemons should.


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Fri, 04 Apr 2003 12:13:37 -0500
Jan Wieck <JanWieck@Yahoo.com>
Re: more contrib: log rotator
Peter Eisentraut <peter_e@gmx.net> writes:
> What would get me a whole lot more excited is if the server could write
> directly to a file and do its own rotating (or at least reopening of
> files).

AFAICS, the only practical way to do this is to have a single process
collecting the stdout/stderr from the postmaster and all its children.
pglog-rotator is one implementation of that approach.

I too would rather this functionality were integrated into the server,
but I haven't noticed anyone stepping up to the plate to do it.

> Considering that your rotator is tailored to a rather specific setup, it
> doesn't do anything better compared to established ones, it prevents the
> use of pg_ctl, it's written in Perl, and it doesn't do anything for
> Windows users, I think it's not suitable for a general audience.

These might be good arguments for not putting it into the mainstream,
but I don't think they have any force if we consider it for contrib.

I feel we really ought to have *some* rotator included in the standard
distro, just so that the Admin Guide can point to a concrete solution
instead of having to arm-wave about what you can get off the net.
If someone can offer a better alternative than Andrew's, great, let's
see it.

			regards, tom lane


Fri, 04 Apr 2003 12:19:49 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: more contrib: log rotator 
On Friday April 4 2003 10:19, Tom Lane wrote:
>
> I feel we really ought to have *some* rotator included in the standard
> distro, just so that the Admin Guide can point to a concrete solution
> instead of having to arm-wave about what you can get off the net.
> If someone can offer a better alternative than Andrew's, great, let's
> see it.

Out of curiosity, are there issues preventing inclusion of Apache's log 
rotation code?  It seems you'd be hard-pressed to find a more 
battle-hardened log rotator.

Obviously some people also wish to rotate based on log file size, so adding 
both to contrib at least seems sensible.

Ed


Fri, 4 Apr 2003 11:10:16 -0700
"Ed L." <pgsql@bluepolka.net>
Re: more contrib: log rotator
On Fri, 4 Apr 2003, Ed L. wrote:

> On Friday April 4 2003 10:19, Tom Lane wrote:
> >
> > I feel we really ought to have *some* rotator included in the standard
> > distro, just so that the Admin Guide can point to a concrete solution
> > instead of having to arm-wave about what you can get off the net.
> > If someone can offer a better alternative than Andrew's, great, let's
> > see it.
> 
> Out of curiosity, are there issues preventing inclusion of Apache's log 
> rotation code?  It seems you'd be hard-pressed to find a more 
> battle-hardened log rotator.
> 
> Obviously some people also wish to rotate based on log file size, so adding 
> both to contrib at least seems sensible.

OK, I'm playing with the pg_ctl script that comes with 7.3, and trying to 
make it startup with apaches rotatelog script, but this line won't pipe 
output.  I'm a total noob at bash shell scripting, so please feel free to 
snicker when you answer.

rotatelogs is in my path and all, it just never sees it.

"$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION 2>&1 &


Fri, 4 Apr 2003 11:41:04 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: more contrib: log rotator
"scott.marlowe" <scott.marlowe@ihs.com> writes:
> rotatelogs is in my path and all, it just never sees it.

You mean the command fails?  Or just that it doesn't capture output?

> "$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION 2>&1 &

Most if not all of the postmaster's log output goes to stderr, so you'd need

"$po_path" ${1+"$@"} </dev/null 2>&1 | $PGPATH/rotatelogs ...

to have any hope of useful results.

			regards, tom lane


Fri, 04 Apr 2003 13:58:46 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: more contrib: log rotator 
On Friday April 4 2003 11:58, Tom Lane wrote:
> "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > rotatelogs is in my path and all, it just never sees it.
>
> You mean the command fails?  Or just that it doesn't capture output?
>
> > "$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION
> > 2>&1 &
>
> Most if not all of the postmaster's log output goes to stderr, so you'd
> need
>
> "$po_path" ${1+"$@"} </dev/null 2>&1 | $PGPATH/rotatelogs ...
>
> to have any hope of useful results.

Hmmm.  I would have agreed 2>&1 was needed, too, but this command seems to 
routinely capture all output, including ERRORs:

	nohup pg_ctl start | nohup rotatelogs server_log.%a 86400

Ed


Fri, 4 Apr 2003 12:10:13 -0700
"Ed L." <pgsql@bluepolka.net>
Re: more contrib: log rotator
On Fri, 4 Apr 2003, Tom Lane wrote:

> "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > rotatelogs is in my path and all, it just never sees it.
> 
> You mean the command fails?  Or just that it doesn't capture output?

The database starts, but rotatelogs doesn't get run.  I.e. it's just like 
everything after the | symbol isn't there.


Fri, 4 Apr 2003 12:49:37 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: more contrib: log rotator 
"Ed L." <pgsql@bluepolka.net> writes:
> Hmmm.  I would have agreed 2>&1 was needed, too, but this command seems to 
> routinely capture all output, including ERRORs:
> 	nohup pg_ctl start | nohup rotatelogs server_log.%a 86400

That's 'cause pg_ctl internally redirects the postmaster's stderr.

			regards, tom lane


Fri, 04 Apr 2003 15:13:44 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: more contrib: log rotator 
On Fri, 4 Apr 2003, Ed L. wrote:

> On Friday April 4 2003 11:58, Tom Lane wrote:
> > "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > > rotatelogs is in my path and all, it just never sees it.
> >
> > You mean the command fails?  Or just that it doesn't capture output?
> >
> > > "$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION
> > > 2>&1 &
> >
> > Most if not all of the postmaster's log output goes to stderr, so you'd
> > need
> >
> > "$po_path" ${1+"$@"} </dev/null 2>&1 | $PGPATH/rotatelogs ...
> >
> > to have any hope of useful results.
> 
> Hmmm.  I would have agreed 2>&1 was needed, too, but this command seems to 
> routinely capture all output, including ERRORs:
> 
> 	nohup pg_ctl start | nohup rotatelogs server_log.%a 86400

OK, So I tried putting the 2>&1 before the | and all.  No matter what I 
try, every from the | on is ignored.  ps doesn't show it, and neither does 
pg_ctl status.  Both show a command line of 
/usr/local/pgsql/bin/postmaster as the only input to start the server.

Now, the thing is, I've tried this with hardcoded values, like:

"$po_path" ${1+"$@"} </dev/null 2>&1 /usr/local/pgsql/bin/rotatelogs 
/mnt/d1/data/logs/pglog 86400

where I know the logs directory exists.  It works if I do:

pg_ctl start | rotatelogs $PGDATA/pglog 86400 2>1&

and puts the log files there.

I've copied rotatelogs into the /usr/local/pgsql/bin directory as well.

So, I'm thinking this is my weakness in shell scripting that's getting me 
here, and that the shell is eating the |, not passing it out with the 
postmaster to be used when it starts.


Fri, 4 Apr 2003 14:17:41 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: more contrib: log rotator
On Friday April 4 2003 2:17, scott.marlowe wrote:
>
> OK, So I tried putting the 2>&1 before the | and all.  No matter what I
> try, every from the | on is ignored.  ps doesn't show it, and neither
> does pg_ctl status.  Both show a command line of
> /usr/local/pgsql/bin/postmaster as the only input to start the server.

Not clear if you're looking at it this way or if this is your problem, but 
you can't really tell there is log rotation going on just by grepping ps 
for postmaster because ps does not typically show the postmaster and the 
rotatelogs together on the same line.  I wouldn't expect pg_ctl status to 
know anything at all about rotatelogs when you pipe it like this.

Ed


Fri, 4 Apr 2003 14:45:07 -0700
"Ed L." <pgsql@bluepolka.net>
Re: more contrib: log rotator
On Fri, 4 Apr 2003, Ed L. wrote:

> On Friday April 4 2003 2:17, scott.marlowe wrote:
> >
> > OK, So I tried putting the 2>&1 before the | and all.  No matter what I
> > try, every from the | on is ignored.  ps doesn't show it, and neither
> > does pg_ctl status.  Both show a command line of
> > /usr/local/pgsql/bin/postmaster as the only input to start the server.
> 
> Not clear if you're looking at it this way or if this is your problem, but 
> you can't really tell there is log rotation going on just by grepping ps 
> for postmaster because ps does not typically show the postmaster and the 
> rotatelogs together on the same line.  I wouldn't expect pg_ctl status to 
> know anything at all about rotatelogs when you pipe it like this.

Hey, do you guys think that a setting of silent_mode = false might affect 
no log files getting created?

I had it right as soon as I added Tom's recommended 2>&1 but spent another 
30 minutes figuring out why my log file wasn't getting created / filled.   

Thanks for the help.


Fri, 4 Apr 2003 15:34:22 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: more contrib: log rotator
"scott.marlowe" <scott.marlowe@ihs.com> writes:
> Hey, do you guys think that a setting of silent_mode = false might affect 
> no log files getting created?

No, but setting it to true would be bad news.

			regards, tom lane


Fri, 04 Apr 2003 17:42:16 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: more contrib: log rotator 
On Fri, 4 Apr 2003, Tom Lane wrote:

> "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > Hey, do you guys think that a setting of silent_mode = false might affect 
> > no log files getting created?
> 
> No, but setting it to true would be bad news.

That's what I'd meant actually.  I had to turn of silent mode...  You know 
you're having a bad day when your email explaining how stupid you are is 
factually incorrect. :-)

 If anyone wants the diff, here it is:

22c22
<   $CMDNAME start   [-w] [-D DATADIR] [-s] [-l FILENAME] [-o \"OPTIONS\"]
---
>   $CMDNAME start   [-w] [-D DATADIR] [-s] [-r DURATION] [-l FILENAME] 
[-o \"OPTIONS\"]
39a40,41
>   -r DURATION             invoke log rotation with DURATION seconds
>                           between rotation of files.
155a158,161
>       -r)
>           DURATION="$2"
>           shift
>           ;;
336c342,346
<         "$po_path" ${1+"$@"} </dev/null >>$logfile 2>&1 &
---
>         if [ -n "$DURATION" ]; then
>             "$po_path" ${1+"$@"} </dev/null 2>&1| $PGPATH/rotatelogs 
$logfile $DURATION 2>&1 &
>         else
>             "$po_path" ${1+"$@"} </dev/null >>$logfile 2>&1 &
>         fi


Fri, 4 Apr 2003 15:50:57 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: more contrib: log rotator 
Tom Lane writes:

> AFAICS, the only practical way to do this is to have a single process
> collecting the stdout/stderr from the postmaster and all its children.

I think not.  It's a little tricky handling it directly in the child
processes, but it's been done before.

> If someone can offer a better alternative than Andrew's, great, let's
> see it.

How about the attached one, which I floated a while ago but which didn't
generate much interest.

-- 
Peter Eisentraut   peter_e@gmx.net
Sat, 5 Apr 2003 02:03:03 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: more contrib: log rotator 
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> AFAICS, the only practical way to do this is to have a single process
>> collecting the stdout/stderr from the postmaster and all its children.

> I think not.  It's a little tricky handling it directly in the child
> processes, but it's been done before.

A "little" tricky?  Thanks, but no thanks ... for one thing, there'd be
no easy way to know when all the children had switched over to writing
the new file.  Also, at least for not-too-long messages, writing on a
single pipe gives atomicity guarantees that AFAIK do not exist when
writing a file through multiple independently opened descriptors.  In
the latter case I think we'd have lots of trouble with interleaving of
messages from different backends.

>> If someone can offer a better alternative than Andrew's, great, let's
>> see it.

> How about the attached one, which I floated a while ago but which didn't
> generate much interest.

Seems like a good bare-bones file writer; but how about all those
frammishes that people ask for like generating date-based filenames,
switching every so many bytes, etc?  Also, it'd be nice not to be
dependent on a cron job to tickle the switchover.

I do think there's an efficiency argument for having the log writer
coded in C, so starting with what you have here and building up might
be a better idea than starting with Andrew's perl script.  But the
important thing in my mind is to get something in there.

We should also take a look at Apache's rotator to see if there's any need
to reinvent the wheel at all.  I have not seen it, am not even sure what
it's written in...

			regards, tom lane


Fri, 04 Apr 2003 19:22:58 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: more contrib: log rotator 
On Fri, 4 Apr 2003, Tom Lane wrote:

> We should also take a look at Apache's rotator to see if there's any need
> to reinvent the wheel at all.  I have not seen it, am not even sure what
> it's written in...

It's written in 140 lines of C (blank lines and all), and has been very
solid in my experience. I don't know of any deficiencies that would
warrant rewriting it.

Jon


Sat, 5 Apr 2003 14:23:25 +0000 (UTC)
Jon Jensen <jon@endpoint.com>
Re: more contrib: log rotator 
...and what kind of performance hit we take (and under what
circumstances) for not having it?


Sat, 05 Apr 2003 16:35:30 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Anyone know why PostgreSQL doesn't support 2 phase execution?
...and if so, what are the current efforts focusing on?


Sat, 05 Apr 2003 16:35:35 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Anyone working on better transaction locking?
AFAIK, there are only 3 general purpose internal sorting techniques
that have O(n) behavior:
1= Insertion Sort for "almost sorted" lists.  Since "almost sorted" is
a fuzzy concept, let's make the first approximation definition that no
more than n^(1/2) of the elements can be disordered.  There are better
definitions in the literature, but I don't remember them off the top
of my head.

2= Sorts from the class of Address Calculation, Counting, or
Distribution Sort.  These need to be able to carry out something more
complex than simply a comparison in order to achieve O(n), and
therefore have high constants in their execution.  For large enough n
though, these are the performance kings.

3= Straight Radix Sort where you minimize the number of passes by
using a base much greater than two for the the radix.  Usually octal
or hexidecimal.  On a 32b or 64b system, this approach will let you
sort in 2 passes.

All of the above have potentially nasty trade-offs in comparision to
the standard heavily tuned median-of-three quicksort used by the sort
unix command.  Nonetheless, I could see some value in providing all of
these with PostgeSQL (including a decent port of the unix sort routine
for the Win folks).  I'll note in passing that quicksort's Achille's
heel is that it's unstable (all of the rest are stable), which can be
a problem in a DB.

In the general case there's a few papers out there that state you can
sort in O(n) if you can throw O(n^2) space at the problem.  That
implies to me that for DB's, we are not going to be able to use O(n)
algorithms for most of our needs.


As for external sorts, everything I've ever read says that some sort
of merge technique is used:  balanced, multiway, or polyphase.  In all
cases, I've seen comments to the effect that reading some part of the
data into internal buffers, sorting them, and then merging them with
already sorted data is the best practice.


All of this seems to imply that instead of mergesort (which is
stable), PostgreSQL might be better off with the 4 sorts I've listed
plus a viciously efficient merge utility for combining partial results
that do fit into memory into result files on disk that don't.


Or am I crazy?

Ron Peacetree


Sun, 06 Apr 2003 16:29:11 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: No merge sort?
Tom Lane writes:

> Seems like a good bare-bones file writer; but how about all those
> frammishes that people ask for like generating date-based filenames,
> switching every so many bytes, etc?  Also, it'd be nice not to be
> dependent on a cron job to tickle the switchover.

Linux systems have a standard system log rotation mechanism (see
logrotate(8)), which can rotate logs by size and time and has a number of
other features.  I would rather depend on that kind of preferred system
mechanism than rolling out our own.  And we already depend on cron for
vacuuming anyway.

-- 
Peter Eisentraut   peter_e@gmx.net


Sun, 6 Apr 2003 23:03:27 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: more contrib: log rotator 

Tom Lane wrote:

> Seems like a good bare-bones file writer; but how about all those
> frammishes that people ask for like generating date-based filenames,
> switching every so many bytes, etc?  Also, it'd be nice not to be
> dependent on a cron job to tickle the switchover.

One of the earlier times this discussion came up I wrote a log
rotation program too.  It will rotate based on time, file size,
and/or SIGHUP:

    ftp://ftp.nemeton.com.au/pub/src/logwrite-1.0alpha.tar.gz

Written in C, BSD license, used on my production systems.  Needs to be
updated to not have 'alpha' in the URL ... no bug reports in 18 months
means that it could at least be beta. ;-) It should be portable: I've
built it on *BSD, HP-UX, and Linux at least.  I'd need help for
Windows.

I took pains to deal with some of the concerns Tom raised last time
about not giving up and exiting when a filesystem fills up.  This is
something that the Apache rotatelogs program didn't do at the time I
looked at it, else I'd have not written this at all.

> I too would rather this functionality were integrated into the server,
> but I haven't noticed anyone stepping up to the plate to do it.

Shouldn't be hard: the server only has to create a pipe, and run the
log rotation program.  For extra robustness re-starting the log
rotation program if it exits ("don't kill -9 the log rotator") is a
good idea.

I'll look at doing this, unless the discussion heads somewhere else
e.g., if Peter decides to integrate his code.  The server changes are
probably independent of whichever log rotation program is preferred
anyway.

Regards,

Giles


Mon, 07 Apr 2003 07:48:24 +1000
Giles Lean <giles@nemeton.com.au>
Re: more contrib: log rotator 
On Sun, Apr 06, 2003 at 11:03:27PM +0200, Peter Eisentraut wrote:
> Linux systems have a standard system log rotation mechanism (see
> logrotate(8)), which can rotate logs by size and time and has a number of
> other features.  I would rather depend on that kind of preferred system

But it's not available on every platform.  And according to my
/usr/share/doc/logrotate/copyright, it's GPL, so you can't
redistribute it with PostgreSQL.  Building it in was one of the
criteria, I thought

> And we already depend on cron for vacuuming anyway.

But that dependency is actually a liability, because there's no way
to say, "Hey, I really need to be vacuumed now." We (at Liberty) are
currently going through a remarkable number of hoops trying to get
around that very limitation

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Sun, 6 Apr 2003 18:17:15 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: more contrib: log rotator
Andrew Sullivan writes:

> But it's not available on every platform.  And according to my
> /usr/share/doc/logrotate/copyright, it's GPL, so you can't
> redistribute it with PostgreSQL.  Building it in was one of the
> criteria, I thought

My point was that log file rotation should be left up to the system
administrator.  Look at other servers on your system (SMTP, DNS,
whatever).  How do they handle it?

> > And we already depend on cron for vacuuming anyway.
>
> But that dependency is actually a liability, because there's no way
> to say, "Hey, I really need to be vacuumed now."

psql -c 'VACUUM' ?

-- 
Peter Eisentraut   peter_e@gmx.net


Mon, 7 Apr 2003 00:42:34 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: more contrib: log rotator
On Mon, Apr 07, 2003 at 12:42:34AM +0200, Peter Eisentraut wrote:
> 
> My point was that log file rotation should be left up to the system
> administrator.  Look at other servers on your system (SMTP, DNS,
> whatever).  How do they handle it?

PostgreSQL is not a system process, and I think it's a mistake to
assume that it is.  We, for instance, do not have root on the
machines we use.  It's important to assume that users needn't be
system administrators to use the system.

I suppose, however, you could make the argument that log rotation
should be the responisibility of the adminisistrator of the
PostgreSQL server.  But that just amounts to an argument that nothing
needs to be done: as we see, there are lots of log management
facilities on offer, and none of them are included with PostgreSQL. 
I don't feel strongly about which one to use.  But since people
frequently complain that the feature is not available in PostgreSQL,
it does seem that, if it's not too much trouble, adding the feature
is worth it.

> > But that dependency is actually a liability, because there's no way
> > to say, "Hey, I really need to be vacuumed now."
> 
> psql -c 'VACUUM' ?

I meant on the part of the back end.  If you have a busy system on
which some tables need very frequent vacuuming, but it gets
unpredictable traffi, you don't just want to say, "Heck, let's vacuum
every hour."  You want to know _actually_ whether the table needs
vacuuming.

So you come up with a bunch of profiles, &c., and do a pile of work
to figure out which tables really need attention.  But that's not
free: it takes cycles on the machine, and the hypotheical case is one
in which the machine is already under heavy load.  So cron is not
really the answer.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Sun, 6 Apr 2003 18:54:52 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: more contrib: log rotator
On Sun, 6 Apr 2003, Peter Eisentraut wrote:

> Tom Lane writes:
> 
> > Seems like a good bare-bones file writer; but how about all those
> > frammishes that people ask for like generating date-based filenames,
> > switching every so many bytes, etc?  Also, it'd be nice not to be
> > dependent on a cron job to tickle the switchover.
> 
> Linux systems have a standard system log rotation mechanism (see
> logrotate(8)), which can rotate logs by size and time and has a number of
> other features.  I would rather depend on that kind of preferred system
> mechanism than rolling out our own.  And we already depend on cron for
> vacuuming anyway.

How about we set up configure to check what we're on and what's available, 
(i.e. rotatelogs, logrotate, joesbiglogrotatorscript, etc...) and 
configure pg_ctl to use one of them?  It's a good probability that most 
flavors of Unix have log rotators of some kind as a built in, and we can 
include a standard one as well.

That way, if you want to use the same log rotator with postgresql as you 
use with the rest of your system, you can, and if you just want the built 
in one, you can use it, and if you don't want any log rotation, everything 
still works the same as before.


Mon, 7 Apr 2003 09:35:07 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: more contrib: log rotator 
On Sunday 06 April 2003 18:54, Andrew Sullivan wrote:
> On Mon, Apr 07, 2003 at 12:42:34AM +0200, Peter Eisentraut wrote:
> > My point was that log file rotation should be left up to the system
> > administrator.  Look at other servers on your system (SMTP, DNS,
> > whatever).  How do they handle it?

> PostgreSQL is not a system process, and I think it's a mistake to
> assume that it is.  We, for instance, do not have root on the
> machines we use.  It's important to assume that users needn't be
> system administrators to use the system.

I personally believe that making the assumption that PostgreSQL is not a 
system process is wrong.  One can run system services as a normal user (in 
fact, it is recommended that as few system services as is possible should run 
as root); but the fact that a daemon is running as a normal user doesn't make 
it not a system process. But that's just a difference of system 
administration opinion.

However, I can see the utility of a bundled simple log rotator.  The key word 
is simple -- we have the full-fledged route now, called syslog.  And if 
someone needs a better logrotator they can certainly get one of the many that 
are already available.

At the same time I don't necessarily want such a log rotator to be the 
default.  We have syslog as the default.  If someone has the particular need 
for a stderr/stdout log rotator, then let it be a configure option.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Mon, 7 Apr 2003 11:45:07 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: more contrib: log rotator
Andrew Sullivan writes:

> PostgreSQL is not a system process, and I think it's a mistake to
> assume that it is.

The point is that PostgreSQL should fit nicely with the customs of the
system that it runs on.  This starts with the oft-discussed file system
layout, the use of syslog in the first place, using 'cron' and 'at'
instead of rolling our own mechanisms to schedule jobs, as is occasionally
requested, fitting in with the startup scripts system, and so on.

> I suppose, however, you could make the argument that log rotation
> should be the responisibility of the adminisistrator of the
> PostgreSQL server.  But that just amounts to an argument that nothing
> needs to be done: as we see, there are lots of log management
> facilities on offer, and none of them are included with PostgreSQL.

That is not the argument.  What we need to do is to make it *possible* to
rotate the logs without shutting down the server, not (necessarily) do the
rotation ourselves.  How can we even begin to do that?  Do we need to
invent a configuration language that can control when to rotate, where to
move the old logs, when to delete the even older logs, etc.?

> I meant on the part of the back end.  If you have a busy system on
> which some tables need very frequent vacuuming, but it gets
> unpredictable traffi, you don't just want to say, "Heck, let's vacuum
> every hour."  You want to know _actually_ whether the table needs
> vacuuming.

That is an argument that manual vacuum is a liability, not the use of
cron for it.

-- 
Peter Eisentraut   peter_e@gmx.net


Mon, 7 Apr 2003 19:06:07 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: more contrib: log rotator
On Sat, 2003-04-05 at 11:35, Ron Peacetree wrote:
> ...and what kind of performance hit we take (and under what
> circumstances) for not having it?

Do you mean 2-phase commits?  If so, how do you take a performance hit
from *not* having it?  PostgreSQL doesn't have it (prepare & forget
phases)) simply because nobody has completed and submitted an
implementation.  Satoshi is working on the problem.

If not, what do you mean by 2 phase execution?

-- 
Rod Taylor <rbt@rbt.ca>

PGP Key: http://www.rbt.ca/rbtpub.asc
07 Apr 2003 13:11:44 -0400
Rod Taylor <rbt@rbt.ca>
Re: Anyone know why PostgreSQL doesn't support 2 phase execution?
Lamar Owen wrote:
> On Sunday 06 April 2003 18:54, Andrew Sullivan wrote:
> > On Mon, Apr 07, 2003 at 12:42:34AM +0200, Peter Eisentraut wrote:
> > > My point was that log file rotation should be left up to the system
> > > administrator.  Look at other servers on your system (SMTP, DNS,
> > > whatever).  How do they handle it?
> 
> > PostgreSQL is not a system process, and I think it's a mistake to
> > assume that it is.  We, for instance, do not have root on the
> > machines we use.  It's important to assume that users needn't be
> > system administrators to use the system.

> I personally believe that making the assumption that PostgreSQL is not
> a system process is wrong.  One can run system services as a normal
> user (in fact, it is recommended that as few system services as is
> possible should run as root); but the fact that a daemon is running as
> a normal user doesn't make it not a system process. But that's just a
> difference of system administration opinion.

I think the mistake lies in making the "design" assumption that
PostgreSQL is either one or the other.

- There are contexts in which it forcibly is "systemy," such as when it
  is used for password authentication using something like PAM.  In that
  case, whatever userid it runs as, it's a forcible "system" dependancy.
  Users can't log in until PostgreSQL is running.

- There are contexts where it will run as a "part of the system," as is
  typically the case when someone uses "apt-get install postgresql" or
  "rpm -i postgres*.rpm"

- In a "hosted" environment, it may be unacceptable to, in any manner,
  treat PostgreSQL or any related services as "part of the system."
  cron obviously *is* a "part of the system," but if you're not the
  system administrator, you may have /no/ ability to connect in to
  "system" logging services.  (In the environment where 
  pgrotatelog runs, that is indeed the case.)

These are /all/ legitimate scenarios for PostgreSQL to be in use.

--> Assuming PostgreSQL /is/ a system process is wrong.
--> Assuming PostgreSQL /is not/ a system process is wrong.

There are situations where either can be true, and it is vital for
PostgreSQL to be able to support both.
--
(concatenate 'string "cbbrowne" "@ntlug.org")
http://www.ntlug.org/~cbbrowne/unix.html
"There  I  was,  lying,  cheating  and back-stabbing  my  way  up  the
corporate ladder, feeling pretty  darn good about myself, when someone
told me  the 'J' in 'WWJD'  meant *Jesus* I thought  it meant *Judas*!
Hoo boy, am I red in the face!"


Mon, 07 Apr 2003 13:40:41 -0400
cbbrowne@cbbrowne.com
Re: more contrib: log rotator 
"Rod Taylor" <rbt@rbt.ca> wrote in message
news:1049735504.40144.35.camel@jester...
> --=-v98gp7DTtZFJa7ee6YMf
> Content-Type: text/plain
> Content-Transfer-Encoding: quoted-printable
>
> On Sat, 2003-04-05 at 11:35, Ron Peacetree wrote:
> > ...and what kind of performance hit we take (and under what
> > circumstances) for not having it?
>
> Do you mean 2-phase commits?  If so, how do you take a performance
hit
> from *not* having it?  PostgreSQL doesn't have it (prepare & forget
> phases)) simply because nobody has completed and submitted an
> implementation.  Satoshi is working on the problem.
>
> If not, what do you mean by 2 phase execution?
>
The performance hit as in "in comparison to DB's that =do= have two
phase execution (and commit for that matter), just how much slower is
PostgreSQL?"

Two phase execution and two phase commit are two different concepts.
Two phase execution splits the execution of queries explicitly into a
"do all the book keeping and setup stuff before execution" phase and
an actual execution phase.  The benefit is that if you are going to
say, step through a largish table in chunks, doing the same query on
each chunk, two phase execution allows the DB to do everything (syntax
checking, query planning, blah, blah) except the actual execution
=once= and reuse it for each subsequent chunk.  Think of it as a close
cousin to loop unrolling.  It also helps parallel performance since
you can hand the "blessed" set up query plan to multiple processes and
those processes can focus on just getting work done.

The lack of two phase =commit= is a also a potential performance hit
in comparison to DB products that have it, but the more important
issue there is that there are SMP/distributed apps that really can't
work acceptably unless a DB product has two phase commit.

The three "biggies" in DB land, SQL Server, Oracle, and DB2, have both
features.  I suspect that PostgreSQL will need to as well...


Mon, 07 Apr 2003 18:59:21 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: Anyone know why PostgreSQL doesn't support 2 phase execution?
"Ron Peacetree" <rjpeace@earthlink.net> writes:

> AFAIK, there are only 3 general purpose internal sorting techniques
> that have O(n) behavior:

Strictly speaking there are no sorting algorithms that have worst-case time
behaviour better than O(nlog(n)). Period.

The typical kind-of O(n) sorts involve either special-case inputs (almost
sorted), or only work if you ignore some aspect of the input size (radix
sort).

So, for example, for radix sort the log(n) factor comes precisely from having
to have log(n) passes. If you use octal that might go to log(n)/8 but it's
still O(log(n)).

If you assume your input fields are limited to a fixed size then O() notation
loses meaning. You can always sort in linear time by just storing bit flags in
a big vector and then scanning your (fixed-size) vector to read out the
values. 

However databases cannot assume fixed size inputs. Regardless of whether it's
on a 32 or 64 bit system postgres still has to sort much larger data
structures. floats are typically 64 - 96 bytes, bigints can be arbitrarily
large.

In fact posgres can sort user-defined datatypes as long as they have < and =
operators. (Actually I'm not sure on the precise constraint.)

Oh, and just to add one last fly in the ointment, postgres has to be able to
sort large datasets that don't even fit in memory. That means storing
temporary data on disk and minimizing the times data has to move from disk to
memory and back. Some alogorithms are better at that than others.

--
greg


07 Apr 2003 15:36:10 -0400
Greg Stark <gsstark@mit.edu>
Re: No merge sort?
Ron Peacetree wrote
> ...and what kind of performance hit we take (and under what
> circumstances) for not having it?

Are you thinking of "two phase commit"?

There is no "performance hit for not having it."

And it does not currently apply to PostgreSQL.  Two phase commit is only
needed when updates need to be applied simultaneously on multiple
databases.

That is, you'd have something like:

CREATE DISTRIBUTED TRANSACTION X1;
  insert into table TBL1 in database ('db1', 'id1', 'auth1') values (1, 2, 100.00, now());
  insert into table TBL2 in database ('db2', 'id2', 'auth2') values (1, 3, -100.00, now());
COMMIT DISTRIBUTED TRANSACTION X1;

where the "in database ('db1', 'id1', 'auth1')" part indicates some form
of connection parameters for the database.

There certainly is merit to having two phase commit; it allows
coordinating updates to multiple databases.

The "degradation of performance" that results from not having this is
that you can't have distributed transactions.  That's not a "performance
hit;" that's a case of "you can't do distributed transactions."  

And distributed transactions are probably /more/ expensive than
nondistributed ones, so it is more readily argued that by not supporting
them, you don't have the problem of performance degrading due to making
use of distributed transactions.
--
output = reverse("gro.gultn@" "enworbbc")
http://www3.sympatico.ca/cbbrowne/nonrdbms.html
Rules of the Evil Overlord #195. "I will not use hostages as bait in a
trap.  Unless  you're going  to use them  for negotiation or  as human
shields, there's no point in taking them."
<http://www.eviloverlord.com/>


Mon, 07 Apr 2003 15:44:06 -0400
cbbrowne@cbbrowne.com
Re: Anyone know why PostgreSQL doesn't support 2 phase execution? 
Ron Peacetree wrote:
> ...and if so, what are the current efforts focusing on?

What is it that you think of as being potentially "better" about some
would-be-alternative "transaction locking" scheme?

PostgreSQL already supports MVCC, which is commonly considered to be the
"better" scheme that eliminates a lot of need to lock data.

Furthermore, the phrase "transaction locking" doesn't seem to describe
what one would want to lock.  I wouldn't want to lock a "transaction;"
I'd want to lock DATA.
--
(concatenate 'string "cbbrowne" "@cbbrowne.com")
http://www.ntlug.org/~cbbrowne/sap.html
Rules of  the Evil  Overlord #153.  "My Legions of  Terror will  be an
equal-opportunity employer. Conversely, when  it is prophesied that no
man  can defeat  me, I  will  keep in  mind the  increasing number  of
non-traditional gender roles." <http://www.eviloverlord.com/>


Mon, 07 Apr 2003 15:48:27 -0400
cbbrowne@cbbrowne.com
Re: Anyone working on better transaction locking? 
On Mon, Apr 07, 2003 at 03:36:10PM -0400, Greg Stark wrote:
> "Ron Peacetree" <rjpeace@earthlink.net> writes:
> 
> > AFAIK, there are only 3 general purpose internal sorting techniques
> > that have O(n) behavior:
> 
> Strictly speaking there are no sorting algorithms that have worst-case time
> behaviour better than O(nlog(n)). Period.
> 

Not true.

http://www.elsewhere.org/jargon/html/entry/bogo-sort.html

-Jay 'Eraserhead' Felice

P.S. <g>


Mon, 7 Apr 2003 16:10:03 -0400
"Jason M. Felice" <jfelice@cronosys.com>
Re: No merge sort?
Ron Peacetree wrote:
> AFAIK, there are only 3 general purpose internal sorting techniques
> that have O(n) behavior:

Hum?  NO "general purpose" sorting technique has O(n) behaviour.

The theoretical best scenario, _in general_, is O(n log n).

Insertion sort is expected to provide O(n^2) behaviour, and radix-like
schemes get arbitrarily memory hungry and have bad pathological results.

> All of the above have potentially nasty trade-offs in comparision to
> the standard heavily tuned median-of-three quicksort used by the sort
> unix command.  Nonetheless, I could see some value in providing all of
> these with PostgeSQL (including a decent port of the unix sort routine
> for the Win folks).  I'll note in passing that quicksort's Achille's
> heel is that it's unstable (all of the rest are stable), which can be
> a problem in a DB.

Making one sort algorithm work very efficiently is quite likely to be a
lot more effective than frittering away time trying to get some special
cases (that you can't regularly use) to work acceptably.

> All of this seems to imply that instead of mergesort (which is
> stable), PostgreSQL might be better off with the 4 sorts I've listed
> plus a viciously efficient merge utility for combining partial results
> that do fit into memory into result files on disk that don't.
> 
> Or am I crazy?

More than likely.  It is highly likely that it will typically take more
computational effort to figure out that one of the 4 sorts provided
/any/ improvement than any computational effort they would save.

That's a /very/ common problem.  There's also a fair chance, seen in
practice, that the action of collecting additional statistics to improve
query optimization will cost more than the savings provided by the
optimizations.
--
(concatenate 'string "cbbrowne" "@acm.org")
http://www3.sympatico.ca/cbbrowne/wp.html
When ever in doubt consult a song. --JT Fletcher 


Mon, 07 Apr 2003 16:20:01 -0400
cbbrowne@cbbrowne.com
Re: No merge sort? 
<cbbrowne@cbbrowne.com> wrote in message
news:20030407202001.1EC3C58E0D@cbbrowne.com...
> Ron Peacetree wrote:
> > AFAIK, there are only 3 general purpose internal sorting
techniques
> > that have O(n) behavior:
>
> Hum?  NO "general purpose" sorting technique has O(n) behaviour.
>
> The theoretical best scenario, _in general_, is O(n log n).
The O(log(n!)) bound is only for comparison based sorts operating on
arbitarily disordered input.  There are general techniques that can
sort in O(n) time if we can "break" either assumption.  In a DB, we
often are dealing with data that is only slightly disordered, or we
are have enough meta knowledge that we can use more powerful ordering
operators than simple comparisons, or both.


> Insertion sort is expected to provide O(n^2) behaviour, and
radix-like
> schemes get arbitrarily memory hungry and have bad pathological
results.
>
None of these comments is accurate.
The sources for the following discussion are
A= Vol 3 of Knuth 2nd ed, (ISBN 0-201-89685-0)
B= Sedgewick's Algorithms in C, (0-201-51425-7)
C= Handbook of Algorithms and Data Structures 2nd ed by Gonnet and
Baeza-Yates. (0-201-41607-7)

1= Insertion sort is O(n) for "almost sorted" input.
p103 of Sedgewick.  There's also discussion on this in Knuth.

2= Distribution Sort and it's "cousins" which use more powerful
ordering operators than simply comparisons are a= reasonably general,
and b= O(n).
Look in all three references.

3= A proper implementation of straight Radix sort using 8b or 16b at a
time a= is NOT pathological, and b= is O(n).
Sedgewick is the most blunt about it on p142-143, but Knuth discusses
this as well.

All of the above are stable, which quicksort is not.  There are no
"pessimal" inputs for any of the above that will force worst case
behavior.  For quicksort there are (they are =very= unlikely however).
In real world terms, if you can use any of these approaches you should
be able to internally sort your data between 2X and 3X faster.


Unfortunately, most of us do not have the luxury of working with
Memory Resident DB's.  In the real world, disk access is an important
part of our DB sorting efforts, and that changes things.  Very fast
internal sorting routines combined with multidisk merging algorithms
that minimize overall disk I/O while maximizing the sequential nature
of that I/O are the best we can do overall in such a situation.


Tue, 08 Apr 2003 01:32:49 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: No merge sort?
<cbbrowne@cbbrowne.com> wrote in message
news:20030407202001.1EC3C58E0D@cbbrowne.com...
> It is highly likely that it will typically take more
> computational effort to figure out that one of the 4 sorts provided
> /any/ improvement than any computational effort they would save.
>
> That's a /very/ common problem.  There's also a fair chance, seen in
> practice, that the action of collecting additional statistics to
improve
> query optimization will cost more than the savings provided by the
> optimizations.
>
"Back in the Day" I heard similar arguments when discussing whether
there should be support for hashing [O(n)], interpolation search
[O(lglg(n))], binary search [O(lg(n))], and sequential search [O(n)]
or for only some subset of these for a DB system I was working on.  To
make a long story short, it was worth it to have support for all of
them because the "useful domain" of each was reasonably large and
reasonably unique compared to the others.

I submit a similar situation exists for sorting.  (and yes, I've been
here before too ;-)

Giving end users of PostgreSQL a reasonable set of tools for building
high performance applications is just good business.


Tue, 08 Apr 2003 02:11:07 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: No merge sort?

"Ron Peacetree" <rjpeace@earthlink.net> wrote in message
news:%cqka.13403$ey1.1154620@newsread1.prod.itd.earthlink.net...
> <cbbrowne@cbbrowne.com> wrote in message
> news:20030407202001.1EC3C58E0D@cbbrowne.com...
> > It is highly likely that it will typically take more
> > computational effort to figure out that one of the 4 sorts
provided
> > /any/ improvement than any computational effort they would save.
> >
> > That's a /very/ common problem.  There's also a fair chance, seen
in
> > practice, that the action of collecting additional statistics to
> improve
> > query optimization will cost more than the savings provided by the
> > optimizations.
> >
> "Back in the Day" I heard similar arguments when discussing whether
> there should be support for hashing [O(n)], interpolation search
TYPO ALERT:  hashing is, of course, O(1)


Tue, 08 Apr 2003 03:11:26 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: No merge sort?
<cbbrowne@cbbrowne.com> wrote in message
news:20030407194827.D0A3A56B1B@cbbrowne.com...
> Ron Peacetree wrote:
> > ...and if so, what are the current efforts focusing on?
>
> What is it that you think of as being potentially "better" about
some
> would-be-alternative "transaction locking" scheme?
>
> PostgreSQL already supports MVCC, which is commonly considered to be
the
> "better" scheme that eliminates a lot of need to lock data.
Agreed.  FTR, the reason MVCC is "better" is that readers and writers
to the same data don't block each other.  In "traditional" locking
schemes, readers don't block each other, but readers and writers to
the same data do.  Clearly, writers to the same data must always block
each other.

Unfortunately, the performance of PostgreSQL MVCC in comparison to say
Oracle (the performance leader amongst MVCC DB's, and pretty much for
all DB's for that matter) is not competitive.  Therefore there is a
need to improve the implementation of MVCC that PostgreSQL uses.  If
someone can post a detailed blow-by-blow comparison of how the two
operate so that the entire list can see it that would be a Good Thing.
If I can, I'll put together the info and post it myself.


> Furthermore, the phrase "transaction locking" doesn't seem to
describe
> what one would want to lock.  I wouldn't want to lock a
"transaction;"
> I'd want to lock DATA.
>
*sigh*.  The accepted terminology within this domain for what we are
talking about is "transaction locking".  Therefore we should use it to
ease communications.  Argue with Codd and Date if you think the term
is a misnomer.  Secondly, you are thinking only in the space
dimension.  Locks have to protect data within a minimum space vs time
"bubble".  That bubble is defined by the beginning and end of a
transaction, hence we call the locking of resources we do during that
bubble as "transaction locking".


Tue, 08 Apr 2003 13:45:25 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: Anyone working on better transaction locking?
On Mon, 2003-04-07 at 14:59, Ron Peacetree wrote:
> Two phase execution and two phase commit are two different concepts.
> Two phase execution splits the execution of queries explicitly into a
> "do all the book keeping and setup stuff before execution" phase and
> an actual execution phase.  The benefit is that if you are going to
> say, step through a largish table in chunks, doing the same query on
> each chunk, two phase execution allows the DB to do everything (syntax
> checking, query planning, blah, blah) except the actual execution
> =once= and reuse it for each subsequent chunk.

If "stepping through a largish table in chunks" is implemented as a
single SQL query, PostgreSQL already does this internally (the parsing,
planning, rewriting, and execution phases are distinct operations inside
the backend).

If the stepping is done as a bunch of similar queries, you can use
prepared queries (as of PostgreSQL 7.3) to do the parsing, planning and
rewriting only once, and then reuse the query plan multiple times.

> It also helps parallel performance since
> you can hand the "blessed" set up query plan to multiple processes and
> those processes can focus on just getting work done.

Prepared queries are per-backend as of PostgreSQL 7.3, so this can't be
done (and I'm a little skeptical that it would be very useful...)

Cheers,

Neil


08 Apr 2003 15:54:50 -0400
Neil Conway <neilc@samurai.com>
Re: Anyone know why PostgreSQL doesn't support 2 phase
On Tue, Apr 08, 2003 at 01:45:25PM +0000, Ron Peacetree wrote:
> Unfortunately, the performance of PostgreSQL MVCC in comparison to say
> Oracle (the performance leader amongst MVCC DB's, and pretty much for
> all DB's for that matter) is not competitive.  Therefore there is a

What, is this a troll?  The question apparently reduces to, "Why
isn't PostgreSQL as good as Oracle?"  I have two things to say about
that:

1.	For what?  There are things that Oracle users will tell you
not to do, because there is a faster way in Oracle.  

2.	How do you know?  I haven't seen any real benchmarks
comparing PostgreSQL and Oracle similarly tuned on similar hardware. 
So I'm sceptical.

But if you have specifica areas which you think need improvement (and
aren't already listed in the TODO), I'll bet people would like to
hear about it.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Tue, 8 Apr 2003 19:05:18 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: Anyone working on better transaction locking?
"Ron Peacetree" <rjpeace@earthlink.net> writes:
> Unfortunately, the performance of PostgreSQL MVCC in comparison to say
> Oracle (the performance leader amongst MVCC DB's, and pretty much for
> all DB's for that matter) is not competitive.

Ron, the tests that I've seen offer no support for that thesis.  If you
want us to accept such a blanket statement as fact, you'd better back
it up with evidence.  Let's see some test cases.

Postgres certainly has plenty of performance issues, but I have no
reason to believe that the fundamental MVCC mechanism is one of them.

			regards, tom lane


Tue, 08 Apr 2003 23:58:19 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Anyone working on better transaction locking? 
"Andrew Sullivan" <andrew@libertyrms.info> wrote in message
news:20030408230518.GB32207@libertyrms.info...
> On Tue, Apr 08, 2003 at 01:45:25PM +0000, Ron Peacetree wrote:
> > Unfortunately, the performance of PostgreSQL MVCC in comparison to
> > say Oracle (the performance leader amongst MVCC DB's, and pretty
much
> > for all DB's for that matter) is not competitive.  Therefore there
is
>
> What, is this a troll?
Time will tell.  Hopefully not.


> The question apparently reduces to, "Why isn't PostgreSQL
> as good as Oracle?"
Actually, you've just used reductio absurdium, not I.  My question
compares PostgreSQL to the performance leaders within this domain
since I'll have to justify my decisions to my bosses based on such
comparisons.  If you think that is unrealistic, then I wish I worked
where you do.  If you think that is unreasonable, then I think you're
treating PostgreSQL as a religion and not a SW product that must
compete against every other DB solution in the real world in order to
be relevant or even survive.


> 1. For what?  There are things that Oracle users will tell you
> not to do, because there is a faster way in Oracle.
>
> 2. How do you know?  I haven't seen any real benchmarks
> comparing PostgreSQL and Oracle similarly tuned on similar hardware.
> So I'm sceptical.
Please see my response(s) to Tom below.


> But if you have specifica areas which you think need improvement
> (and aren't already listed in the TODO), I'll bet people would like
to
> hear about it.
Please see my posts with regards to sorting and searching, two phase
execution, and two phase commit.  I'll mention thread support in
passing, and I'll be bringing up other stuff as I investigate.  Then
I'll hopefully start helping to solve some of the outstanding issues
in priority order...


"Tom Lane" <tgl@sss.pgh.pa.us> wrote in message
news:4096.1049860699@sss.pgh.pa.us...
> Ron, the tests that I've seen offer no support for that thesis.
What tests?  I've seen no tests doing head-to-head,
feature-for-feature comparisons (particularly for low level features
like locking) of PostgreSQL vs the "biggies": DB2, Oracle, and SQL
Server.  What data I have been able to find is application level, and
certainly not head-to-head.  From those performance results, I've had
to try and extrapolate likely causes from behavioral characteristics,
docs, and what internal code I can look at (clearly not much from the
"biggies").

If you have specific head-to-head, feature-for-feature comparison test
results to share, PLEASE do so.  I need the data.


> If you want us to accept such a blanket statement as fact, you'd
> better back it up with evidence.  Let's see some test cases.
Soon as I have the HW and SW to do so, it'll happen.  I have some "bet
the company" decisions to make in the DB realm.

Test cases are, of course, not the only possible evidence.  I'll get
back to you and the list on this.


> Postgres certainly has plenty of performance issues, but I have no
> reason to believe that the fundamental MVCC mechanism is one of
> them.
Where in your opinion are they then?  How bad are they in comparison
to MySQL or any of the "Big Three"?


Wed, 09 Apr 2003 05:41:06 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: Anyone working on better transaction locking?
Ron Peacetree wrote:
> [...]
> The lack of two phase =commit= is a also a potential performance hit
> in comparison to DB products that have it, but the more important
> issue there is that there are SMP/distributed apps that really can't
> work acceptably unless a DB product has two phase commit.
> 
> The three "biggies" in DB land, SQL Server, Oracle, and DB2, have both
> features.  I suspect that PostgreSQL will need to as well...

Ron, do you actually have some ideas how to do 2 phase commits?
Especially things like how to re-lock during startup after a crash and
the like? Or is your knowledge in reality only buzzwords collected from
high glossy tradeshow flyers?


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Wed, 09 Apr 2003 10:13:45 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: Anyone know why PostgreSQL doesn't support 2 phase 
"Jan Wieck" <JanWieck@Yahoo.com> wrote in message
news:3E942A99.A6102F2D@Yahoo.com...
> Ron Peacetree wrote:
> > [...]
> > The lack of two phase =commit= is a also a potential performance
hit
> > in comparison to DB products that have it, but the more important
> > issue there is that there are SMP/distributed apps that really
can't
> > work acceptably unless a DB product has two phase commit.
> >
> > The three "biggies" in DB land, SQL Server, Oracle, and DB2, have
both
> > features.  I suspect that PostgreSQL will need to as well...
>
> Ron, do you actually have some ideas how to do 2 phase commits?
> Especially things like how to re-lock during startup after a crash
and
> the like? Or is your knowledge in reality only buzzwords collected
from
> high glossy tradeshow flyers?
>
If "some ideas" means "do I know how to code it into PostgreSQL right
now", the answer is no.  If "some ideas" means "do I understand the
general problem at a technical level well enough to be thinking about
the algorithms and datastructures needed to support the functionality"
the answer is yes.

So I'd say a fair response to your questions is that my knowledge is
in between the two extremes you've described, but probably closer to
the first than the second ;-).  We can have a private email discussion
on the topic if you wish.


Wed, 09 Apr 2003 16:35:20 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: Anyone know why PostgreSQL doesn't support 2 phase
"Ron Peacetree" <rjpeace@earthlink.net> writes:
> "Tom Lane" <tgl@sss.pgh.pa.us> wrote in message
> news:4096.1049860699@sss.pgh.pa.us...
>> Ron, the tests that I've seen offer no support for that thesis.

> What tests?  I've seen no tests doing head-to-head,
> feature-for-feature comparisons (particularly for low level features
> like locking) of PostgreSQL vs the "biggies": DB2, Oracle, and SQL
> Server.  What data I have been able to find is application level, and
> certainly not head-to-head.

Who said anything about feature-for-feature comparisons?  You made an
(unsupported) assertion about performance, which has little to do with
feature checklists.

The reason I don't believe there's any fundamental MVCC problem is that
no such problem showed up in the head-to-head performance tests that
Great Bridge did about two years ago.  GB is now defunct, and I have
not heard of anyone else willing to stick their neck out far enough to
publish comparative benchmarks against Oracle.  But I still trust the
results they got.

I have helped various people privately with Oracle-to-PG migration
performance problems, and so far the issues have never been MVCC or
transaction issues at all.  What I've seen is mostly planner
shortcomings, such as failure to optimize "foo IN (sub-SELECT)"
decently.  Some of these things are already addressed in development
sources for 7.4.


>> Postgres certainly has plenty of performance issues, but I have no
>> reason to believe that the fundamental MVCC mechanism is one of
>> them.

> Where in your opinion are they then?  How bad are they in comparison
> to MySQL or any of the "Big Three"?

See the TODO list for some of the known problems.  As for "how bad are
they", that depends completely on the particular application and queries
you are looking at ...

			regards, tom lane


Wed, 09 Apr 2003 12:48:04 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Anyone working on better transaction locking? 
On Wed, Apr 09, 2003 at 05:41:06AM +0000, Ron Peacetree wrote:

> Actually, you've just used reductio absurdium, not I.  My question

Nonsense.  You explicitly made the MVCC comparison with Oracle, and
are asking for a "better" locking mechanism without providing any
evidence that PostgreSQL's is bad.  

> compares PostgreSQL to the performance leaders within this domain
> since I'll have to justify my decisions to my bosses based on such
> comparisons.  If you think that is unrealistic, then I wish I worked

Where I work, we test our systems to performance targets having to do
with what we use the database for.  Generic database benchmarks are
not something I have a great deal of faith in.  I repeat my assertion
that, if you have specific areas of concern and the like, and they're
not on the TODO (or in the FAQ), then people would be likely to be
interested; although they'll likely be more interested if the
specifics are not a lot of hand-wavy talk about PostgreSQL not doing
something the right way.

> treating PostgreSQL as a religion and not a SW product that must
> compete against every other DB solution in the real world in order to
> be relevant or even survive.

Actually, given that we are dependent on PostgreSQL's performance and
stability for the whole of the company's revenue, I am pretty certain
that I have as much "real world" experience of PostgreSQL use as
anyone else.  

> Please see my posts with regards to sorting and searching, two phase
> execution, and two phase commit.  

I think your other posts were similar to the one which started this
thread: full of mighty big pronouncements which turned out to depend
on a bunch of not-so-tenable assumptions.  

I'm sorry to be so cranky about this, but I get tired of having to
defend one of my employer's core technologies from accusations based
on half-truths and "everybody knows" assumptions.  For instance,

> I'll mention thread support in passing,

there's actually a FAQ item about thread support, because in the
opinion of those who have looked at it, the cost is just not worth
the benefit.  If you have evidence to the contrary (specific evidence,
please, for this application), and have already read all the previous
discussion of the topic, perhaps people would be interested in
opening that debate again (though I have my doubts).

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Wed, 9 Apr 2003 13:09:26 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: Anyone working on better transaction locking?
Tom Lane wrote:
> The reason I don't believe there's any fundamental MVCC problem is that
> no such problem showed up in the head-to-head performance tests that
> Great Bridge did about two years ago.  GB is now defunct, and I have
> not heard of anyone else willing to stick their neck out far enough to
> publish comparative benchmarks against Oracle.  But I still trust the
> results they got.

<irony-mode-on>
You're missing where Mr Peacetree documented how MVCC performance
degraded by 42.37% between versions 7.1 and 7.3.1, as well as his
extensive statistical analysis of the relative behaviours of
PostgreSQL's semantics versus those of DB/2's MVCC implementation.
</irony-mode-off>

> I have helped various people privately with Oracle-to-PG migration
> performance problems, and so far the issues have never been MVCC or
> transaction issues at all.  What I've seen is mostly planner
> shortcomings, such as failure to optimize "foo IN (sub-SELECT)"
> decently.  Some of these things are already addressed in development
> sources for 7.4.

Ah, but that's just anecdotal evidence...  

And if you used radix sorting, that would probably fix it all.  (At
least until you discovered that you needed 65 bit addressing to set
sort_mem high enough...  Oh, did I neglect to mention anything about
irony?)
--
output = reverse("gro.mca@" "enworbbc")
http://www.ntlug.org/~cbbrowne/oses.html
"Luckily for Microsoft,  it's difficult to see a  naked emperor in the
dark." --- Ted Lewis, (former) editor-in-chief, IEEE Computer


Wed, 09 Apr 2003 13:53:15 -0400
cbbrowne@cbbrowne.com
Re: Anyone working on better transaction locking? 
"Tom Lane" <tgl@sss.pgh.pa.us> wrote in message
news:8236.1049906884@sss.pgh.pa.us...
> "Ron Peacetree" <rjpeace@earthlink.net> writes:
> > "Tom Lane" <tgl@sss.pgh.pa.us> wrote in message
> > news:4096.1049860699@sss.pgh.pa.us...
> >> Ron, the tests that I've seen offer no support for that thesis.
>
> > What tests?  I've seen no tests doing head-to-head,
> > feature-for-feature comparisons (particularly for low level
features
> > like locking) of PostgreSQL vs the "biggies": DB2, Oracle, and SQL
> > Server.  What data I have been able to find is application level,
and
> > certainly not head-to-head.
>
> Who said anything about feature-for-feature comparisons?  You made
an
> (unsupported) assertion about performance, which has little to do
with
> feature checklists.
>
That's not quite fair.  My assertion was about the performance of an
exact feature in comparison to that same feature in another DB
product, not about overall application level performance...  As I
said, I'll get back to you and the list on this.


> The reason I don't believe there's any fundamental MVCC problem is
that
> no such problem showed up in the head-to-head performance tests that
> Great Bridge did about two years ago.  GB is now defunct, and I have
> not heard of anyone else willing to stick their neck out far enough
to
> publish comparative benchmarks against Oracle.  But I still trust
the
> results they got.
>
Last year eWeek did a shoot out that PostgreSQL was notable in its
absence from:
http://www.eweek.com/print_article/0,3668,a=23115,00.asp
Taking those results and adding PostgreSQL to them should be eminently
feasible since the entire environment used for the test is documented
and the actual scripts and data used for the test are also available.
Of course, MySQL has been evolving at such a ferocious rate that even
one year old results, let alone two year old ones, run the risk of not
being accurate for it.


> I have helped various people privately with Oracle-to-PG migration
> performance problems, and so far the issues have never been MVCC or
> transaction issues at all.  What I've seen is mostly planner
> shortcomings, such as failure to optimize "foo IN (sub-SELECT)"
> decently.  Some of these things are already addressed in development
> sources for 7.4.
>
It's probably worth noting that since SQL support was added to
Postgres rather than being part of the product from Day One, certain
"hard" SQL constructs may still be having teething problems.  NOT IN,
for instance, was a problem for both Oracle and SQL Server at some
point in their history (fuzzy memory: pre Oracle 6, not sure about SQL
Server version...)


> >> Postgres certainly has plenty of performance issues, but I have
no
> >> reason to believe that the fundamental MVCC mechanism is one of
> >> them.
>
> > Where in your opinion are they then?  How bad are they in
comparison
> > to MySQL or any of the "Big Three"?
>
> See the TODO list for some of the known problems.  As for "how bad
are
> they", that depends completely on the particular application and
queries
> you are looking at ...
>
Fair enough.


Wed, 09 Apr 2003 17:53:17 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: Anyone working on better transaction locking?
"Andrew Sullivan" <andrew@libertyrms.info> wrote in message
news:20030409170926.GH2255@libertyrms.info...
> On Wed, Apr 09, 2003 at 05:41:06AM +0000, Ron Peacetree wrote:
> Nonsense.  You explicitly made the MVCC comparison with Oracle, and
> are asking for a "better" locking mechanism without providing any
> evidence that PostgreSQL's is bad.
>
Just because someone else's is "better" does not mean PostgreSQL's is
"bad", and I've never said such.  As I've said, I'll get back to Tom
and the list on this.


> > compares PostgreSQL to the performance leaders within this domain
> > since I'll have to justify my decisions to my bosses based on such
> > comparisons.  If you think that is unrealistic, then I wish I
> > worked where you do.
>
> Where I work, we test our systems to performance targets having to
> do with what we use the database for.  Generic database benchmarks
> are not something I have a great deal of faith in.  I repeat my
> assertion that, if you have specific areas of concern and the like,
> and they're not on the TODO (or in the FAQ), then people would be
> likely to be interested; although they'll likely be more interested
if the
> specifics are not a lot of hand-wavy talk about PostgreSQL not doing
> something the right way.
>
There's nothing "hand wavy"about this unless you think anything except
test cases is "hand wavy".  In that case, you're right.  I don't have
the time or resources to provide exhaustive tests between each DB for
each of the issues we are discussing.  If I did, I'd be publishing a
=very= lucrative newsletter for IT decision makers.  Also, there are
other
valid ways to analyze issues than just application level test cases.
In fact, there are some =better= ways, depending on the issue being
discussed.


> > treating PostgreSQL as a religion and not a SW product that must
> > compete against every other DB solution in the real world in order
> > to be relevant or even survive.
>
> Actually, given that we are dependent on PostgreSQL's performance
> and stability for the whole of the company's revenue, I am pretty
> certain that I have as much "real world" experience of PostgreSQL
> use as anyone else.
>
Your experience was not questioned, and there were "if" clauses at the
beginning of my comments that you seem to be ignoring.  I'm not here
to waste my or anyone else's time on flames.  We've all got work to
do.


> > Please see my posts with regards to ...
>
> I think your other posts were similar to the one which started this
> thread: full of mighty big pronouncements which turned out to depend
> on a bunch of not-so-tenable assumptions.
>
Hmmm.  Well, I don't think of algorithm analysis by the likes of
Knuth, Sedgewick, Gonnet, and Baeza-Yates as being "not so tenable
assumptions", but YMMV.  As for "mighty pronouncements", that also
seems a bit misleading since we are talking about quantifiable
programming and computer science issues, not unquantifiable things
like politics.


> I'm sorry to be so cranky about this, but I get tired of having to
> defend one of my employer's core technologies from accusations based
> on half-truths and "everybody knows" assumptions.  For instance,
>
Again, "accusations" is a bit strong.  I thought the discussion was
about the technical merits and costs of various features and various
ways to implement them, particularly when this product must compete
for installed base with other solutions.  Being coldly realistic about
what a product's strengths and weaknesses are is, again, just good
business.  Sun Tzu's comment about knowing the enemy and yourself
seems appropriate here...


> > I'll mention thread support in passing,
>
> there's actually a FAQ item about thread support, because in the
> opinion of those who have looked at it, the cost is just not worth
> the benefit.  If you have evidence to the contrary (specific
> evidence, please, for this application), and have already read all
the
> previous discussion of the topic, perhaps people would be interested
in
> opening that debate again (though I have my doubts).
>
Zeus had a performance ceiling roughly 3x that of Apache when Zeus
supported threading as well as pre-forking and Apache only supported
pre forking.  The Apache folks now support both.  DB2, Oracle, and SQL
Server all use threads.  Etc, etc.

That's an awful lot of very bright programmers and some serious $$
voting that threads are worth it.  Given all that, if PostgreSQL
specific
thread support is =not= showing itself to be a win that's an
unexpected
enough outcome that we should be asking hard questions as to why not.

At their core, threads are a context switching efficiency tweak.
Since DB's switch context a lot under many circumstances, threads
should be a win under such circumstances.  At the least, it should be
helpful in situations where we have multiple CPUs to split query
execution between.

M$'s first implementation of threads was so "heavy" that it didn't
help them (until they actually implemented real threads and called
them "strings"), but that was not due to the inefficacy of the
concept, but rather M$'s implementation and the system environment
within which that implementation was being used.  Perhaps something
similar is going on here?

Certainly it's =possible= that threads have nothing to offer
PostgreSQL, but IMHO it's not =probable=.  Just another thing for me
to add to my TODO heap for looking at...


Wed, 09 Apr 2003 22:09:14 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: Anyone working on better transaction locking?

"Ron Peacetree" <rjpeace@earthlink.net> wrote in message
news:eS0la.16229$ey1.1398978@newsread1.prod.itd.earthlink.net...
> M$'s first implementation of threads was so "heavy" that it didn't
> help them (until they actually implemented real threads and called
> them "strings"),
TYPO ALERT: M$'s better implementation of threads is called "fibers",
not "strings"


Thu, 10 Apr 2003 01:15:26 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: Anyone working on better transaction locking?
Ron Peacetree wrote:
> "Tom Lane" <tgl@sss.pgh.pa.us> wrote in message
> > [...]
> > If you want us to accept such a blanket statement as fact, you'd
> > better back it up with evidence.  Let's see some test cases.
> Soon as I have the HW and SW to do so, it'll happen.  I have some "bet
> the company" decisions to make in the DB realm.

And you are comparing what? Just pure features and/or performace, or
total cost of ownership for your particular case?

It is a common misunderstanding open source would be free software. It
is not because since the software comes as is, without any warranty and
it's usually hard to get support provided or backed by large companies,
it is safe to build you own support team (depends on how much you "bet
the company"). Replacing license fees and support contracts with payroll
entries plus taking the feature and performance differences into account
makes this comparision a very individual, non-portable task.

Unfortunately most manager type people can produce an annoyingly high
volume of questions and suggestions as long as they need more input,
then all of the sudden disappear when they made their decision.


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Thu, 10 Apr 2003 09:12:56 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: Anyone working on better transaction locking?
"Jan Wieck" <JanWieck@Yahoo.com> wrote in message
news:3E956DD8.29432405@Yahoo.com...
> Ron Peacetree wrote:
> > "Tom Lane" <tgl@sss.pgh.pa.us> wrote in message
> > > [...]
> > > If you want us to accept such a blanket statement
> > > as fact, you'd better back it up with evidence.  Let's
> > > see some test cases.
> > Soon as I have the HW and SW to do so, it'll happen.
> > I have some "bet the company" decisions to make.
>
> And you are comparing what? Just pure features and/or
> performance, or total cost of ownership for your
> particular case?
>
Technical Analysis and Business Analysis are two separate, and equally
necessary, activities.  However, before one can accurately measure
things like Total Cost of Ownership, one needs to have accurately and
sufficiently characterized what will be owned and one's choices as to
what could be owned...


> It is a common misunderstanding open source would be
> free software. It is not because since the software comes
> as is, without any warranty and it's usually hard to get
> support provided or backed by large companies, it is safe
> to build you own support team (depends on how much
> you "bet the company"). Replacing license fees and
> support contracts with payroll entries plus taking the
> feature and performance differences into account  makes
> this comparision a very individual, non-portable task.
>
Very valid points, and I was a supporter of the FSF and the LPF when
Usenet was "the net" and backbone nodes communicated by modem, so I've
been wrestling with people's sometimes misappropriate
use/understanding of the operator "free" for some time.

However, a correctly done Technical Analysis =should= be reasonably
portable since among other things you don't want to have to start all
over if your company's business or business model changes.  Clearly
Business Analysis is very context dependant.

It should also be noted that given the prices of some of the solutions
out there, there are many companies who's choices are constrained, but
still need to stay in business...


> Unfortunately most manager type people can produce an
> annoyingly high volume of questions and suggestions as
> long as they need more input, then all of the sudden
> disappear when they made their decision.
>
Word.  Although the phrase "manager type people" could be replaced
with "people" and the above would still be true IMHO.  Thankfully,
most of my bosses are people who have worked their up from the
technical trenches, so the conversation at least rates to be focused
and reasonable while it's occurring...


Thu, 10 Apr 2003 18:20:13 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: Anyone working on better transaction locking?
On Wed, 9 Apr 2003, Ron Peacetree wrote:

> "Andrew Sullivan" <andrew@libertyrms.info> wrote in message
> news:20030409170926.GH2255@libertyrms.info...
> > On Wed, Apr 09, 2003 at 05:41:06AM +0000, Ron Peacetree wrote:
> > Nonsense.  You explicitly made the MVCC comparison with Oracle, and
> > are asking for a "better" locking mechanism without providing any
> > evidence that PostgreSQL's is bad.
> >
> Just because someone else's is "better" does not mean PostgreSQL's is
> "bad", and I've never said such.  As I've said, I'll get back to Tom
> and the list on this.

But you didn't identify HOW it was better.  I think that's the point 
being made.

> > > Please see my posts with regards to ...
> >
> > I think your other posts were similar to the one which started this
> > thread: full of mighty big pronouncements which turned out to depend
> > on a bunch of not-so-tenable assumptions.
> >
> Hmmm.  Well, I don't think of algorithm analysis by the likes of
> Knuth, Sedgewick, Gonnet, and Baeza-Yates as being "not so tenable
> assumptions", but YMMV.  As for "mighty pronouncements", that also
> seems a bit misleading since we are talking about quantifiable
> programming and computer science issues, not unquantifiable things
> like politics.

But the real truth is revealed when the rubber hits the pavement.  
Remember that Linux Torvalds was roundly criticized for his choice of a 
monolithic development model for his kernel, and was literally told that 
his choice would restrict to "toy" status and that no commercial OS could 
scale with a monolithic kernel.

There's no shortage of people with good ideas, just people with the skills 
to implement those good ideas.  If you've got a patch to apply that's been 
tested to show something is faster EVERYONE here wants to see it.

If you've got a theory, no matter how well backed up by academic research, 
it's still just a theory.  Until someone writes to code to implement it, 
the gains are theoretical, and many things that MIGHT help don't because 
of the real world issues underlying your database, like I/O bandwidth or 
CPU <-> memory bandwidth.

> > I'm sorry to be so cranky about this, but I get tired of having to
> > defend one of my employer's core technologies from accusations based
> > on half-truths and "everybody knows" assumptions.  For instance,
> >
> Again, "accusations" is a bit strong.  I thought the discussion was
> about the technical merits and costs of various features and various
> ways to implement them, particularly when this product must compete
> for installed base with other solutions.  Being coldly realistic about
> what a product's strengths and weaknesses are is, again, just good
> business.  Sun Tzu's comment about knowing the enemy and yourself
> seems appropriate here...

No, you're wrong.  Postgresql doesn't have to compete.  It doesn't have to 
win.  it doesn't need a marketing department.  All those things are nice, 
and I'm glad if it does them, but doesn't HAVE TO.  Postgresql has to 
work.  It does that well.

Postgresql CAN compete if someone wants to put the effort into competing, 
but it isn't a priority for me.  Working is the priority, and if other 
people aren't smart enough to test Postgresql to see if it works for them, 
all the better, I keep my edge by having a near zero cost database engine, 
while the competition spends money on MSSQL or Oracle.

Tom and Andrew ARE coldly realistic about the shortcomings of postgresql.  
It has issues, and things that need to be fixed.  It needs more coders.  
It doesn't need every feature that Oracle or DB2 have.  Heck some of their 
"features" would be considered a mis-feature in the Postgresql world.

> > > I'll mention thread support in passing,
> >
> > there's actually a FAQ item about thread support, because in the
> > opinion of those who have looked at it, the cost is just not worth
> > the benefit.  If you have evidence to the contrary (specific
> > evidence, please, for this application), and have already read all
> the
> > previous discussion of the topic, perhaps people would be interested
> in
> > opening that debate again (though I have my doubts).
> >
> Zeus had a performance ceiling roughly 3x that of Apache when Zeus
> supported threading as well as pre-forking and Apache only supported
> pre forking.  The Apache folks now support both.  DB2, Oracle, and SQL
> Server all use threads.  Etc, etc.

Yes, and if you configured your apache server to have 20 or 30 spare 
servers, in the real world, it was nearly neck and neck to Zeus, but since 
Zeus cost like $3,000 a copy, it is still cheaper to just overwhelm it 
with more servers running apache than to use zeus.

> That's an awful lot of very bright programmers and some serious $$
> voting that threads are worth it.  

For THAT application.  for what a web server does, threads can be very 
useful, even useful enough to put up with the problems created by running 
threads on multiple threading libs on different OSes.  

Let me ask you, if Zeus scrams and crashes out, and it's installed 
properly so it just comes right back up, how much data can you lose?

If Postgresql scrams and crashes out, how much data can you lost?

> Given all that, if PostgreSQL
> specific
> thread support is =not= showing itself to be a win that's an
> unexpected
> enough outcome that we should be asking hard questions as to why not.

There HAS been testing on threads in Postgresql.  It has been covered to 
death.  The fact that you're still arguing proves you likely haven't read 
the archive (google has it back to way back when, use that to look it up) 
about this subject.

Threads COULD help on multi-sorted results, and a few other areas, but the 
increase in performance really wasn't that great for 95% of all the cases, 
and for the 5% it was, simple query planner improvements have provided far 
greater performance increases.

The problem with threading is that we can either use the one process -> 
many thread design, which I personally don't trust for something like a 
database, or a process per backend connection which can run 
multi-threaded.  This scenario makes Postgresql just as stable and 
reliable as it was as a multi-process app, but allows threaded performance 
in certain areas of the backend that are parallelizable to run in parallel 
on multi-CPU systems.

the gain, again, is minimal, and on a system with many users accessing it, 
there is NO real world gain.

> At their core, threads are a context switching efficiency tweak.

Except that on the two OSes which Postgresql runs on the most, threads are 
really no faster than processes.  In the Linux kernel, the only real 
difference is how the OS treats them, creation, destruction of threads 
versus processes is virtually identical there.

> Certainly it's =possible= that threads have nothing to offer
> PostgreSQL, but IMHO it's not =probable=.  Just another thing for me
> to add to my TODO heap for looking at...

It's been tested, it didn't help a lot, and it made it MUCH harder to 
maintain, as threads in Linux are handled by a different lib than in say 
Solaris, or Windows or any other OS.  I.e. you can't guarantee the thread 
lib you need will be there, and that there are no bugs.  MySQL still has 
thread bug issues pop up, most of which are in the thread libs themselves.


Fri, 11 Apr 2003 13:31:06 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Anyone working on better transaction locking?
> -----Original Message-----
> From: Ron Peacetree [mailto:rjpeace@earthlink.net] 
> Sent: Wednesday, April 09, 2003 12:38 PM
> To: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] No merge sort?
> 
> 
> ""Dann Corbit"" <DCorbit@connx.com> wrote in message 
> news:D90A5A6C612A39408103E6ECDD77B829408AC6@voyager.corporate.connx.co
> m...
> > Distribution sort is not an algorithm.  It is a general technique.
> Both
> > counting sort and radix sort are distribution sorts.  I think you
> are
> > talking about counting sort here.  In order to insert a technique
> into a
> > database, you must solve the general case.
> >
> Terminology problem.  Please look in the references I posted.
> 
> 
> > > 2= For Radix sort, that's iff ("if and only if") you use 
> > > =characters= as the radix of a radix sort and do a MSB aka 
> > > partition-exchange sort. The appropriate radix here is a 
> =3Dfield=3D 
> > > not a character.  Since there are 3 fields vs 60 characters the 
> > > problem becomes 2/3 wasted passes instead of 40/60.
> >
> > This is lunacy.  If you count the field or use it as a radix, you
> will
> > need a radix of 40*8 bits= 320 bits.  That means you will 
> have 2^320 
> > 2.136e96 bins.
> >
> You only need as many bins as you have unique key values 
> silly ;-) Remember, at its core Radix sort is just a 
> distribution counting sort (that's the name I learned for the 
> general technique).  The simplest implementation uses bits as 
> the atomic unit, but there's nothing saying you have to...  
> In a DB, we know all the values of the fields we currently 
> have stored in the DB.  We can take advantage of that.

By what magic do we know this?  If a database knew a-priori what all the
distinct values were, it would indeed be excellent magic.
 
> As you correctly implied, the goal is to minimize the number 
> of unique bins you have to, err, distribute, items into.  
> That and having as few duplicates as feasible are the 
> important points.
> 
> If, for example, we have <= 255 possible values for the each 
> radix (Using your previous example, let's say you have <= 255 
> unique values for the combined field "Company+Division" in 
> the DB and the same for "Plant" ), and 4 sets of radix to 
> sort over, it doesn't matter if we are sorting a 32b quantity 
> using a radix of 8b, or a 4 field quantity using a radix of 
> one field (TBF, we should use key and index techniques to 
> minimize the amount of actual data we retrieve and manipulate 
> from disk during the sort.  We want to minimize disk I/O, 
> particularly seeking disk I/O).  The situations are analogous.

With 80 bytes, you have 2^320 possible values.  There is no way around
that.  If you are going to count them or use them as a radix, you will
have to classify them.  The only way you will know how many unique
values you have in "Company+Division" is to ... Either sort them or by
some other means discover all that are distinct.  The database does not
know how many there are beforehand.  Indeed, there could be anywhere
from zero to 2^320 (given enough space) distinct values.

I would be interested to see your algorithm that performs a counting or
radix sort on 320 bit bins and that works in the general case without
using extra space.

> Note TANSTAAFL (as Heinlein would've said):  We have to use 
> extra space for mapping the radix values to the radix keys, 
> and our "inner loop" for the sorting operation is 
> considerably more complicated than that for a quicksort or a 
> mergesort.  Hence the fact that even though this is an O(n) 
> technique, in real world terms you can expect only a 2-3X 
> performance improvement over say quicksort for realistic 
> amounts of data.
> 
> Oh and FTR, IME for most "interesting" sorts in the DB 
> domain, even for "internal" sorting techniques, the time to 
> read data from disk and
> (possibly) write results back to disk tends to dominate the 
> time spent actually doing the internal sort...
> 
> I learned tricks like this 20 years ago.  I thought this 
> stuff was part of general lore in the DB field... *shrug*

As my grandfather used to say:
"Picklesmoke."

> ---------------------------(end of 
> broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to 
> majordomo@postgresql.org
> 


Fri, 11 Apr 2003 13:51:01 -0700
"Dann Corbit" <DCorbit@connx.com>
Re: No merge sort?
Ron Peacetree wrote:
> Zeus had a performance ceiling roughly 3x that of Apache when Zeus
> supported threading as well as pre-forking and Apache only supported
> pre forking.  The Apache folks now support both.  DB2, Oracle, and SQL
> Server all use threads.  Etc, etc.

You can't use Apache as an example of why you should thread a database
engine, except for the cases where the database is used much like the
web server is: for numerous short transactions.

> That's an awful lot of very bright programmers and some serious $$
> voting that threads are worth it.  Given all that, if PostgreSQL
> specific thread support is =not= showing itself to be a win that's
> an unexpected enough outcome that we should be asking hard questions
> as to why not.

It's not that there won't be any performance benefits to be had from
threading (there surely will, on some platforms), but gaining those
benefits comes at a very high development and maintenance cost.  You
lose a *lot* of robustness when all of your threads share the same
memory space, and make yourself vulnerable to classes of failures that
simply don't happen when you don't have shared memory space.

PostgreSQL is a compromise in this regard: it *does* share memory, but
it only shares memory that has to be shared, and nothing else.  To get
the benefits of full-fledged threads, though, requires that all memory
be shared (otherwise the OS has to tweak the page tables whenever it
switches contexts between your threads).

> At their core, threads are a context switching efficiency tweak.

This is the heart of the matter.  Context switching is an operating
system problem, and *that* is where the optimization belongs.  Threads
exist in large part because operating system vendors didn't bother to
do a good job of optimizing process context switching and
creation/destruction.

Under Linux, from what I've read, process creation/destruction and
context switching happens almost as fast as thread context switching
on other operating systems (Windows in particular, if I'm not
mistaken).

> Since DB's switch context a lot under many circumstances, threads
> should be a win under such circumstances.  At the least, it should be
> helpful in situations where we have multiple CPUs to split query
> execution between.

This is true, but I see little reason that we can't do the same thing
using fork()ed processes and shared memory instead.

There is context switching within databases, to be sure, but I think
you'll be hard pressed to demonstrate that it is anything more than an
insignificant fraction of the total overhead incurred by the database.
I strongly suspect that much larger gains are to be had by optimizing
other areas of the database, such as the planner, the storage manager
(using mmap for file handling may prove useful here), the shared
memory system (mmap may be faster than System V style shared memory),
etc.

The big overhead in the process model on most platforms is in creation
and destruction of processes.  PostgreSQL has a relatively high
connection startup cost.  But there are ways of dealing with this
problem other than threading, namely the use of a connection caching
middleware layer.  Such layers exist for databases other than
PostgreSQL, so the high cost of fielding and setting up a database
connection is *not* unique to PostgreSQL ... which suggests that while
threading may help, it doesn't help *enough*.

I'd rather see some development work go into a connection caching
process that understands the PostgreSQL wire protocol well enough to
look like a PostgreSQL backend to connecting processes, rather than
see a much larger amount of effort be spent on converting PostgreSQL
to a threaded architecture (and then discover that connection caching
is still needed anyway).

> Certainly it's =possible= that threads have nothing to offer
> PostgreSQL, but IMHO it's not =probable=.  Just another thing for me
> to add to my TODO heap for looking at...

It's not that threads don't have anything to offer.  It's that the
costs associated with them are high enough that it's not at all clear
that they're an overall win.


-- 
Kevin Brown					      kevin@sysexperts.com


Fri, 11 Apr 2003 14:32:59 -0700
Kevin Brown <kevin@sysexperts.com>
Re: Anyone working on better transaction locking?
""Dann Corbit"" <DCorbit@connx.com> wrote in message
news:D90A5A6C612A39408103E6ECDD77B829408ACB@voyager.corporate.connx.co
m...
> > From: Ron Peacetree [mailto:rjpeace@earthlink.net]=20
> > Sent: Wednesday, April 09, 2003 12:38 PM
> > You only need as many bins as you have unique key values=20
> > silly ;-) Remember, at its core Radix sort is just a=20
> > distribution counting sort (that's the name I learned for the=20
> > general technique).  The simplest implementation uses bits as=20
> > the atomic unit, but there's nothing saying you have to...=20=20
> > In a DB, we know all the values of the fields we currently=20
> > have stored in the DB.  We can take advantage of that.
>
> By what magic do we know this?  If a database knew a-priori what all
the
> distinct values were, it would indeed be excellent magic.
For any table already in the DB, this is self evident.  If you are
talking about sorting data =before= it gets put into the DB (say for a
load), then things are different, and the best you can do is used
comparison based methods (and probably some reasonably sophisticated
external merging routines in addition if the data set to be sorted and
loaded is big enough).  The original question as I understood it was
about a sort as part of a query.  That means everything to be sorted
is in the DB, and we can take advantage of what we know.


> With 80 bytes, you have 2^320 possible values.  There is no way
> around that.  If you are going to count them or use them as a
> radix, you will have to classify them.  The only way you will
> know how many unique values you have in
> "Company+Division" is to ...
> Either sort them or by some means discover all that are distinct
Ummm, Indexes, particularly Primary Key Indexes, anyone?  Finding the
unique values in an index should be perceived as trivial... ...and you
often have the index in memory for other reasons already.


> .  The database does not know how many there are
> beforehand.  Indeed, there could be anywhere
> from zero to 2^320 (given enough space) distinct values.
>
> I would be interested to see your algorithm that
> performs a counting or radix sort on 320 bit bins and that
> works in the general case without using extra space.
>
1= See below.  I clearly stated that we need to use extra space
2= If your definition of "general" is "we know nothing about the
data", then of course any method based on using more sophisticated
ordering operators than comparisons is severely hampered, if not
doomed.  This is not a comparison based method.  You have to know some
things about the data being sorted before you can use it.  I've been
=very= clear about that throughout this.


> > Note TANSTAAFL (as Heinlein would've said):  We have to use=20
> > extra space for mapping the radix values to the radix keys,=20
> > and our "inner loop" for the sorting operation is=20
> > considerably more complicated than that for a quicksort or a=20
> > mergesort.  Hence the fact that even though this is an O(n)=20
> > technique, in real world terms you can expect only a 2-3X=20
> > performance improvement over say quicksort for realistic=20
> > amounts of data.
> >=20
> > Oh and FTR, IME for most "interesting" sorts in the DB=20
> > domain, even for "internal" sorting techniques, the time to=20
> > read data from disk and
> > (possibly) write results back to disk tends to dominate the=20
> > time spent actually doing the internal sort...
> >=20
Since you don't believe me, and have implied you wouldn't believe me
even if I posted results of efforts on my part, go sort some 2GB files
by implementing the algorithms in the sources I've given and as
mentioned in this thread.  (I'm assuming that you have access to
"real" machines that can perform this as an internal sort, or we
wouldn't be bothering to have this discussion). Then come back to the
table if you still think there are open issues to be discussed.


Sat, 12 Apr 2003 00:01:40 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: No merge sort?
"Ron Peacetree" <rjpeace@earthlink.net> wrote in message
news:EHIla.19833$ey1.1702921@newsread1.prod.itd.earthlink.net...
> ""Dann Corbit"" <DCorbit@connx.com> wrote in message
>
news:D90A5A6C612A39408103E6ECDD77B829408ACB@voyager.corporate.connx.co
> m...
> > With 80 bytes, you have 2^320 possible values.  There is no way
> > around that.  If you are going to count them or use them as a
> > radix, you will have to classify them.  The only way you will
> > know how many unique values you have in
> > "Company+Division" is to ...
> > Either sort them or by some means discover all that are distinct
> Ummm, Indexes, particularly Primary Key Indexes, anyone?  Finding
the
> unique values in an index should be perceived as trivial... ...and
you
> often have the index in memory for other reasons already.
>
Interesting Note: DB2 and Oracle have switches that turn on a table
feature that keeps track of all the unique values of a field and a
counter for how often each of those unique values occurs.  The
implications for speeding up non write querys involving those tables
should be obvious (again, TANSTAAFL: writes are now going to have the
extra overhead of updating this information)...

Wonder how hard this would be to put into PostgreSQL?


Sat, 12 Apr 2003 02:33:04 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: No merge sort?
On Saturday 12 April 2003 03:02, you wrote:
> Ron Peacetree wrote:
> > Zeus had a performance ceiling roughly 3x that of Apache when Zeus
> > supported threading as well as pre-forking and Apache only supported
> > pre forking.  The Apache folks now support both.  DB2, Oracle, and SQL
> > Server all use threads.  Etc, etc.
>
> You can't use Apache as an example of why you should thread a database
> engine, except for the cases where the database is used much like the
> web server is: for numerous short transactions.

OK. Let me put my experiences. These are benchmarks on a intranet(100MBps lan) 
run off a 1GHZ P-III/IV webserver on mandrake9 for a single 8K file.

apache2044: 1300 rps
boa:  	4500rps
Zeus: 	6500 rps.

Apache does too many things to be a speed daemon and what it offers is pretty 
impressive from performance POV.

But database is not webserver. It is not suppose to handle tons of concurrent 
requests. That is a fundamental difference.

>
> > That's an awful lot of very bright programmers and some serious $$
> > voting that threads are worth it.  Given all that, if PostgreSQL
> > specific thread support is =not= showing itself to be a win that's
> > an unexpected enough outcome that we should be asking hard questions
> > as to why not.
>
> It's not that there won't be any performance benefits to be had from
> threading (there surely will, on some platforms), but gaining those
> benefits comes at a very high development and maintenance cost.  You
> lose a *lot* of robustness when all of your threads share the same
> memory space, and make yourself vulnerable to classes of failures that
> simply don't happen when you don't have shared memory space.

Well. Threading does not necessarily imply one thread per connection model. 
Threading can be used to make CPU work during I/O and taking advantage of SMP 
for things like sort etc. This is especially true for 2.4.x linux kernels 
where async I/O can not be used for threaded apps. as threads and signal do 
not mix together well.

One connection per thread is not a good model for postgresql since it has 
already built a robust product around process paradigm. If I have to start a 
new database project today, a mix of process+thread is what I would choose bu 
postgresql is not in same stage of life.

> > At their core, threads are a context switching efficiency tweak.
>
> This is the heart of the matter.  Context switching is an operating
> system problem, and *that* is where the optimization belongs.  Threads
> exist in large part because operating system vendors didn't bother to
> do a good job of optimizing process context switching and
> creation/destruction.

But why would a database need a tons of context switches if it is not supposed 
to service loads to request simaltenously? If there are 50 concurrent 
connections, how much context switching overhead is involved regardless of 
amount of work done in a single connection? Remeber that database state is 
maintened in shared memory. It does not take a context switch to access it.

The assumption stems from database being very efficient in creating and 
servicing a new connection. I am not very comfortable with that argument.

> Under Linux, from what I've read, process creation/destruction and
> context switching happens almost as fast as thread context switching
> on other operating systems (Windows in particular, if I'm not
> mistaken).

I hear solaris also has very heavy processes. But postgresql has other issues 
with solaris as well.
>
> > Since DB's switch context a lot under many circumstances, threads
> > should be a win under such circumstances.  At the least, it should be
> > helpful in situations where we have multiple CPUs to split query
> > execution between.

Can you give an example where database does a lot of context switching for 
moderate number of connections?

 Shridhar


Sat, 12 Apr 2003 12:21:12 +0530
Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
Re: Anyone working on better transaction locking?
Shridhar Daithankar wrote:
> Apache does too many things to be a speed daemon and what it offers
> is pretty impressive from performance POV.
>
> But database is not webserver. It is not suppose to handle tons of
> concurrent requests. That is a fundamental difference.

I'm not sure I necessarily agree with this.  A database is just a
tool, a means of reliably storing information in such a way that it
can be retrieved quickly.  Whether or not it "should" handle lots of
concurrent requests is a question that the person trying to use it
must answer.

A better answer is that a database engine that can handle lots of
concurrent requests can also handle a smaller number, but not vice
versa.  So it's clearly an advantage to have a database engine that
can handle lots of concurrent requests because such an engine can be
applied to a larger number of problems.  That is, of course, assuming
that all other things are equal...

There are situations in which a database would have to handle a lot of
concurrent requests.  Handling ATM transactions over a large area is
one such situation.  A database with current weather information might
be another, if it is actively queried by clients all over the country.
Acting as a mail store for a large organization is another.  And, of
course, acting as a filesystem is definitely another.  :-)

> Well. Threading does not necessarily imply one thread per connection
> model. Threading can be used to make CPU work during I/O and taking
> advantage of SMP for things like sort etc. This is especially true
> for 2.4.x linux kernels where async I/O can not be used for threaded
> apps. as threads and signal do not mix together well.

This is true, but whether you choose to limit the use of threads to a
few specific situations or use them throughout the database, the
dangers and difficulties faced by the developers when using threads
will be the same.

> One connection per thread is not a good model for postgresql since
> it has already built a robust product around process paradigm. If I
> have to start a new database project today, a mix of process+thread
> is what I would choose bu postgresql is not in same stage of life.

Certainly there are situations for which it would be advantageous to
have multiple concurrent actions happening on behalf of a single
connection, as you say.  But that doesn't automatically mean that a
thread is the best overall solution.  On systems such as Linux that
have fast process handling, processes are almost certainly the way to
go.  On other systems such as Solaris or Windows, threads might be the
right answer (on Windows they might be the *only* answer).  But my
argument here is simple: the responsibility of optimizing process
handling belongs to the maintainers of the OS.  Application developers
shouldn't have to worry about this stuff.

Of course, back here in the real world they *do* have to worry about
this stuff, and that's why it's important to quantify the problem.
It's not sufficient to say that "processes are slow and threads are
fast".  Processes on the target platform may well be slow relative to
other systems (and relative to threads).  But the question is: for the
problem being solved, how much overhead does process handling
represent relative to the total amount of overhead the solution itself
incurs?

For instance, if we're talking about addressing the problem of
distributing sorts across multiple CPUs, the amount of overhead
involved in doing disk activity while sorting could easily swamp, in
the typical case, the overhead involved in creating parallel processes
to do the sorts themselves.  And if that's the case, you may as well
gain the benefits of using full-fledged processes rather than deal
with the problems that come with the use of threads -- because the
gains to be found by using threads will be small in relative terms.

> > > At their core, threads are a context switching efficiency tweak.
> >
> > This is the heart of the matter.  Context switching is an operating
> > system problem, and *that* is where the optimization belongs.  Threads
> > exist in large part because operating system vendors didn't bother to
> > do a good job of optimizing process context switching and
> > creation/destruction.
> 
> But why would a database need a tons of context switches if it is
> not supposed to service loads to request simaltenously? If there are
> 50 concurrent connections, how much context switching overhead is
> involved regardless of amount of work done in a single connection? 
> Remeber that database state is maintened in shared memory. It does
> not take a context switch to access it.

If there are 50 concurrent connections with one process per
connection, then there are 50 database processes.  The context switch
overhead is incurred whenever the current process blocks (or exhausts
its time slice) and the OS activates a different process.  Since
database handling is generally rather I/O intensive as services go,
relatively few of those 50 processes are likely to be in a runnable
state, so I would expect the overall hit from context switching to be
rather low -- I'd expect the I/O subsystem to fall over well before
context switching became a real issue.

Of course, all of that is independent of whether or not the database
can handle a lot of simultaneous requests.

> > Under Linux, from what I've read, process creation/destruction and
> > context switching happens almost as fast as thread context switching
> > on other operating systems (Windows in particular, if I'm not
> > mistaken).
> 
> I hear solaris also has very heavy processes. But postgresql has
> other issues with solaris as well.

Yeah, I didn't want to mention Solaris because I haven't kept up with
it and thought that perhaps they had fixed this...


-- 
Kevin Brown					      kevin@sysexperts.com


Sat, 12 Apr 2003 03:54:52 -0700
Kevin Brown <kevin@sysexperts.com>
Re: Anyone working on better transaction locking?
On Saturday 12 April 2003 16:24, you wrote:
> A better answer is that a database engine that can handle lots of
> concurrent requests can also handle a smaller number, but not vice
> versa.  So it's clearly an advantage to have a database engine that
> can handle lots of concurrent requests because such an engine can be
> applied to a larger number of problems.  That is, of course, assuming
> that all other things are equal...
>
> There are situations in which a database would have to handle a lot of
> concurrent requests.  Handling ATM transactions over a large area is
> one such situation.  A database with current weather information might
> be another, if it is actively queried by clients all over the country.
> Acting as a mail store for a large organization is another.  And, of
> course, acting as a filesystem is definitely another.  :-)

Well, there is another aspect one should consider. Tuning a database engine 
for a specifiic workload is a hell of a job and shifting it to altogether 
other end of paradigm must be justified.

OK. Postgresql is not optimised to handle lots of concurrent connections, at 
least not much to allow one apache request handler to use a connection. Then 
middleware connection pooling like done in php might be a simpler solution to 
go rather than redoing the postgresql stuff. Because it works.

> This is true, but whether you choose to limit the use of threads to a
> few specific situations or use them throughout the database, the
> dangers and difficulties faced by the developers when using threads
> will be the same.

I do not agree. Let's say I put threading functions in posgresql that do not 
touch shared memory interface at all. They would be hell lot simpler to code 
and mainten than converting postgresql to one thread per connection model.

> Of course, back here in the real world they *do* have to worry about
> this stuff, and that's why it's important to quantify the problem.
> It's not sufficient to say that "processes are slow and threads are
> fast".  Processes on the target platform may well be slow relative to
> other systems (and relative to threads).  But the question is: for the
> problem being solved, how much overhead does process handling
> represent relative to the total amount of overhead the solution itself
> incurs?

That is correct. However it would be a fair assumption on part of postgresql 
developers that a process once setup does not have much of processing 
overhead involved as such, given the state of modern server class OS and 
hardware. So postgresql as it is, fits in that model. I mean it is fine that 
postgresql has heavy connections. Simpler solution is to pool them.

That gets me wondering. Has anybody ever benchmarked how much a database 
connection weighs in terms of memory/CPU/IO BW. for different databases on 
different platforms? Is postgresql really that slow?

 Shridhar


Sat, 12 Apr 2003 16:39:56 +0530
Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
Re: Anyone working on better transaction locking?

Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:

> But database is not webserver. It is not suppose to handle tons of concurrent 
> requests. That is a fundamental difference.

And in one fell swoop you've dismissed the entire OLTP database industry. 

Have you ever called a travel agent and had him or her look up a fare in the
airline database within seconds? Ever placed an order over the telephone? 
Ever used a busy database-backed web site?

On database-backed web sites, probably the main application for databases
today, almost certainly the main application for free software databases,
every web page request translates into at least one, probably several database
queries. 

All those database queries must complete within a limited time, measured in
milliseconds. When they complete another connection needs to be context
switched in and run again within milliseconds.

On a busy web site the database machine will have several processors and be
processing queries for several web pages simultaneously, but what really
matters is precisely the context switch time between one set of queries and
another.

The test I'm most interested in in the benchmarks effort is simply an index
lookup or update of a single record from a large table. How many thousands of
transactions per second is postgres going to be able to handle on the same
machine as mysql and oracle? How many hundreds of thousands of transactions
per second will they be able to handle on a 4 processor hyperthreaded machine
with a raid array striped across ten disks?

--
greg


12 Apr 2003 10:59:57 -0400
Greg Stark <gsstark@mit.edu>
Re: Anyone working on better transaction locking?
Scott Marlowe wrote:
> On Wed, 9 Apr 2003, Ron Peacetree wrote:
> 
> > "Andrew Sullivan" <andrew@libertyrms.info> wrote in message
> > news:20030409170926.GH2255@libertyrms.info...
> > > On Wed, Apr 09, 2003 at 05:41:06AM +0000, Ron Peacetree wrote:
> > > Nonsense.  You explicitly made the MVCC comparison with Oracle, and
> > > are asking for a "better" locking mechanism without providing any
> > > evidence that PostgreSQL's is bad.
> > >
> > Just because someone else's is "better" does not mean PostgreSQL's is
> > "bad", and I've never said such.  As I've said, I'll get back to Tom
> > and the list on this.
> 
> But you didn't identify HOW it was better.  I think that's the point 
> being made.

Oh, but he presented such detailed statistics to prove his case, didn't you 
see it?  :-)

> > > > Please see my posts with regards to ...
> > >
> > > I think your other posts were similar to the one which started this
> > > thread: full of mighty big pronouncements which turned out to depend
> > > on a bunch of not-so-tenable assumptions.
> > >
> > Hmmm.  Well, I don't think of algorithm analysis by the likes of
> > Knuth, Sedgewick, Gonnet, and Baeza-Yates as being "not so tenable
> > assumptions", but YMMV.  As for "mighty pronouncements", that also
> > seems a bit misleading since we are talking about quantifiable
> > programming and computer science issues, not unquantifiable things
> > like politics.
> 
> But the real truth is revealed when the rubber hits the pavement.  
> Remember that Linux Torvalds was roundly criticized for his choice of a 
> monolithic development model for his kernel, and was literally told that 
> his choice would restrict to "toy" status and that no commercial OS could 
> scale with a monolithic kernel.

Indeed.  I have the books from all of the above (when I studied databases 
under Gonnet, Baeza-Yates was his TA...).  And I have seen enough cases of the 
conglomeration of multiple algorithms not behaving the way a blind read of 
their books might suggest to refuse to blindly assume that things are so 
simple.

In the /real/ world, the dictates of flushing buffers to help ensure 
robustness can combine with having enough memory to virtually eliminate read 
I/O to substantially change the results from some simplistic O(f(n)) analysis.

Which is NOT to say that computational complexity is unimportant; what it 
indicates is that theoretical results are merely theoretical.  And may only 
represent a small part of what happens in practice.  The nonsense about radix 
sorts was a wonderful example; it would likely only be useful with PostgreSQL 
if you had some fantastical amount of memory that might not actually be able 
to be constructed within the confines of our solar system.

> There's no shortage of people with good ideas, just people with the skills 
> to implement those good ideas.  If you've got a patch to apply that's been 
> tested to show something is faster EVERYONE here wants to see it.
> 
> If you've got a theory, no matter how well backed up by academic research, 
> it's still just a theory.  Until someone writes to code to implement it, 
> the gains are theoretical, and many things that MIGHT help don't because 
> of the real world issues underlying your database, like I/O bandwidth or 
> CPU <-> memory bandwidth.

An unfortunate thing (to my mind) is that *genuinely novel* operating system 
research has pretty much disappeared.  All we see, these days, are rehashes of 
VMS, MVS, and Unix, along with some reimplementations of P-Code under monikers 
like "JVM", ".NET" or "Parrot."

There's good reason for it; if you build something that is much more than 95% 
indistinguishable from Unix, then you'll be left with the *enormous* projects 
of creating completely new infrastructure for compilers, data persistence 
("novel" would mean, to my mind, concepts different from files), program 
editors, and such.  But if it's 95% the same as Unix, then Emacs, GCC, CVS, 
PostgreSQL, and all sorts of "tool chain" are available to you.

What is unfortunate is that it would be nice to try out some things that are 
Very Different.  Unfortunately, it might take five years of slogging through 
recreating compilers and editors in order to get in about 6 months of "solid 
novel work."

Of course, if you don't plan to lift your finger to help make any of it 
happen, it's easy enough to "armchair quarterback" and suggest that someone 
else do all sorts of would-be "neat things."

> > > I'm sorry to be so cranky about this, but I get tired of having to
> > > defend one of my employer's core technologies from accusations based
> > > on half-truths and "everybody knows" assumptions.  For instance,
> > >
> > Again, "accusations" is a bit strong.  I thought the discussion was
> > about the technical merits and costs of various features and various
> > ways to implement them, particularly when this product must compete
> > for installed base with other solutions.  Being coldly realistic about
> > what a product's strengths and weaknesses are is, again, just good
> > business.  Sun Tzu's comment about knowing the enemy and yourself
> > seems appropriate here...

> No, you're wrong.  Postgresql doesn't have to compete.  It doesn't have to 
> win.  it doesn't need a marketing department.  All those things are nice, 
> and I'm glad if it does them, but doesn't HAVE TO.  Postgresql has to 
> work.  It does that well.

Having a bit more of a "marketing department" might be a nice thing; it could 
make it easier for people that would like to deploy PG to get the idea past 
the higher-ups that have a hard time listening to things that *don't* come 
from that department.

> > > > I'll mention thread support in passing,
> > >
> > > there's actually a FAQ item about thread support, because in the
> > > opinion of those who have looked at it, the cost is just not worth
> > > the benefit.  If you have evidence to the contrary (specific
> > > evidence, please, for this application), and have already read all
> > the
> > > previous discussion of the topic, perhaps people would be interested
> > in
> > > opening that debate again (though I have my doubts).
> > >
> > Zeus had a performance ceiling roughly 3x that of Apache when Zeus
> > supported threading as well as pre-forking and Apache only supported
> > pre forking.  The Apache folks now support both.  DB2, Oracle, and SQL
> > Server all use threads.  Etc, etc.
> 
> Yes, and if you configured your apache server to have 20 or 30 spare 
> servers, in the real world, it was nearly neck and neck to Zeus, but since 
> Zeus cost like $3,000 a copy, it is still cheaper to just overwhelm it 
> with more servers running apache than to use zeus.

All quite entertaining.  Andrew was perhaps trolling just a little bit there; 
our resident "algorithm expert" was certainly easily sucked into leaping down 
the path-too-much-trod.  Just as with choices of sorting algorithms, it's easy 
enough for there to be more to things than whatever the latest academic 
propaganda about threading is.

The VITAL point to be made about threading is that there is a tradeoff, and 
it's not the one that "armchair-quarterbacks-that-don't-write-code" likely 
think of.

--> Hand #1:  Implementing a threaded model would require a lot of work, and 
the *ACTUAL* expected benefits are unknown.

--> Hand #2:  So far, other *easier* optimizations have been providing 
significant speedups, requiring much less effort.

At some point in time, it might be that "doing threading" might become the 
strategy most expected to reap the most rewards for the least amount of 
programmer effort.  Until that time, it's not worth worrying about it.

> > That's an awful lot of very bright programmers and some serious $$
> > voting that threads are worth it.  
> 
> For THAT application.  for what a web server does, threads can be very 
> useful, even useful enough to put up with the problems created by running 
> threads on multiple threading libs on different OSes.  
> 
> Let me ask you, if Zeus scrams and crashes out, and it's installed 
> properly so it just comes right back up, how much data can you lose?
> 
> If Postgresql scrams and crashes out, how much data can you lost?

There's another possibility, namely that the "voting" may not have anything to 
do with threading being "best."  Instead, it may be a road to allow the 
largest software houses, that can afford to have enough programmers that can 
"do threading," to crush smaller competitors.  After all, threading offers 
daunting new opportunities for deadlocks, data overruns, and crashes; if only 
those with the most, best thread programmers can compete, that discourages 
others from even /trying/ to compete.
--
output = ("cbbrowne" "@ntlug.org")
http://www3.sympatico.ca/cbbrowne/sgml.html
"I visited  a company  that was doing  programming in BASIC  in Panama
City and I asked them if they resented that the BASIC keywords were in
English.   The answer  was:  ``Do  you resent  that  the keywords  for
control of actions in music are in Italian?''"  -- Kent M Pitman


Sat, 12 Apr 2003 11:00:51 -0400
cbbrowne@cbbrowne.com
Re: Anyone working on better transaction locking? 
On Fri, 2003-04-11 at 17:32, Kevin Brown wrote:
> The big overhead in the process model on most platforms is in creation
> and destruction of processes.  PostgreSQL has a relatively high
> connection startup cost.  But there are ways of dealing with this
> problem other than threading, namely the use of a connection caching
> middleware layer.

Furthermore, IIRC PostgreSQL's relatively slow connection creation time
has as much to do with other per-backend initialization work as it does
with the time to actually fork() a new backend. If there is interest in
optimizing backend startup time, my guess would be that there is plenty
of room for improvement without requiring the replacement of processes
with threads.

Cheers,

Neil


12 Apr 2003 15:29:38 -0400
Neil Conway <neilc@samurai.com>
Re: Anyone working on better transaction locking?
Neil Conway wrote:

> Furthermore, IIRC PostgreSQL's relatively slow connection creation time
> has as much to do with other per-backend initialization work as it does
> with the time to actually fork() a new backend. If there is interest in
> optimizing backend startup time, my guess would be that there is plenty
> of room for improvement without requiring the replacement of processes
> with threads.

I see there is a whole TODO Chapter devoted to the topic. There is the idea
of pre-forked and persistent backends. That would be very useful in an
environment where it's quite hard to use connection pooling. We are
currently working on a mail system for a free webmail. The mda (mail
delivery agent) written in C connects to the pg database to do some queries
everytime a new mail comes in. I didn't find a solution for connection
pooling yet.

About the TODO items, apache has a nice description of their accept()
serialization:
http://httpd.apache.org/docs-2.0/misc/perf-tuning.html

Perhaps this could be useful if someone decided to start implementing those
features.

Regards,
Michael Paesold


Sat, 12 Apr 2003 22:08:40 +0200
"Michael Paesold" <mpaesold@gmx.at>
Re: Anyone working on better transaction locking?
Greg Stark wrote:

>Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:
>
>  
>
>>But database is not webserver. It is not suppose to handle tons of concurrent 
>>requests. That is a fundamental difference.
>>    
>>
>
>And in one fell swoop you've dismissed the entire OLTP database industry. 
>
>Have you ever called a travel agent and had him or her look up a fare in the
>airline database within seconds? Ever placed an order over the telephone? 
>Ever used a busy database-backed web site?
>  
>
That situation is usually handled by means of a TP Monitor that keeps 
open database connections ( e.g, CICS + DB2 ).

I think there is some confusion between "many concurrent connections + 
short transactions" and "many connect / disconnect + short transactions" 
in some of this discussion.

OLTP systems typically fall into the first case - perhaps because their 
db products do not have fast connect / disconnect :-).  Postgresql plus 
some suitable middleware (e.g Php) will handle this configuration *with* 
its current transaction model.

I think you are actually talking about the connect / disconnect speed 
rather than the *transaction* model per se.

best wishes

Mark


Sun, 13 Apr 2003 10:45:30 +1200
Mark Kirkwood <markir@paradise.net.nz>
Re: Anyone working on better transaction locking?

Mark Kirkwood <markir@paradise.net.nz> writes:

> I think there is some confusion between "many concurrent connections + short
> transactions" and "many connect / disconnect + short transactions" in some of
> this discussion.

I was intended to clarify that but left it out. In fact I think that's
precisely one of the confusions that's obscuring things in this ongoing
debate.

Worrying about connection time is indeed a red herring. Most databases have
slow connection times so most database drivers implement some form of cached
connections. A lot of effort has gone into working around this particular
database design deficiency.

However even if you reuse existing database connections, you nonetheless are
still context switching between hundreds or potentially thousands of threads
of execution. The lighter-weight that context switch is, the faster it'll be
able to do that.

For a web site where all the queries are preparsed, all the data is cached in
ram, and all the queries involve quick single record lookups and updates, the
machine is often quite easily driven 100% cpu bound. 

It's tricky to evaluate the cost of the context switches because a big part of
the cost is simply the tlb flushes. Not only does a process context switch
involve swapping in memory maps and other housekeeping, but all future memory
accesses like the data copies that an OLTP system spends most of its time
doing are slowed down.

And the other question is how much memory does having many processes running
consume? Every page those processes are consuming that could have been shared
is a page that isn't being used for disk caching, and another page to pollute
the processor's cache.

So for example, I wonder how fast postgres would be if there were a thousand
connections open, all doing fast one-record index lookups as fast as they can.

People are going to say that would just be a poorly designed system, but I
think they're just not applying much foresight. Reasonably designed systems
easily need several hundred connections now, and future large systems will
undoubtedly need thousands.

Anyways, this is a long standing debate and the FAQ answer is mostly, we'll
find out when someone writes the code. Continuing to debate it isn't going to
be very productive. My only desire here is to see more people realize that
optimizing for tons of short transactions using data cached in ram is at least
as important as optimizing for big complex transactions on huge datasets.

--
greg


12 Apr 2003 19:21:36 -0400
Greg Stark <gsstark@mit.edu>
Re: Anyone working on better transaction locking?
Greg Stark <gsstark@mit.edu> writes:
> However even if you reuse existing database connections, you nonetheless are
> still context switching between hundreds or potentially thousands of threads
> of execution. The lighter-weight that context switch is, the faster it'll be
> able to do that.

> It's tricky to evaluate the cost of the context switches because a big part of
> the cost is simply the tlb flushes. Not only does a process context switch
> involve swapping in memory maps and other housekeeping, but all future memory
> accesses like the data copies that an OLTP system spends most of its time
> doing are slowed down.

So?  You're going to be paying those costs *anyway*, because most of the
process context swaps will be between the application server and the
database.  A process swap is a process swap, and if you are doing only
very short transactions, few of those swaps will be between database
contexts --- app to database to app will be the common pattern.  Unless
you'd like to integrate the client into the same address space as the
database, I do not see that there's an argument here that says multiple
threads in the database will be markedly faster than multiple processes.

			regards, tom lane


Sat, 12 Apr 2003 21:09:23 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Anyone working on better transaction locking? 
Greg Stark wrote:

>So for example, I wonder how fast postgres would be if there were a thousand
>connections open, all doing fast one-record index lookups as fast as they can.
>
Yes - some form of "connection reducing" middleare is probably needed at 
that point ( unless you have fairly highly spec'ed hardware )

>People are going to say that would just be a poorly designed system, but I
>think they're just not applying much foresight. Reasonably designed systems
>easily need several hundred connections now, and future large systems will
>undoubtedly need thousands.
>  
>
I guess the question could be reduced to : whether some form of TP 
Monitor functionality should be built into Postgresql? This *might* be a 
better approach - as there may be a limit to how much faster a Pg 
connection can get. By way of interest I notice that DB2 8.1 has a 
connection concentrator in it - probably for the very reason that we 
have been discussing...

Maybe there should be a TODO list item in the Pg "Exotic  Features" for 
connection pooling / concentrating ???

What do people think ?

Mark


Sun, 13 Apr 2003 16:00:22 +1200
Mark Kirkwood <markir@paradise.net.nz>
Re: Anyone working on better transaction locking?
Shridhar Daithankar wrote:
> > There are situations in which a database would have to handle a lot of
> > concurrent requests.  Handling ATM transactions over a large area is
> > one such situation.  A database with current weather information might
> > be another, if it is actively queried by clients all over the country.
> > Acting as a mail store for a large organization is another.  And, of
> > course, acting as a filesystem is definitely another.  :-)
> 
> Well, there is another aspect one should consider. Tuning a database
> engine for a specifiic workload is a hell of a job and shifting it
> to altogether other end of paradigm must be justified.

Certainly, but that justification comes from the problem being
solved.  If the nature of the problem demands tons of short
transactions (and as I said, a number of problems have such a
requirement), then tuning the database so that it can deal with it is
a requirement if that database is to be used at all.

Now, keep in mind that "tuning the database" here covers a *lot* of
ground and a lot of solutions, including connection-pooling
middleware.

> OK. Postgresql is not optimised to handle lots of concurrent
> connections, at least not much to allow one apache request handler
> to use a connection. Then middleware connection pooling like done in
> php might be a simpler solution to go rather than redoing the
> postgresql stuff. Because it works.

I completely agree.  In fact, I see little reason to change PG's
method of connection handling because I see little reason that a
general-purpose connection pooling frontend can't be developed.

Another method that could help is to prefork the postmaster.

> > This is true, but whether you choose to limit the use of threads to a
> > few specific situations or use them throughout the database, the
> > dangers and difficulties faced by the developers when using threads
> > will be the same.
> 
> I do not agree. Let's say I put threading functions in posgresql
> that do not touch shared memory interface at all. They would be hell
> lot simpler to code and mainten than converting postgresql to one
> thread per connection model.

I think you misunderstand what I'm saying.

There are two approaches we've been talking about thus far:

1.  One thread per connection.  In this instance, every thread shares
    exactly the same memory space.

2.  One process per connection, with each process able to create
    additional worker threads to handle things like concurrent sorts.
    In this instance, threads that belong to the same process all
    share the same memory space (including the SysV shared memory pool
    that the processes use to communicate with each other), but the
    only memory that *all* the threads will have in common is the SysV
    shared memory pool.

Now, the *scope* of the problems introduced by using threading is
different between the two approaches, but the *nature* of the problems
is the same: for any given process, the introduction of threads will
significantly complicate the debugging of memory corruption issues.
This problem will be there no matter which approach you use; the only
difference will be the scale.

And that's why you're probably better off with the third approach:

3.  One process per connection, with each process able to create
    additional worker subprocesses to handle things like concurrent
    sorts.  IPC between the subprocesses can be handled using a number
    of different mechanisms, perhaps including the already-used SysV
    shared memory pool.

The reason you're probably better off with this third approach is that
by the time you need the concurrency for sorting, etc., the amount of
time you'll spend on the actual process of sorting, etc. will be so
much larger than the amount of time it takes to create, manage, and
destroy the concurrent processes (even on systems that have extremely
heavyweight processes, like Solaris and Windows) that there will be no
discernable difference between using threads and using processes.  It
may take a few milliseconds to create, manage, and destroy the
subprocesses, but the amount of work to be done is likely to represent
at least a couple of *hundred* milliseconds for a concurrent approach
to be worth it at all.  And if that's the case, you may as well save
yourself the problems associated with using threads.

Even if you'd gain as much as a 10% speed improvement by using threads
to handle concurrent sorts and such instead of processes (an
improvement that is likely to be very difficult to achieve), I think
you're still going to be better off using processes.  To justify the
dangers of using threads, you'd need to see something like a factor of
two or more gain in overall performance, and I don't see how that's
going to be possible even on systems with very heavyweight processes.


I might add that the case where you're likely to gain significant
benefits from using either threads or subprocesses to handle
concurrent sorts is one in which you probably *won't* get many
concurrent connections...because if you're dealing with a lot of
concurrent connections (no matter how long-lived they may be), you're
probably *already* using all of the CPUs on the machine anyway.  The
situation where doing the concurrent subprocesses or subthreads will
help you is one where the connections in question are relatively
long-lived and are performing big, complex queries -- exactly the
situation in which threads won't help you at all relative to
subprocesses, because the amount of work to do on behalf of the
connection will dwarf (that is, be many orders of magnitude greater
than) the amount of time it takes to create, manage, and tear down a
process.

> > Of course, back here in the real world they *do* have to worry about
> > this stuff, and that's why it's important to quantify the problem.
> > It's not sufficient to say that "processes are slow and threads are
> > fast".  Processes on the target platform may well be slow relative to
> > other systems (and relative to threads).  But the question is: for the
> > problem being solved, how much overhead does process handling
> > represent relative to the total amount of overhead the solution itself
> > incurs?
> 
> That is correct. However it would be a fair assumption on part of
> postgresql developers that a process once setup does not have much
> of processing overhead involved as such, given the state of modern
> server class OS and hardware. So postgresql as it is, fits in that
> model. I mean it is fine that postgresql has heavy
> connections. Simpler solution is to pool them.

I'm in complete agreement here, and it's why I have very little faith
that a threaded approach to any of the concurrency problems will yield
enough benefits to justify the very significant drawbacks that a
threaded approach brings to the table.


-- 
Kevin Brown					      kevin@sysexperts.com


Sat, 12 Apr 2003 21:17:10 -0700
Kevin Brown <kevin@sysexperts.com>
Re: Anyone working on better transaction locking?
cbbrowne@cbbrowne.com wrote:
> > > That's an awful lot of very bright programmers and some serious $$
> > > voting that threads are worth it.  
> > 
> > For THAT application.  for what a web server does, threads can be very 
> > useful, even useful enough to put up with the problems created by running 
> > threads on multiple threading libs on different OSes.  
> > 
> > Let me ask you, if Zeus scrams and crashes out, and it's installed 
> > properly so it just comes right back up, how much data can you lose?
> > 
> > If Postgresql scrams and crashes out, how much data can you lost?
> 
> There's another possibility, namely that the "voting" may not have
> anything to do with threading being "best."  Instead, it may be a
> road to allow the largest software houses, that can afford to have
> enough programmers that can "do threading," to crush smaller
> competitors.  After all, threading offers daunting new opportunities
> for deadlocks, data overruns, and crashes; if only those with the
> most, best thread programmers can compete, that discourages others
> from even /trying/ to compete.

Yes, but any smart small software shop will realize that threading is
more about buzzword compliance than anything else.  In the real world
where things must get done, threading is just another tool to use when
it's appropriate.  And the only time it's appropriate is when the
amount of time it takes to create, manage, and tear down a process is
a very large fraction of the total amount of time it takes to do the
work.

If we're talking about databases, it's going to be very rare that
threads will *really* buy you any significant performance advantage
over concurrent processes + shared memory.


Buzzword compliance is nice but it doesn't get things done.  At the
end of the day, all that matters is whether or not the tool you chose
does the job you need it to do for as little money as possible.  I
hope in this lean economy that people are starting to realize this.


-- 
Kevin Brown					      kevin@sysexperts.com


Sat, 12 Apr 2003 21:31:28 -0700
Kevin Brown <kevin@sysexperts.com>
Re: Anyone working on better transaction locking?

Tom Lane <tgl@sss.pgh.pa.us> writes:

> So?  You're going to be paying those costs *anyway*, because most of the
> process context swaps will be between the application server and the
> database.  

Separating the database and application onto dedicated machines is normally
the first major optimization busy sites do when they discover that having the
two on the same machine never scales well.

--
greg


13 Apr 2003 01:20:26 -0400
Greg Stark <gsstark@mit.edu>
Re: Anyone working on better transaction locking?
On Sunday 13 April 2003 09:47, you wrote:
> Even if you'd gain as much as a 10% speed improvement by using threads
> to handle concurrent sorts and such instead of processes (an
> improvement that is likely to be very difficult to achieve), I think
> you're still going to be better off using processes.  To justify the
> dangers of using threads, you'd need to see something like a factor of
> two or more gain in overall performance, and I don't see how that's
> going to be possible even on systems with very heavyweight processes.

I couldn't agree more. 

There is just a corner case to justify threads. Looking around, it would be a 
fair assumption that on any platforms threads are at least as fast as 
processes. So using threads it is guarenteed that "sub-work" will be lot more 
faster.

Of course that does not justify threads even in 5% of cases. So again, no 
reason to use threads for sort etc. However the subprocesses used should be 
simple enough. A process as heavy as a full database connection might not be 
too good.

 Shridhar


Sun, 13 Apr 2003 11:59:59 +0530
Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
Re: Anyone working on better transaction locking?
On Saturday 12 April 2003 20:29, you wrote:
> Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:
> > But database is not webserver. It is not suppose to handle tons of
> > concurrent requests. That is a fundamental difference.
>
> And in one fell swoop you've dismissed the entire OLTP database industry.
>
> Have you ever called a travel agent and had him or her look up a fare in
> the airline database within seconds? Ever placed an order over the
> telephone? Ever used a busy database-backed web site?

Well, I was involved in designing a database solution for a telco for their 
support system. That was a fairly big database, aroung 600GB.  There was a 
response time limit as well. Though it was not millisecond.

Though project did not go thr. for non-technical reasons, the bechmark we did 
with postgresql/mysql/oracle left an impression that any of mysql/postgresql 
would handle that kind of load with server clustering. Furthermore postgresql 
would have been the choice given the mature SQL capabilities it has. Even 
with oracle, the database had to be clustered to keep the cost low enough.

> On database-backed web sites, probably the main application for databases
> today, almost certainly the main application for free software databases,
> every web page request translates into at least one, probably several
> database queries.

Queries != connection. We are talking about reducing number of connections 
required, not number of queries sent across.

> All those database queries must complete within a limited time, measured in
> milliseconds. When they complete another connection needs to be context
> switched in and run again within milliseconds.
>
> On a busy web site the database machine will have several processors and be
> processing queries for several web pages simultaneously, but what really
> matters is precisely the context switch time between one set of queries and
> another.

Well, If the application is split between application server and database 
server, I would rather put a cluster of low end database machines and have an 
data consolidating layer in middleware. That is cheaper than big iron 
database machine and can be expanded as required.

However this would not work in all cases unless you are able to partition the 
data. Otherwise you need a database that can have single database image 
across machines. 

If and when postgresql moves to mmap based model, postgresql running on mosix 
should be able to do it. Using PITR mechanism, it would get clustering 
abilities as well. This is been discussed before.

Right now, postgresql does not have any of these capabilities. So using 
application level data consolidation is the only answer

> The test I'm most interested in in the benchmarks effort is simply an index
> lookup or update of a single record from a large table. How many thousands
> of transactions per second is postgres going to be able to handle on the
> same machine as mysql and oracle? How many hundreds of thousands of
> transactions per second will they be able to handle on a 4 processor
> hyperthreaded machine with a raid array striped across ten disks?

I did the same test on a 4 way xeon machine with 4GB of RAM and 40GB of data. 
Both mysql and postgresql did lookups at approximately 80% speed of oracle. 
IIRC they were doing 600 queries per second but I could be off. It was more 
than 6 months ago.

 However testing number of clients was not a criteria. We only tested with 10 
concurrent clients. Mysql freezes at high database loads and high number of 
concurrent connection. Postgresql has tendency to hold the load muh longer. 
So more the number of connections, faster will be response time. That should 
be a fairly flat curve for upto 100 concurrent connection. Good enough 
hardware assumed.

 Shridhar


Sun, 13 Apr 2003 12:13:16 +0530
Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
Re: Anyone working on better transaction locking?
""Dann Corbit"" <DCorbit@connx.com> wrote in message
news:D90A5A6C612A39408103E6ECDD77B829408ACB@voyager.corporate.connx.co
m...
> I would be interested to see your algorithm that performs a counting
> or radix sort on 320 bit bins and that works in the general case
> without using extra space.
>
Since I can't be sure of my audience, I'm writing for the widest
reasonably possible audience, and I apologize in advance to those who
know most of this.

First, let's use some examples for the data to be sorted.  I'll
consider two cases, 1= each 320b entity is a row, and 2= the original
example where there are 3 fields per row; the first being 320b, with
each of the others being 160b.
For case 1, a 2GB file has 2^34/320= 53,687,091 records.  Let's call
it 50x10^6 to keep things simple.  Case 2 is 26,843,545 records.
Let's call that 25x10^6.

Let's examine the non space issues first, then explore space
requirements.

The most space and time efficient comparison based sorting algorithm
(median-of-three partitioning Qsort with Isort used when almost
sorted) will on average execute in ~(32/3)*n*ln(n) tyme units
(~9.455x10^9 for case 1; ~4.543x10^9 for case 2).  The use of "tyme"
here is not a typo or a spice.  It's a necessary abstraction because
real time units are very HW and SW context dependant.

Now let's consider a radix address calculation sort based approach.
The best implementation does a  Rsort on the MSBs of the data to the
point of guaranteeing that the data is "almost sorted" then uses Isort
as a "finishing touch".  The Rsort part of the algorithm will execute
on average in 11*p*n-n+O(p*m) tyme units, where p is the number of
passes used and m is the number of unique states in the radix (2^16
for a 16b radix in which we can't prune any possibilities).

For a 320b data value that we know nothing about, we must consider all
possible values for any length radix we use.  The function
11*p*n-n+O(p*m) has a minimum when we use a 16b radix:
11*20*50x10^6-50x10^6+O(20*2^16)= ~10.951x10^9 tyme units.  If we use
enough Rsort passes to get the data "almost ordered" and then use
Isort to finish things we can do a bit better.  Since Qsort needs a
average of ~9.455x10^9 tyme units to sort the same data.  It's not
clear that the effort to code Rsort is worth it for this case.

However, look what happens if we can assume that the number of
distinct key values is small enough that we can sort the data in a
relatively few number of passes: if the 320b key has <= 2^16 distinct
values, we can sort them using one pass in  ~500x10^6+O(2^16) tyme
units.  Even noting that O(2^16) has just become much larger since we
are now manipulating key values rather than bits, this is stupendously
better than Qsort and there should be no question that Rsort is worth
considering in such cases.  FTR, this shows that my naive (AKA "I
didn't do the math") instinct was wrong.  This technique is FAR better
than I originally was estimating under these circumstances.

As long as we can store a "map" of unique keys in a relatively small
amount of space and use relatively few radix passes, this technique is
far better than comparison based sorting methods.  Said mapping is
easily derived from table indexes, and some DB systems (DB2 and
Oracle) will generate and save this data for you as part of their set
of optimization tweaks.  Even if none of that is true, an extra full
scan of a memory resident data set to derive this information is cheap
if the gamble pays off and you find out that the number of distinct
key values is a small subset of the potential universe of said.

Now let's consider that "extra space".  At the least Qsort is going to
need extra space proportional to lg(n) for its call tree (explicit or
system level to support recursion).  If we are using a 16b radix, the
extra space in question in 64KB.  In either case, it's in the noise.

 If we are using a mapping of <= 2^16 keys, the extra space in
question is ~2.5MB.  Considering we are sorting an ~2GB file and the
incredible improvement in running time, this seems to be a very
worthwhile investment.

This would be worth adding support for to PostgreSQL.


Sun, 13 Apr 2003 10:59:38 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: No merge sort?

"Ron Peacetree" <rjpeace@earthlink.net> wrote in message
news:uqbma.22946$4P1.2023021@newsread2.prod.itd.earthlink.net...
>  If we are using a mapping of <= 2^16 keys, the extra space in
> question is ~2.5MB.
EDIT:  add "...for 320b keys"


Sun, 13 Apr 2003 11:17:47 GMT
"Ron Peacetree" <rjpeace@earthlink.net>
Re: No merge sort?
Greg Stark wrote:
> 
> Tom Lane <tgl@sss.pgh.pa.us> writes:
> 
> > So?  You're going to be paying those costs *anyway*, because most of the
> > process context swaps will be between the application server and the
> > database.
> 
> Separating the database and application onto dedicated machines is normally
> the first major optimization busy sites do when they discover that having the
> two on the same machine never scales well.

If there is enough of an "application" to separate it ... :-)

People talk about "database backed websites" when all they need is
thousands of single index lookups. Can someone give me a real world
example of such a website? And if so, what's wrong with using ndbm/gdbm? 

All these hypothetical arguments based on "the little test I made" don't
lead to anything. I can create such tests that push the CPU load to 20
or more or get all the drive LED's nervous in no time. They don't tell
anything, that's the problem. They are purely synthetic. They hammer a
few different simple queries in a totally unrealistic, hysteric fashion
against a database and are called benchmarks. There are absolutely no
means of consistency checks built into the tests and if one really runs
checksum tests after 100 concurrent clients hammered this other super
fast superior sql database for 10 minutes people wonder how inconsistent
it can become after 10 minutes ... without a single error message.

Anyone ever thought about a reference implementation of TPC-W? 


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Sun, 13 Apr 2003 10:00:40 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: Anyone working on better transaction locking?
Ron Peacetree wrote:
> 
> "Jan Wieck" <JanWieck@Yahoo.com> wrote in message
> > And you are comparing what? Just pure features and/or
> > performance, or total cost of ownership for your
> > particular case?
> >
> Technical Analysis and Business Analysis are two separate, and equally
> necessary, activities.  However, before one can accurately measure
> things like Total Cost of Ownership, one needs to have accurately and
> sufficiently characterized what will be owned and one's choices as to
> what could be owned...

Okay, so you are doing the technical analysis for now.

> [...]
> However, a correctly done Technical Analysis =should= be reasonably
> portable since among other things you don't want to have to start all
> over if your company's business or business model changes.  Clearly
> Business Analysis is very context dependant.

However, doing a technical analysis correctly does not mean to blindly
ask about all the advanced features for each subsystem. The technical
analysis is part of the entire evaluation process. That process starts
with collecting the business requirements and continues with specifying
the technical requirements based on that. Not the other way round,
because technology should not drive, it should serve (unless the
technology in question is your business).

Possible changes in business model might slightly change the technical
requirements in the future, so an appropriate security margin is added.
But the attempt to build canned technical analysis for later reuse is
what leads to the worst solutions. How good is a decision based on 2
year old technical information?

Now all the possible candidates get compared against these
"requirements". That candidate "O" has the super duper buzzword feature
"XYZ" candidate "P" does not have is of very little importance unless
"XYZ" is somewhere on the requirements list. The availability of that
extra feature will not result in any gain here.

In an earlier eMail you pointed out that 2 phase commit is essential for
SMP/distributed applications. I know well what a distributed application
is, but what in the world is an SMP application?


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Sun, 13 Apr 2003 11:16:55 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: Anyone working on better transaction locking?
Michael Paesold wrote:
> I see there is a whole TODO Chapter devoted to the topic. There is the idea
> of pre-forked and persistent backends. That would be very useful in an
> environment where it's quite hard to use connection pooling. We are
> currently working on a mail system for a free webmail. The mda (mail
> delivery agent) written in C connects to the pg database to do some queries
> everytime a new mail comes in. I didn't find a solution for connection
> pooling yet.

I am still playing with the model of reusing connections in a
transparent fashion with a pool manager that uses SCM_RIGHTS messages
over UNIX domain socketpairs. I will scribble down some concept anytime
soon. This will include some more advantages than pure startup cost
reduction, okay?


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Sun, 13 Apr 2003 11:25:54 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: Anyone working on better transaction locking?
[ Warning, topic drift ahead ]

Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:
> However this would not work in all cases unless you are able to partition the
> data. Otherwise you need a database that can have single database image 
> across machines. 

> If and when postgresql moves to mmap based model, postgresql running on mosix
> should be able to do it.

In a thread that's been criticizing handwavy arguments for fundamental
redesigns offering dubious performance improvements, you should know
better than to say such a thing ;-)

I don't believe that such a design would work at all, much less have
any confidence that it would give acceptable performance.  Would mosix
shared memory support TAS mutexes?   I don't see how it could, really.
That leaves you needing to come up with some other low-level lock
mechanism and get it to have adequate performance across CPUs.  Even
after you've got the locking to work, what would performance be like?
Postgres is built on the assumption of cheap access to shared data
structures (lock manager, buffer manager, etc) and I don't think this'll
qualify as cheap.

			regards, tom lane


Sun, 13 Apr 2003 11:45:58 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Anyone working on better transaction locking? 
Tom Lane kirjutas P, 13.04.2003 kell 18:45:
> [ Warning, topic drift ahead ]
> 
> Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:
> > However this would not work in all cases unless you are able to partition the
> > data. Otherwise you need a database that can have single database image 
> > across machines. 
> 
> > If and when postgresql moves to mmap based model, postgresql running on mosix
> > should be able to do it.
> 
> In a thread that's been criticizing handwavy arguments for fundamental
> redesigns offering dubious performance improvements, you should know
> better than to say such a thing ;-)
> 
> I don't believe that such a design would work at all, much less have
> any confidence that it would give acceptable performance.  Would mosix
> shared memory support TAS mutexes?   I don't see how it could, really.
> That leaves you needing to come up with some other low-level lock
> mechanism and get it to have adequate performance across CPUs.

Does anybody have any idea how Oracle RAC does it ?

They seem to need to syncronize a lot (at least locks and data cache
coherency) across different machines.

> Even
> after you've got the locking to work, what would performance be like?
> Postgres is built on the assumption of cheap access to shared data
> structures (lock manager, buffer manager, etc) and I don't think this'll
> qualify as cheap.

[OT]

I vaguely remember some messages about getting PG to work well on NUMA
computers, which by definition should have non-uniformly cheap access to
shared data structures. 

They must have faced similar problems.

-------------
Hannu


13 Apr 2003 22:43:06 +0300
Hannu Krosing <hannu@tm.ee>
Re: Anyone working on better transaction locking?
Hannu Krosing <hannu@tm.ee> writes:
> I vaguely remember some messages about getting PG to work well on NUMA
> computers, which by definition should have non-uniformly cheap access to
> shared data structures. 

My recollection of the thread is that we didn't know how to do it ;-)

			regards, tom lane


Sun, 13 Apr 2003 23:37:40 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Anyone working on better transaction locking? 
Mark Kirkwood wrote:

> Maybe there should be a TODO list item in the Pg "Exotic  Features" 
> for connection pooling / concentrating ???
>
>
Oh dear, there already is... (under "Startup Time"), I just missed it :-(


Mark


Mon, 14 Apr 2003 19:19:54 +1200
Mark Kirkwood <markir@paradise.net.nz>
Re: Anyone working on better transaction locking?
Ron Peacetree wrote:
> ""Dann Corbit"" <DCorbit@connx.com> wrote in message
> news:D90A5A6C612A39408103E6ECDD77B829408ACB@voyager.corporate.connx.co
> m...
> > > From: Ron Peacetree [mailto:rjpeace@earthlink.net]=20
> > > Sent: Wednesday, April 09, 2003 12:38 PM
> > > You only need as many bins as you have unique key values=20
> > > silly ;-) Remember, at its core Radix sort is just a=20
> > > distribution counting sort (that's the name I learned for the=20
> > > general technique).  The simplest implementation uses bits as=20
> > > the atomic unit, but there's nothing saying you have to...=20=20
> > > In a DB, we know all the values of the fields we currently=20
> > > have stored in the DB.  We can take advantage of that.
> >
> > By what magic do we know this?  If a database knew a-priori what all
> the
> > distinct values were, it would indeed be excellent magic.
> For any table already in the DB, this is self evident.  If you are
> talking about sorting data =before= it gets put into the DB (say for a
> load), then things are different, and the best you can do is used
> comparison based methods (and probably some reasonably sophisticated
> external merging routines in addition if the data set to be sorted and
> loaded is big enough).  The original question as I understood it was
> about a sort as part of a query.  That means everything to be sorted
> is in the DB, and we can take advantage of what we know.
> 
> > With 80 bytes, you have 2^320 possible values.  There is no way
> > around that.  If you are going to count them or use them as a
> > radix, you will have to classify them.  The only way you will
> > know how many unique values you have in
> > "Company+Division" is to ...
> > Either sort them or by some means discover all that are distinct

> Ummm, Indexes, particularly Primary Key Indexes, anyone?  Finding the
> unique values in an index should be perceived as trivial... ...and you
> often have the index in memory for other reasons already.

But this does not remove the fact that a radix sort on an 80 byte field
requires O(2^320) space, that is, O(N), where N is the size of the state
space of the key...

Perhaps you need to reread Gonnet; it tells you that...
--
output = reverse("moc.enworbbc@" "enworbbc")
http://www3.sympatico.ca/cbbrowne/multiplexor.html
Rules of  the Evil  Overlord #28. "My  pet monster  will be kept  in a
secure cage  from which it  cannot escape and  into which I  could not
accidentally stumble." <http://www.eviloverlord.com/>


Mon, 14 Apr 2003 12:52:57 -0400
cbbrowne@cbbrowne.com
Re: No merge sort? 
On Sun, Apr 13, 2003 at 10:43:06PM +0300, Hannu Krosing wrote:
> Does anybody have any idea how Oracle RAC does it ?

According to some marketing literature I saw, it was licensed
technology; it was supposed to be related to VMS.  Moredetails I
don't have, though.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Mon, 14 Apr 2003 14:48:57 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: Anyone working on better transaction locking?
> On Sun, Apr 13, 2003 at 10:43:06PM +0300, Hannu Krosing wrote:
> > Does anybody have any idea how Oracle RAC does it ?
> 
> According to some marketing literature I saw, it was licensed
> technology; it was supposed to be related to VMS.  More details I
> don't have, though.

That would fit perfectly with it having been part of the purchase of Rdb
from Digital...  There might well be some "harvestable" Rdb information
out there somewhere...

http://citeseer.nj.nec.com/lomet92private.html

(note that (Rdb != /RDB) && (Rdb != Rand RDB); there's an unfortunate
preponderance of "things called RDB")
--
(reverse (concatenate 'string "moc.enworbbc@" "enworbbc"))
http://www.ntlug.org/~cbbrowne/rdbms.html
Rules of the Evil Overlord #15. "I will never employ any device with a
digital  countdown.   If I  find  that  such  a device  is  absolutely
unavoidable, I  will set it to  activate when the  counter reaches 117
and the hero is just putting his plan into operation."
<http://www.eviloverlord.com/>


Mon, 14 Apr 2003 16:56:45 -0400
cbbrowne@cbbrowne.com
Re: Anyone working on better transaction locking? 
Several people have asked if we are losing momentum.  Specifically, they
are concerned about Red Hat dropping their Red Hat Database and instead
distributing PostgreSQL as part of Red Hat Enterprise Server, and they
are concerned about recent press articles about MySQL.

Let me address these.  First, the Red Hat change probably has a lot to
do with Oracle's relationship with Red Hat, and very little to do with
PostgreSQL.  Their pullback is similar to Great Bridge's closing, except
that Red Hat's database group is still around, so we aren't losing Tom
Lane or Patrick MacDonald (who is completing our PITR work for 7.4).

As far as MySQL, they have a company to push articles to the press, and
many writers just dress them up and print them --- you can tell them
because the pushed ones mention only MySQL, while the non=pushed ones
mention MySQL and PostgreSQL.

I have been around the globe enough to know that PostgreSQL is well on
track.  Our user base is growing, we have Win32 and PITR ready for 7.4
(and each had some commercial funding to make them happen.)  Recently, I
have also been fielding questions from several companies that want to
hire PostgreSQL developers to work for the community.

But most importantly, there is mind share.  I get _very_ few questions
about MySQL anymore, and when the database topic comes up on Slashdot,
the MySQL guys usually end up looking foolish for using MySQL.  And my
recent trip to Toronto (who's details I have shared with core but can
not discuss) left no doubt in my mind that PostgreSQL is moving forward
at a rapid rate.

And, I have 1.5k emails to read after a one week trip.  :-)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073


Mon, 14 Apr 2003 18:37:57 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Are we losing momentum?
With the feature set that Postgres has, it isn't going to lose momentum. 

It is lacking in some areas that are slowly being addressed. 

If they weren't being addressed, THEN, postgres would lose momentum.

We're fortunate to have good volunteers and the private donations of
companies as well.

Bruce Momjian wrote:
> 
> Several people have asked if we are losing momentum.  Specifically, they
> are concerned about Red Hat dropping their Red Hat Database and instead
> distributing PostgreSQL as part of Red Hat Enterprise Server, and they
> are concerned about recent press articles about MySQL.
> 
> Let me address these.  First, the Red Hat change probably has a lot to
> do with Oracle's relationship with Red Hat, and very little to do with
> PostgreSQL.  Their pullback is similar to Great Bridge's closing, except
> that Red Hat's database group is still around, so we aren't losing Tom
> Lane or Patrick MacDonald (who is completing our PITR work for 7.4).
> 
> As far as MySQL, they have a company to push articles to the press, and
> many writers just dress them up and print them --- you can tell them
> because the pushed ones mention only MySQL, while the non=pushed ones
> mention MySQL and PostgreSQL.
> 
> I have been around the globe enough to know that PostgreSQL is well on
> track.  Our user base is growing, we have Win32 and PITR ready for 7.4
> (and each had some commercial funding to make them happen.)  Recently, I
> have also been fielding questions from several companies that want to
> hire PostgreSQL developers to work for the community.
> 
> But most importantly, there is mind share.  I get _very_ few questions
> about MySQL anymore, and when the database topic comes up on Slashdot,
> the MySQL guys usually end up looking foolish for using MySQL.  And my
> recent trip to Toronto (who's details I have shared with core but can
> not discuss) left no doubt in my mind that PostgreSQL is moving forward
> at a rapid rate.
> 
> And, I have 1.5k emails to read after a one week trip.  :-)
> 
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/docs/faqs/FAQ.html


Mon, 14 Apr 2003 16:41:28 -0700
Dennis Gearon <gearond@cvc.net>
Re: Are we losing momentum?
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Several people have asked if we are losing momentum.

I don't think we are losing momentum considering the project in
isolation --- things seem to be moving as well as they ever have,
if not better.

But I do sometimes worry that we are losing the mindshare war.
We might be growing fine, but if we're growing slower than MySQL is,
we've got a problem.  I was just in the local Barnes & Noble store
yesterday, and could not help but notice how many books had "MySQL" in
the title.  I didn't notice a single Postgres title (though I did not
look hard, since I was just passing through the computer area).

Mindshare eventually translates into results, if only because it
means that capable developers will gravitate there instead of here.
So we need to worry about it.

There isn't anyone presently willing to spend real money and effort on
marketing PG (as you say, Red Hat won't, for reasons that have nothing
to do with the merits of the product).  That means that MySQL's
marketeers have a free hand to do things like boast about features that
might materialize in a year or so :-(

I don't know what we can do about it, other than maybe push harder to
get some more PG titles into O'Reilly's catalog ... that would help
narrow the bookshelf gap a little, at least.  Any wannabee authors
out there?  (And Bruce, your book is due for a second edition...)

			regards, tom lane


Mon, 14 Apr 2003 19:54:27 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Are we losing momentum? 
On Mon, 14 Apr 2003, Tom Lane wrote:

> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Several people have asked if we are losing momentum.
> 
> I don't think we are losing momentum considering the project in
> isolation --- things seem to be moving as well as they ever have,
> if not better.

I agree. I am surprised at the pace at which new features are added,
considering the relatively small number of people working on the project.

> 
> But I do sometimes worry that we are losing the mindshare war.
> We might be growing fine, but if we're growing slower than MySQL is,
> we've got a problem.  I was just in the local Barnes & Noble store
> yesterday, and could not help but notice how many books had "MySQL" in
> the title.  I didn't notice a single Postgres title (though I did not
> look hard, since I was just passing through the computer area).

I've considered this at length. I put some ideas together in December and
sent it off to the advocacy list. Most/all were not implemented -- not
least because I didn't do anything I said I would :-). But, some of the
most important things, such as a proper media kit, quotes for journos,
press contacts with authority to give fast/correct answers really need to
be implemented.

As for why MySQL has *significantly* more market share: there's not a lot
we can match them on. They have significant financial backing -- important
if you're an IT manager who actually knows very little about the technical
merit of the product. It has close ties to a *very* widely deployed
scripting language (PHP). MySQL AB employs marketing and 'advocacy' staff,
who attend conferences all over the world, speak several languages, and
have a fairly good understanding of the industry, open source, databases,
etc. They have infrastructure: tech support, on site support,
consultancy.

MySQL AB promotes MySQL as a high performance database, easy to use,
uncomplicated, with features implemented in a way which is syntactically
convenient -- not 'complicated' like Oracle, DB2 or Postgres.

Its hard to argue against that. At a *technical* conference I recently
spoke at, I was criticised for delivering a talk which was too advanced
and didn't explain Postgres for MySQL users. During a lecture series at a
university, I was criticised for not discussing Oracle instead of Postgres
-- students told me that Oracle will make them money and Postgres wont.

Regardless, I'm still of the opinion that if you build it, they will come
-- particularly costly features like replication, PITR, etc. But maybe
that is what the BSDs say about Linux?

Gavin


Tue, 15 Apr 2003 10:18:59 +1000 (EST)
Gavin Sherry <swm@linuxworld.com.au>
Re: Are we losing momentum? 
Gavin Sherry wrote:
> During a lecture series at a university, I was criticised for not
> discussing Oracle instead of Postgres -- students told me that
> Oracle will make them money and Postgres wont.

Their impressions are probably based on reality as it was a couple of
years ago before the U.S. economy came crashing down.

But today?  Companies are trying to figure out how to do things
cheaper, and there are a lot of situations for which Postgres is a
good fit but for which MySQL is a bad fit -- if it'll fit at all.


I seriously think the native Win32 port of Postgres will make a big
difference, because it'll be a SQL Server killer.  Especially if it
comes with a nice administrative GUI.  :-)



-- 
Kevin Brown					      kevin@sysexperts.com


Mon, 14 Apr 2003 17:30:27 -0700
Kevin Brown <kevin@sysexperts.com>
Re: Are we losing momentum?
On Mon, 14 Apr 2003, Kevin Brown wrote:

> Gavin Sherry wrote:
> > During a lecture series at a university, I was criticised for not
> > discussing Oracle instead of Postgres -- students told me that
> > Oracle will make them money and Postgres wont.
> 
> Their impressions are probably based on reality as it was a couple of
> years ago before the U.S. economy came crashing down.
> 
> But today?  Companies are trying to figure out how to do things
> cheaper, and there are a lot of situations for which Postgres is a
> good fit but for which MySQL is a bad fit -- if it'll fit at all.
> 
> 
> I seriously think the native Win32 port of Postgres will make a big
> difference, because it'll be a SQL Server killer.  Especially if it
> comes with a nice administrative GUI.  :-)

I've been thinking about this too. Addressing Tom's point: any one with
Windows experience, interested in the native port and willing to write a
Windows book would probably do a lot for the project. For one, I would be
willing to help write parts which were not Windows specific -- as I
haven't used that system in some time :-).

Gavin


Tue, 15 Apr 2003 10:38:05 +1000 (EST)
Gavin Sherry <swm@linuxworld.com.au>
Re: Are we losing momentum?


--On Monday, April 14, 2003 19:54:27 -0400 Tom Lane <tgl@sss.pgh.pa.us> 
wrote:

> Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Several people have asked if we are losing momentum.
>
> I don't think we are losing momentum considering the project in
> isolation --- things seem to be moving as well as they ever have,
> if not better.
>
> But I do sometimes worry that we are losing the mindshare war.
> We might be growing fine, but if we're growing slower than MySQL is,
> we've got a problem.  I was just in the local Barnes & Noble store
> yesterday, and could not help but notice how many books had "MySQL" in
> the title.  I didn't notice a single Postgres title (though I did not
> look hard, since I was just passing through the computer area).
I was in the Local MicroCenter, and found 3 PG titles, in addition to 
Bruce's.

This is MUCH better than a year ago, when there were NONE.

Agreed, that MySQL, has a bigger shelf space.

I did all 3 authors a favor and bought copies.

LER


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Mon, 14 Apr 2003 20:37:00 -0500
Larry Rosenman <ler@lerctr.org>
Re: Are we losing momentum? 
Gretings!

[2003-04-14 19:54] Tom Lane said:
| Bruce Momjian <pgman@candle.pha.pa.us> writes:
| > Several people have asked if we are losing momentum.

| I don't know what we can do about it, other than maybe push harder to
| get some more PG titles into O'Reilly's catalog ... that would help
| narrow the bookshelf gap a little, at least.  Any wannabee authors
| out there?  (And Bruce, your book is due for a second edition...)

  I've wanted to pipe up in a few of these "popularity" 
discussions in the past.  Seeing how I can't make time to
participate in any other meaningful capacity, I'll share
my thoughts on _why_ mysql has the mindshare.


  Applications, specifically applications that _use_ mysql.


  A quick search over at freshmeat returns 1044 results for 
"mysql" and 260 for "postgresql".  Before this turns into a 
cause/effect discussion, I want to state up front that the 
real "effect" of this is that someone is 4 times as likely to 
download an application that uses mysql.  Sure, many are 
"trivial" applications, but I posit that it is _specifically_ 
these "trivial" applications that inoculate the uninitiated 
with the belief that mysql is suitable for use in real, albeit
trivial applications.  Additionally, it these rudimentary 
applications that will be studied by many as the way to write 
a database application.

  It is all good and well that postgres /can/ do, but until
the application developers see that those features are
valuable enough to forgo mysql support, they'll write the 
application to support whatever database is most likely to 
_already_ be installed, which will be mysql.  Granted, 
many developers will also try to support multiple dbs via
the language's db api, but this leaves the less-supported
dbs in an even worse position; being relegated to an
"might work with XXX database".  When anxious user learns
that "might" currently means "doesn't," the second-string
database looks even worse in the eyes of the user.

  How to solve this problem?  This is the hard part, but
luckily ISTM that there are a few ways to succeed.  Neither
of which involves marketing or writing books.

  1) become active in the "also supports postgres" projects,
     and add features that are made available _because_ of
     postgres' superiority.  Eventually, market pressure
     for the cool feature(s) will lead users to choose
     postgres, and mysql could be relegated to the "also
     runs on mysql, with limited featureset"
  2) take a popular project that uses mysql, fork it, and
     add features that can only be implemented using posgres.
  3) release that super-cool code that you've been hacking
     on for years, especially if it is a "trivial" app.
  4) convince your employer that it would be _beneficial_ to
     them to release, as open source, the internal app(s) you've 
     developed, using postgres-specific features.  (This is 
     about all I can claim to be doing at this point in my 
     indentured servitude, and I can't say I'm doing a good
     job... :-/)

  I'm sure this idea is not original, but I'm also sure that
it _is_ the answer to gaining market^Wmindshare in this
database market.

  (I must apologize in advance, that I might not have time
to even follow this thread, in fact, I hope that instead of
replying to this, the potential respondent might consider
helping to increase the number of apps that require postgres
:-)

wishing-I-could-contribute-more-ly yours,
  brent

-- 
"Develop your talent, man, and leave the world something. Records are 
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing."  -- Duane Allman


Mon, 14 Apr 2003 21:52:17 -0400
Brent Verner <brent@rcfile.org>
Re: Are we losing momentum?
>   1) become active in the "also supports postgres" projects,
>      and add features that are made available _because_ of
>      postgres' superiority.  Eventually, market pressure
>      for the cool feature(s) will lead users to choose
>      postgres, and mysql could be relegated to the "also
>      runs on mysql, with limited featureset"

Take, for example, phpPgAdmin.  It was originally forked from phpMyAdmin, but we've just done a complete rewrite (because phpMyAdmin was written my mysql/php weenies who couldn't code nicely to save their lives...).

However, it's me doing 99% of the coding, Rob doing advocacy and a heap of people who send in translations.  Translations are very nice, but I so rarely get actual code contributions.

phpMyAdmin even implements it's OWN comment and foreign key feature!!

Chris

Tue, 15 Apr 2003 10:15:03 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: Are we losing momentum?
Kevin, without the "e", wrote...
> I seriously think the native Win32 port of Postgres will make a big
> difference, because it'll be a SQL Server killer.  Especially if it
> comes with a nice administrative GUI.  :-)

I wouldn't be too sanguine about that, from two perspectives:

 a) There's a moving target, here, in that Microsoft seems to be
    looking for the next "new thing" to be the elimination of
    the use of "files" in favor of the filesystem being treated
    as a database.

 b) We recently were considering how we'd put a sharable Windows box 
    in, at the office.  Were considering using VNC to allow it to be
    accessible.  Then someone thought to read the license, only to
    discover that the license pretty much expressly forbids running
    "foreign, competing applications" on the platform.

It seems pretty plausible that the net result of further development
will be platforms that are actively hostile to foreign software.

If I suggested that the licensing of Win2003 would expressly forbid
installing PostgreSQL, people would rightly accuse me of being a
paranoid conspiracy theorist.

But considering that the thought of VNC being outlawed would have seemed
pretty daft a few years ago, and we see things like DMCA combining with
"Homeland Security."  Anti-"hacking" provisions have been going into
telecom laws that appear to classify network hardware that can do NAT as
"illegal hacking" equipment.  I'm not sure what we'd have to consider
"daft" come 2005...
--
(reverse (concatenate 'string "gro.mca@" "enworbbc"))
http://cbbrowne.com/info/internet.html
"Heuristics (from the  French heure, "hour") limit the  amount of time
spent executing something.  [When using heuristics] it shouldn't take
longer than an hour to do something."


Mon, 14 Apr 2003 22:24:39 -0400
cbbrowne@cbbrowne.com
Re: Are we losing momentum? 
On Mon, 14 Apr 2003, Brent Verner wrote:

>   Applications, specifically applications that _use_ mysql.
>
>   A quick search over at freshmeat returns 1044 results for
> "mysql" and 260 for "postgresql".

That's a pretty reasonable thought. I work for a shop that sells
Postgres support, and even we install MySQL for the Q&D ticket tracking
system we recommend because we can't justify the cost to port it to
postgres. If the postgres support were there, we would surely be using it.

How to fix such a situation, I'm not sure. "MySQL Compatability Mode,"
anyone? :-)

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC


Tue, 15 Apr 2003 11:36:07 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: Are we losing momentum?

I agree, things aren't good when you look at the book shelf and app
support, but fortunately these are things that are shaded more by the
state of things 1-3 years ago rather than currently.  Certainly, we
would have seen an even worse ratio than 1:4 if we had looked last year
--- we aren't on parity yet, but I think we are getting there.

---------------------------------------------------------------------------

Larry Rosenman wrote:
> 
> 
> --On Monday, April 14, 2003 19:54:27 -0400 Tom Lane <tgl@sss.pgh.pa.us> 
> wrote:
> 
> > Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> Several people have asked if we are losing momentum.
> >
> > I don't think we are losing momentum considering the project in
> > isolation --- things seem to be moving as well as they ever have,
> > if not better.
> >
> > But I do sometimes worry that we are losing the mindshare war.
> > We might be growing fine, but if we're growing slower than MySQL is,
> > we've got a problem.  I was just in the local Barnes & Noble store
> > yesterday, and could not help but notice how many books had "MySQL" in
> > the title.  I didn't notice a single Postgres title (though I did not
> > look hard, since I was just passing through the computer area).
> I was in the Local MicroCenter, and found 3 PG titles, in addition to 
> Bruce's.
> 
> This is MUCH better than a year ago, when there were NONE.
> 
> Agreed, that MySQL, has a bigger shelf space.
> 
> I did all 3 authors a favor and bought copies.
> 
> LER
> 
> 
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
> 
> 
> 
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073


Mon, 14 Apr 2003 22:48:30 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Are we losing momentum?

> That's a pretty reasonable thought. I work for a shop that sells
> Postgres support, and even we install MySQL for the Q&D ticket tracking
> system we recommend because we can't justify the cost to port it to
> postgres. If the postgres support were there, we would surely be using it.
> 
> How to fix such a situation, I'm not sure. "MySQL Compatability Mode,"
> anyone? :-)

The real problem is PHP.  PHP is just the cruftiest language ever invented (trust me, I use it every day).  The PHP people are totally dedicated to MySQL, to the exclusion of all rational thought (eg. When I asked Rasmas at a conference about race conditions in his replicated setup, he replied "it's never going to happen - MySQL's replication is just too fast...).

Chris

Tue, 15 Apr 2003 10:50:03 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: Are we losing momentum?
cbbrowne@cbbrowne.com wrote:

> Kevin, without the "e", wrote...
> 
>>I seriously think the native Win32 port of Postgres will make a big
>>difference, because it'll be a SQL Server killer.  Especially if it
>>comes with a nice administrative GUI.  :-)

I agree. I don't think PostgreSQL will be a SQL Server killer,
but my completely ignorant guess is that 90% of the cause of the
*initial* gap between mySQL and PostgreSQL grew out of the fact
that a Win32 version of mySQL was available. Once the gap became
present, one then had to suffer switching costs. If the
features/performance of PostgreSQL > mySQL switching costs, then
PostgreSQL wins in the long term. Without a Win32 port, the
switching costs also include those switching costs associated
with switching from Win32 to Unix.

> 
> I wouldn't be too sanguine about that, from two perspectives:
> 
>  a) There's a moving target, here, in that Microsoft seems to be
>     looking for the next "new thing" to be the elimination of
>     the use of "files" in favor of the filesystem being treated
>     as a database.

They ought to get their database up to speed first, it seems to
me. I agree Microsoft's view of data management is a moving
target. 6 years ago everything, including network resources were
going to be accessed strickly through an OLE2 Compound Document
interface and OLE structured storage. Then the Internet got hot
and all data suddenly had to be accessible through URLs. Now
it's XML that hot. Perhaps the Microsoft filesystem of the
future will be one big XML document ;-)

> 
>  b) We recently were considering how we'd put a sharable Windows box 
>     in, at the office.  Were considering using VNC to allow it to be
>     accessible.  Then someone thought to read the license, only to
>     discover that the license pretty much expressly forbids running
>     "foreign, competing applications" on the platform.
> 
> It seems pretty plausible that the net result of further development
> will be platforms that are actively hostile to foreign software.
> 
> If I suggested that the licensing of Win2003 would expressly forbid
> installing PostgreSQL, people would rightly accuse me of being a
> paranoid conspiracy theorist.

I think you are a paranoid conspiracy theorist. :-)

Mike Mascari
mascarm@mascari.com


Mon, 14 Apr 2003 22:51:41 -0400
Mike Mascari <mascarm@mascari.com>
Re: Are we losing momentum?
Bruce Momjian wrote:
> I agree, things aren't good when you look at the book shelf and app
> support, but fortunately these are things that are shaded more by the
> state of things 1-3 years ago rather than currently.  Certainly, we
> would have seen an even worse ratio than 1:4 if we had looked last year
> --- we aren't on parity yet, but I think we are getting there.

What's missing are the "FOO Applications With PostgreSQL" sorts of
books, where
  (member FOO '(|Web| |PHP| |Perl| |Python| |Application Frameworks|))

The one PostgreSQL book that _does_ have some of this is the O'Reilly
one, where I was disappointed to see how much of the book was devoted to
a framework I /wasn't/ planning to use.

Right at the moment is probably /not/ a good time to be pushing books on
potentially-obscure application areas; my ex-publisher (Wrox) just
became an ex-publisher as a result of trying too hard to too quickly
hawk too many books in obscure application areas.

My suspicion is that this, along with very soft book sales throughout
the publishing industry, is likely to make "obscure application area"
books a tough sell in the short term.  Like it or not, "PostgreSQL +
FOO" is not going to be the easiest sell, particularly in the absence of
the much denigrated "PostgreSQL Marketing Cabal."
--
output = ("cbbrowne" "@cbbrowne.com")
http://cbbrowne.com/info/wp.html
"There  is no  psychiatrist in  the world  like a  puppy  licking your
face."  -- Ben Williams


Mon, 14 Apr 2003 23:12:35 -0400
Christopher Browne <cbbrowne@cbbrowne.com>
Re: Are we losing momentum? 
>>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:

    Tom> But I do sometimes worry that we are losing the mindshare
    Tom> war.  We might be growing fine, but if we're growing slower
    Tom> than MySQL is, we've got a problem.  I was just in the local

This is probably true. Once people get exposed to PostgreSQL then
there is a fair chance of forming an opinion. Today one of the
undergraduates in my class was telling me how after hacking pgsql
internals he has such a different impression of the two systems
(earlier he'd built a site with MySQL going by the "works for
slashdot" philosophy). 

-- 
Peace, at last ?
Sailesh
http://www.cs.berkeley.edu/~sailesh


14 Apr 2003 20:13:55 -0700
Sailesh Krishnamurthy <sailesh@cs.berkeley.edu>
Re: Are we losing momentum?
Mike Mascari wrote:
> cbbrowne@cbbrowne.com wrote:
>>I wouldn't be too sanguine about that, from two perspectives:
>>
>> a) There's a moving target, here, in that Microsoft seems to be
>>    looking for the next "new thing" to be the elimination of
>>    the use of "files" in favor of the filesystem being treated
>>    as a database.
> 
> 
> They ought to get their database up to speed first, it seems to
> me. I agree Microsoft's view of data management is a moving
> target. 

Not to mention the fact that there's a significant number of NT 4 
servers still out there -- what is that, 7 years old?  A lot of places 
aren't upgrading because they don't need to & don't want to shell out 
the cash.  (And it should go without saying that Microsoft is none too 
happy with it.)  With Windows 2K3 just coming out and who knows how much 
longer until the next version (or ther version after that, who knows 
when these "features" will actually show up), there's still a 
significant window in there for conventional database servers, 
especially for the price conscious out there.

----
Jeff Hoffmann
PropertyKey.com


Mon, 14 Apr 2003 22:54:11 -0500
Jeff Hoffmann <jeff@propertykey.com>
Re: Are we losing momentum?
IMVHO it's reference customers/users more than books & windows ports.

If I were a naive middle manager in some company, would I rather
use:

  (a) the database used by Yahoo, Cisco, and Sony?
  (b) the database used by Shannon Med Center, Mohawk SW, Vanten Inc, and BASF.

Now suppose I told that same middle manager there was an open 
source alternative:

  (c) used by Lockheed Martin, Nasdaq, AOL, and Unisys.

As far as I can tell (5-minutes searching) (c) is PostgreSQL.

  http://jobsearch.monster.com/jobsearch.asp?q=postgresql
  http://www.hotjobs.com/cgi-bin/job-search?KEYWORDS=postgres
  http://seeker.dice.com/jobsearch/servlet/JobSearch?op=1002&dockey=xml/8/1/816e9b7e50ae92331bb5c47a791a589f@activejobs0&c=1
  http://seeker.dice.com/jobsearch/servlet/JobSearch?op=1002&dockey=xml/c/8/c8dc5841d18329c6c50b55f67a7ff038@activejobs0&c=1
  http://seeker.dice.com/jobsearch/servlet/JobSearch?op=1002&dockey=xml/1/6/168f30dc84b8f195d1fc35feb6a2f67a@activejobs0&c=1
  "The Nasdaq Stock Market ... currently looking to fill the following 
   positions in Trumbull, CT...Some positions require knowledge of ...Postgre SQL.."


I'm not sure quite what it'd take to get the permission to use
these company's names, but surely we could have a list of links 
to the job postings...   I'd bet that one of monster, hotjobs, 
and/or dice would even provide a datafeed of relevant jobs to
be posted on the postgresql.org site.


If we simply had a list of companies using postgresql highly visible 
somewhere -- not necessarily a complex case study, just simple list 
of "company X uses postgresql for Y" statements -- I think it would 
go a long way.  I'll contribute.  InterVideo uses postgresql (for
running user surveys and some internal reporting and development tools).

    Ron

PS: No offense to Shannon, Mohawk, Vanten, and yes, I know BASF is
    an awesome company.  But they're all, even BASF, less of
    a household name than Sony,Yahoo,Cisco,AOL,Nasdaq,Lockheed.


Mon, 14 Apr 2003 21:38:19 -0700
"Ron Mayer" <ron@intervideo.com>
Re: Are we losing momentum?
On Tuesday 15 April 2003 05:48, you wrote:
> Regardless, I'm still of the opinion that if you build it, they will come
> -- particularly costly features like replication, PITR, etc. But maybe
> that is what the BSDs say about Linux?

That is an unfair comparison. The technical differences between BSD and linux 
are not as much as postgresql and mysql. Besides what is the parallel of SQL 
standard in OS world? POSIX? And both BSD/linux are doing fine sitting next 
to each other on that. 

After porting my small application in less than half an hour from linux to 
freeBSD and vice versa, I really do not agree with that comment. Not even in 
the spirit of it.

 Shridhar


Tue, 15 Apr 2003 11:38:05 +0530
Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
Re: Are we losing momentum?
On Tue, 2003-04-15 at 00:37, Bruce Momjian wrote:
> Several people have asked if we are losing momentum.  Specifically, they
> are concerned about Red Hat dropping their Red Hat Database and instead
> distributing PostgreSQL as part of Red Hat Enterprise Server, and they
> are concerned about recent press articles about MySQL.

Just reading GENERAL every day shows quite the contrary! There are more
and more questions which really belong on NOVICE. More and more
questions about porting applications from MySQL and Access.

RedHat renaming "PostgreSQL" to "PostgreSQL" after a short stint a.k.a.
"Red Hat Database" is a very positive step.

Lots of stupid journalists are starting to write "replace Oracle with
MySQL" rubbish. I am very concerned about this because the DBA who is
stupid enough to do this will fail miserably and get the wrong
impression about free RDBMS. 

Lets forget the "replace MySQL with PostgreSQL" stuff and go looking for
higher end converts. Our marketing push should be "replace Oracle with
PostgreSQL and replace Access with MySQL". This puts the emphasis on
which database can do what... 

Just my 2 EURO cents

Cheers
Tony Grant
(yes I use PostgreSQL where MySQL would suffice...)

-- 
www.tgds.net Library management software toolkit, 
redhat linux on Sony Vaio C1XD, 
Dreamweaver MX with Tomcat and PostgreSQL


15 Apr 2003 08:12:04 +0200
Tony Grant <tony@tgds.net>
Re: Are we losing momentum?
Curt Sampson <cjs@cynic.net> writes:
> That's a pretty reasonable thought. I work for a shop that sells
> Postgres support, and even we install MySQL for the Q&D ticket tracking
> system we recommend because we can't justify the cost to port it to
> postgres. If the postgres support were there, we would surely be using it.

> How to fix such a situation, I'm not sure. "MySQL Compatability Mode,"
> anyone? :-)

What issues are creating a compatibility problem for you?

			regards, tom lane


Tue, 15 Apr 2003 02:40:37 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Are we losing momentum? 
** Reply to message from Tony Grant <tony@tgds.net> on 15 Apr 2003 08:12:04
+0200
Hi,
 I've got to agree with this.
 We have just been through the excercise of porting our quite extensive fleet
maintenance and accounting package from db2 to postgres. While the port was not
without pain :) the results are very good. We run windows clients, (mostly),
and whatever the database of choice performs best on for the backend. We are
seeing performance gains for the 20-100 user bracket of a factor of 10 to 20
for switching from db2 to postgres on the same platform.
 We literally could not port this stuff to mysql. It does not have enough
feature to support us :)
 Definitely the story is not that postgres competes with mysql/access but
rather with db2/oracle, and the point in time recovery coming in the next
release, just strengthens that story a whole lot more too.

And, client reaction to an opensource (free :) database on linux has been very
enthusiastic.

Regards,
Wayne
http://www.bacchus.com.au

> On Tue, 2003-04-15 at 00:37, Bruce Momjian wrote:
> > Several people have asked if we are losing momentum.  Specifically, they
> > are concerned about Red Hat dropping their Red Hat Database and instead
> > distributing PostgreSQL as part of Red Hat Enterprise Server, and they
> > are concerned about recent press articles about MySQL.
> 
> Just reading GENERAL every day shows quite the contrary! There are more
> and more questions which really belong on NOVICE. More and more
> questions about porting applications from MySQL and Access.
> 
> RedHat renaming "PostgreSQL" to "PostgreSQL" after a short stint a.k.a.
> "Red Hat Database" is a very positive step.
> 
> Lots of stupid journalists are starting to write "replace Oracle with
> MySQL" rubbish. I am very concerned about this because the DBA who is
> stupid enough to do this will fail miserably and get the wrong
> impression about free RDBMS. 
> 
> Lets forget the "replace MySQL with PostgreSQL" stuff and go looking for
> higher end converts. Our marketing push should be "replace Oracle with
> PostgreSQL and replace Access with MySQL". This puts the emphasis on
> which database can do what... 
> 
> Just my 2 EURO cents
> 
> Cheers
> Tony Grant
> (yes I use PostgreSQL where MySQL would suffice...)
> 
> -- 
> www.tgds.net Library management software toolkit, 
> redhat linux on Sony Vaio C1XD, 
> Dreamweaver MX with Tomcat and PostgreSQL
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)


Tue, 15 Apr 2003 18:30:25 +1000
"Wayne Armstrong" <wdarmst@bacchus.com.au>
Re: Are we losing momentum?
On Tuesday 15 April 2003 14:00, you wrote:
> ** Reply to message from Tony Grant <tony@tgds.net> on 15 Apr 2003 08:12:04
> +0200
> Hi,
>  I've got to agree with this.
>  We have just been through the excercise of porting our quite extensive
> fleet maintenance and accounting package from db2 to postgres. While the
> port was not without pain :) the results are very good. We run windows
> clients, (mostly), and whatever the database of choice performs best on for
> the backend. We are seeing performance gains for the 20-100 user bracket of
> a factor of 10 to 20 for switching from db2 to postgres on the same
> platform.
>  We literally could not port this stuff to mysql. It does not have enough
> feature to support us :)
>  Definitely the story is not that postgres competes with mysql/access but
> rather with db2/oracle, and the point in time recovery coming in the next
> release, just strengthens that story a whole lot more too.
>
> And, client reaction to an opensource (free :) database on linux has been
> very enthusiastic.

If you don't mind, could you please submit a short write up on your experience 
for submission on postgresql advocacy-Case Study section? You can either post 
it to posgresql advocacy list or send it to me offlist, if required.

See  http://advocacy.postgresql.org

 Shridhar


Tue, 15 Apr 2003 14:12:35 +0530
Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
Re: [GENERAL] Are we losing momentum?
Tom Lane wrote:
> Curt Sampson <cjs@cynic.net> writes:
> > That's a pretty reasonable thought. I work for a shop that sells
> > Postgres support, and even we install MySQL for the Q&D ticket tracking
> > system we recommend because we can't justify the cost to port it to
> > postgres. If the postgres support were there, we would surely be using it.
> 
> > How to fix such a situation, I'm not sure. "MySQL Compatability Mode,"
> > anyone? :-)
> 
> What issues are creating a compatibility problem for you?

Erm...reserved words?  "Freeze" is a reserved word, for instance, and
that actually bit me when converting an MS-SQL database...

I have no problem with reserved words in principle, at least when they
refer to the SQL-standard commands and their options, but it's not
clear that turning options (such as FREEZE) for PG-specific commands
(such as VACUUM) into reserved words is a good idea.  But it may not
be possible to avoid it, unfortunately.  :-(


-- 
Kevin Brown					      kevin@sysexperts.com


Tue, 15 Apr 2003 02:10:42 -0700
Kevin Brown <kevin@sysexperts.com>
Re: Are we losing momentum?
On Tue, 15 Apr 2003, Tom Lane wrote:

> > How to fix such a situation, I'm not sure. "MySQL Compatability Mode,"
> > anyone? :-)
>
> What issues are creating a compatibility problem for you?

We can't unthinkingly point the product at a PostgreSQL server and
have it Just Work. So all we really need is full SQL and wire-protocol
compatability. :-)

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC


Tue, 15 Apr 2003 18:28:01 +0900 (JST)
Curt Sampson <cjs@cynic.net>
Re: Are we losing momentum? 
Hi,
Your experience is very interesting.
But could you tell me what is the Database size ?

    Regards,
    Thierry

Wayne Armstrong wrote:

> ** Reply to message from Tony Grant <tony@tgds.net> on 15 Apr 2003 08:12:04
> +0200
> Hi,
>  I've got to agree with this.
>  We have just been through the excercise of porting our quite extensive fleet
> maintenance and accounting package from db2 to postgres. While the port was not
> without pain :) the results are very good. We run windows clients, (mostly),
> and whatever the database of choice performs best on for the backend. We are
> seeing performance gains for the 20-100 user bracket of a factor of 10 to 20
> for switching from db2 to postgres on the same platform.
>  We literally could not port this stuff to mysql. It does not have enough
> feature to support us :)
>  Definitely the story is not that postgres competes with mysql/access but
> rather with db2/oracle, and the point in time recovery coming in the next
> release, just strengthens that story a whole lot more too.
>
> And, client reaction to an opensource (free :) database on linux has been very
> enthusiastic.
>
> Regards,
> Wayne
> http://www.bacchus.com.au
>
> > On Tue, 2003-04-15 at 00:37, Bruce Momjian wrote:
> > > Several people have asked if we are losing momentum.  Specifically, they
> > > are concerned about Red Hat dropping their Red Hat Database and instead
> > > distributing PostgreSQL as part of Red Hat Enterprise Server, and they
> > > are concerned about recent press articles about MySQL.
> >
> > Just reading GENERAL every day shows quite the contrary! There are more
> > and more questions which really belong on NOVICE. More and more
> > questions about porting applications from MySQL and Access.
> >
> > RedHat renaming "PostgreSQL" to "PostgreSQL" after a short stint a.k.a.
> > "Red Hat Database" is a very positive step.
> >
> > Lots of stupid journalists are starting to write "replace Oracle with
> > MySQL" rubbish. I am very concerned about this because the DBA who is
> > stupid enough to do this will fail miserably and get the wrong
> > impression about free RDBMS.
> >
> > Lets forget the "replace MySQL with PostgreSQL" stuff and go looking for
> > higher end converts. Our marketing push should be "replace Oracle with
> > PostgreSQL and replace Access with MySQL". This puts the emphasis on
> > which database can do what...
> >
> > Just my 2 EURO cents
> >
> > Cheers
> > Tony Grant
> > (yes I use PostgreSQL where MySQL would suffice...)
> >
> > --
> > www.tgds.net Library management software toolkit,
> > redhat linux on Sony Vaio C1XD,
> > Dreamweaver MX with Tomcat and PostgreSQL
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 2: you can get off all lists at once with the unregister command
> >     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
Tue, 15 Apr 2003 11:41:47 +0200
Thierry Missimilly <THIERRY.MISSIMILLY@BULL.NET>
Re: Are we losing momentum?
** Reply to message from Thierry Missimilly <THIERRY.MISSIMILLY@BULL.NET> on
Tue, 15 Apr 2003 11:41:47 +0200
Hi,
 About 450 tables (+ lots of views:)
 Row counts range from 0 to about 3 million.
 Database sizes range from about 600 meg (a bus company with about 50 busses
and a years fueling/servicing/rostering history) to around 12 gig currently :).
 We are not newbies with db2. We have been using/suppling it since version
2.1(We actually started development on ver 1. something, but didn't ship till
after the version 2 release :). We do do a good job of tuning db2.
Nevertheless, performance of postgresql across the board (and we have not
designed for postgres like we did for db2), leaves db2 for dead most places. 

Regards,
Wayne
 
> Hi,
> Your experience is very interesting.
> But could you tell me what is the Database size ?
> 
>     Regards,
>     Thierry
> 
> Wayne Armstrong wrote:
> 
> > ** Reply to message from Tony Grant <tony@tgds.net> on 15 Apr 2003 08:12:04
> > +0200
> > Hi,
> >  I've got to agree with this.
> >  We have just been through the excercise of porting our quite extensive fleet
> > maintenance and accounting package from db2 to postgres. While the port was not
> > without pain :) the results are very good. We run windows clients, (mostly),
> > and whatever the database of choice performs best on for the backend. We are
> > seeing performance gains for the 20-100 user bracket of a factor of 10 to 20
> > for switching from db2 to postgres on the same platform.
> >  We literally could not port this stuff to mysql. It does not have enough
> > feature to support us :)
> >  Definitely the story is not that postgres competes with mysql/access but
> > rather with db2/oracle, and the point in time recovery coming in the next
> > release, just strengthens that story a whole lot more too.
> >
> > And, client reaction to an opensource (free :) database on linux has been very
> > enthusiastic.
> >
> > Regards,
> > Wayne
> > http://www.bacchus.com.au
> >
> > > On Tue, 2003-04-15 at 00:37, Bruce Momjian wrote:
> > > > Several people have asked if we are losing momentum.  Specifically, they
> > > > are concerned about Red Hat dropping their Red Hat Database and instead
> > > > distributing PostgreSQL as part of Red Hat Enterprise Server, and they
> > > > are concerned about recent press articles about MySQL.
> > >
> > > Just reading GENERAL every day shows quite the contrary! There are more
> > > and more questions which really belong on NOVICE. More and more
> > > questions about porting applications from MySQL and Access.
> > >
> > > RedHat renaming "PostgreSQL" to "PostgreSQL" after a short stint a.k.a.
> > > "Red Hat Database" is a very positive step.
> > >
> > > Lots of stupid journalists are starting to write "replace Oracle with
> > > MySQL" rubbish. I am very concerned about this because the DBA who is
> > > stupid enough to do this will fail miserably and get the wrong
> > > impression about free RDBMS.
> > >
> > > Lets forget the "replace MySQL with PostgreSQL" stuff and go looking for
> > > higher end converts. Our marketing push should be "replace Oracle with
> > > PostgreSQL and replace Access with MySQL". This puts the emphasis on
> > > which database can do what...
> > >
> > > Just my 2 EURO cents
> > >
> > > Cheers
> > > Tony Grant
> > > (yes I use PostgreSQL where MySQL would suffice...)
> > >
> > > --
> > > www.tgds.net Library management software toolkit,
> > > redhat linux on Sony Vaio C1XD,
> > > Dreamweaver MX with Tomcat and PostgreSQL
> > >
> > >
> > > ---------------------------(end of broadcast)---------------------------
> > > TIP 2: you can get off all lists at once with the unregister command
> > >     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org


Tue, 15 Apr 2003 20:10:48 +1000
"Wayne Armstrong" <wdarmst@bacchus.com.au>
Re: Are we losing momentum?


Christopher Kings-Lynne wrote:

>>That's a pretty reasonable thought. I work for a shop that sells
>>Postgres support, and even we install MySQL for the Q&D ticket tracking
>>system we recommend because we can't justify the cost to port it to
>>postgres. If the postgres support were there, we would surely be using it.
>>
>>How to fix such a situation, I'm not sure. "MySQL Compatability Mode,"
>>anyone? :-)
>>    
>>
>
>The real problem is PHP.  PHP is just the cruftiest language ever invented (trust me, I use it every day).  The PHP people are totally dedicated to MySQL, to the exclusion of all rational thought (eg. When I asked Rasmas at a conference about race conditions in his replicated setup, he replied "it's never going to happen - MySQL's replication is just too fast...).
>
>  
>
Hey! don't go knocking PHP, it is probably one of the most flexible and 
easy to use systems around. I have done several fairly large projects 
with PHP and while it is an "ugly" environment, it performs well enough, 
has a very usable extension interface, it is quick and easy to even 
large projects done.

As for MySQL, there are two things that PostgreSQL does not do, and 
probably can not do to support MySQL:

(1) REPLACE INTO (I think that's the name) which does either an insert 
or update into a table depending on the existence of a row. I was told 
that this was impossible.

(2) MySQL returns a value on  insert which is usually usable, for instance,
insert into mytable (x,y,z) values(1,2,3);
select rowid from mytable where x=1 and y=2 and z=3;

I have had many discussions with MySQL people, and one common thread 
exists. People who use MySQL do not usually understand databases all 
that well. Arguments about *why* it is a horrible database and barely 
SQL at all, fall on deaf ears. They don't understand PostgreSQL, they 
complain that it is "too big." They complain that it is "too much," 
MySQL is all they need. They complain that it is "too hard" to use.

All of these things are largely imagined. PostgreSQL is not much bigger 
than MySQL, in fact, the difference is negligible with regards to 
average system capability these days. It isn't any more difficult to 
use, its just a little different. They, however, feel safe with MySQL. 
MySQL is the Microsoft of databases, everyone uses it because everyone 
uses it, not because it is better or even adequate.

We need to take projects like Bugzilla (Did RH ever release the PG 
version or am I way out of date?) and port them to PostgreSQL. We need 
to write free articles for Linux and IT magazines about how to take a 
MySQL project over to PostgreSQL easily, why PostgreSQL is much better 
than MySQL, lastly we have to play the MySQL benchmark game .. we need 
to create a Benchmark program that clearly shows how PostgreSQL compares 
to MySQL.


Tue, 15 Apr 2003 07:51:30 -0400
mlw <pgsql@mohawksoft.com>
Re: Are we losing momentum?


Mike Mascari wrote:

>cbbrowne@cbbrowne.com wrote:
>  
>
>> b) We recently were considering how we'd put a sharable Windows box 
>>    in, at the office.  Were considering using VNC to allow it to be
>>    accessible.  Then someone thought to read the license, only to
>>    discover that the license pretty much expressly forbids running
>>    "foreign, competing applications" on the platform.
>>
>>It seems pretty plausible that the net result of further development
>>will be platforms that are actively hostile to foreign software.
>>
>>If I suggested that the licensing of Win2003 would expressly forbid
>>installing PostgreSQL, people would rightly accuse me of being a
>>paranoid conspiracy theorist.
>>    
>>
>
>I think you are a paranoid conspiracy theorist. :-)
>
>Mike Mascari
>mascarm@mascari.com
>  
>
"Just because you're paranoid does not mean they're not out to get you."
Henry Kissinger.


Tue, 15 Apr 2003 08:07:37 -0400
mlw <pgsql@mohawksoft.com>
Re: Are we losing momentum?
> Hey! don't go knocking PHP, it is probably one of the most flexible and 
> easy to use systems around. I have done several fairly large projects 
> with PHP and while it is an "ugly" environment, it performs well enough, 
> has a very usable extension interface, it is quick and easy to even 
> large projects done.

Right. PHP is our friend. In Japan Apache+PHP+PostgreSQL combo is the
standard for Web systems. Very few people uses Apache+PHP+MySQL.
--
Tatsuo Ishii


Tue, 15 Apr 2003 21:09:31 +0900 (JST)
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Are we losing momentum?
Tatsuo, this has always fascinated me.  Any insights you could share about how PostgreSQL achieved the prominence it has in Japan (and how MySQL did not) would be very interesting.

Cheers,
Ned

----- Original Message ----- 
From: "Tatsuo Ishii" <t-ishii@sra.co.jp>
To: <pgsql@mohawksoft.com>
Cc: <chriskl@familyhealth.com.au>; <cjs@cynic.net>; <brent@rcfile.org>; <pgsql-hackers@postgresql.org>
Sent: Tuesday, April 15, 2003 8:09 AM
Subject: Re: [HACKERS] Are we losing momentum?


> Hey! don't go knocking PHP, it is probably one of the most flexible and 
> easy to use systems around. I have done several fairly large projects 
> with PHP and while it is an "ugly" environment, it performs well enough, 
> has a very usable extension interface, it is quick and easy to even 
> large projects done.

Right. PHP is our friend. In Japan Apache+PHP+PostgreSQL combo is the
standard for Web systems. Very few people uses Apache+PHP+MySQL.
--
Tatsuo Ishii


---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org


Tue, 15 Apr 2003 08:38:05 -0400
"Ned Lilly" <ned@nedscape.com>
PostgreSQL vs. MySQL in Japan (was: Are we losing momentum?)
As one of the top search engine campaign optimization companies in the
space, we here at Did-it.com have been using Postgresql for over a year
now. We had serious locking problems with MySQL and even switching to
their Innodb handler did not solve all the issues.

As the DB administrator, I recommended we switch when it came time to
re-write our client platform. That we did, and we have not looked back.
We have millions of listings, keywords and we perform live visitor
tracking in our database. We capture on the order of about 1 million
visitors every day, with each hit making updates, selects and possibly
inserts.

We could not have done this in mySQL. Basically when I see silly posts
over on Slashdot about MySQL being as good a sliced bread, you can check
out the debunking posts that I make as "esconsult1".

Yes, perhaps Postgresql needs a central org that manages press and so
on, but we know that we dont get the press that MySQL does, but quietly
in the background Postgresql is handling large important things.

- Ericson Smith
Web Developer
Db Admin
http://www.did-it.com

-- 
Ericson Smith <eric@did-it.com>


15 Apr 2003 09:12:19 -0400
Ericson Smith <eric@did-it.com>
Re: Are we losing momentum?
Hi,

We should probably put out an FAQ saying, if you have success story, please 
make a write-up and send to us at http://advocacy.postgresql.org.

 Shridhar

On Tuesday 15 April 2003 18:42, Ericson Smith wrote:
> As one of the top search engine campaign optimization companies in the
> space, we here at Did-it.com have been using Postgresql for over a year
> now. We had serious locking problems with MySQL and even switching to
> their Innodb handler did not solve all the issues.
>
> As the DB administrator, I recommended we switch when it came time to
> re-write our client platform. That we did, and we have not looked back.
> We have millions of listings, keywords and we perform live visitor
> tracking in our database. We capture on the order of about 1 million
> visitors every day, with each hit making updates, selects and possibly
> inserts.
>
> We could not have done this in mySQL. Basically when I see silly posts
> over on Slashdot about MySQL being as good a sliced bread, you can check
> out the debunking posts that I make as "esconsult1".
>
> Yes, perhaps Postgresql needs a central org that manages press and so
> on, but we know that we dont get the press that MySQL does, but quietly
> in the background Postgresql is handling large important things.
>
> - Ericson Smith
> Web Developer
> Db Admin
> http://www.did-it.com


Tue, 15 Apr 2003 18:48:46 +0530
Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
Re: [GENERAL] Are we losing momentum?
mlw <pgsql@mohawksoft.com> writes:
> We need to take projects like Bugzilla (Did RH ever release the PG 
> version or am I way out of date?) and port them to PostgreSQL.

See http://bugzilla.redhat.com/bugzilla/ ... note icon at bottom ...
note tarball offered in News ...

			regards, tom lane


Tue, 15 Apr 2003 10:03:19 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Are we losing momentum? 
On Tue, 2003-04-15 at 07:51, mlw wrote:
> Christopher Kings-Lynne wrote:
> >
> >The real problem is PHP.  PHP is just the cruftiest language ever invented 
> > (trust me, I use it every day).  The PHP people are totally dedicated to 
> > MySQL, to the exclusion of all rational thought (eg. When I asked 
> > Rasmas at a conference about race conditions in his replicated 
> > setup, he replied "it's never going to happen - MySQL's replication 
> > is just too fast...).
> >
> Hey! don't go knocking PHP, it is probably one of the most flexible and 
> easy to use systems around. I have done several fairly large projects 
> with PHP and while it is an "ugly" environment, it performs well enough, 
> has a very usable extension interface, it is quick and easy to even 
> large projects done.
> 

The problem is the marriage of PHP and MySql. I've always held the
notion that early on several of the php developers, being windows
hackers, needed an open source database that would run on windows. They
picked mysql (which was probably their best option at the time) and
mysql rode on the shoulders php's success.  

> As for MySQL, there are two things that PostgreSQL does not do, and 
> probably can not do to support MySQL:
> 
> (1) REPLACE INTO (I think that's the name) which does either an insert 
> or update into a table depending on the existence of a row. I was told 
> that this was impossible.
> 
> (2) MySQL returns a value on  insert which is usually usable, for instance,
> insert into mytable (x,y,z) values(1,2,3);
> select rowid from mytable where x=1 and y=2 and z=3;
> 

I'm pretty sure I've seen people create db functions to duplicate these
features, but admittedly that would be more complicated.

<snip>
> 
> We need to take projects like Bugzilla (Did RH ever release the PG 
> version or am I way out of date?) and port them to PostgreSQL. We need 
> to write free articles for Linux and IT magazines about how to take a 
> MySQL project over to PostgreSQL easily, why PostgreSQL is much better 
> than MySQL, 

Red Hat actually did do this, and does make the source available. One
problem I found with porting of mysql apps is that those apps tend to do
a lot of dump things to make up for mysql's missing features.  Unless
you really are willing to fork the code and then maintain it as a new
project, porting applications gets somewhat futile.

Robert Treat


15 Apr 2003 10:19:11 -0400
Robert Treat <xzilla@users.sourceforge.net>
Re: Are we losing momentum?
On Tue, 15 Apr 2003, mlw wrote:

> 
> 
> Christopher Kings-Lynne wrote:
> 
> >>That's a pretty reasonable thought. I work for a shop that sells
> >>Postgres support, and even we install MySQL for the Q&D ticket tracking
> >>system we recommend because we can't justify the cost to port it to
> >>postgres. If the postgres support were there, we would surely be using it.
> >>
> >>How to fix such a situation, I'm not sure. "MySQL Compatability Mode,"
> >>anyone? :-)
> >>    
> >>
> >
> >The real problem is PHP.  PHP is just the cruftiest language ever invented (trust me, I use it every day).  The PHP people are totally dedicated to MySQL, to the exclusion of all rational thought (eg. When I asked Rasmas at a conference about race conditions in his replicated setup, he replied "it's never going to happen - MySQL's replication is just too fast...).
> >
> >  
> >
> Hey! don't go knocking PHP, it is probably one of the most flexible and 
> easy to use systems around. I have done several fairly large projects 
> with PHP and while it is an "ugly" environment, it performs well enough, 
> has a very usable extension interface, it is quick and easy to even 
> large projects done.

I would say that compared to Perl, TCL, and many other scripting languages 
that PHP is actually a far better and more logically designed language.  
the way it handles arrays and global vars is the way every language 
should.


Tue, 15 Apr 2003 10:18:28 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Are we losing momentum?
On Mon, Apr 14, 2003 at 07:54:27PM -0400, Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Several people have asked if we are losing momentum.
> 
> I was just in the local Barnes & Noble store
> yesterday, and could not help but notice how many books had "MySQL" in
> the title.  I didn't notice a single Postgres title (though I did not
> look hard, since I was just passing through the computer area).

2 local stores here:
One has 11 PostgresQL books and 40 MySQL, the other had 5 on
PostgresQL and 23 about MySQL.


Kurt


Tue, 15 Apr 2003 18:36:17 +0200
Kurt Roeckx <Q@ping.be>
Re: Are we losing momentum?
On Tuesday 15 April 2003 07:51, mlw wrote:
> We need to take projects like Bugzilla (Did RH ever release the PG
> version or am I way out of date?) and port them to PostgreSQL.

http://bugzilla.redhat.com/download/rh-bugzilla-pg-LATEST.tar.gz
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Tue, 15 Apr 2003 12:43:39 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Are we losing momentum?
Tom Lane wrote:
> narrow the bookshelf gap a little, at least.  Any wannabee authors
> out there?  (And Bruce, your book is due for a second edition...)

Agreed.  I will contact the publisher and get started, maybe in the
summer.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073


Tue, 15 Apr 2003 12:50:07 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Are we losing momentum?

Having good reference sites is important, and I could list as many
impressive ones as MySQL, but who has time to hunt around and get
permission to list them --- I will tell you who --- the MySQL marketing
guys, while the PostgreSQL guys don't.  :-(

---------------------------------------------------------------------------

Ron Mayer wrote:
> IMVHO it's reference customers/users more than books & windows ports.
> 
> If I were a naive middle manager in some company, would I rather
> use:
> 
>   (a) the database used by Yahoo, Cisco, and Sony?
>   (b) the database used by Shannon Med Center, Mohawk SW, Vanten Inc, and BASF.
> 
> Now suppose I told that same middle manager there was an open 
> source alternative:
> 
>   (c) used by Lockheed Martin, Nasdaq, AOL, and Unisys.
> 
> As far as I can tell (5-minutes searching) (c) is PostgreSQL.
> 
>   http://jobsearch.monster.com/jobsearch.asp?q=postgresql
>   http://www.hotjobs.com/cgi-bin/job-search?KEYWORDS=postgres
>   http://seeker.dice.com/jobsearch/servlet/JobSearch?op=1002&dockey=xml/8/1/816e9b7e50ae92331bb5c47a791a589f@activejobs0&c=1
>   http://seeker.dice.com/jobsearch/servlet/JobSearch?op=1002&dockey=xml/c/8/c8dc5841d18329c6c50b55f67a7ff038@activejobs0&c=1
>   http://seeker.dice.com/jobsearch/servlet/JobSearch?op=1002&dockey=xml/1/6/168f30dc84b8f195d1fc35feb6a2f67a@activejobs0&c=1
>   "The Nasdaq Stock Market ... currently looking to fill the following 
>    positions in Trumbull, CT...Some positions require knowledge of ...Postgre SQL.."
> 
> 
> I'm not sure quite what it'd take to get the permission to use
> these company's names, but surely we could have a list of links 
> to the job postings...   I'd bet that one of monster, hotjobs, 
> and/or dice would even provide a datafeed of relevant jobs to
> be posted on the postgresql.org site.
> 
> 
> If we simply had a list of companies using postgresql highly visible 
> somewhere -- not necessarily a complex case study, just simple list 
> of "company X uses postgresql for Y" statements -- I think it would 
> go a long way.  I'll contribute.  InterVideo uses postgresql (for
> running user surveys and some internal reporting and development tools).
> 
>     Ron
> 
> PS: No offense to Shannon, Mohawk, Vanten, and yes, I know BASF is
>     an awesome company.  But they're all, even BASF, less of
>     a household name than Sony,Yahoo,Cisco,AOL,Nasdaq,Lockheed.
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073


Tue, 15 Apr 2003 12:52:27 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Are we losing momentum?
Shridhar Daithankar wrote:
> On Tuesday 15 April 2003 05:48, you wrote:
> > Regardless, I'm still of the opinion that if you build it, they will come
> > -- particularly costly features like replication, PITR, etc. But maybe
> > that is what the BSDs say about Linux?
> 
> That is an unfair comparison. The technical differences between BSD and linux 
> are not as much as postgresql and mysql. Besides what is the parallel of SQL 
> standard in OS world? POSIX? And both BSD/linux are doing fine sitting next 
> to each other on that. 

Agreed, Linux and BSD are pretty close --- but Linux used to be behind
BSD --- they caught up because both are open source.  The big question
is whether MySQL (which isn't openly developed) will catch up to
PostgreSQL.  And if they do catch up, will we have mind share parity by
that time?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073


Tue, 15 Apr 2003 12:55:38 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Are we losing momentum?
The impression of MySQL is light weight and fast,
the reputation of PostgreSQL is full featured. 
Business chooses PostgreSQL is bacause PostgreSQL is close
to database like Oracle, reliable but without cost.

To compete with MySQL is not a good strategy, IMHO,
PostgreSQL needs to focus adding features such as
table partitioning like Oracle, needs to improve
the performance of subquery, etc. Those lack performance 
features are the choke point (it's easy to get better performance 
for a big table [~100 million] with partitions in oracle than
postgreSQL; it's a nightmare, a mess for using subquery in postgreSQL,
I can't wait 7.4's smarter on this).

If you really have a super product, don't you
worry user will not switch to it with no cost?

just some thoughts ...

johnl


Tue, 15 Apr 2003 12:21:15 -0500
"John Liu" <johnl@emrx.com>
Re: Are we losing momentum?
On Tue, 2003-04-15 at 09:43, Lamar Owen wrote:
> On Tuesday 15 April 2003 07:51, mlw wrote:
> > We need to take projects like Bugzilla (Did RH ever release the PG
> > version or am I way out of date?) and port them to PostgreSQL.
> 
> http://bugzilla.redhat.com/download/rh-bugzilla-pg-LATEST.tar.gz

Of course, the installation instructions that come with it tell you
to install perl's interface to MySQL, not PostgreSQL.  Sigh.

-- 
Steve Wampler <swampler@noao.edu>
National Solar Observatory


15 Apr 2003 10:23:04 -0700
Steve Wampler <swampler@noao.edu>
Re: Are we losing momentum?
IMHO, mySql 5.0 will put more pressure on PostgreSql, when it's
available.

One of the features that PostgreSql must have, IMHO, is support for
cross-db operations (queries, updates, deletes, inserts). 2PC and
cross-server stuff would be nice but it's not as important as simple
cross -db operations across databases on the same server. All major
comercial RDBMS (and even mySql!) support this but for Postgres. Sad.









__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com


Tue, 15 Apr 2003 10:30:19 -0700 (PDT)
ow <oneway_111@yahoo.com>
Re: Are we losing momentum?
> > Regardless, I'm still of the opinion that if you build it, they
> > will come -- particularly costly features like replication, PITR,
> > etc. But maybe that is what the BSDs say about Linux?
> 
> That is an unfair comparison. The technical differences between BSD
> and linux are not as much as postgresql and mysql.  Besides what is
> the parallel of SQL standard in OS world? POSIX? And both BSD/linux
> are doing fine sitting next to each other on that.
> 
> After porting my small application in less than half an hour from
> linux to freeBSD and vice versa, I really do not agree with that
> comment. Not even in the spirit of it.

Yes, that is the joy of POSIX, ANSI, SUS, SUSv2, XPG*, etc.  The
differences in the OS aren't visible at the user level and shouldn't
be (beyond the layout/management).  That said, standards are great,
but all select()/poll() calls weren't created equal, just like all
SELECT statements weren't created equal.  -sc

-- 
Sean Chittenden


Tue, 15 Apr 2003 11:29:49 -0700
Sean Chittenden <sean@chittenden.org>
Re: Are we losing momentum?
On Tue, 2003-04-15 at 13:30, ow wrote:
> IMHO, mySql 5.0 will put more pressure on PostgreSql, when it's
> available.
> 
> One of the features that PostgreSql must have, IMHO, is support for
> cross-db operations (queries, updates, deletes, inserts). 2PC and
> cross-server stuff would be nice but it's not as important as simple
> cross -db operations across databases on the same server. All major
> comercial RDBMS (and even mySql!) support this but for Postgres. Sad.
> 

dblink ?

Robert Treat


15 Apr 2003 14:50:10 -0400
Robert Treat <xzilla@users.sourceforge.net>
Re: Are we losing momentum?
>From my experience, almost every time I talk to a MySQL supporter about
PostgreSQL, the whole "vacuum" issue always seems to come up. Some way
to get vacuum automated (and thus out of sight, out of mind) I think
would make great strides in making PG at least "seem" more friendly to
someone on the outside.

Shared hosting enviroments. I work for a web hosting company that offers
MySQL to all of its customers, our MySQL server has several thousand
databases on it, and I must say it works exceptionally well. 

Creating users/databases/changing passwords is as simple as sending it a
couple queries from our Customer web interface, trouble shooting poor
queries takes seconds when using "mytop" (mtop), and tracking/billing
for disk usage is as simple as running "du /var/lib/mysql/*". I would
like to say the same things for PG, but I'm affrid I can't.

I think it all comes down to how simple PG is to setup and use on a
daily basis. This is what determines the size of its community. Even
just the simple things make a big difference. ie:

\dt

compared to:

show tables;

Yes, once you get over the "hump" PG is quite efficient, but you need to
understand it, and learn some small quriks first. With MySQL, you can
pretty much guess commands, and they often work! Not as much luck with
PG. 

show indexes
show processlist
show columns from <table>

These are all easy/simple commands that make sense to someone who is
just learning the ropes. Short abbreviated, commands are great for the
experts, but can greatly discourage newbies.


On Mon, 2003-04-14 at 18:52, Brent Verner wrote:
> Gretings!
> 
> [2003-04-14 19:54] Tom Lane said:
> | Bruce Momjian <pgman@candle.pha.pa.us> writes:
> | > Several people have asked if we are losing momentum.
> 
> | I don't know what we can do about it, other than maybe push harder to
> | get some more PG titles into O'Reilly's catalog ... that would help
> | narrow the bookshelf gap a little, at least.  Any wannabee authors
> | out there?  (And Bruce, your book is due for a second edition...)
> 
>   I've wanted to pipe up in a few of these "popularity" 
> discussions in the past.  Seeing how I can't make time to
> participate in any other meaningful capacity, I'll share
> my thoughts on _why_ mysql has the mindshare.
> 
> 
>   Applications, specifically applications that _use_ mysql.
> 
> 
>   A quick search over at freshmeat returns 1044 results for 
> "mysql" and 260 for "postgresql".  Before this turns into a 
> cause/effect discussion, I want to state up front that the 
> real "effect" of this is that someone is 4 times as likely to 
> download an application that uses mysql.  Sure, many are 
> "trivial" applications, but I posit that it is _specifically_ 
> these "trivial" applications that inoculate the uninitiated 
> with the belief that mysql is suitable for use in real, albeit
> trivial applications.  Additionally, it these rudimentary 
> applications that will be studied by many as the way to write 
> a database application.
> 
>   It is all good and well that postgres /can/ do, but until
> the application developers see that those features are
> valuable enough to forgo mysql support, they'll write the 
> application to support whatever database is most likely to 
> _already_ be installed, which will be mysql.  Granted, 
> many developers will also try to support multiple dbs via
> the language's db api, but this leaves the less-supported
> dbs in an even worse position; being relegated to an
> "might work with XXX database".  When anxious user learns
> that "might" currently means "doesn't," the second-string
> database looks even worse in the eyes of the user.
> 
>   How to solve this problem?  This is the hard part, but
> luckily ISTM that there are a few ways to succeed.  Neither
> of which involves marketing or writing books.
> 
>   1) become active in the "also supports postgres" projects,
>      and add features that are made available _because_ of
>      postgres' superiority.  Eventually, market pressure
>      for the cool feature(s) will lead users to choose
>      postgres, and mysql could be relegated to the "also
>      runs on mysql, with limited featureset"
>   2) take a popular project that uses mysql, fork it, and
>      add features that can only be implemented using posgres.
>   3) release that super-cool code that you've been hacking
>      on for years, especially if it is a "trivial" app.
>   4) convince your employer that it would be _beneficial_ to
>      them to release, as open source, the internal app(s) you've 
>      developed, using postgres-specific features.  (This is 
>      about all I can claim to be doing at this point in my 
>      indentured servitude, and I can't say I'm doing a good
>      job... :-/)
> 
>   I'm sure this idea is not original, but I'm also sure that
> it _is_ the answer to gaining market^Wmindshare in this
> database market.
> 
>   (I must apologize in advance, that I might not have time
> to even follow this thread, in fact, I hope that instead of
> replying to this, the potential respondent might consider
> helping to increase the number of apps that require postgres
> :-)
> 
> wishing-I-could-contribute-more-ly yours,
>   brent
-- 
Best Regards,
 
Mike Benoit
NetNation Communications Inc.
Systems Engineer
Tel: 604-684-6892 or 888-983-6600
 ---------------------------------------
 
 Disclaimer: Opinions expressed here are my own and not 
 necessarily those of my employer


15 Apr 2003 11:56:50 -0700
Mike Benoit <mikeb@netnation.com>
Re: Are we losing momentum?
Robert Treat <xzilla@users.sourceforge.net> writes:
> On Tue, 2003-04-15 at 13:30, ow wrote:
>> One of the features that PostgreSql must have, IMHO, is support for
>> cross-db operations (queries, updates, deletes, inserts). 2PC and
>> cross-server stuff would be nice but it's not as important as simple
>> cross -db operations across databases on the same server. All major
>> comercial RDBMS (and even mySql!) support this but for Postgres. Sad.

> dblink ?

I'm of the opinion that the availability of schemas solves most of the
problems that people say they need cross-database access for.  If you
want cross-database access, first say why putting your data into several
schemas in a single database doesn't get the job done for you.

(Obviously, this only addresses cases where you'd have put the multiple
databases under one postmaster, but that's the scenario people seem to be
concerned about.)

			regards, tom lane


Tue, 15 Apr 2003 15:17:17 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Are we losing momentum? 
--- Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I'm of the opinion that the availability of schemas solves most of
> the problems that people say they need cross-database access for.  If

> you want cross-database access, first say why putting your data into
> several  schemas in a single database doesn't get the job done for
you.

Some databases contain lots of data, e.g. dbs that contain historical
data. No one wants to have one HUGE db that runs all company's apps,
takes hours (if not days) to recover and when this huge db goes down
none of the apps is available.





__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com


Tue, 15 Apr 2003 12:35:18 -0700 (PDT)
ow <oneway_111@yahoo.com>
Re: Are we losing momentum? 
On Tue, 2003-04-15 at 15:35, ow wrote:
> Some databases contain lots of data, e.g. dbs that contain historical
> data. No one wants to have one HUGE db that runs all company's apps,
> takes hours (if not days) to recover and when this huge db goes down
> none of the apps is available.

Are you talking about queries between databases on the same postmaster
(i.e. running under the same PostgreSQL installation), or queries
between postmasters running on different systems? If the former, I don't
see how putting your data into multiple schemas in a single database is
significantly less reliable than putting it into multiple databases.

Cheers,

Neil


15 Apr 2003 16:27:07 -0400
Neil Conway <neilc@samurai.com>
Re: Are we losing momentum?
Mike Benoit writes:

> Shared hosting enviroments. I work for a web hosting company that offers
> MySQL to all of its customers, our MySQL server has several thousand
> databases on it, and I must say it works exceptionally well.
>
> Creating users/databases/changing passwords is as simple as sending it a
> couple queries from our Customer web interface, trouble shooting poor
> queries takes seconds when using "mytop" (mtop), and tracking/billing
> for disk usage is as simple as running "du /var/lib/mysql/*". I would
> like to say the same things for PG, but I'm affrid I can't.

At least in the latest versions, things are quite easy.

User/database administration?
CREATE USER someuser ENCRYPTED PASSWORD '...' NOCREATEDB NOCREATEUSER;
CREATE DATABASE someuser OWNER someuser ENCODING 'UNICODE';

Disk usage account? Use contrib/dbsize (README for easy setup)
SELECT database_size('someuser');
Done.

Poor queries -> query stats?

Of course, some things are easier in MySQL. On the other hand, what about
InnoDB, "du /var/lib/mysql/*" won't help much...

I just wanted to show that PostgreSQL administration is not that hard in a
hosting environment.

Regards,
Michael Paesold


Tue, 15 Apr 2003 23:51:00 +0200
"Michael Paesold" <mpaesold@gmx.at>
Re: Are we losing momentum?
--- Neil Conway <neilc@samurai.com> wrote:
> Are you talking about queries between databases on the same
> postmaster
> (i.e. running under the same PostgreSQL installation),

Yes

> or queries
> between postmasters running on different systems? If the former, I
> don't
> see how putting your data into multiple schemas in a single database
> is
> significantly less reliable than putting it into multiple databases.

I disagree. For example, suppose you have
app12 that uses db1 and db2,
app23 that uses db2 and db3,
app3 that uses db3.

If db3 goes down then app12 is not affected, app23 could be partially
affected (e.g. user may not be able to run historic queries) and app3
is completely unavailable. This is definitely better than all three
apps are down. Besides, having one huge db makes everything more
difficult and requires (much) more time for backups, restores, etc.

Every major RDBMS vendor (and mySql) finds this feature important and
they support it. Hope Postgresql will too.






__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com


Tue, 15 Apr 2003 14:58:24 -0700 (PDT)
ow <oneway_111@yahoo.com>
Re: Are we losing momentum?
ow <oneway_111@yahoo.com> writes:
> --- Neil Conway <neilc@samurai.com> wrote:
>> Are you talking about queries between databases on the same
>> postmaster

> Yes

> [snip]

> If db3 goes down then app12 is not affected, app23 could be partially
> affected (e.g. user may not be able to run historic queries) and app3
> is completely unavailable.

This is nonsense.  There is no scenario where one DB "goes down" and
other DBs on the same postmaster remain up.  There are advantages to
having separate DBs on one postmaster (like separate copies of the
system catalogs), but there's very little reliability differential
compared to a multi-schema approach.

			regards, tom lane


Tue, 15 Apr 2003 19:28:27 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Are we losing momentum? 

--- Tom Lane <tgl@sss.pgh.pa.us> wrote:
> This is nonsense.  There is no scenario where one DB "goes down" and
> other DBs on the same postmaster remain up.  There are advantages to
> having separate DBs on one postmaster (like separate copies of the
> system catalogs), but there's very little reliability differential
> compared to a multi-schema approach.

Perhaps "goes down" is not the best term. You can replace it with "is
not available" (as in being restored, etc) if you like. 




__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com


Tue, 15 Apr 2003 16:56:24 -0700 (PDT)
ow <oneway_111@yahoo.com>
Re: Are we losing momentum? 
On the "lets make more apps work with Postgres" front people can check out
http://sourceforge.net/projects/bind-dlz

This is a patch for Bind 9.2.1 that allows all DNS data to be stored in an
external database.  Makes DNS administration easy, and changes to DNS data
are reflected immediately.  The project supports multiple databases now, but
the first one was postgres!

Later
Rob


Tue, 15 Apr 2003 20:08:44 -0400
"Rob Butler" <robert.butler5@verizon.net>
Re: Are we losing momentum? 
ow said...
> --- Neil Conway <neilc@samurai.com> wrote:
> > Are you talking about queries between databases on the same
> > postmaster
> > (i.e. running under the same PostgreSQL installation),
> 
> Yes

Based on your later comments, the answer seems to /actually/ be "No."

> > or queries
> > between postmasters running on different systems? If the former, I
> > don't
> > see how putting your data into multiple schemas in a single database
> > is
> > significantly less reliable than putting it into multiple databases.
> 
> I disagree. For example, suppose you have
> app12 that uses db1 and db2,
> app23 that uses db2 and db3,
> app3 that uses db3.
> 
> If db3 goes down then app12 is not affected, app23 could be partially
> affected (e.g. user may not be able to run historic queries) and app3
> is completely unavailable. This is definitely better than all three
> apps are down. Besides, having one huge db makes everything more
> difficult and requires (much) more time for backups, restores, etc.
> 
> Every major RDBMS vendor (and mySql) finds this feature important and
> they support it. Hope Postgresql will too.

If it's all running as just one PostgreSQL instance, then if db1 goes down, 
then, since it's the same postmaster as is supporting db2 and db3, they 
necessarily go down as well.

The only way that you get to take down one DB without affecting the others is 
for them NOT to be running as part of the same PG installation.

By the way, if you only have one PG instance, then you may very well find it 
challenging to suitably parallelize all the loads/dumps of data.  If you have 
three disks, or three arrays, it may make a lot of sense to have separate PG 
instances on each one, as that allows I/O to not need to interfere between 
instances.  (There are, admittedly, other ways of tuning this sort of thing, 
such as moving WAL to a separate disk, or perhaps even specific table files, 
identified by OID...)

But the most general ways of separating things out lead to having quite 
separate DB instances.  And when you've got that, it certainly is attractive 
to have 2PC, as is available for the "expensive guys."
--
output = reverse("gro.mca@" "enworbbc")
http://www3.sympatico.ca/cbbrowne/sap.html
You  know  that  little  indestructible  black box  that  is  used  on
planes---why  can't  they  make  the  whole  plane  out  of  the  same
substance?


Tue, 15 Apr 2003 20:28:25 -0400
cbbrowne@cbbrowne.com
Re: Are we losing momentum? 
> Tatsuo, this has always fascinated me.  Any insights you could share about how PostgreSQL achieved the prominence it has in Japan (and how MySQL did not) would be very interesting.

PostgreSQL started to become popular in 1998(PostgreSQL 6.4 days). In
the year a publisher asked me to write the first PostgreSQL book and
fortunately it has sold very well. From then many PostgreSQL books
have been published and lots of magazine articles have been written
too. As as result, PostgreSQL users could enjoy rich PostgreSQL
information in Japanese. Since most Japanese (including me) is not
very good at English, localized docs for PostgreSQL is the key factor
for the "prominence". On the other hand, almost no good Japanese MySQL
books have ever appeared.

Next point is the community. Japan PostgreSQL Users Group (JPUG) has
been established in 1999 and now has over 1800 registered members
(local ML for PostgreSQL has over 5400 subscribers). I guess MySQL
does not have this kind large community.

These are not proven factors for the popularity of PostgreSQL in
Japan, I believe they definitely could be listed as one of the top 10
reasons.
--
Tatsuo Ishii


Wed, 16 Apr 2003 14:43:08 +0900 (JST)
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: PostgreSQL vs. MySQL in Japan
On Wednesday 16 April 2003 00:26, Mike Benoit wrote:
> From my experience, almost every time I talk to a MySQL supporter about
> PostgreSQL, the whole "vacuum" issue always seems to come up. Some way
> to get vacuum automated (and thus out of sight, out of mind) I think
> would make great strides in making PG at least "seem" more friendly to
> someone on the outside.

Agreed. But that is not an impossible issue for a DBA, is it? I mean some 
learning is required but that can be done.

> Creating users/databases/changing passwords is as simple as sending it a
> couple queries from our Customer web interface, trouble shooting poor
> queries takes seconds when using "mytop" (mtop), and tracking/billing
> for disk usage is as simple as running "du /var/lib/mysql/*". I would
> like to say the same things for PG, but I'm affrid I can't.

Adding users, databases, password changes are as easy in postgresql. Tracking 
disk usage is no different in postgresql barring additional step of using 
oid2name to find out directory you want to du.

In fact I think postgresql is easier to use. Till date, I could never start 
mysql by hand and get it behave sanely. pg_ctl or nohup postmaster has always 
worked for me.

Besides postgresql is true to it's resource usage. You allocate 128MB of 
shared buffers, and they are consumed. You stop postmaster and all the 
buffers are back to system. With mysql, I found that large amount of memory 
was never returned to system even after service shutdown. I hate black-boxes 
on my system where I can not fathom into. Had to reboot the machine. 

> I think it all comes down to how simple PG is to setup and use on a
> daily basis. This is what determines the size of its community. Even
> just the simple things make a big difference. ie:
>
> \dt
>
> compared to:
>
> show tables;

<I assume that show tables is not a standard SQL syntax>

That is very shallow view. \dt is a postgresql terminal client extension where 
as show tables is part of mysql SQL offerings. Such brutal twisting of SQL 
standards encourages dependence on mysql only features, flushing standard 
compliance down the drain.

> Yes, once you get over the "hump" PG is quite efficient, but you need to
> understand it, and learn some small quriks first. With MySQL, you can
> pretty much guess commands, and they often work! Not as much luck with
> PG.
>
> show indexes
> show processlist
> show columns from <table>
>
> These are all easy/simple commands that make sense to someone who is
> just learning the ropes. Short abbreviated, commands are great for the
> experts, but can greatly discourage newbies.

Well, I might get flamed for this but let me clarify. I am not against 
newbies. Everybody once was a newbie. But being a newbie, does not justify 
reluctance to go thr. manuals. If you are reluctant to go thr. manuals., you 
better hire a commercial support.

My advise has always been ,to read postgresql manual start to end before even 
touching it. It takes a day to digest but pays off big later. When I started 
postgresql back in 1999, I started on postgresql and SQL simalteneously. 
Didn't have faintest idea, what any of those stand for. So I read the manual, 
start to end in couple of days. In one day I could do things that worked as 
expected.

RTFM is not an advice thrown to kick out newbies. It is ground fact that 
everybody has to suffer thr. Borg transplants are not yet available here.

 Shridhar


Wed, 16 Apr 2003 12:50:33 +0530
Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
Re: Are we losing momentum?
On Tuesday 15 April 2003 22:25, Bruce Momjian wrote:
> Shridhar Daithankar wrote:
> > That is an unfair comparison. The technical differences between BSD and
> > linux are not as much as postgresql and mysql. Besides what is the
> > parallel of SQL standard in OS world? POSIX? And both BSD/linux are doing
> > fine sitting next to each other on that.
>
> Agreed, Linux and BSD are pretty close --- but Linux used to be behind
> BSD --- they caught up because both are open source.  The big question
> is whether MySQL (which isn't openly developed) will catch up to
> PostgreSQL.  And if they do catch up, will we have mind share parity by
> that time?

That is a tough question. But if we focus on enterprise features and reach 
threshold in decision making circles, that would be great.

Mind share parity certainly matters. Bigger question is in which circles. I 
would better put decision making circle as fist target.

Besides we won't sit still while mysql catches with us. 

 Shridhar


Wed, 16 Apr 2003 13:25:14 +0530
Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
Re: Are we losing momentum?
> BTW, DB2 doesn't have 'em [cross-db queries] either.
> In DB2, you can of course have cross-schema queries but no cross-db
> queries, unless you rig up the federated functionality to connect one
> db to the other.

A while ago I was at a client who wanted to migrate to DB2 and this
questions was raised during discussions with IBM. There was a way to do
this, if I remember correctly the solution involved creating views for
all tables from db2 that you wanted to use in db1 and maybe something
else. Can't tell you for sure, I'm not working with DB2.

Oracle, Sybase, Ms, Informix (? AFAIK) , mySql, they all support
cross-db queries.

Anyway, I thought it was important to bring this up. With large number
of apps and large amount of data having everything in one db is a sure
way for disaster, IMHO.







__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com


Wed, 16 Apr 2003 05:20:17 -0700 (PDT)
ow <oneway_111@yahoo.com>
Re: Are we losing momentum? 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> In fact I think postgresql is easier to use. Till date, I could never start 
> mysql by hand and get it behave sanely. pg_ctl or nohup postmaster has always 
> worked for me.

This is weird, because despite mysql's technical inferiority, it really is 
pretty simple to use. Also seems a little hypocritical of you in light of 
the RTFM rant later on in your email. :)


> Besides postgresql is true to it's resource usage. You allocate 128MB of 
> shared buffers, and they are consumed. You stop postmaster and all the 
> buffers are back to system. With mysql, I found that large amount of memory 
> was never returned to system even after service shutdown. I hate black-boxes 
> on my system where I can not fathom into. Had to reboot the machine.

"Black-boxes"? It's open-source, just like we are. Did you read their manual 
"start to end"? Did you ask on their mailing lists? I'm no MySQL fan, but 
I'd rather let them, not us, dish out the FUD. The original poster had some 
valid points (auto-vacuum and non-intuitive commands) that still need 
addressing, IMO.


- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200304160945

-----BEGIN PGP SIGNATURE-----
Comment: http://www.turnstep.com/pgp.html

iD8DBQE+nV/EvJuQZxSWSsgRAsaXAKCAY3vGFxDzk9dniqojpi+RK3ToUwCgpv5L
Sl6e9Or440U5QeLIhvNsaro=
=k5Np
-----END PGP SIGNATURE-----


Wed, 16 Apr 2003 13:51:28 -0000
greg@turnstep.com
Re: Are we losing momentum?
On Wednesday 16 April 2003 19:21, greg@turnstep.com wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> > In fact I think postgresql is easier to use. Till date, I could never
> > start mysql by hand and get it behave sanely. pg_ctl or nohup postmaster
> > has always worked for me.
>
> This is weird, because despite mysql's technical inferiority, it really is
> pretty simple to use. Also seems a little hypocritical of you in light of
> the RTFM rant later on in your email. :)

Yes. That is correct. But going thr. 3.5MB html to find out things which has 
got tons of options and figuring out interdependencies by trial and error is 
not a good job at that. Whoever thinks that such a style on manual writing is 
good, needs an attitude readjustment. Postgresql manual is ten times better.

> > Besides postgresql is true to it's resource usage. You allocate 128MB of
> > shared buffers, and they are consumed. You stop postmaster and all the
> > buffers are back to system. With mysql, I found that large amount of
> > memory was never returned to system even after service shutdown. I hate
> > black-boxes on my system where I can not fathom into. Had to reboot the
> > machine.
>
> "Black-boxes"? It's open-source, just like we are. Did you read their
> manual "start to end"? Did you ask on their mailing lists? I'm no MySQL
> fan, but I'd rather let them, not us, dish out the FUD. The original poster
> had some valid points (auto-vacuum and non-intuitive commands) that still
> need addressing, IMO.

I didn't go to any mailing list. My point is, if I pierce the startup-shutdown 
chapter in mysql manual and can not get it working by hand, either I am 
stupid or something wrong with mysql. May sound arrogant but I count on 
later.

Have you seen postgresql 101 I wrote? It is at 
http://wiki.ael.be/index.php/PostgresQL101.  It is that simple with 
postgresql. Now this is not the forum but can anybody point me to similar 
document for mysql. /etc/rc.d/init.d/mysql start always works but it does not 
allow me to tweak options for mysqld which is first thing I want.

Anyway I must admit that I was reluctant to use mysql and was turned off 
pretty quickly. Mine is probably a irreproducible bug but I did encounter it.

 Shridhar


Wed, 16 Apr 2003 19:46:37 +0530
Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
Re: Are we losing momentum?
greg@turnstep.com kirjutas K, 16.04.2003 kell 16:51:
> The original poster had some 
> valid points (auto-vacuum and non-intuitive commands) that still need 
> addressing, IMO.

As of 7.3 (or was it 7.2) auto-vacuum is just one line in crontab. In
many scenarios it can be left running continuously with very little
effect on performance. In others it must be run nightly, but having it
kick in at unexpected times may not be what you want at all. So it has
to be configured for good performance weather it is built-in or run in a
separate backend process.

And I can't see how "show tables" is more intuitive than "\dt" - I
expected it to be "list tables" or "tablelist" or "näita tabeleid" .

Once you have found \? it is all there (and you are advised to use \? at
psql startup).

That may also be why PostgreSQL is more popular in Japan - if one has to
remember nonsensical strings, then it is easier to remember short ones
;)

----------------
Hannu


16 Apr 2003 17:17:47 +0300
Hannu Krosing <hannu@tm.ee>
Re: Are we losing momentum?
> > That's a pretty reasonable thought. I work for a shop that sells
> > Postgres support, and even we install MySQL for the Q&D ticket
> > tracking system we recommend because we can't justify the cost to
> > port it to postgres. If the postgres support were there, we would
> > surely be using it.
> 
> > How to fix such a situation, I'm not sure. "MySQL Compatability
> > Mode," anyone? :-)
> 
> What issues are creating a compatibility problem for you?

I don't think these should be hacked into the backend/libpq, but I
think it'd be a huge win to hack in "show *" support into psql for
MySQL users so they can type:

     SHOW (databases|tables|views|functions|triggers|schemas);

I have yet to meet a MySQL user who understands the concept of system
catalogs even though it's just the 'mysql' database (this irritates me
enough as is)... gah, f- it: mysql users be damned, I have three
developers that think that postgresql is too hard to use because they
can't remember "\d [table name]" and I'm tired of hearing them bitch
when I push using PostgreSQL instead of MySQL.  I have better things
to do with my time than convert their output to PostgreSQL.  Here goes
nothing...

I've tainted psql and added a MySQL command compatibility layer for
the family of SHOW commands (psql [-m | --mysql]).


The attached patch does a few things:

1) Implements quite a number of SHOW commands (AGGREGATES, CASTS,
   CATALOGS, COLUMNS, COMMENTS, CONSTRAINTS, CONVERSIONS, DATABASES,
   DOMAINS, FUNCTIONS, HELP, INDEX, LARGEOBJECTS, NAMES, OPERATORS,
   PRIVILEGES, PROCESSLIST, SCHEMAS, SEQUENCES, SESSION, STATUS,
   TABLES, TRANSACTION, TYPES, USERS, VARIABLES, VIEWS)

   SHOW thing
   SHOW thing LIKE pattern
   SHOW thing FROM pattern
   SHOW HELP ON (topic || ALL);
   etc.

   Some of these don't have \ command eqiv's.  :( I was tempted to add
   them, but opted not to for now, but it'd certainly be a nice to
   have.

2) Implements the necessary tab completion for the SHOW commands for
   the tab happy newbies/folks out there.  psql is more friendly than
   mysql's CLI now in terms of tab completion for the show commands.

3) Few trailing whitespace characters were nuked

4) guc.c is now in sync with the list of available variables used for
   tab completion


Few things to note:

1) SHOW INDEXES is the same as SHOW INDEX, I think MySQL is wrong in
   this regard and that it should be INDEXES to be plural along with
   the rest of the types, but INDEX is preserved for compatibility.

2) There are two bugs that I have yet to address

   1) SHOW VARIABLES doesn't work, but "SHOW [TAB][TAB]y" does
   2) "SHOW [variable_of_choice];" doesn't work, but "SHOW
      [variable_of_choice]\n;" does work... not sure where this
      problem is coming from

3) I think psql is more usable as a result of this more verbose
   syntax, but it's not the prettiest thing on the planet (wrote a
   small parser outside of the backend or libraries: I don't want to
   get those dirty with MySQL's filth).

4) In an attempt to wean people over to PostgreSQL's syntax, I
   included translation tips on how to use the psql equiv of the SHOW
   commands.  Going from SHOW foo to \d foo is easy, going from \d foo
   to SHOW foo is hard and drives me nuts.  This'll help userbase
   retention of newbies/converts.  :)

5) The MySQL mode is just a bounce layer that provides different
   syntax wrapping exec_command() so it should provide little in the
   way of maintenance headaches.  Some of the SHOW commands, however,
   don't have \ couterparts, but once they do and that code is
   centralized, this feature should come for zero cost.

6) As an administrator, I'd be interested in having an environment
   variable that I could set that'd turn on MySQL mode for some of my
   bozo users that way they don't complain if they forget the -m
   switch.  Thoughts?


I'll try and iron out the last of those two bugs/features, but at this
point, would like to see this patch get wider testing/feedback.
Comments, as always, are welcome.

PostgreSQL_usability++

-sc

-- 
Sean Chittenden
Wed, 16 Apr 2003 08:45:17 -0700
Sean Chittenden <sean@chittenden.org>
Re: Are we losing momentum?


> -----Original Message-----
> From: Rob Butler [mailto:robert.butler5@verizon.net] 
> Sent: 16 April 2003 16:33
> To: pgsql-hackers@postgresql.org
> Subject: [HACKERS] Many comments (related to "Are we losing 
> momentum?") 
> 
> 
> You 
> want a real shocker...  Visit the gborg homepage (click the 
> gborg link on the bottom left corner of the postgres site).  
> Then read the news on the right side of the gborg homepage.  
> See that url for http://www.greatbridge.org/ in the 
> GreatBridge.Org Version 2.0.0 Release story from way back in 
> 2001!  Go ahead, visit that URL.  What's the first thing you 
> see on that page?  "Purchase IBM DB2 At IBM.com".  Come on!  
> Not even a mention of Postgres, and this is right on the 
> gborg homepage!  

We don't own that site, and we never did. We did manage to inherit the
projects database through it's key programmer and his goodwill.

> The firebird, sapdb and mysql sites are 
> killing postgres here.  The postgres homepage and related 
> links is the first thing someone new to postgres sees!  There 
> shouldn't be news on any of the main pages from back in 2001. 

There isn't. The oldest news is from January 2003.

Wrt your comments on the style of some of the pages linked off the main
site (the archives spring to mind), if there are any volunteers to help
fix that, please raise your hands because my time is limited and I could
do with some committed help on that sort of thing.

Regards, Dave.


Wed, 16 Apr 2003 16:51:17 +0100
"Dave Page" <dpage@vale-housing.co.uk>
Re: Many comments (related to "Are we losing momentum?") 
Sean Chittenden <sean@chittenden.org> writes:
> 4) guc.c is now in sync with the list of available variables used for
>    tab completion

AFAIK, the GUC variables not listed in tab-complete.c were omitted
deliberately.  We could have a discussion about the sensefulness of
those decisions, but please do not consider it a bug to be fixed 
out-of-hand.

> 6) As an administrator, I'd be interested in having an environment
>    variable that I could set that'd turn on MySQL mode for some of my
>    bozo users that way they don't complain if they forget the -m
>    switch.  Thoughts?

Can't you set it in ~/.psqlrc ?

			regards, tom lane


Wed, 16 Apr 2003 11:53:07 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Are we losing momentum? 
> > 4) guc.c is now in sync with the list of available variables used for
> >    tab completion
> 
> AFAIK, the GUC variables not listed in tab-complete.c were omitted
> deliberately.  We could have a discussion about the sensefulness of
> those decisions, but please do not consider it a bug to be fixed
> out-of-hand.

Alright, there weren't many omitted GUC's, but those that were
omitted did have counterparts that were include already so I figured
there was some bit rot going on.

> > 6) As an administrator, I'd be interested in having an environment
> >    variable that I could set that'd turn on MySQL mode for some of my
> >    bozo users that way they don't complain if they forget the -m
> >    switch.  Thoughts?
> 
> Can't you set it in ~/.psqlrc ?

Hrm...  ah, ok, done.  Patch updated.

http://people.freebsd.org/~seanc/patches/patch_postgresql-HEAD::src::bin::psql

-sc

-- 
Sean Chittenden


Wed, 16 Apr 2003 09:17:25 -0700
Sean Chittenden <sean@chittenden.org>
Re: [HACKERS] Are we losing momentum?
On Wed, 2003-04-16 at 11:51, Dave Page wrote:
> > -----Original Message-----
> > From: Rob Butler [mailto:robert.butler5@verizon.net] 
> > Sent: 16 April 2003 16:33
> > To: pgsql-hackers@postgresql.org
> > Subject: [HACKERS] Many comments (related to "Are we losing 
> > momentum?") 
> > 
> > 
> > You 
> > want a real shocker...  Visit the gborg homepage (click the 
> > gborg link on the bottom left corner of the postgres site).  
> > Then read the news on the right side of the gborg homepage.  
> > See that url for http://www.greatbridge.org/ in the 
> > GreatBridge.Org Version 2.0.0 Release story from way back in 
> > 2001!  Go ahead, visit that URL.  What's the first thing you 
> > see on that page?  "Purchase IBM DB2 At IBM.com".  Come on!  
> > Not even a mention of Postgres, and this is right on the 
> > gborg homepage!  
> 
> We don't own that site, and we never did. We did manage to inherit the
> projects database through it's key programmer and his goodwill.
> 

Oddly I don't see the link he's referring to, but I've always thought it
unfortunate that the postgresql community didn't hold onto those domain
names. I was just speaking with a tech writer last night who mentioned
how the company that was initially behind postgresql went kaput ("I
think it was called great river or something?") Took me a minute to set
him straight on that.  

Anyways, I've offered to look into setting up gforge services in the
past as it's a mature project with a good development community and the
founder (Tim Perdue) has always been friendly with the PostgreSQL
community. Course I don't see this as an immediate need, but long term I
think it would be a good idea.

> > The firebird, sapdb and mysql sites are 
> > killing postgres here.  The postgres homepage and related 
> > links is the first thing someone new to postgres sees!  There 
> > shouldn't be news on any of the main pages from back in 2001. 
> 
> There isn't. The oldest news is from January 2003.
> 
> Wrt your comments on the style of some of the pages linked off the main
> site (the archives spring to mind), if there are any volunteers to help
> fix that, please raise your hands because my time is limited and I could
> do with some committed help on that sort of thing.
> 

I'm in the process of adding the newsletters to the main page (expect
questions your way later today ;-) but once I am done with that
"cohesiveness" will definitely be on my radar screen. 

Robert Treat


16 Apr 2003 12:21:55 -0400
Robert Treat <xzilla@users.sourceforge.net>
Re: Many comments (related to "Are we losing momentum?")
Sean Chittenden <sean@chittenden.org> writes:
> 4) guc.c is now in sync with the list of available variables used for
> tab completion
>> 
>> AFAIK, the GUC variables not listed in tab-complete.c were omitted
>> deliberately.  We could have a discussion about the sensefulness of
>> those decisions, but please do not consider it a bug to be fixed
>> out-of-hand.

> Alright, there weren't many omitted GUC's, but those that were
> omitted did have counterparts that were include already so I figured
> there was some bit rot going on.

There could be some of that too.  I was just saying that it's not a
foregone conclusion to me that every parameter known to guc.c should
be in the tab completion list.

			regards, tom lane


Wed, 16 Apr 2003 12:26:58 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Are we losing momentum? 
Robert Treat <xzilla@users.sourceforge.net> writes:
> Oddly I don't see the link he's referring to, but I've always thought it
> unfortunate that the postgresql community didn't hold onto those domain
> names.

We were not offered the chance; Landmark wanted to hold onto the Great
Bridge trademark, apparently.  We did come away with the postgres.com,
postgres.org, and postgresql.com domains, which GB had managed to wrestle
away from some korean domain squatter.

The fact that some of those aren't currently resolving nicely (eg,
www.postgres.org gets you a Horde login page) is our own darn fault.
As Dave Page was mentioning, some additional hands in the webpage
maintenance effort would be great.

			regards, tom lane


Wed, 16 Apr 2003 13:29:01 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Many comments (related to "Are we losing momentum?") 
On Wed, 2003-04-16 at 13:29, Tom Lane wrote:
> Robert Treat <xzilla@users.sourceforge.net> writes:
> > Oddly I don't see the link he's referring to, but I've always thought it
> > unfortunate that the postgresql community didn't hold onto those domain
> > names.
> 
> We were not offered the chance; Landmark wanted to hold onto the Great
> Bridge trademark, apparently.  We did come away with the postgres.com,
> postgres.org, and postgresql.com domains, which GB had managed to wrestle
> away from some korean domain squatter.
> 

Sorry. I knew that but I guess my statement implied different.

> The fact that some of those aren't currently resolving nicely (eg,
> www.postgres.org gets you a Horde login page) is our own darn fault.
> As Dave Page was mentioning, some additional hands in the webpage
> maintenance effort would be great.
> 

Well, that's more of a DNS issue than a webpage, and afaik Marc is the
only one who can make that change. If he's willing to change it now by
all means let's do so. If he's too busy but willing to give someone else
access to do it I'm sure we can dig someone up (like me)

Robert Treat


16 Apr 2003 14:32:32 -0400
Robert Treat <xzilla@users.sourceforge.net>
Re: Many comments (related to "Are we losing momentum?")
Glad to see all the comments about the web page stuff, and see that people
recognize the need for a cohesive look and feel for Postgres.org.  Do people
have any comments about the rest of the stuff in my post?

Like, are the JDBC developers aware that you will be changing the FE/BE
protocol?  Do they know you will (possibly) be adding 2PC?

Any communication going on between core and postgres-r developers to make
sure that replication and 2PC will work together simultaneously as I
described in my earlier e-mail?

Any comments about the section on competition?  Do people agree / disagree
that postgres should not be competing with MySQL (because it's no real
competition) or does everyone think MySQL is a real threat?  Do people agree
/ disagree that the real competition is commercial DB's and Firebird / SAP
db?

Later
Rob


Wed, 16 Apr 2003 16:50:47 -0400
"Rob Butler" <robert.butler5@verizon.net>
Re: Many comments (related to "Are we losing momentum?")

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> Any comments about the section on competition?  Do people agree / disagree
> that postgres should not be competing with MySQL (because it's no real
> competition) or does everyone think MySQL is a real threat?  Do people 
> agree / disagree that the real competition is commercial DB's and Firebird 
> / SAP db?

There has been plenty of discussion on the advocacy list about this: it is 
a much better place for this sort of talk than hackers anyway.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200304161743
-----BEGIN PGP SIGNATURE-----
Comment: http://www.turnstep.com/pgp.html

iD8DBQE+nc6+vJuQZxSWSsgRAmt/AJwKpkYWubiF+fz0mgZoXSIaMBpASACcDgjl
9Ks9DmXP+SjkzHA7DIMF0q8=
=qmPX
-----END PGP SIGNATURE-----


Wed, 16 Apr 2003 21:44:55 -0000
greg@turnstep.com
Re: Many comments (related to "Are we losing momentum?")


Bruce wrote:

>Having good reference sites is important, and I could list as many
>impressive ones as MySQL, but who has time to hunt around and get
>permission to list them --- I will tell you who --- the MySQL marketing
>guys, while the PostgreSQL guys don't.  :-(

Is it a good enough benefit to make the ones we already
have easier to find?

If the content on these pages:

  http://techdocs.postgresql.org/techdocs/supportcontracts.php
  http://advocacy.postgresql.org/casestudies/
  http://archives.postgresql.org/pgsql-announce/2002-11/msg00004.php

could be integrated and put on an easy to find page in the 
advocacy area it'd be a lot easier for new people to see.


I know PostgreSQL's got at least as impressive a list as MySQL.  It's
just that you need to dig harder to find it.

    Ron


Wed, 16 Apr 2003 15:38:31 -0700
"Ron Mayer" <ron@intervideo.com>
Re: Are we losing momentum?
"Rob Butler" <robert.butler5@verizon.net> writes:
> Like, are the JDBC developers aware that you will be changing the FE/BE
> protocol?

The ones who have been participating in the discussion are ;-)

> Do they know you will (possibly) be adding 2PC?

The odds of that appearing in 7.4 are nil, IMHO.

			regards, tom lane


Wed, 16 Apr 2003 23:10:07 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Many comments (related to "Are we losing momentum?") 
Hi Kevin

> -----Original Message-----
> From: Kevin Brown [mailto:kevin@sysexperts.com] 
> Sent: 17 April 2003 03:35
> To: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] pg_clog woes with 7.3.2 - Episode 2
> 
> I'd also start looking carefully through the system logs for 
> SCSI errors.  You should see some if you're getting bad block 
> problems (in particular, you should see bad block remapping 
> attempts that couldn't read the data from the original bad 
> block -- this, or running out of spare blocks, is the only 
> reason you should see errors at all on an otherwise functional setup).

No errors on the system at all apart from PostgreSQL. On the previous
disk, the number of errors reported by badblocks was rising from 64
initially to 80-something when I took it out.

> If badblocks shows errors but you don't see any SCSI errors 
> in the system logs, then it's time to start suspecting the 
> disk controller or perhaps even the PCI bus controller, 
> because it means something really weird is happening on the 
> backend that is entirely invisible.  Cabling or termination 
> could be an issue, but I'd expect to see parity errors, timed 
> out commands, etc. if that's the problem.

Yes, me too. Still, I've now tried a 29160N adaptor, and changed the
cable to a DPT one (both that and the previous Adaptec have hardwired
terminators). I've also removed the DAT drive from the system so the
only things on the SCSI bus are 2 identical disks and the adaptor.

I'm beginning to wonder about my shiny new 2.4.20 kernel...

Regards, Dave.


Thu, 17 Apr 2003 08:32:40 +0100
"Dave Page" <dpage@vale-housing.co.uk>
Re: pg_clog woes with 7.3.2 - Episode 2
I am not a hacker of PgSQL, and new to Databases.  I was using MySQL 
under .NET, but was annoyed by their agressive licence agreements and 
immaturity.  (Their sales personel are also very rude.  One girl once 
told me that if I didn't like their licence terms I should just use 
flat-files instead.)  One of the .NET developers for MySQL advised me to 
look at PostgreSQL, and I have never looked back.

This was not the fist time I looked at PostgreSQL, I initially looked at 
30+ Databases, but rejected PostgreSQL off hand.  There was no Windows 
version.

Speaking stricktly as an ameture, I would like to make a few comments I 
could not see mensioned in this thread.  This is not a dig, more a wish 
list!

Because: With the expanding industry and popularity of IT as part of 
unrelated collage courses (Engineering, Accountancy etc), there are lots 
of ametures and newbe's.  We will never be first class hackers, we will 
probably never get to the end of the manuals or understand what P2C / 
FE/BE is or where it's used.  But we are the silent magority.  (I am 
personally extreamly dyslexic.  I learn from example, talking and brief 
painful trips into the documentation archives.)

Therefore we learn as much as we need to know.  In time I am sure we all 
want to be guru's in everything.  I have lots of ameture friends at this 
level, running ISP's, producing commercial applications, in unrelated 
research, needing office systems...  All needing a DB.  Most using 
MS-SQL or MySQL.

To draw on a popular examples, MySQL helps the ameture:  (I'm putting my 
foot in it that some of this probably exists.  I just haven't found it yet.)

-	A true Windows version which people can learn their craft on.
-	Tools which look like Access, to do row level data editing with no SQL.
-	Easy to use and remember command extensions, like 'CREATE IF NOT 
EXISTS', 'DROP IF EXISTS' which are universal.
-	Centrally located complete documentation in many consistent  easy to 
read formats, of the system and *ALL* API's, including in-line tutorials 
and examples.
-	Data types like 'ENUM' which appeal to ametures.
-	There are no administrative mandatorys.  Eg, VACUUM.  (A stand-alone 
commercial app, like an Email client, will be contrainted by having to 
be an app and a DBA in one.)
-	The tables (not innodb) are in different files of the same name. 
Allowing the OS adminitrator great ability.  EG, putting tables on 
separate partitions and therefore greatly speeding performance.
-	They have extensive backup support.  Including now, concurrent backup 
without user interuption or risk of inconsistency.

Now I have begun to climb the ladder a bit, I know this it of little 
importance compared to working referential constraints, triggers, 
procedures and transactions...  You also have the excelent mailing list 
'novice', with excelent support for Ametures, with the most friendly 
welcome note: 'No problem too minor'!  Thanks to you all for providing 
the system I am now beginning to enjoy using.

PS: I like the '\dt'.  Especially the way it can be used half way 
through a true statement, inspired bit of genious there.

Ben


Thu, 17 Apr 2003 09:45:29 +0000
Ben Clewett <B.Clewett@roadrunner.uk.com>
For the ametures.  (related to "Are we losing momentum?")
On Wed, 16 Apr 2003 greg@turnstep.com wrote:

> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> > In fact I think postgresql is easier to use. Till date, I could never start 
> > mysql by hand and get it behave sanely. pg_ctl or nohup postmaster has always 
> > worked for me.
> 
> This is weird, because despite mysql's technical inferiority, it really is 
> pretty simple to use. Also seems a little hypocritical of you in light of 
> the RTFM rant later on in your email. :)

I hate to join in this thread but...

I don't find it weird. It's probably a different mind set or something but I
find the MySQL documentation discussing something that will be in version 8.34
when they still list 3.23 as the latest production version is so confusing when
it's written with no indication that the thing isn't already in place.

Just my own view. People say MySQL is easy and PostgreSQL is difficult to
learn. I say PostgreSQL is easy and MySQL is difficult to learn.

And as for it being maintenance free while a regular vacuum is something too
difficult a concept for people to grasp. Well, what do these maintenance free
MySQL folk do with the regular tasks that MySQL needs run?


--
Nigel Andrews


Thu, 17 Apr 2003 12:35:07 +0100 (BST)
"Nigel J. Andrews" <nandrews@investsystems.co.uk>
Re: Are we losing momentum?
Sean Chittenden writes:

> I don't think these should be hacked into the backend/libpq, but I
> think it'd be a huge win to hack in "show *" support into psql for
> MySQL users so they can type:
>
>      SHOW (databases|tables|views|functions|triggers|schemas);

Well, we (will) have the information schema, and if you like you can put
it in the path and write

select * from tables;

etc., which seems just as good.

-- 
Peter Eisentraut   peter_e@gmx.net


Thu, 17 Apr 2003 14:45:26 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Are we losing momentum?
On Thursday 17 April 2003 13:35, Nigel J. Andrews wrote:

> I hate to join in this thread but...

me too, but I am suffering from a bout of MySQL :-(

(...)
> Just my own view. People say MySQL is easy and PostgreSQL is difficult to
> learn. I say PostgreSQL is easy and MySQL is difficult to learn.

Having had to use MySQL seriously for the first time for a long time, I am finding
it makes the easy things (appear) easy and the difficult things impossible.
For example, AUTO_INCREMENT is easy to set up and use, but 
is a toy feature compared to real sequences...

> And as for it being maintenance free while a regular vacuum is something
> too difficult a concept for people to grasp. Well, what do these
> maintenance free MySQL folk do with the regular tasks that MySQL needs run?

This is what MySQL recommends:
http://www.mysql.com/doc/en/Maintenance_regimen.html

How about repackaging VACUUM as a "database defragmentation
utility"?  After all many many people have come to accept
disk defragmenters as an essential part of their OS ;-) 


Ian Barwick
barwick@gmx.net


Thu, 17 Apr 2003 14:50:47 +0200
Ian Barwick <barwick@gmx.net>
Re: Are we losing momentum?
Peter Eisentraut <peter_e@gmx.net> writes:
> Sean Chittenden writes:
>> I don't think these should be hacked into the backend/libpq, but I
>> think it'd be a huge win to hack in "show *" support into psql for
>> MySQL users so they can type:
>> 
>> SHOW (databases|tables|views|functions|triggers|schemas);

> Well, we (will) have the information schema, and if you like you can put
> it in the path and write
> select * from tables;
> etc., which seems just as good.

I think Sean's idea is not to make it "just as easy as MySQL", it's to
make it "the *same* as MySQL", for the benefit of those that refuse to
learn differently.  Them as won't adjust to "\dt" in place of "show
tables" aren't likely to adjust to "select * from tables" either.
Not even (maybe especially not) if it's arguably a standard.

I think the idea has some merit; although I wonder whether it wouldn't
be smarter to put the code in the backend so that you don't need a
parser in psql.  The SHOW code could fall back to looking at these
possibilities after it fails to find a match to a GUC variable name.

			regards, tom lane


Thu, 17 Apr 2003 10:05:20 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Are we losing momentum? 
Ben Clewett <B.Clewett@roadrunner.uk.com> writes:
> -	The tables (not innodb) are in different files of the same name. 
> Allowing the OS adminitrator great ability.  EG, putting tables on 
> separate partitions and therefore greatly speeding performance.

FWIW, we used to do it that way too, many releases ago.  We gave it up
because it was impossible to support rollback of table deletion/rename
with that storage rule underneath.  Consider

		BEGIN;
		DROP TABLE a;
		CREATE TABLE a (with-some-other-schema);
		-- oops, think better of it
		ROLLBACK;

With table files named after the table, we could not support the above,
because we'd need two "a"'s in existence at the same time.  Postgres'
catalog mechanisms can handle rollback of the catalog changes, but the
Unix filesystem never heard of rollback :-(

There are other reasons, which some folks have pointed out elsewhere in
this thread, but that was the killer one.

I notice that MySQL seems to be migrating in this direction as well...

			regards, tom lane


Thu, 17 Apr 2003 10:35:31 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: For the ametures. (related to "Are we losing momentum?") 
> >> I don't think these should be hacked into the backend/libpq, but
> >> I think it'd be a huge win to hack in "show *" support into psql
> >> for MySQL users so they can type:
> >> 
> >> SHOW (databases|tables|views|functions|triggers|schemas);
> 
> > Well, we (will) have the information schema, and if you like you
> > can put it in the path and write select * from tables; etc., which
> > seems just as good.

I thought about changing the SHOW commands to executing a SELECT from
the information_schema schema, actually, but unless the various \d
commands are going to get switched to SELECT'ing from the
information_schema, it decreases the likelihood that someone will
successfully switch over to using the \d commands in psql.  I believe
that the \d commands are a good thing and it's good that the backend
doesn't have support for the \d commands.  Having \d or SHOW commands
in the backend would dirty up the sources of the backend, IMHO.

> I think Sean's idea is not to make it "just as easy as MySQL", it's
> to make it "the *same* as MySQL", for the benefit of those that
> refuse to learn differently.  Them as won't adjust to "\dt" in place
> of "show tables" aren't likely to adjust to "select * from tables"
> either.  Not even (maybe especially not) if it's arguably a
> standard.

It's amazing what you can accomplish by taking out your DBAs and
racking up a $20 bar tab (note to would be attempters of this method:
when the bar tab gets to $50, you find out things you didn't really
want to know or weren't intending to hear).  So yeah, as Tom said,
they know \d [table_name], I've been able to get that much through,
but syntactically, it doesn't offer the same syntactic goo that their
fingers are used to typing and they _hate_ that there's no SHOW
command for tables.  It's a usability irk that kills them every time,
they type "SHOW [tab]" him.  There are a few other things I picked up
that evening too.

*) MySQL's CLI tab completion is terrible and the SHOW commands work
   even worse than other tab operations in mysql, this _is_ something
   that they do like about psql.  psql's tab completion is really
   snappy by comparison.

*) The _only_ time that they use the SHOW commands is when they're
   using a CLI.

So as opposed to supporting MySQL's brokenness, I hacked a small
parser into psql and added a TIP to the top of the result set printed
to stderr that tells the user how to use the equiv \d command.

> I think the idea has some merit; although I wonder whether it
> wouldn't be smarter to put the code in the backend so that you don't
> need a parser in psql.  The SHOW code could fall back to looking at
> these possibilities after it fails to find a match to a GUC variable
> name.

Well, I think that the backend should be kept clean of MySQL's
nastiness.  It's ugly to have a small parser in psql, but I think it's
better off than letting MySQL dictate non-existent standards to the
rest of the DB community because its developers have struck a chord
with the newbie database masses.  PostgreSQL is a better database, we
shouldn't have to cater to their hackery.  -sc

-- 
Sean Chittenden


Thu, 17 Apr 2003 12:48:03 -0700
Sean Chittenden <sean@chittenden.org>
Re: [HACKERS] Are we losing momentum?
Sean Chittenden <sean@chittenden.org> writes:
>> I think the idea has some merit; although I wonder whether it
>> wouldn't be smarter to put the code in the backend so that you don't
>> need a parser in psql.  The SHOW code could fall back to looking at
>> these possibilities after it fails to find a match to a GUC variable
>> name.

> Well, I think that the backend should be kept clean of MySQL's
> nastiness.

Keep in mind though that there was already talk of migrating most of the
\d functionality to the backend (primarily as a way of decoupling psql
from catalog version changes).  If we were to do that, it would make
good sense to make it accessible via SHOW as well.  IMHO anyway.

			regards, tom lane


Thu, 17 Apr 2003 15:52:06 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Are we losing momentum? 
> >> I think the idea has some merit; although I wonder whether it
> >> wouldn't be smarter to put the code in the backend so that you
> >> don't need a parser in psql.  The SHOW code could fall back to
> >> looking at these possibilities after it fails to find a match to
> >> a GUC variable name.
> 
> > Well, I think that the backend should be kept clean of MySQL's
> > nastiness.
> 
> Keep in mind though that there was already talk of migrating most of
> the \d functionality to the backend (primarily as a way of
> decoupling psql from catalog version changes).  If we were to do
> that, it would make good sense to make it accessible via SHOW as
> well.  IMHO anyway.

:-/ Yeah, I've been following that from a distance and I'm not so wild
about that.  I really like that the information_schema has been
integrated into the base, but translating the SHOW commands into
SELECTs from information_schema on the backend seems like a bad idea
unless its going to be done abstract enough via some kind of rewrite
engine that allows users to program the database to translate their
verbiage into SQL (ex: KILL -> DROP, GET -> SELECT), which could be
kinda fun.

Getting back to SHOW, what do you want to show or not show?  Does the
backend show what's most user friendly?  If that's the case, do you
only show tables that a user has SELECT access to?  Does SHOW return
tuples like a SELECT?  What if a SHOW statement doesn't show what the
user is interested in (view definitions)?  How about when those view
definitions get really long and hard to visually see on a terminal
screen?  There's no select list available in the SHOW syntax to limit
out excessive bits.

While adding the ability to set MYSQL_MODE as something that a user
could set in their .psqlrc, I thought it'd be the ideal progression to
do a few things: 

1) change the \d commands to the appropriate SELECT from the
   information_schema.  Doing this'll go a long way toward keeping the
   structure of the database contained in the database and psql
   independent.

2) Set a few tunables that specify the select list for the SELECTs
   from the information_schema that way a user can specify what they
   see/don't see.

3) SHOW is syntactic user goo that makes MySQL users feel happy and
   should be in the user interface.  Because SHOW is a user interface
   nicety, real admins that over see database users could change
   users' .psqlrc files to specify the select list that the user/site
   wants, which could possibly be even the entire query.

Hrm, how's this for a more concise argument:

Pushing SHOW/\d into the backend is a bad idea. The backend is a
relational database, not a user interface.  The information_schema.*
tables/views are the SQL sanctioned interface that the backend
provides.  How a user interfaces with the database/information_schema
is something that should be left up to the user interface program
(psql) and not pushed into the backend.  If a user wants to type "SHOW
TABLES LIKE p" instead of "\dt p*", so be it, but that's a user
interface concern, not an SQL concern.  The SQL way of getting the
same data as "SHOW TABLES" is via SELECTing from the
information_schema schema.  Implementing SQL commands in the backend
to make up for MySQL's inability to be forward thinking and
consequently hack in a syntax to wrap around their system catalogs for
newbie DB users is bad juju.  By the same token, doesn't mean
PostgreSQL can't provide the same lovey dovey interface that new users
expect, it should, however mean that the backend should be left alone
to do what it specializes in (being an SQL conformant relational DB)
and that the user interface (psql in this case) should be left alone
to implement what SHOW TABLES really means.

Keep in mind, that the only time that the SHOW commands are used, from
what I've been able to ascertain, is when DBAs are in psql and doing
basic admin work and exploring/creating their corner of the universe.
Anyone who's seriously trying to write a tool to inspect the database
knows PostgreSQL reasonably well and uses SELECT + the system
catalogs.  The target audience for a SHOW syntax isn't the power DBAs
or people writing interfaces to examine PostgreSQL, it's the newbie
creating a table for a hack project via the CLI (psql).  Allowing
users to customize the meaning of the \d/SHOW commands would make psql
much more powerful than it currently is and would address many of
these usability concerns.  I'm now thinking that psql should intercept
all non-standard SQL calls (bits not starting with SELECT, UPDATE,
INSERT, ALTER, etc) and translate them into the appropriate SQL.
Having a generic mechanism for doing this would make psql
significantly cleaner.

Anyway, I'll rest on this topic until I hear whether or not folks
would rather have this done in psql or on the backend, but I'd like to
get this in place somewhere so that I can stop reworking bits from
MySQL to PostgreSQL.  If it's determined that the bits should be done
in psql, I'll gladly finish things up, clean things up, add the docs,
move things over to use the information_schema, and if folks would
like, add the appropriate functionality that'll allow folks to
configure the \d commands/SHOW via their .psqlrc.

-sc

-- 
Sean Chittenden


Thu, 17 Apr 2003 14:09:41 -0700
Sean Chittenden <sean@chittenden.org>
Re: [HACKERS] Are we losing momentum?
It's out of date and I did not receive any feedback on it (so I assume
it's not very useful), but the patch submitted in late February would
easily allow the described 'translation' to occur.

Summary:

Schema in backend translates a \<command> <arg> command into an sql
query which is then to be executed.  Logic is still in psql, but the
available commands and how data is retrieved from the schema was backend
specific.


I'm using it here due to it's ability to add new psql commands (since
it's just data in a table.) 

http://archives.postgresql.org/pgsql-patches/2003-02/msg00216.php


Anyway, might be useful for thoughts.  Simply moving the commands into
the backend still leaves us with unsupported new commands in old clients
or new commands in old databases (leaving errors) unless the SHOW ...
syntax itself is handled by the backend.

> :-/ Yeah, I've been following that from a distance and I'm not so wild
> about that.  I really like that the information_schema has been
> integrated into the base, but translating the SHOW commands into
> SELECTs from information_schema on the backend seems like a bad idea
> unless its going to be done abstract enough via some kind of rewrite
> engine that allows users to program the database to translate their
> verbiage into SQL (ex: KILL -> DROP, GET -> SELECT), which could be
> kinda fun.
-- 
Rod Taylor <rbt@rbt.ca>

PGP Key: http://www.rbt.ca/rbtpub.asc
17 Apr 2003 17:25:15 -0400
Rod Taylor <rbt@rbt.ca>
Re: [HACKERS] Are we losing momentum?
Sean Chittenden <sean@chittenden.org> writes:
>> Keep in mind though that there was already talk of migrating most of
>> the \d functionality to the backend (primarily as a way of
>> decoupling psql from catalog version changes).  If we were to do
>> that, it would make good sense to make it accessible via SHOW as
>> well.  IMHO anyway.

> :-/ Yeah, I've been following that from a distance and I'm not so wild
> about that.  I really like that the information_schema has been
> integrated into the base, but translating the SHOW commands into
> SELECTs from information_schema on the backend seems like a bad idea
> unless its going to be done abstract enough via some kind of rewrite
> engine that allows users to program the database to translate their
> verbiage into SQL (ex: KILL -> DROP, GET -> SELECT), which could be
> kinda fun.

Well, I don't want to convert \d into selects from information_schema,
primarily because that would constrain us to showing only things that
are known to the SQL spec --- goodbye, Postgres-specific features
(such as user-definable operators).

I was, however, wondering whether the backend internal support for
"SHOW tables" couldn't be simply to translate it to "SELECT * FROM
some_view".  Then it'd be possible for people to customize the output by
replacing the view definition.

> Getting back to SHOW, what do you want to show or not show?  Does the
> backend show what's most user friendly?  If that's the case, do you
> only show tables that a user has SELECT access to?  Does SHOW return
> tuples like a SELECT?  What if a SHOW statement doesn't show what the
> user is interested in (view definitions)?

I thought you only wanted MySQL-equivalent functionality here ;-).
Don't tell me they have customizable SHOW output ...

> The information_schema.* tables/views are the SQL sanctioned interface
> that the backend provides.

This argument sounds great in the abstract, but it falls down as soon as
you consider the reality that we want to support things that aren't
SQL-sanctioned.  Now, we could define some views that are *not* exactly
INFORMATION_SCHEMA, but at that point the claim that it's a stable
standard interface is looking a lot weaker :-(

			regards, tom lane


Thu, 17 Apr 2003 17:40:47 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Are we losing momentum? 
> >> Keep in mind though that there was already talk of migrating most
> >> of the \d functionality to the backend (primarily as a way of
> >> decoupling psql from catalog version changes).  If we were to do
> >> that, it would make good sense to make it accessible via SHOW as
> >> well.  IMHO anyway.
> 
> > :-/ Yeah, I've been following that from a distance and I'm not so
> > wild about that.  I really like that the information_schema has
> > been integrated into the base, but translating the SHOW commands
> > into SELECTs from information_schema on the backend seems like a
> > bad idea unless its going to be done abstract enough via some kind
> > of rewrite engine that allows users to program the database to
> > translate their verbiage into SQL (ex: KILL -> DROP, GET ->
> > SELECT), which could be kinda fun.
> 
> Well, I don't want to convert \d into selects from
> information_schema, primarily because that would constrain us to
> showing only things that are known to the SQL spec --- goodbye,
> Postgres-specific features (such as user-definable operators).

::nods:: Good point.  All the more the reason to put this in the
client.  :)

> I was, however, wondering whether the backend internal support for
> "SHOW tables" couldn't be simply to translate it to "SELECT * FROM
> some_view".  Then it'd be possible for people to customize the
> output by replacing the view definition.

Well, my attitude is to arm psql with a good set of defaults (already
has some good ones, IMHO), and having it catch \[token1] [token2] and
SHOW [token1] and translate it into the appropriate query.  If it
works out in psql, then leave it.  If people complain about it not
being available in the backend, then we can move it there in 8.0.  :)

> > Getting back to SHOW, what do you want to show or not show?  Does
> > the backend show what's most user friendly?  If that's the case,
> > do you only show tables that a user has SELECT access to?  Does
> > SHOW return tuples like a SELECT?  What if a SHOW statement
> > doesn't show what the user is interested in (view definitions)?
> 
> I thought you only wanted MySQL-equivalent functionality here ;-).
> Don't tell me they have customizable SHOW output ...

Heh, they don't, but letting psql customize what SHOW means would be a
feature that mysql doesn't have and one that'd be reasonably useful,
IMHO.

> > The information_schema.* tables/views are the SQL sanctioned
> > interface that the backend provides.
> 
> This argument sounds great in the abstract, but it falls down as
> soon as you consider the reality that we want to support things that
> aren't SQL-sanctioned.  Now, we could define some views that are
> *not* exactly INFORMATION_SCHEMA, but at that point the claim that
> it's a stable standard interface is looking a lot weaker :-(

Does the spec preclude us from adding views/tables to the
information_schema that allow information_schema to be a completely
reflective interface into the structure of the backend?  I'm worried
that things are out of control because the existing, already in use
backend's system catalogs aren't user friendly (ex: usename ->
username).

-sc

-- 
Sean Chittenden


Thu, 17 Apr 2003 14:55:31 -0700
Sean Chittenden <sean@chittenden.org>
Re: [HACKERS] Are we losing momentum?
Tom Lane writes:

> I think Sean's idea is not to make it "just as easy as MySQL", it's to
> make it "the *same* as MySQL", for the benefit of those that refuse to
> learn differently.  Them as won't adjust to "\dt" in place of "show
> tables" aren't likely to adjust to "select * from tables" either.
> Not even (maybe especially not) if it's arguably a standard.

"Same as MySQL" is impossible.  We can pick here and there and add tons of
duplicate interfaces, play catch-up when they change them, but there's
always going to be a next feature that "would be *really* nice if
PostgreSQL could support it and it would surely draw *tons* of users to
PostgreSQL".  Freely written MySQL code is basically completely
incompatible with PostgreSQL, so someone who needs to switch will have to
relearn anyway.

-- 
Peter Eisentraut   peter_e@gmx.net


Fri, 18 Apr 2003 02:21:22 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Are we losing momentum? 
Tom Lane wrote:
> Ben Clewett <B.Clewett@roadrunner.uk.com> writes:
> > -	The tables (not innodb) are in different files of the same name. 
> > Allowing the OS adminitrator great ability.  EG, putting tables on 
> > separate partitions and therefore greatly speeding performance.
> 
> FWIW, we used to do it that way too, many releases ago.  We gave it up
> because it was impossible to support rollback of table deletion/rename
> with that storage rule underneath.

It occurs to me that we could make it possible to get some of the
performance gains MySQL gets through its naming conventions by
including the type of object in the path of the object.  For instance,
a table with relfilenode 52715 in database with relfilenode 46722
would have a path of $PGDATA/table/46722/52715, an index in the same
database with OID 98632 would have a path of
$PGDATA/index/46722/98632, etc.  Then you could use symlinks to have
tables, indexes, etc. point to various places on disk where they
really need to live.

Is that even remotely feasible?  I looked into making the changes
required but didn't see an obvious way to get a type string from an
object's RelFileNode internally (once you have that, it's a matter of
changing relpath() appropriately).



-- 
Kevin Brown					      kevin@sysexperts.com


Thu, 17 Apr 2003 21:56:52 -0700
Kevin Brown <kevin@sysexperts.com>
Re: For the ametures. (related to "Are we losing momentum?")
Kevin Brown <kevin@sysexperts.com> writes:
> It occurs to me that we could make it possible to get some of the
> performance gains MySQL gets through its naming conventions by
> including the type of object in the path of the object.

"Performance gains"?  Name one.

We have been there and done that.  I see no reason to go back.

			regards, tom lane


Fri, 18 Apr 2003 01:56:33 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: For the ametures. (related to "Are we losing momentum?") 
Tom Lane wrote:
> Kevin Brown <kevin@sysexperts.com> writes:
> > It occurs to me that we could make it possible to get some of the
> > performance gains MySQL gets through its naming conventions by
> > including the type of object in the path of the object.
> 
> "Performance gains"?  Name one.

Instead of tables and their indexes being on the same platter, you'd
be able to put them on separate platters.  Sounds like it would likely
yield a performance gain to me...

> We have been there and done that.  I see no reason to go back.

I'm not proposing that we return to calling the individual files (or
the database they reside in) by name, only that we include a "type"
identifier in the path so that objects of different types can be
located on different spindles if the DBA so desires.  As it is right
now, tables and indexes are all stored in the same directory, and
moving the indexes to a different spindle is an uncertain operation at
best (you get to shut down the database in order to move any
newly-created indexes, and dropping a moved index will not free the
space occupied by the index as it'll only remove the symlink).

All the current transactional operations (including things like table
rename ops) will still be transactional, with the only difference
being that instead of one directory (base/) to deal with you'd have
several (one for each type of object, thus a base/<type>/ directory
for each object type).  Creating a database would mean creating a
directory for the database in each of the type directories instead of
just one in base/, and dropping it would mean removing said
directories.

It's not like we'd be losing anything by it: the only operations that
you wouldn't necessarily be able to run in a transaction are the ones
that you can't currently run in a transaction anyway, like CREATE
DATABASE.

But the benefit is that you can now safely put indexes on a different
spindle than the data.  That sounds like a net win to me.


-- 
Kevin Brown					      kevin@sysexperts.com


Thu, 17 Apr 2003 23:19:57 -0700
Kevin Brown <kevin@sysexperts.com>
Re: For the ametures. (related to "Are we losing momentum?")
Dave Page wrote:
> Still, I've now tried a 29160N adaptor, and changed the
> cable to a DPT one (both that and the previous Adaptec have hardwired
> terminators). I've also removed the DAT drive from the system so the
> only things on the SCSI bus are 2 identical disks and the adaptor.
> 
> I'm beginning to wonder about my shiny new 2.4.20 kernel...

That's a possibility.  There's one other I'd try, prior to using a
different kernel: try booting the kernel with the "noapic" option.  On
certain SMP systems, this has allowed the kernel to come up properly
and see the SCSI adaptor (among other things), whereas without it the
system would hang on attempts to access the SCSI device.  Even if
you're not running SMP, it may help your system's stability.




-- 
Kevin Brown					      kevin@sysexperts.com


Fri, 18 Apr 2003 02:58:19 -0700
Kevin Brown <kevin@sysexperts.com>
Re: pg_clog woes with 7.3.2 - Episode 2
Kevin Brown wrote:
> Tom Lane wrote:
> > Kevin Brown <kevin@sysexperts.com> writes:
> > > It occurs to me that we could make it possible to get some of the
> > > performance gains MySQL gets through its naming conventions by
> > > including the type of object in the path of the object.
> > 
> > "Performance gains"?  Name one.
> 
> Instead of tables and their indexes being on the same platter, you'd
> be able to put them on separate platters.  Sounds like it would likely
> yield a performance gain to me...
> 
> > We have been there and done that.  I see no reason to go back.
> 
> I'm not proposing that we return to calling the individual files (or
> the database they reside in) by name, only that we include a "type"
> identifier in the path so that objects of different types can be
> located on different spindles if the DBA so desires.  As it is right
> now, tables and indexes are all stored in the same directory, and
> moving the indexes to a different spindle is an uncertain operation at
> best (you get to shut down the database in order to move any
> newly-created indexes, and dropping a moved index will not free the
> space occupied by the index as it'll only remove the symlink).

The thing is, this isn't necessarily particularly useful in managing the 
partitioning of data across disks.

If I have, defined, /disk1, /disk2, /disk3, /disk4, and /disk5, it is highly 
unlikely that my partitioning will be based on the notion of "put indices on 
disk1, tables on disk2, and, well, skip the others."

I'm liable to want WAL separate from all the others, for a start, but then 
look for what to put on different disks based on selecting particular tables 
and indices as candidates.

Your observation about the dropping of a moved index is well taken; that would 
point to the idea that the top level "thing" containing each table/index 
perhaps should be a directory, with two interesting properties:

- By being a directory, and putting files in it, this allows extensions to be 
more clearly tied to the table/index when a file grows towards the 
not-uncommon 2GB barrier;

- In order for the linking to physical devices to be kept under control, 
particularly if an index gets dropped and recreated, the postmaster needs to 
be able to establish the links, suggesting an extension to syntax.  At first 
blush:

  CREATE INDEX FROBOZZ_IDX LOCATION '/disk1/pgindices' on FROBOZZ(ID);

Supposing the OID number was 234231, the postmaster would then create the 
symbolic link from $PGDATA/base/234231 to the freshly-created directory 
/disk1/pgindices/234231, where the index would reside.  (And if the directory 
exists, there should be some complaint :-).)

I have made that up out of whole cloth; it _doesn't_ take into consideration 
how you would specify the location of implicitly-created indices.

But it seems a useful approach that can be robust, and where it's even 
plausible that the postmaster could cope with a request to shift a table or 
index to another location.  (Which would, quite naturally, put a lock on 
access to the object for the duration of the operation.)
--
output = reverse("gro.gultn@" "enworbbc")
http://www.ntlug.org/~cbbrowne/
"The dinosaurs died because they didn't have a space program."
-- Arthur C Clarke


Fri, 18 Apr 2003 08:32:25 -0400
cbbrowne@cbbrowne.com
Re: For the ametures. (related to "Are we losing 
Kevin Brown wrote:
> Dave Page wrote:
> > Still, I've now tried a 29160N adaptor, and changed the
> > cable to a DPT one (both that and the previous Adaptec have hardwired
> > terminators). I've also removed the DAT drive from the system so the
> > only things on the SCSI bus are 2 identical disks and the adaptor.
> > 
> > I'm beginning to wonder about my shiny new 2.4.20 kernel...
> 
> That's a possibility.  There's one other I'd try, prior to using a
> different kernel: try booting the kernel with the "noapic" option.  On
> certain SMP systems, this has allowed the kernel to come up properly
> and see the SCSI adaptor (among other things), whereas without it the
> system would hang on attempts to access the SCSI device.  Even if
> you're not running SMP, it may help your system's stability.

The "noapic" option seems a quasi-magical elixir for many sorts of ailments.

I upgraded a box to 2.4.20 and discovered that my NIC was no longer properly 
recognized until I threw that option in.  Others in my office have 
/apparently/ the same hardware, and found they didn't need the option.

As a "fix," it certainly seems to fall into the "snakeoil/superstition" 
category.  While it often seems to have a useful effect, I haven't located any 
actual explanations as to why it should be expected to work.

But based on the comments in the thread, I would concur that my suspicions 
would be with the kernel as the most likely root of the problem.
--
(reverse (concatenate 'string "moc.enworbbc@" "enworbbc"))
http://www3.sympatico.ca/cbbrowne/nonrdbms.html
Rules of the Evil Overlord #82. "I will not shoot at any of my enemies
if they are standing in front  of the crucial support beam to a heavy,
dangerous, unbalanced structure. <http://www.eviloverlord.com/>


Fri, 18 Apr 2003 08:39:45 -0400
cbbrowne@cbbrowne.com
Re: pg_clog woes with 7.3.2 - Episode 2 
Kevin Brown <kevin@sysexperts.com> writes:
> Tom Lane wrote:
>> "Performance gains"?  Name one.

> Instead of tables and their indexes being on the same platter, you'd
> be able to put them on separate platters.  Sounds like it would likely
> yield a performance gain to me...

That has *nothing* to do with whether we name files after tables or not.
As Andrew pointed out, you don't really want people munging file
locations by hand anyway; until we have a proper tablespace
implementation, it's going to be tedious and error-prone no matter what.

> I'm not proposing that we return to calling the individual files (or
> the database they reside in) by name, only that we include a "type"
> identifier in the path so that objects of different types can be
> located on different spindles if the DBA so desires.

This has been proposed and rejected repeatedly in the tablespace
discussions.  It's too limiting; and what's worse, it's not actually
any easier to implement than a proper tablespace facility.  The
low-level I/O routines still need access to a piece of info they do
not have now.  You may as well make it a tablespace identifier instead
of a file-type identifier.

The real point here is that "put the indexes on a different platter"
is policy.  One should never confuse policy with mechanism, nor build a
mechanism that can only implement one policy.

			regards, tom lane


Fri, 18 Apr 2003 10:06:27 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: For the ametures. (related to "Are we losing momentum?") 
cbbrowne@cbbrowne.com wrote:
> The "noapic" option seems a quasi-magical elixir for many sorts of
> ailments.
>
> I upgraded a box to 2.4.20 and discovered that my NIC was no longer
> properly recognized until I threw that option in.  Others in my
> office have /apparently/ the same hardware, and found they didn't
> need the option.
>
> As a "fix," it certainly seems to fall into the
> "snakeoil/superstition" category.  While it often seems to have a
> useful effect, I haven't located any actual explanations as to why
> it should be expected to work.

Well, when it comes to booting a computer, the placebo effect doesn't
really exist.  :-)

Normally I'd agree that "noapic" sounds and smells like snakeoil.  The
problem is that it has observable and repeatable effects on some
systems, and thus can't really be classified as snakeoil (much as one
might like to!).

Why should it be expected to work?  I don't know...possibly because
the APIC hardware is buggy (perhaps in very subtle ways) on some
systems?  Possibly because the APIC driver is subtlely incompatible
with certain APIC hardware?  Possibly because the APIC driver has
certain subtle bugs that only manifest themselves on certain
motherboards with certain peripheral devices?

Whatever the reason, the "noapic" option *does* work on certain
systems, so it unfortunately isn't something that can be dismissed as
mere superstition -- the computer isn't being asked its opinion of its
own health here, nor does it "know" that it should get "well" when
given different boot options.  No "placebo effect" involved, just
repeatable observation (that the observation isn't terribly repeatable
*across* systems does not diminish the validity of the observation).


-- 
Kevin Brown					      kevin@sysexperts.com


Fri, 18 Apr 2003 13:37:03 -0700
Kevin Brown <kevin@sysexperts.com>
Re: pg_clog woes with 7.3.2 - Episode 2
Tom Lane wrote:
> > I'm not proposing that we return to calling the individual files (or
> > the database they reside in) by name, only that we include a "type"
> > identifier in the path so that objects of different types can be
> > located on different spindles if the DBA so desires.
> 
> This has been proposed and rejected repeatedly in the tablespace
> discussions.  It's too limiting; and what's worse, it's not actually
> any easier to implement than a proper tablespace facility.  

It's not?  This is a little surprising, since the type information is
already stored, is it not?  A proper tablespace implementation
requires the addition of commands to manage it and table
infrastructure to store it.  That seems like a bit more work than
writing a function to translate an object ID into a type name (and
changing CREATE/DROP DATABASE to deal with multiple directories).  But
since you're much more familiar with the internals, I'll take your
word for it.

I figured getting the type name of the object would be a relatively
easy thing to do, obvious to anyone with any real familiarity with the
source.  Guess not...




-- 
Kevin Brown					      kevin@sysexperts.com


Fri, 18 Apr 2003 15:06:19 -0700
Kevin Brown <kevin@sysexperts.com>
Re: For the ametures. (related to "Are we losing momentum?")
Kevin Brown <kevin@sysexperts.com> writes:
> Tom Lane wrote:
>> This has been proposed and rejected repeatedly in the tablespace
>> discussions.  It's too limiting; and what's worse, it's not actually
>> any easier to implement than a proper tablespace facility.  

> It's not?  This is a little surprising, since the type information is
> already stored, is it not?  A proper tablespace implementation
> requires the addition of commands to manage it and table
> infrastructure to store it.

Well, yeah, you do have to provide some user interface stuff ;-)

But the hard, dirty, dangerous stuff is all in the low-level internals
(bufmgr, smgr, etc).  I don't want to put a kluge in there when the same
amount of work will support a non-kluge solution.

Also, you'd still have to provide some user interface stuff for the
kluge, so it's not like you can avoid doing any work at that level.

			regards, tom lane


Fri, 18 Apr 2003 20:11:45 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: For the ametures. (related to "Are we losing momentum?") 
> > Instead of tables and their indexes being on the same platter, you'd
> > be able to put them on separate platters.  Sounds like it would likely
> > yield a performance gain to me...
>
> That has *nothing* to do with whether we name files after tables or not.
> As Andrew pointed out, you don't really want people munging file
> locations by hand anyway; until we have a proper tablespace
> implementation, it's going to be tedious and error-prone no matter what.

Just so people are aware, I'm getting Jim Buttfuoco's tablespaces patch
from him again, and getting it up to CVS.  I'll then see what needs to be
done to it such that it would be accepted...

Chris


Sat, 19 Apr 2003 12:21:01 +0800 (WST)
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: For the ametures. (related to "Are we losing momentum?")

Great.  I think it can be made acceptable with little work.

---------------------------------------------------------------------------

Christopher Kings-Lynne wrote:
> > > Instead of tables and their indexes being on the same platter, you'd
> > > be able to put them on separate platters.  Sounds like it would likely
> > > yield a performance gain to me...
> >
> > That has *nothing* to do with whether we name files after tables or not.
> > As Andrew pointed out, you don't really want people munging file
> > locations by hand anyway; until we have a proper tablespace
> > implementation, it's going to be tedious and error-prone no matter what.
> 
> Just so people are aware, I'm getting Jim Buttfuoco's tablespaces patch
> from him again, and getting it up to CVS.  I'll then see what needs to be
> done to it such that it would be accepted...
> 
> Chris
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073


Sat, 19 Apr 2003 01:21:57 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: For the ametures. (related to "Are we losing momentum?")
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Great.  I think it can be made acceptable with little work.

IIRC, the reason Jim's patch got bounced was exactly that it offered an
implementation of only one policy, with no possibility of extension.

			regards, tom lane


Sat, 19 Apr 2003 01:27:15 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: For the ametures. (related to "Are we losing momentum?") 


On Sat, 19 Apr 2003, Bruce Momjian wrote:

>
> Great.  I think it can be made acceptable with little work.

Just keep in mind my track record of finding things too hard ;)

Jim has offered to help, he was just put off by some negativity...

Chris

>
> Christopher Kings-Lynne wrote:
> > > > Instead of tables and their indexes being on the same platter, you'd
> > > > be able to put them on separate platters.  Sounds like it would likely
> > > > yield a performance gain to me...
> > >
> > > That has *nothing* to do with whether we name files after tables or not.
> > > As Andrew pointed out, you don't really want people munging file
> > > locations by hand anyway; until we have a proper tablespace
> > > implementation, it's going to be tedious and error-prone no matter what.
> >
> > Just so people are aware, I'm getting Jim Buttfuoco's tablespaces patch
> > from him again, and getting it up to CVS.  I'll then see what needs to be
> > done to it such that it would be accepted...
> >
> > Chris
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 3: if posting/reading through Usenet, please send an appropriate
> > subscribe-nomail command to majordomo@postgresql.org so that your
> > message can get through to the mailing list cleanly
> >
>
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
>


Sat, 19 Apr 2003 17:21:46 +0800 (WST)
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: For the ametures. (related to "Are we losing momentum?")
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Great.  I think it can be made acceptable with little work.
>
> IIRC, the reason Jim's patch got bounced was exactly that it offered an
> implementation of only one policy, with no possibility of extension.

I read all the comments regarding Jim's patch, but would you mind stating
exactly what your concern is, Tom?  What do you mean by 'one policy'?

Chris


Sat, 19 Apr 2003 17:22:49 +0800 (WST)
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: For the ametures. (related to "Are we losing momentum?")
Christopher Kings-Lynne wrote:
> > Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > > Great.  I think it can be made acceptable with little work.
> >
> > IIRC, the reason Jim's patch got bounced was exactly that it offered an
> > implementation of only one policy, with no possibility of extension.
> 
> I read all the comments regarding Jim's patch, but would you mind stating
> exactly what your concern is, Tom?  What do you mean by 'one policy'?

As I remember, the patch only put indexes in one place, and tables in
another place.  We need a general tablespace solution where we can
create tablespaces and put tables/indexes in any of those.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073


Sat, 19 Apr 2003 11:33:19 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: For the ametures. (related to "Are we losing momentum?")
Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes:
> I read all the comments regarding Jim's patch, but would you mind stating
> exactly what your concern is, Tom?  What do you mean by 'one policy'?

I don't want something that will only support a policy of "put the
indexes over there".  It should be possible to assign individual tables
or indexes to particular tablespaces if the DBA wants to do that.
I have nothing against making it easy to "put the indexes over there"
--- for example, we might say that a database has a default tablespace
for each kind of object.  But if the mechanism can only support a
per-object-kind determination of tablespace then it's insufficiently
flexible.

I no longer recall any details about Jim's patch, but I believe we felt
that it failed the flexibility criterion.

			regards, tom lane


Sat, 19 Apr 2003 11:33:31 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: For the ametures. (related to "Are we losing momentum?") 
On 16 Apr 2003, Hannu Krosing wrote:

> greg@turnstep.com kirjutas K, 16.04.2003 kell 16:51:
> > The original poster had some 
> > valid points (auto-vacuum and non-intuitive commands) that still need 
> > addressing, IMO.
> 
> As of 7.3 (or was it 7.2) auto-vacuum is just one line in crontab. In
> many scenarios it can be left running continuously with very little
> effect on performance. In others it must be run nightly, but having it
> kick in at unexpected times may not be what you want at all. So it has
> to be configured for good performance weather it is built-in or run in a
> separate backend process.
> 
> And I can't see how "show tables" is more intuitive than "\dt" - I
> expected it to be "list tables" or "tablelist" or "näita tabeleid" .

'show tables' is SQL, and can be run from a script, with the output 
parsed.  For some reason when I run a query of \dt from PHP I get an 
error. :-)

> Once you have found \? it is all there (and you are advised to use \? at
> psql startup).

I love \ commands, but remember, those are psql commands, not postgresql 
commands.  show tables would be a postgreSQL command the backend parser 
would understand.  Apples and Oranges.

> That may also be why PostgreSQL is more popular in Japan - if one has to
> remember nonsensical strings, then it is easier to remember short ones

But, how do I write an app to ask such questions easily?  psql -E is not 
the easiest and most intuitive way to learn how to get the data into a 
structure for parsing in a client side app.


Mon, 21 Apr 2003 09:30:49 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Are we losing momentum?
On Monday 21 April 2003 21:00, scott.marlowe wrote:
> But, how do I write an app to ask such questions easily?  psql -E is not
> the easiest and most intuitive way to learn how to get the data into a
> structure for parsing in a client side app.

How about selecting from pg_class? Nothing could have been more structured..

 Shridhar


Mon, 21 Apr 2003 21:14:33 +0530
Shridhar Daithankar <shridhar_daithankar@persistent.co.in>
Re: Are we losing momentum?
On Mon, 2003-04-21 at 08:30, scott.marlowe wrote:
> On 16 Apr 2003, Hannu Krosing wrote:
> 
> > Once you have found \? it is all there (and you are advised to use \? at
> > psql startup).
> 
> I love \ commands, but remember, those are psql commands, not postgresql 
> commands.  show tables would be a postgreSQL command the backend parser 
> would understand.  Apples and Oranges.
> 
> > That may also be why PostgreSQL is more popular in Japan - if one has to
> > remember nonsensical strings, then it is easier to remember short ones
> 
> But, how do I write an app to ask such questions easily?  psql -E is not 
> the easiest and most intuitive way to learn how to get the data into a 
> structure for parsing in a client side app.

He speaks my mind.
-- 
Steve Wampler <swampler@noao.edu>
National Solar Observatory


21 Apr 2003 08:45:12 -0700
Steve Wampler <swampler@noao.edu>
Re: Are we losing momentum?
On Mon, 2003-04-21 at 11:30, scott.marlowe wrote:
> 'show tables' is SQL, and can be run from a script, with the output 
> parsed.

But "select * from pg_tables" (or the equivalent query on the
information schemas) is SQL, can be run from a script, and can be parsed
by a client application.

> But, how do I write an app to ask such questions easily?  psql -E is not 
> the easiest and most intuitive way to learn how to get the data into a 
> structure for parsing in a client side app.

You're conflating two distinct issues: (1) providing an interface for
CLI use by the DBA (2) providing an API for programmer use in
applications.

If you think the existing system catalogs are not sufficiently
intuitive, then we should fix that problem properly (for example,
through better documentation), not by copying some ad-hoc syntax from
another RDBMS.

If you think the existing CLI interface (\d etc.) is not sufficiently
intuitive (which has been what a couple people in this thread have
argued), I don't see what that has to do with client side applications
or parsing the output.

Cheers,

Neil


21 Apr 2003 13:06:01 -0400
Neil Conway <neilc@samurai.com>
Re: Are we losing momentum?
On Fri, 18 Apr 2003, Kevin Brown wrote:

> cbbrowne@cbbrowne.com wrote:
> > The "noapic" option seems a quasi-magical elixir for many sorts of
> > ailments.
> >
> > I upgraded a box to 2.4.20 and discovered that my NIC was no longer
> > properly recognized until I threw that option in.  Others in my
> > office have /apparently/ the same hardware, and found they didn't
> > need the option.
> >
> > As a "fix," it certainly seems to fall into the
> > "snakeoil/superstition" category.  While it often seems to have a
> > useful effect, I haven't located any actual explanations as to why
> > it should be expected to work.
> 
> Well, when it comes to booting a computer, the placebo effect doesn't
> really exist.  :-)
> 
> Normally I'd agree that "noapic" sounds and smells like snakeoil.  The
> problem is that it has observable and repeatable effects on some
> systems, and thus can't really be classified as snakeoil (much as one
> might like to!).
> 
> Why should it be expected to work?  I don't know...possibly because
> the APIC hardware is buggy (perhaps in very subtle ways) on some
> systems?  Possibly because the APIC driver is subtlely incompatible
> with certain APIC hardware?  Possibly because the APIC driver has
> certain subtle bugs that only manifest themselves on certain
> motherboards with certain peripheral devices?
> 
> Whatever the reason, the "noapic" option *does* work on certain
> systems, so it unfortunately isn't something that can be dismissed as
> mere superstition -- the computer isn't being asked its opinion of its
> own health here, nor does it "know" that it should get "well" when
> given different boot options.  No "placebo effect" involved, just
> repeatable observation (that the observation isn't terribly repeatable
> *across* systems does not diminish the validity of the observation).

Just to add to this, on some of the first SMP systems I messed with there 
was a setting for some SMP version of 1.1 or 1.4, and using 1.1 resulted 
in an unstable box for me.  1.4 fixed all the issues.  SMP on Intel is a 
wild ride, and no two motherboards are equivalent.  I've had good luck 
with Supermicro and Intel SMP motherboards, although both have needed BIOS 
updates at times.


Mon, 21 Apr 2003 11:40:32 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: pg_clog woes with 7.3.2 - Episode 2
On 21 Apr 2003, Neil Conway wrote:

> On Mon, 2003-04-21 at 11:30, scott.marlowe wrote:
> > 'show tables' is SQL, and can be run from a script, with the output 
> > parsed.
> 
> But "select * from pg_tables" (or the equivalent query on the
> information schemas) is SQL, can be run from a script, and can be parsed
> by a client application.

But it's not an answer.  In psql we have the \ commands, which I love.  In 
a client side app, select * from pg_tables is just the beginning.  You've 
got to join that to pg_class and jump through quite a few hoops.

For instance, a \d on a simple table in my database produces this much SQL 
in the backend:

********* QUERY **********
SELECT relhasindex, relkind, relchecks, reltriggers, relhasrules
FROM pg_class WHERE relname='profile'
**************************

********* QUERY **********
SELECT a.attname, format_type(a.atttypid, a.atttypmod), a.attnotnull, 
a.atthasdef, a.attnum
FROM pg_class c, pg_attribute a
WHERE c.relname = 'profile'
  AND a.attnum > 0 AND a.attrelid = c.oid
ORDER BY a.attnum
**************************

********* QUERY **********
SELECT substring(d.adsrc for 128) FROM pg_attrdef d, pg_class c
WHERE c.relname = 'profile' AND c.oid = d.adrelid AND d.adnum = 1
**************************

********* QUERY **********
SELECT c2.relname
FROM pg_class c, pg_class c2, pg_index i
WHERE c.relname = 'profile' AND c.oid = i.indrelid AND i.indexrelid = 
c2.oid
AND NOT i.indisunique ORDER BY c2.relname
**************************

********* QUERY **********
SELECT c2.relname
FROM pg_class c, pg_class c2, pg_index i
WHERE c.relname = 'profile' AND c.oid = i.indrelid AND i.indexrelid = 
c2.oid
AND i.indisprimary AND i.indisunique ORDER BY c2.relname
**************************

********* QUERY **********
SELECT c2.relname
FROM pg_class c, pg_class c2, pg_index i
WHERE c.relname = 'profile' AND c.oid = i.indrelid AND i.indexrelid = 
c2.oid
AND NOT i.indisprimary AND i.indisunique ORDER BY c2.relname
**************************

Yet there is no equivalent materialized view that puts the data together 
for the user.  I don't know about you, but show table tablename is a bit 
easier to grasp for beginners than the above sequence of SQL statements.

> > But, how do I write an app to ask such questions easily?  psql -E is not 
> > the easiest and most intuitive way to learn how to get the data into a 
> > structure for parsing in a client side app.
> 
> You're conflating two distinct issues: (1) providing an interface for
> CLI use by the DBA (2) providing an API for programmer use in
> applications.

Why are those two seperate issues?  Why can't the same answer be easily 
and readily available to both the DBA and the programmer?  Why does one 
have to first use psql -E to figure out the queries needed then figure out 
which ones to use and not use etc...?  I'm not saying the \ commands are 
bad, I'm saying they're implemented in the wrong place.  Having \ in the 
psql monitor is fine.  But it should really be hitting views in the 
background where possible.

> If you think the existing system catalogs are not sufficiently
> intuitive, then we should fix that problem properly (for example,
> through better documentation), not by copying some ad-hoc syntax from
> another RDBMS.

I don't care what MySQL does.  Period.  But, I do think Postgresql has a 
high learning curve because so much of it is hidden from beginners.  

Better documentation won't fix this issue.  The real issue here is that 
psql has a facility (\ commands) that isn't present in the rest of 
postgresql, and really should be.  psql shouldn't be the only interface 
that allows you to easily see how tables are put together etc...

> If you think the existing CLI interface (\d etc.) is not sufficiently
> intuitive (which has been what a couple people in this thread have
> argued), I don't see what that has to do with client side applications
> or parsing the output.

No, I like the psql interface.  It's intuitive to me and has been since 
day one.  It's the lack of intuition on the application side that bothers 
me.


Mon, 21 Apr 2003 14:26:20 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Are we losing momentum?
> It's out of date and I did not receive any feedback on it (so I assume
> it's not very useful), but the patch submitted in late February would
> easily allow the described 'translation' to occur.
> 
> Summary:
> 
> Schema in backend translates a \<command> <arg> command into an sql
> query which is then to be executed.  Logic is still in psql, but the
> available commands and how data is retrieved from the schema was backend
> specific.
> 
> 
> I'm using it here due to it's ability to add new psql commands (since
> it's just data in a table.) 
> 
> http://archives.postgresql.org/pgsql-patches/2003-02/msg00216.php
> 
> 
> Anyway, might be useful for thoughts.  Simply moving the commands into
> the backend still leaves us with unsupported new commands in old clients
> or new commands in old databases (leaving errors) unless the SHOW ...
> syntax itself is handled by the backend.
> 
> > :-/ Yeah, I've been following that from a distance and I'm not so wild
> > about that.  I really like that the information_schema has been
> > integrated into the base, but translating the SHOW commands into
> > SELECTs from information_schema on the backend seems like a bad idea
> > unless its going to be done abstract enough via some kind of rewrite
> > engine that allows users to program the database to translate their
> > verbiage into SQL (ex: KILL -> DROP, GET -> SELECT), which could be
> > kinda fun.

Anyone other than Rod (and now myself) had a chance to look this over?
This doesn't really address the lack of a SHOW syntax for new MySQL
users, but it sure does open up the possibilities for making it easier
to probe the backend.

On a related note, any thoughts on the SHOW stuff given that the topic
has come back to life on -hackers?  -sc

-- 
Sean Chittenden
Mon, 21 Apr 2003 14:17:55 -0700
Sean Chittenden <sean@chittenden.org>
Re: [HACKERS] Are we losing momentum?
On Mon, 2003-04-21 at 16:26, scott.marlowe wrote:
> Yet there is no equivalent materialized view that puts the data together 
> for the user.  I don't know about you, but show table tablename is a bit 
> easier to grasp for beginners than the above sequence of SQL statements.

Granted -- but I don't think that replacing or augmenting the system
catalogs with a set of SHOW commands is a good idea (which is what you
suggested originally). IMHO enhancing the system catalogs by adding
views that encapsulate more of the \ command functionality into the
backend is a good idea, and one that should be implemented eventually.
AFAIK that's been the consensus for some time...

Cheers,

Neil


21 Apr 2003 17:47:08 -0400
Neil Conway <neilc@samurai.com>
Re: Are we losing momentum?
Hi,

> On Mon, 2003-04-21 at 16:26, scott.marlowe wrote:
> > Yet there is no equivalent materialized view that puts the data together
> > for the user.  I don't know about you, but show table tablename is a bit
> > easier to grasp for beginners than the above sequence of SQL statements.
>
> Granted -- but I don't think that replacing or augmenting the system
> catalogs with a set of SHOW commands is a good idea (which is what you
> suggested originally). IMHO enhancing the system catalogs by adding
> views that encapsulate more of the \ command functionality into the
> backend is a good idea, and one that should be implemented eventually.
> AFAIK that's been the consensus for some time...

I think the SHOW commands won't be neccesary when there are views to use.
There is already a good SQL command to get data/information from the
databaseserver: SELECT. Adding SHOW commands to the backend that essentially
do a SELECT on a system view are a bad thing IMHO. The user can just as easy
do a SELECT on the view himself.

All just IMHO ofcourse :)
Sander.


Tue, 22 Apr 2003 00:48:10 +0200
"Sander Steffann" <sander@steffann.nl>
Re: Are we losing momentum?
Keven Brown wrote:
> Normally I'd agree that "noapic" sounds and smells like snakeoil.  The
> problem is that it has observable and repeatable effects on some
> systems, and thus can't really be classified as snakeoil (much as one
> might like to!).

There's a /slight/ difference between 'superstition' and 'snakeoil'; the
latter is something you don't expect to find effectual.  The former may
represent something you don't/can't understand.

Recall Clarke's observation that "Any sufficiently advanced technology
is indistinguishable from magic."

Anything I can't explain is likely to either be:

 a) Something I don't understand yet, or

 b) Perhaps something truly supernatural, that cannot be explained
    based on any sort of natural reasoning.  Perhaps explainable by "God
    did something unexplainable, therefore things are the way they are."

There are disagreements as to where to draw the line.  People of some
degrees of "superstitiousness" may be prepared to explain _everything_
as involving "God decided to make that happen," making _no_ attempt to
understand lower level processes.  Some, more skeptical, may reject that
anything should be able to fall into category b).

On the other hand, some people get themselves rip-roaring drunk and trip
over the line we thought was one of concept:

   "SCSI is *NOT* magic. There are *fundamental technical reasons* why
   it is necessary to sacrifice a young goat to your SCSI chain now and
   then."
--
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne> rate me
http://www3.sympatico.ca/cbbrowne/wp.html
"While preceding your entrance with a grenade is a good tactic in
Quake, it can lead to problems if attempted at work."    -- C Hacking
-- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html


Mon, 21 Apr 2003 21:16:32 -0400
Christopher Browne <cbbrowne@cbbrowne.com>
Re: pg_clog woes with 7.3.2 - Episode 2 
On Tue, Apr 15, 2003 at 08:12:04AM +0200, Tony Grant wrote:
> Lets forget the "replace MySQL with PostgreSQL" stuff and go looking for
> higher end converts. Our marketing push should be "replace Oracle with
> PostgreSQL and replace Access with MySQL". This puts the emphasis on
> which database can do what... 
 
Except going from MS Access to MySQL would be a step backwards. :P
-- 
Jim C. Nasby (aka Decibel!)                    jim@nasby.net
Member: Triangle Fraternity, Sports Car Club of America
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"


Tue, 22 Apr 2003 02:07:01 -0500
"Jim C. Nasby" <jim@nasby.net>
Re: Are we losing momentum?
Neil Conway writes:

> IMHO enhancing the system catalogs by adding
> views that encapsulate more of the \ command functionality into the
> backend is a good idea, and one that should be implemented eventually.

That would be very nice.

	Tilo


Tue, 22 Apr 2003 17:14:07 +0200
Tilo Schwarz <mail@tilo-schwarz.de>
Re: Are we losing momentum?
multilog


andrew@libertyrms.info (Andrew Sullivan) wrote:
> Since now is the time for contrib/ flamewars, this seemed a good time
> to suggest this.
> 
...
> 
> Is anyone interested in having pglog-rotator?


Tue, 6 May 2003 02:40:51 +0000 (UTC)
ngpg@grymmjack.com
Re: more contrib: log rotator
I didn't find any documentation mentioning scheduled jobs. I assume there is no such a feature
yet. I would like to implement it if someone helps me with the development process (I am brand new
to OpenSource projects).

Basically the feature should include scheduling function exeution at:
    - postmaster startup
    - postmaster shutdown
    - a specified moment
    - a time of the day/month/year
    - recurring at a time interval

I know this could be implemented in exernal processes but from an application standpoint it would
be much more consistent if all the database-related functionality is in the database server.
Besides, both Oracle and Microsoft have the feature.

Please advise.

Thanks,
Zlatko


__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


Mon, 12 May 2003 15:03:50 -0700 (PDT)
Zlatko Michailov <zmichailov@yahoo.com>
Scheduled jobs
Zlatko Michailov <zmichailov@yahoo.com> writes:
> I didn't find any documentation mentioning scheduled jobs. I assume
> there is no such a feature yet. I would like to implement it if
> someone helps me with the development process (I am brand new to
> OpenSource projects).

> Basically the feature should include scheduling function exeution at:
>     - postmaster startup
>     - postmaster shutdown
>     - a specified moment
>     - a time of the day/month/year
>     - recurring at a time interval

Use cron.  I see no value in duplicating cron's functionality inside
Postgres.

> Besides, both Oracle and Microsoft have the feature.

They can afford to expend developer time on inventing and maintaining
useless "features".  We have very finite resources and have to be
careful of buying into supporting things that won't really pull their
weight.

			regards, tom lane


Mon, 12 May 2003 18:17:11 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Scheduled jobs 
On Mon, 12 May 2003, Tom Lane wrote:

> Zlatko Michailov <zmichailov@yahoo.com> writes:
> > I didn't find any documentation mentioning scheduled jobs. I assume
> > there is no such a feature yet. I would like to implement it if
> > someone helps me with the development process (I am brand new to
> > OpenSource projects).
> 
> > Basically the feature should include scheduling function exeution at:
> >     - postmaster startup
> >     - postmaster shutdown
> >     - a specified moment
> >     - a time of the day/month/year
> >     - recurring at a time interval
> 
> Use cron.  I see no value in duplicating cron's functionality inside
> Postgres.

I was going to say use cron :)

Only cron can't handle some of those cases listed, but then one could always
patch one's own local installation of pg_ctl etc. to run things at startup and
shutdown.

I'm not sure how specified moment differs from time of day/month/year (except
can cron handle years?)


> 
> > Besides, both Oracle and Microsoft have the feature.
> 
> They can afford to expend developer time on inventing and maintaining
> useless "features".  We have very finite resources and have to be
> careful of buying into supporting things that won't really pull their
> weight.
> 
> 			regards, tom lane

-- 
Nigel J. Andrews


Mon, 12 May 2003 23:22:23 +0100 (BST)
"Nigel J. Andrews" <nandrews@investsystems.co.uk>
Re: Scheduled jobs
"Nigel J. Andrews" <nandrews@investsystems.co.uk> writes:
> Only cron can't handle some of those cases listed, but then one could always
> patch one's own local installation of pg_ctl etc. to run things at startup and
> shutdown.

If you are using a typical init script setup, it's easy to add
additional operations to the init script's start and stop actions.

I'm a tad suspicious of adding on-shutdown actions anyway, as there's
little guarantee they would get done (consider system crash, postmaster
crash, etc).

			regards, tom lane


Mon, 12 May 2003 18:27:26 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Scheduled jobs 
Zlatko Wrote:
> I didn't find any documentation mentioning scheduled jobs. I assume
> there is no such a feature yet. I would like to implement it if
> someone helps me with the development process (I am brand new to
> OpenSource projects).

> Basically the feature should include scheduling function exeution at:
>     - postmaster startup
>     - postmaster shutdown
>     - a specified moment
>     - a time of the day/month/year
>     - recurring at a time interval

Have you ever considered using cron?

- It is available on every Unix.
- It may readily be compiled for Cygwin.

It seems preposterous to imagine that reimplementing the functionality
of cron would significantly add to the functionality of PostgreSQL.

If you really want a unified way of accessing crontabs, then feel free
to write some plpgsql functions that are "wrappers" for cron...
--
(concatenate 'string "aa454" "@freenet.carleton.ca")
http://www.ntlug.org/~cbbrowne/unix.html
FLORIDA: Relax, Retire, Re Vote.


Mon, 12 May 2003 18:38:14 -0400
Christopher Browne <cbbrowne@cbbrowne.com>
Re: Scheduled jobs 
On Mon, 12 May 2003, Tom Lane wrote:

> "Nigel J. Andrews" <nandrews@investsystems.co.uk> writes:
> > Only cron can't handle some of those cases listed, but then one could always
> > patch one's own local installation of pg_ctl etc. to run things at startup and
> > shutdown.
> 
> If you are using a typical init script setup, it's easy to add
> additional operations to the init script's start and stop actions.
> 
> I'm a tad suspicious of adding on-shutdown actions anyway, as there's
> little guarantee they would get done (consider system crash, postmaster
> crash, etc).
> 
> 			regards, tom lane

Absolutely. That's why you'd patch your startup/shutdown scripts. Adding it to
pg_ctl does enable those to kick the necessary stuff without requiring use of
the system's init scripts for manual control of the postmaster. When the
emphasis on the 'controlled' aspect of this is acknowledged then it's just a
toss up between editing pg_ctl or your own wrapper for it. I would go for my
own wrapper since then that still leaves the ability for pg_ctl to be used
_without_ kicking those startup/shutdown actions.


I believe this has arisen several times and each time there's been no
enthusiasm to stick cron into the core which I think is a reasonable stance.


-- 
Nigel J. Andrews


Mon, 12 May 2003 23:38:59 +0100 (BST)
"Nigel J. Andrews" <nandrews@investsystems.co.uk>
Re: Scheduled jobs
For windows, look up wincron, I used it back in the day.  don't know if 
it's still an up to date package, but it was great back when NT4.0 still 
held some small attraction to me.

On Mon, 12 May 2003, Christopher Browne wrote:

> Zlatko Wrote:
> > I didn't find any documentation mentioning scheduled jobs. I assume
> > there is no such a feature yet. I would like to implement it if
> > someone helps me with the development process (I am brand new to
> > OpenSource projects).
> 
> > Basically the feature should include scheduling function exeution at:
> >     - postmaster startup
> >     - postmaster shutdown
> >     - a specified moment
> >     - a time of the day/month/year
> >     - recurring at a time interval
> 
> Have you ever considered using cron?
> 
> - It is available on every Unix.
> - It may readily be compiled for Cygwin.
> 
> It seems preposterous to imagine that reimplementing the functionality
> of cron would significantly add to the functionality of PostgreSQL.
> 
> If you really want a unified way of accessing crontabs, then feel free
> to write some plpgsql functions that are "wrappers" for cron...
> --
> (concatenate 'string "aa454" "@freenet.carleton.ca")
> http://www.ntlug.org/~cbbrowne/unix.html
> FLORIDA: Relax, Retire, Re Vote.
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
> http://archives.postgresql.org
> 


Mon, 12 May 2003 16:56:02 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Scheduled jobs 
-*- Tom Lane <tgl@sss.pgh.pa.us> [ 2003-05-12 22:18 ]:
> Use cron.  I see no value in duplicating cron's functionality inside
> Postgres.

The biggest advantages I see:
 - running tasks as a specific database user without having to store passwords on the server
 - When deploying a database -- maintenance jobs can be created with SQL commands
 - Not everybody have access to the server or don't have another machine to run it from

Just mentioning some pros I see -- I do agree with your point on resources and future maintenance.


-- 
Regards,
Tolli
tolli@tol.li


Mon, 12 May 2003 23:03:20 +0000
=?iso-8859-1?Q?=DE=F3rhallur_H=E1lfd=E1narson?= <tolli@tol.li>
Re: Scheduled jobs
Quoth nandrews@investsystems.co.uk ("Nigel J. Andrews"):
> I believe this has arisen several times and each time there's been
> no enthusiasm to stick cron into the core which I think is a
> reasonable stance.

I think it _would_ be kind of neat to set up some tables to contain
what's in the postgres user's crontab, and have a pair of stored
procedures to move data in and out.  If you had some Truly Appalling
number of cron jobs, manipulating them in a database could well be a
great way to do it.

That is, of course, quite separate from having cron in the core.  And
having a _good_ set of semantics for the push/pull is a nontrivial
matter...
-- 
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne> rate me
http://cbbrowne.com/info/nonrdbms.html
As Will Rogers would have said, "There is no such thing as a free
variable."  -- Alan Perlis


Mon, 12 May 2003 19:29:46 -0400
Christopher Browne <cbbrowne@cbbrowne.com>
Re: Scheduled jobs
On Mon, May 12, 2003 at 23:03:20 +0000,
  Þórhallur Hálfdánarson <tolli@tol.li> wrote:
> 
> The biggest advantages I see:
>  - running tasks as a specific database user without having to store passwords on the server

You can do that now using ident authentication. For some OS's you can do
this using domain sockets and don't have to run an ident server.


Tue, 13 May 2003 06:40:31 -0500
Bruno Wolff III <bruno@wolff.to>
Re: Scheduled jobs
-*- Bruno Wolff III <bruno@wolff.to> [ 2003-05-13 11:39 ]:
> On Mon, May 12, 2003 at 23:03:20 +0000,
>   Þórhallur Hálfdánarson <tolli@tol.li> wrote:
> > 
> > The biggest advantages I see:
> >  - running tasks as a specific database user without having to store passwords on the server
> 
> You can do that now using ident authentication. For some OS's you can do
> this using domain sockets and don't have to run an ident server.

In most of my setups there is only a limited number of users on the system, but many other users in PostgreSQL.  Creating a user on the system for every user I create in the DB and allowing him to run processes is not according to procedures I follow, and I believe that applies to more people.

-- 
Regards,
Tolli
tolli@tol.li


Tue, 13 May 2003 12:40:41 +0000
=?iso-8859-1?Q?=DE=F3rhallur_H=E1lfd=E1narson?= <tolli@tol.li>
Re: Scheduled jobs
On Tue, May 13, 2003 at 12:40:41PM +0000, Þórhallur Hálfdánarson wrote:

> In most of my setups there is only a limited number of users on the
> system, but many other users in PostgreSQL.  Creating a user on the
> system for every user I create in the DB and allowing him to run
> processes is not according to procedures I follow, and I believe that
> applies to more people.

In this case you can put the passwords in ~/.pgpass, I think.

-- 
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"El sentido de las cosas no viene de las cosas, sino de
las inteligencias que las aplican a sus problemas diarios
en busca del progreso." (Ernesto Hernández-Novich)


Tue, 13 May 2003 09:15:46 -0400
Alvaro Herrera <alvherre@dcc.uchile.cl>
Re: Scheduled jobs
-*- Alvaro Herrera <alvherre@dcc.uchile.cl> [ 2003-05-13 13:19 ]:
> On Tue, May 13, 2003 at 12:40:41PM +0000, Þórhallur Hálfdánarson wrote:
> 
> > In most of my setups there is only a limited number of users on the
> > system, but many other users in PostgreSQL.  Creating a user on the
> > system for every user I create in the DB and allowing him to run
> > processes is not according to procedures I follow, and I believe that
> > applies to more people.
> 
> In this case you can put the passwords in ~/.pgpass, I think.

The suggestion on using ident was to eliminate the need for storing passwords in the first place...

-- 
Regards,
Tolli
tolli@tol.li


Tue, 13 May 2003 13:33:25 +0000
=?iso-8859-1?Q?=DE=F3rhallur_H=E1lfd=E1narson?= <tolli@tol.li>
Re: Scheduled jobs
Yes. We need a system table to store scheduled jobs and a single external daemon to fire them up.

The example I have in mind is maintaining active sessions where my app maintains its own user
accounts. When a user logs in, a session row is created in a an app table. Every time a new
request comes through that session, a last_used timestamp is updated. At the same time there must
be a job checking that same table every minute for rows where the last_used timestamp is over 20
minutes old and remove such rows. Since the account registrations are inside the database, it
appeals to me that session maintenance should also be there.

Please think about it again. I can provide a table and SQL command (or stored proc) proposal.

Thanks,
Zlatko


--- Christopher Browne <cbbrowne@cbbrowne.com> wrote:
> Quoth nandrews@investsystems.co.uk ("Nigel J. Andrews"):
> > I believe this has arisen several times and each time there's been
> > no enthusiasm to stick cron into the core which I think is a
> > reasonable stance.
> 
> I think it _would_ be kind of neat to set up some tables to contain
> what's in the postgres user's crontab, and have a pair of stored
> procedures to move data in and out.  If you had some Truly Appalling
> number of cron jobs, manipulating them in a database could well be a
> great way to do it.
> 
> That is, of course, quite separate from having cron in the core.  And
> having a _good_ set of semantics for the push/pull is a nontrivial
> matter...
> -- 
> If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne> rate me
> http://cbbrowne.com/info/nonrdbms.html
> As Will Rogers would have said, "There is no such thing as a free
> variable."  -- Alan Perlis
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)


__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


Tue, 13 May 2003 07:18:52 -0700 (PDT)
Zlatko Michailov <zmichailov@yahoo.com>
Re: Scheduled jobs
On Tue, May 13, 2003 at 01:33:25PM +0000, ??rhallur H?lfd?narson wrote:
> The suggestion on using ident was to eliminate the need for storing
> passwords in the first place...

But how are you going to let them run scheduled jobs inside the
postmaster if they can't be authenticated, then?  You either have to
use .pgpass, user kerberos, or use ident; nothing else is safe in the
context you're discussing.  I don't understand the problem.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Tue, 13 May 2003 10:41:34 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: Scheduled jobs
On Tue, May 13, 2003 at 07:18:52AM -0700, Zlatko Michailov wrote:

> remove such rows. Since the account registrations are inside the
> database, it appeals to me that session maintenance should also be
> there.

I still don't see what this is going to buy which cron will not.  And
what you're asking for is that the whole community pay the cost of
maintaining an expensive and redundant piece of functionality because
of your preference.

If you really want that, you can probably implement it yourself. I
know plenty of people (including me) who would argue very strongly
against putting this sort scheduling function inside any release of
PostgreSQL.  It's a maintenance nightmare which adds nothing to cron;
moreover, it's a potential source of some extremely serious bugs,
including security vulnerabilities.  Finally, the effort that might
be expended in maintaining a redundant piece of code like this is
something that could be expended instead on providing functionality
which is at present not available at all.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Tue, 13 May 2003 10:47:37 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: Scheduled jobs
-*- Andrew Sullivan <andrew@libertyrms.info> [ 2003-05-13 14:42 ]:
> On Tue, May 13, 2003 at 01:33:25PM +0000, ??rhallur H?lfd?narson wrote:
> > The suggestion on using ident was to eliminate the need for storing
> > passwords in the first place...
> 
> But how are you going to let them run scheduled jobs inside the
> postmaster if they can't be authenticated, then?  You either have to
> use .pgpass, user kerberos, or use ident; nothing else is safe in the
> context you're discussing.  I don't understand the problem.

I was simply pointing out some scenarios when scheduled jobs are nice. :-)

I believe you have to be authenticated to *create* jobs... and would probably run as the owner, if it gets implemented.

-- 
Regards,
Tolli
tolli@tol.li


Tue, 13 May 2003 14:48:11 +0000
=?iso-8859-1?Q?=DE=F3rhallur_H=E1lfd=E1narson?= <tolli@tol.li>
Re: Scheduled jobs
> > That is, of course, quite separate from having cron in the core.  And
> > having a _good_ set of semantics for the push/pull is a nontrivial

Sometimes I need some external action to be taken when a table is updated. I 
can't find anything about how to do this with triggers.

The easiest would be to have a server listening on a port. But can I write a 
PL/pgSQL trigger that can talk tcp/ip?

There is a short description about untrusted Perl. This might solve the 
problem of talking to the port, but I'm not sure I would like to run anything 
called "Untrusted" in my server!

-- 
Kaare Rasmussen            --Linux, spil,--        Tlf:        3816 2582
Kaki Data                tshirts, merchandize      Fax:        3816 2501
Howitzvej 75               Åben 12.00-18.00        Email: kar@kakidata.dk
2000 Frederiksberg        Lørdag 12.00-16.00       Web:      www.suse.dk


Tue, 13 May 2003 17:04:52 +0200
Kaare Rasmussen <kar@kakidata.dk>
Re: Scheduled jobs
On Tue, 2003-05-13 at 10:48, Þórhallur Hálfdánarson wrote:
> -*- Andrew Sullivan <andrew@libertyrms.info> [ 2003-05-13 14:42 ]:
> > On Tue, May 13, 2003 at 01:33:25PM +0000, ??rhallur H?lfd?narson wrote:
> > > The suggestion on using ident was to eliminate the need for storing
> > > passwords in the first place...
> > 
> > But how are you going to let them run scheduled jobs inside the
> > postmaster if they can't be authenticated, then?  You either have to
> > use .pgpass, user kerberos, or use ident; nothing else is safe in the
> > context you're discussing.  I don't understand the problem.
> 
> I was simply pointing out some scenarios when scheduled jobs are nice. :-)
> 
> I believe you have to be authenticated to *create* jobs... and would probably run as the owner, if it gets implemented.

Wouldn't it make more sense to modify cron to be able to read scheduling
details out of the database -- rather than trying to modify PostgreSQL
to try to feed cron?

See examples of FTP, DNS, etc. software that can read authentication
elements from databases -- and the lack of DBs that have knowledge of
how to push data into those services.
-- 
Rod Taylor <rbt@rbt.ca>

PGP Key: http://www.rbt.ca/rbtpub.asc
13 May 2003 11:07:55 -0400
Rod Taylor <rbt@rbt.ca>
Re: Scheduled jobs
On Tue, May 13, 2003 at 04:55:01PM +0200, Reinoud van Leeuwen wrote:

> One thing comes in mind: portability. Scripts and cron work different on 
> Unix compared to Windows or other platforms. Even cron is not the same on 
> al Unix variants.
> When the scheduling system is inside the database, it works identical on 
> all platforms...

Unless, of course, they have used different versions of an
identically-named library.  In which case you get different
performance anyway, and so then you end up writing a custom library
which is or is not the default for HP-UX version 9 when compiled with
certain options (see another recent thread about exactly such a
case).

That's exactly the sort of terrible maintenance problem that I can
see by implementing such functionality, and I can't see that it's
anywhere near worth the cost.  Given that the behaviour of /bin/ksh
and cron are both POSIX, you can still rely on some standardisation
across platforms.

It seems to me that, if the price of supporting Windows is that
Postgres has to have its own cron, the cost is too high.  I don't
believe that Postgres _does_ need that, however: a scheduling service
is available on Windows that's good enough for these purposes, and
you cannot really expect perfect portability between any flavour of
UNIX and Windows (as anyone who's had to support such a heterogenous
environment knows).  

Of course it's true that if you re-implement every service of every
supported operating system yourself, you get a more portable system. 
But in that case, perhaps someone should start the PostgrOS project. 
(It's a database!  No, it's an operating system!  No, it's a
data-based operating environment!  Wait.  Someone already did that:
PICK.  Nice system, but not SQL.)

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Tue, 13 May 2003 12:18:25 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: Scheduled jobs
On 13 May 2003, Rod Taylor wrote:

> On Tue, 2003-05-13 at 10:48, Þórhallur Hálfdánarson wrote:
> > -*- Andrew Sullivan <andrew@libertyrms.info> [ 2003-05-13 14:42 ]:
> > > On Tue, May 13, 2003 at 01:33:25PM +0000, ??rhallur H?lfd?narson wrote:
> > > > The suggestion on using ident was to eliminate the need for storing
> > > > passwords in the first place...
> > > 
> > > But how are you going to let them run scheduled jobs inside the
> > > postmaster if they can't be authenticated, then?  You either have to
> > > use .pgpass, user kerberos, or use ident; nothing else is safe in the
> > > context you're discussing.  I don't understand the problem.
> > 
> > I was simply pointing out some scenarios when scheduled jobs are nice. :-)
> > 
> > I believe you have to be authenticated to *create* jobs... and would probably run as the owner, if it gets implemented.
> 
> Wouldn't it make more sense to modify cron to be able to read scheduling
> details out of the database -- rather than trying to modify PostgreSQL
> to try to feed cron?
> 
> See examples of FTP, DNS, etc. software that can read authentication
> elements from databases -- and the lack of DBs that have knowledge of
> how to push data into those services.

Bingo, Rod.  You obviously reached across the miles into my head and stole 
that from my brain, because honestly I was about 30 seconds from posting 
the same thing.


Tue, 13 May 2003 10:59:46 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Scheduled jobs
Tolli wrote:
> -*- Andrew Sullivan <andrew@libertyrms.info> [ 2003-05-13 14:42 ]:
> > On Tue, May 13, 2003 at 01:33:25PM +0000, ??rhallur H?lfd?narson wrote:
> > > The suggestion on using ident was to eliminate the need for storing
> > > passwords in the first place...
> > 
> > But how are you going to let them run scheduled jobs inside the
> > postmaster if they can't be authenticated, then?  You either have to
> > use .pgpass, user kerberos, or use ident; nothing else is safe in the
> > context you're discussing.  I don't understand the problem.

> I was simply pointing out some scenarios when scheduled jobs are
> nice. :-)

"Nice" does not dictate "Someone should be responsible for the
implementation."

In the old fable about the mice and the cat, it would sure be "nice" if
they could put a bell on the cat so the mice could hear the cat coming.
But in the fable, none of the mice were prepared to risk life and limb
getting the bell put onto the cat.

In this case, the fact that you'd like a scheduler does not imply that
anyone will want to take the job on.

> I believe you have to be authenticated to *create* jobs... and would
> probably run as the owner, if it gets implemented.

No, these "jobs" would run as the "postgres" user.  (Or whatever user
it is that the PostgreSQL server runs as.)

And there enters a *big* whack of complexity, particularly if that
isn't the right answer.

It rapidly turns into a *very* complex system that, even with MS-SQL
and Oracle, isn't really part of the database.  Why is it complex?
Because of the need to be able to change user roles to different
system users, which is inherently system-dependent (e.g. - very
different between Unix and Windows) and *highly* security-sensitive.

I agree with the thoughts that it would be a slick idea to come up
with a way of having PostgreSQL be the "data store" for some outside
scheduling tool.  You likely won't have something that anyone will
have compete with Cron or Maestro or [whatever they call the Windows
'scheduler'], but it could be useful to those that care.  And by
keeping it separate, those of us that don't care don't get a bloated
system.
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://www3.sympatico.ca/cbbrowne/spreadsheets.html
Think of C++ as an object-oriented assembly language.


Tue, 13 May 2003 16:35:28 -0400
Christopher Browne <cbbrowne@cbbrowne.com>
Re: Scheduled jobs 
Andrew Sullivan wrote:
> Of course it's true that if you re-implement every service of every
> supported operating system yourself, you get a more portable system. 
> But in that case, perhaps someone should start the PostgrOS project. 
> (It's a database!  No, it's an operating system!  No, it's a
> data-based operating environment!  Wait.  Someone already did that:
> PICK.  Nice system, but not SQL.)

MaVerick apparently implements something Pick-like on top of
PostgreSQL... <http://www.maverick-dbms.org/articles/article1.html>

And IBM Universe and UniData *do* make this into SQL...
http://www-3.ibm.com/software/data/u2/pubs/whitepapers/nested_rdbms.pdf

And it's also worth considering that we have array types that support
something Like MV, although that's quite a separate debate.  

Or perhaps not; let me suggest the thought that it would be more
worthwhile to examine the notion of adding MV SQL keywords to PostgreSQL
than it would be to try adding a batch scheduler...
--
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne> rate me
http://www.ntlug.org/~cbbrowne/multiplexor.html
Rules of  the Evil Overlord #47. "If  I learn that a  callow youth has
begun a  quest to  destroy me,  I will slay  him while  he is  still a
callow youth instead of waiting for him to mature."
<http://www.eviloverlord.com/>


Tue, 13 May 2003 16:35:40 -0400
Christopher Browne <cbbrowne@cbbrowne.com>
Re: Scheduled jobs 
Hi

-*- Christopher Browne <cbbrowne@cbbrowne.com> [ 2003-05-13 20:38 ]:
> Tolli wrote:
> "Nice" does not dictate "Someone should be responsible for the
> implementation."
> 
> In the old fable about the mice and the cat, it would sure be "nice" if
> they could put a bell on the cat so the mice could hear the cat coming.
> But in the fable, none of the mice were prepared to risk life and limb
> getting the bell put onto the cat.
> 
> In this case, the fact that you'd like a scheduler does not imply that
> anyone will want to take the job on.

As I said in my original reply to Tom: "Just mentioning some pros I see -- I do agree with your point on resources and future maintenance."

The point being, which I might have stated explicitly, that if someone (for example Zlatko who originally suggested it) will go on implementing it, I believe it helps is indeed good.  Weather or not it should be included in the main distribution is a matter of a totally seperate debate later on. :-)

> > I believe you have to be authenticated to *create* jobs... and would
> > probably run as the owner, if it gets implemented.
> 
> No, these "jobs" would run as the "postgres" user.  (Or whatever user
> it is that the PostgreSQL server runs as.)
> 
> And there enters a *big* whack of complexity, particularly if that
> isn't the right answer.

Eeek!  What I've been thinking about all along is something for running, err, SQL (which therefor can be run as the owner) or some internal tasks -- nothing with external processes.  

> It rapidly turns into a *very* complex system that, even with MS-SQL
> and Oracle, isn't really part of the database.  Why is it complex?
> Because of the need to be able to change user roles to different
> system users, which is inherently system-dependent (e.g. - very
> different between Unix and Windows) and *highly* security-sensitive.
> 
> I agree with the thoughts that it would be a slick idea to come up
> with a way of having PostgreSQL be the "data store" for some outside
> scheduling tool.  You likely won't have something that anyone will
> have compete with Cron or Maestro or [whatever they call the Windows
> 'scheduler'], but it could be useful to those that care.  And by
> keeping it separate, those of us that don't care don't get a bloated
> system.

I sincerely agree that I'd not like to see PostgreSQL bloated with a cron-wannabe. ;-)

-- 
Regards,
Tolli
tolli@tol.li


Tue, 13 May 2003 21:09:16 +0000
=?iso-8859-1?Q?=DE=F3rhallur_H=E1lfd=E1narson?= <tolli@tol.li>
Re: Scheduled jobs
OK, here's an idea.  You write a set of stored procs that let you do 
something like:

insert into batch_jobs ('.....  I'm not sure what we'd put here...)

then, postgresql has a crontab entry that uses something like redhats 
runparts script to run the SQL commands it finds in the table.

I.e. the jobs could be scheduled by something as simple as a query, and 
removed as well.  Just need a postgresql cron that runs every 5 minutes or 
whatever resolution you need.


Tue, 13 May 2003 15:51:28 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Scheduled jobs
pgsql-core have agreed to put out a 7.3.3 release on Wednesday (5/21),
God willin' an' the creek don't rise.  If anyone's got anything you've
been planning to fix in the 7.3 branch, now is a real good time to get
it done.

			regards, tom lane

Fri, 16 May 2003 01:15:28 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Heads up: 7.3.3 this Wednesday
Tom Lane wrote:
> pgsql-core have agreed to put out a 7.3.3 release on Wednesday (5/21),
> God willin' an' the creek don't rise.  If anyone's got anything you've
> been planning to fix in the 7.3 branch, now is a real good time to get
> it done.
> 

Funny you should ask :-)

Here's a fix for a bug in connectby (crashes when called as a targetlist 
function instead of as a table function). The patch is against cvs HEAD, 
but should also be applied to the 7.3 branch, I think.

Thanks,

Joe
Thu, 15 May 2003 22:49:31 -0700
Joe Conway <mail@joeconway.com>
tablefunc bugfix (was Re: [HACKERS] Heads up: 7.3.3 this Wednesday)
> pgsql-core have agreed to put out a 7.3.3 release on Wednesday (5/21),
> God willin' an' the creek don't rise.  If anyone's got anything you've
> been planning to fix in the 7.3 branch, now is a real good time to get
> it done.

Only thing is that %rowtype and dropped columns business - but I think you
indicated that would be a 7.4 fix...I think it's a bit too complex for me...

Chris


Fri, 16 May 2003 14:01:19 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: Heads up: 7.3.3 this Wednesday
Joe Conway <mail@joeconway.com> writes:
> Here's a fix for a bug in connectby (crashes when called as a targetlist 
> function instead of as a table function). The patch is against cvs HEAD, 
> but should also be applied to the 7.3 branch, I think.

Right-o, applied in both branches.

			regards, tom lane

Fri, 16 May 2003 02:08:32 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: tablefunc bugfix (was Re: [HACKERS] Heads up: 7.3.3 this
	Wednesday)
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes:
> Only thing is that %rowtype and dropped columns business - but I think you
> indicated that would be a 7.4 fix...I think it's a bit too complex for me...

I think only low-risk bug fixes need apply for 7.3 at this point ... the
dropped-col thing needs some study ...

			regards, tom lane

Fri, 16 May 2003 02:12:46 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 
On Fri, 16 May 2003, Tom Lane wrote:

> pgsql-core have agreed to put out a 7.3.3 release on Wednesday (5/21),
> God willin' an' the creek don't rise.  If anyone's got anything you've
> been planning to fix in the 7.3 branch, now is a real good time to get
> it done.

I think Bruce should check his mail box for unapplied patches.
I suspect some of our patches are still waiting for attention.

        Oleg


>
> 			regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>

	Regards,
		Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Fri, 16 May 2003 11:46:27 +0400 (MSD)
Oleg Bartunov <oleg@sai.msu.su>
Re: Heads up: 7.3.3 this Wednesday
Oleg Bartunov <oleg@sai.msu.su> writes:
> On Fri, 16 May 2003, Tom Lane wrote:
>> pgsql-core have agreed to put out a 7.3.3 release on Wednesday (5/21),

> I think Bruce should check his mail box for unapplied patches.
> I suspect some of our patches are still waiting for attention.

Bruce is going to be mostly out of the loop on this release (he's out of
town this weekend, and planning a server upgrade Monday).  So if you've
got any problems, let me know about 'em.

I have just finished digging through the pgsql-patches archives back to
February, and found only a few small things that seemed appropriate for
back-patching.  I do seem to recall seeing some fixes from you and
Teodor recently, though.  Could you check REL7_3_STABLE CVS tip against
what you have, and either resubmit any missing patches or point me to
where they're archived?

			regards, tom lane

Fri, 16 May 2003 11:25:29 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 
I sended it before (24 Apr), but I don't sure that it applied.

This patch fixes nt[]-int[] operation


Tom Lane wrote:
> Oleg Bartunov <oleg@sai.msu.su> writes:
> 
>>On Fri, 16 May 2003, Tom Lane wrote:
>>
>>>pgsql-core have agreed to put out a 7.3.3 release on Wednesday (5/21),
> 
> 
>>I think Bruce should check his mail box for unapplied patches.
>>I suspect some of our patches are still waiting for attention.
> 
> 
> Bruce is going to be mostly out of the loop on this release (he's out of
> town this weekend, and planning a server upgrade Monday).  So if you've
> got any problems, let me know about 'em.
> 
> I have just finished digging through the pgsql-patches archives back to
> February, and found only a few small things that seemed appropriate for
> back-patching.  I do seem to recall seeing some fixes from you and
> Teodor recently, though.  Could you check REL7_3_STABLE CVS tip against
> what you have, and either resubmit any missing patches or point me to
> where they're archived?
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

-- 
Teodor Sigaev                                  E-mail: teodor@sigaev.ru
Fri, 16 May 2003 22:30:32 +0400
Teodor Sigaev <teodor@sigaev.ru>
Re: Heads up: 7.3.3 this Wednesday
Teodor Sigaev <teodor@sigaev.ru> writes:
> I sended it before (24 Apr), but I don't sure that it applied.
> This patch fixes nt[]-int[] operation

Yeah, you're right, it hadn't been applied yet.  Done now; thanks!

			regards, tom lane

Fri, 16 May 2003 14:52:47 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 
Tom Lane wrote:
> pgsql-core have agreed to put out a 7.3.3 release on Wednesday (5/21),
> God willin' an' the creek don't rise.  If anyone's got anything you've
> been planning to fix in the 7.3 branch, now is a real good time to get
> it done.

I've seen no problems with the deferred trigger patch (which addresses
a performance issue with deferred triggers) you gave me some time ago.
Is this something that's likely to be included in 7.3.3?


-- 
Kevin Brown					      kevin@sysexperts.com

Fri, 16 May 2003 12:20:04 -0700
Kevin Brown <kevin@sysexperts.com>
Re: Heads up: 7.3.3 this Wednesday
Kevin Brown <kevin@sysexperts.com> writes:
> I've seen no problems with the deferred trigger patch (which addresses
> a performance issue with deferred triggers) you gave me some time ago.
> Is this something that's likely to be included in 7.3.3?

I've been wondering myself about whether to include Jan's patch that
reduces foreign-key deadlocks.  Neither of these patches have gotten
enough testing (that I know of) to make me feel very comfortable about
dropping them into 7.3.3 ... but on the other hand, they could be pretty
significant fixes.

Any votes pro or con?  Who else can report successful use of either of
these patches?

			regards, tom lane

Fri, 16 May 2003 16:18:25 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 
> > I've seen no problems with the deferred trigger patch (which
> > addresses a performance issue with deferred triggers) you gave me
> > some time ago.  Is this something that's likely to be included in
> > 7.3.3?
> 
> I've been wondering myself about whether to include Jan's patch that
> reduces foreign-key deadlocks.  Neither of these patches have gotten
> enough testing (that I know of) to make me feel very comfortable
> about dropping them into 7.3.3 ... but on the other hand, they could
> be pretty significant fixes.
> 
> Any votes pro or con?  Who else can report successful use of either
> of these patches?

I've been using the deferred triggers patch in production for about a
month without any problems.  I'm definitely for that as well as Jan's
patch, though I haven't used it.  Anything to help out FK's and I'm
game.  :)

If someone needs the clean patch for 7.3 of the deferred triggers, the
patch is at the URL below.  The one originally posted didn't merge 100%
cleanly to my 7.3.2 sources.

http://people.FreeBSD.org/~seanc/patch_postgresql-7.3.2::src::backend::utils::adt::ri_triggers.c

 -sc

-- 
Sean Chittenden

Fri, 16 May 2003 13:26:50 -0700
Sean Chittenden <sean@chittenden.org>
Re: Heads up: 7.3.3 this Wednesday
Is is possible to make it a build time switch so many folks can beta test 
it for us, so to speak?  I'd certainly be willing to test it out a bit, 
but we don't have time to test it before 7.3.3 comes out.

That would allow us to basically test the deferred triggers in a 
relatively stable code base (7.3) on semi-production machines (i.e. the 
ones running batch files at night and such) long before 7.4 goes beta.

Or is it too complex or ugly to put something like that into configure and 
the code?  Just a thought.

On Fri, 16 May 2003, Tom Lane wrote:

> Kevin Brown <kevin@sysexperts.com> writes:
> > I've seen no problems with the deferred trigger patch (which addresses
> > a performance issue with deferred triggers) you gave me some time ago.
> > Is this something that's likely to be included in 7.3.3?
> 
> I've been wondering myself about whether to include Jan's patch that
> reduces foreign-key deadlocks.  Neither of these patches have gotten
> enough testing (that I know of) to make me feel very comfortable about
> dropping them into 7.3.3 ... but on the other hand, they could be pretty
> significant fixes.
> 
> Any votes pro or con?  Who else can report successful use of either of
> these patches?
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/docs/faqs/FAQ.html
> 


Fri, 16 May 2003 15:13:37 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Heads up: 7.3.3 this Wednesday 
"scott.marlowe" <scott.marlowe@ihs.com> writes:
> Is is possible to make it a build time switch so many folks can beta test 
> it for us, so to speak?

I doubt it'd get tested enough to notice, if it's not in the default
build.

I actually think that both of these are pretty good candidates to put
into 7.3.3.  I'm just trying to adopt an appropriately paranoid stance
and ask hard questions about how much they've been tested.  Between
Kevin and Sean it seems that the deferred-triggers change has gotten
enough testing to warrant some trust, but I'm not hearing anything
about the FK-deadlock one :-(.

BTW, if anyone is looking for that patch, it was at
http://archives.postgresql.org/pgsql-hackers/2003-04/msg00260.php

			regards, tom lane

Fri, 16 May 2003 17:36:32 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 
> > Is is possible to make it a build time switch so many folks can beta test 
> > it for us, so to speak?
> 
> I doubt it'd get tested enough to notice, if it's not in the default
> build.
> 
> I actually think that both of these are pretty good candidates to put
> into 7.3.3.  I'm just trying to adopt an appropriately paranoid stance
> and ask hard questions about how much they've been tested.  Between
> Kevin and Sean it seems that the deferred-triggers change has gotten
> enough testing to warrant some trust, but I'm not hearing anything
> about the FK-deadlock one :-(.
> 
> BTW, if anyone is looking for that patch, it was at
> http://archives.postgresql.org/pgsql-hackers/2003-04/msg00260.php

Are there any test cases to get this bug to fire?  I haven't had any
problems with this particular bug so I can apply this patch, but I
can't promise that my use of the patch will result in anything useful
other than, "nothing's broke yet" since I haven't had any real
problems with 7.3.2 other than the deferred trigger speed.  Off hand,
I don't see why this patch would cause any problems if it were
applied.  -sc

-- 
Sean Chittenden

Fri, 16 May 2003 14:44:53 -0700
Sean Chittenden <sean@chittenden.org>
Re: Heads up: 7.3.3 this Wednesday
Sean Chittenden <sean@chittenden.org> writes:
>> BTW, if anyone is looking for that patch, it was at
>> http://archives.postgresql.org/pgsql-hackers/2003-04/msg00260.php

> Are there any test cases to get this bug to fire?  I haven't had any
> problems with this particular bug so I can apply this patch, but I
> can't promise that my use of the patch will result in anything useful
> other than, "nothing's broke yet" since I haven't had any real
> problems with 7.3.2 other than the deferred trigger speed.

"Nothing's broke yet" would be a useful report.  I just would like to
see some more mileage on the beast before we let it loose on the
unsuspecting world.  Test cases aren't really the point --- we know what
we expect them to do.  It's the cases we didn't think of that worry me
at times like this.

			regards, tom lane

Fri, 16 May 2003 17:51:54 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 
On Fri, 16 May 2003, Tom Lane wrote:

> "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > Is is possible to make it a build time switch so many folks can beta test 
> > it for us, so to speak?
> 
> I doubt it'd get tested enough to notice, if it's not in the default
> build.

Well, we could reverse that and make it the default, and if you find a FK 
problem folks can turn it off.

But that probably is sub optimal too.


Fri, 16 May 2003 15:57:20 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Heads up: 7.3.3 this Wednesday 
Did that 'prevent cluster on partial and non-NULL indexes' patch get
backported?

Chris



Sat, 17 May 2003 23:32:50 +0800 (WST)
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: Heads up: 7.3.3 this Wednesday 
Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes:
> Did that 'prevent cluster on partial and non-NULL indexes' patch get
> backported?

This one?

2003-03-02 23:37  tgl

	* src/backend/commands/: cluster.c (REL7_3_STABLE), cluster.c:
	Prevent clustering on incomplete indexes: partial indexes are
	verboten, as are non-amindexnulls AMs unless first column is
	attnotnull.

			regards, tom lane

Sat, 17 May 2003 11:38:13 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 

> I doubt it'd get tested enough to notice, if it's not in the default
> build.
> 
> I actually think that both of these are pretty good candidates to put
> into 7.3.3.  I'm just trying to adopt an appropriately paranoid stance
> and ask hard questions about how much they've been tested.  Between
> Kevin and Sean it seems that the deferred-triggers change has gotten
> enough testing to warrant some trust, but I'm not hearing anything
> about the FK-deadlock one :-(.
> 
> BTW, if anyone is looking for that patch, it was at
> http://archives.postgresql.org/pgsql-hackers/2003-04/msg00260.php

7.3.2: I applied the above patch and did install and restarted postgresql,
but the 'deadlock detected' error on FK update still exist. The below is
the test case. Someone *advice* me, if it the above mentioned patch is not
intended to address the below case.

Test case:

test_pg=# CREATE TABLE prim_test (id int primary key);
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index 
'prim_test_pkey'
for table 'prim_test'
CREATE TABLE
test_pg=# CREATE TABLE for_test (id int references prim_test(id), name 
text);
NOTICE:  CREATE TABLE will create implicit trigger(s) for FOREIGN KEY 
check(s)
CREATE TABLE
test_pg=# INSERT INTO prim_test VALUES ('1');
INSERT 4383707 1
test_pg=# INSERT INTO prim_test VALUES ('2');
INSERT 4383708 1
test_pg=# INSERT INTO for_test VALUES (1, 'foo');
INSERT 4383710 1
test_pg=# INSERT INTO for_test VALUES (2, 'bar');
INSERT 4383711 1

t1:
test_pg=# BEGIN ;
BEGIN
test_pg=# UPDATE for_test set name ='FOO' where id = 1;
UPDATE 1
test_pg=# UPDATE for_test set name ='Bar' where id = 2;
UPDATE 1

t2:
test_pg=# BEGIN ;
BEGIN
test_pg=# UPDATE for_test set name = 'BAR' where id = 2;
UPDATE 1
test_pg=# UPDATE for_test set name = 'Foo' where id = 1;
ERROR:  deadlock detected

regards,
bhuvaneswaran




Mon, 19 May 2003 12:48:24 +0530 (IST)
"A.Bhuvaneswaran" <bhuvanbk@yahoo.com>
Re: Heads up: 7.3.3 this Wednesday 
"A.Bhuvaneswaran" <bhuvanbk@yahoo.com> writes:
> 7.3.2: I applied the above patch and did install and restarted postgresql,
> but the 'deadlock detected' error on FK update still exist. The below is
> the test case. Someone *advice* me, if it the above mentioned patch is not
> intended to address the below case.

That is not a foreign-key deadlock; it's a plain old deadlock.  It would
happen exactly the same way without the foreign key, because the
contention is directly for the rows being updated.

An example of what the patch fixes:

regression=# CREATE TABLE prim_test (id int primary key);
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index 'prim_test_pkey' for table 'prim_test'
CREATE TABLE
regression=# CREATE TABLE for_test (id int, name text,
regression(# ref int references prim_test(id));
NOTICE:  CREATE TABLE will create implicit trigger(s) for FOREIGN KEY check(s)
CREATE TABLE
regression=# INSERT INTO prim_test VALUES ('1');
INSERT 566517 1
regression=# INSERT INTO prim_test VALUES ('2');
INSERT 566518 1
regression=# INSERT INTO for_test VALUES (11, 'foo', 1);
INSERT 566520 1
regression=# INSERT INTO for_test VALUES (12, 'fooey', 1);
INSERT 566521 1
regression=# INSERT INTO for_test VALUES (21, 'fooey', 2);
INSERT 566522 1
regression=# INSERT INTO for_test VALUES (22, 'fooey', 2);
INSERT 566523 1
regression=# begin;
BEGIN
regression=# UPDATE for_test set name ='FOO' where id = 11;
UPDATE 1

-- in client 2 do

regression=# begin;
BEGIN
regression=# UPDATE for_test set name = 'BAR' where id = 22;
UPDATE 1
regression=# UPDATE for_test set name = 'BAR' where id = 12;
UPDATE 1

-- back to client 1, do

regression=# UPDATE for_test set name ='FOO' where id = 21;
UPDATE 1

This deadlocks in 7.3, but works in CVS tip.

			regards, tom lane

Mon, 19 May 2003 10:24:05 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 
> regression=# UPDATE for_test set name ='FOO' where id = 21;
> UPDATE 1
>
> This deadlocks in 7.3, but works in CVS tip.

Hmmm...I suspect this will remove a lot of the FK deadlocks I see in my logs
all the time...does seem like a bug doesn't it?

Chris


Tue, 20 May 2003 10:42:58 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: Heads up: 7.3.3 this Wednesday 
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes:
>> This deadlocks in 7.3, but works in CVS tip.

> Hmmm...I suspect this will remove a lot of the FK deadlocks I see in my logs
> all the time...does seem like a bug doesn't it?

Yeah.  I'm just worried that there might be some downside we've not
spotted yet.  I'd feel better about it if *anyone* had reported
successful production use of the patch in the month since it's been
available.  I thought one or two people had expressed the intention
to run the patch when Jan offered it ... where are they?

			regards, tom lane

Mon, 19 May 2003 23:38:45 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 
Tom Lane wrote:
> Yeah.  I'm just worried that there might be some downside we've not
> spotted yet.  I'd feel better about it if *anyone* had reported
> successful production use of the patch in the month since it's been
> available.  I thought one or two people had expressed the intention
> to run the patch when Jan offered it ... where are they?

I put it in on some DBs we were doing testing against at Liberty; the
results unfortunately were inconclusive.  

We didn't see any noticeable improvement in performance out of it,
albeit with limited use, because we weren't planning to use 7.3 in
production just yet, and so did the vast majority of the testing against
7.2.4, where the patch doesn't even come close to applying...

The result is that I can't suggest either a "yea" or "nay"...
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://www.ntlug.org/~cbbrowne/spiritual.html
If con is the opposite of pro, is Congress the opposite of progress?

Tue, 20 May 2003 07:16:21 -0400
Christopher Browne <cbbrowne@cbbrowne.com>
Re: Heads up: 7.3.3 this Wednesday 
On Mon, 19 May 2003, Tom Lane wrote:

> "Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes:
> >> This deadlocks in 7.3, but works in CVS tip.
> 
> > Hmmm...I suspect this will remove a lot of the FK deadlocks I see in my logs
> > all the time...does seem like a bug doesn't it?
> 
> Yeah.  I'm just worried that there might be some downside we've not
> spotted yet.  I'd feel better about it if *anyone* had reported
> successful production use of the patch in the month since it's been
> available.  I thought one or two people had expressed the intention
> to run the patch when Jan offered it ... where are they?

I'd be glad to test it, but we don't have any issues with fk deadlocks 
since our load is 99% read, 1% write, and most of the tables with fks on 
them only have a handful of writers, so any testing I would do would 
probably just be the "we used it in production and it didn't die" kind of 
testing.


Tue, 20 May 2003 09:31:50 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Heads up: 7.3.3 this Wednesday 
"scott.marlowe" <scott.marlowe@ihs.com> writes:
> I'd be glad to test it, but we don't have any issues with fk deadlocks 
> since our load is 99% read, 1% write, and most of the tables with fks on 
> them only have a handful of writers, so any testing I would do would 
> probably just be the "we used it in production and it didn't die" kind of 
> testing.

That's what I'm looking for mostly: that it does not have any adverse
side-effects.

			regards, tom lane

Tue, 20 May 2003 11:54:11 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 
On Tue, 20 May 2003, Tom Lane wrote:

> "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > I'd be glad to test it, but we don't have any issues with fk deadlocks 
> > since our load is 99% read, 1% write, and most of the tables with fks on 
> > them only have a handful of writers, so any testing I would do would 
> > probably just be the "we used it in production and it didn't die" kind of 
> > testing.
> 
> That's what I'm looking for mostly: that it does not have any adverse
> side-effects.

So where's that patch again?  The search function of the mail archives is 
broken, so I can't seem to find it.




Tue, 20 May 2003 10:08:07 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Heads up: 7.3.3 this Wednesday 
I wrote:
> pgsql-core have agreed to put out a 7.3.3 release on Wednesday (5/21),
> God willin' an' the creek don't rise.

Well, the postgresql.org server move has proven to be much messier than
we hoped, so it seems prudent to delay a couple days while the kinks
get worked out.  New plan is 7.3.3 release on Friday (5/23).

			regards, tom lane

Tue, 20 May 2003 13:04:43 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Heads up: 7.3.3 this, er, Friday
On Tue, 20 May 2003, scott.marlowe wrote:

> On Tue, 20 May 2003, Tom Lane wrote:
>
> > "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > > I'd be glad to test it, but we don't have any issues with fk deadlocks
> > > since our load is 99% read, 1% write, and most of the tables with fks on
> > > them only have a handful of writers, so any testing I would do would
> > > probably just be the "we used it in production and it didn't die" kind of
> > > testing.
> >
> > That's what I'm looking for mostly: that it does not have any adverse
> > side-effects.
>
> So where's that patch again?  The search function of the mail archives is
> broken, so I can't seem to find it.
>

did you try http://fts.postgresql.org/ ? It should works


>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>

	Regards,
		Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Tue, 20 May 2003 21:36:00 +0400 (MSD)
Oleg Bartunov <oleg@sai.msu.su>
Re: Heads up: 7.3.3 this Wednesday 
scott.marlowe wrote:
> On Tue, 20 May 2003, Tom Lane wrote:
> 
> 
>>"scott.marlowe" <scott.marlowe@ihs.com> writes:
>>
>>>I'd be glad to test it, but we don't have any issues with fk deadlocks 
>>>since our load is 99% read, 1% write, and most of the tables with fks on 
>>>them only have a handful of writers, so any testing I would do would 
>>>probably just be the "we used it in production and it didn't die" kind of 
>>>testing.
>>
>>That's what I'm looking for mostly: that it does not have any adverse
>>side-effects.
> 
> 
> So where's that patch again?  The search function of the mail archives is 
> broken, so I can't seem to find it.
> 

Attached.


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #
Tue, 20 May 2003 15:32:55 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: Heads up: 7.3.3 this Wednesday
Thanks!  We should just replace the search screen (that doesn't work 
anyway) on the archives with that one.  That fts site rocks!

I found it on the ftp server by the way, in the patches directory of all 
places.  I'll put it online on our backup server tonight.

On Tue, 20 May 2003, Oleg Bartunov wrote:

> On Tue, 20 May 2003, scott.marlowe wrote:
> 
> > On Tue, 20 May 2003, Tom Lane wrote:
> >
> > > "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > > > I'd be glad to test it, but we don't have any issues with fk deadlocks
> > > > since our load is 99% read, 1% write, and most of the tables with fks on
> > > > them only have a handful of writers, so any testing I would do would
> > > > probably just be the "we used it in production and it didn't die" kind of
> > > > testing.
> > >
> > > That's what I'm looking for mostly: that it does not have any adverse
> > > side-effects.
> >
> > So where's that patch again?  The search function of the mail archives is
> > broken, so I can't seem to find it.
> >
> 
> did you try http://fts.postgresql.org/ ? It should works
> 
> 
> >
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 2: you can get off all lists at once with the unregister command
> >     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> >
> 
> 	Regards,
> 		Oleg
> _____________________________________________________________
> Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> Sternberg Astronomical Institute, Moscow University (Russia)
> Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
> phone: +007(095)939-16-83, +007(095)939-23-83
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/docs/faqs/FAQ.html
> 


Tue, 20 May 2003 14:43:56 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Heads up: 7.3.3 this Wednesday 
Andrew Sullivan <andrew@libertyrms.info> writes:
> Yes, 3 weeks of testing with the patch that Jan provided for 7.2.  We
> had no failures that I know of.  It provided no measurable
> performance gain, but it also never deadlocked, and we tested under
> conditions where it sometimes did in the past.

> Note, however, that we were testing it indirectly; that is, while we
> were testing to see if it would break, our application (which does a
> number of the referential checks itself, multiple-connection-safety
> notwithstanding :( ) tends not to try to violate the foreign keys
> anyway.  I wouldn't want to claim that we've tested it real heavily.

Nonetheless, this does seem to speak to my real concern, which is
whether the patch introduces any unexpected side-effects.  The code
was getting executed, whether or not it detected any FK violations,
so we can have some hope that any bizarre problems would have been
noticed.

I'll go ahead and apply it for 7.3.3.

			regards, tom lane

Wed, 21 May 2003 13:40:02 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 
On Wed, May 21, 2003 at 01:40:02PM -0400, Tom Lane wrote:
> was getting executed, whether or not it detected any FK violations,
> so we can have some hope that any bizarre problems would have been
> noticed.

Certainly, bizarre problems would have been noticed.  It's safe to
apply, if you ask me.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Wed, 21 May 2003 16:52:04 -0400
Andrew Sullivan <andrew@libertyrms.info>
Re: Heads up: 7.3.3 this Wednesday
Tom Lane wrote:
> I actually think that both of these are pretty good candidates to put
> into 7.3.3.  I'm just trying to adopt an appropriately paranoid stance
> and ask hard questions about how much they've been tested.  Between
> Kevin and Sean it seems that the deferred-triggers change has gotten
> enough testing to warrant some trust, but I'm not hearing anything
> about the FK-deadlock one :-(.
> 
> BTW, if anyone is looking for that patch, it was at
> http://archives.postgresql.org/pgsql-hackers/2003-04/msg00260.php

I've been running with this patch ever since I upgraded to 7.3.2 some
time back, without any ill effects at all, FWIW...


-- 
Kevin Brown					      kevin@sysexperts.com

Wed, 21 May 2003 21:56:59 -0700
Kevin Brown <kevin@sysexperts.com>
Re: Heads up: 7.3.3 this Wednesday
Kevin Brown <kevin@sysexperts.com> writes:
> I've been running with this patch ever since I upgraded to 7.3.2 some
> time back, without any ill effects at all, FWIW...

Andrew Sullivan also reported doing a fair bit of testing without
noticing any bad side-effects, so I've gone ahead and applied it for
7.3.3.   Appreciate the confirmation though...

			regards, tom lane

Thu, 22 May 2003 01:15:04 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Heads up: 7.3.3 this Wednesday 
I worked on project at GBorg to implement:

>     - a specified moment
>     - a time of the day/month/year
>     - recurring at a time interval

But now the project is frozen for some time, exist only UML.

Expected features were:

1. All transactions from job spool performed in one connection serially. 
   So on multy processors servers spooled jobs will eat only one CPU. 
And job performed in expected order.
2. Guarantee launch job from spool in none guarantee time, even if 
server crashed when job transaction not ended.

Cron can't implement this.

-- 
Olleg


Thu, 22 May 2003 15:57:12 +0400
Olleg Samojlov <olleg@telecom.mipt.ru>
Re: Scheduled jobs
Peter Eisentraut writes:
> I think not.  It's a little tricky handling it directly in the child
> processes, but it's been done before.

Certainly has...

In the cfengine2 code base, the relevant file is "rotate.c"; it
essentially attaches the file descriptor to a new file "on the fly," and
does so as a separate process.  The code apparently works on NT, too;
there is a comment that indicates that they use chown() rather than
fchown() because the latter doesn't exist on NT.

In fact, that file is well worth taking a look at for strategies on
this.  It's GPLed code, and so may not be suitable for integration, but
there are doubtless some useful techniques to be seen...
--
wm(X,Y):-write(X),write('@'),write(Y). wm('cbbrowne','cbbrowne.com').
http://cbbrowne.com/info/languages.html
"Microsoft is sort of a mixture between the Borg and the
Ferengi. Combine the Borg marketing with Ferengi networking..."
-- Andre Beck in dcouln

Fri, 23 May 2003 22:42:58 -0400
Christopher Browne <cbbrowne@cbbrowne.com>
Re: more contrib: log rotator 

I assume we are not moving in the XML/psql direction, right?  We want it
int he backend, or the psql HTML converted to XHTML?

---------------------------------------------------------------------------

greg@turnstep.com wrote:
[ There is text before PGP section. ]
> 
[ PGP not available, raw data follows ]
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> Patch to add XML output to psql:
> 
> http://www.gtsm.com/xml.patch.txt
> 
> Notes and questions:
> 
> The basic output looks something like this:
> 
> <?xml version="1.0" encoding="SQL_ASCII"?>
> <resultset psql_version="7.4devel" query="select * from foo;">
> 
> <columns>
>   <col num="1">a</col>
>   <col num="2">b</col>
>   <col num="3">c</col>
>   <col num="4">mucho nacho  </col>
> </columns>
> <row num="1">
>   <a>1</a>
>   <b>pizza</b>
>   <c>2003-02-25 15:19:22.169797</c>
>   <"mucho nacho  "></"mucho nacho  ">
> </row>
> <row num="2">
>   <a>2</a>
>   <b>mushroom</b>
>   <c>2003-02-25 15:19:26.969415</c>
>   <"mucho nacho  "></"mucho nacho  ">
> </row>
> <footer>(2 rows)</footer>
> </resultset>
> 
> and with the \x option:
> 
> <?xml version="1.0" encoding="SQL_ASCII"?>
> <resultset psql_version="7.4devel" query="select * from foo;">
> 
> <columns>
>   <col num="1">a</col>
>   <col num="2">b</col>
>   <col num="3">c</col>
>   <col num="4">mucho nacho  </col>
> </columns>
> <row num="1">
>   <cell name="a">1</cell>
>   <cell name="b">pizza</cell>
>   <cell name="c">2003-02-25 15:19:22.169797</cell>
>   <cell name="mucho nacho  "></cell>
> </row>
> <row num="2">
>   <cell name="a">2</cell>
>   <cell name="b">mushroom</cell>
>   <cell name="c">2003-02-25 15:19:26.969415</cell>
>   <cell name="mucho nacho  "></cell>
> </row>
> </resultset>
> 
> 
> The default encoding "SQL-ASCII" is not valid for XML. 
> Should it be automatically changed to something else?
> 
> The flag "-X" is already taken, unfortunately, although \X is not. 
> I used "-L" and "\L" but they are not as memorable as "X". Anyone 
> see a way around this? Can we still use \X inside of psql?
> 
> 
> It would be nice to include the string representation of the column 
> types in the xml output:
> <col type="int8">foo</col>
> ....but I could not find an easy way to do this: PQftype returns the 
> OID only (which is close but not quite there). Is there an 
> existing way to get the name of the type of a column from a 
> PQresult item?
> 
> The HTML, XML, and Latex modes should have better documentation - 
> I'll submit a separate doc patch when/if this gets finalized.
> 
> 
> - --
> Greg Sabino Mullane greg@turnstep.com
> PGP Key: 0x14964AC8 200302261518
> 
> -----BEGIN PGP SIGNATURE-----
> Comment: http://www.turnstep.com/pgp.html
> 
> iD8DBQE+XSR/vJuQZxSWSsgRAi2jAJ9IAKnMBmNcVEEI8TXQBBd/rtm4XQCg0Vjq
> IO9OsCSkdnNJqnrYYutM3jw=
> =9kwY
> -----END PGP SIGNATURE-----
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
> http://archives.postgresql.org
> 
[ Decrypting message... End of raw data. ]

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sat, 24 May 2003 13:37:14 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: XML ouput for psql

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> I assume we are not moving in the XML/psql direction, right?  We want it
> int he backend, or the psql HTML converted to XHTML?

I don't think a consensus was ever reached. It would certainly be better if 
this was done on the backend, but that seems to be a long time away, and 
some have argued that it is not the job of the engine to do this anyway.

I agree at the very least we should update the HTML ourput for psql: I'll 
try to make a patch for that this weekend.

I still think we should at least have a rudimentary xml output option inside 
of psql. It won't be perfect, but we can certainly have the flag toggle 
a backend variable when/if the backend supports XML directly.


- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200305301452
-----BEGIN PGP SIGNATURE-----
Comment: http://www.turnstep.com/pgp.html

iD8DBQE+16npvJuQZxSWSsgRAnTHAJ0UN3HFWVybqDd/5lnsV2CcotRxSgCgp7md
W9Iho/Y1mwUYEl8SX/9oAVc=
=G1Jx
-----END PGP SIGNATURE-----



Fri, 30 May 2003 19:06:38 -0000
greg@turnstep.com
Re: XML ouput for psql
> > I assume we are not moving in the XML/psql direction, right?  We
> > want it int he backend, or the psql HTML converted to XHTML?
> 
> I don't think a consensus was ever reached. It would certainly be
> better if this was done on the backend, but that seems to be a long
> time away, and some have argued that it is not the job of the engine
> to do this anyway.

Few points for the archives regarding XML and databases (spent 9mo
working on this kinda stuff during the .com days):

*) Use libxml2.  MIT Licensed, most complete opensource XML
   implementation available, and fast.  See the XML benchmarks on
   sf.net for details.  To avoid library naming conflicts, the library
   should likely be renamed to pgxml.so and imported into the src
   tree.  Mention java in this context and risk being clubbed to death.

*) There should be two storage formats for XML data:

   a) DOM-esque storage: broken down xmlNodes.  This is necessary for
      indexing specific places in documents (ala XPath queries).
      Actual datums on the disk should be similar in structure to the
      xmlNode struct found in libxml2 (would help with the
      serialization in either direction).  In database xslt
      transformations are also possible with the data stored this way.

   b) SAX-esque storage: basically a single BYTEA/TEXT column.  Not
      all documents need to be indexed/searchable and SAX processing
      of data is generally more efficient if you don't know what
      you're looking for.  This format is the low hanging fruit
      though.

-sc

-- 
Sean Chittenden

Fri, 30 May 2003 13:20:28 -0700
Sean Chittenden <sean@chittenden.org>
Re: XML ouput for psql

Done.  Added to 1.15) How can I financially assist PostgreSQL?

    <P>Also, if you have a success story about PostgreSQL, please submit
    it to our advocacy site at <a href="http://advocacy.postgresql.org">
    http://advocacy.postgresql.org</a>.


---------------------------------------------------------------------------

Shridhar Daithankar wrote:
> Hi,
> 
> We should probably put out an FAQ saying, if you have success story, please 
> make a write-up and send to us at http://advocacy.postgresql.org.
> 
>  Shridhar
> 
> On Tuesday 15 April 2003 18:42, Ericson Smith wrote:
> > As one of the top search engine campaign optimization companies in the
> > space, we here at Did-it.com have been using Postgresql for over a year
> > now. We had serious locking problems with MySQL and even switching to
> > their Innodb handler did not solve all the issues.
> >
> > As the DB administrator, I recommended we switch when it came time to
> > re-write our client platform. That we did, and we have not looked back.
> > We have millions of listings, keywords and we perform live visitor
> > tracking in our database. We capture on the order of about 1 million
> > visitors every day, with each hit making updates, selects and possibly
> > inserts.
> >
> > We could not have done this in mySQL. Basically when I see silly posts
> > over on Slashdot about MySQL being as good a sliced bread, you can check
> > out the debunking posts that I make as "esconsult1".
> >
> > Yes, perhaps Postgresql needs a central org that manages press and so
> > on, but we know that we dont get the press that MySQL does, but quietly
> > in the background Postgresql is handling large important things.
> >
> > - Ericson Smith
> > Web Developer
> > Db Admin
> > http://www.did-it.com
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 30 May 2003 22:26:10 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [GENERAL] Are we losing momentum?

Where do we want to go with this?  It is interesting that it maps MySQL
SHOW commands on top of our existing SHOW syntax in psql.  The patch
doesn't look too big.

Should this be applied?  Sean, I know you have a newer patch the the URL
you posted isn't good anymore.  This also contains GUC change.  Sean,
would you show each one you added and we can discuss if any are
inappropriate.

---------------------------------------------------------------------------

Sean Chittenden wrote:
> > > That's a pretty reasonable thought. I work for a shop that sells
> > > Postgres support, and even we install MySQL for the Q&D ticket
> > > tracking system we recommend because we can't justify the cost to
> > > port it to postgres. If the postgres support were there, we would
> > > surely be using it.
> > 
> > > How to fix such a situation, I'm not sure. "MySQL Compatability
> > > Mode," anyone? :-)
> > 
> > What issues are creating a compatibility problem for you?
> 
> I don't think these should be hacked into the backend/libpq, but I
> think it'd be a huge win to hack in "show *" support into psql for
> MySQL users so they can type:
> 
>      SHOW (databases|tables|views|functions|triggers|schemas);
> 
> I have yet to meet a MySQL user who understands the concept of system
> catalogs even though it's just the 'mysql' database (this irritates me
> enough as is)... gah, f- it: mysql users be damned, I have three
> developers that think that postgresql is too hard to use because they
> can't remember "\d [table name]" and I'm tired of hearing them bitch
> when I push using PostgreSQL instead of MySQL.  I have better things
> to do with my time than convert their output to PostgreSQL.  Here goes
> nothing...
> 
> I've tainted psql and added a MySQL command compatibility layer for
> the family of SHOW commands (psql [-m | --mysql]).
> 
> 
> The attached patch does a few things:
> 
> 1) Implements quite a number of SHOW commands (AGGREGATES, CASTS,
>    CATALOGS, COLUMNS, COMMENTS, CONSTRAINTS, CONVERSIONS, DATABASES,
>    DOMAINS, FUNCTIONS, HELP, INDEX, LARGEOBJECTS, NAMES, OPERATORS,
>    PRIVILEGES, PROCESSLIST, SCHEMAS, SEQUENCES, SESSION, STATUS,
>    TABLES, TRANSACTION, TYPES, USERS, VARIABLES, VIEWS)
> 
>    SHOW thing
>    SHOW thing LIKE pattern
>    SHOW thing FROM pattern
>    SHOW HELP ON (topic || ALL);
>    etc.
> 
>    Some of these don't have \ command eqiv's.  :( I was tempted to add
>    them, but opted not to for now, but it'd certainly be a nice to
>    have.
> 
> 2) Implements the necessary tab completion for the SHOW commands for
>    the tab happy newbies/folks out there.  psql is more friendly than
>    mysql's CLI now in terms of tab completion for the show commands.
> 
> 3) Few trailing whitespace characters were nuked
> 
> 4) guc.c is now in sync with the list of available variables used for
>    tab completion
> 
> 
> Few things to note:
> 
> 1) SHOW INDEXES is the same as SHOW INDEX, I think MySQL is wrong in
>    this regard and that it should be INDEXES to be plural along with
>    the rest of the types, but INDEX is preserved for compatibility.
> 
> 2) There are two bugs that I have yet to address
> 
>    1) SHOW VARIABLES doesn't work, but "SHOW [TAB][TAB]y" does
>    2) "SHOW [variable_of_choice];" doesn't work, but "SHOW
>       [variable_of_choice]\n;" does work... not sure where this
>       problem is coming from
> 
> 3) I think psql is more usable as a result of this more verbose
>    syntax, but it's not the prettiest thing on the planet (wrote a
>    small parser outside of the backend or libraries: I don't want to
>    get those dirty with MySQL's filth).
> 
> 4) In an attempt to wean people over to PostgreSQL's syntax, I
>    included translation tips on how to use the psql equiv of the SHOW
>    commands.  Going from SHOW foo to \d foo is easy, going from \d foo
>    to SHOW foo is hard and drives me nuts.  This'll help userbase
>    retention of newbies/converts.  :)
> 
> 5) The MySQL mode is just a bounce layer that provides different
>    syntax wrapping exec_command() so it should provide little in the
>    way of maintenance headaches.  Some of the SHOW commands, however,
>    don't have \ couterparts, but once they do and that code is
>    centralized, this feature should come for zero cost.
> 
> 6) As an administrator, I'd be interested in having an environment
>    variable that I could set that'd turn on MySQL mode for some of my
>    bozo users that way they don't complain if they forget the -m
>    switch.  Thoughts?
> 
> 
> I'll try and iron out the last of those two bugs/features, but at this
> point, would like to see this patch get wider testing/feedback.
> Comments, as always, are welcome.
> 
> PostgreSQL_usability++
> 
> -sc
> 
> -- 
> Sean Chittenden

[ Attachment, skipping... ]

> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 30 May 2003 23:07:00 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Are we losing momentum?
Hi everyone,

As an additional point of interest, we're still processing all of the 
Case Study submissions received from the last call around February, and 
should be beginning translations within the next week or two.

There is a lot of very good news contained in the submissions, and they 
will definitely assist in bringing into the open just how good a 
database system and Community we truly have.

Hope everyone will be as amazed as I was at just where PostgreSQL is 
being used already.  There are some very clued-in people out there.

Regards and best wishes,

Justin Clift


Bruce Momjian wrote:
> Done.  Added to 1.15) How can I financially assist PostgreSQL?
> 
>     <P>Also, if you have a success story about PostgreSQL, please submit
>     it to our advocacy site at <a href="http://advocacy.postgresql.org">
>     http://advocacy.postgresql.org</a>.
> 
> 
> ---------------------------------------------------------------------------
> 
> Shridhar Daithankar wrote:
> 
>>Hi,
>>
>>We should probably put out an FAQ saying, if you have success story, please 
>>make a write-up and send to us at http://advocacy.postgresql.org.
>>
>> Shridhar
>>
>>On Tuesday 15 April 2003 18:42, Ericson Smith wrote:
>>
>>>As one of the top search engine campaign optimization companies in the
>>>space, we here at Did-it.com have been using Postgresql for over a year
>>>now. We had serious locking problems with MySQL and even switching to
>>>their Innodb handler did not solve all the issues.
>>>
>>>As the DB administrator, I recommended we switch when it came time to
>>>re-write our client platform. That we did, and we have not looked back.
>>>We have millions of listings, keywords and we perform live visitor
>>>tracking in our database. We capture on the order of about 1 million
>>>visitors every day, with each hit making updates, selects and possibly
>>>inserts.
>>>
>>>We could not have done this in mySQL. Basically when I see silly posts
>>>over on Slashdot about MySQL being as good a sliced bread, you can check
>>>out the debunking posts that I make as "esconsult1".
>>>
>>>Yes, perhaps Postgresql needs a central org that manages press and so
>>>on, but we know that we dont get the press that MySQL does, but quietly
>>>in the background Postgresql is handling large important things.
>>>
>>>- Ericson Smith
>>>Web Developer
>>>Db Admin
>>>http://www.did-it.com
>>
>>
>>---------------------------(end of broadcast)---------------------------
>>TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>>
> 
> 



Sat, 31 May 2003 15:39:34 +0800
Justin Clift <justin@postgresql.org>
Re: [GENERAL] Are we losing momentum?
Sean Chittenden kirjutas R, 30.05.2003 kell 23:20:
> > > I assume we are not moving in the XML/psql direction, right?  We
> > > want it int he backend, or the psql HTML converted to XHTML?
> > 
> > I don't think a consensus was ever reached. It would certainly be
> > better if this was done on the backend, but that seems to be a long
> > time away, and some have argued that it is not the job of the engine
> > to do this anyway.
> 
> Few points for the archives regarding XML and databases (spent 9mo
> working on this kinda stuff during the .com days):
> 
> *) Use libxml2.  MIT Licensed, most complete opensource XML
>    implementation available, and fast.  See the XML benchmarks on
>    sf.net for details.  To avoid library naming conflicts, the library
>    should likely be renamed to pgxml.so and imported into the src
>    tree.  Mention java in this context and risk being clubbed to death.

Agree completely on all points ;)

> *) There should be two storage formats for XML data:
> 
>    a) DOM-esque storage: broken down xmlNodes.  This is necessary for
>       indexing specific places in documents (ala XPath queries).
>       Actual datums on the disk should be similar in structure to the
>       xmlNode struct found in libxml2 (would help with the
>       serialization in either direction).  In database xslt
>       transformations are also possible with the data stored this way.
> 
>    b) SAX-esque storage: basically a single BYTEA/TEXT column.  Not
>       all documents need to be indexed/searchable and SAX processing
>       of data is generally more efficient if you don't know what
>       you're looking for.  This format is the low hanging fruit
>       though.

I think that Oleg and Todor very recently proposed somethink that could
use b) and still provide indexed access. 

Most flexible would be some way to define, how much of a tree is kept
together, as xmlNode/tuple would probably be too much overhead for most
operations, whereas xmlFile/tuple would also, just for other ops;)

--------------
Hannu



31 May 2003 13:35:43 +0300
Hannu Krosing <hannu@tm.ee>
Re: XML ouput for psql
Bruce Momjian writes:

> Where do we want to go with this?  It is interesting that it maps MySQL
> SHOW commands on top of our existing SHOW syntax in psql.  The patch
> doesn't look too big.

The response to "Are we losing momentum?" isn't to add redundant syntax
for nonstandard features that we have no control over.

-- 
Peter Eisentraut   peter_e@gmx.net


Sat, 31 May 2003 17:55:16 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Are we losing momentum?
Peter Eisentraut <peter_e@gmx.net> writes:
> Bruce Momjian writes:
>> Where do we want to go with this?  It is interesting that it maps MySQL
>> SHOW commands on top of our existing SHOW syntax in psql.  The patch
>> doesn't look too big.

> The response to "Are we losing momentum?" isn't to add redundant syntax
> for nonstandard features that we have no control over.

I'm of two minds about it myself.  I don't like trying to play follow-the-
leader with a moving target.  But if you think of it as trying to win
over converts from MySQL, it seems a lot more palatable.

It would also be interesting to combine this with Rod's idea of driving
describe-type queries by table instead of hardwired code.  Imagine that
the backend's "show foo" command first looks for "foo" as a GUC
variable, as it does now, but upon failing to find one it looks in a
system table for a query associated with the name "foo".  If it finds
such a query, it runs it and sends back the result.  Now, not only can
we emulate "show tables", but people can easily add application-specific
"show whatever" commands, which seems tremendously cool.

There are some safety and protection issues to be solved here (probably
only superusers should be allowed to modify the query table, and we
should restrict the form of the query to be a single SELECT command)
but I can't think of any showstoppers.  Now that we've abandoned backend
autocommit there are no technical reasons that SHOW shouldn't be allowed
to run a SELECT query.

I did not care for the original patch (which IIRC made psql recognize
"show" commands, rather than doing it in the backend) but something like
the above seems reasonably clean.

			regards, tom lane

Sat, 31 May 2003 12:49:00 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Are we losing momentum? 
> >> Where do we want to go with this?  It is interesting that it maps
> >> MySQL SHOW commands on top of our existing SHOW syntax in psql.
> >> The patch doesn't look too big.
> >
> > The response to "Are we losing momentum?" isn't to add redundant
> > syntax for nonstandard features that we have no control over.
> 
> I'm of two minds about it myself.  I don't like trying to play
> follow-the- leader with a moving target.  But if you think of it as
> trying to win over converts from MySQL, it seems a lot more
> palatable.

The _only_ reason I added this was to aid in the conversion of MySQL
users who don't know how to navigate their way through psql and
PostgreSQL.  If you read through the patch, the extended SHOW commands
are only enabled if you turn them on via the --mysql option or by
setting an option (don't recall at the moment) that way you can turn
these on by default in a person's .psqlrc file.  With Win32 coming
down the pipe (right???  *cough cough* :-P), converting existing users
is going to be important for PostgreSQL's long term success if it
seeks to gain market share and momentum.

> It would also be interesting to combine this with Rod's idea of
> driving describe-type queries by table instead of hardwired code.
> Imagine that the backend's "show foo" command first looks for "foo"
> as a GUC variable, as it does now, but upon failing to find one it
> looks in a system table for a query associated with the name "foo".
> If it finds such a query, it runs it and sends back the result.
> Now, not only can we emulate "show tables", but people can easily
> add application-specific "show whatever" commands, which seems
> tremendously cool.

I really like the ability to program in queries or syntaxes into the
backend, but as it stands, SHOW foo would have to be pretty smart to
handle the LIKE clauses and other bits.  And how would tab completion
be handled in psql?  I can't remember if I implemented stuff like,
SHOW TABLES IN SCHEMA foo, but 'SHOW TABLES LIKE a%' was quite a hit
to the people I have using the patch and I don't see how that'd be
applicable with what has been proposed so far.  The extra verbosity
and tab completion-ability is important for the newbie, it's an
interactive environment that lets them learn and explore in a
proactive manner.  Having IN SCHEMA vs LIKE handled on the backend
would be a rather complex generic model, but in an ideal world, one
that I'd prefer if done right.

> There are some safety and protection issues to be solved here
> (probably only superusers should be allowed to modify the query
> table, and we should restrict the form of the query to be a single
> SELECT command) but I can't think of any showstoppers.  Now that
> we've abandoned backend autocommit there are no technical reasons
> that SHOW shouldn't be allowed to run a SELECT query.
> 
> I did not care for the original patch (which IIRC made psql
> recognize "show" commands, rather than doing it in the backend) but
> something like the above seems reasonably clean.

Well, unless something is done very cleanly and abstractly, and can be
extended to handle IN SCHEMA, LIKE, etc., I'm of the opinion that this
kinda stuff belongs on the client side of things and _not_ in the
backend.  Putting MySQL's brokenness in the backend seems, well, to
perpetuate the brokenness and acknowledge that their brokenness isn't
really that broken: not a message I'm fond of.  Putting MySQL's
brokenness in psql and hiding it behind the --mysql CLI option,
however, is tolerably broken, IMHO and of the same use as ora2pg or
mysql2pg.

Bruce wrote:
> Where do we want to go with this?  It is interesting that it maps
> MySQL SHOW commands on top of our existing SHOW syntax in psql.  The
> patch doesn't look too big.
>
> Should this be applied?  Sean, I know you have a newer patch the the
> URL you posted isn't good anymore.  This also contains GUC change.
> Sean, would you show each one you added and we can discuss if any
> are inappropriate.

The GUC change was just flushing out various GUC options that weren't
available as tab completion options... the usefulness of those GUCs
could be debated, but at least there's a complete list in the CLI
now... having this pulled from the server would be wise, IMHO, but I
didn't spend the time to add that at the time.  If there's interest, I
can do that.

I have the patch someplace, don't worry about that.  The patch isn't
100% correct as it stands: there is a problem in recognizing the end
of a SHOW command so that the query buffer was considered non-empty
(wasn't scanning properly for ;'s to emulate the end of a query:
something that's not necessary with the normal \ syntax).  Other than
that, there are no bugs that I'm aware of and the work around is to
issue a \r: not an earth shattering problem given psql has a history.
Before I finished, I wanted to get some review/thoughts on the patch
before I completed the work and sent it off only to have it rejected.

I know Tom doesn't like the idea because it's not "clean," but I can't
help but feel that adding similar functionality to the backend is some
how contaminating PostgreSQL and acknowledges MySQL's done something
creditable, which is too big of an admission, especially since their
product is pretty fundamentally flawed in terms of spec compliance.
The last thing we want is to have their brokenness turn into spec and
for that to become the norm.  So, as life would have it, I'm an
advocate of adding the SHOW syntax to psql and an opponent of having it
in the backend, the core developers would rather have the SHOW syntax
in the backend because it's not clean to have it in the front end.
*shrug*

psql has to have a quasi-parser in the front end as is to handle tab
completion, so I don't really buy the argument that it's bad to have a
parser in the front end (sorry Tom).  In fact, I actually think that
the parser in psql needs to be _beefed up_ and made _smarter_ that way
psql's more user friendly, even if that means increasing the
complexity of psql.  At the moment, there are short falls with psql's
ability to perform tab completion of column names on aliased tables,
etc.  Some of it's non-trivial and won't ever see the light of day in
psql, but there is room for improvement.  Anyway, usability is a
problem for clients, not for servers.  MySQL usability belongs in the
client, not the backend.... and hidden behind a CLI flag at that
'cause I don't available in my personal development environment.

And before someone says, "that's fine and dandy, who's going to do the
work on psql," I'll gladly work on psql and even go so far as to clean
up the code now that someone's around to apply the patches.  :)

*listens and waits before updating the patch and fixing the above
 bugs* -sc

-- 
Sean Chittenden

Sat, 31 May 2003 18:43:23 -0700
Sean Chittenden <sean@chittenden.org>
Re: [HACKERS] Are we losing momentum?
> It would also be interesting to combine this with Rod's idea of driving
> describe-type queries by table instead of hardwired code.  Imagine that
> the backend's "show foo" command first looks for "foo" as a GUC
> variable, as it does now, but upon failing to find one it looks in a
> system table for a query associated with the name "foo".  If it finds
> such a query, it runs it and sends back the result.  Now, not only can
> we emulate "show tables", but people can easily add application-specific
> "show whatever" commands, which seems tremendously cool.

Easy enough to accomplish for the most part.  I suppose the most
difficult part is whether we support arbitrary syntax?

SHOW INDEXES ON TABLE <bleah>;
SHOW TABLES;
SHOW DATABASES;
SHOW COLUMNS FROM <table>; <-- Long form of DESCRIBE <table>

I believe all of the above is valid MySQL syntax, so we would need to be
able to work with a list of colId, rather than a single one.

This does not help with tab completion or help for the above items. 
Though one could certainly argue help on available commands should come
from the backend, and psql could be taught how to read the table to
determine how to deal with tab completion.

One significant downside is that describe commands would require an
initdb when updated.

Oh, if it's the backend doing the work, the queries should probably be
functions, not free-form queries.  Simply makes it easier to inject
variables into the right place. In the initial patch, psql is using a
prepared statement for this work with the side benefit that the plan is
cached.  An SQL function would accomplish the same thing.

-- 
Rod Taylor <rbt@rbt.ca>

PGP Key: http://www.rbt.ca/rbtpub.asc
31 May 2003 23:08:49 -0400
Rod Taylor <rbt@rbt.ca>
Re: [HACKERS] Are we losing momentum?
[ moving this thread to a more appropriate place ]

Sean Chittenden <sean@chittenden.org> writes:
>> It would also be interesting to combine this with Rod's idea of
>> driving describe-type queries by table instead of hardwired code.
>> Imagine that the backend's "show foo" command first looks for "foo"
>> as a GUC variable, as it does now, but upon failing to find one it
>> looks in a system table for a query associated with the name "foo".
>> If it finds such a query, it runs it and sends back the result.
>> Now, not only can we emulate "show tables", but people can easily
>> add application-specific "show whatever" commands, which seems
>> tremendously cool.

> I really like the ability to program in queries or syntaxes into the
> backend, but as it stands, SHOW foo would have to be pretty smart to
> handle the LIKE clauses and other bits.

It's certainly doable.  I thought more about how to handle parameters
and such, and came up with this sketch:

1. We generalize the SHOW syntax to accept 1 or more identifiers (might
as well allow strings too).  The existing special cases like SHOW TIME
ZONE would be taken out of the grammar and checked for at runtime.

2. The "key" field of the show_queries table is an array of one or more
strings that can be either keywords or parameter placeholders ($n).
There must be at least one keyword.  Then SHOW matches a particular
table entry if there are the right number of words and all the keyword
strings match the corresponding words.  The other words become the
parameter values.

3. The "query" field of the table is a SELECT possibly containing
parameter references $n.  This can be handled the same way as a
preparable statement (we already have mechanisms for resolving the types
of the parameters).

While I haven't studied the MySQL manual to see what-all they allow,
this certainly seems sufficient to support "SHOW TABLE foo" and similar
variants.  And the possibility of user-added extensions to the table
seems really cool.

> And how would tab completion be handled in psql?

You look at the table to see what can come after SHOW.  We already have
database-driven completion, so this doesn't seem out of reach.

One thing that is doable with psql's current hard-wired approach, but
doesn't seem easy to do with this solution, is automatic localization
of strings such as column headings.  Rod had looked at that a little
in his trial patch to convert psql's \d stuff to table-driven form,
but AFAIR he didn't have a satisfactory answer.

			regards, tom lane

Sun, 01 Jun 2003 12:03:08 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Table-driven SHOW (was Re: Are we losing momentum?)
Tom Lane writes:

> Now, not only can we emulate "show tables", but people can easily add
> application-specific "show whatever" commands, which seems tremendously
> cool.

We already have that.  They're called views.

-- 
Peter Eisentraut   peter_e@gmx.net


Sun, 1 Jun 2003 23:52:45 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Are we losing momentum? 
> > Now, not only can we emulate "show tables", but people can easily
> > add application-specific "show whatever" commands, which seems
> > tremendously cool.
> 
> We already have that.  They're called views.

Um, I'm interested in aiding in the conversion of users from MySQL to
PostgreSQL (how ever it happens, I don't really care).  Tom and Rod
are interested in an extensible/programmable SHOW syntax (very cool
and much more interesting than the hack I put together).  Views solve
none of the above problems.  -sc

-- 
Sean Chittenden

Sun, 1 Jun 2003 14:57:27 -0700
Sean Chittenden <sean@chittenden.org>
Re: [HACKERS] Are we losing momentum?
> One thing that is doable with psql's current hard-wired approach, but
> doesn't seem easy to do with this solution, is automatic localization
> of strings such as column headings.  Rod had looked at that a little
> in his trial patch to convert psql's \d stuff to table-driven form,
> but AFAIR he didn't have a satisfactory answer.

I've yet to come up with anything better.  Best answer I've come up with
is to make the column headings a separate query result set and let the
back-end do the translations.

-- 
Rod Taylor <rbt@rbt.ca>

PGP Key: http://www.rbt.ca/rbtpub.asc
01 Jun 2003 20:02:45 -0400
Rod Taylor <rbt@rbt.ca>
Re: Table-driven SHOW (was Re: Are we losing momentum?)
Sean Chittenden writes:

> Um, I'm interested in aiding in the conversion of users from MySQL to
> PostgreSQL (how ever it happens, I don't really care).

Your approach to that reminds me of those

alias dir='ls'
alias md='mkdir'

things that Linux distributors once stuck (or still stick?) in the default
profile files, presumably to help conversion from DOS.  It's pretty
pointless, because Linux is still very different from DOS, and once you
want to do something besides showing or changing directories, you will
need documentation and training.  That is what the MySQL conversion
process needs as well, otherwise you're not converting, you're emulating,
and that is not a game you can win.

-- 
Peter Eisentraut   peter_e@gmx.net


Mon, 2 Jun 2003 21:07:11 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Are we losing momentum?
> > Um, I'm interested in aiding in the conversion of users from MySQL
> > to PostgreSQL (how ever it happens, I don't really care).
> 
> Your approach to that reminds me of those
> 
> alias dir='ls'
> alias md='mkdir'
> 
> things that Linux distributors once stuck (or still stick?) in the
> default profile files, presumably to help conversion from DOS.  It's
> pretty pointless, because Linux is still very different from DOS,
> and once you want to do something besides showing or changing
> directories, you will need documentation and training.

Well, interestingly enough, those commands work for getting people in
the door and to the point that they're able to learn more.  The first
step to any kind of adult education or reeducation is to have concepts
that the people are familiar with (in this case MySQL) be translated
into the concepts of the area that they're trying to learn.  If you'd
have read the original patch that I'd posted, you'd see that I'd done
that by adding a TIP section to the top of the response.

*SNIP*
SHOW COLUMNS FROM [tblname];

     TIP: In psql, "SHOW COLUMNS FROM [tblname]" is natively written as \dt [tblname]

[normal output from \dt tblname]
*END SNIP*

The point of my patch was to aid the conversion, not to gimp along
b0rk3d habits from MySQL.

> That is what the MySQL conversion process needs as well, otherwise
> you're not converting, you're emulating, and that is not a game you
> can win.

Emulation within reason.  dir->ls and md->mkdir worked for a handful
of people that I've transitioned into the UNIX world from Win32 land,
in fact, I have one friend from school who's been so successful that
he's converted from using Win32 on his desktop to using Linux, worked
with me on a job where we were hacking mod_perl on a site pushing in
excess of 80Mbps to 25M people a day, but still types dir to this day.
Not bad for an aero student who graduated with a 4.0 in his major,
exceedingly bright, adaptive, learns fast, etc.  My point is that
regardless of how bright the person or what the right invocation, aids
like these help get people in the door and if they like what they see
once they're through the door, they'll stay.  Once people try and use
PostgreSQL, they stay.  When people try MySQL, they're left wanting or
needing more and are bound by the limits of the software... that's not
really the case with PostgreSQL.

To get people to try, you play the association or emulation game and
it works.  Ask anyone in adult education and they will say the same.
The Internet was an "information super highway" because that was
something that people could grasp, regardless of how flawed it really
is.  Bandwidth is thought of as pipes in various sizes diameters, a
much better analogy.  To geeks, broadcast is explained as the same as
radio and unicast as satellite TV.  To communication majors, TCP is
descried as a letter that's been chopped up into a thousand numbered
pieces and sent to the other side of the US via the postal mail.
Leche is milk to English speakers learning Spanish. One way or
another, you have to play the game of working within the understanding
of the target audience, in this case, MySQL users who use a SHOW
TABLES type syntax.

-sc

-- 
Sean Chittenden

Mon, 2 Jun 2003 12:37:55 -0700
Sean Chittenden <sean@chittenden.org>
Re: [HACKERS] Are we losing momentum?

Wow, this is a good argument!  I must admit I made 'ps -ef' work on BSD
because I was just so used to it on mainframe Unix.  There is that
'fingers type without thinking' thing, and I think that is what he is
talking about.

I wonder if we should just support SHOW TABLES or the most common ones.

Maybe emulation is the wrong approach --- maybe we just need 'finger
thinking' shortcuts.

---------------------------------------------------------------------------

Sean Chittenden wrote:
> > > Um, I'm interested in aiding in the conversion of users from MySQL
> > > to PostgreSQL (how ever it happens, I don't really care).
> > 
> > Your approach to that reminds me of those
> > 
> > alias dir='ls'
> > alias md='mkdir'
> > 
> > things that Linux distributors once stuck (or still stick?) in the
> > default profile files, presumably to help conversion from DOS.  It's
> > pretty pointless, because Linux is still very different from DOS,
> > and once you want to do something besides showing or changing
> > directories, you will need documentation and training.
> 
> Well, interestingly enough, those commands work for getting people in
> the door and to the point that they're able to learn more.  The first
> step to any kind of adult education or reeducation is to have concepts
> that the people are familiar with (in this case MySQL) be translated
> into the concepts of the area that they're trying to learn.  If you'd
> have read the original patch that I'd posted, you'd see that I'd done
> that by adding a TIP section to the top of the response.
> 
> *SNIP*
> SHOW COLUMNS FROM [tblname];
> 
>      TIP: In psql, "SHOW COLUMNS FROM [tblname]" is natively written as \dt [tblname]
> 
> [normal output from \dt tblname]
> *END SNIP*
> 
> The point of my patch was to aid the conversion, not to gimp along
> b0rk3d habits from MySQL.
> 
> > That is what the MySQL conversion process needs as well, otherwise
> > you're not converting, you're emulating, and that is not a game you
> > can win.
> 
> Emulation within reason.  dir->ls and md->mkdir worked for a handful
> of people that I've transitioned into the UNIX world from Win32 land,
> in fact, I have one friend from school who's been so successful that
> he's converted from using Win32 on his desktop to using Linux, worked
> with me on a job where we were hacking mod_perl on a site pushing in
> excess of 80Mbps to 25M people a day, but still types dir to this day.
> Not bad for an aero student who graduated with a 4.0 in his major,
> exceedingly bright, adaptive, learns fast, etc.  My point is that
> regardless of how bright the person or what the right invocation, aids
> like these help get people in the door and if they like what they see
> once they're through the door, they'll stay.  Once people try and use
> PostgreSQL, they stay.  When people try MySQL, they're left wanting or
> needing more and are bound by the limits of the software... that's not
> really the case with PostgreSQL.
> 
> To get people to try, you play the association or emulation game and
> it works.  Ask anyone in adult education and they will say the same.
> The Internet was an "information super highway" because that was
> something that people could grasp, regardless of how flawed it really
> is.  Bandwidth is thought of as pipes in various sizes diameters, a
> much better analogy.  To geeks, broadcast is explained as the same as
> radio and unicast as satellite TV.  To communication majors, TCP is
> descried as a letter that's been chopped up into a thousand numbered
> pieces and sent to the other side of the US via the postal mail.
> Leche is milk to English speakers learning Spanish. One way or
> another, you have to play the game of working within the understanding
> of the target audience, in this case, MySQL users who use a SHOW
> TABLES type syntax.
> 
> -sc
> 
> -- 
> Sean Chittenden
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 2 Jun 2003 15:53:10 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Are we losing momentum?

I assume we agreed against adding a MySQL mode --- just verifying.

---------------------------------------------------------------------------

Sean Chittenden wrote:
> > > That's a pretty reasonable thought. I work for a shop that sells
> > > Postgres support, and even we install MySQL for the Q&D ticket
> > > tracking system we recommend because we can't justify the cost to
> > > port it to postgres. If the postgres support were there, we would
> > > surely be using it.
> > 
> > > How to fix such a situation, I'm not sure. "MySQL Compatability
> > > Mode," anyone? :-)
> > 
> > What issues are creating a compatibility problem for you?
> 
> I don't think these should be hacked into the backend/libpq, but I
> think it'd be a huge win to hack in "show *" support into psql for
> MySQL users so they can type:
> 
>      SHOW (databases|tables|views|functions|triggers|schemas);
> 
> I have yet to meet a MySQL user who understands the concept of system
> catalogs even though it's just the 'mysql' database (this irritates me
> enough as is)... gah, f- it: mysql users be damned, I have three
> developers that think that postgresql is too hard to use because they
> can't remember "\d [table name]" and I'm tired of hearing them bitch
> when I push using PostgreSQL instead of MySQL.  I have better things
> to do with my time than convert their output to PostgreSQL.  Here goes
> nothing...
> 
> I've tainted psql and added a MySQL command compatibility layer for
> the family of SHOW commands (psql [-m | --mysql]).
> 
> 
> The attached patch does a few things:
> 
> 1) Implements quite a number of SHOW commands (AGGREGATES, CASTS,
>    CATALOGS, COLUMNS, COMMENTS, CONSTRAINTS, CONVERSIONS, DATABASES,
>    DOMAINS, FUNCTIONS, HELP, INDEX, LARGEOBJECTS, NAMES, OPERATORS,
>    PRIVILEGES, PROCESSLIST, SCHEMAS, SEQUENCES, SESSION, STATUS,
>    TABLES, TRANSACTION, TYPES, USERS, VARIABLES, VIEWS)
> 
>    SHOW thing
>    SHOW thing LIKE pattern
>    SHOW thing FROM pattern
>    SHOW HELP ON (topic || ALL);
>    etc.
> 
>    Some of these don't have \ command eqiv's.  :( I was tempted to add
>    them, but opted not to for now, but it'd certainly be a nice to
>    have.
> 
> 2) Implements the necessary tab completion for the SHOW commands for
>    the tab happy newbies/folks out there.  psql is more friendly than
>    mysql's CLI now in terms of tab completion for the show commands.
> 
> 3) Few trailing whitespace characters were nuked
> 
> 4) guc.c is now in sync with the list of available variables used for
>    tab completion
> 
> 
> Few things to note:
> 
> 1) SHOW INDEXES is the same as SHOW INDEX, I think MySQL is wrong in
>    this regard and that it should be INDEXES to be plural along with
>    the rest of the types, but INDEX is preserved for compatibility.
> 
> 2) There are two bugs that I have yet to address
> 
>    1) SHOW VARIABLES doesn't work, but "SHOW [TAB][TAB]y" does
>    2) "SHOW [variable_of_choice];" doesn't work, but "SHOW
>       [variable_of_choice]\n;" does work... not sure where this
>       problem is coming from
> 
> 3) I think psql is more usable as a result of this more verbose
>    syntax, but it's not the prettiest thing on the planet (wrote a
>    small parser outside of the backend or libraries: I don't want to
>    get those dirty with MySQL's filth).
> 
> 4) In an attempt to wean people over to PostgreSQL's syntax, I
>    included translation tips on how to use the psql equiv of the SHOW
>    commands.  Going from SHOW foo to \d foo is easy, going from \d foo
>    to SHOW foo is hard and drives me nuts.  This'll help userbase
>    retention of newbies/converts.  :)
> 
> 5) The MySQL mode is just a bounce layer that provides different
>    syntax wrapping exec_command() so it should provide little in the
>    way of maintenance headaches.  Some of the SHOW commands, however,
>    don't have \ couterparts, but once they do and that code is
>    centralized, this feature should come for zero cost.
> 
> 6) As an administrator, I'd be interested in having an environment
>    variable that I could set that'd turn on MySQL mode for some of my
>    bozo users that way they don't complain if they forget the -m
>    switch.  Thoughts?
> 
> 
> I'll try and iron out the last of those two bugs/features, but at this
> point, would like to see this patch get wider testing/feedback.
> Comments, as always, are welcome.
> 
> PostgreSQL_usability++
> 
> -sc
> 
> -- 
> Sean Chittenden

[ Attachment, skipping... ]

> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sat, 16 Aug 2003 19:22:47 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Are we losing momentum?
Bruce Momjian wrote:

> I assume we agreed against adding a MySQL mode --- just verifying.

We agreed that applications that need schema information are much better 
off using the schema views.


Jan

> 
> ---------------------------------------------------------------------------
> 
> Sean Chittenden wrote:
>> > > That's a pretty reasonable thought. I work for a shop that sells
>> > > Postgres support, and even we install MySQL for the Q&D ticket
>> > > tracking system we recommend because we can't justify the cost to
>> > > port it to postgres. If the postgres support were there, we would
>> > > surely be using it.
>> > 
>> > > How to fix such a situation, I'm not sure. "MySQL Compatability
>> > > Mode," anyone? :-)
>> > 
>> > What issues are creating a compatibility problem for you?
>> 
>> I don't think these should be hacked into the backend/libpq, but I
>> think it'd be a huge win to hack in "show *" support into psql for
>> MySQL users so they can type:
>> 
>>      SHOW (databases|tables|views|functions|triggers|schemas);
>> 
>> I have yet to meet a MySQL user who understands the concept of system
>> catalogs even though it's just the 'mysql' database (this irritates me
>> enough as is)... gah, f- it: mysql users be damned, I have three
>> developers that think that postgresql is too hard to use because they
>> can't remember "\d [table name]" and I'm tired of hearing them bitch
>> when I push using PostgreSQL instead of MySQL.  I have better things
>> to do with my time than convert their output to PostgreSQL.  Here goes
>> nothing...
>> 
>> I've tainted psql and added a MySQL command compatibility layer for
>> the family of SHOW commands (psql [-m | --mysql]).
>> 
>> 
>> The attached patch does a few things:
>> 
>> 1) Implements quite a number of SHOW commands (AGGREGATES, CASTS,
>>    CATALOGS, COLUMNS, COMMENTS, CONSTRAINTS, CONVERSIONS, DATABASES,
>>    DOMAINS, FUNCTIONS, HELP, INDEX, LARGEOBJECTS, NAMES, OPERATORS,
>>    PRIVILEGES, PROCESSLIST, SCHEMAS, SEQUENCES, SESSION, STATUS,
>>    TABLES, TRANSACTION, TYPES, USERS, VARIABLES, VIEWS)
>> 
>>    SHOW thing
>>    SHOW thing LIKE pattern
>>    SHOW thing FROM pattern
>>    SHOW HELP ON (topic || ALL);
>>    etc.
>> 
>>    Some of these don't have \ command eqiv's.  :( I was tempted to add
>>    them, but opted not to for now, but it'd certainly be a nice to
>>    have.
>> 
>> 2) Implements the necessary tab completion for the SHOW commands for
>>    the tab happy newbies/folks out there.  psql is more friendly than
>>    mysql's CLI now in terms of tab completion for the show commands.
>> 
>> 3) Few trailing whitespace characters were nuked
>> 
>> 4) guc.c is now in sync with the list of available variables used for
>>    tab completion
>> 
>> 
>> Few things to note:
>> 
>> 1) SHOW INDEXES is the same as SHOW INDEX, I think MySQL is wrong in
>>    this regard and that it should be INDEXES to be plural along with
>>    the rest of the types, but INDEX is preserved for compatibility.
>> 
>> 2) There are two bugs that I have yet to address
>> 
>>    1) SHOW VARIABLES doesn't work, but "SHOW [TAB][TAB]y" does
>>    2) "SHOW [variable_of_choice];" doesn't work, but "SHOW
>>       [variable_of_choice]\n;" does work... not sure where this
>>       problem is coming from
>> 
>> 3) I think psql is more usable as a result of this more verbose
>>    syntax, but it's not the prettiest thing on the planet (wrote a
>>    small parser outside of the backend or libraries: I don't want to
>>    get those dirty with MySQL's filth).
>> 
>> 4) In an attempt to wean people over to PostgreSQL's syntax, I
>>    included translation tips on how to use the psql equiv of the SHOW
>>    commands.  Going from SHOW foo to \d foo is easy, going from \d foo
>>    to SHOW foo is hard and drives me nuts.  This'll help userbase
>>    retention of newbies/converts.  :)
>> 
>> 5) The MySQL mode is just a bounce layer that provides different
>>    syntax wrapping exec_command() so it should provide little in the
>>    way of maintenance headaches.  Some of the SHOW commands, however,
>>    don't have \ couterparts, but once they do and that code is
>>    centralized, this feature should come for zero cost.
>> 
>> 6) As an administrator, I'd be interested in having an environment
>>    variable that I could set that'd turn on MySQL mode for some of my
>>    bozo users that way they don't complain if they forget the -m
>>    switch.  Thoughts?
>> 
>> 
>> I'll try and iron out the last of those two bugs/features, but at this
>> point, would like to see this patch get wider testing/feedback.
>> Comments, as always, are welcome.
>> 
>> PostgreSQL_usability++
>> 
>> -sc
>> 
>> -- 
>> Sean Chittenden
> 
> [ Attachment, skipping... ]
> 
>> 
>> ---------------------------(end of broadcast)---------------------------
>> TIP 3: if posting/reading through Usenet, please send an appropriate
>> subscribe-nomail command to majordomo@postgresql.org so that your
>> message can get through to the mailing list cleanly
> 


-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Sat, 16 Aug 2003 22:41:39 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: [HACKERS] Are we losing momentum?

Personally, I think adding the SHOW commands would be a good thing ...
psql is nice with its \df to get information without having to learn all
the JOINs required ... having that ability easily from any of the
interfaces would definitely be a plus ... to me, its not about MySQL
compatibility, but about a small improvement to ease of use :)

On Sat, 16 Aug 2003, Bruce Momjian wrote:

>
> I assume we agreed against adding a MySQL mode --- just verifying.
>
> ---------------------------------------------------------------------------
>
> Sean Chittenden wrote:
> > > > That's a pretty reasonable thought. I work for a shop that sells
> > > > Postgres support, and even we install MySQL for the Q&D ticket
> > > > tracking system we recommend because we can't justify the cost to
> > > > port it to postgres. If the postgres support were there, we would
> > > > surely be using it.
> > >
> > > > How to fix such a situation, I'm not sure. "MySQL Compatability
> > > > Mode," anyone? :-)
> > >
> > > What issues are creating a compatibility problem for you?
> >
> > I don't think these should be hacked into the backend/libpq, but I
> > think it'd be a huge win to hack in "show *" support into psql for
> > MySQL users so they can type:
> >
> >      SHOW (databases|tables|views|functions|triggers|schemas);
> >
> > I have yet to meet a MySQL user who understands the concept of system
> > catalogs even though it's just the 'mysql' database (this irritates me
> > enough as is)... gah, f- it: mysql users be damned, I have three
> > developers that think that postgresql is too hard to use because they
> > can't remember "\d [table name]" and I'm tired of hearing them bitch
> > when I push using PostgreSQL instead of MySQL.  I have better things
> > to do with my time than convert their output to PostgreSQL.  Here goes
> > nothing...
> >
> > I've tainted psql and added a MySQL command compatibility layer for
> > the family of SHOW commands (psql [-m | --mysql]).
> >
> >
> > The attached patch does a few things:
> >
> > 1) Implements quite a number of SHOW commands (AGGREGATES, CASTS,
> >    CATALOGS, COLUMNS, COMMENTS, CONSTRAINTS, CONVERSIONS, DATABASES,
> >    DOMAINS, FUNCTIONS, HELP, INDEX, LARGEOBJECTS, NAMES, OPERATORS,
> >    PRIVILEGES, PROCESSLIST, SCHEMAS, SEQUENCES, SESSION, STATUS,
> >    TABLES, TRANSACTION, TYPES, USERS, VARIABLES, VIEWS)
> >
> >    SHOW thing
> >    SHOW thing LIKE pattern
> >    SHOW thing FROM pattern
> >    SHOW HELP ON (topic || ALL);
> >    etc.
> >
> >    Some of these don't have \ command eqiv's.  :( I was tempted to add
> >    them, but opted not to for now, but it'd certainly be a nice to
> >    have.
> >
> > 2) Implements the necessary tab completion for the SHOW commands for
> >    the tab happy newbies/folks out there.  psql is more friendly than
> >    mysql's CLI now in terms of tab completion for the show commands.
> >
> > 3) Few trailing whitespace characters were nuked
> >
> > 4) guc.c is now in sync with the list of available variables used for
> >    tab completion
> >
> >
> > Few things to note:
> >
> > 1) SHOW INDEXES is the same as SHOW INDEX, I think MySQL is wrong in
> >    this regard and that it should be INDEXES to be plural along with
> >    the rest of the types, but INDEX is preserved for compatibility.
> >
> > 2) There are two bugs that I have yet to address
> >
> >    1) SHOW VARIABLES doesn't work, but "SHOW [TAB][TAB]y" does
> >    2) "SHOW [variable_of_choice];" doesn't work, but "SHOW
> >       [variable_of_choice]\n;" does work... not sure where this
> >       problem is coming from
> >
> > 3) I think psql is more usable as a result of this more verbose
> >    syntax, but it's not the prettiest thing on the planet (wrote a
> >    small parser outside of the backend or libraries: I don't want to
> >    get those dirty with MySQL's filth).
> >
> > 4) In an attempt to wean people over to PostgreSQL's syntax, I
> >    included translation tips on how to use the psql equiv of the SHOW
> >    commands.  Going from SHOW foo to \d foo is easy, going from \d foo
> >    to SHOW foo is hard and drives me nuts.  This'll help userbase
> >    retention of newbies/converts.  :)
> >
> > 5) The MySQL mode is just a bounce layer that provides different
> >    syntax wrapping exec_command() so it should provide little in the
> >    way of maintenance headaches.  Some of the SHOW commands, however,
> >    don't have \ couterparts, but once they do and that code is
> >    centralized, this feature should come for zero cost.
> >
> > 6) As an administrator, I'd be interested in having an environment
> >    variable that I could set that'd turn on MySQL mode for some of my
> >    bozo users that way they don't complain if they forget the -m
> >    switch.  Thoughts?
> >
> >
> > I'll try and iron out the last of those two bugs/features, but at this
> > point, would like to see this patch get wider testing/feedback.
> > Comments, as always, are welcome.
> >
> > PostgreSQL_usability++
> >
> > -sc
> >
> > --
> > Sean Chittenden
>
> [ Attachment, skipping... ]
>
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 3: if posting/reading through Usenet, please send an appropriate
> > subscribe-nomail command to majordomo@postgresql.org so that your
> > message can get through to the mailing list cleanly
>
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org

Sat, 16 Aug 2003 23:48:59 -0300 (ADT)
The Hermit Hacker <scrappy@postgresql.org>
Re: [HACKERS] Are we losing momentum?
> Bruce Momjian wrote:
> >I assume we agreed against adding a MySQL mode --- just verifying.
> 
> We agreed that applications that need schema information are much better 
> off using the schema views.
> 
> Jan

Heh, I don't think there was any agreement on anything in that thread,
everyone had their own view (no pun intended).

> From: The Hermit Hacker <scrappy@postgresql.org>
>
> Personally, I think adding the SHOW commands would be a good thing
> ...  psql is nice with its \df to get information without having to
> learn all the JOINs required ... having that ability easily from any
> of the interfaces would definitely be a plus ... to me, its not
> about MySQL compatibility, but about a small improvement to ease of
> use :)

Which goes back to the point about there being little agreement on
this patch or its issues.  A handful of folks think it's a _user
interface_ issue (read: psql, phppgadmin, pgadminIII, etc) and would
be good for converting MySQL users to PostgreSQL (or simply because
its easy and less obtuse than a \ command), others thought it was a
fugly hack to have a parser in the front end and that it should be
handled on the backend by extending SQL to conform to MySQL's
interface (that some argue is incorrect and would unjustly bloat the
backend) that way all clients have the SHOW syntax (thus averting a
possible FAQ), and others took a more elitist mindset and simply
thought that everyone should just select from the information schemas.

*shrug* I tabled working on the patch until there was some kind of
agreement from someone with commit privs and am waiting to pick up
quashing the remaining parser state bug until after 7.4's out the door
or there's renewed interest from non-users.

-sc

-- 
Sean Chittenden

Sat, 16 Aug 2003 22:48:16 -0700
Sean Chittenden <sean@chittenden.org>
Re: [HACKERS] Are we losing momentum?
Short summary:

   This patch allows ISO 8601 "time intervals" using the "format 
   with time-unit designators" to specify postgresql "intervals".

   Below I have (A) What these time intervals are, (B) What I
   modified to support them, (C) Issues with intervals I want
   to bring up, and (D) a patch supporting them.

   It's helpful to me.  Any feedback is appreciated.  If you 
   did want to consider including it, let me know what to clean 
   up.  If not, I thought I'd just put it here if anyone else finds
   it useful too.

   Thanks for your time,
  
      Ron Mayer

Longer:

(A) What these intervals are.

   ISO 8601, the standard from which PostgreSQL gets some of it's 
   time syntax, also has a specification for "time-intervals".
  
   In particular, section 5.5.4.2 has a "Representation of
   time-interval by duration only" which I believe maps
   nicely to ISO intervals.

   Compared to the ISO 8601 time interval specification, the
   postgresql interval syntax is quite verbose.  For example:

     Postgresql interval:              ISO8601 Interval
     ---------------------------------------------------
     '1 year 6 months'                'P1Y6M'
     '3 hours 25 minutes 42 seconds'  'PT3H25M42S'

   Yeah, it's uglier, but it sure is short which can make
   for quicker typing and shorter scripts, and if for some
   strange reason you had an application using this format
   it's nice not to have to translate.

   The syntax is as follows:
       Basic extended format:  PnYnMnDTnHnMnS
                               PnW

       Where everything before the "T" is a date-part and everything
       after is a time-part.  W is for weeks.
       In the date-part, Y=Year, M=Month,  D=Day
       In the time-part, H=Hour, M=Minute, S=Second

   Much more info can be found from the draft standard
   ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
   The final standard's only available for $$$ so I didn't
   look at it.  Some other sites imply that this part didn't
   change from the last draft to the standard.


(B) This change was made by adding two functions to "datetime.c"
    next to where DecodeInterval parses the normal interval syntax.

    A total of 313 lines were added, including comments and sgml docs.
    Of these only 136 are actual code, the rest, comments, whitespace, etc.


    One new function "DecodeISO8601Interval" follows the style of
    "DecodeInterval" below it, and trys to strictly follow the ISO
    syntax.  If it doesn't match, it'll return -1 and the old syntax
    will be checked as before.

    The first test (first character of the first field must be 'P', 
    and second character must be 'T' or '\0') should be fast so I don't
    think this will impact performance of existing code.


    The second function ("adjust_fval") is just a small helper-function
    to remove some of the cut&paste style that DecodeInterval used.

    It seems to work.
    =======================================================================
    betadb=# select 'P1M15DT12H30M7S'::interval;
            interval        
    ------------------------
     1 mon 15 days 12:30:07
    (1 row)

    betadb=# select '1 month 15 days 12 hours 30 minutes 7 seconds'::interval;
	    interval        
    ------------------------
    1 mon 15 days 12:30:07
    (1 row)
    =====================================================================



(C) Open issues with intervals, and questions I'd like to ask.

    1.  DecodeInterval seems to have a hardcoded '.' for specifying
        fractional times.  ISO 8601 states that both '.' and ',' are
        ok, but "of these, the comma is the preferred sign".

        In DecodeISO8601Interval I loosened the test to allow
        both but left it as it was in DecodeInterval.  Should
        both be changed to make them more consistant?

    2.  In "DecodeInterval", fractional weeks and fractional months
        can produce seconds; but fractional years can not (rounded
        to months).  I didn't understand the reasoning for this, so
        I left it the same, and followed the same convention for
        ISO intervals.  Should I change this?

    3.  I could save a bunch of copy-paste-lines-of-code from the
        pre-existing DecodeInterval by calling the adjust_fval helper
        function.  The tradeoff is a few extra function-calls when
        decoding an interval.  However I didn't want to risk changes
        to the existing part unless you guys encourage me to do so.


(D) The patch.


Index: doc/src/sgml/datatype.sgml
===================================================================
RCS file: /projects/cvsroot/pgsql-server/doc/src/sgml/datatype.sgml,v
retrieving revision 1.123
diff -u -1 -0 -r1.123 datatype.sgml
--- doc/src/sgml/datatype.sgml	31 Aug 2003 17:32:18 -0000	1.123
+++ doc/src/sgml/datatype.sgml	8 Sep 2003 04:04:58 -0000
@@ -1735,20 +1735,71 @@
       Quantities of days, hours, minutes, and seconds can be specified without
       explicit unit markings.  For example, <literal>'1 12:59:10'</> is read
       the same as <literal>'1 day 12 hours 59 min 10 sec'</>.
      </para>
 
      <para>
       The optional precision
       <replaceable>p</replaceable> should be between 0 and 6, and
       defaults to the precision of the input literal.
      </para>
+
+
+     <para>
+      Alternatively, <type>interval</type> values can be written as 
+      ISO 8601 time intervals, using the "Format with time-unit designators".
+      This format always starts with the character <literal>'P'</>, followed 
+      by a string of values followed by single character time-unit designators.
+      A <literal>'T'</> separates the date and time parts of the interval.
+     </para>
+
+     <para>
+       Format:  PnYnMnDTnHnMnS
+     </para>
+     <para>
+       In this format, <literal>'n'</> gets replaced by a number, and 
+       <literal>Y</> represents years, 
+       <literal>M</> (in the date part) months,
+       <literal>D</> months,
+       <literal>H</> hours,
+       <literal>M</> (in the time part) minutes,
+       and <literal>S</> seconds.
+     </para>
+      
+
+     <table id="interval-example-table">
+	   <title>Interval Example</title>
+	   <tgroup cols="2">
+		<thead>
+		 <row>
+		  <entry>Traditional</entry>
+		  <entry>ISO-8601 time-interval</entry>
+		 </row>
+		</thead>
+		<tbody>
+		 <row>
+		  <entry>1 month</entry>
+		  <entry>P1M</entry>
+		 </row>
+		 <row>
+		  <entry>1 hour 30 minutes</entry>
+		  <entry>PT1H30M</entry>
+		 </row>
+		 <row>
+		  <entry>2 years 10 months 15 days 10 hours 30 minutes 20 seconds</entry>
+		  <entry>P2Y10M15DT10H30M20S</entry>
+		 </row>
+		</tbody>
+	   </thead>
+	  </table>
+	  
+     </para>
     </sect3>
 
     <sect3>
      <title>Special Values</title>
 
      <indexterm>
       <primary>time</primary>
       <secondary>constants</secondary>
      </indexterm>
 
Index: src/backend/utils/adt/datetime.c
===================================================================
RCS file: /projects/cvsroot/pgsql-server/src/backend/utils/adt/datetime.c,v
retrieving revision 1.116
diff -u -1 -0 -r1.116 datetime.c
--- src/backend/utils/adt/datetime.c	27 Aug 2003 23:29:28 -0000	1.116
+++ src/backend/utils/adt/datetime.c	8 Sep 2003 04:04:59 -0000
@@ -30,20 +30,21 @@
 			 struct tm * tm, fsec_t *fsec, int *is2digits);
 static int DecodeNumberField(int len, char *str,
 				  int fmask, int *tmask,
 				  struct tm * tm, fsec_t *fsec, int *is2digits);
 static int DecodeTime(char *str, int fmask, int *tmask,
 		   struct tm * tm, fsec_t *fsec);
 static int	DecodeTimezone(char *str, int *tzp);
 static datetkn *datebsearch(char *key, datetkn *base, unsigned int nel);
 static int	DecodeDate(char *str, int fmask, int *tmask, struct tm * tm);
 static void TrimTrailingZeros(char *str);
+static int  DecodeISO8601Interval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec);
 
 
 int			day_tab[2][13] = {
 	{31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31, 0},
 {31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31, 0}};
 
 char	   *months[] = {"Jan", "Feb", "Mar", "Apr", "May", "Jun",
 "Jul", "Aug", "Sep", "Oct", "Nov", "Dec", NULL};
 
 char	   *days[] = {"Sunday", "Monday", "Tuesday", "Wednesday",
@@ -2872,30 +2873,271 @@
 			default:
 				*val = tp->value;
 				break;
 		}
 	}
 
 	return type;
 }
 
 
+void adjust_fval(double fval,struct tm * tm, fsec_t *fsec, int scale);
+{
+	int	sec;
+	fval	   *= scale;
+	sec		    = fval;
+	tm->tm_sec += sec;
+#ifdef HAVE_INT64_TIMESTAMP
+	*fsec	   += ((fval - sec) * 1000000);
+#else
+	*fsec	   += (fval - sec);
+#endif
+}
+
+
+/* DecodeISO8601Interval()
+ *
+ *  Check if it's a ISO 8601 Section 5.5.4.2 "Representation of
+ *  time-interval by duration only." 
+ *  Basic extended format:  PnYnMnDTnHnMnS
+ *                          PnW
+ *  For more info.
+ *  http://www.astroclark.freeserve.co.uk/iso8601/index.html
+ *  ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
+ *
+ *  Examples:  P1D  for 1 day
+ *             PT1H for 1 hour
+ *             P2Y6M7DT1H30M for 2 years, 6 months, 7 days 1 hour 30 min
+ *
+ *  The first field is exactly "p" or "pt" it may be of this type.
+ *
+ *  Returns -1 if the field is not of this type.
+ *
+ *  It pretty strictly checks the spec, with the two exceptions
+ *  that a week field ('W') may coexist with other units, and that
+ *  this function allows decimals in fields other than the least
+ *  significant units.
+ */
+int
+DecodeISO8601Interval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec) 
+{
+	char	   *cp;
+	int			fmask = 0,
+				tmask;
+	int			val;
+	double		fval;
+	int			arg;
+	int			datepart;
+
+    /*
+	 * An ISO 8601 "time-interval by duration only" must start
+	 * with a 'P'.  If it contains a date-part, 'p' will be the
+	 * only character in the field.  If it contains no date part
+	 * it will contain exactly to characters 'PT' indicating a
+	 * time part.
+	 * Anything else is illegal and will be treated like a 
+	 * traditional postgresql interval.
+	 */
+    if (!(field[0][0] == 'p' &&
+          ((field[0][1] == 0) || (field[0][1] == 't' && field[0][2] == 0))))
+	{
+	  return -1;
+	}
+
+
+    /*
+	 * If the first field is exactly 1 character ('P'), it starts
+	 * with date elements.  Otherwise it's two characters ('PT');
+	 * indicating it starts with a time part.
+	 */
+	datepart = (field[0][1] == 0);
+
+	/*
+	 * Every value must have a unit, so we require an even
+	 * number of value/unit pairs. Therefore we require an
+	 * odd nubmer of fields, including the prefix 'P'.
+	 */
+	if ((nf & 1) == 0)
+		return -1;
+
+	/*
+	 * Process pairs of fields at a time.
+	 */
+	for (arg = 1 ; arg < nf ; arg+=2) 
+	{
+		char * value = field[arg  ];
+		char * units = field[arg+1];
+
+		/*
+		 * The value part must be a number.
+		 */
+		if (ftype[arg] != DTK_NUMBER) 
+			return -1;
+
+		/*
+		 * extract the number, almost exactly like the non-ISO interval.
+		 */
+		val = strtol(value, &cp, 10);
+
+		/*
+		 * One difference from the normal postgresql interval below...
+		 * ISO 8601 states that "Of these, the comma is the preferred 
+		 * sign" so I allow it here for locales that support it.
+		 * Note: Perhaps the old-style interval code below should
+		 * allow for this too, but I didn't want to risk backward
+		 * compatability.
+		 */
+		if (*cp == '.' || *cp == ',') 
+		{
+			fval = strtod(cp, &cp);
+			if (*cp != '\0')
+				return -1;
+
+			if (val < 0)
+				fval = -(fval);
+		}
+		else if (*cp == '\0')
+			fval = 0;
+		else
+			return -1;
+
+
+		if (datepart)
+		{
+			/*
+			 * All the 8601 unit specifiers are 1 character, but may
+			 * be followed by a 'T' character if transitioning between
+			 * the date part and the time part.  If it's not either
+			 * one character or two characters with the second being 't'
+			 * it's an error.
+			 */
+			if (!(units[1] == 0 || (units[1] == 't' && units[2] == 0)))
+				return -1;
+
+			if (units[1] == 't')
+				datepart = 0;
+
+			switch (units[0]) /* Y M D W */
+			{
+				case 'd':
+					tm->tm_mday += val;
+					if (fval != 0)
+					  adjust_fval(fval,tm,fsec, 86400);
+					tmask = ((fmask & DTK_M(DAY)) ? 0 : DTK_M(DAY));
+					break;
+
+				case 'w':
+					tm->tm_mday += val * 7;
+					if (fval != 0)
+					  adjust_fval(fval,tm,fsec,7 * 86400);
+					tmask = ((fmask & DTK_M(DAY)) ? 0 : DTK_M(DAY));
+					break;
+
+				case 'm':
+					tm->tm_mon += val;
+					if (fval != 0)
+					  adjust_fval(fval,tm,fsec,30 * 86400);
+					tmask = DTK_M(MONTH);
+					break;
+
+				case 'y':
+					/*
+					 * Why can fractional months produce seconds,
+					 * but fractional years can't?  Well the older
+					 * interval code below has the same property
+					 * so this one follows the other one too.
+					 */
+					tm->tm_year += val;
+					if (fval != 0)
+						tm->tm_mon += (fval * 12);
+					tmask = ((fmask & DTK_M(YEAR)) ? 0 : DTK_M(YEAR));
+					break;
+
+				default:
+					return -1;  /* invald date unit prefix */
+			}
+		}
+		else
+		{
+			/*
+			 * ISO 8601 time part.
+			 * In the time part, only one-character
+			 * unit prefixes are allowed.  If it's more
+			 * than one character, it's not a valid ISO 8601
+			 * time interval by duration.
+			 */
+			if (units[1] != 0)
+				return -1;
+
+			switch (units[0]) /* H M S */
+			{
+				case 's':
+					tm->tm_sec += val;
+#ifdef HAVE_INT64_TIMESTAMP
+					*fsec += (fval * 1000000);
+#else
+					*fsec += fval;
+#endif
+					tmask = DTK_M(SECOND);
+					break;
+
+				case 'm':
+					tm->tm_min += val;
+					if (fval != 0)
+					  adjust_fval(fval,tm,fsec,60);
+					tmask = DTK_M(MINUTE);
+					break;
+
+				case 'h':
+					tm->tm_hour += val;
+					if (fval != 0)
+					  adjust_fval(fval,tm,fsec,3600);
+					tmask = DTK_M(HOUR);
+					break;
+
+				default:
+					return -1; /* invald time unit prefix */
+			}
+		}
+		fmask |= tmask;
+	}
+
+	if (*fsec != 0)
+	{
+		int			sec;
+
+#ifdef HAVE_INT64_TIMESTAMP
+		sec = (*fsec / INT64CONST(1000000));
+		*fsec -= (sec * INT64CONST(1000000));
+#else
+		TMODULO(*fsec, sec, 1e0);
+#endif
+		tm->tm_sec += sec;
+	}
+	return (fmask != 0) ? 0 : -1;
+}
+
+
 /* DecodeInterval()
  * Interpret previously parsed fields for general time interval.
  * Returns 0 if successful, DTERR code if bogus input detected.
  *
  * Allow "date" field DTK_DATE since this could be just
  *	an unsigned floating point number. - thomas 1997-11-16
  *
  * Allow ISO-style time span, with implicit units on number of days
  *	preceding an hh:mm:ss field. - thomas 1998-04-30
+ * 
+ * Allow ISO-8601 style "Representation of time-interval by duration only"
+ *  of the format 'PnYnMnDTnHnMnS' and 'PnW' - ron 2003-08-30
  */
+
 int
 DecodeInterval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec)
 {
 	int			is_before = FALSE;
 	char	   *cp;
 	int			fmask = 0,
 				tmask,
 				type;
 	int			i;
 	int			dterr;
@@ -2906,20 +3148,37 @@
 
 	type = IGNORE_DTF;
 	tm->tm_year = 0;
 	tm->tm_mon = 0;
 	tm->tm_mday = 0;
 	tm->tm_hour = 0;
 	tm->tm_min = 0;
 	tm->tm_sec = 0;
 	*fsec = 0;
 
+	/*
+	 *  Check if it's a ISO 8601 Section 5.5.4.2 "Representation of
+     *  time-interval by duration only." 
+	 *  Basic extended format:  PnYnMnDTnHnMnS
+	 *                          PnW
+	 *  http://www.astroclark.freeserve.co.uk/iso8601/index.html
+	 *  ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
+	 *  Examples:  P1D  for 1 day
+	 *             PT1H for 1 hour
+	 *             P2Y6M7DT1H30M for 2 years, 6 months, 7 days 1 hour 30 min
+	 *
+	 *  The first field is exactly "p" or "pt" it may be of this type.
+	 */
+	if (DecodeISO8601Interval(field,ftype,nf,dtype,tm,fsec) == 0) {
+	    return 0;
+    }
+
 	/* read through list backwards to pick up units before values */
 	for (i = nf - 1; i >= 0; i--)
 	{
 		switch (ftype[i])
 		{
 			case DTK_TIME:
 				dterr = DecodeTime(field[i], fmask, &tmask, tm, fsec);
 				if (dterr)
 					return dterr;
 				type = DTK_DAY;
@@ -2983,20 +3242,21 @@
 				}
 				/* DROP THROUGH */
 
 			case DTK_DATE:
 			case DTK_NUMBER:
 				val = strtol(field[i], &cp, 10);
 
 				if (type == IGNORE_DTF)
 					type = DTK_SECOND;
 
+				/* should this allow ',' for locales that use it ? */
 				if (*cp == '.')
 				{
 					fval = strtod(cp, &cp);
 					if (*cp != '\0')
 						return DTERR_BAD_FORMAT;
 
 					if (val < 0)
 						fval = -(fval);
 				}
 				else if (*cp == '\0')

===================================================================

Sun, 7 Sep 2003 21:50:49 -0700
"Ron Mayer" <ron@intervideo.com>
ISO 8601 "Time Intervals" of the "format with time-unit deignators"
"Ron Mayer" <ron@intervideo.com> writes:
>    Compared to the ISO 8601 time interval specification, the
>    postgresql interval syntax is quite verbose.  For example:

>      Postgresql interval:              ISO8601 Interval
>      ---------------------------------------------------
>      '1 year 6 months'                'P1Y6M'
>      '3 hours 25 minutes 42 seconds'  'PT3H25M42S'

Er, don't we support that already?  I know I saw code to support
something much like that syntax last time I looked into the datetime
routines.

			regards, tom lane

Mon, 08 Sep 2003 01:47:17 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
	deignators"

Is there a way of producing as well as reading this format? Or did I miss
something?

cheers

andrew

Ron Mayer said:
> Short summary:
>
>   This patch allows ISO 8601 "time intervals" using the "format
>   with time-unit designators" to specify postgresql "intervals".
>
>   Below I have (A) What these time intervals are, (B) What I
>   modified to support them, (C) Issues with intervals I want
>   to bring up, and (D) a patch supporting them.
>
>   It's helpful to me.  Any feedback is appreciated.  If you
>   did want to consider including it, let me know what to clean
>   up.  If not, I thought I'd just put it here if anyone else finds it
>   useful too.
>
>   Thanks for your time,
>
>      Ron Mayer
>
> Longer:
>
> (A) What these intervals are.
>
>   ISO 8601, the standard from which PostgreSQL gets some of it's  time
>   syntax, also has a specification for "time-intervals".
>
>   In particular, section 5.5.4.2 has a "Representation of
>   time-interval by duration only" which I believe maps
>   nicely to ISO intervals.
>
>   Compared to the ISO 8601 time interval specification, the
>   postgresql interval syntax is quite verbose.  For example:
>
>     Postgresql interval:              ISO8601 Interval
>     ---------------------------------------------------
>     '1 year 6 months'                'P1Y6M'
>     '3 hours 25 minutes 42 seconds'  'PT3H25M42S'
>
>   Yeah, it's uglier, but it sure is short which can make
>   for quicker typing and shorter scripts, and if for some
>   strange reason you had an application using this format
>   it's nice not to have to translate.
>
>   The syntax is as follows:
>       Basic extended format:  PnYnMnDTnHnMnS
>                               PnW
>
>       Where everything before the "T" is a date-part and everything
>       after is a time-part.  W is for weeks.
>       In the date-part, Y=Year, M=Month,  D=Day
>       In the time-part, H=Hour, M=Minute, S=Second
>
>   Much more info can be found from the draft standard
>   ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
>   The final standard's only available for $$$ so I didn't
>   look at it.  Some other sites imply that this part didn't
>   change from the last draft to the standard.
>
>
> (B) This change was made by adding two functions to "datetime.c"
>    next to where DecodeInterval parses the normal interval syntax.
>
>    A total of 313 lines were added, including comments and sgml docs.
>    Of these only 136 are actual code, the rest, comments, whitespace,
>    etc.
>
>
>    One new function "DecodeISO8601Interval" follows the style of
>    "DecodeInterval" below it, and trys to strictly follow the ISO
>    syntax.  If it doesn't match, it'll return -1 and the old syntax
>    will be checked as before.
>
>    The first test (first character of the first field must be 'P',  and
>    second character must be 'T' or '\0') should be fast so I don't
>    think this will impact performance of existing code.
>
>
>    The second function ("adjust_fval") is just a small helper-function
>    to remove some of the cut&paste style that DecodeInterval used.
>
>    It seems to work.
>
=======================================================================
>    betadb=# select 'P1M15DT12H30M7S'::interval;
>            interval
>    ------------------------
>     1 mon 15 days 12:30:07
>    (1 row)
>
>    betadb=# select '1 month 15 days 12 hours 30 minutes 7
>    seconds'::interval;
> 	    interval
>    ------------------------
>    1 mon 15 days 12:30:07
>    (1 row)
>    =====================================================================
>
>
>
> (C) Open issues with intervals, and questions I'd like to ask.
>
>    1.  DecodeInterval seems to have a hardcoded '.' for specifying
>        fractional times.  ISO 8601 states that both '.' and ',' are ok,
>        but "of these, the comma is the preferred sign".
>
>        In DecodeISO8601Interval I loosened the test to allow
>        both but left it as it was in DecodeInterval.  Should
>        both be changed to make them more consistant?
>
>    2.  In "DecodeInterval", fractional weeks and fractional months
>        can produce seconds; but fractional years can not (rounded to
>        months).  I didn't understand the reasoning for this, so I left
>        it the same, and followed the same convention for
>        ISO intervals.  Should I change this?
>
>    3.  I could save a bunch of copy-paste-lines-of-code from the
>        pre-existing DecodeInterval by calling the adjust_fval helper
>        function.  The tradeoff is a few extra function-calls when
>        decoding an interval.  However I didn't want to risk changes to
>        the existing part unless you guys encourage me to do so.
>
>
> (D) The patch.
>
>
[snip]



Mon, 8 Sep 2003 02:32:17 -0400 (EDT)
<andrew@dunslane.net>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
	deignators'

Tom wrote:
> "Ron Mayer" <ron@intervideo.com> writes:
> >    Compared to the ISO 8601 time interval specification, the
> >    postgresql interval syntax is quite verbose.  For example:
> 
> >      Postgresql interval:              ISO8601 Interval
> >      ---------------------------------------------------
> >      '1 year 6 months'                'P1Y6M'
> >      '3 hours 25 minutes 42 seconds'  'PT3H25M42S'
> 
> Er, don't we support that already?  I know I saw code to support
> something much like that syntax last time I looked into the datetime
> routines.
> 

Nope.

Postgresql supports a rather bizzare shorthand that has a similar
syntax, but AFAICT, doesn't match ISO 8601 in any way that makes 
it practical.


A disclaimer,  I have the "Final Draft" (ISO/TC 154N 362 
of 2000-12-19) of the spec; but have not seen the official,
expensive, version. 
ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF


For example, if I read it right, I have differences
like this:

     Interval            ISO             Postgres
                         8601            shorthand
     -----------------------------------------------------
     '1 year 1 minute'   'P1YT1M'         '1Y1M'
     '1 year 1 month'    'P1Y1M'          N/A

The best part about the postgresql syntax is that
they omit the required 'P', so it's easy to differentiate
between the two.  :-)


Perhaps one could argue that the postgres shorthand should 
follow the ISO conventions, but I'd not want to break backward
compatability, incase someone out there is using '1H30M' and
expecting minutes instead of months.   If we didn't want to
support two syntaxes, I wouldn't mind eventually depricating
the less-standard one.

   Ron



Mon, 8 Sep 2003 11:59:50 -0700
"Ron Mayer" <ron@intervideo.com>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
	deignators"
andrew@dunslane.net wrote:
> 
> Is there a way of producing as well as reading this format? Or did I miss
> something?

Not yet, but I'd be happy to add it.  

My immediate problem was having some 'P1Y6M' intervals to load. 
I posted this much largely because it was useful to me so might 
help others, and to see if it was of interest to others and 
get feedback on what else to change.

I'd be happy to make it produce the output, and have some style
questions for doing so.

I'd hate to trigger this output on the already-existing 'datestyle' 
of 'ISO', since that would break backward compatability.
I do notice that 8601 has both "basic" and "extended" formats.
The "basic format" is more terse ('19980115' instead of '1998-01-15').

Would it be useful if I added a 'datestyle' of 'ISO basic' which
would produce the most terse formats ('19980115' for dates, 
and 'P1Y1M' for intervals)?

   Ron

PS: What's the best inexpenive way for me to know if this changed 
at all between the final draft and the published standard? 

> Ron Mayer said:
> >   This patch allows ISO 8601 "time intervals" using the "format
> >   with time-unit designators" to specify postgresql "intervals".
> >...
> >   Much more info can be found from the draft standard
> >   ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
> >   The final standard's only available for $$$ so I didn't
> >   look at it.


Mon, 8 Sep 2003 12:16:50 -0700
"Ron Mayer" <ron@intervideo.com>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
	deignators'
"Ron Mayer" <ron@intervideo.com> writes:
> Tom wrote:
>> Er, don't we support that already?

> Postgresql supports a rather bizzare shorthand that has a similar
> syntax, but AFAICT, doesn't match ISO 8601 in any way that makes 
> it practical.

Well, it's *supposed* to match ISO, AFAICT (the comments in the code
talk about "ISO dates").  Unless ISO has put out multiple specs that
cover this?

> Perhaps one could argue that the postgres shorthand should 
> follow the ISO conventions, but I'd not want to break backward
> compatability, incase someone out there is using '1H30M' and
> expecting minutes instead of months.

I doubt anyone is using it, because it's completely undocumented.
If we're going to support the real ISO spec, I'd suggest ripping
out any not-quite-there variant.  (Especially so noting that your
code seems a lot cleaner than the ptype stuff.)

The datetime code is kind of a mess right now, because Thomas Lockhart
walked away from the project while only partway through some significant
additions.  He left some incomplete features and quite a number of bugs
in new-and-untested code.  We've been gradually cleaning up the problems,
but if if you find something that doesn't seem to make sense, it's
likely a bug rather than anything we want to preserve.  In particular,
given the knowledge that it doesn't meet the ISO spec, I'd judge that
the existing code for the ISO shorthand was a work-in-progress.

			regards, tom lane

Mon, 08 Sep 2003 15:59:20 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
	deignators"
"Ron Mayer" <ron@intervideo.com> writes:
> Would it be useful if I added a 'datestyle' of 'ISO basic' which
> would produce the most terse formats ('19980115' for dates, 
> and 'P1Y1M' for intervals)?

I don't really care for using that name for it --- for one thing, you
couldn't do
	set datestyle to iso basic;
because of syntax limitations.  A one-word name is a much better idea.

Perhaps call it "compact" or "terse" datestyle?


> PS: What's the best inexpenive way for me to know if this changed 
> at all between the final draft and the published standard? 

ANSI sells PDFs of ISO specs at their online store
http://webstore.ansi.org/ansidocstore/default.asp
although it looks like they want $81 for 8601, which is not my idea of
"inexpensive".

Usually ISO final drafts differ very little from the published specs;
I think you could just work from the draft and no one would complain.

			regards, tom lane

Mon, 08 Sep 2003 16:17:19 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
	deignators'
Tom Lane wrote:
> > Perhaps one could argue that the postgres shorthand should 
> > follow the ISO conventions, but I'd not want to break backward
> > compatability, incase someone out there is using '1H30M' and
> > expecting minutes instead of months.
> 
> I doubt anyone is using it, because it's completely undocumented.
> If we're going to support the real ISO spec, I'd suggest ripping
> out any not-quite-there variant.  (Especially so noting that your
> code seems a lot cleaner than the ptype stuff.)

Agreed.  Let me put your code in the queue for 7.5.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 8 Sep 2003 16:27:16 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit

This has been saved for the 7.5 release:

	http:/momjian.postgresql.org/cgi-bin/pgpatches2

Feel free to submit an updated patch that rips out the old syntax, as
discussed, or replace this patch with a more comprehensive one.

---------------------------------------------------------------------------

Ron Mayer wrote:
> Short summary:
> 
>    This patch allows ISO 8601 "time intervals" using the "format 
>    with time-unit designators" to specify postgresql "intervals".
> 
>    Below I have (A) What these time intervals are, (B) What I
>    modified to support them, (C) Issues with intervals I want
>    to bring up, and (D) a patch supporting them.
> 
>    It's helpful to me.  Any feedback is appreciated.  If you 
>    did want to consider including it, let me know what to clean 
>    up.  If not, I thought I'd just put it here if anyone else finds
>    it useful too.
> 
>    Thanks for your time,
>   
>       Ron Mayer
> 
> Longer:
> 
> (A) What these intervals are.
> 
>    ISO 8601, the standard from which PostgreSQL gets some of it's 
>    time syntax, also has a specification for "time-intervals".
>   
>    In particular, section 5.5.4.2 has a "Representation of
>    time-interval by duration only" which I believe maps
>    nicely to ISO intervals.
> 
>    Compared to the ISO 8601 time interval specification, the
>    postgresql interval syntax is quite verbose.  For example:
> 
>      Postgresql interval:              ISO8601 Interval
>      ---------------------------------------------------
>      '1 year 6 months'                'P1Y6M'
>      '3 hours 25 minutes 42 seconds'  'PT3H25M42S'
> 
>    Yeah, it's uglier, but it sure is short which can make
>    for quicker typing and shorter scripts, and if for some
>    strange reason you had an application using this format
>    it's nice not to have to translate.
> 
>    The syntax is as follows:
>        Basic extended format:  PnYnMnDTnHnMnS
>                                PnW
> 
>        Where everything before the "T" is a date-part and everything
>        after is a time-part.  W is for weeks.
>        In the date-part, Y=Year, M=Month,  D=Day
>        In the time-part, H=Hour, M=Minute, S=Second
> 
>    Much more info can be found from the draft standard
>    ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
>    The final standard's only available for $$$ so I didn't
>    look at it.  Some other sites imply that this part didn't
>    change from the last draft to the standard.
> 
> 
> (B) This change was made by adding two functions to "datetime.c"
>     next to where DecodeInterval parses the normal interval syntax.
> 
>     A total of 313 lines were added, including comments and sgml docs.
>     Of these only 136 are actual code, the rest, comments, whitespace, etc.
> 
> 
>     One new function "DecodeISO8601Interval" follows the style of
>     "DecodeInterval" below it, and trys to strictly follow the ISO
>     syntax.  If it doesn't match, it'll return -1 and the old syntax
>     will be checked as before.
> 
>     The first test (first character of the first field must be 'P', 
>     and second character must be 'T' or '\0') should be fast so I don't
>     think this will impact performance of existing code.
> 
> 
>     The second function ("adjust_fval") is just a small helper-function
>     to remove some of the cut&paste style that DecodeInterval used.
> 
>     It seems to work.
>     =======================================================================
>     betadb=# select 'P1M15DT12H30M7S'::interval;
>             interval        
>     ------------------------
>      1 mon 15 days 12:30:07
>     (1 row)
> 
>     betadb=# select '1 month 15 days 12 hours 30 minutes 7 seconds'::interval;
> 	    interval        
>     ------------------------
>     1 mon 15 days 12:30:07
>     (1 row)
>     =====================================================================
> 
> 
> 
> (C) Open issues with intervals, and questions I'd like to ask.
> 
>     1.  DecodeInterval seems to have a hardcoded '.' for specifying
>         fractional times.  ISO 8601 states that both '.' and ',' are
>         ok, but "of these, the comma is the preferred sign".
> 
>         In DecodeISO8601Interval I loosened the test to allow
>         both but left it as it was in DecodeInterval.  Should
>         both be changed to make them more consistant?
> 
>     2.  In "DecodeInterval", fractional weeks and fractional months
>         can produce seconds; but fractional years can not (rounded
>         to months).  I didn't understand the reasoning for this, so
>         I left it the same, and followed the same convention for
>         ISO intervals.  Should I change this?
> 
>     3.  I could save a bunch of copy-paste-lines-of-code from the
>         pre-existing DecodeInterval by calling the adjust_fval helper
>         function.  The tradeoff is a few extra function-calls when
>         decoding an interval.  However I didn't want to risk changes
>         to the existing part unless you guys encourage me to do so.
> 
> 
> (D) The patch.
> 
> 
> Index: doc/src/sgml/datatype.sgml
> ===================================================================
> RCS file: /projects/cvsroot/pgsql-server/doc/src/sgml/datatype.sgml,v
> retrieving revision 1.123
> diff -u -1 -0 -r1.123 datatype.sgml
> --- doc/src/sgml/datatype.sgml	31 Aug 2003 17:32:18 -0000	1.123
> +++ doc/src/sgml/datatype.sgml	8 Sep 2003 04:04:58 -0000
> @@ -1735,20 +1735,71 @@
>        Quantities of days, hours, minutes, and seconds can be specified without
>        explicit unit markings.  For example, <literal>'1 12:59:10'</> is read
>        the same as <literal>'1 day 12 hours 59 min 10 sec'</>.
>       </para>
>  
>       <para>
>        The optional precision
>        <replaceable>p</replaceable> should be between 0 and 6, and
>        defaults to the precision of the input literal.
>       </para>
> +
> +
> +     <para>
> +      Alternatively, <type>interval</type> values can be written as 
> +      ISO 8601 time intervals, using the "Format with time-unit designators".
> +      This format always starts with the character <literal>'P'</>, followed 
> +      by a string of values followed by single character time-unit designators.
> +      A <literal>'T'</> separates the date and time parts of the interval.
> +     </para>
> +
> +     <para>
> +       Format:  PnYnMnDTnHnMnS
> +     </para>
> +     <para>
> +       In this format, <literal>'n'</> gets replaced by a number, and 
> +       <literal>Y</> represents years, 
> +       <literal>M</> (in the date part) months,
> +       <literal>D</> months,
> +       <literal>H</> hours,
> +       <literal>M</> (in the time part) minutes,
> +       and <literal>S</> seconds.
> +     </para>
> +      
> +
> +     <table id="interval-example-table">
> +	   <title>Interval Example</title>
> +	   <tgroup cols="2">
> +		<thead>
> +		 <row>
> +		  <entry>Traditional</entry>
> +		  <entry>ISO-8601 time-interval</entry>
> +		 </row>
> +		</thead>
> +		<tbody>
> +		 <row>
> +		  <entry>1 month</entry>
> +		  <entry>P1M</entry>
> +		 </row>
> +		 <row>
> +		  <entry>1 hour 30 minutes</entry>
> +		  <entry>PT1H30M</entry>
> +		 </row>
> +		 <row>
> +		  <entry>2 years 10 months 15 days 10 hours 30 minutes 20 seconds</entry>
> +		  <entry>P2Y10M15DT10H30M20S</entry>
> +		 </row>
> +		</tbody>
> +	   </thead>
> +	  </table>
> +	  
> +     </para>
>      </sect3>
>  
>      <sect3>
>       <title>Special Values</title>
>  
>       <indexterm>
>        <primary>time</primary>
>        <secondary>constants</secondary>
>       </indexterm>
>  
> Index: src/backend/utils/adt/datetime.c
> ===================================================================
> RCS file: /projects/cvsroot/pgsql-server/src/backend/utils/adt/datetime.c,v
> retrieving revision 1.116
> diff -u -1 -0 -r1.116 datetime.c
> --- src/backend/utils/adt/datetime.c	27 Aug 2003 23:29:28 -0000	1.116
> +++ src/backend/utils/adt/datetime.c	8 Sep 2003 04:04:59 -0000
> @@ -30,20 +30,21 @@
>  			 struct tm * tm, fsec_t *fsec, int *is2digits);
>  static int DecodeNumberField(int len, char *str,
>  				  int fmask, int *tmask,
>  				  struct tm * tm, fsec_t *fsec, int *is2digits);
>  static int DecodeTime(char *str, int fmask, int *tmask,
>  		   struct tm * tm, fsec_t *fsec);
>  static int	DecodeTimezone(char *str, int *tzp);
>  static datetkn *datebsearch(char *key, datetkn *base, unsigned int nel);
>  static int	DecodeDate(char *str, int fmask, int *tmask, struct tm * tm);
>  static void TrimTrailingZeros(char *str);
> +static int  DecodeISO8601Interval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec);
>  
>  
>  int			day_tab[2][13] = {
>  	{31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31, 0},
>  {31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31, 0}};
>  
>  char	   *months[] = {"Jan", "Feb", "Mar", "Apr", "May", "Jun",
>  "Jul", "Aug", "Sep", "Oct", "Nov", "Dec", NULL};
>  
>  char	   *days[] = {"Sunday", "Monday", "Tuesday", "Wednesday",
> @@ -2872,30 +2873,271 @@
>  			default:
>  				*val = tp->value;
>  				break;
>  		}
>  	}
>  
>  	return type;
>  }
>  
>  
> +void adjust_fval(double fval,struct tm * tm, fsec_t *fsec, int scale);
> +{
> +	int	sec;
> +	fval	   *= scale;
> +	sec		    = fval;
> +	tm->tm_sec += sec;
> +#ifdef HAVE_INT64_TIMESTAMP
> +	*fsec	   += ((fval - sec) * 1000000);
> +#else
> +	*fsec	   += (fval - sec);
> +#endif
> +}
> +
> +
> +/* DecodeISO8601Interval()
> + *
> + *  Check if it's a ISO 8601 Section 5.5.4.2 "Representation of
> + *  time-interval by duration only." 
> + *  Basic extended format:  PnYnMnDTnHnMnS
> + *                          PnW
> + *  For more info.
> + *  http://www.astroclark.freeserve.co.uk/iso8601/index.html
> + *  ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
> + *
> + *  Examples:  P1D  for 1 day
> + *             PT1H for 1 hour
> + *             P2Y6M7DT1H30M for 2 years, 6 months, 7 days 1 hour 30 min
> + *
> + *  The first field is exactly "p" or "pt" it may be of this type.
> + *
> + *  Returns -1 if the field is not of this type.
> + *
> + *  It pretty strictly checks the spec, with the two exceptions
> + *  that a week field ('W') may coexist with other units, and that
> + *  this function allows decimals in fields other than the least
> + *  significant units.
> + */
> +int
> +DecodeISO8601Interval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec) 
> +{
> +	char	   *cp;
> +	int			fmask = 0,
> +				tmask;
> +	int			val;
> +	double		fval;
> +	int			arg;
> +	int			datepart;
> +
> +    /*
> +	 * An ISO 8601 "time-interval by duration only" must start
> +	 * with a 'P'.  If it contains a date-part, 'p' will be the
> +	 * only character in the field.  If it contains no date part
> +	 * it will contain exactly to characters 'PT' indicating a
> +	 * time part.
> +	 * Anything else is illegal and will be treated like a 
> +	 * traditional postgresql interval.
> +	 */
> +    if (!(field[0][0] == 'p' &&
> +          ((field[0][1] == 0) || (field[0][1] == 't' && field[0][2] == 0))))
> +	{
> +	  return -1;
> +	}
> +
> +
> +    /*
> +	 * If the first field is exactly 1 character ('P'), it starts
> +	 * with date elements.  Otherwise it's two characters ('PT');
> +	 * indicating it starts with a time part.
> +	 */
> +	datepart = (field[0][1] == 0);
> +
> +	/*
> +	 * Every value must have a unit, so we require an even
> +	 * number of value/unit pairs. Therefore we require an
> +	 * odd nubmer of fields, including the prefix 'P'.
> +	 */
> +	if ((nf & 1) == 0)
> +		return -1;
> +
> +	/*
> +	 * Process pairs of fields at a time.
> +	 */
> +	for (arg = 1 ; arg < nf ; arg+=2) 
> +	{
> +		char * value = field[arg  ];
> +		char * units = field[arg+1];
> +
> +		/*
> +		 * The value part must be a number.
> +		 */
> +		if (ftype[arg] != DTK_NUMBER) 
> +			return -1;
> +
> +		/*
> +		 * extract the number, almost exactly like the non-ISO interval.
> +		 */
> +		val = strtol(value, &cp, 10);
> +
> +		/*
> +		 * One difference from the normal postgresql interval below...
> +		 * ISO 8601 states that "Of these, the comma is the preferred 
> +		 * sign" so I allow it here for locales that support it.
> +		 * Note: Perhaps the old-style interval code below should
> +		 * allow for this too, but I didn't want to risk backward
> +		 * compatability.
> +		 */
> +		if (*cp == '.' || *cp == ',') 
> +		{
> +			fval = strtod(cp, &cp);
> +			if (*cp != '\0')
> +				return -1;
> +
> +			if (val < 0)
> +				fval = -(fval);
> +		}
> +		else if (*cp == '\0')
> +			fval = 0;
> +		else
> +			return -1;
> +
> +
> +		if (datepart)
> +		{
> +			/*
> +			 * All the 8601 unit specifiers are 1 character, but may
> +			 * be followed by a 'T' character if transitioning between
> +			 * the date part and the time part.  If it's not either
> +			 * one character or two characters with the second being 't'
> +			 * it's an error.
> +			 */
> +			if (!(units[1] == 0 || (units[1] == 't' && units[2] == 0)))
> +				return -1;
> +
> +			if (units[1] == 't')
> +				datepart = 0;
> +
> +			switch (units[0]) /* Y M D W */
> +			{
> +				case 'd':
> +					tm->tm_mday += val;
> +					if (fval != 0)
> +					  adjust_fval(fval,tm,fsec, 86400);
> +					tmask = ((fmask & DTK_M(DAY)) ? 0 : DTK_M(DAY));
> +					break;
> +
> +				case 'w':
> +					tm->tm_mday += val * 7;
> +					if (fval != 0)
> +					  adjust_fval(fval,tm,fsec,7 * 86400);
> +					tmask = ((fmask & DTK_M(DAY)) ? 0 : DTK_M(DAY));
> +					break;
> +
> +				case 'm':
> +					tm->tm_mon += val;
> +					if (fval != 0)
> +					  adjust_fval(fval,tm,fsec,30 * 86400);
> +					tmask = DTK_M(MONTH);
> +					break;
> +
> +				case 'y':
> +					/*
> +					 * Why can fractional months produce seconds,
> +					 * but fractional years can't?  Well the older
> +					 * interval code below has the same property
> +					 * so this one follows the other one too.
> +					 */
> +					tm->tm_year += val;
> +					if (fval != 0)
> +						tm->tm_mon += (fval * 12);
> +					tmask = ((fmask & DTK_M(YEAR)) ? 0 : DTK_M(YEAR));
> +					break;
> +
> +				default:
> +					return -1;  /* invald date unit prefix */
> +			}
> +		}
> +		else
> +		{
> +			/*
> +			 * ISO 8601 time part.
> +			 * In the time part, only one-character
> +			 * unit prefixes are allowed.  If it's more
> +			 * than one character, it's not a valid ISO 8601
> +			 * time interval by duration.
> +			 */
> +			if (units[1] != 0)
> +				return -1;
> +
> +			switch (units[0]) /* H M S */
> +			{
> +				case 's':
> +					tm->tm_sec += val;
> +#ifdef HAVE_INT64_TIMESTAMP
> +					*fsec += (fval * 1000000);
> +#else
> +					*fsec += fval;
> +#endif
> +					tmask = DTK_M(SECOND);
> +					break;
> +
> +				case 'm':
> +					tm->tm_min += val;
> +					if (fval != 0)
> +					  adjust_fval(fval,tm,fsec,60);
> +					tmask = DTK_M(MINUTE);
> +					break;
> +
> +				case 'h':
> +					tm->tm_hour += val;
> +					if (fval != 0)
> +					  adjust_fval(fval,tm,fsec,3600);
> +					tmask = DTK_M(HOUR);
> +					break;
> +
> +				default:
> +					return -1; /* invald time unit prefix */
> +			}
> +		}
> +		fmask |= tmask;
> +	}
> +
> +	if (*fsec != 0)
> +	{
> +		int			sec;
> +
> +#ifdef HAVE_INT64_TIMESTAMP
> +		sec = (*fsec / INT64CONST(1000000));
> +		*fsec -= (sec * INT64CONST(1000000));
> +#else
> +		TMODULO(*fsec, sec, 1e0);
> +#endif
> +		tm->tm_sec += sec;
> +	}
> +	return (fmask != 0) ? 0 : -1;
> +}
> +
> +
>  /* DecodeInterval()
>   * Interpret previously parsed fields for general time interval.
>   * Returns 0 if successful, DTERR code if bogus input detected.
>   *
>   * Allow "date" field DTK_DATE since this could be just
>   *	an unsigned floating point number. - thomas 1997-11-16
>   *
>   * Allow ISO-style time span, with implicit units on number of days
>   *	preceding an hh:mm:ss field. - thomas 1998-04-30
> + * 
> + * Allow ISO-8601 style "Representation of time-interval by duration only"
> + *  of the format 'PnYnMnDTnHnMnS' and 'PnW' - ron 2003-08-30
>   */
> +
>  int
>  DecodeInterval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec)
>  {
>  	int			is_before = FALSE;
>  	char	   *cp;
>  	int			fmask = 0,
>  				tmask,
>  				type;
>  	int			i;
>  	int			dterr;
> @@ -2906,20 +3148,37 @@
>  
>  	type = IGNORE_DTF;
>  	tm->tm_year = 0;
>  	tm->tm_mon = 0;
>  	tm->tm_mday = 0;
>  	tm->tm_hour = 0;
>  	tm->tm_min = 0;
>  	tm->tm_sec = 0;
>  	*fsec = 0;
>  
> +	/*
> +	 *  Check if it's a ISO 8601 Section 5.5.4.2 "Representation of
> +     *  time-interval by duration only." 
> +	 *  Basic extended format:  PnYnMnDTnHnMnS
> +	 *                          PnW
> +	 *  http://www.astroclark.freeserve.co.uk/iso8601/index.html
> +	 *  ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
> +	 *  Examples:  P1D  for 1 day
> +	 *             PT1H for 1 hour
> +	 *             P2Y6M7DT1H30M for 2 years, 6 months, 7 days 1 hour 30 min
> +	 *
> +	 *  The first field is exactly "p" or "pt" it may be of this type.
> +	 */
> +	if (DecodeISO8601Interval(field,ftype,nf,dtype,tm,fsec) == 0) {
> +	    return 0;
> +    }
> +
>  	/* read through list backwards to pick up units before values */
>  	for (i = nf - 1; i >= 0; i--)
>  	{
>  		switch (ftype[i])
>  		{
>  			case DTK_TIME:
>  				dterr = DecodeTime(field[i], fmask, &tmask, tm, fsec);
>  				if (dterr)
>  					return dterr;
>  				type = DTK_DAY;
> @@ -2983,20 +3242,21 @@
>  				}
>  				/* DROP THROUGH */
>  
>  			case DTK_DATE:
>  			case DTK_NUMBER:
>  				val = strtol(field[i], &cp, 10);
>  
>  				if (type == IGNORE_DTF)
>  					type = DTK_SECOND;
>  
> +				/* should this allow ',' for locales that use it ? */
>  				if (*cp == '.')
>  				{
>  					fval = strtod(cp, &cp);
>  					if (*cp != '\0')
>  						return DTERR_BAD_FORMAT;
>  
>  					if (val < 0)
>  						fval = -(fval);
>  				}
>  				else if (*cp == '\0')
> 
> ===================================================================
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
>                http://archives.postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 8 Sep 2003 16:29:56 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
Tom wrote: 
> "Ron Mayer" <ron@intervideo.com> writes:
> > Tom wrote:
> >> Er, don't we support that already?
> > ...AFAICT, doesn't match ISO 8601...
> 
> Well, it's *supposed* to match ISO....  Unless ISO has put out 
> multiple specs that cover this?

Any way to tell if this is the case.  
8601's the one I see cited the most.


> > ...I'd not want to break backward compatability...'1H30M'
>
> I doubt anyone is using it, because it's completely undocumented.
> If we're going to support the real ISO spec, I'd suggest ripping
> out any not-quite-there variant.

I'm happy to look into it.  Rip out completely?  Ifdef? 

> We've been gradually cleaning up the problems, but if if you find 
> something that doesn't seem to make sense, it's likely a bug rather
> than anything we want to preserve. 

I've seen a few more cases that don't make sense.

For example "why is 0.001 years less than 0.001 months".

  betadb=# select '0.01 years'::interval
   interval
  ----------
   00:00:00

  betadb=# select '0.01 months'::interval
   interval
  ----------
   07:12:00

If I'm breaking backward compatability anyway, I'd be happy to tweak
things like this one too.  Unless, of course someone can give me a 
reason why we want fractional years rounded to months, but fractional 
months are rounded to fractions of a second.

   Ron Mayer.

PS: mailinglist etiquite question... for discussion, should I
    more this to hackers, or continue it here.


Mon, 8 Sep 2003 13:42:11 -0700
"Ron Mayer" <ron@intervideo.com>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
	deignators"
Ron Mayer wrote:
> If I'm breaking backward compatability anyway, I'd be happy to tweak
> things like this one too.  Unless, of course someone can give me a 
> reason why we want fractional years rounded to months, but fractional 
> months are rounded to fractions of a second.
> 
>    Ron Mayer.
> 
> PS: mailinglist etiquite question... for discussion, should I
>     more this to hackers, or continue it here.

Your choice, but you get a larger audience on hackers.  I usually keep
things on patches when I have lots of code to post, and other times move
to hackers.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073


Mon, 8 Sep 2003 17:05:14 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
"Ron Mayer" <ron@intervideo.com> writes:
> For example "why is 0.001 years less than 0.001 months".

And look at this:

regression=# select '0.99 years'::interval;
 interval
----------
 11 mons
(1 row)

regression=#  select '0.99 months'::interval;
     interval
------------------
 29 days 16:48:00
(1 row)

It kinda looks like fractional years are converted to integer months
(truncating) while fractional months are moved to the seconds part
of the interval.  Ick.  The handling ought to be consistent if you
ask me.

> If I'm breaking backward compatability anyway, I'd be happy to tweak
> things like this one too.  Unless, of course someone can give me a 
> reason why we want fractional years rounded to months, but fractional 
> months are rounded to fractions of a second.

Actually, what I'd like to see done with interval is re-implement it as
a three-field entity, separately storing months, days, and seconds.
The separation between months and smaller units is good because a month
isn't a fixed number of any smaller unit, but the same holds true for
days and smaller units (days are not always 24 hours, consider DST
transitions).  This would no doubt cause some backwards compatibility
problems, but overall it would fix many more cases than it breaks.
We see complaints about related issues regularly, every spring and fall...

I'm unsure whether fractional months or fractional days are sensible
to accept, but surely we should accept both or reject both.  (This might
suggest that the underlying storage for the month and day fields should
be float not int, btw, but I am not sure about it.)

> PS: mailinglist etiquite question... for discussion, should I
>     more this to hackers, or continue it here.

At this point it should move to pghackers, I think.

			regards, tom lane

Mon, 08 Sep 2003 17:19:13 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
	deignators"
Tom Lane writes:

> "Ron Mayer" <ron@intervideo.com> writes:
> > Would it be useful if I added a 'datestyle' of 'ISO basic' which
> > would produce the most terse formats ('19980115' for dates,
> > and 'P1Y1M' for intervals)?
>
> I don't really care for using that name for it --- for one thing, you
> couldn't do
> 	set datestyle to iso basic;
> because of syntax limitations.  A one-word name is a much better idea.

iso8601

Keep in mind that SQL itself is also a kind of ISO, so being more specific
is useful.

-- 
Peter Eisentraut   peter_e@gmx.net


Mon, 8 Sep 2003 23:58:09 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> I don't really care for using that name for it ---

> iso8601

> Keep in mind that SQL itself is also a kind of ISO, so being more specific
> is useful.

Yes, but by the same token "iso8601" isn't specific enough either.
Several of the other input formats we support have at least as good a
claim on that name.

			regards, tom lane

Mon, 08 Sep 2003 18:06:11 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
	deignators'
Tom Lane writes:

> Yes, but by the same token "iso8601" isn't specific enough either.
> Several of the other input formats we support have at least as good a
> claim on that name.

The only input formats we support are along the lines of

@ 1 year 2 mons 3 days 4 hours 5 mins 6 secs
@ 1 year 2 mons 3 days 04:05:06

These are also the supported output formats (the first you get for 'sql',
'postgres', and 'german' formats; the second is 'iso').  A quick check of
ISO 8601 shows, however, that neither of these are close to anything
specified in that standard.

-- 
Peter Eisentraut   peter_e@gmx.net


Tue, 9 Sep 2003 00:39:44 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> Yes, but by the same token "iso8601" isn't specific enough either.
>> Several of the other input formats we support have at least as good a
>> claim on that name.

> The only input formats we support are along the lines of

> @ 1 year 2 mons 3 days 4 hours 5 mins 6 secs
> @ 1 year 2 mons 3 days 04:05:06

Sorry, I was thinking of timestamp formats not intervals.
You're right that we don't have anything else particularly ISO-standard
for intervals, but my understanding is that formats like '2003-09-08
18:43:31.046283-04' are ISO8601 compatible for timestamps.  (Possibly
you need to put a T in there for strict compatibility, not sure.)

			regards, tom lane

Mon, 08 Sep 2003 18:45:03 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
	deignators'
Tom wrote...
> At this point it should move to pghackers, I think.

Background for pghackers first, open issues below...

   Over on pgpatches we've been discussing ISO syntax for
   ï¿½time intervalsï¿½ of the ï¿½format with time-unit designatorsï¿½.
    http://archives.postgresql.org/pgsql-patches/2003-09/msg00103.php
   A short summary is that Iï¿½ve submitted a patch that
   accepts intervals of this format..
     Postgresql interval:              ISO8601 Interval
     ---------------------------------------------------
     '1 year 6 months'                'P1Y6M'
     '3 hours 25 minutes 42 seconds'  'PT3H25M42S'
   The final draft is here
     ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF

   This patch was backward-compatable, but further improvements
   discussed on patches may break compatability so I wanted to
   discuss them here before implementing them.   Iï¿½ll also
   be submitting a new datestyle ï¿½iso8601ï¿½ to output these intervals.

Open issues:

1. Postgresql supported a shorthand for intervals that had
   a similar, but not compatable syntax:
     Interval            ISO             Existing postgres
                         8601            shorthand
     -----------------------------------------------------
     '1 year 1 minute'   'P1YT1M'         '1Y1M'
     '1 year 1 month'    'P1Y1M'          N/A

   The current thinking of the thread in pgpatches is to remove
   the existing (undocumented) syntax.

   Removing this will break backward compatability if anyone
   used this feature.  Let me know if you needed it.

2. Some of the parsing for intervals is inconsistant and
   confusing.  For example, note that ï¿½0.01 yearsï¿½ is
   less than ï¿½0.01 monthsï¿½.

  betadb=# select '0.01 month'::interval as hundredth_of_month,
  betadb-#        '0.01 year'::interval  as hundredth_of_year;
   hundredth_of_month | hundredth_of_year
  --------------------+-------------------
   07:12:00           | 00:00:00

   This occurs because the current interval parsing rounds
   fractional years to the month, but fractional months
   to the fraction of a second.

   The current thinking on the thread in patches is
   at the very least to make these consistant, but with
   some open-issues because months arenï¿½t a fixed number
   of days, and days arenï¿½t a fixed number of seconds.

   The easiest and most minimal change would be to assume
   that any fractional part automatically gets turned
   into seconds, assuming things like 30 seconds/month,
   24 hrs/day.  Since all units except years work that way
   today, itï¿½d would have the least impact on existing code.

   A probably better way that Tom recommended would remember
   fractional months and fractional days.  This has the
   advantage that unlike today,
     ï¿½.5 monthsï¿½::interval + ï¿½.5 monthsï¿½::interval
   would then equal 1 month.

   So what should ï¿½.5 yearsï¿½ be?

   Today, itï¿½s ï¿½6 monsï¿½.  But I could just as easily
   argue that it should be 365.2425/2 days, or 4382.91
   seconds.  Each of these will be different (the last
   two are different durring daylight savings).

3. This all is based on the final draft standard of
   ISO 8601, but I havenï¿½t seen the actual expensive
   standard.  If anyone has it handy...

   Also, Iï¿½m curious to know what if anything the SQL
   spec says about intervals and units.  Any pointers.

  Ron

Any other interval annoyances I should hit at the same time?


Mon, 8 Sep 2003 16:05:38 -0700
"Ron Mayer" <ron@intervideo.com>
Re: [PATCHES] ISO 8601 "Time Intervals" of the "format with time-unit
	designators"

Tom wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > Tom Lane writes:
> >> Yes, but by the same token "iso8601" isn't specific enough either.

ISO 8601 gives more specific names.

  ISO 8601 Basic Format:         P2Y10M15DT10H20M30S
  ISO 8601 Alternative Format:   P00021015T102030
  ISO 8601 Extended Format:      P0002-10-15T10:20:30

In a way, the Extended Format is kinda nice, since itï¿½s
almost human readable.

I could put in both the basic and extended ones, and
call the dateformats ï¿½iso8601basicï¿½ and ï¿½iso8601extendedï¿½.
The negative is that to do ï¿½iso8601basicï¿½ right, Iï¿½d also
have to tweak the ï¿½dateï¿½ and ï¿½timeï¿½ parts of the code too.


Mon, 8 Sep 2003 16:46:13 -0700
"Ron Mayer" <ron@intervideo.com>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
	deignators'
[ backtracking a little ]

"Ron Mayer" <ron@intervideo.com> writes:
> Tom wrote: 
>> I doubt anyone is using it, because it's completely undocumented.
>> If we're going to support the real ISO spec, I'd suggest ripping
>> out any not-quite-there variant.

> I'm happy to look into it.  Rip out completely?  Ifdef? 

"Rip" was what I had in mind --- the idea is to simplify the code,
which you will surely agree is too complicated as it stands.  ifdefs
won't simplify the code or make it more understandable, rather the
reverse.

I have no problem with complex code when it's needed, but in this case
the ptype implementation of almost-ISO notation seems to me to affect
much more of the code than it has any right to on a usefulness basis.

			regards, tom lane

Tue, 09 Sep 2003 00:19:48 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
	deignators"

Where did we leave this?

---------------------------------------------------------------------------

Ron Mayer wrote:
> 
> Tom wrote:
> > Peter Eisentraut <peter_e@gmx.net> writes:
> > > Tom Lane writes:
> > >> Yes, but by the same token "iso8601" isn't specific enough either.
> 
> ISO 8601 gives more specific names.
> 
>   ISO 8601 Basic Format:         P2Y10M15DT10H20M30S
>   ISO 8601 Alternative Format:   P00021015T102030
>   ISO 8601 Extended Format:      P0002-10-15T10:20:30
> 
> In a way, the Extended Format is kinda nice, since it?s
> almost human readable.
> 
> I could put in both the basic and extended ones, and
> call the dateformats ?iso8601basic? and ?iso8601extended?.
> The negative is that to do ?iso8601basic? right, I?d also
> have to tweak the ?date? and ?time? parts of the code too.
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 26 Sep 2003 18:49:08 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Where did we leave this?

I thought it was proposed work for 7.5.

			regards, tom lane

Fri, 26 Sep 2003 18:54:09 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
	deignators'
So far,

  I have submitted the input-part.

  I have a working output-part (attached below, but I'm
  still cleaning up the documentation so I'll submit another
  one later).  The output is chosen by setting
  the datestyle to 'iso8601basic'.

  Those two changes don't break backward compatability
  but don't fix too much odd behavior except ISO time interval I/O.


  I was encouraged to look into changing the way timestamp
  math is done (keeping month, and day, and second separate
  until the end).  This is a bigger change and I don't have
  a stable version yet, and it breaks backward compatability. 
  I hope to submit a proposal for changes early enough in the
  7.5 timeframe to submit fixes then as well.  At the very least
  I will fully document the existing interval-math as part of
  this proposal so the docs can be updated even if the proposal
  gets rejected.

     Ron

> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sent: Friday, September 26, 2003 3:54 PM
> To: Bruce Momjian
> Cc: Ron Mayer; Peter Eisentraut; andrew@dunslane.net;
> pgsql-patches@postgresql.org
> Subject: Re: [PATCHES] ISO 8601 'Time Intervals' of the 'format with
> time-unit deignators' 
> 
> 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Where did we leave this?
> 
> I thought it was proposed work for 7.5.
> 
> 			regards, tom lane
> 
>
Fri, 26 Sep 2003 15:54:58 -0700
"Ron Mayer" <ron@intervideo.com>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
	deignators'
>   I have a working output-part (attached below, but I'm
>   still cleaning up the documentation so I'll submit another
>   one later)

Ugh.  Something in this pc quoted some characters in the attachment.
Rather than trying to apply it, wait a couple days and I'll submit
an update where the docs match.

  Ron



Fri, 26 Sep 2003 16:04:37 -0700
"Ron Mayer" <ron@intervideo.com>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
	deignators'

Oh... and I'm pretty sure noone expects this before 7.5. :-)
Looks like I'm  the only one with input files that have these 
wierd-but-iso8601 'P1Y6M' and 'PT30M' inputs; so I can just 
use my own patch. :-)


> 
> So far,
> 
>   I have submitted the input-part.
> 
>   I have a working output-part (attached below, but I'm
>   still cleaning up the documentation so I'll submit another
>   one later).  The output is chosen by setting
>   the datestyle to 'iso8601basic'.
> 
>   Those two changes don't break backward compatability
>   but don't fix too much odd behavior except ISO time interval I/O.
> 
> 
>   I was encouraged to look into changing the way timestamp
>   math is done (keeping month, and day, and second separate
>   until the end).  This is a bigger change and I don't have
>   a stable version yet, and it breaks backward compatability. 
>   I hope to submit a proposal for changes early enough in the
>   7.5 timeframe to submit fixes then as well.  At the very least
>   I will fully document the existing interval-math as part of
>   this proposal so the docs can be updated even if the proposal
>   gets rejected.
> 
>      Ron
> 
> > -----Original Message-----
> > From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> > Sent: Friday, September 26, 2003 3:54 PM
> > To: Bruce Momjian
> > Cc: Ron Mayer; Peter Eisentraut; andrew@dunslane.net;
> > pgsql-patches@postgresql.org
> > Subject: Re: [PATCHES] ISO 8601 'Time Intervals' of the 'format with
> > time-unit deignators' 
> > 
> > 
> > Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > > Where did we leave this?
> > 
> > I thought it was proposed work for 7.5.
> > 
> > 			regards, tom lane
> > 
> > 


Fri, 26 Sep 2003 16:04:37 -0700
"Ron Mayer" <ron@intervideo.com>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
	deignators'

Is this ready for application?  It looks good to me.  However, there is
an "Open issues" section.

---------------------------------------------------------------------------

Ron Mayer wrote:
> Short summary:
> 
>    This patch allows ISO 8601 "time intervals" using the "format 
>    with time-unit designators" to specify postgresql "intervals".
> 
>    Below I have (A) What these time intervals are, (B) What I
>    modified to support them, (C) Issues with intervals I want
>    to bring up, and (D) a patch supporting them.
> 
>    It's helpful to me.  Any feedback is appreciated.  If you 
>    did want to consider including it, let me know what to clean 
>    up.  If not, I thought I'd just put it here if anyone else finds
>    it useful too.
> 
>    Thanks for your time,
>   
>       Ron Mayer
> 
> Longer:
> 
> (A) What these intervals are.
> 
>    ISO 8601, the standard from which PostgreSQL gets some of it's 
>    time syntax, also has a specification for "time-intervals".
>   
>    In particular, section 5.5.4.2 has a "Representation of
>    time-interval by duration only" which I believe maps
>    nicely to ISO intervals.
> 
>    Compared to the ISO 8601 time interval specification, the
>    postgresql interval syntax is quite verbose.  For example:
> 
>      Postgresql interval:              ISO8601 Interval
>      ---------------------------------------------------
>      '1 year 6 months'                'P1Y6M'
>      '3 hours 25 minutes 42 seconds'  'PT3H25M42S'
> 
>    Yeah, it's uglier, but it sure is short which can make
>    for quicker typing and shorter scripts, and if for some
>    strange reason you had an application using this format
>    it's nice not to have to translate.
> 
>    The syntax is as follows:
>        Basic extended format:  PnYnMnDTnHnMnS
>                                PnW
> 
>        Where everything before the "T" is a date-part and everything
>        after is a time-part.  W is for weeks.
>        In the date-part, Y=Year, M=Month,  D=Day
>        In the time-part, H=Hour, M=Minute, S=Second
> 
>    Much more info can be found from the draft standard
>    ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
>    The final standard's only available for $$$ so I didn't
>    look at it.  Some other sites imply that this part didn't
>    change from the last draft to the standard.
> 
> 
> (B) This change was made by adding two functions to "datetime.c"
>     next to where DecodeInterval parses the normal interval syntax.
> 
>     A total of 313 lines were added, including comments and sgml docs.
>     Of these only 136 are actual code, the rest, comments, whitespace, etc.
> 
> 
>     One new function "DecodeISO8601Interval" follows the style of
>     "DecodeInterval" below it, and trys to strictly follow the ISO
>     syntax.  If it doesn't match, it'll return -1 and the old syntax
>     will be checked as before.
> 
>     The first test (first character of the first field must be 'P', 
>     and second character must be 'T' or '\0') should be fast so I don't
>     think this will impact performance of existing code.
> 
> 
>     The second function ("adjust_fval") is just a small helper-function
>     to remove some of the cut&paste style that DecodeInterval used.
> 
>     It seems to work.
>     =======================================================================
>     betadb=# select 'P1M15DT12H30M7S'::interval;
>             interval        
>     ------------------------
>      1 mon 15 days 12:30:07
>     (1 row)
> 
>     betadb=# select '1 month 15 days 12 hours 30 minutes 7 seconds'::interval;
> 	    interval        
>     ------------------------
>     1 mon 15 days 12:30:07
>     (1 row)
>     =====================================================================
> 
> 
> 
> (C) Open issues with intervals, and questions I'd like to ask.
> 
>     1.  DecodeInterval seems to have a hardcoded '.' for specifying
>         fractional times.  ISO 8601 states that both '.' and ',' are
>         ok, but "of these, the comma is the preferred sign".
> 
>         In DecodeISO8601Interval I loosened the test to allow
>         both but left it as it was in DecodeInterval.  Should
>         both be changed to make them more consistant?
> 
>     2.  In "DecodeInterval", fractional weeks and fractional months
>         can produce seconds; but fractional years can not (rounded
>         to months).  I didn't understand the reasoning for this, so
>         I left it the same, and followed the same convention for
>         ISO intervals.  Should I change this?
> 
>     3.  I could save a bunch of copy-paste-lines-of-code from the
>         pre-existing DecodeInterval by calling the adjust_fval helper
>         function.  The tradeoff is a few extra function-calls when
>         decoding an interval.  However I didn't want to risk changes
>         to the existing part unless you guys encourage me to do so.
> 
> 
> (D) The patch.
> 
> 
> Index: doc/src/sgml/datatype.sgml
> ===================================================================
> RCS file: /projects/cvsroot/pgsql-server/doc/src/sgml/datatype.sgml,v
> retrieving revision 1.123
> diff -u -1 -0 -r1.123 datatype.sgml
> --- doc/src/sgml/datatype.sgml	31 Aug 2003 17:32:18 -0000	1.123
> +++ doc/src/sgml/datatype.sgml	8 Sep 2003 04:04:58 -0000
> @@ -1735,20 +1735,71 @@
>        Quantities of days, hours, minutes, and seconds can be specified without
>        explicit unit markings.  For example, <literal>'1 12:59:10'</> is read
>        the same as <literal>'1 day 12 hours 59 min 10 sec'</>.
>       </para>
>  
>       <para>
>        The optional precision
>        <replaceable>p</replaceable> should be between 0 and 6, and
>        defaults to the precision of the input literal.
>       </para>
> +
> +
> +     <para>
> +      Alternatively, <type>interval</type> values can be written as 
> +      ISO 8601 time intervals, using the "Format with time-unit designators".
> +      This format always starts with the character <literal>'P'</>, followed 
> +      by a string of values followed by single character time-unit designators.
> +      A <literal>'T'</> separates the date and time parts of the interval.
> +     </para>
> +
> +     <para>
> +       Format:  PnYnMnDTnHnMnS
> +     </para>
> +     <para>
> +       In this format, <literal>'n'</> gets replaced by a number, and 
> +       <literal>Y</> represents years, 
> +       <literal>M</> (in the date part) months,
> +       <literal>D</> months,
> +       <literal>H</> hours,
> +       <literal>M</> (in the time part) minutes,
> +       and <literal>S</> seconds.
> +     </para>
> +      
> +
> +     <table id="interval-example-table">
> +	   <title>Interval Example</title>
> +	   <tgroup cols="2">
> +		<thead>
> +		 <row>
> +		  <entry>Traditional</entry>
> +		  <entry>ISO-8601 time-interval</entry>
> +		 </row>
> +		</thead>
> +		<tbody>
> +		 <row>
> +		  <entry>1 month</entry>
> +		  <entry>P1M</entry>
> +		 </row>
> +		 <row>
> +		  <entry>1 hour 30 minutes</entry>
> +		  <entry>PT1H30M</entry>
> +		 </row>
> +		 <row>
> +		  <entry>2 years 10 months 15 days 10 hours 30 minutes 20 seconds</entry>
> +		  <entry>P2Y10M15DT10H30M20S</entry>
> +		 </row>
> +		</tbody>
> +	   </thead>
> +	  </table>
> +	  
> +     </para>
>      </sect3>
>  
>      <sect3>
>       <title>Special Values</title>
>  
>       <indexterm>
>        <primary>time</primary>
>        <secondary>constants</secondary>
>       </indexterm>
>  
> Index: src/backend/utils/adt/datetime.c
> ===================================================================
> RCS file: /projects/cvsroot/pgsql-server/src/backend/utils/adt/datetime.c,v
> retrieving revision 1.116
> diff -u -1 -0 -r1.116 datetime.c
> --- src/backend/utils/adt/datetime.c	27 Aug 2003 23:29:28 -0000	1.116
> +++ src/backend/utils/adt/datetime.c	8 Sep 2003 04:04:59 -0000
> @@ -30,20 +30,21 @@
>  			 struct tm * tm, fsec_t *fsec, int *is2digits);
>  static int DecodeNumberField(int len, char *str,
>  				  int fmask, int *tmask,
>  				  struct tm * tm, fsec_t *fsec, int *is2digits);
>  static int DecodeTime(char *str, int fmask, int *tmask,
>  		   struct tm * tm, fsec_t *fsec);
>  static int	DecodeTimezone(char *str, int *tzp);
>  static datetkn *datebsearch(char *key, datetkn *base, unsigned int nel);
>  static int	DecodeDate(char *str, int fmask, int *tmask, struct tm * tm);
>  static void TrimTrailingZeros(char *str);
> +static int  DecodeISO8601Interval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec);
>  
>  
>  int			day_tab[2][13] = {
>  	{31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31, 0},
>  {31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31, 0}};
>  
>  char	   *months[] = {"Jan", "Feb", "Mar", "Apr", "May", "Jun",
>  "Jul", "Aug", "Sep", "Oct", "Nov", "Dec", NULL};
>  
>  char	   *days[] = {"Sunday", "Monday", "Tuesday", "Wednesday",
> @@ -2872,30 +2873,271 @@
>  			default:
>  				*val = tp->value;
>  				break;
>  		}
>  	}
>  
>  	return type;
>  }
>  
>  
> +void adjust_fval(double fval,struct tm * tm, fsec_t *fsec, int scale);
> +{
> +	int	sec;
> +	fval	   *= scale;
> +	sec		    = fval;
> +	tm->tm_sec += sec;
> +#ifdef HAVE_INT64_TIMESTAMP
> +	*fsec	   += ((fval - sec) * 1000000);
> +#else
> +	*fsec	   += (fval - sec);
> +#endif
> +}
> +
> +
> +/* DecodeISO8601Interval()
> + *
> + *  Check if it's a ISO 8601 Section 5.5.4.2 "Representation of
> + *  time-interval by duration only." 
> + *  Basic extended format:  PnYnMnDTnHnMnS
> + *                          PnW
> + *  For more info.
> + *  http://www.astroclark.freeserve.co.uk/iso8601/index.html
> + *  ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
> + *
> + *  Examples:  P1D  for 1 day
> + *             PT1H for 1 hour
> + *             P2Y6M7DT1H30M for 2 years, 6 months, 7 days 1 hour 30 min
> + *
> + *  The first field is exactly "p" or "pt" it may be of this type.
> + *
> + *  Returns -1 if the field is not of this type.
> + *
> + *  It pretty strictly checks the spec, with the two exceptions
> + *  that a week field ('W') may coexist with other units, and that
> + *  this function allows decimals in fields other than the least
> + *  significant units.
> + */
> +int
> +DecodeISO8601Interval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec) 
> +{
> +	char	   *cp;
> +	int			fmask = 0,
> +				tmask;
> +	int			val;
> +	double		fval;
> +	int			arg;
> +	int			datepart;
> +
> +    /*
> +	 * An ISO 8601 "time-interval by duration only" must start
> +	 * with a 'P'.  If it contains a date-part, 'p' will be the
> +	 * only character in the field.  If it contains no date part
> +	 * it will contain exactly to characters 'PT' indicating a
> +	 * time part.
> +	 * Anything else is illegal and will be treated like a 
> +	 * traditional postgresql interval.
> +	 */
> +    if (!(field[0][0] == 'p' &&
> +          ((field[0][1] == 0) || (field[0][1] == 't' && field[0][2] == 0))))
> +	{
> +	  return -1;
> +	}
> +
> +
> +    /*
> +	 * If the first field is exactly 1 character ('P'), it starts
> +	 * with date elements.  Otherwise it's two characters ('PT');
> +	 * indicating it starts with a time part.
> +	 */
> +	datepart = (field[0][1] == 0);
> +
> +	/*
> +	 * Every value must have a unit, so we require an even
> +	 * number of value/unit pairs. Therefore we require an
> +	 * odd nubmer of fields, including the prefix 'P'.
> +	 */
> +	if ((nf & 1) == 0)
> +		return -1;
> +
> +	/*
> +	 * Process pairs of fields at a time.
> +	 */
> +	for (arg = 1 ; arg < nf ; arg+=2) 
> +	{
> +		char * value = field[arg  ];
> +		char * units = field[arg+1];
> +
> +		/*
> +		 * The value part must be a number.
> +		 */
> +		if (ftype[arg] != DTK_NUMBER) 
> +			return -1;
> +
> +		/*
> +		 * extract the number, almost exactly like the non-ISO interval.
> +		 */
> +		val = strtol(value, &cp, 10);
> +
> +		/*
> +		 * One difference from the normal postgresql interval below...
> +		 * ISO 8601 states that "Of these, the comma is the preferred 
> +		 * sign" so I allow it here for locales that support it.
> +		 * Note: Perhaps the old-style interval code below should
> +		 * allow for this too, but I didn't want to risk backward
> +		 * compatability.
> +		 */
> +		if (*cp == '.' || *cp == ',') 
> +		{
> +			fval = strtod(cp, &cp);
> +			if (*cp != '\0')
> +				return -1;
> +
> +			if (val < 0)
> +				fval = -(fval);
> +		}
> +		else if (*cp == '\0')
> +			fval = 0;
> +		else
> +			return -1;
> +
> +
> +		if (datepart)
> +		{
> +			/*
> +			 * All the 8601 unit specifiers are 1 character, but may
> +			 * be followed by a 'T' character if transitioning between
> +			 * the date part and the time part.  If it's not either
> +			 * one character or two characters with the second being 't'
> +			 * it's an error.
> +			 */
> +			if (!(units[1] == 0 || (units[1] == 't' && units[2] == 0)))
> +				return -1;
> +
> +			if (units[1] == 't')
> +				datepart = 0;
> +
> +			switch (units[0]) /* Y M D W */
> +			{
> +				case 'd':
> +					tm->tm_mday += val;
> +					if (fval != 0)
> +					  adjust_fval(fval,tm,fsec, 86400);
> +					tmask = ((fmask & DTK_M(DAY)) ? 0 : DTK_M(DAY));
> +					break;
> +
> +				case 'w':
> +					tm->tm_mday += val * 7;
> +					if (fval != 0)
> +					  adjust_fval(fval,tm,fsec,7 * 86400);
> +					tmask = ((fmask & DTK_M(DAY)) ? 0 : DTK_M(DAY));
> +					break;
> +
> +				case 'm':
> +					tm->tm_mon += val;
> +					if (fval != 0)
> +					  adjust_fval(fval,tm,fsec,30 * 86400);
> +					tmask = DTK_M(MONTH);
> +					break;
> +
> +				case 'y':
> +					/*
> +					 * Why can fractional months produce seconds,
> +					 * but fractional years can't?  Well the older
> +					 * interval code below has the same property
> +					 * so this one follows the other one too.
> +					 */
> +					tm->tm_year += val;
> +					if (fval != 0)
> +						tm->tm_mon += (fval * 12);
> +					tmask = ((fmask & DTK_M(YEAR)) ? 0 : DTK_M(YEAR));
> +					break;
> +
> +				default:
> +					return -1;  /* invald date unit prefix */
> +			}
> +		}
> +		else
> +		{
> +			/*
> +			 * ISO 8601 time part.
> +			 * In the time part, only one-character
> +			 * unit prefixes are allowed.  If it's more
> +			 * than one character, it's not a valid ISO 8601
> +			 * time interval by duration.
> +			 */
> +			if (units[1] != 0)
> +				return -1;
> +
> +			switch (units[0]) /* H M S */
> +			{
> +				case 's':
> +					tm->tm_sec += val;
> +#ifdef HAVE_INT64_TIMESTAMP
> +					*fsec += (fval * 1000000);
> +#else
> +					*fsec += fval;
> +#endif
> +					tmask = DTK_M(SECOND);
> +					break;
> +
> +				case 'm':
> +					tm->tm_min += val;
> +					if (fval != 0)
> +					  adjust_fval(fval,tm,fsec,60);
> +					tmask = DTK_M(MINUTE);
> +					break;
> +
> +				case 'h':
> +					tm->tm_hour += val;
> +					if (fval != 0)
> +					  adjust_fval(fval,tm,fsec,3600);
> +					tmask = DTK_M(HOUR);
> +					break;
> +
> +				default:
> +					return -1; /* invald time unit prefix */
> +			}
> +		}
> +		fmask |= tmask;
> +	}
> +
> +	if (*fsec != 0)
> +	{
> +		int			sec;
> +
> +#ifdef HAVE_INT64_TIMESTAMP
> +		sec = (*fsec / INT64CONST(1000000));
> +		*fsec -= (sec * INT64CONST(1000000));
> +#else
> +		TMODULO(*fsec, sec, 1e0);
> +#endif
> +		tm->tm_sec += sec;
> +	}
> +	return (fmask != 0) ? 0 : -1;
> +}
> +
> +
>  /* DecodeInterval()
>   * Interpret previously parsed fields for general time interval.
>   * Returns 0 if successful, DTERR code if bogus input detected.
>   *
>   * Allow "date" field DTK_DATE since this could be just
>   *	an unsigned floating point number. - thomas 1997-11-16
>   *
>   * Allow ISO-style time span, with implicit units on number of days
>   *	preceding an hh:mm:ss field. - thomas 1998-04-30
> + * 
> + * Allow ISO-8601 style "Representation of time-interval by duration only"
> + *  of the format 'PnYnMnDTnHnMnS' and 'PnW' - ron 2003-08-30
>   */
> +
>  int
>  DecodeInterval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec)
>  {
>  	int			is_before = FALSE;
>  	char	   *cp;
>  	int			fmask = 0,
>  				tmask,
>  				type;
>  	int			i;
>  	int			dterr;
> @@ -2906,20 +3148,37 @@
>  
>  	type = IGNORE_DTF;
>  	tm->tm_year = 0;
>  	tm->tm_mon = 0;
>  	tm->tm_mday = 0;
>  	tm->tm_hour = 0;
>  	tm->tm_min = 0;
>  	tm->tm_sec = 0;
>  	*fsec = 0;
>  
> +	/*
> +	 *  Check if it's a ISO 8601 Section 5.5.4.2 "Representation of
> +     *  time-interval by duration only." 
> +	 *  Basic extended format:  PnYnMnDTnHnMnS
> +	 *                          PnW
> +	 *  http://www.astroclark.freeserve.co.uk/iso8601/index.html
> +	 *  ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
> +	 *  Examples:  P1D  for 1 day
> +	 *             PT1H for 1 hour
> +	 *             P2Y6M7DT1H30M for 2 years, 6 months, 7 days 1 hour 30 min
> +	 *
> +	 *  The first field is exactly "p" or "pt" it may be of this type.
> +	 */
> +	if (DecodeISO8601Interval(field,ftype,nf,dtype,tm,fsec) == 0) {
> +	    return 0;
> +    }
> +
>  	/* read through list backwards to pick up units before values */
>  	for (i = nf - 1; i >= 0; i--)
>  	{
>  		switch (ftype[i])
>  		{
>  			case DTK_TIME:
>  				dterr = DecodeTime(field[i], fmask, &tmask, tm, fsec);
>  				if (dterr)
>  					return dterr;
>  				type = DTK_DAY;
> @@ -2983,20 +3242,21 @@
>  				}
>  				/* DROP THROUGH */
>  
>  			case DTK_DATE:
>  			case DTK_NUMBER:
>  				val = strtol(field[i], &cp, 10);
>  
>  				if (type == IGNORE_DTF)
>  					type = DTK_SECOND;
>  
> +				/* should this allow ',' for locales that use it ? */
>  				if (*cp == '.')
>  				{
>  					fval = strtod(cp, &cp);
>  					if (*cp != '\0')
>  						return DTERR_BAD_FORMAT;
>  
>  					if (val < 0)
>  						fval = -(fval);
>  				}
>  				else if (*cp == '\0')
> 
> ===================================================================
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
>                http://archives.postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sun, 30 Nov 2003 23:52:10 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit

Here is an email on the open issues.

---------------------------------------------------------------------------

Ron Mayer wrote:
> Tom wrote...
> > At this point it should move to pghackers, I think.
> 
> Background for pghackers first, open issues below...
> 
>    Over on pgpatches we've been discussing ISO syntax for
>    ?time intervals? of the ?format with time-unit designators?.
>     http://archives.postgresql.org/pgsql-patches/2003-09/msg00103.php
>    A short summary is that I?ve submitted a patch that
>    accepts intervals of this format..
>      Postgresql interval:              ISO8601 Interval
>      ---------------------------------------------------
>      '1 year 6 months'                'P1Y6M'
>      '3 hours 25 minutes 42 seconds'  'PT3H25M42S'
>    The final draft is here
>      ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
> 
>    This patch was backward-compatable, but further improvements
>    discussed on patches may break compatability so I wanted to
>    discuss them here before implementing them.   I?ll also
>    be submitting a new datestyle ?iso8601? to output these intervals.
> 
> Open issues:
> 
> 1. Postgresql supported a shorthand for intervals that had
>    a similar, but not compatable syntax:
>      Interval            ISO             Existing postgres
>                          8601            shorthand
>      -----------------------------------------------------
>      '1 year 1 minute'   'P1YT1M'         '1Y1M'
>      '1 year 1 month'    'P1Y1M'          N/A
> 
>    The current thinking of the thread in pgpatches is to remove
>    the existing (undocumented) syntax.
> 
>    Removing this will break backward compatability if anyone
>    used this feature.  Let me know if you needed it.
> 
> 2. Some of the parsing for intervals is inconsistant and
>    confusing.  For example, note that ?0.01 years? is
>    less than ?0.01 months?.
> 
>   betadb=# select '0.01 month'::interval as hundredth_of_month,
>   betadb-#        '0.01 year'::interval  as hundredth_of_year;
>    hundredth_of_month | hundredth_of_year
>   --------------------+-------------------
>    07:12:00           | 00:00:00
> 
>    This occurs because the current interval parsing rounds
>    fractional years to the month, but fractional months
>    to the fraction of a second.
> 
>    The current thinking on the thread in patches is
>    at the very least to make these consistant, but with
>    some open-issues because months aren?t a fixed number
>    of days, and days aren?t a fixed number of seconds.
> 
>    The easiest and most minimal change would be to assume
>    that any fractional part automatically gets turned
>    into seconds, assuming things like 30 seconds/month,
>    24 hrs/day.  Since all units except years work that way
>    today, it?d would have the least impact on existing code.
> 
>    A probably better way that Tom recommended would remember
>    fractional months and fractional days.  This has the
>    advantage that unlike today,
>      ?.5 months?::interval + ?.5 months?::interval
>    would then equal 1 month.
> 
>    So what should ?.5 years? be?
> 
>    Today, it?s ?6 mons?.  But I could just as easily
>    argue that it should be 365.2425/2 days, or 4382.91
>    seconds.  Each of these will be different (the last
>    two are different durring daylight savings).
> 
> 3. This all is based on the final draft standard of
>    ISO 8601, but I haven?t seen the actual expensive
>    standard.  If anyone has it handy...
> 
>    Also, I?m curious to know what if anything the SQL
>    spec says about intervals and units.  Any pointers.
> 
>   Ron
> 
> Any other interval annoyances I should hit at the same time?
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
>       subscribe-nomail command to majordomo@postgresql.org so that your
>       message can get through to the mailing list cleanly
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sun, 30 Nov 2003 23:53:13 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [PATCHES] ISO 8601 "Time Intervals" of the "format with

And another open issues email.

---------------------------------------------------------------------------

Ron Mayer wrote:
> 
> Tom wrote:
> > Peter Eisentraut <peter_e@gmx.net> writes:
> > > Tom Lane writes:
> > >> Yes, but by the same token "iso8601" isn't specific enough either.
> 
> ISO 8601 gives more specific names.
> 
>   ISO 8601 Basic Format:         P2Y10M15DT10H20M30S
>   ISO 8601 Alternative Format:   P00021015T102030
>   ISO 8601 Extended Format:      P0002-10-15T10:20:30
> 
> In a way, the Extended Format is kinda nice, since it?s
> almost human readable.
> 
> I could put in both the basic and extended ones, and
> call the dateformats ?iso8601basic? and ?iso8601extended?.
> The negative is that to do ?iso8601basic? right, I?d also
> have to tweak the ?date? and ?time? parts of the code too.
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sun, 30 Nov 2003 23:53:22 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
Bruce Momjian writes:

> Is this ready for application?  It looks good to me.  However, there is
> an "Open issues" section.

It would be more useful to implement the SQL standard for intervals first
instead of inventing more nonstandard formats for it.

>
> ---------------------------------------------------------------------------
>
> Ron Mayer wrote:
> > Short summary:
> >
> >    This patch allows ISO 8601 "time intervals" using the "format
> >    with time-unit designators" to specify postgresql "intervals".
> >
> >    Below I have (A) What these time intervals are, (B) What I
> >    modified to support them, (C) Issues with intervals I want
> >    to bring up, and (D) a patch supporting them.
> >
> >    It's helpful to me.  Any feedback is appreciated.  If you
> >    did want to consider including it, let me know what to clean
> >    up.  If not, I thought I'd just put it here if anyone else finds
> >    it useful too.
> >
> >    Thanks for your time,
> >
> >       Ron Mayer
> >
> > Longer:
> >
> > (A) What these intervals are.
> >
> >    ISO 8601, the standard from which PostgreSQL gets some of it's
> >    time syntax, also has a specification for "time-intervals".
> >
> >    In particular, section 5.5.4.2 has a "Representation of
> >    time-interval by duration only" which I believe maps
> >    nicely to ISO intervals.
> >
> >    Compared to the ISO 8601 time interval specification, the
> >    postgresql interval syntax is quite verbose.  For example:
> >
> >      Postgresql interval:              ISO8601 Interval
> >      ---------------------------------------------------
> >      '1 year 6 months'                'P1Y6M'
> >      '3 hours 25 minutes 42 seconds'  'PT3H25M42S'
> >
> >    Yeah, it's uglier, but it sure is short which can make
> >    for quicker typing and shorter scripts, and if for some
> >    strange reason you had an application using this format
> >    it's nice not to have to translate.
> >
> >    The syntax is as follows:
> >        Basic extended format:  PnYnMnDTnHnMnS
> >                                PnW
> >
> >        Where everything before the "T" is a date-part and everything
> >        after is a time-part.  W is for weeks.
> >        In the date-part, Y=Year, M=Month,  D=Day
> >        In the time-part, H=Hour, M=Minute, S=Second
> >
> >    Much more info can be found from the draft standard
> >    ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
> >    The final standard's only available for $$$ so I didn't
> >    look at it.  Some other sites imply that this part didn't
> >    change from the last draft to the standard.
> >
> >
> > (B) This change was made by adding two functions to "datetime.c"
> >     next to where DecodeInterval parses the normal interval syntax.
> >
> >     A total of 313 lines were added, including comments and sgml docs.
> >     Of these only 136 are actual code, the rest, comments, whitespace, etc.
> >
> >
> >     One new function "DecodeISO8601Interval" follows the style of
> >     "DecodeInterval" below it, and trys to strictly follow the ISO
> >     syntax.  If it doesn't match, it'll return -1 and the old syntax
> >     will be checked as before.
> >
> >     The first test (first character of the first field must be 'P',
> >     and second character must be 'T' or '\0') should be fast so I don't
> >     think this will impact performance of existing code.
> >
> >
> >     The second function ("adjust_fval") is just a small helper-function
> >     to remove some of the cut&paste style that DecodeInterval used.
> >
> >     It seems to work.
> >     =======================================================================
> >     betadb=# select 'P1M15DT12H30M7S'::interval;
> >             interval
> >     ------------------------
> >      1 mon 15 days 12:30:07
> >     (1 row)
> >
> >     betadb=# select '1 month 15 days 12 hours 30 minutes 7 seconds'::interval;
> > 	    interval
> >     ------------------------
> >     1 mon 15 days 12:30:07
> >     (1 row)
> >     =====================================================================
> >
> >
> >
> > (C) Open issues with intervals, and questions I'd like to ask.
> >
> >     1.  DecodeInterval seems to have a hardcoded '.' for specifying
> >         fractional times.  ISO 8601 states that both '.' and ',' are
> >         ok, but "of these, the comma is the preferred sign".
> >
> >         In DecodeISO8601Interval I loosened the test to allow
> >         both but left it as it was in DecodeInterval.  Should
> >         both be changed to make them more consistant?
> >
> >     2.  In "DecodeInterval", fractional weeks and fractional months
> >         can produce seconds; but fractional years can not (rounded
> >         to months).  I didn't understand the reasoning for this, so
> >         I left it the same, and followed the same convention for
> >         ISO intervals.  Should I change this?
> >
> >     3.  I could save a bunch of copy-paste-lines-of-code from the
> >         pre-existing DecodeInterval by calling the adjust_fval helper
> >         function.  The tradeoff is a few extra function-calls when
> >         decoding an interval.  However I didn't want to risk changes
> >         to the existing part unless you guys encourage me to do so.
> >
> >
> > (D) The patch.
> >
> >
> > Index: doc/src/sgml/datatype.sgml
> > ===================================================================
> > RCS file: /projects/cvsroot/pgsql-server/doc/src/sgml/datatype.sgml,v
> > retrieving revision 1.123
> > diff -u -1 -0 -r1.123 datatype.sgml
> > --- doc/src/sgml/datatype.sgml	31 Aug 2003 17:32:18 -0000	1.123
> > +++ doc/src/sgml/datatype.sgml	8 Sep 2003 04:04:58 -0000
> > @@ -1735,20 +1735,71 @@
> >        Quantities of days, hours, minutes, and seconds can be specified without
> >        explicit unit markings.  For example, <literal>'1 12:59:10'</> is read
> >        the same as <literal>'1 day 12 hours 59 min 10 sec'</>.
> >       </para>
> >
> >       <para>
> >        The optional precision
> >        <replaceable>p</replaceable> should be between 0 and 6, and
> >        defaults to the precision of the input literal.
> >       </para>
> > +
> > +
> > +     <para>
> > +      Alternatively, <type>interval</type> values can be written as
> > +      ISO 8601 time intervals, using the "Format with time-unit designators".
> > +      This format always starts with the character <literal>'P'</>, followed
> > +      by a string of values followed by single character time-unit designators.
> > +      A <literal>'T'</> separates the date and time parts of the interval.
> > +     </para>
> > +
> > +     <para>
> > +       Format:  PnYnMnDTnHnMnS
> > +     </para>
> > +     <para>
> > +       In this format, <literal>'n'</> gets replaced by a number, and
> > +       <literal>Y</> represents years,
> > +       <literal>M</> (in the date part) months,
> > +       <literal>D</> months,
> > +       <literal>H</> hours,
> > +       <literal>M</> (in the time part) minutes,
> > +       and <literal>S</> seconds.
> > +     </para>
> > +
> > +
> > +     <table id="interval-example-table">
> > +	   <title>Interval Example</title>
> > +	   <tgroup cols="2">
> > +		<thead>
> > +		 <row>
> > +		  <entry>Traditional</entry>
> > +		  <entry>ISO-8601 time-interval</entry>
> > +		 </row>
> > +		</thead>
> > +		<tbody>
> > +		 <row>
> > +		  <entry>1 month</entry>
> > +		  <entry>P1M</entry>
> > +		 </row>
> > +		 <row>
> > +		  <entry>1 hour 30 minutes</entry>
> > +		  <entry>PT1H30M</entry>
> > +		 </row>
> > +		 <row>
> > +		  <entry>2 years 10 months 15 days 10 hours 30 minutes 20 seconds</entry>
> > +		  <entry>P2Y10M15DT10H30M20S</entry>
> > +		 </row>
> > +		</tbody>
> > +	   </thead>
> > +	  </table>
> > +
> > +     </para>
> >      </sect3>
> >
> >      <sect3>
> >       <title>Special Values</title>
> >
> >       <indexterm>
> >        <primary>time</primary>
> >        <secondary>constants</secondary>
> >       </indexterm>
> >
> > Index: src/backend/utils/adt/datetime.c
> > ===================================================================
> > RCS file: /projects/cvsroot/pgsql-server/src/backend/utils/adt/datetime.c,v
> > retrieving revision 1.116
> > diff -u -1 -0 -r1.116 datetime.c
> > --- src/backend/utils/adt/datetime.c	27 Aug 2003 23:29:28 -0000	1.116
> > +++ src/backend/utils/adt/datetime.c	8 Sep 2003 04:04:59 -0000
> > @@ -30,20 +30,21 @@
> >  			 struct tm * tm, fsec_t *fsec, int *is2digits);
> >  static int DecodeNumberField(int len, char *str,
> >  				  int fmask, int *tmask,
> >  				  struct tm * tm, fsec_t *fsec, int *is2digits);
> >  static int DecodeTime(char *str, int fmask, int *tmask,
> >  		   struct tm * tm, fsec_t *fsec);
> >  static int	DecodeTimezone(char *str, int *tzp);
> >  static datetkn *datebsearch(char *key, datetkn *base, unsigned int nel);
> >  static int	DecodeDate(char *str, int fmask, int *tmask, struct tm * tm);
> >  static void TrimTrailingZeros(char *str);
> > +static int  DecodeISO8601Interval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec);
> >
> >
> >  int			day_tab[2][13] = {
> >  	{31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31, 0},
> >  {31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31, 0}};
> >
> >  char	   *months[] = {"Jan", "Feb", "Mar", "Apr", "May", "Jun",
> >  "Jul", "Aug", "Sep", "Oct", "Nov", "Dec", NULL};
> >
> >  char	   *days[] = {"Sunday", "Monday", "Tuesday", "Wednesday",
> > @@ -2872,30 +2873,271 @@
> >  			default:
> >  				*val = tp->value;
> >  				break;
> >  		}
> >  	}
> >
> >  	return type;
> >  }
> >
> >
> > +void adjust_fval(double fval,struct tm * tm, fsec_t *fsec, int scale);
> > +{
> > +	int	sec;
> > +	fval	   *= scale;
> > +	sec		    = fval;
> > +	tm->tm_sec += sec;
> > +#ifdef HAVE_INT64_TIMESTAMP
> > +	*fsec	   += ((fval - sec) * 1000000);
> > +#else
> > +	*fsec	   += (fval - sec);
> > +#endif
> > +}
> > +
> > +
> > +/* DecodeISO8601Interval()
> > + *
> > + *  Check if it's a ISO 8601 Section 5.5.4.2 "Representation of
> > + *  time-interval by duration only."
> > + *  Basic extended format:  PnYnMnDTnHnMnS
> > + *                          PnW
> > + *  For more info.
> > + *  http://www.astroclark.freeserve.co.uk/iso8601/index.html
> > + *  ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
> > + *
> > + *  Examples:  P1D  for 1 day
> > + *             PT1H for 1 hour
> > + *             P2Y6M7DT1H30M for 2 years, 6 months, 7 days 1 hour 30 min
> > + *
> > + *  The first field is exactly "p" or "pt" it may be of this type.
> > + *
> > + *  Returns -1 if the field is not of this type.
> > + *
> > + *  It pretty strictly checks the spec, with the two exceptions
> > + *  that a week field ('W') may coexist with other units, and that
> > + *  this function allows decimals in fields other than the least
> > + *  significant units.
> > + */
> > +int
> > +DecodeISO8601Interval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec)
> > +{
> > +	char	   *cp;
> > +	int			fmask = 0,
> > +				tmask;
> > +	int			val;
> > +	double		fval;
> > +	int			arg;
> > +	int			datepart;
> > +
> > +    /*
> > +	 * An ISO 8601 "time-interval by duration only" must start
> > +	 * with a 'P'.  If it contains a date-part, 'p' will be the
> > +	 * only character in the field.  If it contains no date part
> > +	 * it will contain exactly to characters 'PT' indicating a
> > +	 * time part.
> > +	 * Anything else is illegal and will be treated like a
> > +	 * traditional postgresql interval.
> > +	 */
> > +    if (!(field[0][0] == 'p' &&
> > +          ((field[0][1] == 0) || (field[0][1] == 't' && field[0][2] == 0))))
> > +	{
> > +	  return -1;
> > +	}
> > +
> > +
> > +    /*
> > +	 * If the first field is exactly 1 character ('P'), it starts
> > +	 * with date elements.  Otherwise it's two characters ('PT');
> > +	 * indicating it starts with a time part.
> > +	 */
> > +	datepart = (field[0][1] == 0);
> > +
> > +	/*
> > +	 * Every value must have a unit, so we require an even
> > +	 * number of value/unit pairs. Therefore we require an
> > +	 * odd nubmer of fields, including the prefix 'P'.
> > +	 */
> > +	if ((nf & 1) == 0)
> > +		return -1;
> > +
> > +	/*
> > +	 * Process pairs of fields at a time.
> > +	 */
> > +	for (arg = 1 ; arg < nf ; arg+=2)
> > +	{
> > +		char * value = field[arg  ];
> > +		char * units = field[arg+1];
> > +
> > +		/*
> > +		 * The value part must be a number.
> > +		 */
> > +		if (ftype[arg] != DTK_NUMBER)
> > +			return -1;
> > +
> > +		/*
> > +		 * extract the number, almost exactly like the non-ISO interval.
> > +		 */
> > +		val = strtol(value, &cp, 10);
> > +
> > +		/*
> > +		 * One difference from the normal postgresql interval below...
> > +		 * ISO 8601 states that "Of these, the comma is the preferred
> > +		 * sign" so I allow it here for locales that support it.
> > +		 * Note: Perhaps the old-style interval code below should
> > +		 * allow for this too, but I didn't want to risk backward
> > +		 * compatability.
> > +		 */
> > +		if (*cp == '.' || *cp == ',')
> > +		{
> > +			fval = strtod(cp, &cp);
> > +			if (*cp != '\0')
> > +				return -1;
> > +
> > +			if (val < 0)
> > +				fval = -(fval);
> > +		}
> > +		else if (*cp == '\0')
> > +			fval = 0;
> > +		else
> > +			return -1;
> > +
> > +
> > +		if (datepart)
> > +		{
> > +			/*
> > +			 * All the 8601 unit specifiers are 1 character, but may
> > +			 * be followed by a 'T' character if transitioning between
> > +			 * the date part and the time part.  If it's not either
> > +			 * one character or two characters with the second being 't'
> > +			 * it's an error.
> > +			 */
> > +			if (!(units[1] == 0 || (units[1] == 't' && units[2] == 0)))
> > +				return -1;
> > +
> > +			if (units[1] == 't')
> > +				datepart = 0;
> > +
> > +			switch (units[0]) /* Y M D W */
> > +			{
> > +				case 'd':
> > +					tm->tm_mday += val;
> > +					if (fval != 0)
> > +					  adjust_fval(fval,tm,fsec, 86400);
> > +					tmask = ((fmask & DTK_M(DAY)) ? 0 : DTK_M(DAY));
> > +					break;
> > +
> > +				case 'w':
> > +					tm->tm_mday += val * 7;
> > +					if (fval != 0)
> > +					  adjust_fval(fval,tm,fsec,7 * 86400);
> > +					tmask = ((fmask & DTK_M(DAY)) ? 0 : DTK_M(DAY));
> > +					break;
> > +
> > +				case 'm':
> > +					tm->tm_mon += val;
> > +					if (fval != 0)
> > +					  adjust_fval(fval,tm,fsec,30 * 86400);
> > +					tmask = DTK_M(MONTH);
> > +					break;
> > +
> > +				case 'y':
> > +					/*
> > +					 * Why can fractional months produce seconds,
> > +					 * but fractional years can't?  Well the older
> > +					 * interval code below has the same property
> > +					 * so this one follows the other one too.
> > +					 */
> > +					tm->tm_year += val;
> > +					if (fval != 0)
> > +						tm->tm_mon += (fval * 12);
> > +					tmask = ((fmask & DTK_M(YEAR)) ? 0 : DTK_M(YEAR));
> > +					break;
> > +
> > +				default:
> > +					return -1;  /* invald date unit prefix */
> > +			}
> > +		}
> > +		else
> > +		{
> > +			/*
> > +			 * ISO 8601 time part.
> > +			 * In the time part, only one-character
> > +			 * unit prefixes are allowed.  If it's more
> > +			 * than one character, it's not a valid ISO 8601
> > +			 * time interval by duration.
> > +			 */
> > +			if (units[1] != 0)
> > +				return -1;
> > +
> > +			switch (units[0]) /* H M S */
> > +			{
> > +				case 's':
> > +					tm->tm_sec += val;
> > +#ifdef HAVE_INT64_TIMESTAMP
> > +					*fsec += (fval * 1000000);
> > +#else
> > +					*fsec += fval;
> > +#endif
> > +					tmask = DTK_M(SECOND);
> > +					break;
> > +
> > +				case 'm':
> > +					tm->tm_min += val;
> > +					if (fval != 0)
> > +					  adjust_fval(fval,tm,fsec,60);
> > +					tmask = DTK_M(MINUTE);
> > +					break;
> > +
> > +				case 'h':
> > +					tm->tm_hour += val;
> > +					if (fval != 0)
> > +					  adjust_fval(fval,tm,fsec,3600);
> > +					tmask = DTK_M(HOUR);
> > +					break;
> > +
> > +				default:
> > +					return -1; /* invald time unit prefix */
> > +			}
> > +		}
> > +		fmask |= tmask;
> > +	}
> > +
> > +	if (*fsec != 0)
> > +	{
> > +		int			sec;
> > +
> > +#ifdef HAVE_INT64_TIMESTAMP
> > +		sec = (*fsec / INT64CONST(1000000));
> > +		*fsec -= (sec * INT64CONST(1000000));
> > +#else
> > +		TMODULO(*fsec, sec, 1e0);
> > +#endif
> > +		tm->tm_sec += sec;
> > +	}
> > +	return (fmask != 0) ? 0 : -1;
> > +}
> > +
> > +
> >  /* DecodeInterval()
> >   * Interpret previously parsed fields for general time interval.
> >   * Returns 0 if successful, DTERR code if bogus input detected.
> >   *
> >   * Allow "date" field DTK_DATE since this could be just
> >   *	an unsigned floating point number. - thomas 1997-11-16
> >   *
> >   * Allow ISO-style time span, with implicit units on number of days
> >   *	preceding an hh:mm:ss field. - thomas 1998-04-30
> > + *
> > + * Allow ISO-8601 style "Representation of time-interval by duration only"
> > + *  of the format 'PnYnMnDTnHnMnS' and 'PnW' - ron 2003-08-30
> >   */
> > +
> >  int
> >  DecodeInterval(char **field, int *ftype, int nf, int *dtype, struct tm * tm, fsec_t *fsec)
> >  {
> >  	int			is_before = FALSE;
> >  	char	   *cp;
> >  	int			fmask = 0,
> >  				tmask,
> >  				type;
> >  	int			i;
> >  	int			dterr;
> > @@ -2906,20 +3148,37 @@
> >
> >  	type = IGNORE_DTF;
> >  	tm->tm_year = 0;
> >  	tm->tm_mon = 0;
> >  	tm->tm_mday = 0;
> >  	tm->tm_hour = 0;
> >  	tm->tm_min = 0;
> >  	tm->tm_sec = 0;
> >  	*fsec = 0;
> >
> > +	/*
> > +	 *  Check if it's a ISO 8601 Section 5.5.4.2 "Representation of
> > +     *  time-interval by duration only."
> > +	 *  Basic extended format:  PnYnMnDTnHnMnS
> > +	 *                          PnW
> > +	 *  http://www.astroclark.freeserve.co.uk/iso8601/index.html
> > +	 *  ftp://ftp.qsl.net/pub/g1smd/154N362_.PDF
> > +	 *  Examples:  P1D  for 1 day
> > +	 *             PT1H for 1 hour
> > +	 *             P2Y6M7DT1H30M for 2 years, 6 months, 7 days 1 hour 30 min
> > +	 *
> > +	 *  The first field is exactly "p" or "pt" it may be of this type.
> > +	 */
> > +	if (DecodeISO8601Interval(field,ftype,nf,dtype,tm,fsec) == 0) {
> > +	    return 0;
> > +    }
> > +
> >  	/* read through list backwards to pick up units before values */
> >  	for (i = nf - 1; i >= 0; i--)
> >  	{
> >  		switch (ftype[i])
> >  		{
> >  			case DTK_TIME:
> >  				dterr = DecodeTime(field[i], fmask, &tmask, tm, fsec);
> >  				if (dterr)
> >  					return dterr;
> >  				type = DTK_DAY;
> > @@ -2983,20 +3242,21 @@
> >  				}
> >  				/* DROP THROUGH */
> >
> >  			case DTK_DATE:
> >  			case DTK_NUMBER:
> >  				val = strtol(field[i], &cp, 10);
> >
> >  				if (type == IGNORE_DTF)
> >  					type = DTK_SECOND;
> >
> > +				/* should this allow ',' for locales that use it ? */
> >  				if (*cp == '.')
> >  				{
> >  					fval = strtod(cp, &cp);
> >  					if (*cp != '\0')
> >  						return DTERR_BAD_FORMAT;
> >
> >  					if (val < 0)
> >  						fval = -(fval);
> >  				}
> >  				else if (*cp == '\0')
> >
> > ===================================================================
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 6: Have you searched our list archives?
> >
> >                http://archives.postgresql.org
> >
>
>

-- 
Peter Eisentraut   peter_e@gmx.net


Mon, 1 Dec 2003 07:20:21 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > Is this ready for application?  It looks good to me.  However, there is
> > an "Open issues" section.
> 
> It would be more useful to implement the SQL standard for intervals first
> instead of inventing more nonstandard formats for it.

OK, patch removed from queue.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 1 Dec 2003 09:12:35 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
> -----Original Message-----
> 
> Is this ready for application?  It looks good to me.  However, there is
> an "Open issues" section.

In my mind there were two categories of open issues
  a) ones that are 100% backward (such as the comment about 
     outputting this format)
and
  b) ones that aren't (such as deprecating the current
     postgresql shorthand of 
         '1Y1M'::interval = 1 year 1 minute
     in favor of the ISO-8601
         'P1Y1M'::interval = 1 year 1 month.

Attached is a patch that addressed all the discussed issues that
did not break backward compatability, including the ability to
output ISO-8601 compliant intervals by setting datestyle to
iso8601basic.

   Ron
Mon, 1 Dec 2003 12:50:47 -0800
"Ron Mayer" <ron@intervideo.com>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
	deignators"
Peter wrote:
> 
> It would be more useful to implement the SQL standard for intervals first
> instead of inventing more nonstandard formats for it.

Much of the postgresql docs talks about ISO-8601 formats, so I would think
of the patch more as a standards-based improvemnt for the current interval 
shortand.  

For example, where today postgresql accepts "1Y1M" as '1 year 1 minute',
with the patch, the ISO-8601-standard "P1Y1M" would mean '1 year 1 month'.

I would be happy to implement the SQL-standard-intervals as well if 
someone can point me to that spec.  It's just that the system I worked
with happend to exchange data with ISO8601 time intervals.

   Ron Mayer

[Moderators of psql-patches... I first replied to Peter from a
separate (personal) account.  I think this reply is now in the 
moderation queue). In retrospect I thought it might be better if 
the archives had the whole thread from the original email address.  
If it's not too late, could you reject my post in the moderation queue?
If not, sorry for the spam.]



Mon, 1 Dec 2003 13:18:20 -0800
"Ron Mayer" <ron@intervideo.com>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
[sNip]
> ISO 8601 gives more specific names.
> 
>   ISO 8601 Basic Format:         P2Y10M15DT10H20M30S
>   ISO 8601 Alternative Format:   P00021015T102030
>   ISO 8601 Extended Format:      P0002-10-15T10:20:30
> 
> In a way, the Extended Format is kinda nice, since it�s
> almost human readable.
> 
> I could put in both the basic and extended ones, and
> call the dateformats �iso8601basic� and �iso8601extended�.
> The negative is that to do �iso8601basic� right, I�d also
> have to tweak the �date� and �time� parts of the code too.

    	Perhaps all three formats should be supported, and if the following 
names were all valid things could be simplified further too:

    	    	iso8601basic
    	    	iso8601bas
    	    	iso8601alternative
    	    	iso8601alt
    	    	iso8601extended
    	    	iso8601ext

    	The reason for allowing shorter names is to simplify database 
management for anyone who may need to store the format name in a column for 
some reason (I can't think of one now, but I get a feeling that someone 
will want to do this type of thing in the future).

    	For that matter, the first letter could be used instead of the first 
three for the short versions.  Any thoughts on this?

-- 
Randolf Richardson - rr@8x.ca
Vancouver, British Columbia, Canada

Please do not eMail me directly when responding
to my postings in the newsgroups.

Wed, 3 Dec 2003 10:20:23 +0000 (UTC)
Randolf Richardson <rr@8x.ca>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit
	deignators'
Randolf Richardson wrote:
> [sNip]
> > ISO 8601 gives more specific names.
> > 
> >   ISO 8601 Basic Format:         P2Y10M15DT10H20M30S
> >   ISO 8601 Alternative Format:   P00021015T102030
> >   ISO 8601 Extended Format:      P0002-10-15T10:20:30
> > 
> > In a way, the Extended Format is kinda nice, since it�s
> > almost human readable.
> > 
> > I could put in both the basic and extended ones, and
> > call the dateformats �iso8601basic� and �iso8601extended�.
> > The negative is that to do �iso8601basic� right, I�d also
> > have to tweak the �date� and �time� parts of the code too.
> 
>     	Perhaps all three formats should be supported, and if the following 
> names were all valid things could be simplified further too:
> 
>     	    	iso8601basic
>     	    	iso8601bas
>     	    	iso8601alternative
>     	    	iso8601alt
>     	    	iso8601extended
>     	    	iso8601ext
> 
>     	The reason for allowing shorter names is to simplify database 
> management for anyone who may need to store the format name in a column for 
> some reason (I can't think of one now, but I get a feeling that someone 
> will want to do this type of thing in the future).
> 
>     	For that matter, the first letter could be used instead of the first 
> three for the short versions.  Any thoughts on this?

Just go with the full spellings.


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 9 Dec 2003 18:28:20 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 'Time Intervals' of the 'format with time-unit

Your patch has been added to the PostgreSQL unapplied patches list at:

	http://momjian.postgresql.org/cgi-bin/pgpatches

I will try to apply it within the next 48 hours.

---------------------------------------------------------------------------


Ron Mayer wrote:
> > -----Original Message-----
> > 
> > Is this ready for application?  It looks good to me.  However, there is
> > an "Open issues" section.
> 
> In my mind there were two categories of open issues
>   a) ones that are 100% backward (such as the comment about 
>      outputting this format)
> and
>   b) ones that aren't (such as deprecating the current
>      postgresql shorthand of 
>          '1Y1M'::interval = 1 year 1 minute
>      in favor of the ISO-8601
>          'P1Y1M'::interval = 1 year 1 month.
> 
> Attached is a patch that addressed all the discussed issues that
> did not break backward compatability, including the ability to
> output ISO-8601 compliant intervals by setting datestyle to
> iso8601basic.
> 
>    Ron

[ Attachment, skipping... ]

> 
> ---------------------------(end of broadcast)---------------------------
> TIP 8: explain analyze is your friend

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 15 Dec 2003 18:30:09 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
Bruce Momjian wrote:
> Your patch has been added to the PostgreSQL unapplied patches list
> at:
>
> 	http://momjian.postgresql.org/cgi-bin/pgpatches
>
> I will try to apply it within the next 48 hours.

I keep reading about open issues, and deprecating certain things, and 
patch removed, and patch readded.  What is going on?


Tue, 16 Dec 2003 01:06:32 +0100
Peter Eisentraut <peter_e@gmx.net>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
Peter Eisentraut wrote:
> Bruce Momjian wrote:
> > Your patch has been added to the PostgreSQL unapplied patches list
> > at:
> >
> > 	http://momjian.postgresql.org/cgi-bin/pgpatches
> >
> > I will try to apply it within the next 48 hours.
> 
> I keep reading about open issues, and deprecating certain things, and 
> patch removed, and patch readded.  What is going on?

I think the patch just added is OK, no?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 15 Dec 2003 19:09:15 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
Bruce Momjian wrote:
> > I keep reading about open issues, and deprecating certain things,
> > and patch removed, and patch readded.  What is going on?
>
> I think the patch just added is OK, no?

I don't know, but earlier the identical patch was rejected by you.


Tue, 16 Dec 2003 01:37:50 +0100
Peter Eisentraut <peter_e@gmx.net>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
Peter Eisentraut wrote:
> Bruce Momjian wrote:
> > > I keep reading about open issues, and deprecating certain things,
> > > and patch removed, and patch readded.  What is going on?
> >
> > I think the patch just added is OK, no?
> 
> I don't know, but earlier the identical patch was rejected by you.

I thought he made an adjustment so no backward compatibility was broken.


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 15 Dec 2003 19:39:00 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
Bruce Momjian wrote:
> Peter Eisentraut wrote:
> > Bruce Momjian wrote:
> > > > I keep reading about open issues, and deprecating certain
> > > > things, and patch removed, and patch readded.  What is going
> > > > on?
> > >
> > > I think the patch just added is OK, no?
> >
> > I don't know, but earlier the identical patch was rejected by you.
>
> I thought he made an adjustment so no backward compatibility was
> broken.

Then I wouldn't have said "identical".


Tue, 16 Dec 2003 01:42:09 +0100
Peter Eisentraut <peter_e@gmx.net>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
Peter Eisentraut wrote:
> Bruce Momjian wrote:
> > Peter Eisentraut wrote:
> > > Bruce Momjian wrote:
> > > > > I keep reading about open issues, and deprecating certain
> > > > > things, and patch removed, and patch readded.  What is going
> > > > > on?
> > > >
> > > > I think the patch just added is OK, no?
> > >
> > > I don't know, but earlier the identical patch was rejected by you.
> >
> > I thought he made an adjustment so no backward compatibility was
> > broken.
> 
> Then I wouldn't have said "identical".

OK, can anyone raise an objection to the patch.  The new description
means to me that he addressed our concerns and that my original
hesitation was unwarranted.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 15 Dec 2003 19:44:11 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit

Patch applied.  Thanks.

---------------------------------------------------------------------------


Ron Mayer wrote:
> > -----Original Message-----
> > 
> > Is this ready for application?  It looks good to me.  However, there is
> > an "Open issues" section.
> 
> In my mind there were two categories of open issues
>   a) ones that are 100% backward (such as the comment about 
>      outputting this format)
> and
>   b) ones that aren't (such as deprecating the current
>      postgresql shorthand of 
>          '1Y1M'::interval = 1 year 1 minute
>      in favor of the ISO-8601
>          'P1Y1M'::interval = 1 year 1 month.
> 
> Attached is a patch that addressed all the discussed issues that
> did not break backward compatability, including the ability to
> output ISO-8601 compliant intervals by setting datestyle to
> iso8601basic.
> 
>    Ron

[ Attachment, skipping... ]

> 
> ---------------------------(end of broadcast)---------------------------
> TIP 8: explain analyze is your friend

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sat, 20 Dec 2003 10:32:45 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: ISO 8601 "Time Intervals" of the "format with time-unit
Hello,

Is there ANY chance to recover data from a database system that suffered disk 
crash, and is not missing the data/global directory?

Version is 7.2.4. Database files seem to be intact as well as pg_clog and 
pg_xlog directories.

Thanks in advance for any ideas.

Daniel


Mon, 23 Aug 2004 16:00:01 +0300
Daniel Kalchev <daniel@digsys.bg>
missing data/global
Daniel Kalchev <daniel@digsys.bg> writes:
> Is there ANY chance to recover data from a database system that suffered disk
> crash, and is not missing the data/global directory?
> Version is 7.2.4. Database files seem to be intact as well as pg_clog and 
> pg_xlog directories.

The hard part I think would be reconstructing pg_database, because you'd
need to get the database OIDs right.  I can't think of any way to do
that that doesn't involve poking at the file with a hex editor.

Here's a sketch of how I'd proceed:

1. Make a tar backup of what you have!  That way you can start over
after you screw up ;-)

2. I assume you know the names and properties of your databases, users,
and groups if any; also the SYSID numbers for the users and groups.
A recent pg_dumpall script would be a good place to get this info.

3. You're also going to need to figure out the OIDs of your databases
(the OIDs are the same as the names of their subdirectories under
$PGDATA/base).  Possibly you can do this just from directory/file sizes.
Note that template1 should be OID 1, and template0 will have the next
lowest number (probably 16555, in 7.2).

4. Initdb a scratch database in some other place (or move aside your
existing files, if that seems safer).  In this scratch DB, create
databases, users, and groups to match your old setup.  You should be
able to duplicate everything except the database OIDs using standard
SQL commands.

5. Shut down scratch postmaster, then hex-edit pg_database to insert the
correct OIDs.  Use pg_filedump or a similar tool to verify that you did
this properly.

6. Restart scratch postmaster, and VACUUM FREEZE pg_database, pg_shadow,
and pg_group (from any database).  This will make the next step safe.

7. Stop scratch postmaster, and then copy over its $PGDATA/global
directory into the old DB.

8. Cross your fingers and start postmaster ...

This will probably *not* work if you had been doing anything to
pg_database, pg_shadow, or pg_group between your last checkpoint and the
crash, because the reconstructed tables are not going to be physically
identical to what they were before, so any actions replayed from WAL
against those tables will be wrong.  Hopefully you won't have that
problem.  If you do, it might work to shut down the postmaster and again
copy the scratch $PGDATA/global directory into the old DB, thereby
overwriting what the WAL replay did.  This is getting into the realm of
speculation though.

			regards, tom lane

Mon, 23 Aug 2004 14:46:17 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: missing data/global 
Tom,

This is basically what I had in mind, but you described it better than I ever 
could :)

What I need from this database system is just one database and probably not 
all of the tables anyway (but some do seem valuable). This database happens to 
be second in the pg_dumpall script. The next databases are rather big (and I 
actually have more recent backup and could eventually recreate the data) The 
valuable database hasn't had significant structure changes since the backup.

Looking at the files, I am confident which is the proper database oid - if 
this cannot be properly fixed, is there .. reasonable way to dump data from 
the (heap) files?

Here is what I have:

su-2.02# du
1747    ./base/1
1693    ./base/16555
1       ./base/77573557/pgsql_tmp
127036  ./base/77573557
1       ./base/13255137/pgsql_tmp
1379190 ./base/13255137
11246   ./base/95521309
1781    ./base/96388007
1       ./base/133512058/pgsql_tmp
11933861        ./base/133512058
13456555        ./base
98209   ./pg_xlog
41315   ./pg_clog
13596100        .

My database should be with oid 77573557, template0 is apparently 16555

Let's see how all this works.

Daniel

>>>Tom Lane said:
 > Daniel Kalchev <daniel@digsys.bg> writes:
 > > Is there ANY chance to recover data from a database system that suffered d
     isk
 > > crash, and is not missing the data/global directory?
 > > Version is 7.2.4. Database files seem to be intact as well as pg_clog and 
 > > pg_xlog directories.
 > 
 > The hard part I think would be reconstructing pg_database, because you'd
 > need to get the database OIDs right.  I can't think of any way to do
 > that that doesn't involve poking at the file with a hex editor.
 > 
 > Here's a sketch of how I'd proceed:
 > 
 > 1. Make a tar backup of what you have!  That way you can start over
 > after you screw up ;-)
 > 
 > 2. I assume you know the names and properties of your databases, users,
 > and groups if any; also the SYSID numbers for the users and groups.
 > A recent pg_dumpall script would be a good place to get this info.
 > 
 > 3. You're also going to need to figure out the OIDs of your databases
 > (the OIDs are the same as the names of their subdirectories under
 > $PGDATA/base).  Possibly you can do this just from directory/file sizes.
 > Note that template1 should be OID 1, and template0 will have the next
 > lowest number (probably 16555, in 7.2).
 > 
 > 4. Initdb a scratch database in some other place (or move aside your
 > existing files, if that seems safer).  In this scratch DB, create
 > databases, users, and groups to match your old setup.  You should be
 > able to duplicate everything except the database OIDs using standard
 > SQL commands.
 > 
 > 5. Shut down scratch postmaster, then hex-edit pg_database to insert the
 > correct OIDs.  Use pg_filedump or a similar tool to verify that you did
 > this properly.
 > 
 > 6. Restart scratch postmaster, and VACUUM FREEZE pg_database, pg_shadow,
 > and pg_group (from any database).  This will make the next step safe.
 > 
 > 7. Stop scratch postmaster, and then copy over its $PGDATA/global
 > directory into the old DB.
 > 
 > 8. Cross your fingers and start postmaster ...
 > 
 > This will probably *not* work if you had been doing anything to
 > pg_database, pg_shadow, or pg_group between your last checkpoint and the
 > crash, because the reconstructed tables are not going to be physically
 > identical to what they were before, so any actions replayed from WAL
 > against those tables will be wrong.  Hopefully you won't have that
 > problem.  If you do, it might work to shut down the postmaster and again
 > copy the scratch $PGDATA/global directory into the old DB, thereby
 > overwriting what the WAL replay did.  This is getting into the realm of
 > speculation though.
 > 
 > 			regards, tom lane



Mon, 23 Aug 2004 23:02:49 +0300
Daniel Kalchev <daniel@digsys.bg>
Re: missing data/global 
If you're not missing your data dir, clog or xlog then what's the problem?

Daniel Kalchev wrote:
> Hello,
> 
> Is there ANY chance to recover data from a database system that suffered disk 
> crash, and is not missing the data/global directory?
> 
> Version is 7.2.4. Database files seem to be intact as well as pg_clog and 
> pg_xlog directories.
> 
> Thanks in advance for any ideas.
> 
> Daniel
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

Tue, 24 Aug 2004 09:11:26 +0800
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: missing data/global
data/base/global is missing and this is where postgres gets all it's startup 
data from (database oids, next oid, transaction id etc).

Let's see how easy to recover from this it will turn to be.

Daniel

>>>Christopher Kings-Lynne said:
 > If you're not missing your data dir, clog or xlog then what's the problem?
 > 
 > Daniel Kalchev wrote:
 > > Hello,
 > > 
 > > Is there ANY chance to recover data from a database system that suffered d
     isk 
 > > crash, and is not missing the data/global directory?
 > > 
 > > Version is 7.2.4. Database files seem to be intact as well as pg_clog and 
 > > pg_xlog directories.
 > > 
 > > Thanks in advance for any ideas.
 > > 
 > > Daniel
 > > 
 > > 
 > > ---------------------------(end of broadcast)---------------------------
 > > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org



Tue, 24 Aug 2004 11:30:20 +0300
Daniel Kalchev <daniel@digsys.bg>
Re: missing data/global 
Ah, you said 'is NOT missing'.

Chris

Daniel Kalchev wrote:

> data/base/global is missing and this is where postgres gets all it's startup 
> data from (database oids, next oid, transaction id etc).
> 
> Let's see how easy to recover from this it will turn to be.
> 
> Daniel
> 
> 
>>>>Christopher Kings-Lynne said:
> 
>  > If you're not missing your data dir, clog or xlog then what's the problem?
>  > 
>  > Daniel Kalchev wrote:
>  > > Hello,
>  > > 
>  > > Is there ANY chance to recover data from a database system that suffered d
>      isk 
>  > > crash, and is not missing the data/global directory?
>  > > 
>  > > Version is 7.2.4. Database files seem to be intact as well as pg_clog and 
>  > > pg_xlog directories.
>  > > 
>  > > Thanks in advance for any ideas.
>  > > 
>  > > Daniel
>  > > 
>  > > 
>  > > ---------------------------(end of broadcast)---------------------------
>  > > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 

Tue, 24 Aug 2004 17:07:22 +0800
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: missing data/global
Tom I did the following:

(found out 7.2.3 does not have pg_database)

1. saved old data etc.

2. created new database, and the database. database oid was 16556;

3. moved data/global to the old data directory.

4. though, that postmaster would actually use the database oid to locate the 
directory, then load everything from there.. old database oid was 77573557, so 
I just linked this to 16556 in the data/base direcotry. (this might be the 
first possible error)

Now I can connect to the 'old' database, but get the error 

FATAL 1:  Index pg_operator_oid_index is not a btree

(if I run postmaster with -P I get not errors, but no tables as well).

By the way, I had to copy over the 'new' files from pg_clog and pg_xlog (this 
is the second possible error) to get the postmaster running. Perhaps better 
would be to use pg_resetxlog or similar?

Daniel

>>>Tom Lane said:
 > Daniel Kalchev <daniel@digsys.bg> writes:
 > > Is there ANY chance to recover data from a database system that suffered d
     isk
 > > crash, and is not missing the data/global directory?
 > > Version is 7.2.4. Database files seem to be intact as well as pg_clog and 
 > > pg_xlog directories.
 > 
 > The hard part I think would be reconstructing pg_database, because you'd
 > need to get the database OIDs right.  I can't think of any way to do
 > that that doesn't involve poking at the file with a hex editor.
 > 
 > Here's a sketch of how I'd proceed:
 > 
 > 1. Make a tar backup of what you have!  That way you can start over
 > after you screw up ;-)
 > 
 > 2. I assume you know the names and properties of your databases, users,
 > and groups if any; also the SYSID numbers for the users and groups.
 > A recent pg_dumpall script would be a good place to get this info.
 > 
 > 3. You're also going to need to figure out the OIDs of your databases
 > (the OIDs are the same as the names of their subdirectories under
 > $PGDATA/base).  Possibly you can do this just from directory/file sizes.
 > Note that template1 should be OID 1, and template0 will have the next
 > lowest number (probably 16555, in 7.2).
 > 
 > 4. Initdb a scratch database in some other place (or move aside your
 > existing files, if that seems safer).  In this scratch DB, create
 > databases, users, and groups to match your old setup.  You should be
 > able to duplicate everything except the database OIDs using standard
 > SQL commands.
 > 
 > 5. Shut down scratch postmaster, then hex-edit pg_database to insert the
 > correct OIDs.  Use pg_filedump or a similar tool to verify that you did
 > this properly.
 > 
 > 6. Restart scratch postmaster, and VACUUM FREEZE pg_database, pg_shadow,
 > and pg_group (from any database).  This will make the next step safe.
 > 
 > 7. Stop scratch postmaster, and then copy over its $PGDATA/global
 > directory into the old DB.
 > 
 > 8. Cross your fingers and start postmaster ...
 > 
 > This will probably *not* work if you had been doing anything to
 > pg_database, pg_shadow, or pg_group between your last checkpoint and the
 > crash, because the reconstructed tables are not going to be physically
 > identical to what they were before, so any actions replayed from WAL
 > against those tables will be wrong.  Hopefully you won't have that
 > problem.  If you do, it might work to shut down the postmaster and again
 > copy the scratch $PGDATA/global directory into the old DB, thereby
 > overwriting what the WAL replay did.  This is getting into the realm of
 > speculation though.
 > 
 > 			regards, tom lane
 > 
 > ---------------------------(end of broadcast)---------------------------
 > TIP 2: you can get off all lists at once with the unregister command
 >     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)



Tue, 24 Aug 2004 12:54:26 +0300
Daniel Kalchev <daniel@digsys.bg>
Re: missing data/global 
Daniel Kalchev <daniel@digsys.bg> writes:
> (found out 7.2.3 does not have pg_database)

You think not?

> By the way, I had to copy over the 'new' files from pg_clog and pg_xlog (this
> is the second possible error) to get the postmaster running.

That was *not* part of the recipe, and is guaranteed *not* to work.

It seems likely though that you are wasting your time --- the index
failure suggests strongly that you have more corruption than just the
loss of the /global subdirectory :-(

			regards, tom lane

Tue, 24 Aug 2004 10:43:41 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: missing data/global 
>>>Tom Lane said:
 > Daniel Kalchev <daniel@digsys.bg> writes:
 > > (found out 7.2.3 does not have pg_database)
 > 
 > You think not?

Not as a file similar to pg_control. pg_database is indeed table in the system 
catalog.

 > > By the way, I had to copy over the 'new' files from pg_clog and pg_xlog (t
     his
 > > is the second possible error) to get the postmaster running.
 > 
 > That was *not* part of the recipe, and is guaranteed *not* to work.

I know that, but wondered if it would help in any way.. By the way, what would 
be the solution to sync WAL with the pg_control contents?

 > 
 > It seems likely though that you are wasting your time --- the index
 > failure suggests strongly that you have more corruption than just the
 > loss of the /global subdirectory :-(

After spending some time to find possible ways to adjust pointers (could 
eventually save part of the data), I decided to move to plan B, which is to 
have few people manually re-enter the data - would have been more effective to 
waste my time anyway - but not if it will take days and the result be not 
guaranteed to be consistent.

Does such toll exist, that could dump data (records?) from the heap files 
given the table structure?

Regards,
Daniel


Wed, 25 Aug 2004 19:07:23 +0300
Daniel Kalchev <daniel@digsys.bg>
Re: missing data/global 
On Wed, Aug 25, 2004 at 07:07:23PM +0300, Daniel Kalchev wrote:

> Does such toll exist, that could dump data (records?) from the heap files 
> given the table structure?

You may want to check pg_filedump (from http://sources.redhat.com/rhdb
IIRC).

(What happened to pg_fsck BTW?)

-- 
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)


Wed, 25 Aug 2004 12:34:34 -0400
Alvaro Herrera <alvherre@dcc.uchile.cl>
Re: missing data/global
Coincidentally I JUST NOW built 8.0 on Solaris 9, and ran into the same
problem. As they say, "this used to work"..... 

We build databases as part of the build of our product, and I'm looking
into what we need to do to upgrade from 7.4.5, and this was the first
thing I ran into. I hadn't gotten as far as truss yet, so thanks Kenneth
for that extra info.

Did initdb previously just assume the -D path existed, and now it is
trying to create the whole path, if necessary?

- DAP

>-----Original Message-----
>From: pgsql-hackers-owner@postgresql.org 
>[mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Kenneth Lareau
>Sent: Thursday, January 27, 2005 5:23 PM
>To: pgsql-hackers@postgresql.org
>Subject: [HACKERS] Strange issue with initdb on 8.0 and 
>Solaris automounts
>
>Folks,
>
>I ran into an interesting issue when installing PostgreSQL 8.0 
>that I'm not sure how to resolve correctly.  My system is a 
>Sun machine (Blade
>1000) running Solaris 9, with relatively recent patches. After 
>install- ing 8.0, I went to run the 'initdb' command and was 
>greeted with the
>following:
>
>[delirium:postgres] ~
>(11) initdb -D /software/postgresql-8.0.0/data The files 
>belonging to this database system will be owned by user "postgres".
>This user must also own the server process.
>
>The database cluster will be initialized with locale C.
>
>creating directory /software/postgresql-8.0.0/data ... initdb: 
>could not create directory "/software/postgresql-8.0.0": 
>Operation not applicable
>
>
>The error message was a bit confusing, so I decided to run a 
>truss on the process to see what might be happening, and this 
>is what I came
>across:
>
>[...]
>8802/1:         write(1, " c r e a t i n g   d i r".., 62)      = 62
>8802/1:         umask(0)                                        = 077
>8802/1:         umask(077)                                      = 0
>8802/1:         mkdir("/software", 0777)                       
> Err#17 EEXIST
>8802/1:         stat64("/software", 0xFFBFC858)                 = 0
>8802/1:         mkdir("/software/postgresql-8.0.0", 0777)      
> Err#89 ENOSYS
>[...]
>
>
>The last error in that section, ENOSYS, is very strange, as 
>the Solaris manpage for 'mkdir' does not mention it as a 
>possible error.  One thing to note in this, however, is that 
>'/software/postgresql-8.0.0' is not a regular directory, but 
>an automount point (which in this case is just a local 
>loopback mount).  So the indication is that Solaris seems to 
>have a bug not in mkdir, but deeper in their VFS code that's 
>causing this seemingly strange issue.
>
>Two workarounds for this problem have been found: running 
>'initdb' with a directory that's *not* an automount point and 
>then moving the 'data'
>directory to its final destination worked fine, along with a 
>suggestion from Andrew Dunstan (on the #postgresql IRC 
>channel) with using a rela- tive path for the data directory.  
>Both were successful in avoiding the issue, but I decided to 
>mention this here in case someone felt it might be worth 
>looking into to see if the Sun problem can be avoided; I am 
>going to notify Sun of their bug, just don't know how long it 
>will take them to actually resolve it (if they ever do).
>
>While I can fully understand that a code change here may not 
>be desire- able, might some notes in the documentation be 
>useful for those who might stumble across the problem as well? 
> Just a suggestion...
>
>I hope I gave sufficient information on the problem, though 
>I'm always willing to give any clarification needed.  Thank 
>you for your time.
>
>
>Ken Lareau
>elessar@numenor.org
>
>---------------------------(end of 
>broadcast)---------------------------
>TIP 6: Have you searched our list archives?
>
>               http://archives.postgresql.org
>

Thu, 27 Jan 2005 17:49:37 -0500
"David Parker" <dparker@tazznetworks.com>
Re: Strange issue with initdb on 8.0 and Solaris automounts
"David Parker" <dparker@tazznetworks.com> writes:
> Did initdb previously just assume the -D path existed, and now it is
> trying to create the whole path, if necessary?

Pre-8.0 it was using mkdir(1), which might possibly contain some weird
workaround for this case on Solaris.

I suppose that manually creating the data directory before running
initdb would also avoid this issue, since the mkdir(2) loop is only
entered if we don't find the directory in existence.

			regards, tom lane

Thu, 27 Jan 2005 18:22:18 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Strange issue with initdb on 8.0 and Solaris automounts 
In message <21723.1106868138@sss.pgh.pa.us>, Tom Lane writes:
>"David Parker" <dparker@tazznetworks.com> writes:
>> Did initdb previously just assume the -D path existed, and now it is
>> trying to create the whole path, if necessary?
>
>Pre-8.0 it was using mkdir(1), which might possibly contain some weird
>workaround for this case on Solaris.
>
>I suppose that manually creating the data directory before running
>initdb would also avoid this issue, since the mkdir(2) loop is only
>entered if we don't find the directory in existence.
>
>			regards, tom lane
>

Actually, creating the 'data' directory first doesn't work either:

[delirium:postgres] ~
(17) mkdir data
[delirium:postgres] ~
(18) initdb -D /software/postgresql-8.0.0/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale C.

fixing permissions on existing directory /software/postgresql-8.0.0/data ... ok
creating directory /software/postgresql-8.0.0/data/global ... initdb: could not create directory "/software/postgresql-8.0.0": Operation not applicable
initdb: removing contents of data directory "/software/postgresql-8.0.0/data"


Since there's subdirectories that need to be created, it still runs into
the problem.  I don't know why the command 'mkdir' doesn't exhibit the
same problem as the function 'mkdir', but running:

   mkdir /software/postgresql-8.0.0

produces the correct error "File exists" on my system.  I suspect the
'mkdir' command probably checks to see if the directory exists first
before trying to create it, which avoids the problem.


Ken Lareau
elessar@numenor.org

Thu, 27 Jan 2005 15:35:41 -0800
Kenneth Lareau <elessar@numenor.org>
Re: Strange issue with initdb on 8.0 and Solaris automounts
Kenneth Lareau <elessar@numenor.org> writes:
> In message <21723.1106868138@sss.pgh.pa.us>, Tom Lane writes:
>> I suppose that manually creating the data directory before running
>> initdb would also avoid this issue, since the mkdir(2) loop is only
>> entered if we don't find the directory in existence.

> Actually, creating the 'data' directory first doesn't work either:

Good point.

> I don't know why the command 'mkdir' doesn't exhibit the
> same problem as the function 'mkdir', but running:

>    mkdir /software/postgresql-8.0.0

> produces the correct error "File exists" on my system.

Could you truss that and see what it does?  It would be a simple change
in initdb to make it stat before mkdir instead of after, but I'm not
totally convinced that would fix the problem.  If mkdir returns a funny
error code then stat might as well ...

			regards, tom lane

Thu, 27 Jan 2005 18:50:48 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Strange issue with initdb on 8.0 and Solaris automounts 
In message <22095.1106869848@sss.pgh.pa.us>, Tom Lane writes:
>Kenneth Lareau <elessar@numenor.org> writes:
>> In message <21723.1106868138@sss.pgh.pa.us>, Tom Lane writes:
>>> I suppose that manually creating the data directory before running
>>> initdb would also avoid this issue, since the mkdir(2) loop is only
>>> entered if we don't find the directory in existence.
>
>> Actually, creating the 'data' directory first doesn't work either:
>
>Good point.
>
>> I don't know why the command 'mkdir' doesn't exhibit the
>> same problem as the function 'mkdir', but running:
>
>>    mkdir /software/postgresql-8.0.0
>
>> produces the correct error "File exists" on my system.
>
>Could you truss that and see what it does?  It would be a simple change
>in initdb to make it stat before mkdir instead of after, but I'm not
>totally convinced that would fix the problem.  If mkdir returns a funny
>error code then stat might as well ...
>
>			regards, tom lane
>

Here's the relevant truss output from 'mkdir /software/postgresql-8.0.0'
on my Solaris 9 system:

10832:  umask(0)                                        = 077
10832:  umask(077)                                      = 0
10832:  mkdir("/software/postgresql-8.0.0", 0777)       Err#89 ENOSYS
10832:  stat64("/software/postgresql-8.0.0", 0xFFBFFA38) = 0
10832:  fstat64(2, 0xFFBFEB78)                          = 0
10832:  write(2, " m k d i r", 5)                       = 5
10832:  write(2, " :  ", 2)                             = 2
10832:  write(2, " c a n n o t   c r e a t".., 24)      = 24
10832:  write(2, " ` / s o f t w a r e / p".., 28)      = 28
10832:  write(2, " :  ", 2)                             = 2
10832:  write(2, " F i l e   e x i s t s", 11)          = 11
10832:  write(2, "\n", 1)                               = 1
10832:  _exit(1)


It's doing the stat after the mkdir attempt it seems, and coming back
with the correct response.  Hmm, maybe I should look at the Solaris 8
code for the mkdir command...


Ken Lareau
elessar@numenor.org

Thu, 27 Jan 2005 16:18:36 -0800
Kenneth Lareau <elessar@numenor.org>
Re: Strange issue with initdb on 8.0 and Solaris automounts


Tom Lane wrote:

>>I don't know why the command 'mkdir' doesn't exhibit the
>>same problem as the function 'mkdir', but running:
>>    
>>
>
>  
>
>>   mkdir /software/postgresql-8.0.0
>>    
>>
>
>  
>
>>produces the correct error "File exists" on my system.
>>    
>>
>
>Could you truss that and see what it does?  It would be a simple change
>in initdb to make it stat before mkdir instead of after, but I'm not
>totally convinced that would fix the problem.  If mkdir returns a funny
>error code then stat might as well ...
>
>
>  
>

There's also a tiny race condition, which I guess isn't worth worrying 
about.

Returning ENOSYS is pretty bogus ...

cheers

andrew

Thu, 27 Jan 2005 19:28:25 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: Strange issue with initdb on 8.0 and Solaris automounts
Kenneth Lareau <elessar@numenor.org> writes:
> In message <22095.1106869848@sss.pgh.pa.us>, Tom Lane writes:
>> Could you truss that and see what it does?

> Here's the relevant truss output from 'mkdir /software/postgresql-8.0.0'
> on my Solaris 9 system:

> 10832:  mkdir("/software/postgresql-8.0.0", 0777)       Err#89 ENOSYS
> 10832:  stat64("/software/postgresql-8.0.0", 0xFFBFFA38) = 0

> It's doing the stat after the mkdir attempt it seems, and coming back
> with the correct response.  Hmm, maybe I should look at the Solaris 8
> code for the mkdir command...

Well, the important point is that the stat does succeed.  I'm not going
to put in anything as specific as a check for ENOSYS, but it seems
reasonable to try the stat first and mkdir only if stat fails.
I've applied the attached patch.

			regards, tom lane

*** src/bin/initdb/initdb.c.orig	Sat Jan  8 17:51:12 2005
--- src/bin/initdb/initdb.c	Thu Jan 27 19:23:49 2005
***************
*** 476,481 ****
--- 476,484 ----
   * this tries to build all the elements of a path to a directory a la mkdir -p
   * we assume the path is in canonical form, i.e. uses / as the separator
   * we also assume it isn't null.
+  *
+  * note that on failure, the path arg has been modified to show the particular
+  * directory level we had problems with.
   */
  static int
  mkdir_p(char *path, mode_t omode)
***************
*** 544,573 ****
  		}
  		if (last)
  			(void) umask(oumask);
! 		if (mkdir(path, last ? omode : S_IRWXU | S_IRWXG | S_IRWXO) < 0)
  		{
! 			if (errno == EEXIST || errno == EISDIR)
! 			{
! 				if (stat(path, &sb) < 0)
! 				{
! 					retval = 1;
! 					break;
! 				}
! 				else if (!S_ISDIR(sb.st_mode))
! 				{
! 					if (last)
! 						errno = EEXIST;
! 					else
! 						errno = ENOTDIR;
! 					retval = 1;
! 					break;
! 				}
! 			}
! 			else
  			{
  				retval = 1;
  				break;
  			}
  		}
  		if (!last)
  			*p = '/';
--- 547,570 ----
  		}
  		if (last)
  			(void) umask(oumask);
! 
! 		/* check for pre-existing directory; ok if it's a parent */
! 		if (stat(path, &sb) == 0)
  		{
! 			if (!S_ISDIR(sb.st_mode))
  			{
+ 				if (last)
+ 					errno = EEXIST;
+ 				else
+ 					errno = ENOTDIR;
  				retval = 1;
  				break;
  			}
+ 		}
+ 		else if (mkdir(path, last ? omode : S_IRWXU | S_IRWXG | S_IRWXO) < 0)
+ 		{
+ 			retval = 1;
+ 			break;
  		}
  		if (!last)
  			*p = '/';

Thu, 27 Jan 2005 19:37:33 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Strange issue with initdb on 8.0 and Solaris automounts 
Andrew Dunstan <andrew@dunslane.net> writes:
> There's also a tiny race condition, which I guess isn't worth worrying 
> about.

Considering that we're not checking ownership or permissions of the
parent directories, I'd say not.

			regards, tom lane

Thu, 27 Jan 2005 19:39:59 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Strange issue with initdb on 8.0 and Solaris automounts 
In message <22687.1106872653@sss.pgh.pa.us>, Tom Lane writes:
>Kenneth Lareau <elessar@numenor.org> writes:
>> In message <22095.1106869848@sss.pgh.pa.us>, Tom Lane writes:
>>> Could you truss that and see what it does?
>
>> Here's the relevant truss output from 'mkdir /software/postgresql-8.0.0'
>> on my Solaris 9 system:
>
>> 10832:  mkdir("/software/postgresql-8.0.0", 0777)       Err#89 ENOSYS
>> 10832:  stat64("/software/postgresql-8.0.0", 0xFFBFFA38) = 0
>
>> It's doing the stat after the mkdir attempt it seems, and coming back
>> with the correct response.  Hmm, maybe I should look at the Solaris 8
>> code for the mkdir command...
>
>Well, the important point is that the stat does succeed.  I'm not going
>to put in anything as specific as a check for ENOSYS, but it seems
>reasonable to try the stat first and mkdir only if stat fails.
>I've applied the attached patch.
>
>			regards, tom lane


Tom, thank you very much for the patch, it worked like a charm.


Ken Lareau
elessar@numenor.org

Thu, 27 Jan 2005 17:10:04 -0800
Kenneth Lareau <elessar@numenor.org>
Re: Strange issue with initdb on 8.0 and Solaris automounts
> My opinion is that this is a very bogus shortcut in the 
> network datatype code.  There are no cases outside the 
> inet/cidr group where an operator doesn't exactly match its 
> underlying function.  (The whole business of inet and cidr 
> being almost but not quite the same type is maldesigned
> anyway...)
> 
> The right solution for you is to declare two SQL functions.  
> Whether you make them point at the same underlying C code is 
> up to you.

Right,...

In that case may I suggest fixing the catalog so network_* functions exists for both datatypes!
Anything less I'd consider inconsistent...

Kind regards,

John

Sun, 30 Jan 2005 13:46:34 +1100
"John Hansen" <john@geeknet.com.au>
Re: Bug in create operator and/or initdb 
"John Hansen" <john@geeknet.com.au> writes:
> In that case may I suggest fixing the catalog so network_* functions exists for both datatypes!

Redesigning the inet/cidr distinction is on the to-do list (though I'm
afraid not very high on the list).  ISTM it should either be one type
with a distinguishing bit in the runtime representation, or two types
with no such bit needed.  Having both is a schizophrenic design.  It's
led directly to bugs in the past, and I think there are still some
corner cases that act oddly (see the archives).

			regards, tom lane

Sat, 29 Jan 2005 22:07:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Bug in create operator and/or initdb 
On Sat, Jan 29, 2005 at 10:07:30PM -0500, Tom Lane wrote:
> "John Hansen" <john@geeknet.com.au> writes:
> > In that case may I suggest fixing the catalog so network_* functions exists for both datatypes!
> 
> Redesigning the inet/cidr distinction is on the to-do list (though I'm
> afraid not very high on the list).  ISTM it should either be one type
> with a distinguishing bit in the runtime representation, or two types
> with no such bit needed.  Having both is a schizophrenic design.  It's
> led directly to bugs in the past, and I think there are still some
> corner cases that act oddly (see the archives).

 From a network engineering point of view the inet type is utterly
bogus. I'm not aware of data of that type being needed or used in
any real application. Given that, the complexity that it causes
simply by existing seems too high a cost.

I suspect that the right thing to do is to kill the inet type
entirely, and replace it with a special case of cidr. (And possibly
then to kill cidr and replace it with something that can be indexed
more effectively.)

For a replacement type, how important is it that it be completely
compatible with the existing inet/cidr types? Is anyone actually using
inet types with a non-cidr mask?

Cheers,
  Steve

Sat, 29 Jan 2005 20:20:10 -0800
Steve Atkins <steve@blighty.com>
Re: [BUGS] Bug in create operator and/or initdb
Steve Atkins <steve@blighty.com> writes:
> For a replacement type, how important is it that it be completely
> compatible with the existing inet/cidr types? Is anyone actually using
> inet types with a non-cidr mask?

If you check the archives you'll discover that our current inet/cidr
types were largely designed and implemented by Paul Vixie (yes, that
Vixie).  I'm disinclined to second-guess Paul about the external
definition of these types; I just want to rationalize the internal
representation a bit.  In particular we've got some issues about
conversions between the two types ...

			regards, tom lane

Sun, 30 Jan 2005 01:46:19 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [BUGS] Bug in create operator and/or initdb 
On Sun, 30 Jan 2005, Tom Lane wrote:

> Steve Atkins <steve@blighty.com> writes:
>> For a replacement type, how important is it that it be completely
>> compatible with the existing inet/cidr types? Is anyone actually using
>> inet types with a non-cidr mask?
>
> If you check the archives you'll discover that our current inet/cidr
> types were largely designed and implemented by Paul Vixie (yes, that
> Vixie).  I'm disinclined to second-guess Paul about the external
> definition of these types; I just want to rationalize the internal
> representation a bit.  In particular we've got some issues about
> conversions between the two types ...
Please do **NOT** break the external representations.  We had enough fights
about that 2-3 releases ago, and I personally don't want to revisit them.

Yes, we do flakey things with inet on the masking stuff.

LER

-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Sun, 30 Jan 2005 21:49:43 -0600 (CST)
Larry Rosenman <ler@lerctr.org>
Re: [BUGS] Bug in create operator and/or initdb
On Sun, Jan 30, 2005 at 09:49:43PM -0600, Larry Rosenman wrote:
> On Sun, 30 Jan 2005, Tom Lane wrote:
> 
> >Steve Atkins <steve@blighty.com> writes:
> >>For a replacement type, how important is it that it be completely
> >>compatible with the existing inet/cidr types? Is anyone actually using
> >>inet types with a non-cidr mask?
> >
> >If you check the archives you'll discover that our current inet/cidr
> >types were largely designed and implemented by Paul Vixie (yes, that
> >Vixie).  I'm disinclined to second-guess Paul about the external
> >definition of these types; I just want to rationalize the internal
> >representation a bit.  In particular we've got some issues about
> >conversions between the two types ...
>
> Please do **NOT** break the external representations.  We had enough fights
> about that 2-3 releases ago, and I personally don't want to revisit them.

> Yes, we do flakey things with inet on the masking stuff.

Well, if you want the ability to store both a host address and a
netmask in the same datatype the inet masking stuff makes sense.
That's not really a useful datatype for any actual use, but it's
fairly well-defined. The problem is that when someone looks at the
docs they'll see inet as the obvious datatype to use to store IP
addresses, and it isn't very good for that.

But that's not all that's flakey, unfortunately.

The CIDR input format is documented to be classful, which in itself is
horribly obsolete and completely useless in this decades internet (and
was when the current code was written in '98).

But the implementation isn't either classful or classless, and the
behaviour disagrees with documented behaviour, and the behaviour you'd
reasonably expect, in many cases.

-- Class A - documented to be 10.0.0.0/8
  steve=# select '10.0.0.0'::cidr;
      cidr     
  -------------
   10.0.0.0/32

-- Class B - documented to be 128.0.0.0/16
  steve=# select '128.0.0.0'::cidr;
       cidr     
  --------------
   128.0.0.0/32

-- Class C - documented to be 223.10.0.0/24
  steve=# select '223.10.0.0'::cidr;
       cidr      
  ---------------
   223.10.0.0/32

-- Class D
  steve=# select '224.10.0.0'::cidr;
  ERROR:  invalid cidr value: "224.10.0.0"
  DETAIL:  Value has bits set to right of mask.

  steve=# select '224.0.0.0'::cidr;
      cidr     
  -------------
   224.0.0.0/4

-- Class E
  steve=# select '240.10.0.0'::cidr;
       cidr      
  ---------------
   240.10.0.0/32

I use postgresql for network-related applications and for IP address
related data mining, so I'm dealing with IP addresses in postgresql
on a daily basis.

The cidr type, including it's external interface, is simply broken.
There is no way to fix it that doesn't change that external interface.

I know of at least two independant implementations of function IP
address types that have been put together for specific projects to
implement a working IP datatype. The ability to use gist indexes on
them to accelerate range-based lookups is a bonus.

If it's not possible (for backwards compatibility reasons) to fix
inet+cidr, would migrating them out to contrib be a possibility?
Data types in the core tend to be widely used, even if they're
broken and there are better datatypes implemented as external
modules.

Cheers,
  Steve

Mon, 31 Jan 2005 07:23:05 -0800
Steve Atkins <steve@blighty.com>
Re: [BUGS] Bug in create operator and/or initdb
Steve Atkins <steve@blighty.com> writes:
> The cidr type, including it's external interface, is simply broken.

That is a large claim that I don't think you have demonstrated.
The only one of your examples that seems to me to contradict the
documentation is this one:

 steve=# select '224.0.0.0'::cidr;
      cidr     
  -------------
   224.0.0.0/4

which should be /32 according to what the docs say:

: If y is omitted, it is calculated using assumptions from the older
: classful network numbering system, except that it will be at least large
: enough to include all of the octets written in the input.

The bogus netmask is in turn responsible for this case:

  steve=# select '224.10.0.0'::cidr;
  ERROR:  invalid cidr value: "224.10.0.0"
  DETAIL:  Value has bits set to right of mask.


Looking at the source code, there seems to be a special case for "class D"
network numbers that causes the code not to extend y to cover the
supplied inputs:

    /* If no CIDR spec was given, infer width from net class. */
    if (bits == -1)
    {
        if (*odst >= 240)        /* Class E */
            bits = 32;
        else if (*odst >= 224)    /* Class D */
            bits = 4;
        else if (*odst >= 192)    /* Class C */
            bits = 24;
        else if (*odst >= 128)    /* Class B */
            bits = 16;
        else                      /* Class A */
            bits = 8;
        /* If imputed mask is narrower than specified octets, widen. */
        if (bits >= 8 && bits < ((dst - odst) * 8))
            ^^^^^^^^^
            bits = (dst - odst) * 8;
    }

I think the test for "bits >= 8" should be removed.  Does anyone know
why it's there?

			regards, tom lane

Mon, 31 Jan 2005 12:16:26 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [BUGS] Bug in create operator and/or initdb 
On Mon, Jan 31, 2005 at 12:16:26PM -0500, Tom Lane wrote:
> Steve Atkins <steve@blighty.com> writes:
> > The cidr type, including it's external interface, is simply broken.
> 
> That is a large claim that I don't think you have demonstrated.
> The only one of your examples that seems to me to contradict the
> documentation is this one:
> 
>  steve=# select '224.0.0.0'::cidr;
>       cidr     
>   -------------
>    224.0.0.0/4
> 
> which should be /32 according to what the docs say:

OK. If this sort of thing is considered a bug, rather than part
of the external interface that shouldn't be changed, then I'd
agree that cidr isn't entirely broken and it may well be possible
to improve it without changing the interface.

/me goes grovelling through the IPv6 inet code...

Cheers,
  Steve


Mon, 31 Jan 2005 09:54:24 -0800
Steve Atkins <steve@blighty.com>
Re: [BUGS] Bug in create operator and/or initdb
when my cidr datatype was integrated into pgsql, the decision was made to
incorporate a copy of bind's inet_net_pton.c rather than add a link-time
dependence to libbind.a (libbind.so).  thus, when this bug was fixed in
2003:

----------------------------
revision 1.14
date: 2003/08/20 02:21:08;  author: marka;  state: Exp;  lines: +10 -4
1580.   [bug]           inet_net_pton() didn't fully handle implicit
                        multicast IPv4 network addresses.

the pgsql "fork" of this code did not benefit from the fix.  the patch was:

Index: inet_net_pton.c
===================================================================
RCS file: /proj/cvs/prod/bind8/src/lib/inet/inet_net_pton.c,v
retrieving revision 1.13
retrieving revision 1.14
diff -u -r1.13 -r1.14
--- inet_net_pton.c     27 Sep 2001 15:08:38 -0000      1.13
+++ inet_net_pton.c     20 Aug 2003 02:21:08 -0000      1.14
@@ -16,7 +16,7 @@
  */
 
 #if defined(LIBC_SCCS) && !defined(lint)
-static const char rcsid[] = "$Id: inet_net_pton.c,v 1.13 2001/09/27 15:08:38 marka Exp $";
+static const char rcsid[] = "$Id: inet_net_pton.c,v 1.14 2003/08/20 02:21:08 marka Exp $";
 #endif
 
 #include "port_before.h"
@@ -59,7 +59,7 @@
  *     Paul Vixie (ISC), June 1996
  */
 static int
-inet_net_pton_ipv4( const char *src, u_char *dst, size_t size) {
+inet_net_pton_ipv4(const char *src, u_char *dst, size_t size) {
        static const char xdigits[] = "0123456789abcdef";
        static const char digits[] = "0123456789";
        int n, ch, tmp = 0, dirty, bits;
@@ -152,7 +152,7 @@
                if (*odst >= 240)       /* Class E */
                        bits = 32;
                else if (*odst >= 224)  /* Class D */
-                       bits = 4;
+                       bits = 8;
                else if (*odst >= 192)  /* Class C */
                        bits = 24;
                else if (*odst >= 128)  /* Class B */
@@ -160,8 +160,14 @@
                else                    /* Class A */
                        bits = 8;
                /* If imputed mask is narrower than specified octets, widen. */
-               if (bits >= 8 && bits < ((dst - odst) * 8))
+               if (bits < ((dst - odst) * 8))
                        bits = (dst - odst) * 8;
+               /*
+                * If there are no additional bits specified for a class D
+                * address adjust bits to 4.
+                */
+               if (bits == 8 && *odst == 224)
+                       bits = 4;
        }
        /* Extend network to cover the actual mask. */
        while (bits > ((dst - odst) * 8)) {

re:

> To: Steve Atkins <steve@blighty.com>
> Cc: pgsql-hackers <pgsql-hackers@postgresql.org>, paul@vix.com
> Subject: Re: [HACKERS] [BUGS] Bug in create operator and/or initdb 
> Comments: In-reply-to Steve Atkins <steve@blighty.com>
> 	message dated "Mon, 31 Jan 2005 07:23:05 -0800"
> Date: Mon, 31 Jan 2005 12:16:26 -0500
> From: Tom Lane <tgl@sss.pgh.pa.us>
> 
> Steve Atkins <steve@blighty.com> writes:
> > The cidr type, including it's external interface, is simply broken.
> 
> That is a large claim that I don't think you have demonstrated.
> The only one of your examples that seems to me to contradict the
> documentation is this one:
> 
>  steve=# select '224.0.0.0'::cidr;
>       cidr     
>   -------------
>    224.0.0.0/4
> 
> which should be /32 according to what the docs say:
> 
> : If y is omitted, it is calculated using assumptions from the older
> : classful network numbering system, except that it will be at least large
> : enough to include all of the octets written in the input.
> 
> The bogus netmask is in turn responsible for this case:
> 
>   steve=# select '224.10.0.0'::cidr;
>   ERROR:  invalid cidr value: "224.10.0.0"
>   DETAIL:  Value has bits set to right of mask.
> 
> 
> Looking at the source code, there seems to be a special case for "class D"
> network numbers that causes the code not to extend y to cover the
> supplied inputs:
> 
>     /* If no CIDR spec was given, infer width from net class. */
>     if (bits == -1)
>     {
>         if (*odst >= 240)        /* Class E */
>             bits = 32;
>         else if (*odst >= 224)    /* Class D */
>             bits = 4;
>         else if (*odst >= 192)    /* Class C */
>             bits = 24;
>         else if (*odst >= 128)    /* Class B */
>             bits = 16;
>         else                      /* Class A */
>             bits = 8;
>         /* If imputed mask is narrower than specified octets, widen. */
>         if (bits >= 8 && bits < ((dst - odst) * 8))
>             ^^^^^^^^^
>             bits = (dst - odst) * 8;
>     }
> 
> I think the test for "bits >= 8" should be removed.  Does anyone know
> why it's there?
> 
> 			regards, tom lane

Mon, 31 Jan 2005 19:28:28 +0000
Paul Vixie <paul@vix.com>
Re: [BUGS] Bug in create operator and/or initdb 
Tom Lane <tgl@sss.pgh.pa.us> writes:

>  steve=# select '224.0.0.0'::cidr;
>       cidr     
>   -------------
>    224.0.0.0/4
> 
> which should be /32 according to what the docs say:

224-239 are multicast addresses. Making it /4 makes the entire multicast
address space one network block which is about as reasonable an answer as
anything else.

>         if (bits >= 8 && bits < ((dst - odst) * 8))
>             ^^^^^^^^^
>             bits = (dst - odst) * 8;
>     }
> 
> I think the test for "bits >= 8" should be removed.  Does anyone know
> why it's there?

I guess Vixie figured network blocks subdividing multicast address space
weren't a sensible concept? It's a bit of a strange constraint to hard code
into the C code though.

Incidentally, how can that code possibly work? It treats odst as a pointer in
some places but then calculates bits using arithmetic on it directly without
dereferencing?

-- 
greg


31 Jan 2005 14:35:58 -0500
Greg Stark <gsstark@mit.edu>
Re: [BUGS] Bug in create operator and/or initdb
Paul Vixie <paul@vix.com> writes:
> when my cidr datatype was integrated into pgsql, the decision was made to
> incorporate a copy of bind's inet_net_pton.c rather than add a link-time
> dependence to libbind.a (libbind.so).

We didn't really want to assume that all platforms are using libbind :-(

> thus, when this bug was fixed in 2003:

> ----------------------------
> revision 1.14
> date: 2003/08/20 02:21:08;  author: marka;  state: Exp;  lines: +10 -4
> 1580.   [bug]           inet_net_pton() didn't fully handle implicit
>                         multicast IPv4 network addresses.

> the pgsql "fork" of this code did not benefit from the fix.  the patch was:

Ah-hah.  Many thanks for supplying the patch --- will integrate it.

			regards, tom lane

Mon, 31 Jan 2005 14:36:33 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [BUGS] Bug in create operator and/or initdb 
> We didn't really want to assume that all platforms are using libbind :-(

i think you could have, at the time, since windows wasn't even a gleam in
pgsql's eye.  even now, libbind would be a dependable universal dependency,
since we publish windows binaries.

> > the pgsql "fork" of this code did not benefit from the fix.  the patch was:
> 
> Ah-hah.  Many thanks for supplying the patch --- will integrate it.

i have two suggestions.  first, look at the rest of the current source file,
in case there are other fixes.  second, track changes this source file during
your release engineering process for each new pgsql version.

Mon, 31 Jan 2005 22:44:38 +0000
Paul Vixie <paul@vix.com>
Re: [BUGS] Bug in create operator and/or initdb 
Paul Vixie <paul@vix.com> writes:
> i have two suggestions.  first, look at the rest of the current source file,
> in case there are other fixes.

Right, I already grabbed the latest.

> second, track changes this source file during
> your release engineering process for each new pgsql version.

Bruce, do you think this is worth adding to RELEASE_CHANGES?
inet_net_ntop.c and inet_net_pton.c are both extracted from the BIND
distribution.  But they're hardly the only files we took from elsewhere.

			regards, tom lane

Mon, 31 Jan 2005 17:52:40 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [BUGS] Bug in create operator and/or initdb 
Tom Lane wrote:
> Paul Vixie <paul@vix.com> writes:
> > i have two suggestions.  first, look at the rest of the current source file,
> > in case there are other fixes.
> 
> Right, I already grabbed the latest.
> 
> > second, track changes this source file during
> > your release engineering process for each new pgsql version.
> 
> Bruce, do you think this is worth adding to RELEASE_CHANGES?
> inet_net_ntop.c and inet_net_pton.c are both extracted from the BIND
> distribution.  But they're hardly the only files we took from elsewhere.

Yes, I do.  Most of the stuff we pull from other OS projects has
clearly-defined behavior, while inet/cidr seem to be still in flux a
little.

Added to release checklist:

	* Update inet/cidr data types with newest Bind patches

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 2 Feb 2005 12:00:05 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [BUGS] Bug in create operator and/or initdb
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Added to release checklist:
> 	* Update inet/cidr data types with newest Bind patches

You should also add "check for zic database updates".

			regards, tom lane

Wed, 02 Feb 2005 12:02:58 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [BUGS] Bug in create operator and/or initdb 
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Added to release checklist:
> > 	* Update inet/cidr data types with newest Bind patches
> 
> You should also add "check for zic database updates".

Uh, we already have:

	* Update timezone data to match latest zic database (see
	  src/timezone/README)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 2 Feb 2005 12:38:04 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [BUGS] Bug in create operator and/or initdb