While applying the NT regression tests, I remember Tom Lane's comment
that people are being much more picky about the regression results.  In
the old days, we could just say that they will have _expected_ errors,
but now they want them to match exactly.

Kind of funny, their standards are going up.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 31 Mar 2000 09:17:58 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Regression tests
Bruce Momjian wrote:
> 
> While applying the NT regression tests, I remember Tom Lane's comment
> that people are being much more picky about the regression results.  In
> the old days, we could just say that they will have _expected_ errors,
> but now they want them to match exactly.

Well, it is NICE if they match exactly, but not essential for me.  I
have been advising intel RPM dist users to _expect_ float8 and geometry
failures -- and the occassional 'random' failure, of course.  And the
situation with RedHat 6.1's locales brings yet another mixed bag -- I
advise intel RPM users to completely disable RedHat's locale support (by
renaming /etc/sysconfig/i18n to something else and rebooting) for
performing regression tests -- then they can restore locale.  Sort order
under RedHat 6.1's locale is _messed_up_.

Thomas mentioned RedHat 6.1 Intel being an appropriate reference
platform -- if that is the case, then RH 6.1 Intel needs exact matches. 
Of course, ANY platform can be the reference -- as long as _one_ is.  If
BSD/os 4 (or whatever rev you're currently running) were to be the
reference, then the regression tests had better match your machine's
run.

My opinion is to select the reference platform that produces the least
amount of failures for other known good platforms.

Also, geometry wouldn't fail if the number of digits of precision was
jacked down one.... :-).

> Kind of funny, their standards are going up.

Is it a case of standards going up, or standards going down? ;-)

As far as PostgreSQL's performance, our standards are definitely going
up -- there has never been a better PostgreSQL, all around, than the one
that is in CURRENT, IMO (and I've used everything since 6.1.1).

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Fri, 31 Mar 2000 09:46:16 -0500
Lamar Owen <lamar.owen@wgcr.org>
Re: Regression tests
At 09:17 AM 3/31/00 -0500, Bruce Momjian wrote:
>While applying the NT regression tests, I remember Tom Lane's comment
>that people are being much more picky about the regression results.  In
>the old days, we could just say that they will have _expected_ errors,
>but now they want them to match exactly.
>
>Kind of funny, their standards are going up.

Is this perhaps a result of a growing audience for Postgres?

For instance, I dealt with one of our web toolkit "early achievers",
new to AOLserver, new to Postgres, new to the toolkit - that's a lot
of "new to's" for someone to deal with in parallel!

He had problems with the regression tests - cockpit error, first
go-around, later diminished to expected errors.  He's hacker enough
to have run the regression tests in the first place (rather than
blindly assume his install went OK) and also to figure out that
the geometry results were probably due simply to FP imprecision,
but wanted to safety-blanket reassurance from myself (and Lamar
Owen) that all was A-OK.  Particularly after his first go-around
of self-inflicted problems (the details of which I don't even
remember at the moment, he figured them out himself).

As PG gets more use, I would expect to see more, not fewer, intellegent
newcomers who aren't steeped in PG lore (i.e. experience with old
versions) who will be full of questions about any seeming abnormality.



- Don Baccus, Portland OR <dhogaza@pacifier.com>
  Nature photos, on-line guides, Pacific Northwest
  Rare Bird Alert Service and other goodies at
  http://donb.photo.net.

Fri, 31 Mar 2000 15:13:22 -0800
Don Baccus <dhogaza@pacifier.com>
Re: Regression tests
Lamar Owen writes:

> Thomas mentioned RedHat 6.1 Intel being an appropriate reference
> platform -- if that is the case, then RH 6.1 Intel needs exact matches. 
> Of course, ANY platform can be the reference -- as long as _one_ is.

Why do we need a reference platform? Just to show "See, all the tests pass
on this machine.", that has exactly zero practical value. If at all I
think whatever hub.org is running should be the reference. A platform with
broken locale certainly shouldn't.

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Sun, 2 Apr 2000 13:33:39 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Regression tests
> > Thomas mentioned RedHat 6.1 Intel being an appropriate reference
> > platform -- if that is the case, then RH 6.1 Intel needs exact matches.
> > Of course, ANY platform can be the reference -- as long as _one_ is.
> Why do we need a reference platform? Just to show "See, all the tests pass
> on this machine.", that has exactly zero practical value.

Sorry, I disagree. When we first started maintaining the regression
tests, they passed on *no* machine, but every machine had different
failures (much as happens now). We need to have one machine defined as
the standard to simplify the testing process and discussion, and until
now that machine has been mine. If this is an issue, then, at least
for the 7.0 release, that machine will continue to be the same one, a
Linux RH5.2 machine at my home.

Until someone else takes *complete* responsibility for the regression
tests, does so for some period of time, and makes sure that these
tests are run regularly, then I can't see the situation changing. But
it is certainly true that for the last while (6 months, 1 year?) it is
clear that there are several people with enough interest and
persistance to take on this responsibility if they would like.

RH6.1 vs some other platform is a minor issue. I'd be happy for the
reference machine have Mandrake 7.0, which I'm running on several
machines.

                        - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Sun, 02 Apr 2000 15:50:12 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: Regression tests