> -----Original Message-----
> From: Jason Earl [mailto:jason.earl@simplot.com] 
> Sent: Friday, June 20, 2003 3:32 PM
> To: Dann Corbit
> Cc: Jason Earl; The Hermit Hacker; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> "Dann Corbit" <DCorbit@connx.com> writes:
> >> 
> >> Why couldn't you just release the win32 version of 7.4 when
> >> it was finished.  If it takes an extra month then that just 
> >> gives you guys the chance to circulate *two* press releases.  
> >> The Native Win32 port is likely to make a big enough splash 
> >> all by itself.
> >
> > A formal release needs a big testing effort.  Two separate releases 
> > will double the work of validation.
> 
> There are several problems with that statement.  The first is 
> that PostgreSQL's "testing effort" happens right here on this 
> mailing list. 

That's not exactly reassuring.  There is no regression test that gets
formal acceptance?!

> The various PostgreSQL hackers code stuff up, 
> and we try and break it. There's very little /effort/ 
> involved.  People that want the new features go out on a limb 
> and start using them.  If they don't work, then they bring it 
> up on the list.  If they do work then very little gets said.
> 
> As it now stands Tom Lane is on the record as stating that 
> the new Win32 version isn't going to be ready for production 
> anyhow.  If anything the Win32 version *should* get released 
> separately simply because we don't want people mistaking the 
> Win32 version as being up to the PostgreSQL teams high 
> standards.  Those people that want the Win32 version to 
> become production ready are going to have to risk their 
> precious data.  Otherwise, the Win32 version will likely 
> remain a second class citizen forever.
> 
> The fact of the matter is that the Win32 specific bits are 
> the parts that are likely to break in the new port.  If 
> anything the Windows version will *benefit* from an earlier 
> *nix release because the *nix users will chase down the bugs 
> in the new PostgreSQL features.  Once the *nix version is up 
> to 7.4.2 (or so) then a Windows release of 7.4.2 should allow 
> the PostgreSQL hackers to simply chase down Windows specific problems.

Then using the same logic, the new Windows version should wait
indefinitely, since the *nix version will always be shaking out bugs.
 
> Adding a new platform--especially a platform as diverse from 
> the rest of PostgreSQL's supported platforms as Windows--is 
> what adds the work. Testing the new platform is relatively 
> easy.  All you need to do is to start using the Win32 version 
> with real live data.

That is not testing.  Using the world as your beta team seems to be a
business model used by a few software giants that is largely frowned
upon.  I would think that there is an opportunity to do things
differently. [Read 'properly'].

We (at CONNX Solutions Inc.) have a formal release procedure that
includes many tens of thousands of automated tests using dozens of
different platforms.  There are literally dozens of machines (I would
guess 70 or so total) running around the clock for 7 days before we even
know if we have a release candidate.  The QA team is distinct from the
development team, and if they say "FLOP!" the release goes nowhere.  No
formal release until QA passes it.

If there is no procedure for PostgreSQL of this nature, then there
really needs to be.  I am sure that MySQL must have something in place
like that.  Their "Crash-Me" test suite shows (at least) that they have
put a large effort into testing.


Fri, 20 Jun 2003 15:39:47 -0700
"Dann Corbit" <DCorbit@connx.com>
Re: Two weeks to feature freeze
On Fri, Jun 20, 2003 at 03:39:47PM -0700, Dann Corbit wrote:

> We (at CONNX Solutions Inc.) have a formal release procedure that
> includes many tens of thousands of automated tests using dozens of
> different platforms.  [...]
> 
> If there is no procedure for PostgreSQL of this nature, then there
> really needs to be.  I am sure that MySQL must have something in place
> like that.  Their "Crash-Me" test suite shows (at least) that they have
> put a large effort into testing.

The regression testing suite in Postgres is one of the things that
impresses me about this software.  It's very rare that a change is even
committed to the main tree if a single regression test doesn't pass.
When it does, a proper fix is quickly put in or the change reverted.

It's even rare that patches with regression failures get posted in
pgsql-patches.  Regression tests are a very handy tool for contributors
to check that their work is "safe".  It's considered good practice to
submit new tests when there's new functionality in a patch.

There probably isn't such a gigantic effort like the one you describe,
but there certainly _is_ a testing procedure.  There's probably room for
improvement, of course, but we don't want the tests to take a full week
to complete, IMHO.

It would be nice to have a system which could receive a patch and
compile and verify that it passes the tests before it goes to Bruce's
queue; or compile on multiple platforms to check for portability
problems, for example.

-- 
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Uno puede defenderse de los ataques; contra los elogios se esta indefenso"

Fri, 20 Jun 2003 19:05:09 -0400
Alvaro Herrera <alvherre@dcc.uchile.cl>
Re: Two weeks to feature freeze
"Dann Corbit" <DCorbit@connx.com> writes:

>> -----Original Message-----
>> From: Jason Earl [mailto:jason.earl@simplot.com] 
>> Sent: Friday, June 20, 2003 3:32 PM
>> To: Dann Corbit
>> Cc: Jason Earl; The Hermit Hacker; PostgreSQL-development
>> Subject: Re: [HACKERS] Two weeks to feature freeze
>> 
>> 
>> "Dann Corbit" <DCorbit@connx.com> writes:
>> >> 
>> >> Why couldn't you just release the win32 version of 7.4 when
>> >> it was finished.  If it takes an extra month then that just 
>> >> gives you guys the chance to circulate *two* press releases.  
>> >> The Native Win32 port is likely to make a big enough splash 
>> >> all by itself.
>> >
>> > A formal release needs a big testing effort.  Two separate releases 
>> > will double the work of validation.
>> 
>> There are several problems with that statement.  The first is 
>> that PostgreSQL's "testing effort" happens right here on this 
>> mailing list. 
>
> That's not exactly reassuring.  There is no regression test that
> gets formal acceptance?!

Yes, there are regression tests, and new tests get invented all of the
time whenever the real world finds new bugs.  Regression tests are
excellent for making sure that you don't make the same mistake twice,
but they aren't a substitute for handing the code over to actual end
users.

>> The various PostgreSQL hackers code stuff up, 
>> and we try and break it. There's very little /effort/ 
>> involved.  People that want the new features go out on a limb 
>> and start using them.  If they don't work, then they bring it 
>> up on the list.  If they do work then very little gets said.
>> 
>> As it now stands Tom Lane is on the record as stating that 
>> the new Win32 version isn't going to be ready for production 
>> anyhow.  If anything the Win32 version *should* get released 
>> separately simply because we don't want people mistaking the 
>> Win32 version as being up to the PostgreSQL teams high 
>> standards.  Those people that want the Win32 version to 
>> become production ready are going to have to risk their 
>> precious data.  Otherwise, the Win32 version will likely 
>> remain a second class citizen forever.
>> 
>> The fact of the matter is that the Win32 specific bits are 
>> the parts that are likely to break in the new port.  If 
>> anything the Windows version will *benefit* from an earlier 
>> *nix release because the *nix users will chase down the bugs 
>> in the new PostgreSQL features.  Once the *nix version is up 
>> to 7.4.2 (or so) then a Windows release of 7.4.2 should allow 
>> the PostgreSQL hackers to simply chase down Windows specific problems.
>
> Then using the same logic, the new Windows version should wait
> indefinitely, since the *nix version will always be shaking out
> bugs.

That's not true at all.  Despite the excellent work by the PostgreSQL
team, and despite the beta testing that will be done by volunteers, if
history repeats itself, there will be problems with version 7.4.0,
even on platforms that have been well supported by PostgreSQL forever.
I am not saying that we should hold off indefinitely on the Win32
port, I am simply saying that it probably wouldn't hurt to shake out
the normal .0 release bugs before throwing the unique Win32 bugs into
the mix.

My guess is that reported Win32 bugs are going blamed on the Win32
specific bits at first no matter what happens.  Unless the bug can be
demonstrated on a *nix version it will be assumed that the problem is
a shortcoming of the Win32 specific code.  That's just common sense.

>> Adding a new platform--especially a platform as diverse from 
>> the rest of PostgreSQL's supported platforms as Windows--is 
>> what adds the work. Testing the new platform is relatively 
>> easy.  All you need to do is to start using the Win32 version 
>> with real live data.
>
> That is not testing.  Using the world as your beta team seems to be
> a business model used by a few software giants that is largely
> frowned upon.  I would think that there is an opportunity to do
> things differently. [Read 'properly'].

Hmm... I must have missed the huge corporation paying for in house
testing of PostgreSQL.  In the Free Software world the "beta team" is
all of those people that need the new features so badly that they are
willing to risk their own data and hardware testing it.  You might not
like the way that this sounds, but in practice it works astoundingly
well.  Chances are you can't name 25 pieces of commercial software
that run on the wide array of hardware platforms and OSes as
PostgreSQL, and PostgreSQL has a earned a well deserved reputation for
being a solid piece of software.  Clearly the PostgreSQL team is doing
*something* right.

> We (at CONNX Solutions Inc.) have a formal release procedure that
> includes many tens of thousands of automated tests using dozens of
> different platforms.  There are literally dozens of machines (I
> would guess 70 or so total) running around the clock for 7 days
> before we even know if we have a release candidate.  The QA team is
> distinct from the development team, and if they say "FLOP!" the
> release goes nowhere.  No formal release until QA passes it.

And yet when you release the software your customers invariably find
bugs, don't they?

Don't get me wrong.  I am all for testing, regression tests, and such,
but the fact of the matter is that there simply is no way that a
centralized authority could afford to really test PostgreSQL on even a
fraction of the supported platforms and configurations.  The way it
stands now the PostgreSQL teams gets the best testbed you could hope
for (the world) for the price of hosting a few web and FTP servers
(thanks Marc).

PostgreSQL betas almost certainly gest tested on an order of magnitude
more systems than the 70 that you boast about.  PostgreSQL gets tested
on everything from Sony Playstations to AMD Opterons to IBM
mainframes.  Heck, there are probably more than 70 machines running
CVS versions of PostgreSQL right this minute (Marc, any download
numbers to back this up?).  More importantly, PostgreSQL gets tested
on a wide variety of real world tasks, and not some lab application or
some test scripts.  Like I have mentioned several times before.
PostgreSQL gets tested by folks that put their actual data into the
beta versions and try it out.  Even with this sort of testing,
however, bugs still make it into the release version.  Even with a
large group of beta testers we simply can't test all of the possible
ways that the software might get used on every available platform.

> If there is no procedure for PostgreSQL of this nature, then there
> really needs to be.  I am sure that MySQL must have something in place
> like that.  Their "Crash-Me" test suite shows (at least) that they have
> put a large effort into testing.

Yow!  Have you read the crash-me script.  It's possible that they have
improved dramatically in the year or so since I last took a look at
them, but it used to be that MySQL's crash-me scripts were the worst
amalgamation of marketeering and poor relational theory ever conceived
by the human mind.  Basically the crash-me scripts were nothing more
than an attempt to put MySQL's competition in the worst light
possible.  Basically any time a competitor differed from MySQL an
error would be generated (despite the fact that it was very likely
that it was MySQL that was wrong).

MySQL even tried to pawn this single-process monstrosity off as a
"benchmark."  What a laugh.  It was a perfectly valid benchmark if
your database was designed to be used by one user at a time and one of
your biggest criteria was the time it took to create a valid
connection from a perl script.

PostgreSQL's regression tests (IMHO) are much better than MySQL's
crash-me scripts.

Jason

Fri, 20 Jun 2003 17:43:03 -0600
Jason Earl <jason.earl@simplot.com>
Re: Two weeks to feature freeze
"Dann Corbit" <DCorbit@connx.com> writes:
> If there is no procedure for PostgreSQL of this nature, then there
> really needs to be.

Are you volunteering to create it?  Step right up.

> I am sure that MySQL must have something in place
> like that.  Their "Crash-Me" test suite shows (at least) that they have
> put a large effort into testing.

...ROTFL...  Crash-Me is not a regression test.  It is a marketing
effort.

			regards, tom lane

Fri, 20 Jun 2003 23:36:23 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Two weeks to feature freeze 
Alvaro Herrera <alvherre@dcc.uchile.cl> writes:
> It would be nice to have a system which could receive a patch and
> compile and verify that it passes the tests before it goes to Bruce's
> queue; or compile on multiple platforms to check for portability
> problems, for example.

It happens not infrequently that Bruce commits some patch that fails the
regression tests.  Whereupon he gets chewed out by whoever notices it
first :-).  I've been guilty of the same on occasion, even though I try
hard to avoid it.  If the regression tests took two seconds to run I'm
sure we'd both set up scripts to ensure we *never* commit without
regression testing first.  On the other hand, if they took a week to run
you can be damn sure that most commits would go in without any
regression testing.  I think that our existing tests are a fairly happy
medium --- they catch most stuff, and the stuff they don't catch is in
my opinion stuff that an automated test is unlikely to catch.  (I do add
things to the regression tests whenever something shows up that looks
like it should have been caught.)

Another point is that passing on one platform doesn't ensure passing on
another.  Here we really rely on the willingness of the pghackers
community to update to CVS tip regularly and run the regression tests
when they do.  Again, tests that take a couple minutes to run are ideal;
if they took a week then the uptake would drop to zero, and we'd not be
ahead.

			regards, tom lane

Fri, 20 Jun 2003 23:49:04 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Two weeks to feature freeze 
Jason Earl <jason.earl@simplot.com> writes:
> Hmm... I must have missed the huge corporation paying for in house
> testing of PostgreSQL.  In the Free Software world the "beta team" is
> all of those people that need the new features so badly that they are
> willing to risk their own data and hardware testing it.

I don't have a lot of faith in huge automated test efforts.  They're
great at ensuring you don't make the same mistakes you made once before,
but in my experience the nastiest bugs are the ones you haven't seen
before and would never in a million years have dreamed to test for.
Thus, the best test team is a bunch of people doing unplanned things
with the software, on a wide variety of platforms...

			regards, tom lane

Fri, 20 Jun 2003 23:57:43 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Two weeks to feature freeze 
Tom Lane wrote:

>Alvaro Herrera <alvherre@dcc.uchile.cl> writes:
>  
>
>>It would be nice to have a system which could receive a patch and
>>compile and verify that it passes the tests before it goes to Bruce's
>>queue; or compile on multiple platforms to check for portability
>>problems, for example.
>>
*snip*

>Another point is that passing on one platform doesn't ensure passing on
>another.  Here we really rely on the willingness of the pghackers
>community to update to CVS tip regularly and run the regression tests
>when they do.  Again, tests that take a couple minutes to run are ideal;
>if they took a week then the uptake would drop to zero, and we'd not be
>ahead.
>  
>
Have you considered something similar to the Mozilla tinderbox approach 
where you have a daemon checkout the cvs, compile, run regression tests, 
and report a status or be able to report a status?



Fri, 20 Jun 2003 23:03:01 -0500
Thomas Swan <tswan@idigx.com>
Re: Two weeks to feature freeze
Thomas Swan <tswan@idigx.com> writes:
> Have you considered something similar to the Mozilla tinderbox approach 
> where you have a daemon checkout the cvs, compile, run regression tests, 
> and report a status or be able to report a status?

Tinderbox is pretty cool.  Who wants to set it up?

			regards, tom lane

Sat, 21 Jun 2003 00:12:18 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Two weeks to feature freeze 
> We (at CONNX Solutions Inc.) have a formal release procedure that
> includes many tens of thousands of automated tests using dozens of
> different platforms.  There are literally dozens of machines (I would
> guess 70 or so total) running around the clock for 7 days before we even
> know if we have a release candidate.  The QA team is distinct from the
> development team, and if they say "FLOP!" the release goes nowhere.  No
> formal release until QA passes it.

PostgreSQL has a comprehensive regression suite that is run by the
developers all the time...

> If there is no procedure for PostgreSQL of this nature, then there
> really needs to be.  I am sure that MySQL must have something in place
> like that.  Their "Crash-Me" test suite shows (at least) that they have
> put a large effort into testing.

No, it means they've put a crap effort into trying to make other databases
look bad...

Chris



Sat, 21 Jun 2003 13:13:32 +0800 (WST)
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: Two weeks to feature freeze
> I don't have a lot of faith in huge automated test efforts.  They're
> great at ensuring you don't make the same mistakes you made once before,
> but in my experience the nastiest bugs are the ones you haven't seen
> before and would never in a million years have dreamed to test for.
> Thus, the best test team is a bunch of people doing unplanned things
> with the software, on a wide variety of platforms...

Which is why I never use a .0 release of PostgreSQL :)

Chris



Sat, 21 Jun 2003 13:19:19 +0800 (WST)
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: Two weeks to feature freeze 
Thomas Swan writes:

> Have you considered something similar to the Mozilla tinderbox approach
> where you have a daemon checkout the cvs, compile, run regression tests,
> and report a status or be able to report a status?

Even if you could achieve near complete coverage of the platforms,
platform versions, and auxilliary software versions and combinations that
PostgreSQL runs with, in most cases, something breaks on a new
version or combination of these things.

-- 
Peter Eisentraut   peter_e@gmx.net


Sat, 21 Jun 2003 14:51:17 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Two weeks to feature freeze
On Fri, 20 Jun 2003, Jason Earl wrote:

> Heck, there are probably more than 70 machines running
> CVS versions of PostgreSQL right this minute (Marc, any download
> numbers to back this up?).

Unfortunately, most ppl testing would be using CVS or CVSup, which don't
(or, at least, I haven't been able to find?) log such ..

download wise, through the FTP site, maybe 20 downloads since the 4th of
June of the snapshot ...


Sat, 21 Jun 2003 12:01:13 -0300 (ADT)
The Hermit Hacker <scrappy@postgresql.org>
Re: Two weeks to feature freeze
Peter Eisentraut <peter_e@gmx.net> writes:
> Thomas Swan writes:
>> Have you considered something similar to the Mozilla tinderbox approach
>> where you have a daemon checkout the cvs, compile, run regression tests,
>> and report a status or be able to report a status?

> Even if you could achieve near complete coverage of the platforms,
> platform versions, and auxilliary software versions and combinations that
> PostgreSQL runs with, in most cases, something breaks on a new
> version or combination of these things.

Still, whenever we're doing something that interacts at all with the OS,
it seems we get breakages that don't show in the original author's
testing, but only pop up days to months later when some beta tester
tries the code on platform P or using option Q.  The current
difficulties with the IPv6 patches are a fine case in point.
If we could get feedback more easily about whether a proposed patch
compiles and passes regression on a variety of platforms, we could
reduce the pain involved by a great deal, simply because the problems
could be fixed while the code is still fresh in mind.

I don't think there is any company involved with Postgres that is
willing to commit the resources to run a Mozilla-style tinderbox setup
singlehanded.  But I wonder whether we couldn't set up something that is
community-based: get a few dozen people with different platforms to
volunteer to check the code regularly on their own machines.  I'm
imagining a cron job that fires daily in the wee hours, pulls the latest
CVS tip, does "make distclean; configure; make; make check", and mails
the results to someplace that puts 'em up on our website.

It's possible that we could adapt the tinderbox software to work this
way, but even if we had to write our own, it seems like a fairly simple
task.  And it'd give *much* better feedback on porting problems than we
have now.  Sure, there will always be corner cases you don't catch,
but the first rule of testing is the sooner you find a bug the cheaper
it is to fix.

			regards, tom lane

Sat, 21 Jun 2003 11:43:17 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Two weeks to feature freeze 


--On Saturday, June 21, 2003 11:43:17 -0400 Tom Lane <tgl@sss.pgh.pa.us> 
wrote:

> Peter Eisentraut <peter_e@gmx.net> writes:
>> Thomas Swan writes:
>>> Have you considered something similar to the Mozilla tinderbox approach
>>> where you have a daemon checkout the cvs, compile, run regression tests,
>>> and report a status or be able to report a status?
>
>> Even if you could achieve near complete coverage of the platforms,
>> platform versions, and auxilliary software versions and combinations that
>> PostgreSQL runs with, in most cases, something breaks on a new
>> version or combination of these things.
>
> Still, whenever we're doing something that interacts at all with the OS,
> it seems we get breakages that don't show in the original author's
> testing, but only pop up days to months later when some beta tester
> tries the code on platform P or using option Q.  The current
> difficulties with the IPv6 patches are a fine case in point.
> If we could get feedback more easily about whether a proposed patch
> compiles and passes regression on a variety of platforms, we could
> reduce the pain involved by a great deal, simply because the problems
> could be fixed while the code is still fresh in mind.
>
> I don't think there is any company involved with Postgres that is
> willing to commit the resources to run a Mozilla-style tinderbox setup
> singlehanded.  But I wonder whether we couldn't set up something that is
> community-based: get a few dozen people with different platforms to
> volunteer to check the code regularly on their own machines.  I'm
> imagining a cron job that fires daily in the wee hours, pulls the latest
> CVS tip, does "make distclean; configure; make; make check", and mails
> the results to someplace that puts 'em up on our website.
>
> It's possible that we could adapt the tinderbox software to work this
> way, but even if we had to write our own, it seems like a fairly simple
> task.  And it'd give *much* better feedback on porting problems than we
> have now.  Sure, there will always be corner cases you don't catch,
> but the first rule of testing is the sooner you find a bug the cheaper
> it is to fix.
>
> 			regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>

I'm willing to run such a job on UnixWare 7.1.3 and OpenUnix 8, as well
as FreeBSD 4.8



-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749




Sat, 21 Jun 2003 10:48:19 -0500
Larry Rosenman <ler@lerctr.org>
Re: Two weeks to feature freeze 
Larry Rosenman wrote:

>
>
> --On Saturday, June 21, 2003 11:43:17 -0400 Tom Lane 
> <tgl@sss.pgh.pa.us> wrote:
>
>> Peter Eisentraut <peter_e@gmx.net> writes:
>>
>>> Thomas Swan writes:
>>>
>>>> Have you considered something similar to the Mozilla tinderbox 
>>>> approach
>>>> where you have a daemon checkout the cvs, compile, run regression 
>>>> tests,
>>>> and report a status or be able to report a status?
>>>
>>
>>> Even if you could achieve near complete coverage of the platforms,
>>> platform versions, and auxilliary software versions and combinations 
>>> that
>>> PostgreSQL runs with, in most cases, something breaks on a new
>>> version or combination of these things.
>>
>>
>> Still, whenever we're doing something that interacts at all with the OS,
>> it seems we get breakages that don't show in the original author's
>> testing, but only pop up days to months later when some beta tester
>> tries the code on platform P or using option Q.  The current
>> difficulties with the IPv6 patches are a fine case in point.
>> If we could get feedback more easily about whether a proposed patch
>> compiles and passes regression on a variety of platforms, we could
>> reduce the pain involved by a great deal, simply because the problems
>> could be fixed while the code is still fresh in mind.
>>
>> I don't think there is any company involved with Postgres that is
>> willing to commit the resources to run a Mozilla-style tinderbox setup
>> singlehanded.  But I wonder whether we couldn't set up something that is
>> community-based: get a few dozen people with different platforms to
>> volunteer to check the code regularly on their own machines.  I'm
>> imagining a cron job that fires daily in the wee hours, pulls the latest
>> CVS tip, does "make distclean; configure; make; make check", and mails
>> the results to someplace that puts 'em up on our website.
>>
>> It's possible that we could adapt the tinderbox software to work this
>> way, but even if we had to write our own, it seems like a fairly simple
>> task.  And it'd give *much* better feedback on porting problems than we
>> have now.  Sure, there will always be corner cases you don't catch,
>> but the first rule of testing is the sooner you find a bug the cheaper
>> it is to fix.
>>
>>             regards, tom lane
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 2: you can get off all lists at once with the unregister command
>>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>>
>
> I'm willing to run such a job on UnixWare 7.1.3 and OpenUnix 8, as well
> as FreeBSD 4.8
>
>
>
I'll have a machine shortly where I can run RH9 SMP tests..


Sat, 21 Jun 2003 15:50:38 -0500
Thomas Swan <tswan@idigx.com>
Re: Two weeks to feature freeze

----- Original Message ----- 
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Peter Eisentraut" <peter_e@gmx.net>
Cc: "Thomas Swan" <tswan@idigx.com>; <pgsql-hackers@postgresql.org>
Sent: Saturday, June 21, 2003 11:43 AM
Subject: Re: [HACKERS] Two weeks to feature freeze


> I don't think there is any company involved with Postgres that is
> willing to commit the resources to run a Mozilla-style tinderbox setup
> singlehanded.  But I wonder whether we couldn't set up something that is
> community-based: get a few dozen people with different platforms to
> volunteer to check the code regularly on their own machines.  I'm
> imagining a cron job that fires daily in the wee hours, pulls the latest
> CVS tip, does "make distclean; configure; make; make check", and mails
> the results to someplace that puts 'em up on our website.
>
> It's possible that we could adapt the tinderbox software to work this
> way, but even if we had to write our own, it seems like a fairly simple
> task.  And it'd give *much* better feedback on porting problems than we
> have now.  Sure, there will always be corner cases you don't catch,
> but the first rule of testing is the sooner you find a bug the cheaper
> it is to fix.
>

Two thoughts:
1. we'd need a matrix of hardware / (OS/version) / other environmental
things to ensure some sort of good coverage.
2. we'd need to test various configuration sets too, e.g. --with-krb5

I too have an old spare x86 machine lying around that I can set up with
whatever free *nix might not have coverage and contribute to the effort.

andrew


Sat, 21 Jun 2003 18:50:06 -0400
"Andrew Dunstan" <andrew@dunslane.net>
Re: Two weeks to feature freeze 
Dann Corbit wrote:
> > Adding a new platform--especially a platform as diverse from 
> > the rest of PostgreSQL's supported platforms as Windows--is 
> > what adds the work. Testing the new platform is relatively 
> > easy.  All you need to do is to start using the Win32 version 
> > with real live data.
> 
> That is not testing.  Using the world as your beta team seems to be a
> business model used by a few software giants that is largely frowned
> upon.  I would think that there is an opportunity to do things
> differently. [Read 'properly'].
> 
> We (at CONNX Solutions Inc.) have a formal release procedure that
> includes many tens of thousands of automated tests using dozens of
> different platforms.  There are literally dozens of machines (I would
> guess 70 or so total) running around the clock for 7 days before we even
> know if we have a release candidate.  The QA team is distinct from the
> development team, and if they say "FLOP!" the release goes nowhere.  No
> formal release until QA passes it.
> 
> If there is no procedure for PostgreSQL of this nature, then there
> really needs to be.  I am sure that MySQL must have something in place
> like that.  Their "Crash-Me" test suite shows (at least) that they have
> put a large effort into testing.

One thing you might be missing is that we have a _very_ close
relationship with our users.  We can send out code and debug/fix things
much faster than a company can that ships binaries.  If you look at the
changes that go into minor releases (post X.X.0 releases) you will see
very few fixes, and the ones we do fine are usually for very esoteric
problem cases.  Maybe it isn't ideal, but given our limited resources,
it works very well.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sat, 21 Jun 2003 23:43:09 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze
Tom Lane wrote:

>Peter Eisentraut <peter_e@gmx.net> writes:
>  
>
>>Thomas Swan writes:
>>    
>>
>>>Have you considered something similar to the Mozilla tinderbox approach
>>>where you have a daemon checkout the cvs, compile, run regression tests,
>>>and report a status or be able to report a status?
>>>      
>>>
>
>  
>
>>Even if you could achieve near complete coverage of the platforms,
>>platform versions, and auxilliary software versions and combinations that
>>PostgreSQL runs with, in most cases, something breaks on a new
>>version or combination of these things.
>>    
>>
>
>Still, whenever we're doing something that interacts at all with the OS,
>it seems we get breakages that don't show in the original author's
>testing, but only pop up days to months later when some beta tester
>tries the code on platform P or using option Q.  The current
>difficulties with the IPv6 patches are a fine case in point.
>If we could get feedback more easily about whether a proposed patch
>compiles and passes regression on a variety of platforms, we could
>reduce the pain involved by a great deal, simply because the problems
>could be fixed while the code is still fresh in mind.
>
>I don't think there is any company involved with Postgres that is
>willing to commit the resources to run a Mozilla-style tinderbox setup
>singlehanded.  But I wonder whether we couldn't set up something that is
>community-based: get a few dozen people with different platforms to
>volunteer to check the code regularly on their own machines.  I'm
>imagining a cron job that fires daily in the wee hours, pulls the latest
>CVS tip, does "make distclean; configure; make; make check", and mails
>the results to someplace that puts 'em up on our website.
>
>It's possible that we could adapt the tinderbox software to work this
>way, but even if we had to write our own, it seems like a fairly simple
>task.  And it'd give *much* better feedback on porting problems than we
>have now.  Sure, there will always be corner cases you don't catch,
>but the first rule of testing is the sooner you find a bug the cheaper
>it is to fix.	
>  
>
Is it possible the sourceforge compile farms could be used for some of 
the automated testing?  I'm not sure how that system works, but it could 
be worth looking into.

>			regards, tom lane
>
>---------------------------(end of broadcast)---------------------------
>TIP 2: you can get off all lists at once with the unregister command
>    (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>  
>



Thu, 26 Jun 2003 11:51:32 -0500
Thomas Swan <tswan@idigx.com>
Re: Two weeks to feature freeze
Thomas Swan <tswan@idigx.com> writes:
> Is it possible the sourceforge compile farms could be used for some of 
> the automated testing?  I'm not sure how that system works, but it could 
> be worth looking into.

The last time I used it (which admittedly was a year or two back), they
didn't really have a very wide range of platforms available.  And I
don't think they were offering cron access on the farm machines, so this
would be much more painful to automate than work done locally on
contributors' machines.

			regards, tom lane

Thu, 26 Jun 2003 13:08:21 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Two weeks to feature freeze 

On Thu, 26 Jun 2003, Thomas Swan wrote:
> >
> Is it possible the sourceforge compile farms could be used for some of 
> the automated testing?  I'm not sure how that system works, but it could 
> be worth looking into.

Isn't the sourceforge license very scary and along the lines of "whatever you
put on here we own it's just we tend not to persue that at the moment as
there's not much money in it for us but that doesn't stop us from claiming it 
at some indeterminate time in the future"?


-- 
Nigel J. Andrews


Thu, 26 Jun 2003 18:08:35 +0100 (BST)
"Nigel J. Andrews" <nandrews@investsystems.co.uk>
Re: Two weeks to feature freeze
Nigel J. Andrews wrote:

>On Thu, 26 Jun 2003, Thomas Swan wrote:
>  
>
>>Is it possible the sourceforge compile farms could be used for some of 
>>the automated testing?  I'm not sure how that system works, but it could 
>>be worth looking into.
>>    
>>
>
>Isn't the sourceforge license very scary and along the lines of "whatever you
>put on here we own it's just we tend not to persue that at the moment as
>there's not much money in it for us but that doesn't stop us from claiming it 
>at some indeterminate time in the future"?
>
If it's that intrusive, then it was a bad idea.  But, I didn't find
anything like that on their Terms of Use
<http://sourceforge.net/docman/display_doc.php?docid=6048&group_id=1>
page.  The compiler farm has a relatively small number of platforms, but
perhaps it would be enough to get started with at least verifying an
automated test would work. See Guide to the Sourceforge Compile Farm
<http://sourceforge.net/docman/display_doc.php?docid=762&group_id=1>.

In terms of implementation, I was thinking of something like the following.

    * clean the source, destination directories
    * pull latest CVS tip down.
    * record environment / installed packages
    * loop - on different options ( w/ or w/o krb5, w/ or w/o ssl, etc. )
          o make clean
          o configure with sets of options
          o compile
                + log messages
                + analyze errors ( perhaps gather statitistics:
                  warnings, failures, notices, etc.)
          o (run / install) if successful
          o run tests
                + output results (perhaps to HTML)
                + compare results with expected
                + record differences if any | gather aggregate information
          o uninstall  / clean up
    * end loop

Perhaps there could be an occasion where the test would be able to put
in a corrupt WAL or a corrupt table to do regression tests for recovery
of errors.

Of course, these are just ideas and I'm not sure how practical it is to
do any of them.  I just am really concerned about the uninstall/clean up
phase and how that can be done in an orderly fashion.   Unless the
process can start from a clean state again, then it won't be valid.   At
one point I had even given thought, vainly,  to purchasing VMWare for
such an occasion.  Suggestions?



Thu, 26 Jun 2003 13:48:01 -0500
Thomas Swan <tswan@idigx.com>
Re: Two weeks to feature freeze
I know I'm new to this list, but is OSDL's testing capabilities out of
the question?


On Thu, 2003-06-26 at 13:48, Thomas Swan wrote:
> Nigel J. Andrews wrote:
> 
> >On Thu, 26 Jun 2003, Thomas Swan wrote:
> >  
> >
> >>Is it possible the sourceforge compile farms could be used for some of 
> >>the automated testing?  I'm not sure how that system works, but it could 
> >>be worth looking into.
> >>    
> >>
> >
> >Isn't the sourceforge license very scary and along the lines of "whatever you
> >put on here we own it's just we tend not to persue that at the moment as
> >there's not much money in it for us but that doesn't stop us from claiming it 
> >at some indeterminate time in the future"?
> >
> If it's that intrusive, then it was a bad idea.  But, I didn't find
> anything like that on their Terms of Use
> <http://sourceforge.net/docman/display_doc.php?docid=6048&group_id=1>
> page.  The compiler farm has a relatively small number of platforms, but
> perhaps it would be enough to get started with at least verifying an
> automated test would work. See Guide to the Sourceforge Compile Farm
> <http://sourceforge.net/docman/display_doc.php?docid=762&group_id=1>.
...
-- 
Austin Gonyou <austin@coremetrics.com>
Coremetrics, Inc.

26 Jun 2003 14:00:58 -0500
Austin Gonyou <austin@coremetrics.com>
Re: Two weeks to feature freeze
On Thu, 2003-06-26 at 15:00, Austin Gonyou wrote:
> I know I'm new to this list, but is OSDL's testing capabilities out of
> the question?

From what I've seen, OSDL is only concerned with a very very small set
of platforms (Linux in a couple of configurations).

-- 
Rod Taylor <rbt@rbt.ca>

PGP Key: http://www.rbt.ca/rbtpub.asc
26 Jun 2003 16:05:30 -0400
Rod Taylor <rbt@rbt.ca>
Re: Two weeks to feature freeze
>     * clean the source, destination directories
>     * pull latest CVS tip down.

Why tip?  Lets simply update the current source tree to the most current
of whatever branch they had checked out initially.

Running it on older stable branches is just as useful.

>     * record environment / installed packages

Virtually impossible, especially if people take to installing some items
by hand (we want to test them as well).  Recording the configure output
on the other hand would be quite useful.

>     * loop - on different options ( w/ or w/o krb5, w/ or w/o ssl, etc. )
>           o make clean
>           o configure with sets of options
>           o compile
>                 + log messages
>                 + analyze errors ( perhaps gather statitistics:
>                   warnings, failures, notices, etc.)

Any warnings, failures, etc. aside from what is in the 'known' file
should be reported -- makes sense.

>           o (run / install) if successful
>           o run tests
>                 + output results (perhaps to HTML)
>                 + compare results with expected
>                 + record differences if any | gather aggregate information

Standard regression tests should suffice.  Any non-ignored errors
reported.

>           o uninstall  / clean up

Skip this.  Cleanup prior to the run.  If something is wrong, we may
need additional information from the environment.

>     * end loop
> 
> Perhaps there could be an occasion where the test would be able to put
> in a corrupt WAL or a corrupt table to do regression tests for recovery
> of errors.

These would be interesting extensions to the standard regression suite
-- but perhaps require a flag

> Of course, these are just ideas and I'm not sure how practical it is to
> do any of them.  I just am really concerned about the uninstall/clean up
> phase and how that can be done in an orderly fashion.   Unless the
> process can start from a clean state again, then it won't be valid.   At

I've not tried, but if PostgreSQL can do an 'out of tree' compile it
could make it much easier.  That said, poor cleanup would be a result of
a broken make clean.

> one point I had even given thought, vainly,  to purchasing VMWare for
> such an occasion.  Suggestions?

Skip VMWare.  Find a few volunteers to cron the script (I'll volunteer).

I think we should replace Bruce's pgtest script with this one -- with an
argument to accept the email address to report to for FAILING cases. 
Success isn't very interesting if it runs regularly.

 
-- 
Rod Taylor <rbt@rbt.ca>

PGP Key: http://www.rbt.ca/rbtpub.asc
26 Jun 2003 16:28:23 -0400
Rod Taylor <rbt@rbt.ca>
Re: Two weeks to feature freeze
On Thu, 26 Jun 2003, Thomas Swan wrote:

> Of course, these are just ideas and I'm not sure how practical it is to
> do any of them.  I just am really concerned about the uninstall/clean up
> phase and how that can be done in an orderly fashion.  Unless the
> process can start from a clean state again, then it won't be valid.  At
> one point I had even given thought, vainly, to purchasing VMWare for
> such an occasion.  Suggestions?

Personally ... if you could build up the test script, I think there are
enough ppl with more platforms on these lists that would be willing ot run
it ... the problem isn't getting the "farm" together, its coming up with
the automated (or even semi-automated) tests :(


Thu, 26 Jun 2003 22:15:13 -0300 (ADT)
The Hermit Hacker <scrappy@postgresql.org>
Re: Two weeks to feature freeze
On Thu, 26 Jun 2003, Rod Taylor wrote:

> I think we should replace Bruce's pgtest script with this one -- with an
> argument to accept the email address to report to for FAILING cases.
> Success isn't very interesting if it runs regularly.

that was why I suggested getting it into the tree ... to at least give a
starting point to work from ...

and I have at least one machine right now that I can run such tests on ...
Dual PII with FreeBSD 5.x-CURRENT on her ...

Thu, 26 Jun 2003 22:17:06 -0300 (ADT)
The Hermit Hacker <scrappy@postgresql.org>
Re: Two weeks to feature freeze
The Hermit Hacker wrote:

>On Thu, 26 Jun 2003, Thomas Swan wrote:
>
>  
>
>>Of course, these are just ideas and I'm not sure how practical it is to
>>do any of them.  I just am really concerned about the uninstall/clean up
>>phase and how that can be done in an orderly fashion.  Unless the
>>process can start from a clean state again, then it won't be valid.  At
>>one point I had even given thought, vainly, to purchasing VMWare for
>>such an occasion.  Suggestions?
>>    
>>
>
>Personally ... if you could build up the test script, I think there are
>enough ppl with more platforms on these lists that would be willing ot run
>it ... the problem isn't getting the "farm" together, its coming up with
>the automated (or even semi-automated) tests :(
>
I'll see what I can do... my shell script skills are pretty good, but 
I'm not sure how to handle the noting changes in the gcc output.  My 
best guess is to just do it a couple of times and force something to 
change (make an intentional mistake) and see if it can catch it, or at 
least what changes.




Thu, 26 Jun 2003 20:54:04 -0500
Thomas Swan <tswan@idigx.com>
Re: Two weeks to feature freeze
Rod Taylor <rbt@rbt.ca> writes:
> I've not tried, but if PostgreSQL can do an 'out of tree' compile it
> could make it much easier.

Yes it can, following the usual procedure for autoconfiscated trees:
just invoke configure from an empty directory, eg
	mkdir build
	cd build
	../pgsql/configure

This is something that breaks regularly because few of the key
developers use it :-(.  If there were automatic tests that used that
build setup, it would be a good thing 'cause it'd keep us honest.

> That said, poor cleanup would be a result of
> a broken make clean.

Not to worry, the developers will notice that case fast enough.
I think the auto test script should just rm -rf build and then
proceed as above.

			regards, tom lane

Thu, 26 Jun 2003 22:55:45 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Two weeks to feature freeze 
On Thu, 26 Jun 2003 22:55:45 -0400, Tom Lane <tgl@sss.pgh.pa.us>
wrote:
>> I've not tried, but if PostgreSQL can do an 'out of tree' compile it
>> could make it much easier.
>
>Yes it can, following the usual procedure for autoconfiscated trees:
>just invoke configure from an empty directory, eg
>	mkdir build
>	cd build
>	../pgsql/configure

I do this all the time.  Works pretty well.  The only minor annoyance
is that some files are created in the source tree:

./src/backend/bootstrap/bootparse.c
./src/backend/bootstrap/bootscanner.c
./src/backend/bootstrap/bootstrap_tokens.h
./src/backend/parser/gram.c
./src/backend/parser/parse.h
./src/backend/parser/scan.c
./src/backend/utils/misc/guc-file.c
./src/bin/psql/sql_help.h
./src/interfaces/ecpg/preproc/pgc.c
./src/interfaces/ecpg/preproc/preproc.c
./src/interfaces/ecpg/preproc/preproc.h
./src/pl/plpgsql/src/pl.tab.h
./src/pl/plpgsql/src/pl_gram.c
./src/pl/plpgsql/src/pl_scan.c

This didn't itch enough to make me scratch ;-)

Servus
 Manfred

Fri, 27 Jun 2003 11:16:19 +0200
Manfred Koizar <mkoi-pg@aon.at>
'out of tree' compile (was: Two weeks to feature freeze)
Manfred Koizar <mkoi-pg@aon.at> writes:
> I do this all the time.  Works pretty well.  The only minor annoyance
> is that some files are created in the source tree:

Yeah, that's deliberate: those files are shipped in distribution
tarballs (so that tarball users don't have to have bison/flex/perl
to build from a source tarball).  So they need to be made in the
source tree.  Since they are platform-independent this isn't any
big problem for the normal uses of out-of-tree building.  You could
get rid of them I believe by doing a "make maintainer-clean", but
for testing purposes I suspect that's just a waste of cycles.  When
the underlying files haven't changed, there's no reason to regenerate
these files.

			regards, tom lane

Fri, 27 Jun 2003 10:17:44 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: 'out of tree' compile (was: Two weeks to feature freeze) 
Tom Lane writes:

> This is something that breaks regularly because few of the key
> developers use it :-(.  If there were automatic tests that used that
> build setup, it would be a good thing 'cause it'd keep us honest.

This should be included in 'make distcheck'.  I'm quite puzzled right now
why it doesn't to it already.  A DESTDIR installation should also be
tested there.  Once that is done, ./configure [options] && make distcheck
&& make maintainer-clean is a quite extensive test.  The only drawback is
that it is hard to isolate the log of the failing part (config.log, build
log, regression test log).

For those who don't know, 'make distcheck' builds a distribution, unpacks
it, configures, builds, and installs the temporary tree, runs the
regression tests, uninstalls, checks whether all files were correctly
uninstalled, and builds another distribution out of the temporary tree to
check whether the distribution is self-contained.

> > That said, poor cleanup would be a result of
> > a broken make clean.
>
> Not to worry, the developers will notice that case fast enough.

This is not my experience, but a check for this could be included in make
distcheck as well.

-- 
Peter Eisentraut   peter_e@gmx.net


Fri, 27 Jun 2003 16:31:54 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Two weeks to feature freeze 
Thomas Swan writes:

> I just am really concerned about the uninstall/clean up phase and how
> that can be done in an orderly fashion.  Unless the process can start
> from a clean state again, then it won't be valid.

The only clean state is if you remove the entire source tree and check it
out again.  (Of course to save bandwidth, you copy the checked out source
tree to a temporary location, do your testing, and then remove that
temporary tree.)  Relying on make clean or make uninstall is flawed,
because those are among the things you want to test.

-- 
Peter Eisentraut   peter_e@gmx.net


Fri, 27 Jun 2003 16:34:04 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Two weeks to feature freeze
Manfred Koizar writes:

> I do this all the time.  Works pretty well.  The only minor annoyance
> is that some files are created in the source tree:

This is intentional, because said files are prebuilt in the distribution,
so they always have to be in the source directory.

-- 
Peter Eisentraut   peter_e@gmx.net


Fri, 27 Jun 2003 16:35:05 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: 'out of tree' compile (was: Two weeks to feature
Peter Eisentraut wrote:

>Thomas Swan writes:
>
>  
>
>>I just am really concerned about the uninstall/clean up phase and how
>>that can be done in an orderly fashion.  Unless the process can start
>>from a clean state again, then it won't be valid.
>>    
>>
>
>The only clean state is if you remove the entire source tree and check it
>out again.  (Of course to save bandwidth, you copy the checked out source
>tree to a temporary location, do your testing, and then remove that
>temporary tree.)  Relying on make clean or make uninstall is flawed,
>because those are among the things you want to test.
>
That sounds plausible.   Should we let everything stay in the compilers
directory.   Something like the configure --prefix=$TEST_ROOT and that
way we can have the whole thing run as one user in one directory so that
system wide impact is minimal.    I guess what I'm concerned with is
running this on a clean system, and then leaving unknown artifacts
behind.   Can/does make install output each file it's copying and where
to.   Capturing that output would make life easier for clean up of
things installed outside of the work directory, and provide a more
controlled environment.


Fri, 27 Jun 2003 10:43:04 -0500
Thomas Swan <tswan@idigx.com>
Re: Two weeks to feature freeze