> -----Original Message-----
> From: Jason Earl [mailto:jason.earl@simplot.com] 
> Sent: Friday, June 20, 2003 4:43 PM
> To: Dann Corbit
> Cc: Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> "Dann Corbit" <DCorbit@connx.com> writes:
> 
> >> -----Original Message-----
> >> From: Jason Earl [mailto:jason.earl@simplot.com]
> >> Sent: Friday, June 20, 2003 3:32 PM
> >> To: Dann Corbit
> >> Cc: Jason Earl; The Hermit Hacker; PostgreSQL-development
> >> Subject: Re: [HACKERS] Two weeks to feature freeze
> >> 
> >> 
> >> "Dann Corbit" <DCorbit@connx.com> writes:
> >> >> 
> >> >> Why couldn't you just release the win32 version of 7.4 
> when it was 
> >> >> finished.  If it takes an extra month then that just gives you 
> >> >> guys the chance to circulate *two* press releases.
> >> >> The Native Win32 port is likely to make a big enough splash 
> >> >> all by itself.
> >> >
> >> > A formal release needs a big testing effort.  Two 
> separate releases
> >> > will double the work of validation.
> >> 
> >> There are several problems with that statement.  The first is
> >> that PostgreSQL's "testing effort" happens right here on this 
> >> mailing list. 
> >
> > That's not exactly reassuring.  There is no regression test 
> that gets 
> > formal acceptance?!
> 
> Yes, there are regression tests, and new tests get invented 
> all of the time whenever the real world finds new bugs.  
> Regression tests are excellent for making sure that you don't 
> make the same mistake twice, but they aren't a substitute for 
> handing the code over to actual end users.

After testing & QA, there is a beta period.  You don't hand beta code
over to actual end users.  In the corporate world it would be a clear
case of both negligence and incompetence.
 
> >> The various PostgreSQL hackers code stuff up,
> >> and we try and break it. There's very little /effort/ 
> >> involved.  People that want the new features go out on a limb 
> >> and start using them.  If they don't work, then they bring it 
> >> up on the list.  If they do work then very little gets said.
> >> 
> >> As it now stands Tom Lane is on the record as stating that
> >> the new Win32 version isn't going to be ready for production 
> >> anyhow.  If anything the Win32 version *should* get released 
> >> separately simply because we don't want people mistaking the 
> >> Win32 version as being up to the PostgreSQL teams high 
> >> standards.  Those people that want the Win32 version to 
> >> become production ready are going to have to risk their 
> >> precious data.  Otherwise, the Win32 version will likely 
> >> remain a second class citizen forever.
> >> 
> >> The fact of the matter is that the Win32 specific bits are
> >> the parts that are likely to break in the new port.  If 
> >> anything the Windows version will *benefit* from an earlier 
> >> *nix release because the *nix users will chase down the bugs 
> >> in the new PostgreSQL features.  Once the *nix version is up 
> >> to 7.4.2 (or so) then a Windows release of 7.4.2 should allow 
> >> the PostgreSQL hackers to simply chase down Windows 
> specific problems.
> >
> > Then using the same logic, the new Windows version should wait 
> > indefinitely, since the *nix version will always be shaking 
> out bugs.
> 
> That's not true at all.  Despite the excellent work by the 
> PostgreSQL team, and despite the beta testing that will be 
> done by volunteers, if history repeats itself, there will be 
> problems with version 7.4.0, even on platforms that have been 
> well supported by PostgreSQL forever. I am not saying that we 
> should hold off indefinitely on the Win32 port, I am simply 
> saying that it probably wouldn't hurt to shake out the normal 
> .0 release bugs before throwing the unique Win32 bugs into the mix.
> 
> My guess is that reported Win32 bugs are going blamed on the 
> Win32 specific bits at first no matter what happens.  Unless 
> the bug can be demonstrated on a *nix version it will be 
> assumed that the problem is a shortcoming of the Win32 
> specific code.  That's just common sense.

You are right that a new feature will add new bugs.  I am saying that
the Win32 port is a new feature that will need a shakedown, but the
shakedown should occur in the testing and beta phase, like any other
feature.
 
> >> Adding a new platform--especially a platform as diverse from
> >> the rest of PostgreSQL's supported platforms as Windows--is 
> >> what adds the work. Testing the new platform is relatively 
> >> easy.  All you need to do is to start using the Win32 version 
> >> with real live data.
> >
> > That is not testing.  Using the world as your beta team 
> seems to be a 
> > business model used by a few software giants that is 
> largely frowned 
> > upon.  I would think that there is an opportunity to do things 
> > differently. [Read 'properly'].
> 
> Hmm... I must have missed the huge corporation paying for in 
> house testing of PostgreSQL.  In the Free Software world the 
> "beta team" is all of those people that need the new features 
> so badly that they are willing to risk their own data and 
> hardware testing it.  

I don't see how this model can possibly succeed then.  You can't just
hope that your end users will:
1.  Exhaustively test
2.  Accurately report the findings

> You might not like the way that this 
> sounds, but in practice it works astoundingly well.  Chances 
> are you can't name 25 pieces of commercial software that run 
> on the wide array of hardware platforms and OSes as 
> PostgreSQL, and PostgreSQL has a earned a well deserved 
> reputation for being a solid piece of software.  Clearly the 
> PostgreSQL team is doing
> *something* right.

I don't argue that PostgreSQL is a good piece of software.  I happen to
like it very much and have been a staunch advocate for its use with our
commercial products as well as in house.  What I am saying is that it
may be possible to improve the process.

If the corporate world knew that the only testing applied to PostgreSQL
was ad-hoc, I doubt that it would be accepted at all anywhere.  The fact
that PostgreSQL does succeed shows that the installed base of users must
be highly intelligent and highly motivated.
 
> > We (at CONNX Solutions Inc.) have a formal release procedure that 
> > includes many tens of thousands of automated tests using dozens of 
> > different platforms.  There are literally dozens of 
> machines (I would 
> > guess 70 or so total) running around the clock for 7 days before we 
> > even know if we have a release candidate.  The QA team is distinct 
> > from the development team, and if they say "FLOP!" the release goes 
> > nowhere.  No formal release until QA passes it.
> 
> And yet when you release the software your customers 
> invariably find bugs, don't they?

Our beta customers do help us to find bugs.  Bugs reported by customers
for released products are extremely rare.  The total issue count is 2495
in our bug tracking database (active since the late 1980's).  There are
82 issues found by the customers in that list, and 7 with an issue ID
over 2000 (recent issues).  Our code base is several hundred thousand
lines of code, and we have many thousands of customers world-wide.  When
I first started here, testing was less rigorous, and largely done by the
programmers instead of separate testing teams.  Since formal testing
procedures have been established, technical support incidents have gone
way down and quality has gone way up.

> Don't get me wrong.  I am all for testing, regression tests, 
> and such, but the fact of the matter is that there simply is 
> no way that a centralized authority could afford to really 
> test PostgreSQL on even a fraction of the supported platforms 
> and configurations.  The way it stands now the PostgreSQL 
> teams gets the best testbed you could hope for (the world) 
> for the price of hosting a few web and FTP servers (thanks Marc).
> 
> PostgreSQL betas almost certainly gest tested on an order of 
> magnitude more systems than the 70 that you boast about.

Maybe it does.  Maybe it doesn't.  You have no way of knowing, since no
formal reporting procedure exists.
  
> PostgreSQL gets tested on everything from Sony Playstations 
> to AMD Opterons to IBM mainframes.  Heck, there are probably 
> more than 70 machines running CVS versions of PostgreSQL 
> right this minute (Marc, any download numbers to back this 
> up?). 

If your count all the end-users workstations, our products have millions
of seats.  We run on UNIX (Solaris, Linux, AIX, Tru64, HP/UX, etc.) and
OpenVMS and MVS and Win32 and OS/400 and others.  As you can well
imagine, we *must* have an enormous testing effort.

> More importantly, PostgreSQL gets tested on a wide 
> variety of real world tasks, and not some lab application or 
> some test scripts. 

Spoken like a programmer.  Yes, real world data *always* turns up things
that neither the testers nor the programmers imagined.  But a huge and
comprehensive conformance testing effort will turn up 99% of the
problems.

> Like I have mentioned several times 
> before. PostgreSQL gets tested by folks that put their actual 
> data into the beta versions and try it out.  Even with this 
> sort of testing, however, bugs still make it into the release 
> version. 

Bug cost as a function of discovery stage is exponential.
1.  Discovered in design phase: nearly free to fix (designer sees bug,
designer fixes bug)
2.  Discovered in unit test phase: very cheap to fix (programmer sees
bug, programmer fixes bug)
3.  Discovered in integration test phase: inexpensive to fix (other
engineers become involved)
4.  Discovered in beta test phase: expensive to fix (customers,
tech-support, sales, programmers, engineers involved)
5.  Discovered in release: catastrophic cost to fix (as above, but now
every single customer must be upgraded, tens of thousands of dollars
lost, minimum -- possibly millions)

> Even with a large group of beta testers we simply 
> can't test all of the possible ways that the software might 
> get used on every available platform.

100% code coverage is impossible.
Program proving is impossible.
0% defect code delivery is impossible.

But you should try to approach the ideal as closely as can be attained.
 
> > If there is no procedure for PostgreSQL of this nature, then there 
> > really needs to be.  I am sure that MySQL must have 
> something in place 
> > like that.  Their "Crash-Me" test suite shows (at least) that they 
> > have put a large effort into testing.
> 
> Yow!  Have you read the crash-me script.  It's possible that 
> they have improved dramatically in the year or so since I 
> last took a look at them, but it used to be that MySQL's 
> crash-me scripts were the worst amalgamation of marketeering 
> and poor relational theory ever conceived by the human mind.  

The tests are good tests.  They cover a wide range of features and
functions and discover if you can cause permanent damage to a database
by simply performing end-user queries.  The scripts are a bit hokey, but
it isn't all that difficult to get them to run.

> Basically the crash-me scripts were nothing more than an 
> attempt to put MySQL's competition in the worst light 
> possible. 

I disagree.  In fact, in their matrix, PostgreSQL looks remarkably good.
In fact, I would choose it over MySQL, if the only examination made was
of the information contained in the matrix (but nobody would really
drive a decision based on that data alone).

> Basically any time a competitor differed from 
> MySQL an error would be generated (despite the fact that it 
> was very likely that it was MySQL that was wrong).

This is unfair and untrue. (I have no connection whatsoever with the
MySQL group, BTW).
 
> MySQL even tried to pawn this single-process monstrosity off 
> as a "benchmark."  What a laugh.  It was a perfectly valid 
> benchmark if your database was designed to be used by one 
> user at a time and one of your biggest criteria was the time 
> it took to create a valid connection from a perl script.

You can call it a conformance benchmark.  It is not touted as a
performance benchmark.  And nobody would fall for it if it were, since
it does not contain time information.
 
> PostgreSQL's regression tests (IMHO) are much better than 
> MySQL's crash-me scripts.

They are less thorough in coverage, but not too bad.

Here is what I suggest:

PostgreSQL has an excellent programming team.  Why not try to recruit a
similar testing team?  I think it would strongly differentiate the tool
set from similar free stuff.

Perhaps all that is needed is some sort of automated, formal reporting
procedure.  For example, a large test set might be created that runs a
thorough regression feature list.  When the test completes, a data file
is emailed to some central repository, parsed, and stored in a database.

If the end-users can simply start some shell script and take off for the
weekend, then it would be possible to collect a large volume of data.

Fri, 20 Jun 2003 17:31:03 -0700
"Dann Corbit" <DCorbit@connx.com>
Re: Two weeks to feature freeze
--- Dann Corbit <DCorbit@connx.com> wrote:
> Why couldn't you just release the win32 version of 7.4 
> when it was finished.

I agree. Don't delay *nix release because of win32 port is not ready. To many
users win32 port is of marginal importance anyway.






__________________________________
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com

Fri, 20 Jun 2003 21:08:17 -0700 (PDT)
ow <oneway_111@yahoo.com>
Re: Two weeks to feature freeze
On Fri, 20 Jun 2003, Dann Corbit wrote:

> > Hmm... I must have missed the huge corporation paying for in
> > house testing of PostgreSQL.  In the Free Software world the
> > "beta team" is all of those people that need the new features
> > so badly that they are willing to risk their own data and
> > hardware testing it.
>
> I don't see how this model can possibly succeed then.  You can't just
> hope that your end users will:
> 1.  Exhaustively test
> 2.  Accurately report the findings

But it does, and has for 10 years now ...

> Our beta customers do help us to find bugs.  Bugs reported by customers
> for released products are extremely rare.

Check the past archives for the mailing lists ... our "bugs reported by
end users for released products" is extremely rare also, and *generally*
is a result of them doing something that nobody had thought to test for
before ...

> Spoken like a programmer.  Yes, real world data *always* turns up things
> that neither the testers nor the programmers imagined.  But a huge and
> comprehensive conformance testing effort will turn up 99% of the
> problems.

And ours do ... I don't believe I can recall us having a release where
we've had a stream of problem reports come flying in afterwards ... we
might get one or two from ppl that have hit a 'never before seen' bug,
that generally gets fixed very quickly ...

> 100% code coverage is impossible.
> Program proving is impossible.
> 0% defect code delivery is impossible.
>
> But you should try to approach the ideal as closely as can be attained.

And we do ...

> The tests are good tests.  They cover a wide range of features and
> functions and discover if you can cause permanent damage to a database
> by simply performing end-user queries.  The scripts are a bit hokey, but
> it isn't all that difficult to get them to run.

Well, if you would like to volunteer to run them against PostgreSQL, and
let us know what fails, we can let you know why said test is wrong in the
first place ... we've been through crash-me several times before, and
'fixing crash-me' was more work then it was worth ...

> > Basically any time a competitor differed from
> > MySQL an error would be generated (despite the fact that it
> > was very likely that it was MySQL that was wrong).
>
> This is unfair and untrue. (I have no connection whatsoever with the
> MySQL group, BTW).

Been there, done that ... even tried to get changes made to make the tests
more accurate ... it was like trying to move a mountain ...

> PostgreSQL has an excellent programming team.  Why not try to recruit a
> similar testing team?  I think it would strongly differentiate the tool
> set from similar free stuff.

Are you volunteering?  We already have a testing team we're happy with,
but if you would like to extend it with your resources, please feel free
to join in ...


Sat, 21 Jun 2003 12:15:22 -0300 (ADT)
The Hermit Hacker <scrappy@postgresql.org>
Re: Two weeks to feature freeze
Dann Corbit wrote:
> Perhaps all that is needed is some sort of automated, formal reporting
> procedure.  For example, a large test set might be created that runs a
> thorough regression feature list.  When the test completes, a data file
> is emailed to some central repository, parsed, and stored in a database.

I do have an automated build/initdb/regression that I run every night
and email the results to myself.


	[ "X$1" != "X-n" ] && CLEAN="-c" && shift
	
	. /etc/profile
	pgstop
	rm -r /u/pg/data
	# return command error value
	(pgmakeall $CLEAN 2>&1; echo "$?" > $TMP/ret) | 
		(tee  $TMP/0; exit `cat $TMP/ret`) &&
	aspg initdb &&
	pgstart &&
	newdb test &&
	cd /pg/test/regress &&
	gmake clean &&
	aspg gmake installcheck
	grep warning $TMP/0 | 
		grep -v setproctitle | 
		grep -v find_rule | 
		grep -v yy_flex_realloc |
		grep -v '\[javac\] 1 warning'

I also run this after I apply patches.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sat, 21 Jun 2003 23:47:13 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze

I have added a cleaned up version of this to CVS as src/tools/pgtest.

---------------------------------------------------------------------------

Bruce Momjian wrote:
> Dann Corbit wrote:
> > Perhaps all that is needed is some sort of automated, formal reporting
> > procedure.  For example, a large test set might be created that runs a
> > thorough regression feature list.  When the test completes, a data file
> > is emailed to some central repository, parsed, and stored in a database.
> 
> I do have an automated build/initdb/regression that I run every night
> and email the results to myself.
> 
> 
> 	[ "X$1" != "X-n" ] && CLEAN="-c" && shift
> 	
> 	. /etc/profile
> 	pgstop
> 	rm -r /u/pg/data
> 	# return command error value
> 	(pgmakeall $CLEAN 2>&1; echo "$?" > $TMP/ret) | 
> 		(tee  $TMP/0; exit `cat $TMP/ret`) &&
> 	aspg initdb &&
> 	pgstart &&
> 	newdb test &&
> 	cd /pg/test/regress &&
> 	gmake clean &&
> 	aspg gmake installcheck
> 	grep warning $TMP/0 | 
> 		grep -v setproctitle | 
> 		grep -v find_rule | 
> 		grep -v yy_flex_realloc |
> 		grep -v '\[javac\] 1 warning'
> 
> I also run this after I apply patches.
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
>       subscribe-nomail command to majordomo@postgresql.org so that your
>       message can get through to the mailing list cleanly
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sun, 22 Jun 2003 00:23:32 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze
Dann Corbit wrote:
>> PostgreSQL's regression tests (IMHO) are much better than 
>> MySQL's crash-me scripts.
> 
> They are less thorough in coverage, but not too bad.

Right, we are still missing this test that proves 10,000 CREATE+DROP 
TABLE will ensure that PostgreSQL is slower than MySQL, if you don't 
VACUUM the catalog ...

> 
> Here is what I suggest:
> 
> PostgreSQL has an excellent programming team.  Why not try to recruit a
> similar testing team?  I think it would strongly differentiate the tool
> set from similar free stuff.
> 
> Perhaps all that is needed is some sort of automated, formal reporting
> procedure.  For example, a large test set might be created that runs a
> thorough regression feature list.  When the test completes, a data file
> is emailed to some central repository, parsed, and stored in a database.

Here is what I suggest:

Get started :-)


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Sun, 22 Jun 2003 08:39:26 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: Two weeks to feature freeze
Bruce Momjian writes:

> I have added a cleaned up version of this to CVS as src/tools/pgtest.

This seems to be a platform-specific reimplementation of 'make clean; make
check'.  Why bother?

-- 
Peter Eisentraut   peter_e@gmx.net


Sun, 22 Jun 2003 21:57:52 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Two weeks to feature freeze
Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > I have added a cleaned up version of this to CVS as src/tools/pgtest.
> 
> This seems to be a platform-specific reimplementation of 'make clean; make
> check'.  Why bother?

Marc requested it.  Is there anything platform-specific except the greps?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Sun, 22 Jun 2003 18:17:36 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze

Let me add my two cents ..

I think something like PostgreSQL needs two test suites - an
acceptance test (to ensure that checkins don't break functionality)
and a regression test suite.

What we call the regression test suite is really an acceptance
test. Tom Lane is absolutely right in asserting that a test suite that
takes a week to run will mean that people won't test at
all. Personally, I can (and have for many years) tolerate acceptance
test suites that take upto an hour. 

The existence of such an acceptance test should not however obviate
the presence of a wider regression test suite. It should be fine to
have an entire suite of regression test buckets take a week to run,
because you only start running them once you have a release candidate
or equivalent. 

Of course, setting up a regression test suite takes effort. There is
no need, however, to spend umpteen amounts of time writing the
buckets. What can be done is to incrementally build it up. So whenever
we have a significant new feature, somebody (preferably not the key
developers) could take the time to set up a set of test cases that try
to test it thoroughly. It's okay if such a test bucket takes 10-15
minutes to run. Then this can get rolled up into the regression suite
while a very small "representative" test case makes it to the
acceptance test suite. 

Of course, in the open source world, these things take resources and
are not easy to do. 

I certainly think that the base regression test suite is great. We
have clearing the pgsql regression test a checkin requirement for
TelegraphCQ developers as our goal is to not break pgsql
functionality.

-- 
Pip-pip
Sailesh
http://www.cs.berkeley.edu/~sailesh


23 Jun 2003 01:07:59 +0200
Sailesh Krishnamurthy <sailesh@cs.berkeley.edu>
Re: Two weeks to feature freeze
On Sun, 22 Jun 2003, Bruce Momjian wrote:

> Peter Eisentraut wrote:
> > Bruce Momjian writes:
> >
> > > I have added a cleaned up version of this to CVS as src/tools/pgtest.
> >
> > This seems to be a platform-specific reimplementation of 'make clean; make
> > check'.  Why bother?
>
> Marc requested it.  Is there anything platform-specific except the greps?

Ya, the script looked like it did a bit more then just a 'make clean; make
check' ... doesn't it?


Sun, 22 Jun 2003 20:22:50 -0300 (ADT)
The Hermit Hacker <scrappy@postgresql.org>
Re: Two weeks to feature freeze
The Hermit Hacker writes:

> Ya, the script looked like it did a bit more then just a 'make clean; make
> check' ... doesn't it?

No.

-- 
Peter Eisentraut   peter_e@gmx.net


Mon, 23 Jun 2003 16:12:58 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Two weeks to feature freeze
Peter Eisentraut wrote:
> The Hermit Hacker writes:
> 
> > Ya, the script looked like it did a bit more then just a 'make clean; make
> > check' ... doesn't it?
> 
> No.

Well, it is a nice test template for people who aren't shell script
experts, and I have been in the habit of pushing stuff I use into /tools
so it is available for others.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 23 Jun 2003 12:44:20 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze
On Mon, 23 Jun 2003, Bruce Momjian wrote:

> Peter Eisentraut wrote:
> > The Hermit Hacker writes:
> > 
> > > Ya, the script looked like it did a bit more then just a 'make clean; make
> > > check' ... doesn't it?
> > 
> > No.
> 
> Well, it is a nice test template for people who aren't shell script
> experts, and I have been in the habit of pushing stuff I use into /tools
> so it is available for others.

Please keep pushing such scripts out.  They're a valuable learning tool 
for many beginners and a cost little in size to be included.


Mon, 23 Jun 2003 10:56:07 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Two weeks to feature freeze
Bruce Momjian writes:

> Well, it is a nice test template for people who aren't shell script
> experts, and I have been in the habit of pushing stuff I use into /tools
> so it is available for others.

I know and I'm not happy about it.  The PostgreSQL source tree isn't a
repository of everyone's favorite shell scripts.  There's an official
method to build and test a PostgreSQL installation.  If that is flawed or
incomplete, then let's talk about it.  But everyone pushing out their own
little test methodology without further consideration is not going to help
this discussion.

-- 
Peter Eisentraut   peter_e@gmx.net


Mon, 23 Jun 2003 21:06:38 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Two weeks to feature freeze
Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > Well, it is a nice test template for people who aren't shell script
> > experts, and I have been in the habit of pushing stuff I use into /tools
> > so it is available for others.
> 
> I know and I'm not happy about it.  The PostgreSQL source tree isn't a
> repository of everyone's favorite shell scripts.  There's an official
> method to build and test a PostgreSQL installation.  If that is flawed or
> incomplete, then let's talk about it.  But everyone pushing out their own
> little test methodology without further consideration is not going to help
> this discussion.

I put stuff in /tools so if something happens to me, you guys can keep
going.  Do I have to be clearer that that?  I have RELEASE_CHANGES,
which Tom used for 7.3.X releases, pgindent, stuff for finding
missing/extraneous includes, static requirements, stuff like that.

Unless you can find someone else who agrees with you, it stays.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 23 Jun 2003 15:14:49 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze
On Mon, 23 Jun 2003, Bruce Momjian wrote:

> Peter Eisentraut wrote:
> > Bruce Momjian writes:
> > 
> > > Well, it is a nice test template for people who aren't shell script
> > > experts, and I have been in the habit of pushing stuff I use into /tools
> > > so it is available for others.
> > 
> > I know and I'm not happy about it.  The PostgreSQL source tree isn't a
> > repository of everyone's favorite shell scripts.  There's an official
> > method to build and test a PostgreSQL installation.  If that is flawed or
> > incomplete, then let's talk about it.  But everyone pushing out their own
> > little test methodology without further consideration is not going to help
> > this discussion.
> 
> I put stuff in /tools so if something happens to me, you guys can keep
> going.  Do I have to be clearer that that?  I have RELEASE_CHANGES,
> which Tom used for 7.3.X releases, pgindent, stuff for finding
> missing/extraneous includes, static requirements, stuff like that.
> 
> Unless you can find someone else who agrees with you, it stays.

Peter is coming off awfully paternalistic here.  I'd rather have a few 
extra scripts to look through to find what I need when I'm trying to 
figure out something than to have a tool that only the hackers know exists 
and I can only get by asking nicely to see the pretty code.

While no one wants to see a contrib or tool directory of a hundred megs, 
lots of little example files and testing scripts can be nice for nothing 
else other than the examples they provide.  I learned a lot when I first 
started using pgsql from the things in the contrib directory.


Mon, 23 Jun 2003 13:27:40 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Two weeks to feature freeze
Bruce Momjian writes:

> I put stuff in /tools so if something happens to me, you guys can keep
> going.

Yes, we keep going with make clean; make check, like everyone else.  Why
don't you consider using that?

-- 
Peter Eisentraut   peter_e@gmx.net


Mon, 23 Jun 2003 21:28:54 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Two weeks to feature freeze
Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > I put stuff in /tools so if something happens to me, you guys can keep
> > going.
> 
> Yes, we keep going with make clean; make check, like everyone else.  Why
> don't you consider using that?

The script is automated to run at night, it captures gmake output while
returning the proper error return code, and exits if it finds any
problems.  There was talk others want to automate nightly builds, so
this a start for them.

Amazing you find 688 bytes worth discussing.  I know you said "what
happens if everyone adds their scripts", but something that would be a
mess if everyone did it isn't always a proper way to judge if something
is appropriate.

If someone wants to submit a better test build script, I will merge it
into the existing one or replace mine.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 23 Jun 2003 15:37:53 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze
Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > I put stuff in /tools so if something happens to me, you guys can keep
> > going.
> 
> Yes, we keep going with make clean; make check, like everyone else.  Why
> don't you consider using that?

Actually, I used to manually do all those tests to test patches, but I
found a single script was less error prone, and allowed me to more
reliably test patches --- in the old days, I would forget the initdb or
regression runs, only to find out later the patch broke something.

There is a value to that script for me, and if someone else does patch
application, I suggest they use it too.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 23 Jun 2003 15:39:37 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze
scott.marlowe wrote:
> Peter is coming off awfully paternalistic here.  I'd rather have a few 
> extra scripts to look through to find what I need when I'm trying to 
> figure out something than to have a tool that only the hackers know exists 
> and I can only get by asking nicely to see the pretty code.
> 
> While no one wants to see a contrib or tool directory of a hundred megs, 
> lots of little example files and testing scripts can be nice for nothing 
> else other than the examples they provide.  I learned a lot when I first 
> started using pgsql from the things in the contrib directory.

Having something that just runs and I don't have to fiddle with it is a
big help --- of course, it took me a few years to realize that this is
the best way to test patches --- kind of embarassing that I didn't think
of it in 1997.

I think the patch came out of complaints that my patch application was
letting in broken code --- since I have been using it, I am mostly down
to weird bugs or platform problem, but the obvious patch problems are
pretty much gone, which some people might have noticed.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 23 Jun 2003 15:42:58 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze

On Mon, 23 Jun 2003, Peter Eisentraut wrote:

> Bruce Momjian writes:
>
> > Well, it is a nice test template for people who aren't shell script
> > experts, and I have been in the habit of pushing stuff I use into /tools
> > so it is available for others.
>
> I know and I'm not happy about it.  The PostgreSQL source tree isn't a
> repository of everyone's favorite shell scripts.  There's an official
> method to build and test a PostgreSQL installation.  If that is flawed or
> incomplete, then let's talk about it.  But everyone pushing out their own
> little test methodology without further consideration is not going to help
> this discussion.

'K, its flawed and incomplete, let's talk about it :)

What I was looking for here was something I could add to cron on a machine
that would update the source, do a base configure (or give me the ability
to give extra options, as the case may be), build, install and test, and
report errors to me via email where applicable ...

The idea is that it could be something that ppl could have run nightly, in
the wee hours, and only when a problem occurs would they be informed so
taht they coudl either fix teh error (ie. out of space), or report it to
the list(s) ...




Mon, 23 Jun 2003 22:27:45 -0300 (ADT)
The Hermit Hacker <scrappy@postgresql.org>
Auto Building / Testing (was: Re: Two weeks to feat..)

Yes, it does some of that, but I don't think it is safe to do a cvs
update in an automated fashion, as least on my machine.

---------------------------------------------------------------------------

The Hermit Hacker wrote:
> 
> On Mon, 23 Jun 2003, Peter Eisentraut wrote:
> 
> > Bruce Momjian writes:
> >
> > > Well, it is a nice test template for people who aren't shell script
> > > experts, and I have been in the habit of pushing stuff I use into /tools
> > > so it is available for others.
> >
> > I know and I'm not happy about it.  The PostgreSQL source tree isn't a
> > repository of everyone's favorite shell scripts.  There's an official
> > method to build and test a PostgreSQL installation.  If that is flawed or
> > incomplete, then let's talk about it.  But everyone pushing out their own
> > little test methodology without further consideration is not going to help
> > this discussion.
> 
> 'K, its flawed and incomplete, let's talk about it :)
> 
> What I was looking for here was something I could add to cron on a machine
> that would update the source, do a base configure (or give me the ability
> to give extra options, as the case may be), build, install and test, and
> report errors to me via email where applicable ...
> 
> The idea is that it could be something that ppl could have run nightly, in
> the wee hours, and only when a problem occurs would they be informed so
> taht they coudl either fix teh error (ie. out of space), or report it to
> the list(s) ...
> 
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if your
>       joining column's datatypes do not match
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 23 Jun 2003 21:30:13 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Auto Building / Testing (was: Re: Two weeks to feat..)
Bruce Momjian writes:

> Amazing you find 688 bytes worth discussing.  I know you said "what
> happens if everyone adds their scripts", but something that would be a
> mess if everyone did it isn't always a proper way to judge if something
> is appropriate.

I said, if everyone adds their test methodologies.  That leads to
discrepancies, more of them down the road if one method changes and the
other doesn't catch up.  For instance, your method just calls pg_ctl,
createdb, etc. from the path.  If people already have a stable
installation of PostgreSQL on their machine, then this will test the wrong
installation.  So, from now on, if someone submits a test result I have to
ask, "which method did you use" -- "don't use that method, because it's
wrong".  That is one instance, and I'm sure you'll fix it, but there might
be more.  What I'm saying is, we were in a discussion about improving the
testing of PostgreSQL, and this is not a step forward.  If we need to
improve the testing mechanisms for various purposes -- patch application,
automated testing, etc. -- let's look at it and see how we can improve the
current infrastructure without inventing a parallel one.  At this point,
I'm not sure why "make check" doesn't serve you.  Perhaps you are not
fully aware of what it does (I guess so, from looking at your script).

-- 
Peter Eisentraut   peter_e@gmx.net


Fri, 27 Jun 2003 16:57:07 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Two weeks to feature freeze

Wow, I am impressed by 'gmake check'.  Who did all that work?  It is
great.

I modified tools/pgtest to use 'gmake check'.  Thanks.

---------------------------------------------------------------------------

Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > Amazing you find 688 bytes worth discussing.  I know you said "what
> > happens if everyone adds their scripts", but something that would be a
> > mess if everyone did it isn't always a proper way to judge if something
> > is appropriate.
> 
> I said, if everyone adds their test methodologies.  That leads to
> discrepancies, more of them down the road if one method changes and the
> other doesn't catch up.  For instance, your method just calls pg_ctl,
> createdb, etc. from the path.  If people already have a stable
> installation of PostgreSQL on their machine, then this will test the wrong
> installation.  So, from now on, if someone submits a test result I have to
> ask, "which method did you use" -- "don't use that method, because it's
> wrong".  That is one instance, and I'm sure you'll fix it, but there might
> be more.  What I'm saying is, we were in a discussion about improving the
> testing of PostgreSQL, and this is not a step forward.  If we need to
> improve the testing mechanisms for various purposes -- patch application,
> automated testing, etc. -- let's look at it and see how we can improve the
> current infrastructure without inventing a parallel one.  At this point,
> I'm not sure why "make check" doesn't serve you.  Perhaps you are not
> fully aware of what it does (I guess so, from looking at your script).
> 
> -- 
> Peter Eisentraut   peter_e@gmx.net
> 
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 27 Jun 2003 17:47:59 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze