> -----Original Message-----
> From: Bruce Momjian [mailto:pgman@candle.pha.pa.us] 
> Sent: Saturday, June 21, 2003 8:50 PM
> To: Dann Corbit
> Cc: Tom Lane; Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> Dann Corbit wrote:
> > That is the worst possible test plan.  It totally lacks 
> organization 
> > and there is no hint to define when the feature set has 
> been covered.  
> > Ad hoc testing is a useful addition, but it cannot replace all the 
> > standard tests that have been used by the industry for decades.
> > 
> > If you run literally hundreds of tests designed to ensure that your 
> > product conforms to ANSI/ISO standards then the bugs that 
> are missed 
> > will be few and far between.  Unless you are bad at designing tests.
> > 
> > Designing tests is busywork.  Desiging tests is boring.  
> Nobody wants 
> > to design tests, let alone interpret the results and define correct 
> > baselines.  But testing is very, very important.
> 
> I remember when I was with Great Bridge they said, "Oh, we 
> are going to have a test setup and do all sorts of testing to 
> improve PostgreSQL."  I told them I doubted their testing was 
> going to shake out many more bugs than our existing testing 
> setup, and you know what, I was pretty much right.  Sure, 
> they found a few, but it wasn't much.

PostgreSQL is a fairly mature product, having been in existence in one
form or another for many years now.

I expect that most of the bugs that surface will be in areas of new
functionality.

Great Bridge had the right idea though.  Let's suppose that they ran
10,000 tests and turned up only one bug.  That would be just as valuable
(if not more so) than turning up 100 bugs.  A large, carefully designed
test system is *proof* of software quality, or at least of the effort to
determine the quality level.  It is also proof of the responsibility of
the software's originators.

Scenario:
You are going to install a tool that your organization will invest its
future in.

Vendor A: "We think our tool is pretty solid and our end users hardly
ever turn up any bugs."

Vendor B:" We think our tool is pretty solid and our 8500 tests
currently show only 3 defects with the released version, and these are
low impact issues.  To view our current database of issues, log onto web
form <page>."

Which tool would you prefer to install?

Mon, 23 Jun 2003 12:08:41 -0700
"Dann Corbit" <DCorbit@connx.com>
Re: Two weeks to feature freeze
Dann Corbit wrote:
> PostgreSQL is a fairly mature product, having been in existence in one
> form or another for many years now.
> 
> I expect that most of the bugs that surface will be in areas of new
> functionality.
> 
> Great Bridge had the right idea though.  Let's suppose that they ran
> 10,000 tests and turned up only one bug.  That would be just as valuable
> (if not more so) than turning up 100 bugs.  A large, carefully designed
> test system is *proof* of software quality, or at least of the effort to
> determine the quality level.  It is also proof of the responsibility of
> the software's originators.

Look at the cost/benefit ratio to that.  If you think we don't have to
care about cost/benefit, well, it would be pretty amazing if we didn't.

> Scenario:
> You are going to install a tool that your organization will invest its
> future in.
> 
> Vendor A: "We think our tool is pretty solid and our end users hardly
> ever turn up any bugs."
> 
> Vendor B:" We think our tool is pretty solid and our 8500 tests
> currently show only 3 defects with the released version, and these are
> low impact issues.  To view our current database of issues, log onto web
> form <page>."
> 
> Which tool would you prefer to install?

I don't think commerical vendors, with those 8500 test, are are doing
any better in reliability than PostgreSQL, and in fact, I think they are
doing worse, and have to expend much more effort than we do.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 23 Jun 2003 15:17:44 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze
On Mon, 23 Jun 2003, Dann Corbit wrote:

> Vendor A: "We think our tool is pretty solid and our end users hardly
> ever turn up any bugs."
> 
> Vendor B:" We think our tool is pretty solid and our 8500 tests
> currently show only 3 defects with the released version, and these are
> low impact issues.  To view our current database of issues, log onto web
> form <page>."
> 
> Which tool would you prefer to install?

The one I've tested and found to meet my needs, both now and by providing 
fixes when I needed it.

Real world example:  We run Crystal Reports Enterprise edition where I 
work.  It's tested thouroughly (supposedly) and has all kinds of QA.  
However, getting it to work right and stay up is a nightmare.  It's taken 
them almost a year to get around to testing against the OpenLDAP LDAP 
server we use.  The box said "LDAP V3 compliant" and they assured us that 
it was.  Well, it doesn't work with our LDAP V3 compliant LDAP server at 
all, and the problem is something they can't fix for months because it 
doesn't fit into their test cycle.


Real world example: Postgresql aggregates in subselects.
Someone found a bug in subselects in Postgresql with inner references to 
outter aggregates.  The postgresql team delivered a patch in less than a 
week.  User tested it and it works.

I'm not against testing and all, but as one of the many beta testers for 
Postgresql, I do feel a bit insulted by your attitude that only a 
cohesive, organized testing effort can result in a reliable product.

I take my support of Postgresql seriously, and answer many questions every 
week in the general, sql, and performance mailing lists.  I'm not paid to 
do it, I stay at work an extra hour or so each day to "pay for it."  I 
test every beta and RC release on our dataset at work, and with a test 
load to make sure it works for us, and it does.

I offered to beta test for Crystal Reports and was told they weren't 
interested, they can do it in house.  Their support, like many big 
commercial houses, is designed more to make my boss's boss happy, not me, 
and it shows every day in how they fail to provide timely support for 
their product while playing CYA to the higherups.


Mon, 23 Jun 2003 13:24:48 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Two weeks to feature freeze