> -----Original Message-----
> From: scott.marlowe [mailto:scott.marlowe@ihs.com] 
> Sent: Monday, June 23, 2003 12:25 PM
> To: Dann Corbit
> Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> On Mon, 23 Jun 2003, Dann Corbit wrote:
> 
> > Vendor A: "We think our tool is pretty solid and our end 
> users hardly 
> > ever turn up any bugs."
> > 
> > Vendor B:" We think our tool is pretty solid and our 8500 tests 
> > currently show only 3 defects with the released version, 
> and these are 
> > low impact issues.  To view our current database of issues, 
> log onto 
> > web form <page>."
> > 
> > Which tool would you prefer to install?
> 
> The one I've tested and found to meet my needs, both now and 
> by providing 
> fixes when I needed it.
> 
> Real world example:  We run Crystal Reports Enterprise 
> edition where I 
> work.  It's tested thouroughly (supposedly) and has all kinds of QA.  
> However, getting it to work right and stay up is a nightmare. 
>  It's taken 
> them almost a year to get around to testing against the OpenLDAP LDAP 
> server we use.  The box said "LDAP V3 compliant" and they 
> assured us that 
> it was.  Well, it doesn't work with our LDAP V3 compliant 
> LDAP server at 
> all, and the problem is something they can't fix for months 
> because it 
> doesn't fit into their test cycle.
> 
> 
> Real world example: Postgresql aggregates in subselects. 
> Someone found a bug in subselects in Postgresql with inner 
> references to 
> outter aggregates.  The postgresql team delivered a patch in 
> less than a 
> week.  User tested it and it works.
> 
> I'm not against testing and all, but as one of the many beta 
> testers for 
> Postgresql, I do feel a bit insulted by your attitude that only a 
> cohesive, organized testing effort can result in a reliable product.

Let me rephrase it:
"Only a  cohesive, organized testing effort can result in a product that
is proven reliable."

Without such an effort, it is only an educated guess as to whether the
product is reliable or not.  The data is the most valuable software
component in an organization.  It is worth more than the hardware and it
is worth more than the software.  If you are going to trust one billion
dollars worth of corporate data on a software system, you ought to
ensure that the system has been carefully tested.  I don't think that is
just an opinion.  It's simply common sense.
 
> I take my support of Postgresql seriously, and answer many 
> questions every 
> week in the general, sql, and performance mailing lists.  I'm 
> not paid to 
> do it, I stay at work an extra hour or so each day to "pay 
> for it."  I 
> test every beta and RC release on our dataset at work, and 
> with a test 
> load to make sure it works for us, and it does.
> 
> I offered to beta test for Crystal Reports and was told they weren't 
> interested, they can do it in house.  Their support, like many big 
> commercial houses, is designed more to make my boss's boss 
> happy, not me, 
> and it shows every day in how they fail to provide timely support for 
> their product while playing CYA to the higherups.

A long test cycle does result in a slower patch.  But when you get the
patch, it is going to work and not introduce new problems.

The resistance to testing is typical of programmers.  The PostgreSQL
group is a group of programmers.  I don't think I can change anyone's
mind, since the most significant people on the list don't think it is
worth the bother.

Therefore, I am going to stop harping on it.

Mon, 23 Jun 2003 12:42:44 -0700
"Dann Corbit" <DCorbit@connx.com>
Re: Two weeks to feature freeze
Dann Corbit wrote:
> Let me rephrase it:
> "Only a  cohesive, organized testing effort can result in a product that
> is proven reliable."
> 
> Without such an effort, it is only an educated guess as to whether the
> product is reliable or not.  The data is the most valuable software
> component in an organization.  It is worth more than the hardware and it
> is worth more than the software.  If you are going to trust one billion
> dollars worth of corporate data on a software system, you ought to
> ensure that the system has been carefully tested.  I don't think that is
> just an opinion.  It's simply common sense.

True, it is an "educated guess", but it turns out our educated guess
method produces more reliable code than the exhaustive testing method,
so though there isn't as much of a _feeling_ of confidence, there is a
_result_ of more reliability.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 23 Jun 2003 15:44:53 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Two weeks to feature freeze
On Mon, 23 Jun 2003, Dann Corbit wrote:

> > -----Original Message-----
> > From: scott.marlowe [mailto:scott.marlowe@ihs.com] 
> > Sent: Monday, June 23, 2003 12:25 PM
> > To: Dann Corbit
> > Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development
> > Subject: Re: [HACKERS] Two weeks to feature freeze
> > 
> > 
> > On Mon, 23 Jun 2003, Dann Corbit wrote:
> > 
> > > Vendor A: "We think our tool is pretty solid and our end 
> > users hardly 
> > > ever turn up any bugs."
> > > 
> > > Vendor B:" We think our tool is pretty solid and our 8500 tests 
> > > currently show only 3 defects with the released version, 
> > and these are 
> > > low impact issues.  To view our current database of issues, 
> > log onto 
> > > web form <page>."
> > > 
> > > Which tool would you prefer to install?
> > 
> > The one I've tested and found to meet my needs, both now and 
> > by providing 
> > fixes when I needed it.
> > 
> > Real world example:  We run Crystal Reports Enterprise 
> > edition where I 
> > work.  It's tested thouroughly (supposedly) and has all kinds of QA.  
> > However, getting it to work right and stay up is a nightmare. 
> >  It's taken 
> > them almost a year to get around to testing against the OpenLDAP LDAP 
> > server we use.  The box said "LDAP V3 compliant" and they 
> > assured us that 
> > it was.  Well, it doesn't work with our LDAP V3 compliant 
> > LDAP server at 
> > all, and the problem is something they can't fix for months 
> > because it 
> > doesn't fit into their test cycle.
> > 
> > 
> > Real world example: Postgresql aggregates in subselects. 
> > Someone found a bug in subselects in Postgresql with inner 
> > references to 
> > outter aggregates.  The postgresql team delivered a patch in 
> > less than a 
> > week.  User tested it and it works.
> > 
> > I'm not against testing and all, but as one of the many beta 
> > testers for 
> > Postgresql, I do feel a bit insulted by your attitude that only a 
> > cohesive, organized testing effort can result in a reliable product.
> 
> Let me rephrase it:
> "Only a  cohesive, organized testing effort can result in a product that
> is proven reliable."

No, it isn't proven reliable PERIOD, it's proven reliable under the exact 
conditions of the testing procedure you've implemented.  And no matter how 
idiot proof we try to make Postgresql and its testing, someone else will 
always make a better idiot.  :-)  Actually, what I'm saying is that the 
corner cases are the ones that are hard to predict, and no amount of 
effort up front is going to find a corner case you haven't thought of yet.

> Without such an effort, it is only an educated guess as to whether the
> product is reliable or not.  The data is the most valuable software
> component in an organization.  It is worth more than the hardware and it
> is worth more than the software.  If you are going to trust one billion
> dollars worth of corporate data on a software system, you ought to
> ensure that the system has been carefully tested.  I don't think that is
> just an opinion.  It's simply common sense.

But if that is true, then Postgresql should cause me no end of problems as 
it crashes down around my feet every other week.  Oddly, the dbas for the 
other systems here at work (Oracle and MSSQL server) have a much higher 
workload keeping their factory tested databases up and running.  In over 
four years of use, we have had exactly ZERO downtime of postgresql.

Carefully testing the system is what I, the DBA of our postgresql servers, 
do.  Only I know how we use the system, so no matter how you or Bruce or 
Tom might test it, I'll always be able to do something you wouldn't think 
of, because you're simply not in my shoes.

It's not an educated guess that postgresql works for us, it's proven over 
and over again every single day by the throrough testing and use of every 
Postgresql user.

> > I take my support of Postgresql seriously, and answer many 
> > questions every 
> > week in the general, sql, and performance mailing lists.  I'm 
> > not paid to 
> > do it, I stay at work an extra hour or so each day to "pay 
> > for it."  I 
> > test every beta and RC release on our dataset at work, and 
> > with a test 
> > load to make sure it works for us, and it does.
> > 
> > I offered to beta test for Crystal Reports and was told they weren't 
> > interested, they can do it in house.  Their support, like many big 
> > commercial houses, is designed more to make my boss's boss 
> > happy, not me, 
> > and it shows every day in how they fail to provide timely support for 
> > their product while playing CYA to the higherups.
> 
> A long test cycle does result in a slower patch.  But when you get the
> patch, it is going to work and not introduce new problems.

Nice theory, but it isn't provable by my experience.  While I've put .0 
releases of postgresql into production many a times, usually with no 
issues at all, I never have and never will put a .0 release of Crystal 
Reports online.  They've taught me well not to trust their initial 
release.

> The resistance to testing is typical of programmers.  The PostgreSQL
> group is a group of programmers.  I don't think I can change anyone's
> mind, since the most significant people on the list don't think it is
> worth the bother.

No, I am NOT A postgresql programmer, I am a Postgresql user.  I do 
program, but that's in support of my job as a PHP dev / database admin.  
I've found myself looking through the postgresql code only an odd half 
dozen times.  I've never NEEDED to hack it, as it seems to just work.

> Therefore, I am going to stop harping on it.

Good move.  You're busily shooting at ghosts right now.

While there's always room for more testing, the best way to do it is to 
roll up your sleaves and write your own tests or add to the regression 
tests.  I've written quite a few load tests in Perl over the years to make 
sure that postgresql can hold up to the kind of load we tend to throw at 
it.  So have hundreds of other Postgresql users you've never met.  Just 
because no one has a centralized plan and document for testing doesn't 
mean it isn't getting done, it just means you can't see it all that well.

I think of your testing methodology as the equivalent of the old Soviet 
Union's Five year plans.  They were thorough and well documented, but 
never worked as well as planned.  Postgresql is like the free market 
system.  no one's completely in charge, and the more you try to control it 
the more it tries to be free.  Ultimately, the chaos of the free market 
won out.

It may be reassuring to think your product is very well tested before it 
goes out the door, but it's a false security, proven over and over by 
commercial products that simply don't work in the field because of 
problems that the original designers never envisioned, and now that they 
have a thorough and long drawn out testing cycle, it simply takes longer 
and longer to get fixes, while providing little, if any, improvement in 
quality.


Mon, 23 Jun 2003 14:00:46 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Two weeks to feature freeze
On Mon, 23 Jun 2003, Dann Corbit wrote:

> > -----Original Message-----
> > From: scott.marlowe [mailto:scott.marlowe@ihs.com] 
> > Sent: Monday, June 23, 2003 12:25 PM
> > To: Dann Corbit
> > Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development
> > Subject: Re: [HACKERS] Two weeks to feature freeze
> > 
> > 
> > On Mon, 23 Jun 2003, Dann Corbit wrote:
> > 
> > > Vendor A: "We think our tool is pretty solid and our end 
> > users hardly 
> > > ever turn up any bugs."
> > > 
> > > Vendor B:" We think our tool is pretty solid and our 8500 tests 
> > > currently show only 3 defects with the released version, 
> > and these are 
> > > low impact issues.  To view our current database of issues, 
> > log onto 
> > > web form <page>."
> > > 
> > > Which tool would you prefer to install?
> > 
> > The one I've tested and found to meet my needs, both now and 
> > by providing 
> > fixes when I needed it.

How about the one that doesn't run tests in order to show how much better it is
than the competition but to actually test operation? In other words Vendor B
has an interest in having the tests pass, what gives you the confidence it just
hasn't listed the ones that fail and that the tests that do pass are not just
testing something vendor B wants to show it can do?


> > Real world example:  We run Crystal Reports Enterprise 
> > edition where I 
> > work.  It's tested thouroughly (supposedly) and has all kinds of QA.  
> > However, getting it to work right and stay up is a nightmare. 
> >  It's taken 
> > them almost a year to get around to testing against the OpenLDAP LDAP 
> > server we use.  The box said "LDAP V3 compliant" and they 
> > assured us that 
> > it was.  Well, it doesn't work with our LDAP V3 compliant 
> > LDAP server at 
> > all, and the problem is something they can't fix for months 
> > because it 
> > doesn't fit into their test cycle.
> > 
> > 
> > Real world example: Postgresql aggregates in subselects. 
> > Someone found a bug in subselects in Postgresql with inner 
> > references to 
> > outter aggregates.  The postgresql team delivered a patch in 
> > less than a 
> > week.  User tested it and it works.
> > 
> > I'm not against testing and all, but as one of the many beta 
> > testers for 
> > Postgresql, I do feel a bit insulted by your attitude that only a 
> > cohesive, organized testing effort can result in a reliable product.
> 
> Let me rephrase it:
> "Only a  cohesive, organized testing effort can result in a product that
> is proven reliable."
> 
> Without such an effort, it is only an educated guess as to whether the
> product is reliable or not.  The data is the most valuable software
> component in an organization.  It is worth more than the hardware and it
> is worth more than the software.  If you are going to trust one billion
> dollars worth of corporate data on a software system, you ought to
> ensure that the system has been carefully tested.  I don't think that is
> just an opinion.  It's simply common sense.

So you've never worked on a project where the data is of high value, since in
those circumstances the customer is always going to apply their own acceptance
testing anyway. If you think that doesn't happen you try sitting through 2
solid days of Y2k testing on _one_ system and tell me customers never do their
own testing.


> Therefore, I am going to stop harping on it.

But there is no need to, as has been mentioned before, if the testing is not
upto your level of testing submit something that makes it so. Having said that
I do believe you mentioned that you didn't have the time to create something
but you would be happy to test it, i.e. test the test.


-- 
Nigel J. Andrews


Mon, 23 Jun 2003 21:29:47 +0100 (BST)
"Nigel J. Andrews" <nandrews@investsystems.co.uk>
Re: Two weeks to feature freeze
On Monday 23 June 2003 15:42, Dann Corbit wrote:
> Let me rephrase it:
> "Only a  cohesive, organized testing effort can result in a product that
> is proven reliable."

One can never 100% prove reliability without time in the field with real-world 
data, testing or no testing.  I would dare say that there are numerous 
reliable software packages out there in OSS-land that have never had that 
sort of testing.  But it really hinges on ones definition of proof, no?

Furthermore, the beta testers here in hackers are not 'end-users' per se.  The 
people in hackers who test the betas are very valuable as our QA team.

PostgreSQL is already proven reliable, to various degrees of reliability, 
enough to where a PostgreSQL backend was approved as the new .ORG registry.  
The track record continues, without mathematically rigorous and cohesive 
testing.  Such testing would be useful, of course, by it is not required for 
our purposes.  

Those who want rigorous testing can do the rigorous testing.  You yourself 
said that your company has a separate QA team from the development team; OK, 
organize a rigorous QA team.  While you won't stop releases (unless you find 
showstopper bugs, like those that have been found by our wonderful hackers 
testers), your input into actually testing PostgreSQL (as opposed to trying 
to convince someone else to test for you) would be valuable.  But you're not 
going to get me to do it; I do enough testing of a varied nature already.  I 
do this for fun.

If you find testing fun, more power to you. :-)
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Mon, 23 Jun 2003 17:59:48 -0400
Lamar Owen <lamar.owen@wgcr.org>
Re: Two weeks to feature freeze
On Mon, 23 Jun 2003, Dann Corbit wrote:

> The resistance to testing is typical of programmers.  The PostgreSQL
> group is a group of programmers.  I don't think I can change anyone's
> mind, since the most significant people on the list don't think it is
> worth the bother.
>
> Therefore, I am going to stop harping on it.

*rofl* I believe several of us have suggested that we would welcome
submissions for improved testing ... so, what, we feel that the test that
is done is sufficient, but are willing to accept submissions to improve
it, but you aren't willing to spend the time/effort to do so?

We might be a bunch of 'typical programmers', but you definitely sound
like a typical "I want you to do something to make life easier for me, but
am not willing to expend the time or effort to do anyting about it" ...


Mon, 23 Jun 2003 22:43:57 -0300 (ADT)
The Hermit Hacker <scrappy@postgresql.org>
Re: Two weeks to feature freeze
scott.marlowe wrote:
> On Mon, 23 Jun 2003, Dann Corbit wrote:
 > [Dann Corbit wrote a lot]
 > [...]
> It may be reassuring to think your product is very well tested before it 
> goes out the door, but it's a false security, proven over and over by 
> commercial products that simply don't work in the field because of 
> problems that the original designers never envisioned, and now that they 
> have a thorough and long drawn out testing cycle, it simply takes longer 
> and longer to get fixes, while providing little, if any, improvement in 
> quality.

Scott, it's worse.

It's been back in the early 90's, when we had WfW-3.11 systems with some 
MS-Word dinosaur, and we just lost 14 days of work because it simply 
crashed on loading the document. The Microsoft support solution was 
something that lost all the formatting, indexing and cross references of 
a structured 250 page concept. I don't remember the exact procedure as 
my brain cells did overcharge, but the dummy on the hotline really 
believed that their thoroughly tested software wasn't the problem and 
that the error lies within our document. That that was a file, written 
by their thoroughly tested software was a point she really didn't catch.

This dumb hotline girl is the type of people, Dann Corbit's test 
strategy will reassure. Plus maybe a few (unfortunately important but 
otherwise useless) managers. Other than that, it'll not make the life of 
the average DBA any better. Big amounts of useless tests just give 
otherwise clueless people the false impression, the error must be 
somewhere else. MySQL's crash-me is a perfect example for that.


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Tue, 24 Jun 2003 01:09:56 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: Two weeks to feature freeze