> -----Original Message-----
> From: Christopher Kings-Lynne [mailto:chriskl@familyhealth.com.au] 
> Sent: Friday, June 20, 2003 10:14 PM
> To: Dann Corbit
> Cc: Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> > We (at CONNX Solutions Inc.) have a formal release procedure that 
> > includes many tens of thousands of automated tests using dozens of 
> > different platforms.  There are literally dozens of 
> machines (I would 
> > guess 70 or so total) running around the clock for 7 days before we 
> > even know if we have a release candidate.  The QA team is distinct 
> > from the development team, and if they say "FLOP!" the release goes 
> > nowhere.  No formal release until QA passes it.
> 
> PostgreSQL has a comprehensive regression suite that is run 
> by the developers all the time...

If you mean the one that comes with PostgreSQL, then I think the MySQL
test is better.  The PostgreSQL test seems to focus more on extensions
than anything else.
 
> > If there is no procedure for PostgreSQL of this nature, then there 
> > really needs to be.  I am sure that MySQL must have 
> something in place 
> > like that.  Their "Crash-Me" test suite shows (at least) that they 
> > have put a large effort into testing.
> 
> No, it means they've put a crap effort into trying to make 
> other databases look bad...

It does not achieve that goal.

Most of the criticism leveled at their efforts sound like fearful hand
waving to me.  True, I have not studied the test as carefully as others
have.  But the PostgreSQL test is not superior to the MySQL test.  I
have put considerable effort into the PostgreSQL regression test.  We
achieved 100% success on the Win32 platform, including dynamic loading
of functions.

Fri, 20 Jun 2003 22:19:12 -0700
"Dann Corbit" <DCorbit@connx.com>
Re: Two weeks to feature freeze
> If you mean the one that comes with PostgreSQL, then I think the MySQL
> test is better.  The PostgreSQL test seems to focus more on extensions
> than anything else.

What the?  It tests no extensions.  The extensions have their own
regression tests.

> Most of the criticism leveled at their efforts sound like fearful hand
> waving to me.  True, I have not studied the test as carefully as others
> have.  But the PostgreSQL test is not superior to the MySQL test.  I
> have put considerable effort into the PostgreSQL regression test.  We
> achieved 100% success on the Win32 platform, including dynamic loading
> of functions.

Notice that it tests absulutely no parallel functionality, whereas
PostgreSQL tests things in parallel to check for concurrency problems:

"Note that this benchmark is single threaded, so it measures the minimum
time for the operations. We plan to in the future add a lot of
multi-threaded tests to the benchmark suite. "

It's said that for at least 4 years now.

Crash-me has nothing to do with testing, it jsut checks to see what
features a db supports:

"crash-me tries to determine what features a database supports and what
its capabilities and limitations are by actually running queries. For
example, it determines:

What column types are supported
How many indexes are supported
What functions are supported
How big a query can be
How big a VARCHAR column can be"

Obviously it has nothing to do with can I index every type in the system,
can I use the index to look up a set of test values, etc., etc.

Chris



Sat, 21 Jun 2003 13:34:45 +0800 (WST)
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: Two weeks to feature freeze
> > PostgreSQL has a comprehensive regression suite that is run 
> > by the developers all the time...
> 
> If you mean the one that comes with PostgreSQL, then I think the MySQL
> test is better.  The PostgreSQL test seems to focus more on extensions
> than anything else.

I would be happy to make additions to the regression tests if you can
give me a list of items that are missing.  I've looked and didn't see
anything obvious aside from inter-connection testing.  Yes, tests run in
parallel on multiple connections, but there is no interaction between
since there is not a method of controlling the timing at the moment.

-- 
Rod Taylor <rbt@rbt.ca>

PGP Key: http://www.rbt.ca/rbtpub.asc
21 Jun 2003 08:50:57 -0400
Rod Taylor <rbt@rbt.ca>
Re: Two weeks to feature freeze
On Sat, 21 Jun 2003, Christopher Kings-Lynne wrote:

> Crash-me has nothing to do with testing, it jsut checks to see what
> features a db supports:

An interesting point is that until recently, crashme said that the 
postgresql backend crashed on very large queries.  The actual problem was 
that postgresql has NO LIMIT to query size, and the crashme script would 
keep feeding the postgresql backend larger and larger amounts of query 
until the internal buffer of the crashme script overran.

This failure was attributed to postgresql when it was, in fact a bug in 
the crashme script.

This is not an isolated behaviour of crashme.  It's a quick dirty hack job 
designed to show the differences between MySQL and all the other 
databases.  If it was truly comprehensive (i.e. SQL92 spec testing) there 
would be hundreds of failure points for MySQL.  but it isn't.  It tests 
only those things that are good in MySQL against other databases (for the 
most part, there is some token effort at including a few things MySQL 
doesn't do).




Mon, 23 Jun 2003 06:46:11 -0600 (MDT)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: Two weeks to feature freeze