> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us] 
> Sent: Friday, June 20, 2003 8:36 PM
> To: Dann Corbit
> Cc: Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze 
> 
> 
> "Dann Corbit" <DCorbit@connx.com> writes:
> > If there is no procedure for PostgreSQL of this nature, then there 
> > really needs to be.
> 
> Are you volunteering to create it?  Step right up.

No.  And as an outsider, I rather doubt if any procedures I developed
would be taken very seriously.  If such procedures are to be developed,
I suspect that they will have to be developed from within if they are to
be successful.

This would be a good start:

A.  Combine:
	1.  Your regression test
	2.  Crashme (or some rough equivalent if you don't like it)
	3.  The NIST validation test suite
B.  Automate:
	1.  Installation of the tests
	2.  Execution of the tests
	3.  Transfer of the test results to a repository
	4.  Analysis of the test results
C.  Assign:
	1.  Criteria for acceptance of a build for release
	2.  Authority for acceptance of a build for release
	3.  Delegation rules for issue resolution
	4.  Procedures for issue resolution
 
> > I am sure that MySQL must have something in place
> > like that.  Their "Crash-Me" test suite shows (at least) that they 
> > have put a large effort into testing.
> 
> ...ROTFL...  Crash-Me is not a regression test.  It is a 
> marketing effort.

Let's see...
Their marketing effort checks for STANDARDS conformance against over
several hundred distinct, important properties.
Their marketing effort checks for a number of interesting and valuable
extensions.
Their marketing effort checks for system safety in a manner that is
better than anything I have ever seen from a commercial vendor.

And the PostgreSQL regression test is superior in what ways?

Look at this:
http://www.mysql.com/information/crash-me.php?mysql_4_1=on&postgres=on

Their marketing effort makes PostgreSQL look superior to MySQL in most
areas.  If it is a marketing effort, then we must applaud them for their
honesty.

Fri, 20 Jun 2003 21:03:40 -0700
"Dann Corbit" <DCorbit@connx.com>
Re: Two weeks to feature freeze 
"Dann Corbit" <DCorbit@connx.com> writes:
>> ...ROTFL...  Crash-Me is not a regression test.  It is a 
>> marketing effort.

> Their marketing effort checks for STANDARDS conformance against over
> several hundred distinct, important properties.

If you'd not spelled STANDARDS in caps I'd not have taken the trouble to
respond ... but I suggest you stop to count exactly how many of their
bullet points are actually grounded in the SQL standard, how many are
not, and how many are in fact counter to spec but agree with MySQL's
deviations from spec (of course those show as green for MySQL's version
of reality, and as "failures" for spec-compliant databases).

I have been through crash-me in some detail, and it left a very bad
taste in my mouth.  Don't bother holding it up as an example of good
practice.

			regards, tom lane

Sat, 21 Jun 2003 00:18:58 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Two weeks to feature freeze 
"Dann Corbit" <DCorbit@connx.com> writes:
> Look at this:
> http://www.mysql.com/information/crash-me.php?mysql_4_1=on&postgres=on

This looks a little cleaner than the last time I looked at it (more than
three years ago), but it's still fundamentally a marketing effort.  It
is not an exercise in spec compliance measurement, because there are
hundreds of bullet points that all look exactly alike, whether they are
measuring spec-required elements, random vendor extensions to the spec,
or spec violations.  To take just one example of the latter,
"Calculate 1--1" is still shown with a green star for MySQL and a
failure for Postgres, when a more correct reading would be "Fails to
recognize SQL-standard -- comment syntax" for MySQL.  And yes, they were
called out on this three years ago, and no they haven't fixed the entry
since then.  I should believe that there is any good faith on their
part?

For another example, take a close look at the "Quoting" section, which
purports to measure compliance to the spec's ideas about how to quote
an identifier.  Postgres accepts double-quoted identifiers per spec,
including doubled double quotes per spec, and rejects bracketed or
backquoted identifiers per spec.  MySQL is apparently spec compliant on
just one of those four points.  Curious that they manage to end up with
a better looking display than us in this section; in particular note
that Postgres is specifically claimed *not* to handle double-quoted
identifiers.  (Memory is fuzzy after three years, but IIRC when you look
at the actual test code being used, it tests more than whether double
quoted identifiers are allowed, and really is failing us on some quite
unrelated detail.)

Another point worth mentioning is that most of the numerical limits
shown in the table have nothing to do with actual server limits, but
with random limitations of their test process.  For instance, I'm not
sure what "max index part length 235328" really means, but I am pretty
sure it's got nothing to do with the Postgres server.  Or look at
"constant string size in SELECT 16777207" ... nope, there's no such
limit.  (If they'd put a "+" in there then it'd be okay, but no.)
I still remember watching crash-me trying to measure the max query
length of Postgres 7.0: the crashme client process dumped core before
Postgres did, after which the controlling script announced that we
weren't crash-safe.

> So far, I have seen three problems pointed out (out of 600+ tests).

These are the high spots from three-year-old memories.  Do you really
want a detailed analysis?  A quick look at their table recalls plenty
of bogosity to my mind.

A last point is that this table is comparing MySQL 4.1 (bleeding edge
alpha release) against PG 7.2 (one full major release behind the times).
While I cannot really blame the MySQL guys for not being up-to-the-
minute on everyone else's releases, this does emphasize the key point,
namely that this isn't a fair comparison run by disinterested parties
but a marketing effort of, by, and for MySQL.

			regards, tom lane

Sat, 21 Jun 2003 02:46:30 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Two weeks to feature freeze 
Tom Lane wrote:
> I have been through crash-me in some detail, and it left a very bad
> taste in my mouth.  Don't bother holding it up as an example of good
> practice.

You seem to miss Dan's point.  The specific implementation of crashme
is undoubtedly flawed in a number of ways, but the idea is very useful
as part of an acceptance testing suite.  In short, it would probably
be beneficial to us to fix crashme so that it tests the proper,
standards-compliant things and reports the actual results, and then
include it in the test suite.

Indeed, we could even go so far as to use it for our own marketing
purposes!  Have it cite, for each test, which part of the SQL spec it's
testing and what the result should be.



-- 
Kevin Brown					      kevin@sysexperts.com

Sat, 21 Jun 2003 15:15:03 -0700
Kevin Brown <kevin@sysexperts.com>
Re: Two weeks to feature freeze
I wrote:
> Tom Lane wrote:
> > I have been through crash-me in some detail, and it left a very bad
> > taste in my mouth.  Don't bother holding it up as an example of good
> > practice.
> 
> You seem to miss Dan's point.  The specific implementation of crashme
> is undoubtedly flawed in a number of ways, but the idea is very useful
> as part of an acceptance testing suite.  In short, it would probably
> be beneficial to us to fix crashme so that it tests the proper,
> standards-compliant things and reports the actual results, and then
> include it in the test suite.

Actually, now that I think about it, it would probably be more beneficial
to merge any correct tests that we aren't already performing into our
existing regression test framework, provided that the end result doesn't
take too long to run (as you pointed out elsewhere, regression tests
that take a really long time to run simply won't be run by most people,
except perhaps in a tinderbox type of environment).

Overall, it might be of some benefit to mark individual regression tests
with a priority, and then make it possible to run only those tests of
a specified priority or higher.  That way, the indvidual developer may
decide for himself which group of regression tests to run based on the
amount of time he's willing to let it take and how much hardware he has
to throw at it.  And at the same time, it would make it easier for new
tests to be included in the suite without worrying about the impact it
would have on people running the tests.


-- 
Kevin Brown					      kevin@sysexperts.com

Sat, 21 Jun 2003 15:41:54 -0700
Kevin Brown <kevin@sysexperts.com>
Re: Two weeks to feature freeze
Dann Corbit wrote:
>> -----Original Message-----
>> From: Tom Lane [mailto:tgl@sss.pgh.pa.us] 
>> 
>> Are you volunteering to create it?  Step right up.
> 
> No.  And as an outsider, I rather doubt if any procedures I developed
> would be taken very seriously.  If such procedures are to be developed,
> I suspect that they will have to be developed from within if they are to
> be successful.

What do you think is the way to become an insider?


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Sun, 22 Jun 2003 08:45:07 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: Two weeks to feature freeze