> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us] 
> Sent: Friday, June 20, 2003 9:19 PM
> To: Dann Corbit
> Cc: Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze 
> 
> 
> "Dann Corbit" <DCorbit@connx.com> writes:
> >> ...ROTFL...  Crash-Me is not a regression test.  It is a
> >> marketing effort.
> 
> > Their marketing effort checks for STANDARDS conformance 
> against over 
> > several hundred distinct, important properties.
> 
> If you'd not spelled STANDARDS in caps I'd not have taken the 
> trouble to respond 

Sorry for shouting.

>... but I suggest you stop to count 
> exactly how many of their bullet points are actually grounded 
> in the SQL standard, how many are not, and how many are in 
> fact counter to spec but agree with MySQL's deviations from 
> spec (of course those show as green for MySQL's version of 
> reality, and as "failures" for spec-compliant databases).
> 
> I have been through crash-me in some detail, and it left a 
> very bad taste in my mouth.  Don't bother holding it up as an 
> example of good practice.

Every single test in their list is interesting and useful.

I see several hundred things which I recognize as part of the ANSI/ISO
SQL Standard.

I have set up and run the tests.  I did not go into great detail (as you
have done) to ensure that they were really testing what they claimed to
test and that correct interpretation of the results was made in each
case.

If they have done something underhanded, then they should be called out
onto the carpet for it.  In any case, the general outline of what they
are doing is a very good idea.  If it can be improved upon, then that
would be an excellent idea.


Fri, 20 Jun 2003 21:25:08 -0700
"Dann Corbit" <DCorbit@connx.com>
Re: Two weeks to feature freeze 
On Fri, Jun 20, 2003 at 09:25:08PM -0700, Dann Corbit wrote:

Citing Tom Lane:
> > I have been through crash-me in some detail, and it left a 
> > very bad taste in my mouth.  Don't bother holding it up as an 
> > example of good practice.
> 
> Every single test in their list is interesting and useful.

At least on the version I just saw there are several results with
Postgres that are weird (table names > 500 chars?).  Other things tested
are clearly wrong (things that are = NULL, dates like '00-00-0000');
results for Postgres that are wrong probably because they are trying a
weird syntax. Etc.

Things like that drive the credibility of the whole thing to the floor.
Maybe something like this should exist for Postgres, but it's not
crash-me.  Maybe the NIST compliance test is adequate.

-- 
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"La conclusion que podemos sacar de esos estudios es que
no podemos sacar ninguna conclusion de ellos" (Tanenbaum)

Sat, 21 Jun 2003 00:59:43 -0400
Alvaro Herrera <alvherre@dcc.uchile.cl>
Re: Two weeks to feature freeze
> Things like that drive the credibility of the whole thing to the floor.
> Maybe something like this should exist for Postgres, but it's not
> crash-me.  Maybe the NIST compliance test is adequate.

Plus I belive the RedHat people are getting PostgreSQL through the NIST
compliance tests at the moment...I'd love to see MySQL pass them...

Chris



Sat, 21 Jun 2003 13:22:00 +0800 (WST)
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: Two weeks to feature freeze
Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes:
>> Maybe the NIST compliance test is adequate.

> Plus I belive the RedHat people are getting PostgreSQL through the NIST
> compliance tests at the moment...I'd love to see MySQL pass them...

FWIW, the first pass of those tests is complete, and it turned up
exactly one bug that we didn't already know of (the
outer-level-aggregate bizarrity that I fixed last week ... which MySQL
wouldn't be subject to since they haven't got subselects ...)

The work is not done, because there are some tests that couldn't be run
because they were blocked by known noncompliances (such as lack of
updatable views).  But I'm not getting a sense that we will learn a
whole lot from the NIST tests.

Further details will be published soon...

			regards, tom lane

Sat, 21 Jun 2003 01:33:36 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Two weeks to feature freeze 

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Cc: "Alvaro Herrera" <alvherre@dcc.uchile.cl>; "Dann Corbit"
<DCorbit@connx.com>; "Jason Earl" <jason.earl@simplot.com>;
"PostgreSQL-development" <pgsql-hackers@postgresql.org>
Sent: Saturday, June 21, 2003 8:33 AM
Subject: Re: [HACKERS] Two weeks to feature freeze


> Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes:
> >> Maybe the NIST compliance test is adequate.
>
> > Plus I belive the RedHat people are getting PostgreSQL through the NIST
> > compliance tests at the moment...I'd love to see MySQL pass them...
>
> FWIW, the first pass of those tests is complete, and it turned up
> exactly one bug that we didn't already know of (the
> outer-level-aggregate bizarrity that I fixed last week ... which MySQL
> wouldn't be subject to since they haven't got subselects ...)
>
> The work is not done, because there are some tests that couldn't be run
> because they were blocked by known noncompliances (such as lack of
> updatable views).  But I'm not getting a sense that we will learn a
> whole lot from the NIST tests.
As far as I can see NIST (http://www.itl.nist.gov/div897/ctg/sql_form.htm)
tests are used *only* for testing SQL92 conformance.
Latest available test suite version 6, dated 12/1996.
Are you using about 6-7 years old test suite?
Perhaps I am wrong, I would be happy If someone could point me up-to-date
info about NIST conformance testing.

regards,
Alvis Tunkelis


Sat, 21 Jun 2003 14:35:53 +0300
<alvis@piladzi-2.biz>
Re: Two weeks to feature freeze 
<alvis@piladzi-2.biz> writes:
> As far as I can see NIST (http://www.itl.nist.gov/div897/ctg/sql_form.htm)
> tests are used *only* for testing SQL92 conformance.
> Latest available test suite version 6, dated 12/1996.
> Are you using about 6-7 years old test suite?
> Perhaps I am wrong, I would be happy If someone could point me up-to-date
> info about NIST conformance testing.

AFAIK, the NIST abandoned that project years ago, so there isn't any
more-up-to-date test suite available.  Not sure this is a big problem,
since SQL92 is not a moving target.  It'd be nice if they had kept
working and developed tests for the non-entry-level features, but
they didn't.

			regards, tom lane

Sat, 21 Jun 2003 12:23:42 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Two weeks to feature freeze 