> -----Original Message-----
> From: Alvaro Herrera [mailto:alvherre@dcc.uchile.cl] 
> Sent: Friday, June 20, 2003 10:00 PM
> To: Dann Corbit
> Cc: Tom Lane; Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> On Fri, Jun 20, 2003 at 09:25:08PM -0700, Dann Corbit wrote:
> 
> Citing Tom Lane:
> > > I have been through crash-me in some detail, and it left a
> > > very bad taste in my mouth.  Don't bother holding it up as an 
> > > example of good practice.
> > 
> > Every single test in their list is interesting and useful.
> 
> At least on the version I just saw there are several results 
> with Postgres that are weird (table names > 500 chars?).  

It does get silly at a point, but I have seen systems with 128
characters for table names, column names, etc.  Some people seem to like
it.  Not me.  Too much typing.

> Other things tested are clearly wrong (things that are = 
> NULL, 

Sounds like testing for the existence of a bug.
X = NULL
X <= NULL
X >= NULL
Etc. must always test false, regardless of the contents of X.  Test for
equality with NULL is a conformance error if NULL == NULL returns true.

> dates like '00-00-0000'); 

Not sure what that might even mean.

> results for Postgres that are 
> wrong probably because they are trying a weird syntax. Etc.
> 
> Things like that drive the credibility of the whole thing to 
> the floor. Maybe something like this should exist for 
> Postgres, but it's not crash-me.  Maybe the NIST compliance 
> test is adequate.

So far, I have seen three problems pointed out (out of 600+ tests).
That's 0.5% defects.  Why not just drop the stupid tests, or bend them
to test for what they ought to be testing.

Besides those three, what other tests are bogus and why?

Fri, 20 Jun 2003 22:04:09 -0700
"Dann Corbit" <DCorbit@connx.com>
Re: Two weeks to feature freeze
> Sounds like testing for the existence of a bug.
> X = NULL
> X <= NULL
> X >= NULL
> Etc. must always test false, regardless of the contents of X.  Test for
> equality with NULL is a conformance error if NULL == NULL returns true.

They should all return NULL, not false...

> > dates like '00-00-0000');

Yes, that's MySQL's idea of a NULL date.  In fact, it's exactly what you
get when you insert a NULL date into a NOT NULL column - hooray the test
proves that MySQL accepts NULL values in NOT NULL columns...

Chris


Sat, 21 Jun 2003 13:24:30 +0800 (WST)
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: Two weeks to feature freeze
Dann Corbit writes:

> So far, I have seen three problems pointed out (out of 600+ tests).
> That's 0.5% defects.  Why not just drop the stupid tests, or bend them
> to test for what they ought to be testing.

The problem with crashme is that it tells you nothing of practical value.
It doesn't tell you whether PostgreSQL works right.  It doesn't tell you
whether PostgreSQL works well.  It doesn't tell you whether PostgreSQL
conforms to some standard.  It doesn't even tell whether PostgreSQL is
compatible to some other product.  The only thing that you could possibly
get out of fixing crashme is to look better in crashme.  And for the above
reasons, there is little interest in working on that.

-- 
Peter Eisentraut   peter_e@gmx.net


Sat, 21 Jun 2003 14:50:56 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: Two weeks to feature freeze
On Fri, Jun 20, 2003 at 10:04:09PM -0700, Dann Corbit wrote:

> > On Fri, Jun 20, 2003 at 09:25:08PM -0700, Dann Corbit wrote:
> > 
> > Citing Tom Lane:
> > > > I have been through crash-me in some detail, and it left a
> > > > very bad taste in my mouth.  Don't bother holding it up as an 
> > > > example of good practice.
> > > 
> > > Every single test in their list is interesting and useful.
> > 
> > At least on the version I just saw there are several results 
> > with Postgres that are weird (table names > 500 chars?).  
> 
> It does get silly at a point, but I have seen systems with 128
> characters for table names, column names, etc.  Some people seem to like
> it.  Not me.  Too much typing.

I meant that the real limit on 7.2 was much lower than that unless they
twiddled with sources at compile time (there's no configure switch for
that AFAIR).  Maybe 31 or 63 chars, I don't remember.  Do you really
trust the rest of the test seeing that they came up with a clearly wrong
answer in such a simple test?

They can't even "make vacuum run reliably" on 7.1.  See the performance
test.  Maybe they want to test 7.3 with lazy vacuum in place.  Why don't
they do that?  7.1 is already 2 years old.

> > Other things tested are clearly wrong (things that are = 
> > NULL, 
> 
> Sounds like testing for the existence of a bug.
> X = NULL
> X <= NULL
> X >= NULL
> Etc. must always test false, regardless of the contents of X.  Test for
> equality with NULL is a conformance error if NULL == NULL returns true.

You see, you are saying "sounds like they are testing".  What does the
code actually test?  Which is the right behaviour?  Which behaviour
gets the green point, MySQL's or the right one?  There are lots of
things like this; I don't want to waste my time actually reading the
code to see what the correct answer for each test is.

About the 1--1 thing Tom mentioned: be aware that Postgres happily
accepts the correct 1 - -1 expression, but also correctly fails to
"calculate" 1--1.  Which one gets the green point?  Of course it's the
non-compliant one.

Also they don't test things they don't support.  Is there a test for
subselects?  What about concurrency?  Transactional issues?  What about
performance when they have their "transaction support" enabled?


> So far, I have seen three problems pointed out (out of 600+ tests).
> That's 0.5% defects.  Why not just drop the stupid tests, or bend them
> to test for what they ought to be testing.

There's already a mechanism for testing inside Postgres.  Maybe more
tests are needed, but crash-me offers no real value.

I just became aware that the NIST test suite is quite old.  Maybe what's
needed is to expand it to SQL3 to develop a way of measuring the
compliance level.  But the cost of doing that is probably prohibitive.

-- 
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Hay quien adquiere la mala costumbre de ser infeliz" (M. A. Evans)

Sat, 21 Jun 2003 13:11:50 -0400
Alvaro Herrera <alvherre@dcc.uchile.cl>
Re: Two weeks to feature freeze
Alvaro Herrera wrote:
> Also they don't test things they don't support.  Is there a test for
> subselects?  What about concurrency?  Transactional issues?  What about
> performance when they have their "transaction support" enabled?

Sure don't they. Like their NUMERIC data type, that can *store* any 
precision, but when you actually calculate with it, it converts to 
floating point internally ... that's not only a spec violation, in many 
countries that's a violation of law if used by a bookkeeping system.


Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #


Sun, 22 Jun 2003 09:03:56 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: Two weeks to feature freeze