> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us] 
> Sent: Friday, June 20, 2003 11:47 PM
> To: Dann Corbit
> Cc: Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze 
> 
> 
> "Dann Corbit" <DCorbit@connx.com> writes:
> > Look at this: 
> > 
> http://www.mysql.com/information/crash-me.php?mysql_4_1=on&postgres=on
> 
> This looks a little cleaner than the last time I looked at it 
> (more than three years ago), but it's still fundamentally a 
> marketing effort.  It is not an exercise in spec compliance 
> measurement, because there are hundreds of bullet points that 
> all look exactly alike, whether they are measuring 
> spec-required elements, random vendor extensions to the spec, 
> or spec violations.  To take just one example of the latter, 
> "Calculate 1--1" is still shown with a green star for MySQL 
> and a failure for Postgres, when a more correct reading would 
> be "Fails to recognize SQL-standard -- comment syntax" for 
> MySQL.  And yes, they were called out on this three years 
> ago, and no they haven't fixed the entry since then.  I 
> should believe that there is any good faith on their part?
> 
> For another example, take a close look at the "Quoting" 
> section, which purports to measure compliance to the spec's 
> ideas about how to quote an identifier.  Postgres accepts 
> double-quoted identifiers per spec, including doubled double 
> quotes per spec, and rejects bracketed or backquoted 
> identifiers per spec.  MySQL is apparently spec compliant on 
> just one of those four points.  Curious that they manage to 
> end up with a better looking display than us in this section; 
> in particular note that Postgres is specifically claimed 
> *not* to handle double-quoted identifiers.  (Memory is fuzzy 
> after three years, but IIRC when you look at the actual test 
> code being used, it tests more than whether double quoted 
> identifiers are allowed, and really is failing us on some 
> quite unrelated detail.)
> 
> Another point worth mentioning is that most of the numerical 
> limits shown in the table have nothing to do with actual 
> server limits, but with random limitations of their test 
> process.  For instance, I'm not sure what "max index part 
> length 235328" really means, but I am pretty sure it's got 
> nothing to do with the Postgres server.  Or look at "constant 
> string size in SELECT 16777207" ... nope, there's no such 
> limit.  (If they'd put a "+" in there then it'd be okay, but 
> no.) I still remember watching crash-me trying to measure the 
> max query length of Postgres 7.0: the crashme client process 
> dumped core before Postgres did, after which the controlling 
> script announced that we weren't crash-safe.
> 
> > So far, I have seen three problems pointed out (out of 600+ tests).
> 
> These are the high spots from three-year-old memories.  Do 
> you really want a detailed analysis?  A quick look at their 
> table recalls plenty of bogosity to my mind.
> 
> A last point is that this table is comparing MySQL 4.1 
> (bleeding edge alpha release) against PG 7.2 (one full major 
> release behind the times). While I cannot really blame the 
> MySQL guys for not being up-to-the- minute on everyone else's 
> releases, this does emphasize the key point, namely that this 
> isn't a fair comparison run by disinterested parties but a 
> marketing effort of, by, and for MySQL.

It seems pretty clear that there are warts on the Crashme test.
Perhaps 70% or so is truly useful.  Maybe the useful subset could be
approximated or modified to be useful as a general tool set.

Not too surprising that a commercial enterprise tries to bend the facts
in their favor a bit.

Some other stuff worth note:
http://osdb.sourceforge.net/
http://sourceforge.net/projects/osdldbt (looks like someone has put a
bunch of PostgreSQL effort into it.
http://sourceforge.net/projects/ltp/ (DOTS)
http://www.mysql.com/portal/software/item-222.html (I won't mention
where it's from)
ftp://ftp.cs.wisc.edu/OO7/


Win32 specific, but has source code:
http://www.mipt.sw.ru/en/install/ots/ (ODBC testing)
http://www.mipt.sw.ru/en/install/ats/ (ADO testing)
Some other interesting stuff is found there too...

Test tools links:
http://www.softwareqatest.com/qattls1.html
http://www.aptest.com/resources.html

Sat, 21 Jun 2003 00:16:07 -0700
"Dann Corbit" <DCorbit@connx.com>
Re: Two weeks to feature freeze 
On Sat, 21 Jun 2003, Dann Corbit wrote:

> It seems pretty clear that there are warts on the Crashme test.
> Perhaps 70% or so is truly useful.  Maybe the useful subset could be
> approximated or modified to be useful as a general tool set.

And we all wait with baited breath for you to develop and submit such
tests ...


Sun, 22 Jun 2003 14:08:34 -0300 (ADT)
The Hermit Hacker <scrappy@postgresql.org>
Re: Two weeks to feature freeze 