Wol,

> I think one MAJOR problem is that most (if not all) MV practitioners are
> not formally qualified in computing ... 
<snip>
> "Relational" is all about theory and proving things mathematically
> correct. "MV" is all about engineering and getting the result. And if
> that means pinching all the best ideas we can find from relational, then
> we're engineers - of course we'll do it :-)

So what you're saying is that you use MV databases becuase you find them 
easier to understand and set up than relational databases.   Sure.  For that 
matter, spreadsheets are easier to understand and set up than relational 
databases -- or MV databases, for that matter.

I've no problem with the idea that MV databases are easier for the neophyte to 
understand.  But you seem to be making this bizarre logical leap that "easier 
to understand without training" == "technically superior".  

And I noticed that you have completely backed off from your assertion that 
there was some kind of MV database theory without ever addressing it.

> As soon as a requirement for a database specifies extraction of the
> maximum power from the box, it OUGHT to rule out all the current
> relational databases. MV flattens it for it for performance. As an MV
> programmer, I *KNOW* that I can find any thing I'm looking for (or find
> out it doesn't exist) with just ONE disk seek. A relational programmer
> has to ask the db "does this exist" and hope the db is optimised to be
> able to return the result quickly. 

Well, frankly, no.   You're confusing your floundering because you find it 
hard to grasp relational database design and/or have a badly designed 
database with a performance problem.   There are several people in our 
community running large genetics databases on PostgreSQL, using good 
relational design, and doing well with them.

> To quote the Pick FAQ "SQL optimises
> the easy task of finding stuff in memory. Pick optimises the hard task
> of getting it into memory in the first place".

And hope to the gods that nobody pulls the power cord.

> "Relational" is all about theory and proving things mathematically
> correct. "MV" is all about engineering and getting the result. And if
> that means pinching all the best ideas we can find from relational, then
> we're engineers - of course we'll do it :-)

"Relational" is all about preserving the *long-term* integrity, utility, and 
accessability of your data.   "MV" is all about getting an expedient result 
immediately and to heck with the future.  

Read some of E.F. Codd's original papers ... he was not concerned with 
expanding some obscure batch of mathematics, he was concerned with the 
problems of data that wasn't clean, or wasn't the same when you queried it 
twice, or data with rules that changed eraticaly over time, and wasn't 
portable at all.  These are real, "engineering" problems that needed solving 
and relational theory solved it.

> . Unless you can use set theory to predict
> the future, relational has nothing to do with science ...

I wasn't aware that clairvoyance was a tenet of the scientific method.  
Science is about reproducing reliable results, which is also what relational 
theory is about.

To conclude:   Your entire advocacy of Pick can be summed up as "I like Pick 
and find it easy to program, therefore it is superior."   Well, Wol, that 
makes it superior for *you* but not for the rest of us.  If you want to use 
Pick and not PostgreSQL, go for it; but if you barge in here and try to 
convince us that Pick is superior based strictly on your say-so, you've 
become one of the crazy preachers on Sproul Plaza.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco

Wed, 22 Oct 2003 09:54:04 -0700
Josh Berkus <josh@agliodbs.com>
Re: Dreaming About Redesigning SQL 
>>>>> "Josh" == Josh Berkus <josh@agliodbs.com> writes:

    >> "Relational" is all about theory and proving things
    >> mathematically correct. "MV" is all about engineering and
    >> getting the result. And if that means pinching all the best
    >> ideas we can find from relational, then we're engineers - of
    >> course we'll do it :-)

    Josh> "Relational" is all about preserving the *long-term*
    Josh> integrity, utility, and accessability of your data.  "MV" is
    Josh> all about getting an expedient result immediately and to
    Josh> heck with the future.

To emphasize the *long-term* - relational is all about separating
physical database representation from a logical view of the data. 

Josh, I'll be happy to meet at Intermezzo - after Nov 24 perhaps. I
have a conference deadline .. the papers that we write "just to secure
funding" - your tax dollars at work. Long live the NSF ! 

BTW, I'm not necessarily that far from your views. There is, however,
more to an XML database than just storing data - relational databases
do just fine there. The tools that are used to build database systems
have other uses - for instance XML message brokers. 

-- 
Pip-pip
Sailesh
http://www.cs.berkeley.edu/~sailesh



Wed, 22 Oct 2003 11:37:08 -0700
Sailesh Krishnamurthy <sailesh@cs.berkeley.edu>
Re: Dreaming About Redesigning SQL
Gregor,

> I'm developing a native XML database (C++) (which is supposed to become
> open source one day) and I'm wondering wheather I could use GiST for it's
> indexes. Is GiST still alive?

Don't know, sorry.   

> Would PostgreSQL fit that requirement? And are you interested in having a
> fast, scalable XML access method?

We would welcome good XML tools for XML stored in databases.  

However, I'm not sure that PostgreSQL is the appropriate platform for an 
all-XML database; we're pretty dogmatically a relational database ... perhaps 
the grandfather of most relational databases.

-- 
-Josh Berkus
 Aglio Database Solutions
 San Francisco


Wed, 22 Oct 2003 12:19:43 -0700
Josh Berkus <josh@agliodbs.com>
Re: is GiST still alive?
josh@agliodbs.com (Josh Berkus) writes:
> Gregor,
>> I'm developing a native XML database (C++) (which is supposed to become
>> open source one day) and I'm wondering wheather I could use GiST for it's
>> indexes. Is GiST still alive?
>
> Don't know, sorry.   
>
>> Would PostgreSQL fit that requirement? And are you interested in having a
>> fast, scalable XML access method?
>
> We would welcome good XML tools for XML stored in databases.  
>
> However, I'm not sure that PostgreSQL is the appropriate platform for an 
> all-XML database; we're pretty dogmatically a relational database ... perhaps 
> the grandfather of most relational databases.

It leaves open the question of what is the appropriate way of
expressing XML entities and attributes and CDATA in database form.

If the point of the exercise is to have some way of storing XML data
in a database, then you may readily declare it thus:

create table some_table (
   -- key columns omitted
   xml character varying,
   -- additional columns omitted
);

And that's perfectly good at storing XML information.

But I think back to the XML generator I wrote for GnuCash; it has the
notion of building up a hierarchy of entities and attributes, each of
which is visible as an identifyable object of some sort.  Mapping that
onto a set of PostgreSQL relations wouldn't work terribly well.
-- 
let name="cbbrowne" and tld="libertyrms.info" in String.concat "@" [name;tld];;
<http://dev6.int.libertyrms.com/>
Christopher Browne
(416) 646 3304 x124 (land)

Wed, 22 Oct 2003 16:48:46 -0400
Christopher Browne <cbbrowne@libertyrms.info>
Re: is GiST still alive?
Christopher Browne wrote:

>But I think back to the XML generator I wrote for GnuCash; it has the
>notion of building up a hierarchy of entities and attributes, each of
>which is visible as an identifyable object of some sort.  Mapping that
>onto a set of PostgreSQL relations wouldn't work terribly well.
>  
>

*nod* I have tried this several times - it just doesn't work well, 
because the maps are too different.

You could do something like this:
   . a table for each element type, fields being the attributes, plus 
the node id.
   . a table to tie everything together (parent_id, child_id, 
child_order, child_type).
In theory you could even generate the DB schema from an XML schema and 
evaluate it with XPath-like expressions.

But why put yourself to such bother? I have never found a good reason to 
do this sort of thing.

cheers

andrew


Wed, 22 Oct 2003 17:43:31 -0400
Andrew Dunstan <andrew@dunslane.net>
Re: is GiST still alive?
On Wed, 22 Oct 2003, Christopher Browne wrote:
> It leaves open the question of what is the appropriate way of
> expressing XML entities and attributes and CDATA in database form.
<snip>

Thanx for your advise, but that's not what I had in mind. The original
idea to have a native xml database was that I doesn't work too well in a
relational database.
I was just wondering wheater I have to reinvent the wheel of database
technology when it comes to transaction processing, ACID, and Indexes,
which a native xml database ought to have as well.

-- 
Gregor Zeitlinger
gregor@zeitlinger.de


Thu, 23 Oct 2003 00:16:21 +0200 (CEST)
Gregor Zeitlinger <zeitling@informatik.hu-berlin.de>
Re: is GiST still alive?
On Wed, 22 Oct 2003, Andrew Dunstan wrote:
> But why put yourself to such bother? I have never found a good reason to
> do this sort of thing.
I think there is a huge potential for XML databases once there are good
ones and people start using them more extensively.
But for having real fast xml databases it's too slow to store them in
relational tables. After all, if you were designing a database for xml
data only - would you come up with tables to store them?

-- 
Gregor Zeitlinger
gregor@zeitlinger.de


Thu, 23 Oct 2003 00:22:55 +0200 (CEST)
Gregor Zeitlinger <zeitling@informatik.hu-berlin.de>
Re: is GiST still alive?
In the last exciting episode, zeitling@informatik.hu-berlin.de (Gregor Zeitlinger) wrote:
> On Wed, 22 Oct 2003, Andrew Dunstan wrote:
>> But why put yourself to such bother? I have never found a good reason to
>> do this sort of thing.
> I think there is a huge potential for XML databases once there are good
> ones and people start using them more extensively.
> But for having real fast xml databases it's too slow to store them in
> relational tables. After all, if you were designing a database for xml
> data only - would you come up with tables to store them?

No, "tables" wouldn't be the right way to do it.

But it's going to be troubled, in any case, because of the
every-popular mixtures of:

 a) Often weird declarations of what character sets are in use;

 b) Pointers to other parts of a document;

 c) What's a "database" going to consist of?  One XML document?  Or
    many?  And if many, then then how do you have a centralized
    reference point to navigate from to find the document that you
    want?

And "navigate" was a carefully chosen word; what you then have is
essentially a network database system, and have to then start making
up ways of describing queries.  XQuery may be better than CODASYL of
yesteryear, but you're still left writing a lot of recursive code.
(Thus making those that understand the Lambda Nature more powerful...)

At the end, do you have a "database?"  Or just a set of documents?
It's hard to tell, a priori.

And do you think this is likely to be useful because:

 a) You have some clear notion as to why this ought to be useful?

   or merely because

 b) XML is a big buzzword, and people have been able to succesfully
    attract "research funds" or "vulture capital" on the basis of
    having that acronym in a proposal?

The latter seems to have been really common the last few years, as
people seem often to add "Java" and "XML" to proposals in order to
attract unwary investors in much the same way that a "lady of the
evening" wears scanty clothing in order to attract her customers.
-- 
wm(X,Y):-write(X),write('@'),write(Y). wm('aa454','freenet.carleton.ca').
http://www3.sympatico.ca/cbbrowne/lisp.html
Eagles may soar, but weasels don't get sucked into jet engines.

Wed, 22 Oct 2003 21:39:54 -0400
Christopher Browne <cbbrowne@acm.org>
Re: is GiST still alive?
Do this:

1. Create a new type called 'xml', based on text.

2. The xmlin function for that type will validate what you are 
enterering is XML

3. Create new functions to implement XPath, SAX, etc. on the xml type.

4. Extend the contrib/ltree gist-based tree indexing scheme to work on 
xml and hence the operations in no.3 above are really fast...

Chris


Andrew Dunstan wrote:

> Christopher Browne wrote:
> 
>> But I think back to the XML generator I wrote for GnuCash; it has the
>> notion of building up a hierarchy of entities and attributes, each of
>> which is visible as an identifyable object of some sort.  Mapping that
>> onto a set of PostgreSQL relations wouldn't work terribly well.
>>  
>>
> 
> *nod* I have tried this several times - it just doesn't work well, 
> because the maps are too different.
> 
> You could do something like this:
>   . a table for each element type, fields being the attributes, plus the 
> node id.
>   . a table to tie everything together (parent_id, child_id, 
> child_order, child_type).
> In theory you could even generate the DB schema from an XML schema and 
> evaluate it with XPath-like expressions.
> 
> But why put yourself to such bother? I have never found a good reason to 
> do this sort of thing.
> 
> cheers
> 
> andrew
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 7: don't forget to increase your free space map settings


Thu, 23 Oct 2003 09:53:30 +0800
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: is GiST still alive?
Christopher Browne wrote:

> No, "tables" wouldn't be the right way to do it.
> 
> But it's going to be troubled, in any case, because of the
> every-popular mixtures of:
>  a) Often weird declarations of what character sets are in use;
>  b) Pointers to other parts of a document;
>  c) What's a "database" going to consist of?  One XML document?  Or
>     many?  And if many, then then how do you have a centralized
>     reference point to navigate from to find the document that you
>     want?
> And "navigate" was a carefully chosen word; what you then have is
> essentially a network database system, and have to then start making
> up ways of describing queries.  XQuery may be better than CODASYL of
> yesteryear, but you're still left writing a lot of recursive code.
> (Thus making those that understand the Lambda Nature more powerful...)
> 
> At the end, do you have a "database?"  Or just a set of documents?
> It's hard to tell, a priori.
> 
[Trimmed comments on utility of such features]

transferred from pgsql-hackers to pgsql-chat.

Coincidentally, just recently looked at Fongs "The design
and implementation of the POSTGRES query optimizer2"  (referenced in the
Developers Guide) which implies that one of the initial goals of
postquel and postgres was to handle similar structures :

"The basic problem here is that the relational model is not well-suited
for representing hierarchical relationships. As a solution, Stonebraker
has proposed embedding queries within data fields and using these
queries to express the hierarchical relationship between the
corresponding tuple and information elsewhere in the database [STON84].
Using this idea, which POSTGRES supports, our complex object example is
now represented as shown in figure 2.5. To retrieve information executed
by the queries embedded within this data field, the user would issue the
following query:

execute (COBJECT.components) where COBJECT.coid = 1.

Thus, n join queries reduce to a single execute query. In addition, 
users can selectively retrieve information
linked through these hierarchies by nesting attributes"

Go's around, comes around.
Cheers, Kurt.


Thu, 23 Oct 2003 16:37:24 +1300
Kurt at DBC <kurtw@dbc.co.nz>
Re: [HACKERS] is GiST still alive?
On Wed, 22 Oct 2003, Christopher Browne wrote:
> No, "tables" wouldn't be the right way to do it.
>
> But it's going to be troubled, in any case, because of the
> every-popular mixtures of:
>
>  a) Often weird declarations of what character sets are in use;
I gotta admit that I haven't spend too much attention on that specific
part. But couln't you just store it in the character set that was
originally used to populate the document?

>  b) Pointers to other parts of a document;
do you mean to the parent element and the child elements?
This is specifially what my custom format is designed for.

>  c) What's a "database" going to consist of?  One XML document?  Or
>     many?
many, each of which can be up to about 1TB

> And if many, then then how do you have a centralized
>     reference point to navigate from to find the document that you
>     want?
This one could be a table, or another xml document.

> And "navigate" was a carefully chosen word; what you then have is
> essentially a network database system, and have to then start making
> up ways of describing queries.  XQuery may be better than CODASYL of
> yesteryear, but you're still left writing a lot of recursive code.
> (Thus making those that understand the Lambda Nature more powerful...)
I don't get your point? XQuery works on one document, IIRC.

> At the end, do you have a "database?"  Or just a set of documents?
> It's hard to tell, a priori.
OK, know waht you mean. I'd say it's a database, because the information
is stored not plain - but in pages and in an optimized format for
insertion, deletion and querying.

> And do you think this is likely to be useful because:
>
>  a) You have some clear notion as to why this ought to be useful?
yes. Modyfing and querying plain xml files sucks performancewise once your
documents get a little larger (100 MB+)

>  b) XML is a big buzzword, and people have been able to succesfully
>     attract "research funds" or "vulture capital" on the basis of
>     having that acronym in a proposal?
That time's over anyways, isn't it?

-- 
Gregor Zeitlinger
gregor@zeitlinger.de


Thu, 23 Oct 2003 13:03:17 +0200 (CEST)
Gregor Zeitlinger <zeitling@informatik.hu-berlin.de>
Re: is GiST still alive?
On Thu, 23 Oct 2003, Christopher Kings-Lynne wrote:
> 4. Extend the contrib/ltree gist-based tree indexing scheme to work on
> xml and hence the operations in no.3 above are really fast...
but then, the plain xml data is still stored in a database colum, if I
understand correctly?

-- 
Gregor Zeitlinger
gregor@zeitlinger.de


Thu, 23 Oct 2003 13:06:03 +0200 (CEST)
Gregor Zeitlinger <zeitling@informatik.hu-berlin.de>
Re: is GiST still alive?

Storing the XML text has problems - you have to parse it every time you 
want something - that has to cause a huge performance hit.

I use XML a lot for all sorts of purposes, but it is appropriate for 
data transfer rather than data storage, IMNSHO.

cheers

andrew

Christopher Kings-Lynne wrote:

> Do this:
>
> 1. Create a new type called 'xml', based on text.
>
> 2. The xmlin function for that type will validate what you are 
> enterering is XML
>
> 3. Create new functions to implement XPath, SAX, etc. on the xml type.
>
> 4. Extend the contrib/ltree gist-based tree indexing scheme to work on 
> xml and hence the operations in no.3 above are really fast...
>
> Chris
>
>
> Andrew Dunstan wrote:
>
>> Christopher Browne wrote:
>>
>>> But I think back to the XML generator I wrote for GnuCash; it has the
>>> notion of building up a hierarchy of entities and attributes, each of
>>> which is visible as an identifyable object of some sort.  Mapping that
>>> onto a set of PostgreSQL relations wouldn't work terribly well.
>>>  
>>>
>>
>> *nod* I have tried this several times - it just doesn't work well, 
>> because the maps are too different.
>>
>> You could do something like this:
>>   . a table for each element type, fields being the attributes, plus 
>> the node id.
>>   . a table to tie everything together (parent_id, child_id, 
>> child_order, child_type).
>> In theory you could even generate the DB schema from an XML schema 
>> and evaluate it with XPath-like expressions.
>>
>> But why put yourself to such bother? I have never found a good reason 
>> to do this sort of thing.
>>
>> cheers
>>
>> andrew
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 7: don't forget to increase your free space map settings
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 7: don't forget to increase your free space map settings
>


Thu, 23 Oct 2003 07:40:33 -0400
Andrew Dunstan <andrew@dunslane.net>
Re: is GiST still alive?
On Thu, 23 Oct 2003, Andrew Dunstan wrote:
> Storing the XML text has problems - you have to parse it every time you
> want something - that has to cause a huge performance hit.
You couldn't have said better what I meant.
I store the xml already parsed. You can navigate right along. To the
parent, the previous, the next elemnt or the first or last child.

> I use XML a lot for all sorts of purposes, but it is appropriate for
> data transfer rather than data storage, IMNSHO.
Right now, you're quite right. But I want to change that.

-- 
Gregor Zeitlinger
gregor@zeitlinger.de


Thu, 23 Oct 2003 13:51:52 +0200 (CEST)
Gregor Zeitlinger <zeitling@informatik.hu-berlin.de>
Re: is GiST still alive?
>>4. Extend the contrib/ltree gist-based tree indexing scheme to work on
>>xml and hence the operations in no.3 above are really fast...
> 
> but then, the plain xml data is still stored in a database colum, if I
> understand correctly?

Yep - which to me seems to be the most useful way to store it :)

Chris



Thu, 23 Oct 2003 21:09:11 +0800
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: is GiST still alive?
> You couldn't have said better what I meant.
> I store the xml already parsed. You can navigate right along. To the
> parent, the previous, the next elemnt or the first or last child.

Which is the whole point of indexing it...

>>I use XML a lot for all sorts of purposes, but it is appropriate for
>>data transfer rather than data storage, IMNSHO.
> 
> Right now, you're quite right. But I want to change that.

No point, it's a data exchange format, it's not usefull for data storage.

Chris



Thu, 23 Oct 2003 21:10:46 +0800
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: is GiST still alive?
> *nod* I have tried this several times - it just doesn't work well, 
> because the maps are too different.
> 
> You could do something like this:
>    . a table for each element type, fields being the attributes, plus 
> the node id.
>    . a table to tie everything together (parent_id, child_id, 
> child_order, child_type).
> In theory you could even generate the DB schema from an XML schema and 
> evaluate it with XPath-like expressions.

Although bits separated from a topic, I am developing XML database enviroment 
based on PostgreSQL, XpSQL.
http://gborg.postgresql.org/project/xpsql/

The enviroment decompose XML documents into fragments and uses PostgreSQL 
to store them in a set of relations. 
Users do not have to know that they are stored as relations.

It provides a variety of means to access XML Documents;
(a) DOM functions to build and traverse XML Documents
(b) XPath engine to extract infomation from documents
(c) update functions to modify documents

Although the development is on going, 
the performance is better than Apache Xindice.
#if you require, I'll send you the result after.

- points -
* node labeling is based on dewey decimal classification,  
 so updating performance is well.
* The index method is general (postgres) B-tree.

this is a programing image.
http://gborg.postgresql.org/project/xpsql/cvs/co.php/XpSQL/doc/USAGE?r=1.4

+-------------------------------------------------------------------+
Makoto Yui <yuin@bb.din.or.jp>
Key fingerprint = 6462 E285 97D8 1323 40C4  F9E5 EB0F 9DE6 1713 219E
+-------------------------------------------------------------------+

Thu, 23 Oct 2003 22:38:09 +0900
Makoto Yui <yuin@bb.din.or.jp>
Re: is GiST still alive?
On Thu, 23 Oct 2003, Christopher Kings-Lynne wrote:
> > You couldn't have said better what I meant.
> > I store the xml already parsed. You can navigate right along. To the
> > parent, the previous, the next elemnt or the first or last child.
>
> Which is the whole point of indexing it...
not quite. Indexing can solve some of the problems, not all.
1) You have to update the index every time you modify the data. My custom
format serves as an index for some queries.

2) The page format is designed in such a way that modifications
(insertion, deletion) are as fast as the original parsing. I'm not sure
how that stacks up to modifying data in a column. I guess it depens on the
strategy to store very large strings in columns.

> >>I use XML a lot for all sorts of purposes, but it is appropriate for
> >>data transfer rather than data storage, IMNSHO.
> >
> > Right now, you're quite right. But I want to change that.
>
> No point, it's a data exchange format, it's not usefull for data storage.
Well, neither one is a data exchange format only or a data storage format
only.
Rather, the difference is that relations are designed to store structued
data while xml is desinged to store semi-structued (not so regular) data.

Which is better suited for data exchange is a matter of convention (where
xml seems to be good), while efficiency and other features of an
implementation determine, which one is suited for data storage.

If your point is that currently xml is not suited for storage, because
there are more efficent RDBMS than xml databases, I agree. Otherwise, I
don't see your point.

-- 
Gregor Zeitlinger
gregor@zeitlinger.de


Thu, 23 Oct 2003 15:59:49 +0200 (CEST)
Gregor Zeitlinger <zeitling@informatik.hu-berlin.de>
Re: is GiST still alive?