Dear all,

A friend of mine needs to import and query a very large amount of data, coming 
from real-time acquisition systems. The database is growing fast, several 
Tetra-bytes a day.

What is the advancement of the community in the field of very large databases? 
Could you point out to me some useful information, techdocs, etc...?

Are there working groups, private fundings in this precise field?

Also, in Brussels, at FOSDEM, I met a very brilliant Russian student 
interested in large indexing technologies. Could he possibly contact me?

Best regards,
Jean-Michel POURE

Thu, 22 May 2003 10:38:19 +0200
Jean-Michel POURE <jm.poure@freesurf.fr>
Tetra-bytes database / large indexes needs
Jean-Michel POURE kirjutas N, 22.05.2003 kell 11:38:
> Dear all,
> 
> A friend of mine needs to import and query a very large amount of data, coming 
> from real-time acquisition systems.

What kind of querying does he need ?

You could check out Telegraph for continuous queries

http://telegraph.cs.berkeley.edu/

>  The database is growing fast, several 
> Tetra-bytes a day.

if you mean terabytes TB then 1 TB/day ~= 12.7 MB/sec, just about as
fast as you can write on an average ide drive if you do nothing else, or
what can come in over a 100 base T ethernet.

even in ide disks (the biggest available disks apiece) you have to add 5
disks a day just to store the incoming 1TB/day.

> What is the advancement of the community in the field of very large databases? 
> Could you point out to me some useful information, techdocs, etc...?

no ;)

> Are there working groups, private fundings in this precise field?

dunno..


You are possibly on the verge of impossible ?

If you can hardly write down the data due to physical constraints
(memory and bus speeds) , it is very hard to also query it, unless you
deploy some terribly clever techniques which extract and compress your
data before it even gets to your DB ;)

Perhaps you (or your friend) are after filtering and not queries at all
?

--------------
Hannu


22 May 2003 22:50:25 +0300
Hannu Krosing <hannu@tm.ee>
Re: [HACKERS] Tetra-bytes database / large indexes needs
>>>>> "Hannu" == Hannu Krosing <hannu@tm.ee> writes:


    Hannu> You are possibly on the verge of impossible ?

Eeep .. it's because of his data rates that I stayed quiet :-)

-- 
Pip-pip
Sailesh
http://www.cs.berkeley.edu/~sailesh


22 May 2003 15:16:18 -0700
Sailesh Krishnamurthy <sailesh@cs.berkeley.edu>
Re: [HACKERS] Tetra-bytes database / large indexes needs


Jean-Michel POURE wrote:

>Dear all,
>
>A friend of mine needs to import and query a very large amount of data, coming 
>from real-time acquisition systems. The database is growing fast, several 
>Tetra-bytes a day.
>
>What is the advancement of the community in the field of very large databases? 
>Could you point out to me some useful information, techdocs, etc...?
>
>Are there working groups, private fundings in this precise field?
>
>Also, in Brussels, at FOSDEM, I met a very brilliant Russian student 
>interested in large indexing technologies. Could he possibly contact me?
>
>  
>

I don't know why but as I was reading this I remembered Frank Sinatra's 
song "Fly Me To The Moon".

Fly me to the moon
Let me play among the stars
Let me see what spring is like
On a-Jupiter and Mars
In other words, hold my Tera_hand
In other words, baby, Tera_kiss me

:-)




Fri, 23 May 2003 10:14:32 +0300
Antonis Antoniou <a.antoniou@albourne.com>
Re: [HACKERS] Tetra-bytes database / large indexes needs