I was wondering what direction we want to take for replication?  There
is a group at:

	http://gborg.postgresql.org/genpage?replication_research

working on a fairly advanced replication solution that they hope to add
to 7.3.  However, has everyone reviewed that information?  Is this the
direction we want to go?  I was personally impressed by it, but I am no
replication expert.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 22 Feb 2002 10:57:36 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Replication direction
> working on a fairly advanced replication solution that they hope to add
> to 7.3.  However, has everyone reviewed that information?  Is this the
> direction we want to go?  I was personally impressed by it, but I am no
> replication expert.

7.3 is probably a reach.  We're hoping to get master-slave working first
and then multimaster,  high quality multimaster for 7.3 might be too fast
to hope for (even it it takes a year as 7.1 -> 7.2 did).

- Brandon

----------------------------------------------------------------------------
 c: 646-456-5455                                            h: 201-798-4983
 b. palmer,  bpalmer@crimelabs.net           pgp:crimelabs.net/bpalmer.pgp5


Fri, 22 Feb 2002 11:33:34 -0500 (EST)
bpalmer <bpalmer@crimelabs.net>
Re: Replication direction
One thought I had about replication is base it off of the WAL files. As
they are created, they can be forward into other servers (both on a LAN
and WAN environment) and imported to mirror the originating database. 
(As opposed to having triggers that force writes into replicated servers
on updates, inserts and deletes)

On Fri, 2002-02-22 at 08:57, Bruce Momjian wrote:
> I was wondering what direction we want to take for replication?  There
> is a group at:
> 
> 	http://gborg.postgresql.org/genpage?replication_research
> 
> working on a fairly advanced replication solution that they hope to add
> to 7.3.  However, has everyone reviewed that information?  Is this the
> direction we want to go?  I was personally impressed by it, but I am no
> replication expert.
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html
-- 

Virtually, 
Ned Wolpert <ned.wolpert@knowledgenet.com>

D08C2F45:  28E7 56CB 58AC C622 5A51  3C42 8B2B 2739 D08C 2F45
22 Feb 2002 09:49:58 -0700
Ned Wolpert <ned.wolpert@knowledgenet.com>
Re: Replication direction
Ned Wolpert wrote:
> One thought I had about replication is base it off of the WAL files. As
> they are created, they can be forward into other servers (both on a LAN
> and WAN environment) and imported to mirror the originating database. 
> (As opposed to having triggers that force writes into replicated servers
> on updates, inserts and deletes)

OK, let me chime in here and give my observations, then I will address
the WAL issue.  First, in all the discussions I have heard, the final
replication solution needs to be:

	multi-master
	good performance
	low maintenance (no conflict resolution rules)

Now, at the URL I mentioned, there are 5+ other pgsql replication
solutions listed:

	http://gborg.postgresql.org/genpage?replication_research

However, I don't think any of these meet this criteria, particularly
because they are not integrated into the backend code.  That doesn't
mean they aren't valuable, but that they don't represent a final
solution for pgsql replication.

As far as I know, the only working project that will meet these needs is
the one at that URL based on Postgres-R.  What impressed me about it was
that it did not use two-phase commit, nor conflict resolution rules, but
rather used a queue of changes that gets broadcast to all the servers in
the replication cluster.  This seems to allow high throughput with low
maintenance.

The actual direction of the project was decided by researching all the
current solutions and educational research to find the strengths and
weaknesses of each one.

As far as WAL, it currently has the tid's in the WAL file which will not
match other servers unless those servers are read-only.  It is tempting
to think that WAL or some other existing mechanism will allow us to do
replication cheaply, but it is my understanding that the interactions of
multiple write server is quite complicated and requires an amount of
overhead mechanism that is similar to our current transaction mechanisms
that allow multiple people to modify the same table.

People know I am all for the quick solution if it fits into our existing
code, but I am afraid replication is one of those items that has to be
designed from the ground up on a foundation that is backed by research
and experts in the field.  I think this replication project has the
potential to give us a replication capability that is better than
the commercial offering of other databases.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 22 Feb 2002 12:20:29 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Replication direction
bpalmer wrote:
> > working on a fairly advanced replication solution that they hope to add
> > to 7.3.  However, has everyone reviewed that information?  Is this the
> > direction we want to go?  I was personally impressed by it, but I am no
> > replication expert.
> 
> 7.3 is probably a reach.  We're hoping to get master-slave working first
> and then multimaster,  high quality multimaster for 7.3 might be too fast
> to hope for (even it it takes a year as 7.1 -> 7.2 did).

Sorry, yes master-slave for 7.3.  The idea is that the code will be
there in 7.3 for master-slave, and we will add on that for 7.4 and
later, but that clearly the infrastructure is there for master-master
from the start.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 22 Feb 2002 12:22:12 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Replication direction
> One thought I had about replication is base it off of the WAL files. As
> they are created, they can be forward into other servers (both on a LAN
> and WAN environment) and imported to mirror the originating database.
> (As opposed to having triggers that force writes into replicated servers
> on updates, inserts and deletes)

In that case,  we would have a master / slave environment.  Also,  how do
we guarantee that the slaves get the info and are able to commit it?  Next
problem:  a wal has all tuples changed.  though "delete from foo" is only
a few characters,  the WAL would be however many rows are in the database
(that were deleted).

That said,  however,  we are looking at using some of the same methods
that are used to generate the WAL info to use for packaging info to go to
other hosts in the network (for pg-repl).

- Brandon

----------------------------------------------------------------------------
 c: 646-456-5455                                            h: 201-798-4983
 b. palmer,  bpalmer@crimelabs.net           pgp:crimelabs.net/bpalmer.pgp5


Fri, 22 Feb 2002 12:42:59 -0500 (EST)
bpalmer <bpalmer@crimelabs.net>
Re: Replication direction
On Fri, 2002-02-22 at 10:20, Bruce Momjian wrote: 
> As far as WAL, it currently has the tid's in the WAL file which will not
> match other servers unless those servers are read-only.  It is tempting
> to think that WAL or some other existing mechanism will allow us to do
> replication cheaply, but it is my understanding that the interactions of
> multiple write server is quite complicated and requires an amount of
> overhead mechanism that is similar to our current transaction mechanisms
> that allow multiple people to modify the same table.

Drat.  I was hoping that the WAL was an overlooked 'easy' solution...
but from the sound of this thread, the WAL misses some key issues.  And
it doesn't help the master-master environment at all, which I forgot. 
Ah well...
 
> People know I am all for the quick solution if it fits into our existing
> code, but I am afraid replication is one of those items that has to be
> designed from the ground up on a foundation that is backed by research
> and experts in the field.  I think this replication project has the
> potential to give us a replication capability that is better than
> the commercial offering of other databases.



-- 

Virtually, 
Ned Wolpert <ned.wolpert@knowledgenet.com>

D08C2F45:  28E7 56CB 58AC C622 5A51  3C42 8B2B 2739 D08C 2F45
22 Feb 2002 11:22:26 -0700
Ned Wolpert <ned.wolpert@knowledgenet.com>
Re: Replication direction
> The actual direction of the project was decided by researching all
> the current solutions and educational research to find the strengths
> and weaknesses of each one.

For those that are interested,

http://www.spread.org/

Spread toolkit is what's being used.  -sc

-- 
Sean Chittenden

Fri, 22 Feb 2002 10:25:04 -0800
Sean Chittenden <sean@chittenden.org>
Re: Replication direction