
> then on restore once all the files are restored move the
> pg_control.backup to its original name.  That gives us the checkpoint
> wal/offset but how do we get the start/stop information.  Is that not
> required?

The checkpoint wal/offset is in pg_control, that is sufficient start 
information. The stop info is only necessary as a safeguard.

> Do we need a checkpoint after the archiving
> starts but before the backup begins?

No.

> Also, when you are in recovery mode, how do you get out of recovery
> mode, meaning if you have a power failure, how do you prevent the system
> from doing another recovery?  Do you remove the recovery.conf file?

pg_control could be updated during rollforward (only if that actually 
does a checkpoint). So if pg_control is also the recovery start info, then 
we can continue from there if we have a power failure.
For the first release it would imho also be ok to simply start over if
you loose power.

I think the filename 'recovery.conf' is misleading, since it is not a 
static configuration file, but a command file for one recovery.
How about 'recovery.command' then 'recovery.inprogress', and on recovery 
completion it should be renamed to 'recovery.done'

Andreas

Fri, 16 Jul 2004 17:25:56 +0200
"Zeugswetter Andreas SB SD" <ZeugswetterA@spardat.at>
Re: Point in Time Recovery
"Zeugswetter Andreas SB SD" <ZeugswetterA@spardat.at> writes:
>> Do we need a checkpoint after the archiving
>> starts but before the backup begins?

> No.

Actually yes.  You have to start at a checkpoint record when replaying
the log, so if no checkpoint occurred between starting to archive WAL
and starting the tar backup, you have a useless backup.

It would be reasonable to issue a CHECKPOINT just before starting the
backup as part of the standard operating procedure for taking PITR
dumps.  We need not require this, but it would help to avoid this
particular sort of mistake; and of course it might save a little bit of
replay effort if the backup is ever used.

As far as the business about copying pg_control first goes: there is
another way to think about it, which is to copy pg_control to another
place that will be included in your backup.  For example the standard
backup procedure could be

1. [somewhat optional] Issue CHECKPOINT and wait till it finishes.

2. cp $PGDATA/global/pg_control $PGDATA/pg_control.dump

3. tar cf /dev/mt $PGDATA

4. do something to record ending WAL position

If we standardized on this way, then the tar archive would automatically
contain the pre-backup checkpoint position in ./pg_control.dump, and
there is no need for any special assumptions about the order in which
tar processes things.

However, once you decide to do things like that, there is no reason why
the copied file has to be an exact image of pg_control.  I claim it
would be more useful if the copied file were plain text so that you
could just "cat" it to find out the starting WAL position; that would
let you determine without any special tools what range of WAL archive
files you are going to need to bring back from your archives.

This is pretty much the same chain of reasoning that Bruce and I went
through yesterday to come up with the idea of putting a label file
inside the tar backups.  We concluded that it'd be worth putting
both the backup starting time and the checkpoint WAL position into
the label file --- the starting time isn't needed for restore but
might be really helpful as documentation, if you needed to verify
which dump file was which.

			regards, tom lane

Fri, 16 Jul 2004 11:47:42 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Point in Time Recovery 
On Fri, 2004-07-16 at 16:25, Zeugswetter Andreas SB SD wrote:

> I think the filename 'recovery.conf' is misleading, since it is not a 
> static configuration file, but a command file for one recovery.
> How about 'recovery.command' then 'recovery.inprogress', and on recovery 
> completion it should be renamed to 'recovery.done'

You understand this and your assessment is correct.

recovery.conf isn't an attempt to persist information. It is a means of
delivering a set of parameters to the recovery process, as well as
signalling overall that archive recovery is required (because the system
default remains the same, which is to recover from the logs it has
locally available to it).

I originally offered a design which used a command, similar to
DB2/Oracle...that was overruled as too complex. The (whatever you call
it) file is just a very simple way of specifying whats required.

There is more to be said here...clearly some explanations are required
and I will provide those later...

Best regards, Simon Riggs


Fri, 16 Jul 2004 18:13:02 +0100
Simon Riggs <simon@2ndquadrant.com>
Re: Point in Time Recovery
On Fri, 2004-07-16 at 16:47, Tom Lane wrote:
> As far as the business about copying pg_control first goes: there is
> another way to think about it, which is to copy pg_control to another
> place that will be included in your backup.  For example the standard
> backup procedure could be
> 
> 1. [somewhat optional] Issue CHECKPOINT and wait till it finishes.
> 
> 2. cp $PGDATA/global/pg_control $PGDATA/pg_control.dump
> 
> 3. tar cf /dev/mt $PGDATA
> 
> 4. do something to record ending WAL position
> 
> If we standardized on this way, then the tar archive would automatically
> contain the pre-backup checkpoint position in ./pg_control.dump, and
> there is no need for any special assumptions about the order in which
> tar processes things.
> 

Sounds good. That would be familiar to Oracle DBAs doing BACKUP
CONTROLFILE. We can document that and off it as a suggested procedure.

> However, once you decide to do things like that, there is no reason why
> the copied file has to be an exact image of pg_control.  I claim it
> would be more useful if the copied file were plain text so that you
> could just "cat" it to find out the starting WAL position; that would
> let you determine without any special tools what range of WAL archive
> files you are going to need to bring back from your archives.

I wouldn't be in favour of a manual mechanism. If you want an automated
mechanism, whats wrong with using the one thats already there? You can
use pg_controldata to read the controlfile, again whats wrong with that?

We agreed some time back that an off-line xlog file inspector would be
required to allow us to inspect the logs and make a decision about where
to end recovery. You'd still need that.

It's scary enough having to specify the end point, let alone having to
specify the starting point as well.

At your request, and with Bruce's idea, I designed and built the
recovery system so that you don't need to know what range of xlogs to
bring back. You just run it, it brings back the right files from archive
and does recovery with them, then cleans up - and it works without
running out of disk space on long recoveries.

I've built it now and it works...

> This is pretty much the same chain of reasoning that Bruce and I went
> through yesterday to come up with the idea of putting a label file
> inside the tar backups.  We concluded that it'd be worth putting
> both the backup starting time and the checkpoint WAL position into
> the label file --- the starting time isn't needed for restore but
> might be really helpful as documentation, if you needed to verify
> which dump file was which.

...if you are doing tar backups...what will you do if you're not using
that mechanism?

If you are: It's common practice to make up a backup filename from
elements such as systemname, databasename, date and time etc. That gives
you the start time, the file last mod date gives you the end time. 

I think its perfectly fine for everybody to do backups any way they
please. There are many licenced variants of PostgreSQL and it might be
appropriate in those to specify particular ways of doing things.

I'll be trusting the management of backup metadata and storage media to
a solution designed for the purpose (open or closed source), just as
I'll be trusting my data to a database solution designed for that
purpose. That for me is one of the good things about PostgreSQL - we use
the filesystem, we don't write our own, we provide language interfaces
not invent our own proprietary server language etc..

Best Regards, Simon Riggs


Fri, 16 Jul 2004 23:35:58 +0100
Simon Riggs <simon@2ndquadrant.com>
Re: Point in Time Recovery

OK, I think I have some solid ideas and reasons for them.

First, I think we need server-side functions to call when we start/stop
the backup.  The advantage of these server-side functions is that they
will do the required work of recording the pg_control values and
creating needed files with little chance for user error.  It also allows
us to change the internal operations in later releases without requiring
admins to change their procedures.  We are even able to adjust the
internal operation in minor releases without forcing a new procedure on
users.

Second, I think once we start a restore, we should rename recovery.conf
to recovery.in_progress, and when complete rename that to
recovery.done.  If the postmaster starts and sees recovery.in_progress,
it will fail to start knowing its recovery was interrupted.  This allows
the admin to take appropriate action.  (I am not sure what that action
would be. Does he bring back the backup files or just keep going?)

Third, I think we need to put a file in the archive location once we
complete a backup, recording the start/stop xid and wal/offsets.  This
gives the admin documentation on what archive logs to keep and what xids
are available for recovery.  Ideally the recover program would read that
file and check the recover xid to make sure it is after the stop xid
recorded in the file.

How would the recover program know the name of that file?  We need to
create it in /data with start contents before the backup, then complete
it with end contents and archive it.

What should we name it?  Ideally it would be named by the WAL
name/offset of the start so it orders in the proper spot in the archive
file listing, e.g.:

	000000000000093a
	000000000000093b
	000000000000093b.032b9.start
	000000000000093c

Are people going to know they need 000000000000093b for
000000000000093b.032b9.start?  I hope so.  Another idea is to do:


	000000000000093a.xlog
	000000000000093b.032b9.start
	000000000000093b.xlog
	000000000000093c.xlog

This would order properly.  It might be a very good idea to add
extensions to these log files now that we are archiving them in strange
places.  In fact, maybe we should use *.pg_xlog to document the
directory they came from.

---------------------------------------------------------------------------


Simon Riggs wrote:
> On Fri, 2004-07-16 at 16:47, Tom Lane wrote:
> > As far as the business about copying pg_control first goes: there is
> > another way to think about it, which is to copy pg_control to another
> > place that will be included in your backup.  For example the standard
> > backup procedure could be
> > 
> > 1. [somewhat optional] Issue CHECKPOINT and wait till it finishes.
> > 
> > 2. cp $PGDATA/global/pg_control $PGDATA/pg_control.dump
> > 
> > 3. tar cf /dev/mt $PGDATA
> > 
> > 4. do something to record ending WAL position
> > 
> > If we standardized on this way, then the tar archive would automatically
> > contain the pre-backup checkpoint position in ./pg_control.dump, and
> > there is no need for any special assumptions about the order in which
> > tar processes things.
> > 
> 
> Sounds good. That would be familiar to Oracle DBAs doing BACKUP
> CONTROLFILE. We can document that and off it as a suggested procedure.
> 
> > However, once you decide to do things like that, there is no reason why
> > the copied file has to be an exact image of pg_control.  I claim it
> > would be more useful if the copied file were plain text so that you
> > could just "cat" it to find out the starting WAL position; that would
> > let you determine without any special tools what range of WAL archive
> > files you are going to need to bring back from your archives.
> 
> I wouldn't be in favour of a manual mechanism. If you want an automated
> mechanism, whats wrong with using the one thats already there? You can
> use pg_controldata to read the controlfile, again whats wrong with that?
> 
> We agreed some time back that an off-line xlog file inspector would be
> required to allow us to inspect the logs and make a decision about where
> to end recovery. You'd still need that.
> 
> It's scary enough having to specify the end point, let alone having to
> specify the starting point as well.
> 
> At your request, and with Bruce's idea, I designed and built the
> recovery system so that you don't need to know what range of xlogs to
> bring back. You just run it, it brings back the right files from archive
> and does recovery with them, then cleans up - and it works without
> running out of disk space on long recoveries.
> 
> I've built it now and it works...
> 
> > This is pretty much the same chain of reasoning that Bruce and I went
> > through yesterday to come up with the idea of putting a label file
> > inside the tar backups.  We concluded that it'd be worth putting
> > both the backup starting time and the checkpoint WAL position into
> > the label file --- the starting time isn't needed for restore but
> > might be really helpful as documentation, if you needed to verify
> > which dump file was which.
> 
> ...if you are doing tar backups...what will you do if you're not using
> that mechanism?
> 
> If you are: It's common practice to make up a backup filename from
> elements such as systemname, databasename, date and time etc. That gives
> you the start time, the file last mod date gives you the end time. 
> 
> I think its perfectly fine for everybody to do backups any way they
> please. There are many licenced variants of PostgreSQL and it might be
> appropriate in those to specify particular ways of doing things.
> 
> I'll be trusting the management of backup metadata and storage media to
> a solution designed for the purpose (open or closed source), just as
> I'll be trusting my data to a database solution designed for that
> purpose. That for me is one of the good things about PostgreSQL - we use
> the filesystem, we don't write our own, we provide language interfaces
> not invent our own proprietary server language etc..
> 
> Best Regards, Simon Riggs
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 16 Jul 2004 19:57:39 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Point in Time Recovery
On Sat, 2004-07-17 at 00:57, Bruce Momjian wrote:
> OK, I think I have some solid ideas and reasons for them.
> 

Sorry for taking so long to reply...

> First, I think we need server-side functions to call when we start/stop
> the backup.  The advantage of these server-side functions is that they
> will do the required work of recording the pg_control values and
> creating needed files with little chance for user error.  It also allows
> us to change the internal operations in later releases without requiring
> admins to change their procedures.  We are even able to adjust the
> internal operation in minor releases without forcing a new procedure on
> users.

Yes, I think we should go down this route. ....there's a "but" and that
is we don't absolutely need it for correctness....and so I must decline
adding it to THIS release. I don't imagine I'll stop be associated with
this code for a while yet....

Can we recommend that users should expect to have to call a start and
end backup routine in later releases? Don't expect you'll agree to
that..

> 
> Second, I think once we start a restore, we should rename recovery.conf
> to recovery.in_progress, and when complete rename that to
> recovery.done.  If the postmaster starts and sees recovery.in_progress,
> it will fail to start knowing its recovery was interrupted.  This allows
> the admin to take appropriate action.  (I am not sure what that action
> would be. Does he bring back the backup files or just keep going?)
> 

Superceded by Tom's actions. Two states are required: start and stop.
Recovery isn't going to be checkpoint-restartable anytime soon, IMHO.

Best regards, Simon Riggs


Mon, 19 Jul 2004 23:57:17 +0100
Simon Riggs <simon@2ndquadrant.com>
Re: Point in Time Recovery
Simon Riggs wrote:
> On Sat, 2004-07-17 at 00:57, Bruce Momjian wrote:
> > OK, I think I have some solid ideas and reasons for them.
> > 
> 
> Sorry for taking so long to reply...
> 
> > First, I think we need server-side functions to call when we start/stop
> > the backup.  The advantage of these server-side functions is that they
> > will do the required work of recording the pg_control values and
> > creating needed files with little chance for user error.  It also allows
> > us to change the internal operations in later releases without requiring
> > admins to change their procedures.  We are even able to adjust the
> > internal operation in minor releases without forcing a new procedure on
> > users.
> 
> Yes, I think we should go down this route. ....there's a "but" and that
> is we don't absolutely need it for correctness....and so I must decline
> adding it to THIS release. I don't imagine I'll stop be associated with
> this code for a while yet....
> 
> Can we recommend that users should expect to have to call a start and
> end backup routine in later releases? Don't expect you'll agree to
> that..

I guess my big question is that if we don't do this for 7.5 how will
people doing restores know if the xid they specify is valid for the
backup they used.  If we recover to most recent time, is there any check
that will tell them their backup is invalid because there are no archive
records that span the time of their backup?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 20 Jul 2004 17:08:20 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Point in Time Recovery

We need someone to code two backend functions to complete PITR.  The
function would be called at start/stop of backup of the data directory.
The functions would be checked during restore to make sure the requested
xid is not between the start/stop xids of the backup.  They would also
contain timestamps so the admin can easily review the archive directory.

The start needs to call checkpoint and create file in the data directory
that contains a few server parameters.  At backup stop the function
needs to move the file to pg_xlog and set the *.ready archive flag so it
is archived.

As for checking during recover, the file needs to be retrieved and
checked to see the xid recovery is valid.  Tom and I can help you with
that detail.

DOn't worry about all the details of the email below.  It is just a
general summary.  We can give you details once you volunteer.

---------------------------------------------------------------------------

Bruce Momjian wrote:
> 
> OK, I think I have some solid ideas and reasons for them.
> 
> First, I think we need server-side functions to call when we start/stop
> the backup.  The advantage of these server-side functions is that they
> will do the required work of recording the pg_control values and
> creating needed files with little chance for user error.  It also allows
> us to change the internal operations in later releases without requiring
> admins to change their procedures.  We are even able to adjust the
> internal operation in minor releases without forcing a new procedure on
> users.
> 
> Second, I think once we start a restore, we should rename recovery.conf
> to recovery.in_progress, and when complete rename that to
> recovery.done.  If the postmaster starts and sees recovery.in_progress,
> it will fail to start knowing its recovery was interrupted.  This allows
> the admin to take appropriate action.  (I am not sure what that action
> would be. Does he bring back the backup files or just keep going?)
> 
> Third, I think we need to put a file in the archive location once we
> complete a backup, recording the start/stop xid and wal/offsets.  This
> gives the admin documentation on what archive logs to keep and what xids
> are available for recovery.  Ideally the recover program would read that
> file and check the recover xid to make sure it is after the stop xid
> recorded in the file.
> 
> How would the recover program know the name of that file?  We need to
> create it in /data with start contents before the backup, then complete
> it with end contents and archive it.
> 
> What should we name it?  Ideally it would be named by the WAL
> name/offset of the start so it orders in the proper spot in the archive
> file listing, e.g.:
> 
> 	000000000000093a
> 	000000000000093b
> 	000000000000093b.032b9.start
> 	000000000000093c
> 
> Are people going to know they need 000000000000093b for
> 000000000000093b.032b9.start?  I hope so.  Another idea is to do:
> 
> 
> 	000000000000093a.xlog
> 	000000000000093b.032b9.start
> 	000000000000093b.xlog
> 	000000000000093c.xlog
> 
> This would order properly.  It might be a very good idea to add
> extensions to these log files now that we are archiving them in strange
> places.  In fact, maybe we should use *.pg_xlog to document the
> directory they came from.
> 
> ---------------------------------------------------------------------------
> 
> 
> Simon Riggs wrote:
> > On Fri, 2004-07-16 at 16:47, Tom Lane wrote:
> > > As far as the business about copying pg_control first goes: there is
> > > another way to think about it, which is to copy pg_control to another
> > > place that will be included in your backup.  For example the standard
> > > backup procedure could be
> > > 
> > > 1. [somewhat optional] Issue CHECKPOINT and wait till it finishes.
> > > 
> > > 2. cp $PGDATA/global/pg_control $PGDATA/pg_control.dump
> > > 
> > > 3. tar cf /dev/mt $PGDATA
> > > 
> > > 4. do something to record ending WAL position
> > > 
> > > If we standardized on this way, then the tar archive would automatically
> > > contain the pre-backup checkpoint position in ./pg_control.dump, and
> > > there is no need for any special assumptions about the order in which
> > > tar processes things.
> > > 
> > 
> > Sounds good. That would be familiar to Oracle DBAs doing BACKUP
> > CONTROLFILE. We can document that and off it as a suggested procedure.
> > 
> > > However, once you decide to do things like that, there is no reason why
> > > the copied file has to be an exact image of pg_control.  I claim it
> > > would be more useful if the copied file were plain text so that you
> > > could just "cat" it to find out the starting WAL position; that would
> > > let you determine without any special tools what range of WAL archive
> > > files you are going to need to bring back from your archives.
> > 
> > I wouldn't be in favour of a manual mechanism. If you want an automated
> > mechanism, whats wrong with using the one thats already there? You can
> > use pg_controldata to read the controlfile, again whats wrong with that?
> > 
> > We agreed some time back that an off-line xlog file inspector would be
> > required to allow us to inspect the logs and make a decision about where
> > to end recovery. You'd still need that.
> > 
> > It's scary enough having to specify the end point, let alone having to
> > specify the starting point as well.
> > 
> > At your request, and with Bruce's idea, I designed and built the
> > recovery system so that you don't need to know what range of xlogs to
> > bring back. You just run it, it brings back the right files from archive
> > and does recovery with them, then cleans up - and it works without
> > running out of disk space on long recoveries.
> > 
> > I've built it now and it works...
> > 
> > > This is pretty much the same chain of reasoning that Bruce and I went
> > > through yesterday to come up with the idea of putting a label file
> > > inside the tar backups.  We concluded that it'd be worth putting
> > > both the backup starting time and the checkpoint WAL position into
> > > the label file --- the starting time isn't needed for restore but
> > > might be really helpful as documentation, if you needed to verify
> > > which dump file was which.
> > 
> > ...if you are doing tar backups...what will you do if you're not using
> > that mechanism?
> > 
> > If you are: It's common practice to make up a backup filename from
> > elements such as systemname, databasename, date and time etc. That gives
> > you the start time, the file last mod date gives you the end time. 
> > 
> > I think its perfectly fine for everybody to do backups any way they
> > please. There are many licenced variants of PostgreSQL and it might be
> > appropriate in those to specify particular ways of doing things.
> > 
> > I'll be trusting the management of backup metadata and storage media to
> > a solution designed for the purpose (open or closed source), just as
> > I'll be trusting my data to a database solution designed for that
> > purpose. That for me is one of the good things about PostgreSQL - we use
> > the filesystem, we don't write our own, we provide language interfaces
> > not invent our own proprietary server language etc..
> > 
> > Best Regards, Simon Riggs
> > 
> 
> -- 
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
>                http://www.postgresql.org/docs/faqs/FAQ.html
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 28 Jul 2004 12:06:32 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Point in Time Recovery
I was wondering about this point - might it not be just as reasonable 
for the copied file to *be* an exact image of pg_control?  Then a very 
simple variant of pg_controldata (or maybe even just adding switches to 
pg_controldata itself) would enable the relevant info to be extracted

P.s : would love to be that volunteer - however up to the eyeballs in 
Business Objects (cringe) and Db2 for the next week or so....

regards

Mark

Bruce Momjian wrote:

>We need someone to code two backend functions to complete PITR. 
>
>>><snippage>
>>>      
>>>
>>>>However, once you decide to do things like that, there is no reason why
>>>>the copied file has to be an exact image of pg_control.  I claim it
>>>>would be more useful if the copied file were plain text so that you
>>>>could just "cat" it to find out the starting WAL position; that would
>>>>let you determine without any special tools what range of WAL archive
>>>>files you are going to need to bring back from your archives.
>>>>        
>>>>
>>>

Thu, 29 Jul 2004 21:34:14 +1200
Mark Kirkwood <markir@coretech.co.nz>
Re: Point in Time Recovery
Mark Kirkwood wrote:
> I was wondering about this point - might it not be just as reasonable 
> for the copied file to *be* an exact image of pg_control?  Then a very 
> simple variant of pg_controldata (or maybe even just adding switches to 
> pg_controldata itself) would enable the relevant info to be extracted

We didn't do that so admins could easily read the file contents.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 29 Jul 2004 13:03:02 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Point in Time Recovery
Quoting Bruce Momjian <pgman@candle.pha.pa.us>:

> Mark Kirkwood wrote:
> > I was wondering about this point - might it not be just as reasonable
> > for the copied file to *be* an exact image of pg_control?  Then a very
> > simple variant of pg_controldata (or maybe even just adding switches to
> > pg_controldata itself) would enable the relevant info to be extracted
>
> We didn't do that so admins could easily read the file contents.
>
Ease of reading is a good thing, no argument there.

However using 'pg_controldata' (or similar) to perform the read is not really
that much harder than using 'cat' - (it is a wee bit harder, I grant you)

When I posted the original mail I was thinking that the pg_control image is good
because it has much more information than just the last wal offset, and could
be used to perform a recovery in the advent of the "actual" pg_control being
unsuitable (e.g. backed up last instead of first on a busy system).

Of couse this thinking didn't make it into the original mail, sorry about that!

regards

Mark



Fri, 30 Jul 2004 15:22:14 +1200
markir@coretech.co.nz
Re: Point in Time Recovery