
Is it my imagination, or is there a problem with the way pg_dump uses off_t 
etc. My understanding is that off_t may be 64 bits on systems with 32 bit 
ints. But it looks like pg_dump writes them as 4 byte values in all cases. 
It also reads them as 4 byte values. Does this seem like a problem to 
anybody else?



----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Tue, 01 Oct 2002 22:13:04 +1000
Philip Warner <pjw@rhyme.com.au>
pg_dump and large files - is this a problem?
Philip Warner <pjw@rhyme.com.au> writes:
> Is it my imagination, or is there a problem with the way pg_dump uses off_t 
> etc. My understanding is that off_t may be 64 bits on systems with 32 bit 
> ints. But it looks like pg_dump writes them as 4 byte values in all cases. 
> It also reads them as 4 byte values. Does this seem like a problem to 
> anybody else?

Yes, it does --- the implication is that the custom format, at least,
can't support dumps > 4Gb.  What exactly is pg_dump writing off_t's
into files for; maybe there's not really a problem?

If there is a problem, seems like we'd better fix it.  Perhaps there
needs to be something in the header to tell the reader the sizeof
off_t.

			regards, tom lane

Tue, 01 Oct 2002 09:59:52 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: pg_dump and large files - is this a problem? 
Tom Lane wrote:
> Philip Warner <pjw@rhyme.com.au> writes:
> > Is it my imagination, or is there a problem with the way pg_dump uses off_t 
> > etc. My understanding is that off_t may be 64 bits on systems with 32 bit 
> > ints. But it looks like pg_dump writes them as 4 byte values in all cases. 
> > It also reads them as 4 byte values. Does this seem like a problem to 
> > anybody else?
> 
> Yes, it does --- the implication is that the custom format, at least,
> can't support dumps > 4Gb.  What exactly is pg_dump writing off_t's
> into files for; maybe there's not really a problem?
> 
> If there is a problem, seems like we'd better fix it.  Perhaps there
> needs to be something in the header to tell the reader the sizeof
> off_t.

BSD/OS has 64-bit off_t's so it does support large files.  Is there
something I can test?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tue, 1 Oct 2002 11:20:44 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
Tom Lane writes:

> Yes, it does --- the implication is that the custom format, at least,
> can't support dumps > 4Gb.  What exactly is pg_dump writing off_t's
> into files for; maybe there's not really a problem?

That's kind of what I was wondering, too.

Not that it's an excuse, but I think that large file access through zlib
won't work anyway.  Zlib uses the integer types in fairly random ways.

-- 
Peter Eisentraut   peter_e@gmx.net


Tue, 1 Oct 2002 22:03:54 +0200 (CEST)
Peter Eisentraut <peter_e@gmx.net>
Re: pg_dump and large files - is this a problem? 
At 09:59 AM 1/10/2002 -0400, Tom Lane wrote:
>If there is a problem, seems like we'd better fix it.  Perhaps there
>needs to be something in the header to tell the reader the sizeof
>off_t.

Yes, and do the peripheral stuff to support old archives etc. We also need 
to be careful about the places where we do file-position-arithmetic - if 
there are any, I can't recall.

I am not sure we need to worry about whether zlib supports large files 
since I am pretty sure we don't use zlib for file IO - we just pass it 
in-memory blocks; so it should work no matter how much data is in the stream.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Wed, 02 Oct 2002 09:42:16 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem? 
At 11:20 AM 1/10/2002 -0400, Bruce Momjian wrote:
>BSD/OS has 64-bit off_t's so it does support large files.  Is there
>something I can test?

Not really since it saves only the first 32 bits of the 64 bit positions it 
will do no worse than a version that supports 32 bits only. It might even 
do slightly better. When this is sorted out, we need to verify that:

- large dump files are restorable

- dump files with 32 bit off_t restore properly on systems with 64 biy off_t

- dump files with 64 bit off_t restore properly on systems with 32 bit off_tAS
LONG AS the offsets are less than 32 bits.

- old dump files restore properly.

- new dump files have a new version number so that old pg_restore will not 
try to restore them.

We probably need to add Read/WriteOffset to pg_backup_archiver.c to read 
the appropriate sized value from a dump file, in the same way that 
Read/WriteInt works now.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Wed, 02 Oct 2002 09:54:13 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem?
At 09:42 AM 2/10/2002 +1000, Philip Warner wrote:
>Yes, and do the peripheral stuff to support old archives etc.

Does silence mean people agree? Does it also mean someone is doing this 
(eg. whoever did the off_t support)? Or does it mean somebody else needs to 
do it?




----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 03 Oct 2002 00:07:16 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem? 
Philip Warner <pjw@rhyme.com.au> writes:
> At 09:42 AM 2/10/2002 +1000, Philip Warner wrote:
>> Yes, and do the peripheral stuff to support old archives etc.

> Does silence mean people agree? Does it also mean someone is doing this 
> (eg. whoever did the off_t support)? Or does it mean somebody else needs to 
> do it?

It needs to get done; AFAIK no one has stepped up to do it.  Do you want
to?

			regards, tom lane

Wed, 02 Oct 2002 11:06:31 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: pg_dump and large files - is this a problem? 
Philip Warner wrote:
> At 09:42 AM 2/10/2002 +1000, Philip Warner wrote:
> >Yes, and do the peripheral stuff to support old archives etc.
> 
> Does silence mean people agree? Does it also mean someone is doing this 
> (eg. whoever did the off_t support)? Or does it mean somebody else needs to 
> do it?

Added to open items:

	Fix pg_dump to handle 64-bit off_t offsets for custom format


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

                               P O S T G R E S Q L

                          7 . 3  O P E N    I T E M S


Current at ftp://momjian.postgresql.org/pub/postgresql/open_items.

Source Code Changes
-------------------
Schema handling - ready? interfaces? client apps?
Drop column handling - ready for all clients, apps?
Fix BeOS, QNX4 ports
Fix AIX large file compile failure of 2002-09-11 (Andreas)
Get bison upgrade on postgresql.org for ecpg only (Marc)
Fix vacuum btree bug (Tom)
Fix client apps for autocommit = off
Change log_min_error_statement to be off by default (Gavin)
Fix return tuple counts/oid/tag for rules, SPI
Add schema dump option to pg_dump
Make SET not start a transaction with autocommit off, document it
Remove GRANT EXECUTE to all /contrib functions?
Change NUMERIC to have 16 digit precision
Handle CREATE CONSTRAINT TRIGGER without FROM in loads from old db's
Fix pg_dump to handle 64-bit off_t offsets for custom format

On Going
--------
Security audit


Documentation Changes
---------------------
Document need to add permissions to loaded functions and languages
Move documation to gborg for moved projects


7.2.X
-----
CLOG
WAL checkpoint
Linux mktime()
Wed, 2 Oct 2002 11:45:40 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: pg_dump and large files - is this a problem?
At 11:06 AM 2/10/2002 -0400, Tom Lane wrote:
>It needs to get done; AFAIK no one has stepped up to do it.  Do you want
>to?

I'll have a look; my main concern at the moment is that off_t and size_t 
are totally non-committal as to structure; in particular I can probably 
safely assume that they are unsigned, but can I assume that they have the 
same endian--ness as int etc?

If so, then will it be valid to just read/write each byte in endian order? 
How likely is it that the 64 bit value will actually be implemented as a 
structure like:

off_t { int lo; int hi; }

which effectively ignores endian-ness at the 32 bit scale?



----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.B.N. 75 008 659 498)          |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|
                                  |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Thu, 03 Oct 2002 14:39:13 +1000
Philip Warner <pjw@rhyme.com.au>
Re: pg_dump and large files - is this a problem? 