Hello.

I am working on a Knoppix distribution of my program.
I do have it working, but the size of the database on
disk is becoming a factor. (I copy it to ramdisk
before starting postmaster).

How can I change (minimize) the size of the db on
disk? Are there any parameters I can set? Maybe limit
the size of logs/journals?

Greg (dostatnig@yahoo.com)

__________________________________
Do you Yahoo!?
Free Pop-Up Blocker - Get it now
http://companion.yahoo.com/

Tue, 25 Nov 2003 14:21:45 -0800 (PST)
Grzegorz Dostatni <dostatnig@yahoo.com>
Size on Disk
you do not mention which version of postgresql or which files are problematic,
but just guessing:
- log/checkpoint files: http://www.varlena.com/GeneralBits/Tidbits/perf.html
- index files: see "reindex" command
- table files: see FSM settings and vacuum more, or vacuum full
  (contrib/pg_autovacuum is helpful here)

SELECT CASE WHEN ((SELECT class3.relname FROM pg_class class3, pg_index WHERE ((class1.oid = pg_index.indexrelid) AND (class3.oid = pg_index.indrelid))) IS NOT NULL) THEN (SELECT class3.relname FROM pg_class class3, pg_index WHERE ((class1.oid = pg_index.indexrelid) AND (class3.oid = pg_index.indrelid))) WHEN (class1.relname IS NOT NULL) THEN class1.relname ELSE NULL::name END AS "table", CASE WHEN (class1.relkind = 'r'::"char") THEN NULL::name ELSE class1.relname END AS "index", (class1.relpages * 8) AS "size (KBytes)" FROM pg_class class1 WHERE ((class1.relkind = 'r'::"char") OR (class1.relkind = 'i'::"char")) ORDER BY CASE WHEN ((SELECT class3.relname FROM pg_class class3, pg_index WHERE ((class1.oid = pg_index.indexrelid) AND (class3.oid = pg_index.indrelid))) IS NOT NULL) THEN (SELECT class3.relname FROM pg_class class3, pg_index WHERE ((class1.oid = pg_index.indexrelid) AND (class3.oid = pg_index.indrelid))) WHEN (class1.relname IS NOT NULL) THEN class1.relname ELSE NULL::name END, CASE WHEN (class1.relkind = 'r'::"char") THEN NULL::name ELSE class1.relname END, (class1.relpages * 8);

might help telling you how big things are. i have problems sometimes where
the sizes for indexes are not correct. you can also run "vacuum verbose full"
and see if it is maybe dead tuples sticking around.

On Tue, 25 Nov 2003, Grzegorz Dostatni wrote:
> I am working on a Knoppix distribution of my program.
> I do have it working, but the size of the database on
> disk is becoming a factor. (I copy it to ramdisk
> before starting postmaster).
>
> How can I change (minimize) the size of the db on
> disk? Are there any parameters I can set? Maybe limit
> the size of logs/journals?


Tue, 25 Nov 2003 17:37:48 -0500 (EST)
Chester Kustarz <chester@arbor.net>
Re: Size on Disk
Thank you for your very quick response Chester.

I guess I should clarify something. Knoppix is a
version of linux that runs entirely in memory. I need
as smallest footprint as possible.

Currently the datase is roughly 80 Megs. About half of
the size is stored in pg_xlog directory. I managed to
figure out that those files are transaction log files?
How can I delete them safely? (after creatation the
database will pretty much be read-only - any changes
will be in memory only). Are there any other tricks I
can use to get the size down? Please keep in mind that
I work with a fairly new database - I'm always
recreating it from a dump so only the initial inserts
are in there, no changes or deletions.

Vacuuming helps a bit, but nowhere near as much as I
would hope.

Greg
Greg

__________________________________
Do you Yahoo!?
Free Pop-Up Blocker - Get it now
http://companion.yahoo.com/

Tue, 25 Nov 2003 20:37:52 -0800 (PST)
Grzegorz Dostatni <dostatnig@yahoo.com>
Re: Size on Disk
Grzegorz Dostatni <dostatnig@yahoo.com> writes:
> Currently the datase is roughly 80 Megs. About half of
> the size is stored in pg_xlog directory. I managed to
> figure out that those files are transaction log files?
> How can I delete them safely?

You can NOT.  Don't even think about going there.

What you can do, if you intend only low-update-volume usage,
is reduce checkpoint_segments to reduce the number of WAL files
the system wants to keep around.

			regards, tom lane

Wed, 26 Nov 2003 00:53:27 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: Size on Disk 
Hello

>SELECT CASE WHEN ((SELECT class3.relname FROM pg_class class3, pg_index WHERE ((class1.oid = pg_index.indexrelid) AND (class3.oid = pg_index.indrelid))) IS NOT NULL) THEN (SELECT class3.relname FROM pg_class class3, pg_index WHERE ((class1.oid = pg_index.indexrelid) AND (class3.oid = pg_index.indrelid))) WHEN (class1.relname IS NOT NULL) THEN class1.relname ELSE NULL::name END AS "table", CASE WHEN (class1.relkind = 'r'::"char") THEN NULL::name ELSE class1.relname END AS "index", (class1.relpages * 8) AS "size (KBytes)" FROM pg_class class1 WHERE ((class1.relkind = 'r'::"char") OR (class1.relkind = 'i'::"char")) ORDER BY CASE WHEN ((SELECT class3.relname FROM pg_class class3, pg_index WHERE ((class1.oid = pg_index.indexrelid) AND (class3.oid = pg_index.indrelid))) IS NOT NULL) THEN (SELECT class3.relname FROM pg_class class3, pg_index WHERE ((class1.oid = pg_index.indexrelid) AND (class3.oid = pg_index.indrelid))) WHEN (class1.relname IS NOT NULL) THEN class1.relname ELSE NULL::name END, CASE WHEN (class1.relkind = 'r'::"char") THEN NULL::name ELSE class1.relname END, (class1.relpages * 8);
>  
>
I was verry interested in your querry but I did not understood it. 
Therefore I rewrote it. Now it is a little simpler to read and does (in 
my opinion) the same?

--
-- Amount of space per object used after vacuum
--
VACUUM;
SELECT c1.relname AS "tablename", c2.relname AS "indexname",
       c2.relpages * 8 AS "size_kb", c2.relfilenode AS "filename"
FROM pg_class c1, pg_class c2, pg_index i
WHERE c1.oid = i.indrelid
  AND i.indexrelid = c2.oid
UNION
SELECT relname, NULL, relpages * 8, relfilenode
FROM pg_class
WHERE relkind = 'r'
ORDER BY tablename, indexname DESC, size_kb;

Caution: This Sktipt does NOT exactly the same... but the results should 
be the same

Regrards
Oli

-------------------------------------------------------

Oli Sennhauser
Database-Engineer (Oracle & PostgreSQL)
Rebenweg 6
CH - 8610 Uster / Switzerland

Phone (+41) 1 940 24 82 or Mobile (+41) 79 450 49 14
e-Mail oli.sennhauser@bluewin.ch
Website http://mypage.bluewin.ch/shinguz/PostgreSQL/

Secure (signed/encrypted) e-Mail with a Free Personal SwissSign ID: http://www.swisssign.ch

Import the SwissSign Root Certificate: http://swisssign.net/cgi-bin/trust/import
Wed, 26 Nov 2003 18:40:18 +0100
Oli Sennhauser <oli.sennhauser@bluewin.ch>
Re: Size on Disk
Hello,
   Try SQlite. It is small DB with SQL 92 standard - almost
implemented. If you need something fast and small it is
just for you.
   BTW: it is embadded system, so concurency... is not in the
focus ;)

--
Regards
   Michal Zaborowski (TeXXaS)
http://sqlite4delphi.sourceforge.net/


Wed, 26 Nov 2003 23:58:46 +0100
Michal Zaborowski <listy.mailowe@wp.pl>
Re: Size on Disk
On Wed, 2003-11-26 at 05:53, Tom Lane wrote:
> Grzegorz Dostatni <dostatnig@yahoo.com> writes:
> > Currently the datase is roughly 80 Megs. About half of
> > the size is stored in pg_xlog directory. I managed to
> > figure out that those files are transaction log files?
> > How can I delete them safely?
> 
> You can NOT.  Don't even think about going there.
> 
> What you can do, if you intend only low-update-volume usage,
> is reduce checkpoint_segments to reduce the number of WAL files
> the system wants to keep around.

The use of the word "log" in the directory name does tend to invite this
error, and some have acted on it without asking first.  I think initdb
should put a README.IMPORTANT file in $PGDATA to say,

        pg_xlog and pg_clog are crucial to the preservation of your
        data. They do not contain standard log files.  Do not even think
        about deleting them to save space; you would destroy your
        database.
        
The cost is only one disk block per cluster, and it might deflect some
of the weaponry pointed at hapless feet...

Patch for initdb.c attached

I notice that pg_clog and pg_xlog are not mentioned in the index to the
documentation, which makes it more difficult for people to find out what
they are.  I therefore also attach a doc patch to add index entries for
those two files.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Who shall ascend into the hill of the LORD? or who 
      shall stand in his holy place? He that hath clean 
      hands, and a pure heart..."            Psalms 24:3,4
Sun, 30 Nov 2003 22:29:35 +0000
Oliver Elphick <olly@lfix.co.uk>
initdb should create a warning message [was Re: [ADMIN] Size on Disk]
Oliver Elphick <olly@lfix.co.uk> writes:
> The use of the word "log" in the directory name does tend to invite
> this error, and some have acted on it without asking first.  I think
> initdb should put a README.IMPORTANT file in $PGDATA to say [...]

If someone deletes something from $PGDATA without understanding what
it is, they deserve what they get.

I do agree that we could stand to document the purpose of pg_clog
and pg_xlog more clearly. However, this information belongs in the
standard documentation, not scattered throughout $PGDATA.

-Neil


Sun, 30 Nov 2003 18:18:54 -0500
Neil Conway <neilc@samurai.com>
Re: [HACKERS] initdb should create a warning message [was Re:
On Sun, 2003-11-30 at 23:18, Neil Conway wrote:
> Oliver Elphick <olly@lfix.co.uk> writes:
> > The use of the word "log" in the directory name does tend to invite
> > this error, and some have acted on it without asking first.  I think
> > initdb should put a README.IMPORTANT file in $PGDATA to say [...]
> 
> If someone deletes something from $PGDATA without understanding what
> it is, they deserve what they get.

People have a distressing tendency to want to shoot themselves in the
foot; and the somewhat unfortunate naming of those files contributes to
the problem.  While it is satisfying to see stupidity properly rewarded,
it is more neighbourly at least to attempt to protect a fool from his
folly.  It is also kinder to those who may be depending on him for the
protection of their data.

> I do agree that we could stand to document the purpose of pg_clog
> and pg_xlog more clearly. However, this information belongs in the
> standard documentation, not scattered throughout $PGDATA.

Then it needs to be stated very prominently.  But the place to put a
sign saying "Dangerous cliff edge" is beside the path that leads along
it.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Who is like unto thee, O LORD, among the gods? who is 
      like thee, glorious in holiness, fearful in praises, 
      doing wonders?"             Exodus 15:11 


Mon, 01 Dec 2003 07:29:57 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: [HACKERS] initdb should create a warning message [was Re:
Oliver Elphick <olly@lfix.co.uk> writes:
> On Sun, 2003-11-30 at 23:18, Neil Conway wrote:
>> I do agree that we could stand to document the purpose of pg_clog
>> and pg_xlog more clearly. However, this information belongs in the
>> standard documentation, not scattered throughout $PGDATA.

> Then it needs to be stated very prominently.  But the place to put a
> sign saying "Dangerous cliff edge" is beside the path that leads along
> it.

How about changing the names of those directories?

			regards, tom lane

Mon, 01 Dec 2003 10:13:11 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] initdb should create a warning message [was Re: [ADMIN]
	Size on Disk]
Tom Lane wrote:
> Oliver Elphick <olly@lfix.co.uk> writes:
> > On Sun, 2003-11-30 at 23:18, Neil Conway wrote:
> >> I do agree that we could stand to document the purpose of pg_clog
> >> and pg_xlog more clearly. However, this information belongs in the
> >> standard documentation, not scattered throughout $PGDATA.
> 
> > Then it needs to be stated very prominently.  But the place to put a
> > sign saying "Dangerous cliff edge" is beside the path that leads along
> > it.
> 
> How about changing the names of those directories?

I thought about that, but what would we call them?  We could change xlog
to wal, I guess.  That might actually be clearer.  xlog could become
xstatus or xactstatus or just xact.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 1 Dec 2003 10:34:14 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] initdb should create a warning message [was Re: [ADMIN]
Bruce Momjian wrote:
> Tom Lane wrote:
> 
>>Oliver Elphick <olly@lfix.co.uk> writes:
>>
>>>On Sun, 2003-11-30 at 23:18, Neil Conway wrote:
>>>
>>>>I do agree that we could stand to document the purpose of pg_clog
>>>>and pg_xlog more clearly. However, this information belongs in the
>>>>standard documentation, not scattered throughout $PGDATA.
>>
>>>Then it needs to be stated very prominently.  But the place to put a
>>>sign saying "Dangerous cliff edge" is beside the path that leads along
>>>it.
>>
>>How about changing the names of those directories?
> 
> 
> I thought about that, but what would we call them?  We could change xlog
> to wal, I guess.  That might actually be clearer.  xlog could become
> xstatus or xactstatus or just xact.
> 

active_xdata
active_cdata

Mike Mascari
mascarm@mascari.com



Mon, 01 Dec 2003 10:46:01 -0500
Mike Mascari <mascarm@mascari.com>
Re: [HACKERS] initdb should create a warning message [was Re: [ADMIN]

Oliver Elphick <olly@lfix.co.uk> writes:

> Then it needs to be stated very prominently.  But the place to put a
> sign saying "Dangerous cliff edge" is beside the path that leads along
> it.

The only way to make this prominent would be a file with the *name* "THIS
DIRECTORY CONTAINS CRITICAL DATA". Not a "README" with that message inside.

-- 
greg


01 Dec 2003 11:16:56 -0500
Greg Stark <gsstark@mit.edu>
Re: initdb should create a warning message [was Re:
Greg Stark wrote:

>Oliver Elphick <olly@lfix.co.uk> writes:
>
>  
>
>>Then it needs to be stated very prominently.  But the place to put a
>>sign saying "Dangerous cliff edge" is beside the path that leads along
>>it.
>>    
>>
>
>The only way to make this prominent would be a file with the *name* "THIS
>DIRECTORY CONTAINS CRITICAL DATA". Not a "README" with that message inside.
>
>  
>

Renaming the directories is the only suggestion I've seen that makes 
sense. The others remind me of the warning that is now placed on coffee 
cup lids at fast food places: "Caution, Contents May Be Hot".

cheers

andrew


Mon, 01 Dec 2003 11:39:02 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: initdb should create a warning message [was Re:
On Mon, 2003-12-01 at 16:39, Andrew Dunstan wrote:
> Renaming the directories is the only suggestion I've seen that makes 
> sense. The others remind me of the warning that is now placed on coffee 
> cup lids at fast food places: "Caution, Contents May Be Hot".

I agree that renaming the directories is the best solution.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK                             http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Who is like unto thee, O LORD, among the gods? who is 
      like thee, glorious in holiness, fearful in praises, 
      doing wonders?"             Exodus 15:11 


Mon, 01 Dec 2003 17:26:35 +0000
Oliver Elphick <olly@lfix.co.uk>
Re: initdb should create a warning message [was Re:
Greg Stark writes:
> Oliver Elphick <olly@lfix.co.uk> writes:
> > Then it needs to be stated very prominently.  But the place to put a
> > sign saying "Dangerous cliff edge" is beside the path that leads along
> > it.Greg Stark <gsstark@mit.edu>, p
>
> The only way to make this prominent would be a file with the *name* "THIS
> DIRECTORY CONTAINS CRITICAL DATA". Not a "README" with that message inside.

That's exacly what I did, after some "root" came along and moved my pgdata 
away while postmaster was running. The data was not that important in that 
case, but nevertheless I put a file with a name like

NEVER_MOVE_THIS_DIRECTORY_WHILE_POSTMASTER_PROCESS_IS_RUNNING.txt

in pgdata and wrote a few lines in that file, how to shutdown postmaster 
properly.

But renaming pgdata to something like that would be even better and could be 
done alrealy (if I'm right).

Regards,
	
	Tilo

Mon, 1 Dec 2003 18:56:46 +0100
Tilo Schwarz <mail@tilo-schwarz.de>
Re: initdb should create a warning message [was Re: