I was just doing some huge operations with PostgreSQL and it all crashed out
with a "too many files open" message plastered all over the place..

Now in /proc/sys/fs/file-max there is only 4096, that limit could have
easily been reached. Does changing the value in the file effectively change
the limit system-wide? I changed it and rebooted but it was set right back
to 4096.. I've been out of the Linux loop for a long time (FreeBSD junkie
now) so I don't know how to set that up to permanently change the limit.

Marc: You use Linux, don't you? Have you ever run into this?

Any help is very appreciated, thanks!

-Mitch



Wed, 13 Sep 2000 13:04:32 -0700
"Mitch Vincent" <mitch@venux.net>
Linux / PostgreSQL question
"Mitch Vincent" <mitch@venux.net> writes:

> I was just doing some huge operations with PostgreSQL and it all crashed out
> with a "too many files open" message plastered all over the place..
> 
> Now in /proc/sys/fs/file-max there is only 4096, that limit could have
> easily been reached. Does changing the value in the file effectively change
> the limit system-wide? I changed it and rebooted but it was set right back
> to 4096.. 

A reboot would reset values in proc...

For Red Hat Linux 6.2 and up, you would add a line to
/etc/sysctl.conf:

# Max open files:
fs.file-max = 8192

-- 
Trond Eivind Glomsrï¿½d
Red Hat, Inc.

13 Sep 2000 16:13:57 -0400
teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=)
Re: Linux / PostgreSQL question
On Wed, Sep 13, 2000 at 01:04:32PM -0700, Mitch Vincent wrote:
> I was just doing some huge operations with PostgreSQL and it all crashed out
> with a "too many files open" message plastered all over the place..
> 

This is really a Linux question, not postgresql, but you knew that...

(I'm keeping hackers on this message, so if it comes up again, the
answer's in the archives with the question)

> Now in /proc/sys/fs/file-max there is only 4096, that limit could have
> easily been reached. Does changing the value in the file effectively change
> the limit system-wide? I changed it and rebooted but it was set right back
> to 4096.. I've been out of the Linux loop for a long time (FreeBSD junkie
> now) so I don't know how to set that up to permanently change the limit.

Almost right. Why'd you reboot? It's a runtime configuration. Proc is not
a file system, it's a pseudo-filesystem interface to kernel internals.

Just do something like:

echo 32768 > /proc/sys/fs/file-max 

And you may need to up the number of inodes, too:

echo 65536 > /proc/sys/fs/inode-max 

You'll probably want to put these in rc.boot, or rc.local, or something, 
to set this at boot time, as well.

Ross

-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005

Wed, 13 Sep 2000 15:18:44 -0500
"Ross J. Reedstrom" <reedstrm@rice.edu>
Re: Linux / PostgreSQL question
I didn't see any mention of it on the TODO so I thought I'd ask if anyone
had thought about full test indexing for 7.1 (I'm guessing not)..

If not, I'd like to suggest it be put on the TODO -- if nothing else so
someone could pick it up in the far future if they wanted to.. It doesn't
seem like too many are worried about it so the request is pretty selfish,
though I'm sure it would help many people especially after 7.1 and TOAST
make text fields unlimited in size.

Thanks!

-Mitch



Mon, 16 Oct 2000 09:37:58 -0700
"Mitch Vincent" <mitch@venux.net>
Full text indexing (Question/request)
See contrib/fulltextindex.

[ Charset ISO-8859-1 unsupported, converting... ]
> I didn't see any mention of it on the TODO so I thought I'd ask if anyone
> had thought about full test indexing for 7.1 (I'm guessing not)..
> 
> If not, I'd like to suggest it be put on the TODO -- if nothing else so
> someone could pick it up in the far future if they wanted to.. It doesn't
> seem like too many are worried about it so the request is pretty selfish,
> though I'm sure it would help many people especially after 7.1 and TOAST
> make text fields unlimited in size.
> 
> Thanks!
> 
> -Mitch
> 
> 
> 


-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 16 Oct 2000 12:45:52 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Full text indexing (Question/request)
Bruce Momjian wrote:
> 
> See contrib/fulltextindex.

An easy answer, but not a very good solution in the real world.

contrib/fulltextindex requires you to jump through hoops in developing
queries to retrieve your data.  It's also very space-inefficient in that
a table with a fulltextindex on a field needs another table with a
complete set of values for that field, as well as any substrings of that
field, and then it wants two indexes on that table.  Add that up!

It would be nice to see a true index which was full text.  It would be
nice to see a true index which allowed an individual field to index to
many entries through a function interface.  This would straightforwardly
allow people to create their own simple functions to perform full-text,
keyword or other indexing schemes quite simply.

It naively appears to me that the function interface is moving closer to
achieving this with the enhancements in 7.1 to the use of setof()
returns combined with the earlier enhancement to indexing on function
results.

If a function fulltextindex(text) returned a setof() the substrings in
its text argument, how hard will it be to index on that return value and
allow WHERE field=fulltextindex('substring') to use that index?

Of course such a fulltextindex() function would have to know not to do
any processing on the string when called in the second situation.  Is it
possible for functions to do this sort of trick?  It seems a bit beyond
the pale!

I would _love_ to see full-text or keyword indexing natively in
PostgreSQL.

Regards,
					Andrew.
-- 
_____________________________________________________________________
            Andrew McMillan, e-mail: Andrew@cat-it.co.nz
Catalyst IT Ltd, PO Box 10-225, Level 22, 105 The Terrace, Wellington
Me: +64 (21) 635 694, Fax: +64 (4) 499 5596, Office: +64 (4) 499 2267

Tue, 17 Oct 2000 22:36:32 +1300
Andrew McMillan <Andrew@catalyst.net.nz>
Re: Full text indexing (Question/request)
Andrew McMillan wrote:
> 
> Bruce Momjian wrote:
> >
> > See contrib/fulltextindex.
> 
> An easy answer, but not a very good solution in the real world.
> 
> contrib/fulltextindex requires you to jump through hoops in developing
> queries to retrieve your data.  It's also very space-inefficient in that
> a table with a fulltextindex on a field needs another table with a
> complete set of values for that field, as well as any substrings of that
> field, and then it wants two indexes on that table.  Add that up!
> 
> It would be nice to see a true index which was full text.  It would be
> nice to see a true index which allowed an individual field to index to
> many entries through a function interface.  This would straightforwardly
> allow people to create their own simple functions to perform full-text,
> keyword or other indexing schemes quite simply.
> 
> It naively appears to me that the function interface is moving closer to
> achieving this with the enhancements in 7.1 to the use of setof()
> returns combined with the earlier enhancement to indexing on function
> results.
> 
> If a function fulltextindex(text) returned a setof() the substrings in
> its text argument, how hard will it be to index on that return value and
> allow WHERE field=fulltextindex('substring') to use that index?
> 
> Of course such a fulltextindex() function would have to know not to do
> any processing on the string when called in the second situation.  Is it
> possible for functions to do this sort of trick?  It seems a bit beyond
> the pale!
> 
> I would _love_ to see full-text or keyword indexing natively in
> PostgreSQL.

I tottally agree with you. FTI is not a good solution. It seems natural
that PostgreSQL will have a built-in (and better) FTI, now that the
TOAST project will be implemented in 7.1.

Poul L. Christiansen

Tue, 17 Oct 2000 12:28:16 +0100
"Poul L. Christiansen" <poulc@cs.auc.dk>
Re: Full text indexing (Question/request)
Andrew McMillan <Andrew@catalyst.net.nz> writes:
> 
> I would _love_ to see full-text or keyword indexing natively in
> PostgreSQL.
> 

I would rather love to see a great fulltext engine integrated with
PostgreSQL. It would be cool to be able to have ranked results and
different ways of searching the index(regex, soundex, synonym). Having a
solution that where pluggable for different languages would be nice. 

regards, 

	Gunnar

17 Oct 2000 14:54:58 +0100
Gunnar R|nning <gunnar@candleweb.no>
Re: Full text indexing (Question/request)