AFAICT, the last remaining textual length limits in the backend are a
couple of fixed-size buffers in rewriteDefine.c and array support.
In the next few days (probably this weekend) I intend to fix that code
and then remove all #define constants that have anything to do with
string length limits.  (Tentative hit list is attached.)

Although the backend is in fairly good shape, there are parts of
interfaces/ and bin/ that are not.  In particular:

pg_dump has a lot of uses of MAX_QUERY_SIZE.  I believe Michael Ansley
is working on revising pg_dump to use expansible string buffers, so for
now I will leave that code alone (I'll just temporarily stick a
definition of MAX_QUERY_SIZE into pg_dump.h).

ecpg's lexer needs the same fixes recently applied to parser/scan.l to
eliminate a fixed-size literal buffer and allow flex's input buffer to
grow as needed.  Michael Meskes usually handles ecpg --- Michael, do
you want to deal with this issue or shall I take a cut at it?

The odbc, python, and cli interfaces will all break because they contain
references to symbols I propose to remove.  Since I don't use any of
these, and they aren't built by default, I can face this prospect
without flinching ;-).  This is a call for whoever maintains these
modules to get busy.

ODBC will probably need some actual thought, since what it seems to be
doing with these symbols is making their values available to client
programs on request.  Does ODBC's API for this function have a concept
of "no specific upper limit"?

			regards, tom lane

Say goodnight to:

MAX_PARSE_BUFFER
MAX_QUERY_SIZE
ERROR_MSG_LENGTH
MAX_LINES
MAX_STRING_LENGTH
REMARK_LENGTH
ELOG_MAXLEN
MAX_BUFF_SIZE
PQ_BUFFER_SIZE
MAXPGPATH
MAX_STRING_LENGTH
YY_BUF_SIZE		remove hacking in makefiles
YY_READ_BUF_SIZE	no need to alter default
MaxHeapTupleSize	(not used)
MaxAttributeSize	(not used)
MaxAttrSize		(where used for buffer sizing)

Suspicious-looking symbols found only in various interfaces/ dirs;
I don't plan to remove these but someone should:

SQL_PACKET_SIZE		odbc
MAX_MESSAGE_LEN		interfaces/cli, odbc
MAX_STATEMENT_LEN	odbc
TEXT_FIELD_SIZE		odbc
MAX_VARCHAR_SIZE	odbc
DRV_VARCHAR_SIZE	odbc
DRV_LONGVARCHAR_SIZE	odbc
MAX_CONNECT_STRING	odbc
MAX_BUFFER_SIZE		interfaces/python
MAX_FIELDS		odbc (does this relate to anything real?)


Wed, 20 Oct 1999 11:38:00 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Planning final assault on query length limits
   AFAICT, the last remaining textual length limits in the backend are a
   couple of fixed-size buffers in rewriteDefine.c and array support.
   In the next few days (probably this weekend) I intend to fix that code
   and then remove all #define constants that have anything to do with
   string length limits.  (Tentative hit list is attached.)

Great!

Jan, does this mean that we can also lose the "rewrite string too big"
problem with rules?

That would be a huge win.

Cheers,
Brook

Wed, 20 Oct 1999 11:18:08 -0600 (MDT)
Brook Milligan <brook@biology.nmsu.edu>
Re: [HACKERS] Planning final assault on query length limits
>    AFAICT, the last remaining textual length limits in the backend are a
>    couple of fixed-size buffers in rewriteDefine.c and array support.
>    In the next few days (probably this weekend) I intend to fix that code
>    and then remove all #define constants that have anything to do with
>    string length limits.  (Tentative hit list is attached.)
> 
> Great!
> 
> Jan, does this mean that we can also lose the "rewrite string too big"
> problem with rules?
> 
> That would be a huge win.


No.  We have to have long tuples.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Wed, 20 Oct 1999 13:41:27 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Planning final assault on query length limits
   > Jan, does this mean that we can also lose the "rewrite string too big"
   > problem with rules?

   No.  We have to have long tuples.

Darn.  Oh well, I guess this is a major step in that direction.

Cheers,
Brook

Wed, 20 Oct 1999 11:59:58 -0600 (MDT)
Brook Milligan <brook@biology.nmsu.edu>
Re: [HACKERS] Planning final assault on query length limits
Brook Milligan <brook@biology.nmsu.edu> writes:
>> Jan, does this mean that we can also lose the "rewrite string too big"
>> problem with rules?

>    No.  We have to have long tuples.

> Darn.  Oh well, I guess this is a major step in that direction.

I'm hoping that once this is done, someone who knows the guts of the
storage managers better than I will feel motivated to work on letting
stored tuples cross block boundaries.  (Paging Vadim...)  That seems
to be the last piece of the puzzle.

			regards, tom lane

Wed, 20 Oct 1999 14:47:46 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Planning final assault on query length limits 
On Wed, Oct 20, 1999 at 11:38:00AM -0400, Tom Lane wrote:
> ecpg's lexer needs the same fixes recently applied to parser/scan.l to
> eliminate a fixed-size literal buffer and allow flex's input buffer to

I see.

> grow as needed.  Michael Meskes usually handles ecpg --- Michael, do
> you want to deal with this issue or shall I take a cut at it?

I'm currently very busy, so I guess it will take some some until I can
tackle this problem. So if you have some spare time left I have no problem
if you make these changes. Otherwise I will once I find time.

Michael
-- 
Michael Meskes                         | Go SF 49ers!
Th.-Heuss-Str. 61, D-41812 Erkelenz    | Go Rhein Fire!
Tel.: (+49) 2431/72651                 | Use Debian GNU/Linux!
Email: Michael@Fam-Meskes.De           | Use PostgreSQL!

Thu, 21 Oct 1999 15:16:57 +0200
Michael Meskes <meskes@postgreSQL.org>
Re: Planning final assault on query length limits
Tom Lane wrote:
> 
> Brook Milligan <brook@biology.nmsu.edu> writes:
> >> Jan, does this mean that we can also lose the "rewrite string too big"
> >> problem with rules?
> 
> >    No.  We have to have long tuples.
> 
> > Darn.  Oh well, I guess this is a major step in that direction.
> 
> I'm hoping that once this is done, someone who knows the guts of the
> storage managers better than I will feel motivated to work on letting
> stored tuples cross block boundaries.  (Paging Vadim...)  That seems
> to be the last piece of the puzzle.

You know that I'm busy with WAL...
And I already made some step in big tuples dirrection
when made memory/disk tuple presentations different -:)

typedef struct HeapTupleData
{
    uint32      t_len;          /* length of *t_data */
    ItemPointerData t_self;     /* SelfItemPointer */
    HeapTupleHeader t_data;     /* */
    ^^^^^^^^^^^^^^^^^^^^^^
    On-disk data

} HeapTupleData;

I hope that something could be added here for tuple chunks...
TupleTableSlot.ttc_buffer (and ttc_shouldFree?) is good candidate
to be moved here from TupleTableSlot. 

As for smgr part - it's not hard at all.

Vadim

Fri, 22 Oct 1999 10:17:21 +0800
Vadim Mikheev <vadim@krs.ru>
Re: [HACKERS] Planning final assault on query length limits
Vadim Mikheev <vadim@krs.ru> writes:
> Tom Lane wrote:
>> I'm hoping that once this is done, someone who knows the guts of the
>> storage managers better than I will feel motivated to work on letting
>> stored tuples cross block boundaries.  (Paging Vadim...)  That seems
>> to be the last piece of the puzzle.

> You know that I'm busy with WAL...

Well, Vadim doesn't want to do it, and I don't want to do it (I've
really got to spend some more time on the planner/optimizer, because
there's too much stuff half-fixed in there).

Any volunteers out there?  It'd be a shame to not have this problem
area completely licked for 7.0.

			regards, tom lane

Thu, 21 Oct 1999 23:42:28 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Planning final assault on query length limits 
> Vadim Mikheev <vadim@krs.ru> writes:
> > Tom Lane wrote:
> >> I'm hoping that once this is done, someone who knows the guts of the
> >> storage managers better than I will feel motivated to work on letting
> >> stored tuples cross block boundaries.  (Paging Vadim...)  That seems
> >> to be the last piece of the puzzle.
> 
> > You know that I'm busy with WAL...
> 
> Well, Vadim doesn't want to do it, and I don't want to do it (I've
> really got to spend some more time on the planner/optimizer, because
> there's too much stuff half-fixed in there).
> 
> Any volunteers out there?  It'd be a shame to not have this problem
> area completely licked for 7.0.

Welcome to the small club, Tom.  For the first 2 & 1/2 years, the only
person who could tackle those big jobs was Vadim.  Now you are in the
club too.

The problem is that there are no more.  I can't imagine anyone is going
to be able to jump out of the woodwork and take on a job like that.  We
will just have to do the best job we can, and maybe save something for
7.1.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 22 Oct 1999 00:12:29 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Planning final assault on query length limits
Bruce Momjian wrote:
> 
> > > You know that I'm busy with WAL...
> >
> > Well, Vadim doesn't want to do it, and I don't want to do it (I've
> > really got to spend some more time on the planner/optimizer, because
> > there's too much stuff half-fixed in there).
> >
> > Any volunteers out there?  It'd be a shame to not have this problem
> > area completely licked for 7.0.
> 
> Welcome to the small club, Tom.  For the first 2 & 1/2 years, the only
> person who could tackle those big jobs was Vadim.  Now you are in the
> club too.
> 
> The problem is that there are no more.  I can't imagine anyone is going
> to be able to jump out of the woodwork and take on a job like that.  We
> will just have to do the best job we can, and maybe save something for
> 7.1.

There is Jan!...
But he's busy too -:)

Let's wait for 7.0 beta - "big tuples" seems as work for 2 weeks...

Vadim

Fri, 22 Oct 1999 12:54:28 +0800
Vadim Mikheev <vadim@krs.ru>
Re: [HACKERS] Planning final assault on query length limits
> > The problem is that there are no more.  I can't imagine anyone is going
> > to be able to jump out of the woodwork and take on a job like that.  We
> > will just have to do the best job we can, and maybe save something for
> > 7.1.
> 
> There is Jan!...
> But he's busy too -:)
> 
> Let's wait for 7.0 beta - "big tuples" seems as work for 2 weeks...

Oops, forgot about him.  Sorry Jan.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 22 Oct 1999 01:22:22 -0400 (EDT)
Bruce Momjian <maillist@candle.pha.pa.us>
Re: [HACKERS] Planning final assault on query length limits
Vadim Mikheev <vadim@krs.ru> writes:
> Bruce Momjian wrote:
>>>> Any volunteers out there?  It'd be a shame to not have this problem
>>>> area completely licked for 7.0.
>> 
>> Welcome to the small club, Tom.  For the first 2 & 1/2 years, the only
>> person who could tackle those big jobs was Vadim.  Now you are in the
>> club too.
>> 
>> The problem is that there are no more.  I can't imagine anyone is going
>> to be able to jump out of the woodwork and take on a job like that.  We
>> will just have to do the best job we can, and maybe save something for
>> 7.1.

> There is Jan!...
> But he's busy too -:)

> Let's wait for 7.0 beta - "big tuples" seems as work for 2 weeks...

Thing is, if Vadim could do it in two weeks (sounds about right), then
maybe I could do it in three or four (I'd have to spend time studying
parts of the backend that Vadim already knows, but I don't).  It seems
to me that some aspiring hacker who's already a little bit familiar
with backend coding could do it in a month or two, with suitable study,
and would in the process make great strides towards gurudom.  This is
a fairly localized task, if I'm not greatly mistaken about it.  And
there's plenty of time left before 7.0.  So this seems like a perfect
project for someone who wants to learn more about the backend and has
some time to spend doing so.

A year ago I didn't know a darn thing about the backend, so I'm a bit
bemused to find myself being called a member of "the small club".
Programming skills don't come out of nowhere, they come out of study
and practice.  (See http://www.tuxedo.org/~esr/faqs/loginataka.html)

In short, I'd like to see the club get bigger...

			regards, tom lane

Fri, 22 Oct 1999 02:04:29 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Planning final assault on query length limits 
Tom Lane wrote:
> 
> > Let's wait for 7.0 beta - "big tuples" seems as work for 2 weeks...
> 
> Thing is, if Vadim could do it in two weeks (sounds about right), then
> maybe I could do it in three or four (I'd have to spend time studying
> parts of the backend that Vadim already knows, but I don't).  It seems
> to me that some aspiring hacker who's already a little bit familiar
> with backend coding could do it in a month or two, with suitable study,
> and would in the process make great strides towards gurudom.  This is
> a fairly localized task, if I'm not greatly mistaken about it.  And
           ^^^^^^^^^^^^^^
I'm not sure. Seems that we'll have to change heap_getattr:
if a column crosses page boundary then we'll have to re-construct
it in memory and pfree it after using...

> there's plenty of time left before 7.0.  So this seems like a perfect
> project for someone who wants to learn more about the backend and has
> some time to spend doing so.

And we always ready to help...

> A year ago I didn't know a darn thing about the backend, so I'm a bit
> bemused to find myself being called a member of "the small club".
> Programming skills don't come out of nowhere, they come out of study
> and practice.  (See http://www.tuxedo.org/~esr/faqs/loginataka.html)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-:)))

Vadim

Fri, 22 Oct 1999 14:45:50 +0800
Vadim Mikheev <vadim@krs.ru>
Re: [HACKERS] Planning final assault on query length limits
Vadim Mikheev <vadim@krs.ru> writes:
>> a fairly localized task, if I'm not greatly mistaken about it.  And
>            ^^^^^^^^^^^^^^
> I'm not sure. Seems that we'll have to change heap_getattr:
> if a column crosses page boundary then we'll have to re-construct
> it in memory and pfree it after using...

I was thinking more along the lines of reconstructing the whole tuple
in palloc'd memory and leaving heap_getattr as-is.  Otherwise we have
problems with the system relations whose tuples are accessed as C
structs.  You'd need to somehow guarantee that those tuples are never
split if you do it as above.

Of course, that just moves the palloc/pfree bookkeeping problem down
a level; it's still going to be tricky to avoid storage leaks.
We might be able to get some win from storing reassembled tuples in
TupleTableSlots, though.

>> there's plenty of time left before 7.0.  So this seems like a perfect
>> project for someone who wants to learn more about the backend and has
>> some time to spend doing so.

> And we always ready to help...

Right.  Questions can be answered.

			regards, tom lane

Fri, 22 Oct 1999 11:15:41 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Planning final assault on query length limits 