I have finished applying Mike Ansley's changes for long queries, along
with a bunch of my own.  The current status is:

* You can send a query string of indefinite length to the backend.
  (This is poorly tested for MULTIBYTE, though; would someone who
  uses MULTIBYTE more than I do try it out?)

* You can get back an EXPLAIN or error message string of indefinite
  length.

* Single lexical tokens within a query are currently limited to 64k
  because of the lexer's use of YY_REJECT.  I have not committed any
  of Leon's proposed lexer changes, since that issue still seems
  controversial.  I would like to see us agree on a solution.
  (ecpg's lexer has the same problem, of course.)

Although I think the backend is in fairly good shape, there are still
a few minor trouble spots.  (The rule deparser will blow up at 8K for
example --- I have some work to do in there and will fix it when
I get a chance.)

In the frontend libraries and clients, both libpq and psql are length-
limit-free.  I have not looked much at any of the other frontend
interface libraries.  I suspect that at least odbc and the python
interface need work, because quick glimpse searches show suspicious-
looking constants:
	MAX_QUERY_SIZE
	ERROR_MSG_LENGTH
	SQL_PACKET_SIZE
	MAX_MESSAGE_LEN
	TEXT_FIELD_SIZE
	MAX_VARCHAR_SIZE
	DRV_VARCHAR_SIZE
	DRV_LONGVARCHAR_SIZE
	MAX_BUFFER_SIZE
	MAX_FIELDS

The real problem in the clients is that pg_dump blithely assumes it
will never need to deal with strings over MAX_QUERY_SIZE.  This is
a bad idea --- it ought to be rewritten to use the expansible-string-
buffer facility that now exists in libpq.  There may be restrictions
in the other programs in bin/ as well, though glimpse didn't turn up
any red flags.

I would like to encourage the odbc and python folks to get rid of the
length limitations in their modules; I don't use either and have no
intention of touching either.  I'd like to find a volunteer other than
myself to fix pg_dump, too.

Now, all we need is someone to implement multiple-disk-block tuples ;-)

			regards, tom lane

Sat, 11 Sep 1999 18:50:16 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Status report: long-query-string changes
Tom Lane wrote:

> 
> * Single lexical tokens within a query are currently limited to 64k
>   because of the lexer's use of YY_REJECT.  I have not committed any
>   of Leon's proposed lexer changes, since that issue still seems
>   controversial.  I would like to see us agree on a solution.

Thomas Lockhart should speak up - he seems the only person who
has objections yet. If the proposed thing is to be declined, something
has to be applied instead in respect to lexer reject feature and
accompanying size limits, as well as grammar inconsistency. Seems there
are only awkward solutions as alternatives. As you probably remember,
the proposed change only breaks constructs like 1+-2, which anyone
in a sane condition should avoid when programming :)

There are more size restrictions there. I noticed (by simply eyeing the
lexer source, without testing) that in case of flex lexer 
(FLEX_LEXER being defined in scan.c) lexer can't
swallow big queries. You (Tom and Michael) aren't using flex,
are you?

-- 
Leon.
-------
He knows he'll never have to answer for any of his theories actually 
being put to test. If they were, they would be contaminated by reality.


Sun, 12 Sep 1999 04:22:01 +0500
Leon <leon@udmnet.ru>
Re: [HACKERS] Status report: long-query-string changes
Leon <leon@udmnet.ru> writes:
> There are more size restrictions there. I noticed (by simply eyeing the
> lexer source, without testing) that in case of flex lexer 
> (FLEX_LEXER being defined in scan.c) lexer can't
> swallow big queries. You (Tom and Michael) aren't using flex,
> are you?

Huh?  flex is the only lexer that works with the Postgres .l files,
as far as I know.  Certainly it's what I'm using.

If you're looking at the "literal" buffer, that would need to be made
expansible, but there's not much point until flex's internal stuff is
fixed.

			regards, tom lane

Sat, 11 Sep 1999 20:38:21 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Status report: long-query-string changes 
> I have finished applying Mike Ansley's changes for long queries, along
> with a bunch of my own.  The current status is:
> 
> * You can send a query string of indefinite length to the backend.
>   (This is poorly tested for MULTIBYTE, though; would someone who
>   uses MULTIBYTE more than I do try it out?)

I'll take care of this.
---
Tatsuo Ishii

Sun, 12 Sep 1999 10:00:46 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: [HACKERS] Status report: long-query-string changes 
Tom Lane wrote:

> 
> If you're looking at the "literal" buffer, that would need to be made
> expansible, but there's not much point until flex's internal stuff is
> fixed.
> 
> 

Look at this piece of code. It seems that when myinput() had been called
once, for the second time it will return 0 even if string isn't
over yet. Parameter 'max' is 8192 bytes on my system. So the query is 
simply truncated to that size.

#ifdef FLEX_SCANNER
/* input routine for flex to read input from a string instead of a file */
static int
myinput(char* buf, int max)
{
	int len, copylen;

	if (parseCh == NULL)
	{
		len = strlen(parseString);
		if (len >= max)
			copylen = max - 1;
		else
			copylen = len;
		if (copylen > 0)
			memcpy(buf, parseString, copylen);
		buf[copylen] = '\0';
		parseCh = parseString;
		return copylen;
	}
	else
		return 0; /* end of string */
}
#endif /* FLEX_SCANNER */
 

-- 
Leon.
-------
He knows he'll never have to answer for any of his theories actually 
being put to test. If they were, they would be contaminated by reality.


Sun, 12 Sep 1999 17:12:02 +0500
Leon <leon@udmnet.ru>
Re: [HACKERS] Status report: long-query-string changes
Leon <leon@udmnet.ru> writes:
> Look at this piece of code. It seems that when myinput() had been called
> once, for the second time it will return 0 even if string isn't
> over yet.

It's always a good idea to pull a fresh copy of the sources
before opinionating about what works or doesn't work in someone's
just-committed changes ;-)

			regards, tom lane

Sun, 12 Sep 1999 11:34:32 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Status report: long-query-string changes 
> Thomas Lockhart should speak up - he seems the only person who
> has objections yet. If the proposed thing is to be declined, something
> has to be applied instead in respect to lexer reject feature and
> accompanying size limits, as well as grammar inconsistency.

Hmm. I'd suggest that we go with the "greedy lexer" solution, which
continues to gobble characters which *could* be an operator until
other characters or whitespace are encountered.

I don't recall any compelling cases for which this would be an
inadequate solution, and we have plenty of time until v6.6 is released
to discover problems and work out alternatives.

Sorry for slowing things up; but fwiw I *did* think about it some more
;)

                    - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Mon, 13 Sep 1999 03:33:08 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Status report: long-query-string changes
> Thomas Lockhart should speak up...
> He knows he'll never have to answer for any of his theories actually
> being put to test. If they were, they would be contaminated by reality.

You talkin' to me?? ;)

So, while you are on the lexer warpath, I'd be really happy if someone
would fix the following behavior:

(I'm doing this from memory, but afaik it is close to correct)

For non-psql applications, such as tcl or ecpg, which do not do any
pre-processing on input tokens, a trailing un-terminated string will
be lost, and no error will be detected. For example,

select * from t1 'abc

sent directly to the server will not fail as it should with that
garbage at the end. The lexer is in a non-standard mode after all
tokens are processed, and the accumulated string "abc" is left in a
buffer and not sent to yacc/bison. I think you can see this behavior
just by looking at the lexer code.

A simple fix would be to check the current size after lexing of that
accumulated string buffer, and if it is non-zero then elog(ERROR) a
complaint. Perhaps a more general fix would be to ensure that you are
never in an exclusive state after all tokens are processed, but I'm
not sure how to do that.

                       - Thomas

-- 
Thomas Lockhart				lockhart@alumni.caltech.edu
South Pasadena, California

Mon, 13 Sep 1999 03:45:28 +0000
Thomas Lockhart <lockhart@alumni.caltech.edu>
Re: [HACKERS] Status report: long-query-string changes
Thomas Lockhart wrote:
> 
> > Thomas Lockhart should speak up - he seems the only person who
> > has objections yet. If the proposed thing is to be declined, something
> > has to be applied instead in respect to lexer reject feature and
> > accompanying size limits, as well as grammar inconsistency.
> 
> Hmm. I'd suggest that we go with the "greedy lexer" solution, which
> continues to gobble characters which *could* be an operator until
> other characters or whitespace are encountered.

'Xcuse my dumbness ;) , but is it in any way different from 
what is proposed (by me and some others?)

-- 
Leon.
-------
He knows he'll never have to answer for any of his theories actually 
being put to test. If they were, they would be contaminated by reality.



Mon, 13 Sep 1999 18:34:56 +0500
Leon <leon@udmnet.ru>
Re: [HACKERS] Status report: long-query-string changes
Thomas Lockhart wrote:
> 
> > Thomas Lockhart should speak up...
> > He knows he'll never have to answer for any of his theories actually
> > being put to test. If they were, they would be contaminated by reality.
> 
> You talkin' to me?? ;)

Nein, nein! Sei still bitte! :)  This is my signature which is a week 
old already :)

> A simple fix would be to check the current size after lexing of that
> accumulated string buffer, and if it is non-zero then elog(ERROR) a
> complaint. Perhaps a more general fix would be to ensure that you are
> never in an exclusive state after all tokens are processed, but I'm
> not sure how to do that.

The solution is obvious - to eliminate exclusive states entirely!
Banzai!!!

-- 
Leon.
-------
He knows he'll never have to answer for any of his theories actually 
being put to test. If they were, they would be contaminated by reality.


Mon, 13 Sep 1999 18:35:12 +0500
Leon <leon@udmnet.ru>
Re: [HACKERS] Status report: long-query-string changes
Thomas Lockhart wrote:
> 
> > The solution is obvious - to eliminate exclusive states entirely!
> > Banzai!!!
> 
> That will complicate the lexer, and make it more brittle and difficult
> to read, since you will have to, essentially, implement the exclusive
> states using flags within each element.
> 
> If you want to try it as an exercise, we *might* find it isn't as ugly
> as I am afraid it will be, but...
> 

Gimme the latest lexer source. (I pay for my Internet on a per 
minute basis, so I can't connect to CVS) You will see what I mean.

-- 
Leon.
-------
He knows he'll never have to answer for any of his theories actually 
being put to test. If they were, they would be contaminated by reality.


Mon, 13 Sep 1999 19:33:27 +0500
Leon <leon@udmnet.ru>
Re: [HACKERS] Status report: long-query-string changes
Leon <leon@udmnet.ru> writes:
>> A simple fix would be to check the current size after lexing of that
>> accumulated string buffer, and if it is non-zero then elog(ERROR) a
>> complaint. Perhaps a more general fix would be to ensure that you are
>> never in an exclusive state after all tokens are processed, but I'm
>> not sure how to do that.

> The solution is obvious - to eliminate exclusive states entirely!
> Banzai!!!

Can we do that?  Seems like a more likely approach is to ensure that
all of the lexer states have rules that ensure they terminate (or
raise an error, as for unterminated quoted string) at end of input.
I do think checking the token buffer is a hack, and changing the rules
a cleaner solution...

			regards, tom lane

Mon, 13 Sep 1999 10:53:19 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Status report: long-query-string changes 
Tom Lane wrote:
> 
> Leon <leon@udmnet.ru> writes:
> >> A simple fix would be to check the current size after lexing of that
> >> accumulated string buffer, and if it is non-zero then elog(ERROR) a
> >> complaint. Perhaps a more general fix would be to ensure that you are
> >> never in an exclusive state after all tokens are processed, but I'm
> >> not sure how to do that.
> 
> > The solution is obvious - to eliminate exclusive states entirely!
> > Banzai!!!
> 
> Can we do that?  Seems like a more likely approach is to ensure that
> all of the lexer states have rules that ensure they terminate (or
> raise an error, as for unterminated quoted string) at end of input.
> I do think checking the token buffer is a hack, and changing the rules
> a cleaner solution...

Hmm, yea, you are right. It is much simpler solution. We can check 
condition in myinput() and input() when we are going to return 
end-of-input (YYSTATE == INITIAL), and raise an error if that's not so.
Well, I give up my idea of total extermination of start conditions :)

BTW, while eyeing the scan.l again, I noticed that C - style comments
can also contain bugs, but I am not completely sure.
-- 
Leon.
-------
He knows he'll never have to answer for any of his theories actually 
being put to test. If they were, they would be contaminated by reality.

Mon, 13 Sep 1999 23:19:50 +0500
Leon <leon@udmnet.ru>
Re: [HACKERS] Status report: long-query-string changes