
What has become of the "dollar quoting" mechanism that we had so much 
discussion about back in August/September?

IIRC, a consensus was reached on the actual format of the quote 
delimiters (either $$ or $identifier$), and Tom had a proof of concept 
patch to the parser to handle it, but work was needed on psql, plpgsql, 
pg_dump (and pg_restore?) + docs.

Is anyone working on it?

cheers

andrew


Thu, 05 Feb 2004 10:07:07 -0500
Andrew Dunstan <andrew@dunslane.net>
dollar quoting
On Thu, 5 Feb 2004, Andrew Dunstan wrote:

> What has become of the "dollar quoting" mechanism that we had so much 
> discussion about back in August/September?
> 
> IIRC, a consensus was reached on the actual format of the quote 
> delimiters (either $$ or $identifier$), and Tom had a proof of concept 
> patch to the parser to handle it, but work was needed on psql, plpgsql, 
> pg_dump (and pg_restore?) + docs.
> 
> Is anyone working on it?

I am, but not very quickly. I hope to have some time in the next month, 
but if someone else beats me to it I'll just be happy it got done.

Jon

Thu, 5 Feb 2004 15:50:20 +0000 (UTC)
Jon Jensen <jon@endpoint.com>
Re: dollar quoting
Andrew Dunstan <andrew@dunslane.net> writes:
> IIRC, a consensus was reached on the actual format of the quote 
> delimiters (either $$ or $identifier$), and Tom had a proof of concept 
> patch to the parser to handle it, but work was needed on psql, plpgsql, 
> pg_dump (and pg_restore?) + docs.

I think someone has to fix psql before we can consider applying the
backend patch.  Fixing the other stuff can come after.

> Is anyone working on it?

I kinda thought you had volunteered to work on the psql part...

			regards, tom lane

Thu, 05 Feb 2004 11:14:00 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: dollar quoting 


Tom Lane wrote:

>Andrew Dunstan <andrew@dunslane.net> writes:
>  
>
>>IIRC, a consensus was reached on the actual format of the quote 
>>delimiters (either $$ or $identifier$), and Tom had a proof of concept 
>>patch to the parser to handle it, but work was needed on psql, plpgsql, 
>>pg_dump (and pg_restore?) + docs.
>>    
>>
>
>I think someone has to fix psql before we can consider applying the
>backend patch.  Fixing the other stuff can come after.
>

Makes sense.

>
>  
>
>>Is anyone working on it?
>>    
>>
>
>I kinda thought you had volunteered to work on the psql part...
>
>  
>


I don't recall being that specific, but you could be right. In any case, 
I didn't want to trip over anyone else, which is why I asked.

I will try to coordinate with Jon.

cheers

andrew


Thu, 05 Feb 2004 11:34:02 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: dollar quoting


Andrew Dunstan wrote:

>
>
> Tom Lane wrote:
>
>>
>> I kinda thought you had volunteered to work on the psql part...
>>
>>  
>>
>
>
> I don't recall being that specific, but you could be right. In any 
> case, I didn't want to trip over anyone else, which is why I asked.
>
> I will try to coordinate with Jon.
>

After staring at the code for a long time, I think I see how to do this. 
It's complicated a bit by the fact that $ is a valid identifier 
character. So my current thinking is to say that if we see $ not in a 
quote and not preceded by a valid identifier char then it is the start 
of a $foo$ sequence. Or have I missed something? Can we validly see $ in 
any other context?

BTW, Tom's proof of concept patch worked just fine for me. I changed the 
allowed pattern to what I think was agreed:

dolqdelim   \$([A-Za-z\200-\377][A-Za-z\200-\377_0-9]*)?\$

and changed some names and comments to remove misleading references to 
"here docs".

cheers

andrew


Sat, 07 Feb 2004 14:07:59 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: dollar quoting
Andrew Dunstan <andrew@dunslane.net> writes:
> After staring at the code for a long time, I think I see how to do this. 
> It's complicated a bit by the fact that $ is a valid identifier 
> character. So my current thinking is to say that if we see $ not in a 
> quote and not preceded by a valid identifier char then it is the start 
> of a $foo$ sequence. Or have I missed something? Can we validly see $ in 
> any other context?

Right, a $ should be considered to start a quote marker only if it's not
part of an identifier.  The backend lexer doesn't have a problem with
this because it's written in flex, but I can imagine that getting it
right in psql's ad-hoc parser might be tricky.

There was some discussion awhile back of converting psql to use flex
for interpreting its input, but I dunno how practical that really is.
I don't know how you get flex to do reasonable stuff with an incomplete
input string.  Still, it might be worth looking into.

			regards, tom lane

Sat, 07 Feb 2004 16:51:02 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: dollar quoting 


Tom Lane wrote:

>Andrew Dunstan <andrew@dunslane.net> writes:
>  
>
>>After staring at the code for a long time, I think I see how to do this. 
>>It's complicated a bit by the fact that $ is a valid identifier 
>>character. So my current thinking is to say that if we see $ not in a 
>>quote and not preceded by a valid identifier char then it is the start 
>>of a $foo$ sequence. Or have I missed something? Can we validly see $ in 
>>any other context?
>>    
>>
>
>Right, a $ should be considered to start a quote marker only if it's not
>part of an identifier.  The backend lexer doesn't have a problem with
>this because it's written in flex, but I can imagine that getting it
>right in psql's ad-hoc parser might be tricky.
>

I think it's doable, though.  I seem to have a working patch, which I 
will send out for review soon.

>
>There was some discussion awhile back of converting psql to use flex
>for interpreting its input, but I dunno how practical that really is.
>I don't know how you get flex to do reasonable stuff with an incomplete
>input string.  Still, it might be worth looking into.
>
>  
>

That's what made me not even think about it. If someone better versed in 
this stuff than me wants to do it then more power to them.

cheers

andrew


Sat, 07 Feb 2004 19:45:57 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: dollar quoting
Andrew Dunstan wrote:

>
>
> Tom Lane wrote:
>
>> Andrew Dunstan <andrew@dunslane.net> writes:
>>  
>>
>>> After staring at the code for a long time, I think I see how to do 
>>> this. It's complicated a bit by the fact that $ is a valid 
>>> identifier character. So my current thinking is to say that if we 
>>> see $ not in a quote and not preceded by a valid identifier char 
>>> then it is the start of a $foo$ sequence. Or have I missed 
>>> something? Can we validly see $ in any other context?
>>>   
>>

I had missed one, though - numbered params in prepared statements. Fixed 
in attached patch.

>>
>> Right, a $ should be considered to start a quote marker only if it's not
>> part of an identifier.  The backend lexer doesn't have a problem with
>> this because it's written in flex, but I can imagine that getting it
>> right in psql's ad-hoc parser might be tricky.
>>
>
> I think it's doable, though.  I seem to have a working patch, which I 
> will send out for review soon.
>

Proof of Concept patch (i.e. not for application) attached for review. 
The scanner changes are based on Tom's original, with some name/comment 
changes and a more liberal pattern. The psql changes are all my own work 
:-).

Comments welcome. Reviewers: I am not sure I got multi-byte stuff right 
in psql/mainloop.c - please pay close attention to that.

If I'm not wildly off course I will polish this up and start on docs.

cheers

andrew
Sun, 08 Feb 2004 15:14:11 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: [HACKERS] dollar quoting
Andrew Dunstan <andrew@dunslane.net> writes:
> Comments welcome. Reviewers: I am not sure I got multi-byte stuff right 
> in psql/mainloop.c - please pay close attention to that.

The i-1 stuff should generally be i-prevlen.  Not sure if there are any
other pitfalls.

A bigger problem here:

> +             else if (!dol_quote && line[i] == '$' && 
> +                      !isdigit(line[i + thislen]) && 
> +                      (dol_end = strchr(line+i+1,'$')) != NULL &&
> +                      (i == 0 || 
> +                       ! ((line[i-1] & 0x80) != 0 || isalnum(line[i-1]) || 
> +                          line[i-1] == '_')))
> +             {

is that you aren't checking that what comes between the two dollar signs
looks like empty-or-an-identifier.  The check for
next-char-isn't-a-digit is part of that but not the only part.

Also I'm not sure about the positioning of these tests relative to the
in_quote and in_xcomment tests.  As you have it, $foo$ will be
recognized within an xcomment, which I think is at variance with the
proposed backend lexing behavior.

Also, the strdup should be pg_strdup.

			regards, tom lane

Mon, 09 Feb 2004 01:25:10 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] dollar quoting 


Tom Lane wrote:

>A bigger problem here:
>
>  
>
>>+             else if (!dol_quote && line[i] == '$' && 
>>+                      !isdigit(line[i + thislen]) && 
>>+                      (dol_end = strchr(line+i+1,'$')) != NULL &&
>>+                      (i == 0 || 
>>+                       ! ((line[i-1] & 0x80) != 0 || isalnum(line[i-1]) || 
>>+                          line[i-1] == '_')))
>>+             {
>>    
>>
>
>is that you aren't checking that what comes between the two dollar signs
>looks like empty-or-an-identifier.  The check for
>next-char-isn't-a-digit is part of that but not the only part.
>  
>


Well, I think the right way to do a full check would be with a regex, 
which I had hoped to avoid. However, I will now try to get one working 
and to address your other concerns.

Thanks for the comments.

cheers

andrew


Mon, 09 Feb 2004 02:05:02 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: [HACKERS] dollar quoting
I think the attached patch addresses Tom's comments.

I ended up not using a regex, which seemed to be a little heavy handed, 
but just writing a small custom recognition function, that should (and I 
think does) mimic the pattern recognition for these tokens used by the 
backend lexer. This patch just puts that function in mainloop.c, but 
perhaps it belongs elsewhere (string_utils.c maybe?). I don't have 
strong opinions on that.

Enjoy

andrew



Tom Lane wrote:

>Andrew Dunstan <andrew@dunslane.net> writes:
>  
>
>>Comments welcome. Reviewers: I am not sure I got multi-byte stuff right 
>>in psql/mainloop.c - please pay close attention to that.
>>    
>>
>
>The i-1 stuff should generally be i-prevlen.  Not sure if there are any
>other pitfalls.
>
>A bigger problem here:
>
>  
>
>>+             else if (!dol_quote && line[i] == '$' && 
>>+                      !isdigit(line[i + thislen]) && 
>>+                      (dol_end = strchr(line+i+1,'$')) != NULL &&
>>+                      (i == 0 || 
>>+                       ! ((line[i-1] & 0x80) != 0 || isalnum(line[i-1]) || 
>>+                          line[i-1] == '_')))
>>+             {
>>    
>>
>
>is that you aren't checking that what comes between the two dollar signs
>looks like empty-or-an-identifier.  The check for
>next-char-isn't-a-digit is part of that but not the only part.
>
>Also I'm not sure about the positioning of these tests relative to the
>in_quote and in_xcomment tests.  As you have it, $foo$ will be
>recognized within an xcomment, which I think is at variance with the
>proposed backend lexing behavior.
>
>Also, the strdup should be pg_strdup.
>
>			regards, tom lane
>
>---------------------------(end of broadcast)---------------------------
>TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>
>  
>
Mon, 09 Feb 2004 10:37:38 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: [HACKERS] dollar quoting
Andrew Dunstan <andrew@dunslane.net> writes:
> I ended up not using a regex, which seemed to be a little heavy handed, 
> but just writing a small custom recognition function, that should (and I 
> think does) mimic the pattern recognition for these tokens used by the 
> backend lexer.

I looked at this and realized that it still doesn't do very well at
distinguishing $foo$ from other random uses of $.  The problem is that
looking back at just the immediately preceding character isn't enough
context to tell whether a $ is part of an identifier.  Consider the
input
	a42$foo$
This is a legal identifier according to PG 7.4.  But how about
	42$foo$
This is a syntax error in 7.4, and we propose to redefine it as an
integer literal '42' followed by a dollar-quote start symbol.

There's no way to tell these apart with a single-character lookback,
or indeed any fixed number of characters of lookback.

I begin to think that we'll really have to bite the bullet and convert
psql's input parser to use flex.  If we're not scanning with exactly the
same rules as the backend uses, we're going to get the wrong answers.

			regards, tom lane

Sat, 14 Feb 2004 11:54:09 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] dollar quoting 


Tom Lane wrote:

>Andrew Dunstan <andrew@dunslane.net> writes:
>  
>
>>I ended up not using a regex, which seemed to be a little heavy handed, 
>>but just writing a small custom recognition function, that should (and I 
>>think does) mimic the pattern recognition for these tokens used by the 
>>backend lexer.
>>    
>>
>
>I looked at this and realized that it still doesn't do very well at
>distinguishing $foo$ from other random uses of $.  The problem is that
>looking back at just the immediately preceding character isn't enough
>context to tell whether a $ is part of an identifier.  Consider the
>input
>	a42$foo$
>This is a legal identifier according to PG 7.4.  But how about
>	42$foo$
>This is a syntax error in 7.4, and we propose to redefine it as an
>integer literal '42' followed by a dollar-quote start symbol.
>

The test in the patch I sent is this:


            else if (!dol_quote && valid_dolquote(line+i) &&
                     (i == 0 ||
                      ! ((line[i-prevlen] & 0x80) != 0 ||
                         isalnum(line[i-prevlen]) ||
                         line[i-prevlen] == '_' ||
                         line[i-prevlen] == '$' )))


The test should not succeed anywhere in the string '42$foo$'.

Note that psql does not change any '$foo$' at all - it just passes it to 
the backend. The reason we need this at all in psql is that it has to 
detect the end of a statement, and it has to prompt correctly, and to do 
that it needs to know if we are in a quote (single, double, dollar) or a 
comment.

psql does not detect many syntax errors, or even lexical errors - that 
is the job of the backend - rightly so, I believe.

>
>There's no way to tell these apart with a single-character lookback,
>or indeed any fixed number of characters of lookback.
>

I'm still not convinced, although maybe there's something I'm not getting.

>
>I begin to think that we'll really have to bite the bullet and convert
>psql's input parser to use flex.  If we're not scanning with exactly the
>same rules as the backend uses, we're going to get the wrong answers.
>
>  
>

Interacting with lexer states would probably be ... unpleasant. Matching 
a stream oriented lexer with a line oriented CLI would be messy I suspect.

cheers

andrew


Sat, 14 Feb 2004 15:04:31 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: [HACKERS] dollar quoting
Andrew Dunstan <andrew@dunslane.net> writes:
> Tom Lane wrote:
>> ... But how about
>> 42$foo$
>> This is a syntax error in 7.4, and we propose to redefine it as an
>> integer literal '42' followed by a dollar-quote start symbol.

> The test should not succeed anywhere in the string '42$foo$'.

No, it won't.  The problem is that it should, because the backend will
see that as '42' followed by a $foo$ quote start.

> Interacting with lexer states would probably be ... unpleasant. Matching 
> a stream oriented lexer with a line oriented CLI would be messy I suspect.

I think it would not be that bad.  We'd have to run the lexer on the
command input buffer and see what state it terminates in.

			regards, tom lane

Sun, 15 Feb 2004 00:45:04 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] dollar quoting 
Tom Lane wrote:

>Andrew Dunstan <andrew@dunslane.net> writes:
>  
>
>>Tom Lane wrote:
>>    
>>
>>>... But how about
>>>42$foo$
>>>This is a syntax error in 7.4, and we propose to redefine it as an
>>>integer literal '42' followed by a dollar-quote start symbol.
>>>      
>>>
>
>  
>
>>The test should not succeed anywhere in the string '42$foo$'.
>>    
>>
>
>No, it won't.  The problem is that it should, because the backend will
>see that as '42' followed by a $foo$ quote start.
>

Ok, I see what you are saying. This mismatch would only happen on 
invalid input, though. I believe that what I did will work on all legal 
input.

I think that this might be cured by having psql recognise a legal 
identifier or keyword and eating it as a word, rather than treating it 
as just another set of bytes in the stream. That would enable us to 
avoid the lookback in the dollar-quote recognition test altogether. The 
attached patch does it that way - the keyword/id test needs to come 
right at the end of the loop to avoid clashing with backslash commands, 
btw.

I *think* that this way psql will recognise the start of a dollar quote 
iff the backend lexer would.

>
>  
>
>>Interacting with lexer states would probably be ... unpleasant. Matching 
>>a stream oriented lexer with a line oriented CLI would be messy I suspect.
>>    
>>
>
>I think it would not be that bad.  We'd have to run the lexer on the
>command input buffer and see what state it terminates in.
>
>  
>

Yeah. I am not enough of a flex wizard to undertake the task, though. It 
would take me lots of time. If we make a decision that we really need 
this in order to do dollar quoting in psql I would need some substantial 
help, at least.

cheers

andrew
Sun, 15 Feb 2004 09:36:58 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: [HACKERS] dollar quoting
Andrew Dunstan <andrew@dunslane.net> writes:
>> No, it won't.  The problem is that it should, because the backend will
>> see that as '42' followed by a $foo$ quote start.

> Ok, I see what you are saying. This mismatch would only happen on 
> invalid input, though. I believe that what I did will work on all legal 
> input.

I'm unconvinced.  Even if there are not any current syntaxes in which a
numeric literal can be adjacent to a string literal (I'm not totally
sure about that), what of the future?  We should solve the problem
rather than assuming it won't bite us.

> I think that this might be cured by having psql recognise a legal 
> identifier or keyword and eating it as a word, rather than treating it 
> as just another set of bytes in the stream.

Hm, might work ... will think about it ...

			regards, tom lane

Sun, 15 Feb 2004 11:42:40 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] dollar quoting 

Another interesting thing abut psql that I noticed when using '$' in 
identifiers is this effect:


andrew=# create table ab$cd$ef (ef$cd$ab text);
CREATE TABLE
andrew=# \d ab$cd$ef
Did not find any relation named "ab$cd$ef".
andrew=# \d ab\$cd\$ef
   Table "public.ab$cd$ef"
  Column  | Type | Modifiers
----------+------+-----------
 ef$cd$ab | text |


which is perhaps slightly less than intuitive.

cheers

andrew

>  
>


Sun, 15 Feb 2004 15:55:36 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: [PATCHES] dollar quoting
Andrew Dunstan <andrew@dunslane.net> writes:
> andrew=# create table ab$cd$ef (ef$cd$ab text);
> CREATE TABLE
> andrew=# \d ab$cd$ef
> Did not find any relation named "ab$cd$ef".

Hmph.  I always thought that "$" was only special at the end of a regex,
but that doesn't seem to be how our implementation treats it.  Anyway
this is not a bug, it is a feature: the argument of \d is a regex.

			regards, tom lane

Sun, 15 Feb 2004 17:17:18 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] dollar quoting 


Tom Lane wrote:

>Andrew Dunstan <andrew@dunslane.net> writes:
>  
>
>>andrew=# create table ab$cd$ef (ef$cd$ab text);
>>CREATE TABLE
>>andrew=# \d ab$cd$ef
>>Did not find any relation named "ab$cd$ef".
>>    
>>
>
>Hmph.  I always thought that "$" was only special at the end of a regex,
>but that doesn't seem to be how our implementation treats it.  Anyway
>this is not a bug, it is a feature: the argument of \d is a regex.
>
>  
>

Arguably this at least is a singularly useless feature, since a 
regex-meaning $ before the end of string is a nonsense, as you rightly 
imply, and one at the end of the string is redundant, as it is implied - 
psql turns 'abc' into '^abc$' when constructing the query.

I don't care that much - I don't use $ in my identifiers.

cheers

andrew


Sun, 15 Feb 2004 21:16:21 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: [PATCHES] dollar quoting
Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
> >> No, it won't.  The problem is that it should, because the backend will
> >> see that as '42' followed by a $foo$ quote start.
> 
> > Ok, I see what you are saying. This mismatch would only happen on 
> > invalid input, though. I believe that what I did will work on all legal 
> > input.
> 
> I'm unconvinced.  Even if there are not any current syntaxes in which a
> numeric literal can be adjacent to a string literal (I'm not totally
> sure about that), what of the future?  We should solve the problem
> rather than assuming it won't bite us.
> 
> > I think that this might be cured by having psql recognise a legal 
> > identifier or keyword and eating it as a word, rather than treating it 
> > as just another set of bytes in the stream.
> 
> Hm, might work ... will think about it ...

I am a little concerned about adding the overhead of lex to psql.  Right
now, some folks have reported that lex/yacc take a considerable amount
of processing time in the backend as part of a query, and adding that to
psql just to do $$ seems questionable.  Of course, we can alway test and
see what the overhead shows.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 16 Feb 2004 10:48:11 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] dollar quoting
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I am a little concerned about adding the overhead of lex to psql.  Right
> now, some folks have reported that lex/yacc take a considerable amount
> of processing time in the backend as part of a query, and adding that to
> psql just to do $$ seems questionable.  Of course, we can alway test and
> see what the overhead shows.

That's not the question to ask --- the question is whether a flex lexer
will be faster or slower than the hand-made lexing code that psql is
currently using.  Lexer generation is a well-understood art, and you
have to be pretty tense to beat out flex with hand-made code.  It's the
same tradeoff as trying to write better assembly code than a compiler
does.  Look at the lexing loop in psql/mainloop.c (that series of
if-tests starting at about line 250) and ask yourself if that's really
going to beat out the state-machine implementation flex uses --- which
looks to be about two table lookups per character, plus a switch()
executed at the end of every token.  I'll bet on flex being faster.

The reason the lexer shows up in the backend is that it has to grovel
over every individual character of a query.  For sufficiently large
query strings that's gonna take awhile no matter how you do it.

But in any case, the argument for moving to flex is not about
performance, it is about making the code more understandable and more
certain to agree with the backend lexer.

			regards, tom lane

Mon, 16 Feb 2004 11:07:46 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] dollar quoting 
Bruce Momjian wrote:

>Tom Lane wrote:
>  
>
>>Andrew Dunstan <andrew@dunslane.net> writes:
>>    
>>
>>>>No, it won't.  The problem is that it should, because the backend will
>>>>see that as '42' followed by a $foo$ quote start.
>>>>        
>>>>
>>>Ok, I see what you are saying. This mismatch would only happen on 
>>>invalid input, though. I believe that what I did will work on all legal 
>>>input.
>>>      
>>>
>>I'm unconvinced.  Even if there are not any current syntaxes in which a
>>numeric literal can be adjacent to a string literal (I'm not totally
>>sure about that), what of the future?  We should solve the problem
>>rather than assuming it won't bite us.
>>
>>    
>>
>>>I think that this might be cured by having psql recognise a legal 
>>>identifier or keyword and eating it as a word, rather than treating it 
>>>as just another set of bytes in the stream.
>>>      
>>>
>>Hm, might work ... will think about it ...
>>    
>>
>
>I am a little concerned about adding the overhead of lex to psql.  Right
>now, some folks have reported that lex/yacc take a considerable amount
>of processing time in the backend as part of a query, and adding that to
>psql just to do $$ seems questionable.  Of course, we can alway test and
>see what the overhead shows.
>
>  
>

I think Tom's point is that the custom lexical recognition done by psql 
is approaching the point of being unmaintainable, and if we have to bite 
the bullet then maybe we might as well do so now.

I'd be surprised if using a flex lexer instead made a huge speed 
difference, but maybe I'm wrong. I'm more concerned that it will be 
difficult to write and maintain and keep in sync with the backend's 
lexical structure - but those are just gut feelings, and I could be way 
off base. Right now psql does just enough recognition to enable it to 
work. Making it recognise the whole sql lexical structure instead 
strikes me as being somewhat redundant - that's what the backend does. 
Maybe we could do a flex lexer somewhat along the lines of the 
minimalist approach that psql currently employs.

Anyway - I did put forward a possible non-flex way to handle the problem 
Tom saw with $$ quoting - let's see what his verdict is on that first 
;-) If he thinks it won't work, I can't see much alternative to using a 
flex-based lexer.

cheers

andrew


Mon, 16 Feb 2004 11:15:50 -0500
Andrew Dunstan <andrew@dunslane.net>
Re: [PATCHES] dollar quoting
Andrew Dunstan <andrew@dunslane.net> writes:
> I'd be surprised if using a flex lexer instead made a huge speed 
> difference, but maybe I'm wrong.

No, I agree --- it's unlikely to make very much difference in the real
world.  Maybe on huge query strings you could notice the difference.

> I'm more concerned that it will be 
> difficult to write and maintain and keep in sync with the backend's 
> lexical structure - but those are just gut feelings, and I could be way 
> off base. Right now psql does just enough recognition to enable it to 
> work. Making it recognise the whole sql lexical structure instead 
> strikes me as being somewhat redundant - that's what the backend does. 

Actually, I thought the way to handle it would be to duplicate the
backend lexer as nearly as possible.  Most of the productions would have
empty bodies probably, but doing it that way would guarantee that in
fact psql and the backend would lex a string the same way, which is
exactly the problem we are facing here.  You'd fall out of the lexer
only upon detecting backslash (unless we want to put backslash command
lexing into the flex code, which might or might not be a good idea),
or upon detecting a ';' at parenthesis depth 0, or upon hitting end of
string.  In the last case the lexer state would indicate which prompt
we need to give.

One unsolved issue in my mind is how to deal with multibyte characters
in non-backend-compatible encodings (the ones where some bytes of a
multibyte character can have values < 128 and thus can look like plain
ASCII characters).  We'd have to make sure that the flex lexer skips
over such bytes properly.  I've thought of a couple of kluges but
nothing I like ...

			regards, tom lane

Mon, 16 Feb 2004 13:18:30 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] dollar quoting 
> Actually, I thought the way to handle it would be to duplicate the
> backend lexer as nearly as possible.  Most of the productions would have
> empty bodies probably, but doing it that way would guarantee that in
> fact psql and the backend would lex a string the same way, which is
> exactly the problem we are facing here.  You'd fall out of the lexer
> only upon detecting backslash (unless we want to put backslash command
> lexing into the flex code, which might or might not be a good idea),
> or upon detecting a ';' at parenthesis depth 0, or upon hitting end of
> string.  In the last case the lexer state would indicate which prompt
> we need to give.

You know what would be really sweet?  If the lexing was made available 
as a public function.  eg. So I could parse queries in phpPgAdmin before 
sending them to the backend, etc...

Chris

Tue, 17 Feb 2004 09:15:46 +0800
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: [PATCHES] dollar quoting
Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes:
> You know what would be really sweet?  If the lexing was made available 
> as a public function.  eg. So I could parse queries in phpPgAdmin before 
> sending them to the backend, etc...

Parsing is a whole nother ball of wax besides lexing.  I wasn't planning
to put *that* into psql.  Remember the only thing psql really wants from
this is to detect where end-of-statement is ...

			regards, tom lane

Mon, 16 Feb 2004 21:22:41 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] dollar quoting 
> Parsing is a whole nother ball of wax besides lexing.  I wasn't planning
> to put *that* into psql.  Remember the only thing psql really wants from
> this is to detect where end-of-statement is ...

Forgive my lameness, but I've never truly figured out where parsing ends 
and lexing begins.  Anyone care to illuminate me on the difference?

Chris

Tue, 17 Feb 2004 10:42:22 +0800
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: [PATCHES] dollar quoting
Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes:
>> Parsing is a whole nother ball of wax besides lexing.

> Forgive my lameness, but I've never truly figured out where parsing ends 
> and lexing begins.  Anyone care to illuminate me on the difference?

The theoretical answer is that you can do lexing with a finite-state
machine, but parsing generally requires a stack, because it supports
nested constructs.  Lexers don't have any way to describe nested
constructs --- a series of tokens is the only level of abstraction there
is.

The practical answer is that you do one with flex and the other with
bison ;-).  If you can do it with flex, and not cheat by implementing
your own state stack, it's lexing.

			regards, tom lane

Mon, 16 Feb 2004 21:54:52 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] dollar quoting 
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I am a little concerned about adding the overhead of lex to psql.  Right
> > now, some folks have reported that lex/yacc take a considerable amount
> > of processing time in the backend as part of a query, and adding that to
> > psql just to do $$ seems questionable.  Of course, we can alway test and
> > see what the overhead shows.
> 
> That's not the question to ask --- the question is whether a flex lexer
> will be faster or slower than the hand-made lexing code that psql is
> currently using.  Lexer generation is a well-understood art, and you
> have to be pretty tense to beat out flex with hand-made code.  It's the
> same tradeoff as trying to write better assembly code than a compiler
> does.  Look at the lexing loop in psql/mainloop.c (that series of
> if-tests starting at about line 250) and ask yourself if that's really
> going to beat out the state-machine implementation flex uses --- which
> looks to be about two table lookups per character, plus a switch()
> executed at the end of every token.  I'll bet on flex being faster.
> 
> The reason the lexer shows up in the backend is that it has to grovel
> over every individual character of a query.  For sufficiently large
> query strings that's gonna take awhile no matter how you do it.
> 
> But in any case, the argument for moving to flex is not about
> performance, it is about making the code more understandable and more
> certain to agree with the backend lexer.

If we go with lex/flex for psql, I would still like someone to test
performance to see that we aren't taking a big hit.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 16 Feb 2004 22:16:01 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] dollar quoting
Christopher Kings-Lynne wrote:
> > Parsing is a whole nother ball of wax besides lexing.  I wasn't planning
> > to put *that* into psql.  Remember the only thing psql really wants from
> > this is to detect where end-of-statement is ...
> 
> Forgive my lameness, but I've never truly figured out where parsing ends 
> and lexing begins.  Anyone care to illuminate me on the difference?

The simplistic answer is the lexing breaks a string up into
words/tokens, and parsing matches those tokens against patterns and
fires actions.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Mon, 16 Feb 2004 23:01:38 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [PATCHES] dollar quoting
> Parsing is a whole nother ball of wax besides lexing.  I wasn't planning
> to put *that* into psql.  Remember the only thing psql really wants from
> this is to detect where end-of-statement is ...

OK, I'm not that great at understanding where lexing ands and parsing 
starts.  These are the things that it would be really nice to know about 
  in phpPgAdmin:

* Start and end of multiline queries, so when executing an SQL script I 
can execute each query separately.

* Knowing how many $n parameters there are in a SELECT statement that is 
to be prepared.

Chris


Fri, 04 Jun 2004 09:45:57 +0800
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: [PATCHES] dollar quoting
Doh, sorry.  I just realised that the lists just gave me a whole bunch 
of mails from back in February, which is when Tom made this post...

Chris

Christopher Kings-Lynne wrote:

>> Parsing is a whole nother ball of wax besides lexing.  I wasn't planning
>> to put *that* into psql.  Remember the only thing psql really wants from
>> this is to detect where end-of-statement is ...
> 
> 
> OK, I'm not that great at understanding where lexing ands and parsing 
> starts.  These are the things that it would be really nice to know about 
>  in phpPgAdmin:
> 
> * Start and end of multiline queries, so when executing an SQL script I 
> can execute each query separately.
> 
> * Knowing how many $n parameters there are in a SELECT statement that is 
> to be prepared.
> 
> Chris
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
>      subscribe-nomail command to majordomo@postgresql.org so that your
>      message can get through to the mailing list cleanly

Fri, 04 Jun 2004 09:57:57 +0800
Christopher Kings-Lynne <chriskl@familyhealth.com.au>
Re: [PATCHES] dollar quoting