Jeroen van Vianen <jeroen@design.nl> writes:
>> Does this work with a non-bison parser?  It looks mighty
>> bison-dependent to me...

> I'm not sure, but it probably is flex dependent (but Postgres always needed 
> flex anyway). I'm not aware of any yacc / byacc / bison dependencies. Don't 
> know if anybody has been successful building Postgres with another parser 
> generator.

Um, you're right of course --- those are lexer not parser datastructures
you're poking into.  Sorry for my confusion.

We do in fact work with non-bison parser generators, or did last time
I tried it (around 6.5 release).  I would not like us to stop working
with non-bison yaccs, since bison's output depends on alloca() which
is not available everywhere.

I'm not sure about the situation with lexers.  We have been saying for
a long time that flex was required, but since we got rid of the
scanner's use of trailing context ('/' rules) I think there is a better
chance that it would work with vanilla lex.  Anyone want to try that
with current sources?

> BTW, as we ship flex's output lex.yy.c (as scan.c) and bison's output
> (gram.c) in the distribution, any user would be able to compile the
> sources, but if they want to start hacking the .l or .y files, they'll
> need appropriate tools.

Right.  I am not aware of any portability problems with flex's output
as there are with bison's, so it may be that the concern is moot.
We may just be able to say "use the prebuilt scan.c or get flex; we
don't care about supporting vendor lexes anymore".

I do see a potential problem with this patch that's not related to
portability questions; it is that you're assuming that the lexer's
furthest penetration into the source text is a good place to point
at for parser errors.  That may not be true always.  In particular,
I've been advocating solving some other problems by inserting a
one-token lookahead buffer between the parser and the lexer.  If that
happens then you'd be off by (at least) one token in some cases.

I think the way that this sort of thing is customarily handled in
"real" compilers is that each token carries along an indication of
just where it was found in the source, and then error messages can
finger the right place without making assumptions about synchronization
between different phases of the scanning/parsing process.  That might
be more work than we can justify for SQL queries; not sure.

BTW, I think that the immediate problem of generating a good error
message for unterminated comments and literals could be solved in other
ways.  This patch or something like it might be cool anyway, but you
should bear in mind that printing out a query and then a marker that's
supposed to line up with something in the query doesn't always work
all that well.  Consider a query that's several dozen lines long,
such as a large table definition.  If we had more control over the
user interface and could highlight the offending token directly,
I'd be more excited about doing something like this.  (Actually, you
could partially address that problem by only printing one line's worth
of query text leading up to the error marker point.  It would still be
tricky to get it right in the presence of newlines, tabs, etc.)

			regards, tom lane

Sun, 20 Feb 2000 20:03:40 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [PATCHES] Patch for more readable parse error messages 
Tom Lane wrote:

> We do in fact work with non-bison parser generators, or did last time
> I tried it (around 6.5 release).  I would not like us to stop working
> with non-bison yaccs, since bison's output depends on alloca() which
> is not available everywhere.

I think GNU alloca should work on any platform because it's written in a
portable way.

Mon, 21 Feb 2000 14:40:54 +1100
Chris Bitmead <chrisb@nimrod.itg.telstra.com.au>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse error
	messages
At 20:03 20-02-00 -0500, Tom Lane wrote:
>I do see a potential problem with this patch that's not related to
>portability questions; it is that you're assuming that the lexer's
>furthest penetration into the source text is a good place to point
>at for parser errors.  That may not be true always.  In particular,
>I've been advocating solving some other problems by inserting a
>one-token lookahead buffer between the parser and the lexer.  If that
>happens then you'd be off by (at least) one token in some cases.

That's true, but the '*' indicator might at least indicate the approximate 
location of the error. I'm not aware of many (programming) languages that 
are able to indicate the error at the correct location all the time, anyway.

>I think the way that this sort of thing is customarily handled in
>"real" compilers is that each token carries along an indication of
>just where it was found in the source, and then error messages can
>finger the right place without making assumptions about synchronization
>between different phases of the scanning/parsing process.  That might
>be more work than we can justify for SQL queries; not sure.

True, but requires a lot more work.

>BTW, I think that the immediate problem of generating a good error
>message for unterminated comments and literals could be solved in other
>ways.  This patch or something like it might be cool anyway, but you
>should bear in mind that printing out a query and then a marker that's
>supposed to line up with something in the query doesn't always work
>all that well.  Consider a query that's several dozen lines long,
>such as a large table definition.  If we had more control over the
>user interface and could highlight the offending token directly,
>I'd be more excited about doing something like this.  (Actually, you
>could partially address that problem by only printing one line's worth
>of query text leading up to the error marker point.  It would still be
>tricky to get it right in the presence of newlines, tabs, etc.)

I try to make a good guess at where the location of the error is, but am 
hesitant to only print a few tokens near the error locations, as you won't 
be able to know where the error was found in complex queries or table 
definitions. Please try with more complex queries and tell me what you think.


Jeroen


Mon, 21 Feb 2000 10:07:11 +0100
Jeroen van Vianen <jeroen@design.nl>
Re: [PATCHES] Patch for more readable parse error messages 
On 2000-02-20, Tom Lane mentioned:

> I would not like us to stop working
> with non-bison yaccs, since bison's output depends on alloca() which
> is not available everywhere.

Couldn't alloca(x) be defined to palloc(x) where missing? The performance
will be worse, but it ought to work.

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Tue, 22 Feb 2000 00:57:23 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse error
	messages
Peter Eisentraut <peter_e@gmx.net> writes:
> On 2000-02-20, Tom Lane mentioned:
>> I would not like us to stop working
>> with non-bison yaccs, since bison's output depends on alloca() which
>> is not available everywhere.

> Couldn't alloca(x) be defined to palloc(x) where missing?

Probably, but I wasn't looking for a workaround; that was just one
quick illustration of a reason not to want to use bison (one that's
bitten me personally, so I knew it offhand).  We should try not to
become dependent on bison when there are near-equivalent tools, just
on general principles of maintaining portability.  For an analogy,
I believe most of the developers use gcc, but it would be a real bad
idea for us to abandon support for other compilers.

For the same sort of reasons I'd prefer that our scanner worked
with vanilla lex, not just flex.  I'm not sure how far away we are
from that; it may be an unrealistic goal.  But if it is within reach
then we shouldn't give it up lightly.

			regards, tom lane

Mon, 21 Feb 2000 22:56:34 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse error
	messages
> Peter Eisentraut <peter_e@gmx.net> writes:
> > On 2000-02-20, Tom Lane mentioned:
> >> I would not like us to stop working
> >> with non-bison yaccs, since bison's output depends on alloca() which
> >> is not available everywhere.
> 
> > Couldn't alloca(x) be defined to palloc(x) where missing?
> 
> Probably, but I wasn't looking for a workaround; that was just one
> quick illustration of a reason not to want to use bison (one that's
> bitten me personally, so I knew it offhand).  We should try not to
> become dependent on bison when there are near-equivalent tools, just
> on general principles of maintaining portability.  For an analogy,
> I believe most of the developers use gcc, but it would be a real bad
> idea for us to abandon support for other compilers.

But I don't see non-bison solutions for finding the location of errors. 
Is it possible?  Could we enable the feature just for bison?


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mon, 21 Feb 2000 23:08:26 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse error
	messages
At 10:56 PM 2/21/00 -0500, Tom Lane wrote:
>Probably, but I wasn't looking for a workaround; that was just one
>quick illustration of a reason not to want to use bison (one that's
>bitten me personally, so I knew it offhand).  We should try not to
>become dependent on bison when there are near-equivalent tools, just
>on general principles of maintaining portability.  For an analogy,
>I believe most of the developers use gcc, but it would be a real bad
>idea for us to abandon support for other compilers.
>
>For the same sort of reasons I'd prefer that our scanner worked
>with vanilla lex, not just flex.  I'm not sure how far away we are
>from that; it may be an unrealistic goal.  But if it is within reach
>then we shouldn't give it up lightly.

I agree entirely with the above.  The more portable the tool, the larger
the potential user base.   Unless the goal is to bundle-up Postgres with
a pre-defined set of software, i.e. GNU in this case (despite the fact
that I don't see Postgres on their site as part of their list of open-source
software, and I think I looked twice), go for the cover-the-earth approach.

SQL syntax isn't particularly difficult.  On the other hand, I realize there's
a legacy to support.  Still, making portions of the product dependent on one
tool or another is an issue that merits close scrutiny.  Shouldn't be done 
except under compelling reasons.

I mean, presuming a reasonably modern C, C tools, and large-scale 
operating-system environment makes sense (no reason to run native on a palm
pilot,
at this point).  But unecessary dependence on particular tools when not
necessary doesn't make much sense.

Just IMO, of course.



- Don Baccus, Portland OR <dhogaza@pacifier.com>
  Nature photos, on-line guides, Pacific Northwest
  Rare Bird Alert Service and other goodies at
  http://donb.photo.net.

Mon, 21 Feb 2000 21:05:51 -0800
Don Baccus <dhogaza@pacifier.com>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse
	error messages
At 00:56 22-02-00 +0100, Peter Eisentraut wrote:
>On 2000-02-20, Jeroen van Vianen mentioned:
>
> > The format of the error messages is changed to:
> >
> > jeroen=# create abc ( a int4, b int4 );
> > ERROR:  parser: parse error at or near "abc":
> > create abc ( a int4, b int4 )
> >         *
>
>I believe this is the wrong approach because it's highly psql specific. If
>you use PHP or JDBC or something not character cell based you will get
>misleading output.
>
>You might want to start thinking about putting a little more information
>into an ERROR than just a text string, such as error codes or
>supplementary information like this. psql could then choose to print a
>star, other interfaces might set the cursor to the specified place, etc.
>Much more flexible.

Good idea. As far as I understand things, libpq uses special datastructures 
to access the error code and message and it's up to the application (psql, 
and others) to do what it wants to do with it (let's say print the error). 
These structures might be enhanced with an error location, but this might 
be breaking things. And my question is how to do this.

Note that this location part is only filled now when yyerror() throws an 
error, but other parts of the backend might use a similar approach. OTOH it 
might be nice then to have every token know its location in the query 
string (as Don suggested), so you might end up with error messages like:

mydb-> select * from t1, t2 where ...
ERROR: table t2 not found:
select * from t1, t2 where ...
                   ^

(which may be nice, or not).

What I see now is something like this (for psql):

psql sends a query
psql reads response
         if response is error
                 get error location and find context in which error occurred
                 print error message, with error location and context
         otherwise
                 do what it used to do

and for the other interfaces nothing changes.

This is something I might be able to implement for 7.1.

What do you think?


Jeroen


Tue, 22 Feb 2000 10:55:13 +0100
Jeroen van Vianen <jeroen@design.nl>
Re: [PATCHES] Patch for more readable parse error messages
On Mon, 21 Feb 2000, Tom Lane wrote:

> For the same sort of reasons I'd prefer that our scanner worked
> with vanilla lex, not just flex.  I'm not sure how far away we are
> from that; it may be an unrealistic goal.  But if it is within reach
> then we shouldn't give it up lightly.

I concur. Somewhere in between vanilla lex and flex is also POSIX lex,
which does support exclusive start conditions but no <<EOF>>.

Anyone for getting rid of GNU make?

-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Tue, 22 Feb 2000 13:18:25 +0100 (MET)
Peter Eisentraut <e99re41@DoCS.UU.SE>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse error
	messages
Peter Eisentraut <e99re41@DoCS.UU.SE> writes:
> ... Somewhere in between vanilla lex and flex is also POSIX lex,
> which does support exclusive start conditions but no <<EOF>>.

I noticed that in the flex manual.  Does it help us any?  That is,
are there a useful number of lexes out there that do the full POSIX
spec?  If flex is our only real choice for exclusive start conditions
then it's pointless to avoid <<EOF>>.

> Anyone for getting rid of GNU make?

No ;-).  GNU make has enough important features that there is no
near-equivalent non-GNU make.  VPATH, for example.  One thing I hope
we will be able to do sometime soon is build in an object directory
tree separate from the source tree... can't realistically do that
with any non-GNU make that I've heard of.

			regards, tom lane

Tue, 22 Feb 2000 10:50:03 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse error
	messages
Jeroen van Vianen <jeroen@design.nl> writes:
> What I see now is something like this (for psql):

> psql sends a query
> psql reads response
>          if response is error
>                  get error location and find context in which error occurred
>                  print error message, with error location and context
>          otherwise
>                  do what it used to do

> and for the other interfaces nothing changes.

> This is something I might be able to implement for 7.1.

This looks much better to me than doing it in the backend.  What still
needs a little thought is how to send back the error location from
backend to client app.

I'd be inclined to say that the location info should be imbedded as
text in the existing textual error message, rather than trying to add
a separate message with a machine-readable location value.  The first
way is much less likely to create compatibility problems with old client
apps.  One way to do it is to say that if the last line of the error
message looks like

Error-location: nnn

then libpq should recognize that, strip the line out of the saved
textual error message, and make the location value available through
a new API call.

The reason I suggest a label is that we could further extend this
protocol to handle some other things that people have been griping
about for a long time: providing identifying error code numbers that
client code could rely on instead of trying to match against the error
text, and separating out the info about which routine generated the
error (which is mighty handy for backend debugging but is useless
info for Joe Average user).  Someday the message being sent back
might look less like

ERROR: relation_info: Relation 12345 not found

and more like

ERROR: Failed to find relation OID 12345 in system catalogs
Error-code: 4242
Reporting-routine: relation_info, plancat.c line 543

of which only the first line is really meant for the user.

Of course, making that happen will be a lot of work, and I'm not
asking you to volunteer for it.  But what you do now should fit
in with further development of the error handling stuff...

			regards, tom lane

Tue, 22 Feb 2000 11:12:14 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse error
	messages
On 2000-02-22, Tom Lane mentioned:

> > Anyone for getting rid of GNU make?
> 
> No ;-).  GNU make has enough important features that there is no
> near-equivalent non-GNU make.  VPATH, for example.

There are other makes that support this too. While I love GNU make, too,
all the talk about allowing vanilla lex, etc. is pointless while GNU make
is required. Users don't see lex at all, they do see make.

OTOH, it is very hard for me to get an overview these days what's actually
out there in terms of other make's, other lex's, other yacc's, other
compilers. You should have an edge there (HPUX and all). Most
installations of commercial Unix vendors I get to nowadays use gcc, gmake,
flex as system tools. Yesterday I read that Sun builds Java proper with
GNU make!

The best way of going about this seems to take one of the perpetrators
(make file, gram.y, etc.) and try to port it to some given non-GNU tool
and take a look at the consequences. For example, if we get PostgreSQL to
compile with FreeBSD's make without crippling everything, that would be a
win for the user base. This may in fact be the first experiment.

> One thing I hope we will be able to do sometime soon is build in an
> object directory tree separate from the source tree... can't
> realistically do that with any non-GNU make that I've heard of.

I'm planning to work on that for 7.1. But here's an interesting tidbit:
Automake does support this feature but in its manual it claims that it
does not use any GNU make specific features. And in fact, VPATH exists in
both System V's and 4.3 BSD's make.


-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Wed, 23 Feb 2000 02:20:12 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse error
	messages
Peter Eisentraut <peter_e@gmx.net> writes:
> On 2000-02-22, Tom Lane mentioned:
>>>> Anyone for getting rid of GNU make?
>> 
>> No ;-).  GNU make has enough important features that there is no
>> near-equivalent non-GNU make.  VPATH, for example.

> There are other makes that support this too. While I love GNU make, too,
> all the talk about allowing vanilla lex, etc. is pointless while GNU make
> is required. Users don't see lex at all, they do see make.

Huh?  Assuming someone will have program X installed is not the same as
assuming they will have program Y installed.  In this particular case,
a more exact way of putting it is that assuming program X is installed
is not the same as assuming that program Y's prebuilt-on-another-machine
output is usable on this platform.

> OTOH, it is very hard for me to get an overview these days what's actually
> out there in terms of other make's, other lex's, other yacc's, other
> compilers.

Not much.  The real problem here is "what set of tool features do you
assume you have, and what's it costing you in portability?"  GNU make
provides a very rich feature set that's widely portable, although you
do have to port the particular implementation.  If you don't want to
assume GNU make but just a generic make, there's a big gap in features
before you drop down to what's actually portable to a wide class of
vendor-provided makes.  VPATH, for example, does exist in *some*
vendor makes, but as a practical matter if you use it then you'd better
tell people "my program requires GNU make".  It's not worth the trouble
to keep track of the exceptions.

I will be the first to admit this is all a matter of judgment calls
rather than certainties.  As far as I can see, it's not worth our
trouble to try to operate with non-GNU makes; it is worth the trouble
to work with non-GNU yaccs, because we're not really using any bison-
specific features; it's looking like we should forget about non-GNU
lexes, but I'm not quite convinced yet.  You're free to hold different
opinions of course.  I've been around for a few years in the portable-
software game, so I tend to think I know where the minefields are, but
perhaps my hard experiences are out of date.

> The best way of going about this seems to take one of the perpetrators
> (make file, gram.y, etc.) and try to port it to some given non-GNU tool
> and take a look at the consequences.

But that only tells you about the one tool; in fact, only about the one
version of the one tool that you test.  In practice, useful knowledge
in this area comes from the school of hard knocks: ship an open-source
program and see what complaints you get.  I'd rather rely on experience
previously gained than learn those lessons again...

>> One thing I hope we will be able to do sometime soon is build in an
>> object directory tree separate from the source tree... can't
>> realistically do that with any non-GNU make that I've heard of.

> I'm planning to work on that for 7.1. But here's an interesting tidbit:
> Automake does support this feature but in its manual it claims that it
> does not use any GNU make specific features.

Yeah?  Do they claim not to need VPATH to do it?  I suppose it might
be possible, if they are willing to write sufficiently ugly and
non-hand-maintainable makefiles.  Not sure that's a good tradeoff
though.

> And in fact, VPATH exists in both System V's and 4.3 BSD's make.

You're still confusing two datapoints with the wide world...

			regards, tom lane

Wed, 23 Feb 2000 00:48:09 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse error
	messages
On Wed, 23 Feb 2000, Tom Lane wrote:

> > And in fact, VPATH exists in both System V's and 4.3 BSD's make.
> 
> You're still confusing two datapoints with the wide world...

I challenge everyone to show me a make without VPATH. In fact, show me two
makes without a feature that you can't live without, and I shall forever
hold my peace. It's certainly easier to say "let's support yacc, because
we actually don't use any non-yacc features" than saying it for make. But
it's not the idea to say "we need GNU make because it has all these
features" when 93% of these features in fact exist in all other reasonable
makes as well. It's not the end of the world but it's something that
shouldn't be ignored.


-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Wed, 23 Feb 2000 14:20:44 +0100 (MET)
Peter Eisentraut <e99re41@DoCS.UU.SE>
GNU make (Re: [HACKERS] Re: [PATCHES] Patch for more readable
	parse error messages)
Peter Eisentraut <e99re41@DoCS.UU.SE> writes:
> On Wed, 23 Feb 2000, Tom Lane wrote:
>>>> And in fact, VPATH exists in both System V's and 4.3 BSD's make.
>> 
>> You're still confusing two datapoints with the wide world...

> I challenge everyone to show me a make without VPATH. In fact, show me two
> makes without a feature that you can't live without, and I shall forever
> hold my peace.

Out of the four systems I have easy access to: HPUX 10, HPUX 9, Linux
(some fairly old RedHat version), and SunOS 4.1.4, two have makes
without VPATH ... and Linux doesn't really count since it's using gmake
anyway.

Now you can argue that HPUX 9 and SunOS 4.1.4 are dinosaurs that should
be put out of their misery, and I wouldn't disagree --- but reality is
that a lot of people are running older systems and don't have the time
or interest to upgrade 'em.  "Portability" doesn't mean "portability to
the newest and most standards-conformant systems", it means portability
to what's actually out there.

> it's not the idea to say "we need GNU make because it has all these
> features" when 93% of these features in fact exist in all other reasonable
> makes as well.

If I thought we were anywhere near that close to being able to use old
makes, I'd be arguing for removing the GNU-make dependency too.  But
I don't think it's going to be practical...

			regards, tom lane

Wed, 23 Feb 2000 11:30:10 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: GNU make (Re: [HACKERS] Re: [PATCHES] Patch for more readable
	parse error messages)
At 11:12 22-02-00 -0500, Tom Lane wrote:
>Jeroen van Vianen <jeroen@design.nl> writes:
> > What I see now is something like this (for psql):
>
> > psql sends a query
> > psql reads response
> >          if response is error
> >                  get error location and find context in which error 
> occurred
> >                  print error message, with error location and context
> >          otherwise
> >                  do what it used to do
>
> > and for the other interfaces nothing changes.
>
> > This is something I might be able to implement for 7.1.
>
>This looks much better to me than doing it in the backend.  What still
>needs a little thought is how to send back the error location from
>backend to client app.
>
>I'd be inclined to say that the location info should be imbedded as
>text in the existing textual error message, rather than trying to add
>a separate message with a machine-readable location value.  The first
>way is much less likely to create compatibility problems with old client
>apps.  One way to do it is to say that if the last line of the error
>message looks like
>
>Error-location: nnn
>
>then libpq should recognize that, strip the line out of the saved
>textual error message, and make the location value available through
>a new API call.

Isn't it possible to get this kind of information from a call to a new API 
struct errorinfo *  PQerrorInfo(conn) where the struct contains info about 
the error message, location and code, rather than a call to 
PQerrorMessage(conn) ?

>The reason I suggest a label is that we could further extend this
>protocol to handle some other things that people have been griping
>about for a long time: providing identifying error code numbers that
>client code could rely on instead of trying to match against the error
>text, and separating out the info about which routine generated the
>error (which is mighty handy for backend debugging but is useless
>info for Joe Average user).  Someday the message being sent back
>might look less like
>
>ERROR: relation_info: Relation 12345 not found
>
>and more like
>
>ERROR: Failed to find relation OID 12345 in system catalogs
>Error-code: 4242
>Reporting-routine: relation_info, plancat.c line 543
>
>of which only the first line is really meant for the user.

This might even allow the client app to write out a customized error 
message, instead of 'foreign key ... violation' write 'You cannot delete 
any ... when there are still ...', based upon error codes.

>Of course, making that happen will be a lot of work, and I'm not
>asking you to volunteer for it.  But what you do now should fit
>in with further development of the error handling stuff...



Jeroen


Thu, 24 Feb 2000 09:35:19 +0100
Jeroen van Vianen <jeroen@design.nl>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse
	error messages
On Wed, 23 Feb 2000, Tom Lane wrote:

> Peter Eisentraut <e99re41@DoCS.UU.SE> writes:
> > I challenge everyone to show me a make without VPATH. In fact, show me two
> > makes without a feature that you can't live without, and I shall forever
> > hold my peace.
> 
> Out of the four systems I have easy access to: HPUX 10, HPUX 9, Linux
> (some fairly old RedHat version), and SunOS 4.1.4, two have makes
> without VPATH

You win. ;)

I surveyed several machines as well (Solaris, IRIX, FreeBSD, HPUX) which
all had this feature. I feel better now with actual data points, I hope
that's fair enough.

-- 
Peter Eisentraut                  Sernanders vaeg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Thu, 24 Feb 2000 13:51:40 +0100 (MET)
Peter Eisentraut <e99re41@DoCS.UU.SE>
Re: GNU make (Re: [HACKERS] Re: [PATCHES] Patch for more readable
	parse error messages)
On 2000-02-24, Jeroen van Vianen mentioned:

> At 11:12 22-02-00 -0500, Tom Lane wrote:

> >I'd be inclined to say that the location info should be imbedded as
> >text in the existing textual error message, rather than trying to add
> >a separate message with a machine-readable location value.

> Isn't it possible to get this kind of information from a call to a new API 
> struct errorinfo *  PQerrorInfo(conn) where the struct contains info about 
> the error message, location and code, rather than a call to 
> PQerrorMessage(conn) ?

IMHO, the use of error messages in PostgreSQL has a big conceptual
problem. It's only too tempting to write elog(ERROR, "I don't know what to
do now.") anywhere and any time. This is very convenient for the
developers but not very nice for client applications that want to
recognize, categorize, and recover from errors. There isn't even a clean
separation of perfectly normal user-level errors ("referential integrity
violation") and internal errors (bugs) ("can't attach node 718 to
T_ParseNodeFoo"). Sure, there's FATAL, but it's not always appropriate.

Chapter 22 of SQL92 defines error codes ("SQLSTATE") for (presumably)
every condition that could come up. It has classes and subclasses and
its code space is extensible. It would be very nice if we could classify
error messages in the backend according to that list and, say, do an

	error(PGE_TRIGGERED_DATA_CHANGE_VIOLATION);

instead. The frontend could then call PQsqlstate(connection) to get this
code, or it could call something equivalent to strerror that would convert
this code to a string (potentially language-dependent even). If someone
wants to communicate an internal yet non-fatal error, there would be a
special code reserved for it, telling the client application that it might
as well forget about it. Legacy applications could still call
PQerrorMessage which would interally call the above two.

A necessary extension to the above would be a way to pass along supportive 
data. The tricky part will be to figure out a syntax that is not too
cumbersome, not too restrictive, and encourages help by the compiler. For
example,

	error(PG_PARSE_ERROR, 2345)
	error(PG_PARSE_ERROR(2345))
	error(PG_PARSE_ERROR, errorIntData(2345))
	error(PG_INTERNAL, errorStrData("I'm way lost"))

or something hopefully much better. If error() is made a macro, then we
could include file and line number and have some libpq accessor function
for them. Somehow, the client should also be able to access the "2345"
directly.

In any case, I believe that the actual error message string should be
assembled in the front-end. I'm not too fond of the idea of letting
clients parse out the interesting parts of an error out of a blob of text.

Comments? Anyone interested? This would be very dear to my heart so I'd be
very willing to spend a lot of time on it.

-- 
Peter Eisentraut                  Sernanders vï¿½g 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden


Fri, 25 Feb 2000 00:38:47 +0100 (CET)
Peter Eisentraut <peter_e@gmx.net>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse  error
	messages
> In any case, I believe that the actual error message string should be
> assembled in the front-end. I'm not too fond of the idea of letting
> clients parse out the interesting parts of an error out of a blob of text.
> 
> Comments? Anyone interested? This would be very dear to my heart so I'd be
> very willing to spend a lot of time on it.

Vadim strongly believes in error mesage numbers.  We certainly should do
better, if only to print a code before the error code or something.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Thu, 24 Feb 2000 18:50:24 -0500 (EST)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse error
	messages
Peter Eisentraut <peter_e@gmx.net> writes:
> IMHO, the use of error messages in PostgreSQL has a big conceptual
> problem. It's only too tempting to write elog(ERROR, "I don't know what to
> do now.") anywhere and any time. This is very convenient for the
> developers but not very nice for client applications that want to
> recognize, categorize, and recover from errors.

The vast majority of the one-off error messages are internal consistency
checks.  It seems to me that a workable compromise is to insist on
standardized error codes/texts for reporting user mistakes, but to
continue to allow spur-of-the-moment messages for internal errors.
Most or all internal errors would have the same classification anyway
from the point of view of an application trying to decide what to do,
so they could all share one or a few "error ID numbers".

> A necessary extension to the above would be a way to pass along supportive 
> data. The tricky part will be to figure out a syntax that is not too
> cumbersome, not too restrictive, and encourages help by the compiler.

A printf/elog-like syntax should still work --- the message catalog that
PGE_TRIGGERED_DATA_CHANGE_VIOLATION indexes into would contain strings
that still have %-escapes, but that shouldn't make life any more
difficult for internationalization.  And we do have the opportunity
to check mistakes with gcc, if we stick to the standard printf escapes.

Or do we?  Hmm ... not if the error message text isn't available at
the call site ...  Here's a thought: suppose that error code macros like
PGE_TRIGGERED_DATA_CHANGE_VIOLATION normally expand to an error code
number, which eventually gets used as an index into a localizable table
of error format strings; but we have the option to run with header files
that define all these macros as the actual error message literal
strings.  Then gcc could check for parameter mismatch in that case.
For development work that might even be the normal thing, and only
in production scenarios would you introduce the extra level of
indirection to get to an error message string.

> In any case, I believe that the actual error message string should be
> assembled in the front-end.

That will not work, because the set of possible error messages will
undoubtedly change with every backend release, and we do *not* want
to get into a situation where out-of-date clients mean you get no
error message (or worse, a wrong error message).  It will be better
to have the message table on the backend side.  As long as the backend
ships an identifying code number along with the message text, I think
that will satisfy the needs of applications to avoid reverse-parsing
error messages.

Other than that, I agree with everything you say ;-)

> Comments? Anyone interested? This would be very dear to my heart so I'd be
> very willing to spend a lot of time on it.

It will take a lot of time to clean this up, but I think everyone agrees
we need to do it.  It's just been a matter of someone taking on the job.

			regards, tom lane

Thu, 24 Feb 2000 19:17:36 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse error
	messages
At 06:50 PM 2/24/00 -0500, Bruce Momjian wrote:
>> In any case, I believe that the actual error message string should be
>> assembled in the front-end. I'm not too fond of the idea of letting
>> clients parse out the interesting parts of an error out of a blob of text.
>> 
>> Comments? Anyone interested? This would be very dear to my heart so I'd be
>> very willing to spend a lot of time on it.
>
>Vadim strongly believes in error mesage numbers.  We certainly should do
>better, if only to print a code before the error code or something.

I do, too.  Anyone else with a language implementation background is likely
to share that bias.

For starters ... you can at least imagine doing things like provide error
messages in languages other than English.  Actually...Vadim could probably
force the issue by commiting a version with all the error messages in
Russian!   Hmmm...wonder if he's thought of that? :)

And for applications it often makes a lot more sense to just get a 
defined code.

When I improved on the AOLserver driver for Postgres, one of my goals
was to have it survive the closing of a backend.  This gets less 
crucial with each bug fix, but, heck ... the backend still pees its
pants and crashes occasionally, let's face it.  In this case, the
driver wants to reestablish the connection to the backend (because
it's being managed as part of a persistent pool of connections by
the web server) but return an error.

Afterwards, all other backends close themselves and pass back a
delightfully wordy message that one should retry their query because
it didn't really crash, but rather is closing just in case shared
memory has been corrupted by the very naughty backend that really
did crash.  In this case, the driver wants to reconnect and 
retry the query, and if it succeeds return normally, with the
web server none the wiser.

(works great, BTW)

There's no documented way to distinguish between the two kinds of
backend closures that I could find.  Interpreting the string in
general seems to be how one is expected to probe to determine exactly
what has happened, not only in this case but with other errors, too.

This sucks, IMO.

It turns out there's a trivial way to distinguish these two particular
cases I mention, without resorting to looking at the actual error message,
but I think it illustrates the general kludginess of returning strings
with no error code.




- Don Baccus, Portland OR <dhogaza@pacifier.com>
  Nature photos, on-line guides, Pacific Northwest
  Rare Bird Alert Service and other goodies at
  http://donb.photo.net.

Thu, 24 Feb 2000 16:20:43 -0800
Don Baccus <dhogaza@pacifier.com>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse
	error messages
At 07:17 PM 2/24/00 -0500, Tom Lane wrote:

>The vast majority of the one-off error messages are internal consistency
>checks.  It seems to me that a workable compromise is to insist on
>standardized error codes/texts for reporting user mistakes, but to
>continue to allow spur-of-the-moment messages for internal errors.
>Most or all internal errors would have the same classification anyway
>from the point of view of an application trying to decide what to do,
>so they could all share one or a few "error ID numbers".

I have no problem with this.  Why not just prepend them with an "internal"
error code?  Clients can't really do much other than gasp "omigosh!" when
confronted with an internal error anyway...

>Or do we?  Hmm ... not if the error message text isn't available at
>the call site ...  Here's a thought: suppose that error code macros like
>PGE_TRIGGERED_DATA_CHANGE_VIOLATION normally expand to an error code
>number, which eventually gets used as an index into a localizable table
>of error format strings; but we have the option to run with header files
>that define all these macros as the actual error message literal
>strings.  Then gcc could check for parameter mismatch in that case.
>For development work that might even be the normal thing, and only
>in production scenarios would you introduce the extra level of
>indirection to get to an error message string.

Something like this sounds like a fine.

>> In any case, I believe that the actual error message string should be
>> assembled in the front-end.
>
>That will not work, because the set of possible error messages will
>undoubtedly change with every backend release, and we do *not* want
>to get into a situation where out-of-date clients mean you get no
>error message (or worse, a wrong error message).  It will be better
>to have the message table on the backend side.

Yes, this is where it belongs.  An application gets an error number,
then asks for a message to go with it if it wants one.  Or, the
error's returned as an error code and message, either way.  

>  As long as the backend
>ships an identifying code number along with the message text, I think
>that will satisfy the needs of applications to avoid reverse-parsing
>error messages.

Yep.

>
>Other than that, I agree with everything you say ;-)
>
>> Comments? Anyone interested? This would be very dear to my heart so I'd be
>> very willing to spend a lot of time on it.
>
>It will take a lot of time to clean this up, but I think everyone agrees
>we need to do it.  It's just been a matter of someone taking on the job.

Go, Peter!



- Don Baccus, Portland OR <dhogaza@pacifier.com>
  Nature photos, on-line guides, Pacific Northwest
  Rare Bird Alert Service and other goodies at
  http://donb.photo.net.

Thu, 24 Feb 2000 16:33:41 -0800
Don Baccus <dhogaza@pacifier.com>
Re: [HACKERS] Re: [PATCHES] Patch for more readable parse
	error messages
Here is more information about it.

> Jeroen van Vianen <jeroen@design.nl> writes:
> >> Does this work with a non-bison parser?  It looks mighty
> >> bison-dependent to me...
> 
> > I'm not sure, but it probably is flex dependent (but Postgres always needed 
> > flex anyway). I'm not aware of any yacc / byacc / bison dependencies. Don't 
> > know if anybody has been successful building Postgres with another parser 
> > generator.
> 
> Um, you're right of course --- those are lexer not parser datastructures
> you're poking into.  Sorry for my confusion.
> 
> We do in fact work with non-bison parser generators, or did last time
> I tried it (around 6.5 release).  I would not like us to stop working
> with non-bison yaccs, since bison's output depends on alloca() which
> is not available everywhere.
> 
> I'm not sure about the situation with lexers.  We have been saying for
> a long time that flex was required, but since we got rid of the
> scanner's use of trailing context ('/' rules) I think there is a better
> chance that it would work with vanilla lex.  Anyone want to try that
> with current sources?
> 
> > BTW, as we ship flex's output lex.yy.c (as scan.c) and bison's output
> > (gram.c) in the distribution, any user would be able to compile the
> > sources, but if they want to start hacking the .l or .y files, they'll
> > need appropriate tools.
> 
> Right.  I am not aware of any portability problems with flex's output
> as there are with bison's, so it may be that the concern is moot.
> We may just be able to say "use the prebuilt scan.c or get flex; we
> don't care about supporting vendor lexes anymore".
> 
> I do see a potential problem with this patch that's not related to
> portability questions; it is that you're assuming that the lexer's
> furthest penetration into the source text is a good place to point
> at for parser errors.  That may not be true always.  In particular,
> I've been advocating solving some other problems by inserting a
> one-token lookahead buffer between the parser and the lexer.  If that
> happens then you'd be off by (at least) one token in some cases.
> 
> I think the way that this sort of thing is customarily handled in
> "real" compilers is that each token carries along an indication of
> just where it was found in the source, and then error messages can
> finger the right place without making assumptions about synchronization
> between different phases of the scanning/parsing process.  That might
> be more work than we can justify for SQL queries; not sure.
> 
> BTW, I think that the immediate problem of generating a good error
> message for unterminated comments and literals could be solved in other
> ways.  This patch or something like it might be cool anyway, but you
> should bear in mind that printing out a query and then a marker that's
> supposed to line up with something in the query doesn't always work
> all that well.  Consider a query that's several dozen lines long,
> such as a large table definition.  If we had more control over the
> user interface and could highlight the offending token directly,
> I'd be more excited about doing something like this.  (Actually, you
> could partially address that problem by only printing one line's worth
> of query text leading up to the error marker point.  It would still be
> tricky to get it right in the presence of newlines, tabs, etc.)
> 
> 			regards, tom lane
> 
> ************
> 


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Fri, 9 Jun 2000 08:40:21 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: [HACKERS] Re: Patch for more readable parse error messages