
For those of you who have missed it, here

http://www.google.com/search?q=cache:web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf+clark+end+to+end&hl=en

is the paper some of us mention, "END-TO-END ARGUMENTS IN SYSTEM DESIGN"
by Saltzer, Reed, and Clark.

The abstract is:

    This paper presents a design principle that helps guide placement of
    functions among the modules of a distributed computer system. The
    principle, called the end-to-end argument, suggests that functions
    placed at low levels of a system may be redundant or of little value
    when compared with the cost of providing them at that low level.
    Examples discussed in the paper include bit error recovery, security
    using encryption, duplicate message suppression, recovery from
    system crashes, and delivery acknowledgement. Low level mechanisms
    to support these functions are justified only as performance
    enhancements.

It was written in 1981 and is undiminished by the subsequent decades.

Nathan Myers
ncm@zembu.com

Thu, 17 May 2001 00:24:28 -0700
ncm@zembu.com (Nathan Myers)
"End-to-end" paper
At 12:24 AM 17-05-2001 -0700, Nathan Myers wrote:
>
>For those of you who have missed it, here
>
>http://www.google.com/search?q=cache:web.mit.edu/Saltzer/www/publications/e
ndtoend/endtoend.pdf+clark+end+to+end&hl=en
>
>is the paper some of us mention, "END-TO-END ARGUMENTS IN SYSTEM DESIGN"
>by Saltzer, Reed, and Clark.
>
>The abstract is:
>
>    This paper presents a design principle that helps guide placement of
>    functions among the modules of a distributed computer system. The
>    principle, called the end-to-end argument, suggests that functions
>    placed at low levels of a system may be redundant or of little value
>    when compared with the cost of providing them at that low level.
>    Examples discussed in the paper include bit error recovery, security
>    using encryption, duplicate message suppression, recovery from
>    system crashes, and delivery acknowledgement. Low level mechanisms
>    to support these functions are justified only as performance
>    enhancements.
>
>It was written in 1981 and is undiminished by the subsequent decades.
>

Maybe I don't understand the paper.

The end-to-end argument might be true if taking the monolithic approach. I
find more useful ideas gleaned from the RFCs, TCP/IP and the OSI 7 layer
model: modularity, "useful standard interfaces", "Be liberal in what you
accept, and conservative in what you send" and so on.

Within a module I figure the end to end argument might hold, but the author
keeps talking about networks and networking.

SSL and TCP are useful. The various CRC checks down the IP stack to the
datalink layer have their uses too.

By splitting stuff up at appropriate points, adding or substituting objects
at various layers becomes so much easier. People can download Postgresql
over token ring, Gigabit ethernet, X.25 and so on.

Splitting stuff up does mean that the bits and pieces now do have a certain
responsibility. If those responsibilities involve some redundancies in
error checking or encryption or whatever, so be it, because if done well
people can use those bits and pieces in interesting ways never dreamed of
initially.

For example SSL over TCP over IPSEC over encrypted WAP works (even though
IPSEC is way too complicated :)). There's so much redundancy there, but at
the same time it's not a far fetched scenario - just someone ordering
online on a notebook pc.

But if a low level module never bothered with error
correction/detection/handling or whatever and was optimized for an
application specific purpose, it's harder to use it for other purposes. And
if you do, some chap could post an article to Bugtraq on it, mentioning
exploit, DoS or buffer overflow.

Cheerio,
Link.




Thu, 17 May 2001 18:04:54 +0800
Lincoln Yeoh <lyeoh@pop.jaring.my>
Re: "End-to-end" paper
On Thu, May 17, 2001 at 06:04:54PM +0800, Lincoln Yeoh wrote:
> At 12:24 AM 17-05-2001 -0700, Nathan Myers wrote:
> >
> >For those of you who have missed it, here
> >
> >http://www.google.com/search?q=cache:web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf+clark+end+to+end&hl=en
> >
> >is the paper some of us mention, "END-TO-END ARGUMENTS IN SYSTEM DESIGN"
> >by Saltzer, Reed, and Clark.
> >
> >The abstract is:
> >
> >    This paper presents a design principle that helps guide placement
> >    of functions among the modules of a distributed computer system.
> >    The principle, called the end-to-end argument, suggests that
> >    functions placed at low levels of a system may be redundant or
> >    of little value when compared with the cost of providing them
> >    at that low level. Examples discussed in the paper include
> >    bit error recovery, security using encryption, duplicate
> >    message suppression, recovery from system crashes, and delivery
> >    acknowledgement. Low level mechanisms to support these functions
> >    are justified only as performance enhancements.
> >
> >It was written in 1981 and is undiminished by the subsequent decades.
>
> Maybe I don't understand the paper.

Yes.  It bears re-reading.

> The end-to-end argument might be true if taking the monolithic
> approach. I find more useful ideas gleaned from the RFCs, TCP/IP and
> the OSI 7 layer model: modularity, "useful standard interfaces", "Be
> liberal in what you accept, and conservative in what you send" and so
> on.

The end-to-end principle has had profound effects on the design of 
Internet protocols, perhaps most importantly in keeping them simpler 
than OSI's.

> Within a module I figure the end to end argument might hold,

The end-to-end principle isn't particularly applicable within a module.
It's a system-design principle. Its prescription for individual modules
is: don't imagine that anybody else gets much value from your complex
error recovery shenanigans; they have to do their own error recovery
anyway. You provide more value by making a good effort.

> but the author keeps talking about networks and networking.

Of course networking is just an example, but it's a particularly
good example. Data storage (e.g. disk) is another good example; in
the context of the paper it may be thought of as a mechanism for
communicating with other (later) times. The point there is that the CRCs
and ECC performed by the disk are not sufficient to ensure reliability
for the system (e.g. database service); for that, end-to-end measures
such as hot-failover, backups, redo logs, and block- or record-level
CRCs are needed. The purpose of the disk CRCs is not reliability, a job
they cannot do alone, but performance: they help make the need to use
the backups and redo logs infrequent enough to be tolerable.

> SSL and TCP are useful. The various CRC checks down the IP stack to
> the datalink layer have their uses too.

Yes, of course they are useful. The authors say so in the paper, and
they say precisely how (and how not).

> By splitting stuff up at appropriate points, adding or substituting
> objects at various layers becomes so much easier. People can download
> Postgresql over token ring, Gigabit ethernet, X.25 and so on.

As noted in the paper, the principle is most useful in helping to decide
what goes in each layer.

> Splitting stuff up does mean that the bits and pieces now do have
> a certain responsibility. If those responsibilities involve some
> redundancies in error checking or encryption or whatever, so be
> it, because if done well people can use those bits and pieces in
> interesting ways never dreamed of initially.
>
> For example SSL over TCP over IPSEC over encrypted WAP works (even
> though IPSEC is way too complicated :)). There's so much redundancy
> there, but at the same time it's not a far fetched scenario - just
> someone ordering online on a notebook pc.

The authors quote a similar example in the paper, even though it was
written twenty years ago.

> But if a low level module never bothered with error
> correction/detection/handling or whatever and was optimized for
> an application specific purpose, it's harder to use it for other
> purposes. And if you do, some chap could post an article to Bugtraq on
> it, mentioning exploit, DoS or buffer overflow.

The point is that leaving that stuff _out_ is how you keep low-level
mechanisms useful for a variety of purposes. Putting in complicated
error-recovery stuff might suit it better for a particular application,
but make it less suitable for others.

This is why, at the IP layer, packets get tossed at the first sign of
congestion. It's why TCP connections often get dropped at the first sign
of a data-format violation. This is a very deep principle; understanding
it thoroughly will make you a much better system designer.

Nathan Myers
ncm@zembu.com

Thu, 17 May 2001 14:52:27 -0700
ncm@zembu.com (Nathan Myers)
Re: Re: "End-to-end" paper
On OBSD from cvs source,  clean checkout:

gcc -O2 -pipe -Wall -Wmissing-prototypes -Wmissing-declarations
-I../../../../src/include  -DLIBDIR=\"/usr/local/pgsql/lib\
" -DDLSUFFIX=\".so\"  -c -o dfmgr.o dfmgr.c
dfmgr.c: In function `load_external_function':
dfmgr.c:118: `RTLD_GLOBAL' undeclared (first use in this function)
dfmgr.c:118: (Each undeclared identifier is reported only once
dfmgr.c:118: for each function it appears in.)
gmake[4]: *** [dfmgr.o] Error 1
gmake[4]: Leaving directory
`/home/bpalmer/APPS/pgsql/src/backend/utils/fmgr'


??  RTLD_GLOBAL problems?

- b


b. palmer,  bpalmer@crimelabs.net
pgp:  www.crimelabs.net/bpalmer.pgp5


Sat, 19 May 2001 20:03:50 -0400 (EDT)
bpalmer <bpalmer@crimelabs.net>
cvs snapshot compile problems
This seems to have been broken for a few days ~monday night

- brandon


b. palmer,  bpalmer@crimelabs.net
pgp:  www.crimelabs.net/bpalmer.pgp5



Sat, 19 May 2001 20:05:28 -0400 (EDT)
bpalmer <bpalmer@crimelabs.net>
cvs snapshot compile problems
On Sat, May 19, 2001 at 08:03:50PM -0400, bpalmer wrote:
> On OBSD from cvs source,  clean checkout:
> 
> gcc -O2 -pipe -Wall -Wmissing-prototypes -Wmissing-declarations
> -I../../../../src/include  -DLIBDIR=\"/usr/local/pgsql/lib\
> " -DDLSUFFIX=\".so\"  -c -o dfmgr.o dfmgr.c
> dfmgr.c: In function `load_external_function':
> dfmgr.c:118: `RTLD_GLOBAL' undeclared (first use in this function)
> dfmgr.c:118: (Each undeclared identifier is reported only once
> dfmgr.c:118: for each function it appears in.)
> gmake[4]: *** [dfmgr.o] Error 1
> gmake[4]: Leaving directory
> `/home/bpalmer/APPS/pgsql/src/backend/utils/fmgr'
> 
> 
> ??  RTLD_GLOBAL problems?

Not a solution, but a few data points: I had a successful build from cvs of
May 19 16:21 GMT under NetBSD/i386, and for me RTLD_GLOBAL is defined in
/usr/include/dlfcn.h ie., system header file, not postgresql.

Hope that helps,

Patrick

Mon, 21 May 2001 12:08:17 +0100
Patrick Welche <prlw1@newn.cam.ac.uk>
Re: cvs snapshot compile problems

This was added by Peter E to allow PL/Perl to compile.

> On Sat, May 19, 2001 at 08:03:50PM -0400, bpalmer wrote:
> > On OBSD from cvs source,  clean checkout:
> > 
> > gcc -O2 -pipe -Wall -Wmissing-prototypes -Wmissing-declarations
> > -I../../../../src/include  -DLIBDIR=\"/usr/local/pgsql/lib\
> > " -DDLSUFFIX=\".so\"  -c -o dfmgr.o dfmgr.c
> > dfmgr.c: In function `load_external_function':
> > dfmgr.c:118: `RTLD_GLOBAL' undeclared (first use in this function)
> > dfmgr.c:118: (Each undeclared identifier is reported only once
> > dfmgr.c:118: for each function it appears in.)
> > gmake[4]: *** [dfmgr.o] Error 1
> > gmake[4]: Leaving directory
> > `/home/bpalmer/APPS/pgsql/src/backend/utils/fmgr'
> > 
> > 
> > ??  RTLD_GLOBAL problems?
> 
> Not a solution, but a few data points: I had a successful build from cvs of
> May 19 16:21 GMT under NetBSD/i386, and for me RTLD_GLOBAL is defined in
> /usr/include/dlfcn.h ie., system header file, not postgresql.
> 
> Hope that helps,
> 
> Patrick
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tue, 22 May 2001 00:09:37 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: cvs snapshot compile problems