
> ISTM aio_write only improves the picture if there's some magic in-kernel
> processing that makes this same kind of judgment as to when to issue the
> "ganged" write for real, and is able to do it on time because it's in
> the kernel.  I haven't heard anything to make me think that that feature
> actually exists.  AFAIK the kernel isn't much more enlightened about
> physical head positions than we are.

Can the magic be, that kaio directly writes from user space memory to the 
disk ? Since in your case all transactions A-E want the same buffer written,
the memory (not it's content) will also be the same. This would automatically 
write the latest possible version of our WAL buffer to disk. 

The problem I can see offhand is how the kaio system can tell which transaction
can be safely notified of the write, or whether the programmer is actually responsible
for not changing the buffer until notified of completion ?

Andreas

Tue, 8 Oct 2002 11:15:41 +0200
"Zeugswetter Andreas SB SD" <ZeugswetterA@spardat.at>
Re: Analysis of ganged WAL writes
On Tue, 2002-10-08 at 04:15, Zeugswetter Andreas SB SD wrote:
> Can the magic be, that kaio directly writes from user space memory to the 
> disk ? Since in your case all transactions A-E want the same buffer written,
> the memory (not it's content) will also be the same. This would automatically 
> write the latest possible version of our WAL buffer to disk. 
> 

*Some* implementations allow for zero-copy aio.  That is a savings.  On
heavily used systems, it can be a large savings.

> The problem I can see offhand is how the kaio system can tell which transaction
> can be safely notified of the write, or whether the programmer is actually responsible
> for not changing the buffer until notified of completion ?

That's correct.  The programmer can not change the buffer contents until
notification has completed for that outstanding aio operation.  To do
otherwise results in undefined behavior.  Since some systems do allow
for zero-copy aio operations, requiring the buffers not be modified,
once queued, make a lot of sense.  Of course, even on systems that don't
support zero-copy, changing the buffered data prior to write completion
just seems like a bad idea to me.

Here's a quote from SGI's aio_write man page:
If the buffer pointed to by aiocbp->aio_buf or the control block pointed
to by aiocbp changes or becomes an illegal address prior to asynchronous
I/O completion then the behavior is undefined.  Simultaneous synchronous
operations using the same aiocbp produce undefined results.

And on SunOS we have:
     The aiocbp argument points to an  aiocb  structure.  If  the
     buffer  pointed  to  by aiocbp->aio_buf or the control block
     pointed to by aiocbp becomes an  illegal  address  prior  to
     asynchronous I/O completion, then the behavior is undefined.
and
     For any system action that changes the process memory  space
     while  an  asynchronous  I/O  is  outstanding to the address
     range being changed, the result of that action is undefined.


Greg
08 Oct 2002 07:34:52 -0500
Greg Copeland <greg@CopelandConsulting.Net>
Re: Analysis of ganged WAL writes
"Zeugswetter Andreas SB SD" <ZeugswetterA@spardat.at> writes:
> Can the magic be, that kaio directly writes from user space memory to the 
> disk ?

This makes more assumptions about the disk drive's behavior than I think
are justified...

> Since in your case all transactions A-E want the same buffer written,
> the memory (not it's content) will also be the same.

But no, it won't: the successive writes will ask to write different
snapshots of the same buffer.

> The problem I can see offhand is how the kaio system can tell which
> transaction can be safely notified of the write,

Yup, exactly.  Whose snapshot made it down to (stable) disk storage?

			regards, tom lane

Tue, 08 Oct 2002 09:45:35 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Analysis of ganged WAL writes 
> > Since in your case all transactions A-E want the same buffer written,
> > the memory (not it's content) will also be the same.
>
> But no, it won't: the successive writes will ask to write different
> snapshots of the same buffer.

Successive writes would write different NON-OVERLAPPING sections of the
same log buffer. It wouldn't make sense to send three separate copies of
the entire block. That could indeed cause problems.

If a separate log writing process was doing all the writing, it could
pre-gang the writes. However, I'm not sure this is necessary. I'll test the
simpler way first.

> > The problem I can see offhand is how the kaio system can tell which
> > transaction can be safely notified of the write,
>
> Yup, exactly.  Whose snapshot made it down to (stable) disk storage?

If you do as above, it can inform the transactions when the blocks get
written to disk since there are no inconsistent writes. If transactions A,
B and C had commits in block 1023, and the aio system writes that block to
the disk, it can notify all three that their transaction write is complete
when that block (or partial block) is written to disk.

It transaction C's write didn't make it into the buffer, I've got to assume
the aio system or the disk cache logic can handle realizing that it didn't
queue that write and therefore not inform transaction C of a completion.

- Curtis


Tue, 8 Oct 2002 10:15:30 -0400
"Curtis Faith" <curtis@galtair.com>
Re: Analysis of ganged WAL writes 
"Curtis Faith" <curtis@galtair.com> writes:
> Successive writes would write different NON-OVERLAPPING sections of the
> same log buffer. It wouldn't make sense to send three separate copies of
> the entire block. That could indeed cause problems.

So you're going to undo the code's present property that all writes are
block-sized?  Aren't you worried about incurring page-in reads because
the kernel can't know that we don't care about data beyond what we've
written so far in the block?

			regards, tom lane

Tue, 08 Oct 2002 10:41:43 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Analysis of ganged WAL writes 
> "Curtis Faith" <curtis@galtair.com> writes:
> > Successive writes would write different NON-OVERLAPPING sections of the
> > same log buffer. It wouldn't make sense to send three separate
> copies of
> > the entire block. That could indeed cause problems.
>
> So you're going to undo the code's present property that all writes are
> block-sized?  Aren't you worried about incurring page-in reads because
> the kernel can't know that we don't care about data beyond what we've
> written so far in the block?

Yes, I'll try undoing the current behavior.

I'm not really worried about doing page-in reads because the disks internal
buffers should contain most of the blocks surrounding the end of the log
file. If the successive partial writes exceed a block (which they will in
heavy use) then most of the time this won't be a problem anyway since the
disk will gang the full blocks before writing.

If the inserts are not coming fast enough to fill the log then the disks
cache should contain the data from the last time that block (or the block
before) was written. Disks have become pretty good at this sort of thing
since writing sequentially is a very common scenario.

It may not work, but one doesn't make significant progress without trying
things that might not work.

If it doesn't work, then I'll make certain that commit log records always
fill the buffer they are written too, with variable length commit records
and something to identify the size of the padding used to fill the rest of
the block.


Tue, 8 Oct 2002 10:57:15 -0400
"Curtis Faith" <curtis@galtair.com>
Re: Analysis of ganged WAL writes 
"Curtis Faith" <curtis@galtair.com> writes:
> I'm not really worried about doing page-in reads because the disks internal
> buffers should contain most of the blocks surrounding the end of the log
> file. If the successive partial writes exceed a block (which they will in
> heavy use) then most of the time this won't be a problem anyway since the
> disk will gang the full blocks before writing.

You seem to be willing to make quite a large number of assumptions about
what the disk hardware will do or not do.  I trust you're going to test
your results on a wide range of hardware before claiming they have any
general validity ...

			regards, tom lane

Tue, 08 Oct 2002 11:08:08 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Analysis of ganged WAL writes 
> "Curtis Faith" <curtis@galtair.com> writes:
> > I'm not really worried about doing page-in reads because the
> disks internal
> > buffers should contain most of the blocks surrounding the end
> of the log
> > file. If the successive partial writes exceed a block (which
> they will in
> > heavy use) then most of the time this won't be a problem
> anyway since the
> > disk will gang the full blocks before writing.
>
> You seem to be willing to make quite a large number of assumptions about
> what the disk hardware will do or not do.  I trust you're going to test
> your results on a wide range of hardware before claiming they have any
> general validity ...

Tom, I'm actually an empiricist. I trust nothing that I haven't tested or
read the code for myself. I've found too many instances of bugs or poor
implementations in the O/S to believe without testing.

On the other hand, one has to make some assumptions in order to devise
useful tests.

I'm not necessarily expecting that I'll come up with something that will
help everyone all the time. I'm just hoping that I can come up with
something that will help those using modern hardware, most of the time.

Even if it works, this will probably become one of those flags that need to
be tested as part of the performance analysis for any given system. Or
perhaps ideally, I'll come up with a LogDiskTester that simulates log
output and determines the best settings to use for a given disk and O/S,
optimized for size or speed, heavy inserts, etc.


Tue, 8 Oct 2002 11:17:28 -0400
"Curtis Faith" <curtis@galtair.com>
Re: Analysis of ganged WAL writes 