HI,

I'm using 7.4.5 on Mac OS X (G5) and was profiling it to see why it is 
SO SLOW at committing  inserts and deletes into a large database. One 
of the many slowdowns was from MemSet. I found an old (2002) thread 
about this and retried the tests  (see below). The main point is that 
the system memset crushes pg's!! Is it possible to add a define to call 
the system memset at build time! This probably isn't the case on other 
systems.

I wanted to know the size of FunctionCallInfoData (in execQual.c) 
because the profiler said that if it was over 128 then use the system 
call.

Here are my results:

pgMemSet
* 64
0.410u 0.000s 0:00.42 97.6%     0+0k 0+0io 0pf+0w
* 128
0.600u 0.000s 0:00.61 98.3%     0+0k 0+0io 0pf+0w
* 176 Size of fcinfo is 176, used in execQual.c which was being very 
slow here!
0.790u 0.000s 0:00.79 100.0%    0+0k 0+0io 0pf+0w
* 256
1.040u 0.000s 0:01.08 96.2%     0+0k 0+0io 0pf+0w
* 512
2.030u 0.000s 0:02.04 99.5%     0+0k 0+0io 0pf+0w
* 1024
3.950u 0.010s 0:03.94 100.5%    0+0k 0+0io 0pf+0w
* 2048
7.710u 0.000s 0:07.75 99.4%     0+0k 0+0io 0pf+0w
* 4096
15.390u 0.000s 0:15.37 100.1%   0+0k 0+0io 0pf+0w

system memset
* 64
0.260u 0.000s 0:00.25 104.0%    0+0k 0+0io 0pf+0w
* 128
0.310u 0.000s 0:00.31 100.0%    0+0k 0+0io 0pf+0w
* 176 Size of fcinfo is 176
0.300u 0.010s 0:00.30 103.3%    0+0k 0+0io 0pf+0w
* 256
0.310u 0.000s 0:00.30 103.3%    0+0k 0+0io 0pf+0w
* 512
0.350u 0.000s 0:00.33 106.0%    0+0k 0+0io 0pf+0w
* 1024
0.590u 0.010s 0:00.63 95.2%     0+0k 0+0io 0pf+0w
* 2048
0.780u 0.000s 0:00.77 101.2%    0+0k 0+0io 0pf+0w
* 4096
1.320u 0.000s 0:01.33 99.2%     0+0k 0+0io 0pf+0w


  #include <string.h>
  #include "postgres.h"
  #include "fmgr.h"

  #undef MEMSET_LOOP_LIMIT
  #define MEMSET_LOOP_LIMIT  1000000

  int
  main(int argc, char **argv)
  {
   int  len = atoi(argv[1]);
   char  buffer[len];
   long long i;

     FunctionCallInfoData fcinfo;
     printf("Size of fcinfo is %d\n", sizeof(fcinfo));

   for (i = 0; i < 9900000; i++)
    MemSet(buffer, 0, len);
    //memset(buffer, 0, len);
   return 0;
  }



Tue, 14 Sep 2004 09:46:55 -0400
Marc Colosimo <mcolosimo@mitre.org>
tweaking MemSet() performance - 7.4.5
Marc Colosimo <mcolosimo@mitre.org> writes:
> I'm using 7.4.5 on Mac OS X (G5) and was profiling it to see why it is 
> SO SLOW at committing  inserts and deletes into a large database. One 
> of the many slowdowns was from MemSet. I found an old (2002) thread 
> about this and retried the tests  (see below). The main point is that 
> the system memset crushes pg's!!

Hmm.  I tried to duplicate this on my G4 laptop, and found that they
were more or less on a par for small-to-middling block sizes (using
"gcc -O2").  Darwin's memset code must have some additional tweaks for
use on G5 hardware.  Good for Apple --- this is the sort of thing that
OS vendors *ought* to be doing.  The fact that we can beat the system
memset on so many platforms is an indictment of those platforms.

> Is it possible to add a define to call 
> the system memset at build time! This probably isn't the case on other 
> systems.

Feel free to hack the definition of MemSet in src/include/c.h.  See the
comments for it for more context.

Note that for small compile-time-constant block sizes (a case your test
program doesn't test, but it's common in pgsql), gcc with a sufficiently
high optimization setting can unroll the loop into a linear sequence of
words zeroings.  I would expect that to beat the system memset up to a
few dozen words, no matter how tense the memset coding is.  So you
probably want to think in terms of reducing MEMSET_LOOP_LIMIT rather
than diking out the macro code altogether.  Or maybe reduce MemSet to
"memset(...)" but leave MemSetAligned and/or MemSetTest/MemSetLoop
as-is.  In any case, reporting results without mentioning the compiler
and optimization level in use isn't going to convince anybody ...

			regards, tom lane

Fri, 17 Sep 2004 15:55:00 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: tweaking MemSet() performance - 7.4.5 

On Sep 17, 2004, at 3:55 PM, Tom Lane wrote:

> Marc Colosimo <mcolosimo@mitre.org> writes:
>> I'm using 7.4.5 on Mac OS X (G5) and was profiling it to see why it is
>> SO SLOW at committing  inserts and deletes into a large database. One
>> of the many slowdowns was from MemSet. I found an old (2002) thread
>> about this and retried the tests  (see below). The main point is that
>> the system memset crushes pg's!!
>
> Hmm.  I tried to duplicate this on my G4 laptop, and found that they
> were more or less on a par for small-to-middling block sizes (using
> "gcc -O2").  Darwin's memset code must have some additional tweaks for
> use on G5 hardware.  Good for Apple --- this is the sort of thing that
> OS vendors *ought* to be doing.  The fact that we can beat the system
> memset on so many platforms is an indictment of those platforms.
>
>> Is it possible to add a define to call
>> the system memset at build time! This probably isn't the case on other
>> systems.
>
> Feel free to hack the definition of MemSet in src/include/c.h.  See the
> comments for it for more context.
>
> Note that for small compile-time-constant block sizes (a case your test
> program doesn't test, but it's common in pgsql), gcc with a 
> sufficiently
> high optimization setting can unroll the loop into a linear sequence of
> words zeroings.  I would expect that to beat the system memset up to a
> few dozen words, no matter how tense the memset coding is.  So you
> probably want to think in terms of reducing MEMSET_LOOP_LIMIT rather
> than diking out the macro code altogether.  Or maybe reduce MemSet to
> "memset(...)" but leave MemSetAligned and/or MemSetTest/MemSetLoop
> as-is.  In any case, reporting results without mentioning the compiler
> and optimization level in use isn't going to convince anybody ...
>

Oops, I used the same setting as in the old hacking message (-O2, gcc 
3.3). If I understand what you are saying, then it turns out yes, PG's 
MemSet is faster for smaller blocksizes (see below, between 32 and 64). 
I just replaced the whole MemSet with memset and it is not very low 
when I profile. I could squeeze more out of it if I spent more time 
trying to understand it (change MEMSET_LOOP_LIMIT to 32 and then add 
memset after that?). I'm now working one understanding  Spin Locks and 
friends. Putting in a sync call (in s_lock.h) is really a time killer 
and bad for performance (it takes up 35 cycles).

run on a single processor G5 (1.8Gz, other was on a DP 2Gz G5)
pgMemSet:
*  4
0.070u 0.000s 0:00.15 46.6%     0+0k 0+0io 0pf+0w
* 8
0.090u 0.000s 0:00.16 56.2%     0+0k 0+0io 0pf+0w
* 16
0.120u 0.000s 0:00.17 70.5%     0+0k 0+0io 0pf+0w
* 32
0.180u 0.000s 0:00.29 62.0%     0+0k 0+0io 0pf+0w
* 64
0.450u 0.000s 0:00.92 48.9%     0+0k 0+0io 0pf+0w


memset:
* 4
0.170u 0.010s 0:00.44 40.9%     0+0k 0+0io 0pf+0w
* 8
0.190u 0.000s 0:00.42 45.2%     0+0k 0+0io 0pf+0w
* 16
0.190u 0.010s 0:00.39 51.2%     0+0k 0+0io 0pf+0w
* 32
0.200u 0.000s 0:00.39 51.2%     0+0k 0+0io 0pf+0w
* 64
0.260u 0.000s 0:00.38 68.4%     0+0k 0+0io 0pf+0w


Marc



Fri, 17 Sep 2004 16:26:47 -0400
Marc Colosimo <mcolosimo@mitre.org>
Re: tweaking MemSet() performance - 7.4.5
Marc Colosimo wrote:

> Oops, I used the same setting as in the old hacking message (-O2, gcc 
> 3.3). If I understand what you are saying, then it turns out yes, PG's 
> MemSet is faster for smaller blocksizes (see below, between 32 and 
> 64). I just replaced the whole MemSet with memset and it is not very 
> low when I profile.

Could you check what the OS-X memset function does internally?
One trick to speed up memset it to bypass the cache and bulk-write 
directly from write buffers to main memory. i386 cpus support that and 
in microbenchmarks it's 3 times faster (or something like that). 
Unfortunately it's a loss in real-world tests: Typically a structure is 
initialized with memset and then immediately accessed. If the memset 
bypasses the cache then the following access will cause a cache line 
miss, which can be so slow that using the faster memset can result in a 
net performance loss.

> I could squeeze more out of it if I spent more time trying to 
> understand it (change MEMSET_LOOP_LIMIT to 32 and then add memset 
> after that?). I'm now working one understanding  Spin Locks and 
> friends. Putting in a sync call (in s_lock.h) is really a time killer 
> and bad for performance (it takes up 35 cycles).
>
That's the price you pay for weakly ordered memory access.
Linux on ppc uses eieio, on ppc64 lwsync is used. Could you check if 
they are faster?

--
    Manfred

Sat, 18 Sep 2004 17:38:36 +0200
Manfred Spraul <manfred@colorfullife.com>
Re: tweaking MemSet() performance - 7.4.5
Manfred Spraul <manfred@colorfullife.com> writes:
> That's the price you pay for weakly ordered memory access.
> Linux on ppc uses eieio, on ppc64 lwsync is used. Could you check if 
> they are faster?

I recall looking at lwsync and being concerned about portability
problems --- older assemblers will fail to recognize it.  I'd want
to see some hard evidence that changing sync to lwsync would be a
significant performance win before taking any portability risk here.

			regards, tom lane

Sat, 18 Sep 2004 13:01:42 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: tweaking MemSet() performance - 7.4.5 