
> Per Tom's request(1000 concurrent backends), I tried current on IBM
> AIX 5L and found that make check hungs:
> 
> parallel group (13 tests): float4 oid varchar
> 
> pgbench hungs too if more than 4 or so concurrent backends are
> involved.

I once had hangs during make check on AIX 4, but after make distclean
and 
rebuild was never able to reproduce.

Can you read the man page for cs(3), AIX 4 sais it is not recommended
suggests to use compare_and_swap, maybe AIX 5 has more to say ?

> Unfortunately gdb does not work well on AIX, so I'm stucked.
> Maybe a new locking code?

Use dbx (and ddd) ?

I don't have access to AIX 5.

Andreas

Mon, 1 Oct 2001 14:33:43 +0200
"Zeugswetter Andreas SB SD" <ZeugswetterA@spardat.at>
Re: Problem on AIX with current
> > Per Tom's request(1000 concurrent backends), I tried current on IBM
> > AIX 5L and found that make check hungs:
> > 
> > parallel group (13 tests): float4 oid varchar
> > 
> > pgbench hungs too if more than 4 or so concurrent backends are
> > involved.
> 
> I once had hangs during make check on AIX 4, but after make distclean
> and 
> rebuild was never able to reproduce.

I thing I did make distclean.

> Can you read the man page for cs(3), AIX 4 sais it is not recommended
> suggests to use compare_and_swap, maybe AIX 5 has more to say ?

    Note&#58; The cs subroutine is only provided to support binary
    compatibility with AIX Version 3 applications&#46; When writing new
    applications, it is not recommended to use this subroutine; it may cause
    reduced performance in the future&#46; Applications should use the
    compare_and_swap (compare_and_swap Subroutine) subroutine, unless they
    need to use unaligned memory locations&#46;

Seems same as AIX 4?

> > Unfortunately gdb does not work well on AIX, so I'm stucked.
> > Maybe a new locking code?
> 
> Use dbx (and ddd) ?

Here is a stack trace using dbx.

semop(??, ??, ??) at 0xd02be73c
IpcSemaphoreLock(??, ??, ??), line 425 in "ipc.c"
LWLockAcquire(??, ??), line 270 in "lwlock.c"
LockAcquire(??, ??, ??, ??, ??), line 482 in "lock.c"
LockRelation(??, ??), line 153 in "lmgr.c"
heap_openr(??, ??), line 512 in "heapam.c"
scan_pg_rel_ind(??, ??), line 380 in "relcache.c"
ScanPgRelation(??, ??), line 307 in "relcache.c"
IndexedAccessMethodInitialize(??, ??, ??), line 994 in "relcache.c"
RelationNameGetRelation(??), line 1484 in "relcache.c"
heap_openr(??, ??), line 502 in "heapam.c"
setTargetTable(??, ??, ??, ??), line 136 in "parse_clause.c"
transformUpdateStmt(??, ??), line 2416 in "analyze.c"
transformStmt(??, ??), line 228 in "analyze.c"
parse_analyze(??, ??), line 92 in "analyze.c"
pg_analyze_and_rewrite(??), line 428 in "postgres.c"
unnamed block $b1877, line 740 in "postgres.c"
unnamed block $b1876, line 740 in "postgres.c"
unnamed block $b1872, line 740 in "postgres.c"
pg_exec_query_string(??, ??, ??), line 740 in "postgres.c"
PostgresMain(??, ??, ??, ??, ??), line 1943 in "postgres.c"
DoBackend(??), line 2104 in "postmaster.c"
BackendStartup(??), line 1837 in "postmaster.c"
unnamed block $b1665, line 917 in "postmaster.c"
ServerLoop(), line 917 in "postmaster.c"
PostmasterMain(??, ??), line 712 in "postmaster.c"
main(argc = 0, argv = (nil)), line 178 in "main.c"

Tue, 02 Oct 2001 00:42:42 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Problem on AIX with current
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
>> Can you read the man page for cs(3), AIX 4 sais it is not recommended
>> suggests to use compare_and_swap, maybe AIX 5 has more to say ?

>     Note&#58; The cs subroutine is only provided to support binary
>     compatibility with AIX Version 3 applications&#46; When writing new
>     applications, it is not recommended to use this subroutine; it may cause
>     reduced performance in the future&#46; Applications should use the
>     compare_and_swap (compare_and_swap Subroutine) subroutine, unless they
>     need to use unaligned memory locations&#46;

> Seems same as AIX 4?

Hmm, does anyone want to produce new s_lock code for AIX that uses
compare_and_swap?  But I'm not sure that's the problem here.

> Here is a stack trace using dbx.

> semop(??, ??, ??) at 0xd02be73c
> IpcSemaphoreLock(??, ??, ??), line 425 in "ipc.c"
> LWLockAcquire(??, ??), line 270 in "lwlock.c"
> LockAcquire(??, ??, ??, ??, ??), line 482 in "lock.c"

This process is waiting to acquire the LockMgr lock.  You need to look
at the rest of the processes and try to figure out who's got the lock.

			regards, tom lane

Mon, 01 Oct 2001 12:22:33 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Problem on AIX with current 
> > Here is a stack trace using dbx.
> 
> > semop(??, ??, ??) at 0xd02be73c
> > IpcSemaphoreLock(??, ??, ??), line 425 in "ipc.c"
> > LWLockAcquire(??, ??), line 270 in "lwlock.c"
> > LockAcquire(??, ??, ??, ??, ??), line 482 in "lock.c"
> 
> This process is waiting to acquire the LockMgr lock.  You need to look
> at the rest of the processes and try to figure out who's got the lock.

Strange enough, there's no other backend (of course except stats
collectors) here. I make sure this with ps and pg_stat_activity view.

BTW pg_stat_activity view shows:

16556 | test    |  197378 |        1 | postgres | update accounts set abalance = abalance + 406, filler = 'added amount to abalance is 406' where aid = 1447
--
Tatsuo Ishii

Tue, 02 Oct 2001 10:09:39 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Problem on AIX with current 
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> Strange enough, there's no other backend (of course except stats
> collectors) here. I make sure this with ps and pg_stat_activity view.

If you have no better way of determining what's going on, it might help
to recompile with LOCK_DEBUG defined, then enable trace_lwlocks in
postgresql.conf (better turn on debug_print_query, log_timestamp, and
log_pid too).  This will generate rather voluminous log output, perhaps
enough to provide a clue.

			regards, tom lane

Tue, 02 Oct 2001 10:41:58 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Problem on AIX with current 
> If you have no better way of determining what's going on, it might help
> to recompile with LOCK_DEBUG defined, then enable trace_lwlocks in
> postgresql.conf (better turn on debug_print_query, log_timestamp, and
> log_pid too).  This will generate rather voluminous log output, perhaps
> enough to provide a clue.

When I recompiled with LOCK_DEBUG and trace_lwlocks = true, it *works*
(and saw lots of lock debugging messages, of course). However if I
turn trace_lwlocks to off, the backend stucks again. Is there anything
I can do?

Note the machine has 4 processors. Is that related to?
--
Tatsuo Ishii

Thu, 04 Oct 2001 13:07:17 +0900
Tatsuo Ishii <t-ishii@sra.co.jp>
Re: Problem on AIX with current 
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> When I recompiled with LOCK_DEBUG and trace_lwlocks = true, it *works*
> (and saw lots of lock debugging messages, of course). However if I
> turn trace_lwlocks to off, the backend stucks again.

Ugh ... ye classic Heisenbug ...

> Is there anything I can do?

Apparently the problem is timing-sensitive, which is hardly surprising
for a lock issue.  You might find that it occurs some of the time if
you repeat the test over and over.

> Note the machine has 4 processors. Is that related to?

Hard to tell at this point, but considering that no one else has
reported a problem so far, it does seem like multiple CPUs at least
help to make the failure more probable.  But it could just be a
portability problem.  Do you have another machine with identical OS
and fewer processors to try for comparison?

Andreas, have you tried CVS tip lately on AIX?  What's your results?

			regards, tom lane

Thu, 04 Oct 2001 00:25:19 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Problem on AIX with current 