On Feb 13, 2003, Tom Lane wrote:
>
> Laurette Cisneros <laurette@nextbus.com> writes:
> > This is the error in the pgsql log:
> > 2003-02-13 16:21:42 [8843]   ERROR:  Index external_signstops_pkey is
> > not a btree
>
> This says that one of two fields that should never change, in fixed
> positions in the first block of a btree index, didn't have the right
> values.  I am not aware of any PG bugs that could overwrite those
> fields.  I think the most likely bet is that you've got hardware
> issues ... have you run memory and disk diagnostics lately?

I am seeing this same problem on two separate machines, one brand new, one 
older.  Not sure yet what is causing it, but seems pretty unlikely that it 
is hardware-related.

Ed


Mon, 31 Mar 2003 15:38:12 -0700
"Ed L." <pgsql@bluepolka.net>
Re: index corruption?
On Monday March 31 2003 3:38, Ed L. wrote:
> On Feb 13, 2003, Tom Lane wrote:
> > Laurette Cisneros <laurette@nextbus.com> writes:
> > > This is the error in the pgsql log:
> > > 2003-02-13 16:21:42 [8843]   ERROR:  Index external_signstops_pkey is
> > > not a btree
> >
> > This says that one of two fields that should never change, in fixed
> > positions in the first block of a btree index, didn't have the right
> > values.  I am not aware of any PG bugs that could overwrite those
> > fields.  I think the most likely bet is that you've got hardware
> > issues ... have you run memory and disk diagnostics lately?
>
> I am seeing this same problem on two separate machines, one brand new,
> one older.  Not sure yet what is causing it, but seems pretty unlikely
> that it is hardware-related.

I am dabbling for the first time with a (crashing) C trigger, so that may be 
the culprit here.

Ed


Mon, 31 Mar 2003 15:41:27 -0700
"Ed L." <pgsql@bluepolka.net>
Re: index corruption?
"Ed L." <pgsql@bluepolka.net> writes:
>> I am seeing this same problem on two separate machines, one brand new,
>> one older.  Not sure yet what is causing it, but seems pretty unlikely
>> that it is hardware-related.

> I am dabbling for the first time with a (crashing) C trigger, so that may be 
> the culprit here.

Could well be, although past experience has been that crashes in C code
seldom lead directly to disk corruption.  (First, the bogus code has to
overwrite a shared disk buffer.  If you follow what I consider the
better path of not making your shared buffers a large fraction of the
address space, the odds of a wild store happening to hit a disk buffer
aren't high.  Second, once it's corrupted a shared buffer, it has to
contrive to cause that buffer to get written out before the core dump
occurs --- in most cases, the fact that the postmaster abandons the
contents of shared memory after a backend crash protects us from this
kind of failure.)

When you find the problem, please take note of whether there's something
involved that increases the chances of corruption getting to disk.  We
might want to try to do something about it ...

			regards, tom lane


Mon, 31 Mar 2003 17:54:22 -0500
Tom Lane <tgl@sss.pgh.pa.us>
Re: index corruption? 
On Monday March 31 2003 3:54, Tom Lane wrote:
> "Ed L." <pgsql@bluepolka.net> writes:
> >> I am seeing this same problem on two separate machines, one brand new,
> >> one older.  Not sure yet what is causing it, but seems pretty unlikely
> >> that it is hardware-related.
> >
> > I am dabbling for the first time with a (crashing) C trigger, so that
> > may be the culprit here.
>
> Could well be, although past experience has been that crashes in C code
> seldom lead directly to disk corruption.  (First, the bogus code has to
> overwrite a shared disk buffer.  If you follow what I consider the
> better path of not making your shared buffers a large fraction of the
> address space, the odds of a wild store happening to hit a disk buffer
> aren't high.  Second, once it's corrupted a shared buffer, it has to
> contrive to cause that buffer to get written out before the core dump
> occurs --- in most cases, the fact that the postmaster abandons the
> contents of shared memory after a backend crash protects us from this
> kind of failure.)
>
> When you find the problem, please take note of whether there's something
> involved that increases the chances of corruption getting to disk.  We
> might want to try to do something about it ...

It is definitely due to some rogue trigger code.  Not sure what exactly, but 
if I remove a certain code segment the problem disappears.

Ed


Mon, 31 Mar 2003 16:15:43 -0700
"Ed L." <pgsql@bluepolka.net>
Re: index corruption?
On Mon, 31 Mar 2003, Ed L. wrote:

> On Feb 13, 2003, Tom Lane wrote:
> >
> > Laurette Cisneros <laurette@nextbus.com> writes:
> > > This is the error in the pgsql log:
> > > 2003-02-13 16:21:42 [8843]   ERROR:  Index external_signstops_pkey is
> > > not a btree
> >
> > This says that one of two fields that should never change, in fixed
> > positions in the first block of a btree index, didn't have the right
> > values.  I am not aware of any PG bugs that could overwrite those
> > fields.  I think the most likely bet is that you've got hardware
> > issues ... have you run memory and disk diagnostics lately?
> 
> I am seeing this same problem on two separate machines, one brand new, one 
> older.  Not sure yet what is causing it, but seems pretty unlikely that it 
> is hardware-related.

Until you've tested them, the likelyhood is unimportant.  If you've tested 
the boxes, and the memory tests good and the hard drives test good, then 
there is still likely to be another explanation, like a runaway kernel bug 
is writing somewhere it should every fifth eon or two.

If you haven't tested the boxes, they're reliability is part of the NULL 
set. :-)


Mon, 31 Mar 2003 16:36:01 -0700 (MST)
"scott.marlowe" <scott.marlowe@ihs.com>
Re: index corruption?
On Monday March 31 2003 4:15, Ed L. wrote:
> On Monday March 31 2003 3:54, Tom Lane wrote:
> > "Ed L." <pgsql@bluepolka.net> writes:
> > >> I am seeing this same problem on two separate machines, one brand
> > >> new, one older.  Not sure yet what is causing it, but seems pretty
> > >> unlikely that it is hardware-related.
> > >
> > > I am dabbling for the first time with a (crashing) C trigger, so that
> > > may be the culprit here.
> >
> > Could well be, although past experience has been that crashes in C code
> > seldom lead directly to disk corruption.  (First, the bogus code has to
> > overwrite a shared disk buffer.  If you follow what I consider the
> > better path of not making your shared buffers a large fraction of the
> > address space, the odds of a wild store happening to hit a disk buffer
> > aren't high.  Second, once it's corrupted a shared buffer, it has to
> > contrive to cause that buffer to get written out before the core dump
> > occurs --- in most cases, the fact that the postmaster abandons the
> > contents of shared memory after a backend crash protects us from this
> > kind of failure.)
> >
> > When you find the problem, please take note of whether there's
> > something involved that increases the chances of corruption getting to
> > disk.  We might want to try to do something about it ...

Well, I fixed it but cannot now remember exactly what change did it amidst a 
bunch of rewrites of some existing stuff, and I cannot get back to that 
state from here.  :(  It was definitely arising from some funky C trigger 
code of my own making.

Ed


Mon, 31 Mar 2003 18:06:35 -0700
"Ed L." <pgsql@bluepolka.net>
Re: index corruption?