{"action":"created","issue":{"url":"https://api.github.com/repos/reorg/pg_repack/issues/92","repository_url":"https://api.github.com/repos/reorg/pg_repack","labels_url":"https://api.github.com/repos/reorg/pg_repack/issues/92/labels{/name}","comments_url":"https://api.github.com/repos/reorg/pg_repack/issues/92/comments","events_url":"https://api.github.com/repos/reorg/pg_repack/issues/92/events","html_url":"https://github.com/reorg/pg_repack/issues/92","id":171801841,"number":92,"title":"pg_repack 1.3.4 server crash","user":{"login":"gdeglin","id":4891,"avatar_url":"https://avatars.githubusercontent.com/u/4891?v=3","gravatar_id":"","url":"https://api.github.com/users/gdeglin","html_url":"https://github.com/gdeglin","followers_url":"https://api.github.com/users/gdeglin/followers","following_url":"https://api.github.com/users/gdeglin/following{/other_user}","gists_url":"https://api.github.com/users/gdeglin/gists{/gist_id}","starred_url":"https://api.github.com/users/gdeglin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gdeglin/subscriptions","organizations_url":"https://api.github.com/users/gdeglin/orgs","repos_url":"https://api.github.com/users/gdeglin/repos","events_url":"https://api.github.com/users/gdeglin/events{/privacy}","received_events_url":"https://api.github.com/users/gdeglin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-08-18T02:04:01Z","updated_at":"2016-08-18T06:19:15Z","closed_at":null,"body":"We've been using 1.3.3 fairly regularly without issue. Today we tried the master branch of pg_repack and experienced a server crash.\r\n\r\nHere is the command we used and the crash output:\r\n```\r\nroot@server-03:/home/user/pg_repack# /usr/lib/postgresql/9.5/bin/pg_repack -d our_database -j 10 -t one_of_our_tables\r\nNOTICE: Setting up workers.conns\r\nINFO: repacking table \"notifications\"\r\nLOG: Initial worker 0 to build index: CREATE INDEX index_250210598 ON repack.table_61541 USING btree (send_after)\r\nLOG: Initial worker 1 to build index: CREATE INDEX index_237745594 ON repack.table_61541 USING btree (app_id, id) WHERE (id < '01010101-0101-0101-0101-010101010101'::uuid)\r\nLOG: Initial worker 2 to build index: CREATE INDEX index_233080738 ON repack.table_61541 USING btree (app_id, send_after) WHERE ((queued_at IS NOT NULL) AND (NOT canceled))\r\nLOG: Initial worker 3 to build index: CREATE INDEX index_80485329 ON repack.table_61541 USING btree (queued_at, canceled, send_after, created_at) WHERE ((queued_at IS NOT NULL) AND (NOT canceled) AND (NOT sending_done))\r\nLOG: Initial worker 4 to build index: CREATE INDEX index_113885354 ON repack.table_61541 USING btree (app_id, send_after)\r\nLOG: Initial worker 5 to build index: CREATE INDEX index_194895629 ON repack.table_61541 USING btree (app_id, queued_at, send_after, created_source)\r\nLOG: Initial worker 6 to build index: CREATE INDEX index_99127176 ON repack.table_61541 USING btree (app_id, queued_at DESC) WHERE (queued_at IS NOT NULL)\r\nLOG: Initial worker 7 to build index: CREATE UNIQUE INDEX index_67055 ON repack.table_61541 USING btree (id)\r\nWARNING:  terminating connection because of crash of another server process\r\nDETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.\r\nHINT:  In a moment you should be able to reconnect to the database and repeat your command.\r\nWARNING:  terminating connection because of crash of another server process\r\nDETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.\r\nHINT:  In a moment you should be able to reconnect to the database and repeat your command.\r\nWARNING:  terminating connection because of crash of another server process\r\nDETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.\r\nHINT:  In a moment you should be able to reconnect to the database and repeat your command.\r\nWARNING:  terminating connection because of crash of another server process\r\nDETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.\r\nHINT:  In a moment you should be able to reconnect to the database and repeat your command.\r\nWARNING:  terminating connection because of crash of another server process\r\nDETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.\r\nHINT:  In a moment you should be able to reconnect to the database and repeat your command.\r\nWARNING:  terminating connection because of crash of another server process\r\nDETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.\r\nHINT:  In a moment you should be able to reconnect to the database and repeat your command.\r\nWARNING:  terminating connection because of crash of another server process\r\nDETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.\r\nHINT:  In a moment you should be able to reconnect to the database and repeat your command.\r\nWARNING:  terminating connection because of crash of another server process\r\nDETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.\r\nHINT:  In a moment you should be able to reconnect to the database and repeat your command.\r\nWARNING: Error fetching async query status: server closed the connection unexpectedly\r\n\tThis probably means the server terminated abnormally\r\n\tbefore or while processing the request.\r\nWARNING:  terminating connection because of crash of another server process\r\nDETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.\r\nHINT:  In a moment you should be able to reconnect to the database and repeat your command.\r\nERROR: query failed: server closed the connection unexpectedly\r\n\tThis probably means the server terminated abnormally\r\n\tbefore or while processing the request.\r\nDETAIL: query was: ROLLBACK\r\nERROR: could not connect to database with \"dbname=our_database \": FATAL:  the database system is in recovery mode\r\nERROR: could not connect to database with \"dbname=our_database \": FATAL:  the database system is in recovery mode\r\n```"},"comment":{"url":"https://api.github.com/repos/reorg/pg_repack/issues/comments/240633975","html_url":"https://github.com/reorg/pg_repack/issues/92#issuecomment-240633975","issue_url":"https://api.github.com/repos/reorg/pg_repack/issues/92","id":240633975,"user":{"login":"gdeglin","id":4891,"avatar_url":"https://avatars.githubusercontent.com/u/4891?v=3","gravatar_id":"","url":"https://api.github.com/users/gdeglin","html_url":"https://github.com/gdeglin","followers_url":"https://api.github.com/users/gdeglin/followers","following_url":"https://api.github.com/users/gdeglin/following{/other_user}","gists_url":"https://api.github.com/users/gdeglin/gists{/gist_id}","starred_url":"https://api.github.com/users/gdeglin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gdeglin/subscriptions","organizations_url":"https://api.github.com/users/gdeglin/orgs","repos_url":"https://api.github.com/users/gdeglin/repos","events_url":"https://api.github.com/users/gdeglin/events{/privacy}","received_events_url":"https://api.github.com/users/gdeglin/received_events","type":"User","site_admin":false},"created_at":"2016-08-18T06:19:15Z","updated_at":"2016-08-18T06:19:15Z","body":"Hmm, in our syslog I do see that postgres triggered the OOM killer:\r\n\r\n```\r\nAug 18 01:56:08 server-03 kernel: [24563682.538181] postgres invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0\r\nAug 18 01:56:08 server-03 kernel: [24563682.538184] postgres cpuset=/ mems_allowed=0-1\r\nAug 18 01:56:08 server-03 kernel: [24563682.538186] CPU: 7 PID: 33651 Comm: postgres Not tainted 3.13.0-62-generic #102-Ubuntu\r\nAug 18 01:56:08 server-03 kernel: [24563682.538187] Hardware name: Quanta OCP-Q301 /OCP-Q301, BIOS F03B3A14.Signed 07/14/2014\r\nAug 18 01:56:08 server-03 kernel: [24563682.538188]  0000000000000000 ffff880010d2b980 ffffffff81723ec0 ffff8800033ee000\r\nAug 18 01:56:08 server-03 kernel: [24563682.538193]  ffff880010d2ba08 ffffffff8171e4c7 ffffffff81067a16 ffff880010d2b9e0\r\nAug 18 01:56:08 server-03 kernel: [24563682.538196]  ffffffff810c79dc 0000000000000000 ffff88107fffce48 0000000000000000\r\nAug 18 01:56:08 server-03 kernel: [24563682.538198] Call Trace:\r\nAug 18 01:56:08 server-03 kernel: [24563682.538204]  [<ffffffff81723ec0>] dump_stack+0x45/0x56\r\nAug 18 01:56:08 server-03 kernel: [24563682.538208]  [<ffffffff8171e4c7>] dump_header+0x7f/0x1f1\r\nAug 18 01:56:08 server-03 kernel: [24563682.538211]  [<ffffffff81067a16>] ? put_online_cpus+0x56/0x80\r\nAug 18 01:56:08 server-03 kernel: [24563682.538213]  [<ffffffff810c79dc>] ? rcu_oom_notify+0xcc/0xf0\r\nAug 18 01:56:08 server-03 kernel: [24563682.538216]  [<ffffffff81152f0e>] oom_kill_process+0x1ce/0x330\r\nAug 18 01:56:08 server-03 kernel: [24563682.538220]  [<ffffffff812d8025>] ? security_capable_noaudit+0x15/0x20\r\nAug 18 01:56:08 server-03 kernel: [24563682.538221]  [<ffffffff81153644>] out_of_memory+0x414/0x450\r\nAug 18 01:56:08 server-03 kernel: [24563682.538224]  [<ffffffff81159980>] __alloc_pages_nodemask+0xa60/0xb80\r\nAug 18 01:56:08 server-03 kernel: [24563682.538228]  [<ffffffff81197f83>] alloc_pages_current+0xa3/0x160\r\nAug 18 01:56:08 server-03 kernel: [24563682.538230]  [<ffffffff8114fac7>] __page_cache_alloc+0x97/0xc0\r\nAug 18 01:56:08 server-03 kernel: [24563682.538232]  [<ffffffff811514d5>] filemap_fault+0x185/0x410\r\nAug 18 01:56:08 server-03 kernel: [24563682.538235]  [<ffffffff811763cf>] __do_fault+0x6f/0x530\r\nAug 18 01:56:08 server-03 kernel: [24563682.538237]  [<ffffffff8117a252>] handle_mm_fault+0x482/0xf10\r\nAug 18 01:56:08 server-03 kernel: [24563682.538240]  [<ffffffff8172a6e2>] ? mutex_lock+0x12/0x2f\r\nAug 18 01:56:08 server-03 kernel: [24563682.538243]  [<ffffffff811c7358>] ? pipe_read+0x2b8/0x3e0\r\nAug 18 01:56:08 server-03 kernel: [24563682.538246]  [<ffffffff8172fff4>] __do_page_fault+0x184/0x570\r\nAug 18 01:56:08 server-03 kernel: [24563682.538249]  [<ffffffff811bdc0a>] ? do_sync_read+0x5a/0x90\r\nAug 18 01:56:08 server-03 kernel: [24563682.538251]  [<ffffffff817303fa>] do_page_fault+0x1a/0x70\r\nAug 18 01:56:08 server-03 kernel: [24563682.538253]  [<ffffffff8172c728>] page_fault+0x28/0x30\r\n```\r\n\r\n"}}
{"actor":{"display_login":"gdeglin"}}