{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/1547","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/1547/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/1547/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/1547/events","html_url":"https://github.com/scylladb/scylla/issues/1547","id":169340947,"number":1547,"title":"[1.3 RC2 AMI] Coredumps observed during 8 hour longevity test","user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"tgrabiec","id":283695,"avatar_url":"https://avatars.githubusercontent.com/u/283695?v=3","gravatar_id":"","url":"https://api.github.com/users/tgrabiec","html_url":"https://github.com/tgrabiec","followers_url":"https://api.github.com/users/tgrabiec/followers","following_url":"https://api.github.com/users/tgrabiec/following{/other_user}","gists_url":"https://api.github.com/users/tgrabiec/gists{/gist_id}","starred_url":"https://api.github.com/users/tgrabiec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgrabiec/subscriptions","organizations_url":"https://api.github.com/users/tgrabiec/orgs","repos_url":"https://api.github.com/users/tgrabiec/repos","events_url":"https://api.github.com/users/tgrabiec/events{/privacy}","received_events_url":"https://api.github.com/users/tgrabiec/received_events","type":"User","site_admin":false},"assignees":[{"login":"tgrabiec","id":283695,"avatar_url":"https://avatars.githubusercontent.com/u/283695?v=3","gravatar_id":"","url":"https://api.github.com/users/tgrabiec","html_url":"https://github.com/tgrabiec","followers_url":"https://api.github.com/users/tgrabiec/followers","following_url":"https://api.github.com/users/tgrabiec/following{/other_user}","gists_url":"https://api.github.com/users/tgrabiec/gists{/gist_id}","starred_url":"https://api.github.com/users/tgrabiec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgrabiec/subscriptions","organizations_url":"https://api.github.com/users/tgrabiec/orgs","repos_url":"https://api.github.com/users/tgrabiec/repos","events_url":"https://api.github.com/users/tgrabiec/events{/privacy}","received_events_url":"https://api.github.com/users/tgrabiec/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2016-08-04T10:18:53Z","updated_at":"2016-08-04T11:21:51Z","closed_at":"2016-08-04T11:21:51Z","body":"*Installation details*\r\nScylla version (or git commit hash): ami-19aa3d0e (copied to us-west-2, ami-5acb023a)\r\n\r\n    2016-08-04 09:38:27,670 nemesis          L0080 INFO | sdcm.nemesis.ChaosMonkey: Report\r\n    2016-08-04 09:38:27,670 nemesis          L0081 INFO | sdcm.nemesis.ChaosMonkey: DB Version:\r\n    2016-08-04 09:38:27,670 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-boost-date-time-1.58.0-11.el7.centos.x86_64\r\n    2016-08-04 09:38:27,671 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-boost-filesystem-1.58.0-11.el7.centos.x86_64\r\n    2016-08-04 09:38:27,671 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-boost-program-options-1.58.0-11.el7.centos.x86_64\r\n    2016-08-04 09:38:27,671 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-boost-system-1.58.0-11.el7.centos.x86_64\r\n    2016-08-04 09:38:27,671 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-boost-test-1.58.0-11.el7.centos.x86_64\r\n    2016-08-04 09:38:27,671 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-boost-thread-1.58.0-11.el7.centos.x86_64\r\n    2016-08-04 09:38:27,671 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-conf-1.3.rc2-20160729.23c340b.el7.centos.x86_64\r\n    2016-08-04 09:38:27,671 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-env-1.0-1.el7.centos.noarch\r\n    2016-08-04 09:38:27,671 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-gdb-7.10.1-30.el7.centos.x86_64\r\n    2016-08-04 09:38:27,671 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-jmx-1.3.rc2-20160729.e87c771.el7.centos.noarch\r\n    2016-08-04 09:38:27,672 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-server-1.3.rc2-20160729.23c340b.el7.centos.x86_64\r\n    2016-08-04 09:38:27,672 nemesis          L0083 INFO | sdcm.nemesis.ChaosMonkey: scylla-tools-1.3.rc2-20160729.472d2da.el7.centos.noarch\r\n    2016-08-04 09:38:27,672 nemesis          L0084 INFO | sdcm.nemesis.ChaosMonkey: Interval: 900 s\r\n    2016-08-04 09:38:27,672 nemesis          L0085 INFO | sdcm.nemesis.ChaosMonkey: Average duration: 1576 s\r\n    2016-08-04 09:38:27,672 nemesis          L0086 INFO | sdcm.nemesis.ChaosMonkey: Total execution time: 29188 s\r\n    2016-08-04 09:38:27,672 nemesis          L0087 INFO | sdcm.nemesis.ChaosMonkey: Times executed: 11\r\n    2016-08-04 09:38:27,672 nemesis          L0088 INFO | sdcm.nemesis.ChaosMonkey: Unhandled exceptions: 0\r\n\r\nCluster size: Cluster scylla-1-3-rc2-scylla-db-cluster-2d0f0f18 (AMI: ami-5acb023a Type: c3.xlarge)\r\nOS (RHEL/CentOS/Ubuntu/AWS AMI): CentOS\r\n\r\nRunning the longevity test for 8 hours on a c3.xlarge cluster, we observed a coredump generated:\r\n\r\n               PID: 11378 (scylla)\r\n               UID: 994 (scylla)\r\n               GID: 1001 (scylla)\r\n            Signal: 6 (ABRT)\r\n         Timestamp: Thu 2016-08-04 07:51:44 UTC (1h 28min ago)\r\n      Command Line: /usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --collectd-address=127.0.0.1:25826 --collectd=1 --collectd-poll-period 3000 --network-stack posix --num-io-queues 4 --max-io-requests 32\r\n        Executable: /usr/bin/scylla\r\n     Control Group: /system.slice/scylla-server.service\r\n              Unit: scylla-server.service\r\n             Slice: system.slice\r\n           Boot ID: 24b8143161b34f27927f2868d5a4f532\r\n        Machine ID: 8ea3e6ea03b64d018847fcdf669935d4\r\n          Hostname: ip-172-31-38-120\r\n          Coredump: /var/lib/systemd/coredump/core.scylla.994.24b8143161b34f27927f2868d5a4f532.11378.1470297104000000.xz\r\n           Message: Process 11378 (scylla) of user 994 dumped core.\r\n                    \r\n                    Stack trace of thread 11379:\r\n                    #0  0x00007f6a5f0555f7 raise (libc.so.6)\r\n                    #1  0x00007f6a5f056ce8 abort (libc.so.6)\r\n                    #2  0x00007f6a5f04e566 __assert_fail_base (libc.so.6)\r\n                    #3  0x00007f6a5f04e612 __assert_fail (libc.so.6)\r\n                    #4  0x0000000000504df2 n/a (scylla)\r\n                    #5  0x000000000051a5dc n/a (scylla)\r\n                    #6  0x00000000004d570e n/a (scylla)\r\n                    #7  0x000000000050326b n/a (scylla)\r\n                    #8  0x00000000005205dd n/a (scylla)\r\n                    #9  0x00000000005522de n/a (scylla)\r\n                    #10 0x00007f6a5f3e8dc5 start_thread (libpthread.so.0)\r\n                    #11 0x00007f6a5f11621d __clone (libc.so.6)\r\n                    \r\n                    Stack trace of thread 11383:\r\n                    #0  0x00007f6a5f3ef22d read (libpthread.so.0)\r\n                    #1  0x00000000004d5fbe n/a (scylla)\r\n                    #2  0x00000000005522de n/a (scylla)\r\n                    #3  0x00007f6a5f3e8dc5 start_thread (libpthread.so.0)\r\n                    #4  0x00007f6a5f11621d __clone (libc.so.6)\r\n                    \r\n                    Stack trace of thread 11384:\r\n                    #0  0x00007f6a5f3ef22d read (libpthread.so.0)\r\n                    #1  0x00000000004d5fbe n/a (scylla)\r\n                    #2  0x00000000005522de n/a (scylla)\r\n                    #3  0x00007f6a5f3e8dc5 start_thread (libpthread.so.0)\r\n                    #4  0x00007f6a5f11621d __clone (libc.so.6)\r\n                    \r\n                    Stack trace of thread 11385:\r\n                    #0  0x00007f6a5f3ef22d read (libpthread.so.0)\r\n                    #1  0x00000000004d5fbe n/a (scylla)\r\n                    #2  0x00000000005522de n/a (scylla)\r\n                    #3  0x00007f6a5f3e8dc5 start_thread (libpthread.so.0)\r\n                    #4  0x00007f6a5f11621d __clone (libc.so.6)\r\n                    \r\n                    Stack trace of thread 11381:\r\n                    #0  0x000000000054e0b2 n/a (scylla)\r\n                    #1  0x000000000076731e n/a (scylla)\r\n                    #2  0x00000000010ab56c n/a (scylla)\r\n                    #3  0x00000000010bbf66 n/a (scylla)\r\n                    #4  0x0000000001000cc3 n/a (scylla)\r\n                    #5  0x0000000000cee447 n/a (scylla)\r\n                    #6  0x0000000000cf05a9 n/a (scylla)\r\n                    #7  0x0000000000d327c4 n/a (scylla)\r\n                    #8  0x0000000000d32a04 n/a (scylla)\r\n                    #9  0x0000000000d34c9d n/a (scylla)\r\n                    #10 0x0000000000cfea05 n/a (scylla)\r\n                    #11 0x0000000000d00745 n/a (scylla)\r\n                    #12 0x0000000000b6043d n/a (scylla)\r\n                    #13 0x0000000000b60b36 n/a (scylla)\r\n                    #14 0x0000000000d4828d n/a (scylla)\r\n                    #15 0x0000000000d48f0d n/a (scylla)\r\n                    #16 0x0000000000acdbe2 n/a (scylla)\r\n                    #17 0x0000000000ad4b88 n/a (scylla)\r\n                    #18 0x0000000000ad54d3 n/a (scylla)\r\n                    #19 0x0000000000ad69d0 n/a (scylla)\r\n                    #20 0x0000000000ad8d3d n/a (scylla)\r\n                    #21 0x0000000000ad9cab n/a (scylla)\r\n                    #22 0x0000000000ada91e n/a (scylla)\r\n                    #23 0x0000000000adaf0e n/a (scylla)\r\n                    #24 0x0000000000adbb80 n/a (scylla)\r\n                    #25 0x0000000000adbca4 n/a (scylla)\r\n                    #26 0x00000000004d570e n/a (scylla)\r\n                    #27 0x000000000050326b n/a (scylla)\r\n                    #28 0x00000000005205dd n/a (scylla)\r\n                    #29 0x00000000005522de n/a (scylla)\r\n                    #30 0x00007f6a5f3e8dc5 start_thread (libpthread.so.0)\r\n                    #31 0x00007f6a5f11621d __clone (libc.so.6)\r\n                    \r\n                    Stack trace of thread 11380:\r\n                    #0  0x00000000005d40cb n/a (scylla)\r\n                    #1  0x000000000101522d n/a (scylla)\r\n                    #2  0x00000000010a8b9d n/a (scylla)\r\n                    #3  0x00000000010b9085 n/a (scylla)\r\n                    #4  0x00000000010b922b n/a (scylla)\r\n                    #5  0x00000000010008bf n/a (scylla)\r\n                    #6  0x000000000125cbee n/a (scylla)\r\n                    #7  0x000000000125d273 n/a (scylla)\r\n                    #8  0x000000000125d55f n/a (scylla)\r\n                    #9  0x000000000125d6f3 n/a (scylla)\r\n                    #10 0x00000000008fff33 n/a (scylla)\r\n                    #11 0x00000000008f9207 n/a (scylla)\r\n                    #12 0x00000000008f99fb n/a (scylla)\r\n                    #13 0x00000000008fa7d1 n/a (scylla)\r\n                    #14 0x000000000125a709 n/a (scylla)\r\n                    #15 0x000000000125a872 n/a (scylla)\r\n                    #16 0x000000000125aad7 n/a (scylla)\r\n                    #17 0x000000000125173c n/a (scylla)\r\n                    #18 0x00000000012573af n/a (scylla)\r\n                    #19 0x00000000004d570e n/a (scylla)\r\n                    #20 0x000000000050326b n/a (scylla)\r\n                    #21 0x00000000005205dd n/a (scylla)\r\n                    #22 0x00000000005522de n/a (scylla)\r\n                    #23 0x00007f6a5f3e8dc5 start_thread (libpthread.so.0)\r\n                    #24 0x00007f6a5f11621d __clone (libc.so.6)\r\n                    \r\n                    Stack trace of thread 11382:\r\n                    #0  0x00007f6a5f3ef22d read (libpthread.so.0)\r\n                    #1  0x00000000004d5fbe n/a (scylla)\r\n                    #2  0x00000000005522de n/a (scylla)\r\n                    #3  0x00007f6a5f3e8dc5 start_thread (libpthread.so.0)\r\n                    #4  0x00007f6a5f11621d __clone (libc.so.6)\r\n                    \r\n                    Stack trace of thread 11378:\r\n                    #0  0x00000000008f7363 n/a (scylla)\r\n                    #1  0x00000000009046de n/a (scylla)\r\n                    #2  0x00000000007ace08 n/a (scylla)\r\n                    #3  0x000000000081a100 n/a (scylla)\r\n                    #4  0x000000000082414e n/a (scylla)\r\n                    #5  0x00000000008248e7 n/a (scylla)\r\n                    #6  0x00000000004d570e n/a (scylla)\r\n                    #7  0x000000000050326b n/a (scylla)\r\n                    #8  0x000000000057ad05 n/a (scylla)\r\n                    #9  0x0000000000\r\n\r\nThe coredump can be downloaded using:\r\n\r\n    INFO | Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-5 [54.187.159.38 | 172.31.38.120] (seed: None): To download it, you may use 'curl --user-agent [user-agent] scylladb-users-upload.s3.amazonaws.com/core.scylla.994.24b8143161b34f27927f2868d5a4f532.11378.1470297104000000/core.scylla.994.24b8143161b34f27927f2868d5a4f532.11378.1470297104000000.xz > core.scylla.994.24b8143161b34f27927f2868d5a4f532.11378.1470297104000000.xz'\r\n\r\nThe cassandra-stress cmd used was:\r\n\r\n    DEBUG| Remote [centos@54.187.84.151]: Running 'cassandra-stress write cl=QUORUM duration=480m -schema 'replication(factor=3)' -port jmx=6868 -mode cql3 native -rate threads=1000   -pop seq=1..10000000 -node 172.31.38.122'\r\n\r\nDisruption functions executed:\r\n\r\n    INFO | sdcm.nemesis.ChaosMonkey: Operation log:\r\n    INFO | sdcm.nemesis.ChaosMonkey: {'duration': 149, 'start': 1470275243, 'operation': 'Drainer Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-3 [54.201.40.197 | 172.31.38.118] (seed: False)', 'end': 1470275392}\r\n    INFO | sdcm.nemesis.ChaosMonkey: {'duration': 4412, 'start': 1470276293, 'operation': 'CorruptThenRebuild Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-4 [54.187.225.24 | 172.31.38.121] (seed: False)', 'end': 1470280705}\r\n    INFO | sdcm.nemesis.ChaosMonkey: {'duration': 144, 'start': 1470281605, 'operation': 'Drainer Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-4 [54.187.225.24 | 172.31.38.121] (seed: False)', 'end': 1470281749}\r\n    INFO | sdcm.nemesis.ChaosMonkey: {'duration': 4303, 'start': 1470282649, 'operation': 'Decommission Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-3 [54.201.112.215 | 172.31.38.118] (seed: False)', 'end': 1470286953}\r\n    INFO | sdcm.nemesis.ChaosMonkey: {'duration': 488, 'start': 1470287853, 'operation': 'StopWaitStartService Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-6 [54.213.247.171 | 172.31.34.19] (seed: False)', 'end': 1470288341}\r\n    INFO | sdcm.nemesis.ChaosMonkey: {'duration': 207, 'start': 1470289241, 'operation': 'Drainer Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-2 [54.186.176.59 | 172.31.38.119] (seed: False)', 'end': 1470289449}\r\n    INFO | sdcm.nemesis.ChaosMonkey: {'duration': 4958, 'start': 1470290349, 'operation': 'Decommission Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-6 [54.213.247.171 | 172.31.34.19] (seed: False)', 'end': 1470295307}\r\n    INFO | sdcm.nemesis.ChaosMonkey: {'duration': 971, 'start': 1470296207, 'operation': 'CorruptThenRepair Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-5 [54.187.159.38 | 172.31.38.120] (seed: False)', 'end': 1470297179}\r\n    INFO | sdcm.nemesis.ChaosMonkey: {'duration': 1079, 'start': 1470298079, 'operation': 'CorruptThenRebuild Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-4 [54.149.178.52 | 172.31.38.121] (seed: False)', 'end': 1470299159}\r\n    INFO | sdcm.nemesis.ChaosMonkey: {'duration': 149, 'start': 1470300059, 'operation': 'Drainer Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-2 [54.191.125.235 | 172.31.38.119] (seed: False)', 'end': 1470300209}\r\n    INFO | sdcm.nemesis.ChaosMonkey: {'duration': 486, 'start': 1470301109, 'operation': 'StopWaitStartService Node scylla-1-3-rc2-scylla-db-node-2d0f0f18-4 [54.149.178.52 | 172.31.38.121] (seed: False)', 'end': 1470301595}\r\n\r\nWe have full logs available on .tar.bz2 format (21 MB) in case that would be interesting."},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/237525800","html_url":"https://github.com/scylladb/scylla/issues/1547#issuecomment-237525800","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/1547","id":237525800,"user":{"login":"tgrabiec","id":283695,"avatar_url":"https://avatars.githubusercontent.com/u/283695?v=3","gravatar_id":"","url":"https://api.github.com/users/tgrabiec","html_url":"https://github.com/tgrabiec","followers_url":"https://api.github.com/users/tgrabiec/followers","following_url":"https://api.github.com/users/tgrabiec/following{/other_user}","gists_url":"https://api.github.com/users/tgrabiec/gists{/gist_id}","starred_url":"https://api.github.com/users/tgrabiec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgrabiec/subscriptions","organizations_url":"https://api.github.com/users/tgrabiec/orgs","repos_url":"https://api.github.com/users/tgrabiec/repos","events_url":"https://api.github.com/users/tgrabiec/events{/privacy}","received_events_url":"https://api.github.com/users/tgrabiec/received_events","type":"User","site_admin":false},"created_at":"2016-08-04T11:21:51Z","updated_at":"2016-08-04T11:21:51Z","body":"Looks like a dup of #1537\r\n\r\n```\r\n(gdb) bt\r\n#0  0x00007f6a5f0555f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56\r\n#1  0x00007f6a5f056ce8 in __GI_abort () at abort.c:90\r\n#2  0x00007f6a5f04e566 in __assert_fail_base (fmt=0x7f6a5f19e228 \"%s%s%s:%u: %s%sAssertion `%s' failed.\\n%n\", \r\n    assertion=assertion@entry=0x17a26c1 \"!_future && _state && !_task\", file=file@entry=0x17c183b \"core/future.hh\", \r\n    line=line@entry=1092, \r\n    function=function@entry=0x17acf00 <promise<io_event>::get_future()::__PRETTY_FUNCTION__> \"future<T ...> promise<T>::get_future() [with T = {io_event}]\") at assert.c:92\r\n#3  0x00007f6a5f04e612 in __GI___assert_fail (assertion=assertion@entry=0x17a26c1 \"!_future && _state && !_task\", \r\n    file=file@entry=0x17c183b \"core/future.hh\", line=line@entry=1092, \r\n    function=function@entry=0x17acf00 <promise<io_event>::get_future()::__PRETTY_FUNCTION__> \"future<T ...> promise<T>::get_future() [with T = {io_event}]\") at assert.c:101\r\n#4  0x0000000000504df2 in promise<io_event>::get_future (this=<optimized out>) at core/future.hh:1092\r\n#5  reactor::<lambda()>::operator() (__closure=<optimized out>) at core/reactor.cc:564\r\n#6  apply_helper<reactor::submit_io(Func) [with Func = posix_file_impl::read_dma(uint64_t, void*, size_t, const io_priority_class&)::<lambda(iocb&)>]::<lambda()>, std::tuple<>&&, std::integer_sequence<long unsigned int> >::apply (args=<optimized out>, \r\n    func=<optimized out>) at core/apply.hh:34\r\n#7  apply<reactor::submit_io(Func) [with Func = posix_file_impl::read_dma(uint64_t, void*, size_t, const io_priority_class&)::<lambda(iocb&)>]::<lambda()> > (args=<optimized out>, func=<optimized out>) at core/apply.hh:42\r\n#8  futurize<future<io_event> >::apply<reactor::submit_io(Func) [with Func = posix_file_impl::read_dma(uint64_t, void*, size_t, const io_priority_class&)::<lambda(iocb&)>]::<lambda()> > (args=<optimized out>, func=<optimized out>) at core/future.hh:1234\r\n#9  future<>::then<reactor::submit_io(Func) [with Func = posix_file_impl::read_dma(uint64_t, void*, size_t, const io_priority_class&)::<lambda(iocb&)>]::<lambda()>, future<io_event> > (func=<optimized out>, this=0x7f6a5aff8590) at core/future.hh:860\r\n#10 reactor::submit_io<posix_file_impl::read_dma(uint64_t, void*, size_t, const io_priority_class&)::<lambda(iocb&)> > (\r\n    prepare_io=..., this=0x6030000c3000) at core/reactor.cc:565\r\n#11 io_queue::<lambda()>::<lambda()>::operator() (__closure=__closure@entry=0x6030000c3040) at core/reactor.cc:780\r\n#12 fair_queue::<lambda()>::operator() (__closure=<optimized out>) at core/fair_queue.hh:217\r\n#13 apply_helper<fair_queue::queue(priority_class_ptr, unsigned int, Func) [with Func = io_queue::queue_request(shard_id, const io_priority_class&, size_t, Func)::<lambda()> [with Func = posix_file_impl::read_dma(uint64_t, void*, size_t, const io_priority_class&)::<lambda(iocb&)>]::<lambda()>; futurize_t<typename std::result_of<Func()>::type> = future<io_event>; priority_class_ptr = lw_shared_ptr<priority_class>]::<lambda()>, std::tuple<>&&, std::integer_sequence<long unsigned int> >::apply (args=<optimized out>, \r\n    func=<optimized out>) at core/apply.hh:34\r\n#14 apply<fair_queue::queue(priority_class_ptr, unsigned int, Func) [with Func = io_queue::queue_request(shard_id, const io_priority_class&, size_t, Func)::<lambda()> [with Func = posix_file_impl::read_dma(uint64_t, void*, size_t, const io_priority_class&)::<lambda(iocb&)>]::<lambda()>; futurize_t<typename std::result_of<Func()>::type> = future<io_event>; priority_class_ptr = lw_shared_ptr<priority_class>]::<lambda()> > (args=<optimized out>, func=<optimized out>) at core/apply.hh:42\r\n#15 futurize<future<io_event> >::apply<fair_queue::queue(priority_class_ptr, unsigned int, Func) [with Func = io_queue::queue_reques---Type <return> to continue, or q <return> to quit---\r\nt(shard_id, const io_priority_class&, size_t, Func)::<lambda()> [with Func = posix_file_impl::read_dma(uint64_t, void*, size_t, const io_priority_class&)::<lambda(iocb&)>]::<lambda()>; futurize_t<typename std::result_of<Func()>::type> = future<io_event>; priority_class_ptr = lw_shared_ptr<priority_class>]::<lambda()> >(<unknown type in /usr/lib/debug/usr/bin/scylla.debug, CU 0x0, DIE 0x1f03c9>, <unknown type in /usr/lib/debug/usr/bin/scylla.debug, CU 0x0, DIE 0x1f03dc>) (\r\n    func=func@entry=<unknown type in /usr/lib/debug/usr/bin/scylla.debug, CU 0x0, DIE 0x1f03c9>, args=<optimized out>)\r\n    at core/future.hh:1234\r\n#16 0x000000000051a5dc in future<>::<lambda(auto:1&&)>::operator()<future_state<> > (\r\n    state=<unknown type in /usr/lib/debug/usr/bin/scylla.debug, CU 0x0, DIE 0x1ff8a3>, __closure=0x60304557f710)\r\n    at core/future.hh:870\r\n#17 continuation<future<T>::then(Func&&) [with Func = fair_queue::queue(priority_class_ptr, unsigned int, Func) [with Func = io_queue::queue_request(shard_id, const io_priority_class&, size_t, Func)::<lambda()> [with Func = posix_file_impl::read_dma(uint64_t, void*, size_t, const io_priority_class&)::<lambda(iocb&)>]::<lambda()>; futurize_t<typename std::result_of<Func()>::type> = future<io_event>; priority_class_ptr = lw_shared_ptr<priority_class>]::<lambda()>; Result = future<io_event>; T = {}]::<lambda(auto:1&&)> >::run(void) (this=0x60304557f700) at core/future.hh:402\r\n#18 0x00000000004d570e in reactor::run_tasks (this=this@entry=0x6030000c3000, tasks=...) at core/reactor.cc:1522\r\n#19 0x000000000050326b in reactor::run (this=0x6030000c3000) at core/reactor.cc:1942\r\n#20 0x00000000005205dd in smp::<lambda()>::operator()(void) const (__closure=0x600000087540) at core/reactor.cc:2810\r\n#21 0x00000000005522de in std::function<void ()>::operator()() const (this=<optimized out>)\r\n    at /opt/scylladb/include/c++/5.3.1/functional:2271\r\n#22 posix_thread::start_routine (arg=<optimized out>) at core/posix.cc:50\r\n#23 0x00007f6a5f3e8dc5 in start_thread (arg=0x7f6a5afff700) at pthread_create.c:308\r\n#24 0x00007f6a5f11621d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113\r\n\r\n```"}}
{"actor":{"display_login":"tgrabiec"}}