{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/1043","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/1043/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/1043/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/1043/events","html_url":"https://github.com/scylladb/scylla/issues/1043","id":140896808,"number":1043,"title":"Scylla AMI 0.19 fail to start, creates coredumps","user":{"login":"tzach","id":170200,"avatar_url":"https://avatars.githubusercontent.com/u/170200?v=3","gravatar_id":"","url":"https://api.github.com/users/tzach","html_url":"https://github.com/tzach","followers_url":"https://api.github.com/users/tzach/followers","following_url":"https://api.github.com/users/tzach/following{/other_user}","gists_url":"https://api.github.com/users/tzach/gists{/gist_id}","starred_url":"https://api.github.com/users/tzach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tzach/subscriptions","organizations_url":"https://api.github.com/users/tzach/orgs","repos_url":"https://api.github.com/users/tzach/repos","events_url":"https://api.github.com/users/tzach/events{/privacy}","received_events_url":"https://api.github.com/users/tzach/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/showstopper","name":"showstopper","color":"e11d21"}],"state":"closed","locked":false,"assignee":{"login":"syuu1228","id":183648,"avatar_url":"https://avatars.githubusercontent.com/u/183648?v=3","gravatar_id":"","url":"https://api.github.com/users/syuu1228","html_url":"https://github.com/syuu1228","followers_url":"https://api.github.com/users/syuu1228/followers","following_url":"https://api.github.com/users/syuu1228/following{/other_user}","gists_url":"https://api.github.com/users/syuu1228/gists{/gist_id}","starred_url":"https://api.github.com/users/syuu1228/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/syuu1228/subscriptions","organizations_url":"https://api.github.com/users/syuu1228/orgs","repos_url":"https://api.github.com/users/syuu1228/repos","events_url":"https://api.github.com/users/syuu1228/events{/privacy}","received_events_url":"https://api.github.com/users/syuu1228/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/1","html_url":"https://github.com/scylladb/scylla/milestones/GA","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/1/labels","id":1164307,"number":1,"title":"GA","description":null,"creator":{"login":"dorlaor","id":1735237,"avatar_url":"https://avatars.githubusercontent.com/u/1735237?v=3","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"open_issues":75,"closed_issues":271,"state":"open","created_at":"2015-06-15T06:37:52Z","updated_at":"2016-03-20T08:07:58Z","due_on":null,"closed_at":null},"comments":22,"created_at":"2016-03-15T07:29:37Z","updated_at":"2016-03-20T08:07:58Z","closed_at":"2016-03-20T08:07:58Z","body":"Launching Scylla  AMI 0.19, on c3.8xlarge, Scylla process fail to start, coredump\r\nLog start below\r\n\r\n```\r\nMar 15 07:03:03 ip-172-30-0-15.ec2.internal scylla_ami_setup[1253]: RAID does not constructed, going to initialize...\r\nMar 15 07:03:03 ip-172-30-0-15.ec2.internal scylla_ami_setup[1253]: Found disk /dev/xvdb\r\nMar 15 07:03:03 ip-172-30-0-15.ec2.internal scylla_ami_setup[1253]: Found disk /dev/xvdc\r\nMar 15 07:03:03 ip-172-30-0-15.ec2.internal scylla_ami_setup[1253]: Creating RAID0 for scylla using 2 disk(s): /dev/xvdb /dev/xvdc\r\nMar 15 07:03:09 ip-172-31-35-116 scylla_ami_setup[1253]: Loaded plugins: fastestmirror, langpacks\r\nMar 15 07:03:15 ip-172-31-35-116 scylla_ami_setup[1253]: Loading mirror speeds from cached hostfile\r\nMar 15 07:03:17 ip-172-31-35-116 scylla_ami_setup[1253]: * base: linux.cc.lehigh.edu\r\nMar 15 07:03:17 ip-172-31-35-116 scylla_ami_setup[1253]: * epel: s3-mirror-us-west-2.fedoraproject.org\r\nMar 15 07:03:17 ip-172-31-35-116 scylla_ami_setup[1253]: * extras: centos-distro.1gservers.com\r\nMar 15 07:03:17 ip-172-31-35-116 scylla_ami_setup[1253]: * updates: mirror.atlanticmetro.net\r\nMar 15 07:03:23 ip-172-31-35-116 scylla_ami_setup[1253]: Package mdadm-3.3.2-7.el7.x86_64 already installed and latest version\r\nMar 15 07:03:23 ip-172-31-35-116 scylla_ami_setup[1253]: Package xfsprogs-3.2.2-2.el7.x86_64 already installed and latest version\r\nMar 15 07:03:23 ip-172-31-35-116 scylla_ami_setup[1253]: Nothing to do\r\nMar 15 07:03:23 ip-172-31-35-116 scylla_ami_setup[1253]: mdadm: /dev/xvdb appears to contain an ext2fs file system\r\nMar 15 07:03:23 ip-172-31-35-116 scylla_ami_setup[1253]: size=335515648K  mtime=Thu Jan  1 00:00:00 1970\r\nMar 15 07:03:23 ip-172-31-35-116 scylla_ami_setup[1253]: mdadm: /dev/xvdc appears to contain an ext2fs file system\r\nMar 15 07:03:23 ip-172-31-35-116 scylla_ami_setup[1253]: size=335515648K  mtime=Thu Jan  1 00:00:00 1970\r\nMar 15 07:03:23 ip-172-31-35-116 scylla_ami_setup[1253]: mdadm: creation continuing despite oddities due to --run\r\nMar 15 07:03:23 ip-172-31-35-116 scylla_ami_setup[1253]: mdadm: Defaulting to version 1.2 metadata\r\nMar 15 07:03:23 ip-172-31-35-116 scylla_ami_setup[1253]: mdadm: array /dev/md0 started.\r\nMar 15 07:03:25 ip-172-31-35-116 scylla_ami_setup[1253]: meta-data=/dev/md0               isize=256    agcount=32, agsize=5240384 blks\r\nMar 15 07:03:25 ip-172-31-35-116 scylla_ami_setup[1253]: =                       sectsz=4096  attr=2, projid32bit=1\r\nMar 15 07:03:25 ip-172-31-35-116 scylla_ami_setup[1253]: =                       crc=0        finobt=0\r\nMar 15 07:03:25 ip-172-31-35-116 scylla_ami_setup[1253]: data     =                       bsize=4096   blocks=167692288, imaxpct=25\r\nMar 15 07:03:25 ip-172-31-35-116 scylla_ami_setup[1253]: =                       sunit=64     swidth=128 blks\r\nMar 15 07:03:25 ip-172-31-35-116 scylla_ami_setup[1253]: naming   =version 2              bsize=4096   ascii-ci=0 ftype=0\r\nMar 15 07:03:25 ip-172-31-35-116 scylla_ami_setup[1253]: log      =internal log           bsize=4096   blocks=81881, version=2\r\nMar 15 07:03:25 ip-172-31-35-116 scylla_ami_setup[1253]: =                       sectsz=4096  sunit=1 blks, lazy-count=1\r\nMar 15 07:03:25 ip-172-31-35-116 scylla_ami_setup[1253]: realtime =none                   extsz=4096   blocks=0, rtextents=0\r\nMar 15 07:03:26 ip-172-31-35-116 scylla_ami_setup[1253]: Setting parameters on /etc/sysconfig/scylla-server\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: Waiting for cloud-init to finish...\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] Started with user data set to:\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] --clustername CassandraCluster --totalnodes 3 --version=community --release=2.1.8\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] Using user data:\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] --clustername CassandraCluster --totalnodes 3 --version=community --release=2.1.8\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] Using instance type: c3.8xlarge\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] meta-data:instance-type: c3.8xlarge\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] meta-data:local-ipv4: 172.31.35.116\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] meta-data:public-hostname: ec2-54-201-39-228.us-west-2.compute.amazonaws.com\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] meta-data:ami-launch-index: 1\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] meta-data:reservation-id: r-5f597a9a\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] Using cluster name: CassandraCluster\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] Using cluster size: 3\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] Using seed indexes: [0, 3, 3]\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] Reflector loop...\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] 03/15/16-07:03:36 Reflector: Received 1 of 1 responses from: [u'172.31.35.115']\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_ami_setup[1253]: [INFO] scylla.yaml configured.\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_io_setup[2556]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_io_setup[2556]: Dload  Upload   Total   Spent    Left  Speed\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_io_setup[2556]: [155B blob data]\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_io_setup[2556]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_io_setup[2556]: Dload  Upload   Total   Spent    Left  Speed\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_io_setup[2556]: [155B blob data]\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_io_setup[2556]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_io_setup[2556]: Dload  Upload   Total   Spent    Left  Speed\r\nMar 15 07:03:36 ip-172-31-35-116 scylla_io_setup[2556]: [155B blob data]\r\nMar 15 07:03:36 ip-172-31-35-116 sudo[2581]:   scylla : TTY=unknown ; PWD=/var/lib/scylla ; USER=root ; COMMAND=/usr/lib/scylla/scylla_prepare\r\nMar 15 07:03:40 ip-172-31-35-116 scylla[2617]: terminate called after throwing an instance of 'std::runtime_error'\r\nMar 15 07:03:40 ip-172-31-35-116 scylla[2617]: what():  Invalid number of IO queues. Asked for 32. Minimum value is 2, maximum 31\r\nMar 15 07:03:40 ip-172-31-35-116 systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT\r\nMar 15 07:03:40 ip-172-31-35-116 sudo[2626]:   scylla : TTY=unknown ; PWD=/var/lib/scylla ; USER=root ; COMMAND=/usr/lib/scylla/scylla_stop\r\nMar 15 07:03:40 ip-172-31-35-116 systemd[1]: Job scylla-jmx.service/start failed with result 'dependency'.\r\nMar 15 07:03:40 ip-172-31-35-116 systemd[1]: Unit scylla-server.service entered failed state.\r\nMar 15 07:03:40 ip-172-31-35-116 systemd[1]: scylla-server.service failed.\r\nMar 15 07:03:40 ip-172-31-35-116 systemd[1]: scylla-server.service holdoff time over, scheduling restart.\r\nMar 15 07:03:40 ip-172-31-35-116 sudo[2633]:   scylla : TTY=unknown ; PWD=/var/lib/scylla ; USER=root ; COMMAND=/usr/lib/scylla/scylla_prepare\r\nMar 15 07:03:41 ip-172-31-35-116 scylla[2670]: terminate called after throwing an instance of 'std::runtime_error'\r\nMar 15 07:03:41 ip-172-31-35-116 scylla[2670]: what():  Invalid number of IO queues. Asked for 32. Minimum value is 2, maximum 31\r\nMar 15 07:03:41 ip-172-31-35-116 systemd[1]: scylla-server.service: main process exited, code=dumped, status=6/ABRT\r\nMar 15 07:03:41 ip-172-31-35-116 sudo[2673]:   scylla : TTY=unknown ; PWD=/var/lib/scylla ; USER=root ; COMMAND=/usr/lib/scylla/scylla_stop\r\nMar 15 07:03:41 ip-172-31-35-116 systemd[1]: Job scylla-jmx.service/start failed with result 'dependency'.\r\nMar 15 07:03:41 ip-172-31-35-116 systemd[1]: Unit scylla-server.service entered failed state.\r\nMar 15 07:03:41 ip-172-31-35-116 systemd[1]: scylla-server.service failed.\r\nMar 15 07:03:41 ip-172-31-35-116 systemd[1]: scylla-server.service holdoff time over, scheduling restart.\r\n...\r\n```\r\n\r\ncoredump\r\n```\r\n$ coredumpctl info 2617\r\n           PID: 2617 (scylla)\r\n           UID: 994 (scylla)\r\n           GID: 1001 (scylla)\r\n        Signal: 6 (ABRT)\r\n     Timestamp: Tue 2016-03-15 07:03:40 UTC (25min ago)\r\n  Command Line: /usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --collectd-address=127.0.0.1:25826 --collectd=1 --collectd-poll-period 3000 --network-stack \r\n    Executable: /usr/bin/scylla\r\n Control Group: /system.slice/scylla-server.service\r\n          Unit: scylla-server.service\r\n         Slice: system.slice\r\n       Boot ID: 9e400ddb6da048ce81fa10a759d4a3ba\r\n    Machine ID: 8ea3e6ea03b64d018847fcdf669935d4\r\n      Hostname: ip-172-31-35-116\r\n      Coredump: /var/lib/systemd/coredump/core.scylla.994.9e400ddb6da048ce81fa10a759d4a3ba.2617.1458025420000000.xz\r\n       Message: Process 2617 (scylla) of user 994 dumped core.\r\n                \r\n                Stack trace of thread 2617:\r\n                #0  0x00007fd4fef4f5f7 raise (libc.so.6)\r\n                #1  0x00007fd4fef50ce8 abort (libc.so.6)\r\n                #2  0x00000000014d58fd _ZN9__gnu_cxx27__verbose_terminate_handlerEv (scylla)\r\n                #3  0x00000000014402f6 _ZN10__cxxabiv111__terminateEPFvvE (scylla)\r\n                #4  0x0000000001440341 _ZSt9terminatev (scylla)\r\n                #5  0x0000000001440449 __cxa_throw (scylla)\r\n                #6  0x000000000054a305 allocate_io_queues (scylla)\r\n                #7  0x0000000000516ebc _ZN3smp9configureEN5boost15program_options13variables_mapE (scylla)\r\n                #8  0x000000000056bc4c _ZN12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla)\r\n                #9  0x00000000004708d8 main (scylla)\r\n                #10 0x00007fd4fef3bb15 __libc_start_main (libc.so.6)\r\n                #11 0x00000000004c74f5 _start (scylla)\r\n```"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/198870087","html_url":"https://github.com/scylladb/scylla/issues/1043#issuecomment-198870087","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/1043","id":198870087,"user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"created_at":"2016-03-20T08:07:58Z","updated_at":"2016-03-20T08:07:58Z","body":"openned a new issue #1062"}}