{"action":"created","issue":{"url":"https://api.github.com/repos/Metaswitch/astaire/issues/71","repository_url":"https://api.github.com/repos/Metaswitch/astaire","labels_url":"https://api.github.com/repos/Metaswitch/astaire/issues/71/labels{/name}","comments_url":"https://api.github.com/repos/Metaswitch/astaire/issues/71/comments","events_url":"https://api.github.com/repos/Metaswitch/astaire/issues/71/events","html_url":"https://github.com/Metaswitch/astaire/issues/71","id":184244301,"number":71,"title":"Astaire crashes under load","user":{"login":"rkd-msw","id":4363492,"avatar_url":"https://avatars.githubusercontent.com/u/4363492?v=3","gravatar_id":"","url":"https://api.github.com/users/rkd-msw","html_url":"https://github.com/rkd-msw","followers_url":"https://api.github.com/users/rkd-msw/followers","following_url":"https://api.github.com/users/rkd-msw/following{/other_user}","gists_url":"https://api.github.com/users/rkd-msw/gists{/gist_id}","starred_url":"https://api.github.com/users/rkd-msw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rkd-msw/subscriptions","organizations_url":"https://api.github.com/users/rkd-msw/orgs","repos_url":"https://api.github.com/users/rkd-msw/repos","events_url":"https://api.github.com/users/rkd-msw/events{/privacy}","received_events_url":"https://api.github.com/users/rkd-msw/received_events","type":"User","site_admin":false},"labels":[{"id":186335813,"url":"https://api.github.com/repos/Metaswitch/astaire/labels/bug","name":"bug","color":"fc2929","default":true},{"id":241899262,"url":"https://api.github.com/repos/Metaswitch/astaire/labels/cat:stability","name":"cat:stability","color":"009800","default":false},{"id":198973923,"url":"https://api.github.com/repos/Metaswitch/astaire/labels/high-priority","name":"high-priority","color":"ff0000","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-10-20T14:24:59Z","updated_at":"2016-11-17T17:16:56Z","closed_at":null,"body":"<!--\nThis page is for reporting issues with Project Clearwater. If you have a question, rather than a bug report, the mailing list at clearwater@lists.projectclearwater.org is a better place for it.\n\nTo give us the best chance of fixing the problem, we've suggested some information to give - please follow these guidelines if possible.\n\nDon't forget that you can attach logs and screenshots to Github issues - this may help us debug a problem.\n-->\n#### Symptoms\n\n<!-- What exactly went wrong? (If possible, copy and paste the exact output you saw) -->\n\nI was running Astaire under low load and it crashed with the following backtrace:\n\n```\n#0  0x00007fd065becc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56\n#1  0x00007fd065bf0028 in __GI_abort () at abort.c:89\n#2  0x000000000042842a in signal_handler (sig=11) at main.cpp:212\n#3  <signal handler called>\n#4  0x000000000042b4f3 in _M_lower_bound (this=<optimized out>, __k=<error reading variable: Cannot access memory at address 0x0>, __y=0x7fd0206969c0, __x=0x7fd0201c9910)\n    at /usr/include/c++/4.8/bits/stl_tree.h:1140\n#5  lower_bound (__k=<error reading variable: Cannot access memory at address 0x0>, this=0x7fd0206969b8) at /usr/include/c++/4.8/bits/stl_tree.h:879\n#6  lower_bound (__x=<error reading variable: Cannot access memory at address 0x0>, this=0x7fd0206969b8) at /usr/include/c++/4.8/bits/stl_map.h:864\n#7  operator[] (__k=<error reading variable: Cannot access memory at address 0x0>, this=0x7fd0206969b8) at /usr/include/c++/4.8/bits/stl_map.h:461\n#8  MemcachedBackend::get_replicas (this=this@entry=0x22b5190, vbucket=vbucket@entry=26, operation=operation@entry=MemcachedBackend::READ) at memcached_backend.cpp:284\n#9  0x000000000042bea8 in MemcachedBackend::read_data (this=0x22b5190, key=\"reg\\\\\\\\sip:2010000278@em7.cw-ngv.com\", data=\"\", cas=@0x7fbdfe3c7e28: 1) at memcached_backend.cpp:383\n#10 0x00000000004290dc in ProxyServer::handle_get (this=this@entry=0x22b5130, get_req=0x7fd02020aca0, connection=connection@entry=0x7fd038036d90) at proxy_server.cpp:330\n#11 0x0000000000429833 in ProxyServer::connection_thread_fn (this=0x22b5130, connection=0x7fd038036d90) at proxy_server.cpp:253\n#12 0x0000000000429a90 in ProxyServer::connection_thread_entry_point (params_arg=0x7fd03801d4b0) at proxy_server.cpp:224\n#13 0x00007fd066d17184 in start_thread (arg=0x7fbdfe3c8700) at pthread_create.c:312\n#14 0x00007fd065cb037d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111\n```\n\nI jumped to frame 8 and checked `_read_replicas`, and it looks corrupt:\n\n```\n(gdb) p _read_replicas\n$7 = std::vector of length 128, capacity 128 = {std::vector of length 0, capacity -17566630078711, std::vector of length 4550841, capacity 4550841 = {Cannot access memory at address 0x0\n```\n#### Impact\n\n<!-- What is this preventing you from doing? Does this stop Clearwater working, or stop some calls being processed? -->\n\nAstaire crashes. The biggest impact of that is likely to be a CPU% increase as we gather diags.\n#### Release and environment\n\n<!-- For example: the \"Doom\" release, whether you're on VMware or OpenStack, etc. -->\n\nPresent in the Onix release.\n"},"comment":{"url":"https://api.github.com/repos/Metaswitch/astaire/issues/comments/261308848","html_url":"https://github.com/Metaswitch/astaire/issues/71#issuecomment-261308848","issue_url":"https://api.github.com/repos/Metaswitch/astaire/issues/71","id":261308848,"user":{"login":"AlexHockey","id":4903205,"avatar_url":"https://avatars.githubusercontent.com/u/4903205?v=3","gravatar_id":"","url":"https://api.github.com/users/AlexHockey","html_url":"https://github.com/AlexHockey","followers_url":"https://api.github.com/users/AlexHockey/followers","following_url":"https://api.github.com/users/AlexHockey/following{/other_user}","gists_url":"https://api.github.com/users/AlexHockey/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexHockey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexHockey/subscriptions","organizations_url":"https://api.github.com/users/AlexHockey/orgs","repos_url":"https://api.github.com/users/AlexHockey/repos","events_url":"https://api.github.com/users/AlexHockey/events{/privacy}","received_events_url":"https://api.github.com/users/AlexHockey/received_events","type":"User","site_admin":false},"created_at":"2016-11-17T17:16:56Z","updated_at":"2016-11-17T17:16:56Z","body":"So normally we don't care too much about crashes during termination. This one was a bit interesting because it ended up with the core file being 73GB in size when unzipped. My guess is that when the `ProxyServer` got deleted we scribbled and accidentally allocated a shed-load of memory. But that could happen in any crash. I'm also not convinced it caused a 73GB file on disk - this would have been more than the total size of the disk (most of the file is zeros which I'm guessing the disk can optimize away). \r\n\r\nTL;DR I think this should be downgraded, but it might be worth scheduling the story to make astaire terminate cleanly. "}}
{"actor":{"display_login":"AlexHockey"}}