{"action":"created","issue":{"url":"https://api.github.com/repos/csirtgadgets/bearded-avenger/issues/396","repository_url":"https://api.github.com/repos/csirtgadgets/bearded-avenger","labels_url":"https://api.github.com/repos/csirtgadgets/bearded-avenger/issues/396/labels{/name}","comments_url":"https://api.github.com/repos/csirtgadgets/bearded-avenger/issues/396/comments","events_url":"https://api.github.com/repos/csirtgadgets/bearded-avenger/issues/396/events","html_url":"https://github.com/csirtgadgets/bearded-avenger/issues/396","id":327357789,"node_id":"MDU6SXNzdWUzMjczNTc3ODk=","number":396,"title":"Searching stops working after a few hours (give or take) of use","user":{"login":"Rickerd0613","id":17521826,"node_id":"MDQ6VXNlcjE3NTIxODI2","avatar_url":"https://avatars0.githubusercontent.com/u/17521826?v=4","gravatar_id":"","url":"https://api.github.com/users/Rickerd0613","html_url":"https://github.com/Rickerd0613","followers_url":"https://api.github.com/users/Rickerd0613/followers","following_url":"https://api.github.com/users/Rickerd0613/following{/other_user}","gists_url":"https://api.github.com/users/Rickerd0613/gists{/gist_id}","starred_url":"https://api.github.com/users/Rickerd0613/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Rickerd0613/subscriptions","organizations_url":"https://api.github.com/users/Rickerd0613/orgs","repos_url":"https://api.github.com/users/Rickerd0613/repos","events_url":"https://api.github.com/users/Rickerd0613/events{/privacy}","received_events_url":"https://api.github.com/users/Rickerd0613/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-05-29T14:54:30Z","updated_at":"2018-06-08T19:56:42Z","closed_at":null,"author_association":"NONE","body":"Hello!\r\n\r\nI have been using CIF (primarily v2) for the past couple years (even have some commits into the CIF chrome extension). On this new CIFv3 installation, I am running into a weird problem I can't quite fix.\r\n\r\n# Expected behavior and actual behavior.\r\nExpected behavior is when I search CIF for an indicator (either through the CLI or API) I get results back.\r\n\r\nActual behavior is after searching for a little while, the search will just hang. `cif -p` will not work as well.\r\n\r\n# Steps to reporduce the problem\r\nUse CIF for a period of time, usually a couple hours but sometimes as short as a few minutes.\r\n\r\n# Relevant logs as a result of the actual behavior\r\n\r\n> May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: [2018-05-29 14:17:02,845] ERROR in app: Exception on /ping [GET]\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: Traceback (most recent call last):\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/flask/app.py\", line 1982, in wsgi_app\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     response = self.full_dispatch_request()\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/flask/app.py\", line 1614, in full_dispatch_request\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     rv = self.handle_user_exception(e)\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/flask_cors/extension.py\", line 161, in wrapped_function\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     return cors_after_request(app.make_response(f(*args, **kwargs)))\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/flask/app.py\", line 1517, in handle_user_exception\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     reraise(exc_type, exc_value, tb)\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/flask/app.py\", line 1612, in full_dispatch_request\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     rv = self.dispatch_request()\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/flask/app.py\", line 1598, in dispatch_request\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     return self.view_functions[rule.endpoint](**req.view_args)\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/flask/views.py\", line 84, in view\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     return self.dispatch_request(*args, **kwargs)\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/flask/views.py\", line 149, in dispatch_request\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     return meth(*args, **kwargs)\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/cif-3.0.0rc2-py2.7.egg/cif/httpd/views/ping.py\", line 24, in get\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     r = Client(remote, pull_token()).ping(write=write)\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/cifsdk/client/zeromq.py\", line 110, in ping\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     return self._send(Msg.PING_WRITE)\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/cifsdk/client/zeromq.py\", line 106, in _send\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     return self._recv(decode=decode)\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/cifsdk/client/zeromq.py\", line 53, in _recv\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     mtype, data = Msg().recv(self.socket)\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/cifsdk/msg.py\", line 72, in recv\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     m = s.recv_multipart()\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"/usr/local/lib/python2.7/dist-packages/zmq/sugar/socket.py\", line 395, in recv_multipart\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     parts = [self.recv(flags, copy=copy, track=track)]\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"zmq/backend/cython/socket.pyx\", line 693, in zmq.backend.cython.socket.Socket.recv\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"zmq/backend/cython/socket.pyx\", line 727, in zmq.backend.cython.socket.Socket.recv\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"zmq/backend/cython/socket.pyx\", line 150, in zmq.backend.cython.socket._recv_copy\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"zmq/backend/cython/socket.pyx\", line 145, in zmq.backend.cython.socket._recv_copy\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:   File \"zmq/backend/cython/checkrc.pxd\", line 19, in zmq.backend.cython.checkrc._check_rc\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]:     raise Again(errno)\r\nMay 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: Again: Resource temporarily unavailable\r\n\r\n# Specifications like the version of the project, operating system, or hardware.\r\nCIFv3 3.0.0rc2.\r\nOS Ubuntu Server 16.04\r\nAWS EC2 m5.2xlarge\r\n\r\n# Other Notes\r\nWhen this happens, I am still able to access parts of the API like `/`, `/help`, and `/help/confidence`. Additionally, CIFv3 was installed using the easy button. Restarting the server fixes the problem for a short period of time.\r\n\r\nLet me know if there are any other logs I can provide you!\r\n"},"comment":{"url":"https://api.github.com/repos/csirtgadgets/bearded-avenger/issues/comments/395872611","html_url":"https://github.com/csirtgadgets/bearded-avenger/issues/396#issuecomment-395872611","issue_url":"https://api.github.com/repos/csirtgadgets/bearded-avenger/issues/396","id":395872611,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NTg3MjYxMQ==","user":{"login":"JustinAzoff","id":65812,"node_id":"MDQ6VXNlcjY1ODEy","avatar_url":"https://avatars2.githubusercontent.com/u/65812?v=4","gravatar_id":"","url":"https://api.github.com/users/JustinAzoff","html_url":"https://github.com/JustinAzoff","followers_url":"https://api.github.com/users/JustinAzoff/followers","following_url":"https://api.github.com/users/JustinAzoff/following{/other_user}","gists_url":"https://api.github.com/users/JustinAzoff/gists{/gist_id}","starred_url":"https://api.github.com/users/JustinAzoff/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JustinAzoff/subscriptions","organizations_url":"https://api.github.com/users/JustinAzoff/orgs","repos_url":"https://api.github.com/users/JustinAzoff/repos","events_url":"https://api.github.com/users/JustinAzoff/events{/privacy}","received_events_url":"https://api.github.com/users/JustinAzoff/received_events","type":"User","site_admin":false},"created_at":"2018-06-08T19:56:42Z","updated_at":"2018-06-08T19:56:42Z","author_association":"COLLABORATOR","body":"I think I see what is going on.  The logs show 2 interleaved errors. The first error was caught elsewhere in cif and wasn't harming anything.\r\n\r\nWith CIF_HUNTER_THREADS=0 everything is fine, but the hunter isn't catching the InvalidIndicator exception though, so if CIF_HUNTER_THREADS is 4, after 4 queries like that, all the hunters are dead and the router blocks.\r\n\r\nI just sent over https://github.com/csirtgadgets/bearded-avenger/pull/397  which should fix it."}}
{"actor":{"display_login":"JustinAzoff"}}