{"action":"created","issue":{"url":"https://api.github.com/repos/matrix-org/synapse/issues/3188","repository_url":"https://api.github.com/repos/matrix-org/synapse","labels_url":"https://api.github.com/repos/matrix-org/synapse/issues/3188/labels{/name}","comments_url":"https://api.github.com/repos/matrix-org/synapse/issues/3188/comments","events_url":"https://api.github.com/repos/matrix-org/synapse/issues/3188/events","html_url":"https://github.com/matrix-org/synapse/issues/3188","id":320106710,"number":3188,"title":"Frequent crashes and spikes in RAM use, several critical errors in logs","user":{"login":"JoKeyser","id":8722846,"avatar_url":"https://avatars1.githubusercontent.com/u/8722846?v=4","gravatar_id":"","url":"https://api.github.com/users/JoKeyser","html_url":"https://github.com/JoKeyser","followers_url":"https://api.github.com/users/JoKeyser/followers","following_url":"https://api.github.com/users/JoKeyser/following{/other_user}","gists_url":"https://api.github.com/users/JoKeyser/gists{/gist_id}","starred_url":"https://api.github.com/users/JoKeyser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoKeyser/subscriptions","organizations_url":"https://api.github.com/users/JoKeyser/orgs","repos_url":"https://api.github.com/users/JoKeyser/repos","events_url":"https://api.github.com/users/JoKeyser/events{/privacy}","received_events_url":"https://api.github.com/users/JoKeyser/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-05-03T22:53:47Z","updated_at":"2018-05-26T10:18:40Z","closed_at":null,"author_association":"NONE","body":"### Description\r\nSince a few days, I notice frequent disconnects to my homeserver; mainly on Riot Android, but also some on Riot desktop.\r\n\r\nIn Synapse's logs, I find several errors that may hint at the problem(s), see below. Sorry that it's a fairly long dump, but I can't judge what parts are useful. Let me know if I can check/add more.\r\n\r\n\r\n```\r\n2018-05-03 23:02:00,056 - twisted - 131 - CRITICAL - - Unexpected exception from synapse.http.client.FileBodyProducer.stopProducing\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/protocols/policies.py\", line 125, in connectionLost\r\n    self.wrappedProtocol.connectionLost(reason)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/web/_newclient.py\", line 964, in dispatcher\r\n    return func(*args, **kwargs)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/web/_newclient.py\", line 1634, in _connectionLost_TRANSMITTING\r\n    self._currentRequest.stopWriting()\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/web/_newclient.py\", line 865, in stopWriting\r\n    _callAppFunction(self.bodyProducer.stopProducing)\r\n--- <exception caught here> ---\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/web/_newclient.py\", line 194, in _callAppFunction\r\n    function()\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/web/client.py\", line 1086, in stopProducing\r\n    self._task.stop()\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/task.py\", line 497, in stop\r\n    self._checkFinish()\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/task.py\", line 507, in _checkFinish\r\n    raise self._completionState\r\ntwisted.internet.task.TaskStopped: \r\n```\r\n\r\nThere are a bunch of ones like this:\r\n```\r\n2018-05-03 23:29:58,103 - synapse.http.server - 191 - ERROR - GET-638- Failed handle request synapse.rest.media.v1.download_resource._async_render_GET on <synapse.rest.media.v1.download_resource.DownloadResource instance at 0x7faf9a3e1d88>: <SynapseRequest at 0x7faf0ecca710 method=GET uri=/_matrix/media/v1/download/matrix.org/jdVtvZBLTBLNCdnSXfkfOibx clientproto=HTTP/1.1 site=8008>: Traceback (most recent call last):\r\nFailure: exceptions.Exception: Consumer asked us to stop producing\r\n2018-05-03 23:29:58,107 - twisted - 131 - CRITICAL - GET-638- Unhandled Error\r\nTraceback (most recent call last):\r\nFailure: exceptions.RuntimeError: Producer was not unregistered for /_matrix/media/v1/download/matrix.org/jdVtvZBLTBLNCdnSXfkfOibx\r\n```\r\n\r\n\r\n```\r\n2018-05-03 23:50:30,669 - synapse.http.server - 191 - ERROR - GET-148- Failed handle request synapse.rest.media.v1.thumbnail_resource._async_render_GET on <synapse.rest.media.v1.thumbnail_resource.ThumbnailResource instance at 0x7fc773a48a28>: <SynapseRequest at 0x7fc73817d200 method=GET uri=/_matrix/media/v1/thumbnail/matrix.org/YqvlPbTsQhGREHeVRXWsHqgL?width=36&height=36&method=crop clientproto=HTTP/1.1 site=8008>: Traceback (most recent call last):\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 653, in _runCallbacks\r\n    current.result = callback(current.result, *args, **kw)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1442, in gotResult\r\n    _inlineCallbacks(r, g, deferred)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n--- <exception caught here> ---\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/http/server.py\", line 165, in wrapped_request_handler\r\n    yield request_handler(self, request)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/rest/media/v1/thumbnail_resource.py\", line 80, in _async_render_GET\r\n    width, height, method, m_type\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/rest/media/v1/thumbnail_resource.py\", line 233, in _respond_remote_thumbnail\r\n    media_info = yield self.media_repo.get_remote_media_info(server_name, media_id)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/rest/media/v1/media_repository.py\", line 273, in get_remote_media_info\r\n    server_name, media_id,\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/rest/media/v1/media_repository.py\", line 323, in _get_remote_me\r\n    server_name, media_id, file_id\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/rest/media/v1/media_repository.py\", line 429, in _download_remo\r\n    filesystem_id=file_id,\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/_base.py\", line 424, in _simple_insert\r\n    self._simple_insert_txn, table, values,\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/_base.py\", line 320, in runInteraction\r\n    result = yield self.runWithConnection(inner_func, *args, **kwargs)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/_base.py\", line 365, in runWithConnection\r\n    inner_func, *args, **kwargs\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py\", line 250, in inContext\r\n    result = inContext.theWork()\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py\", line 266, in <lambda>\r\n    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/context.py\", line 122, in callWithContext\r\n    return self.currentContext().callWithContext(ctx, func, *args, **kw)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/context.py\", line 85, in callWithContext\r\n    return func(*args,**kw)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py\", line 307, in _runWithConnection\r\n    compat.reraise(excValue, excTraceback)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py\", line 298, in _runWithConnection\r\n    result = func(conn, *args, **kw)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/_base.py\", line 361, in inner_func\r\n    return func(conn, *args, **kwargs)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/_base.py\", line 316, in inner_func\r\n    func, *args, **kwargs\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/_base.py\", line 241, in _new_transaction\r\n    r = func(txn, *args, **kwargs)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/_base.py\", line 444, in _simple_insert_txn\r\n    txn.execute(sql, vals)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/_base.py\", line 82, in execute\r\n    self._do_execute(self.txn.execute, sql, *args)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/_base.py\", line 112, in _do_execute\r\n    sql, *args\r\n```\r\n\r\n```\r\n2018-05-04 00:14:07,385 - synapse.federation.federation_server - 620 - ERROR - PUT-1236- Failed to handle edu 'm.receipt'\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/federation/federation_server.py\", line 616, in on_edu\r\n    yield handler(origin, content)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/handlers/receipts.py\", line 84, in _received_remote_receipt\r\n    yield self._handle_new_receipts(receipts)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/handlers/receipts.py\", line 101, in _handle_new_receipts\r\n    room_id, receipt_type, user_id, event_ids, data\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/receipts.py\", line 456, in insert_receipt\r\n    stream_id=stream_id,\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1386, in _inlineCallbacks\r\n    result = g.send(result)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/_base.py\", line 323, in runInteraction\r\n    after_callback(*after_args, **after_kwargs)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/storage/receipts.py\", line 308, in _invalidate_get_users_with_receipts_in_room\r\n    if user_id in res:\r\nTypeError: argument of type 'ObservableDeferred' is not iterable\r\n```\r\n\r\n```\r\n2018-05-04 00:19:13,429 - synapse.federation.transport.server - 188 - ERROR - PUT-1913- authenticate_request failed\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/federation/transport/server.py\", line 181, in new_func\r\n    origin = yield authenticator.authenticate_request(request, content)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1384, in _inlineCallbacks\r\n    result = result.throwExceptionIntoGenerator(g)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/failure.py\", line 408, in throwExceptionIntoGenerator\r\n    return g.throw(self.type, self.value, self.tb)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/federation/transport/server.py\", line 145, in authenticate_request\r\n    yield self.keyring.verify_json_for_server(origin, json_request)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py\", line 1386, in _inlineCallbacks\r\n    result = g.send(result)\r\n  File \"/usr/lib/python2.7/dist-packages/synapse/crypto/keyring.py\", line 768, in _handle_key_deferred\r\n    Codes.UNAUTHORIZED,\r\nSynapseError: 401: Invalid signature for server MY_DOMAIN with key ed25519:a_RBEL\r\n```\r\n\r\n```\r\n2018-05-04 00:23:36,277 - twisted - 131 - CRITICAL - - Unhandled Error\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/log.py\", line 103, in callWithLogger\r\n    return callWithContext({\"system\": lp}, func, *args, **kw)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/log.py\", line 86, in callWithContext\r\n    return context.call({ILogContext: newCtx}, func, *args, **kw)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/context.py\", line 122, in callWithContext\r\n    return self.currentContext().callWithContext(ctx, func, *args, **kw)\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/python/context.py\", line 85, in callWithContext\r\n    return func(*args,**kw)\r\n--- <exception caught here> ---\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py\", line 619, in _doReadOrWrite\r\n    why = selectable.doWrite()\r\n  File \"/usr/lib/python2.7/dist-packages/twisted/internet/base.py\", line 1168, in doWrite\r\n    \"doWrite called on a %s\" % reflect.qual(self.__class__))\r\nexceptions.RuntimeError: doWrite called on a twisted.internet.udp.Port\r\n```\r\n\r\n### Steps to reproduce\r\nRunning Synapse... :/.\r\n\r\n### Version information\r\n\r\nIf not matrix.org:\r\n- **Version**:  Synapse/0.27.2 and 0.28.1.\r\n- **Install method**: Debian's apt.\r\n- **Platform**:  Debian buster, amd64, on dedicated hardware with 4 cores and 4 GiB, http://www.pcengines.ch/apu2c4.htm.\r\n"},"comment":{"url":"https://api.github.com/repos/matrix-org/synapse/issues/comments/392251903","html_url":"https://github.com/matrix-org/synapse/issues/3188#issuecomment-392251903","issue_url":"https://api.github.com/repos/matrix-org/synapse/issues/3188","id":392251903,"user":{"login":"JoKeyser","id":8722846,"avatar_url":"https://avatars1.githubusercontent.com/u/8722846?v=4","gravatar_id":"","url":"https://api.github.com/users/JoKeyser","html_url":"https://github.com/JoKeyser","followers_url":"https://api.github.com/users/JoKeyser/followers","following_url":"https://api.github.com/users/JoKeyser/following{/other_user}","gists_url":"https://api.github.com/users/JoKeyser/gists{/gist_id}","starred_url":"https://api.github.com/users/JoKeyser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoKeyser/subscriptions","organizations_url":"https://api.github.com/users/JoKeyser/orgs","repos_url":"https://api.github.com/users/JoKeyser/repos","events_url":"https://api.github.com/users/JoKeyser/events{/privacy}","received_events_url":"https://api.github.com/users/JoKeyser/received_events","type":"User","site_admin":false},"created_at":"2018-05-26T10:18:40Z","updated_at":"2018-05-26T10:18:40Z","author_association":"NONE","body":"@neilisfragile thanks for your help. I'm the only user on my server, so everything I do is in federated rooms. The breakdown of user number of joined room is:  About 20 1:1 chats, 3x with 10-100 users, 7x 100-1k users, 3x with 2k-7k, and 1x with 15k. I'd be happy to leave the bigger ones to see if it helps, probably there's a server command to do so?\r\n\r\nI've set `SYNAPSE_CACHE_FACTOR = 1`, but even after 12+ hours I still cannot connect, and it keeps crashing (oom_reaper). The only apparent improvement is that only one kind of CRITICAL errors seems to persist, namely `Unhandled error in Deferred`.\r\n\r\nDue to the lack of other reports like mine, I suspect it's the unique state of my server. It was working fine for months, and I didn't add big rooms recently.\r\nWould there be a way to start fresh(er) without losing my joined rooms etc?"}}
{"actor":{"display_login":"JoKeyser"}}