{"action":"created","issue":{"url":"https://api.github.com/repos/matrix-org/synapse/issues/1760","repository_url":"https://api.github.com/repos/matrix-org/synapse","labels_url":"https://api.github.com/repos/matrix-org/synapse/issues/1760/labels{/name}","comments_url":"https://api.github.com/repos/matrix-org/synapse/issues/1760/comments","events_url":"https://api.github.com/repos/matrix-org/synapse/issues/1760/events","html_url":"https://github.com/matrix-org/synapse/issues/1760","id":198865021,"node_id":"MDU6SXNzdWUxOTg4NjUwMjE=","number":1760,"title":"Forward extremities accumulate and lead to poor performance","user":{"login":"ara4n","id":1294269,"node_id":"MDQ6VXNlcjEyOTQyNjk=","avatar_url":"https://avatars2.githubusercontent.com/u/1294269?v=4","gravatar_id":"","url":"https://api.github.com/users/ara4n","html_url":"https://github.com/ara4n","followers_url":"https://api.github.com/users/ara4n/followers","following_url":"https://api.github.com/users/ara4n/following{/other_user}","gists_url":"https://api.github.com/users/ara4n/gists{/gist_id}","starred_url":"https://api.github.com/users/ara4n/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ara4n/subscriptions","organizations_url":"https://api.github.com/users/ara4n/orgs","repos_url":"https://api.github.com/users/ara4n/repos","events_url":"https://api.github.com/users/ara4n/events{/privacy}","received_events_url":"https://api.github.com/users/ara4n/received_events","type":"User","site_admin":false},"labels":[{"id":512032222,"node_id":"MDU6TGFiZWw1MTIwMzIyMjI=","url":"https://api.github.com/repos/matrix-org/synapse/labels/federation-meltdown","name":"federation-meltdown","color":"b60205","default":false},{"id":477045093,"node_id":"MDU6TGFiZWw0NzcwNDUwOTM=","url":"https://api.github.com/repos/matrix-org/synapse/labels/p1","name":"p1","color":"d93f0b","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":25,"created_at":"2017-01-05T02:56:26Z","updated_at":"2018-10-09T16:11:47Z","closed_at":null,"author_association":"MEMBER","body":"TLDR: To determine if you are affected by this problem, run the following query:\r\n\r\n```\r\nselect room_id, count(*) c from event_forward_extremities group by room_id order by c desc limit 20;\r\n```\r\n\r\nAny rows showing a count of more than a handful (say 10) are cause for concern. You can probably gain some respite by running the query at https://github.com/matrix-org/synapse/issues/1760#issuecomment-379183539.\r\n\r\n---\r\nWhilst investigating the cause of heap usage spikes in synapse, correlating jumps in RSZ with logs showed that 'resolving state for !curbaf with 49 groups' loglines took ages to execute and would temporarily take loads of heap (resulting in a permenant hike in RSZ, as python is bad at reclaiming heap).\r\n\r\nOn looking at the groups being resolved, it turns out that these were the extremities of the current room, and whenever the synapse queries the current room state, it has to merge these all together, whose implementation is currently very slow.  To clear the extremities, one has to talk in the room (each message 'heals' 10 extremities, as max prev-events for a message is 10).\r\n\r\nProblems here are:\r\n * [ ] Why are we accumulating so many extremities?  I assume it's whenever there's some downtime the graph breaks, leaving a dangling node.\r\n * [ ] Is there a way to stop them accumulating by healing or discarding them on launch (e.g. by sending a null healing event into the room)?\r\n * [ ] Why is state resolution so incredibly heavy?  There should hardly be any conflicting state here, unless the bifurcation has been going on for months.  Is it because to auth potential conflicts we have to load all auth events, which include every m.room.member \r\n * [ ] Logs of a state resolution happening from arasphere at DEBUG show lots of thrashing on the rejections table too.\r\n * [ ] We're also seeing ominous pauses in the logging of requests which resolve state, as if there's some lock we're contending for.  (This might be the same as #1774)\r\n * [ ] Can we just insert dummy nodes in our local copy of the DAG after doing a successful state resolution, to avoid having to constantly re-calculate it or rely on naive caching?"},"comment":{"url":"https://api.github.com/repos/matrix-org/synapse/issues/comments/428254242","html_url":"https://github.com/matrix-org/synapse/issues/1760#issuecomment-428254242","issue_url":"https://api.github.com/repos/matrix-org/synapse/issues/1760","id":428254242,"node_id":"MDEyOklzc3VlQ29tbWVudDQyODI1NDI0Mg==","user":{"login":"turt2live","id":1190097,"node_id":"MDQ6VXNlcjExOTAwOTc=","avatar_url":"https://avatars2.githubusercontent.com/u/1190097?v=4","gravatar_id":"","url":"https://api.github.com/users/turt2live","html_url":"https://github.com/turt2live","followers_url":"https://api.github.com/users/turt2live/followers","following_url":"https://api.github.com/users/turt2live/following{/other_user}","gists_url":"https://api.github.com/users/turt2live/gists{/gist_id}","starred_url":"https://api.github.com/users/turt2live/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/turt2live/subscriptions","organizations_url":"https://api.github.com/users/turt2live/orgs","repos_url":"https://api.github.com/users/turt2live/repos","events_url":"https://api.github.com/users/turt2live/events{/privacy}","received_events_url":"https://api.github.com/users/turt2live/received_events","type":"User","site_admin":false},"created_at":"2018-10-09T16:11:47Z","updated_at":"2018-10-09T16:11:47Z","author_association":"MEMBER","body":"Yes."}}
{"actor":{"display_login":"turt2live"}}