{"action":"created","issue":{"url":"https://api.github.com/repos/snarfed/bridgy/issues/624","repository_url":"https://api.github.com/repos/snarfed/bridgy","labels_url":"https://api.github.com/repos/snarfed/bridgy/issues/624/labels{/name}","comments_url":"https://api.github.com/repos/snarfed/bridgy/issues/624/comments","events_url":"https://api.github.com/repos/snarfed/bridgy/issues/624/events","html_url":"https://github.com/snarfed/bridgy/issues/624","id":135909483,"number":624,"title":"optimize OPD refetch","user":{"login":"snarfed","id":778068,"avatar_url":"https://avatars.githubusercontent.com/u/778068?v=3","gravatar_id":"","url":"https://api.github.com/users/snarfed","html_url":"https://github.com/snarfed","followers_url":"https://api.github.com/users/snarfed/followers","following_url":"https://api.github.com/users/snarfed/following{/other_user}","gists_url":"https://api.github.com/users/snarfed/gists{/gist_id}","starred_url":"https://api.github.com/users/snarfed/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/snarfed/subscriptions","organizations_url":"https://api.github.com/users/snarfed/orgs","repos_url":"https://api.github.com/users/snarfed/repos","events_url":"https://api.github.com/users/snarfed/events{/privacy}","received_events_url":"https://api.github.com/users/snarfed/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/snarfed/bridgy/labels/listen","name":"listen","color":"207de5"},{"url":"https://api.github.com/repos/snarfed/bridgy/labels/now","name":"now","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2016-02-24T00:19:40Z","updated_at":"2016-02-28T16:51:16Z","closed_at":null,"body":"we've started seeing waves of poll latency spikes in the last week or two. anecdotally, it looks like much of the time is spent in OPD refetch. even without the latency spikes, OPD may be one of the main contributors to load. background in https://github.com/snarfed/bridgy/issues/578#issuecomment-186350518.\r\n\r\nhere are some optimization ideas:\r\n\r\n1. [x] ignore syndication links to silos that aren't the one currently being polled. right now we HEAD them to resolve redirects.\r\n1. [x] more aggressive: ignore all syndication links *except* the current silo. this is a bit scary since we'd stop supporting shortlinks for syndication links and alternate domains like flic.kr. not sure how widespread either of those are.\r\n1. [x] similar to poll frequency, ratchet down refetch frequency when we haven't seen a new syndication link for a given user in N days, weeks, etc.\r\n1. [ ] remember if we've seen a syndication link in a feed before. if we have, don't bother ever fetching permalinks, at least for a while.\r\n1. [ ] merge refetch for the same domain across multiple sources.\r\n  1. [ ] easy first step: let sources detect and store syndication links for other silos.\r\n  1. [ ] more ambitious: decouple refetch from poll and key it on domain so we don't duplicate refetches per domain.\r\n"},"comment":{"url":"https://api.github.com/repos/snarfed/bridgy/issues/comments/189901619","html_url":"https://github.com/snarfed/bridgy/issues/624#issuecomment-189901619","issue_url":"https://api.github.com/repos/snarfed/bridgy/issues/624","id":189901619,"user":{"login":"kylewm","id":950127,"avatar_url":"https://avatars.githubusercontent.com/u/950127?v=3","gravatar_id":"","url":"https://api.github.com/users/kylewm","html_url":"https://github.com/kylewm","followers_url":"https://api.github.com/users/kylewm/followers","following_url":"https://api.github.com/users/kylewm/following{/other_user}","gists_url":"https://api.github.com/users/kylewm/gists{/gist_id}","starred_url":"https://api.github.com/users/kylewm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kylewm/subscriptions","organizations_url":"https://api.github.com/users/kylewm/orgs","repos_url":"https://api.github.com/users/kylewm/repos","events_url":"https://api.github.com/users/kylewm/events{/privacy}","received_events_url":"https://api.github.com/users/kylewm/received_events","type":"User","site_admin":false},"created_at":"2016-02-28T16:51:16Z","updated_at":"2016-02-28T16:51:16Z","body":"Interesting, it seems like the vast majority of time in the parser is spent in `backcompat.apply_rules`"}}