{"action":"created","issue":{"url":"https://api.github.com/repos/mmcdole/gofeed/issues/68","repository_url":"https://api.github.com/repos/mmcdole/gofeed","labels_url":"https://api.github.com/repos/mmcdole/gofeed/issues/68/labels{/name}","comments_url":"https://api.github.com/repos/mmcdole/gofeed/issues/68/comments","events_url":"https://api.github.com/repos/mmcdole/gofeed/issues/68/events","html_url":"https://github.com/mmcdole/gofeed/pull/68","id":192296141,"number":68,"title":"Remember to close non-2xx responses","user":{"login":"Kegsay","id":7190048,"avatar_url":"https://avatars.githubusercontent.com/u/7190048?v=3","gravatar_id":"","url":"https://api.github.com/users/Kegsay","html_url":"https://github.com/Kegsay","followers_url":"https://api.github.com/users/Kegsay/followers","following_url":"https://api.github.com/users/Kegsay/following{/other_user}","gists_url":"https://api.github.com/users/Kegsay/gists{/gist_id}","starred_url":"https://api.github.com/users/Kegsay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Kegsay/subscriptions","organizations_url":"https://api.github.com/users/Kegsay/orgs","repos_url":"https://api.github.com/users/Kegsay/repos","events_url":"https://api.github.com/users/Kegsay/events{/privacy}","received_events_url":"https://api.github.com/users/Kegsay/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-11-29T14:29:03Z","updated_at":"2016-12-02T21:17:00Z","closed_at":"2016-12-02T02:13:25Z","pull_request":{"url":"https://api.github.com/repos/mmcdole/gofeed/pulls/68","html_url":"https://github.com/mmcdole/gofeed/pull/68","diff_url":"https://github.com/mmcdole/gofeed/pull/68.diff","patch_url":"https://github.com/mmcdole/gofeed/pull/68.patch"},"body":"I'm using gofeed to constantly poll some Atom/RSS feeds. There was a memory leak in my project caused by HTTP responses in this library not being closed: I was using `ParseURL`. It turns out any non 2xx response was leaking, because `Close()` was not being called on it.\r\n\r\nThis PR fixes this and also closes responses even if there was a non-nil error. The presence of an `err` does not mean an absence of `resp`, as redirection failures will produce both `err` and `resp` as non-nil.\r\n"},"comment":{"url":"https://api.github.com/repos/mmcdole/gofeed/issues/comments/264564814","html_url":"https://github.com/mmcdole/gofeed/pull/68#issuecomment-264564814","issue_url":"https://api.github.com/repos/mmcdole/gofeed/issues/68","id":264564814,"user":{"login":"Kegsay","id":7190048,"avatar_url":"https://avatars.githubusercontent.com/u/7190048?v=3","gravatar_id":"","url":"https://api.github.com/users/Kegsay","html_url":"https://github.com/Kegsay","followers_url":"https://api.github.com/users/Kegsay/followers","following_url":"https://api.github.com/users/Kegsay/following{/other_user}","gists_url":"https://api.github.com/users/Kegsay/gists{/gist_id}","starred_url":"https://api.github.com/users/Kegsay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Kegsay/subscriptions","organizations_url":"https://api.github.com/users/Kegsay/orgs","repos_url":"https://api.github.com/users/Kegsay/repos","events_url":"https://api.github.com/users/Kegsay/events{/privacy}","received_events_url":"https://api.github.com/users/Kegsay/received_events","type":"User","site_admin":false},"created_at":"2016-12-02T21:17:00Z","updated_at":"2016-12-02T21:17:00Z","body":"> Perhaps you might look at using bloom filter to make it so you don't have to keep the whole set of GUIDs in memory for any given feed. Instead, you can just keep a small bloom filter per feed in memory.\r\n\r\nKeeping the entire set of GUIDs in memory isn't really a problem, given I have ultimate bounds on the GUIDs I end up holding onto via a heuristic. I only need to allocate for them when I parse the feed and can then dispose of it afterwards, and that generally works well.\r\n\r\n> Feeds typically cull older entries after a time too -- so I would be sure to cull old/unused entries that you have persisted once they fall off the feed.\r\n\r\nIf only it were that simple. I've seen feeds in the wild (Yahoo feeds for example) which have the peculiar property that they flip-flop how many items they return, possibly because they end up hitting different backend servers. You end up seeing something like:\r\n```\r\nRequest 1     Request 2      Request 3      Request 4\r\n   A             A               A             A\r\n   B             B               B             B\r\n   C             C               C             C\r\n                 D                             D\r\n                 E                             E\r\n```\r\n\r\nMy original naive solution was to remember the GUIDs for exactly the number of items returned, assuming that feeds would *consistently* return that number of items. This resulted in sending duplicate messages for the above scenario, as on request 3 I would forget that I saw D and E. I ended up applying a heuristic which was to:\r\n - Keep 2N GUIDs (where N=the maximum number of items in the feed **ever seen**)\r\n - If 2N > 1000 then truncate at 1000. This keeps the ultimate bound of GUIDs needed to be stored per feed.\r\n - If the number of GUIDs in the *current* poll is < 2N, then also remember the most recent unique GUIDs from the last poll to fill up 2N.\r\n\r\nThis ends up persisting (for the aforementioned requests):\r\n - 1: [A, B, C, _, _, _]\r\n - 2: [A, B, C, D, E, _, _, _, _, _]\r\n - 3: [A, B, C, D, E, _, _, _, _, _]\r\n - 4: [A, B, C, D, E, _, _, _, _, _]\r\n\r\nThis seems to have worked quite well for the bizarre feeds I'm encountering.\r\n\r\n> they assign a date to them in their own database at the time of first parsing that particular item\r\n\r\nThis is an excellent idea, I didn't think of that!\r\n\r\n"}}
{"actor":{"display_login":"Kegsay"}}