I'm trying to create an EPUB from all StackExchange RSS feeds (to use as a "podcast" to read on an offline ebook reader). This involves parsing the recent questions from all SE sites, which quickly leads to a "Too Many Requests from this IP" error.

This is what the script (a Calibre recipe) does:

from calibre.web.feeds.recipes import BasicNewsRecipe
import feedparser

def get_feeds(url):
        feed = feedparser.parse(url)
        entries = []
        entries.extend( feed["items"] )
        sorted_entries = sorted(entries, key=lambda entry: entry["title"])
        feeds = []
        for entry in sorted_entries:
                feeds.append( "%s/feeds" % entry["id"] )
        return feeds

class StackExchange(BasicNewsRecipe):
        title                 = u'StackExchange recent questions'
        oldest_article        = 10
        auto_cleanup          = True
        language              = 'en'
        __author__            = 'Raphaël Pinson'
        max_articles_per_feed = 100
        cover_url = u'http://i1.sndcdn.com/avatars-000011139586-8ft0oi-crop.jpg?04ad178'
        all_feeds = u'http://stackexchange.com/feeds/sites'
        feeds = get_feeds(all_feeds)

        def print_version(self, url):
                return url + '&template=printart'

What is the proper way of handling this (maybe using the API)?

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

There are a couple questions in here.

The answer to the obvious one, "How do I stop getting a "Too Many Requests from this IP" error?" is... make fewer requests. Stack Overflow is better suited than Meta.SO for information about throttling Calibre scripts. At least for the API, the limit is 30+ requests in <= 5 seconds and 10k requests/day, so that's not a bad guess for a ballpark on the limit you're hitting, but it sounds like you might just be able to use trial and error to determine the limit.

As for the API route, some kind soul might stop by and hand you a Stack App that does what they think you want, but you'll probably have better luck giving it a go yourself and posting on StackApps.com if you run into problems. I'm sure that they'd also be eager to share in the fruits of your labor, so it'd be a nicety to offer it to everyone there.

share|improve this answer
Thanks. I just found out there is a delay parameter to add to Calibre recipes. I will definitely post to StackApps once I get something that works (it's always been my intention :)) –  ℝaphink Oct 11 '12 at 7:06
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .