There are many web-based Application Programming Interfaces (API) that allow data-hungry people to get their hands on all sorts of data. It's a great way to collect data as-needed, instead of looking for an existing database or data set.

But how can I use web-based APIs to gather data?

  • 5
    What exactly is your question?
    – vanthome
    Mar 6, 2014 at 14:55

4 Answers 4


APIs are often offered by websites so that developers can use the web-based data for apps, without having the uncertainty and difficulty of scraping the HTML. But it's not necessary to use the data to build apps, and this means that APIs can be a great source of data for research and analysis. Just to name a few types of API data: weather forecasts, historical stock prices, the twitter live-stream, wikipedia page views, ... and the list goes on and on.

Full list of APIs: ProgrammableWeb has a comprehensive list of APIs (more than 10,000). The list is so comprehensive that it's hard to know where to start. In the answers below you'll find details about popular APIs as well as some instructions for how to connect to data.

Authentication: Some APIs are free and open and don't require authentication, although they will often block your IP address if you make too many requests or request too fast. Other APIs require authentication but are free for basic usage. Each API has its own terms of service, and can restrict unlimited access by either capping the total calls per time period (i.e. 1000 per day), or by using a rate-limiting quota (i.e. 1 call per second).

Data formats: Data from APIs usually flows via a RESTful interface, and common formats are JSON, XML, and sometimes CSV (comma separated values).

Software and code: With a correct URL/URI, anyone can access the raw data from the browser (for example, JSON, XML, or even a CSV file). Here is a handy overview for non-programmers. Programming skills are usually needed to repeat or automate a process. There are some open source tools such as KNIME that allow non-programmers to connect to a RESTful interface and then collect and analyze data. For Python, see the Open Data Python Guide for examples of how to read and use the data files described above.

Here are just a handful of examples of the diverse and useful web-based API data

Wikipedia page views:

  • stats.grok.se

    • Wikipedia page views are useful for many things, including predicting changes in a stock price (paper)

      A flood of views to a company's Wikipedia page may be a sign that their stock price is about to plummet. (news article)

    • You can access the raw data without any authentication, either by download or via an API.

    • For the API, you construct a URL like this: http://stats.grok.se/json/en/201306/Data , which would give you the page views in JSON format for the article 'Data' from June, 2013.
    • It helps very much to first verify that you page and title exists. It does.

    • Sample python code to access page views for two articles ('Advisor' and 'Adviser') for all months between 2008 and 2014 (my reason). Note that days that don't exist (i.e. June 31st) will return 0 (zero) page views.

Weather data (historical):

  • Wunderground.com

    • 500 free API calls per day (with authentication), and offers historical detailed daily weather data going back to 1997 for my location (details).
    • International weather, so it is not only restricted to the US.
    • Data comes in JSON format (example data, click "show response").
    • Sample python code for current and historical weather.

Stock quotes

The Internet Archive

Google Maps Geolocation API

  • Pass search string and receive latitude, longitude and other data.
  • 100 free requests per 24 hours (no authentication required)
  • Example lookup
  • Python code sample


I found this website (http://www.datasciencetoolkit.org/). There is an API collection in there with great tools, especially to "clean" and prepare your open datasets for analysis and visualization.

A list of the tools:

Street Address to Coordinates

Street Address to Location calculates the latitude/longitude coordinates for a postal address. Currently only the US and UK have street-level detail.

Google-style Geocoder

Are you currently using Google's geocoding API and want to switch? Replace maps.googleapis.com with the address of a DSTK server and your code should work without changes.

Coordinates to Political Areas

Returns the country, region, state, county, constituencies and neighborhood a point is inside.

Text to Sentiment

Estimates whether a piece of text comment is complimentary or negative.

Coordinates to Statistics

Returns the population density, elevation, terrain, climate, and other conditions at a point.



IP Address to Coordinates

IP Address to Location calculates country, state, city and latitude/longitude coordinates for IP addresses.

Text to Sentences

Removes the parts of the text that seem to be boilerplate, leaving the real sentences.

HTML to Text

Returns the full text that would actually be displayed in the browser when an HTML document was rendered.

HTML to Story

Takes an HTML document representing a news article or similar page, and extracts just the story text.

Text to People

Spots text fragments that look like people's names or titles, and guesses their gender where possible.

Text to Times

Spots text fragments that look like times or dates, and converts them into a standard form.

File to Text

Converts PDFs, Word Documents, Excel Spreadsheets to text. Recovers text from JPEG, PNG or TIFF images of scanned documents.


It depends what data you want to gather. Twitter is a major source of data that is API-accessible. They have a REST API and a streaming API. There are also a lot of wrappers to make it easier to use those API's from your language of choice. For Twitter and other social media websites, I would suggest looking at the book "Mining the Social Web"

But this is just scratching the surface, and there are a lot of websites with APIs, and which you use depends on what type of data you are looking for.


Microformats are far more implemented than other "technologies", and plenty of big companies, pushing out a lot of data, implement them. There's no great advertisement for them, so you have to figure it out, although there are plug-ins that will tell you when a microformat can be mined/consumed/interacted with.

I would assume you could do the same with microdata/rdfa that's been implemented. So to your question, microformats + microdata + rfda are by far the most implemented APIs.

Also typically you're not going to have many problems/restrictions with API limits. The markup is the API. You just have to parse it, and there are libraries for that all over GitHub. Optimus would be the most popular/famous
and here's microformats parser for indieweb (which is something we all need to check out):

commoncrawl.org has set up webcommonsdata, for getting slices of the web's data, and they specifically touch on crawling for microformats/rdfa in their posts as well.

Over half of the "semantic world" will fight me to death for promoting markup and microformats, but they're so easy, and they're all around us; and they are specifically what you are asking for. They provide a web API, and it's free and open. I don't want credit for the answer, I just cannot emphasize how crucial microformats are, how easy they are to implement, and why no one has an excuse to not use them. Most solid webdevs have been publishing them for years; it's near second-hand knowledge, or rather, the classes just make up patterns I embed into practically everything I do.

  • And if you're planning on publishing something using microformats, see schema.org
    – Joe
    Apr 8, 2014 at 15:17
  • not at all, quite the contrary, microformats.org. schema needs microformats, pretty sure they haven't adopted any of schema's though
    – albert
    Apr 8, 2014 at 18:42

Your Answer

