Web Scraping: working with APIs

APIs present researchers with a diverse set of data sources through a standardised access mechanism: send a pasted together HTTP request, receive JSON or XML in return. Today we tap into a range of APIs to get comfortable sending queries and processing responses.

These are the slides from the final class in Web Scraping through R: Web scraping for the humanities and social sciences


This week we explore how to use APIs in R, focusing on the Google Maps API. We then attempt to transfer this approach to query the Yandex Maps API. Finally, the practice section includes examples of working with the YouTube V2 API, a few 'social' APIs such as LinkedIn and Twitter, as well as APIs less off the beaten track (Cricket scores, anyone?).

I enjoyed teaching this course and hope to repeat and improve on it next year. When designing the course I tried to cram in everything I wish I had been taught early on in my PhD (resulting in information overload, I fear). Still, hopefully it has been useful to students getting started with digital data collection, showing on the one hand what is possible, and on the other giving some idea of key steps in achieving research objectives.


Download the .Rpres file to use in Rstudio here

A regular R script with code-snippets only can be accessed here


Slides from the first session here

Slides from the second session here

Slides from the third session here

UPDATE March 2015:
New 2015 version of slides here
PDFs of slides available here


14 comments:

  1. Nice blog on Web scraping, all the post on web scraping is very interesting and useful. Thanks for sharing a very useful information web scraping.

    ReplyDelete
  2. This comment has been removed by a blog administrator.

    ReplyDelete
  3. wow that's great web scraping working with apis this topic i have searching last few days finally i found you blog it's really great thanks for shearing helpful information.

    ReplyDelete
  4. Thank you very much for sharing this. Your slides really help me learn the basics of digital data collection, which is crucial part of my research which applied text mining.

    ReplyDelete
  5. For straight forward data extraction, json-csv.com could save someone a bit of time. You just need to paste in the JSON API url and it will produce a neatly formatted CSV file which you can work with in Excel.

    ReplyDelete
  6. Online gambling games will bring the player to the rich.

    Ruby888 The online casino is a long-standing online gambling and casino services. The club is also considered one of the beginning of the million. Since winning online gambling games only once, you can make millions of money to yourself already.

    It can be said that a lot of money can be created with the two hands of the risk yourself and the player does not need to be a risky. Because only players are passionate about betting, the bettor can join in the fun and prove the ability to beat the game online casino games on the site of our online casino has been called. The fun and excitement of playing online gambling games will lead the player to the rich and the rich from the online casino games will change. Bets life forever because of gambling online in a format that can transform ordinary people to become rich overnight.

    Players can join the fun and play online casino games with the gamblers around the world 24 hours a day with our online casino site, allowing all gamblers to play. Online Betting Games Along the way, when a bettor thinks about it. Online casino services do not forget to remember the site IBCbet


    ReplyDelete
  7. With a scraper such as google places scraper you can simply collect the data that you want from Google without using any APIs. Use their data to populate your own databases of business listings, send out mass mailing campaigns, or just collect it for lead generation.

    ReplyDelete
  8. Very beautiful if it is a birthday present, it would be nice.
    หวยเด็ดไทยรัฐ

    ReplyDelete
  9. Collagen Refresh Lemonade by Zenith Labs is a clinically proven skin care dietary supplement that repairs dermal collapse and reduces the collagen-destroying enzymes with a high level. It assures to ensure the enhancement of skin health and supports to remove inflamed, swollen or damaged skin cell using the proprietary herbal extract at the right time to experience better skin health.

    learn more here

    ReplyDelete
  10. Shop Online in Auckland Newzealand on a single click. We provide you quality items which you will be buy online at lowest price..! Buy online sofa beds nz at lowest price on Treasurebox store.

    ReplyDelete
  11. watch your favourite show here just go to pinoy channel and enjoy tfc seervice free

    ReplyDelete

  12. Get all the latest clicksud online seriale online of clicksud and all the seriale online daily on this blog.

    ReplyDelete