Web Scraping: working with APIs

APIs present researchers with a diverse set of data sources through a standardised access mechanism: send a pasted together HTTP request, receive JSON or XML in return. Today we tap into a range of APIs to get comfortable sending queries and processing responses.

These are the slides from the final class in Web Scraping through R: Web scraping for the humanities and social sciences


This week we explore how to use APIs in R, focusing on the Google Maps API. We then attempt to transfer this approach to query the Yandex Maps API. Finally, the practice section includes examples of working with the YouTube V2 API, a few 'social' APIs such as LinkedIn and Twitter, as well as APIs less off the beaten track (Cricket scores, anyone?).

I enjoyed teaching this course and hope to repeat and improve on it next year. When designing the course I tried to cram in everything I wish I had been taught early on in my PhD (resulting in information overload, I fear). Still, hopefully it has been useful to students getting started with digital data collection, showing on the one hand what is possible, and on the other giving some idea of key steps in achieving research objectives.


Download the .Rpres file to use in Rstudio here

A regular R script with code-snippets only can be accessed here


Slides from the first session here

Slides from the second session here

Slides from the third session here

UPDATE March 2015:
New 2015 version of slides here
PDFs of slides available here


28 comments:

  1. Nice blog on Web scraping, all the post on web scraping is very interesting and useful. Thanks for sharing a very useful information web scraping.

    ReplyDelete
    Replies
    1. KARNA RASA HATI YANG GEMBIRA BERKAT BANTUAN AKI SOLEH
      MAKANYA SENGAJA NAMA BELIAU SAYA CANTUNKAN DI INTERNET !!!

      assalamualaikum wr, wb, saya IBU PUSPITA WATI saya Mengucapkan banyak2
      Terima kasih kepada: AKI SOLEH
      atas nomor togelnya yang kemarin AKI berikan "4D"
      alhamdulillah ternyata itu benar2 tembus AKI
      dan berkat bantuan AKI SOLEH saya bisa melunasi semua hutan2…
      orang tua saya yang ada di BANK BRI dan bukan hanya itu AKI alhamdulillah,
      sekarang saya sudah bisa bermodal sedikit untuk mencukupi kebutuhan keluarga saya sehari2.
      Itu semua berkat bantuan AKI SOLEH sekali lagi makasih banyak ya, AKI
      yang ingin merubah nasib
      seperti saya...?
      SILAHKAN GABUNG SAMA AKI SOLEH No; { 082-313-336-747 }

      Sebelum Gabung Sama AKI Baca Duluh Kata2 Yang Dibawah Ini
      Apakah anda termasuk dalam kategori di bawah ini...!!
      1: Di kejar2 tagihan hutang..
      2: Selaluh kalah dalam bermain togel
      3: Barang berharga sudah
      terjual buat judi togel..
      4: Sudah kemana2 tapi tidak
      menghasilkan, solusi yang tepat..!
      5: Sudah banyak dukun ditempati minta angka ritual blom dapat juga,
      satu jalan menyelesaikan masalah anda..
      Dijamin anda akan berhasil
      silahkan buktikan sendiri
      Atau Chat/Tlpn di WhatsApp (WA)
      No WA Aki : 082313336747

      TERIMA KASIH YANG PUNYA
      ROOM ATAS TUMPANGANYA SALAM KOMPAK SELALU
      "KLIK DISINI BOCORAN TOGEL SGP HK SDY DAN DLL"

      Delete
    2. KARNA RASA HATI YANG GEMBIRA BERKAT BANTUAN AKI SOLEH
      MAKANYA SENGAJA NAMA BELIAU SAYA CANTUNKAN DI INTERNET !!!

      assalamualaikum wr, wb, saya IBU PUSPITA WATI saya Mengucapkan banyak2
      Terima kasih kepada: AKI SOLEH
      atas nomor togelnya yang kemarin AKI berikan "4D"
      alhamdulillah ternyata itu benar2 tembus AKI
      dan berkat bantuan AKI SOLEH saya bisa melunasi semua hutan2…
      orang tua saya yang ada di BANK BRI dan bukan hanya itu AKI alhamdulillah,
      sekarang saya sudah bisa bermodal sedikit untuk mencukupi kebutuhan keluarga saya sehari2.
      Itu semua berkat bantuan AKI SOLEH sekali lagi makasih banyak ya, AKI
      yang ingin merubah nasib
      seperti saya...?
      SILAHKAN GABUNG SAMA AKI SOLEH No; { 082-313-336-747 }

      Sebelum Gabung Sama AKI Baca Duluh Kata2 Yang Dibawah Ini
      Apakah anda termasuk dalam kategori di bawah ini...!!
      1: Di kejar2 tagihan hutang..
      2: Selaluh kalah dalam bermain togel
      3: Barang berharga sudah
      terjual buat judi togel..
      4: Sudah kemana2 tapi tidak
      menghasilkan, solusi yang tepat..!
      5: Sudah banyak dukun ditempati minta angka ritual blom dapat juga,
      satu jalan menyelesaikan masalah anda..
      Dijamin anda akan berhasil
      silahkan buktikan sendiri
      Atau Chat/Tlpn di WhatsApp (WA)
      No WA Aki : 082313336747

      TERIMA KASIH YANG PUNYA
      ROOM ATAS TUMPANGANYA SALAM KOMPAK SELALU
      "KLIK DISINI BOCORAN TOGEL SGP HK SDY DAN DLL"

      Delete
  2. This comment has been removed by a blog administrator.

    ReplyDelete
  3. wow that's great web scraping working with apis this topic i have searching last few days finally i found you blog it's really great thanks for shearing helpful information.

    ReplyDelete
  4. Thank you very much for sharing this. Your slides really help me learn the basics of digital data collection, which is crucial part of my research which applied text mining.

    ReplyDelete
  5. For straight forward data extraction, json-csv.com could save someone a bit of time. You just need to paste in the JSON API url and it will produce a neatly formatted CSV file which you can work with in Excel.

    ReplyDelete
  6. Online gambling games will bring the player to the rich.

    Ruby888 The online casino is a long-standing online gambling and casino services. The club is also considered one of the beginning of the million. Since winning online gambling games only once, you can make millions of money to yourself already.

    It can be said that a lot of money can be created with the two hands of the risk yourself and the player does not need to be a risky. Because only players are passionate about betting, the bettor can join in the fun and prove the ability to beat the game online casino games on the site of our online casino has been called. The fun and excitement of playing online gambling games will lead the player to the rich and the rich from the online casino games will change. Bets life forever because of gambling online in a format that can transform ordinary people to become rich overnight.

    Players can join the fun and play online casino games with the gamblers around the world 24 hours a day with our online casino site, allowing all gamblers to play. Online Betting Games Along the way, when a bettor thinks about it. Online casino services do not forget to remember the site IBCbet


    ReplyDelete
  7. With a scraper such as google places scraper you can simply collect the data that you want from Google without using any APIs. Use their data to populate your own databases of business listings, send out mass mailing campaigns, or just collect it for lead generation.

    ReplyDelete
  8. Very beautiful if it is a birthday present, it would be nice.
    หวยเด็ดไทยรัฐ

    ReplyDelete
  9. Thank you for sharing your article. Great efforts put it to find the list of articles which is very useful to know, Definitely will share the same to other forums.

    appvn app

    ReplyDelete
  10. Collagen Refresh Lemonade by Zenith Labs is a clinically proven skin care dietary supplement that repairs dermal collapse and reduces the collagen-destroying enzymes with a high level. It assures to ensure the enhancement of skin health and supports to remove inflamed, swollen or damaged skin cell using the proprietary herbal extract at the right time to experience better skin health.

    learn more here

    ReplyDelete
  11. Shop Online in Auckland Newzealand on a single click. We provide you quality items which you will be buy online at lowest price..! Buy online sofa beds nz at lowest price on Treasurebox store.

    ReplyDelete
  12. watch your favourite show here just go to pinoy channel and enjoy tfc seervice free

    ReplyDelete
  13. You can write an article very much. Thank you for making a great article to read.
    Sa gaming สมัคร

    ReplyDelete

  14. Get all the latest clicksud online seriale online of clicksud and all the seriale online daily on this blog.

    ReplyDelete
  15. I really enjoy reading of your article. I wanted to inform you that you have people like me who appreciate your work.

    techwithgeeks
    talesbuzz
    whizzherald

    ReplyDelete
  16. Doramasmp4 esperamos que la estés pasando genial viendo. DoramasMP4 siempre será el primero en tener el episodio, así que por favor marque y agréguenos en Facebook para actualizarlo.
    https://www.evernote.com/shard/s617/sh/badf8185-4565-1a82-d4d9-17abc7c71027/e5be45b0b13252eca414907f2b9ea0ee

    ReplyDelete
  17. One of our clients generated $4.6 million of new sales in the first two business quarters and the direct competitor never knew what hit them. salesforce training in noida

    ReplyDelete
  18. Thanks for the detailed article on this topic. I would like to see more such awesome articles from you. Also you can get the new and best features of GBWhatsapp which are coming in 2022- GBWhatsapp 2022 APK

    ReplyDelete