是什么

API 是网站官方提供给大家的信息抽取方式,调用 API 可以清晰迅速的获得所需信息,不用爬虫去爬那么辛苦(一般网站会有反爬虫技术,不会让你舒舒服服的抓下来那么多数据)。

对于 API 的理解,可以从两方面,一个是官方 API 都提供些什么。另一个是如何使用 API,咱们课程的定位偏重前面,所以这个 Assignment 是要让大家去找找那些网站提供了 API,官方披露了什么信息。

Twitter API

以 Twitter 的API 举例,可以找到这个 Use cases,其实就是从用户角度说明可以用 API 得到什么信息,帮助你的工作更高效(比较复杂的 API 会是这样组织的):
image.png
点击 Learn More,就能发现具体的说明:
image.png

如何获取

爬虫与收费数据

至于如何抓到数据看,是爬虫工程师的工作,感兴趣的同学可以看看 API 的文档和调用代码,在大部分网站的 Dev 相关部分都用提供。当然,也有很多专门收集了很多数据,之后通过 API 让你使用的网站(收费),感兴趣的同学可以参考 扩展下:

  • https://www.zhihu.com/question/19998447
  • https://www.jianshu.com/p/90dce199022f

    API集市

    RapidAPI 是一个很大的 API 集市,比较复杂注册可能需要科学上网,比如天气的数据:https://rapidapi.com/community/api/open-weather-map

    政府或行业数据

    很多政府都提供免费的数据,比如:
    1新加坡政府 https://data.gov.sg/
    2 美国政府 https://data.gov/

    API示例

    和风天气

    注册key

    注册账户后,在【控制台】【应用管理】中创建key,其中QPM是每分钟查询的数量:
    image.png
    创建名字之后,选择API的种类(例如选择Web):
    image.png
    完成后会弹出API地址:https://devapi.qweather.com
    image.png
    配置完成后,就会增加key的记录,其中【Public ID】和【Key】即对应API的辨识信息:
    image.png

    查阅API文档

    在主页上点击链接:
    image.png
    或者直接输入网址:https://dev.qweather.com/docs/api/ 即可查询相关文档:
    image.png

    使用API查询

    根据文档,直接在网页中的链接中输入数据,则可以得到相应的数据,比如在网页中输入:https://devapi.qweather.com/v7/air/now?location=101010100&key=eea3bc477d794e279df3aa1ea2f706c0 拆解如下: :::info https://devapi.qweather.com/v7/ air/now? location=101010100 &key=eea3bc477d794e279df3aa1ea2f706c0
    开发版本API网页请求地址 要查询即时空气质量 位置编号(北京)使用者的key ::: 完成后,则网页中会输出相应数据:
    image.png
    接下来就可以在应用中显示,或者通过编程将数据进行整合了,文本版本输入如下:
    1. {"code":"200","updateTime":"2021-05-14T06:58+08:00","fxLink":"http://hfx.link/2ax4","now":{"pubTime":"2021-05-14T06:00+08:00","aqi":"59","level":"2","category":"良","primary":"PM2.5","pm10":"58","pm2p5":"42","no2":"33","so2":"2","co":"1.1","o3":"28"},"station":[{"pubTime":"2021-05-14T06:00+08:00","name":"密云镇","id":"CNA3697","aqi":"73","level":"2","category":"良","primary":"PM2.5","pm10":"49","pm2p5":"53","no2":"26","so2":"1","co":"1.0","o3":"21"},{"pubTime":"2021-05-14T06:00+08:00","name":"丰台小屯","id":"CNA3696","aqi":"51","level":"2","category":"良","primary":"PM10","pm10":"52","pm2p5":"33","no2":"36","so2":"1","co":"1.1","o3":"19"},{"pubTime":"2021-05-14T06:00+08:00","name":"怀柔新城","id":"CNA3695","aqi":"69","level":"2","category":"良","primary":"PM2.5","pm10":"75","pm2p5":"50","no2":"34","so2":"2","co":"1.1","o3":"17"},{"pubTime":"2021-05-14T06:00+08:00","name":"延庆石河营","id":"CNA3694","aqi":"74","level":"2","category":"良","primary":"PM2.5","pm10":"66","pm2p5":"54","no2":"20","so2":"3","co":"1.0","o3":"66"},{"pubTime":"2021-05-14T06:00+08:00","name":"大兴旧宫","id":"CNA3675","aqi":"57","level":"2","category":"良","primary":"PM2.5","pm10":"45","pm2p5":"40","no2":"26","so2":"1","co":"0.9","o3":"30"},{"pubTime":"2021-05-14T06:00+08:00","name":"房山燕山","id":"CNA3674","aqi":"64","level":"2","category":"良","primary":"PM10","pm10":"78","pm2p5":"43","no2":"49","so2":"3","co":"0.8","o3":"17"},{"pubTime":"2021-05-14T06:00+08:00","name":"通州东关","id":"CNA3673","aqi":"56","level":"2","category":"良","primary":"PM10","pm10":"62","pm2p5":"35","no2":"26","so2":"3","co":"1.4","o3":"37"},{"pubTime":"2021-05-14T06:00+08:00","name":"丰台云岗","id":"CNA3672","aqi":"65","level":"2","category":"良","primary":"PM2.5","pm10":"46","pm2p5":"47","no2":"27","so2":"4","co":"1.2","o3":"34"},{"pubTime":"2021-05-14T06:00+08:00","name":"门头沟三家店","id":"CNA3671","aqi":"80","level":"2","category":"良","primary":"PM2.5","pm10":"74","pm2p5":"59","no2":"40","so2":"1","co":"1.1","o3":"33"},{"pubTime":"2021-05-14T06:00+08:00","name":"密云新城","id":"CNA3418","aqi":"87","level":"2","category":"良","primary":"PM2.5","pm10":"61","pm2p5":"64","no2":"27","so2":"1","co":"1.0","o3":"27"},{"pubTime":"2021-05-14T06:00+08:00","name":"平谷新城","id":"CNA3417","aqi":"59","level":"2","category":"良","primary":"PM2.5","pm10":"63","pm2p5":"42","no2":"39","so2":"1","co":"1.4","o3":"1"},{"pubTime":"2021-05-14T06:00+08:00","name":"延庆夏都","id":"CNA3281","aqi":"62","level":"2","category":"良","primary":"PM2.5","pm10":"68","pm2p5":"44","no2":"19","so2":"1","co":"0.9","o3":"66"},{"pubTime":"2021-05-14T06:00+08:00","name":"古城","id":"CNA1012","aqi":"68","level":"2","category":"良","primary":"PM2.5","pm10":"74","pm2p5":"49","no2":"37","so2":"1","co":"1.0","o3":"24"},{"pubTime":"2021-05-14T06:00+08:00","name":"奥体中心","id":"CNA1011","aqi":"59","level":"2","category":"良","primary":"PM2.5","pm10":"52","pm2p5":"42","no2":"35","so2":"1","co":"0.9","o3":"32"},{"pubTime":"2021-05-14T06:00+08:00","name":"昌平镇","id":"CNA1010","aqi":"48","level":"1","category":"优","primary":"NA","pm10":"45","pm2p5":"33","no2":"20","so2":"3","co":"0.9","o3":"45"},{"pubTime":"2021-05-14T06:00+08:00","name":"怀柔镇","id":"CNA1009","aqi":"53","level":"2","category":"良","primary":"PM2.5","pm10":"46","pm2p5":"37","no2":"29","so2":"1","co":"0.9","o3":"32"},{"pubTime":"2021-05-14T06:00+08:00","name":"顺义新城","id":"CNA1008","aqi":"62","level":"2","category":"良","primary":"PM10","pm10":"73","pm2p5":"42","no2":"37","so2":"3","co":"1.5","o3":"20"},{"pubTime":"2021-05-14T06:00+08:00","name":"海淀区万柳","id":"CNA1007","aqi":"48","level":"1","category":"优","primary":"NA","pm10":"48","pm2p5":"27","no2":"39","so2":"2","co":"1.0","o3":"25"},{"pubTime":"2021-05-14T06:00+08:00","name":"官园","id":"CNA1006","aqi":"59","level":"2","category":"良","primary":"PM2.5","pm10":"52","pm2p5":"42","no2":"31","so2":"3","co":"1.0","o3":"23"},{"pubTime":"2021-05-14T06:00+08:00","name":"农展馆","id":"CNA1005","aqi":"50","level":"1","category":"优","primary":"NA","pm10":"50","pm2p5":"33","no2":"33","so2":"3","co":"1.2","o3":"34"},{"pubTime":"2021-05-14T06:00+08:00","name":"天坛","id":"CNA1004","aqi":"54","level":"2","category":"良","primary":"PM2.5","pm10":"45","pm2p5":"38","no2":"52","so2":"1","co":"1.0","o3":"10"},{"pubTime":"2021-05-14T06:00+08:00","name":"东四","id":"CNA1003","aqi":"57","level":"2","category":"良","primary":"PM2.5","pm10":"52","pm2p5":"40","no2":"37","so2":"1","co":"1.0","o3":"27"},{"pubTime":"2021-05-14T06:00+08:00","name":"定陵","id":"CNA1002","aqi":"62","level":"2","category":"良","primary":"PM2.5","pm10":"53","pm2p5":"44","no2":"24","so2":"1","co":"1.0","o3":"34"},{"pubTime":"2021-05-14T06:00+08:00","name":"万寿西宫","id":"CNA1001","aqi":"51","level":"2","category":"良","primary":"PM10","pm10":"52","pm2p5":"30","no2":"43","so2":"1","co":"0.9","o3":"10"}],"refer":{"sources":["cnemc"],"license":["no commercial use"]}}

    玖房数据

    是一家提供中国房产数据的数据公司,公司提供API服务,文档网址 http://91my.net/dataapi/?id=a4 ,API权限或数据需要联系公司获取(比较专业的数据提供基本都不是免费):
    image.png