URL与HTTP

image.png
image.png

使用urllib 下载文件

  1. # Import package
  2. from urllib.request import urlretrieve
  4. # Import pandas
  5. import pandas as pd
  7. # Assign url of file: url
  8. url = 'https://s3.amazonaws.com/assets.datacamp.com/production/course_1606/datasets/winequality-red.csv'
  11. # Save file locally
  12. urlretrieve(url, 'winequality-red.csv')
  14. # Read file into a DataFrame and print its head
  15. df = pd.read_csv('winequality-red.csv', sep=';')
  16. print(df.head())

获取request方式

想要得到网页内容，必须得先向该网页发送一个request，以获得访问。

通过urllib

image.png

  1. # Import packages
  2. from urllib.request import urlopen, Request
  4. # Specify the url
  5. url = "http://www.datacamp.com/teach/documentation"
  7. # This packages the request
  8. request = Request(url)
  10. # Sends the request and catches the response: response
  11. response = urlopen(request)
  13. # Extract the response: html
  14. html = response.read()
  16. # Print the html
  17. print(html)
  19. # Be polite and close the response!
  20. response.close()

通过request

image.png

  1. # Import package
  2. import requests
  4. # Specify the url: url
  5. url = "http://www.datacamp.com/teach/documentation"
  7. # Packages the request, send the request and catch the response: r
  8. r = requests.get(url)
  10. # Extract the response: text
  11. text = r.text
  13. # Print the html
  14. print(text)

通过bs4获取HTML 内容

image.png
ps：关于bs4 库详细用法，可以参考010_爬虫学习004-006内容。

  1. # Import packages
  2. import requests
  3. from bs4 import BeautifulSoup
  5. # Specify url
  6. url = 'https://www.python.org/~guido/'
  8. # Package the request, send the request and catch the response: r
  9. r = requests.get(url)
  11. # Extracts the response as html: html_doc
  12. html_doc = r.text
  14. # create a BeautifulSoup object from the HTML: soup
  15. soup = BeautifulSoup(html_doc)
  17. # Print the title of Guido's webpage
  18. print(soup.title)
  20. # Find all 'a' tags (which define hyperlinks): a_tags
  21. a_tags = soup.find_all('a')
  23. # Print the URLs to the shell
  24. for link in a_tags:
  25.     print(link.get('href'))

通过APIs 获取信息

image.png

通常来说，APIs 传输文件类型为JSON。

JSON

image.png
采用有类型的key-value，适合程序使用（直接作为程序的一部分）。

  1. # Load JSON: json_data
  2. with open("a_movie.json") as json_file:
  3.     json_data = json.load(json_file)
  5. # Print each key-value pair in json_data
  6. for k in json_data.keys():
  7.     print(k + ': ', json_data[k])

获取APIs 过程

image.png
image.png

  1. # Import package
  2. import requests
  4. # Assign URL to variable: url
  5. url = 'https://en.wikipedia.org/w/api.php?action=query&prop=extracts&format=json&exintro=&titles=pizza'
  7. # Package the request, send the request and catch the response: r
  8. r = requests.get(url)
  10. # Decode the JSON data into a dictionary: json_data
  11. json_data = r.json()
  13. # Print the Wikipedia page extract
  14. pizza_extract = json_data['query']['pages']['24768']['extract']
  15. print(pizza_extract)

获取推特API 进行分析

  • 创建推特账户
  • 获取token 与API 与相关的密钥
  • 推特提供许多APIs 选择合适的（streaming API，public stream）
  • 使用get statuses，获取小量随机的public statuses。如果获取大量的实际public statuses，可以使用firehose API。
  • 获取JSON文件。

代码操作：
1）通过token 与API 与相关的密钥访问，获得授权(authorization)
image.png

2）设置一个类，用于监听数据流。
image.png

3）抓取指定数据流
image.png

实战：获取推特random_statuses 并进行分析

获取推特API

  1. # Import package
  2. import json
  4. # String of path to file: tweets_data_path
  5. tweets_data_path = 'tweets.txt'
  7. # Initialize empty list to store tweets: tweets_data
  8. tweets_data = []
  10. # Open connection to file
  11. tweets_file = open(tweets_data_path, "r")
  13. # Read in tweets and store in list: tweets_data
  14. for line in tweets_file:
  15.     tweet = json.loads(line)
  16.     tweets_data.append(tweet)
  18. # Close connection to file
  19. tweets_file.close()

导入为pandas 表格

  1. # Import package
  2. import pandas as pd
  4. # Build DataFrame of tweet texts and languages
  5. df = pd.DataFrame(tweets_data, columns=['text', 'lang'])
  7. # Print head of DataFrame
  8. print(df.head())

分析内容：找出单词出现次数

  1. # Initialize list to store tweet counts
  2. [clinton, trump, sanders, cruz] = [0, 0, 0, 0]
  4. # Iterate through df, counting the number of tweets in which
  5. # each candidate is mentioned
  6. for index, row in df.iterrows():
  7.     clinton += word_in_text('clinton', row['text'])
  8.     trump += word_in_text('trump', row['text'])
  9.     sanders += word_in_text('sanders', row['text'])
  10.     cruz += word_in_text('cruz', row['text'])

使用matplotlib 与seaborn 进行作图

  1. # Import packages
  2. import matplotlib.pyplot as plt
  3. import seaborn as sns
  5. # Set seaborn style
  6. sns.set(color_codes=True)
  8. # Create a list of labels:cd
  9. cd = ['clinton', 'trump', 'sanders', 'cruz']
  11. # Plot the bar chart
  12. ax = sns.barplot(cd, [clinton, trump, sanders, cruz])
  13. ax.set(ylabel="count")
  14. plt.show()

image.png