思路

爬取蜗牛网站下的多个文章

  1. 找出网站跳转规律 ，一般通过浏览器上的网站变化判断
  2. 找出对应的xpath路径
  3. 进到一篇文章内找出想要爬下来的内容
  4. 组合代码

1.首先爬取网站的所有链接

爬蜗牛网站内多个文档
image.png

  1. #导入所需的库
  2. import requests,json
  3. from lxml import etree
  4. from bs4 import BeautifulSoup
  5. import re
  6. from urllib.parse import urljoin
  8. base_url = "https://www.woniuxy.com/"
  9. url = 'https://www.woniuxy.com/note/page-1'            #要爬取的网站
  10. headers = {
  11.     'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36'
  12. }
  13. rsp = requests.get(url,headers=headers)                #向网站发起请求
  14. html = etree.HTML(rsp.text)                            #将获取的请求用etree中HTML转化为HTML格式
  15. woniu_urls = html.xpath('//div[@class="title"]/a')    #通过xpath获取所有子链接
  16. for url in woniu_urls:                                #if循环列出爬取的链接
  17.     full_url = urljoin(base_url, url.get('href'))    #将爬取下来的不是完整的链接，所有要合并成完整的链接
  18.     print(full_url)
  20. ---------------------------------------------------------------------------------
  21. 输出效果展示：
  22. D:\学习软件工具\pycharm\openstack-api\venv\Scripts\python.exe D:/学习软件工具/pycharm/openstack-api/venv/flavor/flavor003.py
  23. https://www.woniuxy.com/note/820
  24. https://www.woniuxy.com/note/819
  25. https://www.woniuxy.com/note/818
  26. .................

2.抓取每篇文章的作者等信息

image.png

  1. import requests,json
  2. from bs4 import BeautifulSoup
  3. from lxml import etree
  4. import re
  6. url = "https://www.woniuxy.com/note/820"
  7. r = requests.get(url)
  8. html = etree.HTML(r.text)
  9. titl_obj = html.xpath("//div[contains(@class, 'info')]")[1]
  10. #titl_obj = html.xpath("/html/body/div[8]/div/div[1]/div[1]/div[3]/@class")
  11. result = re.findall('作者：(.*?)\s+类型：(.*?)\s+.*日期：(.*?)\s+阅读：(.*?)次', titl_obj.text, re.S)
  12. (user, type, date, read_num) = result[0]
  13. print("作者:" + user)
  14. print("类型:" + type)
  15. print("时间:" + date)
  16. print("阅读次数:" + read_num)
  19. -------------------------------------------------------------------------------------
  20. 输出展示：
  21. D:\学习软件工具\pycharm\openstack-api\venv\Scripts\python.exe D:/学习软件工具/pycharm/openstack-api/venv/pachong/pachong001.py
  22. 作者:管理员
  23. 类型:学院动态
  24. 时间:2021-12-10
  25. 阅读次数:225