//TODO 修改成能用的

程序设计步骤

image.png

建立工程和Spider模板

  1. \> scrapy startproject stock_spider
  2. \> cd BaiduStocks
  3. \> scrapy genspider stocks baidu.com

进一步修改spiders/stocks.py文件

编写Spider

image.png

  • 修改前 ```python

    -- coding: utf-8 --

    import scrapy

class StocksSpider(scrapy.Spider): name = ‘stocks’ allowed_domains = [‘baidu.com’] start_urls = [‘http://baidu.com/‘]

  1. def parse(self, response):
  2.     pass
  2. - 修改后
  3. ```python
  4.  # -*- coding: utf-8 -*-
  5. import scrapy
  8.     def parse(self, response):
  9.         for href in response.css('a::attr(href)).extract():
  10.             try:
  11.                 stock = re.findall(r"[s][hz]\d{6}", href)[0]
  12.                 url = 'https://gupiao/baidu.com/stock/' + stock + '.html'
  13.                 yield scrapy.Request(url, callback=parse_stock)
  14.             except:
  15.                 continue
  17.     def parse_stock(self, response):
  18.         info_dict = {}
  19.         stock_info = response.css('.stock-bets')
  20.         name = stock_info.css('.bets-name').extract()[0]
  21.         key_list = stock_info.css('dd').extract()
  22.         value_list = stock_info.css('dd').extract()
  23.         for i in range(len(key_list):
  24.             key = re.findall(r'>.*</dt>', key_list[i])[0][1:-5]
  25.             try:
  26.                 value = re.findall(r'\d+\.?.*</dd>', value_list[i])[0][0:-5]
  27.             except:
  28.                 value = '--'
  29.             info_dict[key] = value
  31.         info_dict.update({'股票名称': re.findall('\s.*\(', name)[0].split()[0] + 
  32.                          re.findall('\>.*\<', name)[0][1:-1]})
  34.          yield info_dict

编写Pipelines

image.png
文件在stock_spider目录下

  • 修改前 ```python

    -- coding: utf-8 --

Define your item pipelines here

#

Don’t forget to add your pipeline to the ITEM_PIPELINES setting

See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html

class StockSpiderPipeline: def process_item(self, item, spider): return item

  2. - 修改后
  3.    - 这里我们新建一个类，并在配置文件中添加配置，使它识别我们新编写的类
  4. ```python
  5. # -*- coding: utf-8 -*-
  7. # Define your item pipelines here
  8. #
  9. # Don't forget to add your pipeline to the ITEM_PIPELINES setting
  10. # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
  13. class StockSpiderPipeline:
  14.     def process_item(self, item, spider):
  15.         return item
  17. class StockInfoPipeline():
  18.     def open_spider(self, spider):
  19.         self.f = open('StockInfo.txt', 'w')
  21.     def close_spider(self, spider):
  22.         self.f.close()
  25.     def process_item(self, item, spider):
  26.         try:
  27.             line = str(dict(item)) + '\n'
  28.             self.f.write(line)
  29.         except:
  30.             pass
  31.         return item
  • 修改配置文件

配置settings.py文件

  1. # Configure item pipelines
  2. # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
  3. ITEM_PIPELINES = {
  4.     'stock_spider.pipelines.StockInfoPipeline': 300,
  5. }

执行

  1. /> python -m scrapy crawl stocks

优化——改善性能

image.png