02 Jsoup 实现简单爬虫

jsoup是一个用于解析、提取、操作HTML的开源Java函式库。

引入对应包

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.14.3</version>
</dependency>

基本案例

Connection connection = Jsoup.connect(url).ignoreContentType(true);
connection.header("Accept", "text/html, application/xhtml+xml, */*");
connection.header("Content-Type", "application/x-www-form-urlencoded");
connection.header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36");
connection.proxy("127.0.0.1", 10809);
connection.timeout(3000);
Document doc = connection.get();

Jsoup 解决异步Ajax请求

HtmlUnit

<dependency>
    <groupId>net.sourceforge.htmlunit</groupId>
  <artifactId>htmlunit</artifactId>
  <version>2.56.0</version>
</dependency>

请求案例

WebClient browser = new WebClient(BrowserVersion.CHROME);
browser.getOptions().setThrowExceptionOnScriptError(false);
browser.getOptions().setThrowExceptionOnFailingStatusCode(false);
browser.getOptions().setCssEnabled(false);
browser.getOptions().setJavaScriptEnabled(true);
// 设置网页超时时间
browser.getOptions().setTimeout(6000);
browser.setAjaxController(new NicelyResynchronizingAjaxController());
// 异步JS执行需要耗时,所以这里线程要阻塞6秒,等待异步JS执行结束
browser.waitForBackgroundJavaScript(6000);
HtmlPage htmlPage = browser.getPage(url);

Jsoup 常见方法介绍

Jsoup 类下常见方法