Java广度优先爬虫示例(抓取复旦新闻信息)

时间：2022-03-26 01:02:44

以下内容仅供学习交流使用，请勿做他用，否则后果自负。

一.使用的技术

这个爬虫是近半个月前学习爬虫技术的一个小例子,比较简单,怕时间久了会忘,这里简单总结一下.主要用到的外部Jar包有HttpClient4.3.4,HtmlParser2.1,使用的开发工具(IDE)为intelij 13.1,Jar包管理工具为Maven,不习惯用intelij的同学,也可以使用eclipse新建一个项目.

二.爬虫基本知识

1.什么是网络爬虫?(爬虫的基本原理)

网络爬虫,拆开来讲,网络即指互联网,互联网就像一个蜘蛛网一样,爬虫就像是蜘蛛一样可以到处爬来爬去,把爬来的数据再进行加工处理.

百科上的解释:网络爬虫（又被称为网页蜘蛛，网络机器人，在FOAF社区中间，更经常的称为网页追逐者），是一种按照一定的规则，自动的抓取万维网信息的程序或者脚本。另外一些不常使用的名字还有蚂蚁，自动索引，模拟程序或者蠕虫。

基本原理:传统爬虫从一个或若干初始网页的URL开始，获得初始网页上的URL，在抓取网页的过程中，不断从当前页面上抽取新的URL放入队列,直到满足系统的一定停止条件，流程图所示。聚焦爬虫的工作流程较为复杂，需要根据一定的网页分析算法过滤与主题无关的链接，保留有用的链接并将其放入等待抓取的URL队列。然后，它将根据一定的搜索策略从队列中选择下一步要抓取的网页URL，并重复上述过程，直到达到系统的某一条件时停止

Java广度优先爬虫示例(抓取复旦新闻信息)

2.常用的爬虫策略有哪些?

网页的抓取策略可以分为深度优先、广度优先和最佳优先三种。深度优先在很多情况下会导致爬虫的陷入(trapped)问题，目前常见的是广度优先和最佳优先方法。

2.1广度优先(Width-First)

广度优先遍历是连通图的一种遍历策略。因为它的思想是从一个顶点V0开始，辐射状地优先遍历其周围较广的区域,故得名.

其基本思想:

1)、从图中某个顶点V0出发，并访问此顶点；
2)、从V0出发，访问V0的各个未曾访问的邻接点W1，W2，…,Wk;然后,依次从W1,W2,…,Wk出发访问各自未被访问的邻接点；
3)、重复步骤2，直到全部顶点都被访问为止。

如下图所示:

Java广度优先爬虫示例(抓取复旦新闻信息)

2.2深度优先(Depth-First)

假设初始状态是图中所有顶点都未被访问，则深度优先搜索方法的步骤是：
1）选取图中某一顶点Vi为出发点，访问并标记该顶点；
2）以Vi为当前顶点，依次搜索Vi的每个邻接点Vj，若Vj未被访问过，则访问和标记邻接点Vj，若Vj已被访问过，则搜索Vi的下一个邻接点；
3）以Vj为当前顶点，重复步骤2，直到图中和Vi有路径相通的顶点都被访问为止；
4）若图中尚有顶点未被访问过（非连通的情况下），则可任取图中的一个未被访问的顶点作为出发点，重复上述过程，直至图中所有顶点都被访问。

下面以一个有向图和一个无向图为例:

Java广度优先爬虫示例(抓取复旦新闻信息)

广度和深度和区别:

广度优先遍历是以层为顺序，将某一层上的所有节点都搜索到了之后才向下一层搜索；而深度优先遍历是将某一条枝桠上的所有节点都搜索到了之后，才转向搜索另一条枝桠上的所有节点。

2.3 最佳优先搜索

最佳优先搜索策略按照一定的网页分析算法，预测候选URL与目标网页的相似度，或与主题的相关性，并选取评价最好的一个或几个URL进行抓取。它只访问经过网页分析算法预测为“有用”的网页。这种搜索适合暗网数据的爬取,只要符合要求的内容.

3.本文爬虫示例图

本文介绍的例子是抓取新闻类的信息,因为一般新闻类的信息,重要的和时间近的都会放在首页,处在网络层中比较深的信息的重要性一般将逐级降低,所以广度优先算法更适合,下图是本文将要抓取的网页结构图:

Java广度优先爬虫示例(抓取复旦新闻信息)

三.广度优先爬虫示例

1.需求:抓取复旦新闻信息(只抓取100个网页信息)

这里只抓取100条信息,并用url必须以new.fudan.edu.cn开头.

2.代码实现

使用maven引入外部jar包:

       <dependency>

            <groupId>org.apache.httpcomponents</groupId>

            <artifactId>httpclient</artifactId>

            <version>4.3.4</version>

        </dependency>

        <dependency>

            <groupId>org.htmlparser</groupId>

            <artifactId>htmlparser</artifactId>

            <version>2.1</version>

        </dependency>

程序主入口:

package com.amos.crawl;

import java.util.Set;

/**

 * Created by amosli on 14-7-10.

 */

public class MyCrawler {

    /**

     * 使用种子初始化URL队列

     *

     * @param seeds

     */

    private void initCrawlerWithSeeds(String[] seeds) {

        for (int i = 0; i < seeds.length; i++) {

            LinkQueue.addUnvisitedUrl(seeds[i]);

        }

    }

    public void crawling(String[] seeds) {

        //定义过滤器,提取以http://news.fudan.edu.cn/的链接

        LinkFilter filter = new LinkFilter() {

            @Override

            public boolean accept(String url) {

                if (url.startsWith("http://news.fudan.edu.cn")) {

                    return true;

                }

                return false;

            }

        };

        //初始化URL队列

        initCrawlerWithSeeds(seeds);

        int count=0;

        //循环条件:待抓取的链接不为空抓取的网页最多100条

        while (!LinkQueue.isUnvisitedUrlsEmpty() && LinkQueue.getVisitedUrlNum() <= 100) {

            System.out.println("count:"+(++count));

            //附头URL出队列

            String visitURL = (String) LinkQueue.unVisitedUrlDeQueue();

            DownLoadFile downloader = new DownLoadFile();

            //下载网页

            downloader.downloadFile(visitURL);

            //该URL放入怩访问的URL中

            LinkQueue.addVisitedUrl(visitURL);

            //提取出下载网页中的URL

            Set<String> links = HtmlParserTool.extractLinks(visitURL, filter);

            //新的未访问的URL入列

            for (String link : links) {

                System.out.println("link:"+link);

                LinkQueue.addUnvisitedUrl(link);

            }

        }

    }

    public static void main(String args[]) {

        //程序入口

        MyCrawler myCrawler = new MyCrawler();

        myCrawler.crawling(new String[]{"http://news.fudan.edu.cn/news/"});

    }

}

工具类:Tools.java

package com.amos.tool;

import java.io.*;

import java.net.URI;

import java.net.URISyntaxException;

import java.net.UnknownHostException;

import java.security.KeyManagementException;

import java.security.KeyStoreException;

import java.security.NoSuchAlgorithmException;

import java.security.cert.CertificateException;

import java.security.cert.X509Certificate;

import java.util.Locale;

import javax.net.ssl.SSLContext;

import javax.net.ssl.SSLException;

import org.apache.http.*;

import org.apache.http.client.CircularRedirectException;

import org.apache.http.client.CookieStore;

import org.apache.http.client.HttpRequestRetryHandler;

import org.apache.http.client.RedirectStrategy;

import org.apache.http.client.config.RequestConfig;

import org.apache.http.client.methods.HttpGet;

import org.apache.http.client.methods.HttpHead;

import org.apache.http.client.methods.HttpUriRequest;

import org.apache.http.client.methods.RequestBuilder;

import org.apache.http.client.protocol.HttpClientContext;

import org.apache.http.client.utils.URIBuilder;

import org.apache.http.client.utils.URIUtils;

import org.apache.http.conn.ConnectTimeoutException;

import org.apache.http.conn.HttpClientConnectionManager;

import org.apache.http.conn.ssl.SSLConnectionSocketFactory;

import org.apache.http.conn.ssl.SSLContextBuilder;

import org.apache.http.conn.ssl.TrustStrategy;

import org.apache.http.cookie.Cookie;

import org.apache.http.impl.client.*;

import org.apache.http.impl.conn.BasicHttpClientConnectionManager;

import org.apache.http.impl.cookie.BasicClientCookie;

import org.apache.http.protocol.HttpContext;

import org.apache.http.util.Args;

import org.apache.http.util.Asserts;

import org.apache.http.util.TextUtils;

import org.omg.CORBA.Request;

/**

 * Created by amosli on 14-6-25.

 */

public class Tools {

    /**

     * 写文件到本地

     *

     * @param httpEntity

     * @param filename

     */

    public static void saveToLocal(HttpEntity httpEntity, String filename) {

        try {

            File dir = new File(Configuration.FILEDIR);

            if (!dir.isDirectory()) {

                dir.mkdir();

            }

            File file = new File(dir.getAbsolutePath() + "/" + filename);

            FileOutputStream fileOutputStream = new FileOutputStream(file);

            InputStream inputStream = httpEntity.getContent();

            byte[] bytes = new byte[1024];

            int length = 0;

            while ((length = inputStream.read(bytes)) > 0) {

                fileOutputStream.write(bytes, 0, length);

            }

            inputStream.close();

            fileOutputStream.close();

        } catch (Exception e) {

            e.printStackTrace();

        }

    }

    /**

     * 写文件到本地

     *

     * @param bytes

     * @param filename

     */

    public static void saveToLocalByBytes(byte[] bytes, String filename) {

        try {

            File dir = new File(Configuration.FILEDIR);

            if (!dir.isDirectory()) {

                dir.mkdir();

            }

            File file = new File(dir.getAbsolutePath() + "/" + filename);

            FileOutputStream fileOutputStream = new FileOutputStream(file);

                fileOutputStream.write(bytes);

                //fileOutputStream.write(bytes, 0, bytes.length);

                fileOutputStream.close();

        } catch (Exception e) {

            e.printStackTrace();

        }

    }

    /**

     * 输出

     * @param string

     */

    public static void println(String string){

        System.out.println("string:"+string);

    }

    /**

     * 输出

     * @param string

     */

    public static void printlnerr(String string){

        System.err.println("string:"+string);

    }

    /**

     * 使用ssl通道并设置请求重试处理

     * @return

     */

    public static CloseableHttpClient createSSLClientDefault() {

        try {

            SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustStrategy() {

                //信任所有

                public boolean isTrusted(X509Certificate[] chain,String authType) throws CertificateException {

                    return true;

                }

            }).build();

            SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext);

            //设置请求重试处理,重试机制,这里如果请求失败会重试5次

            HttpRequestRetryHandler retryHandler = new HttpRequestRetryHandler() {

                @Override

                public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {

                    if (executionCount >= 5) {

                        // Do not retry if over max retry count

                        return false;

                    }

                    if (exception instanceof InterruptedIOException) {

                        // Timeout

                        return false;

                    }

                    if (exception instanceof UnknownHostException) {

                        // Unknown host

                        return false;

                    }

                    if (exception instanceof ConnectTimeoutException) {

                        // Connection refused

                        return false;

                    }

                    if (exception instanceof SSLException) {

                        // SSL handshake exception

                        return false;

                    }

                    HttpClientContext clientContext = HttpClientContext.adapt(context);

                    HttpRequest request = clientContext.getRequest();

                    boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);

                    if (idempotent) {

                        // Retry if the request is considered idempotent

                        return true;

                    }

                    return false;

                }

            };

            //请求参数设置,设置请求超时时间为20秒,连接超时为10秒,不允许循环重定向

            RequestConfig requestConfig = RequestConfig.custom()

                    .setConnectionRequestTimeout(20000).setConnectTimeout(20000)

                    .setCircularRedirectsAllowed(false)

                    .build();

            Cookie cookie ;

            return HttpClients.custom().setSSLSocketFactory(sslsf)

                    .setUserAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36")

                    .setMaxConnPerRoute(25).setMaxConnPerRoute(256)

                    .setRetryHandler(retryHandler)

                    .setRedirectStrategy(new SelfRedirectStrategy())

                    .setDefaultRequestConfig(requestConfig)

                    .build();

        } catch (KeyManagementException e) {

            e.printStackTrace();

        } catch (NoSuchAlgorithmException e) {

            e.printStackTrace();

        } catch (KeyStoreException e) {

            e.printStackTrace();

        }

        return HttpClients.createDefault();

    }

    /**

     * 带cookiestore

     * @param cookieStore

     * @return

     */

    public static CloseableHttpClient createSSLClientDefaultWithCookie(CookieStore cookieStore) {

        try {

            SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustStrategy() {

                //信任所有

                public boolean isTrusted(X509Certificate[] chain,String authType) throws CertificateException {

                    return true;

                }

            }).build();

            SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext);

            //设置请求重试处理,重试机制,这里如果请求失败会重试5次

            HttpRequestRetryHandler retryHandler = new HttpRequestRetryHandler() {

                @Override

                public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {

                    if (executionCount >= 5) {

                        // Do not retry if over max retry count

                        return false;

                    }

                    if (exception instanceof InterruptedIOException) {

                        // Timeout

                        return false;

                    }

                    if (exception instanceof UnknownHostException) {

                        // Unknown host

                        return false;

                    }

                    if (exception instanceof ConnectTimeoutException) {

                        // Connection refused

                        return false;

                    }

                    if (exception instanceof SSLException) {

                        // SSL handshake exception

                        return false;

                    }

                    HttpClientContext clientContext = HttpClientContext.adapt(context);

                    HttpRequest request = clientContext.getRequest();

                    boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);

                    if (idempotent) {

                        // Retry if the request is considered idempotent

                        return true;

                    }

                    return false;

                }

            };

            //请求参数设置,设置请求超时时间为20秒,连接超时为10秒,不允许循环重定向

            RequestConfig requestConfig = RequestConfig.custom()

                    .setConnectionRequestTimeout(20000).setConnectTimeout(20000)

                    .setCircularRedirectsAllowed(false)

                    .build();

            return HttpClients.custom().setSSLSocketFactory(sslsf)

                    .setUserAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36")

                    .setMaxConnPerRoute(25).setMaxConnPerRoute(256)

                    .setRetryHandler(retryHandler)

                    .setRedirectStrategy(new SelfRedirectStrategy())

                    .setDefaultRequestConfig(requestConfig)

                    .setDefaultCookieStore(cookieStore)

                    .build();

        } catch (KeyManagementException e) {

            e.printStackTrace();

        } catch (NoSuchAlgorithmException e) {

            e.printStackTrace();

        } catch (KeyStoreException e) {

            e.printStackTrace();

        }

        return HttpClients.createDefault();

    }

}

将网页写入到本地的下载类:DownLoadFile.java

package com.amos.crawl;

import com.amos.tool.Configuration;

import com.amos.tool.Tools;

import org.apache.http.*;

import org.apache.http.client.HttpClient;

import org.apache.http.client.HttpRequestRetryHandler;

import org.apache.http.client.config.RequestConfig;

import org.apache.http.client.methods.HttpGet;

import org.apache.http.client.protocol.HttpClientContext;

import org.apache.http.conn.ClientConnectionManager;

import org.apache.http.conn.ConnectTimeoutException;

import org.apache.http.impl.client.AutoRetryHttpClient;

import org.apache.http.impl.client.DefaultHttpClient;

import org.apache.http.protocol.HttpContext;

import javax.net.ssl.SSLException;

import java.io.*;

import java.net.UnknownHostException;

/**

 * Created by amosli on 14-7-9.

 */

public class DownLoadFile {

    public String getFileNameByUrl(String url, String contentType) {

        //移除http http://

        url = url.contains("http://") ? url.substring(7) : url.substring(8);

        //text/html类型

        if (url.contains(".html")) {

            url = url.replaceAll("[\\?/:*|<>\"]", "_");

        } else if (contentType.indexOf("html") != -1) {

            url = url.replaceAll("[\\?/:*|<>\"]", "_") + ".html";

        } else {

            url = url.replaceAll("[\\?/:*|<>\"]", "_") + "." + contentType.substring(contentType.lastIndexOf("/") + 1);

        }

        return url;

    }

    /**

     * 将网页写入到本地

     * @param data

     * @param filePath

     */

    private void saveToLocal(byte[] data, String filePath) {

        try {

            DataOutputStream out = new DataOutputStream(new FileOutputStream(new File(filePath)));

            for(int i=0;i<data.length;i++){

                out.write(data[i]);

            }

            out.flush();

            out.close();

        } catch (Exception e) {

            e.printStackTrace();

        }

    }

    /**

     * 写文件到本地

     *

     * @param httpEntity

     * @param filename

     */

    public static void saveToLocal(HttpEntity httpEntity, String filename) {

        try {

            File dir = new File(Configuration.FILEDIR);

            if (!dir.isDirectory()) {

                dir.mkdir();

            }

            File file = new File(dir.getAbsolutePath() + "/" + filename);

            FileOutputStream fileOutputStream = new FileOutputStream(file);

            InputStream inputStream = httpEntity.getContent();

            if (!file.exists()) {

                file.createNewFile();

            }

            byte[] bytes = new byte[1024];

            int length = 0;

            while ((length = inputStream.read(bytes)) > 0) {

                fileOutputStream.write(bytes, 0, length);

            }

            inputStream.close();

            fileOutputStream.close();

        } catch (Exception e) {

            e.printStackTrace();

        }

    }

    public String downloadFile(String url)  {

        //文件路径

        String filePath=null;

        //1.生成HttpClient对象并设置参数

        HttpClient httpClient = Tools.createSSLClientDefault();

        //2.HttpGet对象并设置参数

        HttpGet httpGet = new HttpGet(url);

        //设置get请求超时5s

        //方法1

        //httpGet.getParams().setParameter("connectTimeout",5000);

        //方法2

        RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(5000).build();

        httpGet.setConfig(requestConfig);

        try {

            HttpResponse httpResponse = httpClient.execute(httpGet);

            int statusCode = httpResponse.getStatusLine().getStatusCode();

            if(statusCode!= HttpStatus.SC_OK){

                System.err.println("Method failed:"+httpResponse.getStatusLine());

                filePath=null;

            }

            filePath=getFileNameByUrl(url,httpResponse.getEntity().getContentType().getValue());

            saveToLocal(httpResponse.getEntity(),filePath);

        } catch (Exception e) {

            e.printStackTrace();

        }

        return filePath;

    }

    public static void main(String args[]) throws IOException {

        String url = "http://websearch.fudan.edu.cn/search_dep.html";

        HttpClient httpClient = new DefaultHttpClient();

        HttpGet httpGet = new HttpGet(url);

        HttpResponse httpResponse = httpClient.execute(httpGet);

        Header contentType = httpResponse.getEntity().getContentType();

        System.out.println("name:" + contentType.getName() + "value:" + contentType.getValue());

        System.out.println(new DownLoadFile().getFileNameByUrl(url, contentType.getValue()));

    }

}

创建一个过滤接口:LinkFilter.java

package com.amos.crawl;

/**

 * Created by amosli on 14-7-10.

 */

public interface LinkFilter {

    public boolean accept(String url);

}

使用HtmlParser的过滤url的方法:HtmlParserTool.java

package com.amos.crawl;

import org.htmlparser.Node;

import org.htmlparser.NodeFilter;

import org.htmlparser.Parser;

import org.htmlparser.filters.NodeClassFilter;

import org.htmlparser.filters.OrFilter;

import org.htmlparser.tags.LinkTag;

import org.htmlparser.util.NodeList;

import java.util.HashSet;

import java.util.Set;

/**

 * Created by amosli on 14-7-10.

 */

public class HtmlParserTool {

    public static Set<String> extractLinks(String url, LinkFilter filter) {

        Set<String> links = new HashSet<String>();

        try {

            Parser parser = new Parser(url);

            parser.setEncoding("GBK");

            //过滤<frame>标签的filter,用来提取frame标签里的src属性

            NodeFilter framFilter = new NodeFilter() {

                @Override

                public boolean accept(Node node) {

                    if (node.getText().contains("frame src=")) {

                        return true;

                    } else {

                        return false;

                    }

                }

            };

            //OrFilter来设置过滤<a>标签和<frame>标签

            OrFilter linkFilter = new OrFilter(new NodeClassFilter(LinkTag.class), framFilter);

            //得到所有经过过滤的标签

            NodeList list = parser.extractAllNodesThatMatch(linkFilter);

            for (int i = 0; i < list.size(); i++) {

                Node tag = list.elementAt(i);

                if (tag instanceof LinkTag) {

                    tag = (LinkTag) tag;

                    String linkURL = ((LinkTag) tag).getLink();

                    //如果符合条件那么将url添加进去

                    if (filter.accept(linkURL)) {

                        links.add(linkURL);

                    }

                } else {//frame 标签

                    //frmae里src属性的链接,如<frame src="test.html" />

                    String frame = tag.getText();

                    int start = frame.indexOf("src=");

                    frame = frame.substring(start);

                    int end = frame.indexOf(" ");

                    if (end == -1) {

                        end = frame.indexOf(">");

                    }

                    String frameUrl = frame.substring(5, end - 1);

                    if (filter.accept(frameUrl)) {

                        links.add(frameUrl);

                    }

                }

            }

        } catch (Exception e) {

            e.printStackTrace();

        }

        return links;

    }

}


管理网页url的实现队列: Queue.java

package com.amos.crawl;

import java.util.LinkedList;

/**

 * Created by amosli on 14-7-9.

 */

public class Queue {

    //使用链表实现队列

    private LinkedList queueList = new LinkedList();

    //入队列

    public void enQueue(Object object) {

        queueList.addLast(object);

    }

    //出队列

    public Object deQueue() {

        return queueList.removeFirst();

    }

    //判断队列是否为空

    public boolean isQueueEmpty() {

        return queueList.isEmpty();

    }

    //判断队列是否包含ject元素..

    public boolean contains(Object object) {

        return queueList.contains(object);

    }

    //判断队列是否为空

    public boolean empty() {

        return queueList.isEmpty();

    }

}

网页链接进出队列的管理:LinkQueue.java

package com.amos.crawl;

import java.util.HashSet;

import java.util.Set;

/**

 * Created by amosli on 14-7-9.

 */

public class LinkQueue {

    //已经访问的队列

    private static Set visitedUrl = new HashSet();

    //未访问的队列

    private static Queue unVisitedUrl = new Queue();

    //获得URL队列

    public static Queue getUnVisitedUrl() {

        return unVisitedUrl;

    }

    public static Set getVisitedUrl() {

        return visitedUrl;

    }

    //添加到访问过的URL队列中

    public static void addVisitedUrl(String url) {

        visitedUrl.add(url);

    }

    //删除已经访问过的URL

    public static void removeVisitedUrl(String url){

        visitedUrl.remove(url);

    }

    //未访问的URL出队列

    public static Object unVisitedUrlDeQueue(){

        return unVisitedUrl.deQueue();

    }

    //保证每个URL只被访问一次,url不能为空,同时已经访问的URL队列中不能包含该url,而且因为已经出队列了所未访问的URL队列中也不能包含该url

    public static void addUnvisitedUrl(String url){

        if(url!=null&&!url.trim().equals("")&&!visitedUrl.contains(url)&&!unVisitedUrl.contains(url))

        unVisitedUrl.enQueue(url);

    }

    //获得已经访问过的URL的数量

    public static int getVisitedUrlNum(){

        return visitedUrl.size();

    }

    //判断未访问的URL队列中是否为空

    public static boolean isUnvisitedUrlsEmpty(){

        return unVisitedUrl.empty();

    }

}

抓取思路是:首先给出要抓取的url==>查询符合条件的url,并将其加入到队列中==>按顺序取出队列中的url,并访问之,同时取出符合条件的url==>下载队列中的url网页,即按层探索,最多限制100条数据.

3.3　截图

Java广度优先爬虫示例(抓取复旦新闻信息)

相关文章