企业搜索引擎开发之连接器connector（十八）

时间：2024-12-07 12:05:56

创建并启动连接器实例之后，连接器就会基于Http协议向指定的数据接收服务器发送xmlfeed格式数据，我们可以通过配置http代理服务器抓取当前基于http协议格式的数据（或者也可以通过其他网络抓包工具抓取）

// 设置代理

            /Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("IP地址", "端口"));

            synchronized (this) {

                uc = (HttpURLConnection) feedUrl.openConnection();

            }

如此设置之后，我们就可以打开代理工具清楚的观察到连接器发送的具体数据了

POST http://127.0.0.1:8080/hedgehog-searchEngine/xmlfeed HTTP/1.1

Content-Type: multipart/form-data; boundary=<<

Cache-Control: no-cache

Pragma: no-cache

User-Agent: Java/1.6.0_45

Host: 127.0.0.1:8080

Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

Connection: keep-alive

Content-Length: 31621

--<<

Content-Disposition: form-data; name="datasource"

Content-Type: text/plain

default_collectionName_dbconnector_1401370320421

--<<

Content-Disposition: form-data; name="feedtype"

Content-Type: text/plain

incremental

--<<

Content-Disposition: form-data; name="data"

Content-Type: text/xml

<?xml version='1.0' encoding='UTF-8'?><!DOCTYPE gsafeed PUBLIC "-//Google//DTD GSA Feeds//EN" "gsafeed.dtd">

<gsafeed>

<header>

<datasource>default_collectionName_dbconnector_1401370320421</datasource>

<feedtype>incremental</feedtype>

</header>

<group>

<record url="googleconnector://default_collectionName_dbconnector_1401370320421.localhost/doc?docid=B/9795" displayurl="dbconnector://default_collectionName_dbconnector_1401370320421.localhost/B/9795" action="add" mimetype="text/html">

<metadata>

<meta name="google:displayurl" content="dbconnector://default_collectionName_dbconnector_1401370320421.localhost/B/9795"/>

<meta name="google:mimetype" content="text/html"/>

</metadata>

<content encoding="base64binary">

PGh0bWw+DQo8dGl0bGU+RGF0YWJhc2UgQ29ubmVjdG9yIFJlc3VsdCBkb2NJRD05Nzk1PC90aXRsZT4NCjxib2R5Pg0KPHRhYmxlIGJvcmRlcj0iMSI+DQo8dHIgYmdjb2xvcj0iIzlhY2QzMiI+DQo8dGg+ZG9jX3Rhc2tpZDwvdGg+PHRoPmRvY19zaXRlSUQ8L3RoPjx0aD5kb2NfZGF0ZTwvdGg+PHRoPmRvY190aXRsZTwvdGg+PHRoPmRvY19ocmVmPC90aD48dGg+ZG9jSUQ8L3RoPjx0aD5kb2NfY2F0ZUlEPC90aD48dGg+ZG9jX2NoaWxkY2F0ZUlEPC90aD4NCjwvdHI+DQo8dHI+DQo8dGQ+MTwvdGQ+PHRkPjE1PC90ZD48dGQ+MjAxMi0wOC0wMzwvdGQ+PHRkPuWMl+S6rOaWsOS4lue6qumlreW6l+WKnuWFrOalvDwvdGQ+PHRkPmh0dHA6Ly8yMTAuNzUuMjExLjUzL2djanN6bC5wclByb2plY3QucHJHQ0pTX1pMX1ZfUFJPSl9BUFBSX0lORk9fUVVFUlkuZG8/Y29kZT0zMTQ5NjImYW1wO3NlY1RhZz1wcm9qZWN0JmFtcDtzeXNvcmdhbmlkPTc1PC90ZD48dGQ+OTc5NTwvdGQ+PHRkPjE8L3RkPjx0ZD4xPC90ZD4NCjwvdHI+DQo8L3RhYmxlPg0KPC9ib2R5Pg0KPC9odG1sPg0K

</content>

</record>

……

</group>

</gsafeed>

--<<--

分析上面的数据格式，可以观察到发送方式为POST，发送元素项为datasource feedtype data（datasouce为连接实例名，feedtype表示增量信息，data即为xmlfeed数据）

xmlfeed数据部分，我们可以参考官方的dtd文件

<?xml version="1.0" encoding="UTF-8"?>

    <!ELEMENT gsafeed (header, group+)>

    <!ELEMENT header (datasource, feedtype)>

    <!-- datasource name should match the regex [a-zA-Z_][a-zA-Z0-9_-]*,

        the first character must be a letter or underscore,

        the rest of the characters can be alphanumeric, dash, or underscore. -->

    <!ELEMENT datasource (#PCDATA)>

    <!-- feedtype must be either 'full', 'incremental', or 'metadata-and-url' -->

    <!ELEMENT feedtype (#PCDATA)>

    <!-- group element lets you group records together and

        specify a common action for them -->

    <!ELEMENT group (record*)>

    <!-- record element can have attribute that overrides group's element-->

    <!ELEMENT record (metadata*,content*)>

    <!ELEMENT metadata (meta*)>

    <!ELEMENT meta EMPTY>

    <!ELEMENT content (#PCDATA)>

    <!-- last-modified date as per RFC822 -->

    <!-- default is 'add' -->

    <!ATTLIST group action (add|delete) "add">

    <!ATTLIST record

        url CDATA #REQUIRED

        displayurl CDATA #IMPLIED

        action (add|delete) #IMPLIED

        mimetype CDATA #IMPLIED

        last-modified CDATA #IMPLIED

        lock (true|false) "false"

        authmethod (none|httpbasic|ntlm|httpsso) "none">

    <!ATTLIST meta

        name CDATA #REQUIRED

        content CDATA #REQUIRED>

    <!-- if encoding is specified it must be base64binary as that is the only

        binary encoding that is supported -->

    <!ATTLIST content encoding (base64binary) #IMPLIED>

接下来我们便可以在数据接收服务器端接收这些数据并解析之

具体解析过程不再描述，读者可以参考下面的相关资料，本人推荐woodstox这款解析器(符合stax规范)

使用 StAX 解析 XML，第 1 部分: Streaming API for XML (StAX) 简介 
http://www.ibm.com/developerworks/cn/xml/x-stax1.html

使用 StAX 解析 XML，第 2 部分: 拉式解析和事件 
http://www.ibm.com/developerworks/cn/xml/x-stax2.html

使用 StAX 解析 XML，第 3 部分: 使用定制事件和编写 XML 
http://www.ibm.com/developerworks/cn/xml/x-stax3.html

Geronimo 叛逆者: 使用集成软件包：Codehaus 的 Woodstox 
http://www.ibm.com/developerworks/cn/opensource/os-ag-renegade15/

Woodstox官网

http://woodstox.codehaus.org/

---------------------------------------------------------------------------

本系列企业搜索引擎开发之连接器connector系本人原创

转载请注明出处 博客园 刺猬的温驯

本人邮箱： chenying998179@163#com （#改为.）

本文链接 http://www.cnblogs.com/chenying99/p/3765047.html

相关文章