Lucene 查询分页技术

常用的Lucene查询代码如下所示，该代码的作用是将path路径下的所有索引信息返回

 public String matchAll(String path) {
         try {
             Directory directory = FSDirectory.open(new File(path));
             DirectoryReader reader = DirectoryReader.open(directory);
             IndexSearcher searcher = new IndexSearcher(reader);
             MatchAllDocsQuery query = new MatchAllDocsQuery();

             ScoreDoc[] hits = searcher.search(query, null, Integer.MAX_VALUE).scoreDocs;
             StringBuffer buffer = new StringBuffer();
             for (int i = 0; i < hits.length; i++) {
                 Document hitDocument = searcher.doc(hits[i].doc);
 //                System.out.println(hitDocument.get("key")
 //                        + "......"+hitDocument.get("value"));
                 buffer.append(hitDocument.get("key")+";"+hitDocument.get("value")+"|");
             }
             return buffer.toString();
         } catch (IOException e) {
             e.printStackTrace();
         }
         return null;
     }

但是当该文件夹下索引的数目比较巨大，那么在执行以下代码的时候，则会出现java.lang.OutOfMemoryError: Java heap space的提示

ScoreDoc[] hits = searcher.search(query, null, Integer.MAX_VALUE).scoreDocs;

这时候，我们可以考虑使用分页技术，比如以前大约1亿条数据，我们可以将其分成100个100W的页，每次对100W条索引数据进行处理，这样就可以避免上述情况的发生。在Lucene 中，我们使用searchAfter的方法实现上述功能。它的官方API介绍如下所示：

public TopDocs searchAfter(ScoreDoc after,
Query query,
                  int n)
                    throws IOException

Finds the top n hits for query where all results are after a previous result (after).

By passing the bottom result from a previous page as after, this method can be used for efficient 'deep-paging' across potentially large result sets.

Throws:

BooleanQuery.TooManyClauses

IOException

 private String transToContent(IndexSearcher searcher,TopDocs topDocs) throws IOException {
         ScoreDoc[] scoreDocs = topDocs.scoreDocs;
         StringBuffer sb = new StringBuffer();
         for(int i=0; i<scoreDocs.length; i++) {
             Document doc  = searcher.doc(scoreDocs[i].doc);
             sb.append(doc.get("key")+";"+doc.get("value")+"|");
         }
         return sb.toString();
     }

     private void matchAll(String path) {
         try {
             Directory directory = FSDirectory.open(new File(path));
             DirectoryReader reader = DirectoryReader.open(directory);
             IndexSearcher searcher = new IndexSearcher(reader);

             ScoreDoc after = null;
             TopDocs topDocs = searcher.searchAfter(after, new MatchAllDocsQuery(), Preference.PAGE_SIZE);
             int curPage = 1;
             while(topDocs.scoreDocs.length > 0) {
                 System.out.println("Current Page:"+ (curPage++) );
                 System.out.println(transToContent(searcher, topDocs));
                 after = topDocs.scoreDocs[topDocs.scoreDocs.length -1];
                 topDocs = searcher.searchAfter(after, new MatchAllDocsQuery(), Preference.PAGE_SIZE);
             }
         } catch (IOException e) {
             e.printStackTrace();
         }
     }