轉載請注明出處:[http://blog.csdn.net/xiaojimanman/article/details/44015983](http://blog.csdn.net/xiaojimanman/article/details/44015983)
[http://www.llwjy.com/blogdetail/5757ce8c007754704b563dd6a47ca1ca.html](http://www.llwjy.com/blogdetail/5757ce8c007754704b563dd6a47ca1ca.html)
個人的博客小站也搭建成功,網址:[www.llwjy.com](http://www.llwjy.com) ,歡迎大家來吐槽~
在前一篇博客中,對實時索引的實現原理做了一些簡單的介紹,這里就介紹下,如何利用Lucene來實現索引的管理(Lucene中已經實現了大部分的功能,我們只需要對其再次封裝即可)。
**逐個擊破**
在Lucene4.3.1中,實現實時索引時,需要將IndexWrite的相關操作委托給TrackingIndexWriter來處理,具體代碼實現如下:
ps:關于如何創建索引這里就不再介紹了,可以參照之前的博客或者該博客后面的完整代碼。
~~~
this.trackingIndexWriter = new TrackingIndexWriter(this.indexWriter);
~~~
同時初始化索引管理對象,代碼如下:
~~~
this.nrtManager = new NRTManager(this.trackingIndexWriter, new SearcherFactory());
~~~
到這里還需要開啟兩個守護線程:內存索引重讀線程和內存數據commit線程。內存索引重讀線程執行的頻率也就是實時索引的時差,由于內存中的數據不會太多,所以這個延時一般也就是在十幾毫秒左右;內存數據commit線程是將內存中的數據寫到磁盤上,不至于數據丟失,如果研究過Lucene源碼的童鞋也許會發現,即使你不執行commit操作,到內存中的數據達到一定的程度,也會將一部分數據寫到磁盤上,只不過重啟服務這部分數據就丟失了同時還會造成一系列的問題,[http://bbs.csdn.net/topics/390677902](http://bbs.csdn.net/topics/390677902) 這個地址下就是commit線程死掉之后造成的一系列問題,感興趣的童鞋可以了解下。
內存重讀線程我們只需要配置下參數啟動即可,代碼如下:
~~~
this.nrtManagerReopenThread = new NRTManagerReopenThread(this.nrtManager, indexReopenMaxStaleSec, indexReopenMinStaleSec);
this.nrtManagerReopenThread.setName("NRTManager Reopen Thread");
this.nrtManagerReopenThread.setPriority(Math.min(Thread.currentThread().getPriority()+2, Thread.MAX_PRIORITY));
this.nrtManagerReopenThread.setDaemon(true);
this.nrtManagerReopenThread.start();
~~~
內存數據commit線程需要自己寫代碼實現,然后啟動該線程即可,代碼如下:
~~~
private class IndexCommitThread extends Thread{
private boolean flag;
public IndexCommitThread(String name){
super(name);
}
@SuppressWarnings("deprecation")
public void run(){
flag = true;
while(flag) {
try {
indexWriter.commit();
if (bprint) {
System.out.println(new Date().toLocaleString() + "\t" + IndexManagerName + "\tcommit");
}
TimeUnit.SECONDS.sleep(indexCommitSeconds);
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e1) {
e1.printStackTrace();
}
}
}
}
~~~
~~~
this.indexCommitThread = new IndexCommitThread(IndexManagerName + "Index Commit Thread");
this.indexCommitThread.setDaemon(true);
this.indexCommitThread.start();
~~~
那又如何像普通的索引那樣使用IndexSearcher呢?當然NrtManager類也提供了相關的方法,可以獲取最新可用的IndexSearcher,代碼如下:
~~~
public IndexSearcher getIndexSearcher(){
try {
return this.nrtManager.acquire();
} catch (IOException e) {
e.printStackTrace();
return null;
}
}
~~~
當然在使用之后別忘記釋放,代碼如下:
~~~
public void release(IndexSearcher searcher){
try {
nrtManager.release(searcher);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
~~~
**另類單例模式**
在之前的博客中,我也多次提到,加載索引是一個相當消耗資源的事情,所以我們不可能每一次索引操作都加載一次索引,所以我們就必須使用單例模式來實現IndexManager類。這里的單例模式又和我們常見的單例模式有所區別,普通的單例模式該類只有一個對象,這里的單例模式是該類有多個對象,下面就簡單的介紹下此處另類的單例模式。
通過前一篇博客最后,你也許會注意到,系統中關于索引的配置信息是存在HashSet對象中,這也就是說這里IndexManager類會實例化多少次取決于HashSet對象,也就是你配置文件讓他實例化多少次就會實例化多少次。既然這樣,怎么還能叫單例模式呢?這里的單例是索引的單例,也就是說一個索引只有一個IndexManager對象,不會存在兩個IndexManager對象去操作同一個索引的情況。具體代碼實現如下:
~~~
/**
* Initialization on Demand Holder式初始化IndexManager
*/
private static class LazyLoadIndexManager {
private static final HashMap<String, IndexManager> indexManager = new HashMap<String, IndexManager>();
static {
for (ConfigBean configBean : IndexConfig.getConfigBean()) {
indexManager.put(configBean.getIndexName(), new IndexManager(configBean));
}
}
}
/**
*@Description: IndexManager私有構造方法
*@Author: lulei
*@Version: 1.1.0
*/
private IndexManager(ConfigBean configBean){
//...
}
public static IndexManager getIndexManager(String indexName){
return LazyLoadIndexManager.indexManager.get(indexName);
}
~~~
這樣我們就可以通過索引名獲取到該索引的IndexManager對象。
**廬山真面目**
說了這么多,下面就把IndexManager的源碼附在最后,感興趣的童鞋可以試試(里面還有一些其他的方法,相信不用介紹也都可以看的懂)
~~~
/**
*@Description: 索引管理類
*/
package com.lulei.lucene.index.manager;
import java.io.File;
import java.io.IOException;
import java.util.Date;
import java.util.HashMap;
import java.util.concurrent.TimeUnit;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.IndexWriterConfig.OpenMode;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.NRTManager;
import org.apache.lucene.search.NRTManager.TrackingIndexWriter;
import org.apache.lucene.search.NRTManagerReopenThread;
import org.apache.lucene.search.SearcherFactory;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.NIOFSDirectory;
import org.apache.lucene.util.Version;
import com.lulei.lucene.index.model.ConfigBean;
import com.lulei.lucene.index.model.IndexConfig;
public class IndexManager {
private IndexWriter indexWriter;
//更新索引文件的IndexWriter
private TrackingIndexWriter trackingIndexWriter;
//索引文件采用的分詞器
private Analyzer analyzer;
//索引管理對象
private NRTManager nrtManager;
//索引重讀線程
private NRTManagerReopenThread nrtManagerReopenThread;
//索引寫入磁盤線程
private IndexCommitThread indexCommitThread;
//索引地址
private String indexPath;
//索引重讀最大、最小時間間隔
private double indexReopenMaxStaleSec;
private double indexReopenMinStaleSec;
//索引commit時間
private int indexCommitSeconds;
//索引名
private String IndexManagerName;
//commit時是否輸出相關信息
private boolean bprint = true;
/**
* Initialization on Demand Holder式初始化IndexManager
*/
private static class LazyLoadIndexManager {
private static final HashMap<String, IndexManager> indexManager = new HashMap<String, IndexManager>();
static {
for (ConfigBean configBean : IndexConfig.getConfigBean()) {
indexManager.put(configBean.getIndexName(), new IndexManager(configBean));
}
}
}
/**
*@Description: IndexManager私有構造方法
*@Author: lulei
*@Version: 1.1.0
*/
private IndexManager(ConfigBean configBean){
//設置相關屬性
analyzer = configBean.getAnalyzer();
indexPath = configBean.getIndexPath();
IndexManagerName = configBean.getIndexName();
indexReopenMaxStaleSec = configBean.getIndexReopenMaxStaleSec();
indexReopenMinStaleSec = configBean.getIndexReopenMinStaleSec();
indexCommitSeconds = configBean.getIndexCommitSeconds();
bprint = configBean.isBprint();
String indexFile = indexPath + IndexManagerName + "/";
//創建或打開索引
IndexWriterConfig indexWriterConfig = new IndexWriterConfig(Version.LUCENE_43, analyzer);
indexWriterConfig.setOpenMode(OpenMode.CREATE_OR_APPEND);
Directory directory = null;
try {
directory = NIOFSDirectory.open(new File(indexFile));
if (IndexWriter.isLocked(directory)){
IndexWriter.unlock(directory);
}
this.indexWriter = new IndexWriter(directory, indexWriterConfig);
this.trackingIndexWriter = new TrackingIndexWriter(this.indexWriter);
this.nrtManager = new NRTManager(this.trackingIndexWriter, new SearcherFactory());
} catch(IOException e){
e.printStackTrace();
}
//開啟守護進程
this.setThread();
}
/**
* @Author: lulei
* @Description: 創建索引管理線程
*/
private void setThread(){
this.nrtManagerReopenThread = new NRTManagerReopenThread(this.nrtManager, indexReopenMaxStaleSec, indexReopenMinStaleSec);
this.nrtManagerReopenThread.setName("NRTManager Reopen Thread");
this.nrtManagerReopenThread.setPriority(Math.min(Thread.currentThread().getPriority()+2, Thread.MAX_PRIORITY));
this.nrtManagerReopenThread.setDaemon(true);
this.nrtManagerReopenThread.start();
this.indexCommitThread = new IndexCommitThread(IndexManagerName + "Index Commit Thread");
this.indexCommitThread.setDaemon(true);
this.indexCommitThread.start();
}
/**
* @return
* @Author:lulei
* @Description: 重啟索引commit線程
*/
public String setCommitThread() {
try {
if (this.indexCommitThread.isAlive()){
return "is alive";
}
this.indexCommitThread = new IndexCommitThread(IndexManagerName + "Index Commit Thread");
this.indexCommitThread.setDaemon(true);
this.indexCommitThread.start();
} catch (Exception e) {
e.printStackTrace();
return "failed";
}
return "reload";
}
/**
*@Description: 索引commit線程
*@Author: lulei
*@Version: 1.1.0
*/
private class IndexCommitThread extends Thread{
private boolean flag;
public IndexCommitThread(String name){
super(name);
}
@SuppressWarnings("deprecation")
public void run(){
flag = true;
while(flag) {
try {
indexWriter.commit();
if (bprint) {
System.out.println(new Date().toLocaleString() + "\t" + IndexManagerName + "\tcommit");
}
TimeUnit.SECONDS.sleep(indexCommitSeconds);
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e1) {
e1.printStackTrace();
}
}
}
}
/**
* @return IndexManager
* @Author: lulei
* @Description: 獲取索引管理類
*/
public static IndexManager getIndexManager(String indexName){
return LazyLoadIndexManager.indexManager.get(indexName);
}
/**
* @@Description:釋放IndexSearcher資源
* @param searcher
*/
public void release(IndexSearcher searcher){
try {
nrtManager.release(searcher);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
/**
* @return IndexSearcher
* @Author: lulei
* @Description: 返回IndexSearcher對象,使用完之后,調用release方法進行釋放
*/
public IndexSearcher getIndexSearcher(){
try {
return this.nrtManager.acquire();
} catch (IOException e) {
e.printStackTrace();
return null;
}
}
public NRTManager getNRTManager(){
return this.nrtManager;
}
public IndexWriter getIndexWriter(){
return this.indexWriter;
}
public TrackingIndexWriter getTrackingIndexWriter(){
return this.trackingIndexWriter;
}
public Analyzer getAnalyzer(){
return analyzer;
}
/**
* @return
* @Author: lulei
* @Description: 獲取索引中的記錄條數
*/
public int getIndexNum(){
return indexWriter.numDocs();
}
}
~~~
ps:最近發現其他網站可能會對博客轉載,上面并沒有源鏈接,如想查看更多關于 [基于lucene的案例開發](http://blog.csdn.net/xiaojimanman/article/category/2841877) 請[點擊這里](http://blog.csdn.net/xiaojimanman/article/category/2841877)。或訪問網址http://blog.csdn.net/xiaojimanman/article/category/2841877 或?[www.llwjy.com](http://www.llwjy.com)
- 前言
- 寫在開始之前
- lucene初始認知
- 索引數學模型
- 索引文件結構
- 創建索引
- 搜索索引
- 分詞器介紹
- Query查詢
- IndexSearcher中檢索方法
- 更新說明
- 案例初識
- JsonUtil &amp; XmlUtil
- 基ClassUtil &amp; CharsetUtil
- ParseUtil &amp; ParseRequest
- 數據庫連接池
- 實現實時索引基本原理
- 實時索引管理類IndexManager
- 實時索引的檢索
- 實時索引的修改
- 查詢語句創建PackQuery
- 縱橫小說更新列表頁抓取
- 縱橫小說簡介頁采集
- 縱橫小說章節列表采集
- 縱橫小說閱讀頁采集
- 縱橫小說數據庫設計
- 縱橫小說數據庫操作
- 縱橫小說分布式采集