Question

我正在创建一个程序，可以索引不同文件夹中的许多文本文件。所以这意味着每个文件夹都有索引，其索引存储在另一个文件夹中。所以这另一个文件夹就像我计算机中所有文件的通用索引。我正在使用lucene来实现这一点，因为lucene完全支持增量更新。这是我用它来编制索引的源代码。

public class SimpleFileIndexer {


public static void main(String[] args) throws Exception   {

    int i=0;
    while(i<2) {
    File indexDir = new File("C:/Users/Raden/Documents/myindex");
    File dataDir = new File("C:/Users/Raden/Documents/indexthis");
    String suffix = "txt";

    SimpleFileIndexer indexer = new SimpleFileIndexer();

    int numIndex = indexer.index(indexDir, dataDir, suffix);

    System.out.println("Total files indexed " + numIndex);
    i++;
    Thread.sleep(1000);

    }
}


private int index(File indexDir, File dataDir, String suffix) throws Exception {
    RAMDirectory ramDir = new RAMDirectory();          // 1
    @SuppressWarnings("deprecation")
    IndexWriter indexWriter = new IndexWriter(
            ramDir,                                    // 2
            new StandardAnalyzer(Version.LUCENE_CURRENT),
            true,
            IndexWriter.MaxFieldLength.UNLIMITED);
    indexWriter.setUseCompoundFile(false);
    indexDirectory(indexWriter, dataDir, suffix);
    int numIndexed = indexWriter.maxDoc();
    indexWriter.optimize();
    indexWriter.close();

    Directory.copy(ramDir, FSDirectory.open(indexDir), false); // 3

    return numIndexed;
}


private void indexDirectory(IndexWriter indexWriter, File dataDir, String suffix)  throws IOException {
    File[] files = dataDir.listFiles();
    for (int i = 0; i < files.length; i++) {
        File f = files[i];
        if (f.isDirectory()) {
            indexDirectory(indexWriter, f, suffix);
        }
        else {
            indexFileWithIndexWriter(indexWriter, f, suffix);
        }
    }
}

private void indexFileWithIndexWriter(IndexWriter indexWriter, File f, String suffix) throws IOException {
    if (f.isHidden() || f.isDirectory() || !f.canRead() || !f.exists()) {
        return;
    }
    if (suffix!=null && !f.getName().endsWith(suffix)) {
        return;
    }
    System.out.println("Indexing file " + f.getCanonicalPath());

    Document doc = new Document();
    doc.add(new Field("contents", new FileReader(f)));      
doc.add(new Field("filename", f.getCanonicalPath(), Field.Store.YES, Field.Index.ANALYZED));
    indexWriter.addDocument(doc);
} }

这是我用来搜索lucene创建的索引的源代码

public class SimpleSearcher {

public static void main(String[] args) throws Exception {

    File indexDir = new File("C:/Users/Raden/Documents/myindex");
    String query = "revolution";
    int hits = 100;

    SimpleSearcher searcher = new SimpleSearcher();
    searcher.searchIndex(indexDir, query, hits);

}

private void searchIndex(File indexDir, String queryStr, int maxHits) throws Exception {

    Directory directory = FSDirectory.open(indexDir);

    IndexSearcher searcher = new IndexSearcher(directory);
    @SuppressWarnings("deprecation")
    QueryParser parser = new QueryParser(Version.LUCENE_30, "contents", new StandardAnalyzer(Version.LUCENE_CURRENT));
    Query query = parser.parse(queryStr);

    TopDocs topDocs = searcher.search(query, maxHits);

    ScoreDoc[] hits = topDocs.scoreDocs;
    for (int i = 0; i < hits.length; i++) {
        int docId = hits[i].doc;
        Document d = searcher.doc(docId);
        System.out.println(d.get("filename"));
    }

    System.out.println("Found " + hits.length);

}

}

我现在遇到的问题是我上面创建的索引程序似乎无法进行任何增量更新。我的意思是我可以搜索一个文本文件，但只能搜索我已编入索引的最后一个文件夹中存在的文件，而我已编入索引的另一个上一个文件夹似乎在搜索结果中缺失并且没有显示。你能告诉我我的代码出了什么问题吗？我只是希望能够在源代码中使用增量更新功能。所以从本质上讲，我的程序似乎用新的索引覆盖现有索引而不是合并它。

谢谢

Answer 1

Directory.copy()会覆盖目标目录，您需要使用IndexWriter.addIndexes()将新目录索引合并到主目录中。

您也可以重新打开主索引并直接向其添加文档。 RAMDirectory不一定比正确调整的缓冲区和合并因子设置更有效（参见IndexWriter docs）。

更新：代替Directory.copy()，您需要打开ramDir进行阅读，indexDir进行撰写并致电.addIndexes作者indexDir并将其传递给ramDir读者。或者，您可以使用.addIndexesNoOptimize并直接传递ramDir（无需打开阅读器）并在关闭前优化索引。

但实际上，首先跳过RAMDir并在indexDir上打开一个作家可能更容易。还可以更轻松地更新已更改的文件。

示例

private int index(File indexDir, File dataDir, String suffix) throws Exception { RAMDirectory ramDir = new RAMDirectory(); IndexWriter indexWriter = new IndexWriter(ramDir, new StandardAnalyzer(Version.LUCENE_CURRENT), true, IndexWriter.MaxFieldLength.UNLIMITED); indexWriter.setUseCompoundFile(false); indexDirectory(indexWriter, dataDir, suffix); int numIndexed = indexWriter.maxDoc(); indexWriter.optimize(); indexWriter.close(); IndexWriter index = new IndexWriter(FSDirectory.open(indexDir), new StandardAnalyzer(Version.LUCENE_CURRENT), true, IndexWriter.MaxFieldLength.UNLIMITED); index.addIndexesNoOptimize(ramDir); index.optimize(); index.close(); return numIndexed; }

但是，这也很好：

private int index(File indexDir, File dataDir, String suffix) throws Exception { IndexWriter index = new IndexWriter(FSDirectory.open(indexDir), new StandardAnalyzer(Version.LUCENE_CURRENT), true, IndexWriter.MaxFieldLength.UNLIMITED); // tweak the settings for your hardware index.setUseCompoundFile(false); index.setRAMBufferSizeMB(256); index.setMergeFactor(30); indexDirectory(index, dataDir, suffix); index.optimize(); int numIndexed = index.maxDoc(); index.close(); // you'll need to update indexDirectory() to keep track of indexed files return numIndexed; }

lucene中增量更新的问题

1 个答案: