English 中文(简体)
Merge Index in Lucene
原标题:Merge index in Lucene
  • 时间:2011-11-21 11:26:29
  •  标签:
  • java
  • lucene

基本上,我是新来的,我利用70份电子邮件文件制定了指数。 第一个指数是头29个文件,其余41个文件用于另一个指数编制。

I tried to search in first indexed file by using lucene and it gave me results as i want... But whenever I just try to merge both the indexes it never do this for me.. For index creation

import java.io.BufferedReader;
import java.io.File;
import java.io.FileFilter;
import java.io.FileReader;
import java.io.IOException;

import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;

public class Indexer {

    public static void main(String[] args) throws Exception {
        if (args.length != 0) {
            throw new IllegalArgumentException("Usage: java "
                    + Indexer.class.getName() + " <index dir> <data dir>");
        }
        String indexDir = "docsOPDir"; //1
        String dataDir = "docsDir"; //2
        long start = System.currentTimeMillis();
        Indexer indexer = new Indexer(indexDir);
        int numIndexed;
        try {
            numIndexed = indexer.index(dataDir, new TextFilesFilter());
        } finally {
            indexer.close();
        }
        long end = System.currentTimeMillis();
        System.out.println("Indexing " + numIndexed + " files took "
                + (end - start) + " milliseconds");
    }

    private IndexWriter writer;

    public Indexer(String indexDir) throws IOException {
        File INDEX_DIR = new File(indexDir);

        INDEX_DIR.mkdir();
        Directory dir = FSDirectory.getDirectory(INDEX_DIR);
        writer = new IndexWriter(dir, new StandardAnalyzer(), true);
        writer.setMergeFactor(1000);
        writer.setRAMBufferSizeMB(50);

    }

    public void close() throws IOException {
        writer.close(); //4
    }

    public int index(String dataDir, FileFilter filter) throws Exception {
        File[] files = new File(dataDir).listFiles();
        for (File f : files) {
            System.out.println("Reading File:"+f);
            if (!f.isDirectory() && !f.isHidden() && f.exists() && f.canRead()
                    && (filter == null || filter.accept(f))) {
                indexFile(f);
            }
        }
        return writer.numRamDocs(); //5
    }

    private static class TextFilesFilter implements FileFilter {
        public boolean accept(File path) {
            return !path.getName().toLowerCase() //6
                    .startsWith("541"); //6
        }
    }

    protected Document getDocument(File f) throws Exception {
        Document doc = new Document();
        doc.add(new Field("subject", getSubject(f),Field.Store.YES, Field.Index.TOKENIZED)); //7
        doc.add(new Field("filename", f.getName(), //8
                Field.Store.YES, Field.Index.NO));//8
        doc.add(new Field("fullpath", f.getCanonicalPath(), //9
                Field.Store.YES, Field.Index.NO));//9
        return doc;
    }
     private String getSubject(File f) throws Exception {
            BufferedReader br = new BufferedReader( new FileReader(f));
            String line ;
            while (( line = br.readLine()) != null) {
                if(line.toUpperCase().startsWith("SUBJECT")){
                   return line; 
                }
                }
            return "NO Subject Found";

        }
    private void indexFile(File f) throws Exception {
        System.out.println("Indexing " + f.getCanonicalPath());
        Document doc = getDocument(f);
        writer.addDocument(doc); //10
    }
}

合并指数:

File INDEXES_DIR  = new File("\docsOP2");
        File INDEX_DIR    = new File("\docs");

        INDEX_DIR.mkdir();

        Date start = new Date();

        try {

            IndexWriter writer = new IndexWriter(INDEX_DIR, 
                                                new StandardAnalyzer(), 
                                                true);
            writer.setMergeFactor(1000);
            writer.setRAMBufferSizeMB(50);

            Directory indexes[] = new Directory[INDEXES_DIR.list().length];

            for (int i = 0; i < INDEXES_DIR.list().length; i++) {
                System.out.println("Adding: " + INDEXES_DIR.list()[i]);
                indexes[i] = FSDirectory.getDirectory(INDEXES_DIR.getAbsolutePath() 
                                                    + "/" + INDEXES_DIR.list()[i]);
                System.out.println(indexes[i]);
            }

            System.out.print("Merging added indexes...");
            writer.addIndexes(indexes);
            System.out.println("done");

            System.out.print("Optimizing index...");
            writer.optimize();
            writer.close();
            System.out.println("done");

            Date end = new Date();
            System.out.println("It took: "+((end.getTime() - start.getTime()) / 1000) 
                                            + """);
问题回答

守则是正确的。 为了帮助你追踪这一问题,将新指数倾斜,看看它包含的内容。

The is a GIST with some code:





相关问题
Spring Properties File

Hi have this j2ee web application developed using spring framework. I have a problem with rendering mnessages in nihongo characters from the properties file. I tried converting the file to ascii using ...

Logging a global ID in multiple components

I have a system which contains multiple applications connected together using JMS and Spring Integration. Messages get sent along a chain of applications. [App A] -> [App B] -> [App C] We set a ...

Java Library Size

If I m given two Java Libraries in Jar format, 1 having no bells and whistles, and the other having lots of them that will mostly go unused.... my question is: How will the larger, mostly unused ...

How to get the Array Class for a given Class in Java?

I have a Class variable that holds a certain type and I need to get a variable that holds the corresponding array class. The best I could come up with is this: Class arrayOfFooClass = java.lang....

SQLite , Derby vs file system

I m working on a Java desktop application that reads and writes from/to different files. I think a better solution would be to replace the file system by a SQLite database. How hard is it to migrate ...

热门标签