Question

我正试图“翻译”一系列案文。更具体地说,我需要证明投入流,在专门的字典中逐年研究,并得出相应的标语“翻译”。然而,我也希望保留投入中的所有原始白色空间、中词等,以便使产出的格式与投入相同,而不是最终成为译文。因此,如果我的投入是这样的话,那么我的意见就是这样。

Term1: Term2 Stopword! Term3 Term4

那么,我想看一下产出。

Term1 : Term2 Stopword! Term3 Term4

(Termi 译自Termi)

Term1 Term2 Term3 Term4

我目前正在做以下工作:

PatternAnalyzer pa = new PatternAnalyzer(Version.LUCENE_31,
                             PatternAnalyzer.WHITESPACE_PATTERN,
                             false, 
                             WordlistLoader.getWordSet(new File(stopWordFilePath)));
TokenStream ts = pa.tokenStream(null, in);
CharTermAttribute charTermAttribute = ts.getAttribute(CharTermAttribute.class);

while (ts.incrementToken()) { // loop over tokens
     String termIn = charTermAttribute.toString(); 
     ...
}

但这当然会失去所有白色空间等。我怎么能够修改,以便重新纳入产出? 感谢!

============ UPDATE!

我试图将原来的行文分为“口号”和“非口号”。看来,这只是罚款。不知道它是否最有效率,但:

public ArrayList splitToWords(String sIn) {

if (sIn == null || sIn.length() == 0) {
    return null;
}

char[] c = sIn.toCharArray();
ArrayList<Token> list = new ArrayList<Token>(); 
int tokenStart = 0;
boolean curIsLetter = Character.isLetter(c[tokenStart]);
for (int pos = tokenStart + 1; pos < c.length; pos++) {
    boolean newIsLetter = Character.isLetter(c[pos]);
    if (newIsLetter == curIsLetter) {
        continue;
    }
    TokenType type = TokenType.NONWORD;
    if (curIsLetter == true)
    {
        type = TokenType.WORD;
    }

    list.add(new Token(new String(c, tokenStart, pos - tokenStart),type));
    tokenStart = pos;

    curIsLetter = newIsLetter;
}
TokenType type = TokenType.NONWORD;
if (curIsLetter == true)
{
    type = TokenType.WORD;
}
list.add(new Token(new String(c, tokenStart, c.length - tokenStart),type));

return list;

}

Answer 1

确实,这并没有真正失去白色空间,你仍然有原来的案文:

So I think you should make use of OffsetAttribute, which contains startOffset() and endOffset() of each term into your original text. This is what lucene uses, for example, to highlight snippets of search results from the original text.

I wrote up a quick test (uses EnglishAnalyzer) to demonstrate: The input is:

Just a test of some ideas. Let s see if it works.

产出如下:

just a test of some idea. let see if it work.

// just for example purposes, not necessarily the most performant.
public void testString() throws Exception {
  String input = "Just a test of some ideas. Let s see if it works.";
  EnglishAnalyzer analyzer = new EnglishAnalyzer(Version.LUCENE_35);
  StringBuilder output = new StringBuilder(input);
  // in some cases, the analyzer will make terms longer or shorter.
  // because of this we must track how much we have adjusted the text so far
  // so that the offsets returned will still work for us via replace()
  int delta = 0;

  TokenStream ts = analyzer.tokenStream("bogus", new StringReader(input));
  CharTermAttribute termAtt = ts.addAttribute(CharTermAttribute.class);
  OffsetAttribute offsetAtt = ts.addAttribute(OffsetAttribute.class);
  ts.reset();
  while (ts.incrementToken()) {
    String term = termAtt.toString();
    int start = offsetAtt.startOffset();
    int end = offsetAtt.endOffset();
    output.replace(delta + start, delta + end, term);
    delta += (term.length() - (end - start));
  iii
  ts.close();

System.out.println(output.toString());

iii

友情链接