ArabicLetterTokenizer (Lucene 2.9.4 API)

Skip navigation links

All Classes

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

java.lang.Object
- org.apache.lucene.util.AttributeSource
- - org.apache.lucene.analysis.TokenStream
  - - org.apache.lucene.analysis.Tokenizer
    - - org.apache.lucene.analysis.CharTokenizer
      - org.apache.lucene.analysis.LetterTokenizer
        
        org.apache.lucene.analysis.ar.ArabicLetterTokenizer

```
public class ArabicLetterTokenizer
extends org.apache.lucene.analysis.LetterTokenizer
```
Tokenizer that breaks text into runs of letters and diacritics.
The problem with the standard Letter tokenizer is that it fails on diacritics. Handling similar to this is necessary for Indic Scripts, Hebrew, Thaana, etc.

Nested Class Summary
- Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
  org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Field Summary
- Fields inherited from class org.apache.lucene.analysis.Tokenizer
  input

Constructor Summary

Constructors
Constructor and Description
`ArabicLetterTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, java.io.Reader in)`
`ArabicLetterTokenizer(org.apache.lucene.util.AttributeSource source, java.io.Reader in)`
`ArabicLetterTokenizer(java.io.Reader in)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type Method and Description

protected boolean isTokenChar(char c)
Allows for Letter category or NonspacingMark category
- Methods inherited from class org.apache.lucene.analysis.CharTokenizer
  end, incrementToken, next, next, normalize, reset
- Methods inherited from class org.apache.lucene.analysis.Tokenizer
  close, correctOffset
- Methods inherited from class org.apache.lucene.analysis.TokenStream
  getOnlyUseNewAPI, reset, setOnlyUseNewAPI
- Methods inherited from class org.apache.lucene.util.AttributeSource
  addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
- Methods inherited from class java.lang.Object
  clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ArabicLetterTokenizer

public ArabicLetterTokenizer(java.io.Reader in)

ArabicLetterTokenizer

public ArabicLetterTokenizer(org.apache.lucene.util.AttributeSource source,
                             java.io.Reader in)

ArabicLetterTokenizer

public ArabicLetterTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
                             java.io.Reader in)

Method Detail
- isTokenChar
```
protected boolean isTokenChar(char c)
```
  Allows for Letter category or NonspacingMark category
  
  Overrides:
  
  isTokenChar in class org.apache.lucene.analysis.LetterTokenizer
  
  See Also:
  
  LetterTokenizer.isTokenChar(char)

Skip navigation links

All Classes

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

Copyright © 2000-2016 Apache Software Foundation. All Rights Reserved.