fr.umlv.tatoo.runtime.lexer
Class Tokenizer<R,B extends LexerBuffer>

java.lang.Object
  fr.umlv.tatoo.runtime.lexer.Tokenizer<R,B>

Type Parameters:

R - type of rules

B - type of the buffer

Direct Known Subclasses:

Scanner

public class Tokenizer<R,B extends LexerBuffer>
extends Object

Provide a simple tokenizer that can be used to implement LL algorithm. The buffer must support the read operation. If you want an iterator, use Scanner instead.

Author:

Julien

See Also:

Scanner

Method Summary

static <R,B extends LexerBuffer> Tokenizer<R,B>	createTokenizer(LexerTable<R> lexerTable, B buffer) Creates a tokenizer that extracts character from a buffer and matches them against rules.
static <R,B extends LexerBuffer> Tokenizer<R,B>	createTokenizer(LexerTable<R> lexerTable, B buffer, Iterable<? extends R> blanks) Creates a scanner that extracts character from a buffer and matches them against rules.
boolean	eof() Indicated wether eod-of-file is reached.
B	getBuffer() Returns the underlying buffer.
LexerTable<R>	getLexerTable() Returns the rule tables for this process
R	getNext() Returns the rule successfully matched by a prior call to hasNext.
boolean	hasNext(Iterable<? extends R> rules) Indicates if input contains one of the tokens passed as argument.
boolean	hasNext(R... rules) This method wraps its arguments and calls hasNext(Iterable).
R	next(Iterable<? extends R> rules) Returns the next matching rule.
R	next(R... rules) This method wraps its arguments and calls next(Iterable).
void	reset(B buffer) Reset the tokenizer to perform a new analysis.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

createTokenizer

public static <R,B extends LexerBuffer> Tokenizer<R,B> createTokenizer(LexerTable<R> lexerTable,
                                                                       B buffer)

Creates a tokenizer that extracts character from a buffer and matches them against rules. The buffer must support the read operation.

Type Parameters:

R - type of rules.

B - type of buffer

Parameters:

lexerTable - data table of the scanner.

buffer - the buffer used to extract characters. the lexing process

Returns:

a new scanner

createTokenizer

public static <R,B extends LexerBuffer> Tokenizer<R,B> createTokenizer(LexerTable<R> lexerTable,
                                                                       B buffer,
                                                                       Iterable<? extends R> blanks)

Creates a scanner that extracts character from a buffer and matches them against rules.

Type Parameters:

R - type of rules.

B - type of buffer

Parameters:

lexerTable - data table of the scanner.

buffer - the buffer used to extract characters. the lexing process

blanks - the tokens to be skipped

Returns:

a new scanner

hasNext

public boolean hasNext(Iterable<? extends R> rules)
                throws IOException

Indicates if input contains one of the tokens passed as argument. A call to this method discards the previous token from the buffer.

Parameters:

rules - the tokens to match

Returns:

true if input contains one of these tokens

Throws:

IOException

hasNext

public boolean hasNext(R... rules)
                throws IOException

This method wraps its arguments and calls hasNext(Iterable).

Parameters:

rules - the tokens to match

Returns:

true if input contains one of these tokens

Throws:

IOException

See Also:

hasNext(Iterable)

next

public R next(Iterable<? extends R> rules)
       throws IOException

Returns the next matching rule. To extract the character corresponding to the rule, the developer needs to use the method getBuffer() and the devoted method of the buffer. A call to this method discards the previous token from the buffer.

Parameters:

rules - the tokens to match

Returns:

the next matching rule.

Throws:

IOException

See Also:

TokenBuffer.view()

next

public R next(R... rules)
       throws IOException

This method wraps its arguments and calls next(Iterable).

Parameters:

rules - the tokens to match

Returns:

the next matching rule.

Throws:

IOException

See Also:

next(Iterable)

getNext

public R getNext()

Returns the rule successfully matched by a prior call to hasNext. This method can only be called after hasNext has returned true.

Returns:

the rule successfully matched by a prior call to hasNext.

See Also:

hasNext(Iterable)

getBuffer

public B getBuffer()

Returns the underlying buffer.

Returns:

the underlying buffer.

getLexerTable

public LexerTable<R> getLexerTable()

Returns the rule tables for this process

Returns:

the rule tables for this process

reset

public void reset(B buffer)

Reset the tokenizer to perform a new analysis. All current states are discarded. The new buffer is the buffer supplied.

eof

public boolean eof()
            throws IOException

Indicated wether eod-of-file is reached.

Returns:

true if end-of-file is reached

Throws:

IOException - when an i/o error occurs