fr.umlv.tatoo.runtime.lexer
Class Tokenizer<R,B extends LexerBuffer>

java.lang.Object
  extended by fr.umlv.tatoo.runtime.lexer.Tokenizer<R,B>
Type Parameters:
R - type of rules
B - type of the buffer
Direct Known Subclasses:
Scanner

public class Tokenizer<R,B extends LexerBuffer>
extends Object

Provide a simple tokenizer that can be used to implement LL algorithm. The buffer must support the read operation. If you want an iterator, use Scanner instead.

Author:
Julien
See Also:
Scanner

Method Summary
static
<R,B extends LexerBuffer>
Tokenizer<R,B>
createTokenizer(LexerTable<R> lexerTable, B buffer)
          Creates a tokenizer that extracts character from a buffer and matches them against rules.
static
<R,B extends LexerBuffer>
Tokenizer<R,B>
createTokenizer(LexerTable<R> lexerTable, B buffer, Iterable<? extends R> blanks)
          Creates a scanner that extracts character from a buffer and matches them against rules.
 boolean eof()
          Indicated wether eod-of-file is reached.
 B getBuffer()
          Returns the underlying buffer.
 LexerTable<R> getLexerTable()
          Returns the rule tables for this process
 R getNext()
          Returns the rule successfully matched by a prior call to hasNext.
 boolean hasNext(Iterable<? extends R> rules)
          Indicates if input contains one of the tokens passed as argument.
 boolean hasNext(R... rules)
          This method wraps its arguments and calls hasNext(Iterable).
 R next(Iterable<? extends R> rules)
          Returns the next matching rule.
 R next(R... rules)
          This method wraps its arguments and calls next(Iterable).
 void reset(B buffer)
          Reset the tokenizer to perform a new analysis.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Method Detail

createTokenizer

public static <R,B extends LexerBuffer> Tokenizer<R,B> createTokenizer(LexerTable<R> lexerTable,
                                                                       B buffer)
Creates a tokenizer that extracts character from a buffer and matches them against rules. The buffer must support the read operation.

Type Parameters:
R - type of rules.
B - type of buffer
Parameters:
lexerTable - data table of the scanner.
buffer - the buffer used to extract characters. the lexing process
Returns:
a new scanner

createTokenizer

public static <R,B extends LexerBuffer> Tokenizer<R,B> createTokenizer(LexerTable<R> lexerTable,
                                                                       B buffer,
                                                                       Iterable<? extends R> blanks)
Creates a scanner that extracts character from a buffer and matches them against rules.

Type Parameters:
R - type of rules.
B - type of buffer
Parameters:
lexerTable - data table of the scanner.
buffer - the buffer used to extract characters. the lexing process
blanks - the tokens to be skipped
Returns:
a new scanner

hasNext

public boolean hasNext(Iterable<? extends R> rules)
                throws IOException
Indicates if input contains one of the tokens passed as argument. A call to this method discards the previous token from the buffer.

Parameters:
rules - the tokens to match
Returns:
true if input contains one of these tokens
Throws:
IOException

hasNext

public boolean hasNext(R... rules)
                throws IOException
This method wraps its arguments and calls hasNext(Iterable).

Parameters:
rules - the tokens to match
Returns:
true if input contains one of these tokens
Throws:
IOException
See Also:
hasNext(Iterable)

next

public R next(Iterable<? extends R> rules)
       throws IOException
Returns the next matching rule. To extract the character corresponding to the rule, the developer needs to use the method getBuffer() and the devoted method of the buffer. A call to this method discards the previous token from the buffer.

Parameters:
rules - the tokens to match
Returns:
the next matching rule.
Throws:
IOException
See Also:
TokenBuffer.view()

next

public R next(R... rules)
       throws IOException
This method wraps its arguments and calls next(Iterable).

Parameters:
rules - the tokens to match
Returns:
the next matching rule.
Throws:
IOException
See Also:
next(Iterable)

getNext

public R getNext()
Returns the rule successfully matched by a prior call to hasNext. This method can only be called after hasNext has returned true.

Returns:
the rule successfully matched by a prior call to hasNext.
See Also:
hasNext(Iterable)

getBuffer

public B getBuffer()
Returns the underlying buffer.

Returns:
the underlying buffer.

getLexerTable

public LexerTable<R> getLexerTable()
Returns the rule tables for this process

Returns:
the rule tables for this process

reset

public void reset(B buffer)
Reset the tokenizer to perform a new analysis. All current states are discarded. The new buffer is the buffer supplied.


eof

public boolean eof()
            throws IOException
Indicated wether eod-of-file is reached.

Returns:
true if end-of-file is reached
Throws:
IOException - when an i/o error occurs