net.sf.saxon.expr
public final class Tokenizer extends Object
Field Summary | |
---|---|
static int | BARE_NAME_STATE
State in which a name is NOT to be merged with what comes next, for example "(" |
int | currentToken
The number identifying the most recently read token |
int | currentTokenStartOffset
The position in the input expression where the current token starts |
String | currentTokenValue
The string value of the most recently read token |
static int | DEFAULT_STATE
Initial default state of the Tokenizer |
String | input
The string being parsed |
int | inputOffset
The current position within the input string |
static int | OPERATOR_STATE
State in which the next thing to be read is an operator |
int | startLineNumber
The starting line number (for XPath in XSLT, the line number in the stylesheet) |
static int | SEQUENCE_TYPE_STATE
State in which the next thing to be read is a SequenceType |
Method Summary | |
---|---|
int | getColumnNumber()
Get the column number of the current token |
int | getColumnNumber(int offset) |
long | getLineAndColumn(int offset)
Get the line and column number corresponding to a given offset in the input expression,
as a long value with the line number in the top half
and the column number in the lower half |
int | getLineNumber()
Get the line number of the current token |
int | getLineNumber(int offset) |
int | getState() |
void | lookAhead()
Look ahead by one token. |
void | next()
Get the next token from the input expression. |
char | nextChar()
Read next character directly. |
String | recentText()
Get the most recently read text (for use in an error message) |
void | setState(int state) |
void | tokenize(String input, int start, int end, int lineNumber)
Prepare a string for tokenization.
|
void | treatCurrentAsOperator()
Force the current token to be treated as an operator if possible |
void | unreadChar()
Step back one character. |
Returns: the line and column number, packed together
Throws: net.sf.saxon.trans.StaticError if a lexical error occurs
Throws: net.sf.saxon.trans.StaticError if a lexical error is detected
Returns: the next character from the input
Throws: StringIndexOutOfBoundsException if an attempt is made to read beyond the end of the string. This will only occur in the event of a syntax error in the input.
Parameters: input the string to be tokenized start start point within the string end end point within the string (last character not read): -1 means end of string
Throws: net.sf.saxon.trans.StaticError if a lexical error occurs, e.g. unmatched string quotes