org.antlr.runtime
Class TokenRewriteStream

java.lang.Object
  org.antlr.runtime.CommonTokenStream
      org.antlr.runtime.TokenRewriteStream

All Implemented Interfaces:

IntStream, TokenStream

public class TokenRewriteStream
extends CommonTokenStream

Useful for dumping out the input stream after doing some augmentation or other manipulations. You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :) Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1. Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example, CharStream input = new ANTLRFileStream("input"); TLexer lex = new TLexer(input); TokenRewriteStream tokens = new TokenRewriteStream(lex); T parser = new T(tokens); parser.startRule(); Then in the rules, you can execute Token t,u; ... input.insertAfter(t, "text to put after t");} input.insertAfter(u, "text after u");} System.out.println(tokens.toString()); Actually, you have to cast the 'input' to a TokenRewriteStream. :( You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer: tokens.insertAfter("pass1", t, "text to put after t");} tokens.insertAfter("pass2", u, "text after u");} System.out.println(tokens.toString("pass1")); System.out.println(tokens.toString("pass2")); If you don't use named rewrite streams, a "default" stream is used as the first example shows.

Field Summary

static java.lang.String	DEFAULT_PROGRAM_NAME
protected java.util.Map	lastRewriteTokenIndexes Map String (program name) -> Integer index
static int	MIN_TOKEN_INDEX
static int	PROGRAM_INIT_SIZE
protected java.util.Map	programs You may have multiple, named streams of rewrite operations.

Fields inherited from class org.antlr.runtime.CommonTokenStream

channel, channelOverrideMap, discardOffChannelTokens, discardSet, lastMarker, p, tokens, tokenSource

Constructor Summary

TokenRewriteStream()

TokenRewriteStream(TokenSource tokenSource)

TokenRewriteStream(TokenSource tokenSource, int channel)

Method Summary

protected void	addToSortedRewriteList(java.lang.String programName, org.antlr.runtime.TokenRewriteStream.RewriteOperation op) Add an instruction to the rewrite instruction list ordered by the instruction number (use a binary search for efficiency).
protected void	addToSortedRewriteList(org.antlr.runtime.TokenRewriteStream.RewriteOperation op) If op.index > lastRewriteTokenIndexes, just add to the end.
void	delete(int index)
void	delete(int from, int to)
void	delete(java.lang.String programName, int from, int to)
void	delete(java.lang.String programName, Token from, Token to)
void	delete(Token indexT)
void	delete(Token from, Token to)
void	deleteProgram()
void	deleteProgram(java.lang.String programName) Reset the program so that no instructions exist
int	getLastRewriteTokenIndex()
protected int	getLastRewriteTokenIndex(java.lang.String programName)
protected java.util.List	getProgram(java.lang.String name)
protected void	init()
void	insertAfter(int index, java.lang.Object text)
void	insertAfter(java.lang.String programName, int index, java.lang.Object text)
void	insertAfter(java.lang.String programName, Token t, java.lang.Object text)
void	insertAfter(Token t, java.lang.Object text)
void	insertBefore(int index, java.lang.Object text)
void	insertBefore(java.lang.String programName, int index, java.lang.Object text)
void	insertBefore(java.lang.String programName, Token t, java.lang.Object text)
void	insertBefore(Token t, java.lang.Object text)
void	replace(int from, int to, java.lang.Object text)
void	replace(int index, java.lang.Object text)
void	replace(java.lang.String programName, int from, int to, java.lang.Object text)
void	replace(java.lang.String programName, Token from, Token to, java.lang.Object text)
void	replace(Token indexT, java.lang.Object text)
void	replace(Token from, Token to, java.lang.Object text)
void	rollback(int instructionIndex)
void	rollback(java.lang.String programName, int instructionIndex) Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.
protected void	setLastRewriteTokenIndex(java.lang.String programName, int i)
java.lang.String	toDebugString()
java.lang.String	toDebugString(int start, int end)
java.lang.String	toOriginalString()
java.lang.String	toOriginalString(int start, int end)
java.lang.String	toString()
java.lang.String	toString(int start, int end) Return the text of all tokens from start to stop, inclusive.
java.lang.String	toString(java.lang.String programName)
java.lang.String	toString(java.lang.String programName, int start, int end)

Methods inherited from class org.antlr.runtime.CommonTokenStream

consume, discardOffChannelTokens, discardTokenType, fillBuffer, get, getTokens, getTokens, getTokens, getTokens, getTokens, getTokenSource, index, LA, LB, LT, mark, release, rewind, rewind, seek, setTokenSource, setTokenTypeChannel, size, skipOffTokenChannels, skipOffTokenChannelsReverse, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_PROGRAM_NAME

public static final java.lang.String DEFAULT_PROGRAM_NAME

See Also:

Constant Field Values

PROGRAM_INIT_SIZE

public static final int PROGRAM_INIT_SIZE

See Also:

Constant Field Values

MIN_TOKEN_INDEX

public static final int MIN_TOKEN_INDEX

See Also:

Constant Field Values

programs

protected java.util.Map programs

You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) -> rewrite (List)

lastRewriteTokenIndexes

protected java.util.Map lastRewriteTokenIndexes

Map String (program name) -> Integer index

Constructor Detail

TokenRewriteStream

public TokenRewriteStream()

TokenRewriteStream

public TokenRewriteStream(TokenSource tokenSource)

TokenRewriteStream

public TokenRewriteStream(TokenSource tokenSource,
                          int channel)

Method Detail

init

protected void init()

rollback

public void rollback(int instructionIndex)

rollback

public void rollback(java.lang.String programName,
                     int instructionIndex)

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

deleteProgram

public void deleteProgram()

deleteProgram

public void deleteProgram(java.lang.String programName)

Reset the program so that no instructions exist

addToSortedRewriteList

protected void addToSortedRewriteList(org.antlr.runtime.TokenRewriteStream.RewriteOperation op)

If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear

addToSortedRewriteList

protected void addToSortedRewriteList(java.lang.String programName,
                                      org.antlr.runtime.TokenRewriteStream.RewriteOperation op)

Add an instruction to the rewrite instruction list ordered by the instruction number (use a binary search for efficiency). The list is ordered so that toString() can be done efficiently. When there are multiple instructions at the same index, the instructions must be ordered to ensure proper behavior. For example, a delete at index i must kill any replace operation at i. Insert-before operations must come before any replace / delete instructions. If there are multiple insert instructions for a single index, they are done in reverse insertion order so that "insert foo" then "insert bar" yields "foobar" in front rather than "barfoo". This is convenient because I can insert new InsertOp instructions at the index returned by the binary search. A ReplaceOp kills any previous replace op. Since delete is the same as replace with null text, i can check for ReplaceOp and cover DeleteOp at same time. :)

insertAfter

public void insertAfter(Token t,
                        java.lang.Object text)

insertAfter

public void insertAfter(int index,
                        java.lang.Object text)

insertAfter

public void insertAfter(java.lang.String programName,
                        Token t,
                        java.lang.Object text)

insertAfter

public void insertAfter(java.lang.String programName,
                        int index,
                        java.lang.Object text)

insertBefore

public void insertBefore(Token t,
                         java.lang.Object text)

insertBefore

public void insertBefore(int index,
                         java.lang.Object text)

insertBefore

public void insertBefore(java.lang.String programName,
                         Token t,
                         java.lang.Object text)

insertBefore

public void insertBefore(java.lang.String programName,
                         int index,
                         java.lang.Object text)

replace

public void replace(int index,
                    java.lang.Object text)

replace

public void replace(int from,
                    int to,
                    java.lang.Object text)

replace

public void replace(Token indexT,
                    java.lang.Object text)

replace

public void replace(Token from,
                    Token to,
                    java.lang.Object text)

replace

public void replace(java.lang.String programName,
                    int from,
                    int to,
                    java.lang.Object text)

replace

public void replace(java.lang.String programName,
                    Token from,
                    Token to,
                    java.lang.Object text)

delete

public void delete(int index)

delete

public void delete(int from,
                   int to)

delete

public void delete(Token indexT)

delete

public void delete(Token from,
                   Token to)

delete

public void delete(java.lang.String programName,
                   int from,
                   int to)

delete

public void delete(java.lang.String programName,
                   Token from,
                   Token to)

getLastRewriteTokenIndex

public int getLastRewriteTokenIndex()

getLastRewriteTokenIndex

protected int getLastRewriteTokenIndex(java.lang.String programName)

setLastRewriteTokenIndex

protected void setLastRewriteTokenIndex(java.lang.String programName,
                                        int i)

getProgram

protected java.util.List getProgram(java.lang.String name)

toOriginalString

public java.lang.String toOriginalString()

toOriginalString

public java.lang.String toOriginalString(int start,
                                         int end)

toString

public java.lang.String toString()

Overrides:

toString in class CommonTokenStream

toString

public java.lang.String toString(java.lang.String programName)

toString

public java.lang.String toString(int start,
                                 int end)

Description copied from interface: TokenStream

Return the text of all tokens from start to stop, inclusive. If the stream does not buffer all the tokens then it can just return "" or null; Users should not access $ruleLabel.text in an action of course in that case.

Specified by:

toString in interface TokenStream

Overrides:

toString in class CommonTokenStream

toString

public java.lang.String toString(java.lang.String programName,
                                 int start,
                                 int end)

toDebugString

public java.lang.String toDebugString()

toDebugString

public java.lang.String toDebugString(int start,
                                      int end)