I'm trying to do a Java interpreter for a new language and for this I need to tokenize each one of the characters. I found Jasic that is an already developed interpreter developed in Java so I'm using it as a guide. For the tokenizer I'm trying to change the the if statement that define that a character is a comment. Here is the code:
public static List<Token> tokenize(String source) {
List<Token> tokens = new ArrayList<Token>();
int n=0;
String token = "";
TokenizeState state = TokenizeState.DEFAULT;
// Many tokens are a single character, like operators and ().
String charTokens = "\n=+-*/<>()";
TokenType[] tokenTypes = { TokenType.LINE, TokenType.EQUALS,
TokenType.OPERATOR, TokenType.OPERATOR, TokenType.OPERATOR,
TokenType.OPERATOR, TokenType.OPERATOR, TokenType.OPERATOR,
TokenType.LEFT_PAREN, TokenType.RIGHT_PAREN
};
// Scan through the code one character at a time, building up the list
// of tokens.
for (int i = 0; i < source.length(); i++) {
char c = source.charAt(i);
switch (state) {
case DEFAULT:
if (charTokens.indexOf(c) != -1) {
tokens.add(new Token(Character.toString(c),
tokenTypes[charTokens.indexOf(c)]));
} else if (Character.isLetter(c)) {
token += c;
state = TokenizeState.WORD;
} else if (Character.isDigit(c)) {
token += c;
state = TokenizeState.NUMBER;
} else if (c == '"') {
state = TokenizeState.STRING;
} else if (c == '/'){ // This comparision won't work
state = TokenizeState.COMMENT;
}
/* This is how it was before I tried to change it
else if (c == '\'') {
state = TokenizeState.COMMENT;
}*/
break;
case WORD:
// Here is more code, but not relevant for this question
}
I already tried even comparing the hashcode of the character and the type, also I tried to use Character.valueOf(c) to get only the value and compare it, like this: Character.valueOf(c) == '/' but nothing worked.
I've searched for answers but everything is about backslashes. So, my question is, do you know why it isn't working? or what I'm doing wrong that the if statement isn't accepting the comparision?
Aucun commentaire:
Enregistrer un commentaire