Question

我在AntlWorks中创建了一个简单的语法。然后我生成了代码，我有两个文件：grammarLexer.java和grammarParser.java。我的目标是创建将语法映射到java语言。接下来我该怎么做才能实现它？

FOR        :     WS* 'for' WS+ VARIABLE WS+ DIGIT+ WS+ DIGIT+ WS* ENTER  ( FOR | WHILE | IF | PRINT | DECLARATION | ENTER | (WS* FUNCTION) | INC_DEC )* WS* 'end' WS* ENTER;
WHILE        :     WS* 'while' WS+ (VARIABLE | DIGIT+) WS* EQ_OPERATOR WS* (VARIABLE | DIGIT+) WS* ENTER  (FOR | WHILE | IF | PRINT | DECLARATION | ENTER | (WS* FUNCTION) | (WS* INC_DEC))* WS* 'end' WS* ENTER;
IF        :         WS* 'if' WS+ ( FUNCTION | VARIABLE | DIGIT+) WS* EQ_OPERATOR WS* (VARIABLE | DIGIT+) WS* ENTER (FOR | WHILE | IF | PRINT | DECLARATION | ENTER | (WS* FUNCTION) | INC_DEC)* ( WS* 'else' ENTER (FOR | WHILE | IF | PRINT | DECLARATION | ENTER | (WS* FUNCTION) | (WS* INC_DEC))*)? WS* 'end' WS* ENTER;

CHAR        :     ('a'..'z'|'A'..'Z')+;
EQ_OPERATOR    :    ('<' | '>' | '==' | '>=' | '<=' | '!=');
DIGIT        :     '0'..'9'+;
ENTER        :     '\n';
WS        :     ' ' | '\t';

PRINT_TEMPLATE  :     WS+ (('"' (CHAR | DIGIT | WS)* '"') | VARIABLE | DIGIT+ | FUNCTION | INC_DEC);
PRINT             :     WS* 'print' PRINT_TEMPLATE (',' PRINT_TEMPLATE)*  WS* ENTER;

VARIABLE        :    CHAR(CHAR|DIGIT)*;
FUN_TEMPLATE    :    WS* (VARIABLE | DIGIT+ | '"' (CHAR | DIGIT | WS)* '"');
FUNCTION        :    VARIABLE '(' (FUN_TEMPLATE (WS* ',' FUN_TEMPLATE)*)? ')' WS* ENTER*;

DECLARATION     :    WS* VARIABLE WS* ('=' WS* (DIGIT+ | '"' (CHAR | DIGIT | WS)* '"' | VARIABLE)) WS* ENTER;
FUNCTION_DEC    :    WS*'def' WS* FUNCTION ( FOR | WHILE | IF | PRINT | DECLARATION | ENTER | (WS* FUNCTION) | INC_DEC )* WS* 'end' WS* ENTER*;

INC_DEC            :    VARIABLE ('--' | '++') WS* ENTER*;`

这是解析器的主类： `
import org.antlr.runtime.ANTLRStringStream; import org.antlr.runtime.CommonToken; import org.antlr.runtime.CommonTokenStream; import org.antlr.runtime.Parser;

public class Main {
    public static void main(String[] args) throws Exception {  
        // the input source  
        String source =   
            "for i 1 3\n " +
            "printHi()\n " +
            "end\n " +
            "if fun(y, z) == 0\n " +
            "end\n ";
// create an instance of the lexer  
         grammarLexer lexer = new grammarLexer(new ANTLRStringStream(source));  

         // wrap a token-stream around the lexer  
         CommonTokenStream tokens = new CommonTokenStream(lexer);  

         // traverse the tokens and print them to see if the correct tokens are created  
         int n = 1;  
         for(Object o : tokens.getTokens()) {  
           CommonToken token = (CommonToken)o;  
           System.out.println("token(" + n + ") = " + token.getText().replace("\n", "\\n"));  
           n++;  
         }
         grammarParser parser = new grammarParser(tokens);
         parser.file();
}
}
`

Answer 1

正如我在评论中已提到的：过度使用词法分析器规则是错误的。将词法规则视为您语言的基本构建块。就像你如何描述化学中的水一样。您不会像这样描述水：

WATER
 : 'HHO'
 ;

即：作为单个元素。水应该被描述为3个独立的元素：

water
 : Hydrogen Hydrogen Oxygen
 ;

Hydrogen : 'H';
Oxygen   : 'O';

其中Hydrogen和Oxygen是基本构建块（词法规则），water是复合（解析器规则）。

一个好的经验法则是，如果您正在创建包含其他几个词法分析器规则的词法分析器规则，那么您的语法中可能存在一些可疑的东西。当然，情况并非总是如此。

我们假设您要解析以下输入：

for i 1 3
  print(i)
end

if fun(y, z) == 0
  print('foo')
end

语法可以看起来像这样：

grammar T;

options {
  output=AST;
}

tokens {
  BLOCK;
  CALL;
  PARAMS;
}

// parser rules
parse
 : block EOF!
 ;

block
 : stat* -> ^(BLOCK stat*)
 ;

stat
 : for_stat
 | if_stat
 | func_call
 ;

for_stat
 : FOR^ ID expr expr block END!
 ;

if_stat
 : IF^ expr block END!
 ;

expr
 : eq_expr
 ;

eq_expr
 : atom (('==' | '!=')^ atom)*
 ;

atom
 : func_call
 | INT
 | ID
 | STR
 ;

func_call
 : ID '(' params ')' -> ^(CALL ID params)
 ;

params
 : (expr (',' expr)*)? -> ^(PARAMS expr*)
 ;

// lexer rules
FOR : 'for';
END : 'end';
IF  : 'if';
ID  : ('a'..'z' | 'A'..'Z')+;
INT : '0'..'9'+;
STR : '\'' ~('\'')* '\'';
SP  : (' ' | '\t' | '\r' | '\n')+ {skip();};

如果您现在运行此测试类：

import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;
import org.antlr.stringtemplate.*;

public class Main {
  public static void main(String[] args) throws Exception {
    String src = 
        "for i 1 3          \n" + 
        "  print(i)         \n" + 
        "end                \n" + 
        "                   \n" + 
        "if fun(y, z) == 0  \n" + 
        "  print('foo')     \n" + 
        "end                \n";
    TLexer lexer = new TLexer(new ANTLRStringStream(src));
    TParser parser = new TParser(new CommonTokenStream(lexer));
    CommonTree tree = (CommonTree)parser.parse().getTree();
    DOTTreeGenerator gen = new DOTTreeGenerator();
    StringTemplate st = gen.toDOT(tree);
    System.out.println(st);
  }
}

您将看到一些输出打印到控制台，对应于以下AST：

enter image description here

Antlworks语法分析器

1 个答案: