是否有可能为规则设定优先级以避免“最长最早”的匹配模式？

Another simple question : is there any way to tell flex to prefer a rule that matches a short thing over a rule that matches a longer thing ? I can't find any good documentation about that.

另一个简单的问题：是否有任何方法可以告诉flex更喜欢匹配较短事物的规则与匹配较长事物的规则？我找不到任何关于这方面的好文件。

Here is why I need that : I parse a file for a pseudo language that contains some keywords corresponding to control instructions. I'd like them to be the absolute priority so that they're not parsed as parts of an expression. I actually need this priority thing because I don't have to write a full grammar for my project (that would be totally overkill in my case since I perform structural analysis on the program parsed, I don't need to know the details...), so I can't use a fine grammar tuning to be sure that those blocks won't be parsed into an expression.

这就是我需要的原因：我为一个伪语言解析一个文件，其中包含一些与控制指令相对应的关键字。我希望它们是绝对的优先级，这样它们就不会被解析为表达式的一部分。我实际上需要这个优先级的东西，因为我不需要为我的项目写一个完整的语法（在我的情况下，因为我对解析的程序执行结构分析，我不需要知道细节，这将是完全矫枉过正的... 。），所以我不能使用精细的语法调整来确保这些块不会被解析成表达式。

Any help will be appreciated.

任何帮助将不胜感激。

Here is an example of a file parsed :

以下是解析文件的示例：

If a > 0 Then read(b); Endif
c := "If I were...";
While d > 5 Do d := d + 1 Endwhile

I just want to collect info on the Ifs, Thens, Endifs etc... The rest doesn't matter to me. That's why I'd like the Ifs, Thens etc... related rules to be prioritized without to have to write a grammar.

我只想收集关于Ifs，Thens，Endifs等的信息......其余对我来说无关紧要。这就是为什么我喜欢Ifs，Thens等...相关规则要优先考虑而不必编写语法。

1 个解决方案

#1

From the Dragon Book 2nd edition, Section 3.5.3 "Conflict Resolution in Lex":

从Dragon Book第2版，第3.5.3节“Lex中的冲突解决”：

We have alluded to the two rules that Lex uses to decide on the proper lexeme
to select, when several prefixes of the input match one or more patterns:
    1. Always prefer a longer prefix to a shorter prefix.
    2. If the longest possible prefix matches two or more patterns, prefer the
       pattern listed first in the Lex program.

The rule above also applies to Flex. Here is what the Flex manual says (Chapter 7: How the input is matched.)

上面的规则也适用于Flex。这是Flex手册所说的内容（第7章：输入如何匹配。）

When the generated scanner is run, it analyzes its input looking for strings 
which match any of its patterns. If it finds more than one match, it takes the 
one matching the most text (for trailing context rules, this includes the length 
of the trailing part, even though it will then be returned to the input). If it 
finds two or more matches of the same length, the rule listed first in the flex 
input file is chosen.

If I understood correctly, your lexer treats keywords like Endif as an identifier, so it will be considered as part of an expression afterwards. If this is your problem, simply put the rules of keywords on top of your specification, such as the following: (suppose each word in uppercase is a predefined enum corresponding to a token)

如果我理解正确，你的词法分析器会将像Endif这样的关键字视为标识符，因此它将被视为表达式的一部分。如果这是您的问题，只需将关键字规则放在您的规范之上，如下所示:(假设每个单词都是一个对应于令牌的预定义枚举）

"If"                      { return IF;         }
"Then"                    { return THEN;       }
"Endif"                   { return ENDIF;      }
"While"                   { return WHILE;      }
"Do"                      { return DO;         }
"EndWhile"                { return ENDWHILE;   }
\"(\\.|[^\\"])*\"         { return STRING;     }
[a-zA-Z_][a-zA-Z0-9_]*    { return IDENTIFIER; }

Then the keywords will always matched before the identifier due to Rule No. 2.

然后，由于规则2，关键字将始终在标识符之前匹配。

EDIT:

编辑：

Thank you for your comment, kol. I forgot to add the rule for string. But I don't think my solution is wrong. for example, if an identifier called If_this_is_an_identifier, rule 1 will apply, thus the identifier rule will take effect (Since it matches the longest string). I wrote a simple test case and saw no problem in my solution. Here is my lex.l file:

谢谢你的评论，kol。我忘了为字符串添加规则。但我不认为我的解决方案是错误的。例如，如果将应用名为If_this_is_an_identifier的标识符，则应用规则1，因此标识符规则将生效（因为它匹配最长的字符串）。我写了一个简单的测试用例，在我的解决方案中没有遇到任何问这是我的lex.l文件：

%{
  #include <iostream>
  using namespace std;
%}

ID       [a-zA-Z_][a-zA-Z0-9_]*

%option noyywrap
%%

"If"                      { cout << "IF: " << yytext << endl;         }
"Then"                    { cout << "THEN: " << yytext << endl;       }
"Endif"                   { cout << "ENDIF: " << yytext << endl;      }
"While"                   { cout << "WHILE: " << yytext << endl;      }
"Do"                      { cout << "DO: " << yytext << endl;         }
"EndWhile"                { cout << "ENDWHILE: " << yytext << endl;   }
\"(\\.|[^\\"])*\"         { cout << "STRING: " << yytext << endl;     }
{ID}                      { cout << "IDENTIFIER: " << yytext << endl; }
.                         { cout << "Ignore token: " << yytext << endl; }

%%

int main(int argc, char* argv[]) {
  ++argv, --argc;  /* skip over program name */
  if ( argc > 0 )
    yyin = fopen( argv[0], "r" );
  else
    yyin = stdin;

  yylex();
}

I tested my solution with the following test case:

我用以下测试用例测试了我的解决方案：

If If_this_is_an_identifier > 0 Then read(b); Endif
    c := "If I were...";
While While_this_is_also_an_identifier > 5 Do d := d + 1 Endwhile

and it gives me the following output (other output not relevant to the problem you mentioned is ignored.)

它给了我以下输出（忽略了与你提到的问题无关的其他输出。）

IF: If
IDENTIFIER: If_this_is_an_identifier
......
STRING: "If I were..."
......
WHILE: While
IDENTIFIER: While_this_is_also_an_identifier

The lex.l program is modified base on an example from the flex manual: (which use the same method to match keyword out of identifiers)

根据flex手册中的一个示例修改lex.l程序:(使用相同的方法匹配标识符中的关键字）

Also have a look at the ANSI C grammar, Lex specification.

另请参阅ANSI C语法，Lex规范。

I also used this approach in my personal project, and so far I didn't find any problem.

我也在我的个人项目中使用了这种方法，到目前为止我没有发现任何问题。

#1