使用UTF-8在非Ascii字符上运行Ascii正则表达式

#include <boost/regex.hpp>

#include <string>
#include <vector>
#include <iostream>

int main(int argc, char* argv[]) {

    std::string text = argv[1];
    std::string patterns = argv[2];

    boost::regex regex = boost::regex(patterns);

    boost::smatch match;

    std::cout << boost::regex_search(text, match, regex) << std::endl;    
}

If I run the program over the input hello¿ ¿ (containing a non-ASCII character with UTF-8 encoding) it returns 0 i.e. not found, but if I run it over the input hel√ √ (again containing non-ascii) it returns 1, i.e. found.

如果我运行程序输入你好害怕害怕(包含非ascii字符的utf - 8编码)它返回0即没有发现,但是如果我运行它在输入冥界√√(同样包含非ascii)返回1,即发现。

My question: What is the expected behavior of boost::regex (i.e. the ascii version) when run over utf characters?

我的问题是:当运行utf字符时，boost::regex(即ascii版本)的预期行为是什么?

---

Edit: Thanks for all the comments, I am still interested as to why exactly 1 is output, since both the text and the regex contain non-ascii characters. My guess would be that the bytes are interpreted as ascii and thus they match.

编辑:感谢所有的评论，我仍然对为什么1是输出感兴趣，因为文本和regex都包含非ascii字符。我的猜测是字节被解释为ascii，因此它们是匹配的。

2 个解决方案

#1

1. Using regular expressions on ASCII strings, is about using "bytes" to find a pattern in.
   Using regular expressions on UTF-8 strings, is about using regular expressions on "multi-byte" sequences, where a sequence represents a Unicode code point.
   
   Thus the regular expression gets applied to a Unicode string with an encoding with variable byte-count per character.
   
   UTF-8 strings contain multi-byte sequences with 1 to 4 bytes, which representing a Unicode "character". In UTF-8 only ASCII 7 bit characters are 1 byte "wide".
   
   So - using an ASCII regular expression engine on an UTF-8 encoded string, ignores the multi-byte sequences in the UTF-8 encoded string and causes a pattern matching byte by byte. The results of this ASCII regular expression engine usage on an UTF-8 encoded string is invalid.
   
   Please take a look at http://utfcpp.sourceforge.net.
   
   To get the regular expressions working on UTF-8 encoded strings, you have to …
   

   在ASCII字符串中使用正则表达式，就是使用“字节”来查找一个模式。在UTF-8字符串上使用正则表达式，是关于在“多字节”序列上使用正则表达式，其中序列表示Unicode代码点。因此，将正则表达式应用到具有每个字符的变量字节计数的编码的Unicode字符串。UTF-8字符串包含1到4字节的多字节序列，表示Unicode“字符”。在UTF-8中，只有ASCII 7位字符是1字节“宽”。因此——在UTF-8编码的字符串上使用ASCII正则表达式引擎，会忽略UTF-8编码字符串中的多字节序列，并导致逐字节的模式匹配。这个ASCII正则表达式引擎在UTF-8编码字符串上的使用结果无效。请查看http://utfcpp.sourceforge.net。要使正则表达式在UTF-8编码的字符串上工作，您必须…

   • have UTF-8 string iterators usable with the regular expressions, or
   • 让UTF-8字符串迭代器可以用于正则表达式，还是
   • use std::codecvt_utf8 in combination of setting temporarily the global locale to get the regular expression working, or
   • 使用std::codecvt_utf8结合临时设置全局语言环境以使正则表达式工作，或
   • have to convert the UTF-8 encoded string into a UTF-16 encoded string to be used with an Unicode regular expression engine - based on std::wstring.
   • 必须将UTF-8编码的字符串转换为UTF-16编码的字符串，以与Unicode正则表达式引擎(基于std::wstring)一起使用。

2. The regex_search function returns a boolean and true on a match.
   In your case the ASCII regular expression pattern matches a part of the UTF-8 encoded string, which is parsed invalidly as ASCII string - as you assumed!
   If you have English text in an UTF-8 encoded string, then an ASCII regular expression engine can be used safely. Leaving the ASCII 7 bit range, makes the result of the ASCII regular expression engine unreliable.

   regex_search函数在匹配时返回一个布尔值和true值。在您的示例中，ASCII正则表达式模式匹配UTF-8编码字符串的一部分，该字符串被无效地解析为ASCII字符串——正如您所假定的那样!如果您有UTF-8编码字符串中的英语文本，那么可以安全地使用ASCII正则表达式引擎。离开ASCII 7位范围，使ASCII正则表达式引擎的结果不可靠。

#2

It is a bug and not a feature: I tried your example on a better system (g++ 4.9.2 on Windows MinGW) and all happens well:

这是一个bug，而不是特性:我在一个更好的系统(Windows MinGW的g++ 4.9.2)上尝试了您的示例，一切都很顺利:

#include <iostream>
#include <string>
#include <regex>
int main()
{ std::string text ="hello¿"; // or "hello√"
  std::string patterns ="¿";  // or "√"
  std::regex regex = std::regex(patterns);
  std::smatch match;
  std::cout << std::regex_search(text, match, regex) << std::endl;
}

with output:

输出:

1