807down vote
|
|
|
@Sorin Sbarnea: UTF-8 could take 1-6 bytes, but apparently the standard limits it to 1-4. See en.wikipedia.org/wiki/UTF8#Description for more information. – paercebal Jan 13 '10 at 13:10
|
||
|
While this examples produces different results on Linux and Windows the C++ program contains implementation-defined behavior as to whether
olè is encoded as UTF-8 or not. Further more, the reason you cannot natively stream wchar_t * to std::cout is because the types are incompatible resulting in an ill-formed program and it has nothing to do with the use of encodings. It's worth pointing out that whether you use std::string or std::wstring depends on your own encoding preference rather than the platform, especially if you want your code to be portable. – John Leidegren Aug 9 '12 at 9:37
|
||
|
@paercebal Whatever the platform supports is entirely arbitrary and besides the point. If you store all strings internally as UTF-8 on Windows you'll have to convert them to either ANSI or UTF-16 and call the corresponding Win32 function but if you know your UTF-8 strings are just plain ASCII strings you don't have to do anything. The platform doesn't dictate how you use strings as much as the circumstances. – John Leidegren Aug 9 '12 at 16:35
|
||
|
Windows actually uses UTF-16 and have been for quite some time, older versions of Windows did use UCS-2 but this is not the case any longer. My only issue here is the conclusion that
std::wstring should be used on Windows because it's a better fit for the Unicode Windows API which I think is fallacious. If your only concern was calling into the Unicode Windows API and not marshalling strings then sure but I don't buy this as the general case. – John Leidegren Aug 9 '12 at 18:15
|
||
|
@ John Leidegren :
If your only concern was calling into the Unicode Windows API and not marshalling strings then sure : Then, we agree. I'm coding in C++, not JavaScript. Avoiding useless marshalling or any other potentially costly processing at runtime when it can be done at compile time is at the heart of that language. Coding against WinAPI and using std::string is just an unjustified wasting runtime resources. You find it fallacious, and it's Ok, as it is your viewpoint. My own is that I won't write code with pessimization on Windows just because it looks better from the Linux side. – paercebal Aug 9 '12 at 19:48
|
So, every reader here now should have a clear understanding about the facts, the situation. If not, then you must read paercebal's outstandingly comprehensive answer [btw: thanks!]. My pragmatical conclusion is shockingly simple: all that C++ (and STL) "character encoding" stuff is substantially broken and useless. Blame it on Microsoft or not, that will not help anyway. My solution, after in-depth investigation, much frustration and the consequential experiences is the following:
The conversions are straightforward, google should help here ... That's it. Use UTF8String wherever memory is precious and for all UTF-8 I/O. Use UCS2String wherever the string must be parsed and/or manipulated. You can convert between those two representations any time. Alternatives & Improvements
ICU or other unicode libraries? |
|||||||||||||||||||||
|
I recommend avoiding My view is summarized in http://utf8everywhere.org of which I am a co-author. Unless your application is API-call-centric, e.g. mainly UI application, the suggestion is to store Unicode strings in std::string and encoded in UTF-8, performing conversion near API calls. The benefits outlined in the article outweigh the apparent annoyance of conversion, especially in complex applications. This is doubly so for multi-platform and library development. And now, answering your questions:
|
||||
|
|||||||||||||||||||||
|
I frequently use std::string to hold utf-8 characters without any problems at all. I heartily recommend doing this when interfacing with API's which use utf-8 as the native string type as well. For example, I use utf-8 when interfacing my code with the Tcl interpreter. The major caveat is the length of the std::string, is no longer the number of characters in the string.
|
|||||||||||||||||||||
|
|
|||||||||||||
|
Applications that are not satisfied with only 256 different characters have the options of either using wide characters (more than 8 bits) or a variable-length encoding (a multibyte encoding in C++ terminology) such as UTF-8. Wide characters generally require more space than a variable-length encoding, but are faster to process. Multi-language applications that process large amounts of text usually use wide characters when processing the text, but convert it to UTF-8 when storing it to disk. The only difference between a Practically every compiler uses a character set whose first 128 characters correspond with ASCII. This is also the case with compilers that use UTF-8 encoding. The important thing to be aware of when using strings in UTF-8 or some other variable-length encoding, is that the indices and lengths are measured in bytes, not characters. The data type of a wstring is If you don't need multi-language support, you might be fine with using only regular strings. On the other hand, if you're writing a graphical application, it is often the case that the API supports only wide characters. Then you probably want to use the same wide characters when processing the text. Keep in mind that UTF-16 is a variable-length encoding, meaning that you cannot assume |
|||||||||||||||||
|
1) As mentioned by Greg, wstring is helpful for internationalization, that's when you will be releasing your product in languages other than english 4) Check this out for wide character http://en.wikipedia.org/wiki/Wide_character |
|||
|
|||||||||||||||||||||
|
A good question! I think DATA ENCODING (sometime CHARSET also involved) is a MEMORY EXPRESSION MECHANISM in order to save data to file or transfer data via network, so I answer this question as: 1.When should I use std::wstring over std::string? If the programming platform or API function is a single-byte one, and we want to process or parse some unicode datas, e.g read from Windows' .REG file or network 2-byte stream, we should declare std::wstring variable to easy process them. e.g.: wstring ws=L"中国a"(6 octets memory: 0x4E2D 0x56FD 0x0061), we can use ws[0] to get character '中' and ws[1] to get character '国' and ws[2] to get character 'a', etc. 2.Can std::string hold the entire ASCII character set, including the special characters? Yes. But notice: American ASCII, means each 0x00~0xFF octet stand for one character ,including printable text such as "123abc&*_&" and you said special one, mostly print it as a '.' avoid confusing editors or terminals. And some other countries extend their own "ASCII" charset ,e.g. Chinese, use 2 octets to stand for one character. 3.Is std::wstring supported by all popular C++ compilers? Maybe, or mostly. I have used: VC++6 and GCC 3.3, YES 4.What is exactly a "wide character"? wide character mostly indicate using 2 octets or 4 octets to hold all countries's characters. 2 octets UCS2 is a representative sample, and further e.g. English 'a', its memory is 2 octet of 0x0061(vs in ASCII 'a's memory is 1 octet 0x61) |
https://*.com/questions/402283/stdwstring-vs-stdstring