UTF-8 with windows mixed ANSI/WIDECHAR programming

[Michele]

This other example shows how to use the manifest to change the process codepage for a window program.
When we change the process codepage to UTF-8 the ANSI version of many (warning not all!) windows messages automatically translates from process codepage to internal UTF-16 (unicode) format.
Winapi explicitly allows the mixed use of ANSI and WIDE messages providing behind the scenes the translation from between the process codepage and unicode. This means that you can create an ANSI window/control and send to it wide messages or the reverse, and also means that you are free to create a unicode or ansi application as you like and use which encoding you prefer.
But if your codepage doesn't cover the whole range of alphabets, i.e. if you use a typical latin western 1252 encoding, supplying an utf8 string will produce garbage output.
The trick is to set the process codepage to UTF-8 using the specific manifest command, but this feature has benn introduced starting from Windows 10 Version 1903 (Build 18362), and will be silently ignored in previous versions. The manifest code is:

Code Select Expand

<application xmlns="urn:schemas-microsoft-com:asm.v3">
  <windowsSettings>
    <activeCodePage xmlns="http://schemas.microsoft.com/SMI/2019/WindowsSettings">UTF-8</activeCodePage>
  </windowsSettings>
</application>
The attached sample demonstrate how it works. You can test it in both versions ANSI and WIDECHAR, just change the '#if 1' to '#if 0' to compile in ANSI mode.

The idea was born creating a program based on sqlite3 that is UTF-8 based.

If your system does'nt supports the process codepage change you can use the standar conversion functions:

Code Select Expand

#include <windows.h>
#include <stdlib.h>
#include <stdio.h>

// 1. UTF-8 (multibyte char*) -> UTF-16 (wchar_t*)
wchar_t* Utf8ToUtf16(const char* utf8_str) {
    if (!utf8_str) return NULL;

    // Get required buffer size in wchar_t elements (including null-terminator)
    int size_needed = MultiByteToWideChar(CP_UTF8, 0, utf8_str, -1, NULL, 0);
    if (size_needed == 0) return NULL;

    wchar_t* wstr = (wchar_t*)malloc(size_needed * sizeof(wchar_t));
    if (!wstr) return NULL;

    // Perform conversion
    MultiByteToWideChar(CP_UTF8, 0, utf8_str, -1, wstr, size_needed);
    return wstr; // Caller must free()
}

// 2. UTF-16 (wchar_t*) -> UTF-8 (multibyte char*)
char* Utf16ToUtf8(const wchar_t* wstr) {
    if (!wstr) return NULL;

    // Get required buffer size in bytes (including null-terminator)
    int size_needed = WideCharToMultiByte(CP_UTF8, 0, wstr, -1, NULL, 0, NULL, NULL);
    if (size_needed == 0) return NULL;

    char* utf8_str = (char*)malloc(size_needed);
    if (!utf8_str) return NULL;

    // Perform conversion
    WideCharToMultiByte(CP_UTF8, 0, wstr, -1, utf8_str, size_needed, NULL, NULL);
    return utf8_str; // Caller must free()
}