_readdir() translates some non-ASCII characters in file names

[RobertSteed]

Dear forum,
My code includes the following:

Code Select Expand

#include <dirent.h>
#include <locale.h>
...
setlocale(LC_ALL, "");
...
_DIR *dir;
struct _dirent *dirent;
dir = _opendir(path); // Open the folder
...
dirent = _readdir(dir); // Read first entry in folder // Read first entry in folder
if (dirent == NULL)
{ perror(path);
return 1;
}
while (dirent != NULL) // While not at end of folder
{
printf("dirent->d_name = %s\n", dirent->d_name);
...

Which results in

Code Select Expand

dirent->d_name = Art of Beksinski 1978 (MHS) (2).jpgLater code tries to open the files found:

Code Select Expand

FILE *fh = fopen(fn, "rb");which of course results in an error message:

Code Select Expand

Couldn't open test data\Art of Beksinski 1978 (MHS) (2).jpg: No such file or directoryThe actual filename is "Art of Beksiński 1978 (MHS) (2).jpg" with a diacritic on the "n". _readdir() worked as expected for "00tést.jpg" which has a diacritic on the "e".
So how can I get _opendir to return all non-ASCII file names as they are, without translating some characters?

[TimoVJL]

Function fopen might be a problem too, as it works with ANSI filenames.
I did some tests, even i don't like those functions, as i rather use Win32 API.

[John Z]

Hi,

Well first I've never used these functions so a grain of salt with my comments.

Looking into the header there is no Unicode or wide character support for the functions.  So support will be limited to code pages.  If using ANSI code page ( I guess for the UK ?) for example then the 233 é  exists and can be shown.  However the ANSI code page does not include an n with diacritic in fact the only n is 241 ñ, so the closest substitution is made.

So try a different code page?  I was going to suggest trying to get the 8.3 filename but it seems the same issue would hinder that too.

John Z

For example codepage 852  E4 is n with diacritic

[TimoVJL]

For example codepage 852  E4 is n with diacritic

https://www.ascii-codes.com/cp852.html
https://www.asciigrid.com/codepages/dos/852
E4 is Lower case n with acute

https://altcodeunicode.com/alt-codes-letter-n-with-accents/

[John Z]

E4 is Lower case n with acute

Yup, thanks, I did not remember if it was acute or grave (up, or down) so just general term diacritic.  Been a while since my French

John Z

[RobertSteed]

Function fopen might be a problem too, as it works with ANSI filenames.
I did some tests, even i don't like those functions, as i rather use Win32 API.

That's a good point. What do you use for Unicode files?
Why do you use the Win32 API rather than _findfirst64() etc?

[RobertSteed]

Hi,

Well first I've never used these functions so a grain of salt with my comments.

Looking into the header there is no Unicode or wide character support for the functions.  So support will be limited to code pages.  If using ANSI code page ( I guess for the UK ?) for example then the 233 é  exists and can be shown.  However the ANSI code page does not include an n with diacritic in fact the only n is 241 ñ, so the closest substitution is made.

So try a different code page?  I was going to suggest trying to get the 8.3 filename but it seems the same issue would hinder that too.

John Z

For example codepage 852  E4 is n with diacritic

Cheers. I'm loathe to use codepages! What do you do for this kind of thing where you're dealing with Unicode file names? I guess the wide-character versions might work.

[John Z]

Hi,

You've not mentioned whether coding for DOS or Windows, Linux?
These non-std functions were provide to improve portability according to the help file.

Windows filenames are all in Unicode UTF-16.  However AFAIK all OS files are ASCII and ASCII is supported intact within the lower UTF-16 codes, so 98% of the files on the system can be opened without using wide.  The other 2% are user files which can be using any character in the UTF-16 set, which includes all (well almost all) characters sets in the world. A swag as the percentages but you get the gist.

IMO, you are better off using _wfindfirst64,  and associated functions, then you are covered for all language nuances.

Otherwise codepages are not hard to use, the hard part is figuring out which one to activate....

John Z

[Robert]

Brecht Sanders has a cross-platform library specific to readdir.

https://github.com/brechtsanders/libdirtrav

Warning! I have not tried using it, so I have no idea if it will work with Pelles C.

CodePages ?
The example file, "Art of Beksiński 1978 (MHS) (2).jpg", includes Polish characters. What if the next file is "Miró's L'escala de l'evasió.jpg"? How could you know that the filename is Spanish? And even if your app could figure that out, how could you switch codepages within the app?

[John Z]

Hi,

Well here is Mr. Sanders brief:
"The libdirtrav library provides a way to recursively read folder contents using callback functions. This is done using opendir() and readdir() functions, except for Windows where native FindFirstFile()/FindFirstFileEx() and FindNextFile() functions are used. On Windows a wide version is available which uses wchar_t* as string type for Unicode UTF-16 support"

So one sees he also recommends FindFirst etc for windows.

There are two issues 1) codepages being used for Filenames  2) codepages being used within a program or data.  #1 is most difficult to handle as you pointed out it is possible in windows which uses UTF-16 to have many different languages for filenames .  #2 is not so hard.  Most legacy programs or data files have some identifier to inform about the codepage.  VCF contact files for example before VCF 3 used CHARSET=  some name (not always standardized).  VCF3 pushed UTF-8 as a recommendation and VCF4 requires it.

Handling codepage changes within a program is not difficult.  In most cases the end user will be able to see the information is weird so can change the codepage and re-read the data.  An example dialog from my vCardz_i program for vcf contact cards for changing the code page as needed.  Data is read in as binary and and converted from the codepage into UTF-8 or UTF-16 to display.  If looks bogus another codepage can be selected, tedious to be sure and hopefully not often needed. 

John Z