STR #1496: scandir for Win32 (src/win32) is not Unicode-aware

STR #1496

Application:	FLTK Library
Status:	1 - Closed w/Resolution
Priority:	1 - Request for Enhancement, e.g. asking for a feature
Scope:	2 - Specific to an operating system
Subsystem:	WIN32
Summary:	scandir for Win32 (src/win32) is not Unicode-aware
Version:	2.0-feature
Created By:	exhu
Assigned To:	bgbnbigben
Fix Version:	2.0-current (SVN: v8328)
Update Notification:	Receive EMails Don't Receive EMails

Trouble Report Files:

No files

Trouble Report Comments:

#1	exhu 00:40 Nov 13, 2006

FindFirstW etc. must be used if is_unicode() returns 1 for scanning the directory and retreiving file names because non-ascii characters in the file name under Win32 are shown as trash in FileChooser (ANSI is treated as utf-8).

#2	spitzak 12:03 Jun 06, 2007

Is there any way to force the current encoding to be UTF8? This would make all the mb api take and return UTF8 and remove a lot of problems. I know you can do it in the registry, but can a program do it to itself?

#3	exhu 00:21 Jun 07, 2007

It's all rather awkward in Win32 with separate ANSI and UNICODE functions for all text-receiving/returning functions. You can set the code page only for a console not for the file APIs (SetFileApisToANSI/OEM cannot set to UTF-8) :( So anyway a wrapper is needed or your own string class/functions.

#4	tamidit 01:45 Dec 10, 2007

To fix file browser widget under Windows, I have changed scandir.c to use Unicode API. The filenames are then recoded into UTF8 so that FLTK could handle them. I also had to change filename_isdir.cxx to recode from UTF8 back to Unicode (I use _wstat() instead of stat() so that the Unicode filenames were processed correctly). This fixes the FileBrowser widget for Russian WinXP, but there are issues nevertheless: 1. Directories with cyrillic names come before everything else in the list (even the ".." directory). 2. Filenames returned by the widget are in utf8 encoding, so you cannot simply pass them to, say, fopen() and hope it will work. I agree with exhu - FLTK must use its own mechanism for handling Unicode strings. I see two options: - true Unicode strings (like QString class in Qt). I personally like this option (see below). - UTF8 strings (like in GTK). Unicode strings are ugly unless you have a really nice class that handles them. All functions of the toolkit must be rewritten so that they could use the new class; also, the code gets additional bloat from this class. But the prize is big enough: 1. No problems with Unicode. 2. Each character occupies exactly one position in the string (this is not the case in UTF8, where a character can take 1, 2 or even more bytes) 3. You are FORCED to think of a proper encoding/decoding function when you pass strings to functions that take char* as arguments. UTF8 strings are another option. They can be handled like usual ASCIIZ strings, which is good. But there are drawbacks: 1. You can't tell by the type if the string is in Unicode or in local 8-bit encoding. This will surely lead to difficult-to-trace errors. 2. It is difficult to predict the length of the UTF8 string. For example, converting latin letters from Unicode to UTF8 reduces the length by two. But converting Russian text from Unicode to UTF8 does not reduce the length of the string: each character takes 2 bytes. Moreover, some Unicode characters can produce more than 2 bytes of UTF8 output.

#5	fabien 10:37 Mar 20, 2008

Changing the format of strings in fltk2 is certainly greatly interesting but implies huge modifications, so I change this priority into an RFE

#6	bgbnbigben 00:55 Jan 29, 2011

Please contact your Linux distributor for this issue. SVN r8328 should fix the issue of "scandir for Win32 is not Unicode-aware". I'll re-submit a STR with fabien's idea to make _all_ fltk2 strings utf8 aware.

#7	bgbnbigben 00:57 Jan 29, 2011

Fixed in Subversion repository. Whoops, included the wrong "standard message". Fixed in SVN. :-)