Go Back   Cockos Incorporated Forums > REAPER Forums > REAPER Bug Reports

Reply
 
Thread Tools Display Modes
Old 12-04-2017, 05:49 AM   #1
aciddose
Human being with feelings
 
Join Date: Dec 2017
Posts: 22
Default Non-USASCII usernames on Windows, Reaper fails to load/save configuration (FIXED)

Reaper fails to load or save configuration (and other) files on any version of Windows where the filename and/or path contain non-USASCII characters.

For example using the Japanese "tesuto" = "テスト" ("test") as a user/account name will reproduce this issue. This is the username I use on my Windows systems for QA/testing to reproduce such types of bugs.

Unfortunately Windows does not natively support utf-8 encoded strings passed to fopen or other cstdlib functions but supports only US-ASCII strings. Instead it supports a 16-bit "wide char" format (Windows.h: WCHAR) and unique non-standard functions (_wfopen, ...) for accessing system resources using non-USASCII filenames or paths.

Typical conversion from char array (utf-8) to WCHAR array is like so:
Code:
const char *source_string;

const int destination_length = MultiByteToWideChar(CP_UTF8, 0, source_string, -1, nullptr, 0);

// RAII of course, but to keep it short:
WCHAR *destination_string = allocate(destination_length);

int result_length = MultiByteToWideChar(CP_UTF8, 0, source_string, -1, destination_string, destination_length);

// if file I/O is handled with cstdio functions, _wfopen can be used in-place without modification of the dependent code.
FILE *handle = _wfopen(wfilename, wmode);

free(destination_string);
The addition of such code to the file I/O wrapper classes used in Reaper would allow it to begin to function with non-USASCII user/account names on all supported versions of Windows.

Last edited by aciddose; 12-04-2017 at 06:36 AM.
aciddose is offline   Reply With Quote
Old 12-04-2017, 06:54 AM   #2
EvilDragon
Human being with feelings
 
EvilDragon's Avatar
 
Join Date: Jun 2009
Location: Croatia
Posts: 24,790
Default

Oh hi thar, aciddose! Nice to see you around here
EvilDragon is offline   Reply With Quote
Old 12-04-2017, 08:51 AM   #3
aciddose
Human being with feelings
 
Join Date: Dec 2017
Posts: 22
Default

Don't jinx it!

I've really been putting off reporting this issue because I figured it was something that needed some time to work out in the code and was already known. After years though I wonder if there are issues that make it "more complicated than that", or if my suggestion (WCHAR conversion) is really as complicated as it has to be.

Hopefully it's an easy fix!
aciddose is offline   Reply With Quote
Old 12-04-2017, 09:06 AM   #4
EvilDragon
Human being with feelings
 
EvilDragon's Avatar
 
Join Date: Jun 2009
Location: Croatia
Posts: 24,790
Default

Relevant post from Justin, perhaps:

https://forum.cockos.com/showpost.ph...&postcount=327
EvilDragon is offline   Reply With Quote
Old 12-04-2017, 09:44 AM   #5
aciddose
Human being with feelings
 
Join Date: Dec 2017
Posts: 22
Default

I would need to test installing on such an account to confirm whether the installer has been fixed (NSIS?), but regardless it would be a pointless exercise if Reaper itself installs correctly but then can't load or save its configuration from such a path

I would honestly be very surprised if this issue wasn't known. After experiencing it way back in 2012 and still seeing no fix I just wanted to ensure a proper bug report was posted to be certain it is a known issue.

It's fairly well known that using non-US anything with a lot of lower quality software (which software is higher quality ... ?) will tend to cause issues. So I have no doubt that most users are aware they should be using a US-ASCII user name if they want their software to work.

That said the solution to this problem was only a few lines of code across my entire codebase as I always use a single file I/O object for all types of I/O on all platforms. I suspect the Reaper codebase may be written in a way which makes it very difficult to solve such problems so easily... but I feel it is essential to take the first steps before you can continue to move forward.

So even if Reaper can be made to load its configuration files and then fail in 1000 other horrifying ways; I still feel it is worthwhile to start to fix this issue now and aim to have new code written in a compatible way moving forward.
aciddose is offline   Reply With Quote
Old 12-04-2017, 05:46 PM   #6
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 15,721
Default

Please try the 5.65rc (see the prerelease forum), it should address these issues.

Almost everything does support UTF-8 in 5.62, but we forgot about GetModuleFileName(), hence the issue with the installation path (user paths with UTF-8 chars should work in 5.62, as long as the executable is in a path representable by the configured ANSI codepage).
Justin is offline   Reply With Quote
Old 12-05-2017, 12:45 AM   #7
aciddose
Human being with feelings
 
Join Date: Dec 2017
Posts: 22
Default

I can confirm 565rc2 does work with a non-USASCII user name and paths regardless of the codepage (I use a US codepage setting, which is the source of the problem requiring usage of the Windows WCHAR functions.)

So those functions used to access user profile folders such as SHGetFolderPathW(NULL, CSIDL_APPDATA, ...) are already in use in the RC version.

Funny then that I finally bothered to report the issue at the in-between versions stage where it had already been fixed in an RC!
aciddose is offline   Reply With Quote
Old 12-05-2017, 03:30 AM   #8
ivansc
Human being with feelings
 
Join Date: Aug 2007
Location: Near Cambridge UK and Near Questembert, France
Posts: 22,754
Default

SPIDEY SENSE AT WORK!


(cue Twiglet Zone theme)

__________________
Ici on parles Franglais
ivansc is offline   Reply With Quote
Old 12-05-2017, 11:06 AM   #9
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 15,721
Default

Hmm I may have been incorrect, there are some AppData utf8 path issues remaining, stay tuned
Justin is offline   Reply With Quote
Old 12-05-2017, 05:48 PM   #10
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 15,721
Default

5.65rc3 should finish the job, if you want to test that (I just tested with win10 and a username of テスト and it worked OK!).
Justin is offline   Reply With Quote
Old 12-06-2017, 05:44 AM   #11
aciddose
Human being with feelings
 
Join Date: Dec 2017
Posts: 22
Default

I can verify rc3 also works on Windows 7 using username テスト.

I should probably look into what Microsoft's aims are for utf-8 support in Win10 as the last time I researched this issue I was still concerned about WinXP compatibility (~2012).

It would be absolutely great if they finally fix Windows to allow the system codepage to be set to utf-8! Especially now with the codebase across different platforms (Linux, MacOS) where Windows is the odd one of the bunch.

You never know, it might happen with things like the Linux subsystem working.


... that is if they don't implement a proprietary utf-9 and use it to label a series of gates ...
aciddose is offline   Reply With Quote
Old 12-15-2017, 07:16 PM   #12
cfillion
Human being with feelings
 
cfillion's Avatar
 
Join Date: May 2015
Location: Québec, Canada
Posts: 4,937
Default

Saving and restoring window pin states doesn't work with Unicode paths in 5.70.

EDIT: Also WDL doesn't have UTF-8 implementations of GetPrivateProfileSection and WritePrivateProfileSection that SWS needs (#935, #934).



Last edited by cfillion; 12-15-2017 at 09:16 PM.
cfillion is offline   Reply With Quote
Old 12-15-2017, 09:50 PM   #13
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 15,721
Default

Quote:
Originally Posted by cfillion View Post
Saving and restoring window pin states doesn't work with Unicode paths in 5.70.

EDIT: Also WDL doesn't have UTF-8 implementations of GetPrivateProfileSection and WritePrivateProfileSection that SWS needs (#935, #934).


Thanks -- fixing the pin state issue. Re: ProfileSection() APIs, could implement that in SWS, or (eventually) in WDL.
Justin is offline   Reply With Quote
Old 12-17-2017, 05:53 AM   #14
Spurk
Human being with feelings
 
Join Date: Oct 2016
Posts: 126
Default

Hey,

Not really relative but concerns capital letters with accent in Reaper.

Some letters such as 'Ç' or 'È' are well handeled but for this one É Reaper gives a nice '?'. It appears in track name, marker/region but not in the explorer search bar for instance.

Of course I changed Reapers language but I think this is not related.

Please, can you look at that or may be it's from an option I have to change?

cheers
Spurk is offline   Reply With Quote
Old 12-17-2017, 02:39 PM   #15
cfillion
Human being with feelings
 
cfillion's Avatar
 
Join Date: May 2015
Location: Québec, Canada
Posts: 4,937
Default

Quote:
Originally Posted by Justin View Post
Re: ProfileSection() APIs, could implement that in SWS, or (eventually) in WDL.
OK, I made a patch to add these API wrappers in SWS at least for now.

One more thing that used to work before 5.60: REAPER & SWS's .ini files are now limited to storing only characters in the ANSI codepage if the filename has any non-latin characters. This is because the wide PrivateProfiles convert the values to ANSI replacing unknown characters with '?' unless the file already contains UTF-16 (eg. a BOM).



Last edited by cfillion; 02-27-2018 at 11:43 AM.
cfillion is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 08:03 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.