FLTK logo

Re: [MOD] STR #3436: use of isspace(), ispunct(), and others must correctly test unicode characters

FLTK matrix user chat room
(using Element browser app)   FLTK gitter user chat room   GitHub FLTK Project   FLTK News RSS Feed  
  FLTK Library      Forums      Links      Apps     Login 
[  All Forums  |  Back to fltk.bugs  ]
 
[ Previous Message ] [ New Message | Reply ] [ Next Message ]

Re: [MOD] STR #3436: use of isspace(), ispunct(), and others must correctly test unicode characters Albrecht Schlosser May 22, 2024  
 
DO NOT REPLY TO THIS MESSAGE.  INSTEAD, POST ANY RESPONSES TO THE LINK BELOW.

[STR Pending]

Link:         https://www.fltk.org/str.php?L3436
Version:      1.4-current
Fix Version:  1.4.0
Git Commit:   9df6dc2aeb37682a2ef1906e04b9e0436e63a6ba
Assigned To:  AlbrechtS

Reopening because the underlying problems are not resolved so it doesn't
get forgotten.

Git commit 9df6dc2aeb37682a2ef1906e04b9e0436e63a6ba may have changed some
minor issues but the overall problem still exists.

Addition to the original post: two more functions need to be considered:
 - toupper()
 - tolower()

The current status - tested with VS 2019 - *seems* to be that Visual Studio
in Debug mode does no longer raise an assert error. Thus I lowered the
priority to 3 (Moderate). I can't test with older or other VS versions
though; there might be assert errors with older VS versions.

Recap and summary of the remaining problem:

1) All the mentioned functions can only called with ASCII characters or
with 'unsigned char' values that are propagated by the call to 'int' in the
range 0 ... 255.

2) Wherever we call one of the mentioned functions with a 'char' as
argument, all non-ASCII (UTF-8) characters have the high bit set in each of
their two to four bytes. Thus, propagation to 'int' (done by the compiler)
calls the function(s) with invalid (negative) values. This results in
undefined behavior - although it appears to be benign in many cases.

3) The "naïve fix" of (2) would be to mask each byte with '&255' or
'&0xff' before calling any of these functions. An alternative would be to
cast to 'unsigned [char]' This fixes the "negative value" issue but not the
problem: it still results in unintended results. Example: the Euro
character U+20ac is three bytes (hex: e2, 82, ac) in UTF-8 encoding. It is
obvious that calling any of the functions either for the first or for all
three bytes consecutively can't answer whether '€' is uppercase or
lowercase or any of the other "questions".

3) Thus every occurrence of any of these functions must be considered in
its context and - if necessary or useful - we'd have to call
Unicode-capable replacement functions per *character* and not per byte.

Anyway, this should be kept open and fixed in the future. It should not
block the release of 1.4.x though since it doesn't seem to be high priority
anymore.

My take on this: let's fix it in 1.5.0.

An example of unexpected results can be seen in fluid if the filename
contains non-ASCII characters, e.g. 'test.fl' in Russian (cyrillic
alphabet) would be 'Тест.fl' (see uploaded file). The generated header
'Тест.h' looks like:
```
// Minimal fluid example
// generated by Fast Light User Interface Designer (fluid) version 1.0400

#ifndef __________h
#define __________h
#include <FL/Fl.H>
#include <FL/Fl_Box.H>
#endif
```

The '#ifndef' and '#define' lines are obviously not what we want, otherwise
all header files with 4 (cyrillic) letters would output the same header
guards.

This is *only* an example (the relevant code is in 'fluid/code.cxx').


Link:         https://www.fltk.org/str.php?L3436
[ Direct Link to Message ]
 
     
[ Previous Message ] [ New Message | Reply ] [ Next Message ]
 
 

Comments are owned by the poster. All other content is copyright 1998-2026 by Bill Spitzak and others. This project is hosted by The FLTK Team. Please report site problems to 'erco@seriss.com'.