FLTK logo

STR #1903

FLTK matrix user chat room
(using Element browser app)   FLTK gitter user chat room   GitHub FLTK Project   FLTK News RSS Feed  
  FLTK Apps      FLTK Library      Forums      Links     Login 
 Home  |  Articles & FAQs  |  Bugs & Features  |  Documentation  |  Download  |  Screenshots  ]
 

Return to Bugs & Features | SVN ⇄ GIT ]

STR #1903

Application:FLTK Library
Status:1 - Closed w/Resolution
Priority:1 - Request for Enhancement, e.g. asking for a feature
Scope:3 - Applies to all machines and operating systems
Subsystem:Unicode support
Summary:Check for missing functionalities from original UTF8 patch
Version:1.4-feature
Created By:matt
Assigned To:AlbrechtS
Fix Version:1.3.0
Fix Commit:unknown
Update Notification:

Receive EMails Don't Receive EMails

Trouble Report Files:


Name/Time/Date Filename/Size  
 
#1 timothy.lee
03:10 May 15, 2008
fltk-1.3-utf8.patch
1.9M
 
 
#2 timothy.lee
17:43 Feb 09, 2009
fltk-1.3-u8wrapper.patch
28k
 
 
#3 timothy.lee
00:36 Feb 10, 2009
fltk-1.3-xft.patch
17k
 
 
#4 michaelbaeuerle
05:27 May 27, 2014
str_1903_r10174_xft_fontset_v1.patch
28k
 
 
#5 michaelbaeuerle
05:27 May 27, 2014
str_1903_r10174_xft_fontset_v1.png
15k
 
 
#6 ianmacarthur
16:20 Jun 04, 2014
alt-xft-glyph-subs-V1.zip
46k
 
 
#7 ianmacarthur
14:21 Jun 07, 2014
alt-xft-glyph-subs-V2.zip
46k
 
 
#8 ianmacarthur
16:05 Jun 08, 2014
alt-xft-glyph-subs-V3.zip
47k
 
 
#9 ianmacarthur
15:02 Jun 09, 2014
alt-xft-glyph-subs-V4.zip
48k
 
 
#10 ianmacarthur
14:59 Jun 11, 2014
alt-xft-glyph-subs-V5.zip
54k
 
 
#11 ianmacarthur
16:56 Jun 17, 2014
alt-xft-glyph-subs-V6.zip
55k
 
 
#12 ianmacarthur
15:58 Jun 19, 2014
alt-xft-glyph-subs-V7.zip
55k
 
 
#13 michaelbaeuerle
06:12 Sep 10, 2014
alt-xft-glyph-subs-V7_for_r10299.tar.bz2
45k
 
     

Trouble Report Comments:


Name/Time/Date Text  
 
#1 matt
13:42 Mar 27, 2008
Merge UTF-8 support into the FLTK 1.3 source tree  
 
#2 timothy.lee
03:13 May 15, 2008
I've posted FLTK-UTF8 patch updated for 1.3 branch.  Hopefully this will speed up development.

The patch compiles and runs cleanly on linux, mingw and MSVC.  Don't have access to Mac.
 
 
#3 ianmacarthur
00:57 May 16, 2008
I haven't had a chance to look at the patch yet.
Is it based on one of the existing utf8 patches? And if so, which one?
 
 
#4 ianmacarthur
04:46 May 17, 2008
OK - I started to look at this patch, but it's taking more time than I had allowed... And I've only had a chance to try it on my linux box so far, so...

Initial comments: Patch applied OK to current (at time of writing) SVN 1.3 tree, and code builds.

Observations: the header files Xutf8.h and fl_utf8.H - patch seems to have included all the text twice in each file. Odd. Still builds OK.

The XFT fontopen code has been extended to handle a list of fonts, this seems like a Good Thing.
There are some uses of Xft*16 functions - these always make me uneasy - I think I'd rather go the Xft*UCS4 route, at least on *nix, for anything that can't be readily done directly as utf8. I worry that at we need to handle glyphs that can't be represented in a single 16-bit field, and going to a UCS4 type is more future proof.

Something in the patch breaks GL text rendering - see this in the cube and fractals demos, where the FPS text in the GL window is not displayed (1.1.9 and 1.3.x both seem fine.)
This is a nuisance for me, as I have need of that facility in my code!

Time permitting, more comments to follow. Also (possibly!) OSX testing...

--
Ian
 
 
#5 timothy.lee
19:27 May 18, 2008
The patch was initially extracted from http://www.oksid.ch/xd640/fltk-utf8_1.1.6-1.tar.gz.  Since then, I've been adapting for each of the FLTK releases (up to 1.1.9).

This patch is different from oksid's in the following ways:

1. The -utf8 suffix have been removed from the output libraries.

2. No extra classes like Fl_Shaped_Window, Fl_Table or Fl_Tree.  Just the bare minimum to get UTF-8 content working.

3. UTF-8 Xft support under X11.

And yes, OpenGL's probably broken -- I don't use GL, so haven't been testing that.  Sorry.
 
 
#6 ianmacarthur
01:06 May 19, 2008
Ah, OK... I guess we've been working in parallel a bit then...

I had done somework on the Oksi'D patch to bring it into 1.1.8 (see here) http://www.imm.uklinux.net/fltk/fltk118-utf8-2008-02-24.tar.bz2

This of course differs from both the Oksi'D patch, and from your patch!
In particular I did a chunk of work on Xft, GL and OSX support.
I ripped out a lot of the code Oksi'D had added to support IM's on older winXX variants (I don't think we are supporting win9x anymore...)
Also, I hacked around a lot of the utf utility functions to align them more closely with the fltk2 variants.

Anyway, some of my stuff is in 1.1.8/1.1.9 and 1.3.x, but not the utf8 specific parts...
Can you take a look at my stuff and see where it fits with yours, and I'll try and do the same from this end? Mind you, it's been a while since I worked on this - I can't remeber what I did!

Cheers,
--
Ian
 
 
#7 astrand
06:54 Sep 30, 2008
fltk118-utf8-2008-02-24.tar.bz2 was easy to build and the utf8 test looks good. Is this version supposed to work on all major platforms (Solaris/Linux/Windows/OS X)? Any known stability problems?  
 
#8 ianmacarthur
07:07 Oct 02, 2008
The fltk118-utf8-2008-02-24.tar.bz2 tarball is deprecated now in favour of the fltk-1.3 svn which now incorporates most of the functionality from that tarball. (Missing some of the functionality that is in the timothy.lee patch though, that should probably be considered.)

From my testing, fltk-1.3 seems OK on OSX, win32 and Linux with XFT, others have tested Linux without XFT and reported it OK.

I can't vouch for Solaris - probably OK, as the X implementation is good on linux.
--
Ian
 
 
#9 fabien
01:51 Jan 02, 2009
renamed this rfe to more accurate/useful title
(was: add UTF-8 support)
 
 
#10 timothy.lee
18:11 Feb 09, 2009
I finally had time to merge my UTF-8 changes into the latest SVN trunk (r6639).

The flkt-1.3-u8wrapper patch changes all filename-related system calls (fopen, access, getenv, chdir, system) to their equivalent UTF-8 wrapper function.
 
 
#11 timothy.lee
00:38 Feb 10, 2009
The flkt-1.3-xft patch allows a Unicode string to be drawn with a font set (poor man's Pango).  
 
#12 timothy.lee
17:27 Feb 22, 2009
Dear Ian,

> Is there a list of what functionalities this particular patch adds?
>
> Also, the other patches you have posted - do they need to be applied in
> a particular order?
>
> These patches seem to be mainly linux oriented - have you had a chance
> to test on other platforms? Are there any known issues? How has the
> testing gone, generally?

My patches are independent of each other.  I have not had any issues using these patches together.

The Xft and iconv patches are geared towards Linux, and I do not have the opportunity to test on other platforms.  However since conditional compilation is used (ie: USE_XFT and HAVE_LIBC_ICONV macros), the new code should have no effect on other platforms.  It would be great if others could verify this assumption.

Below is a summary of the patches I've posted so far:

fltk-1.3-u8wrapper (under STR #1903):  This patch enforces the use of UTF-8 wrapper functions to ease future porting.  All filename-related system calls (fopen, access, getenv, chdir, system) are changed to their equivalent UTF-8 wrapper function.

fltk-1.3-xft (under STR #1903):  This patch allows a Unicode string to be drawn with an Xft font set (poor man's Pango).

fltk-1.3-iconv.patch (under STR #2148):  This patch makes use of libc's iconv() to convert between UTF-8 and other encodings, instead of placing the conversion table inside libfltk.  External iconv() implementations are not considered for such use.

How do I proceed to get these patches applied to the trunk?
 
 
#13 matt
12:56 Nov 14, 2010
Not essential for the 1.3.0 release. Temporarily moved to 1.4 .  
 
#14 andreldm
17:40 May 26, 2014
This patch would be much appreciated if merge, as in my case, a music player needs to display text in different languages at the same time.  
 
#15 michaelbaeuerle
05:34 May 27, 2014
The Xft fontset V1 patch is the first attempt to get glyph substitution working again for 1.3 current. Many things may still be horribly broken, but the attached picture show that FLTK now can at least mix latin, cyrillic and japanese characters from different fonts in a single string.

I have copied the strings from Greg's demo program and added a test file test/fontset.cxx for demonstration.
 
 
#16 ianmacarthur
16:19 Jun 04, 2014
I've posted my first pass at my proposed alternative way of achieving glyph substitution "on the fly" with fontconfig/XFT on X11 systems.

The attached zip alt-xft-glyph-subs-V1.zip is not a patch per se, it is the full modified files:

src/Fl_Font.H
src/fl_font_xft.cxx
src/fl_set_font.cxx
src/fl_utf.c
FL/Enumerations.H
FL/fl_utf8.h
test/unittest_text.cxx
test/rotated_text.cxx
test/utf8.cxx

My intent is to lower the barrier for testing - anyone can drop these files onto a recent weekly snapshot and rebuild, and it should Just Work. Probably...

Measurement of text extents is still a bit borked, but glyphs substitution and text rotation and so forth all seem to work "well enough".

You will need to have the ABI guard in Enumerations.H set though.

I have tweaked 3 of the test programs to illustrate the usage:

- unittest is tweaked just to use a "compound" string (EN, Ru, Jp) using the UTF8 values from Greg's demo code. View the "rendering text" page of this test.

- rotated_text is modified to use the "compound" string, and to draw a bounding box around the rotated string.

- utf8 is modified to use the substitution for the "whole Unicode" page... This can be Quite Slow the first time it loads and searches all the faces... Note that the UTF8 test calls Fl::set_fonts(*) to load *all* the fonts on the system and the glyph substitution logic then searches all of them...

The other tests I modified only load the fltk built-in fonts and one other CJK face that this box has installed; this speeds things up a lot since my glyph subs logic only tests fonts the app has loaded, *not* all the fonts on the system (unless you use set_fonts() to load all the fonts of course!)

In testing, you may need to tweak the "extra" CJK font I use to match something that exists on your system.

I have other code that uses fontconfig to locate a system font that is a Good Fit for a given body of text, but that is not ready yet and is not in this post. Once it is cleaned up, the idea is that you can pass it a chunk of UTF8 text and it will suggest which font that you can load that *should* be able to render that text...
 
 
#17 ianmacarthur
14:20 Jun 07, 2014
Revised version of my "patch" posted as alt-xft-glyph-subs-V2.zip

This *does not* resolve the issue with text_extents, nor does it include my functions for using fontconfig to "search" for a suitable font before loading it. Both these features are in work, but I just have not had the time to actually get them ready...

This patch does:

- Address a bug in the glyph substitution such that, if the composite string mixed both TT and Type-1 fonts, the rotation went awry.

- As per Michael B's suggestion, I have added a musical G clef to exercise glyphs from above the BMP

- As per Michael's suggestion, I have added a few more font options to the load path (FreeSerif and Kochi Mincho) so the test should work on a wider range of hosts

- As per Michael's suggestion, the files in the zip now ought to be path as per the fltk folder layout, so the zip file can be unziped over a stock fltk tree

Feedback welcome.
 
 
#18 ianmacarthur
16:05 Jun 08, 2014
Another revised version of my "patch", alt-xft-glyph-subs-V3.zip.

This *still* does not fix the text extents issue, but adds a demonstration version of the scheme I'm proposing for finding a font to fit a block of text.

The function fl_xft_font_finder_for_UTF8_text(...) is added (somewhat arbitrarily) to fl_font_xft.cxx, and if passed a block of UTF8 encoded text, it will then *print to stdio* a list of suggested fonts, and how many of the needed glyphs that *does not* have.

This is *not* useful as a run time feature, it is just to let us explore what fontconfig can do for us.

Note, also, that I have not implemented yet a way to create a *set* of fonts that, taken together, will cover all the needed glyphs...

I think this is possible, but... well, not yet understood!

Feedback welcomed. Also, fixes!
 
 
#19 ianmacarthur
15:01 Jun 09, 2014
Now posting alt-xft-glyph-subs-V4.zip, which implements a function fl_xft_font_set_find() in fl_font_xft.cxx that, when passed a corpus of UTF8 encoded text, returns an array of (up to 5) char* names for fonts to load.

This mechanism is used in the modified rotated_text demo to load a set of fonts sufficient to execute the demo without me loading any fonts by name, so should work on any host that has any suitable fonts available.

(If there are no CJK fonts on your system though, this will still fail I imagine!)

This works, but the actual code is a mess! The name mechanism is especially clunky...

Anyway, as ever, feedback on whether this is useful, or the right direction to take, is welcomed.
 
 
#20 ianmacarthur
14:58 Jun 11, 2014
Revised version of the font picking function: I have added the function fl_xft_font_set_find_similar(...) that takes in a Fl_Font value referring to some already laoded font, and then tries to find a font with simialr metrics to use for the replacement glyphs.

rotated_text modified to exercise this "new, improved" font finding scheme.

Note that I have not actually modified fl_font_xft to take advanatge of this yet however, so there's not much difference on the display...

See: alt-xft-glyph-subs-V5.zip
 
 
#21 ianmacarthur
17:00 Jun 17, 2014
V6 version of my patch that actually tries to use a glyph with "similar" metrics when substituting... i.e. it will try to follow the bold / italic / serif / sans nature of the base font when picking a replacemnt glyph.

The rotated_text and unittest demos, and the utf8 demo, should now be using this scheme in the patch.

The utf8 demo is *slow* to start the first time, a lot of font picking is going on there. The other two seem to be "OK" since they only load a few extra faces...
 
 
#22 ianmacarthur
16:01 Jun 19, 2014
Test alt-xft-glyph-subs-V7.zip posted: when searching for a substitute face this checks to see of the base font is AA, and if it is it tries to force things so that only AA substitutes are considered.

This may address an issue that Michael has been seeing (though I confess I have failed to reproduce it) where the picker chooses a bitmapped font with extensive glyph coverage, rather than other better looking AA fonts with less extensive, but still adequate, coverage...
 
 
#23 michaelbaeuerle
06:16 Sep 10, 2014
I have attached an archive with Ians V7 patches modified to work with r10299.

I have used a bzip2 compressed tar file because my zip utility don't accept the symbolic link. The content are whole files to copy over the old ones as before.
 
 
#24 michaelbaeuerle
08:37 Sep 11, 2014
Some results with r10299 and the V7 patch:
http://micha.freeshell.org/tmp/flnews-0.9gs2_glyphen.png
using this font set:
|
| Chose font:  0 Baekmuk Batang, still missing 5 glyphs
| Chose font:  1 FreeSerif, still missing 4 glyphs
| Chose font:  2 TakaoPGothic, still missing 3 glyphs
| Chose font:  3 Lohit Kannada, still missing 2 glyphs
| Chose font:  4 Lohit Telugu, still missing 1 glyph
| Chose font:  5 AR PL KaitiM Big5, still missing 0 glyphs

"Kannada" failed for some unknown reason, but the rest is not
looking bad.
The clipping on the end of some lines should be the wrong
calculated width that we had already seen on the bounding
box of the "rotated text" demo.
 
 
#25 ianmacarthur
14:52 Sep 13, 2014
Thanks Michael - glad to hear the V7 patch more or less works as expected.

Dunno about the Kannada issue though.

Regarding the bounding box issue, I more or less know how to fix it, I think, but there's a big restructuring of my code needed to make it work; and I just can't bring myself to face the change - there's a hill to climb...!
 
 
#27 AlbrechtS
11:39 Jan 15, 2023
Fixed in Git repository.

I assume this is all already done, maybe in 1.3.x or later. Looking at the old patches doesn't seem useful anymore, after more than 14 or at least 7 years (last modified: Sep 18, 2014).

Changed status to "4 - Pending".

Can we close this STR? It appears to be obsolete.

I'll close this if the original authors agree or in one week, whichever comes first.
 
 
#28 matt
06:31 Jan 16, 2023
The current 1.4 implementation has been working well for the last 14 years. SO we can assume that this is done. Most (all?) patches no longer apply anyway.  
     

Return to Bugs & Features ]

 
 

Comments are owned by the poster. All other content is copyright 1998-2024 by Bill Spitzak and others. This project is hosted by The FLTK Team. Please report site problems to 'erco@seriss.com'.