UTF-8 Problems

We try to make use of UTF-8 encoded character strings in wini0@ calls and in write statements using ftn95 version 7.10.

We have observed the following problems.

1. The German character 'ö' is not displayed correctly in a winio@ call displaying a menu item. However, the German characters 'ä','ü','Ä','Ü' and 'Ö' are displayed correctly in the same situation. In the error case of character 'ö' blanks are displayed in the menu item.

2. We run into problems when trying to make use of write statements displaying UTF-8 characters, if no winio@ call has been executed before the write statement. In case of having executed a winio@ call before the first write statement the write statements work fine.

Please use the url following to get an image with more informations

http://www11.pic-upload.de/06.10.14/akw991qhnde.jpg

Regards, Dietmar

I have logged this for investigation.

Paul,

how do you proceed with the investigation?

We are interested in moving our complete code basis to UTF-8 code but as long as these bugs exist we will not do so.

Do you have an idea when we could expect an update where these UTF-8 problems are fixed?

Kind regards, Dietmar

Dietmar

We are working flat out on the port to 64 bit at the moment, so I am not able to give you a firm date for this. Hopefully it will all be sorted before the end of the first quarter of 2015.

Paul,

we observed another problem in this context:

we would like to enter UTF-8 characters using the winio format string %rs. We currently enter the UTF-8 characters by Copy &Paste from another document. In our example we use Chinese characters for copying to the %rs box. The buffer of the %rs box will contain characters '?' for the Chinese UTF-8 characters copied. We would have expected that the buffer contains the UTF-8 code of the Chinese UTF-8 characters.

We have created a picture you may refer to by the link:

http://www11.pic-upload.de/19.12.14/gt1ljpr8tj9m.jpg

The first and second line of this picture refer to the %rs box after copying the Chinese characters. The first line contains the contents of the buffer (no Chinese characters) whereas the second line contains Chinese characters int the %rs box. For control reasons we have added a test string 'test' left to the Chinese characters.

We have read in the UTF-8 documentation about problems in this context concerning the %eb box, but nothing concerning %rs boxes.

Do we have a chance to bypass this for %rs boxes or is there a general problem concerning user supplied UTF characters to %rs boxes, as well?

Regards, Dietmar

I don't have an immediate answer to this query. I will try to find some time to look into it.

Dietmar

I have looked briefly at these issues today and it would be a great help if you could supply some short sample programs, one for each issue raised.

The following program works OK for copying characters from the 'Character Map' utility.

winapp
program main
integer i,winio@
character(len=20) txt
C_EXTERNAL ENABLE_UTF8@ '__enable_utf8' (VAL)
call ENABLE_UTF8@(1)
txt = ' '
i = winio@('%rs', txt)
end

but there are no Chinese characters in this utility.

It turns out that you don't need the call to enable_utf8@ in the above sample.

ClearWin+ just uses the standard Microsoft edit control and pasting is done by the API. This means that it is basically a Microsoft issue.

If I can find any further information on the Internet then I will let you know.

Paul wrote 'but there are no Chinese characters in this utility'. In order to get some characters displayed properly in CharMap, you need to first select (inside CharMap) a font that contains those characters. For example, choosing Arial Unicode MS will enable you to select Chinese PinYin characters which, when copied and pasted into the box displayed by running the example program shown above, are displayed correctly.

Many thanks for the information.

Paul,

the code following is an example of what goes wrong to our opinion. It is your example slightly modified.

      winapp 
      program main 
       integer i,winio@ 
       character(len=20) txt 
       C_EXTERNAL ENABLE_UTF8@ '__enable_utf8' (VAL) 
       call ENABLE_UTF8@(1) 
       txt = ' ' 
       i = winio@('%fn[Arial Unicode MS]&')
       i = winio@('%`rs%nl&', txt) 
       i = winio@('%rs', txt) 
      end

We copied a text with Chinese characters from another document, then pasted it to the edit field [%rs] in question. What is displayed on the screen seems to be correct, see web link http://www11.pic-upload.de/22.01.15/m7rv8gb75asw.jpg , however, the txt buffer does not contain any Chinese character (what we expected to be the case).

Regards, Dietmar

Thanks for the sample. I can now see what is expected.

I have not found a way to copy Chinese characters from the input edit box to the read-only edit box. ClearWin+ uses WM_GETTEXT which is equivalent to GetWindowText and this works fine for other UNICODE characters but not for Chinese characters.

Can you explain the wider context of what you want to do? There may be some other way to get the characters. For example, they may be accessible directly from the clipboard.

Paul,

our intent was to show that buffer txt does not contain the Chinese characters although we thought it should. Instead txt contained 2 characters '?'. This is the first line of the picture referred to by the web link of our previous post. We need the correct Chinese UTF values in the buffer because in our main application which we want to move from ASCI to UTF characters we would like to process the UTF characters in further parts of the code.

Here is another example which writes a text file containing the buffer txt without Chinese characters after having pasted the Chinese characters to the edit filed.

winapp 
      program main 
       integer*4 i,winio@, f_write 
       character(len=20) txt     
       common /COMTXT/ txt
       EXTERNAL f_write
       C_EXTERNAL ENABLE_UTF8@ '__enable_utf8' (VAL) 
       call ENABLE_UTF8@(1) 
       txt = ' ' 
       i = winio@('%fn[Arial Unicode MS]&')
       i = winio@('%`rs%nl&', txt) 
       i = winio@('%^rs', txt,f_write) 
      end

      function f_write()
       integer*4 f_write
       character(len=20) txt 
       C_EXTERNAL ENABLE_UTF8@ '__enable_utf8' (VAL) 
       common /COMTXT/ txt
       call ENABLE_UTF8@(1)
       open(10,file='text.log') 
        write(10,'(A)') txt
       close(10)

       f_write=2
      end

The web link following contains a screenshot from the debugger session after pasting the Chinese characters.

http://www11.pic-upload.de/23.01.15/vol5hcu8344q.jpg

Regards, Dietmar

This issue is proving to be very tricky. Clearly the Microsoft edit box used for input is able to receive Chinese characters so it must be working in UNICODE mode. But my attempts to retrieve the characters via a call to GetWindowTextW is not working. Also IsWindowUnicode is returning false which is clearly wrong.

The best I can do at the moment is to copy the text to the clipboard and then I can retrieve it from there.

One way forward may be to get something working using only the API and not using ClearWin+ at all, to see if that provided any clues but that would be a significant task right now.

I will take a look at the problem with the German characters in a menu to see if any progress can be made there.

The first issue in the initial post on this thread has now been fixed for the next release (German character in menu name).

Do you have a short sample for the second item in the initial post?

Paul,

we have written the following code concerning the second issue of our initial post:

      winapp 
      program main 
       integer*4 i,winio@ 
       character(len=20) txt     
       common /COMTXT/ txt
       C_EXTERNAL ENABLE_UTF8@ '__enable_utf8' (VAL) 
       call ENABLE_UTF8@(1) 
       txt = 'hallo 先生' 

C      before using first winio call all utf8 outputs are wrong
       write(*,'(A)') txt    !+ wrong output of characters
       write(*,'(A)') txt    !+ wrong output of characters

       i = winio@('%fn[Arial Unicode MS]&')
       write(*,'(A)') txt    !+ correct output of characters 
       i = winio@('%`rs%nl&', txt)  !+ correct output of characters 
       i = winio@('%rs', txt)  !+ !+ wrong output of characters (thats strange! new bug?)
      end

Please have a look at web link

http://www11.pic-upload.de/23.01.15/dkq36vowrspd.jpg

for a picture corresponding to the executable compile from that code.

It turned out that we found another problem. If buffer txt has been set to a string containing Chinese characters, then the output of the edit field [%rs] is wrong. Please see the comments in the example above.

Regards, Diemar

The main bug in the last sample above has now been fixed for the next release.

The 'new bug' referred to above is in fact a different manifestation of the remaining outstanding bug on this thread - Chinese characters are not copied correctly from a %rs edit to the user's character buffer. Hence they are not transmitted to another control that uses the same buffer.

All of the outstanding bugs described on this thread have now been fixed for the next release. However, for the moment, the problem with Chinese characters and %rs has been fixed by disabling user popup menus in this context. So for the time being you will not be able to attach a user popup menu to a %rs edit box that has UTF8 enabled. However you should still get the standard Microsoft popup menu for an Edit control and this provides options for right to left reading order etc.