UTF-8 support is broken

[thomas.goorden@…]

Even when setting the Lighttp serven "LANG" to "en_US.utf-8", the Rocketwiki engine breaks on UTF-8 characters like é or ü. Looking at the rocketwiki syntax, it seems there is only pseudo support of unicode for specific German characters like {Ae} (Ä), etc.

There doesn't seem to be a good way to provide an alternative to the rocketwiki engine, so we're kind of stuck. Many pages break because of this.

It's in a sense a Rocketwiki "bug", but there seems to be no way to file it.

[dark]

Rocketwiki supports UTF-8, entities for German umlauts are only provided in addition.

You didn't specify in which way rocketwiki breaks by using UTF8 characters for you. Please provide more specific information.

Please provide more information to reproduce the problem, like server configuration and an affected wiki document.

[thomas.goorden@…]

On the lighttpd side, the configuration has the following :

$HTTP["url"] =~ "^/lqfb/" {
  setenv.add-environment += (
    "LANG" => "en_US.UTF-8"
    ...

(As suggested in another ticket.)

However, when submitting text (preview an issue for instance) that is run through the parser, which includes UTF-8 characters like "é" or "ü", the application breaks with this type of error:

/opt/lqfbprod/env/format/wiki_text.lua:7: Wiki parser process returned with error code 1

Stack trace follows:
[C]: in function 'error'
/opt/lqfbprod/env/format/wiki_text.lua:7: in function 'wiki_text'
/opt/lqfbprod/app/main/draft/new.lua:46: in function 'content'
../env/ui/tag.lua:38: in function <../env/ui/tag.lua:15>
(tail call): ?
/opt/lqfbprod/app/main/draft/new.lua:43: in function 'content'
../env/ui/form.lua:154: in function 'content'
../env/ui/tag.lua:38: in function 'tag'
../env/ui/form.lua:143: in function 'form'
/opt/lqfbprod/app/main/draft/new.lua:19: in function 'func'
...
../env/execute/wrapped.lua:21: in function <../env/execute/wrapped.lua:11>
(tail call): ?
(tail call): ?
../env/execute/filtered_view.lua:28: in function 'filtered_view'
webmcp.lua:380: in function <webmcp.lua:257>
[C]: in function 'xpcall'
webmcp.lua:256: in main chunk
[C]: in function 'pcall'
/opt/lqfbprod/webmcp/cgi-bin/webmcp-wrapper.lua:7: in main chunk
[C]: ?

The wiki engine does this in both "Rocketwiki" and "Classic" setting. The installation is all "latest and greatest":
Software Versie Licentie
LiquidFeedback Frontend
2.0.2
LiquidFeedback Core
2.0.11
WebMCP
1.2.4
Lua
5.1
PostgreSQL
8.4.12
BSD

[dark]

Configuration looks correct for me. I have another idea: Do you have the locale en_US.UTF-8 actually built on your system? This was also discussed here: http://dev.liquidfeedback.org/pipermail/main/2012-July/000135.html

Please tell me, if it helped or not.

[thomas.goorden@…]

That solved it! Using "locale -a" it turned out this particular system only had "en_GB.utf8" built, which works great. Maybe this is a step that should be added to the installation guide?

[dr.ib@…]

i played a little bit with the hs code.

adding
import System.IO (hSetEncoding, stdin, stdout, utf8)
to the beginning of rocketwiki-lqfb.hs/rocketwiki-lqfb-compat.hs

and changing the main "equation" to
main = do

hSetEncoding stdin utf8
hSetEncoding stdout utf8
interact wikiParse

makes the encoding independent from the locale-setting of the process environment

ingo bremer