Feature/russian language

[Fairyon]

Has some issues with displaying some options. Probably lies on some fonts and needs to be fixed. (Tested by replacing the original files before start of the application)
Open question: how to integrate into the current translation and where to switch the languages?

[dgelessus]

I've had a quick look over the files. I can't really read Russian, so I can only comment on the technical aspects, not the translations.

Apparently the Git LFS upload of the binary files (.p2f, .bmp) didn't work. I get some errors when trying to fetch your branch:

[bcae551ad7b1fd24b3a98c27849e92af9e31731c320af680146fb0d2f0bac864] Not Found: [404] Not Found
[66393f065831b5b074378940f628dc93d0bd8c47144825f0e46a309006ac8f1a] Not Found: [404] Not Found
[f63a5bd6544ba437eee0c62f7728cca35031d2488c7fc748b8fb0530eb9ccd34] Not Found: [404] Not Found
[c5a44a17c7b8ddfc67537d64de7bfd167543fd0cc4fb01bfd7885fb7b55d51c0] Not Found: [404] Not Found
...

Looking at the .loc files: Most of the text is incorrectly encoded - for example, °äàÞ should be Афро. I'm guessing that you replaced some existing characters in the fonts with Cyrillic letters, so it shows up correctly in the game. Ideally, the fonts should be extended with the Cyrillic letters at their proper Unicode positions, without replacing existing characters, so that the Russian translations can be encoded correctly in the .loc files and don't conflict with the other translations.

(For anyone wondering: the Russian text is ISO 8859-5, mis-decoded as ISO 8859-1, then saved as UTF-8. So this can be fixed programmatically e. g. with s.encode("latin-1").decode("iso-8859-5") in Python.)

Open question: how to integrate into the current translation and where to switch the languages?

To clarify: the H'uru client already has a language selector on the login screen, where you can select the existing translations (French, German, etc.). You can see that when playing on the Gehn shard for example.

The list of supported languages is hardcoded, so adding Russian to the list requires code changes in the client (as mentioned in H-uru/Plasma#1384). I can't promise anything, but I might try to do this in the next few days. Once that's implemented, the ...Russian.loc files can be changed to use <translation language="Russian"> instead of English.

[Fairyon]

Looking at the .loc files: Most of the text is incorrectly encoded - for example, °äàÞ should be Афро. I'm guessing that you replaced some existing characters in the fonts with Cyrillic letters, so it shows up correctly in the game. Ideally, the fonts should be extended with the Cyrillic letters at their proper Unicode positions, without replacing existing characters, so that the Russian translations can be encoded correctly in the .loc files and don't conflict with the other translations.

Yes, I just used the fonts from other translations and never worked on them myself, so the easiest way for me was to save the files with the encoding needed for proper rendering in the game.
So the encoding is not a problem.
Anyway, these fonts somehow crash some texts in the video settings tab, so I guess we need to find out which font is used and fix the problem. I need some help to find out which file needs to be fixed.

The list of supported languages is hardcoded, so adding Russian to the list requires code changes in the client (as mentioned in H-uru/Plasma#1384). I can't promise anything, but I might try to do this in the next few days. > Once that's implemented, the ...Russian.loc files can be changed to use instead of English.

It would be great if someone could change that! :)

[dgelessus]

Yes, I just used the fonts from other translations and never worked on them myself, so the easiest way for me was to save the files with the encoding needed for proper rendering in the game.
So the encoding is not a problem.

It won't be a problem for Russian, but it will be a problem for the other translations 🙂 The font format and the engine have proper Unicode support, so we should put all the characters in the right places in one font file, instead of using weird encoding tricks and multiple copies of the fonts.

Anyway, these fonts somehow crash some texts in the video settings tab, so I guess we need to find out which font is used and fix the problem. I need some help to find out which file needs to be fixed.

As mentioned above, the Git LFS upload didn't work, so I can't download the font files.

Also, it's possible that H'uru clients won't have this crash. The MOULa client shipped by Cyan isn't based on H'uru and is missing many Unicode-related bug fixes. You should probably test on the Destiny testing shard, using a current H'uru client (download from GitHub Actions - click on the topmost build, then download "plasma-windows-x64-internal-release"), plus the SDL files and server.ini from this thread.

@ others: Do we have a more up-to-date tutorial for setting up a current H'uru client for Destiny? I see Mystler's forum post still points to the old moul-scripts repo, and it suggests copying data files from Cyan's shard, which won't work as-is with current H'uru scripts...

[Fairyon]

As mentioned above, the Git LFS upload didn't work, so I can't download the font files.

Somehow I see all the .p2f files on github commit here: 00b61ac

It won't be a problem for Russian, but it will be a problem for the other translations 🙂 The font format and the engine have proper Unicode support, so we should put all the characters in the right places in one font file, instead of using weird encoding tricks and multiple copies of the fonts.

There are many different fonts. I can try to replace everything by myself. The question is: how? The old PlasmaShop does not allow to change the character placement. The new one does not open the .p2f files...

You should probably test on the Destiny testing shard, using a current H'uru client (download from GitHub Actions - click on the topmost build, then download "plasma-windows-x64-internal-release"), plus the SDL files and server.ini from this thread.

There is a totally different structure with .age and .fni files in the 'dat' folder. I don't know how to change fonts ant texts here.

[dgelessus]

Somehow I see all the .p2f files on github commit here: 00b61ac

The way Git LFS works is a bit confusing 🙁 The commit itself is pushed correctly, but the LFS files that the commit points to are not pushed. This can happen if you push your changes and then cancel the git.guildofwriters.org login - then the commits are pushed to GitHub, but the files are not pushed to the GoW server.

I don't know what's the right way to fix this... possibly git lfs push is enough?

There are many different fonts. I can try to replace everything by myself. The question is: how? The old PlasmaShop does not allow to change the character placement. The new one does not open the .p2f files...

The newest PlasmaShop version is 2023.03.21 and that can open .p2f files... But I see what you mean - it only lets you adjust some numbers, but you can't change the graphics or add/remove characters 🙁 I guess we will have to write a custom script for merging the fonts then...

There is a totally different structure with .age and .fni files in the 'dat' folder. I don't know how to change fonts ant texts here.

The download from the Plasma repo is not complete, you still need to add the data files. Because you already have moul-assets cloned, you can just copy the avi, dat, sfx folders from moul-assets/compiled into the "client" folder.

[dgelessus]

@Hoikas just reminded me about his TrollLand test shard, which should be easier to set up than Destiny. It has a pre-built client that you can just extract into a new folder and run, and everything else will update itself automatically. (You can still save some download time if you copy avi, dat, sfx from moul-assets.)

[Fairyon]

I guess we will have to write a custom script for merging the fonts then...

Who can do that?
I could do it by myself, but I don't knoe the .p2f files structure and cannot find it out on the PlasmaShop repo. And the old OHB projects are not available...
Where Can I get the sctucture of the file? Or maybe can someone write a script for me, which moves the cyrillic letters to the right positions?

[dgelessus]

Who can do that?
I could do it by myself, but I don't knoe the .p2f files structure and cannot find it out on the PlasmaShop repo.

You can look at plFont::ReadRaw in the Plasma repo (not PlasmaShop), which is the code that loads .p2f files. It's relatively short and simple, so it's not that hard to see the file format from the code.

If you can do the merging yourself with this info, that would be great 🙂 Feel free to ask if you need to know anything else.

Or maybe can someone write a script for me, which moves the cyrillic letters to the right positions?

I could probably write a script for that, but probably not soon - I haven't even done the other code change I talked about...

Also, can you please try the Git LFS upload again? I still get errors when I check out your branch.

[Fairyon]

I already tried git lfs push
here are the cyrillic fonts, if someone need them: https://1drv.ms/f/s!Au5bFJVqllRMgosXDjng5eBNmhwHig?e=OXCcR8

You can look at plFont::ReadRaw in the Plasma repo (not PlasmaShop)

Thank you for the link, I will look into that :)

---

I looked into it and managed to extract data, but I don't see the placement of the characters in the encoding map... There are only placements on the bitmap.

I need some explanation, how to place the characters at the right unicode positions, it is not inuitively visible in the font data

[dgelessus]

I've submitted H-uru/Plasma#1397, which adds Russian (and a few other languages) to the language switcher. Hoikas has already updated TrollLand, so you can try your changes there with the new patch. (You'll have to change language="English" to language="Russian" in all the .loc files.)

I need some explanation, how to place the characters at the right unicode positions, it is not inuitively visible in the font data

Sorry for not answering earlier, I didn't see your edit. The Unicode values aren't stored directly, they come from the placement of the characters in the font. So for example U+0411 (Б) has to be at index 1041 (0x411) in the fCharacters array. The unused characters in between have to be included in fCharacters too, with all fields set to 0 (they don't need any data in the bitmap).

[Hoikas]

We might want to consider modifying plFontConverter to automatically produce .p2f files with the required characters. Since this is open source, I don't see a problem with introducing some form of compression for the characters if this would cause the .p2f files to become overly bloated on disc.

[Fairyon]

We might want to consider modifying plFontConverter to automatically produce .p2f files with the required characters.

It would be awesome! *_*

@dgelessus Thanks for the Tip, I will try to create a converter for my fonts

[Fairyon]

Ok, I managed to convert the fonts, but only the books are showing correctly, somehow GUI with the font Collegiate-*.p2f uses some othe encoding. Can you tell me which one?

[dgelessus]

All the .p2f fonts in Uru use the same format. What exactly isn't working with the GUI fonts? Perhaps it's not a problem with the fonts, but a rendering bug or an encoding mixup somewhere in the engine.

[Fairyon]

Somehow, my GUI Font is broken and some messages in the Options pane stay empty. I just skip this part in the translation and keep GUI untranslated.
I wanted to upload the changes, but git push thrown a new error:
Fatal error: Server error &{%!!(string=https) %!!(string=) %!!(*url.Userinfo=) %!!(string=git.guildofwriters.org) %!!(string=/H-uru/moul-assets.git/info/lfs/locks/verify) %!!(string=) %!!(bool=false) %!!(string=) %!!(string=) %!!(string=)}s(MISSING) from HTTP 502

Has it something to do with guildofwriters LFS rights?

[dgelessus]

Yeah, looks like git.guildofwriters.org is down - I get a 502 error as well when pulling. (ping @Hoikas)

[Hoikas]

Should be up now.

[Fairyon]

Can someone help me! There is a problem with Collegiate fonts.. when I change them, Video settings are not being displayed. I uploaded fonts to my own repository https://github.com/Fairyon/uru_p2f
I will appreciate if someone finds time to look over it and tell what is wrong here...
All other fonts by Collegiate are working fine by me.

Just put those files during you are on the login screen in MOULa (where you enter your password) and when you in the game open video settings no text are being displayed...

[dgelessus]

Just put those files during you are on the login screen in MOULa (where you enter your password) and when you in the game open video settings no text are being displayed...

Yeah, the OpenUru MOULa client from Cyan has known Unicode/translation bugs, so that's not surprising. You need to test with a H'uru client, e. g. TrollLand as described above.

[Fairyon]

Hoikas has already updated TrollLand, so you can try your changes there with the new patch.

I downloaded the TrollLand client from the link, you submitted earlier and there is still no selector for Russian language available..

Same here, when I use Collegiate fonts, the texts in the Graphics Settings are not visible. Can you check my fonts at https://github.com/Fairyon/uru_p2f ?

[dgelessus]

I downloaded the TrollLand client from the link, you submitted earlier and there is still no selector for Russian language available..

You need to run the internal client (plUruLauncher.exe/plClient.exe) to get Russian in the language list. The external client (UruLauncher.exe/UruExplorer.exe) has Russian disabled, because the translations aren't integrated yet.

Same here, when I use Collegiate fonts, the texts in the Graphics Settings are not visible. Can you check my fonts at https://github.com/Fairyon/uru_p2f ?

Your Collegiate p2f files all look normal to me, so it's probably a rendering problem. I'll see if I can find anything.

[dgelessus]

Did some more testing and encountered some bugs when rendering characters that aren't supported by a font. Fixes pushed to H-uru/Plasma#1397. I'm not sure if that helps with this problem though.

I can reproduce your problem with the missing text in the options menu, but only if I use the fonts from your OneDrive (where you replace the Latin-1 characters with Cyrillic ones).

If I use the new fonts from https://github.com/Fairyon/uru_p2f (where the Cyrillic characters are at the proper positions) and change the Russian.loc files to real UTF-8, then everything works correctly:

[Fairyon]

If you load the English or another language with new fonts enabled (from https://github.com/Fairyon/uru_p2f) and try to load the "graphics settings", that one in the middle, not the left one, than no texts are displayed.
There only trouble with this one and not with other ones, I noticed.

[dgelessus]

Ah, I checked the wrong screen, oops >.> Yes, I can reproduce this, even with the new fonts.

Okay, the problem is that the Cyrillic у (U+0443) character is a few pixels too tall. This increases the height of the entire font (calculated internally by the engine). The text boxes in the graphics settings have very tight sizes, so the font renderer cuts off the entire text because it's now too tall.

It looks like the Cyrillic у should have the same size as the Latin y, but the bitmap for the Cyrillic у has a few invisible pixels at the bottom, which are causing the problem. The difference is invisible in PlasmaShop, but you can see it in a hex editor:

You'll have to change all the Cyrillic у characters to the same pixel height as the Latin y of the same font size. (I think you don't need to change the bitmap - after you reduce the character height, the engine will ignore the extra pixel rows.)

[Fairyon]

Thanks for your help, now the fonts are properly working. I pushed the changes to the repo. Now it is waiting for the final merge:)
Tested by me.
If the fonts are still not loading, they can be downloaded here: https://github.com/Fairyon/uru_p2f

[dgelessus]

Thank you! The newest commit downloads properly for me - so I think your Git LFS upload works now. But for the older commits, the files are still missing, so we can't merge it like this 🙁

If you know how to use git rebase --interactive: can you drop all commits except the newest one, then force-push? If not, you can create a new branch, commit all the files again, then push that and make a new PR.

At first look, this all looks good from the technical side (I can't give any feedback on the actual translations 🙂). I'll test a bit more later.

[Deledrius]

For future information, you can push the new (local) branch to this (remote) one in order to update the PR without needing to create a new PR.

( Bookkeeping Note: This PR was recreated as #237 )