Support for RTL languages

[WissamAntoun]

Hey Jay,

Thank you for this great library.

I was trying to make ecco work for arabic with aragpt2. I had to bump the transformer version to 4.2.0 and add my own custom gpt2 classes.

But I couldn't figure out how to correctly display the text in a right-to-left direction on colab

this is the link to my fork:
https://github.com/WissamAntoun/ecco-aragpt2

[jalammar]

Hi Wissam! It would be wonderful for Ecco to support RTL languages.

If I'm to break down what needs to be tweaked for the display, it would be mostly in eccojs. Namely:
1- The tokens are rendered inside of boxes. HTML div elements. So the first thing would be to switch the direction/float of these boxes and how they're stacked. This seems to be a simple CSS change. in eccojs. Perhaps ecco needs to know if the model/output is Arabic by setting some flag.
2- More difficult, is getting the text linked properly. In English, if a word is broken to two tokens: Say, "Lang" + "uage", you can get away with making them two separate boxes by close together. In Arabic, however, if the tokens are "ال" + "لغة، displaying the two divs would make the word appear broken, "ال لغة" instead of "اللغة". Which makes me wonder if this can be done in Arabic given the way way eccojs is built (the separate divs are needed to allow coloring them with highlighters (for saliency and activation views).

A solution might be to add the "ـ" character to turn "ال" to "الـ" when we detect it should link up with the next token. But this seems to greatly expand the problem scope.

Btw, how common are subword tokens in the Arabic tokenizers? I haven't experimented with them enough yet. If they're mostly whole words, this can be a smaller issue than indicated above.

[WissamAntoun]

I think solution 1 is enough since the text will be at least readable. As for the subword, I say they occur in 1-2% of the tokens which can be ignored for now.

Also, I noticed that some characters aren't being printed correctly. check this notebook for the example https://colab.research.google.com/drive/1X6liDXWoP2uDtPRyCv-dn4BpW086G6FH?usp=sharing

[jalammar]

I have created an issue in eccojs to track this issue: jalammar/eccojs#2

The character issue I suspect is related to BPE. "لبلورة" becomes the two tokens "لب�" "�ورة" as BPE split that word halfway through the bytes that make up that center لام.

I found an issue discussing this: huggingface/tokenizers#508