Give us the power of code-first-i18n

[earloc]

introduction

Inspired by the (at time of writing this) still ongoing discussion over at #70 (and partially #38) I´m currently starting to think about code-first-i18n - similar to what code-first-databases are offering to us.
I´ll try to sketch out the idea emerging in my mind.

When it comes to translating apps, the first thing a dev needs to do is place some simple string ("Hello App") or templated-string ("Hello {0}") somewhere within the app.

I´ll refer to those as localizables

In .net-land this probably would be utilizing plain old resx-files (or resw in UWP, etc.).

There are other options, indeed, like OrchardCore.Localization for po-files (which I currently really need to explore more) or other offerings from - mostly paid - third parties (like poeditor or localizely, just to name a few).

As I understand currently, all those offerings mostly are based on the fact that there IS already some sort of tech-stack native localization file (in .net-land some resx- or maybe an xliff-file), which is used as the point of interchange.

Getting to those files currently isn´t very intuitive or "developer-friendly", at least from my personal point-of-view.

So, what am I proposing?

code-first i18n

From a developer´s point-of-view, all I really want is some sort if Interface I can inject and leverage existing IDE refactorings to introduce new localizables.

Why isn´t introducing a new localizable as easy as writing a new method (inlcuding parameters, if needed) and hitting CTRL+. and selecting Generate method:

nice idea, but how does this help?

Exactly, this doesn´t really help now, as there´s nowhere some usable implementation around.
This is where TypealizR could kick in. By marking the Interface with something like [Typealize], a source-generator could emit a discrete implementation, as well as some sort of actual default exchange-file for storing actual localizables. This could be an ordinary resx-file OR a xliff / po, whatever. What the actual implementation really does, is currently not specified. Could be looking up from some file, or just honoring the requested Culture and deliver a static string - doesn´t really matter.

Inspired by what MAT does, this could then be used to generate even more files intended for interchange with translators or tools like poeditor / localizely / whatever.
There´d need to be - of course - some mechanism to re-import such an exchange-file, to support workflows involving external tools / translation-providers. But this looks kind-of solvable, atm. Even if that would involve taking translatables back and incorporating those into the static strings, which might be returned by generated code.

Any opinions?

[earloc]

Here´s a possible "workflow", at least how I imagine code-first-i18n to work, atm.

the yellow spots are just for reference, where other toolings could pick up and be incorporated in the overall flow.

1. A develper authors a plain-old Interface with the editor of choice. (This might need to be marked with an appropriate Attribute)

[CodeFirstLocalizable]
interface ILocalizables
{
    ///<summary>
    /// Hello '<paramref name="world"/>'
    ///</summary>
    /// <param name="world">the name of the world to be greeted</param>
    /// <returns></returns>
    LocalizedString Hello(string world);
}

2. A source-generator analyzes said interface
3. The source-generator generates a class which implements the Interface. It may use the xml-comment to populate a meaningful default translation, otherwise a generated string could be deduced from the member´s signature (properties or methods). Implementation details do not matter, atm. Could be generating a hard-coded string, or falling back to e.g. IStringLocalizer<T>...dunno, yet.

partial class Localizables : ILocalizables
{
    public LocalizedString Hello(string world) => ...;
}

4. A yet-to-be developed CLI kicks in and reads the generated classes. Either via out-of-process reflection or roslyn.
5. The CLI synchronizes all strings into plain-old Resx-files (Other file-formats are definetly thinkable)., possibly placing some breadcrumbs along the way to ease up sync on subsequent invocations. We now have a default file that can be modified as needed.
6. Tools like MAT or whatever could enhance the overall process by doing their thing(s). For MAT, this is to generate an XLIFF-file out of all Resx-files within a project and assembling those into locale specific XLIFF-files.
7. Those files are then used to populate locale specific Resx-files
8. MAT further synchronizes things into existing XLIF-files, which then can get translated by the developer or a third party (tool).
9. The changes get reflected within the locale specific Resx-files

[MarkPflug]

I'm still trying to wrap my head around your proposal here, so I don't have much to comment about it yet. However, I thought I'd point out a similar effort that I've worked on in the past: https://github.com/MarkPflug/Sylvan.BuildTools.Resources. It is a package that provides localization support very similar to what .resx files offer, but rather than using XML, it uses JSON (.resj) files. The project readme outlines a few of the advantages over resx, if you're interested.

My package uses build-time code generation (an MSBuild task), rather than compile-time code gen (source generator) as your project uses. I don't see a huge advantage to one over the other. The one limitation I found with source generators was that it isn't (yet) possible to add embedded resources to the compilation. My project only came about as a proof-of-concept, for a larger project I was working on professionally, so I haven't really been adding much to it recently.

Resx uses a combination of design-time (Visual Studio custom tool) for code generation and build-time (MSBuild task) for creating the embedded resources. I was trying to find a way to do something similar that didn't require Visual Studio, so other developers could use VSCode and still be able to manage the resj files.

One idea from your project that I really like is the strongly-typed formatter method design. My project, much like RESX, only produces localized strings, so if a localized string is a format string, it is up to the consumer to apply the correct formatting, which wouldn't have any compile time checks. string.Format(MyResources.GreetingFormat, username), for example. That code doesn't have any checks that the correct number/types of arguments is supplied for the format string. It would be a nice enhancement to also generate a formatter method on the resource accessor class so I could also write: MyResources.Greeting(username), which could provide compile-time checks that the arguments were correct.

It looks like your project is mostly targeted at the new IStringLocalizer, which I'm only passingly familiar with from ASP.NET examples I've seen. My understanding is that it is meant to provide an abstraction on top of a concrete implementation such as .RESX. This makes me wonder if your projects might provide a layer on top of what I've created.

Anyway, just wanted to say hi. I'll be keeping an eye on your project's progress.

Cheers,
-Mark

[earloc]

Hey @MarkPflug .

Checkout #88, I think with this in place we're not far away to also incorporate your resj format. Just need to think about how: hardcoded or via some kind of plugin-model , maybe.

[earloc]

Tracking work in #95

[earloc]

@MarkPflug maybe this helps with head-wrapping / grasping the idea behind code-fist-resources ;)
https://github.com/earloc/TypealizR/blob/main/docs/code-first.md

[MarkPflug]

Thanks for the reminder. I need to find some time to dig into this.

[earloc]

Just dropping in this here, so I'll later find it for reference:

https://medium.com/younited-tech-blog/json-based-blazor-localization-395b1b31e8ae

[ivanhofer]

Hi, I'm Ivan and I have done something similar to TypealizR in the JavaScript ecosystem: https://github.com/ivanhofer/typesafe-i18n

i18n is a pain for developers. Making it as effortless as possible is certainly a goal tooling should strive for. In my opinion the main problem is that the place where you call the translation function is not the place where your strings are stored (key-based translations). If you need to add a new string to the application, you need to come up with a new key, write the string into a dictionary-file and then call the translation function inside the code.

If I understand everything correctly, you want to eliminate the step of requiring a developer to manually write something into the dictionary file. He does not need to leave the file where he wants to call the translation function. I like that approach as it eliminates a pain-point when dealing with i18n.

I'm currently working on a draft of a concept to make i18n more accessible to web developers: https://github.com/pipeli18ne/RFC
The goal is that you just annotate code parts that you want to be internationalized, and tooling does the rest for you. No need to manually handle dictionary files, no need to configure it for a certain framework etc.. Everything could be automated to make it as effortless as possible to use.

[ivanhofer]

I'll take a look at this when I find some time :)

[ivanhofer]

I did read it. I have to admit that I don't know anything about .NET (even if it was my first language/platform when I started programming😅).

Overall it looks good to me. The instructions seems easy to follow. Where I had troubles is where you mention structured xml comments. In the example above that point, you mark that codeFirst.Hello outputs "Hello". Is this auto generated from the interface? For me it is not really clear where this comes from.

And where you show the xml comments, why does Greet have a <returns></returns> at the end but Hello not?
You are also using paramref and param there. Is param needed or could it also work if you just define paramref?

I also don't know anything about *.resx files, but why are the parameters 0 and 1 and not userName and planetName like in the interface?

[earloc]

Thx @ivanhofer, I really appreciate your time and effort on this. Thank you so much.

I revised the documenation a bit to clarify / simplify things over at #105.

In the example above that point, you mark that codeFirst.Hello outputs "Hello". Is this auto generated from the interface?

Yes, exaclty. That´s one of the main points where the generator kicks in and let you just start with a naked, un-customized localizable, to not break dev-flow.

And where you show the xml comments, why does Greet have a at the end but Hello not?
You are also using paramref and param there. Is param needed or could it also work if you just define paramref?

Those are just left overs from what you get within Visual Studio once you type ///. The sample will also work exactly the same without those paramref and return-elements.
See xmlcomments, which is an editor-assisted way to specify code-comments since .nets-initial incarnation back in 2002, I guess. Very similar to what e.g. jsdoc do.

I also don't know anything about *.resx files, but why are the parameters 0 and 1 and not userName and planetName like in the interface?

This is because, internally, the implementation ultimately utilizes a call to string.Format, which happens to have no support for named paramaters, only positional ones. That´s just how it works, sadly ;( . But I´m thinking of a way to let users customize even this, similar to what the resource-first- approach currently offers, see #39.

[ivanhofer]

I revised the documenation a bit to clarify / simplify things over at #105.

Great!

and let you just start with a naked, un-customized localizable, to not break dev-flow.

Nice detail!

The sample will also work exactly the same without those paramref and return-elements.

Maybe you could mention this somewhere.

This is because, internally, the implementation ultimately utilizes a call to string.Format, which happens to have no support for named paramaters, only positional ones

This may lead to problems when you have multiple parameters in a message and the position of those change in a translation. Named parameters solve this issue, but I guess you already know that.

[earloc]

Gnaah, I really meant to say index-based, rather than positional 😉. And that's only true for the values itself. So it's absolutely ok, if the order between different languages might change during the translations Process. The index gets fixed based on the positional order arguments get passend down to that internal call. And this static code won't change during translation at all - and when, then only on purpose. 😇

But yeah, having just those indexes at hand really kind of sucks a little bit 😉.

[earloc]

Hey ya´ll.

I recently made some huge steps making this happen, first MVP is about to land soon. See #88.

[earloc]

https://github.com/earloc/TypealizR/releases/tag/v0.9.2 just landed.

See also https://www.reddit.com/r/dotnet/comments/10ufygg/introducing_codefirsti18n_internationalization/?utm_source=share&utm_medium=web2x&context=3

[samuelstroschein]

Inspired by what MAT does, this could then be used to generate even more files intended for interchange with translators or tools like poeditor / localizely / whatever.
There´d need to be - of course - some mechanism to re-import such an exchange-file, to support workflows involving external tools / translation-providers.

I introduce https://github.com/inlang/inlang. One of the apps lets translators manage translations directly on a git repo without the need for re-import/sync.

There seems to be a general overlap in the goal of TypealizR and inlang: make i18n easier, especially for developers.

[earloc]

Hey, cool stuff there. Will definetly check out how this might suite in the overall vision. Thx for spreading the word! ✌️