Replacing a custom scripting language

[np-c0mp1ete]

Reasons to replace it:

• custom interpreter is slow
• custom interpreter adds 5k+ LOC to maintain, not counting other scripts-related code like an in-game inspector/debugger

Requirements for the successful transition:

• transition should be as seamless as possible, so tools for automated scripts conversion should be provided.
• games should stay easily moddable

[q4a]

I don't understand, why are you trying to replace working scripts? Do you know that Lua is slow and guys who used it tried to replace it:
https://github.com/OpenXRay/xray-16 - wanted to keep old Lua scripts support and used with LuaJIT for that, but still has problems. What is devs says:

Russian:

**Прапорщик**:
LuaJIT сам по себе быстрый. Сборка мусора в нём - вот где беда...
Я как-то пробовал собрать с GC64 и отключил сборку мусора, так пока память не закончилась, всё было офигенно плавно - по Юпитеру прошел из конца в конец безо всяких микрофризов.
**Султан**:
Есть идея, которая мне не очень нравится, но тем не менее: раз в определённое количество времени останавливать всю работу со скриптами, но при этом рендерить кадр и пока он рендерится – всё это свободное время отдать на сборку мусора.
Т.е. выделить чисто один кадр раз в сколько-то кадров в котором рендер будет работать так же как обычно, но логика обновляться не будет, а только мусор собираться.
Но, вообще, движок бы распоточить сначала, наверное...
**Прапорщик**:
Вот  только его не распоточишь из-за LuaJIT =)

English:

**Ensign**:
LuaJIT itself is fast. Garbage collection in it - where the trouble is ...
I once tried to collect from GC64 and turned off garbage collection, so until the memory ran out, everything was awesome smoothly - I went through Jupiter from end to end without any microfreezes.
**Sultan**:
There is an idea that I don't really like, but nevertheless: once in a certain amount of time, stop all work with scripts, but at the same time render the frame and while it is being rendered - spend all this free time on garbage collection.
Those. select purely one frame every few frames in which the render will work as usual, but the logic will not be updated, but only garbage will be collected.
But, in general, the engine would have to add multithreading first, probably ...
**Ensign**:
But you can't add multithreading because of LuaJIT =)

Stalker engine was available since April 2014, from that time at least 3 teams replace it:

Russian:

**q4a**:
Всем привет. Мне кажется был проект, где пробовали заменить Lua на какой-то другой язык. Подскажите, как проект называется?
(PS я знаю, что в OpenXRay от Lua не откажутся, просто хочу иметь пример того, почему не стоит переходить на Lua в новых проектах + посмотреть далеко ли в этом продвинулись)
**Collector**:
Привет
Рох переходит на c#
oxygen переходит(перешел) на c#
Кордис переходит на c++

English + links:

**q4a**:
Hello everyone. It seems to me there was a project where they tried to replace Lua with some other language. Tell me, what is the name of the project?
(PS I know that Lua will not be abandoned in OpenXRay, I just want to have an example of why you should not switch to Lua in new projects + see how far they have progressed in this)
**Collector**:
Hello
https://roh-online.com/ replacing it with c#
https://github.com/DJYar/xray-oxygen - replacing (replaced) it with c#
https://github.com/diamondhat/cordisproject - replacing it with с++

I helped to add Linux and ARM support to OpenXRay and you can join to it channel and ask directly about cons of using Lua: https://discord.gg/sjRMQwv

[sharkydima]

In one of self projects i wrote a backend scripts logic on C#. I uses C++/CLI wrapper for native C++ core, and this wrapper has a good perfomance. But that actual only for Windows platform. Many game engines (like Unity or NeoAxis) using C# as a primary language for developing with embedded MONO framework.

[Yakvi]

Scripting language could use API cleanup, sure. That said, I'm not in favor of replacing it entirely for several reasons. A small write-up of my thoughts is below.

First, you will need to re-implement all the legacy API. Remember that, as an open and established game engine, we rely on the old stuff working exactly the way it used to, no less no more. Otherwise this could kill/split the modding community.

Second, as the saying goes, if something's not broken, don't fix it. This engine has a lot of stuff that could use fixing or bringing up to date. Scripting language in the game right now has some parts that are broken, inconsistent or otherwise undesirable, yes. These should be addressed individually. But overall? It's doing its job as it has done for years. Instead, a focused effort should be done at providing a proper toolkit to support it: scripting language debugger and/or profiler, fixing old API, a new API (without breaking the old one!).

On a more personal note, the Storm's engine runtime compiler was what fascinating me the most. It always felt to me like the magical part that made everything come together, it allowed the modding scene to... exist, and preserving that is something that feels the closest to my heart.

Last, I feel that replacing things with new and shiny just for the sake of it is just wrong. Such an implementation will inevitably run into unexpected edge cases, complications, performance loss... you name it. As such, the project might be abandoned midway and the effort that could be used elsewhere would be wasted. I'm a strong opponent of the idea that just because a thing is new, it's better than the old one. There should be more to it. I already cited Jon Blow earlier, but engine devs in major AAA studios would say you the same. Jason Gregory says in his Game Engine Architecture book: "Just because a [cool new C++] feature exists doesn't mean your team needs to immediately start making use of it. At Naughty Dog, we tend to take a conservative approach to adopting new language features. [...] We have a list of those C++ features that are approved for use in our runtime code, and another somewhat more liberal list of language features that are allowed in [dev] tools code".

So yeah, these are my 5 cents. I don't have years of experience in this or another language, but I like listening to other people and trying to understand their point of view. Gamedev is quite a beast, different from anything you might see elsewhere.

[np-c0mp1ete]

I know 0 A.D. uses JS VM for scripting. Here's an interesting read regarding their decision-making process for choosing it over lua.

[espkk]

There are the main points descending by significance:

1. existing scripts are EXTREMELY slow
2. they are not standardized -> it's hard to maintain compiler/VM behind SE scripts, it cannot be replaced by some random clang
3. Object-Oriented scripting could be used since the engine is already using OO

But keep in mind that:

1. This is definitely not a priority task. It will require big efforts
2. We can easily keep a legacy script processor. We can even combine them in order to rewrite the code gradually yet having the whole logic working