Overhauling Northstar mod system in 2023

[GeckoEidechse]

Problem

When Northstar was first publicly released we had no mod distribution platform or the like. Further the mod system in Northstar is derived from Icepick which uses a mod.json to define various aspects of a mod.

Now, 1.5 years after initial release Northstar has largely transitioned to Thunderstore as the sole mod provider which has it's own manifest.json to define mod author etc.

Further we recently introduced wider support for plugins in v1.13 of Northstar which are in a separate folder compared to mods.

Basically a typically Northstar setup looks like this:

R2Northstar/
├── enabledmods.json  <------------------- Defines which mods are enabled
├── logs/   <----------------------------- log folder, written to and created by Northstar
├── mods/
│   ├── some-random-mod/  <--------------- 3rd party mod installed by user manually or mod-manager
│   │   ├── manifest.json  <-------------- Thunderstore manifest added by mod-manager
│   │   ├── mod/    <--------------------- contains Squirrel scripts that are part of mod
│   │   ├── mod.json    <----------------  defines mod for Northstar
│   │   └── thunderstore_author.txt   <--- added by mod-manager
│   ├── Northstar.Client/
│   │   ├── mod/
│   │   └── mod.json
│   ├── Northstar.Custom/
│   └── Northstar.CustomServers/
└── plugins/
    ├── cat_or_not-kyurid-1.0.0/  <------- plugin installed by mod-manager, exact format still tbd
    │   ├── kyurid.dll   <---------------- plugin
    │   └── manifest.json  <-------------- Thunderstore manifest added by mod-manager
    └── DiscordRPC.dll   <---------------- plugin shipped with Northstar

With this we are now in a bit of an awkward situation where mod-managers need to make some weird dance to extract mods and plugins into the right location.

Verified mods

Then there's also the idea of "verified mods" which allows Northstar to auto-download mods required a by a server (e.g. gamemodes or maps) upon joining said server.

For more info on them see:

• https://github.com/R2Northstar/VerifiedMods
• feat: Verified mods auto-downloading NorthstarLauncher#362
• feat: Verified mods auto-downloading NorthstarMods#542

Discussion

Overall, things are starting to get really messy so we probably wanna consider re-structuring our mod system.

This is what this issue is about.

Considerations

Some things to consider are:

• We have mountains of old documentation out on YouTube and various webpages on how to install mods for Northstar which will take ages to replace. As such we might want to consider supporting "the old system" for manual install either temporarily or permanently.
• Manual install doesn't delete files so unless we transition to 100% mod-manager we will never be able to move R2Northstar.XXXX mods to a different location...
• r2modman is the only mod-manager not controlled by us and also has to support other games as such will take longer to update.

[GeckoEidechse]

Naturally given our reliance on Thunderstore nowadays, we should use it as a basis for a new system.

A "simple" solution would be to enable a second mods folder, which could be called something like thunderstore-mods. In it one would simply place the extracted Thunderstore zip

R2Northstar/
├── enabledmods.json
├── logs/
├── mods/
│   └── some-random-mod/  <--------------- some random mod manually installed by user that cannot be moved by mod-manager
│   │   ├── mod/    <--------------------- contains Squirrel scripts that are part of mod
│   │   ├── mod.json    <----------------  defines mod for Northstar
│   │   └── thunderstore_author.txt   <--- added by mod-manager
│   ├── Northstar.Client/
│   │   ├── mod/
│   │   └── mod.json
│   ├── Northstar.Custom/
│   └── Northstar.CustomServers/
│
├── plugins/
│   └── DiscordRPC.dll
│
└── thunderstore-mods/  <-------------------- this new
    ├── ts_author1-ts_name1-4.2.0  <--------- installed by mod manager
    │   └── mods/
    │       ├── some-random-mod/  <---------- Squirrel mod
    │       │   ├── mod/    
    │       │   └── mod.json
    │       └── some-other-random-mod/  <---- Another squirrel mod
    │           ├── mod/
    │           └── mod.json
    │
    └── ts_author2-ts_name2-0.6.9
        ├── mods/
        │   └── some-random-mod/  <---------- Squirrel mod
        │       ├── mod/    
        │       └── mod.json
        └── plugins/
            └── kyurid.dll   <--------------- plugin

Of course this solution isn't without caveats... Like what happens if two Thunderstore mods contain the same Northstar mod? Or if a Northstar mod is both installed via mod-manager and manually.

[pg9182]

I agree that we should have a separate thunderstore dir which preserves their structure.

Regarding the transition, we should transition to a completely new structure which separates core mods, doesn't include default config, has a separate dir for user config, and has a separate dir for thunderstore/custom/plugins/etc. We can continue loading stuff from the old path, but show a warning, and always override it with stuff from the new layout. This will allow us to solve the configuration and launcher issue all in one go, while not breaking existing installations/tutorials. I'm thinking of something like this:

R2Northstar:
  *old-stuff
ns:
  core:
    plugins:
    mods:
      Northstar.Client:
      Northstar.Custom:
      Northstar.CustomServers:
  profiles: # you would be able to specify a profile name or full path on the cli
    default: # would be created automatically on first launch if a profile isn't explicitly specified
      settings.json
      banlist.txt
      cfg:
        config_1.cfg
        config_2.cfg
      mods: # if it is named the same as a core mod, it overrides it with a warning
        Example.Example:
      thunderstore:
        ts_author1-ts_name1-4.2.0:
      plugins:

Also, I don't really like enabledmods.json, but I'm not sure what a better solution would be yet. At the minimum, it would be nice to have some control from the CLI arguments.

[GeckoEidechse]

Uh two issues with this. Using ns as a folder name isn't really descriptive but I'm also aware that naming isn't set in stone :p

More importantly however, don't think making a profiles subdir makes sense. Profiles encompass the entire R2Northstar folder and can be placed anywhere on the user filesystem and with R2Northstar/NorthstarLauncher#451 they basically contain everything except NorthstarLauncher.exe.

[pg9182]

Profiles encompass the entire R2Northstar folder

I'd like to separate core mods from the rest. We could allow specifying a different core dir too.

[ASpoonPlaysGames]

I'd like to separate core mods from the rest. We could allow specifying a different core dir too.

I agree, to an extent. They should not be separate from a modloading perspective, other than the warning when disabling Northstar.Client or Northstar.CustomServers. Technically speaking Northstar.Custom isn't really a "core mod" in the same way as the other two, it can be disabled and re-enabled without issue and is designed to be optional

[pg9182]

By "core mod", I mean anything which we ship and should be overwritten entirely on updates.

[GeckoEidechse]

Tbf, every mod should be fully overwritten when updating that mod :P

[F1F7Y]

I'm very conflicted on a few things here, but here's my thoughts:

1. No backwards compatibility
   Doing this will introduce code complexity with no real value. We could do a warning like Detected a mod %s with old scheme at most.
2. Enforce plugin structure
   With #460 plugins can be loaded from subdirs. This is bad imo and we should do a mod-like structure, possibly moving the manifest outside of the dll ( This imo is a separate topic of its own ).
3. Mod dependencies should have a toggle to not load the mod if dependencies are missing, which we should visualize in-game.

When it comes to the structure i think something like this would work best:

R2Northstar/
├── mods/ ( End-User installed mods )
│   ├── Fifty.SimplifiedMenu
│   ├── Northstar.ArcCannon (Northstar.Custom separated into individual mods)
│   └── Northstar.GameModes
├── remote_mods/ ( Auto-download mods )
│   └── Fifty.Blockout/
│       ├── mod.vdf
│       └── scripts/
│           └── vscripts/
│               └── scripts.rson
├── mods.vdf ( enabledmods.json equivalent )
├── scripts/...
├── resource/...
├── ...
        ( profile dir works the same as `platform` folder in source, basically what the `mod` folder in a mod is.
          We'd have all retail scripts here commited as Respawn so one could git blame, Would also fix
          PRs waiting on vanilla file being commited. This would be Northstar.Client and Northstar.CustomServers)

I'm very much in favor of using engine functions where ever possible ( hence the switch to vdf ). Having a similar system to r5r will also make it easier to have multi-game mods / to mod for both games.

This is very much inspired by the unfinished mod system r5reloaded has on their indev branch

There are quite a few things i havent mentioned such as using squirrel definition constants to distinguish between running northstar and vanilla in script.

We should probably make a thread on discord to discuss this ,as github issues are unreadable after a hefty discussion, and post only agreed up on things here to keep track of things.

[GeckoEidechse]

1. No backwards compatibility
   Doing this will introduce code complexity with no real value. We could do a warning like Detected a mod %s with old scheme at most.

100% agree on slashing backwards compatibility. Mod-managers can be used to "transform" old structure to new one. Though the value in backwards compatibility is that we don't break users installs. If a mod-manager can apply a fix, then that's fine.

2. Enforce plugin structure
   With #460 plugins can be loaded from subdirs. This is bad imo and we should do a mod-like structure, possibly moving the manifest outside of the dll ( This imo is a separate topic of its own ).

Also agreed. The allowing plugins to be loaded from subdirs was done to somehow map plugins to Thunderstore mods which hopefully the new system won't require us to do anymore.

Keeping manifest in plugin is kinda nice though cause it makes everything self contained.

3. Mod dependencies should have a toggle to not load the mod if dependencies are missing, which we should visualize in-game.

This means we need to parse manifest.json in Northstar which is doable but now this is where things start to get tricky. In particular, the main is issue is the difference between Thunderstore mods and Northstar mods. We will basically always have to support the Northstar mods layout as otherwise we are breaking 100% of mods on Thunderstore right now.

So right now we have Northstar mods packaged inside Thunderstore mods and in order to auto-update a mod we need to map back the Northstar mod to said Thunderstore mods. So we need some kind of indicator of linkage.

We could also abandon the idea of a Northstar mod entirely but again this will require updating 100% of Thunderstore as well as every piece of tooling we have in place. We can certainly try to do this but then we need some (temporary) backwards compatibility as otherwise we are breaking every mod in existence x_x

When it comes to the structure i think something like this would work best:

R2Northstar/
├── mods/ ( End-User installed mods )
│   ├── Fifty.SimplifiedMenu
│   ├── Northstar.ArcCannon (Northstar.Custom separated into individual mods)
│   └── Northstar.GameModes
├── remote_mods/ ( Auto-download mods )
│   └── Fifty.Blockout/
│       ├── mod.vdf
│       └── scripts/
│           └── vscripts/
│               └── scripts.rson
├── mods.vdf ( enabledmods.json equivalent )
├── scripts/...
├── resource/...
├── ...
        ( profile dir works the same as `platform` folder in source, basically what the `mod` folder in a mod is.
          We'd have all retail scripts here commited as Respawn so one could git blame, Would also fix
          PRs waiting on vanilla file being commited. This would be Northstar.Client and Northstar.CustomServers)

I'm very much in favor of using engine functions where ever possible ( hence the switch to vdf ).

This is a noble idea but I'd rather not switch from JSON to VDF as the latter is rather poorly defined and lacks library support by most languages which is a bit of an issue for mod-managers ^^

Having a similar system to r5r will also make it easier to have multi-game mods / to mod for both games.

I'm not sure if the two games are compatible enough to allow this tbh. If they are, I'm not opposed to this ^^

This is very much inspired by the unfinished mod system r5reloaded has on their indev branch

There are quite a few things i havent mentioned such as using squirrel definition constants to distinguish between running northstar and vanilla in script.

👀

We should probably make a thread on discord to discuss this ,as github issues are unreadable after a hefty discussion, and post only agreed up on things here to keep track of things.

Best to make a thread in #research then ping everyone in this discussion so far. Although updates should be posted to this thread as Discord is an information blackhole

[GeckoEidechse]

Enforce plugin structure

Should be covered with packages dir, the removal of subdir loading, and the introduction of lib subdir R2Northstar/NorthstarLauncher#590

[GeckoEidechse]

superseded by #618