Make Settings Pydantic and use the power of BaseSettings to simplify CLI

[stan-dot]

Fixes #495

Using Pydantic for logic to do with file loading. It has a bit of magic but will lower our lines of code. ApplicationConfig will extend BaseSettings, while the nested configs like ScratchConfig are still BlueapiBaseModels.

[stan-dot]

File "/workspaces/blueapi/src/blueapi/service/interface.py", line 21, in <module>
INTERNALERROR>     _CONFIG: ApplicationConfig = ApplicationConfig()

now it's the time when the use of global constants bites us

[stan-dot]

@callumforrester why do we get another instance of ApplicationConfig inside the interface.py?

"""This module provides interface between web application and underlying Bluesky
context and worker"""


_CONFIG: ApplicationConfig = ApplicationConfig()


def config() -> ApplicationConfig:
    return _CONFIG


def set_config(new_config: ApplicationConfig):
    global _CONFIG

    _CONFIG = new_config

[callumforrester]

@stan-dot that's for the subprocess, you need an instance in the main process, which is passed down to the subprocess via set_config. I'm not sure why it defaults to the default config rather than None, though.

[stan-dot]

I think that the global keyword made it more confusing to me. Would SUBPROCESS_CONFIG be a correct name?

why would we want it to be None? Pydantic expects it to not be None.

I haven't worked with python subprocesses before and not sure how to debug successfully

[DiamondJoseph]

Looks like settings_customise_sources in the docs?

[DiamondJoseph]

class Settings(BaseSettings):
    foobar: str
    nested: Nested
    model_config = SettingsConfigDict(toml_file='config.toml')

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: Type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> Tuple[PydanticBaseSettingsSource, ...]:
        return (TomlConfigSettingsSource(settings_cls),)

Looking at how a toml file is used by an equivalent

And

settings_customise_sources takes four callables as arguments and returns any number of callables as a tuple. In turn these callables are called to build the inputs to the fields of the settings class.

I think you need the signature to exactly match what is shown?

[stan-dot]

yeah, it needs to be an override. Not sure why did this got changes

[callumforrester]

why would we want it to be None? Pydantic expects it to not be None.

So currently the following sequence of events happens:

1. Subprocess spawned by main process
2. config set to default config, the result of calling ApplicationConfig()
3. Main process calls set_config(), overriding the default config with whatever was in the config file

We could instead make it None at step 2 (and yes, change the type annotation), since whatever it is gets immediately erased anyway.

[stan-dot]

@callumforrester do main and subprocess share a global variable? you didn't mention where does step 2 happen - whether in main process or the subprocess

[callumforrester]

@stan-dot No they can't share a global variable. The main process passes the idea into the subprocess via set_config

[stan-dot]

The main process passes the idea into the subprocess via set_config

I thought usually it's values or references passed, not ideas...

[callumforrester]

Can you tell I haven't finished my morning coffee yet? Honestly no idea what I intended to type there, "config", maybe?

[stan-dot]

no, I cannot tell that. I'd appreciate concise technical terms used

[stan-dot]

very odd errors, trying to fix now

[stan-dot]

trying to refactor the interface to make it testable more easily.

@callumforrester it looks like unlike the patterns I can see in the codebase for worker and rest api the interface was designed to work as a callable module holding its own state, not as a class in its own.

It makes me feel confused, where can I read the ADR for this?

[stan-dot]

huh, I'd need to refactor main as well

[callumforrester]

Could: Can we remove this now?

[callumforrester]

Should: I'm concerned this. mutation of global state is an antipattern. Could you instead subclass the config?

Suggested change

	ApplicationConfig.model_config["yaml_file"] = config
	app_config = ApplicationConfig() # Instantiates with customized sources
	class YamlConfig(ApplicationConfig):
	model_config = {**ApplicationConfig.model_config, **{"yaml_file": config}}
	app_config = YamlConfig()

[stan-dot]

that's how Pydantic uses it and this is not regular state but settings

also this is not Java

[callumforrester]

Mutation of global state is an antipattern in any programming language (good thread).

Now I have to remember to address this if I use ApplicationConfig somewhere else, just like you have:

ApplicationConfig.model_config["yaml_file"] = None

There is nothing that makes this obvious to me as a developer.

[stan-dot]

looking into the base settings docs now

re: mutation - it was there anyway and this config is just the init, and anywhere else you'd sooner use configManager getter methods, should you need access to the config.

[stan-dot]

the mutable settings option is here:
and we're not using it
https://docs.pydantic.dev/latest/concepts/pydantic_settings/#removing-sources

[callumforrester]

It looks like what we're trying to do was just not a well-envisioned use case: pydantic/pydantic-settings#346

Should maybe reconsider the use of BaseSettings and/or raise a PR upstream?

[stan-dot]

that is true, turns out that pydantic-settings isn't as well-finished product like pydantic itself

[callumforrester]

Should: Again, I think you should find an alternative to mutating global state, if you fix the instance in CLI then you shouldn't need this line.