Mounting extra files to ArchiveClasses & BspClasses loaded from archives

[snake-biscuits]

From #191:
We should also look at some standard method for linking related files

sega.Gdi, respawn.RPak, respawn.Vpk & valve.Vpk could all use this too
since their .read() methods can't function without grabbing data from other files

@classmethod
def from_archive(cls, archive: base.Archive, filename: str) -> Union[ArchiveClass, Bsp]:
    out = cls.from_bytes(archive.read(filename))
    for related_filename in cls.related_filenames(archive.namelist()):
        out.link(archive, related_filename)
    return out

Places where this'd be useful

ArchiveClasses

• alcohol.Mds
• golden_hawk.Cue
• respawn.RPak
• respawn.Vpk
• sega.Gdi
• valve.Vpk

BspClasses

• ValveBsp
• RespawnBsp (respawn.ExternalLumpManager)

NOTE: all BspClasses could use a .related_filenames() method
-- generate a list of files that could be mounted (external lumps, navmeshes etc.)
-- shouldn't include material & script dependencies, those require more advanced parsing

Related

• Split related files handler out of BspClass.from_file #156

[snake-biscuits]

really, this has 2 parts
The listed BspClasses have external files we parse
But .link can be in base.Bsp

.from_archive will likely be per class

[snake-biscuits]

started looking at this now
archives.base.Archive has a .from_archive @classmethod __init__

planning to add .mount & .unmount methods to base.Archive to link "external files"
these can provide some kind of file handles to keep the data present

.unmount will be to reduce memory usage
unmounting a file that was added automatically will probably be really painful

[snake-biscuits]

implementing .mount for BspClasses will likely be more complex
since RespawnBsp has ExternalLumpManager to try and cache files

we might need to keep the archive open to dynamically mount .bsp_lump

bsp.mount(archive, filename) makes sense
but I'd also like to mount from files, streams & bytes

might be a while until I land on an implementation

[snake-biscuits]

.link can be in base.Bsp

.from_archive will likely be per class

.from_archive has been implemented in base.Bsp
If we have a .related_filenames() function we can use that to automatically check for & mount files
This should probably be optional & off by default to reduce memory usage

Some RespawnBsp (Apex Season 10 onwards) will need external .bsp_lump to be useful though

We already have .mount_lump methods, so I'm thinking .mount_file should work as a name
These should do the job:

def mount_file(self, filename: str, archive=None):
    if archive is None:
        self.external_files[filename] = open(filename, "rb")
    else:
        self.external_files[filename] = io.BytesIO(archive.read(filename))

def unmount_file(self, filename: str):
    self.external_files.pop(filename)

Afaik all external files would be binary
If not we can always add plaintext=False to .mount_file

Could be useful to have some kind of .stream method in ArchiveClasses to reduce memory usage
Likely a lot slower and would have to be tailored to each individual ArchiveClass
Maybe keep it in mind for if we come back to optimise this system

[snake-biscuits]

particle manifests would be a plaintext extra file for ValveBsp
though as discussed in #156, we're going to focus on mounting lump data
(extra lighting information & external lumps)

[snake-biscuits]

.related_filenames isn't going to work

.lmp filenames include version numbers¹
we should probably use fnmatch patterns instead
this could be used with archive.namelist() or os.listdir()
though we'll need some path splitting solution for working out mod-relative paths

path_tuple in archives.base could be useful
probably handy for autodetect.naps too
maybe we add a new filesystem module?

Footnotes

1. Valve Developer Community: Pathing levels with lump files ↩

[snake-biscuits]

we could extract files we want to mount inside archives
if we use temporary files we'd be reducing ram usage
load time would only increase by the write time
we have to get every byte of the file from the archive either way

until we work out a system for streaming assets in archives anyway
implementations of streaming would be unique to each ArchiveClass
io.BytesIO(archive.read(filename)) might work as a default

[snake-biscuits]

extras seems like a good name to reference "external files"
helps keep variable names short
(mount_extras vs. mount_external_files)

totally didn't get the idea from the Guilty Gear soundtrack playing in another tab

[snake-biscuits]

ArchiveClasses & BspClasses now automatically mount extras

each specific class which uses external files now needs to access them via self.extras
they also need class-specific .extra_patterns implementations

[snake-biscuits]

We need archives we can use to actually test file mounting
I currently don't have any .bsps inside archives w/ supplementary files

Might have to make an id_software.Pak w/ .lit & .vis files
Can't use respawn.Vpk because we don't have compression yet

Tho I guess for ArchiveClasses I could test a sega.Gdi inside a pkware.Zip

[snake-biscuits]

Could add a raw bytedata arg to .mount_file for testing

def mount_file(self, filename: str, archive=None, raw_data: bytes = None):
    if raw_data is not None:
        self.extras[filename] = io.BytesIO(raw_data)
    else:
        ...  # load from file or archive as before

I'm sure this could also be useful outside of testing

[snake-biscuits]

ArchiveClasses need to start using filename & folder attributes
This will lets us generate extra_patterns & scan the local folder for attached files

[snake-biscuits]

Not every ArchiveClass needs to track folder & filename
.from_archive & .from_file handle that for us anyway

[snake-biscuits]

got sega.Gdi implemented
but to test it, I tried updating sega.GDRom to read the data
unfortunately the .gdis I'm using don't use a single track for the GD-ROM Area

base.Archive is not at all the right tool to map disc-related formats
really need to make a new type of object for presenting a virtual disc
and initialising cdrom.Iso from that (while maintaining .iso file compatibility)

NOTE: cdrom.Iso is an ArchiveClass, since it has an actual filesystem

would also reduce memory costs by "streaming" data from discs