Skip to content

Commit

Permalink
gh-74: support compression in archives
Browse files Browse the repository at this point in the history
  • Loading branch information
EgorOrachyov committed May 17, 2024
1 parent 63437b7 commit 585949c
Show file tree
Hide file tree
Showing 499 changed files with 590 additions and 413 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/clang-format.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ on:
branches: [ main ]
paths:
- '.github/**'
- 'engine/code/**'
- 'engine/runtime/**'
- 'engine/plugins/**'
- '.clang-format'
pull_request:
Expand All @@ -19,7 +19,7 @@ jobs:
fail-fast: false
matrix:
path:
- 'engine/code'
- 'engine/runtime'
- 'engine/plugins'
steps:
- uses: actions/checkout@v2
Expand Down
4 changes: 2 additions & 2 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"program": "${workspaceRoot}/build-Debug/template/template.exe",
"args": [
"--root_remap=${workspaceRoot}/template",
"--engine_remap=root://../engine/"
"--engine_remap=../engine/"
],
"stopAtEntry": false,
"cwd": "${workspaceRoot}/build-Debug/template",
Expand All @@ -25,7 +25,7 @@
"program": "${workspaceRoot}/build-Release/template/template.exe",
"args": [
"--root_remap=${workspaceRoot}/template",
"--engine_remap=root://../engine/"
"--engine_remap=../engine/"
],
"stopAtEntry": false,
"cwd": "${workspaceRoot}/build-Release/template",
Expand Down
2 changes: 1 addition & 1 deletion engine/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
###########################################################
# Engine runtime source code

add_subdirectory(code)
add_subdirectory(runtime)

###########################################################
# Plugins sources (standard plugins for engine)
Expand Down
133 changes: 0 additions & 133 deletions engine/code/io/archive_file.cpp

This file was deleted.

6 changes: 3 additions & 3 deletions engine/config/engine.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ log_to_console_level = "Info"
workers = 4

[file_system]
engine_path = "root://engine/"
asset_path = "root://assets/"
local_path = "root://.wgengine/"
engine_path = "engine/"
asset_path = "assets/"
local_path = ".wgengine/"
cache_path = "local://cache/"
debug_path = "local://debug/"
logs_path = "local://logs/"
Expand Down
4 changes: 2 additions & 2 deletions engine/config/game.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,5 @@ width = 1280
height = 720
vsync = 1
exit = 1
icon_default = "root://icons/wmoge-128.png"
icon_small = "root://icons/wmoge-64.png"
icon_default = "icons/wmoge-128.png"
icon_small = "icons/wmoge-64.png"
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion engine/code/engine.hpp → engine/runtime/engine.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -143,11 +143,11 @@

#include "io/archive.hpp"
#include "io/archive_file.hpp"
#include "io/archive_memory.hpp"
#include "io/base64.hpp"
#include "io/compression.hpp"
#include "io/context.hpp"
#include "io/enum.hpp"
#include "io/ini.hpp"
#include "io/serialization.hpp"
#include "io/yaml.hpp"

Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@
#include "glsl/glsl_include_processor.hpp"
#include "grc/shader_compiler_task_manager.hpp"
#include "io/archive.hpp"
#include "io/archive_memory.hpp"
#include "io/enum.hpp"
#include "profiler/profiler.hpp"
#include "system/ioc_container.hpp"
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@
namespace wmoge {

Status PsoFile::read(const std::string& path, FilePsoData& pso_data) {
ArchiveReaderFile archive;
WG_CHECKED(archive.open(path));
ArchiveFile archive;
WG_CHECKED(archive.open(path, {FileOpenMode::In, FileOpenMode::Binary}));

IoContext context;
WG_ARCHIVE_READ(context, archive, pso_data);
Expand All @@ -42,8 +42,8 @@ namespace wmoge {
}

Status PsoFile::write(const std::string& path, const FilePsoData& pso_data) {
ArchiveWriterFile archive;
WG_CHECKED(archive.open(path));
ArchiveFile archive;
WG_CHECKED(archive.open(path, {FileOpenMode::Out, FileOpenMode::Binary}));

IoContext context;
WG_ARCHIVE_WRITE(context, archive, pso_data);
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -116,10 +116,10 @@ namespace wmoge {
Status ShaderCache::load_cache(const std::string& file_path, GfxShaderPlatform platform, bool allow_missing) {
WG_AUTO_PROFILE_GRC("ShaderCache::load_cache");

ArchiveReaderFile archive;
IoContext context;
ArchiveFile archive;
IoContext context;

if (!archive.open(file_path)) {
if (!archive.open(file_path, {FileOpenMode::In, FileOpenMode::Binary})) {
if (allow_missing) {
return WG_OK;
}
Expand All @@ -129,6 +129,7 @@ namespace wmoge {
}

FileShaderProgramCache cache;

WG_ARCHIVE_READ(context, archive, cache);

if (cache.platform != platform) {
Expand Down Expand Up @@ -169,15 +170,16 @@ namespace wmoge {
}
}

ArchiveWriterFile archive;
IoContext context;
ArchiveFile archive;
IoContext context;

if (!archive.open(file_path)) {
if (!archive.open(file_path, {FileOpenMode::Out, FileOpenMode::Binary})) {
WG_LOG_ERROR("failed to open shader cache " << file_path << " for platform " << Enum::to_str(platform));
return StatusCode::FailedOpenFile;
}

WG_ARCHIVE_WRITE(context, archive, cache);

WG_LOG_INFO("save " << file_path << " at=" << cache.timestamp << " entries=" << cache.programs.size());

return WG_OK;
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -229,16 +229,18 @@ namespace wmoge {
WG_AUTO_PROFILE_GRC("ShaderLibrary::load_cache");

const std::string file_path = make_cache_file_name(folder, platform);
ArchiveReaderFile archive;
ArchiveFile archive;
IoContext context;

if (!archive.open(file_path)) {
if (!archive.open(file_path, {FileOpenMode::In, FileOpenMode::Binary})) {
WG_LOG_ERROR("failed to open shader library " << file_path << " for platform " << Enum::to_str(platform));
return StatusCode::FailedOpenFile;
}

FileShaderLibrary library;
WG_CHECKED(archive.begin_compressed());
WG_ARCHIVE_READ(context, archive, library);
WG_CHECKED(archive.end_compressed());

std::unique_lock lock(m_mutex);

Expand Down Expand Up @@ -288,15 +290,17 @@ namespace wmoge {
}

const std::string file_path = make_cache_file_name(folder, platform);
ArchiveWriterFile archive;
ArchiveFile archive;
IoContext context;

if (!archive.open(file_path)) {
if (!archive.open(file_path, {FileOpenMode::Out, FileOpenMode::Binary})) {
WG_LOG_ERROR("failed to open shader library " << file_path << " for platform " << Enum::to_str(platform));
return StatusCode::FailedOpenFile;
}

WG_CHECKED(archive.begin_compressed());
WG_ARCHIVE_WRITE(context, archive, library);
WG_CHECKED(archive.end_compressed());

WG_LOG_INFO("save " << file_path
<< " at=" << library.timestamp
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -47,27 +47,9 @@ namespace wmoge {
}

void on_add_cmd_line_options(CmdLine& cmd_line) override {
cmd_line.add_string("root_config", "path to exe config", "root://config/");
}

Status on_process(CmdLine& cmd_line) override {
Config* config = IocContainer::iresolve_v<Config>();
FileSystem* fs = IocContainer::iresolve_v<FileSystem>();

const std::string path = cmd_line.get_string("root_config");

if (!config->load(path + "/engine.cfg")) {
std::cerr << "failed to load exe engine.cfg file, check your configuration file or path";
}
if (!config->load(path + "/game.cfg")) {
std::cerr << "failed to load exe game.cfg file, check your configure file of path";
}
if (!config->load(path + "/cvars.cfg")) {
std::cerr << "failed to load exe cvars.cfg file, check your configure file of path";
}

fs->setup_mappings();

return WG_OK;
}
};
Expand Down
Loading

0 comments on commit 585949c

Please sign in to comment.