Skip to content

Latest commit

 

History

History
91 lines (62 loc) · 3.46 KB

README.md

File metadata and controls

91 lines (62 loc) · 3.46 KB
Raw

Update system

A very efficient, reliable update system with a static repository.

Efficiency is achieved by:

  • minimizing download size by only downloading required parts,
  • minimizing download request: the update path is often a single big chunk of bytes
  • on the fly decompression/patching
  • on the fly integrity checks
  • resumable update process
  • Unreal Engine 4 pak file patcher (no more multigigabytes diff file)
  • very fast and with good ratio lossless codecs (brotli, zstd, ue4pak)

Reliability is achieved by:

  • enforcing download integrity
  • enforcing file integrity
  • auto repair process if something is wrong
  • rust borrowing rules and typings (no unwrap, no unsafe outside of codecs, ...)

Easy to use

  • static repository with JSON metadata
  • repository management cli tool
  • update test cli tool
  • logging infrastructure (rust log)
  • C binding with update progress reporting

Dependencies

To achieve thoses goals this library depends on:

  • rust: a memory safe programming language
  • future-rs: zero-cost Futures in Rust
  • reqwest: an ergonomic, batteries-included HTTP Client for Rust,
  • native-tls: a wrapper over a platform's native TLS implementation

Supported codecs

  • ue4pak: Unreal Engine 4 pak file direct support for hyper efficient patch files
  • zstd: a good compressor with very fast decompression speed (~ 800MB/s)
  • brotli: a very good compressor with fast decompression speed (~ 300MB/s)
  • lzma: an awesome compressor with slow decompression speed (~ 50MB/s)
  • vcdiff-rs: for vcdiff decoding

Documentation

The documentation of repository format and some internal things is in the doc folder.

Terminology

  • version: a node in the transition graph
  • package: an edge between two versions in the transition graph
  • operation: a way to synchronize a file or directory for a given relative file path
  • repository: a collection of packages and versions forming a transition graph with a pointer to the current version
  • workspace: local files managed and updated by this system

Update logic

The update system only uses 2 threads:

  • one to manages download
  • one to apply changes to the workspace

The update logic is made of multiples stages

  • prepare: find the shortest path of packages to download
  • update: do the actual update
  • recover: if something did go wrong in the update pass, try to fix it

Repository

Here is an overview of repository files content:

Project layout

  • bin/repository: rust cli tool to manage repository (set current version, add version, ...)
  • bin/workspace: rust cli tool to manage workspace (check integrity, update, downgrade, ...)
  • bindings/c: C bindings to integrate workspace functionality to your program
  • src: update system core library

Future improvements

Split big files into smaller chunks so the repair process can only download corrupted chunks.

Add a 4th stage workaround: if something did go wrong again in the update pass, try to fix it by doing something else.