Skip to content

Latest commit

 

History

History
220 lines (183 loc) · 11.7 KB

README.md

File metadata and controls

220 lines (183 loc) · 11.7 KB
Raw

Backup me

This tool is designed for backups. It allows backing up various databases and files to S3 compatible storages.

It is designed to be run as a Docker container.

Important

This tool is definitively NOT production ready, and will certainly never be. You shouldn't use it in a production critical environment. I know my way around databases and devops tools, but I don't claim to be a database backup expert.

I have built this tool more as a hobby (yes I know, weird hobby...) and for a personal use cases. If you'd like to use, I'd be more than happy to help you.

Please use more production ready and complete tools if you have real and serious use cases.

Installation

Docker images are build for version and published on the GitHub container registry.

The available versions can be browsed in the Packages of the repo.

docker run ghcr.io/bastantoine/backup-me:latest

Configuration

The tool can be configured with a config.json file, with the following structure:

{
    "app": "<APP NAME>",
    "sources": [
        [...]
    ],
    "destinations": [
        [...]
    ]
}
  • app: the name of the application currently being handled. This will be used in the name of the backups.
  • sources is a list of one or more sources that needs to be backed up. The available sources are listed below.
  • destinations is a list of one or more destination where to send the backups once they have been generated. The available destinations are listed below.

Some parameters can be configured from env var. They are marked as sensitive below.

The environment variables should be named <source/destination type>_<name>_<param name> where:

  • <source/destination type> is the uppercased value of the type field of the source/destination being configured. See below the list of allowed values. being the name of the destination uppercased, and with spaces replaced by underscores.
  • <name> is the uppercased value of the name field of the source/destination being configured, with spaces replaced by underscores.
  • <param name> is the uppercased value of the parameter being configured.

If a param is provided both in the config file and in the environement, the value of the config file will have precedence.

Available sources

MySQL

{
    "type": "mysql",
    "name": "mysql_db",
    "backup_filename": "mysql_backup",
    "host": "localhost",
    "port": "3306",
    "username": "username",
    "password": "password",
    "database": "--all-databases",
    "all_databases": true,
    "mysql_dump_bin": "/usr/local/opt/mysql-client/bin/mysqldump"
}
Key Description Required Default value Sensitive
type Type of source Yes "mysql" No
name Name of source Yes No
backup_filename Name of the backup file generated Yes   No
host Hostname of the MySQL server Yes No
port Port to connect to the server No 3306 No
username Username to connect to the server Yes Yes
password Password to connect to the server Yes   Yes
database Name of the database to backup No* No
all_databases Whether to backup all databases of the server No* False No
mysql_dump_bin Path to the mysql_dump_bin tool used for the backup No "/usr/local/opt/mysql-client/bin/mysqldump" No

Tip

The database and all_databases parameters are mutually exclusive. One of them must be provided at all time. Config validation will fail if none are provided. If both are provided at the same time all_databases will have precedence.

PostgreSQL

{
    "type": "postgres",
    "name": "pg_db",
    "backup_filename": "pg_backup",
    "host": "0.0.0.0",
    "port": "5432",
    "username": "username",
    "password": "password",
    "database": "db",
    "all_databases": true,
    "pg_dump_bin": "/usr/local/opt/libpq/bin/pg_dump",
    "pg_dumpall_bin": "/usr/local/opt/libpq/bin/pg_dumpall"
}
Key Description Required Default value Sensitive
type Type of source Yes "postgres" No
name Name of source Yes No
backup_filename Name of the backup file generated Yes   No
host Hostname of the PostgreSQL server Yes No
port Port to connect to the server No 5432 No
username Username to connect to the server Yes Yes
password Password to connect to the server Yes   Yes
database Name of the database to backup No* No
all_databases Whether to backup all databases of the server No* False No
pg_dump_bin Path to the pg_dump_bin tool used for the backup No "/usr/local/opt/libpq/bin/pg_dump" No
pg_dumpall_bin Path to the pg_dumpall_bin tool used for the backup No "/usr/local/opt/libpq/bin/pg_dumpall" No

Tip

The database and all_databases parameters are mutually exclusive. One of them must be provided at all time. Config validation will fail if none are provided. If both are provided at the same time all_databases will have precedence.

Tip

The pg_dump_bin is used when backing up only one database (ie. when the database param is provided). When backing up the entire cluster, pg_dumpall_bin is used (ie. when the all_databases param is provided).

Raw files

{
    "type": "files",
    "name": "files",
    "backup_filename": "config",
    "archive_type": "tar",
    "files": [
        "~/.vimrc",
        "~/.sqliterc",
        "~/.zshrc"
    ]
}
Key Description Required Default value
type Type of source Yes "files"
name Name of source Yes
backup_filename Name of the backup file generated Yes  
files List of paths to files to back up in the archive Yes
archive_type Type of archive to create (zip or tar) No "tar"

Tip

The paths in files can either be absolute, relative to the current user homedir, or relative the current working directory.

HTTP call

{
    "type": "http",
    "name": "http",
    "url": "https://httpbin.org/json",
    "method": "GET",
    "request_params": {
        "headers": {
            "Authorization": "Bearer THIS_IS_A_TOKEN"
        }
    }
}
Key Description Required Default value
type Type of source Yes "http"
name Name of source Yes
url URL to call Yes  
method HTTP method to use to do the API call Yes
request_params Any extra param required for the call. No

Tip

The request_params is provided as kwargs to the requests.request. Check out the documentation to see which parameter is available.

Tip

The archive created after the backup is a JSON with the following format:

{
    "url": "<the URL called>",
    "timestamp": "<backup timestamp>",
    "result": "<API call status code>",
    "detail": "<API response when JSON>",
    "msg": "<API response body when not JSON>"
}

Available destinations

S3

{
    "type": "s3",
    "name": "<destination name>",
    "bucket": "my-bucket.for.backups",
    "group": true,
    "access_key_id": "XXXX",
    "secret_access_key": "XXXX",
    "prefix": null,
    "endpoint_url": "https://<s3 service>",
    "region": "us-east-1"
}
Key Description Required Default value Sensitive
type Type of destination Yes "s3" No
name Name of the destination Yes No
bucket Name of the bucket to upload to Yes No
group In case of multiple sources, whether to group all backups in a single tar archive Yes No
access_key_id Access key to login to the S3 service Yes Yes
secret_access_key Secret access key to login to the S3 service Yes Yes
prefix Prefix to the backup file No No
endpoint_url Endpoint to the S3 service No No
region Region of the S3 service No "us-east-1" No