jkaninda/mysql-bkup
1
1 Fork 0
mirror of https://github.com/jkaninda/mysql-bkup.git synced 2026-11-13 08:37:15 +00:00
Code Issues Actions Projects Releases Wiki Activity

feat: mysql-bkup v2 on bkupkit

Browse Source 
MysQL backup and restore built on the shared bkupkit core:
streaming end to end, with no dump held in memory or written to
local disk.
This commit is contained in:
jkaninda committed 2026-10-28 11:18:56 +01:00
1 parent ccb3fcdf26
commit 28de0d7b52
65 files changed
+3453 -3871
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/reference/index.md
+238 -69
View File
@@ -6,81 +6,250 @@ nav_order: 3
# Configuration Reference
MySQL backup, restore, and migration processes can be configured using **environment variables** or **CLI flags**.
mysql-bkup is configured with **environment variables**, **CLI flags** and an optional **YAML file**. For every setting the order is: flag, environment variable, deprecated name, YAML `env:` section, default. Every secret (`DB_PASSWORD`, `AWS_SECRET_KEY`, `GPG_PASSPHRASE`, ...) can also be read from a file with `<NAME>_FILE`.
## CLI Utility Usage
Run `mysql-bkup config` to see the resolved configuration and where each value comes from (secrets redacted).
The `mysql-bkup` CLI provides commands and options to manage MySQL backups efficiently.
## Commands
| Option | Short Flag | Description |
|-------------------------|------------|-----------------------------------------------------------------------------------------|
| `mysql-bkup` | `bkup` | CLI tool for managing MySQL backups, restoration, and migration. |
| `backup` | | Executes a backup operation. |
| `restore` | | Restores a database from a backup file. |
| `migrate` | | Migrates a database from one instance to another. |
| `--storage` | `-s` | Specifies the storage type (`local`, `s3`, `ssh`, etc.). Default: `local`. |
| `--file` | `-f` | Defines the backup file name for restoration. |
| `--path` | | Sets the storage path (e.g., `/custom_path` for S3 or `/home/foo/backup` for SSH). |
| `--config` | `-c` | Provides a configuration file for multi-database backups (e.g., `/backup/config.yaml`). |
| `--dbname` | `-d` | Specifies the database name to back up or restore. |
| `--port` | `-p` | Defines the database port. Default: `3306`. |
| `--disable-compression` | | Disables compression for database backups. |
| `--cron-expression` | `-e` | Schedules backups using a cron expression (e.g., `0 0 * * *` or `@daily`). |
| `--all-databases` | `-a` | Backs up all databases separately (e.g., `backup --all-databases`). |
| `--all-in-one` | `-A` | Backs up all databases in a single file (e.g., `backup --all-databases --single-file`). |
| `--custom-name` | `` | Sets custom backup name for one time backup |
| `--help` | `-h` | Displays the help message and exits. |
| `--version` | `-V` | Shows version information and exits. |
| Command | Description |
|---|---|
| `backup` | Back up now, or as a daemon with `--cron-expression` |
| `restore` | Restore a backup: `-f FILE [-d DB]` |
| `migrate` | Copy a database (or every database with `-a`) into `TARGET_DB_*`, streamed |
| `verify` | Download, decrypt and decompress a backup and check its sha256 |
| `list` | Backup history from the index (`--json`, `--storage-scan`) |
| `delete` | Delete a backup, its sidecar and its index entry (`-f FILE`) |
| `prune` | Apply the retention policy (`--dry-run`) |
| `reconcile` | Rebuild the backup index (`--from-index OLD`, `--dry-run`) |
| `config` | Show the resolved configuration; `--env-docs` prints the tables below |
| `health` | Exit 0 when the daemon is healthy (Docker `HEALTHCHECK`) |
| `version` | Print the version |
---
## Flags
## Environment Variables
| Flag | Short | Commands | Variable |
|---|---|---|---|
| `--dbname` | `-d` | all | `DB_NAME` |
| `--storage` | `-s` | all | `STORAGE` (`local`, `s3`, `gcs`, `azure`, `sftp`/`ssh`/`remote`, `ftp`) |
| `--path` (`--remote-path`) | `-P` | all | `REMOTE_PATH` |
| `--file` | `-f` | restore, verify, delete | `FILE_NAME` |
| `--config` | `-c` | backup, list, prune, config | `BACKUP_CONFIG_FILE` |
| `--cron-expression` | `-e` | backup | `BACKUP_CRON_EXPRESSION` |
| `--all-databases` | `-a` | backup, migrate | `BACKUP_ALL_DATABASES` |
| `--all-in-one` | `-A` | backup | `BACKUP_ALL_IN_ONE` |
| `--exclude-db` | | backup | `BACKUP_EXCLUDE_DATABASES` |
| `--custom-name` | | backup | `BACKUP_CUSTOM_NAME` (fixed name, no timestamp) |
| `--name` | `-n` | backup, list, prune | `BACKUP_NAME` |
| `--disable-compression` | | backup | `DISABLE_COMPRESSION` |
| `--compression` | | backup | `COMPRESSION` (`gzip`, `zstd`, `none`) |
| `--encryption` | | backup | `ENCRYPTION` (`auto`, `none`, `gpg`, `age`) |
| `--index-name` | | all | `BACKUP_INDEX_NAME` |
| `--result-file` | | all | `RESULT_FILE` |
| `--help` | `-h` | all | |
| `--version` | `-v` | root | |
| Name | Requirement | Description |
|--------------------------------|--------------------------------------|----------------------------------------------------------------------------|
| `DB_PORT` | Optional (default: `3306`) | Database port number. |
| `DB_HOST` | Required | Database host. |
| `DB_NAME` | Optional (if provided via `-d` flag) | Database name. |
| `DB_USERNAME` | Required | Database username. |
| `DB_PASSWORD` | Required | Database password. |
| `DB_SSL_CA` | Optional | Database client CA certificate file |
| `DB_SSL_MODE` | Optional(`0 or 1`) default: `0` | Database client Enable CA validation |
| `AWS_ACCESS_KEY` | Required for S3 storage | AWS S3 Access Key. |
| `AWS_SECRET_KEY` | Required for S3 storage | AWS S3 Secret Key. |
| `AWS_BUCKET_NAME` | Required for S3 storage | AWS S3 Bucket Name. |
| `AWS_REGION` | Required for S3 storage | AWS Region. |
| `AWS_DISABLE_SSL` | Optional | Disable SSL for S3 storage. |
| `AWS_FORCE_PATH_STYLE` | Optional | Force path-style access for S3 storage. |
| `FILE_NAME` | Optional (if provided via `--file`) | File name for restoration (e.g., `.sql`, `.sql.gz`). |
| `GPG_PASSPHRASE` | Optional | GPG passphrase for encrypting/decrypting backups. |
| `GPG_PUBLIC_KEY` | Optional | GPG public key for encrypting backups (e.g., `/config/public_key.asc`). |
| `BACKUP_CRON_EXPRESSION` | Optional (flag `-e`) | Cron expression for scheduled backups. |
| `BACKUP_RETENTION_DAYS` | Optional | Delete backups older than the specified number of days. |
| `BACKUP_CONFIG_FILE` | Optional (flag `-c`) | Configuration file for multi database backup. (e.g: `/backup/config.yaml`) |
| `SSH_HOST` | Required for SSH storage | SSH remote hostname or IP. |
| `SSH_USER` | Required for SSH storage | SSH remote username. |
| `SSH_PASSWORD` | Optional | SSH remote user's password. |
| `SSH_IDENTIFY_FILE` | Optional | SSH remote user's private key. |
| `SSH_PORT` | Optional (default: `22`) | SSH remote server port. |
| `REMOTE_PATH` | Required for SSH/FTP storage | Remote path (e.g., `/home/toto/backup`). |
| `FTP_HOST` | Required for FTP storage | FTP hostname. |
| `FTP_PORT` | Optional (default: `21`) | FTP server port. |
| `FTP_USER` | Required for FTP storage | FTP username. |
| `FTP_PASSWORD` | Required for FTP storage | FTP user password. |
| `TARGET_DB_HOST` | Required for migration | Target database host. |
| `TARGET_DB_PORT` | Optional (default: `5432`) | Target database port. |
| `TARGET_DB_NAME` | Required for migration | Target database name. |
| `TARGET_DB_USERNAME` | Required for migration | Target database username. |
| `TARGET_DB_PASSWORD` | Required for migration | Target database password. |
| `TARGET_DB_URL` | Optional | Target database URL in JDBC URI format. |
| `TG_TOKEN` | Required for Telegram notifications | Telegram token (`BOT-ID:BOT-TOKEN`). |
| `TG_CHAT_ID` | Required for Telegram notifications | Telegram Chat ID. |
| `TZ` | Optional | Time zone for scheduling. |
| `AZURE_STORAGE_CONTAINER_NAME` | Required for Azure Blob Storage | Azure storage container name. |
| `AZURE_STORAGE_ACCOUNT_NAME` | Required for Azure Blob Storage | Azure storage account name. |
| `AZURE_STORAGE_ACCOUNT_KEY` | Required for Azure Blob Storage | Azure storage account key. |
## Exit codes
| Code | Meaning |
|---|---|
| `0` | success |
| `1` | failure |
| `2` | configuration error |
| `3` | database connection or authentication |
| `4` | storage |
| `5` | encryption or decryption |
## Environment variables
### storage
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `STORAGE` | `--storage` `-s` | `local` | | Storage backend: local, s3, gcs, azure, sftp (ssh, remote), ftp (local, s3, gcs, azure, sftp, ssh, ftp, remote) |
| `STORAGE_PATH` | | `/backup` | | Root directory of the local storage backend |
| `REMOTE_PATH` | `--path` `-P` | | `AWS_S3_PATH` (deprecated), `S3_PATH` (deprecated) | Folder inside the storage backend (without file name) |
### general
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `BACKUP_NAME` | `--name` `-n` | | | Base name of the artifact; a timestamp is appended (default: the database or volume name) |
| `BACKUP_CUSTOM_NAME` | `--custom-name` | | | Fixed artifact name without timestamp (overwrites the previous one) |
| `FILE_NAME` | `--file` `-f` | | | Backup file to restore, verify or delete |
| `BACKUP_CONFIG_FILE` | `--config` `-c` | | | YAML configuration file (default /config/config.yaml if present) |
| `BACKUP_TIMEOUT` | | | | Abort a backup after this duration (0 = none) |
| `RESTORE_TIMEOUT` | | | | Abort a restore after this duration (0 = none) |
| `TMP_DIR` | | `/tmp/backup` | `BACKUP_TMP_DIR` (deprecated) | Scratch directory (only for secret files and disk-based modes) |
| `COMPRESSION` | `--compression` | `gzip` | | Compression: gzip, zstd or none (gzip, zstd, none) |
| `COMPRESSION_LEVEL` | | | | Compression level (0 = default) |
| `DISABLE_COMPRESSION` | `--disable-compression` | | | Same as COMPRESSION=none |
| `RESTORE_VERIFY_FIRST` | `--verify-first` | `auto` | | Verify the whole artifact before restoring (auto: on for GPG-encrypted volumes) (auto, true, false) |
| `LOG_LEVEL` | | `info` | | Log level (debug, info, warn, error) |
| `LOG_FORMAT` | | `text` | | Log format (text, json) |
| `TIME_FORMAT` | | `2006-01-02 15:04:05` | | Time layout used in notifications |
| `RESULT_FILE` | `--result-file` | | | Write the run result as JSON to this path |
### scheduling
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `BACKUP_CRON_EXPRESSION` | `--cron-expression` `-e` | | | Run as a daemon on this cron schedule (5 or 6 fields, @daily, @every 6h) |
| `BACKUP_RUN_ON_START` | | | | In daemon mode, run one backup immediately at start |
| `BACKUP_RESCUE_MODE` | | | | Deprecated: runs always continue with the remaining targets after a failure |
| `METRICS_ADDR` | | | | Serve Prometheus /metrics and /healthz on this address in daemon mode (e.g. :9090) |
| `HEALTH_FILE` | | `/tmp/bkup-health.json` | | State file read by the health command |
| `HEALTH_MAX_AGE` | | | | health fails when the last success is older than this (default: 2x the schedule interval) |
### retention
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `BACKUP_RETENTION_DAYS` | | `0` | | Delete backups older than N days (0 = keep) |
| `BACKUP_RETENTION_COUNT` | | `0` | | Keep only the newest N backups (0 = no limit) |
| `BACKUP_RETENTION_ALLOW_ROOT` | | | | Allow pattern-based retention at the storage root (index disabled) |
| `BACKUP_RETENTION_ADOPT_LEGACY` | | | | Import pre-index backups matching this tool's naming into the index |
| `BACKUP_INDEX` | | `true` | | Keep the backup index file and per-backup .meta.json sidecars |
| `BACKUP_INDEX_NAME` | `--index-name` | `{tool}-metadata.json` | | Index file name; placeholders {tool} {engine} {name} {db} |
### storage-s3
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `AWS_S3_ENDPOINT` | | | `S3_ENDPOINT` (deprecated), `AWS_ENDPOINT_URL_S3`, `AWS_ENDPOINT_URL` | S3 endpoint (empty = AWS) |
| `AWS_S3_BUCKET_NAME` | | | `BUCKET_NAME` (deprecated), `AWS_BUCKET_NAME` (deprecated) | S3 bucket |
| `AWS_REGION` | | `us-east-1` | `AWS_DEFAULT_REGION` | S3 region |
| `AWS_ACCESS_KEY` | | | `ACCESS_KEY` (deprecated), `AWS_ACCESS_KEY_ID` | S3 access key (empty = default AWS credential chain) |
| `AWS_SECRET_KEY` | | | `SECRET_KEY` (deprecated), `AWS_SECRET_ACCESS_KEY`, `AWS_SECRET_KEY_FILE` | S3 secret key |
| `AWS_SESSION_TOKEN` | | | `AWS_SESSION_TOKEN_FILE` | S3 session token |
| `AWS_PROFILE` | | | | AWS shared config profile |
| `AWS_DISABLE_SSL` | | | | Use http for the S3 endpoint |
| `AWS_FORCE_PATH_STYLE` | | | | Path-style S3 addressing (MinIO) |
| `S3_STORAGE_CLASS` | | | | S3 storage class |
| `S3_SSE` | | | | Server-side encryption: AES256 or aws:kms (aes256, aws:kms) |
| `S3_SSE_KMS_KEY_ID` | | | | KMS key id for aws:kms |
| `S3_PART_SIZE` | | `64MiB` | | Multipart part size; max object size is 10000 x this |
| `S3_UPLOAD_CONCURRENCY` | | `2` | | Parallel part uploads (memory = part size x this) |
### storage-sftp
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `SFTP_HOST` | | | | SFTP host |
| `SFTP_PORT` | | `22` | | SFTP port |
| `SFTP_USER` | | | | SFTP user |
| `SFTP_PASSWORD` | | | `SFTP_PASSWORD_FILE` | SFTP password |
| `SFTP_IDENTITY_FILE` | | | | SFTP private key file (the SSH agent is also used when SSH_AUTH_SOCK is set) |
| `SFTP_IDENTITY_PASSPHRASE` | | | `SFTP_IDENTITY_PASSPHRASE_FILE` | Passphrase of the SFTP private key |
| `SFTP_KNOWN_HOSTS` | | | | known_hosts file used to verify the SFTP host key (default ~/.ssh/known_hosts) |
| `SFTP_INSECURE_IGNORE_HOST_KEY` | | | | Do not verify the SFTP host key (insecure) |
### storage-ftp
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `FTP_HOST` | | | `FTP_HOST_NAME` (deprecated) | FTP host |
| `FTP_PORT` | | `21` | | FTP port |
| `FTP_USER` | | | | FTP user |
| `FTP_PASSWORD` | | | `FTP_PASSWORD_FILE` | FTP password |
| `FTP_TLS` | | | | Use explicit FTPS |
### storage-azure
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `AZURE_STORAGE_ACCOUNT_NAME` | | | | Azure storage account |
| `AZURE_STORAGE_ACCOUNT_KEY` | | | `AZURE_STORAGE_ACCOUNT_KEY_FILE` | Azure storage account key |
| `AZURE_STORAGE_CONTAINER_NAME` | | | | Azure blob container |
| `AZURE_STORAGE_ENDPOINT` | | | | Azure blob endpoint (Azurite, sovereign clouds) |
| `AZURE_STORAGE_SAS_TOKEN` | | | `AZURE_STORAGE_SAS_TOKEN_FILE` | Azure SAS token (instead of the account key) |
### storage-gcs
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `GCS_BUCKET` | | | | GCS bucket |
| `GCS_CREDENTIALS_FILE` | | | `GOOGLE_APPLICATION_CREDENTIALS` | GCS service account JSON file |
| `GCS_CREDENTIALS_JSON` | | | `GCS_CREDENTIALS_JSON_FILE` | GCS service account JSON content |
| `GCS_ENDPOINT` | | | | GCS endpoint (emulators) |
### encryption
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `ENCRYPTION` | `--encryption` | `auto` | | auto (from the keys set), none, gpg or age (auto, none, gpg, age) |
| `ENCRYPTION_ARMOR` | | | | ASCII-armor encrypted output |
| `GPG_PASSPHRASE` | | | `GPG_PASSPHRASE_FILE` | GPG symmetric passphrase |
| `GPG_PUBLIC_KEY` | | | | GPG public key(s): file path or inline key (default /config/gnupg/public_key.{asc,gpg}) |
| `GPG_PRIVATE_KEY` | | | `GPG_PRIVATE_KEY_FILE` | GPG private key for restore: file path or inline key (default /config/gnupg/private_key.{asc,gpg}) |
| `GPG_PRIVATE_KEY_PASSPHRASE` | | | `GPG_PRIVATE_KEY_PASSPHRASE_FILE` | Passphrase of the GPG private key (default GPG_PASSPHRASE) |
| `AGE_RECIPIENTS` | | | | age recipients (age1..., ssh-ed25519 ...), comma or newline separated |
| `AGE_RECIPIENTS_FILE` | | | | File with age recipients |
| `AGE_PASSPHRASE` | | | `AGE_PASSPHRASE_FILE` | age passphrase (scrypt) |
| `AGE_IDENTITY` | | | `AGE_IDENTITY_FILE` | age identity (AGE-SECRET-KEY-...) for restore |
| `AGE_IDENTITY_FILE` | | | | age identity file for restore (default /config/age/identity.txt) |
### notifications
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `MAIL_HOST` | | | | SMTP host |
| `MAIL_PORT` | | `587` | | SMTP port |
| `MAIL_USERNAME` | | | | SMTP user |
| `MAIL_PASSWORD` | | | `MAIL_PASSWORD_FILE` | SMTP password |
| `MAIL_FROM` | | | | Sender address |
| `MAIL_TO` | | | | Recipients, comma separated |
| `MAIL_SKIP_TLS` | | | | Deprecated: true skips certificate verification (use MAIL_INSECURE_SKIP_VERIFY) |
| `MAIL_INSECURE_SKIP_VERIFY` | | | | Skip SMTP certificate verification |
| `MAIL_TLS` | | `starttls` | | SMTP TLS mode (starttls, tls, none) |
| `TG_TOKEN` | | | `TELEGRAM_BOT_TOKEN`, `TG_TOKEN_FILE` | Telegram bot token |
| `TG_CHAT_ID` | | | `TELEGRAM_CHAT_ID` | Telegram chat id |
| `POSTA_URL` | | | | Posta base URL (https://posta.example.com) for email notifications through Posta |
| `POSTA_API_KEY` | | | `POSTA_API_KEY_FILE` | Posta API key with the send scope |
| `POSTA_WORKSPACE_ID` | | | | Posta workspace id (only for account-wide API keys) |
| `POSTA_FROM` | | | | Posta sender address (default MAIL_FROM) |
| `POSTA_TO` | | | | Posta recipients, comma separated (default MAIL_TO) |
| `POSTA_TEMPLATE` | | | | Posta template name for notifications; the event fields are passed as template data (default: built-in HTML) |
| `POSTA_TEMPLATE_ERROR` | | | | Posta template name for failures (default POSTA_TEMPLATE) |
| `POSTA_LANGUAGE` | | | | Posta template language |
| `WEBHOOK_URL` | | | | Generic JSON webhook URL |
| `WEBHOOK_SECRET` | | | `WEBHOOK_SECRET_FILE` | HMAC-SHA256 secret for the X-Bkup-Signature header |
| `SLACK_WEBHOOK_URL` | | | `SLACK_WEBHOOK_URL_FILE` | Slack incoming webhook URL |
| `DISCORD_WEBHOOK_URL` | | | `DISCORD_WEBHOOK_URL_FILE` | Discord webhook URL |
| `NTFY_URL` | | | | ntfy topic URL |
| `NTFY_TOKEN` | | | `NTFY_TOKEN_FILE` | ntfy access token |
| `NOTIFY_ON` | | `always` | | When to notify (always, failure, success, never) |
| `HEALTHCHECK_PING_URL` | | | | healthchecks.io-style URL pinged on start (/start), success and failure (/fail) |
| `BACKUP_REFERENCE` | | | | Free text included in notifications |
| `TEMPLATES_DIR` | | `/config/templates` | | Directory with custom notification templates |
### database
| Variable | Flag | Default | Fallbacks | Description |
|---|---|---|---|---|
| `DB_URL` | | | `DB_URL_FILE` | Connection URL mysql://user:password@host:port/database?ssl-mode=required (overrides host, port, user, password) |
| `DB_HOST` | | `localhost` | | Database host (a path starting with / is a Unix socket) |
| `DB_PORT` | | `3306` | | Database port |
| `DB_NAME` | `--dbname` `-d` | | | Database name |
| `DB_USER` | | | `DB_USERNAME` | Database user |
| `DB_PASSWORD` | | | `DB_PASSWORD_FILE` | Database password |
| `DB_SSL_MODE` | | `disabled` | | TLS mode: disabled, preferred, required, verify_ca, verify_identity (1.x values 0 and 1 mean disabled and required) |
| `DB_SSL_CA` | | `/etc/ssl/certs/ca-certificates.crt` | | CA certificate file, used with DB_SSL_MODE=verify_ca or verify_identity |
| `DB_SSL_CERT` | | | | Client certificate file |
| `DB_SSL_KEY` | | | | Client key file |
| `BACKUP_ALL_DATABASES` | `--all-databases` `-a` | | | Back up every database, one file each |
| `BACKUP_ALL_IN_ONE` | `--all-in-one` `-A` | | | Back up every database into a single file |
| `BACKUP_EXCLUDE_DATABASES` | `--exclude-db` | | | Databases to skip with --all-databases, comma separated |
| `TARGET_DB_URL` | | | `TARGET_DB_URL_FILE` | migrate: target connection URL |
| `TARGET_DB_HOST` | | | | migrate: target host |
| `TARGET_DB_PORT` | | `3306` | | migrate: target port |
| `TARGET_DB_NAME` | | | | migrate: target database (default: the source name) |
| `TARGET_DB_USER` | | | `TARGET_DB_USERNAME` | migrate: target user |
| `TARGET_DB_PASSWORD` | | | `TARGET_DB_PASSWORD_FILE` | migrate: target password |
| `TARGET_DB_SSL_MODE` | | `disabled` | | migrate: target TLS mode (same values as DB_SSL_MODE) |
| `TARGET_DB_SSL_CA` | | `/etc/ssl/certs/ca-certificates.crt` | | migrate: target CA certificate file |
| `MYSQL_CLIENT` | | `auto` | | Client tools: auto (mariadb-dump for MariaDB, Oracle mysqldump for MySQL), mysql or mariadb (auto, mysql, mariadb) |
---
## Scheduled Backups
@@ -91,7 +260,7 @@ The `mysql-bkup` CLI provides commands and options to manage MySQL backups effic
### Cron Syntax
The cron syntax consists of five fields:
The cron syntax consists of five fields (a sixth, leading seconds field is also accepted):
```conf
* * * * * command