11 Commits

Author SHA1 Message Date
602e2bb329 adjust testing procedure
All checks were successful
CI / Build Docker image (pull_request) Successful in 17s
2025-09-04 20:32:45 +02:00
fc1967536f drop deprecated compose version 2025-09-04 20:32:34 +02:00
ed8121a7eb sync
Some checks failed
CI / Build Docker image (pull_request) Failing after 15s
2025-09-04 20:20:01 +02:00
e6e01975b9 feat: add restore functionality
All checks were successful
CI / Build Docker image (pull_request) Successful in 18s
2025-09-04 20:19:17 +02:00
fb31691451 feat: make gotify optional (#16)
All checks were successful
CD / Check changes (push) Successful in 13s
CD / Create tag (push) Successful in 11s
CD / Build and push (amd64) (push) Successful in 24s
CD / Build and push (arm64) (push) Successful in 1m22s
CD / Create manifest (push) Successful in 25s
Reviewed-on: #16
Co-authored-by: Timo Behrendt <t.behrendt@t00n.de>
Co-committed-by: Timo Behrendt <t.behrendt@t00n.de>
2025-09-04 20:18:24 +02:00
0c0305301e Revert "feat: make gotify optional (#14)"
All checks were successful
CD / Check changes (push) Successful in 7s
CD / Create tag (push) Successful in 17s
CD / Build and push (amd64) (push) Successful in 28s
CD / Build and push (arm64) (push) Successful in 1m23s
CD / Create manifest (push) Successful in 11s
This reverts commit cd92ce697e.
2025-09-04 20:03:40 +02:00
cd92ce697e feat: make gotify optional (#14)
All checks were successful
CD / Check changes (push) Successful in 6s
CD / Create tag (push) Successful in 14s
CD / Build and push (amd64) (push) Successful in 24s
CD / Build and push (arm64) (push) Successful in 1m25s
CD / Create manifest (push) Successful in 21s
Reviewed-on: #14
Co-authored-by: Timo Behrendt <t.behrendt@t00n.de>
Co-committed-by: Timo Behrendt <t.behrendt@t00n.de>
2025-09-04 19:34:53 +02:00
4cfb634397 ci: feat add semver conven versioning (#11)
All checks were successful
CD / Check changes (push) Successful in 10s
CD / Create tag (push) Successful in 15s
CD / Build and push (amd64) (push) Successful in 23s
CD / Build and push (arm64) (push) Successful in 5m18s
CD / Create manifest (push) Successful in 8s
Reviewed-on: #11
Co-authored-by: t.behrendt <t.behrendt@t00n.de>
Co-committed-by: t.behrendt <t.behrendt@t00n.de>
2025-09-01 13:13:54 +02:00
cb466747cd chore(deps): update alpine docker tag to v3.22 (#9)
All checks were successful
Build and Release Docker Image / Build and Push amd64 (push) Successful in 27s
Build and Release Docker Image / Build and Push arm64 (push) Successful in 1m24s
Build and Release Docker Image / Create and Push Multi-Arch Manifest (push) Successful in 14s
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| alpine | final | minor | `3.21` -> `3.22` |

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4yNjMuMSIsInVwZGF0ZWRJblZlciI6IjM5LjI2My4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Reviewed-on: #9
Reviewed-by: t.behrendt <t.behrendt@noreply.localhost>
Co-authored-by: Renovate Bot <renovate@t00n.de>
Co-committed-by: Renovate Bot <renovate@t00n.de>
2025-07-30 10:02:46 +02:00
3f544409f1 chore: Configure Renovate (#8)
Some checks failed
Build and Release Docker Image / Build and Push amd64 (push) Successful in 13s
Build and Release Docker Image / Create and Push Multi-Arch Manifest (push) Has been cancelled
Build and Release Docker Image / Build and Push arm64 (push) Has been cancelled
Welcome to [Renovate](https://github.com/renovatebot/renovate)! This is an onboarding PR to help you understand and configure settings before regular Pull Requests begin.

🚦 To activate Renovate, merge this Pull Request. To disable Renovate, simply close this Pull Request unmerged.

---
### Detected Package Files

 * `Dockerfile` (dockerfile)
 * `.gitea/workflows/cd.yaml` (github-actions)

### What to Expect

It looks like your repository dependencies are already up-to-date and no Pull Requests will be necessary right away.

---

 Got questions? Check out Renovate's [Docs](https://docs.renovatebot.com/), particularly the Getting Started section.
If you need any further assistance then you can also [request help here](https://github.com/renovatebot/renovate/discussions).

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

<!--renovate-config-hash:94693a990c975907e7f13da3309b9d56ba02b3983519b41786edf5cf031e457c-->

Reviewed-on: #8
Co-authored-by: Renovate Bot <renovate@t00n.de>
Co-committed-by: Renovate Bot <renovate@t00n.de>
2025-04-18 13:58:29 +02:00
6ff6e8759e refactor: v2 (#5)
Some checks failed
Build and Release Docker Image / Create and Push Multi-Arch Manifest (push) Has been cancelled
Build and Release Docker Image / Build and Push arm64 (push) Has been cancelled
Build and Release Docker Image / Build and Push amd64 (push) Has been cancelled
Reviewed-on: #5
Co-authored-by: Timo Behrendt <t.behrendt@t00n.de>
Co-committed-by: Timo Behrendt <t.behrendt@t00n.de>
2025-02-07 20:56:02 +01:00
10 changed files with 1028 additions and 152 deletions

View File

@@ -1,86 +1,111 @@
name: Build and Release Docker Image
name: CD
on:
push:
branches:
- v2
- main
env:
DOCKER_REGISTRY: gitea.t000-n.de
jobs:
build_amd64:
name: Build and Push amd64
runs-on: [ubuntu-latest, linux_amd64]
check-changes:
name: Check changes
runs-on: ubuntu-latest
outputs:
changes: ${{ steps.filter.outputs.code }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Checkout
uses: actions/checkout@v5
- name: Log in to registry
uses: docker/login-action@v3
- name: Get changed files
id: filter
uses: dorny/paths-filter@v3
with:
registry: gitea.t000-n.de
username: ${{ secrets.REGISTRY_USERNAME }}
password: ${{ secrets.REGISTRY_PASSWORD }}
filters: |
code:
- 'src/**'
- 'Dockerfile'
- 'gitea/workflows/**'
- name: Build Docker image for amd64
run: |
docker build \
--platform linux/amd64 \
-f Dockerfile \
-t gitea.t000-n.de/t.behrendt/backupsidecar:${{ github.sha }}-amd64 \
.
- name: Push Docker image for amd64
run: |
docker push gitea.t000-n.de/t.behrendt/backupsidecar:${{ github.sha }}-amd64
build_arm64:
name: Build and Push arm64
runs-on: [ubuntu-latest, linux_arm64]
build_and_push:
name: Build and push
needs:
- check-changes
if: ${{ needs.check-changes.outputs.changes != '0' }}
strategy:
matrix:
arch:
- amd64
- arm64
runs-on:
- ubuntu-latest
- linux_${{ matrix.arch }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Log in to registry
uses: docker/login-action@v3
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: gitea.t000-n.de
username: ${{ secrets.REGISTRY_USERNAME }}
registry: ${{ env.DOCKER_REGISTRY }}
username: ${{ secrets.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_PASSWORD }}
- name: Build Docker image for arm64
- id: meta
run: |
docker build \
--platform linux/arm64 \
-f Dockerfile \
-t gitea.t000-n.de/t.behrendt/backupsidecar:${{ github.sha }}-arm64 \
.
echo REPO_NAME=$(echo ${GITHUB_REPOSITORY} | awk -F"/" '{print $2}' | tr '[:upper:]' '[:lower:]') >> $GITHUB_OUTPUT
echo REPO_VERSION=$(git describe --tags --always | sed 's/^v//') >> $GITHUB_OUTPUT
- uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile
platforms: linux/${{ matrix.arch }}
push: true
provenance: false
tags: |
${{ env.DOCKER_REGISTRY }}/t.behrendt/${{ steps.meta.outputs.REPO_NAME }}:${{ steps.meta.outputs.REPO_VERSION }}-${{ matrix.arch }}
- name: Push Docker image for arm64
create_tag:
name: Create tag
needs:
- check-changes
if: ${{ needs.check-changes.outputs.changes != '0' }}
runs-on: ubuntu-latest
outputs:
tag: ${{ steps.tag.outputs.new-tag }}
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- uses: https://gitea.t000-n.de/t.behrendt/conventional-semantic-git-tag-increment@0.0.2
id: tag
with:
token: ${{ secrets.GITEA_TOKEN }}
- run: |
git tag ${{ steps.tag.outputs.new-tag }}
git push origin ${{ steps.tag.outputs.new-tag }}
- name: Set output
run: |
docker push gitea.t000-n.de/t.behrendt/backupsidecar:${{ github.sha }}-arm64
echo "tag=${{ steps.tag.outputs.new-tag }}" >> $GITHUB_OUTPUT
manifest:
name: Create and Push Multi-Arch Manifest
needs: [build_amd64, build_arm64]
create_manifest:
name: Create manifest
needs:
- build_and_push
- create_tag
runs-on: ubuntu-latest
steps:
- name: Log in to registry
uses: docker/login-action@v3
- uses: actions/checkout@v5
- id: meta
run: |
echo REPO_NAME=$(echo ${GITHUB_REPOSITORY} | awk -F"/" '{print $2}' | tr '[:upper:]' '[:lower:]') >> $GITHUB_OUTPUT
echo REPO_VERSION=$(git describe --tags --always | sed 's/^v//') >> $GITHUB_OUTPUT
- uses: docker/login-action@v3
with:
registry: gitea.t000-n.de
username: ${{ secrets.REGISTRY_USERNAME }}
registry: ${{ env.DOCKER_REGISTRY }}
username: ${{ secrets.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_PASSWORD }}
- run: |
docker manifest create ${{ env.DOCKER_REGISTRY }}/t.behrendt/${{ steps.meta.outputs.REPO_NAME }}:${{ needs.create_tag.outputs.tag }} \
${{ env.DOCKER_REGISTRY }}/t.behrendt/${{ steps.meta.outputs.REPO_NAME }}:${{ steps.meta.outputs.REPO_VERSION }}-amd64 \
${{ env.DOCKER_REGISTRY }}/t.behrendt/${{ steps.meta.outputs.REPO_NAME }}:${{ steps.meta.outputs.REPO_VERSION }}-arm64
- name: Create and push manifest with git SHA
run: |
docker manifest create gitea.t000-n.de/t.behrendt/backupsidecar:${{ github.sha }} \
gitea.t000-n.de/t.behrendt/backupsidecar:${{ github.sha }}-amd64 \
gitea.t000-n.de/t.behrendt/backupsidecar:${{ github.sha }}-arm64
docker manifest push gitea.t000-n.de/t.behrendt/backupsidecar:${{ github.sha }}
- name: Optionally tag manifest as latest
if: ${{ inputs.latest == true }}
run: |
docker manifest create gitea.t000-n.de/t.behrendt/backupsidecar:latest \
gitea.t000-n.de/t.behrendt/backupsidecar:${{ github.sha }}-amd64 \
gitea.t000-n.de/t.behrendt/backupsidecar:${{ github.sha }}-arm64
docker manifest push gitea.t000-n.de/t.behrendt/backupsidecar:latest
docker manifest push ${{ env.DOCKER_REGISTRY }}/t.behrendt/${{ steps.meta.outputs.REPO_NAME }}:${{ needs.create_tag.outputs.tag }}

24
.gitea/workflows/ci.yaml Normal file
View File

@@ -0,0 +1,24 @@
name: CI
on:
pull_request:
jobs:
build:
name: Build Docker image
runs-on:
- ubuntu-latest
- linux_amd64
steps:
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3
- name: Build image
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile
platforms: linux/amd64
push: false
provenance: false
tags: |
backupsidecar:ci-test

View File

@@ -1,4 +1,4 @@
FROM alpine:3.21
FROM alpine:3.22
RUN apk update && apk add --no-cache \
bash \

162
README.md
View File

@@ -1,6 +1,6 @@
# BackupSidecar
BackupSidecar is a lightweight backup solution designed to run as a cron job in Kubernetes. It automates backups using Restic and supports both directory and PostgreSQL database backups. Notifications are sent via Gotify to keep you informed of backup results.
BackupSidecar is a lightweight backup and restore solution designed to run as a cron job in Kubernetes. It automates backups and restores using Restic and supports both directory and PostgreSQL database operations. Optional notifications can be sent via Gotify to keep you informed of operation results.
## Configuration
@@ -8,34 +8,53 @@ BackupSidecar is configured through environment variables. Below is a breakdown
### General Settings
These variables apply to both directory and PostgreSQL backups.
These variables apply to both backup and restore operations.
- **`OPERATION_MODE`** _(optional)_ - Defines the operation type (`backup` or `restore`). Defaults to `backup`.
- **`BACKUP_MODE`** _(optional)_ - Defines the backup type (`directory` or `postgres`). Defaults to `directory`.
- **`RESTIC_PASSWORD`** _(required)_ - The encryption password for Restic.
- **`RESTIC_REPOSITORY`** _(required)_ - The URI of the Restic repository (e.g., `rest:http://your-rest-server:8000/backup`).
- **`RESTIC_REST_USERNAME`** _(optional)_ - The username for REST server authentication.
- **`RESTIC_REST_PASSWORD`** _(optional)_ - The password for REST server authentication.
- **`GOTIFYHOST`** _(required)_ - The Gotify server URL.
- **`GOTIFYTOKEN`** _(required)_ - The API token for Gotify.
- **`GOTIFYTOPIC`** _(required)_ - The topic under which backup notifications will be sent.
- **`ENABLE_GOTIFY`** _(optional)_ - Enable Gotify notifications. Set to `true` to enable, any other value or unset disables notifications. Defaults to `true`.
- **`GOTIFYHOST`** _(required when ENABLE_GOTIFY=true)_ - The Gotify server URL.
- **`GOTIFYTOKEN`** _(required when ENABLE_GOTIFY=true)_ - The API token for Gotify.
- **`GOTIFYTOPIC`** _(required when ENABLE_GOTIFY=true)_ - The topic under which backup notifications will be sent.
### Directory Backup
### Directory Operations
When running in `directory` mode, the following variable must be set:
When running in `directory` mode, the following variables must be set:
**For Backup Operations:**
- **`SOURCEDIR`** _(required)_ - The path of the directory to be backed up.
### PostgreSQL Backup
**For Restore Operations:**
- **`RESTOREDIR`** _(required)_ - The path where files should be restored to.
- **`RESTORE_SNAPSHOT_ID`** _(optional)_ - The specific snapshot ID to restore (defaults to `latest`).
### PostgreSQL Operations
For `postgres` mode, the following database-related variables are required:
**Common Variables:**
- **`PGHOST`** _(required)_ - The hostname of the PostgreSQL server.
- **`PGDATABASE`** _(required)_ - The name of the database to back up.
- **`PGDATABASE`** _(required)_ - The name of the database.
- **`PGUSER`** _(required)_ - The PostgreSQL username.
- **`PGPORT`** _(optional)_ - The port for PostgreSQL (defaults to `5432`).
- **`PGPASSWORD`** _(optional)_ - The password for authentication. Setting this prevents interactive prompts.
**Backup-Specific Variables:**
- **`PG_DUMP_ARGS`** _(optional)_ - Additional flags for `pg_dump`.
**Restore-Specific Variables:**
- **`RESTORE_SNAPSHOT_ID`** _(optional)_ - The specific snapshot ID to restore (defaults to `latest`).
- **`PSQL_ARGS`** _(optional)_ - Additional flags for `psql` (e.g., `--single-transaction`).
## Dependencies
Ensure the following commands are available in the container:
@@ -43,10 +62,13 @@ Ensure the following commands are available in the container:
- `restic`
- `curl`
- `jq`
- `pg_dump` _(only required for `postgres` mode)_
- `pg_dump` _(only required for PostgreSQL backup operations)_
- `psql` _(only required for PostgreSQL restore operations)_
## Usage
### Backup Operations
Example Kubernetes CronJob manifest for running BackupSidecar as a cron job for directory backups in minimal configuration:
```yaml
@@ -82,6 +104,8 @@ spec:
value: "directory" # or "postgres"
- name: SOURCEDIR
value: "/data/source"
- name: ENABLE_GOTIFY
value: "true"
- name: GOTIFYHOST
value: "http://gotify.example.com"
- name: GOTIFYTOKEN
@@ -102,12 +126,116 @@ spec:
claimName: source-data-pvc
```
### Restore Operations
Example Kubernetes Job manifest for running BackupSidecar to restore a directory:
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: backupsidecar-restore
namespace: authentik
spec:
backoffLimit: 3
activeDeadlineSeconds: 600
template:
spec:
restartPolicy: OnFailure
containers:
- name: backupsidecar
image: backupsidecar:latest
env:
- name: OPERATION_MODE
value: "restore"
- name: BACKUP_MODE
value: "directory"
- name: RESTOREDIR
value: "/data/restore"
- name: RESTORE_SNAPSHOT_ID
value: "abc123def456" # optional, defaults to latest
- name: RESTIC_REPOSITORY
value: "rest:http://rest-server:8000/backup"
- name: RESTIC_PASSWORD
valueFrom:
secretKeyRef:
name: backupsidecar-secret
key: restic_password
- name: GOTIFYHOST
value: "http://gotify.example.com"
- name: GOTIFYTOKEN
valueFrom:
secretKeyRef:
name: backupsidecar-secret
key: gotify_token
- name: GOTIFYTOPIC
value: "Restore Notification"
volumeMounts:
- name: restore-data
mountPath: /data/restore
volumes:
- name: restore-data
persistentVolumeClaim:
claimName: restore-data-pvc
```
Example Kubernetes Job manifest for running BackupSidecar to restore a PostgreSQL database:
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: backupsidecar-postgres-restore
namespace: authentik
spec:
backoffLimit: 3
activeDeadlineSeconds: 600
template:
spec:
restartPolicy: OnFailure
containers:
- name: backupsidecar
image: backupsidecar:latest
env:
- name: OPERATION_MODE
value: "restore"
- name: BACKUP_MODE
value: "postgres"
- name: PGHOST
value: "postgres.example.com"
- name: PGDATABASE
value: "mydatabase"
- name: PGUSER
value: "myuser"
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: password
- name: PGPORT
value: "5432"
- name: RESTORE_SNAPSHOT_ID
value: "abc123def456" # optional, defaults to latest
- name: PSQL_ARGS
value: "--single-transaction" # optional
- name: RESTIC_REPOSITORY
value: "rest:http://rest-server:8000/backup"
- name: RESTIC_PASSWORD
valueFrom:
secretKeyRef:
name: backupsidecar-secret
key: restic_password
- name: GOTIFYHOST
value: "http://gotify.example.com"
- name: GOTIFYTOKEN
valueFrom:
secretKeyRef:
name: backupsidecar-secret
key: gotify_token
- name: GOTIFYTOPIC
value: "Database Restore Notification"
```
## Notifications
The script sends success or failure notifications via Gotify.
Example success notification:
```
Backup successful. Snapshot 56ff6a909a44e01f67d2d88f9a76aa713d437809d7ed14a2361e28893f38befb: files new: 1, files changed: 0, data added: 1019 bytes in 0.277535184 sec
```
The script can send success or failure notifications via Gotify when enabled. To enable notifications, set `ENABLE_GOTIFY=true` and provide the required Gotify configuration variables (`GOTIFYHOST`, `GOTIFYTOKEN`, `GOTIFYTOPIC`). When notifications are disabled, backup status messages are still logged to the console.

354
TESTING.md Normal file
View File

@@ -0,0 +1,354 @@
# Backup Script Testing Guide
This document provides step-by-step instructions for testing the backup script locally using Docker Compose with a Restic server and PostgreSQL database.
## Prerequisites
- Docker and Docker Compose installed
- The backup script (`src/backup.sh`) and its dependencies (restic, curl, jq, pg_dump, psql)
- Basic understanding of bash and PostgreSQL
## Setup
### 1. Start the Test Environment
```bash
# Start the services
docker-compose up -d
# Wait for services to be ready (about 10-15 seconds)
sleep 15
# Verify services are running
docker-compose ps
```
### 3. Initialize Restic Repositories
Before running any backup operations, you need to initialize the restic repositories:
```bash
# Initialize repository for directory backups
RESTIC_PASSWORD="testpassword123" restic -r "rest:http://localhost:8000/test-backup" init
# Initialize repository for PostgreSQL backups
RESTIC_PASSWORD="testpassword123" restic -r "rest:http://localhost:8000/postgres-backup" init
```
**Note**: The repositories only need to be initialized once. If you recreate the restic server container, you'll need to reinitialize the repositories.
## Directory Backup Testing
This section tests the directory backup and restore functionality.
### 1. Prepare Test Data
```bash
# Create a test directory with sample files
mkdir -p test_data
cd test_data
# Create various types of files
echo "This is a text file" > sample.txt
echo '{"name": "test", "value": 123}' > data.json
mkdir -p subdir
echo "Nested file content" > subdir/nested.txt
echo "Binary data" | base64 > binary.dat
# Create some larger files for testing
dd if=/dev/urandom of=large_file.bin bs=1M count=10 2>/dev/null
# Go back to project root
cd ..
```
### 2. Perform Directory Backup
```bash
# Set environment variables for directory backup
export OPERATION_MODE="backup"
export BACKUP_MODE="directory"
export RESTIC_PASSWORD="testpassword123"
export RESTIC_REPOSITORY="rest:http://localhost:8000/test-backup"
export SOURCEDIR="/tmp/test-data"
export ENABLE_GOTIFY="false"
# Run the backup
./src/backup.sh
```
### 3. Verify Backup
```bash
# List snapshots
restic -r "rest:http://localhost:8000/test-backup" snapshots --password-file <(echo "testpassword123")
# Check backup contents
restic -r "rest:http://localhost:8000/test-backup" ls latest --password-file <(echo "testpassword123")
```
### 4. Perform Directory Restore
```bash
# Create restore directory
RESTOR_DIR="/tmp/restored_data"
mkdir -p $RESTOR_DIR
# Set environment variables for directory restore
export OPERATION_MODE="restore"
export RESTOREDIR=$RESTOR_DIR
export RESTORE_SNAPSHOT_ID="latest"
# Run the restore
./src/backup.sh
```
### 5. Verify Directory Restore
```bash
# Compare original and restored directories
diff -r /tmp/test-data $RESTOR_DIR
# Check file contents
echo "Original file:"
cat test_data/sample.txt
echo "Restored file:"
cat restored_data/sample.txt
# Verify binary file integrity
md5sum test_data/large_file.bin
md5sum restored_data/large_file.bin
# Check directory structure
tree test_data
tree restored_data
```
### 6. Cleanup Directory Test
```bash
# Remove test directories
rm -rf test_data restored_data
```
## PostgreSQL Backup Testing
This section tests the PostgreSQL database backup and restore functionality.
### 1. Generate Test Data
```bash
# Generate test data in PostgreSQL
./generate_test_data.sh
```
### 2. Verify Initial Data
```bash
# Check that data exists
docker exec postgres-test psql -U testuser -d testdb -c "
SELECT 'customers' as table_name, COUNT(*) as row_count FROM customers
UNION ALL
SELECT 'orders' as table_name, COUNT(*) as row_count FROM orders;
"
```
### 3. Perform PostgreSQL Backup
```bash
# Set environment variables for PostgreSQL backup
export OPERATION_MODE="backup"
export BACKUP_MODE="postgres"
export RESTIC_PASSWORD="testpassword123"
export RESTIC_REPOSITORY="rest:http://localhost:8000/postgres-backup"
export PGHOST="localhost"
export PGPORT="5432"
export PGDATABASE="testdb"
export PGUSER="testuser"
export PGPASSWORD="testpass"
export ENABLE_GOTIFY="false"
# Run the backup
./src/backup.sh
```
### 4. Verify PostgreSQL Backup
```bash
# List snapshots
restic -r "rest:http://localhost:8000/postgres-backup" snapshots --password-file <(echo "testpassword123")
# Check backup contents
restic -r "rest:http://localhost:8000/postgres-backup" ls latest --password-file <(echo "testpassword123")
```
### 5. Clear the Database
```bash
# Drop all tables to simulate data loss
docker exec postgres-test psql -U testuser -d testdb -c "
DROP TABLE IF EXISTS orders CASCADE;
DROP TABLE IF EXISTS customers CASCADE;
"
# Verify database is empty
docker exec postgres-test psql -U testuser -d testdb -c "
SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public';
"
```
### 6. Perform PostgreSQL Restore
```bash
# Set environment variables for PostgreSQL restore
export OPERATION_MODE="restore"
export RESTORE_SNAPSHOT_ID="latest"
# Run the restore
./src/backup.sh
```
### 7. Verify PostgreSQL Restore
```bash
# Check that data has been restored
docker exec postgres-test psql -U testuser -d testdb -c "
SELECT 'customers' as table_name, COUNT(*) as row_count FROM customers
UNION ALL
SELECT 'orders' as table_name, COUNT(*) as row_count FROM orders;
"
# Verify data integrity
docker exec postgres-test psql -U testuser -d testdb -c "
SELECT
c.name as customer_name,
COUNT(o.id) as order_count,
SUM(o.price * o.quantity) as total_spent
FROM customers c
LEFT JOIN orders o ON c.id = o.customer_id
GROUP BY c.id, c.name
ORDER BY total_spent DESC
LIMIT 5;
"
# Check foreign key constraints
docker exec postgres-test psql -U testuser -d testdb -c "
SELECT
tc.constraint_name,
tc.table_name,
kcu.column_name,
ccu.table_name AS foreign_table_name,
ccu.column_name AS foreign_column_name
FROM information_schema.table_constraints AS tc
JOIN information_schema.key_column_usage AS kcu
ON tc.constraint_name = kcu.constraint_name
AND tc.table_schema = kcu.table_schema
JOIN information_schema.constraint_column_usage AS ccu
ON ccu.constraint_name = tc.constraint_name
AND ccu.table_schema = tc.table_schema
WHERE tc.constraint_type = 'FOREIGN KEY'
AND tc.table_name IN ('customers', 'orders');
"
```
## Advanced Testing Scenarios
### 1. Test with Different Snapshot IDs
```bash
# List all snapshots
restic -r "rest:http://localhost:8000/test-backup" snapshots --password-file <(echo "testpassword123")
# Restore a specific snapshot
export RESTORE_SNAPSHOT_ID="<snapshot-id>"
./src/backup.sh
```
### 2. Test Error Handling
```bash
# Test with invalid repository
export RESTIC_REPOSITORY="rest:http://localhost:8000/nonexistent"
./src/backup.sh
# Test with wrong password
export RESTIC_PASSWORD="wrongpassword"
./src/backup.sh
```
### 3. Test with Gotify Notifications (Optional)
```bash
# If you have a Gotify server running
export ENABLE_GOTIFY="true"
export GOTIFYHOST="http://your-gotify-server:80"
export GOTIFYTOKEN="your-token"
export GOTIFYTOPIC="Backup Test"
./src/backup.sh
```
## Cleanup
### 1. Stop Services
```bash
# Stop and remove containers
docker-compose down
# Remove volumes (optional - this will delete all data)
docker-compose down -v
```
### 2. Clean Up Test Files
```bash
# Remove any remaining test files
rm -rf test_data restored_data
```
## Troubleshooting
### Common Issues
1. **Connection refused to restic server**: Wait a bit longer for the container to start up
2. **PostgreSQL connection failed**: Ensure the database container is fully initialized
3. **Permission denied**: Make sure the backup script is executable (`chmod +x src/backup.sh`)
4. **Restic repository not found**: Check that the repository URL is correct and the server is running
5. **Script exits early with no output**: The restic repository hasn't been initialized yet. Run the initialization commands in step 3 above.
### Debug Commands
```bash
# Check container logs
docker-compose logs restic-server
docker-compose logs postgres
# Test restic connectivity
restic -r "rest:http://localhost:8000/test-backup" snapshots --password-file <(echo "testpassword123")
# Test PostgreSQL connectivity
docker exec postgres-test psql -U testuser -d testdb -c "SELECT 1;"
```
## Expected Results
### Directory Backup Test
- ✅ Backup completes successfully
- ✅ Files are backed up to restic repository
- ✅ Restore completes successfully
- ✅ Restored files match original files exactly
- ✅ Directory structure is preserved
### PostgreSQL Backup Test
- ✅ Database backup completes successfully
- ✅ Database dump is backed up to restic repository
- ✅ Database can be cleared successfully
- ✅ Database restore completes successfully
- ✅ All data is restored correctly
- ✅ Foreign key relationships are maintained
- ✅ Data integrity is preserved
This testing procedure ensures that both directory and PostgreSQL backup/restore functionality works correctly and can be used as a foundation for automated testing in CI/CD pipelines.

19
compose.yaml Normal file
View File

@@ -0,0 +1,19 @@
services:
restic-server:
image: restic/rest-server:latest
container_name: restic-server
ports:
- "8000:8000"
command: rest-server --no-auth --path /data
restart: unless-stopped
postgres:
image: postgres:17
container_name: postgres-test
ports:
- "5432:5432"
environment:
POSTGRES_DB: testdb
POSTGRES_USER: testuser
POSTGRES_PASSWORD: testpass
restart: unless-stopped

51
generate_dir_test_data.sh Executable file
View File

@@ -0,0 +1,51 @@
#!/bin/bash
set -euo pipefail
# Script to generate test data for Directory backup testing
# This script creates a few directories with a few files and directories in each and populates it with test data
# Create base test directory
TEST_DIR="/tmp/test-data"
echo "Creating test directory structure in $TEST_DIR..."
# Remove existing test directory if it exists
rm -rf "$TEST_DIR"
mkdir -p "$TEST_DIR"
# Create various subdirectories
mkdir -p "$TEST_DIR/documents/reports"
mkdir -p "$TEST_DIR/documents/contracts"
mkdir -p "$TEST_DIR/data/logs"
mkdir -p "$TEST_DIR/data/backups"
# Create text files with content
echo "This is the annual report for 2023" > "$TEST_DIR/documents/reports/annual_2023.txt"
echo "Q4 financial summary" > "$TEST_DIR/documents/reports/q4_summary.txt"
echo "Contract terms and conditions" > "$TEST_DIR/documents/contracts/agreement.txt"
# Create JSON files
cat << 'EOF' > "$TEST_DIR/data/config.json"
{
"app_name": "TestApp",
"version": "1.0.0",
"settings": {
"debug": true,
"max_retries": 3,
"timeout": 30
}
}
EOF
# Create some log files
for i in {1..3}; do
echo "$(date) - Log entry $i" >> "$TEST_DIR/data/logs/app.log"
echo "$(date) - Error $i: Sample error message" >> "$TEST_DIR/data/logs/error.log"
done
# Create symbolic links
ln -s "../reports/annual_2023.txt" "$TEST_DIR/documents/contracts/report_link"
ln -s "../../data/config.json" "$TEST_DIR/documents/reports/config_link"
echo "Test data generation completed successfully!"
echo "Created directory structure:"
tree "$TEST_DIR"

124
generate_sql_test_data.sh Executable file
View File

@@ -0,0 +1,124 @@
#!/bin/bash
set -euo pipefail
# Script to generate test data for PostgreSQL backup testing
# This script creates two tables with a foreign key relationship and populates them with test data
# Database connection parameters
PGHOST="${PGHOST:-localhost}"
PGPORT="${PGPORT:-5432}"
PGDATABASE="${PGDATABASE:-testdb}"
PGUSER="${PGUSER:-testuser}"
PGPASSWORD="${PGPASSWORD:-testpass}"
# Export password for psql
export PGPASSWORD
echo "Generating test data for PostgreSQL database..."
# Create tables
echo "Creating tables..."
psql -h "$PGHOST" -p "$PGPORT" -U "$PGUSER" -d "$PGDATABASE" << 'EOF'
-- Drop tables if they exist
DROP TABLE IF EXISTS orders CASCADE;
DROP TABLE IF EXISTS customers CASCADE;
-- Create customers table
CREATE TABLE customers (
id SERIAL PRIMARY KEY,
name VARCHAR(100) NOT NULL,
email VARCHAR(100) UNIQUE NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Create orders table with foreign key to customers
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id INTEGER NOT NULL REFERENCES customers(id) ON DELETE CASCADE,
product_name VARCHAR(100) NOT NULL,
quantity INTEGER NOT NULL CHECK (quantity > 0),
price DECIMAL(10,2) NOT NULL CHECK (price >= 0),
order_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Create indexes for better performance
CREATE INDEX idx_orders_customer_id ON orders(customer_id);
CREATE INDEX idx_customers_email ON customers(email);
CREATE INDEX idx_orders_order_date ON orders(order_date);
EOF
# Insert test data
echo "Inserting test data..."
psql -h "$PGHOST" -p "$PGPORT" -U "$PGUSER" -d "$PGDATABASE" << 'EOF'
-- Insert customers
INSERT INTO customers (name, email) VALUES
('John Doe', 'john.doe@example.com'),
('Jane Smith', 'jane.smith@example.com'),
('Bob Johnson', 'bob.johnson@example.com'),
('Alice Brown', 'alice.brown@example.com'),
('Charlie Wilson', 'charlie.wilson@example.com'),
('Diana Davis', 'diana.davis@example.com'),
('Eve Miller', 'eve.miller@example.com'),
('Frank Garcia', 'frank.garcia@example.com'),
('Grace Lee', 'grace.lee@example.com'),
('Henry Taylor', 'henry.taylor@example.com');
-- Insert orders
INSERT INTO orders (customer_id, product_name, quantity, price) VALUES
(1, 'Laptop', 1, 999.99),
(1, 'Mouse', 2, 25.50),
(2, 'Keyboard', 1, 75.00),
(2, 'Monitor', 1, 299.99),
(3, 'Headphones', 1, 150.00),
(3, 'Webcam', 1, 89.99),
(4, 'Tablet', 1, 399.99),
(4, 'Stylus', 1, 49.99),
(5, 'Smartphone', 1, 699.99),
(5, 'Case', 1, 19.99),
(6, 'Desktop', 1, 1299.99),
(6, 'RAM', 2, 79.99),
(7, 'SSD', 1, 199.99),
(7, 'Graphics Card', 1, 599.99),
(8, 'Motherboard', 1, 199.99),
(8, 'CPU', 1, 399.99),
(9, 'Power Supply', 1, 149.99),
(9, 'Cooling Fan', 2, 29.99),
(10, 'Cable Set', 1, 39.99),
(10, 'USB Hub', 1, 24.99);
EOF
# Verify data
echo "Verifying test data..."
psql -h "$PGHOST" -p "$PGPORT" -U "$PGUSER" -d "$PGDATABASE" << 'EOF'
-- Show table counts
SELECT 'customers' as table_name, COUNT(*) as row_count FROM customers
UNION ALL
SELECT 'orders' as table_name, COUNT(*) as row_count FROM orders;
-- Show sample data
SELECT 'Sample customers:' as info;
SELECT id, name, email FROM customers LIMIT 5;
SELECT 'Sample orders:' as info;
SELECT o.id, c.name as customer_name, o.product_name, o.quantity, o.price
FROM orders o
JOIN customers c ON o.customer_id = c.id
LIMIT 5;
-- Show foreign key relationship
SELECT 'Foreign key relationship check:' as info;
SELECT
c.name as customer_name,
COUNT(o.id) as order_count,
SUM(o.price * o.quantity) as total_spent
FROM customers c
LEFT JOIN orders o ON c.id = o.customer_id
GROUP BY c.id, c.name
ORDER BY total_spent DESC;
EOF
echo "Test data generation completed successfully!"
echo "Database contains:"
echo "- 10 customers"
echo "- 20 orders with foreign key relationships"
echo "- Various data types and constraints"

3
renovate.json Normal file
View File

@@ -0,0 +1,3 @@
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json"
}

278
src/backup.sh Normal file → Executable file
View File

@@ -1,71 +1,6 @@
#!/bin/bash
set -euo pipefail
#######################################
# Determine backup mode from the environment only.
# Valid values: "directory" or "postgres".
# Default to "directory" if not provided.
#######################################
BACKUP_MODE="${BACKUP_MODE:-directory}"
#######################################
# Check for required external commands.
#######################################
REQUIRED_CMDS=(restic curl jq)
if [ "$BACKUP_MODE" = "postgres" ]; then
REQUIRED_CMDS+=(pg_dump)
fi
for cmd in "${REQUIRED_CMDS[@]}"; do
if ! command -v "$cmd" &>/dev/null; then
echo "Error: Required command '$cmd' is not installed." >&2
exit 1
fi
done
#######################################
# Validate common required environment variables.
#######################################
# Gotify notification settings.
: "${GOTIFYHOST:?Environment variable GOTIFYHOST is not set}"
: "${GOTIFYTOKEN:?Environment variable GOTIFYTOKEN is not set}"
: "${GOTIFYTOPIC:?Environment variable GOTIFYTOPIC is not set}"
# Restic encryption password.
: "${RESTIC_PASSWORD:?Environment variable RESTIC_PASSWORD is not set}"
# Use the repository URI directly from the environment.
# Example: export RESTIC_REPOSITORY="rest:http://your-rest-server:8000/backup"
: "${RESTIC_REPOSITORY:?Environment variable RESTIC_REPOSITORY is not set}"
#######################################
# Validate mode-specific environment variables.
#######################################
case "$BACKUP_MODE" in
directory)
: "${SOURCEDIR:?Environment variable SOURCEDIR is not set (required for directory backup mode)}"
;;
postgres)
: "${PGHOST:?Environment variable PGHOST is not set (required for PostgreSQL backup mode)}"
: "${PGDATABASE:?Environment variable PGDATABASE is not set (required for PostgreSQL backup mode)}"
: "${PGUSER:?Environment variable PGUSER is not set (required for PostgreSQL backup mode)}"
# Optional: default PGPORT to 5432.
: "${PGPORT:=5432}"
if [ -z "${PGPASSWORD:-}" ]; then
echo "Warning: Environment variable PGPASSWORD is not set. pg_dump may fail if authentication is required."
fi
;;
*)
echo "Error: Unknown backup mode '$BACKUP_MODE'. Valid modes are 'directory' and 'postgres'." >&2
exit 1
;;
esac
#######################################
# Build the Gotify URL.
#######################################
GOTIFYURL="${GOTIFYHOST}/message?token=${GOTIFYTOKEN}"
#######################################
# Date format for logging.
#######################################
@@ -80,6 +15,111 @@ log() {
echo "$(date +"$LOG_DATE_FORMAT") - $*"
}
#######################################
# Determine operation mode from the environment only.
# Valid values: "backup" or "restore".
# Default to "backup" if not provided.
#######################################
OPERATION_MODE="${OPERATION_MODE:-backup}"
#######################################
# Determine backup mode from the environment only.
# Valid values: "directory" or "postgres".
# Default to "directory" if not provided.
#######################################
BACKUP_MODE="${BACKUP_MODE:-directory}"
#######################################
# Check for required external commands.
#######################################
REQUIRED_CMDS=(restic curl jq)
if [ "$BACKUP_MODE" = "postgres" ]; then
if [ "$OPERATION_MODE" = "backup" ]; then
REQUIRED_CMDS+=(pg_dump)
elif [ "$OPERATION_MODE" = "restore" ]; then
REQUIRED_CMDS+=(psql)
fi
fi
for cmd in "${REQUIRED_CMDS[@]}"; do
if ! command -v "$cmd" &>/dev/null; then
log "Error: Required command '$cmd' is not installed."
exit 1
fi
done
#######################################
# Validate common required environment variables.
#######################################
# Gotify notification settings (optional).
# Set ENABLE_GOTIFY to "true" to enable notifications, any other value or unset disables them.
ENABLE_GOTIFY="${ENABLE_GOTIFY:-true}"
if [ "$ENABLE_GOTIFY" = "true" ]; then
: "${GOTIFYHOST:?Environment variable GOTIFYHOST is not set (required when ENABLE_GOTIFY=true)}"
: "${GOTIFYTOKEN:?Environment variable GOTIFYTOKEN is not set (required when ENABLE_GOTIFY=true)}"
: "${GOTIFYTOPIC:?Environment variable GOTIFYTOPIC is not set (required when ENABLE_GOTIFY=true)}"
else
log "Gotify notifications disabled. Backup status will be logged to console only."
fi
# Restic encryption password.
: "${RESTIC_PASSWORD:?Environment variable RESTIC_PASSWORD is not set}"
# Use the repository URI directly from the environment.
# Example: export RESTIC_REPOSITORY="rest:http://your-rest-server:8000/backup"
: "${RESTIC_REPOSITORY:?Environment variable RESTIC_REPOSITORY is not set}"
#######################################
# Validate operation mode.
#######################################
case "$OPERATION_MODE" in
backup|restore)
;;
*)
echo "Error: Unknown operation mode '$OPERATION_MODE'. Valid modes are 'backup' and 'restore'." >&2
exit 1
;;
esac
#######################################
# Validate mode-specific environment variables.
#######################################
case "$BACKUP_MODE" in
directory)
if [ "$OPERATION_MODE" = "backup" ]; then
: "${SOURCEDIR:?Environment variable SOURCEDIR is not set (required for directory backup mode)}"
elif [ "$OPERATION_MODE" = "restore" ]; then
: "${RESTOREDIR:?Environment variable RESTOREDIR is not set (required for directory restore mode)}"
fi
;;
postgres)
: "${PGHOST:?Environment variable PGHOST is not set (required for PostgreSQL mode)}"
: "${PGDATABASE:?Environment variable PGDATABASE is not set (required for PostgreSQL mode)}"
: "${PGUSER:?Environment variable PGUSER is not set (required for PostgreSQL mode)}"
# Optional: default PGPORT to 5432.
: "${PGPORT:=5432}"
if [ -z "${PGPASSWORD:-}" ]; then
if [ "$OPERATION_MODE" = "backup" ]; then
echo "Warning: Environment variable PGPASSWORD is not set. pg_dump may fail if authentication is required."
elif [ "$OPERATION_MODE" = "restore" ]; then
echo "Warning: Environment variable PGPASSWORD is not set. psql may fail if authentication is required."
fi
fi
;;
*)
echo "Error: Unknown backup mode '$BACKUP_MODE'. Valid modes are 'directory' and 'postgres'." >&2
exit 1
;;
esac
#######################################
# Build the Gotify URL (only if Gotify is enabled).
#######################################
if [ "$ENABLE_GOTIFY" = "true" ]; then
GOTIFYURL="${GOTIFYHOST}/message?token=${GOTIFYTOKEN}"
fi
#######################################
# Send a notification via Gotify.
# Arguments:
@@ -87,6 +127,13 @@ log() {
#######################################
send_notification() {
local message="$1"
# Only send notification if Gotify is enabled
if [ "$ENABLE_GOTIFY" != "true" ]; then
log "$message"
return 0
fi
if ! curl -s -X POST "$GOTIFYURL" -F "title=${GOTIFYTOPIC}" -F "message=${message}" >/dev/null; then
log "Warning: Failed to send notification with message: ${message}"
fi
@@ -157,6 +204,89 @@ backup_postgres() {
run_restic_backup "${TEMP_BACKUP_DIR}"
}
#######################################
# Run the restore using restic.
# Arguments:
# $1 - The target directory to restore to.
# $2 - Optional snapshot ID to restore (defaults to latest).
#######################################
run_restic_restore() {
local target_dir="$1"
local snapshot_id="$2"
log "Starting restore from repository ${RESTIC_REPOSITORY} to '${target_dir}'"
log "Using snapshot: ${snapshot_id}"
# Create target directory if it doesn't exist
mkdir -p "${target_dir}"
# Capture both stdout and stderr in a variable
restore_output=$(restic -r "${RESTIC_REPOSITORY}" restore "${snapshot_id}" --target "${target_dir}" --no-cache --json --verbose 2>&1)
# Optionally, also print the output to the console:
echo "$restore_output"
# Parse the JSON lines output for the summary message
summary=$(echo "$restore_output" | jq -r 'select(.message_type=="summary") | "Restore completed: " + (.files_restored|tostring) + " files restored, " + (.bytes_restored|tostring) + " bytes in " + (.total_duration|tostring) + " sec"' 2>/dev/null || echo "Restore completed")
# Check exit code of restic restore
if [ $? -eq 0 ]; then
msg="Restore successful. $summary"
log "$msg"
send_notification "$msg"
else
exit_code=$?
msg="Restore failed with error code ${exit_code}. $restore_output"
log "$msg"
send_notification "$msg"
exit "$exit_code"
fi
}
#######################################
# Restore a directory (regular mode).
#######################################
restore_directory() {
local snapshot_id="${RESTORE_SNAPSHOT_ID:-latest}"
run_restic_restore "${RESTOREDIR}" "${snapshot_id}"
}
#######################################
# Restore a PostgreSQL database.
# Restores the database dump from the backup and applies it to the database.
#######################################
restore_postgres() {
local snapshot_id="${RESTORE_SNAPSHOT_ID:-latest}"
log "Starting PostgreSQL restore for database '${PGDATABASE}' on host '${PGHOST}'"
# Create a temporary directory for the restore.
TEMP_RESTORE_DIR=$(mktemp -d)
log "Created temporary directory: ${TEMP_RESTORE_DIR}"
# Restore the backup to the temporary directory
run_restic_restore "${TEMP_RESTORE_DIR}" "${snapshot_id}"
local dump_file="${TEMP_RESTORE_DIR}/dump.sql"
if [ ! -f "${dump_file}" ]; then
local msg="PostgreSQL restore failed. Database dump file not found at ${dump_file}"
log "$msg"
send_notification "$msg"
exit 1
fi
log "Restoring PostgreSQL database from ${dump_file}..."
if psql -h "${PGHOST}" -p "${PGPORT}" -U "${PGUSER}" -d "${PGDATABASE}" ${PSQL_ARGS:-} < "${dump_file}"; then
local msg="PostgreSQL database restored successfully"
log "$msg"
send_notification "$msg"
else
local exit_code=$?
local msg="PostgreSQL restore failed with error code ${exit_code}"
log "$msg"
send_notification "$msg"
exit "$exit_code"
fi
}
#######################################
# Cleanup temporary resources.
#######################################
@@ -165,6 +295,10 @@ cleanup() {
rm -rf "${TEMP_BACKUP_DIR}"
log "Removed temporary directory ${TEMP_BACKUP_DIR}"
fi
if [ -n "${TEMP_RESTORE_DIR:-}" ] && [ -d "${TEMP_RESTORE_DIR}" ]; then
rm -rf "${TEMP_RESTORE_DIR}"
log "Removed temporary directory ${TEMP_RESTORE_DIR}"
fi
}
trap cleanup EXIT
@@ -172,6 +306,8 @@ trap cleanup EXIT
# Main routine.
#######################################
main() {
case "$OPERATION_MODE" in
backup)
case "$BACKUP_MODE" in
directory)
backup_directory
@@ -180,6 +316,18 @@ main() {
backup_postgres
;;
esac
;;
restore)
case "$BACKUP_MODE" in
directory)
restore_directory
;;
postgres)
restore_postgres
;;
esac
;;
esac
}
# Trap termination signals to log and exit cleanly.