sync

ci: add bash syntax check (#18 )
Reviewed-on: #18 Co-authored-by: Timo Behrendt <t.behrendt@t00n.de> Co-committed-by: Timo Behrendt <t.behrendt@t00n.de>
2025-09-06 08:24:06 +02:00 · 2025-09-04 20:48:13 +02:00 · 2025-09-04 20:36:40 +02:00
8 changed files with 248 additions and 551 deletions
--- a/.gitea/workflows/ci.yaml
+++ b/.gitea/workflows/ci.yaml
@@ -22,3 +22,12 @@ jobs:
          provenance: false
          tags: |
            backupsidecar:ci-test
+
+  check-syntax:
+    name: Check syntax
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - name: Check syntax
+        run: |
+          bash -n src/backup.sh
--- a/4
+++ b/4
@@ -7,6 +7,10 @@ RUN apk update && apk add --no-cache \
    postgresql-client \
    jq

+RUN curl -O https://dl.min.io/client/mc/release/linux-amd64/mc \
+    && chmod +x mc \
+    && mv mc /usr/local/bin/
+
 WORKDIR /app

 COPY src/backup.sh /app/backup.sh
--- a/README.md
+++ b/README.md
@@ -11,7 +11,7 @@ BackupSidecar is configured through environment variables. Below is a breakdown
 These variables apply to both backup and restore operations.

 - **`OPERATION_MODE`** _(optional)_ - Defines the operation type (`backup` or `restore`). Defaults to `backup`.
- **`BACKUP_MODE`** _(optional)_ - Defines the backup type (`directory` or `postgres`). Defaults to `directory`.
+- **`BACKUP_MODE`** _(optional)_ - Defines the backup type (`directory`, `postgres`, or `s3`). Defaults to `directory`.
 - **`RESTIC_PASSWORD`** _(required)_ - The encryption password for Restic.
 - **`RESTIC_REPOSITORY`** _(required)_ - The URI of the Restic repository (e.g., `rest:http://your-rest-server:8000/backup`).
 - **`RESTIC_REST_USERNAME`** _(optional)_ - The username for REST server authentication.
@@ -55,6 +55,22 @@ For `postgres` mode, the following database-related variables are required:
 - **`RESTORE_SNAPSHOT_ID`** _(optional)_ - The specific snapshot ID to restore (defaults to `latest`).
 - **`PSQL_ARGS`** _(optional)_ - Additional flags for `psql` (e.g., `--single-transaction`).

+### S3 Operations
+
+For `s3` mode, the following S3-related variables are required:
+
+**Common Variables:**
+
+- **`S3_BUCKET`** _(required)_ - The name of the S3 bucket to backup/restore.
+- **`S3_ENDPOINT`** _(required)_ - The S3 endpoint URL (e.g., `http://minio:9000` for MinIO).
+- **`MINIO_ACCESS_KEY`** _(required)_ - The MinIO access key for S3 authentication.
+- **`MINIO_SECRET_KEY`** _(required)_ - The MinIO secret key for S3 authentication.
+- **`S3_PREFIX`** _(optional)_ - Optional path prefix within the bucket to backup/restore.
+
+**Restore-Specific Variables:**
+
+- **`RESTORE_SNAPSHOT_ID`** _(optional)_ - The specific snapshot ID to restore (defaults to `latest`).
+
 ## Dependencies

 Ensure the following commands are available in the container:
@@ -64,6 +80,7 @@ Ensure the following commands are available in the container:
 - `jq`
 - `pg_dump` _(only required for PostgreSQL backup operations)_
 - `psql` _(only required for PostgreSQL restore operations)_
+- `mc` _(only required for S3 operations)_

 ## Usage

@@ -236,6 +253,120 @@ spec:
              value: "Database Restore Notification"
 ```

+Example Kubernetes Job manifest for running BackupSidecar to backup an S3 bucket:
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: backupsidecar-s3-backup
+  namespace: authentik
+spec:
+  backoffLimit: 3
+  activeDeadlineSeconds: 600
+  template:
+    spec:
+      restartPolicy: OnFailure
+      containers:
+        - name: backupsidecar
+          image: backupsidecar:latest
+          env:
+            - name: OPERATION_MODE
+              value: "backup"
+            - name: BACKUP_MODE
+              value: "s3"
+            - name: S3_BUCKET
+              value: "my-bucket"
+            - name: S3_ENDPOINT
+              value: "http://minio:9000"
+            - name: S3_PREFIX
+              value: "data" # optional
+            - name: MINIO_ACCESS_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: minio-secret
+                  key: access_key
+            - name: MINIO_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: minio-secret
+                  key: secret_key
+            - name: RESTIC_REPOSITORY
+              value: "rest:http://rest-server:8000/backup"
+            - name: RESTIC_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: backupsidecar-secret
+                  key: restic_password
+            - name: GOTIFYHOST
+              value: "http://gotify.example.com"
+            - name: GOTIFYTOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: backupsidecar-secret
+                  key: gotify_token
+            - name: GOTIFYTOPIC
+              value: "S3 Backup Notification"
+```
+
+Example Kubernetes Job manifest for running BackupSidecar to restore an S3 bucket:
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: backupsidecar-s3-restore
+  namespace: authentik
+spec:
+  backoffLimit: 3
+  activeDeadlineSeconds: 600
+  template:
+    spec:
+      restartPolicy: OnFailure
+      containers:
+        - name: backupsidecar
+          image: backupsidecar:latest
+          env:
+            - name: OPERATION_MODE
+              value: "restore"
+            - name: BACKUP_MODE
+              value: "s3"
+            - name: S3_BUCKET
+              value: "my-bucket"
+            - name: S3_ENDPOINT
+              value: "http://minio:9000"
+            - name: S3_PREFIX
+              value: "data" # optional
+            - name: RESTORE_SNAPSHOT_ID
+              value: "abc123def456" # optional, defaults to latest
+            - name: MINIO_ACCESS_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: minio-secret
+                  key: access_key
+            - name: MINIO_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: minio-secret
+                  key: secret_key
+            - name: RESTIC_REPOSITORY
+              value: "rest:http://rest-server:8000/backup"
+            - name: RESTIC_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: backupsidecar-secret
+                  key: restic_password
+            - name: GOTIFYHOST
+              value: "http://gotify.example.com"
+            - name: GOTIFYTOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: backupsidecar-secret
+                  key: gotify_token
+            - name: GOTIFYTOPIC
+              value: "S3 Restore Notification"
+```
+
 ## Notifications

 The script can send success or failure notifications via Gotify when enabled. To enable notifications, set `ENABLE_GOTIFY=true` and provide the required Gotify configuration variables (`GOTIFYHOST`, `GOTIFYTOKEN`, `GOTIFYTOPIC`). When notifications are disabled, backup status messages are still logged to the console.
--- a/TESTING.md
+++ b/TESTING.md
@@ -1,354 +0,0 @@
-# Backup Script Testing Guide
-
-This document provides step-by-step instructions for testing the backup script locally using Docker Compose with a Restic server and PostgreSQL database.
-
-## Prerequisites
-
- Docker and Docker Compose installed
- The backup script (`src/backup.sh`) and its dependencies (restic, curl, jq, pg_dump, psql)
- Basic understanding of bash and PostgreSQL
-
-## Setup
-
-### 1. Start the Test Environment
-
-```bash
-# Start the services
-docker-compose up -d
-
-# Wait for services to be ready (about 10-15 seconds)
-sleep 15
-
-# Verify services are running
-docker-compose ps
-```
-
-### 3. Initialize Restic Repositories
-
-Before running any backup operations, you need to initialize the restic repositories:
-
-```bash
-# Initialize repository for directory backups
-RESTIC_PASSWORD="testpassword123" restic -r "rest:http://localhost:8000/test-backup" init
-
-# Initialize repository for PostgreSQL backups
-RESTIC_PASSWORD="testpassword123" restic -r "rest:http://localhost:8000/postgres-backup" init
-```
-
-**Note**: The repositories only need to be initialized once. If you recreate the restic server container, you'll need to reinitialize the repositories.
-
-## Directory Backup Testing
-
-This section tests the directory backup and restore functionality.
-
-### 1. Prepare Test Data
-
-```bash
-# Create a test directory with sample files
-mkdir -p test_data
-cd test_data
-
-# Create various types of files
-echo "This is a text file" > sample.txt
-echo '{"name": "test", "value": 123}' > data.json
-mkdir -p subdir
-echo "Nested file content" > subdir/nested.txt
-echo "Binary data" | base64 > binary.dat
-
-# Create some larger files for testing
-dd if=/dev/urandom of=large_file.bin bs=1M count=10 2>/dev/null
-
-# Go back to project root
-cd ..
-```
-
-### 2. Perform Directory Backup
-
-```bash
-# Set environment variables for directory backup
-export OPERATION_MODE="backup"
-export BACKUP_MODE="directory"
-export RESTIC_PASSWORD="testpassword123"
-export RESTIC_REPOSITORY="rest:http://localhost:8000/test-backup"
-export SOURCEDIR="/tmp/test-data"
-export ENABLE_GOTIFY="false"
-
-# Run the backup
-./src/backup.sh
-```
-
-### 3. Verify Backup
-
-```bash
-# List snapshots
-restic -r "rest:http://localhost:8000/test-backup" snapshots --password-file <(echo "testpassword123")
-
-# Check backup contents
-restic -r "rest:http://localhost:8000/test-backup" ls latest --password-file <(echo "testpassword123")
-```
-
-### 4. Perform Directory Restore
-
-```bash
-# Create restore directory
-RESTOR_DIR="/tmp/restored_data"
-mkdir -p $RESTOR_DIR
-
-# Set environment variables for directory restore
-export OPERATION_MODE="restore"
-export RESTOREDIR=$RESTOR_DIR
-export RESTORE_SNAPSHOT_ID="latest"
-
-# Run the restore
-./src/backup.sh
-```
-
-### 5. Verify Directory Restore
-
-```bash
-# Compare original and restored directories
-diff -r /tmp/test-data $RESTOR_DIR
-
-# Check file contents
-echo "Original file:"
-cat test_data/sample.txt
-echo "Restored file:"
-cat restored_data/sample.txt
-
-# Verify binary file integrity
-md5sum test_data/large_file.bin
-md5sum restored_data/large_file.bin
-
-# Check directory structure
-tree test_data
-tree restored_data
-```
-
-### 6. Cleanup Directory Test
-
-```bash
-# Remove test directories
-rm -rf test_data restored_data
-```
-
-## PostgreSQL Backup Testing
-
-This section tests the PostgreSQL database backup and restore functionality.
-
-### 1. Generate Test Data
-
-```bash
-# Generate test data in PostgreSQL
-./generate_test_data.sh
-```
-
-### 2. Verify Initial Data
-
-```bash
-# Check that data exists
-docker exec postgres-test psql -U testuser -d testdb -c "
-SELECT 'customers' as table_name, COUNT(*) as row_count FROM customers
-UNION ALL
-SELECT 'orders' as table_name, COUNT(*) as row_count FROM orders;
-"
-```
-
-### 3. Perform PostgreSQL Backup
-
-```bash
-# Set environment variables for PostgreSQL backup
-export OPERATION_MODE="backup"
-export BACKUP_MODE="postgres"
-export RESTIC_PASSWORD="testpassword123"
-export RESTIC_REPOSITORY="rest:http://localhost:8000/postgres-backup"
-export PGHOST="localhost"
-export PGPORT="5432"
-export PGDATABASE="testdb"
-export PGUSER="testuser"
-export PGPASSWORD="testpass"
-export ENABLE_GOTIFY="false"
-
-# Run the backup
-./src/backup.sh
-```
-
-### 4. Verify PostgreSQL Backup
-
-```bash
-# List snapshots
-restic -r "rest:http://localhost:8000/postgres-backup" snapshots --password-file <(echo "testpassword123")
-
-# Check backup contents
-restic -r "rest:http://localhost:8000/postgres-backup" ls latest --password-file <(echo "testpassword123")
-```
-
-### 5. Clear the Database
-
-```bash
-# Drop all tables to simulate data loss
-docker exec postgres-test psql -U testuser -d testdb -c "
-DROP TABLE IF EXISTS orders CASCADE;
-DROP TABLE IF EXISTS customers CASCADE;
-"
-
-# Verify database is empty
-docker exec postgres-test psql -U testuser -d testdb -c "
-SELECT table_name FROM information_schema.tables
-WHERE table_schema = 'public';
-"
-```
-
-### 6. Perform PostgreSQL Restore
-
-```bash
-# Set environment variables for PostgreSQL restore
-export OPERATION_MODE="restore"
-export RESTORE_SNAPSHOT_ID="latest"
-
-# Run the restore
-./src/backup.sh
-```
-
-### 7. Verify PostgreSQL Restore
-
-```bash
-# Check that data has been restored
-docker exec postgres-test psql -U testuser -d testdb -c "
-SELECT 'customers' as table_name, COUNT(*) as row_count FROM customers
-UNION ALL
-SELECT 'orders' as table_name, COUNT(*) as row_count FROM orders;
-"
-
-# Verify data integrity
-docker exec postgres-test psql -U testuser -d testdb -c "
-SELECT
-    c.name as customer_name,
-    COUNT(o.id) as order_count,
-    SUM(o.price * o.quantity) as total_spent
-FROM customers c
-LEFT JOIN orders o ON c.id = o.customer_id
-GROUP BY c.id, c.name
-ORDER BY total_spent DESC
-LIMIT 5;
-"
-
-# Check foreign key constraints
-docker exec postgres-test psql -U testuser -d testdb -c "
-SELECT
-    tc.constraint_name,
-    tc.table_name,
-    kcu.column_name,
-    ccu.table_name AS foreign_table_name,
-    ccu.column_name AS foreign_column_name
-FROM information_schema.table_constraints AS tc
-JOIN information_schema.key_column_usage AS kcu
-    ON tc.constraint_name = kcu.constraint_name
-    AND tc.table_schema = kcu.table_schema
-JOIN information_schema.constraint_column_usage AS ccu
-    ON ccu.constraint_name = tc.constraint_name
-    AND ccu.table_schema = tc.table_schema
-WHERE tc.constraint_type = 'FOREIGN KEY'
-    AND tc.table_name IN ('customers', 'orders');
-"
-```
-
-## Advanced Testing Scenarios
-
-### 1. Test with Different Snapshot IDs
-
-```bash
-# List all snapshots
-restic -r "rest:http://localhost:8000/test-backup" snapshots --password-file <(echo "testpassword123")
-
-# Restore a specific snapshot
-export RESTORE_SNAPSHOT_ID="<snapshot-id>"
-./src/backup.sh
-```
-
-### 2. Test Error Handling
-
-```bash
-# Test with invalid repository
-export RESTIC_REPOSITORY="rest:http://localhost:8000/nonexistent"
-./src/backup.sh
-
-# Test with wrong password
-export RESTIC_PASSWORD="wrongpassword"
-./src/backup.sh
-```
-
-### 3. Test with Gotify Notifications (Optional)
-
-```bash
-# If you have a Gotify server running
-export ENABLE_GOTIFY="true"
-export GOTIFYHOST="http://your-gotify-server:80"
-export GOTIFYTOKEN="your-token"
-export GOTIFYTOPIC="Backup Test"
-./src/backup.sh
-```
-
-## Cleanup
-
-### 1. Stop Services
-
-```bash
-# Stop and remove containers
-docker-compose down
-
-# Remove volumes (optional - this will delete all data)
-docker-compose down -v
-```
-
-### 2. Clean Up Test Files
-
-```bash
-# Remove any remaining test files
-rm -rf test_data restored_data
-```
-
-## Troubleshooting
-
-### Common Issues
-
-1. **Connection refused to restic server**: Wait a bit longer for the container to start up
-2. **PostgreSQL connection failed**: Ensure the database container is fully initialized
-3. **Permission denied**: Make sure the backup script is executable (`chmod +x src/backup.sh`)
-4. **Restic repository not found**: Check that the repository URL is correct and the server is running
-5. **Script exits early with no output**: The restic repository hasn't been initialized yet. Run the initialization commands in step 3 above.
-
-### Debug Commands
-
-```bash
-# Check container logs
-docker-compose logs restic-server
-docker-compose logs postgres
-
-# Test restic connectivity
-restic -r "rest:http://localhost:8000/test-backup" snapshots --password-file <(echo "testpassword123")
-
-# Test PostgreSQL connectivity
-docker exec postgres-test psql -U testuser -d testdb -c "SELECT 1;"
-```
-
-## Expected Results
-
-### Directory Backup Test
-
- ✅ Backup completes successfully
- ✅ Files are backed up to restic repository
- ✅ Restore completes successfully
- ✅ Restored files match original files exactly
- ✅ Directory structure is preserved
-
-### PostgreSQL Backup Test
-
- ✅ Database backup completes successfully
- ✅ Database dump is backed up to restic repository
- ✅ Database can be cleared successfully
- ✅ Database restore completes successfully
- ✅ All data is restored correctly
- ✅ Foreign key relationships are maintained
- ✅ Data integrity is preserved
-
-This testing procedure ensures that both directory and PostgreSQL backup/restore functionality works correctly and can be used as a foundation for automated testing in CI/CD pipelines.
--- a/compose.yaml
+++ b/compose.yaml
@@ -1,19 +0,0 @@
-services:
-  restic-server:
-    image: restic/rest-server:latest
-    container_name: restic-server
-    ports:
-      - "8000:8000"
-    command: rest-server --no-auth --path /data
-    restart: unless-stopped
-
-  postgres:
-    image: postgres:17
-    container_name: postgres-test
-    ports:
-      - "5432:5432"
-    environment:
-      POSTGRES_DB: testdb
-      POSTGRES_USER: testuser
-      POSTGRES_PASSWORD: testpass
-    restart: unless-stopped
--- a/generate_dir_test_data.sh
+++ b/generate_dir_test_data.sh
@@ -1,51 +0,0 @@
-#!/bin/bash
-set -euo pipefail
-
-# Script to generate test data for Directory backup testing
-# This script creates a few directories with a few files and directories in each and populates it with test data
-
-# Create base test directory
-TEST_DIR="/tmp/test-data"
-echo "Creating test directory structure in $TEST_DIR..."
-
-# Remove existing test directory if it exists
-rm -rf "$TEST_DIR"
-mkdir -p "$TEST_DIR"
-
-# Create various subdirectories
-mkdir -p "$TEST_DIR/documents/reports"
-mkdir -p "$TEST_DIR/documents/contracts" 
-mkdir -p "$TEST_DIR/data/logs"
-mkdir -p "$TEST_DIR/data/backups"
-
-# Create text files with content
-echo "This is the annual report for 2023" > "$TEST_DIR/documents/reports/annual_2023.txt"
-echo "Q4 financial summary" > "$TEST_DIR/documents/reports/q4_summary.txt"
-echo "Contract terms and conditions" > "$TEST_DIR/documents/contracts/agreement.txt"
-
-# Create JSON files
-cat << 'EOF' > "$TEST_DIR/data/config.json"
-{
-  "app_name": "TestApp",
-  "version": "1.0.0",
-  "settings": {
-    "debug": true,
-    "max_retries": 3,
-    "timeout": 30
-  }
-}
-EOF
-
-# Create some log files
-for i in {1..3}; do
-  echo "$(date) - Log entry $i" >> "$TEST_DIR/data/logs/app.log"
-  echo "$(date) - Error $i: Sample error message" >> "$TEST_DIR/data/logs/error.log"
-done
-
-# Create symbolic links
-ln -s "../reports/annual_2023.txt" "$TEST_DIR/documents/contracts/report_link"
-ln -s "../../data/config.json" "$TEST_DIR/documents/reports/config_link"
-
-echo "Test data generation completed successfully!"
-echo "Created directory structure:"
-tree "$TEST_DIR"
--- a/generate_sql_test_data.sh
+++ b/generate_sql_test_data.sh
@@ -1,124 +0,0 @@
-#!/bin/bash
-set -euo pipefail
-
-# Script to generate test data for PostgreSQL backup testing
-# This script creates two tables with a foreign key relationship and populates them with test data
-
-# Database connection parameters
-PGHOST="${PGHOST:-localhost}"
-PGPORT="${PGPORT:-5432}"
-PGDATABASE="${PGDATABASE:-testdb}"
-PGUSER="${PGUSER:-testuser}"
-PGPASSWORD="${PGPASSWORD:-testpass}"
-
-# Export password for psql
-export PGPASSWORD
-
-echo "Generating test data for PostgreSQL database..."
-
-# Create tables
-echo "Creating tables..."
-psql -h "$PGHOST" -p "$PGPORT" -U "$PGUSER" -d "$PGDATABASE" << 'EOF'
-- Drop tables if they exist
-DROP TABLE IF EXISTS orders CASCADE;
-DROP TABLE IF EXISTS customers CASCADE;
-
-- Create customers table
-CREATE TABLE customers (
-    id SERIAL PRIMARY KEY,
-    name VARCHAR(100) NOT NULL,
-    email VARCHAR(100) UNIQUE NOT NULL,
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
-);
-
-- Create orders table with foreign key to customers
-CREATE TABLE orders (
-    id SERIAL PRIMARY KEY,
-    customer_id INTEGER NOT NULL REFERENCES customers(id) ON DELETE CASCADE,
-    product_name VARCHAR(100) NOT NULL,
-    quantity INTEGER NOT NULL CHECK (quantity > 0),
-    price DECIMAL(10,2) NOT NULL CHECK (price >= 0),
-    order_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP
-);
-
-- Create indexes for better performance
-CREATE INDEX idx_orders_customer_id ON orders(customer_id);
-CREATE INDEX idx_customers_email ON customers(email);
-CREATE INDEX idx_orders_order_date ON orders(order_date);
-EOF
-
-# Insert test data
-echo "Inserting test data..."
-psql -h "$PGHOST" -p "$PGPORT" -U "$PGUSER" -d "$PGDATABASE" << 'EOF'
-- Insert customers
-INSERT INTO customers (name, email) VALUES
-    ('John Doe', 'john.doe@example.com'),
-    ('Jane Smith', 'jane.smith@example.com'),
-    ('Bob Johnson', 'bob.johnson@example.com'),
-    ('Alice Brown', 'alice.brown@example.com'),
-    ('Charlie Wilson', 'charlie.wilson@example.com'),
-    ('Diana Davis', 'diana.davis@example.com'),
-    ('Eve Miller', 'eve.miller@example.com'),
-    ('Frank Garcia', 'frank.garcia@example.com'),
-    ('Grace Lee', 'grace.lee@example.com'),
-    ('Henry Taylor', 'henry.taylor@example.com');
-
-- Insert orders
-INSERT INTO orders (customer_id, product_name, quantity, price) VALUES
-    (1, 'Laptop', 1, 999.99),
-    (1, 'Mouse', 2, 25.50),
-    (2, 'Keyboard', 1, 75.00),
-    (2, 'Monitor', 1, 299.99),
-    (3, 'Headphones', 1, 150.00),
-    (3, 'Webcam', 1, 89.99),
-    (4, 'Tablet', 1, 399.99),
-    (4, 'Stylus', 1, 49.99),
-    (5, 'Smartphone', 1, 699.99),
-    (5, 'Case', 1, 19.99),
-    (6, 'Desktop', 1, 1299.99),
-    (6, 'RAM', 2, 79.99),
-    (7, 'SSD', 1, 199.99),
-    (7, 'Graphics Card', 1, 599.99),
-    (8, 'Motherboard', 1, 199.99),
-    (8, 'CPU', 1, 399.99),
-    (9, 'Power Supply', 1, 149.99),
-    (9, 'Cooling Fan', 2, 29.99),
-    (10, 'Cable Set', 1, 39.99),
-    (10, 'USB Hub', 1, 24.99);
-EOF
-
-# Verify data
-echo "Verifying test data..."
-psql -h "$PGHOST" -p "$PGPORT" -U "$PGUSER" -d "$PGDATABASE" << 'EOF'
-- Show table counts
-SELECT 'customers' as table_name, COUNT(*) as row_count FROM customers
-UNION ALL
-SELECT 'orders' as table_name, COUNT(*) as row_count FROM orders;
-
-- Show sample data
-SELECT 'Sample customers:' as info;
-SELECT id, name, email FROM customers LIMIT 5;
-
-SELECT 'Sample orders:' as info;
-SELECT o.id, c.name as customer_name, o.product_name, o.quantity, o.price 
-FROM orders o 
-JOIN customers c ON o.customer_id = c.id 
-LIMIT 5;
-
-- Show foreign key relationship
-SELECT 'Foreign key relationship check:' as info;
-SELECT 
-    c.name as customer_name,
-    COUNT(o.id) as order_count,
-    SUM(o.price * o.quantity) as total_spent
-FROM customers c
-LEFT JOIN orders o ON c.id = o.customer_id
-GROUP BY c.id, c.name
-ORDER BY total_spent DESC;
-EOF
-
-echo "Test data generation completed successfully!"
-echo "Database contains:"
-echo "- 10 customers"
-echo "- 20 orders with foreign key relationships"
-echo "- Various data types and constraints"
--- a/src/backup.sh
+++ b/src/backup.sh
@@ -24,7 +24,7 @@ OPERATION_MODE="${OPERATION_MODE:-backup}"

 #######################################
 # Determine backup mode from the environment only.
-# Valid values: "directory" or "postgres".
+# Valid values: "directory", "postgres", or "s3".
 # Default to "directory" if not provided.
 #######################################
 BACKUP_MODE="${BACKUP_MODE:-directory}"
@@ -39,6 +39,8 @@ if [ "$BACKUP_MODE" = "postgres" ]; then
    elif [ "$OPERATION_MODE" = "restore" ]; then
        REQUIRED_CMDS+=(psql)
    fi
+elif [ "$BACKUP_MODE" = "s3" ]; then
+    REQUIRED_CMDS+=(mc)
 fi

 for cmd in "${REQUIRED_CMDS[@]}"; do
@@ -107,8 +109,16 @@ case "$BACKUP_MODE" in
            fi
        fi
        ;;
+    s3)
+        : "${S3_BUCKET:?Environment variable S3_BUCKET is not set (required for S3 mode)}"
+        : "${S3_ENDPOINT:?Environment variable S3_ENDPOINT is not set (required for S3 mode)}"
+        : "${MINIO_ACCESS_KEY:?Environment variable MINIO_ACCESS_KEY is not set (required for S3 mode)}"
+        : "${MINIO_SECRET_KEY:?Environment variable MINIO_SECRET_KEY is not set (required for S3 mode)}"
+        # Optional: S3 path prefix
+        : "${S3_PREFIX:=}"
+        ;;
    *)
-        echo "Error: Unknown backup mode '$BACKUP_MODE'. Valid modes are 'directory' and 'postgres'." >&2
+        echo "Error: Unknown backup mode '$BACKUP_MODE'. Valid modes are 'directory', 'postgres', and 's3'." >&2
        exit 1
        ;;
 esac
@@ -287,6 +297,91 @@ restore_postgres() {
    fi
 }

+#######################################
+# Backup an S3 bucket.
+# Syncs the S3 bucket to a temporary directory and then backs it up.
+#######################################
+backup_s3() {
+    log "Starting S3 backup for bucket '${S3_BUCKET}' at endpoint '${S3_ENDPOINT}'"
+    
+    # Create a temporary directory for the S3 sync.
+    TEMP_BACKUP_DIR=$(mktemp -d)
+    log "Created temporary directory: ${TEMP_BACKUP_DIR}"
+
+    # Configure MinIO Client alias
+    local alias_name="backupsidecar"
+    if ! mc alias set "${alias_name}" "${S3_ENDPOINT}" "${MINIO_ACCESS_KEY}" "${MINIO_SECRET_KEY}"; then
+        local msg="Failed to configure MinIO client alias"
+        log "$msg"
+        send_notification "$msg"
+        exit 1
+    fi
+
+    # Build S3 path
+    local s3_path="${alias_name}/${S3_BUCKET}"
+    if [ -n "${S3_PREFIX}" ]; then
+        s3_path="${s3_path}/${S3_PREFIX}"
+    fi
+
+    log "Syncing S3 bucket from ${s3_path} to ${TEMP_BACKUP_DIR}..."
+    if mc mirror "${s3_path}" "${TEMP_BACKUP_DIR}" --remove; then
+        log "S3 sync completed successfully."
+    else
+        local exit_code=$?
+        local msg="S3 sync failed with error code ${exit_code}"
+        log "$msg"
+        send_notification "$msg"
+        exit "$exit_code"
+    fi
+
+    # Back up the directory containing the S3 content.
+    run_restic_backup "${TEMP_BACKUP_DIR}"
+}
+
+#######################################
+# Restore an S3 bucket.
+# Restores the S3 content from the backup and syncs it back to S3.
+#######################################
+restore_s3() {
+    local snapshot_id="${RESTORE_SNAPSHOT_ID:-latest}"
+    log "Starting S3 restore for bucket '${S3_BUCKET}' at endpoint '${S3_ENDPOINT}'"
+    
+    # Create a temporary directory for the restore.
+    TEMP_RESTORE_DIR=$(mktemp -d)
+    log "Created temporary directory: ${TEMP_RESTORE_DIR}"
+
+    # Restore the backup to the temporary directory
+    run_restic_restore "${TEMP_RESTORE_DIR}" "${snapshot_id}"
+
+    # Configure MinIO Client alias
+    local alias_name="backupsidecar"
+    if ! mc alias set "${alias_name}" "${S3_ENDPOINT}" "${MINIO_ACCESS_KEY}" "${MINIO_SECRET_KEY}"; then
+        local msg="Failed to configure MinIO client alias"
+        log "$msg"
+        send_notification "$msg"
+        exit 1
+    fi
+
+    # Build S3 path
+    local s3_path="${alias_name}/${S3_BUCKET}"
+    if [ -n "${S3_PREFIX}" ]; then
+        s3_path="${s3_path}/${S3_PREFIX}"
+    fi
+
+    log "Syncing restored content from ${TEMP_RESTORE_DIR} to ${s3_path}..."
+    if mc mirror "${TEMP_RESTORE_DIR}" "${s3_path}" --remove; then
+        local msg="S3 restore completed successfully"
+        log "$msg"
+        send_notification "$msg"
+    else
+        local exit_code=$?
+        local msg="S3 restore failed with error code ${exit_code}"
+        log "$msg"
+        send_notification "$msg"
+        exit "$exit_code"
+    fi
+}
+
 #######################################
 # Cleanup temporary resources.
 #######################################
@@ -315,6 +410,9 @@ main() {
                postgres)
                    backup_postgres
                    ;;
+                s3)
+                    backup_s3
+                    ;;
            esac
            ;;
        restore)
@@ -325,6 +423,9 @@ main() {
                postgres)
                    restore_postgres
                    ;;
+                s3)
+                    restore_s3
+                    ;;
            esac
            ;;
    esac
Author	SHA1	Message	Date
Timo Behrendt	5a25eca929	sync All checks were successful CI / Check syntax (pull_request) Successful in 10s Details CI / Build Docker image (pull_request) Successful in 21s Details	2025-09-06 08:24:06 +02:00
Timo Behrendt	afc568a4e9	ci: add bash syntax check (#18 ) All checks were successful CD / Check changes (push) Successful in 6s Details CD / Create tag (push) Successful in 14s Details CD / Build and push (amd64) (push) Successful in 23s Details CD / Build and push (arm64) (push) Successful in 1m14s Details CD / Create manifest (push) Successful in 8s Details Reviewed-on: #18 Co-authored-by: Timo Behrendt <t.behrendt@t00n.de> Co-committed-by: Timo Behrendt <t.behrendt@t00n.de>	2025-09-04 20:48:13 +02:00
Timo Behrendt	73ce57b122	feat: add restore functionality (#12 ) All checks were successful CD / Check changes (push) Successful in 14s Details CD / Create tag (push) Successful in 11s Details CD / Build and push (amd64) (push) Successful in 26s Details CD / Build and push (arm64) (push) Successful in 1m14s Details CI / Build Docker image (pull_request) Successful in 18s Details CD / Create manifest (push) Successful in 25s Details Reviewed-on: #12 Co-authored-by: Timo Behrendt <t.behrendt@t00n.de> Co-committed-by: Timo Behrendt <t.behrendt@t00n.de>	2025-09-04 20:36:40 +02:00