Proxmox offside encrypted backups with PGP on untrusted storage server

Neoon · 6:40AM

Hi,

I got a cheap storages boxes like 1TB NAT for 10$/y and I had to put it into use.
However not fully trusting them with raw vm images, so PGP will do.

The backups are pgp encrypted before they are getting pulled by the storage server.
The proxmox host has never any access to the backup server and the backup server never has any access to the vm images since they are encrypted.

Whatever gets compromised, your should still be safe, that's the idea.
Downside is obviously, disk usage, for every backup we do in Proxmox, we have to do another encrypted copy and wait for the backup server to pull it before we can clear the disk space.

This can easily be done by using the Proxmox hooks, so you can configure your backup schedule like you normaly would do, only the STOREID has to match.
The only thing you have to tweak is, the backup server when it has to pull the backups.

1. Install the hook.

I do daily/weekly offside backups and keep them for 4 weeks.
Example for weekly, script can be modded though.

https://pastebin.com/raw/31jMWKTz
Wouldn't format properly...

Put the script to /usr/local/bin/vzdump-hook.sh and make it executable.
Don't forget to create the user "weekly" and the folder structure (/mnt/weekly/dump/ or whatever folder you want to use)
You also should install and ssh key for the backup server to be able to login into the user "weekly".

In my case, Proxmox creates usuable backups to /mnt/weekly/dump and the encrypted ones are put to /home/weekly/backups for pulling.

2. Edit /etc/vzdump.conf

Replace
#script: FILENAME
with
script: /usr/local/bin/vzdump-hook.sh

3. Either generate or import a existing pgp key

gpg --gen-key

or

gpg --import mahkey

Make SURE you backup this key.

4. Make a test backup and check if the encrypted backup is there.
There should be zero errors in the proxmox backup log.

If you do, you might have to trust that pgp key.
gpg --edit-key mahkey
and type "trust"

5. Setup a cronjob on the remote storage server

5 5 * * * /home/weekly/backups.sh

You might have to adjust the time.
rsync will do.

#!/bin/bash
set -e
if pgrep -fl backup.sh &>/dev/null; then
        rsync -Pav -e "ssh -i weekly" weekly@mahserverip:/home/weekly/backups/* /home/weekly/backups/
        find /home/weekly/backups/* -mtime +30 -exec rm {} +
fi

Don't forgot to make it executable and do a testrun.

6. Profit.

Thanks for reading my TED TALK.

bliss · 8:29AM

I've made a similar backup with syncthing (untrusted devices), maybe your solution is more elegant.

wankel · 8:29AM

Thanks for the detailed write-up!

I opted to use this kind of VPS as Proxmox PBS "Remote".

The local PBS stores the backups encrypted.
The PBS "Remote" has limited access (via Proxmox access control) to backups on the local PBS
The "Remote" pulls backups according to its schedule
Local PBS prunes backups according to its own schedule

Would this fit your use case?

Perceived benefit is have everything managed by Proxmox.

The drawback I notice (but have not yet encountered) in this scenario is that I don't check whether a backup has been pulled to remote before being pruned.

How would you value this way of working, compared to the one you outlined?

Neoon · 8:45AM

@wankel said:
Thanks for the detailed write-up!

I opted to use this kind of VPS as Proxmox PBS "Remote".

The local PBS stores the backups encrypted.

The PBS "Remote" has limited access (via Proxmox access control) to backups on the local PBS

The "Remote" pulls backups according to its schedule

Local PBS prunes backups according to its own schedule

Would this fit your use case?

Perceived benefit is have everything managed by Proxmox.

The drawback I notice (but have not yet encountered) in this scenario is that I don't check whether a backup has been pulled to remote before being pruned.

How would you value this way of working, compared to the one you outlined?

To much overhead, I tried PBS but memory requirements yada yada...

Neoon · 3:37PM

@bliss said:
I've made a similar backup with syncthing (untrusted devices), maybe your solution is more elegant.

Actually... it could be better and safer.
Since no SSH access is needed.

Would still encrypt it with PGP though and let syncthing handle the transfer.

legendary · 4:34PM

set -e is a disaster.

Proxmox offside encrypted backups with PGP on untrusted storage server

Comments