VPS maintenance frameworks?
Hi all,
I've managed to keep most of my systems somewhat up-to-date by running a hodgepodge of bash scripts that would blindly try to apt update and upgrade each of them.
It has somewhat worked, with some manual intervention here and there, but 'top notch' is not the stamp I'd put on the works.
With the xz business going on, I see some more updates coming our way, so this could be the moment to improve the upgrading process overall.
My machines mostly run Debian (11 and 12, a single one perhaps 10), with a few Alpines.
What do you use or suggest to centralize maintenance of at least the Debian machines, or LInux servers in general?
Comments
From a central server that has ssh access to all of your other servers:
Note the different ssh keys (you can use the same one if you want) and use of non standard ssh port, different for each server.
You can also use a bastion server, but you have to figure it out yourself on how to set it up as I have yet to set one up for myself.
Websites have ads, I have ad-blocker.
How do you deal with services that need to be restarted? It looks like you are blindly updating all packages.
VirtFusion Affordable, Reliable virtualization management software for the hosting industry · Connect with us on Discord
Indeed, at the moment that is what I do
That's a nice touch... though, I'm not quite sure in which case it helps me more than that it complicates matters (an attacker that has access to two of my machines can see that it has the same public key and infer that the corresponding private key might have access to more, but once someone got hold of the private key, they'd have access to the 'central' server and thus to the separate keys as well.
I have a list of servers that I feed to a somewhat similar, slightly more dynamic, script, I'd hoped someone already had taken the trouble to create something to help managing a set of machines. Searching online gives Netbox (network management), Kubernetes (container management), Cobbler (server provisioning) as solutions to related, but slightly different problems.
/edit/ I don't really have an idea of the name of the class of software I'd be looking for, any hints?
I am giving an example here. You can replace the
sudo apt update && sudo apt upgrade
with your preferred command or set of command. You can even specify a specific bash script on your server to run instead of this.You can also use unattended updates: https://www.baeldung.com/linux/automatic-updates-status-configuration
Sorry, you lost me there... The server that runs and manages the updates needs access to all of your other servers. All the methods you mention bellow uses the same method. Moreover, your servers can have different public/private keys. I believe I mentioned that clearly in my post.
So you have 1 pair of public/private key for each server. Your central master server holds all private keys while your slave servers (the ones that gets updated) gets 1 public key each. Usually your central server is not accessible over the internet (to prevent misuse).
Since I do not know what you need to do or run, i cannot tell you want to do for your specific use case. If I use one of my servers as an example, I usually run:
Websites have ads, I have ad-blocker.
Thanks for taking the time to elaborate!
Yes, you did! I wondered in which case it would increase security or usability.
I now have a bunch of scripts that ssh into $1 with a maintenance user, some of which execute $2. In that directory I
touch new.domain.tld
when I got a new VPS, so that bash helps me completing $1 when calling a script.That is OK for one-off commands for a specific server, but having bash accept an array as input comes (in case of my limited bash-fu) with some headaches. Meaning I can not drop a whole bunch of domains in the script as separate variables (to elaborate on that: I intended to have a management.domain.tld for each server, so I can call a script as
./serverupdate.sh management.*
, by having each server referenced by a management.-domain as empty file in that directory).You can also make use of bash arrays to pass a list of hosts:
And us them in your bash in a array:
Websites have ads, I have ad-blocker.
I use NixOS plus colmena.
The all seeing eye sees everything...
Ansible.
It's a sysadmin framework that abstracts away a lot of the OS specifics, but still lets you write simple code that can either just run pure bash, or do simple/advanced sysadmin things like user management, configure services etc.
You run them from your local computer, and they SSH in to your hosts. No dependencies expect python.
Here's an example that does what you need. You need to configure your inventory file, then run the below playbook with
ansible-playbook -i inventory ./upgrade_all.yml
https://www.jeffgeerling.com/blog/2022/ansible-playbook-upgrade-ubuntudebian-servers-and-reboot-if-needed
Thanks a lot guys!
I'm glad to see that my script-based approach is not totally outdated. Thanks for the array tip :-)
I heard good things about NixOS over the years, so it definitely is on the list to try out once.
Ansible may just be the answer I have been looking for, thanks for the introduction!