Overview
Wee all need to backup our machines. But we also need to keep the data private and the backup procedure secured. In UNIX and Linux machines, we need to run the backup operation as root in order to read everything on the machines. But allowing remote connections as the root user is not exactly a good idea. So how to we proceed? We use rsnapshot(1) and rssh(1) together with OpenSSH to secure the whole process. Here's how to do it on CentOS 6.
In case you're running a heterogeneous network, please note that I've successfully configured this process on FreeBSD, PC-BSD, RedHat, Ubuntu, AIX and Solaris servers.
In this example, our backup server is called backup.company.com and is running CentOS 6 while the clients are :
Server Configuration (part 1 of 2)
Select a server which has a lot of hard disk space or that can grow it's storage without too much trouble. Don't forget that this solution is a disk-based backup solution, so we need disk space. Not an enormous quantity, thanks to rsnapshot, but enough to hold all of your current and future clients. If you can, I'd suggest that you use either a FreeBSD or a Solaris machine and configure the /backup filesystem on ZFS to benefit from it's data integrity feature and eliminate the risk of silent data corruption. There is ZFS on Linux, but I haven't tried it yet since it's still a release candidate. And BRTFS is also not production ready at the time of this writing. Since this is our production data, I don't want to use a non-production ready file system to store it.
So, once you selected a machine, install a minimal CentOS 6 on it and make sure to create a seperate file system with a mount point of /backup.
Connect to the backup server and setup the /backup directories.
ssh backup.company.com
sudo mkdir /backup/{conf,data,key,log,run,scripts}
sudo chown -R root:root /backup
Then install some more packages.
sudo yum -y install openssh-clients rsnapshot
The rsnapshot installation will also install rsync as a dependency.
Next create a backup group and a backup user. Notice that our backup user has the UID zero (same as the root user). This is crucial for our purposes. Also keep in mind that you may select any GID and UID for the backup group and user. The important thing is to make sure that the backup GID and UID are exactly the same across your entire infrastructure.
sudo groupadd -g 911 backup
sudo useradd -c "Remote Backup User" -d /home/backup -o -u 0 -g 911 -m -s /bin/bash backup
Make sure to assign a password to our new backup user. This way it's account won't be locked.
sudo passwd backup
Switch to the backup user and create an SSH key pair. I use DSA here because all my other keys are RSA. Do not assign a passphrase to the new key.
sudo su - backup
ssh-keygen -t dsa -b 1024
exit
Move the key into the /backup/key directory.
sudo cp ~backup/.ssh/id_dsa ~backup/.ssh/id_dsa.pub /backup/key
sudo chmod 700 /backup/key
Create two wrapper scripts to help the process.
sudo vi /backup/scripts/backup_runner.sh
sudo vi /backup/scripts/ssh_wrapper.sh
Make sure both scripts are executable and that they don't have any syntax errors in them.
sudo chmod a+x /backup/scripts/*.sh
sudo sh -n /backup/scripts/backup_runner.sh
sudo sh -n /backup/scripts/ssh_wrapper.sh
Exclude some files from the CentOS/RedHat machines.
Configure the clients. WARNING : rsnapshot is very sensitive with spaces and tabs. DO NOT USE ANY SPACES IN THE CONFIGURATION FILE! You have been warned :)
sudo vi /backup/conf/alice.company.com
Make sure our backup log files don't consume too much disk space.
sudo vi /etc/logrotate.d/backup
And make sure our new logrotate configuration is still valid.
sudo logrotate -d /etc/logrotate.conf
Before we can backup a machine, we must make sure the client allows the connection and has a copy of the backup user's public ssh key. Of course, replace my username by yours in the commands below...
sudo scp /backup/key/id_dsa.pub drobilla@alice.company.com:/tmp
sudo scp /backup/key/id_dsa.pub drobilla@charlie.company.com:/tmp
Client Configurations
In this section we will configure the CentOS and FreeBSD / PC-BSD clients.
CentOS 6 Client Configuration
We need to install rssh on each clients, but it's not in the default CentOS repositories. Thankfiully, rssh is available in RPMforge, one of the Additional CentOS Repositories. Simply follow the instructions to install the RPMforge repository. Once this is done, install rssh and rsync.
sudo yum -y install rssh rsync
We also need the same backup group and user on each clients. But here there is a subtle but very important difference : the backup user's shell is set to /usr/bin/rssh.
sudo groupadd -g 911 backup
sudo useradd -c "Remote Backup User" -d /home/backup -o -u 0 -g 911 -m -s /usr/bin/rssh backup
Make sure to assign a password to our new backup user. This way it's account won't be locked.
Then place a copy of the backup user's public ssh key into the client's backup authorized_keys files.
sudo mkdir ~backup/.ssh
sudo mv /tmp/id_dsa.pub ~backup/.ssh/authorized_keys
Now edit the authorized_keys file to add the from="backup.company.com" keyword at the start of the file. This will restrict the use of this key to our backup server only. Any other machine trying to use this key will not be permitted. This is an extra layer of security to the whole setup. See this URL for more information on the authorized_keys file syntax. Keep in mind that the file below is just an example!
sudo vi ~backup/.ssh/authorized_keys
Make sure it has the right permissions.
sudo chmod 600 ~backup/.ssh/authorized_keys
sudo chown root:root ~backup/.ssh/authorized_keys
sudo chmod 750 ~backup/.ssh
Before we can use rssh, we must configure it. Our goal is to use it only for the root/backup user, so let's do this now.
sudo vi /etc/rssh.conf
We must also add rssh to the available shells on the machine.
sudo vi /etc/shells
And of course, we must configure OpenSSH to allow our backup group to login, to permit the root user to login, but without password. Your sshd_config file may be different, so keep in mind that the important configuration keywords are AllowGroups and PermitRootLogin which are set like this :
sudo vi /etc/ssh/sshd_config
AllowGroups backup sysadmin
PermitRootLogin without-password
Restart the sshd(8) daemon so that it knows about the new configuration.
sudo /etc/init.d/sshd restart
And make sure it starts when the client comes up.
sudo chkconfig sshd on
The client machine is now ready. To test the backup, we must return to our backup server, try the ssh key and answer YES to the very first connection and then setup the crontab so that the backup happens every night without having to manually manage it.
PC-BSD 9.0 / FreeBSD 9.0 Client Configuration
Connect to the client machine.
ssh charlie.company.com
Update the ports tree.
sudo portsnap fetch
sudo portsnap update
Install rssh. We don't need rdist support, but you may compile it in if you want, that's not a problem.
cd /usr/ports/shells/rssh
sudo make install clean
Install rsync using all the default options.
cd /usr/ports/net/rsync
sudo make install clean
Create the backup group and backup user.
sudo pw group add -n backup -g 911
sudo pw user add -o -u 0 -n backup -d /usr/home/backup -g backup -c "Remote Backup User" -m -s /usr/local/bin/rssh -w random
We don't need to assign a password to our new backup user because we chose to use a random one when we created it (see the « -w random » option in the command above). You can forget about this password since you'll never need it.
Run a quick chmod(1) to fix an error if you forget to run it.
sudo chown root:wheel /usr/home/backup/.login_conf
Then place a copy of the backup user's public ssh key into the client's backup authorized_keys files.
sudo mkdir ~backup/.ssh
sudo mv /tmp/id_dsa.pub ~backup/.ssh/authorized_keys
Now edit the authorized_keys file to add the from="backup.company.com" keyword at the start of the file. This will restrict the use of this key to our backup server only. Any other machine trying to use this key will not be permitted. This is an extra layer of security to the whole setup. See this URL for more information on the authorized_keys file syntax. Keep in mind that the file below is just an example!
sudo vi ~backup/.ssh/authorized_keys
Make sure it has the right permissions.
sudo chmod 600 ~backup/.ssh/authorized_keys
sudo chown root:wheel ~backup/.ssh/authorized_keys
sudo chmod 750 ~backup/.ssh
Before we can use rssh, we must configure it. Our goal is to use it only for the root/backup user, so let's do this now.
sudo vi /usr/local/etc/rssh.conf
And of course, we must configure OpenSSH to allow our backup group to login, to permit the root user to login, but without password. Your sshd_config file may be different, so keep in mind that the important configuration keywords are AllowGroups and PermitRootLogin which are set like this :
sudo vi /etc/ssh/sshd_config
AllowGroups backup sysadmin
PermitRootLogin without-password
And make sure it starts when the client comes up. Just add a single line to /etc/rc.conf.
sudo vi /etc/rc.conf
sshd_enable="YES"
sshd_enable="YES"
Restart the sshd(8) daemon so that it knows about the new configuration.
sudo /etc/rc.d/sshd restart
The client machine is now ready. To test the backup, we must return to our backup server, try the ssh key and answer YES to the very first connection and then setup the crontab so that the backup happens every night without having to manually manage it.
Server Configuration (part 2 of 2)
Connect to the backup server.
ssh backup.company.com
Connect to the client using the backup user's key and answer YES. This is very important and you only need to do this once.
sudo ssh -i /backup/key/id_dsa backup@alice.company.com
The authenticity of host 'alice.company.com (192.168.1.20)' can't be established.
RSA key fingerprint is 60:0c:db:21:a2:c3:6b:0d:ae:03:f4:45:be:b5:e5:01.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'alice.company.com (192.168.1.20)' (DSA) to the list of known hosts.
Last login: Tue Jun 12 09:24:37 2012 from backup.company.com
This account is restricted by rssh.
Allowed commands: rsync
If you believe this is in error, please contact your system administrator.
As you can see, the rssh shell prevented us from have a shell on the client. But we still managed to connect to the client. All this in an encrypted ssh tunnel. Great!
IMPORTANT : make sure you answer YES once for all your backup clients!
Now let's try a backup now. Note that the first sudo -l is just there to prevent a password prompt which is usefull when you use an « & » at the end of the next sudo command.
sudo -l
sudo /backup/scripts/backup_runner.sh /backup/conf/alice.company.com daily &
And check the log as it happens.
sudo tail -F /backup/log/alice.company.com
Once the backup is over, confirm that you have the data on the backup server.
sudo ls -alFR /backup/data/alice.company.com/daily.0
And fiinally, configure cron(8) to manage all this automatically.
A good idea now would be to backup your backup server's data! Encrypt the data and send it offsite.
Recovery
Human Error
Recover with this setup is quite easy because you can simply navigate the backup directory and retrive any files as you normally do on a filesystem. For example, let's say you accidentally deleted the /etc/sysconfig/ifcfg-eth0 on host alice (and that you don't use RCS). Then simply connect to the backup server and send the file back to the client alice.company.com via scp(1). Again, replace my own username with yours in the scp(1) command below...
ssh backup.company.com
sudo scp /backup/data/alice.company.com/daily.0/etc/sysconfig/network-scripts/ifcfg-eth0 drobilla@alice.company.com:/tmp
Then connect to the client and move the file where it belongs.
ssh alice.company.com
sudo mv /tmp/ifcfg-eth0 /etc/sysconfig/network-scripts
sudo chown root:root /etc/sysconfig/network-scripts/ifcfg-eth0
There you go! Simple, secured and easy!
Total Client Failure
In the case where you suffer a total client failure, simply reinstall the OS, reconfigure the backup user but give him shell access (i.e. /bin/bash instead of /usr/bin/rssh) and use tar(1) to dump the content of the backups over to the newly installed client. Then reset the client's backup user's shell back to /usr/bin/rssh. For example :
ssh backup.company.com
sudo su -
cd /backup/data/alice.company.com/daily.0/etc
tar zcf - . | ssh -i /backup/key/id_dsa backup@alice.company.com "cd /etc; tar zxvf -"
cd /backup/data/alice.company.com/daily.0/home
tar zcf - . | ssh -i /backup/key/id_dsa backup@alice.company.com "cd /home; tar zxvf -"
cd /backup/data/alice.company.com/daily.0/root
tar zcf - .| ssh -i /backup/key/id_dsa backup@alice.company.com "cd /root; tar zxvf -"
cd /backup/data/alice.company.com/daily.0/var/log
tar zcf - ./var | ssh -i /backup/key/id_dsa backup@alice.company.com "cd /var/log; tar zxvf -"
cd /backup/data/alice.company.com/daily.0/var/lib/ldap
tar zcf - ./etc | ssh -i /backup/key/id_dsa backup@alice.company.com "cd /var/lib/ldap; tar zxvf -"
You can of course simply enable root access via SSH and do this only via the root user, but that would render the whole process a bit less secure.
HTH,
DA+
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.