Tuesday, May 15, 2012

HOWTO : OpenLDAP 2.4 Backup & Recovery on CentOS 6.2

This blog post will explain how to backup and restore our OpenLDAP 2.4 server. This is goal number nine.
  1. Install OpenLDAP 2.4.
  2. Configure Transport Layer Security (TLS).
  3. Manage users and groups in OpenLDAP.
  4. Configure pam_ldap to authenticate users via OpenLDAP.
  5. Use OpenLDAP as sudo's configuration repository.
  6. Use OpenLDAP as automount map repository for autofs.
  7. Use OpenLDAP as NFS netgroup repository again for autofs.
  8. Use OpenLDAP as the Kerberos principal repository.
  9. Setup OpenLDAP backup and recovery.
  10. Setup OpenLDAP replication.
The Maintenance chapter in the OpenLDAP Administrator's Guide on this topic is not very explicit. We hope this blog post will be more helpful.


Backup


We need to backup our OpenLDAP data, the OpenLDAP configuration and the OpenLDAP directory along with the system-wide LDAP configuration. Why take both the configuration and the directory? Because the configuration in LDIF format does not have the Kerberos keytab nor the /etc/sysconfig/ldap file.

Both the data and the configuration are dumped in LDIF files with the help of slapcat(8C). The directory and daemon configuration file are placed together in a compressed tar(1) file.

A quick shell script takes care of the backup procedure.

sudo mkdir -p /root/scripts
sudo vi /root/scripts/backup.slapd.sh


#!/bin/sh

# /root/scripts/backup.slapd.sh
#
# Backup the OpenLDAP data and configuration as compressed LDIF files.
# Also backup the entire OpenLDAP directory and daemon configuration.
#
# David Robillard, April 23rd, 2012.

umask 022

PATH="/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin"
export PATH

DATE=`date +%Y%m%d`

BACKUP_DIR="/root/backup/slapd"

BACKUP_DATA_FILENAME="slapd.data.${DATE}.ldif"

BACKUP_CONFIG_FILENAME="slapd.config.${DATE}.ldif"

BACKUP_TAR_FILENAME="slapd.directory.${DATE}.tar.gz"

CA_TLS_CERT="/etc/pki/tls/certs/companyCA.crt"

DIT_CONFIG="cn=config"

DIT_SUFFIX="dc=company,dc=com"

SLAPD_CONFIG_FILENAME="/etc/sysconfig/ldap"

SLAPD_DIR="/etc/openldap"

SLAPD_LOG_ROTATION="/etc/logrotate.d/slapd"

SLAPD_TLS_CERT="/etc/pki/tls/certs/alice.company.com.pem"

SLAPD_TLS_KEY="/etc/pki/tls/certs/alice.company.com.key"

SLAPCAT_OPTIONS="-F /etc/openldap/slapd.d"

LOGFILE="/var/log/backup/slapd.log"

KEEP="30"

# Make sure we have a log file.
if [ ! -f ${LOGFILE} ]; then
touch ${LOGFILE}

if [ "$?" -ne "0" ]; then
echo "ERROR: could not create the log file."
exit 1
fi
fi

# Check if root is running this script.
if [ `id -u` -ne "0" ]; then
echo "ERROR: only root can run this script." | tee -a ${LOGFILE}
exit 1
fi

# Make sure we have a backup directory.
if [ ! -d ${BACKUP_DIR} ]; then
mkdir -p ${BACKUP_DIR}

if [ "$?" -ne "0" ]; then
echo "ERROR: could not create the backup directory." | tee -a ${LOGFILE}
exit 1
fi
fi

# Make sure we don't have too much backup files piling up in our backup directory.
FILES=`find ${BACKUP_DIR} -type f -name "slapd.*" -print | wc -l`

if [ "${FILES}" -gt "${KEEP}" ]; then

OVER=`echo ${FILES}-${KEEP} | bc`
RMFILES=`find ${BACKUP_DIR} -type f -name "slapd.*" -print | sort -r | tail -${OVER}`
echo "NOTE: removing ${RMFILES} from the backup directory." >> ${LOGFILE}
rm ${RMFILES}
fi

# Backup the DIT data.
slapcat ${SLAPCAT_OPTIONS} -b ${DIT_SUFFIX} -l ${BACKUP_DIR}/${BACKUP_DATA_FILENAME} >/dev/null 2>&1

if [ "$?" -eq "0" ]; then

gzip -f ${BACKUP_DIR}/${BACKUP_DATA_FILENAME} 2>&1 >> ${LOGFILE}

if [ "$?" -ne "0" ] ; then
echo "ERROR: dump file compression problem." | tee -a ${LOGFILE}
exit 1
fi
else
echo "ERROR: problem running slapcat(8C) for the DIT data backup." | tee -a ${LOGFILE}
rm ${BACKUP_DIR}/${BACKUP_DATA_FILENAME}
exit 1
fi

# Backup the DIT config as an LDIF file.
slapcat ${SLAPCAT_OPTIONS} -b ${DIT_CONFIG} -l ${BACKUP_DIR}/${BACKUP_CONFIG_FILENAME} >/dev/null 2>&1

if [ "$?" -eq "0" ]; then

gzip -f ${BACKUP_DIR}/${BACKUP_CONFIG_FILENAME} 2>&1 >> ${LOGFILE}

if [ "$?" -ne "0" ] ; then
echo "ERROR: dump file compression problem." | tee -a ${LOGFILE}
exit 1
fi
else
echo "ERROR: problem running slapcat(8C) for the DIT config backup." | tee -a ${LOGFILE}
rm ${BACKUP_DIR}/${BACKUP_CONFIG_FILENAME}
exit 1
fi

# Backup the entire configuration directory.
BACKUP_FILES_LIST="${CA_TLS_CERT} ${SLAPD_CONFIG_FILENAME} ${SLAPD_DIR} ${SLAPD_LOG_ROTATION} ${SLAPD_TLS_CERT} ${SLAPD_TLS_KEY}"

tar zcf ${BACKUP_DIR}/${BACKUP_TAR_FILENAME} ${BACKUP_FILES_LIST} >/dev/null 2>&1

if [ "$?" -ne "0" ]; then
echo "ERROR: problem running config directory tar." | tee -a ${LOGFILE}
rm ${BACKUP_DIR}/${BACKUP_TAR_FILENAME}
exit 1
fi

# EOF


Make sure this new script is executable.

sudo chmod a+x /root/scripts/backup.slapd.sh

Place the script in root's crontab.

sudo crontab -e

# root's crontab
# The time and date fields are:
#  field          allowed values
#  -----          --------------
#  minute         0-59
#  hour           0-23
#  day of month   1-31
#  month          1-12 (or names, see below)
#  day of week    0-7 (0 or 7 is Sun, or use names)

# Backup the OpenLDAP data, config, directory and daemon config.
00 22 * * * /root/scripts/backup.slapd.sh

# EOF

Come back tonight after 10 PM (or tomorrow :) to see if you have backup files.

IMPORTANT : make sure you copy the backup files off site!

Recovery


In the event of a catastrophic failure of the OpenLDAP server, we have everything to revive it. How we do it depends on the scenario.
  1. Complete Server Lost
  2. ACL Problems
  3. Data Corruption (or Human Error...)
  4. File System Full

Complete Server Lost


If the OpenLDAP server crashes beyond repair, the first thing one needs to do is simply find another computer (or have this one fixed), install CentOS and OpenLDAP. Then simply copy the backup files back to the new machine and run these commands :

sudo yum -y openldap-servers krb5-server-ldap nss-pam-ldapd
sudo mv /etc/openldap /etc/openldap.install
cd / && sudo tar zxvf /tmp/slapd.directory.20120515.tar.gz
sudo cp `rpm -ql openldap-servers | grep DB_CONFIG` /var/lib/ldap/DB_CONFIG
sudo chown ldap:ldap /var/lib/ldap/DB_CONFIG
sudo chkconfig slapd on
sudo /etc/init.d/slapd start

That should get you started. Of course, the rsyslog configuration will need to be reconfigured. See this blog post on how to proceed.

ACL Problems


It happens mostly when we build the server, but one can lock itself out of his own OpenLDAP server if the ACLs are modified. This one is quite easy to get back on our feet :

sudo /etc/init.d/nslcd stop
sudo /etc/init.d/slapd stop
sudo mv /etc/openldap /etc/openldap.broken
cd / && sudo tar zxvf /root/backup/slapd/slapd.directory.20120515.tar.gz
sudo /etc/init.d/slapd start
sudo /etc/init.d/nslcd start

Data Corruption (or Human Error...)


Well, sometimes even the administrators make mistakes. Normally one simply has to change it back to what it was via his prefered LDAP browser. But say one admin made changes on friday and left to go climbing. The other admin has no idea what happened and things are broken saterday morning at 2 AM. Well, in those circumstances, we admins tend to have less debug patience. So the easiest (and fastest!) way of fixing things is to bring back thursday's backup with slapadd(8C) while slapd(8C) is NOT running.

sudo /etc/init.d/nslcd stop
sudo /etc/init.d/slapd stop
sudo mv /var/lib/ldap /var/lib/ldap.broken
sudo mkdir /var/lib/ldap
sudo cp `rpm -ql openldap-servers | grep DB_CONFIG` /var/lib/ldap/DB_CONFIG
sudo zcat /root/backup/slapd/slapd.data.20120512.ldif.gz > /tmp/slapd.data.ldif
sudo slapadd -v < /tmp/slapd.data.ldif
sudo chown -R ldap:ldap /var/lib/ldap
sudo /etc/init.d/slapd start
sudo /etc/init.d/nslcd start

This will bring the data back to which ever backup data file we selected.

File System Full


Sometimes the /var filesystem gets filled up. If the LDAP data doesn't live in it's own /var/lib filesystem and the server generates a lot of logs, this can happen quite often. (HINT : place your LDAP data on a seperate filesystem! :) The /var fills up, one will see these errors in the log :

sudo: ldap_simple_bind_s: Can't contact LDAP server
Checking configuration files for slapd: bdb_db_open: DB_CONFIG for suffix dc=company,dc=com
has changed.
Performing database recovery to activate new settings.
bdb_db_open: Recovery skipped in read-only mode. Run manual recovery if errors are encountered.
bdb(dc=company,dc=com): file id2entry.bdb (meta pgno = 0) has LSN [1][1984522].
bdb(dc=company,dc=com): end of log is [1][28]
bdb(dc=company,dc=com): /var/lib/ldap/id2entry.bdb: unexpected file type or format
bdb_db_open: db_open(/var/lib/ldap/id2entry.bdb) failed: Invalid argument (22)
bdb(dc=company,dc=com): Unknown locker ID: 0
backend_startup_one: bi_db_open failed! (22)
slap_startup failed (test would succeed using the -u switch)

Fixing this error is quite easy. First, clean-up up /var!

Once /var has some space left, run the db_recover tool. This tool will make sure that all committed transactions are guaranteed to appear after db_recover has run, and all uncommitted transactions will be completely undone.

sudo /etc/init.d/nslcd stop
sudo /etc/init.d/slapd stop
sudo db_recover -vh /var/lib/ldap
sudo /etc/init.d/slapd start
sudo /etc/init.d/nslcd start

And that's it. If some data is missing, then use yesterday's backup.

We're now done with the backup and recovery of our OpenLDAP 2.4 server. This was goal number nine.
  1. Install OpenLDAP 2.4.
  2. Configure Transport Layer Security (TLS).
  3. Manage users and groups in OpenLDAP.
  4. Configure pam_ldap to authenticate users via OpenLDAP.
  5. Use OpenLDAP as sudo's configuration repository.
  6. Use OpenLDAP as automount map repository for autofs.
  7. Use OpenLDAP as NFS netgroup repository again for autofs.
  8. Use OpenLDAP as the Kerberos principal repository.
  9. Setup OpenLDAP backup and recovery.
  10. Setup OpenLDAP replication.
The next post will explain how to configure OpenLDAP replication.
HTH,

DA+

15 comments:

  1. Thanks, great backup script. Have been running OpenLDAP on CentOS 5 for a long time, and have now migrated it to CentOS 6, and this was a huge help.

    ReplyDelete
    Replies
    1. Hello Simon,

      Glad I could help! :)

      Cheers,

      DA+

      Delete
  2. Hi David,

    Does it work on OpenLDAP 2.3? coz it different to 2.4, I plan to upgrade but I am afraid in doing it, please can you give me the courage on how to do it? Or it is the same steps as you presented.

    I found it very useful your backup scripts. Thanks for sharing your knowledge.

    Ewan Kho

    ReplyDelete
    Replies
    1. Hello Ewann,

      Upgrades can be a little scary, I totally understand you! If you can, try to build an OpenLDAP 2.4 server on a different machine then your current OpenLDAP 2.3 production machines. This way you will be able to learn version 2.4 and see if it works for you or not? If it does, then you simply have to update your client machine's configuration files to point to this new server instead of the old one. Or simply shutdown the old 2.3 and assign it's hostname and IP to the new machine and voilĂ !

      As far as my backup script, I haven't tested it on a 2.3 server. It's using the same LDAP commands, but the directory where the configuration files are stored might be different. Again, test everything on a seperate server (or a VM) to make sure you're satisfied with it before you go live.

      And thanks for the kind words :)

      Cheers,

      DA+

      Delete
  3. FYI, the purging of the old backups will purge your "data" and "config" tarballs following the logic you have here. At least it did for me when I used it. After a quick look, the easiest way to resolve this was to change the file name formats to have the ${DATE} before the type.
    ie "slapd.directory.${DATE}.tar.gz" becomes "slapd.${DATE}.directory.tar.gz"
    and then it handles the flush appropriately.
    I set my keep to "90" and that maintains 30 days worth of backups

    thanks for this excellent blog.

    ReplyDelete
    Replies
    1. Hello HoustonH,

      Thanks for the good words :)

      What do you mean exactly by « ... will purge your "data" and "config" tarballs » All the tarballs are deleted? Or just the ones older that $KEEP?

      Thanks,

      DA+

      Delete
  4. As a comment for locking yourself out with ACLs, I find a good strategy is to change the rootDN of my databases to idNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth - since ACLs are not evaluated for rootDN, you'll always be able to do
    ldapmodify -H ldapi:/// -Y external
    provided you have root login on the ldap host.

    ReplyDelete
    Replies
    1. That's a very interesting idea. Thanks for sharing!

      Delete
  5. Hello,

    I try to implement a full OpenLDAP server on my LAN and your HowTos are very well written and helpful for me; THANKS for your very good job !

    I try to use your Backup N recorevy script to be safe but I have an error and I don't succeed to find why.

    Could you help me please ?

    This is my errors :
    ERROR: problem running config directory tar.

    When I modify the script I'm able to say problem is on /Backups/slapd/slapd.directory.20140330.tar.gz

    Second problem is that cron never run backup script and I don't understand why :
    My /etc/crontab contains :
    30 23 01 * * /home/fredouille/Scripts/Backup.slapd.sh
    and crond service is running...

    If you have an idea....

    Thanks again for your job...

    ReplyDelete
    Replies
    1. Hi there,

      For the cron error, I'd make sure the your backup script is executable. So try running this command :

      sudo chmod a+x /home/fredouille/Scripts/Backup.slapd.sh

      Then check your /var/log/cron file to see if there are any errors?

      For the tar error, what you can do is run the backup script in verbose mode and isolate the error. Like this :

      sudo sh -x /home/fredouille/Scripts/Backup.slapd.sh

      HTH,

      DA+

      P.S. Thanks for the nice words, I'm glad I could help!

      Delete
    2. Hello David,

      thanks four your reply,

      There's no problem to execute script, because :
      [root@dof3]# ls -l /home/fredouille/Scripts/
      -rwxrwxr-x. 1 fredouille fredouille 3597 30 mars 23:50 Backup.slapd.sh

      And about cron's log file I don't see anything about any error, just bad command :
      Mar 30 22:52:01 dof3 crond[1987]: (*system*) RELOAD (/etc/crontab)
      Mar 30 22:52:01 dof3 crond[1987]: (CRON) bad command (/etc/crontab)
      Mar 30 23:00:01 dof3 CROND[3901]: (root) CMD (/usr/lib64/sa/sa1 1 1)
      Mar 30 23:01:01 dof3 CROND[3906]: (root) CMD (run-parts /etc/cron.hourly)
      Mar 30 23:01:01 dof3 run-parts(/etc/cron.hourly)[3906]: starting 0anacron
      Mar 30 23:01:01 dof3 run-parts(/etc/cron.hourly)[3916]: finished 0anacron
      Mar 30 23:01:48 dof3 crontab[3949]: (root) BEGIN EDIT (root)
      Mar 30 23:10:01 dof3 CROND[3991]: (root) CMD (/usr/lib64/sa/sa1 1 1)

      And about sudo -x /home/fredouille/Scripts/Backup.slapd.sh :
      [root@dof3 ~]# sh -x /home/fredouille/Scripts/Backup.slapd.sh
      + umask 022
      + PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin
      + export PATH
      ++ date +%Y%m%d
      + DATE=20140331
      + BACKUP_DIR=/home/fredouille/Backups/slapd
      + BACKUP_DATA_FILENAME=slapd.data.20140331.ldif
      + BACKUP_CONFIG_FILENAME=slapd.config.20140331.ldif
      + BACKUP_TAR_FILENAME=slapd.directory.20140331.tar.gz
      + CA_TLS_CERT=/etc/pki/tls/certs/rootca.crt
      + DIT_CONFIG=cn=config
      + DIT_SUFFIX=dc=berok,dc=org
      + SLAPD_CONFIG_FILENAME=/etc/sysconfig/ldap
      + SLAPD_DIR=/etc/openldap
      + SLAPD_LOG_ROTATION=/etc/logrotate.d/slapd
      + SLAPD_TLS_CERT=/etc/pki/tls/certs/dof3.berok.org.cer
      + SLAPD_TLS_KEY=/etc/pki/tls/certs/dof3.berok.org
      + SLAPCAT_OPTIONS='-F /etc/openldap/slapd.d'
      + LOGFILE=/var/log/slapd.log
      + KEEP=30
      + '[' '!' -f /var/log/slapd.log ']'
      ++ id -u
      + '[' 0 -ne 0 ']'
      + '[' '!' -d /home/fredouille/Backups/slapd ']'
      ++ find /home/fredouille/Backups/slapd -type f -name 'slapd.*' -print
      ++ wc -l
      + FILES=5
      + '[' 5 -gt 30 ']'
      + slapcat -F /etc/openldap/slapd.d -b dc=berok,dc=org -l /home/fredouille/Backups/slapd/slapd.data.20140331.ldif
      + '[' 0 -eq 0 ']'
      + gzip -f /home/fredouille/Backups/slapd/slapd.data.20140331.ldif
      + '[' 0 -ne 0 ']'
      + slapcat -F /etc/openldap/slapd.d -b cn=config -l /home/fredouille/Backups/slapd/slapd.config.20140331.ldif
      + '[' 0 -eq 0 ']'
      + gzip -f /home/fredouille/Backups/slapd/slapd.config.20140331.ldif
      + '[' 0 -ne 0 ']'
      + BACKUP_FILES_LIST='/etc/pki/tls/certs/rootca.crt /etc/sysconfig/ldap /etc/openldap /etc/logrotate.d/slapd /etc/pki/tls/certs/dof3.berok.org.cer /etc/pki/tls/certs/dof3.berok.org'
      + tar zcf /home/fredouille/Backups/slapd/slapd.directory.20140331.tar.gz /etc/pki/tls/certs/rootca.crt /etc/sysconfig/ldap /etc/openldap /etc/logrotate.d/slapd /etc/pki/tls/certs/dof3.berok.org.cer /etc/pki/tls/certs/dof3.berok.org
      + '[' 2 -ne 0 ']'
      + echo 'ERROR: problem running config directory tar.'
      + tee -a /var/log/slapd.log
      ERROR: problem running config directory tar.
      + rm /home/fredouille/Backups/slapd/slapd.directory.20140331.tar.gz
      + exit 1

      Honestly I don't see where I made a mistake.....

      Fredouille

      Delete
    3. Hello Fredouille,

      The key here are these lines :

      + BACKUP_FILES_LIST='/etc/pki/tls/certs/rootca.crt /etc/sysconfig/ldap /etc/openldap /etc/logrotate.d/slapd /etc/pki/tls/certs/dof3.berok.org.cer /etc/pki/tls/certs/dof3.berok.org'
      + tar zcf /home/fredouille/Backups/slapd/slapd.directory.20140331.tar.gz /etc/pki/tls/certs/rootca.crt /etc/sysconfig/ldap /etc/openldap /etc/logrotate.d/slapd /etc/pki/tls/certs/dof3.berok.org.cer /etc/pki/tls/certs/dof3.berok.org
      + '[' 2 -ne 0 ']'
      + echo 'ERROR: problem running config directory tar.'

      The '[' 2 -ne 0 ']' means that the script only checks the return code of the previous tar command. In this case, the value is 2 while it should be zero. This is what triggers the ERROR line.

      To debug this, try running this command :

      sudo tar zcf /home/fredouille/Backups/slapd/slapd.directory.20140331.tar.gz /etc/pki/tls/certs/rootca.crt /etc/sysconfig/ldap /etc/openldap /etc/logrotate.d/slapd /etc/pki/tls/certs/dof3.berok.org.cer /etc/pki/tls/certs/dof3.berok.org

      What happens? What is the error message?

      HTH,

      DA+

      Delete
    4. Oups David,

      I think I've found my mistake...
      By reading my answer I discover that one of my parameter was wrong :
      + SLAPD_TLS_KEY=/etc/pki/tls/certs/dof3.berok.org

      and right definition is + SLAPD_TLS_KEY=/etc/pki/tls/certs/dof3.berok.org.key

      Now script is working, I'll schedule cron to verify this night...

      I'm really stupid and so sorry to disturb you with my mistakes.....

      Again, thanks a lot for your help and for your HowTos which helps me a lot...

      Should it be possible to contact you for some others specific points that I would enable on OpenLDAP (samba, Squid, named and dhcp, to have complete SSO on ldap) ??

      => I will understand if you will refuse...
      Here is my mail address : berok37@hotmail.com

      Delete
    5. Hello Fredouille,

      No worries mate, there are no stupid questions, just stupid answers ;)

      I'm glad you found your issue, that's the main thing.

      As for a full SSO on LDAP, if you follow my other posts in this OpenLDAP series and do the Kerberos one, that should give you SSO.

      HTH,

      David

      Delete
    6. Thanks David,

      It's pleasant to have found myself my issue, and again thanks for your answers..

      About SSO on LDAP, I try to understand each point on your other posts and wish I'll succeed...
      And if you have any link to follow about linking Samba and Squid, I would appreciate a lot.

      Again, please continue your posts and thanks for your help....

      Bye,

      Fredouille

      Delete

Note: Only a member of this blog may post a comment.