Friday, June 8, 2012

HOWTO : OpenLDAP 2.4 Replication on CentOS 6.2

We continue our OpenLDAP 2.4 on CentOS 6.2 with a description on how to setup  between two OpenLDAP 2.4 servers. This happens to be the final bullet point in our list of goals :
  1. Install OpenLDAP 2.4.
  2. Configure Transport Layer Security (TLS).
  3. Manage users and groups in OpenLDAP.
  4. Configure pam_ldap to authenticate users via OpenLDAP.
  5. Use OpenLDAP as sudo's configuration repository.
  6. Use OpenLDAP as automount map repository for autofs.
  7. Use OpenLDAP as NFS netgroup repository again for autofs.
  8. Use OpenLDAP as the Kerberos principal repository.
  9. Setup OpenLDAP backup and recovery.
  10. Setup OpenLDAP replication.
Of course the first thing to do in order to be able to replication our DIT is to have another CentOS machine. So go ahead and install it on a seperate computer. We will continue with our example two machines : alice and bob. Alice is the current OpenLDAP server while bob was the client. At the end of this document, bob will be the second OpenLDAP server. Which in OpenLDAP syncrepl parlance, we have these entities :
  • provider : alice.company.com (a.k.a. master server)
  • consumer : bob.company.com (a.k.a. replica server)
Another important thing to do is to read the OpenLDAP replication chapter in the administrator's guide. The following exerpt is of particular interest :
Syncrepl supports both pull-based and push-based synchronization. In its basic refreshOnly synchronization mode, the provider uses pull-based synchronization where the consumer servers need not be tracked and no history information is maintained. The information required for the provider to process periodic polling requests is contained in the synchronization cookie of the request itself. To optimize the pull-based synchronization, syncrepl utilizes the present phase of the LDAP Sync protocol as well as its delete phase, instead of falling back on frequent full reloads. To further optimize the pull-based synchronization, the provider can maintain a per-scope session log as a history store. In its refreshAndPersist mode of synchronization, the provider uses a push-based synchronization. The provider keeps track of the consumer servers that have requested a persistent search and sends them necessary updates as the provider replication content gets modified.
Replication is handeled by an OpenLDAP overlay. Check the slapo-syncprov(5) man page for the provider overlay information. With that in mind, we will setup a refreshAndPersist replication using the delta-syncrepl replication scheme. Note that, as the official documentation says :
As you can see, you can let your imagination go wild using Syncrepl and slapd-ldap(8) tailoring your replication to fit your specific network topology.

Provider Configuration


Setting up delta-syncrepl requires configuration changes on both the master (i.e. provider) and replica (i.e. consumer) servers. We will start by configuring the provider machine (i.e. alice.company.com) and then continue to the consumer machine (i.e. bob.company.com).

So, connect to the provider server.

ssh alice.company.com

On this machine, we need to setup several things :
  1. A cn=module configuration.
  2. An accesslog database to store the accesslog data (i.e. cn=accesslog)
  3. A syncprov overlay over the accesslog database.
  4. Two overlays over our primary database (i.e. dc=company,dc=com)
  5. A new user object to authenticate and fetch the data.
  6. Limits and ACLs to the new object.
Let's configure those items one at a time.

Provider Module Configuration


To configure a module on the provider, we first need to check if we have one? Since we have a Kerberos realm and SASL GSSAPI authentication setup, let's use this to simplify our queries. Note that you can always use the cn=admin,dc=company,dc=com RootDN to perform all the tasks in this blog post, but the queries are longer to write.

kinit -p drobilla/admin@COMPANY.COM
ldapsearch -ZLLLb cn=config olcModulePath

The above query did not return the olcModulePath object. So we need to create it. But what is our module path? The cn=module documentation shows us that module names end in « .la ». With that info, a simple rpm query will show us where they're stored on the filesystem.

rpm -ql openldap-servers | grep '\.la$'
/usr/lib/openldap/accesslog.la
/usr/lib/openldap/auditlog.la
/usr/lib/openldap/collect.la
/usr/lib/openldap/constraint.la
/usr/lib/openldap/dds.la
/usr/lib/openldap/deref.la
/usr/lib/openldap/dyngroup.la
/usr/lib/openldap/dynlist.la
/usr/lib/openldap/memberof.la
/usr/lib/openldap/pcache.la
/usr/lib/openldap/ppolicy.la
/usr/lib/openldap/refint.la
/usr/lib/openldap/retcode.la
/usr/lib/openldap/rwm.la
/usr/lib/openldap/seqmod.la
/usr/lib/openldap/smbk5pwd.la
/usr/lib/openldap/sssvlv.la
/usr/lib/openldap/syncprov.la
/usr/lib/openldap/translucent.la
/usr/lib/openldap/unique.la
/usr/lib/openldap/valsort.la

Ok, so we know our olcModulePath is /usr/lib/openldap. We also know that we have quite a bunch of different modules available. Let's write another LDIF file to set the olcModulePath, but also which modules we want to load. Since we need both the accesslog and the syncprov overlays, we might as well load them right?!


Load this new configuration.

ldapmodify -aZf ~/ldap/module.ldif

Check to see if it's installed? Then what's in it?

ldapsearch -ZLLLb cn=config dn | grep module
ldapsearch -ZLLLb cn=module{0},cn=config
dn: cn=module{0},cn=config
objectClass: olcModuleList
cn: module{0}
olcModulePath: /usr/lib/openldap
olcModuleLoad: {0}accesslog.la
olcModuleLoad: {1}syncprov.la

Good. We can proceed to the next provider objective.

Provider Accesslog Database


Now that we have the accesslog overlay module loaded, we must create a database in which to store the accesslog data. We of course do this with another LDIF file.

vi ~/ldap/accesslog.ldif 

As we can see, this database uses a new directory that must be created first. We also need to drop a DB_CONFIG file in there and fix permissions.

sudo mkdir -p /var/lib/ldap/accesslog
sudo cp `rpm -ql openldap-servers | grep DB_CONFIG` /var/lib/ldap/accesslog/DB_CONFIG
sudo chown -R ldap:ldap /var/lib/ldap

Alright, we can now create the new accesslog database. Note that we're using the hdb instead of the bdb for this database. There's no real reason, choose whichever you prefer.

ldapmodify -aZf ~/ldap/accesslog.ldif

We should now have a new database. Let's see if that's true?

ldapsearch -ZLLLb cn=config dn | grep hdb
dn: olcDatabase={3}hdb,cn=config

What does it contain?

ldapsearch -ZLLLb olcDatabase={3}hdb,cn=config
dn: olcDatabase={3}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {3}hdb
olcDbDirectory: /var/lib/ldap/accesslog
olcSuffix: cn=accesslog
olcRootDN: cn=admin,dc=company,dc=com
olcDbIndex: default eq
olcDbIndex: entryCSN,objectClass,reqEnd,reqResult,reqStart

Great! Let's continue with our next objective.

Provider Syncprov Overlay Over the Accesslog Database


Our next objective is to setup a syncprov overlay on the new accesslog database. Create this LDIF file :

And add the new setup to our OpenLDAP server.

ldapmodify -aZf ~/ldap/overlay.accesslog.ldif 

We should now have a new dn: in our olcDatabase={3}hdb,cn=config database.

ldapsearch -ZLLLb olcDatabase={3}hdb,cn=config

dn: olcDatabase={3}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {3}hdb
olcDbDirectory: /var/lib/ldap/accesslog
olcSuffix: cn=accesslog
olcRootDN: cn=admin,dc=company,dc=com
olcDbIndex: default eq
olcDbIndex: entryCSN,objectClass,reqEnd,reqResult,reqStart

dn: olcOverlay={0}syncprov,olcDatabase={3}hdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: {0}syncprov
olcSpNoPresent: TRUE
olcSpReloadHint: TRUE

Sure enough, it's there. We can now continue with our setup.

Provider Overlays On Primary Database


Do do this objective, we must of course write another LDIF file in which we will a) setup new indexes to our primary database, b) add the syncprov overlay and c) add the accesslog overlay.

vi ~/ldap/overlay.primary.ldif

We then add this new LDIF into our system.

ldapmodify -aZf ~/ldap/overlay.primary.ldif 

Which then gives us two new overlays on the primary database.

ldapsearch -ZLLLb olcDatabase={1}bdb,cn=config dn
dn: olcDatabase={1}bdb,cn=config
dn: olcOverlay={0}syncprov,olcDatabase={1}bdb,cn=config
dn: olcOverlay={1}accesslog,olcDatabase={1}bdb,cn=config

And sure enough, we do have the two new overlays.

Provider Replication User


We need a user for the replication. That user will be used to authenticate the replication server and read the data. Period. Once again, we need an LDIF file.

vi ~/ldap/replication.ldif 


Apply this LDIF file.

ldapmodify -aZf ~/ldap/replication.ldif 

This user needs a real password. Make sure to record this password and keep it safe.

ldappasswd -xZ -S cn=replication,dc=company,dc=com

Note that in this example our replication user is named just « replication ». If you plan on having more than one replicated server, then ideally choose a unique name for each of the replicated server. This way you can audit which machine replicates and when. You can also decide which part of the DIT gets replicated to which machine. But that's another story.

Provider Limits and ACLs to the Replication User


We now need to give access via ACLs and some limits to the new user. Let's do it in two steps starting with the limits (or lack of actually :)

vi ~/ldap/limits.ldif

Then apply this LDIF.

ldapmodify -Zf ~/ldap/limits.ldif

Next step is to give read access to the replication user. Before we do this, it's usually a good idea to double-check our current ACL list.

ldapsearch -LLLb cn=config olcAccess

We can now create another LDIF file to update our ACLs. The idea here is to have the same sets of ACLs on both the provider and the consumer. Except that on the consumer, we don't need any ACLs with regards to the replication user (i.e. the one defined in olcSyncRepl:).

vi ~/ldap/consumer.acl.ldif

Enable the ACL changes.

ldapmodify -f ~/ldap/consumer.acl.ldif

Check the ACL listing.

ldapsearch -Zb olcDatabase={1}bdb,cn=config olcAccess

Test to see if we can access our entire DIT with the replication DN?

ldapsearch -xZWD cn=replication,dc=company,dc=com -H ldap://alice.company.com

This is of course crucial. Do not continue with this blog post until you can access the entire DIT with the replication DN. Once it works, we need to setup our consumer OpenLDAP machine.

Consumer Configuration


Consumer OpenLDAP Installation


Install another CentOS 6 machine and connect to it.

ssh bob.company.com

Make sure that ntpd(8) is running and that both alice and bob's clocks are synchronized. All of your servers' clocks must be tightly synchronized using either NTP(See http://www.ntp.org/ for info on NTP), an atomic clock or some other reliable time reference.

ntpq -p

Then connect to bob.company.com and install a few packages.

sudo yum -y install openldap-servers cyrus-sasl-gssapi pam_krb5 krb5-server-ldap

Configure an empty OpenLDAP server. I've already covered how to do this in a previous blog post, but we now need to change this a little. So we can't really follow the other post. Let's do it all over again then. So the first thing we must do is upgrade the sudo package to have the latest sudo schema. I've explained how to achieve this goal in another blog post. But I'll show here just what we need which starts by installing the latest (as of this writing) sudo package from the sudo website. Note that if your machine is 32 bit, then the package would be sudo-1.8.4-5.el6.i386.rpm.

wget http://www.sudo.ws/sudo/dist/packages/Centos/6/sudo-1.8.4-5.el6.x86_64.rpm

Upgrade sudo.

sudo rpm -U ./sudo-1.8.4-5.el6.x86_64.rpm

This sudo package comes with an OpenLDAP schema. Get the path to the file. Keep note of this path as we will include it in our slapd configuration file.

rpm -ql sudo | grep -i openldap
/usr/share/doc/sudo-1.8.5-1.el6/schema.OpenLDAP

We now need to create the config file. It needs the RootDN password, so record the output of the slappasswd(8C) command

slappasswd
New password: 
Re-enter new password: 
{SSHA}JGjPUbxyCn7wa/pt8YM5rzK7s/hUGncW

Plug the above output into the file below. To understand the syncrepl syntax, refer to the official syncrepl documentation.

mkdir ~/ldap
vi ~/ldap/slapd.conf.consumer

Create the configuration.

sudo slapcat -f ~/ldap/slapd.conf.consumer -F /tmp -n 0

Remove the old configuration.

sudo rm -rf /etc/openldap/slapd.d/*

Install the new one.

sudo cp -rp /tmp/cn\=config* /etc/openldap/slapd.d
sudo chown -R ldap:ldap /etc/openldap/slapd.d

Check to see if the new configuration is ok?

sudo slaptest -uF /etc/openldap/slapd.d
config file testing succeeded

Install the DB_CONFIG file.

egrep -vi '^$|^#' `rpm -ql openldap-servers | grep DB_CONFIG` > /tmp/DB_CONFIG
sudo mv /tmp/DB_CONFIG /var/lib/ldap/DB_CONFIG

sudo chown -R ldap:ldap /var/lib/ldap

Prepare the log system.

sudo vi /etc/rsyslog.conf

Touch the new log file.

sudo touch /var/log/slapd.log

Make sure the log file doesn't grow to humongous proportions.

sudo vi /etc/logrotate.d/slapd

Restart the rsyslog daemon so that it knows about the changes.

sudo /etc/init.d/rsyslog restart

Edit the slapd(8) system configuration file.

sudo vi /etc/sysconfig/ldap

Start the slapd(8) daemon.

sudo /etc/init.d/slapd start

Make sure the daemon starts when the server boots.

sudo chkconfig slapd on

Configure a system-wide LDAP client configuration. This is to simplifiy our life and reduce typing later on. Don't worry about the TLS configurations for now. We will configure them later, but it doesn't hurt to have them in the file at the moment. Don't forget that in this blog post, the OpenLDAP server's FQDN is bob.company.com and not alice.company.com as we used to. This is important because here we don't want to query nor modify the provider (i.e. alice.company.com), but only the consumer (i.e. bob.company.com).


Check if our admin user can connect? Double check the logs to make sure you're not binding to the provider!

ldapwhoami -WD cn=admin,dc=company,dc=com
Enter LDAP Password:
dn:cn=admin,dc=company,dc=com

Consumer TLS Configuration


Ok, let's configure TLS for the consumer. I'll still use a Windows CA for this post.

openssl req -newkey rsa:2048 -keyout `hostname`.key -nodes -out `hostname`.req -subj /CN=bob.company.com/O=Company/C=CA/ST=QC/L=Montreal

Upload the .req file to the CA machine sign it.

C:\> certreq -submit -attrib "CertificateTemplate:WebServer" -config "caserver.company.com\Company CA" bob.company.com.req bob.company.com.pem

Upload the .pem file to our consumer OpenLDAP server then place both the .pem and .key files into the proper location with the appropriate permissions.

sudo mv bob.company.com.pem bob.company.com.key /etc/pki/tls/certs
sudo chown ldap:ldap /etc/pki/tls/certs/bob.company.com.*
sudo chmod 600 /etc/pki/tls/certs/bob.company.com.key

Grab a copy of the CA's certificate from our provider server.

scp alice.company.com:/etc/pki/tls/certs/companyCA.crt /tmp
sudo mv /tmp/companyCA.crt /etc/pki/tls/certs
sudo chown root:root /etc/pki/tls/certs/companyCA.crt

Configure TLS in our consumer OpenLDAP server.

vi ~/ldap/tls.consumer.ldif

Add the TLS configuration to the cn=config base.

ldapmodify -WD cn=admin,dc=company,dc=com -H ldapi:/// -f ~/ldap/tls.consumer.ldif

Check if our configuration has been installed?

sudo ldapsearch -LLLY EXTERNAL -H ldapi:/// -b cn=config -s base | grep olcTLS

Check to see if we can connect with TLS (i.e. the « -Z » switch).

ldapwhoami -xZWD cn=admin,dc=company,dc=com -H ldap://bob.company.com

Revist the LDAP client configuration file to enable the TLS configs.

sudo vi /etc/openldap/ldap.conf

Modify the consumer cn=config and database configurations.

vi ~/ldap/consumer.ldif

Apply the changes.

ldapmodify -WD cn=admin,dc=company,dc=com -H ldapi:/// -f ~/ldap/consumer.ldif 

Check the configuration changes to both the cn=config and the primary database.

sudo ldapsearch -LLLY EXTERNAL -H ldapi:/// -b cn=config "(|(cn=config)(olcDatabase={1}bdb))"

Consumer SASL GSSAPI Configuration


Enable SASL GSSAPI authentication.

vi ~/ldap/consumer.gssapi.ldif

Add the modification to the server.

ldapmodify -xZWD cn=admin,dc=company,dc=com -H ldap://bob.company.com -f ~/ldap/consumer.gssapi.ldif

Change the system-wide OpenLDAP daemin configuration to add a Kerberos keytab.

sudo vi /etc/sysconfig/ldap

Create the OpenLDAP kerberos keytab, the the host's principal key and the autofsclient key while we're in the kadmin dialogue. IMPORTANT : notice how we place the host/ and autofsclient/ principals in the /etc/krb5.keytab while the ldap/ principal gets stored in /etc/openldap/krb5.keytab.

sudo kadmin -p drobilla/admin@COMPANY.COM
kadmin:   addprinc -randkey ldap/bob.company.com@COMPANY.COM
kadmin:   ktadd -k /etc/openldap/krb5.keytab ldap/bob.company.com@COMPANY.COM

kadmin:   addprinc -randkey host/bob.company.com@COMPANY.COM
kadmin:   ktadd -k /etc/krb5.keytab host/bob.company.com@COMPANY.COM

kadmin:   addprinc -randkey autofsclient/bob.company.com@COMPANY.COM
kadmin:   ktadd -k /etc/krb5.keytab autofsclient/bob.company.com@COMPANY.COM
kadmin:   exit

Fix permissions on the new Kerberos keytab. The goal here is to let the ldap group be able to read the ldap/ principal stored in the keytab.

sudo chown root:ldap /etc/openldap/krb5.keytab 
sudo chmod 640 /etc/openldap/krb5.keytab

Restart the slapd(8) daemon.

sudo /etc/init.d/slapd restart

Test the SASL GSSAPI authentication.

kdestroy
kinit -p drobilla/admin@COMPANY.COM
ldapwhoami

Consumer DIT Initial Load


One last step before we start the replication is to load the DIT from our provider into our consumer. It's not required, but if your DIT is large, this will save quite a lot of time. Do to this, follow these steps :

Connect to the provider.

ssh alice.company.com

Create a new LDIF file with the entire DIT of the provider.

sudo slapcat | tee -a /tmp/provider.slapcat.ldif

Transfer the LDIF file over to the consumer.

scp /tmp/provider.slapcat.ldif bob.company.com:/tmp

Remove the temporary file. After all, it's the entire DIT that's in there in clear text...

sudo rm /tmp/provider.slapcat.ldif

Connect to the consumer.

ssh bob.company.com

Stop slapd.

sudo /etc/init.d/slapd stop

Destroy the database, but save the DB_CONFIG file.

sudo cp /var/lib/ldap/DB_CONFIG /tmp
sudo rm -rf /var/lib/ldap
sudo mkdir /var/lib/ldap
sudo mv /tmp/DB_CONFIG /var/lib/ldap

Load the provider LDIF file into the consumer's database. Notice the « -w » switch to slapadd(8C) which will write syncrepl context information. Once all entries are added, the contextCSN will be updated with the greatest CSN in the database. That's pretty handy in our case :)

sudo slapadd -l /tmp/provider.slapcat.ldif -w

Again, remove the temporary file. After all, it's the entire DIT that's in there in clear text...

sudo rm /tmp/provider.slapcat.ldif

Start the consumer daemon.

sudo /etc/init.d/slapd start

Check to see if the entire DIT is now on the consumer machine?

ldapsearch -xZWD cn=admin,dc=company,dc=com -b dc=company,dc=com -H ldap://bob.company.com

Check to see if the SASL GSSAPI user has the proper ACLs to this DIT on the consumer?

kinit -p drobilla/admin@COMPANY.COM
ldapsearch -ZLLLb dc=company,dc=com -H ldap://bob.company.com

Replication Test


We now have a TLS and SASL GSSAPI enabled OpenLDAP server bob.company.com configured as the consumer of our provider alice.company.com machine. Let's see if it works? Do test this, connect to the provider and change something. In this example, we will change our test.user's shell.

Connect to the provider server.

ssh alice.company.com

Check the current value for loginShell.
ldapsearch -LLLZb cn=test.user,ou=users,dc=company,dc=com loginShell
loginShell: /bin/bash

Change the loginShell value to /bin/sh.

ldapmodify <<-EOF
dn: cn=test.user,ou=users,dc=company,dc=com
changetype: modify
replace: loginShell
loginShell: /bin/sh
EOF

Keep an eye on the slapd.log file. You should now see these lines on the consumer :

slapd[6620]: do_syncrep2: rid=000 cookie=rid=000,csn=20120608192235.282075Z#000000#000#000000
slapd[6620]: syncrepl_entry: rid=000 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY)
slapd[6620]: syncrepl_entry: rid=000 be_search (0)
slapd[6620]: syncrepl_entry: rid=000 cn=test.user,ou=users,dc=company,dc=com
slapd[6620]: slap_queue_csn: queing 0xa1601040 20120608192235.282075Z#000000#000#000000
slapd[6620]: slap_graduate_commit_csn: removing 0xa16050c8 20120608192235.282075Z#000000#000#000000
slapd[6620]: syncrepl_entry: rid=000 be_modify cn=test.user,ou=users,dc=company,dc=com (0)
slapd[6620]: slap_queue_csn: queing 0xa1601040 20120608192235.282075Z#000000#000#000000
slapd[6620]: slap_graduate_commit_csn: removing 0xa1604a48 20120608192235.282075Z#000000#000#000000

And if you query the consumer, you should see that the loginShell has indeed replicated to the consumer.

ssh bob.company.com

ldapsearch -LLLZb cn=test.user,ou=users,dc=company,dc=com loginShell
loginShell: /bin/sh

Success! :)

Consumer Kerberos Slave Setup


The whole point of replicating our DIT is to have two copies of it should the provider fails. But our DIT also supports our Kerberos infrastructure. We must thus make sure that our consumer can also act as a Kerberos slave should our provier -and Kerberos master- fails.

To enable the consumer machine to become a Kerberos slave, we must of course install the required packages and we already did that (see above). We must then send several provider's Kerberos files over to our consumer. Those files are the Kerberos private key and the Kerberos stash keyfile.

ssh alice.company.com

sudo scp /var/kerberos/krb5kdc/.k5.COMPANY.COM drobilla@bob.company.com:/tmp
sudo scp /etc/krb5.d/stash.keyfile drobilla@kong.caprion.com:/tmp

Now connect to the consumer and move the files to their proper locations and fix permissions.

ssh bob.company.com
sudo mv /tmp/.k5.COMPANY.COM /var/kerberos/krb5kdc
sudo mkdir /etc/krb5.d
sudo mv /tmp/stash.keyfile /etc/krb5.d
sudo chown -R root:root /var/kerberos/krb5kdc /etc/krb5.d

Make sure to edit the Kerberos ACL file on the consumer. This file contains a single line.

sudo vi /var/kerberos/krb5kdc/kadm5.acl
*/admin@COMPANY.COM *

Edit the consumer's Kerberos kdc.conf file.


Grab a copy of the krb5.conf file on the provider machine.

scp alice.company.com:/etc/krb5.conf /tmp
sudo mv /tmp/krb5.conf /etc
sudo chown root:root /etc/krb5.conf

Modify the database to include several new indexes which will help the Kerberos LDAP lookups. While we're at it, let's also add several other required indexes :)

vi ~/ldap/kerberos.indexes.ldif
ldapmodify -xZWD cn=admin,dc=company,dc=com -H ldap://bob.company.com -f ~/ldap/kerberos.indexes.ldif

Check to see if we have the new indexes in place?

ldapsearch -ZLLLb olcDatabase={1}bdb,cn=config olcDbIndex

Make sure the krb5kdc daemon is running when the machine boots. Note that we do not run the kadmin daemon on the slave KDC.

sudo chkconfig krb5kdc on

Start the krb5kdc daemon.

sudo /etc/init.d/krb5kdc start

Consumer Backup


Don't forget to backup the consumer now that it's working. See this blog post on how to do just that.

Check Replication Status


Sometimes the log files are not clear in order to determine if the replication is finished or not. To check this, simply query both the provider and the consumer servers for the contextSCN. If the values are the same, then replication is finished.

kinit -p drobilla/admin@COMPANY.COM

ldapsearch -LLLH ldap://alice.company.com -s base -b "dc=company,dc=com" 
dn: dc=company,dc=com
contextCSN: 20140207211614.110524Z#000000#000#000000

ldapsearch -LLLH ldap://bob.company.com -s base -b "dc=company,dc=com" 
dn: dc=company,dc=com
contextCSN: 20140207211614.110524Z#000000#000#000000

As we can see, both contextSCN values are the same, so we're good.

Client Configuration


Now that we have two OpenLDAP servers, we need to configure the client machines to use them both. So perform these configuration changes to all your LDAP client machines :

ssh client.company.com

Change the sudoers LDAP configuration file.

sudo vi /etc/ldap.conf

Change the system LDAP configuration file.

sudo vi /etc/openldap/ldap.conf

Change the client's nslcd configuration file.

sudo vi /etc/nslcd.conf

Change the pam_ldap configuration file.

sudo vi /etc/pam_ldap.conf

Change the Kerberos configuation file.

sudo vi /etc/krb5.conf

As you can see, this requires quite a few changes. So you should probably script these changes. I usually setup an admin server that runs Apache with configuration files on it. Clients can thus simply wget them. Easy. Or you can use Puppet. Even better!

With the new client configuration, try shutting down the LDAP and Kerberos services on alice.company.com and see if the clients can still work by using bob.company.com.

Troubleshooting


olcLogLevel


If you're not sure about the syncrepl engine, then enable logging for this.

ldapmodify -Z <<-EOF
dn: cn=config
changetype: modify
replace: olcLogLevel
olcLogLevel: stats sync
EOF

You can do this on both the consumer and the provider.

syncrepl_message_to_entry


If you get this type of error :

syncrepl_message_to_entry: rid=000 mods check (objectClass: value #1 invalid per syntax)

It means that you're missing a schema in on the consumer server. Of course the rid=000 can be different on your server. It's the replication ID configured in the olcSyncrepl: config of the consumer server. Compare the schemas on both machines and fix the consumer so that it has exactly the same schemas as the provider. So, the first thing you must do is check the schemas on the provider :

ssh alice.company.com

ldapsearch -ZLLLb cn=schema,cn=config dn

Then connect to the consumer and check the schemas there :

ssh bob.company.com
ldapsearch -ZLLLb cn=schema,cn=config dn

If the consumer is missing some schemas present on the provider, then add those missing schemas to the consumer and try the replication again.

Missing DB_CONFIG, but it's there?


Your logs show these error messages when you start slapd :

slapd[7722]: hdb_db_open: warning - no DB_CONFIG file found in directory /var/lib/ldap/accesslog: (14).#012Expect poor performance for suffix "cn=accesslog".
slapd[7722]: slapd starting
slapd[7722]: <= bdb_equality_candidates: (objectClass) not indexed
slapd[7722]: <= bdb_inequality_candidates: (reqStart) not indexed

But when you look into the /var/lib/ldap/accesslog directory, there is a DB_CONFIG file and the permissions are good.

What's the problem?

Well, it's simply that the HDB has the wrong « objectClass: olcDdbConfig ». It should be « objectClass: olcHdbConfig ». Notice the small, but very critical, difference?

That's it! 

That means we finished our initial goals :
  1. Install OpenLDAP 2.4.
  2. Configure Transport Layer Security (TLS).
  3. Manage users and groups in OpenLDAP.
  4. Configure pam_ldap to authenticate users via OpenLDAP.
  5. Use OpenLDAP as sudo's configuration repository.
  6. Use OpenLDAP as automount map repository for autofs.
  7. Use OpenLDAP as NFS netgroup repository again for autofs.
  8. Use OpenLDAP as the Kerberos principal repository.
  9. Setup OpenLDAP backup and recovery.
  10. Setup OpenLDAP replication.
The next set of goals are coming. I want to enable Referential Integrity in the LDAP DIT (i.e. when you delete a user it is also deleted from the various groups he's part of). I'm also interested in pulling information, such as users, passwords and grous from Active Directory servers and thus remove Samba.

HTH, Let me know if I made a mistake as this post was not easy to put together.
DA+

31 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. How to make this kind of query without Kerberos?

    ldapmodify -aZf ~/ldap/module.ldif

    Thanks a lot for the great article series! ;)

    ReplyDelete
    Replies
    1. ldapmodify -Y EXTERNAL -H ldapi:/// -af ~/ldap/module.ldif

      Delete
  3. @Roberto : anywhere that you see an ldapmodify and/or an ldapquery, you can always use a simple bind with your RootDN by using the following options :

    ldapmodify -xZWD cn=admin,dc=company,dc=com -af file.ldif

    HTH,

    DA+

    ReplyDelete
  4. Thanks, David! My lab is going fine till now.

    Just a little mistake you did - you've missed the ldapmodify query on section "Provider Overlays On Primary Database".

    Thanks a lot, again.

    ReplyDelete
  5. @Roberto : ah well, mistakes happen! Could you be so kind as to point the exact command that I've forgotten and where is goes? This way I can update the blog post.

    Many thanks! :)

    DA+

    ReplyDelete
  6. Is it possible to use multiple LDAP servers? in which the clients will chose from or maybe in round robin fashion.

    ReplyDelete
    Replies
    1. @Aih : I've been asking myself this very same question. But admit I don't have enough free time to test the various ideas that came to my mind. They are :

      a) DNS CNAME.
      b) Very short timeout values in client's ldap.conf.
      c) Configure LVS in front of OpenLDAP servers.
      d) Use hardware load balancers.

      Now, here's a rundown of what I *think* might happen for those ideas. Recall that I haven't tested any of them.

      a) DNS CNAME.

      Create a CNAME in your DNS, say ldap.company.com. which points in a round-robin fashion (search the web for « bind dns round robin configuration » and see what I mean). It would work. But the various problems I can see are :

      1. What if one LDAP server is down? The DNS resolver on the client will have cached the IP of your CNAME and will thus always retry that IP. Even if your timeouts in your client's ldap.conf file are low, the DNS will always return the same IP for the duration of the DNS TTL. This could cause quite a bunch of frustration.

      2. TLS. How will you configure TLS when the IP of your CNAME changes? Not only that, but the hostname of your various LDAP servers will like as not be the same. So certificate verification will result in an error on the client side. That means you have to configure clients not to check certificates, which reduces your overall site's security.

      b) Very short timeout values.

      This could probably work. It doesn't break TLS and if a server is down, the clients will see it. But it would mean testing and testing to get the right time out values.

      c) Configure LVS in front of OpenLDAP servers.

      See LVS if you don't know that great software yet! http://www.linuxvirtualserver.org/
      I've successfully deployed LVS for Oracle database servers and Tomcat application servers. But in both cases, neither were using TLS. Again, LVS would be great to test. But the TLS issue would still nag me. At least if an OpenLDAP server is down, then the LVS cluster would not send it's IP to the clients.

      d) Use hardware load balancers.

      Well, one would need hardware load balancers to test this. I don't. And I haven't yet seen an Open Source hardware load balancer :) Maybe the Elfiq link load balancers would cut it? See http://www.elfiq.com/ for more info.

      HTH,

      DA+

      Delete
    2. Actually I'm in the process of using HA Proxy instead of LVS. It's appears to be a more complete software load balancing solution.

      Check this nice blog post on HA Proxy
      https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts

      HTH,

      David

      Delete
  7. Hi David,

    Thank you for providing this how-to guide for OpenLDAP on CentOS 6.2, it has been very helpful.

    I have been tasked with the duty of setting up a replication of my employers current Red Hat 6 OpenLDAP server,and have been following your guide, but I have run into a bit of a brick wall.

    For testing purposes, I have been working on a VM installation of CentOS 6, and have been mostly successful, until I reached the "Provider Overlays On Primary Database" section in the last bullet point.

    Someone had mentioned before that you forgot to include a ldapmodify command for that section, but he apparently didn't follow up with the exact location of the missing command.

    As mentioned it is in the "Provider Overlays On Primary Database" section, where you stated:

    "Do do this objective, [ You said do do ;-) ] we must of course write another LDIF file in which we will a) setup new indexes to our primary database, b) add the syncprov overlay and c) add the accesslog overlay."

    You then listed the vi command "vi ~/ldap/overlay.primary.ldif"

    And then stated:

    "We then should have new overlays on the primary database."

    It's after the vi command that you forgot to include the ldapmodify command.

    I'm not running a Kerberos server either, so I have been using the simple bind method with my Root DN. I attempted to add the "overlay.primary.ldif" this way, but the ldif you have posted appears to have a syntax error in it, that I tried to correct.

    In this section of the "overlay.primary.ldif" you have:

    # Add the syncprov overlay on the dc=caprion,dc=com database.
    dn: olcOverlay=syncprov,olcDatabase={1}bdb,cn=config
    objectClass: olcOverlayConfig
    objectClass: olcSyncProvConfig
    olcOverlay: syncprov
    olcSpCheckPoint: 500 15

    The commented out dc=caprion,dc=com part, I presume was suppossed to be dc=company,dc=com, referring to the main database.

    But in the next section, you forgot to comment out the first line:
    "Add the accesslo overlay on the dc=company,dc=com database.", which I did add a comment to my ldif file, so the two lines are commented out.
    # Add the accesslo overlay on the dc=company,dc=com database.
    # scan the accesslog DB every day, and purge entries older than 7 days

    I then ran this command to add the ldif file:
    ldapmodify -xWD cn=Manager,dc=company,dc=com -af ~/ldap/overlay.primary.ldif - NOTE: I'm using Manager instead of admin.

    This appeared to add the ldif file, but didn't appear to add the syncprov or accesslog sections of the overlay.primary.ldif file.

    When I run the search command it returns only the following:
    ldapsearch -xLLLWD cn=Manager,dc=company,dc=com -b olcDatabase={1}bdb,cn=config dn
    Enter LDAP Password:
    dn: olcDatabase={1}bdb,cn=config

    Was the command I used to add the ldif incorrect, or is there something missing in the ldif file?

    I'm not real familar with the ldap commands, but I used what I thought was correct.

    Again, thanks for providing this guide.

    Kevin

    ReplyDelete
    Replies
    1. Hello Kevin,

      Thanks for pointing out errors in my blog. I try to debug everything before I post, but even if I'd like to think I'm perfect (!) I'm obviously not! lol :)

      I'm at home right now, so I'll check this out tomorrow at work where I have all the servers configured.

      But until then, I don't see any problem if you use the RootDN user (admin, Manager, root, whatever). It should work.

      Anyway, talk to you soon!

      DA+

      Delete
    2. Hello Kevin,

      Ok, I've fixed the "Provider Overlays On Primary Database" section with the appropriate ldapmodify command. I must admit I haven't tested it, so be wary! I did keep the "Do do" part ;)

      I also fixed the syntax errors in the "overlay.primary.ldif" file. It should not list my current employer anymore and have the correct comment lines starting with a pound (#) character.

      Thanks for letting me know about all those errors.

      Now, concerning your ldapsearch that doesn't return the info you're looking for. When I run this command :

      ldapsearch -xLLLWD cn=admin,dc=company,dc=com -b olcDatabase={1}bdb,cn=config dn
      Enter LDAP Password:
      dn: olcDatabase={1}bdb,cn=config

      dn: olcOverlay={0}syncprov,olcDatabase={1}bdb,cn=config

      dn: olcOverlay={1}accesslog,olcDatabase={1}bdb,cn=config

      dn: olcOverlay={2}refint,olcDatabase={1}bdb,cn=config

      You say this is not what you're getting?

      Make sure you installed the "overlay.primary.ldif" to the right OpenLDAP server. In other words, this LDIF file is for the primary OpenLDAP machine and NOT for the slave one. To be totally sure of which machine you're working on, use the -H flag to any ldap commands. In your case, try this (replace master.company.com by your own master server of course) :

      ldapsearch -xLLLWD cn=Manager,dc=company,dc=com -H ldap://master.company.com -b olcDatabase={1}bdb,cn=config dn

      Another way to find out if the "overlay.primary.ldif" file was properly installed is to look into the OpenLDAP directory. Like this :

      sudo ls -1 /etc/openldap/slapd.d/cn\=config/olcDatabase\=\{1\}bdb

      It should return these three lines :

      olcOverlay={0}syncprov.ldif
      olcOverlay={1}accesslog.ldif
      olcOverlay={2}refint.ldif

      If not, then we have to investigate further. If you see those lines, then you may have some ACL issues preventing you from seeing this. But since you're working as the RootDN, this should not be a problem as the RootDN always has access to the data.

      You could also make sure that your database is {1} and not {2} or something else. To list all the olcDatabase configured in your server, try this :

      ldapsearch -xLLLWD cn=admin,dc=company,dc=com -b cn=config -s one dn
      Enter LDAP Password:
      dn: cn=module{0},cn=config

      dn: cn=schema,cn=config

      dn: olcDatabase={-1}frontend,cn=config

      dn: olcDatabase={0}config,cn=config

      dn: olcDatabase={1}bdb,cn=config

      dn: olcDatabase={2}monitor,cn=config

      dn: olcDatabase={3}hdb,cn=config

      Also, if you use "hdb" instead of "bdb", make sure you update your commands accordingly.

      HTH,

      DA+

      Delete
  8. David
    I followed this to setup between Debian slapd 2.4.23-7.2 and CentOS openldap-servers-2.4.23-26.el6_3.2.i686 both using cn=config method. All was perfect until the actual replication. The initial sync of the cn=admin,dc=example,dc=com conflicts with the core schema - that says for a simpleSecurityObject a MUST for the userPassword. You can create and LDIF with a MAY for the initial sync ( when slave DB empty ) then reset to MUST after. The specific olcObjectClasses is ( don't forget to include all of them from /etc/openldap/schema/core.ldif
    olcObjectClasses: ( 0.9.2342.19200300.100.4.19 NAME 'simpleSecurityObject'
    DESC 'RFC1274: simple security object'
    SUP top AUXILIARY
    MAY userPassword )
    This only is required for whole initial sync from provider.

    BTW - the write up is perfect otherwise. It may just have been my stuff but you may want to just put this in as a now

    ReplyDelete
    Replies
    1. Hello Anonymous,

      Thanks for the information. I must admit I haven't tried to configure OpenLDAP replication between to different OS. But still, one would not think the OS could cause any OpenLDAP issues.

      So what is the end result of your specific setup? Is the replication working? What error messages did you receive and what have you done to fix them?

      Finally, may I ask why you use different OS for your OpenLDAP servers? Legacy perhaps?

      Many thanks,

      DA+

      Delete
  9. David,

    I'm racking my brain here trying to get past applying the limits.ldif file to the master. I did enough of the Kerberos portion so as to enable users to change their LDAP passwords but didn't proceed any further-- not sure if that's part of the problem or not-- it doesn't seem like it should interfere. Anyway, without a KDC up and running I tried applying the limits.ldif like so:

    ldapmodify -a -xZWD cn=admin,dc=company,dc=com -f limits.ldif

    The output is:

    adding new entry "olcDatabase={3}hbd,cn=config"
    ldap_add: Undefined attribute type (17)
    additional info: add: attribute type undefined

    I looked around on Google and there was mention of the ldif file needing white space between dn entries, which yours has. I even tried splitting them into separate ldif files and no luck with either. It seems to me like it doesn't even understand what olcLimits even is? Or perhaps it's choking on one of its attributes? Any help would be greatly appreciated-- I've absolutely LOVED this series of tutorials and can't thank you enough!

    -LP

    ReplyDelete
    Replies
    1. Hi LP,

      Maybe you have different database index numbers than I have? What does this query returns? (Note, if you don't have a Kerberos user/admin@REALM.COM user, then use the second version of the query)

      # a) Kerberos version...
      ldapsearch -LLLZb cn=config -s one dn

      # b) OpenLDAP admin user version...
      ldapsearch -xLLLWD cn=admin,dc=company,dc=com -b cn=config -s one dn

      For example, mine returns this :

      dn: cn=module{0},cn=config
      dn: cn=schema,cn=config
      dn: olcDatabase={-1}frontend,cn=config
      dn: olcDatabase={0}config,cn=config
      dn: olcDatabase={1}bdb,cn=config
      dn: olcDatabase={2}monitor,cn=config
      dn: olcDatabase={3}hdb,cn=config

      As you can see, I have four databases indexed from -1 to 3. The limits.ldif file has to apply to the same ones. Make sure that is the case with your own setup.

      Let me know how things turn up?

      And thanks for the good words! :)

      HTH,

      DA+

      Delete
    2. Going the OpenLDAP admin user route, I received the same output to your query mentioned above. The limits.ldif file is:

      # limits.ldif
      dn: olcDatabase={3}hdb,cn=config
      add: olcLimits
      olcLimits: dn.exact="cn=replication,dc=company,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited

      dn: olcDatabase={1}bdb,cn=config
      add: olcLimits
      olcLimits: dn.exact="cn=replication,dc=company,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited

      # EOF

      And it's still giving me the same error. Here's what /var/log/slapd.log is reporting:

      conn=1070 op=2 ADD dn="olcDatabase={3}hdb,cn=config"
      conn=1070 op=2 RESULT tag=105 err=17 text=add: attribute type undefined
      conn=1070 op=3 UNBIND

      And the error on the screen is the same error I mentioned above... What do you think the problem could be?

      Thanks,

      -LP

      Delete
  10. David,

    Well it's a perfect day for egg on my face... Did some googling around and wound up here: http://support.apple.com/kb/TS4462?viewlocale=en_US and I thought "What the Hell, why not throw a changeType: modify into each of the entries in the limits.ldif file" and it worked! So I used the following command:

    ldapmodify -v -a -xZWD cn=admin,dc=company,dc=com -f limits.ldif

    to apply this limits.ldif file:
    -----------
    # Add limits to the cn=accesslog database.
    dn: olcDatabase={3}hdb,cn=config
    changetype: modify
    add: olcLimits
    olcLimits: dn.exact="cn=replication,dc=company,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited

    # Add limits to the dc=company,dc=com database.
    dn: olcDatabase={1}bdb,cn=config
    changetype: modify
    add: olcLimits
    olcLimits: dn.exact="cn=replication,dc=company,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited

    # EOF
    -----------

    And it worked perfectly! I can't believe I didn't see that from all the previous ACL modifications! Anyway, I'm moving on and will be happy to post feedback when I'm done. Thanks again!

    -LP

    ReplyDelete
    Replies
    1. Hello LP,

      Great, that's good news! Thanks for sharing the solution. I'll update the limits.ldif file on my blog so that others can benefit from your work.

      Cheers!

      DA+

      Delete
  11. Hi David,
    Could you please help me regarding the below errors. I am facing issue while adding the module.ldif
    docsldapmodify -xZWD cn=admin,dc=mdt,dc=ftcrm,dc=accenture,dc=com -af module.ldif
    Enter LDAP Password:
    adding new entry "cn=module,cn=config"
    ldap_add: Other (e.g., implementation specific) error (80)
    additional info: internal error (cannot create file)

    ReplyDelete
    Replies
    1. [root@pvzftldap02 ldap]# ldapmodify -v -xWD cn=admin,dc=mdt,dc=ftcrm,dc=accenture,dc=com -af module.ldif
      ldap_initialize( )
      Enter LDAP Password:
      add objectClass:
      olcModuleList
      add cn:
      module
      add olcModulePath:
      /usr/lib64/openldap
      add olcModuleLoad:
      accesslog.la
      syncprov.la
      adding new entry "cn=module,cn=config"
      ldap_add: Other (e.g., implementation specific) error (80)
      additional info: handler exited with 1

      Delete
    2. Hello Anonymous,

      The « cannot create file » error seems to me that the OpenLDAP server doesn't have write access to it's configuration directory. That's the first place I would check.

      Can you please double check your /etc/openldap permissions and make sure the slapd process has the right to write in there?

      HTH,

      DA+

      Delete
  12. David,

    Some pretty cool stuff! I got replication working but the wheels fall off when it comes to the Kerberos replication and I have it narrowed down it being an ACL problem. Would you be so kind as to post an expected ACL dump of both alice and bob when replication is all done? That would definitely help me understand this stuff better. Again, great work!

    Thanks!

    -LP

    ReplyDelete
    Replies
    1. Hi LP,

      Unfortunately, my OpenLDAP machines are located in a corporation for which I don't work for anymore. So I'll have to rebuild my entire setup from scratch! How funny can that be hey? But I'm supposed to go there next week for a brain dump to the person taking my old position. I'll try to do an ACL dump on both master and slave.

      Stay tuned!

      And thanks for the great words, glad I could help!

      DA+

      Delete
    2. Ugh that sucks! But rebuilding can help catch things and refine them further. I've always kept in the back of my mind just how one could use puppet to augment some of this but haven't gotten to that point of tinkering with it yet. I'll keep sloggin' away and see if I can figure it out. Staying tuned.

      -LP

      Delete
  13. Hi David,

    Is possible to pointing slave server from ldap master server ?
    Commonly on many reference about ldap replication always give configuration about provider = ldap://[provider] on slave server.

    I have case which i build many master server to replicate to one slave ldap server.Master server have dynamic ip.So not possible for me to pointing ldap provider from ldap slave server.So i want to configure provider to pointing their ldap server is.

    You can see my thread on here :

    http://www.linuxquestions.org/questions/linux-server-73/how-to-pointing-ldap-slave-from-ldap-master-on-syncrepl-replication-4175471107/


    Thanks

    ReplyDelete
  14. Hi David,

    I found this http://www.openldap.org/doc/admin24/replication.html#Syncrepl Proxy

    I think my case need method like that.

    But that reference still using slapd.conf.

    i don't know how i can implement using cn=config.

    Any reference for that ?

    ReplyDelete
  15. Hi David.
    I am getting below error while adding the modules

    [root@host4 ldap]# ldapmodify -aZf ~/ldap/module.ldif
    SASL/GSSAPI authentication started
    ldap_sasl_interactive_bind_s: Local error (-2)
    additional info: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_0' not found)

    ReplyDelete
    Replies
    1. Hello Sreekanth,

      Do you have a Kerberos realm setup? If so, then make sure you do a « kinit -p user/admin@YOUR.REALM » before you try the « ldapmodify command.

      If you don't have Kerberos, then try to authenticate to your OpenLDAP server using a different credential. Like the Admin user with -WD cn=admin,dc=example,dc=org or with the super-user with something like « sudo ldapmodify -Y EXTERNAL -H ldapi:/// -f module.ldif » instead.

      HTH,

      DA+

      Delete
  16. Thank you for your guide. Not only did it help get set up with LDAP and Kerberos but also helped me to understand the details. Sometimes a steep learning curve but for me the best guide on the topic currently available on the web.

    ReplyDelete

Note: Only a member of this blog may post a comment.