Maddening problem with LDAP and Apache

I've spent the last three days fighting with one of the most frustrating problems I've ever had. We replaced the hardware for the BCRC server -- an old Solaris server with a new Ubuntu box. We had done this before in the ISB and, other than a minor hiccup or two, everything switched over smoothly. I assumed this would be the same and almost everything was. Except for LDAP in Apache.

We use LDAP for centralized authentication. It's not perfect by any means, but it's been a huge efficiency in how we manage accounts and services. We use it for shell accounts (cf ssh), samba (file sharing and printing), and via apache (http basic authentication and in PHP). It was no problem to get it set up everwhere except for apache. LDAP only failed in apache. But the same configuration we were using on the other server wouldn't work on this one.

I spent one day just denying that it was anything to be concerned about. Then I spent a day double-checking everything: config files, permissions & ownerships, typos. Then I spent a day trying stuff: configuration changes, re-installing software -- even rebooting. Then I spent a day hiding from it (maybe two). Finally, on Sunday, I went in to the office in the evening, rolled up my sleeves, and made the commitment to just stay there working on it until it was solved or I was dead. About three hours in, I found it.

The errors I was getting didn't make sense. The first error, a generic "couldn't contact ldap server" wasn't helpful, especially as the ldap_connect function was working -- it was failing at ldap_bind. I figued out how to turn on debugging with this line of PHP code:

ldap_set_option(NULL, LDAP_OPT_DEBUG_LEVEL, 7);

But the error I was getting back didn't make much sense.

TLS: only one of certfile and keyfile specified

This error is so rare that google mostly just returns links to the source code.

The logging on the LDAP server was the equally vague "TLS Negotiation Error".

Eventually, I figured out that the configuration for setting up SSL for HTTPS also governs the connections the server makes to the LDAP server. And then I found it:

#   Allow insecure renegotiation with clients which do not yet support the
#   secure renegotiation protocol. Default: Off
SSLInsecureRenegotiation on

This line was commented out on the server where it worked. I commented out the line, restarted the webserver, and it just started working.

I'm wondering if this is the point where people start to say, "I'm gettin' too old for this kind of shit!"