Topic: cert verify failed

[mansky]

Hi all,

We are having a problem with Apache still throwing a "certificate verify failed" even though on both the server, and the gateway machine the chain certificate from Let's Encrypt has been updated and the expired root legacy certificate has been removed.

Apache 2.4.37
OpenSSL 1.1.1g
mod_perl 2.0.11
Perl 5.26.3

Any ideas, or suggestions, about what other option in ssl.conf or other config file we need to update, would be greatly appreciated.

Thanks,

--Ed

[tangent]

You don't explain the difference between your Apache and Gateway machine roles, and whether the certificate verify error is server side, or reported by clients.

Checking your Apache SSL configuration, according to http://httpd.apache.org/docs/current/mod/mod_ssl.html#sslcertificatechainfile

SSLCertificateChainFile became obsolete with version 2.4.8, when SSLCertificateFile was extended to also load intermediate CA certificates from the server certificate file.

So have you updated the Intermediate CA certificates in the server certificate file, or left an obsolete certificate in there?

If Apache runs, then you should be able to check the advertised certificate chain through a test site service such as https://www.ssllabs.com/ssltest, or locally with Openssl or Curl. The following site has a good write up on checking certificate chains - https://medium.com/@superseb/get-your-certificate-chain-right-4b117a9c0fce

[mansky]

Hi,

The initial request starts on a User's client machine which sends the SOAP request first to the gateway machine. The gateway machine then sends the SOAP request on to the server for processing.

The error "certificate verify failed" is seen in the Apache error logs on the gateway machine, not the server.

We are still using the SSLCertificateChainFile directive on the server, running Apache 2.4.37.

In testing the server and the gateway machines with the SSL Checker website, the gateway machine checked out fine.

The server however showed chain issues, specifically extra certificates.

We had SSLCertificateChainFile pointing to the fullchain.pem file on the server, which contained the certificate for the machine itself, the LE certificate and the root certificate (both latter certificates were up-to-date).

Changing SSLCertificateChainFile to chain.pem, containing just the LE and root certificates resulted in the SSL Checker website passing the server now as well.

However, we are still getting that error message, even after the above change to SSLCertificateChainFile, and restarting Apache on both machines.

What else might need changing ?

Thanks,

--Ed

[tangent]

Since both your Gateway and Server certificate chains check out ok now, using the SSL Labs checker, I'd consider the verify error in the Apache log is triggered when the Gateway connects to your Server.

Have you defined or updated the SSLProxyCACertificateFile in your Gateway Apache proxy configuration? https://httpd.apache.org/docs/current/mod/mod_ssl.html#sslproxycacertificatefile

This directive sets the all-in-one file where you can assemble the Certificates of Certification Authorities (CA) whose remote servers you deal with. These are used for Remote Server Authentication. Such a file is simply the concatenation of the various PEM-encoded Certificate files, in order of preference.

[mansky]

I found the source of the problem. I checked Apache and mod_security and both were using an up-to-date version of OpenSSL. The problem was in the backend Perl code being executed.

Specifically, the SOAP::Lite module from CPAN was a very old version = 0.715, which in turn is using LWP::UserAgent and Net::SSLeay for SSL authentication.

While the versions of LWP::UserAgent and Net::SSLeay was more recent, I think the issue was the version of SOAP::Lite not accepting the hash ssl_opts to the proxy constructor. I think later versions do accept ssl_opts as an argument.

Hence I had to skip SSL Peer authentication by defining the environment variable PERL_LWP_SSL_VERIFY_HOSTNAME

Adding the line: