THRIFT-6170: Fix TSSLSocket build with OpenSSL 4.0

[jaipaulcheernam]

OpenSSL 4.0 removes SSLv3_method() and per-version TLS method functions (TLSv1_method, TLSv1_1_method, TLSv1_2_method), the deprecated ASN1_STRING_data() function, and returns const pointers from X509_get_subject_name(), X509_NAME_get_entry(), and X509_NAME_ENTRY_get_data().

• Guard SSLv3_method and TLS version methods with version check
• Replace ASN1_STRING_data with ASN1_STRING_get0_data
• Add const qualifiers for X509_NAME, X509_NAME_ENTRY, ASN1_STRING

This is backward-compatible with OpenSSL >= 1.1.0 since all replacement APIs exist since that version.

• Did you create an Apache Jira ticket? (Request account here, not required for trivial changes)
• If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
• Did you squash your changes to a single commit? (not required, but preferred)
• Did you do your best to avoid breaking changes? If one was needed, did you label the Jira ticket with "Breaking-Change"?
• If your change does not involve any code, include [skip ci] anywhere in the commit message to free up build resources.

[Jens-G]

This branch has conflicts that must be resolved

[loqs]

Is it expected that SecurityTest and SecurityFromBufferTest both hang/timeout with OpenSSL 4.0.2 and the test expectations will be changed in a future pull request?

OPENSSL_NO_SSL3 is not defined for OpenSSSL 4 so #ifdef OPENSSL_NO_SSL3 passes and the tests try to use SSL3 and timeout. TLSv1.0/1.1 require SSL_CTX_set_min_proto_version when using TLS_method instead of the per version method. Without it being set the TLSv1.0 and TLSv1.1 tests timeout like the SSL 3 tests.

[jaipaulcheernam]

This branch has conflicts that must be resolved

Done. rebased and please check

[jaipaulcheernam]

Is it expected that SecurityTest and SecurityFromBufferTest both hang/timeout with OpenSSL 4.0.2 and the test expectations will be changed in a future pull request?

OPENSSL_NO_SSL3 is not defined for OpenSSSL 4 so #ifdef OPENSSL_NO_SSL3 passes and the tests try to use SSL3 and timeout. TLSv1.0/1.1 require SSL_CTX_set_min_proto_version when using TLS_method instead of the per version method. Without it being set the TLSv1.0 and TLSv1.1 tests timeout like the SSL 3 tests.

Thanks for the feedback! Updated the PR:

• Rebased onto latest master (conflicts resolved)
• TLSv1.0/1.1/1.2 now use TLS_method() + SSL_CTX_set_min_proto_version() / SSL_CTX_set_max_proto_version() on OpenSSL 4.0, so tests should no longer hang
• SSLv3 is guarded out on OpenSSL 4.0 (returns error since the protocol is truly removed)
• ERR_remove_state() calls removed (no-op since OpenSSL 1.1)

I haven't been able to run the SecurityTest/SecurityFromBufferTest locally — if there are still test timeouts, happy to iterate.

[loqs]

Thank you for working on this. There are still timeouts which I was able to fix with:

--- a/lib/cpp/test/SecurityFromBufferTest.cpp
+++ b/lib/cpp/test/SecurityFromBufferTest.cpp
@@ -229,7 +229,7 @@ BOOST_AUTO_TEST_CASE(ssl_security_matrix) {
           continue;
         }
 
-#ifdef OPENSSL_NO_SSL3
+#if defined(OPENSSL_NO_SSL3) || OPENSSL_VERSION_NUMBER >= 0x40000000L
         if (si == 2 || ci == 2) {
           // Skip all SSLv3 cases - protocol not supported
           continue;
diff --git a/lib/cpp/test/SecurityTest.cpp b/lib/cpp/test/SecurityTest.cpp
index 86640bd68..b133ef0f8 100644
--- a/lib/cpp/test/SecurityTest.cpp
+++ b/lib/cpp/test/SecurityTest.cpp
@@ -357,7 +357,7 @@ BOOST_AUTO_TEST_CASE(ssl_security_matrix)
                     continue;
                 }
 
-#ifdef OPENSSL_NO_SSL3
+#if defined(OPENSSL_NO_SSL3) || OPENSSL_VERSION_NUMBER >= 0x40000000L
                 if (si == 2 || ci == 2)
                 {
                     // Skip all SSLv3 cases - protocol not supported

All tests then pass for C and C++ however I believe that is due to lack of coverage in testtransportsslsocket.c compared to SecurityTest.cpp for SSLv3/TLSv1_0/TLSv1_1. I think the C code needs SSL_CTX_set_min_proto_version() / SSL_CTX_set_max_proto_version() to match the C++ code.