Support TLS 1.3 certificate compression for dynamic certificates

[huntkalio]

Environment

• OpenResty Version: 1.31.1.1 (or latest)
• OpenSSL Version: OpenSSL 3.2.0+ (with RFC 8879 support)
• Subsystem: ssl_certificate_by_lua*, ngx.ssl (lua-resty-core)

---

Problem Description

In Nginx 1.29.1+ compiled with OpenSSL 3.2.0+, TLS 1.3 Certificate Compression (RFC 8879) is supported via the ssl_certificate_compression on; directive.

When using statically configured certificates (ssl_certificate and ssl_certificate_key), Nginx calls SSL_CTX_compress_certs(ssl->ctx, 0) during configuration initialization, and OpenSSL successfully returns CompressedCertificate records during TLS 1.3 handshakes.

However, when certificates are set dynamically during the TLS handshake via ssl_certificate_by_lua* using the standard OpenResty paradigm (ngx.ssl.clear_certs(), ngx.ssl.set_cert(), ngx.ssl.set_priv_key()), the server sends the dynamic certificate chain uncompressed.

---

Root Cause Analysis

1. ngx.ssl.clear_certs() resets SSL connection state:
   Calling ngx.ssl.clear_certs() invokes OpenSSL's SSL_certs_clear(ssl_conn) in C (ngx_http_lua_ssl_certby.c). In OpenSSL 3.2+, SSL_certs_clear() clears the connection's SSL * certificate structure and resets connection-level certificate compression negotiation flags.

2. ngx.ssl.set_cert() does not trigger connection-level compression:
   ngx.ssl.set_cert() calls SSL_use_certificate() and SSL_add1_chain_cert(). While this attaches the new X509 * certificate chain to ssl_conn, it does not re-enable or re-trigger certificate compression for the newly attached chain on ssl_conn.

3. Missing C/FFI Bindings:
   OpenResty currently does not expose OpenSSL's SSL_compress_certs(SSL *ssl, int alg) via FFI in ngx_http_lua_ssl_certby.c or ngx.ssl in lua-resty-core.

---

Feature Request

I hope the OpenResty team can add support for connection-level certificate compression in lua-nginx-module and lua-resty-core