Fix/cross sign cert expired recovery#860
Open
mrnovalles wants to merge 4 commits into
Open
Conversation
OTP's ssl_certificate:find_cross_sign_root_paths/4 recovers from an
expired cross-signed root by locating an alternative valid root with
the same public key in the trust store. It only triggers when path
validation reports root_cert_expired.
ssl_verify_hostname:verify_fun/3 returns {fail, {bad_cert, cert_expired}}
verbatim, which terminates the handshake before OTP's recovery can run.
Wrap the verify_fun in check_hostname_opts/1 to intercept cert_expired
and rewrite it to root_cert_expired. All other events are delegated to
ssl_verify_hostname unchanged, so hostname checking is unaffected.
Confirmed against rest.fra-01.braze.eu (Let's Encrypt chain containing
the ISRG Root X2 cross-signed by ISRG Root X1, expired 2025-09-15)
using hackney 1.25.0, certifi 2.15.0, OTP 27.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
This fix came out of a production incident. Our push notification service recently migrated to Let's Encrypt certificates. After the migration, all HTTPS calls made through hackney started failing with:
The leaf certificate was not expired. The issue is structural: Let's Encrypt chains include an ISRG Root X2 cross-signed by ISRG Root X1, whose validity period ran 2020-09-04 → 2025-09-15 and is now past.
Root cause
OTP ships recovery logic for exactly this scenario in
ssl_certificate:find_cross_sign_root_paths/4. It triggers when path validation reports root_cert_expired: OTP searches the trust store for a cert with the same public key as the expired root and, if found, re-validates the chain anchored at the still-valid self-signed copy.The recovery never runs with hackney due to the following sequence:
{fail, …}and aborts the handshake immediately.find_cross_sign_root_paths/4is never reached.The OTP recovery only triggers on root_cert_expired, not cert_expired.
Fix
Wrap
ssl_verify_hostname:verify_fun/3incheck_hostname_opts/1with a one-clause prefix that rewrites{bad_cert, cert_expired}to{fail, {bad_cert, root_cert_expired}}. All other events are delegated tossl_verify_hostnameunchanged, so hostname checking is unaffected.