Skip to content

PoC: low level TLS machine#128744

Open
wfurt wants to merge 41 commits into
dotnet:mainfrom
wfurt:TlsSession
Open

PoC: low level TLS machine#128744
wfurt wants to merge 41 commits into
dotnet:mainfrom
wfurt:TlsSession

Conversation

@wfurt
Copy link
Copy Markdown
Member

@wfurt wfurt commented May 29, 2026

early prototype -> ignore.

wfurt and others added 11 commits May 27, 2026 02:40
Introduces internal-preview public types TlsContext, TlsSession, and
TlsOperationStatus exposing a non-blocking, transport-agnostic TLS
state machine. The PoC implementation is Linux/FreeBSD-only and reuses
the existing SslStreamPal (AcceptSecurityContext / InitializeSecurityContext
/ EncryptMessage / DecryptMessage). On other platforms the types are
present as PlatformNotSupportedException stubs so ApiCompat passes.

SslStream is intentionally untouched (Stage 1 of the proposal).

Adds TlsSessionTests covering:
  - Server TlsSession against a real SslStream client (handshake + ping/pong)
  - ArgumentNullException from TlsContext.Create
  - InvalidOperationException from Encrypt/Decrypt before handshake
Adds an internal wedge in SslStream that routes NextMessage / Encrypt /
Decrypt through TlsSession on Linux/FreeBSD. On other platforms the
partial method stubs return false and the existing PAL path runs
unchanged.

TlsSession additions:
  - TlsContext.WrapShared() so the wedge can reuse SslStream's own
    SslAuthenticationOptions bag (preserves SNI / cert selection results
    and avoids double Dispose).
  - HandshakeStepForSslStream / EncryptForSslStream / DecryptForSslStream
    surface that keeps SslStream's ProtocolToken-based plumbing intact.
  - SecurityContext / CredentialsHandle accessors so SslStream can mirror
    the SafeHandles back into its own fields for cert validation, channel
    binding, ProcessHandshakeSuccess, and Dispose to continue working.
  - TlsOperationStatus.WantCredentials surfaced from ProcessHandshake
    when the PAL reports CredentialsNeeded (OpenSSL client cert path).
  - Decrypt now treats Renegotiate as transparent: OpenSSL handles
    renegotiation internally inside SSL_read, so we just consume the
    frame and ask for more input.

The wedge calls AcquireServerCredentials / AcquireClientCredentials on
the very first server / client handshake step to preserve the legacy
GenerateToken bootstrap (resolves the cert via the various delegates
and assigns CertificateContext, which Interop.OpenSsl asserts on).

Test status: 4969 of 4977 pass with wedge active; remaining 8 are TLS
1.3 post-handshake auth scenarios (mutual auth + client-cert callback
returning null) that need follow-up. No new tests broken vs. baseline.
Adds public TlsOperationStatus Shutdown(Span<byte>, out int) to TlsSession
on Linux/FreeBSD plus a PNSE stub on other platforms. Drives
SslStreamPal.ApplyShutdownToken followed by one PAL handshake step to
extract the close_notify bytes into pending output. Idempotent across
drains; returns Closed once fully drained.

Adds test ServerSession_Shutdown_DeliversCloseNotifyToSslStreamClient
verifying an SslStream client observes EOF after the server-side shutdown.
Adds public ChannelBinding? GetChannelBinding(ChannelBindingKind) to
TlsSession on Linux/FreeBSD plus a PNSE stub on other platforms.
Delegates to SslStreamPal.QueryContextChannelBinding so the binding
material matches what SslStream produces over the same session.

Adds test ServerSession_ChannelBinding_MatchesSslStreamClient that
authenticates an SslStream client against a TlsSession server and
verifies both sides derive the same tls-unique channel binding bytes.
Adds two public APIs on TlsSession (Linux/FreeBSD) with PNSE stubs on
other platforms:

- X509Certificate2? LocalCertificate { get; } returns the server cert on
  a server session, or the negotiated client cert on a client session
  (using CertificateValidationPal.IsLocalCertificateUsed to gate the
  client-side case after handshake).
- TlsOperationStatus RequestClientCertificate(Span<byte>, out int) drives
  SslStreamPal.Renegotiate which, on TLS 1.3, issues a post-handshake
  CertificateRequest, and on TLS 1.2 initiates renegotiation. The
  generated bytes flow through the pending-output buffer. After the
  caller forwards them and continues normal Decrypt, the peer's response
  is processed transparently by OpenSSL and the new certificate becomes
  observable via GetRemoteCertificate.

The mutual-auth round trip is not yet end-to-end covered by a TlsSession
test because TlsSession lacks plumbing for RemoteCertificateValidationCallback
on the standalone path (Interop.OpenSsl.CertVerifyCallback derefs
options.SslStream, which TlsSession does not own). That gap also affects
any initial-handshake mutual-auth scenario on a standalone TlsSession
and is tracked as a separate follow-up.
Standalone TlsSession instances did not invoke the user-supplied
RemoteCertificateValidationCallback because the OpenSSL CertVerifyCallback
dereferenced options.SslStream, which is only set when an SslStream owns
the session. Mutual-auth (including TLS 1.3 PHA) failed with
'certificate verify failed'.

Decouple the verify callback from SslStream:
- Add VerifyRemoteCertificateCallback delegate + RemoteCertificateValidator
  hook on SslAuthenticationOptions, plus a SafeSslHandle slot populated by
  SafeSslHandle.Create.
- Extract SslStream.VerifyRemoteCertificate's core into a static helper
  (VerifyRemoteCertificateCore) shared by SslStream and TlsSession.
- TlsSession.Create registers its own validator on the options when one
  isn't already set (SslStream wedge keeps its own).
- Interop.OpenSsl.CertVerifyCallback now drives the hook + handle on
  options instead of options.SslStream.

Add a server-side mutual-auth test that verifies the standalone
TlsSession surfaces the client certificate to the user callback.
The Encrypt/Decrypt wedge was a pure pass-through: both the wedged and
direct paths called SslStreamPal.{Encrypt,Decrypt}Message on the same
_securityContext (TlsSession mirrors that handle back to SslStream
after each handshake step). There was no PAL difference to hide, just
an extra hop, two partial methods, and the TlsSession-side helpers.

Remove TryEncryptViaTlsSession / TryDecryptViaTlsSession (both partial
declarations and Unix/NotUnix implementations), inline the PAL call in
SslStream.Encrypt / SslStream.Decrypt, and drop the now-dead
EncryptForSslStream / DecryptForSslStream helpers on TlsSession.

The handshake wedge stays — that one owns credentials acquisition and
the ProtocolToken plumbing, so it's a meaningful demonstration that
TlsSession can host SslStream's TLS engine.
Two new tests:

1. TwoSessions_HandshakeAndPingPong_InMemory_Succeeds
   Pure in-memory TlsSession <-> TlsSession: no transport, no async at all.
   Drives both sides synchronously through ProcessHandshake by swapping
   byte arrays. Proves the TlsSession surface is a self-contained TLS
   engine that doesn't need SslStream or any I/O abstraction.

   Pinned to TLS 1.2: a pure in-memory two-session loop currently does
   not handle the TLS 1.3 post-handshake NewSessionTicket records OpenSSL
   emits after the server consumes the client Finished. With SslStream
   on one side those records are absorbed by SslStream's data path. The
   standalone TlsSession surface does not yet handle them (same scope
   as the PHA / renegotiation gap).

2. ClientSession_AgainstSslStreamServer_HandshakeAndPingPong_Succeeds
   Mirror of the existing server-side test: TlsSession is the client,
   SslStream is the server. Confirms the handshake driver is role-
   agnostic. Renamed DriveServerHandshakeAsync to DriveHandshakeAsync
   to reflect that it works for either role.
Two changes:

1. Fix the in-memory two-session TLS 1.3 case. The previous 'bad MAC'
   failure was not a bug -- it was OpenSSL's normal TLS 1.3 state
   machine behavior surfacing in an unusual call pattern. After the
   server consumes the client Finished it emits NewSessionTicket
   records on the server->client side. The client MUST consume those
   records (via Decrypt) before its first Encrypt call: otherwise
   OpenSSL on the client has not yet finalized its write-key
   transition from client_handshake_traffic_secret to
   client_application_traffic_secret, and the server (which has
   already moved its receive key) rejects the resulting ciphertext as
   'decryption failed or bad record mac'. In real network deployments
   the client's receive pump always consumes these bytes before
   sending; in this synchronous in-memory loop we drain them
   explicitly. Test is now a [Theory] covering both TLS 1.2 and
   TLS 1.3.

2. Add ServerSession_OnNonBlockingSocket_AgainstSslStreamClient_Succeeds.
   Drives a TlsSession-as-server against a real loopback Socket in
   non-blocking mode (Socket.Blocking = false). Sends and receives go
   through Socket.Send/Receive directly and handle WouldBlock by
   polling. The peer is a plain SslStream client over a NetworkStream.
   Exercises the 'give me raw socket bytes, I don't care about your
   I/O model' contract end to end: TlsSession itself never blocks on
   I/O; the caller is responsible for transport readiness.
On TLS 1.3, SChannel surfaces SEC_I_RENEGOTIATE from DecryptMessage for post-handshake records (NewSessionTicket, KeyUpdate, post-handshake CertificateRequest). The decrypted inner record must be fed back into AcceptSecurityContext/InitializeSecurityContext or the next DecryptMessage returns SEC_E_CONTEXT_EXPIRED.

Decrypt now invokes a new ProcessPostHandshakeMessage helper and returns Complete so the caller's loop re-enters to process any application data that arrived in the same TCP segment as the NST.

All TlsSessionTests pass (13/13).
@wfurt wfurt added the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label May 29, 2026
Copilot AI review requested due to automatic review settings May 29, 2026 05:21
@dotnet-policy-service
Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR prototypes a low-level, caller-driven TLS state machine for System.Net.Security, exposing TlsContext, TlsSession, and TlsOperationStatus, and wiring Linux/FreeBSD SslStream handshakes through that implementation as a wedge.

Changes:

  • Adds new public TLS session/context APIs and platform-specific implementations/stubs.
  • Refactors certificate validation plumbing so OpenSSL callbacks can be shared by SslStream and TlsSession.
  • Adds functional tests for handshake, encryption/decryption, ALPN, SNI, shutdown, channel binding, and renegotiation scenarios.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/libraries/System.Net.Security/ref/System.Net.Security.cs Adds public API surface for TlsContext, TlsSession, and TlsOperationStatus.
src/libraries/System.Net.Security/src/System.Net.Security.csproj Includes new TLS session and SslStream wedge files by platform.
src/libraries/System.Net.Security/src/System/Net/Security/TlsContext.cs Adds reusable TLS configuration wrapper over SslAuthenticationOptions.
src/libraries/System.Net.Security/src/System/Net/Security/TlsOperationStatus.cs Adds operation status enum for non-blocking TLS operations.
src/libraries/System.Net.Security/src/System/Net/Security/TlsSession.cs Implements the low-level TLS state machine.
src/libraries/System.Net.Security/src/System/Net/Security/TlsSession.Stub.cs Adds unsupported-platform stub implementation.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Unix.cs Routes Unix SslStream handshake steps through TlsSession.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.NotUnix.cs Keeps non-Unix SslStream on the existing path.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Protocol.cs Adds wedge hook and shared certificate validation core.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs Wires OpenSSL certificate validation callback through options.
src/libraries/System.Net.Security/src/System/Net/Security/SslAuthenticationOptions.cs Adds callback and handle slots for OpenSSL validation plumbing.
src/libraries/System.Net.Security/src/System/Net/Security/NetEventSource.Security.cs Generalizes certificate validation logging sender type.
src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Ssl.cs Stores the created SafeSslHandle on authentication options.
src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs Uses the new options-based validation callback path.
src/libraries/System.Net.Security/tests/FunctionalTests/System.Net.Security.Tests.csproj Includes new TLS session tests on Unix/Windows targets.
src/libraries/System.Net.Security/tests/FunctionalTests/TlsSessionTests.cs Adds functional coverage for the prototype API.
src/libraries/System.Net.Security/tests/UnitTests/Fakes/FakeSslStream.Implementation.cs Updates fake options to match the new callback field.

Comment on lines +871 to +882
SslStream.VerifyRemoteCertificateCore(
this,
_context.Options,
_securityContext,
ref _remoteCertificate,
ref _connectionInfo,
cert,
chain,
trust: null,
ref alertToken,
out _,
out _);
TLS_ECDHE_PSK_WITH_AES_128_CCM_8_SHA256 = (ushort)53251,
TLS_ECDHE_PSK_WITH_AES_128_CCM_SHA256 = (ushort)53253,
}
public enum TlsOperationStatus
// can drive the user RemoteCertificateValidationCallback even for a
// standalone TlsSession. If SslStream wraps this session (wedge mode),
// it sets its own validator first and we leave it untouched.
context.Options.RemoteCertificateValidator ??= session.VerifyRemoteCertificate;
Comment on lines +862 to +864
bool needsValidation = !_context.IsServer || _context.Options.RemoteCertRequired;
if (needsValidation)
{
Mirror legacy GenerateToken's bookkeeping: when AcquireClientCredentials is
re-run with newCredentialsRequested=true (in response to SChannel's
CredentialsNeeded after parsing the server's CertificateRequest), set
refreshCredentialNeeded so the finally-block publishes the new cert-bound
credential to SslSessionsCache. Without this, subsequent connections always
missed the cache (anonymous cred cached, cert-bound cred discarded) and the
SChannel session ticket never resumed.

Also routes credential ownership through TlsContext so the wedge and standalone
TlsContext.Create paths share lifetime semantics, and removes diagnostic
file-logging used during root-cause analysis.

Validated: System.Net.Security.Tests full suite — Total 5247, Failed 0, Skipped 40.
Copilot AI review requested due to automatic review settings May 29, 2026 20:26
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Comment on lines +690 to +698
public enum TlsOperationStatus
{
Complete = 0,
WantRead = 1,
WantWrite = 2,
Closed = 3,
WantCredentials = 4,
NeedsCertificateValidation = 5,
}
Comment on lines +85 to +90
// Provide a default cert validation hook so OpenSSL's CertVerifyCallback
// can drive the user RemoteCertificateValidationCallback even for a
// standalone TlsSession. If SslStream wraps this session (wedge mode),
// it sets its own validator first and we leave it untouched.
context.Options.RemoteCertificateValidator ??= session.VerifyRemoteCertificate;
}
Comment on lines +235 to +236
SetRemoteCertificateValidationResult(ok ? SslPolicyErrors.None : sslPolicyErrors);
return sslPolicyErrors;
Comment on lines +1070 to +1081
SslStream.VerifyRemoteCertificateCore(
this,
_context.Options,
_securityContext,
ref _remoteCertificate,
ref _connectionInfo,
cert,
chain,
trust: null,
ref alertToken,
out _,
out _);
wfurt added 2 commits May 29, 2026 15:04
… 3.0+

Wires up the OpenSSL 3.0 SSL_set_retry_verify lightup so SslStream-style
external certificate validation can suspend the TLS handshake mid-stream
on 3.0+ and resume after the application accepts/rejects the peer cert.
Falls back to the existing post-handshake suspension on 1.1.x.

Native:
 - Add CryptoNative_SslSetRetryVerify export (LIGHTUP_FUNCTION pattern).
 - Add PAL_SSL_ERROR_WANT_RETRY_VERIFY (= 12) and forward-declare
   SSL_set_retry_verify in osslcompat_30.h; ifndef-guard the error code
   for 1.1.x headers.

Managed:
 - Add SslErrorCode.SSL_ERROR_WANT_RETRY_VERIFY and matching P/Invoke.
 - DoSslHandshake maps the new error to NeedsRemoteCertificateValidation.
 - CertVerifyCallback calls SslSetRetryVerify and returns -1 to suspend
   when DeferCertificateValidation is set and no managed validator is
   provided; otherwise falls through to the legacy accept-and-validate
   path.
 - Add internal SslAuthenticationOptions.DeferCertificateValidation
   (OpenSSL only).
 - TlsSession enables DeferCertificateValidation, drops the dummy
   AcceptAllForExternalValidation callback, and tracks
   _externalValidationResolved so the second ProcessHandshake call after
   validation succeeds returns Complete rather than throwing.

Tests: System.Net.Security functional suite 4965 / 0 fail / 19 skip on
macOS arm64.
…erOptions)

Allow TlsContext.Create to be called with null SslServerAuthenticationOptions
so the caller can inspect the peer's ClientHello (SNI, supported versions)
before supplying server options. ProcessHandshake parses the first ClientHello,
exposes it via TlsSession.ClientHelloInfo, and returns NeedsServerOptions with
consumed = 0. The caller then calls SetServerOptions(...) and re-feeds the
same input buffer to resume the handshake.

- TlsOperationStatus.NeedsServerOptions = 6
- TlsContext.Create(SslServerAuthenticationOptions?) accepts null
- TlsSession.ClientHelloInfo / SetServerOptions surface and resume the suspend
- ProcessHandshake parses ClientHello via TlsFrameHelper, suspends with
  consumed=0, and re-returns NeedsServerOptions on subsequent calls until
  options are supplied; on OpenSSL the retry-verify suspension semantics
  are restored after ApplyServerOptions
- Ref assembly and Stub updated to match
Copilot AI review requested due to automatic review settings May 29, 2026 22:48
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 6 comments.

[Fact]
public void TlsContext_RejectsNullOptions()
{
Assert.Throws<ArgumentNullException>(() => TlsContext.Create((SslServerAuthenticationOptions)null!));
TLS_ECDHE_PSK_WITH_AES_128_CCM_8_SHA256 = (ushort)53251,
TLS_ECDHE_PSK_WITH_AES_128_CCM_SHA256 = (ushort)53253,
}
public enum TlsOperationStatus
Comment on lines +458 to +460
// CertVerifyCallback needs the SafeSslHandle to stash a
// CertificateValidationException; expose it via the options.
options.SafeSslHandle = handle;
Comment on lines +222 to +225
// On success VerifyRemoteCertificateCore set _remoteCertificate = _externalPendingCert, so
// SetRemoteCertificateValidationResult below leaves it alone. On failure we must dispose the
// pending cert ourselves because no one adopted it.
SetRemoteCertificateValidationResult(ok ? SslPolicyErrors.None : sslPolicyErrors);
Comment on lines +97 to +100
public string? TargetHostName
{
get => _context.Options.TargetHost;
set => _context.Options.TargetHost = value ?? string.Empty;
Comment on lines +320 to +326
_context.ApplyServerOptions(options);
#if !TARGET_WINDOWS && !SYSNETSECURITY_NO_OPENSSL
// Preserve the retry-verify suspension semantics that TlsSession.Create
// would have configured up front had server options been available then.
_context.Options.DeferCertificateValidation = true;
#endif
_clientHelloInfo = null;
wfurt added 3 commits June 1, 2026 10:11
- DriveHandshakeNonBlocking: handle NeedsCertificateValidation so server-side
  flow does not stall when a user validator is supplied.
- ServerSession_OptionalClientCert_NoCertSent_HandshakeCompletesWithoutValidatorCall:
  align with SslStream semantics; the user validator is invoked even when the
  client sends no cert, with null cert and RemoteCertificateNotAvailable.
- ClientSession_WantCredentials_SetClientCertificateContext_ResumesHandshake:
  convert to [Theory] over Tls12 and Tls13, and disable AllowTlsResume on both
  peers so a session cached by a sibling parallel test cannot let this client
  resume and skip the CertificateRequest (which was making the test flaky).
Surfaces the distinguished names the server listed in its TLS 1.2 CertificateRequest or TLS 1.3 certificate_authorities extension. Intended to be called while suspended on WantCredentials so the caller can pick a client certificate that chains to one of the listed CAs. Returns null when no security context exists yet, when the peer sent no hints, or on a server-side session.
… false

Two related fixes from PR review:

1) RemoteCertificateValidationCallback can reject a chain-clean cert by returning false with sslPolicyErrors == None. AcceptWithDefaultValidation was forwarding the original None to SetRemoteCertificateValidationResult and silently accepting. Synthesize RemoteCertificateChainErrors when the callback rejects so the session faults.

2) VerifyRemoteCertificateCore assigns _remoteCertificate to the candidate before knowing whether validation succeeded; on the reject path the rejected leaf sat in the canonical slot. Clear it (and dispose if we own it) in the rejection branch of SetRemoteCertificateValidationResult so the rejected cert isn't retained.

Regression test covers the callback-returns-false case end-to-end.
Copilot AI review requested due to automatic review settings June 1, 2026 19:27
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.

Comment thread TlsSession-proposal.md
Comment on lines +133 to +135
public static TlsContext Create(SslServerAuthenticationOptions options);
public static TlsContext Create(SslClientAuthenticationOptions options);

Comment thread TlsSession-proposal.md
Comment on lines +167 to +168
/// <summary>SNI / target hostname. Set before the first handshake call.</summary>
public string? TargetHostName { get; set; }
Comment thread TlsSession-proposal.md
| `NeedsServerOptions` | `SSL_CTX_set_client_hello_cb` → `SSL_CLIENT_HELLO_RETRY` | Same | Pre-parse ClientHello, defer first `AcceptSecurityContext` | Our state machine |
| `NeedsCertificateValidation` | `set_cert_verify_callback` returning `-1` (`SSL_ERROR_WANT_RETRY_VERIFY`) | `PlatformNotSupportedException` — caller validates post-handshake | `SCH_CRED_MANUAL_CRED_VALIDATION`; we buffer Finished output | Our state machine |

OpenSSL 1.1.1 is EOL September 2026; the `PlatformNotSupportedException` window is small.
Comment on lines +65 to +66
<Compile Include="System\Net\Security\SslStream.TlsSessionWedge.cs"
Condition="'$(TargetPlatformIdentifier)' == 'linux' or '$(TargetPlatformIdentifier)' == 'freebsd' or '$(TargetPlatformIdentifier)' == 'windows'" />
Copy link
Copy Markdown
Member

@am11 am11 Jun 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can be just !TargetsOSX since other Unices also use libssl or its API compatible flavor.

wfurt added 11 commits June 1, 2026 19:19
- TlsSession.cs: alias the security-context field to SafeDeleteContext on
  Apple so it matches the OSX PAL ref parameter type; handle the
  HandshakeStarted PAL status by parsing the ClientHello, running
  SelectApplicationProtocol with the raw ALPN bytes, and re-stepping the
  handshake with an empty input. Rewrote Decrypt for the new 6-arg
  DecryptMessage contract (destination + leftover offset/length),
  unifying SChannel in-place semantics with the Unix
  write-to-destination semantics. RequestClientCertificate throws
  PlatformNotSupportedException on macOS (no SecureTransport
  renegotiation primitive).
- SslStreamPal.OSX.cs: align AcceptSecurityContext and
  InitializeSecurityContext credential ref parameters with the
  Linux/Windows nullable signatures.
- Strings.resx: add net_ssl_renegotiate_not_supported.
- csproj: include TlsSession.cs and exclude the stub on osx; same for
  the TlsSessionTests compile.
- TlsSessionTests.cs: extend PlatformSpecific to OSX; skip the four
  scenarios SecureTransport does not implement (deferred client-cred
  prompt, post-handshake client-cert request, TLS 1.3,
  tls-server-end-point channel binding).

Validates: full System.Net.Security functional suite on macOS, 4983
tests, 0 failures, 19 unrelated skips. Network.framework backend
deferred to a follow-up.
Make TlsContext own a long-lived SafeSslContextHandle on OpenSSL-backed
Unix platforms (linux/freebsd/non-Apple/non-Android). Every TlsSession
created from the same TlsContext now reuses that SSL_CTX instead of
re-allocating one per session via the global SslContextCacheKey path.

Implementation:
- TlsContext / SslAuthenticationOptions made partial; OpenSsl partials
  carry the SafeSslContextHandle (and PreallocatedSslContext property
  the PAL reads) and dispose with the TlsContext.
- AllocateSslHandle honors PreallocatedSslContext: borrows the handle,
  bypasses the static cache + resume cache, and skips disposing on exit.
- Wedge mode (SslStream) and server-side ServerCertificateSelectionCallback
  (cert not known at SSL_CTX creation) keep the legacy per-session path.
When TlsSession.Create(TlsContext, SafeSocketHandle) is called on an OpenSSL platform, plumb the socket fd through SslAuthenticationOptions.SocketFd so SafeSslHandle.Create skips the ManagedSpanBio installation and binds the SSL object to the socket directly via SSL_set_fd. The socket-bound Handshake/Read/Write methods then drive ciphertext I/O via SSL_do_handshake/SSL_read/SSL_write, bypassing the managed scratch-buffer loop. This avoids a memcpy on each record and enables kernel TLS offload. The non-socket TlsSession + SslStream paths are unchanged (SocketFd defaults to -1, ManagedSpanBio path used).
Replace raw int fd plumbing with SafeSocketHandle:

- CryptoNative_SslSetFd takes intptr_t (matches SafeHandle marshaling)
- Managed SslSetFd P/Invoke takes SafeSocketHandle so the marshaller handles AddRef/Release across the call
- SslAuthenticationOptions.SocketFd (int) -> SocketHandle (SafeSocketHandle?)
- TlsSession.Create no longer constructs a Socket wrapper or extracts a raw IntPtr in fd mode; the handle is stashed on options and the interop layer deals with it
- ThrowIfNotSocketBound now checks _socketHandle (works in both fd and buffer modes)
Split GetOrCreateSslContextHandle so a caller can request a non-cached SSL_CTX
while still controlling whether session resumption tickets are enabled.
TlsContext.AttachSharedNativeContext now passes allowCached:false plus the
options' AllowTlsResume so a per-context SSL_CTX honors the option.
…input

When a peer coalesces handshake-finished with the first app-data record into
one TCP segment, OpenSSL absorbs the ciphertext into its read BIO during the
handshake step. A subsequent Decrypt call with empty input would return
WantRead and deadlock waiting on the socket. On Linux/FreeBSD/Android the
Decrypt early-return now probes the PAL once with an empty input span; if it
produces plaintext we return Complete instead of WantRead.
ServerSession_TlsResume_HonorsAllowTlsResumeOption: a 4-case Theory
(Tls12/Tls13 x true/false) verifying that AllowTlsResume controls whether a
second connection through the same TlsContext resumes the prior session.
macOS TLS 1.3 is skipped (SecureTransport ticket behavior).
… scratch, trace harness

- Add AllowResume parameter and TLS 1.3 coverage.
- Drive the server side on a dedicated thread so SslStream's TP continuations
  don't compete for threads with BDN's InProcessEmit loop.
- Rent/return all scratch buffers via ArrayPool to remove bench-side noise from
  MemoryDiagnoser numbers.
- Tighten Job settings (IterationCount=15, WarmupCount=5, InvocationCount=64)
  for stable measurement of cheap resumed handshakes.
- Add a --trace harness for diagnosing buffered-vs-fd behavior with verbose
  per-step logging.
Copilot AI review requested due to automatic review settings June 3, 2026 03:11
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 8 comments.

Comment on lines +36 to +45
SafeSslHandle ssl = EnsureFdSslHandle();
int ret = Interop.Ssl.SslDoHandshake(ssl, out Interop.Ssl.SslErrorCode err);
if (ret == 1)
{
OnHandshakeCompleted();
result = TlsOperationStatus.Complete;
return;
}
result = MapSslError(err, "SSL_do_handshake");
}
Comment on lines +47 to +60
partial void TryFastRead(Span<byte> buffer, ref int bytesRead, ref TlsOperationStatus? result)
{
if (!_useFdMode)
{
return;
}

if (buffer.IsEmpty)
{
result = TlsOperationStatus.Complete;
return;
}

SafeSslHandle ssl = (SafeSslHandle)_securityContext!;
Comment on lines +71 to +84
partial void TryFastWrite(ReadOnlySpan<byte> buffer, ref int bytesWritten, ref TlsOperationStatus? result)
{
if (!_useFdMode)
{
return;
}

if (buffer.IsEmpty)
{
result = TlsOperationStatus.Complete;
return;
}

SafeSslHandle ssl = (SafeSslHandle)_securityContext!;
Comment on lines +24 to +36
/// <summary>
/// Non-blocking TLS state machine wrapper around the existing
/// <see cref="SslStreamPal"/>. The caller owns I/O and drives ciphertext
/// in and out via byte spans. Supported on Linux/FreeBSD (OpenSSL) and
/// Windows (SChannel). Provides <see cref="ProcessHandshake"/>,
/// <see cref="Encrypt"/>, <see cref="Decrypt"/>, and a pending-output queue.
/// </summary>
/// <remarks>
/// <para>
/// The session never performs any I/O. The caller drives ciphertext in/out
/// via byte spans. Any ciphertext the TLS layer needs to send (handshake
/// records, alerts, encrypted application data) is staged in an internal
/// pending-output buffer and drained via <see cref="DrainPendingOutput"/>.
Comment on lines +704 to +706
public static System.Net.Security.TlsContext Create(System.Net.Security.SslServerAuthenticationOptions? options) { throw null; }
public static System.Net.Security.TlsContext Create(System.Net.Security.SslClientAuthenticationOptions options) { throw null; }
public void Dispose() { }
Comment on lines +21 to +22
[PlatformSpecific(TestPlatforms.Linux | TestPlatforms.FreeBSD | TestPlatforms.Windows | TestPlatforms.OSX)]
public class TlsSessionTests
Comment on lines +1000 to +1006
public async Task ClientSession_ExternalCertificateValidation_AcceptWithDefaultValidation_FailsOnUntrustedCert()
{
// The test cert chain isn't installed in the system trust store, so the default
// validation policy must report at least RemoteCertificateChainErrors.
using X509Certificate2 serverCert = TestCertificates.GetServerCertificate();
string serverName = serverCert.GetNameInfo(X509NameType.SimpleName, forIssuer: false);

Comment thread TlsSession-proposal.md
Comment on lines +332 to +337
The handshake **suspends before** the client's Finished (or the server's
Finished in the mTLS case). If the verdict is reject, the session emits a
fatal alert at exactly the right protocol state and the peer never observes a
successful handshake. No application data is ever exchanged with a rejected
peer. This matches the recent `SslStream` fix that ensured the validation
callback's verdict actually gates the Finished.
@wfurt
Copy link
Copy Markdown
Member Author

wfurt commented Jun 3, 2026

@EgorBot -intel -runtime main vs pr

Interop.OpenSsl.cs (pulled into System.Net.Security.Unit.Tests via Common)
references SslAuthenticationOptions.PreallocatedSslContext, which is defined
in the SslAuthenticationOptions.OpenSsl.cs partial. The unit-tests csproj was
not pulling that partial, breaking the linux/unix build.

> [!NOTE]
> This commit message was AI/Copilot-generated.
@wfurt
Copy link
Copy Markdown
Member Author

wfurt commented Jun 3, 2026

@EgorBot -intel

@wfurt
Copy link
Copy Markdown
Member Author

wfurt commented Jun 3, 2026

@EgorBot -intel --filter SslStream

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-System.Net.Security NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants