PlaceOrderAsync masks CERTInext response body; sandbox rate-limit surfaces as misleading "Inactive Account User."

[spbsoluble]

Summary

Two related diagnostic issues observed during integration of the plugin
with the Keyfactor AnyCA REST Gateway running in a kind-based lab (kfclab):

1. CERTInextClient.PlaceOrderAsync throws with only meta.ErrorMessage
   in the exception text. The raw HTTP response body is never logged, so
   when CERTInext returns a generic / mis-categorized error the operator
   has no recourse short of attaching a network sniffer.

2. The CERTInext sandbox at https://sandbox-us-api.certinext.io/emSignHub-API
   appears to apply a burst rate limit that surfaces as the error
   string "Inactive Account User.". This is exceptionally misleading —
   it reads like an account-state condition, but the same access key /
   account works perfectly fine seconds before and after.

The combination of the two means an operator hitting the rate limit
spends a lot of time chasing a non-issue (verifying the account, rotating
keys, contacting CERTInext support) when the actual root cause is "submit
fewer orders per unit time."

Repro / evidence

Lab: kfclab (kind, k3s, Command 26.2.1, AnyCA REST Gateway 25.5.0,
certinext-caplugin built locally from main post the IDomainValidatorFactory
field-type fix — private volatile object _domainValidatorFactory;).

Sandbox account: AccessKey auth mode, AccountNumber=4873378853,
GroupNumber=3162337784, OrganizationNumber=8258978.

Timeline

Time (PDT)	Source	Action	Result
16:23	kfclab gateway pod	First smoke run, 16 enrollments fired in tight succession	First 2 (DV SSL × PFX/CSR) submitted to CERTInext in pending-approval state; remaining 14 returned Inactive Account User.
16:31	kfclab gateway pod	Second smoke run, same 16 entries	All 16 return Inactive Account User.
09:34	local Mac (xUnit)	LifecycleTests.Enroll_Synchronize_Revoke_FullLifecycle (single enroll)	PASS, OrderNumber 7518968666 created in pending_approval
09:35	local Mac (xUnit)	New KfclabFieldProbeTests.Probe_FieldByField, two back-to-back enrollments	PASS, OrderNumbers 8744449464 and 2924858122 created in pending_approval
09:50	kfclab gateway pod	Smoke run trimmed to one CERTInext canary	PASS
09:55	kfclab gateway pod	Smoke run with full 16 entries restored	First 10 PASS in pending_approval, last 6 (OV Wildcard / OV UCC / OV Wildcard UCC × PFX/CSR) HTTP-timed out — unrelated client-side issue

The kfclab gateway and the local probe Mac use identical plugin
config values (verified down to the SHA256 of the DLL inside the pod
matching the locally built artifact). The only difference between the
runs that fail vs. pass is submission velocity.

I also wrote a field-by-field probe that submits both the
integration-test default config (SignerIp=127.0.0.1, mobile=
0000000000, SignerPlace=Gateway, RequestorName with literal quote
chars — see issue 3 below) and the kfclab field shape (SignerIp=0.0.0.0,
empty mobile, SignerPlace=Lab, unquoted RequestorName). Both succeed.
Field shape is not the cause.

Exception text observed on the gateway side

fail: AnyGatewayREST.Controllers.CertificateV1Controller[0]
      Unsuccessful Request CERTInext order failed: Inactive Account User.. See gateway logs for details.
         at Keyfactor.Extensions.CAPlugin.CERTInext.Client.CERTInextClient.PlaceOrderAsync(...) in .../CERTInextClient.cs:line 213
         at Keyfactor.Extensions.CAPlugin.CERTInext.Client.CERTInextClient.EnrollCertificateAsync(...) in .../CERTInextClient.cs:line 566
         at Keyfactor.Extensions.CAPlugin.CERTInext.CERTInextCAPlugin.EnrollNewAsync(...) in .../CERTInextCAPlugin.cs:line 912
         at Keyfactor.Extensions.CAPlugin.CERTInext.CERTInextCAPlugin.Enroll(...) in .../CERTInextCAPlugin.cs:line 575

The exception message "CERTInext order failed: Inactive Account User.. See gateway logs for details." (note the doubled period — meta.ErrorMessage already ends in ., the plugin appends .) is all the operator sees. There is no log entry containing the raw GenerateOrderSSL response body, so we don't know whether CERTInext returned additional structured detail (errorCode, retryAfter, accountId mismatch, etc.).

Asks

1. Log the raw response body on meta.IsSuccess=false

CERTInextClient.cs:212-215:

if (result.Meta != null && !result.Meta.IsSuccess)
    throw new Exception(
        $"CERTInext order failed: {result.Meta.ErrorMessage ?? result.Meta.ErrorCode}. " +
        "See gateway logs for details.");

Before the throw, please add a Logger.LogWarning (or LogError) that
captures the raw resp.Content (or resp.RawBytes for binary safety) so
the operator can see exactly what CERTInext returned. Same change is
warranted in SubmitCsrAsync (line 248), TrackOrderAsync, and any
other path that throws on a non-success meta block.

Suggested log shape (one of):

Logger.LogWarning(
    "CERTInext API non-success. Path={Path}, HttpStatus={Status}, " +
    "ErrorCode={Code}, ErrorMessage={Msg}, RawBody={Body}",
    Constants.Api.GenerateOrderSslPath,
    (int)resp.StatusCode,
    result.Meta.ErrorCode,
    result.Meta.ErrorMessage,
    Truncate(resp.Content, 4096));

The truncation matters — some responses can be very large. ~4 KB is
plenty for diagnostic context without flooding logs.

2. Document "Inactive Account User." as a rate-limit surface

Either in the README's "Troubleshooting" section or as a callout in
PlaceOrderAsync's XML doc comment, please note:

The CERTInext sandbox returns "Inactive Account User." as a generic
error string for several conditions, including rate-limit / burst-quota
rejection. If you see this error and you have confirmed the account is
active (e.g. by a successful Ping immediately before), throttle order
submissions to one per ~1-2 seconds and retry. Sustained 16+ orders in
under 10 seconds consistently trips the limit on the US sandbox.

A second-best alternative would be a defensive client-side retry in
PlaceOrderAsync when meta.ErrorMessage matches "Inactive Account User"
(case-insensitive). I'm less keen on this — it papers over the misleading
error string and burns the operator's order quota on retries — but it
would at least surface as a LogWarning that names the condition.

3. IntegrationTestFixture.LoadEnvFile does not strip surrounding quotes

Latent bug, low priority, but worth fixing while looking at the file.

IntegrationTestFixture.cs:161-163:

string key = line.Substring(0, idx).Trim();
string val = line.Substring(idx + 1).Trim();
result[key] = val;

A .env_certinext line like:

CERTINEXT_REQUESTOR_NAME="Keyfactor Plugin Test"

is parsed as the 23-character string "Keyfactor Plugin Test",
quote chars and all. The integration tests then send that to CERTInext
as the requestor name. CERTInext appears tolerant of the literal quotes
(it treats RequestorName as a free-text display field), but the same
file consumed by anything that respects quote-as-string-delimiter
semantics (kfclab's secrets pipeline did, for instance) will produce a
mismatch with the live integration test's request shape, complicating
side-by-side debugging.

Suggested fix:

string val = line.Substring(idx + 1).Trim();
if (val.Length >= 2 &&
    ((val[0] == '"' && val[val.Length - 1] == '"') ||
     (val[0] == '\'' && val[val.Length - 1] == '\'')))
{
    val = val.Substring(1, val.Length - 2);
}

Context

This was diagnosed during the kfclab effort to validate the upstream
Phase 2 fix (IDomainValidatorFactory field re-typing) end-to-end. The
fix itself works correctly — the plugin loads cleanly and orders are
submitted to CERTInext. The two issues above are the diagnostic and
documentation gaps that surfaced during validation.

Happy to send a PR for any of (1) / (2) / (3). Let me know which you'd
like to take and which you'd prefer I leave to your team.

[github-actions]

Thanks for opening this issue! Our team has been notified and will follow up here.

[spbsoluble]

All three asks plus a follow-up retry landed on PR #2 in commit aab1847. Summary:

Ask 1 — log raw response body on meta-failure throws

Added a shared LogApiFailure(operationContext, resp, errorCode, errorMessage) helper in CERTInextClient.cs that emits a structured LogWarning capturing the HTTP status, the CERTInext error code + message, and a 4 KB-capped raw response body. Wired into every meta-failure throw site: ValidateCredentials, PlaceOrderAsync, SubmitCsrAsync (HTTP-fail), TrackOrderAsync, DownloadCertificateAsync, RevokeCertificateAsync, VerifyDcvAsync, GetDcvAsync. The exception text stays sanitized for the gateway UI (matches the prior C2 audit finding), but operators now actually have the raw body to inspect when chasing a misleading CA error.

Ask 2 — Troubleshooting section

New docsource/overview.md and a mirrored Troubleshooting section in README.md covering:

• "Inactive Account User." as the sandbox rate-limit surface (your wording was lifted nearly verbatim)
• EMS-956 Invalid Request for this API from GetDcv
• Enrollment returning EXTERNALVALIDATION then being picked up by sync
• The issue Plugin fails to load on gateway 25.4.0: "Could not load type IDomainValidatorFactory" from Keyfactor.AnyGateway.IAnyCAPlugin v3.2.0.0 #7 TypeLoadException (now closed in v1.0)

Ask 3 — env-file quote stripping

Extracted IntegrationTestFixture.LoadEnvFile's value-parsing into IntegrationTestFixture.ParseEnvValue (internal static) with matching-pair double/single-quote stripping per your snippet. New IntegrationTestFixtureTests pins 13 cases: quoted, unquoted, whitespace-only, mismatched quotes, empty, embedded quotes.

Follow-up — auto-retry on the rate-limit surface

After thinking about your hesitation re: the defensive retry, we landed on a narrow version: PlaceOrderAsync now retries only when meta.ErrorMessage matches "Inactive Account User" (case-insensitive substring via the IsRateLimitSurface predicate). Up to 5 attempts with exponential backoff + ±25% jitter — nominal delays 1s/2s/4s/8s/16s, max total wait ~31s. Every retry refreshes the meta block so txn IDs stay unique. After exhaustion the original exception is propagated unchanged, so a genuinely-inactive account surfaces identically to today. RateLimitRetryTests pins the predicate and the backoff schedule with unit tests.

Verification

• Release build clean (0 warnings, 0 errors) across net8.0 and net10.0.
• Unit tests: 171/171 pass (156 → 171 with the new tests).
• Live ping/sync against the sandbox still authenticates after the PlaceOrderAsync refactor.

OK to close this issue from your end once you've had a chance to look at the change. Thanks for the structured repro + the three-asks-in-one format — made it straightforward to address.