Fix runtime data races, memory leak, and shutdown safety by bmehta001 · Pull Request #1429 · microsoft/cpp_client_telemetry

bmehta001 · 2026-04-28T22:59:45Z

Fixes runtime thread-safety, shutdown-safety, response-lifetime, and resource-leak issues that affect normal SDK operation. Split out from #1415 per reviewer request so runtime behavior changes stay separate from CI/build/test fixes.

Fix HTTP cancellation and callback tracking

HttpClient_Apple.mm

Scope cancellation to the current request’s m_dataTask instead of canceling every task on the shared static NSURLSession.
Fix the CancelAllRequests() wait loop so m_requests.empty() is checked while holding m_requestsMtx.

HttpClientManager.cpp

Fix the cancelAllRequests() wait loop so m_httpCallbacks.empty() is checked while holding m_httpCallbacksMtx.

HttpClient_WinInet.cpp

Close the WinInet session handle under its own null check. Previously m_hWinInetSession was closed only inside the if (m_hWinInetRequest != nullptr) block, so if HttpOpenRequestA failed after InternetConnectA succeeded (session set, request null) the session handle leaked.

Fix HTTP response lifetime on abort/network-failure paths

HttpResponseDecoder.cpp

Preserve ctx->httpResponse through requestAborted(ctx) and temporaryNetworkFailure(ctx) so downstream storage/statistics handlers can still read status and headers.
Avoid leaking aborted/network-failure responses by keeping ownership with EventsUploadContext, whose clear() path deletes the response.

HttpResponseDecoderTests.cpp

Add regression coverage that aborted and network-failure decode routes still receive the response object with result/status intact.

Fix WorkerThread shutdown safety and task leak

ITaskDispatcher.hpp / TaskDispatcher.hpp

Add a QueueWithResult(Task*) interface method that reports whether the dispatcher accepted the task (returns false if it was dropped, e.g. during shutdown). It is declared after Cancel and defaulted to delegate to Queue() so existing dispatchers and vtable slot indices are unchanged.

WorkerThread.cpp

Add an explicit shutdown gate (m_shuttingDown): late Queue()/QueueWithResult() calls are rejected (and the task deleted) once teardown starts, and QueueWithResult returns false so callers can drop a pointer they retained for cancellation.
Enqueue the shutdown sentinel only once under the queue lock.
Delete pending queued/timer tasks on shutdown instead of only logging that they exist (previously they leaked). On a normal join() the owning thread deletes them; on the self-Join() path it detaches and the worker drains and deletes its own remaining tasks before exiting, so neither path leaks.
Log join/detach failures instead of silently swallowing all exceptions.

TaskDispatcherCAPITests.cpp

Add coverage for the QueueWithResult behavior on the C-API dispatcher.

Fix Flush teardown deadlock

OfflineStorageHandler.cpp

Flush() could early-return (when StartActivity() fails during teardown) without posting flush completion, so WaitForFlush() blocked forever. Signal completion on the early-return path so teardown cannot deadlock.

BasicFuncTests.cpp

Add a smoke test that tears down while an upload is in flight (MAX_TEARDOWN_TIME = 0, large payloads against the slow endpoint) and asserts shutdown completes cleanly; it also asserts the /slow/ endpoint rewrite actually happened so the coverage can’t silently lapse.

Make TransmissionPolicyManager scheduling consistently mutex-guarded

TransmissionPolicyManager.cpp / .hpp

Guard m_isUploadScheduled, m_runningLatency, and m_scheduledUploadTime consistently with m_scheduledUploadMutex.
Avoid holding m_scheduledUploadMutex across potentially blocking cancellation during stop/shutdown.
Keep force/zero-delay no-wait cancellation under the scheduler mutex so a competing delayed schedule cannot suppress an immediate upload.
Use an explicit std::chrono::milliseconds value in the bandwidth-controller reschedule path, and cast std::chrono counts to long long in the %lld LOG_TRACE calls (the rep is long on LP64, a -Wformat mismatch in logging-enabled builds).

TransmissionPolicyManagerTests.cpp

Add regression coverage for the force/zero-delay scheduling race.

Fix Logger static-destruction-order crash

Logger.cpp

Remove destructor logging from Logger::~Logger() because it can run after logging infrastructure has already been destroyed, causing crashes during static teardown.

Known parity gap (separate repo, follow-up): AIHttpResponseDecoder::handleDecode in the lib/modules submodule (lib/modules/azmon/AIHttpResponseDecoder.cpp:105,118) still sets ctx->httpResponse = nullptr; on the aborted/failure paths — the same response-lifetime leak fixed here in lib/http/HttpResponseDecoder.cpp. It lives in a different repository (the lib/modules submodule), so it must be fixed there and pulled in via a submodule bump; it is out of scope for this PR.

- HttpClient_Apple: scope Cancel() to m_dataTask only instead of blanket-cancelling every task on the shared session. Fix torn read on m_requests.empty() in CancelAllRequests spin loop. - HttpClientManager: fix torn read on m_httpCallbacks.empty() in cancelAllRequests spin loop — read under lock. - HttpResponseDecoder: add missing delete ctx->httpResponse before nullptr in Abort and RetryNetwork paths (memory leak). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Only delete queued tasks after successful join (not after detach, where the thread may still access them — undefined behavior) - Replace catch(...) with std::system_error and std::exception handlers that log error code and message - Log pending queue sizes in both join and detach paths Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Both variables are read and written from different threads during normal upload scheduling. Declare as std::atomic to eliminate data races per the C++ memory model. Add .load() for variadic LOG_TRACE calls. Add comment explaining why unlocked stores in uploadAsync are safe. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove LOG_TRACE from Logger destructor — it triggers a crash on iOS simulator when the recursive_mutex used by logging has already been destroyed during static destruction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Reject new worker-thread tasks once shutdown starts so queue cleanup cannot race with late producers, and move the TPM scheduled-upload state back under a single mutex so latency/next-upload decisions stay consistent without mixed atomic and mutex access. Files changed: - lib/pal/WorkerThread.cpp - lib/tpm/TransmissionPolicyManager.cpp - lib/tpm/TransmissionPolicyManager.hpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR targets runtime correctness in the SDK by addressing thread-safety issues, shutdown safety, and a per-request memory leak in HTTP response handling.

Changes:

Tighten HTTP request cancellation scoping and fix torn reads in request/callback tracking loops.
Fix a SimpleHttpResponse leak on aborted/network-failure decode paths.
Rework worker-thread shutdown behavior to avoid unsafe queue cleanup after detach() and improve error logging.
Refactor TransmissionPolicyManager scheduling state synchronization (mutex + new cancelUploadTaskLocked() helper).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
lib/tpm/TransmissionPolicyManager.hpp	Changes upload-scheduling state fields and adds a locked cancellation helper declaration.
lib/tpm/TransmissionPolicyManager.cpp	Moves upload-scheduling state access under a mutex and adjusts cancellation/scheduling flow.
lib/pal/WorkerThread.cpp	Makes shutdown/join behavior safer and improves exception handling/logging during join/detach.
lib/http/HttpResponseDecoder.cpp	Deletes `ctx->httpResponse` on Abort/RetryNetwork paths to prevent leaks.
lib/http/HttpClient_Apple.mm	Limits cancellation to the instance’s task and fixes a torn read in the shutdown wait loop.
lib/http/HttpClientManager.cpp	Fixes a torn read in the shutdown wait loop by locking around empty-check.
lib/api/Logger.cpp	Removes destructor logging to avoid iOS static-destruction-order crash.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Keep the scheduled-upload state mutex-based, but stop holding m_scheduledUploadMutex across DeferredCallbackHandle::Cancel so shutdown and pause paths do not block uploadAsync behind the same lock. While touching the path, use std::chrono::milliseconds for the bandwidth-controller reschedule call so ENABLE_BW_CONTROLLER builds cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Keep forced upload scheduling atomic around no-wait cancellation and preserve HTTP responses until downstream abort/network-failure handlers finish. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

When scheduleUpload is called with force=true (or zero delay) and the previously scheduled upload task is currently executing on the worker, the no-wait cancel returns false and m_isUploadScheduled stays set. The existing m_isUploadScheduled check then skipped scheduling a new task, silently dropping the requested latency for force-scheduled profile changes. Propagate the requested latency to m_runningLatency under the same mutex when this race occurs. uploadAsync re-reads m_runningLatency inside its own LOCKGUARD, so a task that hasn't yet entered that critical section will pick up the new latency. If uploadAsync has already cleared m_isUploadScheduled (past its LOCKGUARD), the existing fallthrough at line 184 schedules a fresh task with the new latency. Add a regression test using a fake dispatcher whose Cancel always returns false, asserting that a force-scheduled call updates m_runningLatency without enqueueing a duplicate task. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use the existing LOCKGUARD helper because scheduled upload cancellation does not need movable lock ownership. Consolidate the duplicated Issue 388 cancellation note so the PR keeps the remaining limitation documented without repeating the same TODO. Files changed: - lib/tpm/TransmissionPolicyManager.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace tight current-time assertions with a direct comparison against the original delayed schedule time. This keeps coverage for the forced immediate upload race while reducing timing sensitivity in CI. Files changed: - tests/unittests/TransmissionPolicyManagerTests.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restore the existing Issue 388 wording in the remaining cancellation comment while keeping the duplicated helper comment removed. Files changed: - lib/tpm/TransmissionPolicyManager.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Fix printf-style logging arguments for scheduled upload delays and queued worker task pointers. Ensure the blocking cancel test releases the dispatcher before failing so async futures cannot hang the test runner. Files changed: - lib/pal/WorkerThread.cpp - lib/tpm/TransmissionPolicyManager.cpp - tests/unittests/TransmissionPolicyManagerTests.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Addresses three Copilot review comments (TransmissionPolicyManager.cpp:119, 202, 266). This PR changed these LOG_TRACE format strings from %d to %lld but passed std::chrono::milliseconds::rep directly. That rep is implementation- defined and is long on LP64 (Linux/macOS), so %lld (which expects long long) is a -Wformat mismatch -- an error under the project's -Wall -Werror in logging-enabled (HAVE_MAT_LOGGING) builds, and formally UB in the varargs call. Cast each count() to long long so the format always matches on every data model. This mirrors the cast this PR already applies to delta (static_cast<unsigned long long> with %llu) a few lines up. Verified: clang 18 -Wall -Werror -Wextra flags the uncast %lld as "format specifies type 'long long' but the argument has type 'rep' (aka 'long')" and accepts the cast form. TransmissionPolicyManagerTests (40) pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Reword the comment to describe the test without citing a tracking number; no code change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Reword the three `enabled` comments to describe the behavior without citing a tracking number; no code change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

# Conflicts: # lib/pal/WorkerThread.cpp # tests/unittests/TaskDispatcherCAPITests.cpp

…own thread The process-wide PAL WorkerThread is shared by reference count. A task running on the worker thread can drop the last reference (e.g. by tearing down its LogManager/PAL), which ran ~WorkerThread -> Join() synchronously inside the task: Join() detached the thread and returned, freeing the object while threadFunc was still on the stack below the task. threadFunc then kept touching freed members (m_itemInProgress, the locks, and the queues it drains at shutdown) -- a use-after-free / heap corruption confirmed by AddressSanitizer. Give the worker a custom shared_ptr deleter: when the last reference is released on the worker thread itself, detach and defer destruction to the thread, which deletes itself only after its loop has broken and all member access is done. On any other thread the object is deleted immediately as before (~WorkerThread joins the worker first). Add a PalTests regression test that drops the last reference from within a task running on the worker thread; it is clean under AddressSanitizer with the fix and reports heap-use-after-free without it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

onLastReferenceReleased() decided whether it was running on its own worker thread via m_hThread.get_id(), which returns the default not-a-thread id after detach(). If Join() had already run on the worker thread (its self-path detaches m_hThread), a later last-reference drop on that same thread would miss the self-check and delete the object while threadFunc was still executing below it -- the same UAF this change set fixes. Capture the worker's id in an atomic at threadFunc start and compare against that instead, so detection is correct regardless of detach ordering. Not reachable through current SDK code (the default WorkerThread is never explicitly Join()-ed), so this is defense-in-depth. Validated: ASAN dispatcher tests (PalTests incl WorkerThreadSelfDisposeOnOwnThreadIsSafe, TaskDispatcherCAPITests) all pass clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Two issues raised on the previous commit: - std::atomic<std::thread::id> is not portable (std::thread::id is not guaranteed trivially copyable). Store m_workerId as a plain std::thread::id guarded by the existing recursive m_lock instead; the self-dispose check reads it under the lock. - Passing std::thread::id to LOG_INFO's printf-style '%u' is undefined behavior (varargs). Format the id with std::hash<std::thread::id> and '%zu' at both log sites (the constructor's 'Started new thread' and threadFunc's 'Running thread'). This was pre-existing; the surrounding change touches these lines. Validated: ASAN dispatcher tests (PalTests incl WorkerThreadSelfDisposeOnOwnThreadIsSafe, TaskDispatcherCAPITests) 15/15 pass clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

bmehta001 requested a review from a team as a code owner April 28, 2026 22:59

bmehta001 mentioned this pull request Apr 29, 2026

Fix CI workflows, compiler flags, and flaky tests #1415

Merged

bmehta001 and others added 4 commits April 29, 2026 14:41

bmehta001 force-pushed the bhamehta/runtime-fixes branch from 3f0289c to de46cb2 Compare April 29, 2026 21:42

bmehta001 requested a review from Copilot April 30, 2026 01:16

Copilot started reviewing on behalf of bmehta001 April 30, 2026 01:17 View session

Copilot AI reviewed Apr 30, 2026

View reviewed changes

Comment thread lib/tpm/TransmissionPolicyManager.cpp Outdated

Comment thread lib/tpm/TransmissionPolicyManager.hpp

Comment thread lib/tpm/TransmissionPolicyManager.cpp Outdated

bmehta001 added a commit to bmehta001/cpp_client_telemetry that referenced this pull request May 1, 2026

Remove runtime overlap with microsoft#1429

826cc9a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

bmehta001 self-assigned this May 3, 2026

bmehta001 requested a review from Copilot May 3, 2026 05:46

Copilot started reviewing on behalf of bmehta001 May 3, 2026 05:47 View session

Copilot AI reviewed May 3, 2026

View reviewed changes

Comment thread lib/http/HttpClientManager.cpp Outdated

Comment thread lib/pal/WorkerThread.cpp Outdated

Comment thread lib/pal/WorkerThread.cpp

Comment thread lib/tpm/TransmissionPolicyManager.cpp Outdated

Comment thread lib/tpm/TransmissionPolicyManager.cpp Outdated

bmehta001 requested a review from Copilot May 4, 2026 12:09

Copilot started reviewing on behalf of bmehta001 May 4, 2026 12:10 View session

Copilot AI reviewed May 4, 2026

View reviewed changes

Comment thread lib/tpm/TransmissionPolicyManager.cpp Outdated

Comment thread lib/http/HttpResponseDecoder.cpp Outdated

Comment thread lib/http/HttpResponseDecoder.cpp Outdated

bmehta001 and others added 6 commits May 4, 2026 07:26

Address runtime review comments

2cdf817

Keep forced upload scheduling atomic around no-wait cancellation and preserve HTTP responses until downstream abort/network-failure handlers finish. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge branch 'main' into bhamehta/runtime-fixes

11820ae

bmehta001 requested a review from Copilot May 11, 2026 16:57

Copilot AI reviewed May 11, 2026

View reviewed changes

Comment thread lib/tpm/TransmissionPolicyManager.cpp Outdated

Comment thread lib/pal/WorkerThread.cpp Outdated

Comment thread tests/unittests/TransmissionPolicyManagerTests.cpp

bmehta001 force-pushed the bhamehta/runtime-fixes branch from 4178021 to d357260 Compare May 12, 2026 23:33

Copilot started reviewing on behalf of bmehta001 July 8, 2026 23:22 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Comment thread lib/tpm/TransmissionPolicyManager.cpp Outdated

Comment thread lib/tpm/TransmissionPolicyManager.cpp Outdated

Comment thread lib/tpm/TransmissionPolicyManager.cpp Outdated

bmehta001 requested a review from Copilot July 9, 2026 01:16

Copilot started reviewing on behalf of bmehta001 July 9, 2026 01:16 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Drop issue-number reference from teardown smoke-test comment

c10f636

Reword the comment to describe the test without citing a tracking number; no code change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

bmehta001 requested a review from Copilot July 9, 2026 16:08

Copilot started reviewing on behalf of bmehta001 July 9, 2026 16:09 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Drop issue-number reference from metastats opt-in comments

21233a6

Reword the three `enabled` comments to describe the behavior without citing a tracking number; no code change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

bmehta001 requested a review from Copilot July 9, 2026 16:26

Copilot started reviewing on behalf of bmehta001 July 9, 2026 16:26 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

bmehta001 requested a review from Copilot July 10, 2026 16:01

Copilot started reviewing on behalf of bmehta001 July 10, 2026 16:02 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

bmehta001 and others added 2 commits July 10, 2026 13:38

Merge remote-tracking branch 'msft/main' into bhamehta/runtime-fixes

46e8b1d

# Conflicts: # lib/pal/WorkerThread.cpp # tests/unittests/TaskDispatcherCAPITests.cpp

bmehta001 requested a review from Copilot July 10, 2026 19:24

Copilot started reviewing on behalf of bmehta001 July 10, 2026 19:25 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

bmehta001 requested a review from Copilot July 11, 2026 05:28

Copilot started reviewing on behalf of bmehta001 July 11, 2026 05:28 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

Comment thread lib/pal/WorkerThread.cpp

Comment thread lib/pal/WorkerThread.cpp

bmehta001 requested a review from Copilot July 12, 2026 15:47

Copilot started reviewing on behalf of bmehta001 July 12, 2026 15:47 View session

Copilot AI reviewed Jul 12, 2026

View reviewed changes

Uh oh!

Conversation

bmehta001 commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fix HTTP cancellation and callback tracking

Fix HTTP response lifetime on abort/network-failure paths

Fix WorkerThread shutdown safety and task leak

Fix Flush teardown deadlock

Make TransmissionPolicyManager scheduling consistently mutex-guarded

Fix Logger static-destruction-order crash

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

bmehta001 commented Apr 28, 2026 •

edited

Loading