-
-
Notifications
You must be signed in to change notification settings - Fork 191
Expand file tree
/
Copy paththreadsafety_stress.cpp
More file actions
1427 lines (1333 loc) · 66.6 KB
/
Copy paththreadsafety_stress.cpp
File metadata and controls
1427 lines (1333 loc) · 66.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
/*
This file is part of libhttpserver
Copyright (C) 2011-2026 Sebastiano Merlino
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301
USA
*/
// Thread-safety contract stress test.
//
// Sub-test A — concurrent_register_block_from_handlers_no_data_race
// Drives the PUBLIC mutating surface (register_path, unregister_path,
// deny_ip, remove_denied_ip) AND the **server-wide**
// lifecycle-hook registration surface (webserver::add_hook +
// hook_handle::remove) AND the **per-resource**
// hook bus via http_resource::add_hook on both a small pool of
// shared resources (same-slot null-vs-installed timing races in
// ensure_table()) and a shared resource whose hook_table_ is
// already installed (load-acquire short-circuit branch). 16 curl
// clients × N seconds at default port 0 (kernel-assigned).
// The hook ops here exercise the server-wide tier AND
// the resource (middle) tier of the documented
// `route_table_mutex_ → resource hook_table → server-wide hook_table`
// lock order under TSan. Standalone per-resource CAS-race coverage
// lives in Sub-test D.
// The TSan-clean rerun is the headline acceptance:
// `make clean && CXXFLAGS=-fsanitize=thread … && make check` re-runs
// this binary under TSan via the CI matrix entry `build-type: tsan`
// in .github/workflows/verify-build.yml (no workflow edit required).
//
// Wall-clock: 60 seconds by default (per the §9-testing-item-6
// acceptance criterion: "at least 60 seconds"). Override locally with
// HTTPSERVER_STRESS_SECONDS=N for fast iteration.
//
// Sub-test B — stop_from_handler_deadlocks_as_documented
// The negative case: stop() called from a handler thread
// triggers libmicrohttpd to self-join → on this MHD version, an
// abort with "Failed to join a thread."; on others, a silent
// deadlock. The test forks a child process to contain the abort
// so the parent test binary stays healthy. Either a non-zero child
// exit within 5 s or a 5 s timeout (child SIGKILLed by parent)
// counts as positive observation of the contract; a zero child exit
// would be a regression. Now run in per-PR CI on the baseline
// Linux gcc lane via `make -C test check-stop-from-handler`, which sets
// HTTPSERVER_RUN_STOP_FROM_HANDLER=1. Local runs remain opt-in — the
// sub-test SKIPs unless that env var is set.
//
// **Local TSan reproduction (already covered automatically by the tsan CI lane):**
// Rebuild with `CXXFLAGS="-fsanitize=thread -g -O1"
// LDFLAGS="-fsanitize=thread"` and re-run this binary; expect no
// "WARNING: ThreadSanitizer: data race" output. Same pattern as
// route_table_concurrency.cpp.
// Linux-only: pthread_setaffinity_np for the noise-reduction pin in
// adversarial_segments_registration_no_latency_spike.
// _GNU_SOURCE must be defined before the first system header is included.
#if defined(__linux__) && !defined(_WIN32)
#ifndef _GNU_SOURCE
#define _GNU_SOURCE
#endif
#endif
#include <curl/curl.h>
#ifndef _WIN32
#include <sys/wait.h>
#include <unistd.h>
#endif
#if defined(__linux__) && !defined(_WIN32)
#include <pthread.h>
#include <sched.h>
#endif
#include <algorithm>
#include <array>
#include <atomic>
#include <chrono>
#include <cstdint>
#include <cstdio>
#include <csignal>
#include <cstdlib>
#include <functional>
#include <iostream>
#include <latch>
#include <memory>
#include <mutex>
#include <random>
#include <string>
#include <string_view>
#include <thread>
#include <utility>
#include <vector>
#include "./httpserver.hpp"
#include "./littletest.hpp"
namespace ht = httpserver;
namespace {
// Named constants — replaces hex magic literals throughout.
// kDynSlots: number of competing dynamic path slots (slot = i & (kDynSlots-1)).
// kIpRange: number of distinct test IPs (ip suffix = i & (kIpRange-1)).
// kExitCodeStopReturned: child exit when stop() returns from a handler (regression).
// kExitCodeCurlCompleted: child exit when curl completes without abort (regression).
constexpr int kDynSlots = 8;
constexpr int kIpRange = 256;
// kCasPoolSize: number of shared resources op 6 rotates through so
// concurrent clients can land on the SAME hook_table_ slot.
constexpr int kCasPoolSize = 4;
constexpr int kExitCodeStopReturned = 42; // stop() returned — should not happen
constexpr int kExitCodeCurlCompleted = 43; // curl completed — stop() did not block
// Discard libcurl response bodies — we only care about completing the
// round-trip, not the body content.
size_t discard_write(char* /*ptr*/, size_t size, size_t nmemb,
void* /*userdata*/) {
return size * nmemb;
}
// Counters bumped from handler threads. atomic<int> so the
// LT_CHECK(*_ops > 0) gates at the end are race-free reads.
struct OpCounters {
std::atomic<int> register_ok{0};
std::atomic<int> unregister_ok{0};
std::atomic<int> deny_ok{0};
std::atomic<int> remove_denied_ok{0};
std::atomic<int> hook_add_ok{0}; // server-wide hook add
std::atomic<int> hook_remove_ok{0}; // server-wide hook remove
std::atomic<int> cas_resource_hook_ok{0}; // op 6: pooled resources (null-vs-installed timing race)
std::atomic<int> cas_existing_hook_ok{0}; // op 7: load-acquire short-circuit branch
std::atomic<int> handler_calls{0};
};
// Hook-handle bag retained across iterations so add/remove
// ops race against each other AND against the route-table ops above.
// Raw pointer (passed to driver_body) to a stack-allocated bag;
// lifetime is the test body, so no ownership transfer is needed.
struct HookBag {
std::mutex mtx;
std::vector<ht::hook_handle> handles; // armed handles
static constexpr std::size_t kCap = 256; // cap to keep RSS bounded
};
// Minimal resource shared by all sub-tests; where a test needs to
// re-arm the lazy `hook_table_` CAS in ensure_table(), it simply
// allocates a fresh stack-local instance whose slot starts null
// (Sub-test D).
class noop_resource : public ht::http_resource {
public:
ht::http_response render_get(
const ht::http_request&) override {
return ht::http_response::string("ok");
}
};
// Register an armed hook on a randomly-chosen phase. Each
// hook overload is a distinct std::function<...> signature, so the
// phase selection happens at compile time via this switch (the runtime
// `phase` value is purely the argument to add_hook). Returns an armed
// hook_handle bound to the chosen phase.
// NOTE: the modulo constant (11u) MUST match the number of hook_phase
// enumerators in hook_phase.hpp. If a new phase is added, update both
// the modulo and the switch cases here.
ht::hook_handle install_random_hook(ht::webserver* ws, unsigned phase_idx) {
switch (phase_idx % 11u) {
case 0:
return ws->add_hook(ht::hook_phase::connection_opened,
std::function<void(const ht::connection_open_ctx&)>(
[](const ht::connection_open_ctx&) {}));
case 1:
return ws->add_hook(ht::hook_phase::accept_decision,
std::function<void(const ht::accept_ctx&)>(
[](const ht::accept_ctx&) {}));
case 2:
return ws->add_hook(ht::hook_phase::request_received,
std::function<ht::hook_action(ht::request_received_ctx&)>(
[](ht::request_received_ctx&) {
return ht::hook_action::pass();
}));
case 3:
return ws->add_hook(ht::hook_phase::body_chunk,
std::function<ht::hook_action(ht::body_chunk_ctx&)>(
[](ht::body_chunk_ctx&) {
return ht::hook_action::pass();
}));
case 4:
return ws->add_hook(ht::hook_phase::route_resolved,
std::function<void(const ht::route_resolved_ctx&)>(
[](const ht::route_resolved_ctx&) {}));
case 5:
return ws->add_hook(ht::hook_phase::before_handler,
std::function<ht::hook_action(ht::before_handler_ctx&)>(
[](ht::before_handler_ctx&) {
return ht::hook_action::pass();
}));
case 6:
return ws->add_hook(ht::hook_phase::handler_exception,
std::function<ht::hook_action(const ht::handler_exception_ctx&)>(
[](const ht::handler_exception_ctx&) {
return ht::hook_action::pass();
}));
case 7:
return ws->add_hook(ht::hook_phase::after_handler,
std::function<ht::hook_action(ht::after_handler_ctx&)>(
[](ht::after_handler_ctx&) {
return ht::hook_action::pass();
}));
case 8:
return ws->add_hook(ht::hook_phase::response_sent,
std::function<void(const ht::response_sent_ctx&)>(
[](const ht::response_sent_ctx&) {}));
case 9:
return ws->add_hook(ht::hook_phase::request_completed,
std::function<void(const ht::request_completed_ctx&)>(
[](const ht::request_completed_ctx&) {}));
default:
return ws->add_hook(ht::hook_phase::connection_closed,
std::function<void(const ht::connection_close_ctx&)>(
[](const ht::connection_close_ctx&) {}));
}
}
// Driver handler: each request decodes `op` and `i` from the query
// string and re-enters the public webserver API. Catches the documented
// `std::invalid_argument` on duplicate-registration races (per
// register_path's contract — not a data race).
ht::http_response driver_body(const ht::http_request& req,
ht::webserver* ws, OpCounters* counters,
HookBag* hooks,
std::array<noop_resource,
kCasPoolSize>* cas_pool,
ht::http_resource* shared_cas_resource) {
counters->handler_calls.fetch_add(1, std::memory_order_relaxed);
int op = 0;
int i = 0;
try {
std::string op_s{req.get_arg("op")};
std::string i_s{req.get_arg("i")};
if (!op_s.empty()) op = std::stoi(op_s);
if (!i_s.empty()) i = std::stoi(i_s);
} catch (const std::exception& e) {
// Malformed query — still return 200; we only need lock
// exercise. Surface it so malformed URLs are visible in CI logs,
// mirroring the unregister_path exception diagnostic below.
std::cerr << "[stress] malformed op/i query param: " << e.what()
<< '\n';
}
const int slot = i & (kDynSlots - 1);
const std::string dyn_path =
"/dyn/" + std::to_string(slot);
const std::string ip =
"198.51.100." + std::to_string(i & (kIpRange - 1));
// Eight ops total: 0..3 are the route-table / ban-list
// mutators; 4..5 are the webserver-side hook bus churn;
// 6..7 are the per-resource hook bus churn (op 6 rotates
// across a small pool of shared resources so concurrent clients
// can hit the SAME hook_table_ slot — null early in the run,
// installed thereafter; op 7 hits the load-acquire short-circuit
// branch on the shared_cas_resource whose hook_table_ is already
// installed after the first op-7 call lands). `op % 8` keeps the
// distribution roughly uniform.
// NOTE: register_prefix / unregister_prefix are intentionally not
// exercised here because they share the same lock path as
// register_path / unregister_path (register_impl_ with family=true
// vs false); the existing cases already cover the mutex contention.
switch (op % 8) {
case 0:
try {
ws->register_path(dyn_path,
std::make_shared<noop_resource>());
counters->register_ok.fetch_add(
1, std::memory_order_relaxed);
} catch (const std::invalid_argument&) {
// Duplicate-registration race is contract, not a bug.
}
break;
case 1:
try {
ws->unregister_path(dyn_path);
counters->unregister_ok.fetch_add(
1, std::memory_order_relaxed);
} catch (const std::invalid_argument&) {
// Path not registered yet — expected race, not a bug.
} catch (const std::exception& e) {
// Surface unexpected exceptions so they are visible in
// test logs (e.g. bad_alloc, logic_error).
std::cerr << "[stress] unexpected unregister_path exception: "
<< e.what() << '\n';
}
break;
case 2:
ws->deny_ip(ip);
counters->deny_ok.fetch_add(1, std::memory_order_relaxed);
break;
case 3:
ws->remove_denied_ip(ip);
counters->remove_denied_ok.fetch_add(
1, std::memory_order_relaxed);
break;
case 4: {
// Install a hook on a random phase. Bag is capped to
// prevent unbounded growth under net-add pressure (the
// remove ops below drain it but a streak of 4s could
// outrun them).
ht::hook_handle h = install_random_hook(
ws, static_cast<unsigned>(i));
std::lock_guard<std::mutex> lk(hooks->mtx);
if (hooks->handles.size() >= HookBag::kCap) {
// Recycle: move the oldest handle out, erase the slot,
// then call remove() so the moved-from dtor is a no-op.
// erase(begin()) is O(n) on vector, so this already
// costs up to a 255-element shift at today's kCap=256
// (~25 evictions/s in practice) — judged acceptable now,
// not merely a future-growth risk; switch to std::deque
// if kCap grows significantly.
ht::hook_handle dead = std::move(hooks->handles.front());
hooks->handles.erase(hooks->handles.begin());
dead.remove(); // deregisters; dtor is now a no-op
}
hooks->handles.push_back(std::move(h));
counters->hook_add_ok.fetch_add(
1, std::memory_order_relaxed);
break;
}
case 5: {
std::lock_guard<std::mutex> lk(hooks->mtx);
if (!hooks->handles.empty()) {
// Pop the back: most recently added, most likely
// still in cache; removes pressure-test the writer-
// lock path on hook_table_mutex_.
// Move out first, pop the slot, then call remove() so
// the (now-empty) bag entry's dtor is a no-op — mirrors
// the recycle pattern in case 4.
ht::hook_handle dead = std::move(hooks->handles.back());
hooks->handles.pop_back();
dead.remove();
counters->hook_remove_ok.fetch_add(
1, std::memory_order_relaxed);
}
break;
}
case 6: {
// Per-resource hook bus on a small pool of
// shared resources, rotated by `i`. Because the pool
// outlives every request, concurrent clients that land
// on the same still-null slot early in the run race the
// ensure_table() CAS on the SAME hook_table_ slot — the
// null-vs-installed race driven by relative timing
// across concurrent clients. Once a slot's table is
// installed, later hits keep churning add_hook on it.
// The *guaranteed* contended-null CAS coverage lives in
// Sub-test D, which re-arms a fresh null slot per
// iteration behind a latch. While this handler is in
// flight, register_path / unregister_path (cases 0/1)
// on other threads are holding route_table_mutex_
// shared, so this case exercises the full three-tier
// order
// route_table_mutex_ (shared, this thread is a reader
// inside dispatch) -> resource hook_table_ (this
// thread's CAS or short-circuit in ensure_table()) ->
// server-wide hook_table (other threads'
// webserver::add_hook in cases 4/5).
// Note this is exercised as a cross-thread, TSan-observable
// concurrent interleaving across separate handler
// invocations — not the single-thread sequential nested
// lock acquisition documented in architecture doc §5.6.
// Sub-test D provides the standalone per-resource CAS proof.
// remove() runs immediately so registrations on the
// long-lived pool resources do not accumulate.
ht::hook_handle h =
(*cas_pool)[i & (kCasPoolSize - 1)].add_hook(
ht::hook_phase::request_completed,
std::function<void(const ht::request_completed_ctx&)>(
[](const ht::request_completed_ctx&) {}));
h.remove();
counters->cas_resource_hook_ok.fetch_add(
1, std::memory_order_relaxed);
break;
}
case 7: {
// Shared resource whose hook_table_ is
// installed after the first op-7 call. Subsequent calls
// take the load-acquire short-circuit branch in
// ensure_table() (`if (existing) return existing;`),
// complementing case 6's contended-null branch. The
// handle is dropped immediately; remove() runs against
// the still-alive resource at end of scope.
ht::hook_handle h = shared_cas_resource->add_hook(
ht::hook_phase::request_completed,
std::function<void(const ht::request_completed_ctx&)>(
[](const ht::request_completed_ctx&) {}));
h.remove();
counters->cas_existing_hook_ok.fetch_add(
1, std::memory_order_relaxed);
break;
}
}
return ht::http_response::string("ok");
}
// Stress duration: default 60 s (acceptance criterion), overridable
// via HTTPSERVER_STRESS_SECONDS for fast local iteration.
// Capped at 3600 s to prevent runaway in CI (CWE-1284).
// NOTE: this single knob budgets wall-clock for BOTH Sub-test A's client
// loop and Sub-test C's watchdog — an operator setting a large
// value (up to the 3600s cap) should budget for both sub-tests combined.
int stress_seconds() {
if (const char* s = std::getenv("HTTPSERVER_STRESS_SECONDS")) {
try {
int v = std::stoi(s);
if (v > 0 && v <= 3600) return v;
} catch (const std::out_of_range&) {
std::cerr << "[WARN] HTTPSERVER_STRESS_SECONDS value out of "
"range, using default\n";
} catch (...) {
}
}
return 60;
}
// Characterisation knob. When set to N>1, the
// adversarial_segments_registration_no_latency_spike sub-test runs its
// gate computation N times back-to-back, printing one [STATS] line per
// run. Used to build per-lane CDFs of the p95/median ratio when
// investigating CI flakes. Default 1 (single run, no behaviour change).
// Capped at 200 to prevent runaway in CI.
int stress_repeats() {
if (const char* s = std::getenv("HTTPSERVER_STRESS_REPEATS")) {
try {
int v = std::stoi(s);
if (v > 0 && v <= 200) return v;
} catch (const std::out_of_range&) {
std::cerr << "[WARN] HTTPSERVER_STRESS_REPEATS value out of "
"range, using default\n";
} catch (...) {
}
}
return 1;
}
// Linux-only noise-reduction knob. When HTTPSERVER_STRESS_PIN_CPU
// is set to a non-negative integer, the four writer threads of the
// adversarial_segments sub-test are pinned to that CPU via
// pthread_setaffinity_np. Pinning all writers to the same CPU is
// counter-intuitive but correct for this test: the writers contend on
// route_table_mutex_, so they are effectively serialised — forcing them
// onto one CPU eliminates cross-CPU cache misses on segment-trie node
// memory and removes scheduler migration jitter from the p95 tail. macOS
// has no equivalent (thread_policy_set is a hint widely reported as
// ineffective on Apple Silicon), so the knob is a no-op there. Returns
// -1 when unset / out of range, meaning "do not pin".
int stress_pin_cpu() {
if (const char* s = std::getenv("HTTPSERVER_STRESS_PIN_CPU")) {
try {
int v = std::stoi(s);
if (v >= 0 && v < 4096) return v;
} catch (const std::out_of_range&) {
std::cerr << "[WARN] HTTPSERVER_STRESS_PIN_CPU value out of "
"range, using default\n";
} catch (...) {
}
}
return -1;
}
// Pin the calling thread to `cpu_id` on Linux; no-op elsewhere.
// Returns true on success, false on failure (no diagnostic — pinning
// is a best-effort optimisation, not a contract).
bool pin_this_thread_to_cpu(int cpu_id) {
#if defined(__linux__) && !defined(_WIN32)
if (cpu_id < 0) return false;
cpu_set_t set;
CPU_ZERO(&set);
CPU_SET(cpu_id, &set);
return pthread_setaffinity_np(pthread_self(), sizeof(set), &set) == 0;
#else
(void)cpu_id;
return false;
#endif
}
} // namespace
LT_BEGIN_SUITE(threadsafety_stress_suite)
void set_up() {
curl_global_init(CURL_GLOBAL_ALL);
}
void tear_down() {
curl_global_cleanup();
}
LT_END_SUITE(threadsafety_stress_suite)
LT_BEGIN_AUTO_TEST(threadsafety_stress_suite,
concurrent_register_block_from_handlers_no_data_race)
OpCounters counters;
HookBag hooks; // bag retained for the duration of the test
// Shared per-resource CAS target for op 7. The first
// op-7 call installs the hook_table_ via the contended-null path;
// every subsequent op-7 call across all client threads takes the
// load-acquire short-circuit branch in ensure_table(), driving
// concurrent registration + dispatch on the middle tier.
noop_resource shared_cas_resource;
// Op-6 pool. Long-lived shared resources rotated by the
// request's `i`, so concurrent clients can contend on the SAME
// hook_table_ slot (null-vs-installed depending on relative
// timing early in the run). See the case-6 comment in driver_body.
std::array<noop_resource, kCasPoolSize> cas_pool;
// Port 0 lets the kernel pick a free port; read it back via
// get_bound_port() to avoid hard-coded-port collisions when this
// 60-s test runs alongside other integration tests under
// `make -j check`.
ht::webserver ws{
ht::create_webserver(0)
.start_method(ht::http::http_utils::INTERNAL_SELECT)
.max_threads(8)};
ws.on_get("/driver",
[&ws, &counters, &hooks, &cas_pool, &shared_cas_resource](
const ht::http_request& req) {
return driver_body(req, &ws, &counters, &hooks,
&cas_pool, &shared_cas_resource);
});
ws.start(false);
const uint16_t port = ws.get_bound_port();
LT_CHECK_GT(port, 0);
const std::string base =
"http://127.0.0.1:" + std::to_string(port) + "/driver";
std::atomic<bool> stop{false};
constexpr int kClients = 16;
std::vector<std::thread> clients;
clients.reserve(kClients);
for (int c = 0; c < kClients; ++c) {
clients.emplace_back([&, c] {
// Per-thread curl handle: curl_easy_* is per-handle
// thread-safe; each thread owns its handle.
CURL* curl = curl_easy_init();
if (!curl) return; // resource exhaustion — skip this thread
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, discard_write);
// 198.51.100.0/24 is TEST-NET-2 (RFC 5737) — block/unblock
// ops on these IPs cannot blacklist the loopback driver
// traffic. Belt-and-braces: also bind curl to 127.0.0.1.
curl_easy_setopt(curl, CURLOPT_INTERFACE, "127.0.0.1");
curl_easy_setopt(curl, CURLOPT_TIMEOUT_MS, 2000L);
std::mt19937 rng(static_cast<uint32_t>(c) * 0x9e3779b9u);
while (!stop.load(std::memory_order_relaxed)) {
// Eight ops: 0..3 route-table/ban; 4..5 webserver-side
// hook bus; 6..7 per-resource hook bus
// (pooled shared-slot churn + load-acquire
// branch).
const int op = static_cast<int>(rng() % 8u);
const int i = static_cast<int>(rng() & (kIpRange - 1));
const std::string url =
base + "?op=" + std::to_string(op) +
"&i=" + std::to_string(i);
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_perform(curl);
// 5 ms inter-request sleep to rate-limit each thread to
// ~200 req/s. Under TSan (5–20× slower) this keeps total
// lock pressure and shadow-memory churn within the CI
// wall-clock budget without reducing lock-path coverage.
std::this_thread::sleep_for(std::chrono::milliseconds(5));
}
curl_easy_cleanup(curl);
});
}
const auto deadline =
std::chrono::steady_clock::now() +
std::chrono::seconds(stress_seconds());
while (std::chrono::steady_clock::now() < deadline) {
std::this_thread::sleep_for(std::chrono::milliseconds(250));
}
stop.store(true, std::memory_order_relaxed);
for (auto& t : clients) t.join();
// Acceptance criterion: 60 s of concurrent register/lookup/block.
// We don't pin exact counts — the gate is "all eight mutating ops
// executed at least once without deadlock or crash, and (under the
// documented TSan rebuild) no data race fired". The gate also covers
// the webserver-side hook add/remove pair and the per-
// resource CAS-driven pair.
LT_CHECK_GT(counters.handler_calls.load(), 0);
LT_CHECK_GT(counters.register_ok.load(), 0);
LT_CHECK_GT(counters.unregister_ok.load(), 0);
LT_CHECK_GT(counters.deny_ok.load(), 0);
LT_CHECK_GT(counters.remove_denied_ok.load(), 0);
LT_CHECK_GT(counters.hook_add_ok.load(), 0);
LT_CHECK_GT(counters.hook_remove_ok.load(), 0);
LT_CHECK_GT(counters.cas_resource_hook_ok.load(), 0);
LT_CHECK_GT(counters.cas_existing_hook_ok.load(), 0);
// Drain the hook bag explicitly before the webserver stops so the
// hook_handle destructors run while the impl is still alive. (The
// hook_handle dtor is safe after webserver tear-down too — it
// would simply find the impl gone and remove() becomes a no-op —
// but the explicit drain keeps the lifetimes obvious in the test.)
{
std::lock_guard<std::mutex> lk(hooks.mtx);
hooks.handles.clear();
}
ws.stop();
LT_END_AUTO_TEST(concurrent_register_block_from_handlers_no_data_race)
LT_BEGIN_AUTO_TEST(threadsafety_stress_suite,
stop_from_handler_deadlocks_as_documented)
// Gate: skip unless explicitly opted in. The deadlock case is by
// design; the test exists to PIN the documented behaviour
// and is opt-in because reproducing the deadlock requires _Exit()
// to escape the wedged process.
const char* run = std::getenv("HTTPSERVER_RUN_STOP_FROM_HANDLER");
if (run == nullptr || std::string_view(run) != "1") {
std::cout << "[SKIP] stop_from_handler_deadlocks_as_documented"
" — set HTTPSERVER_RUN_STOP_FROM_HANDLER=1 to run\n";
return;
}
#ifdef _WIN32
// fork()/waitpid() are POSIX-only; the wedge cannot be contained in a
// child process on Windows. Skip — the contract is verified by the POSIX
// lanes, and Windows is not a release-blocking target for this gate.
std::cout << "[SKIP] stop_from_handler_deadlocks_as_documented"
" — fork()/waitpid() unavailable on Windows\n";
return;
#else
// Run the wedge in a forked child so the unsafe stop() call does
// not kill the test binary. The expected observable on this MHD
// version is fatal-abort: libmicrohttpd detects the self-join
// attempt (pthread_join on the current thread returns EDEADLK)
// and aborts with "Failed to join a thread." A silent deadlock
// (process still alive after 5 s) is the alternative outcome
// the contract documents — both qualify as "unsafe; do not do this."
//
// A `ready` result with a normal-zero exit would be a regression
// against the contract: it would mean stop() returned successfully from
// a handler, contradicting the documented contract.
const pid_t child = fork();
LT_CHECK(child >= 0);
if (child < 0) return; // fork failed; waitpid(-1,...) would reap unrelated processes
if (child == 0) {
// Child: trigger the contract violation. We do not care about
// the child's stdout — silence it so the test log stays
// readable.
::close(STDOUT_FILENO);
::close(STDERR_FILENO);
ht::webserver ws{
ht::create_webserver(0)
.start_method(ht::http::http_utils::INTERNAL_SELECT)
.max_threads(4)};
ws.on_get("/wedge", [&ws](const ht::http_request&) {
// Call stop() directly on the handler's MHD worker
// thread → the documented unsafe path.
ws.stop();
// Below is unreachable. If reached, the contract is
// broken — exit with a distinctive code so the parent
// can flag the regression.
std::_Exit(kExitCodeStopReturned);
return ht::http_response::string("unreachable");
});
ws.start(false);
const uint16_t port = ws.get_bound_port();
const std::string url = "http://127.0.0.1:" +
std::to_string(port) + "/wedge";
CURL* curl = curl_easy_init();
if (!curl) {
// Resource exhaustion — cannot exercise the curl side of the
// contract; exit with the same sentinel as a completed-without-
// abort curl call so the parent's outcome classification stays
// simple (mirrors the `if (!curl) return;` guard used in the
// Sub-test A client threads).
std::_Exit(kExitCodeCurlCompleted);
}
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, discard_write);
// 3 s < parent's 5 s deadline: curl's window expires before the
// parent SIGKILLs the child, keeping the two timeouts ordered.
curl_easy_setopt(curl, CURLOPT_TIMEOUT_MS, 3000L);
curl_easy_perform(curl);
curl_easy_cleanup(curl);
// If we ever reach here, stop()-from-handler did NOT abort
// or deadlock as documented → regression. Use a distinct
// sentinel so the parent can flag it separately from the
// unreachable-after-stop() case above.
std::_Exit(kExitCodeCurlCompleted);
}
// Parent: bounded wait on the child. SIGKILL it after 5 s if it
// is still running (the silent-deadlock branch of the contract). Any
// outcome OTHER than a zero exit is a positive observation of
// the contract; a zero exit (or sentinel codes) is a regression.
int status = 0;
auto child_deadline =
std::chrono::steady_clock::now() + std::chrono::seconds(5);
bool reaped = false;
while (std::chrono::steady_clock::now() < child_deadline) {
pid_t r = ::waitpid(child, &status, WNOHANG);
if (r == child) {
reaped = true;
break;
}
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
if (!reaped) {
// Silent-deadlock branch: kill the child, then reap.
::kill(child, SIGKILL);
::waitpid(child, &status, 0);
std::cout << "[OK] stop_from_handler — silent deadlock "
"reproduced (child still alive after 5s)\n";
} else if (WIFSIGNALED(status)) {
std::cout << "[OK] stop_from_handler — child aborted with "
"signal " << WTERMSIG(status)
<< " (MHD self-join detection)\n";
} else if (WIFEXITED(status)) {
const int code = WEXITSTATUS(status);
// kExitCodeStopReturned / kExitCodeCurlCompleted mark
// regressions seeded inside the child. Zero exit means
// stop() returned cleanly from a handler.
LT_CHECK(code != 0 &&
code != kExitCodeStopReturned &&
code != kExitCodeCurlCompleted);
std::cout << "[OK] stop_from_handler — child exited with "
"code " << code
<< " (non-zero exit on contract violation)\n";
}
#endif // _WIN32
LT_END_AUTO_TEST(stop_from_handler_deadlocks_as_documented)
// ---------------------------------------------------------------------
// Sub-test C — adversarial_segments_registration_no_latency_spike
//
// Hammer the registration path with an adversarial corpus of sibling
// path segments to confirm the segment trie's per-segment children
// container is DoS-resistant. With the std::map swap landed in this
// task, per-segment lookup is O(log n) regardless of input shape, so
// even a corpus designed to maximise per-probe cost completes within
// a bounded wall-clock budget without latency spikes.
//
// Corpus shape (union of plan options β + γ):
// - 3 parent prefixes (/api, /v1, /svc) keep three independent
// radix sub-trees populated in parallel so child-map growth at
// each parent node is exercised, not just at root.
// - Each parent gets up to N sibling segments (default 5 000, total
// 15 000 routes) where each segment is a 32-byte string sharing a
// common 24-byte prefix and differing only in the trailing 8
// bytes. The long common prefix is the worst case for
// std::map<std::string>::find: every comparison must scan past
// the shared bytes before discriminating. The high entropy in the
// trailing 8 bytes prevents the tokeniser from collapsing the
// siblings into a deeper sub-tree (each is a distinct radix node
// directly under the parent).
//
// Latency gate (from the noise-floor study): capture per-op
// insert times in nanoseconds via per-thread sample buffers (no
// hot-path lock), then assert
// p95 < 20 × median_of_first_quarter_of_samples.
// This is the deterministic encoding of the task's "no dispatch
// latency spikes > 10× baseline" criterion. We anchor the baseline on
// the first quarter (warmup at low cardinality) and compare against
// the tail (high cardinality, worst case for an O(log n) tree).
// The 20× threshold and p95 statistic come from the noise-floor sweep
// — see `test/PERFORMANCE.md § Methodology — threadsafety_stress
// adversarial_segments latency gate` and the inline justification at
// the gate assertion for the full rationale.
//
// Operating mode: no HTTP server — registration is a webserver API
// call, the daemon is unnecessary, and excluding it removes a noise
// source. Four writer threads contend on route_table_mutex_ to keep
// the writer lock saturated and surface any lock-induced regression.
//
// Duration: bounded by both wall-clock (HTTPSERVER_STRESS_SECONDS,
// default 60 s) AND total ops (kMaxRoutesPerParent), whichever
// completes first. The latter is the principled bound (so the test
// makes the same observation regardless of host speed); the former
// is the safety net for slow hosts (TSan, valgrind, CI).
// ---------------------------------------------------------------------
namespace {
// Generate the i-th adversarial segment for `parent_tag`. Shape:
// "<24-byte-padding><parent-tag>_<8-hex-digits-of-i>"
// The 24-byte padding is identical across all siblings so the per-probe
// strcmp cost in std::map::find scans the shared bytes on every compare
// before reaching the discriminating tail.
std::string adversarial_segment(std::string_view parent_tag, uint32_t i) {
static constexpr char kPad[] = "aaaaaaaaaaaaaaaaaaaaaaaa"; // 24 bytes
char tail[16];
std::snprintf(tail, sizeof(tail), "_%08x", i);
std::string s;
s.reserve(24 + parent_tag.size() + 9);
s.append(kPad, 24);
s.append(parent_tag);
s.append(tail);
return s;
}
// Per-run stats for one adversarial_segments stress round. Hoisted to
// namespace scope (next to adversarial_segment()) rather than declared
// inline in the test body for searchability.
struct round_result {
bool gate_ran = false;
int64_t warmup_median = 0; // ns
int64_t median = 0; // ns
int64_t p95 = 0; // ns
int64_t p99 = 0; // ns
int64_t p999 = 0; // ns
int64_t max_ns = 0;
size_t samples = 0;
int collisions = 0;
};
} // namespace
LT_BEGIN_AUTO_TEST(threadsafety_stress_suite,
adversarial_segments_registration_no_latency_spike)
using StressClock = std::chrono::steady_clock;
using ns = std::chrono::nanoseconds;
constexpr int kWriterThreads = 4;
constexpr int kMaxRoutesPerParent = 5000; // 15 000 total
constexpr std::array<const char*, 3> kParents = {"api", "v1", "svc"};
const int repeats = stress_repeats();
const int pin_cpu = stress_pin_cpu();
// Per-run sampler. Each call performs one full 15 000-op stress
// round on a fresh webserver and returns the gathered stats. Wrapping
// the round in a lambda lets HTTPSERVER_STRESS_REPEATS=N drive N
// back-to-back rounds for noise-floor characterisation without
// touching the surrounding test harness.
auto run_one_round = [&]() -> round_result {
round_result r;
ht::webserver ws{
ht::create_webserver(0)
.start_method(ht::http::http_utils::INTERNAL_SELECT)
.max_threads(2)};
// No ws.start() — registration does not need a running daemon.
std::atomic<bool> stop{false};
// Stabilisation: per-thread sample buffers. The
// previous design pushed each sample into a shared
// std::vector<int64_t> under a global std::mutex INSIDE the
// writer loop. Even though the timing window closed BEFORE the
// lock acquisition, the prior-iteration lock-wait jitter
// shifted cache lines and induced scheduler pressure that
// leaked into the next sample. Per-thread buffers (merged
// once at thread exit) make the hot path lock-free.
std::array<std::vector<int64_t>, kWriterThreads> per_thread_samples;
for (auto& v : per_thread_samples) {
v.reserve(static_cast<size_t>(
kMaxRoutesPerParent * kParents.size() / kWriterThreads
+ kParents.size()));
}
std::atomic<int> register_ok{0};
std::atomic<int> register_collision{0};
auto writer = [&](int tid) {
// Stabilisation: optional Linux CPU pinning.
// Pinning all writers to the same CPU is correct for THIS
// test because they contend on route_table_mutex_ (effectively
// serialised) — single-CPU placement eliminates cross-CPU
// cache misses on segment-trie node memory. macOS / Windows:
// no-op (pin_this_thread_to_cpu returns false). Failure to
// pin is silent (best-effort optimisation, not a contract).
if (pin_cpu >= 0) {
(void)pin_this_thread_to_cpu(pin_cpu);
}
auto& samples = per_thread_samples[tid];
for (uint32_t i = static_cast<uint32_t>(tid);
!stop.load(std::memory_order_relaxed)
&& i < static_cast<uint32_t>(kMaxRoutesPerParent);
i += kWriterThreads) {
for (const char* parent : kParents) {
const std::string path = std::string("/") + parent + "/"
+ adversarial_segment(parent, i);
const auto t0 = StressClock::now();
try {
ws.register_path(
path, std::make_shared<noop_resource>());
const auto dt = std::chrono::duration_cast<ns>(
StressClock::now() - t0).count();
register_ok.fetch_add(
1, std::memory_order_relaxed);
samples.push_back(dt);
} catch (const std::invalid_argument&) {
// Cross-thread duplicate race — contract, not bug.
register_collision.fetch_add(
1, std::memory_order_relaxed);
}
}
}
};
std::vector<std::thread> writers;
writers.reserve(kWriterThreads);
for (int t = 0; t < kWriterThreads; ++t) {
writers.emplace_back(writer, t);
}
// Wall-clock safety net: a watchdog thread flips `stop` when the
// deadline expires. Writers poll `stop` between ops, so they exit
// cleanly even if the corpus would otherwise outrun the budget.
std::thread watchdog([&] {
const auto deadline = StressClock::now()
+ std::chrono::seconds(stress_seconds());
while (StressClock::now() < deadline
&& register_ok.load(std::memory_order_relaxed)
< kMaxRoutesPerParent
* static_cast<int>(kParents.size())) {
std::this_thread::sleep_for(
std::chrono::milliseconds(100));
}
stop.store(true, std::memory_order_relaxed);
});
for (auto& t : writers) t.join();
stop.store(true, std::memory_order_relaxed);
watchdog.join();
r.collisions = register_collision.load();
// Flatten per-thread buffers into one insertion-ordered vector
// (interleave round-robin to roughly preserve wall-clock order,
// so the "first quarter = warmup" baseline still corresponds to
// the low-cardinality regime). Exact ordering across threads is
// impossible without synchronised timestamps, but a round-robin
// merge gives each thread equal weight in the warmup window,
// which is sufficient for the normal case.
//
// PRECONDITION: the round-robin warmup-window approximation is
// valid only when all threads make roughly similar progress.
// If the wall-clock watchdog fires early (stop=true before the
// 15 000-route corpus completes), per-thread buffers may have
// wildly different sizes. The round-robin loop then mixes warmup
// samples from fast threads with tail samples from slow threads
// in unpredictable positions. The minimum-samples guard below
// (samples_ns.size() < 100) rejects extreme cutoff cases, but
// partial-corpus runs with uneven thread progress can still
// produce a skewed warmup_median. This is acceptable for the
// CI use-case: the gate is intentionally skipped when the corpus
// does not complete (rounds_ran == 0 trips the explicit check
// below). An assertion at the gate site verifies this.
std::vector<int64_t> samples_ns;
size_t total = 0;
for (auto& v : per_thread_samples) total += v.size();
samples_ns.reserve(total);
size_t idx = 0;
bool any = true;
while (any) {
any = false;
for (auto& v : per_thread_samples) {
if (idx < v.size()) {
samples_ns.push_back(v[idx]);
any = true;
}
}
++idx;
}
if (samples_ns.size() < 100) {
// Too few samples for a meaningful percentile gate (would
// happen only if the wall-clock deadline cut us short before
// 100 ops landed). Skip the latency gate but pass the test
// — the deadlock-free completion above is itself a pass.
std::cout << "[INFO] adversarial_segments: only "
<< samples_ns.size() << " samples — skipping "
"latency gate (deadlock-free completion is "