Raise worker_threads default to 1, make dispatch thread ingress-only - #8404
Amaury Chamayou (achamayou) merged 4 commits into
Conversation
a106afb to
5f579fd
Compare
worker_threads default to 1, make dispatch thread ingress-only
There was a problem hiding this comment.
🟡 Changes recommended
The dispatch loop now unconditionally waits on a coalesced work beacon each iteration, which can introduce avoidable ingress latency when read_n hits its per-iteration budget and there is still queued ringbuffer work.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
This PR updates CCF’s threading/configuration defaults so the enclave dispatch thread (Enclave::run_main) is ingress-only, and task execution always happens on dedicated worker thread(s). It raises the effective minimum/default worker_threads to ensure there is always at least one task executor, while preserving compatibility by coercing worker_threads: 0 to 1.
Changes:
- Make the dispatch thread ingress-only by removing task draining from
Enclave::run_mainand relying onrun_workerthreads for task execution. - Raise the default
worker_threadsto1across code, schema, build defaults, tests, and sample configs; coerce0 -> 1in host config validation with a log message. - Update documentation and release notes to describe the new default and coercion behavior.
Custom instructions used (custom):
/.github/copilot-instructions.md/.github/instructions/changelog.instructions.md/.github/instructions/reviewing.instructions.md
File summaries
| File | Description |
|---|---|
| tests/infra/e2e_args.py | Update CLI default --worker-threads to 1 for e2e runs. |
| src/host/run.cpp | Coerce worker_threads < 1 to 1 during config processing. |
| src/enclave/enclave.h | Remove task draining from dispatch loop; keep worker threads as task executors. |
| samples/config/start_config.json | Update sample worker_threads to 1. |
| samples/config/recover_config.json | Update sample worker_threads to 1. |
| samples/config/join_config.json | Update sample worker_threads to 1. |
| include/ccf/node/startup_config.h | Update worker_threads default to 1 and document coercion behavior. |
| doc/operations/resource_usage.rst | Document default 1 and 0 -> 1 coercion. |
| doc/host_config_schema/host_config.json | Update schema default/description for worker_threads. |
| doc/architecture/threading.rst | Document default 1 and 0 -> 1 coercion. |
| CMakeLists.txt | Update cached WORKER_THREADS default to 1 for test args. |
| CHANGELOG.md | Add release note entry for new default/coercion behavior. |
Review details
- Files reviewed: 12/12 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The enclave dispatch thread (Enclave::run_main) previously drained up to 256 tasks from the main job board on every iteration, in addition to draining the inbound ringbuffer. This was a compatibility bodge from the old task system: it let opaque, potentially blocking tasks (ledger fsync, TLS work, historical deserialisation, ...) execute on the consensus ingress thread, and with worker_threads=0 that thread was the only task executor. run_main() is now ingress-only: it waits on the work beacon, drains ringbuffer messages (node-to-node inbound, ticks), and handles stop/stop-notice. Task execution happens exclusively on run_worker() host threads. The job board no longer needs to wake the dispatch thread when tasks become available, so the work beacon is no longer wired into JobBoard::set_work_beacon from the Enclave constructor. Since the host libuv loop is not a suitable task executor either (blocking tasks there would stall node-to-node and RPC socket polling), there must always be at least one run_worker thread. worker_threads now defaults to 1: - include/ccf/node/startup_config.h and doc/host_config_schema/host_config.json (and the config_schema.h generated from it) declare the new default. - CMakeLists.txt's WORKER_THREADS cache variable and tests/infra/e2e_args.py's --worker-threads default are updated to match, along with the sample configuration files. To avoid a breaking change in a patch release, worker_threads: 0 is not rejected: src/host/run.cpp's validate_and_coerce_worker_threads coerces it up to 1 and logs a LOG_FAIL_FMT message noting the substitution, before the node starts or --check returns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
5f579fd to
256cc10
Compare
…aise-worker-threads-minimum-to-one
DescriptionComparing 1 available run from this branch (#8404) against the trend of the last 30 Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent Axis labels show the latest branch value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where the latest run improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory. A benchmark which does not exist on Throughput (tx/s)---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(4){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(6){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(7){fill:#E5484D!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic Blocking 100ms: 3,061 tx/s ▼ 1%"]
axis b1["Basic Blocking 20ms: 15,199 tx/s ▬ -1%"]
axis b2["Basic Blocking 2ms: 39,923 tx/s ▼ 22%"]
axis b3["Basic JS: 12,536 tx/s ▼ 20%"]
axis b4["Historical Queries: 711,812 tx/s ▼ 27%"]
axis b5["L…g Certificate Blocking: 28,837 tx/s ▼ 2%"]
axis b6["Logging JWT Blocking: 15,254 tx/s ▼ 1%"]
curve stddev2_high["main EWMA + 2 std dev"]{100.47, 104.87, 108.73, 110.40, 111.60, 100.83, 100.35}
curve stddev1_high["main EWMA + 1 std dev"]{100.24, 102.44, 104.37, 105.20, 105.80, 100.41, 100.18}
curve stddev1_low["main EWMA - 1 std dev"]{99.76, 97.56, 95.63, 94.80, 94.20, 99.59, 99.82}
curve stddev2_low["main EWMA - 2 std dev"]{99.53, 95.13, 91.27, 89.60, 88.40, 99.17, 99.65}
curve branch_0["#8404"]{99.06, 99.24, 78.10, 80.04, 73.09, 97.93, 99.34}
graticule polygon
max 126
min 59
ticks 0
showLegend false
Latency (ms)---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(4){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
.radarAxisLabel:nth-of-type(7){fill:#E5484D!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic Blocking 100ms: 99 ms ▬ 0%"]
axis b1["Basic Blocking 20ms: 19 ms ▬ 0%"]
axis b2["Basic Blocking 2ms: 7 ms ▲ 35%"]
axis b3["Basic JS: 24 ms ▲ 23%"]
axis b4["Historical Queries: 42 ms ▲ 36%"]
axis b5["Logging Certificate Blocking: 19 ms ▬ 0%"]
axis b6["Logging JWT Blocking: 20 ms ▲ 5%"]
curve stddev2_high["main EWMA + 2 std dev"]{100.61, 100.00, 115.24, 111.06, 110.44, 100.00, 101.88}
curve stddev1_high["main EWMA + 1 std dev"]{100.30, 100.00, 107.62, 105.53, 105.22, 100.00, 100.94}
curve stddev1_low["main EWMA - 1 std dev"]{99.70, 100.00, 92.38, 94.47, 94.78, 100.00, 99.06}
curve stddev2_low["main EWMA - 2 std dev"]{99.39, 100.00, 84.76, 88.94, 89.56, 100.00, 98.12}
curve branch_0["#8404"]{100.04, 100.00, 134.76, 122.93, 136.35, 100.00, 104.74}
graticule polygon
max 155
min 66
ticks 0
showLegend false
Memory (bytes)---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(2){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(3){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(4){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(7){fill:#2DA44E!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic Blocking 100ms: 85.7 MiB ▼ 2%"]
axis b1["Basic Blocking 20ms: 89.6 MiB ▲ 2%"]
axis b2["Basic Blocking 2ms: 88.3 MiB ▼ 2%"]
axis b3["Basic JS: 90.4 MiB ▼ 6%"]
axis b4["Historical Queries: 149 MiB ▬ 0%"]
axis b5["Logging Certificate Blocking: 109 MiB ▼ 3%"]
axis b6["Logging JWT Blocking: 84.4 MiB ▼ 2%"]
curve stddev2_high["main EWMA + 2 std dev"]{102.15, 102.70, 102.28, 102.43, 101.61, 102.01, 102.13}
curve stddev1_high["main EWMA + 1 std dev"]{101.08, 101.35, 101.14, 101.22, 100.81, 101.00, 101.07}
curve stddev1_low["main EWMA - 1 std dev"]{98.92, 98.65, 98.86, 98.78, 99.19, 99.00, 98.93}
curve stddev2_low["main EWMA - 2 std dev"]{97.85, 97.30, 97.72, 97.57, 98.39, 97.99, 97.87}
curve branch_0["#8404"]{98.09, 101.98, 97.96, 94.45, 99.89, 96.65, 97.67}
graticule polygon
max 107
min 90
ticks 0
showLegend false
Rate (ops/s)---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(5){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(6){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
.radarAxisLabel:nth-of-type(8){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(9){fill:#2DA44E!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["CCF c…n c…t lifecycle: 21,407 ops/s ▬ +2%"]
axis b1["CCF fresh JS invocation: 19,704 ops/s ▲ 3%"]
axis b2["CHAMP get: 63,964,020 ops/s ▬ +1%"]
axis b3["CHAMP put: 8,232,802 ops/s ▲ 3%"]
axis b4["KV deserialisation: 2,820,874 ops/s ▲ 16%"]
axis b5["KV serialisation: 2,502,502 ops/s ▲ 20%"]
axis b6["KV s…t deserialisation: 6,147 ops/s ▬ -1%"]
axis b7["KV snapshot serialisation: 4,420 ops/s ▼ 7%"]
axis b8["Q…S s…d context lifecycle: 26,271 ops/s ▲ 2%"]
curve stddev2_high["main EWMA + 2 std dev"]{104.60, 104.96, 104.18, 105.12, 108.10, 109.28, 104.80, 105.82, 104.61}
curve stddev1_high["main EWMA + 1 std dev"]{102.30, 102.48, 102.09, 102.56, 104.05, 104.64, 102.40, 102.91, 102.30}
curve stddev1_low["main EWMA - 1 std dev"]{97.70, 97.52, 97.91, 97.44, 95.95, 95.36, 97.60, 97.09, 97.70}
curve stddev2_low["main EWMA - 2 std dev"]{95.40, 95.04, 95.82, 94.88, 91.90, 90.72, 95.20, 94.18, 95.39}
curve branch_0["#8404"]{102.11, 102.65, 101.40, 103.06, 116.50, 120.22, 99.18, 92.78, 102.40}
graticule polygon
max 131
min 80
ticks 0
showLegend false
|
Motivation
The enclave dispatch thread (
Enclave::run_main) previously drained up to 256 tasks fromccf::tasks::get_main_job_board()on every iteration, in addition to the inbound ringbuffer (node-to-node inbound, ticks). This was a compatibility bodge from the old task system: it let opaque, potentially blocking tasks (JWT key auto-refresh and quote endorsement HTTP requests, historical query LFS access, retired-node cleanup requests, ...) execute on the consensus ingress thread, and withworker_threads=0(the previous default/minimum) that thread was the only task executor.Implementation summary
Enclave::run_main()(src/enclave/enclave.h) is now ingress-only: it waits on the work beacon, drains ringbuffer messages, and handles stop/stop-notice. It no longer executes tasks.run_worker()threads are now the sole task executors. The job board no longer needs to wake the dispatch thread, soJobBoard::set_work_beaconis no longer wired up from theEnclaveconstructor/destructor.run_workerthread.worker_threadsnow defaults to1(include/ccf/node/startup_config.h,doc/host_config_schema/host_config.json,CMakeLists.txt'sWORKER_THREADScache variable,tests/infra/e2e_args.py, and the sample configuration files).worker_threads: 0is not rejected:src/host/run.cpp's newvalidate_and_coerce_worker_threadscoerces it up to1and logs a[fail]-level message noting the substitution, before the node starts or--checkreturns.doc/architecture/threading.rst,doc/operations/resource_usage.rst, andCHANGELOG.mdto describe the new default and the0->1coercion.Safety and compatibility
node_inbound, ledger entry range, and tick message handling on the dispatch thread are unchanged, only the (unrelated) task-draining loop that used to run alongside them is removed.worker_threads=0now always run on a dedicated worker thread. Existing configurations withworker_threads: 0keep working (coerced to1, logged) rather than being rejected, so this is not a breaking change for a patch release; new/default configurations getworker_threads=1.