Skip to content

Raise worker_threads default to 1, make dispatch thread ingress-only - #8404

Merged
Amaury Chamayou (achamayou) merged 4 commits into
mainfrom
agents/raise-worker-threads-minimum-to-one
Sep 21, 2026
Merged

Amaury Chamayou (achamayou) merged 4 commits into
mainfrom
agents/raise-worker-threads-minimum-to-one

Conversation

@eddyashton

@eddyashton Eddy Ashton (eddyashton) commented Sep 18, 2026

Copy link
Copy Markdown
Member

Motivation

The enclave dispatch thread (Enclave::run_main) previously drained up to 256 tasks from ccf::tasks::get_main_job_board() on every iteration, in addition to the inbound ringbuffer (node-to-node inbound, ticks). This was a compatibility bodge from the old task system: it let opaque, potentially blocking tasks (JWT key auto-refresh and quote endorsement HTTP requests, historical query LFS access, retired-node cleanup requests, ...) execute on the consensus ingress thread, and with worker_threads=0 (the previous default/minimum) that thread was the only task executor.

Implementation summary

  • Enclave::run_main() (src/enclave/enclave.h) is now ingress-only: it waits on the work beacon, drains ringbuffer messages, and handles stop/stop-notice. It no longer executes tasks. run_worker() threads are now the sole task executors. The job board no longer needs to wake the dispatch thread, so JobBoard::set_work_beacon is no longer wired up from the Enclave constructor/destructor.
  • Since the host libuv loop is not a suitable task executor either (blocking tasks there would stall node-to-node and RPC socket polling), there must always be at least one run_worker thread. worker_threads now defaults to 1 (include/ccf/node/startup_config.h, doc/host_config_schema/host_config.json, CMakeLists.txt's WORKER_THREADS cache variable, tests/infra/e2e_args.py, and the sample configuration files).
  • To avoid a breaking change in a patch release, worker_threads: 0 is not rejected: src/host/run.cpp's new validate_and_coerce_worker_threads coerces it up to 1 and logs a [fail]-level message noting the substitution, before the node starts or --check returns.
  • Updated doc/architecture/threading.rst, doc/operations/resource_usage.rst, and CHANGELOG.md to describe the new default and the 0 -> 1 coercion.

Safety and compatibility

  • No ledger/wire format or API changes. This is a node-process threading/config behaviour change only, with no impact on mixed-version operation, recovery, or consensus safety: node_inbound, ledger entry range, and tick message handling on the dispatch thread are unchanged, only the (unrelated) task-draining loop that used to run alongside them is removed.
  • Behaviour change: applications/tasks that used to opportunistically run on the dispatch thread when worker_threads=0 now always run on a dedicated worker thread. Existing configurations with worker_threads: 0 keep working (coerced to 1, logged) rather than being rejected, so this is not a breaking change for a patch release; new/default configurations get worker_threads=1.
  • No new security-sensitive surface: task execution moves entirely off the consensus-ingress thread, which is a defense-in-depth improvement (opaque/blocking tasks can no longer stall ringbuffer processing or elections).

@eddyashton
Eddy Ashton (eddyashton) requested a review from a team as a code owner September 18, 2026 15:29
Copilot AI lite review requested due to automatic review settings September 18, 2026 15:29
@eddyashton
Eddy Ashton (eddyashton) force-pushed the agents/raise-worker-threads-minimum-to-one branch from a106afb to 5f579fd Compare September 18, 2026 15:31
@eddyashton Eddy Ashton (eddyashton) changed the title host: Raise worker_threads default to 1, make dispatch thread ingress-only Raise worker_threads default to 1, make dispatch thread ingress-only Sep 18, 2026
@eddyashton Eddy Ashton (eddyashton) changed the title Raise worker_threads default to 1, make dispatch thread ingress-only Raise worker_threads default to 1, make dispatch thread ingress-only Sep 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The dispatch loop now unconditionally waits on a coalesced work beacon each iteration, which can introduce avoidable ingress latency when read_n hits its per-iteration budget and there is still queued ringbuffer work.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR updates CCF’s threading/configuration defaults so the enclave dispatch thread (Enclave::run_main) is ingress-only, and task execution always happens on dedicated worker thread(s). It raises the effective minimum/default worker_threads to ensure there is always at least one task executor, while preserving compatibility by coercing worker_threads: 0 to 1.

Changes:

  • Make the dispatch thread ingress-only by removing task draining from Enclave::run_main and relying on run_worker threads for task execution.
  • Raise the default worker_threads to 1 across code, schema, build defaults, tests, and sample configs; coerce 0 -> 1 in host config validation with a log message.
  • Update documentation and release notes to describe the new default and coercion behavior.

Custom instructions used (custom):

  • /.github/copilot-instructions.md
  • /.github/instructions/changelog.instructions.md
  • /.github/instructions/reviewing.instructions.md
File summaries
File Description
tests/infra/e2e_args.py Update CLI default --worker-threads to 1 for e2e runs.
src/host/run.cpp Coerce worker_threads < 1 to 1 during config processing.
src/enclave/enclave.h Remove task draining from dispatch loop; keep worker threads as task executors.
samples/config/start_config.json Update sample worker_threads to 1.
samples/config/recover_config.json Update sample worker_threads to 1.
samples/config/join_config.json Update sample worker_threads to 1.
include/ccf/node/startup_config.h Update worker_threads default to 1 and document coercion behavior.
doc/operations/resource_usage.rst Document default 1 and 0 -> 1 coercion.
doc/host_config_schema/host_config.json Update schema default/description for worker_threads.
doc/architecture/threading.rst Document default 1 and 0 -> 1 coercion.
CMakeLists.txt Update cached WORKER_THREADS default to 1 for test args.
CHANGELOG.md Add release note entry for new default/coercion behavior.
Review details
  • Files reviewed: 12/12 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/enclave/enclave.h Outdated
@eddyashton Eddy Ashton (eddyashton) added the run-long-test Run Long Test job label Sep 18, 2026
The enclave dispatch thread (Enclave::run_main) previously drained up
to 256 tasks from the main job board on every iteration, in addition
to draining the inbound ringbuffer. This was a compatibility bodge
from the old task system: it let opaque, potentially blocking tasks
(ledger fsync, TLS work, historical deserialisation, ...) execute on
the consensus ingress thread, and with worker_threads=0 that thread
was the only task executor.

run_main() is now ingress-only: it waits on the work beacon, drains
ringbuffer messages (node-to-node inbound, ticks), and handles
stop/stop-notice. Task execution happens exclusively on run_worker()
host threads. The job board no longer needs to wake the dispatch
thread when tasks become available, so the work beacon is no longer
wired into JobBoard::set_work_beacon from the Enclave constructor.

Since the host libuv loop is not a suitable task executor either
(blocking tasks there would stall node-to-node and RPC socket
polling), there must always be at least one run_worker thread.
worker_threads now defaults to 1:
- include/ccf/node/startup_config.h and doc/host_config_schema/host_config.json
  (and the config_schema.h generated from it) declare the new default.
- CMakeLists.txt's WORKER_THREADS cache variable and
  tests/infra/e2e_args.py's --worker-threads default are updated to
  match, along with the sample configuration files.

To avoid a breaking change in a patch release, worker_threads: 0 is
not rejected: src/host/run.cpp's validate_and_coerce_worker_threads
coerces it up to 1 and logs a LOG_FAIL_FMT message noting the
substitution, before the node starts or --check returns.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@eddyashton
Eddy Ashton (eddyashton) force-pushed the agents/raise-worker-threads-minimum-to-one branch from 5f579fd to 256cc10 Compare September 18, 2026 15:54
@eddyashton
Eddy Ashton (eddyashton) marked this pull request as draft September 21, 2026 14:17
@eddyashton
Eddy Ashton (eddyashton) marked this pull request as ready for review September 21, 2026 15:53
@achamayou
Amaury Chamayou (achamayou) merged commit 4fa6651 into main Sep 21, 2026
11 of 12 checks passed
@achamayou
Amaury Chamayou (achamayou) deleted the agents/raise-worker-threads-minimum-to-one branch September 21, 2026 16:24
@github-actions

Copy link
Copy Markdown
Description

Comparing 1 available run from this branch (#8404) against the trend of the last 30 main runs.

Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent main runs, using a 7-run half-life. The orange line is this branch's latest run; the darker blue band is the main baseline +/- 1 std dev and the lighter blue band around it is +/- 2 std dev.

Axis labels show the latest branch value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where the latest run improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory.

A benchmark which does not exist on main yet has no baseline of its own, so its earliest available run from this branch is used as its reference and its band is measured across this branch's runs. Its axis is normalized, scaled and coloured like any other, but the comparison is against this branch rather than against main.

Throughput (tx/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(4){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(6){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(7){fill:#E5484D!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic Blocking 100ms: 3,061 tx/s ▼ 1%"]
  axis b1["Basic Blocking 20ms: 15,199 tx/s ▬ -1%"]
  axis b2["Basic Blocking 2ms: 39,923 tx/s ▼ 22%"]
  axis b3["Basic JS: 12,536 tx/s ▼ 20%"]
  axis b4["Historical Queries: 711,812 tx/s ▼ 27%"]
  axis b5["L…g Certificate Blocking: 28,837 tx/s ▼ 2%"]
  axis b6["Logging JWT Blocking: 15,254 tx/s ▼ 1%"]
  curve stddev2_high["main EWMA + 2 std dev"]{100.47, 104.87, 108.73, 110.40, 111.60, 100.83, 100.35}
  curve stddev1_high["main EWMA + 1 std dev"]{100.24, 102.44, 104.37, 105.20, 105.80, 100.41, 100.18}
  curve stddev1_low["main EWMA - 1 std dev"]{99.76, 97.56, 95.63, 94.80, 94.20, 99.59, 99.82}
  curve stddev2_low["main EWMA - 2 std dev"]{99.53, 95.13, 91.27, 89.60, 88.40, 99.17, 99.65}
  curve branch_0["#8404"]{99.06, 99.24, 78.10, 80.04, 73.09, 97.93, 99.34}
  graticule polygon
  max 126
  min 59
  ticks 0
  showLegend false
Loading

Latency (ms)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(4){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(7){fill:#E5484D!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic Blocking 100ms: 99 ms ▬ 0%"]
  axis b1["Basic Blocking 20ms: 19 ms ▬ 0%"]
  axis b2["Basic Blocking 2ms: 7 ms ▲ 35%"]
  axis b3["Basic JS: 24 ms ▲ 23%"]
  axis b4["Historical Queries: 42 ms ▲ 36%"]
  axis b5["Logging Certificate Blocking: 19 ms ▬ 0%"]
  axis b6["Logging JWT Blocking: 20 ms ▲ 5%"]
  curve stddev2_high["main EWMA + 2 std dev"]{100.61, 100.00, 115.24, 111.06, 110.44, 100.00, 101.88}
  curve stddev1_high["main EWMA + 1 std dev"]{100.30, 100.00, 107.62, 105.53, 105.22, 100.00, 100.94}
  curve stddev1_low["main EWMA - 1 std dev"]{99.70, 100.00, 92.38, 94.47, 94.78, 100.00, 99.06}
  curve stddev2_low["main EWMA - 2 std dev"]{99.39, 100.00, 84.76, 88.94, 89.56, 100.00, 98.12}
  curve branch_0["#8404"]{100.04, 100.00, 134.76, 122.93, 136.35, 100.00, 104.74}
  graticule polygon
  max 155
  min 66
  ticks 0
  showLegend false
Loading

Memory (bytes)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(2){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(3){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(4){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(5){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(6){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(7){fill:#2DA44E!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic Blocking 100ms: 85.7 MiB ▼ 2%"]
  axis b1["Basic Blocking 20ms: 89.6 MiB ▲ 2%"]
  axis b2["Basic Blocking 2ms: 88.3 MiB ▼ 2%"]
  axis b3["Basic JS: 90.4 MiB ▼ 6%"]
  axis b4["Historical Queries: 149 MiB ▬ 0%"]
  axis b5["Logging Certificate Blocking: 109 MiB ▼ 3%"]
  axis b6["Logging JWT Blocking: 84.4 MiB ▼ 2%"]
  curve stddev2_high["main EWMA + 2 std dev"]{102.15, 102.70, 102.28, 102.43, 101.61, 102.01, 102.13}
  curve stddev1_high["main EWMA + 1 std dev"]{101.08, 101.35, 101.14, 101.22, 100.81, 101.00, 101.07}
  curve stddev1_low["main EWMA - 1 std dev"]{98.92, 98.65, 98.86, 98.78, 99.19, 99.00, 98.93}
  curve stddev2_low["main EWMA - 2 std dev"]{97.85, 97.30, 97.72, 97.57, 98.39, 97.99, 97.87}
  curve branch_0["#8404"]{98.09, 101.98, 97.96, 94.45, 99.89, 96.65, 97.67}
  graticule polygon
  max 107
  min 90
  ticks 0
  showLegend false
Loading

Rate (ops/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(5){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(6){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(8){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(9){fill:#2DA44E!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["CCF c…n c…t lifecycle: 21,407 ops/s ▬ +2%"]
  axis b1["CCF fresh JS invocation: 19,704 ops/s ▲ 3%"]
  axis b2["CHAMP get: 63,964,020 ops/s ▬ +1%"]
  axis b3["CHAMP put: 8,232,802 ops/s ▲ 3%"]
  axis b4["KV deserialisation: 2,820,874 ops/s ▲ 16%"]
  axis b5["KV serialisation: 2,502,502 ops/s ▲ 20%"]
  axis b6["KV s…t deserialisation: 6,147 ops/s ▬ -1%"]
  axis b7["KV snapshot serialisation: 4,420 ops/s ▼ 7%"]
  axis b8["Q…S s…d context lifecycle: 26,271 ops/s ▲ 2%"]
  curve stddev2_high["main EWMA + 2 std dev"]{104.60, 104.96, 104.18, 105.12, 108.10, 109.28, 104.80, 105.82, 104.61}
  curve stddev1_high["main EWMA + 1 std dev"]{102.30, 102.48, 102.09, 102.56, 104.05, 104.64, 102.40, 102.91, 102.30}
  curve stddev1_low["main EWMA - 1 std dev"]{97.70, 97.52, 97.91, 97.44, 95.95, 95.36, 97.60, 97.09, 97.70}
  curve stddev2_low["main EWMA - 2 std dev"]{95.40, 95.04, 95.82, 94.88, 91.90, 90.72, 95.20, 94.18, 95.39}
  curve branch_0["#8404"]{102.11, 102.65, 101.40, 103.06, 116.50, 120.22, 99.18, 92.78, 102.40}
  graticule polygon
  max 131
  min 80
  ticks 0
  showLegend false
Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bench-ab run-long-test Run Long Test job

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants