Skip to content

ScriptTrigger/CommandsTrigger: exitCondition regex can run forever; Ruby timeout leaks commonPool threads #450

Description

@jymaire

Summary

ScriptTrigger / CommandsTrigger compile the user's exitCondition as a regex and match it against the script's output on every poll. Most modules put no time limit on this match. Ruby has one, but it only stops waiting for the match. Either way, a catastrophic-backtracking exitCondition keeps a CPU core busy forever:

  • Guarded (Ruby): one more runaway ForkJoinPool.commonPool thread per poll, until the JVM's common pool is exhausted.
  • Unguarded (all the others): the evaluating thread itself is blocked inside Matcher.find().

Actual Behaviour

Guarded implementations (Ruby on main, Perl in PR #446):

var future = CompletableFuture.supplyAsync(() -> pattern.matcher(haystack).find());
return future.get(5, TimeUnit.SECONDS);
} catch (TimeoutException te) { return haystack.contains(cond); }

future.get times out and the trigger falls back to a substring match, so evaluation returns and the trigger correctly doesn't fire. But the Matcher keeps running on its commonPool thread: CompletableFuture.cancel can't interrupt it, and Matcher.find() doesn't check for interrupts anyway. Each poll leaves one more thread at 100% CPU.

Once commonPool is saturated (parallelism = cores − 1), later matches just queue up behind the runaway threads. Everything else in the same JVM that relies on commonPool stalls too: CompletableFuture.*Async without an executor, parallel streams, and so on.

Observed with interval: PT10S on a 16-core host. The flow was enabled at 14:17:59:

Time (UTC) Container CPU Threads at ~100% CPU
14:18:25 108% 1
14:18:36 210% 2
14:18:48 328% 3
14:19:10 421% 4
14:19:21 522% 5
after disabling the flow ~657% 6 (still spinning until the container restarted)

In top -H, the hot threads were all ForkJoinPool.commonPool-worker-*.

Unguarded implementations (Bun, Deno, .NET, Go, Node, PowerShell, Python, Shell):

return Pattern.compile(cond).matcher(haystack).find();

The catch (Exception) fallback never fires, because catastrophic backtracking doesn't throw. The thread evaluating the trigger stays stuck in find(). This case wasn't exercised in QA; it follows from the code and from the timing check below.

The #388 security audit flagged this as MEDIUM-001. The timeout was only added to Ruby, and it has the leak described above.

Test gap: the natural test pattern (a+)+$ fails in about 0ms on JDK 25 because the regex engine memoises that loop, so a test using it passes without ever reaching the timeout path. Measured against "{k=" + "a".repeat(40) + "!}" on JDK 25:

Pattern Result
(a+)+$, (a|aa)+$, (a*)*b, (x+x+)+y false in 0ms
(.*a){20}$, (\w+)*\1!\d, (.*){1,32000}[bc] still running after 4s

Expected Behaviour

  • A slow or pathological exitCondition is abandoned within a bounded time and falls back to a substring match. That fallback already exists.
  • Abandoning a match leaves no work running: no leaked threads, no CPU burned.
  • commonPool is never occupied by user-controlled work.
  • The behaviour is the same in every trigger module.

Reproducer

  1. Start Kestra OSS (kestra/kestra:v1.3.39) with plugin-scripts from main.
  2. Create this flow:
id: ruby_redos_repro
namespace: company.team

triggers:
  - id: redos
    type: io.kestra.plugin.scripts.ruby.ScriptTrigger
    interval: PT10S
    exitCondition: "(.*a){20}$"
    edge: true
    script: |
      puts '::{"outputs":{"k":"' + 'a' * 40 + '!"}}::'

tasks:
  - id: log
    type: io.kestra.plugin.core.log.Log
    message: "should never fire: {{ trigger.vars }}"
  1. Watch the Kestra container for about 1 minute:
docker stats --no-stream <kestra-container>
docker exec <kestra-container> sh -c "top -H -b -n1 | head -20"
  1. Actual: the trigger never fires, which is correct. But CPU goes up by about one core per poll, and new ForkJoinPool.commonPool-worker-* threads stay at 100%. Disabling the flow doesn't free them; only restarting the JVM does.
  2. For an unguarded module, run the same flow with io.kestra.plugin.scripts.shell.ScriptTrigger and script: echo '::{"outputs":{"k":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!"}}::'.

The evidence above was gathered with the Perl triggers from PR #446, which carry the same guard as Ruby line for line.

Logs / Stack Trace

No log entries or exceptions. The only symptom is CPU use and the hot commonPool threads.

Environment

  • Kestra version: 1.3.39 (OSS, Docker image kestra/kestra:v1.3.39, standalone with H2, JDK 25.0.4 in the container)
  • Plugin version: plugin-scripts 1.10.1-SNAPSHOT (main at a03cd85, plus PR feat(perl): add script commands trigger #446 for Perl)
  • Deployment: OSS self-hosted (local Docker)

Additional Context

Affected on main:

  • Guarded but leaking: plugin-script-ruby (ScriptTrigger, CommandsTrigger). Also Perl, open in feat(perl): add script commands trigger #446.
  • Unguarded: plugin-script-bun, -deno, -dotnet, -go, -node, -powershell, -python (AbstractPythonTrigger), -shell. Both triggers in each, where applicable.

Suggested fix: reuse the fix already in Kestra core. The same problem in Pebble regex filters (kestra-io/kestra#15488) was fixed by kestra-io/kestra#15490. That PR added io.kestra.core.utils.RegexUtils, which wraps the input CharSequence so that every charAt() checks a deadline. Once the deadline passes it throws RegexTimeoutException, so the match stops on the caller's thread and no extra thread or executor is involved. Its API includes matches(Pattern, CharSequence[, Duration]), matcher(...), isSafeUserRegex(...) and syntaxError(...), and the timeout is configured globally by RegexConfiguration.

  • RegexUtils ships in Kestra 2.0.0 and isn't in 1.3.x. plugin-scripts currently compiles against kestraVersion=1.3.39.
  • When the plugin targets Kestra ≥ 2.0: replace every matchesCondition regex call with RegexUtils.matches(pattern, haystack). Catch RegexTimeoutException (plus PatternSyntaxException) and fall back to haystack.contains(cond). Remove Ruby's CompletableFuture guard. Optionally reject a pathological exitCondition up front with isSafeUserRegex.
  • While the plugin still targets 1.3.x: copy the same small deadline-CharSequence wrapper into plugin-script as a temporary shim that mirrors RegexUtils' behaviour, and swap it for RegexUtils when the plugin is bumped.

Either way, put the logic in a shared helper in plugin-script (see the AbstractScriptTrigger TODO) so all trigger modules behave the same.

Tests:

  • Use a pattern that is still catastrophic on JDK 21+/25, such as (.*a){20}$, not (a+)+$.
  • Assert both the fallback result and that it returns within about 1–2× the deadline.
  • Where possible, also assert that no extra commonPool threads are left busy afterwards.

See also: #388 (security audit, MEDIUM-001) and kestra-io/kestra#15488 (the same issue in core, fixed by kestra-io/kestra#15490).


View as Artifact

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/pluginPlugin-related issue or feature requestkind/performancePerformance-related issuekind/securitySecurity-related issue

    Type

    Fields

    Stage

    Backlog

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions