:farmer: rclcpp TestBusyWaiting failing in nightly linux repeated/rhel debug by a clock difference

[miguelgonrod]

Generated by Generative AI

No response

Operating System:

Ubuntu resolute

ROS version or commit hash:

Rolling

RMW implementation (if applicable):

No response

RMW Configuration (if applicable):

No response

Client library (if applicable):

No response

'ros2 doctor --report' output

No response

Steps to reproduce issue

1. Go to https://ci.ros2.org/job/nightly_linux_repeated
2. Build a job
3. See test failing

Expected behavior

To build without test regressions

Actual behavior

Building with test regressions

Additional information

Reference Build

• nightly_linux_repeated#4278

Failing tests

• projectroot.test.rclcpp.test_executors_busy_waiting
• rclcpp.TestBusyWaiting/EventsCBGExecutor.test_spin

Log output

[       OK ] TestBusyWaiting/EventsCBGExecutor.test_spin_some (16 ms)
[ RUN      ] TestBusyWaiting/EventsCBGExecutor.test_spin
/home/jenkins-agent/workspace/nightly_linux_repeated/ws/src/ros2/rclcpp/rclcpp/test/rclcpp/executors/test_executors_busy_waiting.cpp:87: Failure
Expected: (std::chrono::steady_clock::now() - start_time) < (max_duration / 2), actual: 10000440627ns vs 5s
executor took a long time to execute when it should have done nothing and should not have blocked either, but this could be a false negative if the computer is really slow

[  FAILED  ] TestBusyWaiting/EventsCBGExecutor.test_spin, where TypeParam = rclcpp::executors::EventsCBGExecutor (10016 ms)
[----------] 3 tests from TestBusyWaiting/EventsCBGExecutor (10050 ms total)

Flakyness 15 days

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
nightly_linux_repeated	2026-05-21	2026-05-08	14	13	92.86
nightly_linux-rhel_repeated	2026-05-21	2026-05-10	13	8	61.54
nightly_linux_debug	2026-05-18	2026-05-18	14	1	7.14
nightly_linux-aarch64_repeated	2026-05-17	2026-05-17	12	1	8.33
nightly_linux-rhel_release	2026-05-16	2026-05-09	14	3	21.43
nightly_linux_release	2026-05-12	2026-05-12	14	1	7.14

This have happened for a long time so jenkins history doesn't show the first time.

executor took a long time to execute when it should have done nothing and should not have blocked either, but this could be a false negative if the computer is really slow

In regression trend history it seems inconsistent some times, could this time difference be related to buildfarm load? Maybe when the jobs load is too high this gets affected, the time difference is around 5 seconds

CC: @Crola1702

[fujitatomoya]

i am unable to reproduce this issue with my local env.

root@tomoyafujita-B760M-Pro-RS-D4:~/ros2_ws/colcon_ws# colcon test --event-handlers console_direct+ --packages-select rclcpp --retest-until-fail 100 --ctest-args -R test_executors_busy_waiting
...
1/1 Test #99: test_executors_busy_waiting ......   Passed    0.29 sec

The following tests passed:
        test_executors_busy_waiting

100% tests passed, 0 tests failed out of 1

Label Time Summary:
gtest    =   0.29 sec*proc (1 test)

Total Test time (real) =  29.42 sec
[Processing: rclcpp]
Finished <<< rclcpp [30.0s]

Summary: 1 package finished [30.7s]

[jmachowinski]

This smells like some race condition, I a planing on looking into it, but don't have time right now, I try to make time for it this Friday.

[jmachowinski]

Should be fixed now, please reopen if the error returns