tests: make sure cleanup is called as expect when tests time out

[chen1195585098]

Modification includes:

1. change 'echo' cmd in function force_terminate as a subshell, to ensure immediate buffer flush. Otherwise, extra stdout buffer in function cleanup may be redirected to stderr, in such a case, function call in cleanup may get something wrong.
2. change --foreground as a optional setting, since it affects signal detection and cleanup procedure.

With this patch, by default, even if the use case times out, the cleanup process will proceed as expected. And the test env always keep clean after testcase exits.

Fixes: #4610

Related patches:
run-tests.sh: stop test on Ctrl-C
1f03309

tests: call cleanup on receiving external signals INT, TERM and HUP
ea980a8

[chen1195585098]

Hi, @xhernandez @pkalever , could you please review this patch when you have a moment？

Thanks :)

[ThalesBarretto]

Hi @chen1195585098 — your diagnosis here was right: 1f03309's --foreground did break timeout cleanup, and force_terminate is exactly the mechanism that stopped firing. Thank you for the report and the analysis in #4610.

While verifying this change we found the problem runs deeper than the flag: timeout stops supervising the moment prove dies, and bash's job notice can SIGPIPE the test shell before its trap runs — so cleanup stays unreliable even with group signalling, and interactive Ctrl-C cleanup turns out to have been affected as well. The full defect chain is written up in #4737.

We have proposed #4738, which keeps your group-signal insight and adds harness-side supervision (plus an idempotent sweep, so the outcome no longer depends on the race), with reproduction drivers and A/B results on real hardware. Your review there would be very welcome.