fix(offload-test): a dead stage's marker keeps the platform's own incident id - #133
Merged
Merged
Conversation
…ident id `ExecFailed: exec failed (exit -1): internal error; reference = aggq3f5m407e2vb2ht76l2vf` is what a dying container reports. The marker recorded the CLASS and dropped the reference — which is backwards: the class is the half a consumer can already infer from the fact that their stage died, and the reference is the only half that identifies the incident to the platform's operator. It matters now because the resource explanations are gone. On the consumer that motivated the staging work, the heaviest stage peaks at 2.2 GiB of 11.9 GiB with 8.4 GB of disk to spare, and dies at 137s, 647s and 1284s against successes at 2128s and 2176s. Nothing is scarce and no duration is safe, so the next question is one only the platform can answer — and answering it needs the id. EXTRACTED rather than interpolated. The marker is shell-quoted into a `printf '%s\n' '<line>'`, so free vendor text could close the quote; a `[A-Za-z0-9_-]` reference and a signed integer exit code cannot. The test pins that charset rather than trusting the shape of today's message.
![flaredispatch-fractalboxdev[bot]](https://avatars.githubusercontent.com/in/4273665?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem & Insight
A dying container reports:
The staged marker recorded
stage=<label> error=ExecFailed elapsedMs=<n>— the class, and not the reference. That is backwards. The class is the half a consumer can already infer from the fact that their stage died; the reference is the only half that identifies the incident to the platform's operator, and it was being thrown away at the one moment someone needs it.It matters now because the resource explanations are gone. On the consumer whose gate motivated staged mode, with the per-stage memory probe running:
Nothing is scarce, and no duration is safe. Whatever is killing these containers is not the workload — which makes the next question one only the platform can answer, and answering it needs the id.
Take
The marker carries
exit=<n>andreference=<id>when the failure supplied them.Extracted, not interpolated. The marker is shell-quoted into a
printf '%s\n' '<line>', so free vendor text could close the quote and change what the container executes. A[A-Za-z0-9_-]{1,64}reference and a signed integer cannot. The test pins that charset rather than trusting the shape of today's message.Key actions
ExecFailedtext: the marker keeps the reference, and whatever lands afterreference=matches^[A-Za-z0-9_-]+$