Add a distinct "prepared" state for pull-labs jobs

[nuclearcat]

Pull-labs jobs currently use state=available to mean "ready for an external lab to pick up". Elsewhere in the pipeline, available means "this node has produced a usable result/artifact".

This overload confused send_kcidb: it listens for available nodes and can submit pull-labs jobs to KCIDB before they are actually finished, producing dashboard records with status: NULL.

Short-term fix:

• send_kcidb should only submit job, test, and process nodes after state=done.
  (46cb4a1 in pipeline)

Long-term fix:

• Add a distinct state such as prepared for pull-labs jobs waiting to be picked up.
• Make pull-labs workers poll state=prepared.
• Keep available for nodes whose outputs/results are actually available.

[bhcopeland]

Makes sense.

Right now it goes state = available -> state = done.

Your idea:

state = running -> state = prepared -> state = done

Stops the available from being overloaded.

But what about:

state = running -> state = prepared -> state = running -> state = done

FWIW, state = done (also includes result pass | fail ...)

My runner doesn't filter on states, it filters on group. I could add state = running as a changeset my side (and document it in pull labs).

[nuclearcat]

Now it is: running -> available -> closing -> done
My proposal: prepared -> running -> available -> closing -> done
Stage prepared is optional, in case of existing schedulers since we create job definition and immediately consuming it, it will be as before: running -> available -> closing -> done

[nuclearcat]

prepared = job definition exists and is waiting for a pull-lab worker
running = lab has picked up the job and is executing it
available = output/result/artifacts are now available
closing = waiting for child nodes / hierarchy closure
done = terminal state, with result pass/fail/etc.
(we might skip also available/closing for pull-labs)

[bhcopeland]

Makes sense to me, I'll wire up the running state my side.