Add the first part of a multirack join service

[andrewjstone]

Add a new multirack join service tokio task that runs inside the
sled-agent on demand. It's run in the same manner as RSS, shares
the same context, and is mutually exclusive with RSS.

Two new bootstrap-agent-lockstep APIs were added in order to
trigger a multirack join and get its ongoing status. Unlike
RSS, a MultirackJoinRequest can be resent in the case of a typo,
misconfiguration, or runtime error. In many cases this will allow the
join process to correct itself without requiring a clean slate of the
rack. This is all managed via input and output watch channels and
therefore obviates the need for worrying about channel bounds.

As this PR started to get large, I have only implemented the first
part of the multirack join service which sets up the trust quorum.
This was enough to validate the starting of the service, the
watch channel plumbing, and the ability to correct mistakes in the
MultirackJoinRequest that triggers the behavior of the service.

In order to test this new API, I used
voxel with a 2 rack
configuration of 3 sleds each. On the second, non-RSS sled, I logged
into g3 and ran curl requests against the bootstrap-agent-lockstep
server. In order to help this testing, I created a one off tool to
generate example JSON output from a hardcoded configuration. I expect
this to be useful for the remainder of the implementation and then
removed once the implementation is complete.

Another large chunk of this code is just integrating with the existing
bootstrap agent and wicketd functionality. This should remain static for
the remainder of the implementation.

[andrewjstone]

This is part of #10637

[andrewjstone]

Some testing:

root@g3:~# curl -X POST  -H "Content-Type: application/json" --data-binary @bad-multirack_join_request.json http://[fdb0:a840:2500:7::1]:8080/multirack-join

root@g3:~# curl http://[fdb0:a840:2500:7::1]:8080/multirack-join | /opt/ooce/bin/jq "."
  % Total    % Received % Xferd  Average Speed  Time    Time    Time   Current
                                 Dload  Upload  Total   Spent   Left   Speed
100    734 100    734   0      0 434.1k      0                              0
{
  "TrustQuorumPreparing": {
    "config": {
      "rack_id": "9bb7e39a-7b40-4c7a-a137-53bdcd2902e5",
      "epoch": 1,
      "coordinator": {
        "part_number": "913-0000019",
        "serial_number": "2FAKE003"
      },
      "members": [
        [
          {
            "part_number": "913-0000019",
            "serial_number": "2FAKE003"
          },
          "650973a80d3169d626a9accbd4f1b7f01f15413d55f3917fdfbe646f780e1d74"
        ],
        [
          {
            "part_number": "913-0000019",
            "serial_number": "2FAKE004"
          },
          "68eb1f19bddde623cf77ad0343e0020c781d12916f5a4e7a2642e7f564ccab63"
        ],
        [
          {
            "part_number": "913-0000019",
            "serial_number": "2FAKE009"
          },
          "599f3ecc8b0dc299bef0dd0c7fed09492bcaa56254a7e4a283952b083ae40658"
        ]
      ],
      "threshold": 2,
      "encrypted_rack_secrets": null
    },
    "acked_prepares": [
      {
        "part_number": "913-0000019",
        "serial_number": "2FAKE003"
      },
      {
        "part_number": "913-0000019",
        "serial_number": "2FAKE004"
      }
    ]
  }
}

root@g3:~# curl -X POST  -H "Content-Type: application/json" --data-binary @multirack_join_request.json http://[fdb0:a840:2500:7::1]:8080/multirack-join
root@g3:~# curl http://[fdb0:a840:2500:7::1]:8080/multirack-join | /opt/ooce/bin/jq "."
  % Total    % Received % Xferd  Average Speed  Time    Time    Time   Current
                                 Dload  Upload  Total   Spent   Left   Speed
100     11 100     11   0      0   9337      0                              0
"Completed"

2FAKE009 is the bad/typo'd sled. We fix that by changing it to 2FAKE005 in the second POST operation. TQ commits quicker than we can inspect manually after that.

[karencfv]

Got curious about this PR and left some passer-by nit comments 😄

[karencfv]

It'd be nice for this match statement to be exhaustive

[andrewjstone]

In this case I actually don't want the match to be exhaustive. Exhaustive matches are useful when you want the compilation to fail if a new state gets added, because the behavior of the some code will need to change in that spot. However, in this case, all I want to know is if the code is already in the TrustQuorumPreparing state. If it's not then we know that we need to return true so an update gets sent on the watch channel, or we have to check if the state data changed if we are already in this state to determine whether to return true or false.

Since this code is not complete, and more unrelated states are going to be added, I don't want to have compilation fail and force me to update this code unnecessarily.

[karencfv]

Fair. I haven't tried it, but would something like this work?

self.output_tx.send_if_modified(|state| {
    let new_state = MultirackJoinServiceState::TrustQuorumPreparing(status);
    if *state == new_state {
        false
    } else {
        *state = new_state;
        true
    }
});

It's a bit easier to read imo

[andrewjstone]

Ooh. I like this a lot better. Thanks @karencfv !