When clients send a message to services, require services to send back an error message if they can't respond within a given time period

[Lance-Drane]

Currently, clients can send messages to services, but can't have any expectation of guaranteeing that a message can be handled in a certain timeframe. (An exception is the service-to-service request/response workflow, where a service can specify a timeout but will just silently drop the message from its external_requests loop if too much time has passed.)

What we want to happen is to allow for a Client (be it Client or Service) to specify a timeout in the message header, and require the Service to act on the message within the bounds of the timeout. (This could potentially be any number of "when the service starts processing the message" or "when the service completes processing the message".) The Service should attempt to send an error message back if it was unable to handle the message within the required timeframe (generating an error message is trivial and requires almost no resources other than an active thread to handle).

This to a limited extent may supersede #15 , depending on whether or not the (service-to-service | client) code wants to explicitly wait on the message within their function.

[Lance-Drane]

Some ideas from discussion:

• have a separate client timeout callback function which only gets called in the event of the client deciding a timeout has been reached
• make sure to set expiration parameters on brokers for messages in an attempt to avoid the Service handling messages the Client has already given up on (pretty sure I know how to do this with AMQP and MQTT 5.0 but don't think it's supported with MQTT 3?)
• send the timeout in the (userspace) message headers to the Service.

We must handle the usecase where the client gives up on the message during the Service processing the message.

potential way the class could look:

class TimeoutInformation:
    """key = message id, value = this class"""
    current_retries: int
    max_retries: int
    timeout_span: datetime.timedelta
    current_timeout: datetime.datetime

    # user provides constructor params but does not handle creation/modification of this class themselves
    def __init__(self, timeout_span: datetime.timedelta, max_retries: int = 3) -> None:
        self.max_retries = max_retries
        self.timeout_span = timeout_span