Better error handling for values that are serialized outside of user code

[rwb27]

In LabThings, user code is often responsible for returning a value (e.g. action functions, or property getters) and that value is then serialized later by LabThings and/or FastAPI. This leads to the unfortunate situation that exceptions can occur outside of user code (and often outside of LabThings code) resulting in long, unsightly, and completely useless stack traces with often-unhelpful errors.

The problem

Values are always serialised in LabThings using a pydantic.BaseModel - either a BaseModel subclass supplied by the Thing code, or a LabThingsRootModelWrapper dynamically generated by LabThings. For both actions and getters, the procedure is:

1. A model is created, either by the user code or by LabThings wrapping a type hint.
2. The user-supplied code is run, and its return value is obtained.
3. A model is instantiated using the return value.
4. The model is serialized to JSON.

At the moment, errors often occur in 3 and 4, within FastAPI. Because the exception occurs after the code that generated the problematic value, the stack trace doesn't include which bit of user-supplied code is relevant, which seriously hampers debugging.

Proposed solution

It probably makes sense to use slightly different solutions for (3) and (4).

Validating models from user-supplied code (3)

I propose we explicitly create instances of the models we want to return. This moves (3) into LabThings code, so we can catch the error and handle it ourselves, for example adding an indication of where the value came from.

I don't believe this changes the behaviour - anything that worked before should still work, and anything that failed before should still fail. Currently, we'd need to do this for:

• action return values
• functional property getters
• reading the value of data properties.

If validate_properties_on_set becomes the only behaviour, we'll be able to skip validation when reading data properties, because they can hold a validated model instance instead of a plain value, to avoid revalidating when they're read.

Catching serialisation errors (4)

To fix (4), we must handle errors that occur when models are serialized. This could be done in a few different ways:

• Wrap all values that might fail in a custom RootModel that adds context to the serialization error. This provides error handling in the right place, but results in a complicated model, a slightly opaque structure, and less-than-ideal error messages that get re-wrapped by pydantic. This is done in an earlier commit in this branch.
• We could serialise the model in LabThings and return a Response directly, giving us full control over error handling. The major drawback here is that the API documentation might become less clear. We'd also have to repeat the error handling code in each endpoint that needs it.
• We could add an error handler for PydanticSerializationError to ThingServer.app and pass the extra context to it using a context variable. I'm not wild on adding another context variable, but I think it would be reasonable to use one here.

I have tried both of the first two - I think the first is pretty ugly. The second has merit, but I'd want to make sure the openapi documentation isn't broken. The third is the smallest change to the codebase, but requires context variables. I'll see how it looks.