wit/bindgen: support JSON encoding of WIT types

[lxfontes]

This started in #225

Goal: JSON Serialize / Deserialize WIT Records based on WIT field names.

Progress

• record - wit/bindgen: Add JSON tag to Record structs #265
• list - cm: Implement json Marshal/Unmarshal for List type #266
• enum
• tuple
• variant
• result
• option
• resource handle types (decide behavior)
• error-context
• stream
• future

Approach

Centralize JSON serialization in the cm package, keeping codegen changes to a minimum.

Given a complex record type:

  record response {
      headers: list<tuple<string, list<string>>>,
      http-code: u16,
      body: response-body
  }
  variant response-body {
    ok(list<list-element>),
    err(function-error),
    platform-err(platform-error)
  }
  record list-element {
      optional-int: option<u64>,
      optional-bool: option<bool>,
  }
  record function-error {
      error: string
  }
  record platform-error {
    code: string,
    message: string
  }

and filling it up:

	hdrVals := cm.ToList([]string{"value1", "value2"})
	hdrTuple := cm.Tuple[string, cm.List[string]]{
		F0: "header-name",
		F1: hdrVals,
	}
	headers := cm.ToList([]cm.Tuple[string, cm.List[string]]{hdrTuple})
	v := somefunctioninterface.Response{
		Headers:  headers,
		HTTPCode: 200,
		Body:     somefunctioninterface.ResponseBodyErr(somefunctioninterface.FunctionError{Error: "failed"}),
	}

We should serialize it to JSON as:

{"headers":[["header-name",["value1","value2"]]],"http-code":200,"body":{"err":{"error":"failed"}}}

For comparison, this is what is produced today:

{"Headers":{},"Body":{"RequiredParam":"required","OptionalParam":{}}}

Type encoding

Whenever possible, reuse standard mappings. string -> string, u32 -> uint32, etc.

Tuple handling

Tuples are encoded as json arrays with explicit nulls.

Tuple[string,int] -> [ "some string", 42 ]
Tuple3[string, *string, int] -> [ "some string", null, 42 ]

Variant handling

Variants are encoded as json dictionaries, so they can carry the variant data.

 record somerecord {
   myvariant: somevar,
 }
  variant somevar {
    ok,
    err(string),
  }

// JSON
{ "myvariant": { "ok": true }}
{ "myvariant": { "error": "error value" }}

Option handling

Options rely on explicit null for the "None" case.

{ "myoptional": null } // cm.None
{ "myoptional": "this" } // cm.Option[string]

Questions

Zero Value Variants

Today they end up with tag = 0, this impacts de-serialization of variants. We need the ability to distinct between:

• var v Variant
• v := NewVariantWithZeroTagValue()
• v := Some(NewVariantWithZeroTagValue)

atm only the Some() path de-serializes correctly ( pointers ).

[ydnar]

Can we move the design and task list for this to an issue?

Then we can have discrete PRs that implement JSON support for different types?

[ydnar]

Proposal for option, result, and variant JSON:

Option

The current proposal specifies null for option<T> when it is the none case. If option values are nested, this creates an ambiguity as to which option is some versus none, e.g. option<option<T>>.

An alternate proposal would be to specify {"none": ...} instead of null to disambiguate nested options.

For an option<string> with the some case:

{"some":"foo"}

For an option<string> with the none case:

{"none":null}

For an option<option<string>> where the second option is the none case:

{"some":{"none":null}}

Result

result<T, E> values would serialize similarly to option as a simplified variant.

For result<string> in the OK case (no error type):

{"ok": "some OK value"}

For result<string> in the error case:

{"error": null}

[ydnar]

For resource, stream, future, and error-context types, I think JSON serialization should be an error, similar to how chan and func types cannot be marshaled into JSON in Go today.

[ydnar]

Inspired by #301, I think option values should be represented as:

• None: {}
• Some: {"some": 123}

[brooksmtownsend]

@ydnar would love to discuss that a bit, I think that representing option values in that way gives us the best ability to understand when it's a Some(t) value, my concern would be that optional fields from other tools (e.g. OpenAPI / JSON schema) aren't encoded in this way.

Generally what I would expect an optional field to be in JSON is either omitting the field entirely or having the field be set to null, and when present it would just be the value itself. Does this make sense / am I missing something that would bite us during unmarshaling?

What I have in my PR corresponds to the following behavior:

• Marshal Some(T) -> {"field": T}
• Marshal None(T) -> {"field": null}
• Unmarshal {"field": null} -> None
• Unmarshal {} -> None
• Unmarshal {"field": T} -> Some(T)

[ydnar]

Is it a goal to support symmetrical round-trip of WIT values to JSON? If so, then this would require unambiguous encoding of option<T> values.

Given a WIT type option<option<T>, the following have different semantic, and in-memory value:

1. None(T)
2. Some(None(T)

How should these be serialized?

1. null
2. null

Or:

1. {}
2. {"some":{}}

Using null to represent the none case loses information, and cannot be symmetrically encoded and decoded.