Adds Batching Proposal

[michaelstaib]

Within the Composite Schema WG we have discussed a new batching format that is in the first place meant for Subgraphs/Source Schema in a federated graph. This new variable batching allows the distributed GraphQL executor to execute the same operation with a set of variables.

graphql/composite-schemas-spec#25

[Shane32]

Does the distributed GraphQL executor currently support 'regular' batching requests?

[michaelstaib]

@Shane32 we will specify this as well ... we call it at the moment request batching ... the idea is that a request batch can also consist of variable batches.

[Shane32]

So it would support variable batching within request batching then?

[michaelstaib]

Yes, this is the current discussion. There are a lot of constraints we will put in place for the first iteration of this, we have explored this also in combination with subscriptions and all. But for this initial appendix we are focusing on variable batching first as this will be the minimum requirement for the composite schema spec.

[Shane32]

Ok. I would suggest that we do not use the jsonl format.

1. If there is variable batching within request batching, there is no clear way to format such a response. Obviously wrapping a jsonl response in a JSON list produces unparsable JSON due to the missing commas:

[
{"data":{"hello":"world-1"}}
{"data":{"hello":"world-2"}}
,
{"data":{"name":"john-1"}}
{"data":{"name":"john-2"}}
]

Perhaps it could be specified that when request batching was layered on top of variable batching, then the lists are flattened. I'm not sure this is the best approach, but it's feasible.

2. If this scenario is not supported, implying that the request is request batching OR variable batching, then there is no reason to differentiate the response formats.

3. Parsing jsonl is not common in JSON environments, at least not the ones I'm familiar with. Attempting to parse such a response within .NET will throw an exception upon encountering the next line in the jsonl response. Similarly the JSON.parse method of javascript does not support jsonl and will throw. In any of these environments, one would have to code another layer to separate the responses before decoding the results, adding code complexity in addition to having a likely-slower implementation. It seems much cleaner and easier to parse the JSON and then iterate through the response list.

Let's use the response format that is commonplace now and supported by various servers and clients alike.

Perhaps as a separate appendix, the jsonl format is described as an optional response format for batching requests (request batching or variable batching). There it can state that if multiple batching approaches are used, the lists are flattened. But I just don't see the benefit of adding another response format.

[michaelstaib]

There actually is we have specified that there is a requestIndex and a variableIndex in the response structure ... we do need these also as there is @defer and @stream involved. Every execution could yield to a stream of responses. We have this also already implemented with reference implementations ... I will get through the spec text this week and then you will see how this plays out.

[michaelstaib]

BTW ... we will introduce requestIndex and variableIndex also since the server should be able reorder responses and not make the consumer wait just in order to return completed results in order.

[michaelstaib]

@Shane32 I have put a bit more about the response in.

[michaelstaib]

After discussion in the composite schema working group we will have a single appendix about batching as this would allow for better example that show also both request batching and variable batching in combination.

[michaelstaib]

@benjie shall we do also some example in here with defer and stream ... I asked that because defer and stream is not yet handled in the spec.

[benjie]

For now: no; but would be good to have the text handy anyway.

[benjie]

We should remove some of the repetition of the core spec in this appendix, try and see it more as "additions" (extra fields, extra rules, changes in behavior) rather than a re-implementation.

[benjie]

There's a few issues with this:

+jsonl isn't allowed; see: https://www.rfc-editor.org/rfc/rfc6838.html#section-4.2.8

The primary guideline for whether a structured type name suffix is
registrable is that it be described by a readily available
description, preferably within a document published by an established
standards-related organization, and for which there's a reference
that can be used in a Normative References section of an RFC.

application/graphql is incorrect (that's normally used for GraphQL documents, not GraphQL responses) - I think you meant application/graphql-response?

This isn't a response but instead a number of responses.

Suggestion: just use application/jsonl as mentioned by https://jsonlines.org/ - this is sufficient to solve the issues that application/json has w.r.t. status codes/etc.

[benjie]

I don't think we should use the word "stream" here. I think "list" is fine; I think the default media type would likely be e.g. application/graphql-response-list+json (a JSON array of GraphQL responses) but application/jsonl would allow for easier early parsing of responses when the client is stream-capable.

[benjie]

The definition of the application/graphql-response type should be modified such that other keys are allowed iff they are specified as part of a specification or appendix hosted under the graphql.org domain; then this spec can "extend" that type with these keys. May want to go a slightly different direction on this in future.

[benjie]

The combination of MUST and optional is confusing. I would change this to something more along the lines of "in addition to the regular fields expected on a GraphQL response, each entry in a variable batching response MUST include the variableIndex field..."

[benjie]

3. Parsing jsonl is not common in JSON environments, at least not the ones I'm familiar with. Attempting to parse such a response within .NET will throw an exception upon encountering the next line in the jsonl response. Similarly the JSON.parse method of javascript does not support jsonl and will throw. In any of these environments, one would have to code another layer to separate the responses before decoding the results, adding code complexity in addition to having a likely-slower implementation. It seems much cleaner and easier to parse the JSON and then iterate through the response list.

It's not really fair to expect a JSON parser to parse JSONL... it's a different format. However; to make JSONL parseable as JSON is very straightforward; here's a JSONL parser for JS:

const parseJSONL = (jsonl) => JSON.parse(`[${jsonl.trim().replaceAll('\n', ',')}]`);

I'm not sure that you can really say that this significantly "increases the code complexity"? I would expect .NET to also be capable of:

• trimming whitespace (including newlines) from both ends of a string
• replacing all instances of a newline in the string with a comma
• prepending an [ and appending a ]
• parsing the result as JSON

JSONL is an incredibly straightforward format. Most importantly for this use case, it allows you to process the values as they stream down because you just scan through it looking for the next \n and when you hit it everything up until that point is parseable as JSON - you don't need a streaming parser as you would if you were to use JSON as the data type.

All that said, I think we should RECOMMEND that clients and servers implement JSONL, but we should REQUIRE that servers support plain JSON arrays. Which I think is already the case in this text (since SHOULD is equivalent to RECOMMEND according to RFC2119).

[Shane32]

here's a JSONL parser for JS:

const parseJSONL = (jsonl) => JSON.parse(`[${jsonl.trim().replaceAll('\n', ',')}]`);

Fair enough; I didn't think of that.

I'm not sure that you can really say that this significantly "increases the code complexity"? I would expect .NET to also be capable of:

True, .NET could perform string manipulation also fairly easily. However, the typical .NET JSON parser performs on a UTF8 byte stream so that there is no additional memory allocations beyond the minimum required to deserialize the data. Converting the incoming data to a string, performing a number of string manipulations on it, and then parsing it would considerably slow down the JSON engine. To maintain speed on par with the default implementation, you'd have to write a specialized wrapper that parsed the characters as they were being read, looking for \n and either (a) reading each JSON line as it is streamed, or (b) creating a virtual list of 'sub-streams' for further parsing. Or perhaps a streaming wrapper that changed the characters similar to the JS conversion above so it could be read as JSON (but that would be even more complex).

JSONL is an incredibly straightforward format. Most importantly for this use case, it allows you to process the values as they stream down because you just scan through it looking for the next \n and when you hit it everything up until that point is parseable as JSON - you don't need a streaming parser as you would if you were to use JSON as the data type.

Probably true. I may have another viewpoint if I was more familiar with the use case (which I am not).

[benjie]

Probably true. I may have another viewpoint if I was more familiar with the use case (which I am not).

In general batching is used when the client has a lot of queries to execute all at the same time (e.g. as the result of rendering a React tree). In traditional batching, the server receives this as a list, executes them in parallel, and then returns the resulting array - the result being that none of the components can render until the slowest of all the queries has finished executing. By allowing a) the server to return the results in order of execution completion (rather than in request order) and b) the client to determine easily when a result is ready (e.g. by scanning the response for a \n character) the client can choose to read and render results from faster queries whilst slower queries are still executing, and this can be achieved without needing to use an alternative protocol such as SSE or websockets. If the client can't be bothered to do this, then nothing changes and they just read and process the final JSONL result in much the same way they would with JSON.

[benjie]

Or perhaps a streaming wrapper that changed the characters similar to the JS conversion above so it could be read as JSON (but that would be even more complex).

Would it? You can initialize your binary UTF8 data buffer with 0x5b ([) (rather than being empty as you normally would), read the JSONL from the network and write into the buffer, then finally write a closing 0x5d (]) to the buffer. Once complete, just before you would parse the JSON, first check if the penultimate byte is 0x0a (\n) and if so overwrite it with 0x20 ( ) to remove the trailing newline (if any); then scan over this entire buffer and overwrite each 0x0a (\n) with a 0x2c (,). That's it - there's no resizing the buffer, it's safe to perform against UTF8 (even though it's a multi-byte format) because we're in the ASCII plane, it's just a simple prepend, append, byte-wise find and replace, and then using your regular JSON parser. The performance impact should be minimal; but you could speed it up with SIMD if you really wanted to.

[Shane32]

Probably true. I may have another viewpoint if I was more familiar with the use case (which I am not).

In general batching is used when the client has a lot of queries to execute all at the same time (e.g. as the result of rendering a React tree). In traditional batching, the server receives this as a list, executes them in parallel, and then returns the resulting array - the result being that none of the components can render until the slowest of all the queries has finished executing. By allowing a) the server to return the results in order of execution completion (rather than in request order) and b) the client to determine easily when a result is ready (e.g. by scanning the response for a \n character) the client can choose to read and render results from faster queries whilst slower queries are still executing, and this can be achieved without needing to use an alternative protocol such as SSE or websockets. If the client can't be bothered to do this, then nothing changes and they just read and process the final JSONL result in much the same way they would with JSON.

I see. Why does this PR focus on allowing any return order for variable batching vs regular batching then? If anything, variable batching would be much more likely to have a consistent execution time across each request and so having the responses return in any order is much less useful. Maybe we should focus on a flag of some sort to allow this behavior for traditional batching requests, and then take the solution to this PR for consistency.

[Shane32]

Or perhaps a streaming wrapper that changed the characters similar to the JS conversion above so it could be read as JSON (but that would be even more complex).

Would it? You can initialize your binary UTF8 data buffer with 0x5b ([) (rather than being empty as you normally would), read the JSONL from the network and write into the buffer, then finally write a closing 0x5d (]) to the buffer. Once complete, just before you would parse the JSON, first check if the penultimate byte is 0x0a (\n) and if so overwrite it with 0x20 ( ) to remove the trailing newline (if any); then scan over this entire buffer and overwrite each 0x0a (\n) with a 0x2c (,). That's it - there's no resizing the buffer, it's safe to perform against UTF8 (even though it's a multi-byte format) because we're in the ASCII plane, it's just a simple prepend, append, byte-wise find and replace, and then using your regular JSON parser. The performance impact should be minimal; but you could speed it up with SIMD if you really wanted to.

I meant a streaming wrapper. One that does not read the entire result into memory at any time. Yes, I believe it would be more complex than the other two options I presented. For instance, if you read a blob of data and the last character is a LF, you can't pass it to the JSON parser but instead must cache it until the next read so you know if this is the EOF and it must be overwritten with ] or if there is more data and it must be ,. It just would get sorta complex. And it would require you to read the JSON in a streaming fashion too, which would be even more complex, if you wanted to read the results as they were streamed in.

I'm okay with the complexity if it's optional and has benefits, mind you. I would agree JSONL seems to be a better format for reading streaming responses, either in .NET or JS, where you want to read individual GraphQL responses as they are transmitted.

[Shane32]

Maybe we should focus on a flag of some sort to allow this behavior for traditional batching requests, and then take the solution to this PR for consistency.

For example, maybe anytime a JSONL format is requested for a batching request (traditional or variable batching), a batchIndex is added and results are returned in any order.

However for JSON format, no extra property is returned and the results are returned sequentially.

This would simplify response processing for users that do not spend the time to write streaming response processing code (in either JS or .NET).

[benjie]

This should probably use HTTP 207 (Multi-status) as the status code?

[martinbonnin]

One nit: the name of the appendix file should be just Batching. LGTM otherwise.

[benjie]

The more I look at this, the more I feel it should be baked into the main spec. Variable batching at least... I could happily do without operation batching (in fact, I'd argue for us to avoid specifying it).

I think we might need to abstract the spec a little to make way for Trusted Docs and batching

Like:

flowchart TD
  HTTPReq --> GraphQLHTTPRequest
  GraphQLHTTPRequest --> GraphQLRequest
  GraphQLRequest --> Execute
  Execute --> GraphQLResponse
  GraphQLResponse --> GraphQLHTTPResponse
  GraphQLHTTPResponse --> HTTPResponse

Loading

Then we can do:

flowchart TD
  HTTPReq --> TrustedDocRequest["TrustedDocsRequest {id:'abc', variables: [{...}, ...]}"]
  TrustedDocRequest --> GraphQLHTTPRequest["GraphQLHTTPRequest {query:'...', variables: [{...}, ...]}"]

  GraphQLHTTPRequest --> GraphQLReq1["GraphQLRequest {query:'...', variables: {...}}"]
  GraphQLReq1 --> Execute1["Execute"]
  Execute1 --> GraphQLResponse1["GraphQLResponse"]
  GraphQLResponse1 --> GraphQLHTTPResponse

  GraphQLHTTPRequest --> GraphQLReq2["GraphQLRequest {query:'...', variables: {...}}"]
  GraphQLReq2 --> Execute2["Execute"]
  Execute2 --> GraphQLResponse2["GraphQLResponse"]
  GraphQLResponse2 --> GraphQLHTTPResponse

  GraphQLHTTPRequest --> GraphQLReq3["GraphQLRequest {query:'...', variables: {...}}"]
  GraphQLReq3 --> Execute3["Execute"]
  Execute3 --> GraphQLResponse3["GraphQLResponse"]
  GraphQLResponse3 --> GraphQLHTTPResponse

  GraphQLHTTPResponse --> HTTPResponse["HTTPResponse 207 Multi-Status"]

Loading

[benjie]

Item 3 implies GAPs can extend it. I don't think that's wise - GAPs should use extensions. Is this specifically so the composite schemas spec can extend it? What are you thinking about here?

[michaelstaib]

My thinking was exactly that there might be transport specific spec bound to something like the composite schema spec. But I do not have anything in mind yet, just thought this gives flexibility. But i maybe not needed.

I was not thinking about GAPs at the time.

[benjie]

Let's remove it and add it back when needed.

[benjie]

"field-level batching" is not well defined.

[benjie]

We need to encourage them to discard requests where the total number of requests to execute is over some threshold. But also, I think this is more part of execution behavior than request shape.

Suggested change

	If a request object's `{variables}` value is a list, the server MUST execute
	that request once per variables entry.
	If the request body is a list, each entry in that list MUST be processed as an
	independent _GraphQL-over-HTTP request_.

[michaelstaib]

If the server is allowed to discard requests then this must be specified in some way. Like it cannot simply discard requests in the middle without any notice. Otherwise we need another marker to pin which requests have to be reissued. I a server like executes 20 requests and then says limit reached a stop now that could be handled... although to be honest I would find it better if the server would simply reject it all and say i can only process 20 requests at a time so i reject your batch.

This ties btw back in with capabilities. In the gateway case the gateway could inspect the capabilities and send exactly the correct amount of requests in a batch.

[benjie]

Yes, I think all or none - it should decide up front if it's attempting them or not.

That said, mutations would be serialized and may hit timeouts, and the side effects might not be able to be undone, so... 😬 I guess just execute them all and the later ones produce errors...

... We really need to add some standard error codes to GraphQL errors.

[michaelstaib]

@benjie regarding request batching ... we need to spec it, like for Gateways the ability to pack two different requests to the same subgraph really saves on pressure and improves perf. In apollo federation this is solved with alias batching as they can pass in untyped data into the representation they can basically do the same as combining variable batching and request batching combined.

The pair does work with trusted documents. I would agree to moving variable batching into the main spec and leaving request batching as an appendix. I would not drop it.

[benjie]

I think there's enough evidence out there in the ecosystem that GraphQL needs variable batching independent of Federation that I think it's not unreasonable to include it as a recommended feature. We can indeed use service capabilities to indicate not only which types of batching are supported but also what their limits should be (and whether or not they can be combined).