[FEATURE] schema_driven_xml: pass scalar payload siblings through the output envelope (pagination response tokens are dropped)

[jeffreyaven]

Problem

schema_driven_xml_v0.1.0 emits only {"<listProperty>": [rows]}. Any scalar siblings of the row carrier in the response payload - notably pagination tokens (<nextToken>, <NextMarker>, <Marker>, S3's <NextContinuationToken>) - are dropped.

The pagination machinery extracts the response token via JSONPath against the response rawBody (extractNextPageTokenFromBody -> basicResponse.ExtractElement, which reads r.rawBody). For ops using this transform, rawBody IS the transformed document (getOverridenResponse unmarshals the transformer's out-stream), so a config.pagination.responseToken of e.g. $.nextToken can never resolve, and traversal silently ends after page one.

Net effect: config.pagination works for untransformed responses (rest-json / aws-json AWS protocols) but cannot work for query / ec2 / rest-xml, which all route through this walker.

Proposed fix

In pkg/stream_transform/schema_driven_xml.go, pass scalar (string-typed) payload siblings through to the output envelope alongside the row list. Sketch (validated locally against the AWS provider; all existing walker tests pass):

func (t *schemaDrivenXMLTransformer) write(rows []interface{}, payload map[string]interface{}) error {
	out := map[string]interface{}{t.listProperty: rows}
	// Pass scalar payload siblings (pagination tokens - <nextToken>,
	// <NextMarker>, ... - plus inert metadata like <requestId>) through to
	// the output envelope: the pagination machinery extracts the response
	// token from the transformed document, so dropping them here would end
	// traversal after page one.
	for k, v := range payload {
		if k == t.listProperty {
			continue
		}
		if s, ok := v.(string); ok {
			out[k] = s
		}
	}
	b, err := json.Marshal(out)
	if err != nil {
		return err
	}
	_, writeErr := t.outStream.Write(b)
	return writeErr
}

with Transform() passing payload in, and nil at the two empty-result call sites (return t.write(make([]interface{}, 0), nil)).

Only top-level strings are passed through, so row projection, objectKey descent and column inference are unaffected (the schema_override still only declares the list property; extra raw keys are invisible to DDL).

Notes from local validation:

• Existing walker tests keep passing, but the test helper runWalker unmarshals the envelope as map[string][]map[string]interface{} and needs loosening to map[string]interface{}, since envelopes may now carry scalar siblings (e.g. requestId; also <Stacks/> empty self-closing lists surface as an empty-string sibling - harmless).
• Suggested regression test: an ec2-archetype list body containing <nextToken>tok-page-2</nextToken>, asserting env["nextToken"] == "tok-page-2" and that line_items is unchanged.

Downstream

The stackql AWS provider generator already emits config.pagination.{requestToken,responseToken} blocks for rest-json / aws-json list ops (~2,650 methods, verified live: DynamoDB ListTables with Limit=1 traverses all pages, re-injecting ExclusiveStartTableName per page). Emission for query / ec2 / rest-xml is gated off pending this fix - once released, the provider only needs the protocol gate lifted in _pagination_breadcrumbs (response token key = $. + the output member's XML wire name, which this passthrough preserves verbatim).