Add Variant, Decimal, and Timestamp CEL functions - #524
Open
Robert Yokota (rayokota) wants to merge 36 commits into
Open
Add Variant, Decimal, and Timestamp CEL functions#524Robert Yokota (rayokota) wants to merge 36 commits into
Robert Yokota (rayokota) wants to merge 36 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR expands the Schema Registry JS client’s CEL and serde capabilities by adding first-class Decimal/Timestamp/Variant CEL support and introducing schema-declared inline validation rules across Avro, JSON Schema, and Protobuf, with extensive new parity-oriented tests.
Changes:
- Add inline validation rules (schema-level
confluent:rules/ protobuf meta rules) with configurable execution timing and aggregated error reporting. - Add CEL built-ins and runtime wiring for Decimal, Timestamp, and Variant (including Avro logical type support and Variant path navigation).
- Strengthen caching keys for parsed schemas to include references (avoiding cross-subject conflation) and add regression/parity test coverage.
Reviewed changes
Copilot reviewed 33 out of 34 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| schemaregistry/test/serde/validation-serdes.spec.ts | Serializer-level tests for inline validation wiring across Avro/JSON/Protobuf, plus JSON schema caching and rule firing counts. |
| schemaregistry/test/serde/validate-message.spec.ts | Walker-level tests verifying per-format validation traversal and field-path reporting; includes now and dynamic message checks. |
| schemaregistry/test/serde/test/validation_widget_pb.ts | Generated protobuf test fixture carrying inline validation rules in options/meta. |
| schemaregistry/test/serde/protobuf.spec.ts | Adds CEL decimal/timestamp coverage and nested-type CEL field transform coverage for protobuf serde. |
| schemaregistry/test/serde/protobuf-walk-parity.spec.ts | Ensures parity between protobuf validation walk and transform walk, plus schema-view edge cases. |
| schemaregistry/test/serde/json.spec.ts | Adds coverage for JSON Schema enum typing (enum/const without explicit type) in rule walking. |
| schemaregistry/test/serde/avro.spec.ts | Adds CEL decimal/timestamp behavior tests, including fail-closed semantics and Avro logical-type handling. |
| schemaregistry/test/rules/cel/cel-executor.spec.ts | Regression test ensuring CEL program caching is not incorrectly reused across registries. |
| schemaregistry/test/confluent/types/decimal-utils.spec.ts | Tests for exact decimal conversions beyond decimal.js default precision limits. |
| schemaregistry/serde/serde.ts | Introduces inline validation framework types/helpers, schema cache key helper, and serializer config options for validation. |
| schemaregistry/serde/json.ts | Adds inline JSON validation walker and switches caches to use schema+references keying; avoids schema mutation on subtype checks. |
| schemaregistry/serde/avro.ts | Adds inline Avro validation walker, schema+references cache keying, and an Avro variant logical type integration. |
| schemaregistry/rules/cel/variant-path.ts | Implements JSONPath-like subset parser/walker for Variant navigation. |
| schemaregistry/rules/cel/variant-funcs.ts | Adds CEL variant(...) constructor and variants.* accessors + JSON parsing/serialization. |
| schemaregistry/rules/cel/timestamp-funcs.ts | Adds CEL timestamp constructor overloads with spec-aligned epoch semantics and range checks. |
| schemaregistry/rules/cel/is-funcs.ts | Adds CEL string-format validators (isEmail, isHostname, etc.). |
| schemaregistry/rules/cel/decimal-funcs.ts | Adds CEL Decimal constructor and decimals.* operators with BigDecimal-like scale/precision semantics and numeric equality. |
| schemaregistry/rules/cel/cel-validator.ts | Adds CEL-backed inline validation rule executor with protobuf-registry-aware evaluation. |
| schemaregistry/rules/cel/cel-field-executor.ts | Improves field-value presentation to CEL using protobuf field descriptors and Avro logical-type wrapping/unwrapping. |
| schemaregistry/rules/cel/cel-executor.ts | Extends CEL executor with Decimal/Timestamp/Variant funcs, registry-aware plan caching, fail-closed behavior, and Avro logical-type wrapping. |
| schemaregistry/package.json | Bumps @bufbuild/cel, bumps avsc, and adds decimal.js. |
| schemaregistry/index.ts | Exports Variant types/utilities and clarifies Rule type exports to avoid name collisions. |
| schemaregistry/confluent/types/variant_pb.ts | Generated protobuf for confluent.type.Variant. |
| schemaregistry/confluent/types/decimal-utils.ts | Adds decimal conversion utilities between decimal.js and confluent.type.Decimal. |
| schemaregistry/confluent/meta_pb.ts | Updates meta proto generation to include inline rules and adds confluent.Rule message. |
| proto/test/schemaregistry/serde/validation_widget.proto | Test proto defining inline validation rule fixtures. |
| proto/confluent/types/variant.proto | Defines the confluent.type.Variant proto. |
| proto/confluent/meta.proto | Extends Meta with repeated Rule and defines the Rule message. |
| package-lock.json | Lockfile updates for dependency bumps/additions. |
| CHANGELOG.md | Notes inline validation rules enhancement. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+33
to
+35
| export function schemaCacheKey(info: SchemaInfo): string { | ||
| return stringify(minimize(info)) | ||
| } |
Comment on lines
+1467
to
+1477
| let detail: string | undefined | ||
| if (this.message) { | ||
| detail = this.message | ||
| } else if (this.rule?.doc) { | ||
| detail = this.rule.doc | ||
| } else if (this.rule?.sql) { | ||
| detail = this.rule.sql | ||
| } else { | ||
| detail = this.rule?.expr | ||
| } | ||
| let result = `${path}: ${name}: ${detail}` |
Comment on lines
+142
to
+154
| function readIndex(cur: Cursor, path: string): number { | ||
| if (cur.hasMore() && cur.peek() === "-") { | ||
| throw new Error("negative indices are not supported in variant path: " + path); | ||
| } | ||
| const start = cur.pos; | ||
| while (cur.hasMore() && cur.peek() >= "0" && cur.peek() <= "9") { | ||
| cur.next(); | ||
| } | ||
| if (cur.pos === start) { | ||
| throw new Error("expected integer index in variant path: " + path); | ||
| } | ||
| return Number(cur.src.slice(start, cur.pos)); | ||
| } |
Comment on lines
41
to
46
| async transform(ctx: RuleContext, msg: any): Promise<any> { | ||
| const args = { | ||
| message: msg | ||
| message: this.wrapForCel(ctx, msg), | ||
| } | ||
| return await this.execute(ctx, msg, args) | ||
| } |
| 1. Support generating a JSON Schema title from a JSON payload (#505) | ||
| 2. Add support for saving Azure key version with DEK (#507) | ||
| 3. Pass context when clients make KEK calls to DEK Registry (#508) | ||
| 4. Add support for inline validation rules (#522) |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please prefix all TypeScript pull-requests with
[Typescript]What
Checklist
References
JIRA:
Test & Review
Open questions / Follow-ups