Skip to content

Support proto2 syntax and custom proto file headers - #1305

Open
jwils wants to merge 2 commits into
joshuaw/proto-timestampfrom
joshuaw/proto-syntax-headers
Open

Support proto2 syntax and custom proto file headers#1305
jwils wants to merge 2 commits into
joshuaw/proto-timestampfrom
joshuaw/proto-syntax-headers

Conversation

@jwils

@jwils jwils commented Jul 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Why

Some consumers need the generated messages to reference proto2 types — protoc forbids a proto3 message from referencing a proto2 enum — and language-specific file options (like java_package) shouldn't require baking any convention into the gem.

What

proto_schema_artifacts gains two options:

  • syntax: :proto2 emits a proto2 file instead of the default proto3, labeling every field optional or repeated
  • headers: injects file-level lines (such as option declarations) verbatim as a contiguous section after the package declaration

Risk Assessment

Low — only affects the unreleased elasticgraph-proto_ingestion extension; both options are opt-in.

References

Update — 2026-07-10

This PR is stacked directly on #1306 (→ #1304 → #1080).

@jwils jwils mentioned this pull request Jul 8, 2026
Open
@jwils
jwils force-pushed the joshuaw/proto-field-numbers branch from 67b8198 to f62a859 Compare July 8, 2026 23:40
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch 2 times, most recently from 9b72525 to 3396a37 Compare July 9, 2026 16:00
@jwils
jwils changed the base branch from joshuaw/proto-field-numbers to joshuaw/proto-timestamp July 9, 2026 16:00
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 3396a37 to 98fac03 Compare July 9, 2026 17:29
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 5a2e67d to 1537869 Compare July 9, 2026 17:29
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 98fac03 to b3029f8 Compare July 9, 2026 18:13
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch 2 times, most recently from 721fe97 to d516c31 Compare July 10, 2026 15:44
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from b3029f8 to d837b3a Compare July 10, 2026 15:44
@jwils jwils mentioned this pull request Jul 10, 2026
Merged
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from d837b3a to 6f1b1b1 Compare July 10, 2026 15:52
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch 2 times, most recently from 9e7369b to 9e2b508 Compare July 11, 2026 13:36
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 6f1b1b1 to d4471ca Compare July 11, 2026 13:38
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 9e2b508 to 06e1531 Compare July 11, 2026 13:54
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from d4471ca to 5eb990f Compare July 11, 2026 13:54
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 06e1531 to ee20f41 Compare July 11, 2026 14:00
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch 2 times, most recently from d40ff2e to 40be848 Compare July 11, 2026 14:14
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 508c016 to 4a6aa37 Compare July 15, 2026 23:52
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 7b9a2b7 to c8b5e5e Compare July 15, 2026 23:52
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from c8b5e5e to 020eb17 Compare July 16, 2026 14:21
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch 2 times, most recently from 1aa384b to b94fd0e Compare July 17, 2026 15:05
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 020eb17 to 9a7765f Compare July 17, 2026 15:05
jwils added a commit that referenced this pull request Jul 20, 2026
@jwils
Generate a protobuf schema artifact from ElasticGraph schemas (#1080)
48add3e 
## Why
- continue the pluggable ingestion serializer proposal after pulling
JSON Schema into its own gem
- revive the earlier protobuf prototype from #1056 on top of the new
serializer extension points

## What
- fill in the `elasticgraph-proto_ingestion` extension gem with core
generation: `schema_artifacts:dump` emits a `proto3` `schema.proto`
covering the schema's indexed types
- map built-in ElasticGraph scalars to proto types, with `t.protobuf
type:` for custom scalars (resolved via
`type_ref.with_reverted_override` so built-ins renamed with
`type_name_overrides` keep working)
- generate messages for object/interface/union types and enums (with a
zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and
wrapping lists of lists so the output stays valid
- keep `schema.proto` on public GraphQL field names; validate proto
package names
- hold extension state on a `ProtoIngestionState` container behind a
single `proto_ingestion_state` reader (matching #1281)

Field and enum value numbers are assigned sequentially in definition
order in this PR; the stacked follow-up adds the
`proto_field_numbers.yaml` sidecar that keeps them wire-stable across
schema evolution.

## Stacked follow-ups
1. this PR — core `schema.proto` generation
2. wire-stable field/enum value numbers via a `proto_field_numbers.yaml`
sidecar
3. `syntax: :proto2` support and custom file-level `headers:`
4. #1286 — enum value sourcing from existing proto enums + external
proto type references

## Verification
- `script/run_gem_specs elasticgraph-proto_ingestion` (100% line +
branch coverage at this commit)
- `script/type_check`, `script/lint`, `script/spellcheck`
- `script/quick_build` green at the stack head (whose tree is identical
to the previously reviewed single-PR revision)

## References
- #1059
- #1056
- #1079

## Update — 2026-07-10
- The current stack is #1080 → #1304 → #1306 → #1305 → #1286.
- Proto extension state now uses a mutable Struct, and keyword
package-name segments are validated without being rewritten.
- Lists of lists now raise an actionable schema error instead of
generating wrapper messages; this supersedes the earlier wrapping note
above.
---

## Update — 2026-07-15
- Interface and union messages now wrap concrete subtype messages in a
`oneof`, matching the JSON Schema `oneOf` representation.
- Concrete subtype messages omit the redundant `__typename`
discriminator.
- Proto type rendering is now stateless: the generator selects the
reachable type graph up front, then each extended type renders itself
without mutating shared traversal state.
- No-block extension coverage now completes the definitions so the
fixture remains valid under CI GraphQL-schema validation.

---

## Update — 2026-07-19
- Replaced keyword suffixing with fully qualified local message and enum
references, preserving source type and field names while disambiguating
contextual protobuf words and built-in scalar names.
- Removed the now-unnecessary keyword collision tracking and verified a
generated schema containing contextual names with `protoc` 35.1.
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from b94fd0e to d676254 Compare July 20, 2026 02:16
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from d676254 to 1a80948 Compare July 31, 2026 03:39
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 9a7765f to bae3345 Compare July 31, 2026 03:41
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 1a80948 to e08778f Compare July 31, 2026 03:45
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from bae3345 to 0f21d16 Compare July 31, 2026 03:45
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from e08778f to b4fe4d2 Compare July 31, 2026 13:31
jwils added 2 commits July 31, 2026 08:31
@jwils
Support proto2 syntax and custom proto file headers
b50855f 
`proto_schema_artifacts` gains two options:

- `syntax: :proto2` emits a proto2 file instead of the default proto3
  (labeling every field `optional` or `repeated`). This is useful when the
  generated messages need to reference proto2 types — `protoc` forbids a
  proto3 message from referencing a proto2 enum.
- `headers:` injects file-level lines (such as `option` declarations)
  verbatim as a contiguous section after the `package` declaration, so
  language-specific options can be set without baking any particular
  convention into the gem.
@jwils
Test mixed protobuf header validation
abf4228
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 0f21d16 to abf4228 Compare July 31, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BrianSigafoos-SQ BrianSigafoos-SQ Awaiting requested review from BrianSigafoos-SQ BrianSigafoos-SQ is a code owner
@bsorbo bsorbo Awaiting requested review from bsorbo bsorbo is a code owner
@jwondrusch jwondrusch Awaiting requested review from jwondrusch jwondrusch is a code owner
@myronmarston myronmarston Awaiting requested review from myronmarston myronmarston is a code owner
@marcdaniels-toast marcdaniels-toast Awaiting requested review from marcdaniels-toast marcdaniels-toast is a code owner
@ellisandrews-toast ellisandrews-toast Awaiting requested review from ellisandrews-toast ellisandrews-toast is a code owner
@rossroberts-toast rossroberts-toast Awaiting requested review from rossroberts-toast rossroberts-toast is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jwils