feat: add the Kafka consumer for Studio's indexing requests - #692
Open
MoonBoi9001 wants to merge 4 commits into
Open
feat: add the Kafka consumer for Studio's indexing requests#692MoonBoi9001 wants to merge 4 commits into
MoonBoi9001 wants to merge 4 commits into
Conversation
Studio announces how many indexers a subgraph deployment wants over Redpanda; the dipper will consume those messages. This vendors Studio's protobuf schema, generates Rust bindings for it, and adds one field, the network the subgraph indexes, which the consumer needs and Studio has.
The dipper could only send Kafka events, not read the indexing requests Studio publishes. The new consumer finds the topic's partitions from broker metadata, refuses to start if the topic is missing, and leaves offset tracking to the caller; SASL/TLS setup is now shared.
The Kafka client library retries an unreachable broker with no deadline, so connecting while Redpanda is down waited endlessly, dodging every retry and shutdown path around it. Connects now time out (60s default), reject an empty broker list, and metadata can be re-read on demand.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the receiving half of the Studio-to-dipper bridge: a Kafka consumer for the subgraph indexing request events Subgraph Studio publishes over Redpanda, plus a vendored copy of Studio's protobuf schema with generated Rust bindings. Studio already produces these messages when a developer asks for their subgraph to be indexed; nothing on the dipper side could read them. A follow-up PR wires the consumer into the service.
The consumer discovers the topic's partitions from broker metadata and refuses to start when the configured topic does not exist; the topic name deliberately has no default, after a Studio consumer crash-looped in staging because its configured topic did not match the producer's. Connects are bounded by a configurable timeout, because the underlying client retries an unreachable broker with no deadline and would otherwise never return. rskafka has no consumer groups, so offset tracking belongs to the caller (the follow-up persists offsets in the dipper's database).
One deliberate divergence in the vendored schema: the propose payload carries an extra field,
indexed_network_caip2id, naming the network the subgraph indexes. The dipper needs it to key indexing requests, but the field still needs to land on Studio's side; the proto file flags the divergence.