Skip to content

dataconnect(chore): realtime refactor in preparation for deduplicating cached and server results#8236

Draft
dconeybe wants to merge 14 commits into
mainfrom
dconeybe/dataconnect/RealtimeCacheUpdate3
Draft

dataconnect(chore): realtime refactor in preparation for deduplicating cached and server results#8236
dconeybe wants to merge 14 commits into
mainfrom
dconeybe/dataconnect/RealtimeCacheUpdate3

Conversation

@dconeybe
Copy link
Copy Markdown
Contributor

@dconeybe dconeybe commented May 29, 2026

This PR refactors the internal query source tracking architecture in data connect by introducing a sealed DataSource hierarchy (Server and Cache) and a TaggedReference utility class. These changes enable the cache and realtime layers to propagate SqliteSequenceNumber details, laying the groundwork for optimizing realtime updates and preventing duplicate events.

Highlights

  • Introduce Internal DataSource Sealed Hierarchy: Replaced the public-facing DataSource enum in internal components with a new internal sealed DataSource interface (supporting Server and Cache(SqliteSequenceNumber?)), carrying sequence number information down from the local database.
  • Refactor to generic TaggedReference Utility: Renamed and generalized the query-bound DataSourcePair utility into a generic-covariant TaggedReference<out Tag, out T> class with functional .map() operations to transform reference data without losing tag context.
  • Integrate Cache Sequence Numbers into Realtime Streams: Updated the gRPC pipeline and RealtimeQueryManager stream flows to cache incoming realtime frames and tag their corresponding stream responses with the resulting SqliteSequenceNumber.
  • Adopt Safe and Explicit Null Branching: Refactored database query decoding branches in DataConnectGrpcRPCs to use an explicit Kotlin when block for transparent handling of empty cache rows.
  • Enhance Test Coverage for Backward Compatibility: Updated the Kotest dataSource generator to produce nullable sequence numbers (modeling backward compatibility scenarios where the database lacks sequence numbers) and added complete property-based tests for TaggedReference.

Changelog

See affected files
  • DataConnectBidiConnectStream.kt: Removed redundant auto-generated component functions from ExecuteResponse.
  • DataConnectGrpcClient.kt: Updated OperationResult to return the new internal DataSource hierarchy, propagating gRPC-level cache sequence numbers.
  • DataConnectGrpcRPCs.kt: Refactored ExecuteQueryResult.FromCache to hold cache sequence numbers and updated return branching using an explicit when block.
  • DataSource.kt: Created a new internal DataSource sealed interface representing internal sources (Server or Cache) and a utility to convert to the public enum types.
  • QueryRefImpl.kt: Updated destructured components from TaggedReference and public DataSource enum mapping.
  • QuerySubscriptionImpl.kt: Refactored supplementary non-realtime update handlers and realtime stream updates to work with modern TaggedReference structures and safe error-checking flows.
  • LiveQuery.kt: Swapped DataSourcePair references to use TaggedReference<DataSource, T>.
  • QueryManager.kt: Swapped signatures from DataSourcePair to TaggedReference<DataSource, T>.
  • RealtimeQueryManager.kt: Refactored query subscription and stream caching pipes to track nullable SqliteSequenceNumber alongside responses via the new generic TaggedReference utility.
  • RegisteredDataDeserialzer.kt: Updated deserializer flow references and deserialization routines to construct TaggedReference<DataSource, T>.
  • DataConnectCacheDatabase.kt: Added Comparable interface to SqliteSequenceNumber value class to support ordering comparison of cache event numbers.
  • TaggedReference.kt (Renamed from DataSourcePair.kt): Refactored into a general-purpose, generic-covariant utility supporting an inline .map() transformation block.
  • DataConnectGrpcClientUnitTest.kt: Updated client mocks, tests, and verifications to adopt new DataSource structure with nullable sequence numbers.
  • MutationRefImplUnitTest.kt: Updated OperationResult assertions to reflect new DataSource hierarchy.
  • QueryRefImplUnitTest.kt: Refactored cache query test cases to pass TaggedReference with correct core to public DataSource mappings.
  • DataConnectCacheDatabaseUnitTest.kt: Refactored helper and mock updates to construct type-safe SqliteSequenceNumber instances.
  • arbs.kt: Added custom sqliteSequenceNumber and updated dataSource test generators to generate nullable sequence numbers for database backward compatibility checks.
  • TaggedReferenceUnitTest.kt: Created complete property-based unit tests verifying constructor behaviors, standard equals, hashCode, toString contracts, and inline map operations.

@gemini-code-assist
Copy link
Copy Markdown
Contributor

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

@dconeybe
Copy link
Copy Markdown
Contributor Author

/gemini review

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the internal tracking of data sources by introducing a sealed DataSource interface with Server and Cache variants, allowing the cache variant to carry a SqliteSequenceNumber. Additionally, DataSourcePair has been replaced with a more generic TaggedReference utility to associate metadata with query results across the subscription and query manager flows. A critical issue was identified in QuerySubscriptionImpl.kt where failures in non-realtime query updates are silently ignored using getOrNull() ?: return instead of being propagated to the subscriber, and a code suggestion has been provided to correctly handle and send these failures.

@dconeybe dconeybe changed the title Dconeybe/dataconnect/realtime cache update3 Refactor core and cache query sources to propagate sequence numbers using a sealed DataSource and generic TaggedReference May 29, 2026
@dconeybe dconeybe changed the title Refactor core and cache query sources to propagate sequence numbers using a sealed DataSource and generic TaggedReference dataconnect(chore): realtime refactor in preparation for deduplicating cached and server results May 29, 2026
@github-actions
Copy link
Copy Markdown
Contributor

📝 PRs merging into main branch

Our main branch should always be in a releasable state. If you are working on a larger change, or if you don't want this change to see the light of the day just yet, consider using a feature branch first, and only merge into the main branch when the code complete and ready to be released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant