feat(bigquery-jdbc): add EnableProjectDiscovery connection property for metadata methods

[keshavdandeva]

b/499078725

This PR implements the EnableProjectDiscovery connection property into the BigQuery JDBC driver. Enabling this property (default false) allows JDBC database metadata methods (like getCatalogs() and getSchemas()) to discover and query datasets across all Google Cloud projects accessible to the client credentials, rather than being confined to the single default ProjectId specified in the connection URL.

Changes

• Connection Parameter Parsing: Added parsing for EnableProjectDiscovery in BigQueryJdbcUrlUtility and configured it in DataSource.
• SDK Integration: Implemented BigQueryConnection.getDiscoveredProjects() to utilize the BigQuery.listProjects() core SDK method
• Resilience and Caching:
  • Connection-scoped caching is implemented via discoveredProjectsCache.
  • Non-transient errors (HTTP 400, 401, 403) cache an empty list to avoid redundant, failing calls to the BigQuery API.
  • Transient errors (like 500) are not cached, allowing subsequent metadata scans to trigger retries.
• Metadata Integration: Integrated discovered projects inside BigQueryDatabaseMetaData.getAccessibleCatalogNames() so that catalog and schema metadata fetches automatically query across all accessible projects when EnableProjectDiscovery is enabled.

[gemini-code-assist]

Code Review

This pull request introduces an EnableProjectDiscovery configuration property to automatically discover and list all accessible Google Cloud projects as catalogs. To support this, schema fetching in BigQueryDatabaseMetaData has been refactored to run concurrently using an executor service. Feedback on these changes highlights two main areas for improvement: first, replacing the fragile use of reflection to access the low-level BigQuery client with the standard public BigQuery.listProjects() API; second, ensuring that outstanding asynchronous tasks are properly cancelled if the schema fetching loop is interrupted to prevent resource leaks.

[keshavdandeva]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces an automatic project discovery feature to the BigQuery JDBC driver, allowing users to discover all accessible Google Cloud projects via a new EnableProjectDiscovery connection property. It also updates BigQueryDatabaseMetaData to parallelize schema fetching across multiple projects. The review feedback highlights several areas for improvement: reusing a shared ExecutorService instead of creating a new one on every getSchemas() call, avoiding caching empty results on transient exceptions in getDiscoveredProjects(), using GsonFactory.getDefaultInstance() for better resource reuse, and preserving the stack trace when logging execution exceptions.

[logachev]

I'd suggest some refactoring for this method, few ideas to simplify:

• Move code to fetch list of schemas in a specific catalog to a separate function;
• Keep this method simple - single catalog only, essentially calls into helper & transforms to jsonResultSet. No need for background threads since it is single catalog
• Refactor getSchemas() to be the one that fans out multiple requests & assembles data.

Probably breaking up CLs in 2 will be easier:

• Add proper threadPool to connection-layer (and remove static threadPool in statement, it is not used)
• update getSchemas

Also I'd suggest to reuse more code. There are 4 metadata methods that can generate large # of rest calls:

• getCatalogs()
• getSchemas()
• getTables()
• getColumns()
• getProcedures()

In a way, they build results on top of each other. Right now we duplicate a lot, e.g. listDatasets() is called in 7 different places.

Also please don't tackle all of them in a single CL, lets do it incrementally :)
(Otherwise it is pain to review)

[keshavdandeva]

Yeah, makes sense. I have created 3 bugs:

1. b/520400589 - Refactor Metadata Thread Pool Management to Reuse Connection-Scoped Executor
2. b/520407325 - Refactor getSchemas for Catalog-Based Routing (Synchronous & Async Fan-out)
3. b/520406763 - Deduplicate metadata API calls

[keshavdandeva]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces an automatic project discovery feature to the BigQuery JDBC driver, allowing users to discover all accessible Google Cloud projects via the EnableProjectDiscovery connection property. The implementation includes caching of discovered projects and error handling for transient and non-transient BigQuery exceptions. The review feedback correctly identifies compilation errors in BigQueryConnection.java where invalid LOG.warning(e, ...) calls are used instead of LOG.log(Level.WARNING, ..., e).