Search before asking
I searched existing Apache Doris issues for external metadata cache statistics and ExternalMetaCacheMgr.getCatalogCacheStats. The closed tracking issue #60686 lists unified monitoring as a framework goal, but I did not find a dedicated issue covering this remaining gap.
Background
PR #65126 migrates the ExternalCatalog and ExternalDatabase name/object caches to MetaCacheEntry. Four cache entries are owned directly by the catalog hierarchy rather than by an engine cache registered in ExternalMetaCacheMgr:
- ExternalCatalog database names
- ExternalCatalog database objects
- ExternalDatabase table names
- ExternalDatabase table objects
MetaCacheEntry exposes statistics for these entries, but ExternalMetaCacheMgr.getCatalogCacheStats(catalogId) currently reports only engine caches registered in its catalog cache registry. As a result, the unified statistics endpoint omits the central catalog/database name and object caches.
Problem
Operators cannot observe capacity, estimated size, hits, misses, load success/failure, eviction, invalidation, or the last load error for these four cache categories through the unified per-catalog statistics path. This makes cache sizing and cache-consistency diagnosis incomplete even after the MetaCacheEntry migration.
Expected behavior
ExternalMetaCacheMgr.getCatalogCacheStats(catalogId), or the public statistics API built on it, should include both:
- Engine metadata caches registered in ExternalMetaCacheMgr.
- Direct MetaCacheEntry instances owned by ExternalCatalog and its ExternalDatabase objects.
Statistics collection must remain cache-only: it must not initialize a catalog, enumerate remote metadata, load a database/table object, or turn a cold cache hot merely to produce statistics.
Cache names should be stable and distinguish at least catalog database-name/object entries from database table-name/object entries. If table entries are aggregated across databases, the aggregation semantics should be documented; otherwise each database should have a stable owner-qualified cache name.
Suggested implementation
- Add a catalog-side statistics collector for the direct MetaCacheEntry instances.
- Let ExternalMetaCacheMgr merge those statistics with the registered engine-cache statistics.
- Traverse only already initialized/local catalog and database state.
- Preserve lifecycle safety for refresh, unregister, and replay paths.
Suggested tests
- An uninitialized catalog returns statistics without triggering initialization or a remote call.
- Initialized but cold entries are reported with zero size and remain cold.
- Hot database-name/object and table-name/object entries report their own counters and sizes.
- Multiple databases are represented or aggregated deterministically.
- Refresh/reset/unregister removes stale owners from the statistics result.
Related
Search before asking
I searched existing Apache Doris issues for external metadata cache statistics and ExternalMetaCacheMgr.getCatalogCacheStats. The closed tracking issue #60686 lists unified monitoring as a framework goal, but I did not find a dedicated issue covering this remaining gap.
Background
PR #65126 migrates the ExternalCatalog and ExternalDatabase name/object caches to MetaCacheEntry. Four cache entries are owned directly by the catalog hierarchy rather than by an engine cache registered in ExternalMetaCacheMgr:
MetaCacheEntry exposes statistics for these entries, but ExternalMetaCacheMgr.getCatalogCacheStats(catalogId) currently reports only engine caches registered in its catalog cache registry. As a result, the unified statistics endpoint omits the central catalog/database name and object caches.
Problem
Operators cannot observe capacity, estimated size, hits, misses, load success/failure, eviction, invalidation, or the last load error for these four cache categories through the unified per-catalog statistics path. This makes cache sizing and cache-consistency diagnosis incomplete even after the MetaCacheEntry migration.
Expected behavior
ExternalMetaCacheMgr.getCatalogCacheStats(catalogId), or the public statistics API built on it, should include both:
Statistics collection must remain cache-only: it must not initialize a catalog, enumerate remote metadata, load a database/table object, or turn a cold cache hot merely to produce statistics.
Cache names should be stable and distinguish at least catalog database-name/object entries from database table-name/object entries. If table entries are aggregated across databases, the aggregation semantics should be documented; otherwise each database should have a stable owner-qualified cache name.
Suggested implementation
Suggested tests
Related