Skip to content

feat(schema): add resourceType dimension to events and gauges (column + index + projection) #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lohanidamodar wants to merge 1 commit into from
Closed

feat(schema): add resourceType dimension to events and gauges (column + index + projection) #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 25 additions & 2 deletions src/Usage/Adapter/ClickHouse.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -682,6 +682,7 @@ private function formatParamValue(mixed $value): string
['name' => 'p_by_path', 'dims' => ['path']],
['name' => 'p_by_country', 'dims' => ['country']],
['name' => 'p_by_service', 'dims' => ['service']],
['name' => 'p_by_resourceType', 'dims' => ['resourceType']],
];

/**
Expand All @@ -695,8 +696,10 @@ private function formatParamValue(mixed $value): string
private const GAUGE_PROJECTIONS = [
['name' => 'p_by_service', 'dims' => ['service']],
['name' => 'p_by_resource', 'dims' => ['resource']],
['name' => 'p_by_resourceType', 'dims' => ['resourceType']],
['name' => 'p_by_resourceId', 'dims' => ['resourceId']],
['name' => 'p_by_resource_resourceId', 'dims' => ['resource', 'resourceId']],
['name' => 'p_by_resourceType_resource', 'dims' => ['resourceType', 'resource']],
];

/**
Expand Down Expand Up @@ -741,6 +744,7 @@ public function setup(): void
);

$this->ensureGaugeDimColumns();
$this->ensureEventDimColumns();

// --- Per-dim projections on the events / gauges base tables ---
$this->setLightweightMutationProjectionMode($this->getEventsTableName());
Expand Down Expand Up @@ -777,7 +781,7 @@ private function setLightweightMutationProjectionMode(string $baseTable): void
}

/**
* Backfill the service / resource columns on an existing gauges table.
* Backfill late-added dim columns on an existing gauges table.
* setup() uses CREATE TABLE IF NOT EXISTS, so deployments that came up
* before these columns were added never receive them — the gauge
* projections would then fail because their SELECT references columns
Expand All @@ -790,7 +794,26 @@ private function ensureGaugeDimColumns(): void

$sql = "ALTER TABLE {$gaugesTable} "
. 'ADD COLUMN IF NOT EXISTS service LowCardinality(Nullable(String)), '
. 'ADD COLUMN IF NOT EXISTS resource LowCardinality(Nullable(String))';
. 'ADD COLUMN IF NOT EXISTS resource LowCardinality(Nullable(String)), '
. 'ADD COLUMN IF NOT EXISTS resourceType LowCardinality(Nullable(String))';

$this->query($sql);
}

/**
* Backfill late-added dim columns on an existing events table. Same
* reasoning as ensureGaugeDimColumns — CREATE TABLE IF NOT EXISTS won't
* pick up columns added to the schema after the table was first created,
* and a per-dim projection on `resourceType` cannot be materialized until
* the source column exists on the base table.
*/
private function ensureEventDimColumns(): void
{
$eventsTable = $this->escapeIdentifier($this->database)
. '.' . $this->escapeIdentifier($this->getEventsTableName());

$sql = "ALTER TABLE {$eventsTable} "
. 'ADD COLUMN IF NOT EXISTS resourceType LowCardinality(Nullable(String))';
Comment on lines +815 to +816

@greptile-apps greptile-apps Bot Jun 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Update daily rollups

The events base table gets resourceType, but the daily events table, daily materialized view, DAILY_COLUMNS, and findDaily() grouping still omit it. Calls that use the daily APIs with a resourceType filter are rejected by validateDailyAttributeName(), and closed-day event sums filtered by resourceType cannot use the daily rollup because the router treats the column as not daily-safe. This leaves resourceType incomplete as a first-class events dimension for historical usage reads.


$this->query($sql);
Comment on lines 795 to 818

@greptile-apps greptile-apps Bot Jun 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Migrate the indexes

This migration only adds the new resourceType columns. On existing ClickHouse tables, createTable() is skipped by CREATE TABLE IF NOT EXISTS, so the new index-resourceType definitions from Metric::get*Indexes() are never added. After an upgrade, WHERE resourceType = ... queries can still scan without the promised skip index, while fresh installs get different behavior.

Artifacts

Repro: generated migration harness

  • Contains supporting evidence from the run (text/x-php; charset=utf-8).

Repro: harness output showing missing resourceType index migration

  • Keeps the command output available without making the summary code-heavy.

View artifacts

T-Rex Ran code and verified through T-Rex

}
Expand Down
30 changes: 23 additions & 7 deletions src/Usage/Metric.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
* 'status' => '201',
* 'service' => 'storage',
* 'resource' => 'bucket',
* 'resourceType' => 'buckets',
* 'resourceId' => 'abc123',
* 'resourceInternalId' => '42',
* 'teamId' => 'team_x',
Expand All @@ -51,7 +52,7 @@ class Metric extends ArrayObject
*/
public const EVENT_COLUMNS = [
'path', 'method', 'status',
'service', 'resource', 'resourceId', 'resourceInternalId',
'service', 'resource', 'resourceType', 'resourceId', 'resourceInternalId',
'teamId', 'teamInternalId',
'country', 'region', 'hostname',
'osCode', 'osName', 'osVersion',
Expand All @@ -63,7 +64,7 @@ class Metric extends ArrayObject
/**
* Gauge-specific column names that are extracted from tags into dedicated columns.
*/
public const GAUGE_COLUMNS = ['service', 'resource', 'teamId', 'teamInternalId', 'resourceId', 'resourceInternalId'];
public const GAUGE_COLUMNS = ['service', 'resource', 'resourceType', 'teamId', 'teamInternalId', 'resourceId', 'resourceInternalId'];

/**
* Construct a new metric object.
Expand All @@ -79,7 +80,7 @@ class Metric extends ArrayObject
* Event-only dimension columns (see EVENT_COLUMNS):
* - path / method / status: HTTP shape
* - service: API service segment (storage, databases, …)
* - resource / resourceId / resourceInternalId: resource identity
* - resource / resourceType / resourceId / resourceInternalId: resource identity (`resource` is singular like 'bucket', `resourceType` is plural like 'buckets')
* - teamId / teamInternalId: owning team identity
* - country / region / hostname: geographic + caller origin
* - osCode / osName / osVersion: parsed user-agent OS fields
Expand Down Expand Up @@ -236,6 +237,19 @@ public function getResourceId(): ?string
return is_string($resourceId) ? $resourceId : null;
}

/**
* Get the API-level resource type the metric belongs to (e.g. 'functions',
* 'sites', 'buckets', 'databases'). Plural, contrast with `resource`
* which is the singular type of the row itself (e.g. 'deployment',
* 'function', 'bucket'). Low cardinality — useful as a group-by /
* filter dimension across a project's whole stats surface.
*/
public function getResourceType(): ?string
{
$v = $this->getAttribute('resourceType', null);
return is_string($v) ? $v : null;
}

/**
* Get country code (event metrics only).
*
Expand Down Expand Up @@ -611,6 +625,7 @@ public static function getEventSchema(): array
$stringColumn('status', 16),
$stringColumn('service', 256),
$stringColumn('resource', 256),
$stringColumn('resourceType', 64),
$stringColumn('resourceId', 255),
$stringColumn('resourceInternalId', 255),
$stringColumn('teamId', 255),
Expand Down Expand Up @@ -684,6 +699,7 @@ public static function getGaugeSchema(): array
],
$stringColumn('service', 256),
$stringColumn('resource', 256),
$stringColumn('resourceType', 64),
$stringColumn('teamId', 255),
$stringColumn('teamInternalId', 255),
$stringColumn('resourceId', 255),
Expand Down Expand Up @@ -713,13 +729,13 @@ public static function getEventIndexes(): array
{
$indexed = [
'path', 'method', 'status',
'service', 'resource', 'resourceId', 'resourceInternalId',
'service', 'resource', 'resourceType', 'resourceId', 'resourceInternalId',
'teamId', 'teamInternalId',
'country', 'region', 'hostname',
'osName', 'clientType', 'clientName', 'deviceName',
];

$setIndexed = ['status', 'method', 'country', 'service', 'clientType', 'osName'];
$setIndexed = ['status', 'method', 'country', 'service', 'resourceType', 'clientType', 'osName'];

return array_map(
static function (string $col) use ($setIndexed): array {
Expand All @@ -745,9 +761,9 @@ static function (string $col) use ($setIndexed): array {
*/
public static function getGaugeIndexes(): array
{
$indexed = ['service', 'resource', 'resourceId', 'resourceInternalId', 'teamId', 'teamInternalId'];
$indexed = ['service', 'resource', 'resourceType', 'resourceId', 'resourceInternalId', 'teamId', 'teamInternalId'];

$setIndexed = ['service', 'resource'];
$setIndexed = ['service', 'resource', 'resourceType'];

return array_map(
static fn (string $col): array => [
Expand Down
Loading