Utopia Usage

Utopia framework usage library is a simple and lite library for managing application usage statistics. This library is aiming to be as simple and easy to learn and use. This library is maintained by the Appwrite team.

Although this library is part of the Utopia Framework project it is dependency free and can be used as standalone with any other PHP project or framework.

Features

Two Table Architecture: Separate events and gauges tables optimized for their access patterns
Events Table: Request-level metrics with dedicated columns for every dimension we filter on (path/method/status, service/resource ids, team ids, country/region, hostname, parsed UA fields)
Gauges Table: Simple resource snapshots (storage size, user count, etc.)
Query-Time Aggregation: No write-time period fan-out — aggregate by any interval at query time
Daily Materialized View: Pre-aggregated daily SummingMergeTree for fast billing queries
Pluggable Adapters: ClickHouse (production) and Database (development/testing)
In-Memory Buffering: Collect metrics and flush in batch for high-throughput scenarios
Rich Queries: Filter, sort, paginate using Utopia\Query\Query objects
Multi-Tenant: Shared tables with tenant isolation via string tenant IDs
LowCardinality Columns: Country uses LowCardinality(String) for efficient storage
Bloom Filter Indexes: Fast filtering on all event columns

Getting Started

Install using composer:

composer require utopia-php/usage

Using ClickHouse Adapter

<?php

use Utopia\Usage\Usage;
use Utopia\Usage\Adapter\ClickHouse;

// Configuration is fixed at construction. The adapter is fully stateless —
// the tenant is passed explicitly on every call (see below).
$adapter = new ClickHouse(
    host: 'clickhouse-server',
    username: 'default',
    password: '',
    port: 8123,
    secure: false,
    namespace: 'my_app',
    sharedTables: true,
);

$usage = new Usage($adapter);
$usage->setup(); // Creates events, gauges, and daily MV tables

Using Database Adapter

<?php

use Utopia\Usage\Usage;
use Utopia\Usage\Adapter\Database as DatabaseAdapter;

$adapter = new DatabaseAdapter($database); // Utopia\Database\Database instance
$usage = new Usage($adapter);
$usage->setup();

Multi-tenancy

Usage is stateless: every query/mutation takes the tenant as its first argument, and addBatch carries a tenant on each metric row (so one batch can span tenants). This makes a single Usage instance safe to share across tenants and coroutines.

$usage->getTotal('project_123', 'bandwidth');           // read for one tenant
$usage->addBatch([
    ['tenant' => 'project_123', 'metric' => 'bandwidth', 'value' => 5000, 'tags' => []],
    ['tenant' => 'project_456', 'metric' => 'bandwidth', 'value' => 2000, 'tags' => []],
], Usage::TYPE_EVENT);

Callers that only ever touch one tenant can bind it once with the Tenant decorator, which forwards to Usage with the tenant pre-filled (and stamps it onto every addBatch row):

use Utopia\Usage\Tenant;

$tenant = new Tenant($usage, 'project_123');
$tenant->getTotal('bandwidth');                          // no tenant argument
$tenant->addBatch([
    ['metric' => 'bandwidth', 'value' => 5000, 'tags' => []], // tenant stamped automatically
], Usage::TYPE_EVENT);

Metric Types

Events (Additive)

Events are request-level metrics like bandwidth, executions, API calls. They are summed when aggregated.

Event-specific columns (see Metric::EVENT_COLUMNS): path, method, status, service, resource, resourceId, resourceInternalId, teamId, teamInternalId, country, region, hostname, osCode, osName, osVersion, clientType, clientCode, clientName, clientVersion, clientEngine, clientEngineVersion, deviceName, deviceBrand, deviceModel.

use Utopia\Usage\Accumulator;

// Buffer metrics in memory and flush them in batch
$accumulator = new Accumulator($usage);

// Collect events — tenant first; values accumulate in-memory (summed per tenant+metric)
$accumulator->collect('project_123', 'bandwidth', 5000, Usage::TYPE_EVENT, [
    'path' => '/v1/storage/files',
    'method' => 'POST',
    'status' => '201',
    'service' => 'storage',
    'resource' => 'bucket',
    'resourceId' => 'abc123',
    'resourceInternalId' => '42',
    'teamId' => 'team_x',
    'teamInternalId' => '7',
    'country' => 'US',
    'region' => 'us-east',
    'hostname' => 'app.example.com',
    'osName' => 'iOS',
    'clientName' => 'Appwrite SDK',
    'deviceName' => 'smartphone',
]);

Strict tag keys

All keys passed in the tags array must map to a known event or gauge column (see Metric::EVENT_COLUMNS and Metric::GAUGE_COLUMNS). Unknown keys throw at write time — there is no JSON catch-all. To add a new dimension, widen the schema and bump the library version.

Gauges (Point-in-Time)

Gauges are resource snapshots like storage size, user count, file count. Last-write-wins semantics.

Gauge-specific columns (see Metric::GAUGE_COLUMNS): teamId, teamInternalId, resourceId, resourceInternalId.

// Collect gauges — last value wins per tenant+metric in buffer
$accumulator->collect('project_123', 'users', 1500, Usage::TYPE_GAUGE);
$accumulator->collect('project_123', 'storage.size', 1048576, Usage::TYPE_GAUGE, [
    'teamId' => 'team_x',
    'teamInternalId' => '7',
    'resourceId' => 'abc123',
    'resourceInternalId' => '42',
]);

Flushing

The accumulator exposes raw signals — count() (buffered entries) and elapsedSeconds() (seconds since last flush) — and leaves the flush policy to the caller.

// Flush when the buffer grows large or enough time has passed
if ($accumulator->count() >= 5000 || $accumulator->elapsedSeconds() >= 10) {
    $accumulator->flush(); // Writes events to events table, gauges to gauges table
}

Batch Writes

// Write directly without buffering — each row carries its tenant
$usage->addBatch([
    ['tenant' => 'project_123', 'metric' => 'requests', 'value' => 100, 'tags' => ['path' => '/v1/users', 'method' => 'GET']],
    ['tenant' => 'project_123', 'metric' => 'bandwidth', 'value' => 50000, 'tags' => ['country' => 'DE', 'region' => 'fra']],
], Usage::TYPE_EVENT);

$usage->addBatch([
    ['tenant' => 'project_123', 'metric' => 'users', 'value' => 42, 'tags' => ['teamId' => 'team_x']],
], Usage::TYPE_GAUGE);

Querying Metrics

Find with Query Objects

use Utopia\Query\Query;

// Find events filtered by metric and time range (tenant first)
$metrics = $usage->find('project_123', [
    Query::equal('metric', ['bandwidth']),
    Query::equal('country', ['US']),
    Query::greaterThanEqual('time', '2026-01-01'),
    Query::orderDesc('time'),
    Query::limit(100),
], Usage::TYPE_EVENT);

// Find gauges
$gauges = $usage->find('project_123', [
    Query::equal('metric', ['users', 'storage.size']),
], Usage::TYPE_GAUGE);

// Query both tables (type = null)
$all = $usage->find('project_123', [
    Query::equal('metric', ['bandwidth']),
]);

Totals

// Get total for a single metric (SUM for events, latest for gauges)
$total = $usage->getTotal('project_123', 'bandwidth', [
    Query::greaterThanEqual('time', '2026-03-01'),
], Usage::TYPE_EVENT);

// Batch totals — single query with GROUP BY
$totals = $usage->getTotalBatch(
    'project_123',
    ['bandwidth', 'executions', 'requests'],
    [Query::greaterThanEqual('time', '2026-03-01')],
    Usage::TYPE_EVENT
);

Time Series

// Get time series with query-time aggregation
// Events: SUM per bucket, Gauges: argMax per bucket
$series = $usage->getTimeSeries(
    tenant: 'project_123',
    metrics: ['bandwidth', 'requests'],
    interval: '1d',        // '1h' or '1d'
    startDate: '2026-03-01',
    endDate: '2026-04-01',
    zeroFill: true,        // Fill gaps with zeros
    type: Usage::TYPE_EVENT
);

// Returns: ['bandwidth' => ['total' => 5000000, 'data' => [['value' => 100, 'date' => '...'], ...]]]

Billing Queries (Daily MV)

The daily materialized view pre-aggregates events by metric + tenant + day for fast billing:

// Sum a single metric from the daily table
$total = $usage->sumDaily('project_123', [
    Query::equal('metric', ['bandwidth']),
    Query::between('time', '2026-03-01', '2026-04-01'),
]);

// Sum multiple metrics in one query
$totals = $usage->sumDailyBatch(
    'project_123',
    ['bandwidth', 'executions', 'storage.size'],
    [Query::between('time', '2026-03-01', '2026-04-01')]
);
// Returns: ['bandwidth' => 5000000, 'executions' => 12345, 'storage.size' => 0]

// Find daily aggregated rows
$rows = $usage->findDaily('project_123', [
    Query::equal('metric', ['bandwidth']),
    Query::orderDesc('time'),
    Query::limit(30),
]);

Purge

// Purge all event metrics older than 90 days
$usage->purge('project_123', [
    Query::lessThan('time', '2026-01-01'),
], Usage::TYPE_EVENT);

// Purge all gauge metrics
$usage->purge('project_123', [], Usage::TYPE_GAUGE);

Architecture

ClickHouse Tables

Table	Engine	Purpose
`{ns}_usage_events`	MergeTree	Raw request events with full metadata
`{ns}_usage_gauges`	MergeTree	Resource snapshot gauges
`{ns}_usage_events_daily`	SummingMergeTree	Pre-aggregated daily event totals
`{ns}_usage_events_daily_mv`	Materialized View	Auto-populates daily table on insert

Events Table Schema

Column	Type	Description
id	String	UUID
metric	String	Metric name (e.g. bandwidth, requests)
value	Int64	Metric value
time	DateTime64(3)	Event timestamp
path	Nullable(String)	API endpoint path
method	Nullable(String)	HTTP method
status	Nullable(String)	HTTP status code
service	LowCardinality(Nullable(String))	API service (storage, databases, …)
resource	LowCardinality(Nullable(String))	Resource type (bucket, file, …)
resourceId	Nullable(String)	External resource id
resourceInternalId	Nullable(String)	Internal resource sequence
teamId	Nullable(String)	External team id
teamInternalId	Nullable(String)	Internal team sequence
country	LowCardinality(Nullable(String))	ISO country code (lowercased)
region	LowCardinality(Nullable(String))	Region code (lowercased)
hostname	Nullable(String)	Caller origin host
osCode, osName	LowCardinality(Nullable(String))	Parsed OS short code / name
osVersion	Nullable(String)	Parsed OS version
clientType, clientCode, clientName, clientEngine	LowCardinality(Nullable(String))	Parsed client identity
clientVersion, clientEngineVersion	Nullable(String)	Parsed client versions
deviceName, deviceBrand	LowCardinality(Nullable(String))	Parsed device identity
deviceModel	Nullable(String)	Parsed device model
tenant	Nullable(String)	Tenant ID (shared tables)

Gauges Table Schema

Column	Type	Description
id	String	UUID
metric	String	Metric name
value	Int64	Current value
time	DateTime64(3)	Snapshot timestamp
teamId	Nullable(String)	External team id
teamInternalId	Nullable(String)	Internal team sequence
resourceId	Nullable(String)	External resource id
resourceInternalId	Nullable(String)	Internal resource sequence
tenant	Nullable(String)	Tenant ID (shared tables)

Daily Table Schema

Column	Type	Description
metric	String	Metric name
value	Int64	Aggregated daily sum
time	DateTime64(3)	Day start timestamp
resource	LowCardinality(Nullable(String))	Resource type
resourceId	Nullable(String)	External resource id
resourceInternalId	Nullable(String)	Internal resource sequence
teamId	Nullable(String)	External team id
teamInternalId	Nullable(String)	Internal team sequence
tenant	Nullable(String)	Tenant ID (shared tables)

Creating Custom Adapters

Extend Utopia\Usage\Adapter and implement:

getName(), setup(), healthCheck()
addBatch(array $metrics, string $type, int $batchSize): bool (each metric carries its own tenant)
find(string $tenant, array $queries, ?string $type): array
count(string $tenant, array $queries, ?string $type, ?int $max): int
sum(string $tenant, array $queries, string $attribute, string $type): int
getTimeSeries(string $tenant, array $metrics, string $interval, string $startDate, string $endDate, array $queries, bool $zeroFill, ?string $type): array
getTotal(string $tenant, string $metric, array $queries, ?string $type): int
getTotalBatch(string $tenant, array $metrics, array $queries, ?string $type): array
findDaily(string $tenant, array $queries): array
sumDaily(string $tenant, array $queries, string $attribute): int
sumDailyBatch(string $tenant, array $metrics, array $queries): array
purge(string $tenant, array $queries, ?string $type): bool

All configuration — namespace, database, shared-tables mode, async inserts, query logging — is set once via the adapter constructor (see "Using ClickHouse Adapter" above). Adapters hold no per-request state; the tenant is passed explicitly on every call, so one instance is safe to share across tenants and coroutines.

System Requirements

Utopia Framework requires PHP 8.0 or later. We recommend using the latest PHP version whenever possible.

Copyright and license

The MIT License (MIT) http://www.opensource.org/licenses/mit-license.php

Name		Name	Last commit message	Last commit date
Latest commit History 198 Commits
.github/workflows		.github/workflows
src/Usage		src/Usage
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE.md		LICENSE.md
README.md		README.md
composer.json		composer.json
composer.lock		composer.lock
docker-compose.yml		docker-compose.yml
phpstan.neon		phpstan.neon
phpunit.xml		phpunit.xml
pint.json		pint.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Utopia Usage

Features

Getting Started

Using ClickHouse Adapter

Using Database Adapter

Multi-tenancy

Metric Types

Events (Additive)

Strict tag keys

Gauges (Point-in-Time)

Flushing

Batch Writes

Querying Metrics

Find with Query Objects

Totals

Time Series

Billing Queries (Daily MV)

Purge

Architecture

ClickHouse Tables

Events Table Schema

Gauges Table Schema

Daily Table Schema

Creating Custom Adapters

System Requirements

Copyright and license

About

Uh oh!

Releases 10

Packages

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

Utopia Usage

Features

Getting Started

Using ClickHouse Adapter

Using Database Adapter

Multi-tenancy

Metric Types

Events (Additive)

Strict tag keys

Gauges (Point-in-Time)

Flushing

Batch Writes

Querying Metrics

Find with Query Objects

Totals

Time Series

Billing Queries (Daily MV)

Purge

Architecture

ClickHouse Tables

Events Table Schema

Gauges Table Schema

Daily Table Schema

Creating Custom Adapters

System Requirements

Copyright and license

About

Resources

License

Code of conduct

Contributing

Uh oh!

Stars

Watchers

Forks

Releases 10

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages