Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
289 changes: 135 additions & 154 deletions apps/docs/content/guides/telemetry/advanced-log-filtering.mdx

Large diffs are not rendered by default.

186 changes: 87 additions & 99 deletions apps/docs/content/guides/telemetry/logs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -261,66 +261,52 @@ You can access the following logs from the **Sources** drop-down:

## Querying with the Logs Explorer

The Logs Explorer uses BigQuery and supports all [available SQL functions and operators](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators).
The Logs Explorer runs on **ClickHouse**. Every log line from every source is a single row in the `logs` table, tagged by a `source` column. Structured fields live in a `log_attributes` map whose values are strings, and the raw line is in `event_message`.

<Admonition type="caution">
<Admonition type="note">

The Logs Explorer now defaults to ClickHouse, where every source shares a single `logs` table and nested fields are read from a `log_attributes` map rather than with `cross join unnest(metadata)`. The examples below still use the legacy BigQuery syntax and are being migrated. Adjust field access accordingly on projects already on ClickHouse.
ClickHouse has been the default engine since June 2026. Projects created before this date use BigQuery, whose `cross join unnest(metadata)` syntax is deprecated. We recommend rewriting those queries in the ClickHouse syntax shown in this guide.

</Admonition>

### Timestamp display and behavior

Each log entry is stored with a `timestamp` as a `TIMESTAMP` data type. Use the appropriate [timestamp function](https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp) to use the `timestamp` field in a query.
Read fields with bracket access, keeping the full dotted key, for example `log_attributes['request.path']` rather than `path`. Wrap numeric values in `toInt32OrZero(...)`, which returns `0` for a missing or non-numeric value. Use `count()` rather than `count(*)`.

Raw top-level timestamp values are rendered as unix microsecond. To render the timestamps in a human-readable format, use the `DATETIME()` function to convert the unix timestamp display into an ISO-8601 timestamp.
For example, to find failing API requests:

```sql
-- timestamp column without datetime()
select timestamp from ....
-- 1664270180000

-- timestamp column with datetime()
select datetime(timestamp) from ....
-- 2022-09-27T09:17:10.439Z
select timestamp,
toInt32OrZero(log_attributes['response.status_code']) as status,
log_attributes['request.path'] as path
from logs
where source = 'edge_logs'
and toInt32OrZero(log_attributes['response.status_code']) >= 400
order by timestamp desc
limit 100;
```

### Unnesting arrays

Each log event stores metadata an array of objects with multiple levels, and can be seen by selecting single log events in the Logs Explorer. To query arrays, use `unnest()` on each array field and add it to the query as a join. This allows you to reference the nested objects with an alias and select their individual fields.

For example, to query the edge logs without any joins:
For example, to find a specific Postgres SQLSTATE (`42501` permission denied, `42P01` relation missing, `23505` duplicate key):

```sql
select timestamp, metadata from edge_logs as t;
select timestamp, log_attributes['parsed.user_name'] as role, event_message
from logs
where source = 'postgres_logs'
and log_attributes['parsed.sql_state_code'] = '42501'
order by timestamp desc
limit 100;
```

The resulting `metadata` key is rendered as an array of objects in the Logs Explorer. In the following diagram, each box represents a nested array of objects:

{/* <!-- Scene is here https://app.excalidraw.com/s/8gj16loJfGZ/3HzccK9MyLx --> */}

![Without Unnesting](/docs/img/unnesting-none.png)

Perform a `cross join unnest()` to work with the keys nested in the `metadata` key.

To query for a nested value, add a join for each array level:
Do not guess `log_attributes` keys. A missing key returns an empty string rather than an error, so a wrong key makes a working query look empty instead of failing. Discover the real keys for a source, or read `event_message`, which always holds the full line. Statement text and error detail live there, not in `parsed.query` or `parsed.detail`, which are usually empty:

```sql
select timestamp, request.method, header.cf_ipcountry
from
edge_logs as t
cross join unnest(t.metadata) as metadata
cross join unnest(metadata.request) as request
cross join unnest(request.headers) as header;
select arrayJoin(mapKeys(log_attributes)) as key, count() as n
from logs
where source = 'postgres_logs'
group by key
order by n desc
limit 100;
```

This surfaces the following columns available for selection:
![With Two Level Unnesting](/docs/img/unnesting-2.png)

This allows you to select the `method` and `cf_ipcountry` columns. In JS dot notation, the full paths for each selected column are:

- `metadata[].request[].method`
- `metadata[].request[].headers[].cf_ipcountry`
If you use the [Supabase MCP server](/docs/guides/getting-started/mcp), the `query_logs` tool runs a custom ClickHouse query like the ones above on hosted projects. The `get_logs` tool returns a service's recent logs without SQL; it is deprecated on hosted projects in favor of `query_logs`, and remains the option for local and self-hosted projects.

### LIMIT and result row limitations

Expand All @@ -339,22 +325,15 @@ When querying large objects, the columnar storage engine selects each column ass
Instead, select only the values required.

```sql
-- ❌ Avoid doing this
select
datetime(timestamp),
m as metadata -- <- metadata contains many nested keys
from
edge_logs as t
cross join unnest(t.metadata) as m;

-- ✅ Do this
select
datetime(timestamp),
r.method -- <- select only the required values
from
edge_logs as t
cross join unnest(t.metadata) as m
cross join unnest(m.request) as r;
-- ❌ Avoid this: selecting the whole attributes map
select timestamp, log_attributes
from logs
where source = 'edge_logs';

-- ✅ Do this: select only the keys you need
select timestamp, log_attributes['request.method'] as method
from logs
where source = 'edge_logs';
```

3. **Query one source at a time.**
Expand All @@ -367,51 +346,60 @@ Identify which service owns the problem from the error or status code first, the

A misspelled or non-existent field name either errors or silently returns nothing, which leaves a working query look empty. Confirm field names in the [field reference](#logs-field-reference), or select `event_message` and inspect a sample row first.

### Examples and templates

The Logs Explorer includes **Templates** (available in the Templates tab or the dropdown in the Query tab) to help you get started.

For example, you can enter the following query in the SQL Editor to retrieve each user's IP address:

```sql
select datetime(timestamp), h.x_real_ip
from
edge_logs
cross join unnest(metadata) as m
cross join unnest(m.request) as r
cross join unnest(r.headers) as h
where h.x_real_ip is not null and r.method = "GET";
```

### Logs field reference

Refer to the full field reference for each available source below. Do note that in order to access each nested key, you would need to perform the [necessary unnesting joins](#unnesting-arrays)
Refer to the full field reference for each source below. Each source's structured fields are listed as ClickHouse `log_attributes` keys, alongside the base columns (`id`, `timestamp`, `event_message`, `severity_text`, `source`) that every source has.

<SharedData data="logConstants">
{(logConstants) => (
<Tabs scrollable size="small" type="underlined" defaultActiveId="edge_logs" queryGroup="source">
{logConstants.schemas.map((schema) => (
<TabPanel id={schema.reference} key={schema.reference} label={schema.name}>
<table>
<thead>
<tr>
<th className="font-bold">Path</th>
<th className="font-bold">Type</th>
</tr>
</thead>
<tbody>
{schema.fields
.sort((a, b) => a.path - b.path)
.map((field) => (
{(logConstants) => {
const otelBaseFields = [
{ path: 'id', type: 'string' },
{ path: 'timestamp', type: 'datetime' },
{ path: 'event_message', type: 'string' },
{ path: 'severity_text', type: 'string' },
{ path: 'source', type: 'string' },
]
const baseColumns = new Set(otelBaseFields.map((field) => field.path))
return (
<Tabs
scrollable
size="small"
type="underlined"
defaultActiveId="edge_logs"
queryGroup="source"
>
{logConstants.schemas.map((schema) => {
const fields = [
...otelBaseFields,
...schema.fields
.filter((field) => !baseColumns.has(field.path))
.map((field) => ({
path: `log_attributes['${field.path.replace(/^metadata\./, '')}']`,
type: field.type,
})),
]
return (
<TabPanel id={schema.reference} key={schema.reference} label={schema.name}>
<table>
<thead>
<tr>
<td className="font-mono">{field.path}</td>
<td className="font-mono">{field.type}</td>
<th>Path</th>
<th>Type</th>
</tr>
))}
</tbody>
</table>
</TabPanel>
))}
</Tabs>
)}
</thead>
<tbody>
{fields.map((field) => (
<tr>
<td>{field.path}</td>
<td>{field.type}</td>
</tr>
))}
</tbody>
</table>
</TabPanel>
)
})}
</Tabs>
)
}}
</SharedData>
1 change: 1 addition & 0 deletions apps/docs/features/docs/Reference.api.utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ export interface IApiEndPoint {
security?: Array<ISecurityOption>
'x-oauth-scope'?: string
'x-allowed-plans'?: string[]
'x-fga-permissions'?: string[][]
}

export type ISchema =
Expand Down
5 changes: 1 addition & 4 deletions apps/docs/features/docs/Reference.sections.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -275,10 +275,7 @@ async function ApiEndpointSection({ link, section, servicePath }: ApiEndpointSec
: await getApiEndpointById(section.id)
if (!endpointDetails) return null

const endpointFgaPermissionGroups =
endpointDetails.security
?.filter((sec) => 'fga_permissions' in sec)
.map((sec) => sec.fga_permissions) ?? []
const endpointFgaPermissionGroups = endpointDetails['x-fga-permissions'] ?? []
const pathParameters = (endpointDetails.parameters ?? []).filter((param) => param.in === 'path')
const queryParameters = (endpointDetails.parameters ?? []).filter((param) => param.in === 'query')
const bodyParameters =
Expand Down
4 changes: 4 additions & 0 deletions apps/studio/components/grid/components/editor/JsonEditor.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ import { isTableLike } from '@/data/table-editor/table-editor-types'
import { useGetCellValueMutation } from '@/data/table-rows/get-cell-value-mutation'
import { useSelectedProjectQuery } from '@/hooks/misc/useSelectedProject'
import { prettifyJSON, removeJSONTrailingComma, tryParseJson } from '@/lib/helpers'
import { RoleImpersonationState } from '@/lib/role-impersonation'
import { useRoleImpersonationStateSnapshot } from '@/state/role-impersonation-state'

const verifyJSON = (value: string) => {
try {
Expand Down Expand Up @@ -86,6 +88,7 @@ export const JsonEditor = <TRow, TSummaryRow = unknown>({
const applyChangesLabel = isQueueOperationsEnabled ? 'Queue changes' : 'Save changes'

const { mutate: getCellValue, isPending, isSuccess } = useGetCellValueMutation()
const roleImpersonationState = useRoleImpersonationStateSnapshot()

const loadFullValue = () => {
if (selectedTable === undefined || project === undefined || !isTableLike(selectedTable)) return
Expand All @@ -104,6 +107,7 @@ export const JsonEditor = <TRow, TSummaryRow = unknown>({
pkMatch,
projectRef: project?.ref,
connectionString: project?.connectionString,
roleImpersonationState: roleImpersonationState as RoleImpersonationState,
},
{
onSuccess: (data) => {
Expand Down
4 changes: 4 additions & 0 deletions apps/studio/components/grid/components/editor/TextEditor.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ import { useTableEditorQuery } from '@/data/table-editor/table-editor-query'
import { isTableLike } from '@/data/table-editor/table-editor-types'
import { useGetCellValueMutation } from '@/data/table-rows/get-cell-value-mutation'
import { useSelectedProjectQuery } from '@/hooks/misc/useSelectedProject'
import { RoleImpersonationState } from '@/lib/role-impersonation'
import { useRoleImpersonationStateSnapshot } from '@/state/role-impersonation-state'

export const TextEditor = <TRow, TSummaryRow = unknown>({
row,
Expand Down Expand Up @@ -58,6 +60,7 @@ export const TextEditor = <TRow, TSummaryRow = unknown>({
const applyChangesLabel = isQueueEnabled ? 'Queue changes' : 'Save changes'

const { mutate: getCellValue, isPending, isSuccess } = useGetCellValueMutation()
const roleImpersonationState = useRoleImpersonationStateSnapshot()

const isTruncated = isValueTruncated(initialValue)

Expand All @@ -78,6 +81,7 @@ export const TextEditor = <TRow, TSummaryRow = unknown>({
pkMatch,
projectRef: project?.ref,
connectionString: project?.connectionString,
roleImpersonationState: roleImpersonationState as RoleImpersonationState,
},
{ onSuccess: (data) => setValue(data) }
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ export const isOAuthInstalled = ({
)
}

if (integration.id === 'grafana') {
if (integration.id === 'grafana' || integration.id === 'grafana-cloud') {
// Grafana is not yet sending integration status, so just use presence of API key.
return (
isOAuthAppAuthorized(projectData, integration) ||
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,21 @@ const STABLE_INTEGRATIONS = [
navigate: () => null,
navigation: [{ route: 'overview', label: 'Overview' }],
},
{
id: 'grafana-cloud',
name: 'Grafana',
type: 'oauth',
source: 'Partner',
description: 'Grafana',
content: 'Grafana overview content',
docsUrl: null,
siteUrl: null,
author: { name: 'Grafana Labs' },
oauthAppId: 'grafana-app',
icon: () => null,
navigate: () => null,
navigation: [{ route: 'overview', label: 'Overview' }],
},
]
const STABLE_EMPTY_ARR = [] as unknown[]
vi.mock('@/components/interfaces/Integrations/Landing/useAvailableIntegrations', () => ({
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ export type { MarketplaceSource } from '@/components/interfaces/Integrations/Lan
// Defines featured integrations and their order in the featured hero
export const FEATURED_INTEGRATION_IDS = [
'grafana',
'grafana-cloud',
'cron',
'queues',
'stripe_sync_engine',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ const FEATURED_INTEGRATION_IMAGES: Record<string, { dark: string; light?: string
dark: `${BASE_PATH}/img/integrations/covers/grafana-cover.png`,
light: `${BASE_PATH}/img/integrations/covers/grafana-cover-light.webp`,
},
'grafana-cloud': {
dark: `${BASE_PATH}/img/integrations/covers/grafana-cover.png`,
light: `${BASE_PATH}/img/integrations/covers/grafana-cover-light.webp`,
},
}

interface ThemedImage {
Expand Down
Loading
Loading