Skip to content

Commit ce22524

Browse files
cabljaccabljac
authored
Merge pull request #2214 from firebase/next
docs(firestore-bigquery-export): add observability guide
2 parents 792c14d + 8266204 commit ce22524

File tree

1 file changed

+241
-0
lines changed

1 file changed

+241
-0
lines changed

‎firestore-bigquery-export/guides/OBSERVABILITY.md

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
# Observability Guide for Firestore BigQuery Export Extension
2+
3+
This guide provides instructions on how to monitor and check logs for the **firestore-bigquery-export** Firebase Extension for versions `>=0.1.56`. Monitoring your extension's logs is essential for debugging, understanding its behavior, and ensuring it operates as expected.
4+
5+
## Table of Contents
6+
7+
- [Overview](#overview)
8+
- [Identifying Extension Functions](#identifying-extension-functions)
9+
- [Viewing Logs in the Firebase Console](#viewing-logs-in-the-firebase-console)
10+
- [Viewing Logs in the Google Cloud Console](#viewing-logs-in-the-google-cloud-console)
11+
- [Using Cloud Functions Logs](#using-cloud-functions-logs)
12+
- [Using Logs Explorer](#using-logs-explorer)
13+
- [Understanding Log Entries](#understanding-log-entries)
14+
- [Filtering Logs with Queries](#filtering-logs-with-queries)
15+
- [Common Log Messages and Troubleshooting](#common-log-messages-and-troubleshooting)
16+
- [Additional Resources](#additional-resources)
17+
18+
## Overview
19+
20+
The `firestore-bigquery-export` extension listens to changes in your specified Firestore collection and exports them to BigQuery. It consists of Cloud Functions that handle these events and write data to BigQuery. Monitoring the logs of these functions helps you ensure the data is being exported correctly and troubleshoot any issues that may arise.
21+
22+
## Identifying Extension Functions
23+
24+
When the extension is installed, it deploys several Cloud Functions. The primary functions related to data export are:
25+
26+
- **OnWrite Function**: Handles Firestore `onWrite` events to synchronize data with BigQuery
27+
28+
- **Function Name Format**: `ext-<EXTENSION_INSTANCE_ID>-fsexportbigquery`
29+
30+
- **OnDispatch Function**: Handles tasks dispatched to synchronize data with BigQuery if the initial onWrite attempts fail.
31+
- **Function Name Format**: `ext-<EXTENSION_INSTANCE_ID>-syncBigQuery`
32+
33+
Replace `<EXTENSION_INSTANCE_ID>` with the instance ID of your installed extension.
34+
35+
For example, if your extension instance ID is `firestore-bigquery-export`, your functions will be named:
36+
37+
- `ext-firestore-bigquery-export-fsexportbigquery`
38+
- `ext-firestore-bigquery-export-syncBigQuery`
39+
40+
## Viewing Logs in the Firebase Console
41+
42+
1. Navigate to the [Firebase Console](https://console.firebase.google.com)
43+
2. Select your project
44+
3. In the left navigation menu, click on **Extensions**
45+
4. Find your instance of the Firestore BigQuery Export extension
46+
5. Click on **View logs** to see the logs specifically for your extension instance
47+
48+
## Viewing Logs in the Google Cloud Console
49+
50+
### Using Cloud Run Functions Logs
51+
52+
1. Navigate to the [Google Cloud Console](https://console.cloud.google.com/)
53+
2. Ensure your project is selected at the top
54+
3. Go to **Cloud Run Functions** by clicking on **Navigation Menu** (☰) > **Cloud Functions**
55+
4. In the list of functions, find your extension's functions:
56+
- `ext-<EXTENSION_INSTANCE_ID>-fsexportbigquery`
57+
- `ext-<EXTENSION_INSTANCE_ID>-syncBigQuery`
58+
5. Click on the function name to view its details
59+
6. In the function details page, click on the **Logs** tab
60+
7. You will see logs specific to that function
61+
62+
### Using Logs Explorer
63+
64+
For more advanced log querying and analysis, use the **Logs Explorer**:
65+
66+
1. In the Google Cloud Console, click on **Navigation Menu** (☰) > **Logging** > **Logs Explorer**
67+
2. Use the following steps to filter logs for your extension
68+
69+
#### Filtering by Function Name and Searching for Firestore event logs from extension
70+
71+
For the fsexportbigquery function (onWrite events):
72+
73+
```plaintext
74+
(resource.type="cloud_function" resource.labels.function_name=("ext-firestore-bigquery-export-fsexportbigquery") resource.labels.region="us-central1") OR (resource.type="cloud_run_revision" resource.labels.service_name=("ext-firestore-bigquery-export-fsexportbigquery") resource.labels.location="us-central1") SEARCH("Firestore event received by onWrite trigger")
75+
```
76+
77+
For the syncBigQuery function (onDispatch events):
78+
79+
```plaintext
80+
(resource.type="cloud_function" resource.labels.function_name=("ext-firestore-bigquery-export-syncBigQuery") resource.labels.region="us-central1") OR (resource.type="cloud_run_revision" resource.labels.service_name=("ext-firestore-bigquery-export-syncBigQuery") resource.labels.location="us-central1") SEARCH("Firestore event received by onDispatch trigger")
81+
```
82+
83+
Replace `firestore-bigquery-export` with your extension instance ID. These queries will show logs from both traditional Cloud Functions and Cloud Run deployments for each respective function.
84+
85+
## Understanding Log Entries
86+
87+
The extension uses structured logging to provide detailed information about its operation. Here's an example of a log entry from the onWrite trigger:
88+
89+
```json
90+
{
91+
"insertId": "672ca19d0008f1be9873dda5",
92+
"jsonPayload": {
93+
"document_name": "projects/dev-extensions-testing/databases/(default)/documents/posts/x0F2QodvO09MJGX3DUfm",
94+
"event_id": "1b8f5ae5-983a-40f6-b921-560118e5e321-1",
95+
"message": "Firestore event received by onWrite trigger",
96+
"operation": 0
97+
},
98+
"labels": {},
99+
"logName": "projects/dev-extensions-testing/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
100+
"receiveTimestamp": "2024-11-07T11:16:45.817442537Z",
101+
"resource": {},
102+
"severity": "INFO",
103+
"timestamp": "2024-11-07T11:16:45.586174Z",
104+
"trace": "projects/dev-extensions-testing/traces/6b1e13de24821178b8937f6ed72cc113"
105+
}
106+
```
107+
108+
Similarly, here's an example log entry from the onDispatch trigger:
109+
110+
```json
111+
{
112+
"insertId": "672ca19d0008f1be9873dda6",
113+
"jsonPayload": {
114+
"document_name": "projects/dev-extensions-testing/databases/(default)/documents/posts/x0F2QodvO09MJGX3DUfm",
115+
"event_id": "1b8f5ae5-983a-40f6-b921-560118e5e321-2",
116+
"message": "Firestore event received by onDispatch trigger",
117+
"operation": 0
118+
},
119+
"labels": {},
120+
"logName": "projects/dev-extensions-testing/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
121+
"receiveTimestamp": "2024-11-07T11:17:45.817442537Z",
122+
"resource": {},
123+
"severity": "INFO",
124+
"timestamp": "2024-11-07T11:17:45.586174Z",
125+
"trace": "projects/dev-extensions-testing/traces/6b1e13de24821178b8937f6ed72cc113"
126+
}
127+
```
128+
129+
Key fields to note:
130+
131+
- `insertId`: A unique identifier for the log entry
132+
- `jsonPayload`: Contains the structured data of the log entry, including:
133+
- `document_name`: The Firestore document path
134+
- `event_id`: The unique event ID
135+
- `message`: The log message indicating which trigger received the event
136+
- `operation`: The type of change that occurred (0 = CREATE, 1 = DELETE, 2 = UPDATE, 3 = IMPORT). This corresponds to the `ChangeType` enum in the extension's code.
137+
- `logName`: The full path of the log in Cloud Logging
138+
- `receiveTimestamp`: When the log was received by Cloud Logging
139+
- `timestamp`: When the event occurred
140+
- `trace`: A unique identifier for tracing the request through the system
141+
142+
## Filtering Logs with Queries
143+
144+
To effectively monitor and troubleshoot, you can use queries in the Logs Explorer to filter logs.
145+
146+
### Filtering by Severity
147+
148+
To view only error logs:
149+
150+
```plaintext
151+
severity="ERROR"
152+
```
153+
154+
### Filtering by Operation Type
155+
156+
To filter logs for a specific operation, such as UPDATE:
157+
158+
```plaintext
159+
jsonPayload.operation=2
160+
```
161+
162+
### Filtering by Log Messages
163+
164+
To find logs related to events received by the onDispatch trigger:
165+
166+
```plaintext
167+
jsonPayload.message="Firestore event received by onDispatch trigger"
168+
```
169+
170+
### Combining Filters
171+
172+
You can combine filters using AND and OR.
173+
174+
Example:
175+
176+
```plaintext
177+
resource.labels.function_name="ext-<EXTENSION_INSTANCE_ID>-syncBigQuery"
178+
AND jsonPayload.message="Firestore event received by onDispatch trigger"
179+
```
180+
181+
This query shows logs from the syncBigQuery function where Firestore events were received by the onDispatch trigger.
182+
183+
## Common Log Messages and Troubleshooting
184+
185+
### "Firestore event received by onWrite trigger"
186+
187+
**Description**: Indicates that the `fsexportbigquery` function received a Firestore event via the onWrite trigger. This is a normal informational message.
188+
189+
**Action**: No action needed if everything is working as expected.
190+
191+
### "Firestore event received by onDispatch trigger"
192+
193+
**Description**: Indicates that the `syncBigQuery` function received a task dispatched via the onDispatch trigger to synchronize data with BigQuery.
194+
195+
**Action**: No action needed if everything is working as expected.
196+
197+
### "Failed to write event to BigQuery"
198+
199+
**Description**: An error occurred when attempting to write data to BigQuery.
200+
201+
**Possible Causes**:
202+
203+
- BigQuery dataset or table does not exist
204+
- Insufficient permissions
205+
- Data schema mismatch
206+
207+
**Troubleshooting Steps**:
208+
209+
1. Check BigQuery Dataset and Table:
210+
211+
- Ensure that the dataset and table specified in your extension configuration exist
212+
- If they do not exist, verify that the extension has the permissions to create them
213+
214+
2. Verify Permissions:
215+
216+
- The service account running the Cloud Function must have the BigQuery Data Editor role
217+
- Check IAM permissions in the Google Cloud Console
218+
219+
3. Check Data Schema:
220+
- Ensure that the data being written matches the schema of the BigQuery table
221+
- Look for any changes in Firestore data that might affect the schema
222+
223+
### "Error when mirroring data to BigQuery"
224+
225+
**Description**: A general error occurred during the data export process.
226+
227+
**Troubleshooting Steps**:
228+
229+
1. Review the Error Details:
230+
231+
- Expand the log entry to view the error stack trace and message
232+
- Identify any specific error codes or messages
233+
234+
2. Inspect Firestore Data:
235+
236+
- Check the Firestore document that caused the error for any anomalies
237+
- Ensure data types and structures are consistent
238+
239+
3. Check Extension Configuration:
240+
- Verify that all configuration parameters are correct
241+
- Ensure that any transformation functions are working properly

0 commit comments

Comments
 (0)