From 8e089d6c49fe73ab2d2f18f573f9ac9e1e3ed772 Mon Sep 17 00:00:00 2001 From: Rohit Kumar Date: Thu, 6 Aug 2026 05:48:42 +0000 Subject: [PATCH 1/4] feat: add dc-import-postmortem-doc skill and exclude troubleshooting artifacts from git --- .agents/skills.json | 3 + .gitignore | 4 + .../skills/dc-import-postmortem-doc/SKILL.md | 124 ++++++++++++++++++ 3 files changed, 131 insertions(+) create mode 100644 agents/skills/dc-import-postmortem-doc/SKILL.md diff --git a/.agents/skills.json b/.agents/skills.json index ec9887a7cf..de88709dcb 100644 --- a/.agents/skills.json +++ b/.agents/skills.json @@ -2,6 +2,9 @@ "entries": [ { "path": "agents/skills/dc-import-info" + }, + { + "path": "agents/skills/dc-import-postmortem-doc" } ] } diff --git a/.gitignore b/.gitignore index c263eb0b38..87f3d76b24 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,7 @@ import-automation/executor/config_override.json .agents/* !.agents/skills.json +# Ignore generated troubleshooting post-mortem documents +agents/troubleshooting/ + + diff --git a/agents/skills/dc-import-postmortem-doc/SKILL.md b/agents/skills/dc-import-postmortem-doc/SKILL.md new file mode 100644 index 0000000000..8e2038a923 --- /dev/null +++ b/agents/skills/dc-import-postmortem-doc/SKILL.md @@ -0,0 +1,124 @@ +--- +name: dc-import-postmortem-doc +description: Generates standardized troubleshooting post-mortem documents from conversation context after diagnosing or fixing an import failure. Captures discovered infrastructure identifiers, root causes, CI/CD gaps, fixes, and long-term preventions into agents/troubleshooting/YYYY-MM-DD/_YYYYMMDD.md for offline review and pattern extraction. +--- + +# Generate Import Troubleshooting Post-Mortem Documentation + +Use this skill following a troubleshooting or debugging session for a Data Commons import to generate a comprehensive, standardized post-mortem report. + +These documents form an offline repository in `agents/troubleshooting/` to help engineering teams track failure history, extract failure patterns, and design systemic architectural improvements. + +--- + +## 1. Document Target Path + +Always create the post-mortem report at: +```text +agents/troubleshooting//_.md +``` +* ``: The date the incident occurred or was investigated (e.g. `2026-08-06`). +* ``: The exact simple name of the import (e.g. `USCensusPEP_Sex`). +* ``: The date without dashes (e.g. `20260806`). + +Example: +`agents/troubleshooting/2026-08-06/USCensusPEP_Sex_20260806.md` + +--- + +## 2. Evidence Gathering Rules + +- **Rely on Context**: Synthesize the post-mortem exclusively from facts, logs, exit codes, and infrastructure identifiers surfaced during the current conversation session. +- **No Extra Cloud Queries**: Do not run additional cloud commands during documentation generation. +- **Explicit Unresolved Values**: If a particular piece of metadata was not queried or discovered (e.g., source commit, workflow ID), record it explicitly as `null` or `not_discovered`. Never invent or guess values. +- **Use Repository-Relative Paths**: Do NOT use local machine-specific absolute filesystem paths (e.g. `/usr/local/google/home/...`). Always cite files relative to the repository root (e.g. `scripts/us_census/pep/us_pep_sex/process.py`) so documents remain portable across environments. +- **Handle Partial or Unresolved Sessions**: Troubleshooting sessions do not always conclude with a fix or definitive root cause. + - Set `resolution_status: "RESOLVED"`, `"UNRESOLVED"`, or `"IN_PROGRESS"` in the frontmatter. + - Capture all sections for which evidence was found. + - If no fix was implemented, replace 'Fix Applied & Verification' with a **'Future Investigation & Next Steps'** section detailing open questions, unverified hypotheses, or required access. + +--- + +## 3. Failure Taxonomy + +Populate `failure_category` in the YAML frontmatter with one of the standard categories below, and provide more specific detail in `sub_category`: + +| `failure_category` | When to Use | Example `sub_category` | +|---|---|---| +| `dependency_drift` | Package/library updates, version mismatches, or deprecated APIs | `pandas_delim_whitespace_removed` | +| `oom_memory_pressure` | Task ran out of memory, kernel OOM-killer invoked, VM hung/unresponsive | `batch_50002_mcf_load_oom` | +| `upstream_source_error` | External data source unavailable, download 404/500, format changed | `download_url_404` | +| `schema_mismatch` | MCF/TMCF syntax errors, unknown StatVar properties or nodes | `unresolved_statvar_property` | +| `code_logic_error` | Python syntax error, unhandled exception, regex mismatch, parsing bug | `index_out_of_range` | +| `infra_timeout` | Batch job or workflow exceeded maximum allotted execution duration | `batch_timeout_exceeded` | +| `unknown` | Cause could not be definitively determined from available evidence | `unresolved_crash` | + +--- + +## 4. Standard Document Template + +Every generated troubleshooting document must strictly conform to the following template: + +```markdown +--- +import_name: "" +date: "" +status: "FAILURE" +resolution_status: "" +failure_category: "" +sub_category: "" +manifest_path: "" +absolute_import_name: "" +environment: "" +job_id: "" +job_uid: "" +exit_code: +image_uri: "" +source_commit: "" +workflow_id: "" +gcs_latest_version: "" +execution_start_time: "" +execution_end_time: "" +--- + +# Troubleshooting Post-Mortem: + +## 1. Executive Summary & Impact +* **Incident Description**: High-level summary of what happened. +* **Impact**: Affected import output, state in Cloud Spanner `ImportStatus`, and downstream implications. + +## 2. Infrastructure & Execution Trace +| Field | Value | Source / Discovery Method | +|---|---|---| +| **Import Name** | `` | Repository Manifest / Spanner | +| **Manifest Path** | [``]() | Local repository catalog | +| **Spanner State** | `FAILURE` | Cloud Spanner `ImportStatus` | +| **Batch Job ID** | `` | Spanner / Batch API | +| **Batch Job UID** | `` | `gcloud batch jobs describe` | +| **Exit Code** | `` | `gcloud batch tasks list` | +| **Container Image** | `` | `gcloud batch jobs describe` | +| **Timestamps** | Start: `` \| End: `` | Cloud Batch / Logging | +| **Latest Version URI** | `` | Cloud Spanner / GCS Pointer | + +## 3. Root Cause Analysis +Detailed technical breakdown of why the failure occurred, citing specific error messages, tracebacks, or system constraints. + +## 4. Debugging Trail & Evidence +Step-by-step narrative of the investigation: +1. **Initial Discovery**: How the failed state was identified. +2. **Infrastructure Tracing**: Navigating from Spanner record to Batch job and Task list. +3. **Log Extraction**: Key log lines and stack traces retrieved from Cloud Logging. +4. **Environment Audit**: Any package version, resource limit, or configuration checks performed. + +## 5. CI/CD & Testing Gap Analysis +* **Why Unit Tests Did Not Catch It**: Explain whether unit tests exist, why they failed to catch the issue (e.g. missing `__init__.py` test discovery bypass, mock differences, missing test coverage). +* **Environment Differences**: Note any dependency drift between local test environments and production Docker images. + +## 6. Fix Applied & Verification (or Future Investigation & Next Steps) +* **If Resolved**: Show code diffs/snippets of the fix applied and local test/lint verification outcomes. +* **If Unresolved / In Progress**: List unresolved questions, hypotheses to test, required permissions/access, or follow-up debugging steps. + +## 7. Long-Term Prevention & Recommendations +* **Short-Term Actions**: Follow-ups needed for this specific import. +* **Systemic / Architectural Recommendations**: Suggestions to prevent entire classes of similar bugs across Data Commons (e.g. test discovery enforcement, dependency pinning, resource allocation improvements). +``` From 350d04a7b1c04d2b6441e3c06f4a06dc2aba3f80 Mon Sep 17 00:00:00 2001 From: Rohit Kumar Date: Thu, 6 Aug 2026 05:56:07 +0000 Subject: [PATCH 2/4] docs: update SKILL.md to standardize YAML frontmatter null formatting and add permission error category --- .../skills/dc-import-postmortem-doc/SKILL.md | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/agents/skills/dc-import-postmortem-doc/SKILL.md b/agents/skills/dc-import-postmortem-doc/SKILL.md index 8e2038a923..244a42d881 100644 --- a/agents/skills/dc-import-postmortem-doc/SKILL.md +++ b/agents/skills/dc-import-postmortem-doc/SKILL.md @@ -31,6 +31,7 @@ Example: - **Rely on Context**: Synthesize the post-mortem exclusively from facts, logs, exit codes, and infrastructure identifiers surfaced during the current conversation session. - **No Extra Cloud Queries**: Do not run additional cloud commands during documentation generation. - **Explicit Unresolved Values**: If a particular piece of metadata was not queried or discovered (e.g., source commit, workflow ID), record it explicitly as `null` or `not_discovered`. Never invent or guess values. +- **Null Value Formatting**: When values are null or unresolved, write unquoted `null` in the YAML frontmatter (not `"null"`) so YAML parsers treat them as true null values rather than literal strings. - **Use Repository-Relative Paths**: Do NOT use local machine-specific absolute filesystem paths (e.g. `/usr/local/google/home/...`). Always cite files relative to the repository root (e.g. `scripts/us_census/pep/us_pep_sex/process.py`) so documents remain portable across environments. - **Handle Partial or Unresolved Sessions**: Troubleshooting sessions do not always conclude with a fix or definitive root cause. - Set `resolution_status: "RESOLVED"`, `"UNRESOLVED"`, or `"IN_PROGRESS"` in the frontmatter. @@ -47,6 +48,7 @@ Populate `failure_category` in the YAML frontmatter with one of the standard cat |---|---|---| | `dependency_drift` | Package/library updates, version mismatches, or deprecated APIs | `pandas_delim_whitespace_removed` | | `oom_memory_pressure` | Task ran out of memory, kernel OOM-killer invoked, VM hung/unresponsive | `batch_50002_mcf_load_oom` | +| `permission_or_auth_error` | Insufficient IAM permissions, authentication failures, expired API tokens | `gcs_permission_denied` | | `upstream_source_error` | External data source unavailable, download 404/500, format changed | `download_url_404` | | `schema_mismatch` | MCF/TMCF syntax errors, unknown StatVar properties or nodes | `unresolved_statvar_property` | | `code_logic_error` | Python syntax error, unhandled exception, regex mismatch, parsing bug | `index_out_of_range` | @@ -67,18 +69,18 @@ status: "FAILURE" resolution_status: "" failure_category: "" sub_category: "" -manifest_path: "" -absolute_import_name: "" +manifest_path: +absolute_import_name: environment: "" -job_id: "" -job_uid: "" +job_id: +job_uid: exit_code: -image_uri: "" -source_commit: "" -workflow_id: "" -gcs_latest_version: "" -execution_start_time: "" -execution_end_time: "" +image_uri: +source_commit: +workflow_id: +gcs_latest_version: +execution_start_time: +execution_end_time: --- # Troubleshooting Post-Mortem: From 4147be628a6b933fd1dbe529cef0f05a98bc99c5 Mon Sep 17 00:00:00 2001 From: Rohit Kumar Date: Thu, 6 Aug 2026 10:31:36 +0000 Subject: [PATCH 3/4] docs: update post-mortem file naming convention to include UTC timestamp and add created_at field to template --- agents/skills/dc-import-postmortem-doc/SKILL.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/agents/skills/dc-import-postmortem-doc/SKILL.md b/agents/skills/dc-import-postmortem-doc/SKILL.md index 244a42d881..a32bb5a646 100644 --- a/agents/skills/dc-import-postmortem-doc/SKILL.md +++ b/agents/skills/dc-import-postmortem-doc/SKILL.md @@ -1,6 +1,6 @@ --- name: dc-import-postmortem-doc -description: Generates standardized troubleshooting post-mortem documents from conversation context after diagnosing or fixing an import failure. Captures discovered infrastructure identifiers, root causes, CI/CD gaps, fixes, and long-term preventions into agents/troubleshooting/YYYY-MM-DD/_YYYYMMDD.md for offline review and pattern extraction. +description: Generates standardized troubleshooting post-mortem documents from conversation context after diagnosing or fixing an import failure. Captures discovered infrastructure identifiers, root causes, CI/CD gaps, fixes, and long-term preventions into agents/troubleshooting/YYYY-MM-DD/_YYYYMMDD_HHMMSS.md for offline review and pattern extraction. --- # Generate Import Troubleshooting Post-Mortem Documentation @@ -15,14 +15,15 @@ These documents form an offline repository in `agents/troubleshooting/` to help Always create the post-mortem report at: ```text -agents/troubleshooting//_.md +agents/troubleshooting//__.md ``` -* ``: The date the incident occurred or was investigated (e.g. `2026-08-06`). +* ``: The date the incident occurred or was investigated in UTC (e.g. `2026-08-06`). * ``: The exact simple name of the import (e.g. `USCensusPEP_Sex`). -* ``: The date without dashes (e.g. `20260806`). +* ``: The UTC date without dashes (e.g. `20260806`). +* ``: The current UTC time of document generation (24-hour format, e.g. `102735` for 10:27:35 UTC). This ensures unique filenames even if multiple investigations or runs occur on the same day. Example: -`agents/troubleshooting/2026-08-06/USCensusPEP_Sex_20260806.md` +`agents/troubleshooting/2026-08-06/USCensusPEP_Sex_20260806_102735.md` --- @@ -65,6 +66,7 @@ Every generated troubleshooting document must strictly conform to the following --- import_name: "" date: "" +created_at: "" status: "FAILURE" resolution_status: "" failure_category: "" @@ -93,7 +95,7 @@ execution_end_time: | Field | Value | Source / Discovery Method | |---|---|---| | **Import Name** | `` | Repository Manifest / Spanner | -| **Manifest Path** | [``]() | Local repository catalog | +| **Manifest Path** | `` | Local repository catalog | | **Spanner State** | `FAILURE` | Cloud Spanner `ImportStatus` | | **Batch Job ID** | `` | Spanner / Batch API | | **Batch Job UID** | `` | `gcloud batch jobs describe` | From 5dbfdbd396ae7ec69b7e19b728317848b1f5b4cf Mon Sep 17 00:00:00 2001 From: Rohit Kumar Date: Thu, 6 Aug 2026 10:42:06 +0000 Subject: [PATCH 4/4] docs: update post-mortem documentation template and terminology consistency --- .../skills/dc-import-postmortem-doc/SKILL.md | 43 +++++++------------ 1 file changed, 15 insertions(+), 28 deletions(-) diff --git a/agents/skills/dc-import-postmortem-doc/SKILL.md b/agents/skills/dc-import-postmortem-doc/SKILL.md index a32bb5a646..b01a5a80b7 100644 --- a/agents/skills/dc-import-postmortem-doc/SKILL.md +++ b/agents/skills/dc-import-postmortem-doc/SKILL.md @@ -32,10 +32,10 @@ Example: - **Rely on Context**: Synthesize the post-mortem exclusively from facts, logs, exit codes, and infrastructure identifiers surfaced during the current conversation session. - **No Extra Cloud Queries**: Do not run additional cloud commands during documentation generation. - **Explicit Unresolved Values**: If a particular piece of metadata was not queried or discovered (e.g., source commit, workflow ID), record it explicitly as `null` or `not_discovered`. Never invent or guess values. -- **Null Value Formatting**: When values are null or unresolved, write unquoted `null` in the YAML frontmatter (not `"null"`) so YAML parsers treat them as true null values rather than literal strings. +- **Null Value Formatting**: When values are null or unresolved, write unquoted `null` in the YAML metadata block (not `"null"`) so YAML parsers treat them as true null values rather than literal strings. - **Use Repository-Relative Paths**: Do NOT use local machine-specific absolute filesystem paths (e.g. `/usr/local/google/home/...`). Always cite files relative to the repository root (e.g. `scripts/us_census/pep/us_pep_sex/process.py`) so documents remain portable across environments. - **Handle Partial or Unresolved Sessions**: Troubleshooting sessions do not always conclude with a fix or definitive root cause. - - Set `resolution_status: "RESOLVED"`, `"UNRESOLVED"`, or `"IN_PROGRESS"` in the frontmatter. + - Set `resolution_status: "RESOLVED"`, `"UNRESOLVED"`, or `"IN_PROGRESS"` in the metadata block. - Capture all sections for which evidence was found. - If no fix was implemented, replace 'Fix Applied & Verification' with a **'Future Investigation & Next Steps'** section detailing open questions, unverified hypotheses, or required access. @@ -43,7 +43,7 @@ Example: ## 3. Failure Taxonomy -Populate `failure_category` in the YAML frontmatter with one of the standard categories below, and provide more specific detail in `sub_category`: +Populate `failure_category` in the YAML metadata block with one of the standard categories below, and provide more specific detail in `sub_category`: | `failure_category` | When to Use | Example `sub_category` | |---|---|---| @@ -60,10 +60,12 @@ Populate `failure_category` in the YAML frontmatter with one of the standard cat ## 4. Standard Document Template -Every generated troubleshooting document must strictly conform to the following template: +Every generated troubleshooting document must strictly conform to the following template. Include the execution start time in UTC in the title if known (e.g. `# Troubleshooting Post-Mortem: USCensusPEP_Sex (2026-08-05T01:00:31Z)`), or omit it if unknown: -```markdown ---- +````markdown +# Troubleshooting Post-Mortem: [ ()] + +```yaml import_name: "" date: "" created_at: "" @@ -83,46 +85,31 @@ workflow_id: gcs_latest_version: execution_start_time: execution_end_time: ---- - -# Troubleshooting Post-Mortem: +``` ## 1. Executive Summary & Impact * **Incident Description**: High-level summary of what happened. * **Impact**: Affected import output, state in Cloud Spanner `ImportStatus`, and downstream implications. -## 2. Infrastructure & Execution Trace -| Field | Value | Source / Discovery Method | -|---|---|---| -| **Import Name** | `` | Repository Manifest / Spanner | -| **Manifest Path** | `` | Local repository catalog | -| **Spanner State** | `FAILURE` | Cloud Spanner `ImportStatus` | -| **Batch Job ID** | `` | Spanner / Batch API | -| **Batch Job UID** | `` | `gcloud batch jobs describe` | -| **Exit Code** | `` | `gcloud batch tasks list` | -| **Container Image** | `` | `gcloud batch jobs describe` | -| **Timestamps** | Start: `` \| End: `` | Cloud Batch / Logging | -| **Latest Version URI** | `` | Cloud Spanner / GCS Pointer | - -## 3. Root Cause Analysis +## 2. Root Cause Analysis Detailed technical breakdown of why the failure occurred, citing specific error messages, tracebacks, or system constraints. -## 4. Debugging Trail & Evidence +## 3. Debugging Trail & Evidence Step-by-step narrative of the investigation: 1. **Initial Discovery**: How the failed state was identified. 2. **Infrastructure Tracing**: Navigating from Spanner record to Batch job and Task list. 3. **Log Extraction**: Key log lines and stack traces retrieved from Cloud Logging. 4. **Environment Audit**: Any package version, resource limit, or configuration checks performed. -## 5. CI/CD & Testing Gap Analysis +## 4. CI/CD & Testing Gap Analysis * **Why Unit Tests Did Not Catch It**: Explain whether unit tests exist, why they failed to catch the issue (e.g. missing `__init__.py` test discovery bypass, mock differences, missing test coverage). * **Environment Differences**: Note any dependency drift between local test environments and production Docker images. -## 6. Fix Applied & Verification (or Future Investigation & Next Steps) +## 5. Fix Applied & Verification (or Future Investigation & Next Steps) * **If Resolved**: Show code diffs/snippets of the fix applied and local test/lint verification outcomes. * **If Unresolved / In Progress**: List unresolved questions, hypotheses to test, required permissions/access, or follow-up debugging steps. -## 7. Long-Term Prevention & Recommendations +## 6. Long-Term Prevention & Recommendations * **Short-Term Actions**: Follow-ups needed for this specific import. * **Systemic / Architectural Recommendations**: Suggestions to prevent entire classes of similar bugs across Data Commons (e.g. test discovery enforcement, dependency pinning, resource allocation improvements). -``` +````