diff --git a/.agents/skills.json b/.agents/skills.json index ec9887a7cf..de88709dcb 100644 --- a/.agents/skills.json +++ b/.agents/skills.json @@ -2,6 +2,9 @@ "entries": [ { "path": "agents/skills/dc-import-info" + }, + { + "path": "agents/skills/dc-import-postmortem-doc" } ] } diff --git a/.gitignore b/.gitignore index c263eb0b38..87f3d76b24 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,7 @@ import-automation/executor/config_override.json .agents/* !.agents/skills.json +# Ignore generated troubleshooting post-mortem documents +agents/troubleshooting/ + + diff --git a/agents/skills/dc-import-postmortem-doc/SKILL.md b/agents/skills/dc-import-postmortem-doc/SKILL.md new file mode 100644 index 0000000000..b01a5a80b7 --- /dev/null +++ b/agents/skills/dc-import-postmortem-doc/SKILL.md @@ -0,0 +1,115 @@ +--- +name: dc-import-postmortem-doc +description: Generates standardized troubleshooting post-mortem documents from conversation context after diagnosing or fixing an import failure. Captures discovered infrastructure identifiers, root causes, CI/CD gaps, fixes, and long-term preventions into agents/troubleshooting/YYYY-MM-DD/_YYYYMMDD_HHMMSS.md for offline review and pattern extraction. +--- + +# Generate Import Troubleshooting Post-Mortem Documentation + +Use this skill following a troubleshooting or debugging session for a Data Commons import to generate a comprehensive, standardized post-mortem report. + +These documents form an offline repository in `agents/troubleshooting/` to help engineering teams track failure history, extract failure patterns, and design systemic architectural improvements. + +--- + +## 1. Document Target Path + +Always create the post-mortem report at: +```text +agents/troubleshooting//__.md +``` +* ``: The date the incident occurred or was investigated in UTC (e.g. `2026-08-06`). +* ``: The exact simple name of the import (e.g. `USCensusPEP_Sex`). +* ``: The UTC date without dashes (e.g. `20260806`). +* ``: The current UTC time of document generation (24-hour format, e.g. `102735` for 10:27:35 UTC). This ensures unique filenames even if multiple investigations or runs occur on the same day. + +Example: +`agents/troubleshooting/2026-08-06/USCensusPEP_Sex_20260806_102735.md` + +--- + +## 2. Evidence Gathering Rules + +- **Rely on Context**: Synthesize the post-mortem exclusively from facts, logs, exit codes, and infrastructure identifiers surfaced during the current conversation session. +- **No Extra Cloud Queries**: Do not run additional cloud commands during documentation generation. +- **Explicit Unresolved Values**: If a particular piece of metadata was not queried or discovered (e.g., source commit, workflow ID), record it explicitly as `null` or `not_discovered`. Never invent or guess values. +- **Null Value Formatting**: When values are null or unresolved, write unquoted `null` in the YAML metadata block (not `"null"`) so YAML parsers treat them as true null values rather than literal strings. +- **Use Repository-Relative Paths**: Do NOT use local machine-specific absolute filesystem paths (e.g. `/usr/local/google/home/...`). Always cite files relative to the repository root (e.g. `scripts/us_census/pep/us_pep_sex/process.py`) so documents remain portable across environments. +- **Handle Partial or Unresolved Sessions**: Troubleshooting sessions do not always conclude with a fix or definitive root cause. + - Set `resolution_status: "RESOLVED"`, `"UNRESOLVED"`, or `"IN_PROGRESS"` in the metadata block. + - Capture all sections for which evidence was found. + - If no fix was implemented, replace 'Fix Applied & Verification' with a **'Future Investigation & Next Steps'** section detailing open questions, unverified hypotheses, or required access. + +--- + +## 3. Failure Taxonomy + +Populate `failure_category` in the YAML metadata block with one of the standard categories below, and provide more specific detail in `sub_category`: + +| `failure_category` | When to Use | Example `sub_category` | +|---|---|---| +| `dependency_drift` | Package/library updates, version mismatches, or deprecated APIs | `pandas_delim_whitespace_removed` | +| `oom_memory_pressure` | Task ran out of memory, kernel OOM-killer invoked, VM hung/unresponsive | `batch_50002_mcf_load_oom` | +| `permission_or_auth_error` | Insufficient IAM permissions, authentication failures, expired API tokens | `gcs_permission_denied` | +| `upstream_source_error` | External data source unavailable, download 404/500, format changed | `download_url_404` | +| `schema_mismatch` | MCF/TMCF syntax errors, unknown StatVar properties or nodes | `unresolved_statvar_property` | +| `code_logic_error` | Python syntax error, unhandled exception, regex mismatch, parsing bug | `index_out_of_range` | +| `infra_timeout` | Batch job or workflow exceeded maximum allotted execution duration | `batch_timeout_exceeded` | +| `unknown` | Cause could not be definitively determined from available evidence | `unresolved_crash` | + +--- + +## 4. Standard Document Template + +Every generated troubleshooting document must strictly conform to the following template. Include the execution start time in UTC in the title if known (e.g. `# Troubleshooting Post-Mortem: USCensusPEP_Sex (2026-08-05T01:00:31Z)`), or omit it if unknown: + +````markdown +# Troubleshooting Post-Mortem: [ ()] + +```yaml +import_name: "" +date: "" +created_at: "" +status: "FAILURE" +resolution_status: "" +failure_category: "" +sub_category: "" +manifest_path: +absolute_import_name: +environment: "" +job_id: +job_uid: +exit_code: +image_uri: +source_commit: +workflow_id: +gcs_latest_version: +execution_start_time: +execution_end_time: +``` + +## 1. Executive Summary & Impact +* **Incident Description**: High-level summary of what happened. +* **Impact**: Affected import output, state in Cloud Spanner `ImportStatus`, and downstream implications. + +## 2. Root Cause Analysis +Detailed technical breakdown of why the failure occurred, citing specific error messages, tracebacks, or system constraints. + +## 3. Debugging Trail & Evidence +Step-by-step narrative of the investigation: +1. **Initial Discovery**: How the failed state was identified. +2. **Infrastructure Tracing**: Navigating from Spanner record to Batch job and Task list. +3. **Log Extraction**: Key log lines and stack traces retrieved from Cloud Logging. +4. **Environment Audit**: Any package version, resource limit, or configuration checks performed. + +## 4. CI/CD & Testing Gap Analysis +* **Why Unit Tests Did Not Catch It**: Explain whether unit tests exist, why they failed to catch the issue (e.g. missing `__init__.py` test discovery bypass, mock differences, missing test coverage). +* **Environment Differences**: Note any dependency drift between local test environments and production Docker images. + +## 5. Fix Applied & Verification (or Future Investigation & Next Steps) +* **If Resolved**: Show code diffs/snippets of the fix applied and local test/lint verification outcomes. +* **If Unresolved / In Progress**: List unresolved questions, hypotheses to test, required permissions/access, or follow-up debugging steps. + +## 6. Long-Term Prevention & Recommendations +* **Short-Term Actions**: Follow-ups needed for this specific import. +* **Systemic / Architectural Recommendations**: Suggestions to prevent entire classes of similar bugs across Data Commons (e.g. test discovery enforcement, dependency pinning, resource allocation improvements). +````