Added runbooks for RCA and replaced GCS URLs with the repsective paths

tools/import_validation/troubleshooting_guides/Golden Checks.md

@@ -0,0 +1,199 @@
+# **Implementation Guide: Golden Set Validations**
+
+## **1\. Overview**
+
+Due to recurring data deletion failures, we protect our imports using **Golden Set Validations (`GOLDENS_CHECK`)**. This guide helps understand how to generate "golden files" (baselines of expected data) and configure the automated validation system to prevent accidental data regressions.
+
+Currently, to mitigate issues related to data loss, we implement two primary validations supported by specific golden baselines:
+
+1. **`Check_goldens_output_csv`**: Verifies the final output data against an established baseline.  
+2. **`Check_goldens_summary_report`**: Validates structural metrics (like number of places and dates) against a summary baseline.  
+3. More regarding golden checks can be seen : [github link](https://github.com/datacommonsorg/data/blob/master/tools/import_validation/Validations.md)
+
+## **2\. Directory Architecture**
+
+To implement these checks, your import script folder must contain the following file structure:
+
+Plaintext
+
+```
+your_import_folder/
+│
+├── manifest.json                  # Main import configuration
+├── validation_config.json          # Custom validation rules
+│
+└── golden_data/                   # Folder holding your baseline files
+    ├── golden_summary_report.csv  # Generated golden summary
+    └── golden_observations.csv    # Generated golden output data
+```
+
+## **3\. Step-by-Step: How to Create Golden Files**
+
+Golden files are created by extracting a snapshot of known-good data using the script `validator_goldens.py`. Run these commands from your terminal inside your import directory.
+
+### **Step 3.1: Generate the Summary Report Golden File**
+
+This step tracks metadata properties like Statistical Variables (StatVars), the number of places, and date ranges to ensure future runs don't accidentally drop the entire series.
+
+From the data/tools/import\_validation/validator\_goldens.py directory, execute the following command:
+
+```
+python3 validator_goldens.py --validate_goldens_input=summary_report.csv  --generate_goldens=golden_data/golden_summary_report.csv  --generate_goldens_property_sets="StatVar|NumPlaces|MinDate|MeasurementMethods|Units|ScalingFactors|observationPeriods"
+```
+
+### **Step 3.2: Generate the Output Data Golden File (only use if needed)** 
+
+This step targets critical combinations of highly utilized StatVars and top geographical regions to ensure key data points are always preserved.
+
+From the data/tools/import\_validation/validator\_goldens.py directory, execute the following command:
+
+```
+python3  validator_goldens.py --validate_goldens_input=output.csv --generate_goldens=golden_data/golden_observations.csv --goldens_must_include="observationAbout:gs://unresolved_mcf/import_validation/top_100k_places.csv" --generate_goldens_property_sets="observationAbout"
+```
+
+## **4\. Configuring `validation_config.json`**
+
+Create a file named `validation_config.json` in your import script directory. Paste the configuration below.
+
+This configuration does two things:
+
+* Overrides the default deletion tolerance rule (`check_deleted_records_percent`) to a  threshold as per history deletions & current deletions should not be more than  **0.1%**.  
+* Activates the two required golden check rules pointing to the files you created in Section 3\.
+
+JSON
+
+```
+{
+    "schema_version": "1.0",
+    "rules": [
+        {
+            "rule_id": "check_deleted_records_percent",
+            "description": "Strictly enforce historical deletion average threshold of 0.1%",
+            "validator": "DELETED_RECORDS_PERCENT",
+            "params": {
+                "threshold": 0.1
+            }
+        },
+        {
+            "rule_id": "check_goldens_summary_report",
+            "description": "Validates summary_report.csv against the golden summary data",
+            "validator": "GOLDENS_CHECK",
+            "params": {
+                "golden_files": ["golden_data/golden_summary_report.csv"]
+            }
+        },
+        {
+            "rule_id": "check_goldens_output_csv",
+            "description": "Verifies the generated output CSV data matches established critical golden records",
+            "validator": "GOLDENS_CHECK",
+            "params": {
+                "golden_files": ["golden_data/golden_observations.csv"],
+                "input_files": ["output/observations.csv"]
+            }
+        }
+    ]
+}
+```
+
+##  **Below is how the validation\_config.json is structured for imports with multiple output CSVs:**
+
+JSON
+
+```
+{
+    "schema_version": "1.0",
+    "rules": [
+        {
+            "rule_id": "check_deleted_records_percent",
+            "description": "Checks that the percentage of deleted records for the entire import is within threshold.",
+            "validator": "DELETED_RECORDS_PERCENT",
+            "params": { "threshold": 0.1 }
+        },
+        {
+            "rule_id": "check_goldens_national",
+            "description": "Validates national and state-level 2000+ data against its golden summary report.",
+            "validator": "GOLDENS_CHECK",
+            "params": {
+                "golden_files": ["../../../../golden_data/golden_summary_report_national.csv"],
+                "input_files": ["../../input0/genmcf/summary_report.csv"]
+            }
+        },
+        {
+            "rule_id": "check_goldens_before_2000",
+            "description": "Validates data before 2000 against its golden summary report.",
+            "validator": "GOLDENS_CHECK",
+            "params": {
+                "golden_files": ["../../../../golden_data/golden_summary_report_before_2000.csv"],
+                "input_files": ["../../input1/genmcf/summary_report.csv"]
+            }
+        },
+        {
+            "rule_id": "check_goldens_after_2000",
+            "description": "Validates county-level 2000+ data against its golden summary report.",
+            "validator": "GOLDENS_CHECK",
+            "params": {
+                "golden_files": ["../../../../golden_data/golden_summary_report_after_2000.csv"],
+                "input_files": ["../../input2/genmcf/summary_report.csv"]
+            }
+        },
+        {
+            "rule_id": "Check_goldens_output_csv_before_2000",
+            "description": "Verifies the generated output CSV data matches established critical golden records",
+            "validator": "GOLDENS_CHECK",
+            "params": {
+                "golden_files": ["../../../../golden_data/golden_observations_before_2000.csv"],
+                "input_files": ["../../../../output/USA_Population_Count_by_Race_before_2000.csv"]
+            }
+        },
+        {
+            "rule_id": "Check_goldens_output_csv_after_2000",
+            "description": "Verifies the generated output CSV data matches established critical golden records",
+            "validator": "GOLDENS_CHECK",
+            "params": {
+                "golden_files": ["../../../../golden_data/golden_observations_after_2000.csv"],
+                "input_files": ["../../../../output/USA_Population_Count_by_Race_county_after_2000.csv"]
+            }
+        },
+        {
+            "rule_id": "Check_goldens_output_csv_state",
+            "description": "Verifies the generated output CSV data matches established critical golden records",
+            "validator": "GOLDENS_CHECK",
+            "params": {
+                "golden_files": ["../../../../golden_data/golden_observations_national.csv"],
+                "input_files": ["../../../../output/USA_Population_Count_by_Race_National_state_2000.csv"]
+            }
+        }
+    ]
+}
+```
+
+##  **5\. Activating the Config in `manifest.json`**
+
+The auto-refresh pipeline will only notice your new rules if you explicitly link them in your main `manifest.json` file.
+
+Add the `validation_config_file` parameter pointing to your file inside the `config_overrides` object:
+
+JSON
+
+```
+{
+    "import_spec": {
+        "name": "Your_Import_Name_Here"
+    },
+    "config_overrides": {
+        "validation_config_file": "validation_config.json"
+    }
+}
+```
+
+*(Note: Remember that StatVar updates in this environment are driven systematically via the manifest configuration flags rather than manual file renaming.)*
+
+## **6\. How to Read Validation Failures**
+
+If a future data refresh breaks these rules, the pipeline will fail, and a report JSON will be generated.
+
+* **If `Check_goldens_summary_report` fails:** It means a StatVar or a specific geographic series has unexpectedly disappeared from the pipeline.  
+* **If `Check_goldens_output_csv` fails:** The specific rows of data present in your baseline file but missing in the new run will be explicitly listed in the output log.
+
+> **When to update golden files:** Only regenerate golden files using the scripts in if a data change is intentional (e.g., source data deprecation, structural schema updates). Always have the changes reviewed by a peer before committing new golden baselines.
+

tools/import_validation/troubleshooting_guides/Major Deletions.md

@@ -0,0 +1,113 @@
+# **Validating Deletions for the WorldDevelopmentIndicators Import**
+
+To address frequent failures in the **WorldDevelopmentIndicators** import caused by source data deletions—occurring 4–5 times already this year—you must validate these deletions. The resolution process involves : examining the cloud job logs, replicating the import run on a local machine, and identifying the cause of record deletions to implement a permanent fix.
+
+## **Check the Cloud Logs to See What Failed**
+
+First, we need to find out why the job failed by looking at the cloud logs and the error messages.
+
+1. **Check the Cloud Batch Job:**  
+   Go to the Cloud Batch Jobs console and find the latest run for the import.  
+   * Here is the job I looked at for this import: Cloud Batch Job: `worlddevelopmentindicators-1781002802` (Project: `datcom-import-automation-prod`, Region: `us-central1`)  
+2. **Look for Errors in the Logs:**  
+   Even if the job status says "Succeeded," it might still have errors. Look through the logs (you can use the errors filter to jump straight to them). In my case, the job failed some validation checks because of deletions.  
+   * **Error Message:** Found 0.87% deleted records, which is over the threshold of 0%.  
+3. **Check the Production Bucket:**  
+   Next, head over to the Datacom production cloud bucket to find the specific files for this job run. Navigate down the correct path for the import:  
+   Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/world_bank/wdi/WorldDevelopmentIndicators`
+   * Go to the folder with the latest timestamp as the job you just looked at: 2026\_06\_09T04\_07\_31\_200635\_07\_00  
+4. **Review the Validation Files:**  
+   Go into the input0/validation/ folder. Here, you want to look at two specific files:  
+   * **validation\_output.csv:** This tells you the exact reason the checks failed. In my case, 3 checks passed, but the check\_deleted\_records\_percent failed because 3,908 records were deleted. (See table below)  
+      * Location: Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/world_bank/wdi/WorldDevelopmentIndicators/2026_06_09T04_07_31_200635_07_00/input0/validation/validation_output.csv`
+   * **Nodes\_deleted.mcf:** This file shows you the actual list of records that were deleted.  
+      * Location: Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/world_bank/wdi/WorldDevelopmentIndicators/2026_06_09T04_07_31_200635_07_00/input0/validation/obs_diff_log.csv`
+
+## **Run the Import Locally**
+
+Now, we need to run the whole import process on our own computer. This helps us confirm if the deletions are really happening, or if it was just a weird glitch in the cloud job.
+
+1. **Clone the GitHub Repository:**  
+   Clone the Datacommons data repository to your machine: [https://github.com/datacommonsorg/data.git](https://github.com/datacommonsorg/data.git). Once cloned, navigate to the exact same import path we looked at in the bucket.  
+2. **Run the Scripts:**  
+   Check the [manifest.json](https://github.com/niveditasing/data/blob/e28894cc4701578d43d937bd484d91d8333ffa07/scripts/world_bank/wdi/manifest.json) file in that folder to understand how the job runs, and manually run the scripts and processes it outlines.
+
+**Note**: The manifest.json file defines the number of output and TMCF files generated by the import; the presence of directories like input0 or input1 in the bucket is a direct result of these manifest specifications.
+
+
+
+3. **Generate the MCF File:**  
+   Run the lint and genmcf tests to generate the table.mcf file, which we will need for the differ tool. Run these commands:  
+   * java \-jar java-jar.jar lint output.csv output.tmcf  
+   * java \-jar java-jar.jar genmcf output.csv output.tmcf  
+4. **Get the Previous Data:**  
+   To run the differ tool, we need to compare the table.mcf file we just created (the current data) with the table.mcf file from the previous successful run.  
+   * Find the timestamp of the last successful run by checking the `latest_version.txt` file in the cloud bucket. GCS Location: Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/world_bank/wdi/WorldDevelopmentIndicators/latest_version.txt`
+   * Go to that timestamp's folder in the bucket and download its table.mcf file.  
+5. **Run the Differ Tool:**  
+   Now, run this command to compare the two files:  
+   python3 import\_differ.py \--current\_data= \--previous\_data= \--output\_location= \--file\_format=mcf \--runner\_mode=local  
+6. **Confirm the Results:**  
+   Open the dc\_generated/nodes\_deleted.mcf folder and the validation\_output.csv on your local machine. If they match the results we saw in the cloud bucket during Phase 1, we know the results are accurate\!
+
+## **Validate Deletions from the Source**
+
+Now that we know the deletions are real, we need to validate them against the original data source to confirm the source actually removed the data.
+
+1. **Locate the Textproto File:** To find out exactly where the data came from, check the `textproto` file for this import. For WorldDevelopmentIndicators, the file path is: `google3/datacommons/import/mcf/manifest/international_stats/WorldDevelopmentIndicators.textproto`  
+   * You can search for this file using Google Code Search: [WorldDevelopmentIndicators.textproto Link](https://source.corp.google.com/piper///depot/google3/datacommons/import/mcf/manifest/international_stats/WorldDevelopmentIndicators.textproto;l=2?q=worlddevelopmentindicators&sq=package:piper%20file:%2F%2Fdepot%2Fgoogle3%20-file:google3%2Fexperimental)  
+2. **Find the Source URL:** Open the `textproto` file and look for the line that says: `provenance_url: "https://datatopics.worldbank.org/world-development-indicators/"` This URL tells us exactly where the data is downloaded from.  
+3. **Navigate the Source Website:** Open that source URL. On the World Bank website, click on the **Explore Data** option, and then click on **Access Data**. This will allow you to see their entire dataset.  
+4. **Manually Verify Deleted Records:** check at your `nodes_deleted.mcf` file and pick out 5 to 6 specific deleted records. Go back to the World Bank website and manually filter the data by entering the parameters for those specific records.  
+   * *If you do not get any data for them (or the data is missing), this confirms that the data was accurately deleted from the source.*
+
+   *Example :* The job failed because some specific data points present in the previous production version are missing in the current version.
+
+* **Source Deletion:** The record  has been removed from the source.  
+1) [Source Screenshot](https://screenshot.googleplex.com/3oK73RXvox3xeTM) | [Differ Screenshot](https://screenshot.googleplex.com/BWYDfWvTNZoixSE)   
+2) [Source Screenshot](https://screenshot.googleplex.com/8znJ3XadmGGRmdL) | [Differ Screenshot](https://screenshot.googleplex.com/4JKbhAxnJz6SGbJ)  
+3) [Source Screenshot](https://screenshot.googleplex.com/5Nw3dA9a3iH7GAM) | [Differ Screenshot](https://screenshot.googleplex.com/7ovdMycUe7tiADw)   
+* These deletions are confirmed as intentional source-side changes from the World Bank’s April 2026 update.
+
+  WDI April 8, 2026 Changelog [Screenshot](https://screenshot.googleplex.com/497zku465XJzsJt) | [Link](https://datatopics.worldbank.org/world-development-indicators/release-note/apr-2026.html)
+
+5. In my case, the amount of data getting deleted from the source was very huge. When this happens, we cannot just ignore it. We need to:  
+   * **Create a validation error document:** Document the massive deletion properly so there is a clear record of why the job threshold failed and what was removed. [BLS\_CES\_State\_20\_04\_2026](https://docs.google.com/document/d/1QK9pMoSFR7STb78aFgN_NOupcJzU_IdWp4tyIvzrwis/edit?resourcekey=0-VWdGo38x-FfuyBd8xGu68w&tab=t.0#heading=h.shg2oxa69t3d)  
+   * **Store the data historically:** Keep a record of the deleted data. (need approval from core team)  
+6. Check for the affected SVs deleted that we find out by taking unique deleted SVs  from the differ & check in BigQuery (Table: `datcom-store.dc_kg_latest.NLStatVars`) if these Svs are present in the NL SVs table.  
+7. Because the deletions are minor & from the source the next would be store Historical data & because it had recurring failure  golden checks \+ threshold increase as per history deletions will also be implemented ( All these steps must be mentioned in the validation error document because we need core team approval to store historical & update the latest\_version.txt)
+
+   ## **How to implement golden checks?**
+
+   Consult the following resource for instructions on incorporating goldens into your import process: [Implementation Guide: Golden Set Validations](https://docs.google.com/document/d/14Fpe5e9jSzzJ5_QTcqQ1AFb2oqjJzXuc1_BmQCX-uns/edit?tab=t.0#heading=h.4n225wbxsh8v)
+
+   ## **How to store historical data ?**
+
+1.  resolve **validation errors**  
+   Preserve the deleted data by storing it in a historical file, which should then be copied to the CNS. Below are the steps:   
+   1. Verify the production version of the import via [Data Commons/Version](https://datacommons.org/version).  
+      2. Locate and download the production CSV from the storage bucket, matching the date specified in the Data Commons version.  
+      3. Run the script in your local environment, then utilize Python code to perform a difference comparison between the latest and production output CSVs.  
+      4. Run the differ on output & historical data table\_mcf\_nodes.mcf files together & ensure no deletion flags are there.  
+      5. Once the deleted rows are identified & no deletions with this historical file through the comparison, save them to a file and upload it to CNS as a historical record, path shown below.
+
+```
+mcf_proto_url: "/cns/jv-d/home/datcom/v3_resolved_mcf/us_bls/ces/state/latest/historical_data/graph.tfrecord@1.gz"
+ table {
+   mapping_path: "/cns/jv-d/home/datcom/v3_mcf/wdi/WorldDevelopmentIndicators/historical_data/worldbank.tmcf"
+   csv_path: "/cns/jv-d/home/datcom/v3_mcf/wdi/WorldDevelopmentIndicators/historical_data/*.csv"
+ }
+```
+
+   b. To stop these “Deleted” flags from appearing as errors, **the latest\_version.txt file must be updated only after the core team approves.**  This ensures the differ recognizes the change as deliberate to the dataset rather than a data loss error. 
+
+```
+experimental/users/ajaits/datcom/scripts/import_info.sh -i WorldDevelopmentIndicator-set_latest 2026_04_20_02_42_50_536488_08_00 -note 'details of deletion analysis in b/500945912 reviewed by: <ldap of the approver>'
+```
+
+   C. Rerun the pipeline to verify it finishes without issues and check that all tests in *validation\_output.csv* have passed.
+
+
+
+
+