diff --git a/tools/import_validation/troubleshooting_guides/Golden Checks.md b/tools/import_validation/troubleshooting_guides/Golden Checks.md new file mode 100644 index 0000000000..ec0c23d67d --- /dev/null +++ b/tools/import_validation/troubleshooting_guides/Golden Checks.md @@ -0,0 +1,199 @@ +# **Implementation Guide: Golden Set Validations** + +## **1\. Overview** + +Due to recurring data deletion failures, we protect our imports using **Golden Set Validations (`GOLDENS_CHECK`)**. This guide helps understand how to generate "golden files" (baselines of expected data) and configure the automated validation system to prevent accidental data regressions. + +Currently, to mitigate issues related to data loss, we implement two primary validations supported by specific golden baselines: + +1. **`Check_goldens_output_csv`**: Verifies the final output data against an established baseline. +2. **`Check_goldens_summary_report`**: Validates structural metrics (like number of places and dates) against a summary baseline. +3. More regarding golden checks can be seen : [github link](https://github.com/datacommonsorg/data/blob/master/tools/import_validation/Validations.md) + +## **2\. Directory Architecture** + +To implement these checks, your import script folder must contain the following file structure: + +Plaintext + +``` +your_import_folder/ +│ +├── manifest.json # Main import configuration +├── validation_config.json # Custom validation rules +│ +└── golden_data/ # Folder holding your baseline files + ├── golden_summary_report.csv # Generated golden summary + └── golden_observations.csv # Generated golden output data +``` + +## **3\. Step-by-Step: How to Create Golden Files** + +Golden files are created by extracting a snapshot of known-good data using the script `validator_goldens.py`. Run these commands from your terminal inside your import directory. + +### **Step 3.1: Generate the Summary Report Golden File** + +This step tracks metadata properties like Statistical Variables (StatVars), the number of places, and date ranges to ensure future runs don't accidentally drop the entire series. + +From the data/tools/import\_validation/validator\_goldens.py directory, execute the following command: + +``` +python3 validator_goldens.py --validate_goldens_input=summary_report.csv --generate_goldens=golden_data/golden_summary_report.csv --generate_goldens_property_sets="StatVar|NumPlaces|MinDate|MeasurementMethods|Units|ScalingFactors|observationPeriods" +``` + +### **Step 3.2: Generate the Output Data Golden File (only use if needed)** + +This step targets critical combinations of highly utilized StatVars and top geographical regions to ensure key data points are always preserved. + +From the data/tools/import\_validation/validator\_goldens.py directory, execute the following command: + +``` +python3 validator_goldens.py --validate_goldens_input=output.csv --generate_goldens=golden_data/golden_observations.csv --goldens_must_include="observationAbout:gs://unresolved_mcf/import_validation/top_100k_places.csv" --generate_goldens_property_sets="observationAbout" +``` + +## **4\. Configuring `validation_config.json`** + +Create a file named `validation_config.json` in your import script directory. Paste the configuration below. + +This configuration does two things: + +* Overrides the default deletion tolerance rule (`check_deleted_records_percent`) to a threshold as per history deletions & current deletions should not be more than **0.1%**. +* Activates the two required golden check rules pointing to the files you created in Section 3\. + +JSON + +``` +{ + "schema_version": "1.0", + "rules": [ + { + "rule_id": "check_deleted_records_percent", + "description": "Strictly enforce historical deletion average threshold of 0.1%", + "validator": "DELETED_RECORDS_PERCENT", + "params": { + "threshold": 0.1 + } + }, + { + "rule_id": "check_goldens_summary_report", + "description": "Validates summary_report.csv against the golden summary data", + "validator": "GOLDENS_CHECK", + "params": { + "golden_files": ["golden_data/golden_summary_report.csv"] + } + }, + { + "rule_id": "check_goldens_output_csv", + "description": "Verifies the generated output CSV data matches established critical golden records", + "validator": "GOLDENS_CHECK", + "params": { + "golden_files": ["golden_data/golden_observations.csv"], + "input_files": ["output/observations.csv"] + } + } + ] +} +``` + +## **Below is how the validation\_config.json is structured for imports with multiple output CSVs:** + +JSON + +``` +{ + "schema_version": "1.0", + "rules": [ + { + "rule_id": "check_deleted_records_percent", + "description": "Checks that the percentage of deleted records for the entire import is within threshold.", + "validator": "DELETED_RECORDS_PERCENT", + "params": { "threshold": 0.1 } + }, + { + "rule_id": "check_goldens_national", + "description": "Validates national and state-level 2000+ data against its golden summary report.", + "validator": "GOLDENS_CHECK", + "params": { + "golden_files": ["../../../../golden_data/golden_summary_report_national.csv"], + "input_files": ["../../input0/genmcf/summary_report.csv"] + } + }, + { + "rule_id": "check_goldens_before_2000", + "description": "Validates data before 2000 against its golden summary report.", + "validator": "GOLDENS_CHECK", + "params": { + "golden_files": ["../../../../golden_data/golden_summary_report_before_2000.csv"], + "input_files": ["../../input1/genmcf/summary_report.csv"] + } + }, + { + "rule_id": "check_goldens_after_2000", + "description": "Validates county-level 2000+ data against its golden summary report.", + "validator": "GOLDENS_CHECK", + "params": { + "golden_files": ["../../../../golden_data/golden_summary_report_after_2000.csv"], + "input_files": ["../../input2/genmcf/summary_report.csv"] + } + }, + { + "rule_id": "Check_goldens_output_csv_before_2000", + "description": "Verifies the generated output CSV data matches established critical golden records", + "validator": "GOLDENS_CHECK", + "params": { + "golden_files": ["../../../../golden_data/golden_observations_before_2000.csv"], + "input_files": ["../../../../output/USA_Population_Count_by_Race_before_2000.csv"] + } + }, + { + "rule_id": "Check_goldens_output_csv_after_2000", + "description": "Verifies the generated output CSV data matches established critical golden records", + "validator": "GOLDENS_CHECK", + "params": { + "golden_files": ["../../../../golden_data/golden_observations_after_2000.csv"], + "input_files": ["../../../../output/USA_Population_Count_by_Race_county_after_2000.csv"] + } + }, + { + "rule_id": "Check_goldens_output_csv_state", + "description": "Verifies the generated output CSV data matches established critical golden records", + "validator": "GOLDENS_CHECK", + "params": { + "golden_files": ["../../../../golden_data/golden_observations_national.csv"], + "input_files": ["../../../../output/USA_Population_Count_by_Race_National_state_2000.csv"] + } + } + ] +} +``` + +## **5\. Activating the Config in `manifest.json`** + +The auto-refresh pipeline will only notice your new rules if you explicitly link them in your main `manifest.json` file. + +Add the `validation_config_file` parameter pointing to your file inside the `config_overrides` object: + +JSON + +``` +{ + "import_spec": { + "name": "Your_Import_Name_Here" + }, + "config_overrides": { + "validation_config_file": "validation_config.json" + } +} +``` + +*(Note: Remember that StatVar updates in this environment are driven systematically via the manifest configuration flags rather than manual file renaming.)* + +## **6\. How to Read Validation Failures** + +If a future data refresh breaks these rules, the pipeline will fail, and a report JSON will be generated. + +* **If `Check_goldens_summary_report` fails:** It means a StatVar or a specific geographic series has unexpectedly disappeared from the pipeline. +* **If `Check_goldens_output_csv` fails:** The specific rows of data present in your baseline file but missing in the new run will be explicitly listed in the output log. + +> **When to update golden files:** Only regenerate golden files using the scripts in if a data change is intentional (e.g., source data deprecation, structural schema updates). Always have the changes reviewed by a peer before committing new golden baselines. + diff --git a/tools/import_validation/troubleshooting_guides/Major Deletions.md b/tools/import_validation/troubleshooting_guides/Major Deletions.md new file mode 100644 index 0000000000..8a9ba38559 --- /dev/null +++ b/tools/import_validation/troubleshooting_guides/Major Deletions.md @@ -0,0 +1,113 @@ +# **Validating Deletions for the WorldDevelopmentIndicators Import** + +To address frequent failures in the **WorldDevelopmentIndicators** import caused by source data deletions—occurring 4–5 times already this year—you must validate these deletions. The resolution process involves : examining the cloud job logs, replicating the import run on a local machine, and identifying the cause of record deletions to implement a permanent fix. + +## **Check the Cloud Logs to See What Failed** + +First, we need to find out why the job failed by looking at the cloud logs and the error messages. + +1. **Check the Cloud Batch Job:** + Go to the Cloud Batch Jobs console and find the latest run for the import. + * Here is the job I looked at for this import: Cloud Batch Job: `worlddevelopmentindicators-1781002802` (Project: `datcom-import-automation-prod`, Region: `us-central1`) +2. **Look for Errors in the Logs:** + Even if the job status says "Succeeded," it might still have errors. Look through the logs (you can use the errors filter to jump straight to them). In my case, the job failed some validation checks because of deletions. + * **Error Message:** Found 0.87% deleted records, which is over the threshold of 0%. +3. **Check the Production Bucket:** + Next, head over to the Datacom production cloud bucket to find the specific files for this job run. Navigate down the correct path for the import: + Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/world_bank/wdi/WorldDevelopmentIndicators` + * Go to the folder with the latest timestamp as the job you just looked at: 2026\_06\_09T04\_07\_31\_200635\_07\_00 +4. **Review the Validation Files:** + Go into the input0/validation/ folder. Here, you want to look at two specific files: + * **validation\_output.csv:** This tells you the exact reason the checks failed. In my case, 3 checks passed, but the check\_deleted\_records\_percent failed because 3,908 records were deleted. (See table below) + * Location: Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/world_bank/wdi/WorldDevelopmentIndicators/2026_06_09T04_07_31_200635_07_00/input0/validation/validation_output.csv` + * **Nodes\_deleted.mcf:** This file shows you the actual list of records that were deleted. + * Location: Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/world_bank/wdi/WorldDevelopmentIndicators/2026_06_09T04_07_31_200635_07_00/input0/validation/obs_diff_log.csv` + +## **Run the Import Locally** + +Now, we need to run the whole import process on our own computer. This helps us confirm if the deletions are really happening, or if it was just a weird glitch in the cloud job. + +1. **Clone the GitHub Repository:** + Clone the Datacommons data repository to your machine: [https://github.com/datacommonsorg/data.git](https://github.com/datacommonsorg/data.git). Once cloned, navigate to the exact same import path we looked at in the bucket. +2. **Run the Scripts:** + Check the [manifest.json](https://github.com/niveditasing/data/blob/e28894cc4701578d43d937bd484d91d8333ffa07/scripts/world_bank/wdi/manifest.json) file in that folder to understand how the job runs, and manually run the scripts and processes it outlines. + +**Note**: The manifest.json file defines the number of output and TMCF files generated by the import; the presence of directories like input0 or input1 in the bucket is a direct result of these manifest specifications. + + + +3. **Generate the MCF File:** + Run the lint and genmcf tests to generate the table.mcf file, which we will need for the differ tool. Run these commands: + * java \-jar java-jar.jar lint output.csv output.tmcf + * java \-jar java-jar.jar genmcf output.csv output.tmcf +4. **Get the Previous Data:** + To run the differ tool, we need to compare the table.mcf file we just created (the current data) with the table.mcf file from the previous successful run. + * Find the timestamp of the last successful run by checking the `latest_version.txt` file in the cloud bucket. GCS Location: Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/world_bank/wdi/WorldDevelopmentIndicators/latest_version.txt` + * Go to that timestamp's folder in the bucket and download its table.mcf file. +5. **Run the Differ Tool:** + Now, run this command to compare the two files: + python3 import\_differ.py \--current\_data= \--previous\_data= \--output\_location= \--file\_format=mcf \--runner\_mode=local +6. **Confirm the Results:** + Open the dc\_generated/nodes\_deleted.mcf folder and the validation\_output.csv on your local machine. If they match the results we saw in the cloud bucket during Phase 1, we know the results are accurate\! + +## **Validate Deletions from the Source** + +Now that we know the deletions are real, we need to validate them against the original data source to confirm the source actually removed the data. + +1. **Locate the Textproto File:** To find out exactly where the data came from, check the `textproto` file for this import. For WorldDevelopmentIndicators, the file path is: `google3/datacommons/import/mcf/manifest/international_stats/WorldDevelopmentIndicators.textproto` + * You can search for this file using Google Code Search: [WorldDevelopmentIndicators.textproto Link](https://source.corp.google.com/piper///depot/google3/datacommons/import/mcf/manifest/international_stats/WorldDevelopmentIndicators.textproto;l=2?q=worlddevelopmentindicators&sq=package:piper%20file:%2F%2Fdepot%2Fgoogle3%20-file:google3%2Fexperimental) +2. **Find the Source URL:** Open the `textproto` file and look for the line that says: `provenance_url: "https://datatopics.worldbank.org/world-development-indicators/"` This URL tells us exactly where the data is downloaded from. +3. **Navigate the Source Website:** Open that source URL. On the World Bank website, click on the **Explore Data** option, and then click on **Access Data**. This will allow you to see their entire dataset. +4. **Manually Verify Deleted Records:** check at your `nodes_deleted.mcf` file and pick out 5 to 6 specific deleted records. Go back to the World Bank website and manually filter the data by entering the parameters for those specific records. + * *If you do not get any data for them (or the data is missing), this confirms that the data was accurately deleted from the source.* + + *Example :* The job failed because some specific data points present in the previous production version are missing in the current version. + +* **Source Deletion:** The record has been removed from the source. +1) [Source Screenshot](https://screenshot.googleplex.com/3oK73RXvox3xeTM) | [Differ Screenshot](https://screenshot.googleplex.com/BWYDfWvTNZoixSE) +2) [Source Screenshot](https://screenshot.googleplex.com/8znJ3XadmGGRmdL) | [Differ Screenshot](https://screenshot.googleplex.com/4JKbhAxnJz6SGbJ) +3) [Source Screenshot](https://screenshot.googleplex.com/5Nw3dA9a3iH7GAM) | [Differ Screenshot](https://screenshot.googleplex.com/7ovdMycUe7tiADw) +* These deletions are confirmed as intentional source-side changes from the World Bank’s April 2026 update. + + WDI April 8, 2026 Changelog [Screenshot](https://screenshot.googleplex.com/497zku465XJzsJt) | [Link](https://datatopics.worldbank.org/world-development-indicators/release-note/apr-2026.html) + +5. In my case, the amount of data getting deleted from the source was very huge. When this happens, we cannot just ignore it. We need to: + * **Create a validation error document:** Document the massive deletion properly so there is a clear record of why the job threshold failed and what was removed. [BLS\_CES\_State\_20\_04\_2026](https://docs.google.com/document/d/1QK9pMoSFR7STb78aFgN_NOupcJzU_IdWp4tyIvzrwis/edit?resourcekey=0-VWdGo38x-FfuyBd8xGu68w&tab=t.0#heading=h.shg2oxa69t3d) + * **Store the data historically:** Keep a record of the deleted data. (need approval from core team) +6. Check for the affected SVs deleted that we find out by taking unique deleted SVs from the differ & check in BigQuery (Table: `datcom-store.dc_kg_latest.NLStatVars`) if these Svs are present in the NL SVs table. +7. Because the deletions are minor & from the source the next would be store Historical data & because it had recurring failure golden checks \+ threshold increase as per history deletions will also be implemented ( All these steps must be mentioned in the validation error document because we need core team approval to store historical & update the latest\_version.txt) + + ## **How to implement golden checks?** + + Consult the following resource for instructions on incorporating goldens into your import process: [Implementation Guide: Golden Set Validations](https://docs.google.com/document/d/14Fpe5e9jSzzJ5_QTcqQ1AFb2oqjJzXuc1_BmQCX-uns/edit?tab=t.0#heading=h.4n225wbxsh8v) + + ## **How to store historical data ?** + +1. resolve **validation errors** + Preserve the deleted data by storing it in a historical file, which should then be copied to the CNS. Below are the steps: + 1. Verify the production version of the import via [Data Commons/Version](https://datacommons.org/version). + 2. Locate and download the production CSV from the storage bucket, matching the date specified in the Data Commons version. + 3. Run the script in your local environment, then utilize Python code to perform a difference comparison between the latest and production output CSVs. + 4. Run the differ on output & historical data table\_mcf\_nodes.mcf files together & ensure no deletion flags are there. + 5. Once the deleted rows are identified & no deletions with this historical file through the comparison, save them to a file and upload it to CNS as a historical record, path shown below. + +``` +mcf_proto_url: "/cns/jv-d/home/datcom/v3_resolved_mcf/us_bls/ces/state/latest/historical_data/graph.tfrecord@1.gz" + table { + mapping_path: "/cns/jv-d/home/datcom/v3_mcf/wdi/WorldDevelopmentIndicators/historical_data/worldbank.tmcf" + csv_path: "/cns/jv-d/home/datcom/v3_mcf/wdi/WorldDevelopmentIndicators/historical_data/*.csv" + } +``` + + b. To stop these “Deleted” flags from appearing as errors, **the latest\_version.txt file must be updated only after the core team approves.** This ensures the differ recognizes the change as deliberate to the dataset rather than a data loss error. + +``` +experimental/users/ajaits/datcom/scripts/import_info.sh -i WorldDevelopmentIndicator-set_latest 2026_04_20_02_42_50_536488_08_00 -note 'details of deletion analysis in b/500945912 reviewed by: ' +``` + + C. Rerun the pipeline to verify it finishes without issues and check that all tests in *validation\_output.csv* have passed. + + + + + diff --git a/tools/import_validation/troubleshooting_guides/Minor Deletions.md b/tools/import_validation/troubleshooting_guides/Minor Deletions.md new file mode 100644 index 0000000000..0d730ad97a --- /dev/null +++ b/tools/import_validation/troubleshooting_guides/Minor Deletions.md @@ -0,0 +1,85 @@ +# **Validating Deletions for the EurostatData\_Fertility Import** + +To address occasional, low-impact source data deletions in the import **EurostatData\_Fertility**, you must validate that these minor data drops are expected and benign. The resolution process involves: examining the cloud job logs to confirm the deletion percentage is small, verifying the scope of the missing records, and adjusting the validation thresholds to allow the job to succeed. + +## **Check the Cloud Logs to See What Failed** + +First, we need to find out why the job failed by looking at the cloud logs and the error messages. + +1. **Check the Cloud Batch Job:** + Go to the Cloud Batch Jobs console and find the latest run for the import. + * Here is the job I looked at for this import: Cloud Batch Job: `eurostatdata-fertility-1781582401` (Project: `datcom-import-automation-prod`, Region: `us-central1`) +2. **Look for Errors in the Logs:** + Even if the job status says "Succeeded," it might still have errors. Look through the logs (you can use the errors filter to jump straight to them). In my case, the job failed some validation checks because of deletions. + * **Error Message:** Found 0.06% deleted records, which is over the threshold of 0%. +3. **Check the Production Bucket:** + Next, head over to the Datacom production cloud bucket to find the specific files for this job run. Navigate down the correct path for the import: + Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/eurostat/regional_statistics_by_nuts/fertility_rate_mother_age/EurostatData_Fertility/` + * Go to the folder with the latest timestamp as the job you just looked at: 2026\_06\_15T04\_07\_31\_200635\_07\_00 +4. **Review the Validation Files:** + Go into the input0/validation/ folder. Here, you want to look at two specific files: + * **validation\_output.csv:** This tells you the exact reason the checks failed. In my case, 3 checks passed, but the check\_deleted\_records\_percent failed because 36 records were deleted. (See table below) + * Location: Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/eurostat/regional_statistics_by_nuts/fertility_rate_mother_age/EurostatData_Fertility/2026_06_15T21_03_10_391174_07_00/input0/validation/validation_output.csv` + * **obs\_diff\_log.csv:** This file shows you the actual list of records that were deleted. + * Location: Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/eurostat/regional_statistics_by_nuts/fertility_rate_mother_age/EurostatData_Fertility/2026_06_15T21_03_10_391174_07_00/input0/validation/obs_diff_log.csv` + +## **Run the Import Locally** + +Now, we need to run the whole import process on our own computer. This helps us confirm if the deletions are really happening, or if it was just a weird glitch in the cloud job. + +1. **Clone the GitHub Repository:** + Clone the Datacommons data repository to your machine: [https://github.com/datacommonsorg/data.git](https://github.com/datacommonsorg/data.git). Once cloned, navigate to the exact same import path we looked at in the bucket. +2. **Run the Scripts:** + Check the [manifest.json](https://github.com/datacommonsorg/data/blob/master/scripts/eurostat/regional_statistics_by_nuts/fertility_rate_mother_age/manifest.json) file in that folder to understand how the job runs, and manually run the scripts and processes it outlines. + +**Note**: The manifest.json file defines the number of output and TMCF files generated by the import; the presence of directories like input0 or input1 in the bucket is a direct result of these manifest specifications. + + + +3. **Generate the MCF File:** + Run the lint and genmcf tests to generate the table.mcf file, which we will need for the differ tool. Run these commands: + * java \-jar java-jar.jar lint output.csv output.tmcf + * java \-jar java-jar.jar genmcf output.csv output.tmcf +4. **Get the Previous Data:** + To run the differ tool, we need to compare the table.mcf file we just created (the current data) with the table.mcf file from the previous successful run. + * Find the timestamp of the last successful run by checking the `latest_version.txt` file in the cloud bucket. GCS Location: Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/eurostat/regional_statistics_by_nuts/fertility_rate_mother_age/EurostatData_Fertility/latest_version.txt` + * Go to that timestamp's folder in the bucket and download its table.mcf file. +5. **Run the Differ Tool:** + Now, run this command to compare the two files: + python3 import\_differ.py \--current\_data= \--previous\_data= \--output\_location= \--file\_format=mcf \--runner\_mode=local +6. **Confirm the Results:** + Open the dc\_generated/obs\_diff\_logs.csv folder and the validation\_output.csv on your local machine. If they match the results we saw in the cloud bucket during Phase 1, we know the results are accurate\! + +## **Validate Deletions from the Source** + +Now that we know the deletions are real, we need to validate them against the original data source to confirm the source actually removed the data. + +1. **Locate the Textproto File:** To find out exactly where the data came from, check the `textproto` file for this import. For WorldDevelopmentIndicators, the file path is: `datacommons/import/mcf/manifest/international_stats/EurostatData_Fertility.textproto` + * You can search for this file using Google Code Search: [EurostatData\_Fertility.textproto Link](https://source.corp.google.com/piper///depot/google3/datacommons/import/mcf/manifest/international_stats/EurostatData_Fertility.textproto;l=2?q=eurostatdata&sq=package:piper%20file:%2F%2Fdepot%2Fgoogle3%20-file:google3%2Fexperimental) +2. **Find the Source URL:** Open the `textproto` file and look for the line that says: `provenance_url: "https://ec.europa.eu/eurostat/databrowser/view/demo_r_find3/default/table?lang=en"` This URL tells us exactly where the data is downloaded from. +3. **Navigate the Source Website:** Open that source URL. On the World Bank website, click on the **Explore Data** option, and then click on **Access Data**. This will allow you to see their entire dataset. +4. **Manually Verify Deleted Records:** check at your `obs_diff_logs.csv` file and pick out 5 to 6 specific deleted records. Go back to the EurostatData website and manually filter the data by entering the parameters for those specific records. + * *If you do not get any data for them (or the data is missing), this confirms that the data was accurately deleted from the source.* + + *Example :* The job failed because some specific data points present in the previous production version are missing in the current version. + +* **Source Deletion:** The record has been removed from the source. + + [Source Screenshot](https://screenshot.googleplex.com/3VNMybHoDVt6kFR) | [Differ Screenshot](https://screenshot.googleplex.com/B25DugjwV2js2TJ) + + +* These deletions are confirmed as intentional source-side changes from the Eurostat Official Website. + +5. In my case, the amount of data getting deleted from the source was very less. When this happens, we just ignore it. We need to: + * **Create a validation error document:** Document the deletion properly so there is a clear record of why the job threshold failed and what was removed. + * Add goldens checks & increase minor threshold +6. Check for the affected SVs deleted that we find out by taking unique deleted SVs from the differ & check in [bigquery](https://screenshot.googleplex.com/4pprS7vyu4Cba6C) if these SVs are present in the NL SVs table. + + + ## **How to implement golden checks?** + + Consult the following resource for instructions on incorporating goldens into your import process: [Implementation Guide: Golden Set Validations](https://docs.google.com/document/d/14Fpe5e9jSzzJ5_QTcqQ1AFb2oqjJzXuc1_BmQCX-uns/edit?tab=t.0#heading=h.4n225wbxsh8v) + + + + \ No newline at end of file diff --git a/tools/import_validation/troubleshooting_guides/Most Recurring Failures.md b/tools/import_validation/troubleshooting_guides/Most Recurring Failures.md new file mode 100644 index 0000000000..9529363394 --- /dev/null +++ b/tools/import_validation/troubleshooting_guides/Most Recurring Failures.md @@ -0,0 +1,62 @@ +# Analysis of Recurring Production Job Failures + +## 1\. Overview + +This document outlines several recurring production job failures, some of these failures are caused by minor or temporary data deletions. By increasing the threshold, we can prevent these jobs from failing due to non-permanent issues. However, before adjusting these thresholds, we will implement new validation rules to ensure that crucial data is never lost, even during minor deletions. +More details can be found here [DC - Imports Execution Learnings ](https://docs.google.com/spreadsheets/d/1lAm-EPB6o9U9btQHbdRruGhx_DFerlhQF43Ba6skI4Q/edit?gid=1651779522#gid=1651779522) + +## 2\. Most Recurring Job Failures + +In the previous quarter, the following production jobs experienced the highest frequency of failures: + +| Import name | Occurrences | Bug IDs | +| :---- | :---- | :---- | +| BLS\_CES\_State | 4 | [b/482946661](http://b/482946661), [b/500945912](http://b/500945912), [b/502090898](http://b/502090898) | +| USCensusPEP\_Sex | 5 | [b/472605922](http://b/472605922), [b/478186511](http://b/478186511), [b/483219293](http://b/483219293) | +| WorldDevelopmentIndicators | 3 | [b/470415967](http://b/470415967), [b/482902862,](http://b/482902862) [b/489948342](http://b/489948342) | +| EurostatData\_Education\_Enrollment | 3 | [b/474326901](http://b/474326901), [b/481243356,](http://b/481243356) [b/496059688](http://b/496059688) | +| USCensusPEP\_By\_Sex\_Race | 5 | [b/485260648](http://b/485260648), [b/502079403](http://b/502079403) | +| USCensusPEP\_PopulationEstimatebyRace | 5 | [b/486801970](http://b/486801970), [b/493190090](http://b/493190090), [b/497802532](http://b/497802532) | +| EurostatData\_Education\_Attainment | 3 | [b/472606851](http://b/472606851), [b/481245546](http://b/481245546), [b/504879314](http://b/504879314) | +| WorldBankDatasets | 3 | [b/472258775](http://b/472258775), [b/506961224](http://b/506961224) | +| EurostatData\_Fertility | 3 | [b/498154643](http://b/498154643) | +| USCensusPEP\_AgeSexRace | 3 | [b/500622108](http://b/500622108) | +| USCensusPEP\_Annual\_Population | 3 | [b/479399481](http://b/479399481), [b/4935600377](http://b/4935600377) | + +### 2.1 Minor Deletions imports + +The following imports frequently experience minor data losses due to broken source URLs or the removal of individual data points at the origin. To address this, we recommend increasing the deletion thresholds based on historical percentages, supplemented by automated validation rules to maintain data integrity. + +1. USCensusPEP\_Sex +2. USCensusPEP\_By\_Sex\_Race +3. USCensusPEP\_PopulationEstimatebyRace +4. USCensusPEP\_AgeSexRace +5. USCensusPEP\_Annual\_Population +6. EurostatData\_Education\_Attainment +7. EurostatData\_Fertility +8. EurostatData\_Education\_Enrollment + +### 2.2 Major Deletions Imports + +The following import jobs have experienced significant data deletions resulting from annual benchmarking conducted by the data sources, who have officially announced these changes: + +1. BLS\_CES\_State +2. WorldDevelopmentIndicators +3. WorldBankDatasets + +While these major deletions are unavoidable due to source updates, we intend to raise the threshold to accommodate and manage any minor deletions that may occur with some validation rules. + +**Note:** The current baseline threshold for deletions is **0.01%** (doesn’t contain crucial data) any deletion greater than the threshold should be stored as historical. + +## 3\. Imports with a Single Deletion Incident + +The following imports have experienced exactly one minor deletion incident from the source. If these deletions recur, we will increase the thresholds and apply automated validation rules. + +1. EurostatData\_Employment\_Per\_Sector +2. EIA\_Electricity +3. EIA\_NaturalGas +4. EIA\_Petroleum +5. EIA\_SEDS +6. EIA\_NuclearOutages +7. EurostatData\_GDP + diff --git a/tools/import_validation/troubleshooting_guides/Production Challenges.md b/tools/import_validation/troubleshooting_guides/Production Challenges.md new file mode 100644 index 0000000000..df81dea63e --- /dev/null +++ b/tools/import_validation/troubleshooting_guides/Production Challenges.md @@ -0,0 +1,98 @@ +## **Production Challenges: Source-Side Complications** + +### 1\. Structural and Schema Modifications + +Pipeline disruptions frequently arise from unexpected modifications to source data structures. These breaking changes—including altered formats, deleted records, or revised schema definitions—interfere with established ingestion and processing workflows. + +Information regarding the most frequent types of import failures can be found [DC - Imports Execution Learnings ](https://docs.google.com/spreadsheets/d/1lAm-EPB6o9U9btQHbdRruGhx_DFerlhQF43Ba6skI4Q/edit?gid=1651779522#gid=1651779522). + +*Historical instances requiring code adjustments:* + +1. WorldBankDatasets: [Reference Documentation](https://docs.google.com/document/d/1ExMv5_J4vSEY1OVwWLoXbvnH1ltjRsfcBjUqSyw6uXo/edit?tab=t.0) +2. EurostatData\_lifeexpectency: [Reference Documentation](https://docs.google.com/document/d/1h4ETCMddDTWPCCZnE8OaHNYKFAH2WHtUX9tGCxV5caY/edit?resourcekey=0-h6BHDUjBSyHS2v-mCfCKUw&tab=t.0#heading=h.by7ahuiow2gz) +3. UsMontlyRetailSales: [Reference Documentation](https://docs.google.com/document/d/1mVoeHRkdlnPvTpu-IMee-6d2GQ-htlLDRP2ZYdGW04w/edit?tab=t.0#heading=h.by7ahuiow2gz) + +**Standard Remediation Protocol** + +1. Verify the failure by searching for the specific import name within *validation\_output.csv* located in the `datcom-prod-imports` bucket. +2. In cases involving data loss, investigate whether the records were intentionally removed at the source or if a mismatch has occurred, then establish the necessary correction. + +### 1\. Data structure changes (Schema changes) + +Modifications to source data are causing pipeline failures. When a data source alters its format, removes records, or changes schema definitions without notice, it introduces breaking changes that disrupt our ingestion and processing workflows. +For example, code fixes were required after changes occurred in the following datasets: + +1. WorldBankDatasets: [WorldBankDatasets ](https://docs.google.com/document/d/1ExMv5_J4vSEY1OVwWLoXbvnH1ltjRsfcBjUqSyw6uXo/edit?tab=t.0) +2. EurostatData\_lifeexpectency: [EurostatData\_LifeExpectancy](https://docs.google.com/document/d/1h4ETCMddDTWPCCZnE8OaHNYKFAH2WHtUX9tGCxV5caY/edit?resourcekey=0-h6BHDUjBSyHS2v-mCfCKUw&tab=t.0#heading=h.by7ahuiow2gz) +3. UsMontlyRetailSales : [ USMontlyRetailsales](https://docs.google.com/document/d/1mVoeHRkdlnPvTpu-IMee-6d2GQ-htlLDRP2ZYdGW04w/edit?tab=t.0#heading=h.by7ahuiow2gz) + + + +**Standard procedures for addressing the problem** + +1. Analyze *validation\_output.csv* in the `datcom-prod-imports` bucket to confirm the import failure using the import name. +2. If the error is due to deletions, verify if data was removed at the source or if there is a data mismatch. Determine the appropriate fix. +3. If a 'missing reference' error occurs, identify the missing entity or mapping (e.g., place DCID or statistical variable). Update the relevant MCF file in Cider and submit a CL. + + + +### 2\. Data deletion at source + +The data has been officially removed from the source system. In this case, the standard deletion handling rules apply: + +* For minor data deletions, implement golden checks to protect critical information and do a subsequent increase in the import threshold. + * In instances of large data removal, ensure that all deleted records are preserved as historical data. + +1. In case of deletions, we check for generated o/p CSV \-\> in o/p folder of an import/timestamp folder +2. + + For example, Historical has been stored in [EurostatData\_Employment\_Per\_Sector](https://docs.google.com/document/d/1CPyqi9DBjv0t4eWanNYMRfJTIGjhubN6Op2Sao5S-9Y/edit?resourcekey=0-lNpBO1oS4JQOJaGE2uj2Rg&tab=t.0#heading=h.by7ahuiow2gz) & latest\_version.txt has been updated in [EIA\_NuclearOutages](https://docs.google.com/document/d/1GfF_sdUCg4d3fmkJoRCarQpFoGtLIPfezoKwoEXXL84/edit?tab=t.0#heading=h.10kuxd1t3o0u) + +**Standard procedures for addressing the problem** + +1. Review the latest run of the job in the `datcom-prod-imports` bucket under the `datcom` project using the import name. +2. If the job folder timestamp is older than one week, re-trigger the job to generate the latest output. +3. Examine `input0/validation/nodes_deleted.mcf` to review deletion details. +4. Validate whether the deletions originated from the source or were caused by a pipeline/code issue. +5. Apply the appropriate resolution based on the identified deletion scenario. + +### 3\. Downtime or modifications to source URLs + +These failures are typically caused by one of the following reasons: + +* The source URL is completely broken or no longer active. +* The external source changed the URL structure or moved the data to a new location. +* The external source's server is temporarily unresponsive. +* A firewall is blocking our connection to the source URL. + The `UsCensusPep_xxx` data import pipeline has experienced frequent failures (28 occurrences to date) due to issues with the source URLs. eg: [USCensusPEP\_AgeSexRace](https://docs.google.com/document/d/10_kFWkpkon9xhVOlOXjIFF0fSyO2IGFKWq6H1pcAcVE/edit?tab=t.0#heading=h.by7ahuiow2gz) + +**Standard procedures for addressing the problem** + +1. Restart the pipeline; if the issue persists, monitor the URL performance over the next several hours/days. +2. In cases where the source URLs have been fully replaced or relocated, modify the codebase or the relevant configuration settings accordingly. +3. If the URL is completely deleted and there is no new link: +* Use **historical data** if a large amount of data is missing. (only if core teams approves) +* Update the **`latest_version.txt`** file to keep the pipeline running.(only if core team approves) + +### 4\. API Issues (Limits & Failures) + +The pipeline can fail if we hit the API rate limit (too many requests) or if the API stops working entirely. + +The BLS\_CES import process utilizes an API for data retrieval; however, executing the import twice can occasionally trigger API rate limits: [BLS Imports Issues](https://docs.google.com/document/d/1bQbwKMVPtcehUZ3zI3r2kGlidMiE6UjEN9faKlJgtUs/edit?resourcekey=0-wJrxn96bZK7RUR196cSNfA&tab=t.0#heading=h.8rkahuqo26x9) + +**Standard procedures for addressing the problem** + +1. **Wait it Out:** If we hit a temporary rate limit or timeout issue, wait for the lockout period to end and try again later. +2. **Switch the API:** If the API is completely broken, no longer supported, or constantly failing, change the code to use an alternative API with a fallback logic. + +### 5.Missing reference errors due to additional source data + +The pipeline fails with a `missingReferenceObservationAbout` or `missingReferencesVariableMeasured` error when the source website adds new data (like new locations or new variables) that do not exist in our system yet. Because Data Commons doesn't recognize these new entities, the import fails. + +The EIA\_SEDS & US\_SAT\_ACT\_Participation imports has some data additions [EIA\_SEDS-2026-](https://docs.google.com/document/d/1Kvhz-7AaWpaxSvSN1wHDP75-3CVXTfZT7dffBgZNTaY/edit?resourcekey=0-YG8JZHFyO_xnEdqmg_4NMQ&tab=t.0#heading=h.by7ahuiow2gz) [Support P2 - Auto Refresh Failed Imports: US\_SAT\_ACT\_Participation](https://b.corp.google.com/issues/507394518) + +**Standard procedures for addressing the problem** + +1. Identify the missing place DCID or Statistical Variable from the error log. Add it to the existing `.mcf` file if the additions are valid. +2. Raise a CL to merge the updates and fix the pipeline. + diff --git a/tools/import_validation/troubleshooting_guides/RCA Runbook.md b/tools/import_validation/troubleshooting_guides/RCA Runbook.md new file mode 100644 index 0000000000..d71cba041f --- /dev/null +++ b/tools/import_validation/troubleshooting_guides/RCA Runbook.md @@ -0,0 +1,79 @@ +# Production Failure RCA Runbook + +# 1\. Overview + +This document provides an overview of the process that can be followed for doing the Root Cause Analysis (RCA) of the errors popping up in the auto refresh pipelines of Data Commons. + +# 2\. Background on Auto Refresh Pipelines + +The auto refresh pipelines are configured in the Cloud Batch service of the project “datcom-import-automation-prod”. The purpose of these pipelines is to refresh the data on a regular basis for the eligible datasets that have been ingested into Data Commons already. The pipelines perform the following tasks: + +1. Download the data from the source +2. Pre-process the data as required +3. Generate .csv and .tmcf files (via PV-MAPS or custom .py scripts) +4. Validate data against the last execution +5. Ingest the data to Data Commons if every step is successfully completed + +# 3\. Errors in auto refresh pipelines + +While these pipelines are meant to refresh the data in Data Commons, it is necessary to apply checks and perform data validation before the final ingestion. These checks ensure that the right data is ingested to Data Commons and reduce the probability of wrong data ingestion in Data Commons. Consequently, these pipelines fail in case the necessary checks and validations fail while data processing. + +Apart from data checks and validation there could be other probable reasons for the pipeline failures including the compute resources exhaustion, unavailability of data at source, deletion of source, addition of new places in the source data etc. + +The process to tackle each of these issues can be different. However, the steps to perform the RCA can be generalised. The next section will cover some of the important steps that can be taken into consideration for performing RCA. + +# 4\. Steps for performing RCA + +This section highlights the steps that can be taken to perform RCA for the failed production pipelines. + +Step 1: Go to the looker [dashboard](https://lookerstudio.google.com/c/reporting/e88fda74-50c9-46c6-88aa-c84342ceba48/page/eaXdF) and get the latest status of all the pipelines from the dashboard. The probable different states of each pipelines could be as below: + +1. VALIDATION: Data validation failure. +2. FAILURE: Import job failed. +3. STAGING: Import job completed, ready for ingestion. +4. SUCCESS: Data ingestion completed +5. SKIP: Incoming data is same as production data + +Pipelines having the state as “VALIDATION” or “FAILED” are the candidates for performing RCA. + +Step 2: Next, for each failed pipeline check the priority status. The same can be determined from the [Code Search](https://source.corp.google.com/) portal. Search for the import name in the Code Search portal and determine the respective import group for each of the pipelines from the respective manifest.json files. + +| Classification | Priority Ranking | import\_groups | +| :---- | :---- | :---- | +| **P0** | 1 | SearchBranch | +| **P0** | 2 | LaeLaps | +| **P0** | 3 | SearchAim | +| **P2** | 4 | Auto1W | +| **P2** | 5 | Auto2W | +| **P2** | 6 | USCensus | +| **P2** | 7 | USBLS | +| **P2** | 8 | WorldBank | +| **P2** | 8 | OECD | +| **P2** | 9 | CDC | +| **P2** | 10 | EuroStat | +| **P2** | 11 | UNSDG | +| **P2** | 12 | BRFSS | + +Step 3: Next, go to the Cloud Batch service of the `datcom-import-automation-prod` project and search for the respective import name in the search bar. [Screenshot](https://screenshot.googleplex.com/Bah7SXkdpNu5r7u.png) + +Step 4: Based on the type of the failure check the logs for the latest execution of the pipeline. +**Note: For validation failures, the Cloud Batch portal might show the pipeline “Succeeded” but it is important to check the logs to verify the same** + +Step 5: Next, traverse the logs of the pipelines and search for the respective reason of the failures. A few common reasons of the failure are as mentioned below: + +1. Exit code 137 : Signifies that the pipeline failed due to exhaustion of resources +2. Validation and lint errors: Signifies that the pipeline failed because of validation and lint errors. [Screenshot](https://screenshot.googleplex.com/3nosGztEJKr2pZV) +3. Pre processing script failed: Signifies that the script responsible for either downloading the data or processing the data has failed. [Screenshot](https://screenshot.googleplex.com/9KjTNR3EF8644v8) + +Step 6: Based on RCA, plan the error resolution. Discuss issues with the CORE TEAM on a need basis. Also, prepare a document capturing the issues and the next actions. [Template](https://docs.google.com/document/d/1PyBmcN-1C_p9y-ML93eaBFyspg1XD5zwkT7EqeTsQsI/edit?resourcekey=0-B5pj3_KBQDpdfNs6AGllXQ&tab=t.0#heading=h.ne6ee5rvmv4h) + +Step 7: Change the code (if required) and raise [CL/PR](http://cl/PR) as appropriate. + +Step 8: Once the code is merged with production, force execute the production pipelines using the command below: + +``` +~/Desktop/DataCommons/data/import-automation/executor/run_import.sh -p -d dc-test-executor-$USER -cloud -a -batch +``` + + + diff --git a/tools/import_validation/troubleshooting_guides/Resolution_Runbook.md b/tools/import_validation/troubleshooting_guides/Resolution_Runbook.md new file mode 100644 index 0000000000..4ddafe3d8e --- /dev/null +++ b/tools/import_validation/troubleshooting_guides/Resolution_Runbook.md @@ -0,0 +1,118 @@ +**PRODUCTION INCIDENT** + +**RESOLUTION RUNBOOK** + +**Organization / Product: DATA COMMONS** + +*\[\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\]* + +| Version | *\[e.g1. 1.0\]* | +| :---- | :---- | +| **Last Reviewed** | *\[Date\]* | +| **Runbook Owner** | *\[Name / Team\]* | +| **On-call Rotation** | *\[Link or name\]* | +| **Status Page URL** | *\[URL\]* | +| **Incident Channel** | *\[e.g. \#incidents\]* | + +## **Overview:** + +This runbook outlines resolutions for production failures, serving as a continuous guide for troubleshooting and support. + +## **Types of Production Failure** + +| TYPE | ERROR NAME | MEANING | +| :---- | :---- | :---- | +| **VALIDATION ERROR** | Found 0.01% deleted records | Production environment data deletion detected | +| **LINT ERROR** | Existence\_MissingReference\_observationAbout | Data Commons does not contain the specified mapping/place DCID. | +| **MISSING REFERENCE** | Existence\_MissingReference\_variableMeasured | The Data Commons repository is currently missing the required SV/dcid reference. | +| **Command Failed ExitCode 1** | Subprocess Failed | Failed due to code/development issue | +| **FALSE NEGATIVE** | Could be any failure out of above 4 | Failed once because of an issue with the cloud, server, or source. | + +1. ## **Validation Failure** + +| If a job fails due to data deletions, first identify the source of the deletion. Data deletions can occur in two scenarios: | +| :---- | + + **Case 1: Source-Level Deletion** + The data has been officially removed from the source system. In this case, the standard deletion handling rules apply: + + * If the deleted data is less than **0.01%** and does not contain critical data, the import threshold can be increased. + * If the deleted data is greater than **0.01%**, the deleted records must be stored as historical data. + + **Case 2: Pipeline/Code Issue** + The data deletion is caused unintentionally due to a pipeline, transformation, or code issue. In this scenario, investigate and fix the underlying issue before proceeding further. + +**Resolution Steps** + +1. Review the latest run of the job in the `datcom-prod-imports` bucket under the `datcom` project using the import name. +2. If the job folder timestamp is older than one week, re-trigger the job to generate the latest output. +3. Examine `input0/validation/nodes_deleted.mcf` to review deletion details. +4. Validate whether the deletions originated from the source or were caused by a pipeline/code issue. +5. Apply the appropriate resolution based on the identified deletion scenario. + +**How to store Historical data** + +1. Verify the production version of the import via [Data Commons/Version](https://datacommons.org/version). +2. Locate and download the production CSV from the storage bucket, matching the date specified in the Data Commons version. +3. Run the script in your local environment, then utilize Python code to perform a difference comparison between the latest and production output CSVs. +4. Run the differ on output & historical data table\_mcf\_nodes.mcf files together & ensure no deletion flags are there. +5. Once the deleted rows are identified & no deletions with this historical file through the comparison, +6. Upload this historical data (along with its tmcf file) to the 'unresolved' path for that import, and use the importer to write it to the Knowledge Graph (KG). +7. Once the data is saved to the Knowledge Graph, use the COPY Service to move the file to CNS. +8. Add the historical path where you have added the file in the CNS as a historical record + +**Note:** If a folder for historical data already exists for this import, place the new file inside it. If not, create a new folder. + + + +## + +## + +## **LINT ERROR** + +| When dcid/mapping is missing in Data Commons | +| :---- | + +1. Review the latest run of the job in the `datcom-prod-imports` bucket under the `datcom` project using the import name. +2. If the job folder timestamp is older than one week, re-trigger the job to generate the latest output. +3. Check for **Existence\_MissingReference\_observationAbout** in the report.json inside the input0/genmcf/report.json +4. Look for the observationAbouts’s that are throwing the error +5. Run the import locally an update or create the MCF file with the necessary mappings, keep adding mappings until the error is gone from report.json. +6. Add these mappings to the current CNS file or create a new one by creating a CL. + +| When the Data Commons API fails to locate the dcid or mapping | +| :---- | + + 1\. Existence\_FailedDcCall\_observationAbout +Existence\_FailedDcCall\_observationAbout + +## **MISSING REFERENCE** + +## + +| When SV or schema is missing in Data Commons | +| :---- | + +1. Review the latest run of the job in the `datcom-prod-imports` bucket under the `datcom` project using the import name. +2. If the job folder timestamp is older than one week, re-trigger the job to generate the latest output. +3. Check for **Existence\_MissingReference\_variableMeasured** in the report.json inside the input0/genmcf/report.json +4. Look for the Variable measured (value-ref) that are throwing the warnings +5. Run the import locally and update or create the MCF file with the necessary mappings,keep adding mappings until the error is gone from report.json +6. Add these mappings to the current CNS file or create a new one by creating a CL. + +## **Command Failed ExitCode 1** + +| When the code fails due to any issue | +| :---- | + +1. Restart the job in the cloud environment; if it succeeds, the previous failure likely resulted from a transient environmental problem. +2. If the issue persists after re-triggering the job, execute the process in a local environment to identify and resolve code defects. + +## **FALSE NEGATIVE** + +| When the code fails due to any random temporary issue | +| :---- | + +1. +