Skip to content

Download Script Fix + PVmap Changes in Mongolia Imports (Mongolia_Education, Mongolia_Health, Mongolia_Employment, Mongolia_Demographics #2128

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kartik-s21 wants to merge 41 commits into
base: master
Choose a base branch
from
Open

Download Script Fix + PVmap Changes in Mongolia Imports (Mongolia_Education, Mongolia_Health, Mongolia_Employment, Mongolia_Demographics #2128

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
ea4cdd2
Fixed Code for 3 files
kartik-s21 Apr 7, 2026
91d22a7
NameError Resolved
kartik-s21 Apr 8, 2026
c3084c4
Merge branch 'master' into code_fix_unenergy
kartik-s21 Apr 10, 2026
297374c
Merge branch 'master' into code_fix_unenergy
kartik-s21 Apr 12, 2026
34bb357
Merge pull request #1 from kartik-s21/code_fix_unenergy
kartik-s21 Apr 13, 2026
ee1e0b2
Merge branch 'datacommonsorg:master' into master
kartik-s21 Apr 30, 2026
8707bba
Merge branch 'datacommonsorg:master' into master
kartik-s21 May 5, 2026
5527dcd
Merge branch 'datacommonsorg:master' into master
kartik-s21 May 5, 2026
e41a8a7
Merge branch 'datacommonsorg:master' into master
kartik-s21 May 5, 2026
7f17aa3
Resolve merge conflict in energy process script
kartik-s21 May 17, 2026
59bcad2
Merge branch 'datacommonsorg:master' into master
kartik-s21 May 17, 2026
4fc439a
Merge branch 'datacommonsorg:master' into master
kartik-s21 May 28, 2026
9e5417c
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jun 2, 2026
cb72442
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jun 4, 2026
d9bb0a6
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jun 5, 2026
2515f96
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jun 7, 2026
c099907
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jun 10, 2026
1359560
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jun 12, 2026
0dfeecc
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jun 15, 2026
b09267f
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jun 17, 2026
776feff
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jun 18, 2026
4b06212
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jun 30, 2026
8bb164a
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jul 8, 2026
7ebbdce
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jul 13, 2026
7eb092e
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jul 20, 2026
2f9a79b
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jul 20, 2026
fb658bd
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jul 22, 2026
29db8fa
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jul 24, 2026
3026f30
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jul 29, 2026
7a815af
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jul 30, 2026
cb02292
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jul 31, 2026
305fb69
Merge branch 'datacommonsorg:master' into master
kartik-s21 Jul 31, 2026
a146ef9
Merge branch 'datacommonsorg:master' into master
kartik-s21 Aug 3, 2026
c7ad25e
Merge branch 'datacommonsorg:master' into master
kartik-s21 Aug 3, 2026
2b81be4
Merge branch 'datacommonsorg:master' into master
kartik-s21 Aug 4, 2026
51bcc1c
Mongolia_imports fix
kartik-s21 Aug 4, 2026
f4ad95d
gemini suggested changes done
kartik-s21 Aug 4, 2026
c0d08f1
retained changes
kartik-s21 Aug 5, 2026
bccbf34
added ignore lines back to mongolia demographics
kartik-s21 Aug 5, 2026
7e87915
added output counters
kartik-s21 Aug 5, 2026
1b44cd2
mongolia employment fixes
kartik-s21 Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
317 changes: 241 additions & 76 deletions statvar_imports/mongolia_imports/common_download_script.py
View file
Open in desktop

Large diffs are not rendered by default.

27 changes: 19 additions & 8 deletions ...olia_demographics/resident_population_by_agegroup_15_and_over_and_maritalstatus_pvmap.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,24 @@ Single,maritalStatus,Unmarried,,,,,,,,
2000,observationDate,2000,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2010,observationDate,2010,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2015,observationDate,2015,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2020,observationDate,2020,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2021,observationDate,2021,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2022,observationDate,2022,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2023,observationDate,2023,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2024,observationDate,2024,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2025,observationDate,2025,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2026,observationDate,2026,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2027,observationDate,2027,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2028,observationDate,2028,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2029,observationDate,2029,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
2030,observationDate,2030,value,{Number},observationAbout,country/MNG,populationType,Person,measuredProperty,count
,,,,,,,,,,
Age Group:Population aged 15 and over-Total,age,[15 - Years],,,,,,,,
Age Group:15-19,#ignore,skip,,,,,,,,
Age Group:70+,#ignore,skip,,,,,,,,
Age Group:Total,age,[15 - Years],,,,,,,,
Age Group:20-29,age,[20 29 Years],,,,,,,,
Age Group:30-39,age,[30 39 Years],,,,,,,,
Age Group:40-49,age,[40 49 Years],,,,,,,,
Age Group:50-59,age,[50 59 Years],,,,,,,,
Age Group:60-69,age,[60 69 Years],,,,,,,,
,,,,,,,,,,
20-29,age,[20 29 Years],,,,,,,,
30-39,age,[30 39 Years],,,,,,,,
40-49,age,[40 49 Years],,,,,,,,
50-59,age,[50 59 Years],,,,,,,,
60-69,age,[60 69 Years],,,,,,,,
Male,#ignore,skip,,,,,,,,
Female,#ignore,skip,,,,,,,,
15 changes: 8 additions & 7 deletions statvar_imports/mongolia_imports/mongolia_demographics/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,21 @@
set -e

SCRIPT_PATH=$(realpath "$(dirname "$0")")
mkdir -p "$SCRIPT_PATH/counters"

python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/mid_year_total_population_by_region.csv --pv_map=$SCRIPT_PATH/mid_year_total_population_by_region_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/mid_year_total_population_by_region_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --places_resolved_csv=$SCRIPT_PATH/mongolia_place_resolver.csv || { echo "Error: Processing mid_year_total_population_by_region failed!"; exit 1; }
python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/mid_year_total_population_by_region.csv --pv_map=$SCRIPT_PATH/mid_year_total_population_by_region_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/mid_year_total_population_by_region_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --places_resolved_csv=$SCRIPT_PATH/mongolia_place_resolver.csv --output_counters=$SCRIPT_PATH/counters/mid_year_total_population_by_region_counters.csv || { echo "Error: Processing mid_year_total_population_by_region failed!"; exit 1; }

python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/number_of_households_by_region_and_urban_rural.csv --pv_map=$SCRIPT_PATH/number_of_households_by_region_and_urban_rural_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/number_of_households_by_region_and_urban_rural_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --places_resolved_csv=$SCRIPT_PATH/mongolia_place_resolver.csv || { echo "Error: Processing number_of_households_by_region_and_urban_rural failed!"; exit 1; }
python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/number_of_households_by_region_and_urban_rural.csv --pv_map=$SCRIPT_PATH/number_of_households_by_region_and_urban_rural_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/number_of_households_by_region_and_urban_rural_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --places_resolved_csv=$SCRIPT_PATH/mongolia_place_resolver.csv --output_counters=$SCRIPT_PATH/counters/number_of_households_by_region_and_urban_rural_counters.csv || { echo "Error: Processing number_of_households_by_region_and_urban_rural failed!"; exit 1; }

python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/number_of_households_by_region.csv --pv_map=$SCRIPT_PATH/number_of_households_by_region_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/number_of_households_by_region_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --places_resolved_csv=$SCRIPT_PATH/mongolia_place_resolver.csv || { echo "Error: Processing number_of_households_by_region failed!"; exit 1; }
python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/number_of_households_by_region.csv --pv_map=$SCRIPT_PATH/number_of_households_by_region_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/number_of_households_by_region_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --places_resolved_csv=$SCRIPT_PATH/mongolia_place_resolver.csv --output_counters=$SCRIPT_PATH/counters/number_of_households_by_region_counters.csv || { echo "Error: Processing number_of_households_by_region failed!"; exit 1; }

python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/resident_population_by_agegroup_15_and_over_and_maritalstatus.csv --pv_map=$SCRIPT_PATH/resident_population_by_agegroup_15_and_over_and_maritalstatus_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/resident_population_by_agegroup_15_and_over_and_maritalstatus_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf || { echo "Error: Processing resident_population_by_agegroup_15_and_over_and_maritalstatus_pvmap failed!"; exit 1; }
python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/resident_population_by_agegroup_15_and_over_and_maritalstatus.csv --pv_map=$SCRIPT_PATH/resident_population_by_agegroup_15_and_over_and_maritalstatus_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/resident_population_by_agegroup_15_and_over_and_maritalstatus_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --output_counters=$SCRIPT_PATH/counters/resident_population_by_agegroup_15_and_over_and_maritalstatus_counters.csv || { echo "Error: Processing resident_population_by_agegroup_15_and_over_and_maritalstatus_pvmap failed!"; exit 1; }

python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/total_population_by_age_group_and_sex.csv --pv_map=$SCRIPT_PATH/total_population_by_age_group_and_sex_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/total_population_by_age_group_and_sex_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf || { echo "Error: Processing total_population_by_age_group_and_sex failed!"; exit 1; }
python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/total_population_by_age_group_and_sex.csv --pv_map=$SCRIPT_PATH/total_population_by_age_group_and_sex_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/total_population_by_age_group_and_sex_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --output_counters=$SCRIPT_PATH/counters/total_population_by_age_group_and_sex_counters.csv || { echo "Error: Processing total_population_by_age_group_and_sex failed!"; exit 1; }

python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/total_population_by_region_and_urban_rural.csv --pv_map=$SCRIPT_PATH/total_population_by_region_and_urban_rural_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/total_population_by_region_and_urban_rural_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --places_resolved_csv=$SCRIPT_PATH/mongolia_place_resolver.csv || { echo "Error: Processing total_population_by_region_and_urban_rural failed!"; exit 1; }
python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/total_population_by_region_and_urban_rural.csv --pv_map=$SCRIPT_PATH/total_population_by_region_and_urban_rural_pvmap.csv --config_file=$SCRIPT_PATH/mongolia_metadata.csv --output_path=$SCRIPT_PATH/output_files/total_population_by_region_and_urban_rural_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --places_resolved_csv=$SCRIPT_PATH/mongolia_place_resolver.csv --output_counters=$SCRIPT_PATH/counters/total_population_by_region_and_urban_rural_counters.csv || { echo "Error: Processing total_population_by_region_and_urban_rural failed!"; exit 1; }

python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/total_population_by_sex_and_urban_rural.csv --pv_map=$SCRIPT_PATH/total_population_by_sex_and_urban_rural_pvmap.csv --config_file=$SCRIPT_PATH/total_population_by_sex_and_urban_rural_metadata.csv --output_path=$SCRIPT_PATH/output_files/total_population_by_sex_and_urban_rural_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf || { echo "Error: Processing total_population_by_sex_and_urban_rural failed!"; exit 1; }
python3 $SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py --input_data=$SCRIPT_PATH/input_files/total_population_by_sex_and_urban_rural.csv --pv_map=$SCRIPT_PATH/total_population_by_sex_and_urban_rural_pvmap.csv --config_file=$SCRIPT_PATH/total_population_by_sex_and_urban_rural_metadata.csv --output_path=$SCRIPT_PATH/output_files/total_population_by_sex_and_urban_rural_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --output_counters=$SCRIPT_PATH/counters/total_population_by_sex_and_urban_rural_counters.csv || { echo "Error: Processing total_population_by_sex_and_urban_rural failed!"; exit 1; }

echo "All processing steps completed successfully."
exit 0
3 changes: 2 additions & 1 deletion ...mongolia_education/graduates_of_universities_and_colleges_by_professional_field_pvmap.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Biological and related sciences,bachelorsDegreeMajor,BiologicalAndBiomedicalScie
Environment,bachelorsDegreeMajor,EnvironmentMajor,,,,,,
Natural sciences,bachelorsDegreeMajor,NaturalSciencesMajor,,,,,,
Mathematics and statistics,bachelorsDegreeMajor,MathAndStatisticsMajor,,,,,,
Information and Communication Technologies,bachelorsDegreeMajor,InformationAndCommunicationsTechnologiesMajor ,,,,,,
Information and Communication Technologies,bachelorsDegreeMajor,InformationAndCommunicationsTechnologiesMajor,,,,,,
"Engineering, manufacturing and construction",bachelorsDegreeMajor,EngineeringMajor__EngineeringManufacturing__ConstructionEngineering,,,,,,
Architecture and construction,bachelorsDegreeMajor,ArchitectureAndRelatedServicesMajor__ConstructionEngineering,,,,,,
Engineering between industries,bachelorsDegreeMajor,EngineeringMajor__Industries,,,,,,
Expand All @@ -22,6 +22,7 @@ Production and processing,bachelorsDegreeMajor,ProductionEngineering__Processing
"Agriculture, forestry, fisheries and veterinary",bachelorsDegreeMajor,AgriculturalEngineering__ForestryEngineering__Fisheries__Veterinary,,,,,,
Veterinary medicine,bachelorsDegreeMajor,VeterinaryMedicineMajor,,,,,,
"Agriculture, forestry, fisheries",bachelorsDegreeMajor,AgriculturalEngineering__ForestryEngineering__Fisheries,,,,,,
Health and welfare,bachelorsDegreeMajor,HealthRelatedMajor,,,,,,
Comment thread
kartik-s21 marked this conversation as resolved.
Health and social protection,bachelorsDegreeMajor,HealthRelatedMajor,,,,,,
Oral and Maxillofacial Studies,bachelorsDegreeMajor,OralAndMaxillofacialStudies,,,,,,
Nursing and midwifery,bachelorsDegreeMajor,NursingAndMidwifery,,,,,,
Expand Down
8 changes: 7 additions & 1 deletion ...olia_education/number_of_full_time_teachers_in_universities_and_colleges_by_sex_pvmap.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,10 @@ Number of teacher:Total,populationType,Teacher,measuredProperty,count,employment
2021,observationDate,2021,value,{Number},observationAbout,country/MNG,populationType,Teacher,measuredProperty,count
2022,observationDate,2022,value,{Number},observationAbout,country/MNG,populationType,Teacher,measuredProperty,count
2023,observationDate,2023,value,{Number},observationAbout,country/MNG,populationType,Teacher,measuredProperty,count
2024,observationDate,2024,value,{Number},observationAbout,country/MNG,populationType,Teacher,measuredProperty,count
2024,observationDate,2024,value,{Number},observationAbout,country/MNG,populationType,Teacher,measuredProperty,count
2025,observationDate,2025,value,{Number},observationAbout,country/MNG,populationType,Teacher,measuredProperty,count
2026,observationDate,2026,value,{Number},observationAbout,country/MNG,populationType,Teacher,measuredProperty,count
2027,observationDate,2027,value,{Number},observationAbout,country/MNG,populationType,Teacher,measuredProperty,count
2028,observationDate,2028,value,{Number},observationAbout,country/MNG,populationType,Teacher,measuredProperty,count
2029,observationDate,2029,value,{Number},observationAbout,country/MNG,populationType,Teacher,measuredProperty,count
2030,observationDate,2030,value,{Number},observationAbout,country/MNG,populationType,Teacher,measuredProperty,count
8 changes: 7 additions & 1 deletion ...r_imports/mongolia_imports/mongolia_education/number_of_kindergartens_by_region_pvmap.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,10 @@ Aimag,observationAbout,{Data},schoolGradeLevel,Kindergarten,populationType,Schoo
2021,observationDate,2021,value,{Number},,,,
2022,observationDate,2022,value,{Number},,,,
2023,observationDate,2023,value,{Number},,,,
2024,observationDate,2024,value,{Number},,,,
2024,observationDate,2024,value,{Number},,,,
2025,observationDate,2025,value,{Number},,,,
2026,observationDate,2026,value,{Number},,,,
2027,observationDate,2027,value,{Number},,,,
2028,observationDate,2028,value,{Number},,,,
2029,observationDate,2029,value,{Number},,,,
2030,observationDate,2030,value,{Number},,,,
8 changes: 7 additions & 1 deletion ...ts/mongolia_education/number_of_students_in_universities_and_colleges_by_region_pvmap.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,10 @@ Female,gender,Female,,,,
2021,observationDate,2021,value,{Number},,
2022,observationDate,2022,value,{Number},,
2023,observationDate,2023,value,{Number},,
2024,observationDate,2024,value,{Number},,
2024,observationDate,2024,value,{Number},,
2025,observationDate,2025,value,{Number},,
2026,observationDate,2026,value,{Number},,
2027,observationDate,2027,value,{Number},,
2028,observationDate,2028,value,{Number},,
2029,observationDate,2029,value,{Number},,
2030,observationDate,2030,value,{Number},,
Loading
Loading