Experts estimate that 40% of the world's languages are currently endangered, but keeping track of a language's endangerment status is currently a highly manual and time intensive task. This project leverages machine learning techniques to answer the question "Can publicly available socioeconomic and geographic data approximate language endangerment status without requiring on-the-ground linguistic assessment?".
This project leverages machine learning to predict the endangerment level of languages worldwide using demographic, technological, and legal status indicators. Drawing on five integrated datasets, it explores the relationship between factors such as speaker count, urbanization rates, internet use, and legal recognition, and their influence on language vitality. Multiple models, including ensemble methods, gradient boosting, and neural networks, were trained and evaluated, with the highest predictive performance reaching 88% accuracy. The results highlight the critical role of speaker population size while underscoring opportunities for improvement through the inclusion of additional socio-economic and historical features.
| Dataset | Source | Description |
|---|---|---|
| Endangered Languages Project | Endangered Languages Project | Catalogue of Endangered Languages. 2023. University of Hawaii at Manoa. http://www.endangeredlanguages.com. Contains a list of ~3,000 languages with varying levels of endangerment. |
| List of languages by total number of speakers | Wikipedia | List of 20 non-endangered languages and their speaker counts. |
| List of official languages by country and territory | Wikipedia | List of countries and their politically recognized official, regional, minority, national and widely spoken languages. |
| GDP per capita (current US$) | World Bank | Countries and their GDP per capita. |
| Individuals using the internet (% of population) | World Bank | Percentage of population using the internet in each country. |
| Urban population (% of total population) | World Bank | Percent of the total population in a country that lives in an urban area |
├── data/ # processed datasets
├── archive/ # version 2 work in progress
│ ├── notebooks/
│ ├── data/
│ └── README.md
├── notebooks/ # models and data analysis notebooks
├── images/ # charts and figures used in the README
├── README.md
└── requirements.txt
# Clone the repo
git clone https://github.com/jandersen12/Machine-Learning-Endangered-Languages.git
cd Machine-Learning-Endangered-Languages
# Install dependencies
pip install -r requirements.txt # Python
Environment: Python 3.10
- Integrated datasets from multiple sources with Regex techniques and feature engineering to produce a final, model-ready dataset.
- Ran a baseline model that predicted the majority class achieved 46% accuracy on the test set.
- Handled class imbalance with SMOTE and tree-based models.
- Used different machine learning models to see which performed best in predicting language endangerment: ensemble, neural networks, extra trees and gradient boosting.
- Predictive performance of a model is highly related to speaker count. This feature often requires manual on-the-ground techniques to measure, so further research could potentially benefit from feature engineering that attempts to predict the speaker count of a language in order to then determine its level of endangerment.
- Gradient Boosting and Ensemble models achieved the highest performance ~88% test accuracy.
- These models relied on current socioeconomic data, but the inclusion of historical factors such as history of emigration and conflict could further enhance the model's performance.
This project required turning an ambiguous research question into a multi-source predictive model. The main challenge in this project was in designing a data integration process with socioeconomic features that held true to the original problem. Through the project I also gained hands-on experience with machine learning techniques using Tensorflow and Scikit-Learn to make predictions and interpret results.
Helin Yilmaz | Courtney Chen | Brian Woods
UC Berkeley MIDS | July 2025