🔍 Data Reliability Watchdog

Automated Monitoring of Tamil Nadu Government Election Data

The Data Reliability Watchdog is an automated data quality monitoring system designed to clean, validate, analyze, and monitor the reliability of Tamil Nadu government election data.

The project automatically identifies data quality issues, performs validation checks, calculates a reliability score, and generates detailed reports.

---

🎯 Project Objective

The main objective of this project is to ensure that government election datasets are:

• Clean
• Complete
• Consistent
• Valid
• Reliable

The system provides an automated workflow for checking the quality and reliability of election data.

---

📊 Dataset

The project uses Tamil Nadu election result data containing candidate-level information.

The dataset contains 3,955 rows and 12 columns.

Dataset Columns

• Constituency
• Candidate_No
• Candidate_Name
• Gender
• Age
• Category
• Party
• General_Votes
• Postal_Votes
• Total_Votes
• Vote_Percentage
• Elector_Percentage

---

⚙️ Project Workflow

The project follows an automated data reliability pipeline:

Raw Election Data
        ↓
Data Cleaning
        ↓
Data Validation
        ↓
Reliability Score Calculation
        ↓
Report Generation
        ↓
Streamlit Dashboard

---

🧹 Data Cleaning

The cleaning stage loads the raw election dataset and handles missing Constituency values using forward filling.

Before Cleaning

Rows: 3955
Columns: 12
Missing Constituency Values: 2323

After Cleaning

Missing Constituency Values: 0
Output:
data/cleaned_election_results.csv

---

✅ Data Validation

The project performs six automated data quality checks.

Validation Checks

1. Missing Values
2. Required Columns
3. Data Types
4. Duplicate Rows
5. Vote Consistency
6. Percentage Range

Current Validation Result

Missing Values: PASS
Required Columns: PASS
Data Types: PASS
Duplicate Rows: PASS
Vote Consistency: PASS
Percentage Range: PASS

Passed Checks: 6
Failed Checks: 0

---

📊 Reliability Score

The reliability score is calculated using:

Reliability Score =
(Passed Checks / Total Checks) × 100

Current Result

Total Checks: 6
Passed Checks: 6
Failed Checks: 0

Reliability Score: 100.00%
Reliability Level: EXCELLENT

---

📄 Generated Reports

The automated pipeline generates the following reports:

reports/
├── validation_report.csv
├── reliability_report.txt
└── validation_report.pdf

These reports provide detailed information about the validation results and overall dataset reliability.

---

🧪 Automated Testing

The project uses Pytest for automated testing.

All six validation tests are currently passing.

Test Result

6 passed

Run the tests using:

python -m pytest

---

🖥️ Streamlit Dashboard

The project includes an interactive Streamlit dashboard for monitoring data reliability.

Dashboard Preview

The dashboard displays:

• Reliability score
• Passed checks
• Failed checks
• Validation results
• Validation report
• Reliability report
• Downloadable reports

Run the dashboard using:

streamlit run dashboard.py

---

🚀 Running the Complete Pipeline

To execute the complete automated workflow:

python run_pipeline.py

The pipeline performs:

1. Data Cleaning
2. Data Validation
3. Reliability Score Calculation
4. Report Generation

---

🛠️ Technologies Used

• Python – Core programming language
• Pandas – Data cleaning and data manipulation
• NumPy – Numerical operations
• Streamlit – Interactive dashboard
• Pytest – Automated testing
• ReportLab – PDF report generation
• CSV – Dataset storage and processing
• Git & GitHub – Version control and project management

---

📁 Project Structure

Data Reliability Watchdog/
│
├── data/
│   ├── election_results.csv
│   └── cleaned_election_results.csv
│
├── reports/
│   ├── validation_report.csv
│   ├── reliability_report.txt
│   └── validation_report.pdf
│
├── src/
│   ├── data_cleaning.py
│   ├── data_validation.py
│   ├── reliability_score.py
│   ├── report_generator.py
│   └── __init__.py
│
├── tests/
│   └── test_validation.py
│
├── dashboard.py
├── run_pipeline.py
├── requirements.txt
├── README.md
└── .gitignore

---

📈 Results

The Data Reliability Watchdog successfully processes the Tamil Nadu election dataset and generates automated data quality results.

Final Pipeline Result

Dataset Rows: 3,955
Dataset Columns: 12

Missing Constituency Values Fixed: 2,323

Validation Checks: 6
Passed Checks: 6
Failed Checks: 0

Reliability Score: 100.00%
Reliability Level: EXCELLENT

The system successfully completed all validation checks and generated the required reliability reports.

---

🔮 Future Enhancements

The project can be further enhanced with the following features:

• Real-time monitoring of government datasets
• Automated data quality alerts
• Advanced anomaly detection
• Historical reliability score tracking
• Interactive data quality visualizations
• Email notifications for validation failures
• Integration with additional Tamil Nadu government datasets
• Automated scheduled pipeline execution

---

🏁 Conclusion

The Data Reliability Watchdog provides an automated approach for monitoring and evaluating the quality of Tamil Nadu government election data.

By combining data cleaning, automated validation, reliability scoring, report generation, and an interactive dashboard, the system helps identify data quality issues and provides a clear measure of dataset reliability.

The project demonstrates how Python, data validation, automation, and visualization can be combined to build a practical data quality monitoring solution.