Skip to content

Data-Engineering

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

Data-Engineering

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
15 Common Pandas Polars SQL PySpark Translations.md
15 Common Pandas Polars SQL PySpark Translations.md
 
 
7 Strategies to Scale Your Database.md
7 Strategies to Scale Your Database.md
 
 
Advanced SQL Concepts with Python.md
Advanced SQL Concepts with Python.md
 
 
Advanced SQL Techniques CTEs, Subqueries, and More.md
Advanced SQL Techniques CTEs, Subqueries, and More.md
 
 
Apache Spark Concepts and Python Examples.md
Apache Spark Concepts and Python Examples.md
 
 
Building End-to-End Data Pipelines with Python.md
Building End-to-End Data Pipelines with Python.md
 
 
Choosing the Right Python Tool for Large Datasets Pandas, Dask, or PySpark.md
Choosing the Right Python Tool for Large Datasets Pandas, Dask, or PySpark.md
 
 
Comparing ETL, ELT, and EtLT Approaches in Python.md
Comparing ETL, ELT, and EtLT Approaches in Python.md
 
 
Comprehensive Guide to SQL Window Functions with Python.md
Comprehensive Guide to SQL Window Functions with Python.md
 
 
Concurrency Control in Databases.md
Concurrency Control in Databases.md
 
 
Concurrency in Databases Ensuring Data Consistency.md
Concurrency in Databases Ensuring Data Consistency.md
 
 
Data Cleaning with Python and SQL.md
Data Cleaning with Python and SQL.md
 
 
Discover Pandas UDFs in PySpark.md
Discover Pandas UDFs in PySpark.md
 
 
ETL, ELT, and EtLT Processes in Python.md
ETL, ELT, and EtLT Processes in Python.md
 
 
Exploring ETL Data Pipeline Processes.md
Exploring ETL Data Pipeline Processes.md
 
 
Grouping Sets, Rollup, and Cube in SQL with Python.md
Grouping Sets, Rollup, and Cube in SQL with Python.md
 
 
Introduction to Aggregate and Transform Functions in Apache Spark.md
Introduction to Aggregate and Transform Functions in Apache Spark.md
 
 
Introduction to PySpark Using Python.md
Introduction to PySpark Using Python.md
 
 
Kafka for Real-Time Data Pipelines in Python.md
Kafka for Real-Time Data Pipelines in Python.md
 
 
Key Concepts of Database Sharding with Python.md
Key Concepts of Database Sharding with Python.md
 
 
Local-First Text-to-SQL Tool with Python.md
Local-First Text-to-SQL Tool with Python.md
 
 
Mastering PostgreSQL Fundamentals for Backend Development.md
Mastering PostgreSQL Fundamentals for Backend Development.md
 
 
Mastering SQL Query Execution Order.md
Mastering SQL Query Execution Order.md
 
 
Mastering SQL in Python with PandaSQL.md
Mastering SQL in Python with PandaSQL.md
 
 
Optimizing SQL Joins with Table Size Awareness.md
Optimizing SQL Joins with Table Size Awareness.md
 
 
README.md
README.md
 
 
SQL vs PySpark Comparative .md
SQL vs PySpark Comparative .md
 
 
SQL vs. Pandas Mastering Data Analysis Tools.md
SQL vs. Pandas Mastering Data Analysis Tools.md
 
 
SQL's Execution Flow.md
SQL's Execution Flow.md
 
 
Spark User Defined Functions with Python.md
Spark User Defined Functions with Python.md
 
 
Spark Window Functions for Time-Series Analysis in PySpark.md
Spark Window Functions for Time-Series Analysis in PySpark.md
 
 
Spark vs MapReduce Python-Powered Distributed Data Processing.md
Spark vs MapReduce Python-Powered Distributed Data Processing.md
 
 
Splitting and Distributing Databases with Python.md
Splitting and Distributing Databases with Python.md
 
 
The Evolution of Databases From File Systems to Modern Marvels.md
The Evolution of Databases From File Systems to Modern Marvels.md
 
 
Time-Series Analysis with PySpark Window Functions.md
Time-Series Analysis with PySpark Window Functions.md
 
 
Transitioning from SQL to Pandas DataFrames Using Python.md
Transitioning from SQL to Pandas DataFrames Using Python.md
 
 
Understanding Apache Kafka with Python.md
Understanding Apache Kafka with Python.md
 
 
Understanding SQL's Execution Order.md
Understanding SQL's Execution Order.md
 
 
User Defined Functions in Apache Spark with Python.md
User Defined Functions in Apache Spark with Python.md
 
 
Vector Embeddings, Databases, and Search in Python.md
Vector Embeddings, Databases, and Search in Python.md
 
 
Window Functions in Apache Spark.md
Window Functions in Apache Spark.md
 
 

README.md

Data Engineering

40 articles on data engineering — SQL, Apache Spark, Kafka, ETL pipelines, and database architecture.

Part of the xbe.at knowledge base. ← Back to index

What's inside

SQL

  • Query execution order, window functions, CTEs, subqueries
  • Grouping Sets, Rollup, Cube
  • Optimizing SQL joins, text-to-SQL with local LLMs
  • SQL vs Pandas comparison
  • PostgreSQL fundamentals, PandaSQL

Apache Spark & PySpark

  • PySpark fundamentals, Spark vs MapReduce
  • Spark window functions for time-series analysis
  • User-defined functions (UDFs) in Spark
  • Pandas UDFs in PySpark, aggregate/transform functions
  • Pandas, Dask, PySpark — choosing the right tool

Data pipelines & ETL

  • ETL, ELT, and EtLT approaches compared
  • Building end-to-end data pipelines with Python
  • Kafka for real-time data pipelines
  • Data Engineering workflow stages

Databases

  • Database sharding, splitting and distributing databases
  • Concurrency control and data consistency
  • Evolution of databases: from file systems to modern architectures
  • Data Mesh vs Data Fabric architectures
  • Vector databases and search

Translations

  • Pandas / Polars / SQL / PySpark syntax translations (15 common operations)

Highlights

  • Kafka for Real-Time Data Pipelines in Python
  • 15 Common Pandas Polars SQL PySpark Translations
  • Building End-to-End Data Pipelines with Python
  • Spark Window Functions for Time-Series Analysis in PySpark
  • The Evolution of Databases: From File Systems to Modern Marvels