End-to-end Azure data engineering pipeline processing 38M+ NYC taxi trips using ADF, Databricks, Delta Lake and PySpark
-
Updated
Jun 26, 2026 - Python
#
incrementalloading
Here are 5 public repositories matching this topic...
A metadata-driven, CDC-based Medallion architecture pipeline that ingests a Spotify-style streaming warehouse from Azure SQL, processes it through Bronze → Silver → Gold using Databricks Autoloader and Delta Live Tables, and models it as a Star Schema in Unity Catalog — with SCD Type 2 history, Git-based CI/CD, and automated failure alerting.
python jinja2 azure pyspark azure-functions dlt jinja2-templates azuredatafactory azuredatabricks delta-lake azuresql starschema scd-type-2 azurelogicapp deltalivetables incrementalloading
-
Updated
Jul 25, 2026 - Python
End-to-end Azure data engineering pipeline using Spotify dummy data with Medallion Architecture, Databricks, Delta Live Tables, and incremental loading.
etl azure data-engineering databricks azure-databricks deltalake medallion-architecture backfilling scdtype2 deltalivetables incrementalloading
-
Updated
Dec 28, 2025 - Jupyter Notebook
Configuration-driven ETL framework in Python — YAML-defined pipelines, plugin architecture, incremental loading, data quality checks, DuckDB warehouse.
-
Updated
Aug 2, 2026 - Python
This project demonstrates an end-to-end ETL pipeline built using Snowflake modern layered architecture. pipeline ingests retail data from CSV files, validates , transforms the data, loads clean records into curated tables, stores invalid records in reject tables, supports incremental loading using MERGE, and maintains execution logs for monitoring.
-
Updated
Sep 5, 2026
Add this topic to your repo
To associate your repository with the incrementalloading topic, visit your repo's landing page and select "manage topics."