Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.
The simplest way to scale Python.
Data pipelines from re-usable components
The open-source Useful SDK. One python decorator in the Useful library allows for full observability of Python functions within an ETL.
Data Cleaning for Pyspark
A project structure for doing and sharing data engineer work.
Lien de l'application
Master the AWS Data Stack! 🚀 This repository features 15+ Industrial Data Engineering Projects covering Serverless ETL, Real-Time Streaming, & Data Warehousing. Hands-on labs for S3, Lambda, Spark, Airflow, Snowflake, Redshift, Kinesis, & Glue. Includes production-grade CICD pipelines. A complete roadmap to becoming a top Data Professional.
End To End MLOPS Project With ETL Pipelines- Building Network Security System
Lightweight, production-hardened Python utilities for data engineering: resilient HTTP, PostgreSQL pooling, and structured JSON logging with automatic secret redaction.
DataSift auto applies a data pre-processing pipeline to Data Science Projects.
Python ETL pipeline normalizing Ahmedabad transit APIs (BRTS, AMTS, Metro) into unified JSON/GTFS datasets for GraphHopper and OpenTripPlanner
Big Data ETL pipeline for Brazilian e-commerce data. Implements data ingestion, transformation, and storage using Apache Spark, Hadoop, and SQL. Designed for scalable data processing and analytics.
This project demonstrates a comprehensive data warehousing and analytics solution, from building a data warehouse to generating actionable insights. Designed as a portfolio project, it highlights industry best practices in data engineering and analytics.
End-to-end Azure Data Factory project transforming raw sales data into customer-level insights using pivot transformation and storing results in Blob Storage.
Complete portfolio of data engineering projects from Udacity's Data Engineering with AWS Nanodegree.
A robust ETL pipeline using MySQL for data transformation and Python for programmatic visualization of the 2018 NYC Squirrel Census (3,000+ records).
🚀 A comprehensive showcase of projects and skills from the IBM Data Engineering Professional Certificate! 📚 Features include: 🔄 ETL pipelines, 🗄️ data warehousing, ⚡ big data processing with Spark/Hadoop, 🛠️ database administration, and 📈 business intelligence dashboards. Built with 🦾 to demonstrate real-world data engineering capabilities!
🗄️ IBM Relational Database Administrator with GenAI Certificate Portfolio – A comprehensive collection of projects, labs, and assignments showcasing expertise in relational database administration, 🏘️data warehousing, 🔁ETL pipelines, and 🤖Generative AI integration for modern database management.