I design and run production `data pipelines` and `lakehouse` platforms on AWS and Microsoft Fabric, from batch and event-driven ingestion through validated, analytics-ready models. I build validation, reconciliation and monitoring into every stage so data is `reliable`, `replayable` and `traceable`. Right now I lead data engineering on a multi-country finance reporting platform at AmaliTech, working with Finance, Controlling and engineering teams to replace manual reporting with trusted data. Before data, I was a software engineer building Django REST APIs and PostgreSQL schemas, so I still care a lot about clean contracts and well-modelled tables.
Lakehouses, event-driven pipelines and streaming systems on AWS and Microsoft Fabric. Here are a few of my favourites.
End-to-end data engineering and data science platform on Microsoft Fabric for a fictional Ghanaian retail bank. Unifies siloed core banking, cards, mobile, loans, CRM, GL and credit bureau sources into one source of truth, a BI layer and an ML layer for profitability, churn, fraud and credit risk.
Production-grade batch ETL: CSV/XLSX drops on S3 are validated and deduplicated by Glue PySpark into Delta Lake, exposed via Athena. S3 → EventBridge → SQS → debounced Lambda → Step Functions, with SNS alerting and full Terraform IaC.
Microsoft Fabric batch analytics for a fictional cross-border payments company. Customers, FX rates and fee schedules flow from OneLake files to Delta silver tables to a gold star schema for corridor, currency and revenue reporting.
Event-driven serverless pipeline: dropping a streams CSV wakes Step Functions, which runs three Glue jobs (validate → transform → ingest) to compute daily KPIs, serves them from a single DynamoDB table and emails on failure.
Containerised real-time pipeline: a producer emits customer heartbeat events to Kafka, a consumer validates them with Pydantic, upserts idempotently into PostgreSQL and routes bad messages to a DLQ topic. Grafana for visualisation.
Configuration-driven pipeline that profiles, validates, cleans and masks customer data before sharing. Produces quality diagnostics, before/after validation results, a masked analytics-safe dataset and an execution audit report.
Airflow-orchestrated ETL for Bangladesh flight price data: ingest raw CSV, validate and transform with Spark, compute KPIs, stage in MySQL and publish to a PostgreSQL analytics database. Fully containerised.
AI university assistant grounded in official institutional documents. Retrieves handbook and policy passages, answers only from that content, cites page sources and ships an admin dashboard for re-indexing.
More on GitHub.
Building a data platform or need pipelines you can trust? Send me an email or DM me on X and I'll get back to you. Thank you ❤️