Bioinformatics Data Engineering: Genomics to Analytics
Build bioinformatics data pipelines with sample identity, reproducible genomics processing, governed analytics and careful ML evaluation from ingestion to release.
Read article07 / Journal
Thoughts from buildingreal AI systems.
Notes on LLM and RAG systems, MLOps, data engineering and the craft of shipping software that has to keep working.
- 39
- Articles published
- 21
- Topics covered
02 / All articles
Everything I’vewritten down.
Filter by topic or search for a specific problem.
4 articles of 38
Modern Lakehouse Architecture with Apache Iceberg, dbt and Airflow
Define clear responsibilities for storage, table metadata, SQL transformations and orchestration. Build a lakehouse that supports repeatable runs and trustworthy datasets.
Read article
Data Contracts and Data Quality Checks for Reliable Pipelines
Turn dataset expectations into versioned contracts and useful quality checks. Learn how to handle schema changes, quarantine failures and recover without hiding broken data.
Read article
Real-Time Data Pipelines with Kafka and Change Data Capture
Build real-time pipelines around durable change events, explicit ordering and replay. Understand Kafka, CDC snapshots, idempotent consumers and recovery at the sink.
Read articleDiscuss an idea
Working on somethingyou want to talk through?
If one of these posts is close to a problem you have, tell me about it - I’m happy to compare notes.