Most teams drown in events before they extract insight. We start with Kafka (or managed equivalents) for durable streaming, clear topic contracts, and consumers that idempotently land data in Snowflake or BigQuery without duplicating business facts.
dbt is where analytics becomes engineering. Staging models normalize messy source tables; mart models expose metrics product and finance teams can trust. Tests on freshness, uniqueness, and referential integrity run in CI–broken pipelines do not reach production schedules.
Airflow orchestrates the long-running jobs dbt should not own: third-party API syncs, backfills after schema changes, and cross-system reconciliation. Python workers handle transforms that do not belong in SQL while keeping lineage visible.
The dashboard layer–whether Grafana, Looker, or embedded analytics in the product–reads from curated marts, not raw tables. That separation is what keeps executives confident and engineers from rewriting the same JOIN every sprint.
