On a single whiteboard we sketched three boxes —
ETL → Warehouse → BI — then pencils the real logos the team just picked: connectors flow into the warehouse, the warehouse feeds the BI layer. Arrow labels show refresh frequency (nightly pull from CRM, hourly pull from payments, on-demand for ad spend). Nobody argues about Visio perfection; the goal is a poster-sized picture the whole office can read.
Compliance and accessNext, overlay security tape. Edit rights stay with the analyst and one backup; everyone else is view-only. If customer emails or transaction IDs are present, tag the table with a GDPR / PII flag so Looker or Power BI can automatically mask those fields on export. The investor role gets a stripped-down dashboard with no personally identifiable rows — just totals and trends.
Concrete outputThe end deliverables are deliberately minimal:
- Stack diagram — that whiteboard flow, recreated in a single PPT slide so it can live in onboarding decks.
- Access matrix — a table that lists each user group (executive, analyst, sales lead, investor) and the exact permissions (view, edit, download) for every layer of the stack.
These two artefacts become the build-team’s north star; if a future request doesn’t fit the diagram or the matrix, it waits for a new design cycle rather than derailing the current sprint.
3.
Build (≈ 20-30 h)Wiring the pipes, shaping the tables, and pushing the first dashboards to users.
Hooking up the data feedsOpen Zapier (or Make, if the team already uses it) and drop in ready-made connectors: one scenario pulls Bitrix24 deals every hour, another fetches Shopify orders every fifteen minutes, a third grabs Google Ads cost once a night after the billing API settles. If the company already pushes six-figure rows per day, swap those no-code flows for Airbyte or Fivetran so schema drift and bulk loads stay painless. Every connector ends with a scheduled timestamp—5 a.m. or midnight—so the warehouse fills before office hours.
Modelling the warehouseOnce fresh rows land, flip to the SQL editor: raw column names like
paid_amount_cents become business terms such as
Net_Revenue; timestamp fields are normalised to UTC; and a thin
date dimension is added so every chart slices cleanly by day, week, month, quarter. Just enough to make sense, not enough to paint ourselves into a corner.
Building and stress-testing the dashboardsNext, open Looker Studio (or Power BI, depending on the ecosystem) and lay out three starter views: an executive KPI snapshot, a funnel drill-down, and a financial overview. Before anyone outside the data team sees a pixel, the analyst runs a full click-through: every filter, every drill, every total cross-checked against the warehouse. Our rule of thumb is: if you can’t reconcile a single value in under ten seconds, the dashboard isn’t ready.
What ships at the endBy the twentieth hour—or thirtieth in a larger stack—the company has
working dashboards that refresh overnight and a small SQL file (or a Google Sheet) that documents every metric definition. Stakeholders open their browsers at 9 a.m., see data stamped “updated at 05:07,” and never touch the old Excel workbooks again.
4. Sustain (≈ 8-12 h upfront, then 1 h/month)The stack is live, dashboards are humming, but we refuse to hand-over until the team can run it “lights-out.” We block one final session to hard-wire three guard-rails.
Automated pipeline monitoringFirst, hook each ETL job to a Slack channel named
#data-status. A Zapier step (or an Airbyte webhook) fires a green
“Load finished in 3 min 12 s” message after every successful run; a red alert pings “Orders_to_Postgres failed at 02:48 — 503 from Shopify API” if something breaks. That instant feedback means the on-call analyst can retry the task over coffee instead of discovering stale numbers at noon.
Living KPI documentationNext open Confluence and spin up a
“Metrics Catalogue.” Each page follows the same template:
- Name: Customer Acquisition Cost (CAC)
- SQL view / LookML: vw_cac_monthly
- Owner: Marketing Operations
- Refresh cadence: nightly at 05:00 UTC
- Formula notes: (Ads_Spend + Sales_Salaries) / New_Customers
- Because every metric is a link, anyone can click straight into the SQL view to trace lineage. Undocumented KPIs rot; documented KPIs scale.
Weekly backups — the five-minute ritualFinally, schedule a weekly cron job that dumps the Postgres database to cloud storage and exports each Looker (or Power BI) dashboard as a JSON definition. The task finishes in under five minutes and emails a checksum to IT; if the company ever needs to restore after a rogue schema change or licence mix-up, they’re one
pg_restore away from full recovery.
DeliverablesThe Sustain phase produces a concise
run-book: one-page SOPs for “Restart a failed load,” “Add a new metric,” and “Rollback from backup,” plus the Slack/Teams alerting hooks. With those pieces in place, the data stack runs hands-off—owners just glance at the green ticks each morning and get on with driving the business.