Best Practices

The platform is underpinned by the following non-negotiable design principles that govern every architectural decision.

• Sovereignty — runs entirely in-jurisdiction with no mandatory external data transfer.
• Open standards — Apache Iceberg, Kafka, Trino, Spark, and dbt throughout — avoiding proprietary lock-in.
• Governance first — Atlas wraps every layer; governance is built in, not retrofitted.
• Medallion quality pattern — raw data preserved while consumers always receive certified, validated data.
• Single semantic truth — one definition of every KPI, exposed consistently via the semantic layer.
• Separation of concerns — ingestion, storage, processing, intelligence, and consumption are decoupled; changes do not cascade without governed promotion.
• AI-native — vector stores, feature stores, MLOps, and multi-agent orchestration are first-class citizens.

Golden rules

Every rule here exists because breaking it caused a real incident.

Ingestion

1. Decide partitioning before you deploy. Use none or month; never day() on big backfills. Why: the partition spec is fixed at deploy, and day-partitioned backfills crashed ingestion with out-of-memory errors and created millions of tiny files.
2. End dates are exclusive, bounds are midnight, and never in the future. Why: whole-day, half-open ranges. Future days get recorded as loaded-but-empty.
3. Check "Already loaded" before any historical load. Why: historical loads append without de-duplication, so an overlap doubles rows.
4. Backfill one large table at a time. Why: concurrent large backfills competed for memory and produced short or duplicated counts.
5. Verify counts after a backfill (monthly counts vs the source, or Reconciliation). Why: a green run means the steps finished, not that the table is complete.
6. Don't re-trigger in a panic. Wait for the 5-minute reconciler, and check chunk status first. Why: repeated triggers during a backfill multiply work and confuse state.
7. Leave Tuning defaults alone. Output Batch Size 0, Concurrent Tasks 1, Avro logical types on. Why: a batch size of 1 once turned 350K rows into billions.
8. Pick a bookmark column that is indexed, never NULL, and updated on change. Why: otherwise updates are missed or every run scans the whole source.
9. One destination per pipeline, one pipeline per table. Why: shared ownership caused type drift and write crashes.
10. Nothing runs until it's scheduled. Why: the most common "stale data" complaint was simply an unscheduled pipeline.

Transformation

11. In the dbt terminal: commit, then push immediately. Why: the terminal workspace resets to the repository copy, and unpushed commits are destroyed.
12. Commit packages.yml, package-lock.yml and dbt_packages/. Why: air-gapped schedulers can't download packages.
13. Keep the whole dbt project compiling. Why: scheduling parses every model, so one broken file breaks all jobs.
14. Always pass --target in the terminal, and keep each model to one engine's dialect. Why: the terminal's default target may differ from the UI's.
15. De-duplicate and enforce NOT NULL in silver. Why: bronze is raw and nullable by design.
16. Re-save a job after changing its models. Why: jobs run their own copy of the project.

Serving & analytics

17. Cubes and dashboards read gold, never bronze.
18. Pre-join one-to-many relationships in dbt gold. Why: fan-out inflates metrics and blocks materialization.
19. Add a Refresh cubes node after gold in the job. Why: warehouse MVs over Iceberg don't notice new data by themselves.
20. Use Export CSV for large results, not a bigger Limit. Why: the grid caps at 10k rows, while export is complete and governed.
21. Filter on dates, select only needed columns, filter before joining. Why: per-query memory is finite, and Iceberg rewards pruning.

Notebooks

22. Aggregate in Spark; .toPandas() only small results. Why: the kernel has only a few GB.
23. Experiment in sandbox_<you>, productionise in dbt + Jobs. Why: notebooks stop after 30 minutes idle and aren't scheduled.
24. After two failed spawns, stop and report. Why: the hub shuts itself down after five.

Security & access

25. Sensitive analysis happens on per-user governed surfaces (Workbench Lakehouse, JDBC, cubes, impersonated BI connections). Why: Voir, the warehouse tab, notebooks and hand-made BI connections use shared identities.
26. Scope governance policies narrowly and test with a non-admin user. Why: one catalog-wide rule once hid every schema.
27. Update saved passwords in tools right after changing yours. Why: stale retries trigger lockout.
28. Don't build on Lakebase (internal Postgres). Why: it holds platform state, and changes there can break pipelines.