Transforming data (dbt)

Transformations are dbt projects. You edit them in Data Engineering → Workbench → DBT Editor, run them from the editor or its terminal, and schedule them with Jobs.

Where your code lives

There are three copies of your dbt project:

 Version Control (git: dbt-models repo)   ◄── the source of truth
        ▲                     │
   editor saves          cloned into
   directly here              │
        │           ┌─────────┴───────────┐
   DBT Editor       ▼                     ▼
               Terminal workspace     Job copy (scheduled runs)
               (/workspace, temporary) (refreshed when you save a job)

• The DBT Editor saves straight to the repository.
• The terminal works on its own clone. It refreshes from the repository when a terminal session opens. Edits made in the editor appear there after you open a new terminal session.
• Scheduled jobs use their own copy, refreshed when you save the job.

⚠️ The terminal workspace is temporary

The terminal workspace is wiped whenever the dbt shell restarts. That happens on platform upgrades, maintenance and node moves, and after 30 minutes idle on installs where idle scale-down is enabled. You won't get a warning. On refresh the workspace is reset to the repository copy:

• Committed but unpushed work is destroyed. This has caused real loss of dozens of model files.
• Untracked files survive the reset.

The rule:

git add -A && git commit -m "…" && git push      # always push immediately
git status -sb                                   # must NOT show [ahead N]

If you can't push right now, park the work on a rescue branch:

git push origin HEAD:refs/heads/rescue-$(date +%Y%m%d-%H%M)

Targets (engines)

Target	Engine	Writes to	Use for
spark	Spark 4 Connect	lakehouse.<schema> (Iceberg)	Heavy transforms, large joins. Default for new projects and jobs.
trino	Trino	lakehouse.<schema> (Iceberg)	Lighter SQL models; models reading federated sources (postgres, dnw)
dnw	DN Warehouse (StarRocks)	your warehouse database	Serving-layer tables built natively in the warehouse (models/warehouse/)

Pick one engine per model and stay with it. The dialects differ:

• file_format='iceberg' and other Spark configs fail on Trino with INVALID_TABLE_PROPERTY.
• Models reading relational catalogs (postgres, dnw) fail on Spark with "Cannot set database in spark!".

⚠️ In the terminal, a bare dbt run uses the profiles.yml default, which is trino (catalog lakehouse, schema default), even if the UI button shows Spark. Always pass the target:

dbt run --target spark --select silver.orders_clean+
dbt build --target trino --select tag:finance

Recommended project layout

models/
  staging/  or  bronze/    # thin views over lakehouse.bronze_<src>.* : rename, cast
  silver/                  # cleaned, de-duplicated, conformed entities
  intermediate/            # reusable joins
  marts/                   # gold: facts & dimensions per domain (core/, finance/, …)
  warehouse/               # dnw-only models (build with --target dnw)
snapshots/                 # SCD2 history
tests/  macros/  seeds/

Schema naming. With the default profile schema default, a folder config of +schema: silver builds into lakehouse.default_silver, and +schema: gold into lakehouse.default_gold. That's standard dbt behaviour: <target schema>_<custom schema>.

Declare bronze tables as sources, so lineage and freshness work:

sources:
  - name: bronze_crm
    database: lakehouse
    schema: bronze_crm
    tables:
      - name: customers
        loaded_at_field: last_modified

Credentials

You never type or export database passwords. The shell refreshes Trino, warehouse and Postgres credentials automatically, and overwrites any password you export yourself. If the warehouse target returns error 1045 after a credential rotation, open a new session in the DBT Editor.

Run history and lineage

• Runs started from the UI and runs typed in the terminal appear under Atlas → Runs. Terminal runs are picked up after they finish, as long as run_results.json is fresh.
• Lineage and docs read the target/ artifacts, which UI runs commit back to the repository.
  • Don't add target/ to .gitignore.
  • If git status shows [ahead N] after UI runs, an automatic push failed silently. Run git push, and tell your admin if it keeps happening.

Rules for projects that get scheduled

1. Commit your packages. Air-gapped installs can't run dbt deps at schedule time.

   git add -f packages.yml package-lock.yml dbt_packages/

   Without them, tests fail with 'dbt_expectations' is undefined.
2. The whole project must parse. Scheduling parses every model, not just the ones in your job. One broken model or missing source breaks every job.
3. Stay compatible with the scheduler's dbt version. The scheduled environment can run an older dbt than the terminal. Avoid version-specific config such as flags.warn_error_options: its error names differ by version and break scheduled imports.
4. Re-save the job after changing models. Saving refreshes the job's copy of the project.
5. Use tests. Add not_null / unique on silver keys and relationships between facts and dimensions. Bronze is nullable on purpose, so silver is where quality is enforced.

Modelling tips

• De-duplicate in silver. Incremental ingestion can deliver a changed row more than once. Keep the latest row per key:

  select * from (
    select *, row_number() over (partition by id order by last_modified desc) rn
    from {{ source('bronze_crm','customers') }}
  ) where rn = 1

• Use incremental models for large facts (materialized='incremental', unique_key, filtered on your load timestamp). Full rebuilds of big tables are the most common reason jobs are slow.
• Pre-aggregate in gold what dashboards and cubes need. Cubes should sit on gold, never on bronze.
• Custom generate_schema_name: if you emit bare schema names, prefix them with the workspace schema. Otherwise they may not appear in the DN Warehouse browser.

Troubleshooting dbt

Symptom	Cause	Fix
Model files vanished after a restart	Committed, never pushed; workspace reset	Always push. Admin may recover from the pod if it's still alive.
Editor change not visible in terminal	Terminal refreshes only when a session opens	Open a new terminal session
dbt ls … does not match any enabled nodes / empty models/	Terminal clone didn't sync	Contact your admin (restart the dbt shell)
Git asks Username for … in the terminal	Repository credential rotated	Contact your admin (restart the shell after you push)
Cannot set database in spark!	Trino-shaped model run on Spark	--target trino, or rewrite the model
INVALID_TABLE_PROPERTY	Spark-only config run on Trino	Switch the job's engine to Spark
CANNOT_MODIFY_CONFIG / Failed to set spark.sql.catalogImplementation warnings	Harmless dbt-spark noise	Ignore
NotAuthorized / S3V4RestSignerClient.sign on Spark	Shared Spark server's catalog session went stale	Contact your admin (Spark restart), then re-run. See Using Notebooks & Spark.
Warehouse 1045 Access denied	Password rotated mid-session	New DBT Editor session