Appearance
Jobs & scheduling
Data Engineering → Jobs is where anything gets scheduled: ingestion, dbt and cube refreshes, in one ordered graph.
Nothing runs until it is scheduled or triggered. A deployed pipeline or a finished dbt model does not refresh by itself.
Job kinds
| Template | What it does | Notes |
|---|---|---|
| Ingest only | Schedules or triggers one Connect pipeline | Sets the pipeline's own schedule. A blank cron removes the schedule. |
| Transform only | Runs dbt snapshots / bronze / silver / gold models | Choose by layer, folder or individual model |
| Ingest + Transform | Pipeline → dbt → (optional) Refresh cubes | Takes over the pipeline's own schedule so data isn't loaded twice |
Building a job
- + Create job → pick a template.
- On the canvas, add nodes:
- Ingestion: a deployed pipeline.
- dbt: Snapshots / Bronze / Silver / Gold, selecting All, a Folder or a Model. Set the Engine per node (Spark by default, or Trino).
- Refresh cubes: add this after gold if you use materialized cubes (Semantic Layer (Cubes)).
- Turn on Run dbt tests (recommended).
- Trigger: Scheduled (Hourly / Daily / Weekly / Monthly / Yearly / custom cron, with a timezone) or on demand.
- Save. The job appears in orchestration within about 10–30 seconds. For big dbt projects the first parse can take a few minutes.
Guardrails you may hit
| Message | Meaning |
|---|---|
409 conflict | A pipeline or model can belong to only one job. Remove it from the other job first. |
422 zero / unknown models | Your selection matches nothing in the current project. Check the names, and push your dbt changes. |
| Save refused: pipeline not deployed / workflow never published | Deploy the pipeline first |
| Can't trigger: already running | One run of a job at a time |
Timing rules
- The first run is the next cron slot in the chosen timezone. Missed slots are not replayed (no catch-up).
- Jobs created before the timezone fix ran on UTC regardless of the timezone picked. Re-save older jobs once after an upgrade and check the next-run time.
- ⚠️ Re-save a job after changing its dbt models. Otherwise it may run an old copy of the project. The symptom is a run that "succeeds" in milliseconds with zero tasks. Newer releases fail loudly instead.
- Schedule big ingestion outside source business hours, and stagger heavy jobs rather than starting them all at 00:00.
Monitoring
- Job detail: Trigger now, run history with scheduled / manual / backfill badges, and View log per task.
- Atlas → Monitoring (pipelines) and Atlas → Runs (dbt).
- A skipped step reading no work due is normal. Nothing was due, or a backfill is handling the table.
- Run times may be shown in UTC on some screens. Check the timezone label.
Troubleshooting
| Symptom | Cause | Do |
|---|---|---|
| Job never appears in orchestration | Sync delay, or a slow first parse on big projects | Wait 1–3 min; contact your admin if you see an import timeout |
| Import error "not a valid dbt error name" | Version-specific dbt flags | Remove flags.warn_error_options from dbt_project.yml |
Tests fail 'dbt_expectations' is undefined | dbt_packages/ not committed | Commit packages (Transformation (dbt)), re-save job |
| Job fires at the wrong hour | Saved before the timezone fix | Re-save |
| "Success" in milliseconds, no tasks | Stale project copy | Re-save the job |
dag_not_ready on trigger | Workflow not yet in orchestration | Retry shortly; contact your admin if it persists |
dag_import_error | Broken generated workflow | Contact your admin |
| Many jobs fail together with git-sync errors | Version-control service briefly unavailable | Re-trigger once it recovers (about 5 min); contact your admin if it doesn't |