Data Engineering

DataNature Data Engineering brings raw data into the platform and progressively refines it into business-ready datasets — spanning ingestion, distributed transformation, and Medallion promotion, all catalogued and lineage-tracked in Atlas.

Concepts

Batch vs. real-time

Batch and real-time are two processing semantics used in DataNature for data ingestion, transformation, and operational processing. Scheduled loads are handled by DataNature Connect, while continuous, near-zero-latency event flows are handled by DataNature Stream over Kafka and CDC.

Batch semantics

With batch processing, DataNature processes all data currently available at the time the job runs. It does not track what has already been processed, so each run may reprocess earlier data to produce a complete, corrected result. This works well for scheduled loads, periodic reconciliation, and workloads where freshness measured in minutes or hours is acceptable.

Real-time semantics

With real-time processing, DataNature tracks incoming changes and processes only new events after the last checkpoint. This is the right fit for Kafka-based event flows and CDC-driven pipelines where low latency matters more than full reprocessing on every run. The result is continuous delivery of new inserts, updates, and deletes as they happen.

Batch vs. real-time

Batch is simpler to operate and reason about, especially when you want a complete refresh of a dataset. Real-time is more efficient for incremental changes and supports sub-second or near-real-time use cases, but it becomes more complex when you need stateful logic such as joins, aggregations, deduplication, late-arriving events, or out-of-order data handling.

Recommended usage

Use DataNature Connect for scheduled ingestion, backfills, periodic syncs, and workloads where a full or partial refresh is acceptable. Use DataNature Stream for operational events, CDC replication, event-driven integrations, and scenarios where downstream systems need immediate updates.

Tables and views

Tables and views are the core data objects for data engineering in DataNature. Data persists as Apache Iceberg tables, which provide ACID transactions, schema evolution, and time travel, while certified views and KPIs are exposed through the Atlas semantic layer.

Data object	Role in DataNature	Storage behavior	Typical use
Table	Persistent system of record	Stored as Apache Iceberg with ACID guarantees, schema evolution, and time travel	Durable ingestion, curated datasets, historical analysis
View	Logical abstraction over tables	No separate data storage; computed at query time	Simplifying access, hiding complexity, standardizing business logic
Certified view	Governed business abstraction	Exposed through the Atlas semantic layer	Trusted, reusable definitions for reporting and applications
KPI	Governed metric definition	Managed in the Atlas semantic layer	Consistent metric calculation across dashboards and consumers

Table — A table is the persistent storage layer in DataNature and is typically implemented as an Apache Iceberg table. Iceberg supports transactional consistency, schema evolution without rewriting data files, and time-travel queries through versioned snapshots.

View — A view is a virtual abstraction over one or more tables that does not duplicate underlying data. Views are useful for simplifying complex logic, standardizing access patterns, and presenting curated business-friendly structures on top of Iceberg-backed datasets.

Certified views and KPIs — Certified views and KPIs are governed semantic assets exposed through the Atlas semantic layer. This gives downstream users a consistent business definition for metrics and dimensions, so teams can reuse trusted logic instead of redefining KPIs in each report or application.

Tables vs. views — Tables are the system of record in DataNature, optimized for durable storage and lifecycle management. Views, especially certified semantic views, are the consumption layer that simplifies access and enforces consistent business meaning across analytics and operational use cases.

Capture, promotion & governance

• Change Data Capture & snapshots — CDC captures row-level INSERT/UPDATE/DELETE from source logs; Connect also supports full and incremental, watermark-based loads.
• Medallion promotion — Data is refined Bronze → Silver → Gold; promotion is explicit and validated, and lower layers are never deleted, preserving a complete audit trail.
• Governance by default — Every ingestion and transformation is catalogued, access-controlled, PII-masked, lineage-tracked, and audit-logged by Atlas.

Ingestion

Three managed channels land data in the Bronze layer, each governed end-to-end by Atlas.

DataNature Connect · 200+ connectors · batch / incremental / full

• Over 200 pre-built adapters for relational databases, SaaS/REST APIs, SAP/ERP, and flat files (CSV/JSON/XML).
• Full-snapshot, incremental (watermark-based), and change-based load modes, with schema mapping and normalisation at load time.
• Built-in scheduling, orchestration, retries, and monitoring of status and data volumes.

DataNature Stream · Kafka · CDC · real-time ingestion

• Managed Apache Kafka (topics, partitions) ingesting event streams, database change logs, IoT/device telemetry, and clickstream events.
• CDC capture with in-flight stream processing, a schema registry for event contracts, and exactly-once delivery with ordering and deduplication.

DataNature DocuContext · unstructured → vector knowledge base

• Connectors for SharePoint, SFTP, and DMS; an extraction engine with text parsing, OCR, and ASR for audio.
• Field classification and extraction with confidence scoring, validation, and embedding generation into a governed vector store / KB that grounds Voir and AI.

Transformation Workbench (dbt · Spark)

• dbt — SQL-based transformation framework for Bronze → Silver → Gold pipelines.
• Apache Spark — distributed engine for petabyte-scale processing, aggregation, and feature engineering.
• Integrated query editor — exploration, testing, and ad hoc analysis, with scheduled DAG orchestration.
• Voir Copilot — an embedded assistant that explains, optimises, and fixes dbt models.
• Full lineage sync — every dbt run writes metadata back to DataNature Atlas.

Medallion Architecture (Bronze → Silver → Gold)

• Bronze — raw, immutable, exact copy of source; timestamped, partitioned, and catalogued.
• Silver — cleansed and standardised — deduplication, null handling, referential integrity, and business-rule validation.
• Gold — business-ready dimensional and fact tables; certified and version-controlled; consumed by AI, AI360, Voir, BI, and third-party apps.