Works above the source technology
Because comparison occurs on the records returned to the pipeline, the same approach can be used across APIs, databases, files, and application data.
Architectural deep dive
Synthetic CDC
Identify meaningful inserts, updates, and deletions from business data even when the source cannot provide a native change stream or transaction log.
Key-based comparison
Source-independent
Net changes
Schema-aware logs
Change capture without transaction-log access
Native CDC is powerful, but it is not always available. SaaS APIs, files, reports, legacy applications, and restricted databases may expose only the current state. Synthetic CDC compares that state with the last known state and emits the logical changes that matter downstream.
Keys define record identity
One or more key columns identify a unique business record. DataZen uses those identities to determine whether a record is new, changed, unchanged, or deleted.
Observe only meaningful fields
Volatile values such as retrieval timestamps can be excluded from change identification so they do not create false updates on every execution.
Persist a usable change log
When changes exist, the capture result is stored with schema and execution metadata so downstream readers can process and replay it consistently.
Compare the current state with the previous state
- Read a candidate data set. Retrieve the current records from the source, optionally bounded by a source-side watermark or window.
- Identify records. Apply the configured business keys and determine which fields should participate in change detection.
- Calculate the differential. Compare the current state with the prior captured state and classify inserts, updates, and deletions.
- Publish only changes. Create a change log when at least one record changed, then make it available to downstream processing.
First execution establishes state
The first keyed capture has no previous state to compare against, so the returned records form the initial capture. A reinitialization intentionally resets that baseline. Without keys, capture does not perform the differential and the complete retrieved data set is captured.
Combine watermarks and Synthetic CDC
A high watermark reduces what must be requested from a cooperative source. Synthetic CDC determines what actually changed within the returned records. They solve different problems and can be used together to reduce source reads and downstream processing.
- APIs and SaaS applications: detect changes from periodic snapshots when no event feed is available.
- Restricted databases: capture business changes without requiring access to transaction logs.
- Files and reports: compare repeated extracts using stable record keys.
- Downstream synchronization: send only changed records instead of reprocessing every unchanged row.
Synthetic CDC turns snapshots into a governed stream of business changes, independent of native source CDC support.
Back to DataZen