LOAD

Overview

The Load operation allows you to load changes from a previously created change log and indicates that the SQL CDC script is a write pipeline. It must be the first and only load operation in a SQL CDC script when used.

Unless the full name of a change log is specified, the Load operation will use the next available change log. By design, write pipelines keep a high watermark on their last successful execution timestamp. This timestamp is also part of the naming convention of change logs; this allows the write pipeline to filter the list of change logs available for execution from the last successful write.

The Load operation is not supported in a read pipeline (a pipeline that starts with the SELECT operation).

Syntax

Loads the next available change log, or the log name specified, and optionally limits the data stream to upserted or deleted records.

LOAD 
    < { UPSERTS } { DELETES } > 
    
    -- the change log name
    FROM { [CONNECTION] } '...'
    
    { PATH '...' } 

    { WITH 
        { KEY_COLUMNS '<field>,...' }
        { CHECK_SIGNATURE }
        { PREVIEW_FILE '...' }
    }
;

PATH	The directory, path, bucket or container to use
KEY_COLUMNS	Optional comma-separated columns to use for UI support when generating scripts
CHECK_SIGNATURE	When specified, checks the signature of the change log
PREVIEW_FILE	Loads the specified change log for sample data in preview operations

Example 1

-- Load the next set of records previously captured in cloud folder using
-- CAPTURE 'mycdc' INSERT UPDATE ON KEYS 'guid' WITH PATH [adls] '/logs'

-- Specify the key columns to use for UI script generation 
LOAD FROM 'mycdc' PATH [adls] '/logs'
    WITH KEY_COLUMNS 'guid';

-- Save into a database
SINK INTO DB [sql2017] 
    TABLE 'myrssfeed' 
    WITH RECREATE;

Example 2

-- Load the next set of deleted records previously captured in a local folder using
-- CAPTURE 'mycdc' INSERT UPDATE DELETE ON KEYS 'guid' WITH PATH 'c:\tmp\logs'

LOAD DELETES FROM 'mycdc'
    PATH [adls] '/logs'
    ;

-- Save into a database
SINK INTO DB [sql2017] 
    TABLE 'table001_deleted' 
    WITH RECREATE;

Example 3

-- Load the next inserts/updates previously captured in a cloud folder using
-- CAPTURE 'mycdc' INSERT UPDATE DELETE ON KEYS 'guid' WITH PATH [adls2.0] 'container'

LOAD UPSERTS FROM [adls2.0] 'mycdc' PATH '/container' KEY_COLUMNS 'guid';

-- Save into a database
SINK INTO DB [sql2017] 
    TABLE 'table001_deleted' 
    WITH RECREATE;

Example 4

-- Load a specific change log previously captured in a local folder
-- CAPTURE 'pipeline001' INSERT UPDATE DELETE ON KEYS 'guid' WITH PATH 'c:\tmp\logs'

LOAD UPSERTS FROM 'pipeline001_1750675200000.eds' PATH 'c:\tmp\logs';

-- Save into a database
SINK INTO DB [sql2017] 
    TABLE 'table001_deleted' 
    WITH RECREATE;