DataZen HTTP API Documentation

Enzo DataZen


Introduction

The DataZen HTTP API allows developers to control and program DataZen agents and jobs. The HTTP API provided in this documentation is the same used by the DataZen Manager application.

In order to successfully access DataZen agents and jobs using the HTTP API, you will need the following:

  • A DataZen agent installed and running
  • A valid license key
  • HTTP API Authentication Keys

The sample screenshots provided in this documentation use Fiddler; however, you can use any HTTP utility using the payload samples provided.

Follow the steps provided in the Installation Guide to configure the agent and apply your license key.

License Key

A valid license key needs to be applied to the DataZen agent against which the HTTP APIs will be executed. The easiest way to apply the license key is to use the DataZen Manager application.

Security Configuration

The HTTP API requires the use of Access Keys. A DataZen agent can be configured to restrict access to the API using the following mechanisms:

  • API Access: the HTTP calls can be restriced to accessing either the Agent API or the Jobs API
  • HTTP Get: the HTTP calls can be restricted to access GET operations only (read-only)

Shared Access Keys are automatically generated when an agent is configured the first time. To view the Shared Access Keys, start the DataZen Manager, select the agent, and choose Configuration -> Agent Settings, and choose the Shared Access Keys tab.



Restrictions

A few HTTP API handlers are not available for security reasons and require the DataZen Manager to administer:

  • Agent Configuration: Any agent configuration setting, including setting the license key, cannot be modified using the HTTP API
  • Central Connection Strings: The Central Connection Strings used in jobs cannot be listed or managed using the HTTP API

Sample HTTP Call

The following HTTP call shows you how to query an agent, listening on port 9559, authenticate using a Shared Access Key (using the Authorization Header) and returns a JSON payload that provides the current version of the agent:

All HTTP calls to a DataZen agent must be authenticated using the Authentication header. The only authentication mode supported in this release is SharedKey.

HTTP Request

GET http://localhost:9559/version  
Authorization: SharedKey 123456789426478564563123

Response

{
"URL":"http://localhost:9559",
"Running":true,
"Major":1,
"Minor":0,
"Revision":23770,
"Build":7866,
"JobsRunning":0
}

Parameters

If an API requires parameters, they are to be added as part of the Query String of the URL. The following example shows how to call the /job/output endpoint:

GET http://localhost:9559/job/output?guid=B427BC0D-57DB-4C88-A847-7B4D07AFBCC5 
Authorization: SharedKey 123456789426478564563123


Agent API Section

This section covers the HTTP APIs available to manage and monitor DataZen agents.


Start

Changes the status of the agent to "running" and resumes job processing. Any active job that missed its processing window will start.

/start
POST

Resume job processing.

Input Parameters (none)
 
Response
OK Success response code


Stop

Changes the status of the agent to "stopped" and stops job processing. Does not cancel jobs that are still running, but prevents future active jobs from starting on schedule.

/stop
POST

Stop job processing.

Input Parameters (none)
 
Response
OK Success response code


Status

Provides the current status of an agent.

/status
GET

Retrieves configuration settings of an agent.

Input Parameters (none)
 
Response
string URL The URL this agent is listening on (ex: http://localhost:9559)
bool Running When true, the agent is in a running status
string StageDatabaseServer The name of the database server where the Stage database is located
string StageDatabaseName The name of the Stage database
string StageLogin The UserId used for the Stage database
string LicenseFile The License File for the agent
object DataZenLicense show/hide
datazenLicenseTemplate
object Version show/hide
agentVersionTemplate

Sample Output

show/hide
{
    "URL":"http://localhost:9559",
    "Running":true,
    "StageDatabaseServer":"devlap04",
    "StageDatabaseName":"EnzoStaging",
    "StageLogin":"",
    "LicenseFile":"...",
    "DataZenLicense": {
        "AcceptSource":true,
        "AcceptTarget":true,
        "AllowDatabases":true,
        "SchedulerAllowed":true,
        "DataPipelineAllowed":true,
        "DirectJobsOnly":false,
        "AllowMessagingHubs":true,
        "AllowDriveFiles":true,
        "AllowPGPEncryption":true,
        "MaxJobs":-1,
        "AllowDataCopy":true,
        "AllowRBAC":true,
        "AllowHeuristics":true,
        "Edition":"Enterprise",
        "LicenseFile":"...",
        "IsValid":true,
        "LicenseValidationError":null,
        "LicenseValidationResult":4,
        "LastValidationMessage":null
    },
    "Version": {
        "URL":"http://localhost:9559",
        "Running":true,
        "Major":1,
        "Minor":0,
        "Revision":23770,
        "Build":7866,
        "JobsRunning":0
    }
}


Version

Provides the version of an agent.

/version
GET

Retrieves the current version of an agent.

Input Parameters (none)
 
Response
object Version show/hide
agentVersionTemplate

Sample Output

show/hide
{
    "Version": {
        "URL":"http://localhost:9559",
        "Running":true,
        "Major":1,
        "Minor":0,
        "Revision":23770,
        "Build":7866,
        "JobsRunning":0
    }
}


Job Debug Output

Provides the latest debug output for of given job. The output is limited to the last available 200 entries.

/job/output?guid=123456789
GET

Gets the last 200 debug messages of a job given its job identifier.

Input Parameters
string guid The job identifier
 
Response
object[] OutputMessage show/hide
long SequenceA session unique identifier for the message; resets to 1 when the agent restarts
string GuidThe unique identifier for the job
DateTime DateAddedDate/time when the message was recorded
string MessageThe output message
int LevelThe level of the message (0: debug, 1: info, 2: warning, 3: error)

Sample Output

show/hide
[
    {
        "Sequence":1,"
        Guid":"2b76c83008174135849ab0c7619a6d9e",
        "DateAdded":"2021-07-16T14:16:49.5078227Z",
        "Message":"Reader starting...",
        "Level":1
    },
    {
        "Sequence":2,"
        Guid":"2b76c83008174135849ab0c7619a6d9e",
        "DateAdded":"2021-07-16T14:16:49.6019398Z",
        "Message":"Strategy: full",
        "Level":1
    },
    {
        "Sequence":3,
        "Guid":"2b76c83008174135849ab0c7619a6d9e",
        "DateAdded":"2021-07-16T14:16:49.7131806Z",
        "Message":"Schema extracted successfully",
        "Level":1
    }
]


Job History

Provides the execution history of a job.

/job/history?guid=123456789
/job/history?guid=123456789&limit=500&listenerType=writer&filter=all
GET

Gets the execution history of a job.

Input Parameters
string guid The job identifier
limit Maximum number of rows to return (default: 100)
listenerType The listener type (reader, writer) (default: reader). Direct Jobs are considered reader jobs.
filter When empty or not provided (default), only returns execution history that identified or processed records, or that contain errors. When the filter is set to all, includes execution history that did not have any records identified or processed.
 
Response
object[] SyncStatus show/hide
statusTemplate

Sample Output

show/hide
[
    {
        "Guid":"2b76c83008174135849ab0c7619a6d9e",
        "ListenerType":"reader",
        "ElapsedSQLTime":0.0218252,
        "ElapsedMapRecudeTime":0.0929266,
        "RecordCount":34,
        "RecordsAvailable":4,
        "DeleteCount":0,
        "ErrorMessage":"",
        "Success":true,
        "JobKey":"SQLDatabases",
        "ExecutionId":"1626445009724",
        "PackageFile":"C:\\tmp\\datasync_reader\\test\\SQLDatabases_1626445009724.eds",
        "CreatedOn":"2021-07-16T14:16:50"
    }
]


Jobs Info

Provides the configuration settings of a job, or all jobs, including any direct writer settings and last job status.

/jobs/info
/jobs/info?guid=123456789
GET

Provides the configuration settings of a job.

Input Parameters
string guid The job identifier (optional)
 
Response
object[] JobInfoPackage show/hide
string GuidThe job unique identifier
int JobTypeThe type of job (1: reader, 2: writer)
string JobKeyThe name of the job
bool ActiveFlag indicating whether the job is active
object JobReader show/hide
jobReaderTemplate
object JobWriter show/hide
jobWriterTemplate
object LastStatus show/hide
statusTemplate
bool IsDirectModeFlag indicating whether this is a Direct Job
object DirectLastStatus show/hide
statusTemplate

Sample Output

show/hide
[
    {
        "Guid":"5acae8076f8c4c849e3c3aab38976c0d",
        "JobType":1,
        "JobKey":"Films",
        "Active":true,
        "JobReader": {
            "Guid":"5acae8076f8c4c849e3c3aab38976c0d",
            "JobKey":"Films",
            "SqlRead":"SELECT * FROM Film",
            "Path":"C:\\tmp\\datasync_reader\\films",
            "UpsertColumns":"film_id",
            "TimestampCol":"",
            "LastTsPointer":"",
            "LastTsDelPointer":"",
            "ConnectionString":"mysql-sakila",
            "PropagateDelete":true,
            "SyncStrategy":"Full",
            "CreatedOn":"2021-05-08T19:20:45",
            "Status":"ready",
            "Initialize":false,
            "Active":true,
            "LastRunTime":"2021-07-16T14:27:00",
            "CronSchedule":"",
            "JsonData": {
                "AuditLogEnabled": true,
                "EncryptionFile": "",
                "BypassTopN": true,
                "TopNOverride": 0,
                "DirectWriter": null,
                "ExecutionTimeout": 0,
                "BypassSchemaDetection": false,
                "DeletedRecordsMethod": 0,
                "DeletedRecordsSQL": null,
                "CDCTokenDeletedField": "",
                "DriveFilePattern": "",
                "DriveSchemaFile": ""
                },
            "SystemName": "MySQL",
            "PipelineDef": "[]",
            "JobType":1,
            "IsDirty":false,
            "IsDBSource":true,
            "IsDriveSource":false,
            "TrackLastTs":false,
            "DirectWriter":null, 
            "AuditLogEnabled":true,
            "IsDirect":false
        },
        "JobWriter":null,
        "LastStatus":
            {
                "Guid":"5acae8076f8c4c849e3c3aab38976c0d",
                "ListenerType":"reader",
                "ElapsedSQLTime":0.0,
                "ElapsedMapRecudeTime":0.0,
                "RecordCount":0,
                "RecordsAvailable":0,
                "DeleteCount":0,
                "ErrorMessage": "There was an error connecting to the database",
                "Success":false,
                "JobKey":"Films",
                "ExecutionId":null,
                "PackageFile":null,
                "CreatedOn":"0001-01-01T00:00:00"
        },
        "IsDirectMode":false,
        "DirectLastStatus":null
    }
]


Job

Creates, updates or deletes a Job Reader, Job Writer, or Direct Job.

/job
PUT

Updates a Job Reader, Job Writer, or Direct Job. When updating a DirectJob, you need to provide both the JobReader and the JobWriter.

Input Parameters
object JobCreation show/hide
object JobReader show/hide
jobReaderTemplate
object JobWriter show/hide
jobWriterTemplate
 
Sample Request Payload
show/hide
{
    "JobReader": {
        "Guid":"764f836aaf2c4cb49fdc9a72beee38df",
        "JobKey":"PaymentSource",
        "SqlRead":"SELECT * FROM `datalake`.`payment` ",
        "Path":"C:\tmp\output\payment",
        "UpsertColumns":"payment_id",
        "TimestampCol":"",
        "LastTsPointer":"",
        "LastTsDelPointer":"",
        "ConnectionString":"mysql-sakila",
        "PropagateDelete":true,
        "SyncStrategy":"Full",
        "CreatedOn":"2021-05-13T18:53:02",
        "Status":"ready",
        "Initialize":false,
        "Active":true,
        "LastRunTime":"2021-07-02T15:43:10",
        "CronSchedule":"",
        "JsonData": {
            "AuditLogEnabled":false,
            "EncryptionFile": "",
            "BypassTopN": false,
            "TopNOverride": 0,
            "DirectWriter": null,
            "ExecutionTimeout":0,
            "BypassSchemaDetection":false,
            "DeletedRecordsMethod":0,
            "DeletedRecordsSQL":null,
            "CDCTokenDeletedField":"",
            "DriveFileFormat":"",
            "DriveFilePattern":"",
            "DriveSchemaFile":"",
            "DriveFileSelectedColumns":"",
            "DriveUpdatedFilesOnly":false,
            "DriveIncludeSubfolders":false,"DriveFolderColumnName":"",
            "DriveAddFolderColumn":false },
        "SystemName":"MySQL",
        "PipelineDef": [
            {
                "ColumnName":"col1",
                "DataType":"Guid",
                "ExpressionType":"Enzo Expression (Enzo Functions)",
                "SQLExpression":"#rndguid()",
                "MaxLength":0,
                "IgnoreIfExists":false,
                "Name": "DataDynamicColumn",
                "Disabled":false
            }]
        },
        "JobWriter":null}
Response
OK Success response code

POST

Creates a Job Reader, Job Writer, or Direct Job. When updating a DirectJob, you need to provide both the JobReader and the JobWriter.

Input Parameters
object JobCreation show/hide
object JobReader show/hide
jobReaderTemplate
object JobWriter show/hide
jobWriterTemplate
 
Response
OK Success response code

DELETE

Deletes a Job Reader, Job Writer, or Direct Job. When deleting a DirectJob, only the JobReader identifier is needed. This operation does not delete the execution log of the job.

Input Parameters
string guid The job identifier
 
Response
OK Success response code


Job Ts Pointer

Updates a Job Reader's Timestamp pointer used to determine the next starting time window for either read or delete operations. The Timestamp window is used when a Timestamp Column is used in a Job Reader to select records from a known last High Water Mark, or when a Job Reader uses a Change Data Capture stream to identify deleted records from a source system.

/job/settspointer?guid=123456789&type=read
POST

Updates the last Timestamp value of a Job Reader for either read or delete operations.

Input Parameters
string guid The job identifier
string type The type of window to modify (read, delete)
string val The new value for the time window (optional); if empty or not provided, the Timestamp is set to an empty value.
 
Response
OK Success response code


Enable/Disable Job

Enables or disables a job. When re-enabled, job processing will resume at the next scheduled interval; if the last interval was missed, the job will start immediately.

/job/enable?guid=123456789&active=0
POST

Enables or disables a job.

Input Parameters
string guid The job identifier
string active The flag to disable or enable a job (0:disable, 1:enable)
 
Response
OK Success response code


Start Job

Starts a job immediately. If the job is already running, this operation does nothing.

/job/startnow?guid=123456789
POST

Start a job.

Input Parameters
string guid The job identifier
 
Response
OK Success response code


Stop Job

Stops a job as soon as possible. If the job is not currently running, this operation does nothing. If the job is currently running, this operation sends cancellation requests to all pending operations and returns as soon as possible.

/job/stopprocessing?guid=123456789
POST

Stops a job.

Input Parameters
string guid The job identifier
 
Response
OK Success response code