DataZen Functions
DataZen Functions are available throughout DataZen Manager in most fields that display a function icon or show a function replacement window. DataZen Functions fall under the following categories, some of which may not be available depending on the context.
- Enzo Functions: a list of built-in functions that can be used inline
- Dataset Schema: a list of fields available from the data source or after being modified by a data pipeline to choose from
- Job Parameters: a list of job-specific parameters that can be used inline
- Dynamic Parameters: a list of parameters that are available for certain source systems
Enzo Functions can be used in multiple places throughout DataZen, including:
- Data Pipelines Dynamic Columns
- URL or payload of a job that uses an HTTP connection
- Custom SQL or HTTP Scripts in a target operation
- Target object names for database tables, no-sql tables, and file names
- Initialization and Finalization database scripts when using a DB Target
- Most data pipeline configuration settings
DataZen functions can be called using a chaining operation instead of nesting separate calls using a dot operation.
For example, #tohex64(#trim( this is a test )) can be rewritten as #trim( this is a test ).tohex64() or ##( this is a test ).trim().tohex64().
To apply the same logic on a field value during a pipeline transformation, use the {{field}} identifier: ##({{field}}).trim().tohex64()
Example: DataZen Functions in HTTP Payload
In this example, the HTTP Payload provided to a target system contains a DataZen Function and a field from the source data: - a random guid: #rndguid() - a field name replacement: {{ids}}
To insert a field or function, place the cursor at the desired insertion point, then double-click on the field or function to insert.
Certain fields display a DataZen Function icon
to show a secondary screen with a similar layout.
Click on a DataZen function to see a description and example on how to use it.
You can nest or chain functions by adding them together in a call.
For example, the following operation gets the current time in UTC, returns
an ISO8601 compliant date format, and URL encodes the output so it can be
passed in a URL function:
#urlencode(#toiso8601(#utcnow()))
#urlencode(#utcnow(o))
#utcnow(o).urlencode()
List of Functions
| Function | Description | Comments |
| ##(a) | The # operator identifies the content provided as input to the next chained operation. |
##({{field1}}).toiso8601() ##({{field1}}).trim().tolower().tohex64() |
|
#decrement(a) #decrement(a, b) |
Decrements a value by one, or by the amount specified. | #decrement({{field1}}, 1) |
| #compress(a) #compressu(a) |
Compresses a string into a byte array returned as a Base64 encoded utf-8, or unicode (u), string using the zlib algorithm | The output of this operation is a Base64 encoded string. #compress({{fieldToCompress}}) |
|
#dateadd(date, timespan) #dateadd(date, timespan, format) #utcdateadd(date, timespan) #utcdateadd(date, timespan, format) |
Adds (or substracts) a timestamp to a date provided as a local datetime (or utc datetime) and returns the new date in the format specified as a local datetime (or utc datetime). The timestamp can contain the following characters: ymdhns, and start with - for a negative timestamp. Default format: s (sortable). | #dateadd(#now(), -5d30n, s) |
| #doublequoted(a) | Surrounds the string with double quotes if needed, and it is not null. | #doublequoted(TEST) |
| #format(a, b, …) | Uses the C# string.Format syntax to build a string with any number of parameters | The C# syntax uses {0} for the first parameter, {1} for the second, and so forth. #format('User {0} lives in {1}', {{userId}}, {{state}}) |
| #fromhex64(a) | Converts a Hexadecimal string into a string representation. | #fromhex64(74657374) |
| #formatdate(date, format) #utcformatdate(date, format) | Formats a string value into a date with a specific format (following the C# DateTime.ToString(format) notation) | #formatDate(11/01/2000T08:00:00.0000, 0:MM/dd/yy H:mm:ss) |
| #fromepoch(a) #fromepoch(a,b) #fromepoch_sec(a) #fromepoch_sec(a,b) |
Converts an epoch number, provided as seconds, into a date/time. Parameter 'b' indicates the date formatting to use. If no date formatting is used, the format provided is unspecified and not guaranteed to remain the same over time. |
#fromepoch(1728440034, G) #fromepoch_sec(1728440034, o) |
| #fromepoch_ms(a) #fromepoch_ms(a,b) |
Converts an epoch number, provided as milliseconds, into a date/time. Parameter 'b' indicates the date formatting to use. If no date formatting is used, the format provided is unspecified and not guaranteed to remain the same over time. |
#fromepoch_ms(1729540001453, G) #fromepoch_ms(1729540001453, o) |
| #increment(a) #increment(a, b) | Increments a value by one, or by the amount specified. | #increment({{field1}}, 25) |
|
#http_get(…) #http_post(…) #http_put(…) #http_delete(…) (DEPRECATED) |
(DEPRECATED) Please use the Data Pipeline HTTP Endpoint function instead. Performs an HTTP/S call to an internet resource and returns a string as output. If the call to the HTTP resource is an image, the content is returned as an Hexadecimal representation of the bytes of the image. |
See the HTTP Functions section below for further information #http_get({{field_url}}) |
| #isnotnull(a,b) | Returns value ‘b’ if value ‘a’ is not a NULL value | You can use this function to replace a non-NULL value with a constant, or the content of another field #isnotnull({{field1}}, 0) |
| #isnotnullorempty(a,b) | Returns value ‘b’ if value ‘a’ is not a NULL value nor an empty string | You can use this function to replace a non-NULL value or non-empty value with a constant, or the content of another field #isnotnullorempty({{field1}}, 0) |
| #isnull(a,b) | Returns value ‘b’ if value ‘a’ is a NULL value | You can use this function to replace a NULL value with a constant, or the content of another field #isnull({{field1}}, 0) |
| #isnullorempty(a,b) | Returns value ‘b’ if value ‘a’ is a NULL value or an empty string | You can use this function to replace a NULL value with a constant, or the content of another field #isnullorempty({{field1}}, 0) |
| #jsonformat(a) | Parses the JSON content (provided as a string) and applies JSON formatting settings specified by #setjsonsettings | #setjsonsettings(nullvaluehandling:ignore) #jsonformat({\"id\": 1, \"name\": null}) |
| #parse(a, b, c) | Parses the content 'a' (provided as a string) as either a JSON, XML, or CSV content, using Path 'b' (may be null), and returns the value of node 'c' given its name or index | |
| #parsedate(a, b, c) | Returns a date/time representation of the string provided as 'a', using 'b' as the date format to use for 'a', and 'c' as an optional output date format. | #parsedate(31 12 2018 23:50, dd MM yyyy HH:mm, G) |
| #parsejson(a, b, c) | Parses the JSON content 'a' (provided as a string), using the JSON Path 'b' (may be null), and returns the attribute 'c' given its name or index | #parsejson({\"id\": 1, \"name\": \"john\" }, null, name) |
| #parsexml(a, b, c) | Parses the XML content 'a' (provided as a string), using the XML Path 'b' (may be null), and returns the value of node 'c' given its name or index | #parsexml(<root><id>1</id><name>john<name></root>, null, name) |
| #left(a, n) | Returns the left-most n characters of a string | #left(‘this is a test’, 5) |
| #len(a) | Returns the length of a string | #len(‘this is a test’) |
| #now() #now(a) |
Returns the current datetime in the local timezone of the server on which the agent is running with an optional date formatting parameter |
The following returns the current local date of the server running the agent using the ISO 8601 standard #now(o) |
| #pick(a,b,c…) | Randomly selects a value from the list provided as parameters | Any number of parameters can be provided |
| #replace(a, b, c) | Replaces all occurrences of b with c, in string a | #replace(‘this is a cat’, ‘cat’, ‘dog’) |
| #rnddouble(a,b) | Generates a random double value between two numbers | Both the lower and upper bound values are required |
| #rndguid() | Returns a new random GUID | |
| #rndint(a,b) | Generates a random integer value (Int32) between two numbers | Both the lower and upper bound values are required |
| #setjsonsettings(P1:V1,P2:V2...) | JSON formatting options to be used by the next JSON function; must be called immediately before the JSON function. Supports multiple options. | #setjsonsettings(nullvaluehandling:ignore,formatting:none,dateformatstring:MM/dd/yyy hh:mm:ss) #jsonformat({\"id\": 1, \"name\": null}) |
| #skip(a,n) | Skips the first n characters from the string provided as the first parameter | #skip(my name, 3) |
| #singlequoted(a) | Surrounds the string with single quotes if needed, and it is not null. | #singlequoted(TEST) |
| #toepoch(a) #toepoch(a,b) #toepoch_sec(a) #toepoch_sec(a,b) |
Converts a date/time value (parameter 'a') to an epoch number, in seconds. Parameter 'b' indicates whether the date provided is in UTC already (1 or true); this is useful if the date string does not contain timezone information. If the date string already contains timezone information, parameter 'b' is ignored. |
#toepoch('10/9/2024 2:13:54 AM', 1) #toepoch_sec('2024-09-10T02:13:54.000Z') |
| #toepoch_ms(a) #toepoch_ms(a,b) |
Converts a date/time value (parameter 'a') to an epoch number, in millseconds. Parameter 'b' indicates whether the date provided is in UTC already (1 or true); this is useful if the date string does not contain timezone information. If the date string already contains timezone information, parameter 'b' is ignored. |
#toepoch_ms('10/9/2024 2:13:54 AM', 1) #toepoch_ms('2024-09-10T02:13:54.000Z') |
| #tolower(a) | Converts a string into a lower-case representation | #tolower('TEST') |
| #toupper(a) | Converts a string into an upper-case representation | #tolower('test') |
| #tohex64(a) | Transforms a value into an Hex64 representation | #tohex64(‘this is a test’) |
| #toiso8601(x) | Converts a datetime field into a ISO 8601 compliant format | This function converts a datetime field into a string in the ISO 8601 format assuming the date is in the current timezone |
| #torfc1123(x) | Converts a datetime field into a RFC 1123 compliant format | |
| #trim(x) | Removes leading and trailing spaces from the string provided | #trim(' this is a test ') |
| #uncompress(a) #uncompressu(a) |
Decompresses a byte array of compressed utf-8 data, or unicode (u), into a string using the zlib algorithm |
The output of this operation is a string.
#uncompress({{fieldToUncompress}}) |
| #unquote(a) | Removes surrounding quotes or double-quotes if any |
This example returns a string that can be interpreted as an integer #unquote('123456789') This will return the HEX representation of TEST without surrounding quotes #tohex64('TEST').unquote() |
| #unzip(a) #unzipu(a) |
Decompresses a byte array of compressed utf-8 data, or unicode (u), into a string using the GZip algorithm | The output of this operation is a string. #unzip({{fieldToUnzip}}) |
| #utcnow() #utcnow(o) |
Returns the current datetime in UTC format with an optional date formatting parameter |
The following returns the current UTC date using the ISO 8601 standard #utcnow(o) |
| #utctoiso8601(x) | Converts a UTC datetime field into a ISO 8601 compliant format | This function converts a datetime field into a string in the ISO 8601 format assuming the date is in the UTC timezone |
| #utctorfc1123(c) | Converts a UTC datetime field into a RFC 1123 compliant format | |
| #urldecode(c) | Decodes a string from its escaped representation | |
| #urlencode(c) | URL encodes the value provided so it is safe to pass as a URL parameter. | |
| #zip(a) #zipu(a) |
Compresses a string into a byte array returned as a Base64 encoded utf-8, or unicode (u), string using the GZip algorithm | The output of this operation is a string. #zip({{fieldToUnzip}}) |