Job Reader: Messaging Consumer

A Messaging Consumer is a specialized job that does not run on a schedule; instead, it either listens continuously for incoming messages or is in a stopped status. The job automatically reconnects to the source messaging platform in case of temporary network connectivity problems and resumes capturing messages from where it left off.

Messaging consumers are not available with DataZen Cloud Agents; a self-hosted agent is needed.

When creating a new Messaging Consumer, you can choose from two options:

  • New Messaging Consumer: create a messaging consumer that either stores messages in a Change Log or forward them to another platform. This option gives you three separate replication modes:
    • Passthrough: same as a passthrough consumer
    • Message to Dataset: converts a message payload, header, and properties into a data row for further processing
    • Payload to Dataset: converts the JSON/XML message payload into a dataset (rows and columns)
  • New Passthrough Consumer: create a messaging consumer that sends incoming messages directly to another messaging platform

Before creating a Messaging Consumer job, you need to create a Connection String and ensure you can connect to the messaging platform successfully. Connection strings can be created for the following platforms:

  • AWS SQS
  • AWS Event Hub
  • Azure Service Bus
  • Google Pub/Sub
  • Kafka
  • MSMQ
  • RabbitMQ

To avoid data-loss, messages are first persisted and encrypted to disk before being acknowledged and consumed from the source system. Then, at the interval specified (or as soon as possible), these temporary files are processed and either forwarded to the target system specified or stored into a Change Log.

Create or Edit a Messaging Consumer

This screen allows you to configure your messaging consumer by selecting the source messaging platform (Connection String). The Message Processing option determines how the messages will be forwarded to the target system. Some of the job options available vary based on this setting.

When applicable, select the Queue/Topic to use. When applicable, select or type the metadata and/or headers to forward.

If the source system has no messages available to peek from, you can manually enter a sample payload to simulate how the data will be transformed below. The parse sample payload option becomes available when the Sample Payload field is provided.

If the source messaging platform has messages available, you can also click on peek from source queue to retrieve a message and preview the output.

When you are editing a Messaging Consumer, you must first stop the listener if it is running.

Create a Passthrough Consumer

When choosing to create a Passthrough Consumer from DataZen Manager, a simpler screen is displayed to faciliate creating the job. Most of the basic options are similar than when creating a Messaging Consumer; however, you must select a Target system. No other job options are provided on this screen.

To switch to the full screen and change other options, and preview the data before creating the job, or to use different target systems, click on Open Advanced Editor.

This screen is only available when creating Passthrough Consumers. Editing the job later will open the full editor described above.

Advanced Options

The Messaging Consumer job allows you to group messages together so they are sent in a batch to the target system, if the target supports batched records. To enable this feature, use the Advanced Settings tab and check the Batch incoming messages... option and specify the number of messages that should be sent as a batch. When this option is checked, you need to specify the TTL in seconds (Time To Live) to flush the messages present in the internal DataZen queue if the Batch threshold has not been met.