Consume messages action in the Kafka package

The Consume message action in the Kafka package enables you to poll an Apache Kafka topic for a batch of messages using a consumer group, with configurable batch size, poll timeout, offset reset strategy, and commit strategy, returning all consumed messages as a table variable for processing in your automation workflow.

Prerequisites

  • An active Kafka session established using the Connect action. See Connect action in the Kafka package.
  • The target Kafka topic exists on the broker and the authenticated user has consumer permissions for that topic.
  • A consumer group ID is available.
  • If you plan to use the manual commit strategy, ensure a Commit offsets action is added after this action in the automation workflow to confirm processing. See Commit offsets action in the Kafka package.

Settings

Use the Consume message action to poll an Apache Kafka topic for a batch of messages using a consumer group. The action returns a table variable where each row represents one message. You can configure the maximum number of messages per poll, the poll timeout in milliseconds, the offset reset strategy (earliest or latest), and whether offsets are committed automatically or manually. Use manual commit strategy when you require exactly-once processing semantics; the automation must then call the Commit offsets action after successfully processing each message.

  • Session name: Select one of the following options to specify the Kafka session to be used:
    • Session name: Select the active Kafka session returned by the Connect action, or accept the default session name Default.
    • Variable: Select a session variable that includes the active Kafka session configuration.
  • Topic: Enter the exact name of the Kafka topic to consume from. For example, transactions-data.
  • Consumer group ID: Enter the consumer group identifier used to track consumed offsets. All automations sharing the same group ID share partition assignments and offset state. For example, erp-sync-group.
  • Max messages: Enter a positive integer (1 or greater) representing the maximum number of messages to retrieve per poll request. The default value is 10. Fewer messages might be returned if the topic has fewer available records.
  • Poll timeout (ms): Enter a non-negative integer representing the maximum number of milliseconds the Consume message action waits before returning messages. The default value is 5000 (5 seconds). Enter 0 to return immediately with any available messages.
  • Offset reset: Select how the consumer positions itself when no committed offset exists for the group and topic. The default is earliest. Select one of the following options:
    • earliest: Starts reading from the oldest available message in the topic. Use this option when you want to process all historical messages from the beginning.
    • latest: Starts reading from the newest message produced after the consumer subscribes. Use this option when you only want to process new messages going forward.
  • Commit strategy: Select how the consumer commits offsets after polling. The default is Auto commit. Select one of the following options:
    • Auto commit: The consumer automatically commits offsets at a fixed interval. Use this option for at-least-once processing where occasional reprocessing of messages on failure is acceptable.
    • Manual: Offsets are committed only when the automation explicitly calls the Commit offsets action. Use this option for exactly-once processing semantics where you must confirm successful processing before advancing the consumer position.
  • (Optional) CSV file path: Enter an absolute file path to save the consumed messages as a CSV file. If left empty, messages are returned only in the table output variable and no file is written.
  • Assign output to a variable: Specify the variable where the Consume message action stores the result. The Consume message action returns a table variable containing up to the specified maximum number of messages from the Kafka topic. Each row in the table represents one message. Use a Loop action to iterate over the rows and process each message in subsequent automation steps.