Kafka Consumer Connector

Read Apache Kafka topics into FrameworX tags.

Connectors Library → Browse By Connector Group → IT-Cloud and AI Connectors → Kafka Consumer Connector

Overview

The Kafka Consumer connector reads messages from an Apache Kafka cluster into FrameworX tags. Each point subscribes to a topic and takes the latest matching message: the whole message, one field of a JSON message, or the value, quality and timestamp of a message from the Kafka Producer Connector. Each tag holds the latest value of its topic, not a log of every message. The connector does not create tags from topics.

This page covers the channel, node and point settings, the message mapping and delivery behavior. To publish tag values to Kafka, see Kafka Producer Connector. For MQTT brokers, see MQTT Client Connector. For general channel, node and point concepts, see Devices Module.

Prerequisites

• FrameworX 10.1.6 or later.
• A Windows x64 computer running the FrameworX runtime. See Platform Support below for Windows ARM64.
• The Microsoft Visual C++ 2015-2022 x64 runtime. The FrameworX installer installs it by default. If you clear this component in a Custom install and the computer does not have the runtime, the connector reads no messages.
• An Apache Kafka broker reachable from the runtime computer on its listener port, for example 9092.
• The topics to read exist on the broker.
• For SASL authentication or a cluster with access control: a Kafka user with read permission on the topics and on the consumer group of the node. See Consumer Group below for the group name.

Procedure

1. In Devices → Channels, create a channel and select the protocol Kafka Consumer Connector.
2. Open the channel Protocol Options and set the consumer group, start position, payload format and timestamp source. See Channel Configuration.
3. In Devices → Nodes, create a node on the channel and set the Station fields: broker list, security protocol and credentials. See Node Configuration.
4. In Devices → Points, add one point per tag to update. Set Address to Topic;Key;ValuePath and AccessType to a read type with polling, for example Read. See Points Configuration.
5. Start the runtime. On each polling cycle, each tag receives the latest matching message.

Channel Configuration

Protocol Options

Field order in the options string: GroupId;AutoOffsetReset;PayloadFormat;TimestampSource

The runtime reads the Protocol Options at start. After a change, restart the runtime.

Options string examples:

Default values:                                 ;Latest;Auto;Payload
Restore the last value of each key at start:    ;Earliest;Auto;Payload
Named group, text payload, arrival time:        PLANT1.SCADA;Latest;Text;Arrival

Consumer Group

Each node consumes as its own Kafka consumer group. With Latest, the group keeps the read position of the node, so a restart resumes where the node stopped.

• GroupId blank: the group name is FrameworX.<ComputerName>.<SolutionName>.<CHANNEL>.<NODE>, with the channel and node names in upper case. Two solutions, or two computers, reading the same topic therefore use different groups, and each receives every message.
• GroupId set: the group name is <GROUPID>.<NODE>. The runtime converts the GroupId to upper case, so plant1.scada becomes PLANT1.SCADA. Do not use = or ; in the GroupId. Grant the Kafka read permission to the upper-case group name.
• Two runtimes of the same solution on the same computer, for example a Development and a Production profile, share the default group and split the topic partitions between them. Set a different GroupId on one of them.

Node Configuration

Station Configuration

Station syntax: <BootstrapServers>;<SecurityProtocol>;<SaslMechanism>;[SaslUsername];[SaslPassword];[SslCaLocation]

The Station fields are the same as in the Kafka Producer Connector, so a producer node and a consumer node of the same cluster use the same Station string. The node uses its primary Station only. The connector does not switch to a backup Station.

Points Configuration

Address

Address syntax: <Topic>;[Key];[ValuePath]

Address examples:

Messages with key Line1Temp on topic plant.line1:   plant.line1;Line1Temp;
Every message on topic sensors.raw:                  sensors.raw;;
Field temp of the JSON messages on sensors.json:     sensors.json;;$.temp

AccessType

Assign a read AccessType with polling, for example the predefined Read type. The polling period is the tag update period: on each polling cycle the tag receives the latest matching message received since start. The connector ignores writes to its points. See Devices AccessTypes Reference.

Message Mapping

For each point, the connector maps a matching message to a value, a quality and a timestamp, in this order:

1. PayloadFormat Text: the whole message as a string, quality Good (192). The ValuePath is not used.
2. ValuePath set: the JSON value at the ValuePath, quality Good. A message without a value at this path, or not in JSON format, leaves the tag unchanged.
3. Message from the Kafka Producer Connector: a JSON object with tag and value fields. The tag receives value, the quality field when it is an integer, and the timestamp field with TimestampSource Payload.
4. Any other message: the whole message as a string, quality Good.

Producer message example:

{"tag":"K_PlainCounter","value":265,"quality":192,"timestamp":"2026-10-08T10:11:00.6674091Z"}

• Types: JSON numbers become integer or floating-point values, true and false become Boolean values, strings stay strings, and JSON objects and arrays reach the tag as JSON text. The runtime converts the value to the tag type.
• Not mapped: a JSON null value, and a Kafka record without a payload, for example a delete marker in a compacted topic, leave the tag unchanged.
• Dates in JSON strings stay strings.
• Quality: the connector uses the quality field of Kafka Producer Connector messages only. For other messages it sets Good.

Delivery Behavior

• Latest value per tag. Between two polling cycles, the last matching message wins. Intermediate values between two cycles do not reach the tag or the historian. A shorter polling period skips fewer values. To record every message, read the topic with another Kafka client.
• Message order. Kafka orders messages within a partition. When the messages of one point come from several partitions, for example a blank Key on a multi-partition topic, the tag keeps the message with the newest Kafka record time.
• Catch-up at start. When the node starts, or Kafka reassigns its partitions, the connector reads up to the end of each partition before it updates the tags. A replay with Earliest therefore writes only the last value of each point, not the intermediate values. A partition without data or end-of-partition for 10 seconds counts as read.
• Startup. Until Kafka assigns the partitions to the node, the tags keep their startup value and quality. When no assignment arrives within 30 seconds, the tags go Bad after the channel retries.
• Connection loss. When the client loses all brokers, or the broker rejects the credentials, the tags go Bad after the channel retries. While the brokers are unreachable, the connector checks the cluster every 2 seconds. When the connection returns, the tags go Good again with the recovery time as timestamp, so the historian keeps the Bad sample before the Good one. A point with no message since start stays Bad until its first message.
• Read position. The consumer group commits the read position on its own. A message read but not yet applied to a tag when the runtime stops is not read again with Latest.
• Timestamps of replayed values. A value replayed at start, or received after a connection loss, carries the start or recovery time when its own timestamp is older. This applies to every TimestampSource.
• Configuration changes. When the node Station changes, the connector rebuilds the consumer with the new settings. A change of BootstrapServers also discards the values received from the previous cluster. Each tag keeps its current value until a message from the new cluster arrives.
• Loops. Do not read and publish the same tag on the same topic.

Station Examples

1. Local Apache Kafka, no authentication, no TLS

Station string:

localhost:9092;Plaintext;Plain;;;

Channel Protocol Options: the default values. Point: Address sensors.raw;;, AccessType Read.

2. Apache Kafka with SASL_SSL, SCRAM-SHA-256 and a private CA file

The broker listener on port 9093 uses SASL_SSL with SCRAM-SHA-256, and ca.pem holds the CA certificate in PEM format. The broker certificate lists kafka01.plant.local as a host name. Create the SCRAM user on the broker with kafka-configs, and grant it read permission on the topics and on the consumer group, before you start the runtime.

Platform Support

The connector depends on the native Kafka client library for Windows x64. On a host where this library does not load, the node stays Bad.

Troubleshooting

The connector writes these messages to the Trace Window (see Runtime Diagnostics Reference). Check Devices in the Trace Window settings. Messages marked (Debug) appear only when you also check Debug. For rows with a quoted message, the first column shows the start of the message.

Third-Party Components

The connector uses the same components as the Kafka Producer Connector, installed in the Protocols folder of the FrameworX installation:

License notices for these components ship in <FrameworX installation>\Protocols\Kafka-ThirdPartyNotices.txt.

Driver Revision History

Related Pages

• Kafka Producer Connector
• Devices Module
• Devices Module Reference
• Devices AccessTypes Reference
• Runtime Diagnostics Reference
• MQTT Client Connector
• IT-Cloud and AI Connectors

In this section...