Login
- Table of Contents
-
- 13-Network Management and Monitoring Configuration Guide
- 00-Preface
- 01-System maintenance and debugging configuration
- 02-NQA configuration
- 03-iNQA configuration
- 04-NTP configuration
- 05-iFIT configuration
- 06-SNMP configuration
- 07-RMON configuration
- 08-Event MIB configuration
- 09-NETCONF configuration
- 10-CWMP configuration
- 11-EAA configuration
- 12-Process monitoring and maintenance configuration
- 13-Sampler configuration
- 14-Mirroring configuration
- 15-NetStream configuration
- 16-IPv6 NetStream configuration
- 17-sFlow configuration
- 18-Fast log output configuration
- 19-Performance management configuration
- 20-Flow log configuration
- 21-Information center configuration
- 22-GOLD configuration
- 23-eMDI configuration
- 24-Kafka configuration
- 25-SQA configuration
- Related Documents
-
| Title | Size | Download |
|---|---|---|
| 24-Kafka configuration | 93.63 KB |
Configuring Kafka
About Kafka
Kafka is a distributed message publish-subscribe system based on messages. Kafka uses a client/server communication model. In the Kafka system, the devices, acting as clients, encapsulate locally generated logs and alarms into Kafka messages and then publish them to the Kafka servers (brokers). The integration of the devices into the Kafka system allows you to manage the devices through the Kafka system.
Benefits
The Kafka system has the following features:
· High throughput—Supports producing and consuming millions of messages per second.
· Persistence—Kafka provides a complete set of message storage mechanism, supporting efficient, secure, and stable persistent storage of terabyte-scale messages.
Kafka has been widely used in big data analysis and integration services.
Concepts
· Broker—Server in the Kafka communication model that receives and stores messages. Multiple brokers can form a Kafka cluster.
· Producer—Message producer or publisher, a client in the Kafka communication model that produces and sends messages to the Kafka brokers.
· Consumer—Message consumer or subscriber, a client in the Kafka communication model that reads messages from Kafka brokers. Multiple consumers can form a consumer group.
· Topic—Message topic, used for classification and management of messages. The device supports configuring topics according to software modules (such as the alarm module). When logs and alarms generated by the same software module are encapsulated into Kafka messages, the same topic is added.
· Kafka service instance—Used for managing and configuring Kafka services. In a Kafka service instance view, you can bind Kafka brokers and configure the topic and format for messages.
Operating mechanism
As shown in Figure 1, Kafka works as follows:
1. The devices, acting as Kafka message producers, publish produced messages with specific topics to Kafka brokers in the specified format.
The device supports sending Kafka messages such as:
¡ Alarm and alarm clear messages for CPU usage threshold exceeding.
¡ Alarm and alarm clear messages for free memory dropping below threshold.
¡ Alarm and alarm clear messages for interface status changing from up to down.
¡ Alarm and alarm clear messages for expiration of basic licenses.
¡ Alarm and alarm clear messages for routing table entry threshold exceeding.
¡ Alarm and alarm clear messages for ARP table entry threshold exceeding.
¡ Alarm and alarm clear messages for BGP neighbor disconnections.
2. The consumers subscribe to messages based on the topics and obtain messages from Kafka brokers.
Figure 1 Operating mechanism of Kafka
Configuring a Kafka message producer
system-view
2. Create a Kafka service instance, or enter the view of an existing Kafka service instance.
kafka-service service-name
By default, no Kafka service instance exists.
3. Configure a Kafka topic.
topic module topic-name
By default, no Kafka topic is configured.
4. Specify the format for messages sent to Kafka brokers.
format string
By default, no format is specified for messages sent to Kafka brokers.
5. Bind a Kafka broker to the Kafka service instance.
broker [ vpn-instance vpn-instance-name ] { host-name | ipv4-address | ipv6 ipv6-address } port port-number
By default, no Kafka broker is bound to the Kafka service instance.
Kafka configuration examples
Example: Configuring a basic Kafka network
Network configuration
As shown in Figure 2, integrate the device into the Kafka system. Configure the system to detect important events that you are concerned about and configure the device to inform the consumer of Kafka messages for management of the device through the system.
Procedure
1. Assign IP addresses for devices. (Details not shown.)
2. Configure static routing or dynamic routing protocols. Make sure the devices can reach one another. (Details not shown.)
3. Configure the producer:
# Create Kafka service instance suyan and enter its view.
<Producer> system-view
[Producer] kafka-service suyan
# Configure the Kafka topic as suyan-alarm.
[Producer-kafka-service-suyan] topic alarm suyan-alarm
# Specify the format for messages sent to the Kafka broker as suyan-cloud.
[Producer-kafka-service-suyan] format suyan-cloud
# Bind a Kafka broker with IP address 1.1.10.1 and port number 5145 to the Kafka service instance.
[Producer-kafka-service-suyan] broker 1.1.10.1 port 5145
[Producer-kafka-service-suyan] quit
Verifying the configuration
Execute first the shutdown command and then the undo shutdown command on an idle interface of the producer. Verify that the consumer can receive an interface status change message with topic suyan-alarm.
