Apache Kafka Operations
Apache Kafka operations covers the day-to-day management of Kafka clusters: broker configuration, partition rebalancing, replication tuning, monitoring, upgrades, and capacity planning to keep a distributed event-streaming platform running reliably under production workloads.
itDistributed systems, messaging, and integration | OpenSkills.info
Intro
Apache Kafka Operations
Operating Kafka means protecting three things at once: metadata, replicated logs, and client progress. A healthy process list is not enough. You need enough controllers for metadata decisions, enough in-sync replicas for safe writes, and enough consumer capacity to keep lag within the workload's objective.
Use this operational path as your mental model:
controllers decide → brokers store and serve → replicas copy → clients produce and consume → metrics expose pressure
Every routine change touches part of that path. Topic creation adds partition logs. A broker restart moves leadership. A reassignment copies data. An offset reset changes what a consumer group processes next.
The two planes
Modern Kafka uses KRaft for cluster metadata. Controllers form a metadata quorum. One controller is active, while the others remain ready to take over.
Brokers form the data plane. They store partition replicas and handle client requests. A Kafka process can have the broker role, controller role, or both roles. Combined roles suit small environments, but the official guidance does not recommend them for critical deployments.
The controller quorum needs a majority to stay available. A three-controller quorum tolerates one controller failure. A five-controller quorum tolerates two. Losing the quorum stops metadata decisions even if broker disks still hold partition data.
Kafka 4.3 supports static and dynamic controller quorums. Dynamic quorums use controller.quorum.bootstrap.servers. Static quorums use controller.quorum.voters. Treat quorum membership as its own maintenance procedure, not as a broker scaling side effect.
Replication defines the failure budget
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://kafka.apache.org/43/operations/kraft/
Supports
- Broker, controller, and combined process roles
- Controller majority and three-controller or five-controller failure tolerance
- Static and dynamic quorum configuration and membership operations
- Metadata quorum inspection and debugging tools
- https://kafka.apache.org/43/operations/basic-kafka-operations/
Supports
- Topic creation, modification, description, and deletion
- One-way partition expansion and its key-placement effects
- Graceful shutdown and preferred leader election
- Consumer-group inspection and offset-reset preconditions
- Partition reassignment generate, execute, verify, rollback, and throttle workflow
- Rack-aware placement and broker expansion behavior
- https://kafka.apache.org/43/operations/monitoring/
Supports
- Offline log directory, offline partition, under-replicated, and under-minimum-ISR metrics
- Controller, metadata error, queue, throughput, and network metrics
- Consumer records-lag-max semantics
- Preferred replica imbalance and election metrics
- https://kafka.apache.org/43/configuration/broker-configs/
Supports
- Essential KRaft broker settings and update modes
- Data log directories, listeners, and advertised listener behavior
- Unclean leader election data-loss risk
- Broker-wide replication, storage, quota, and shutdown settings
- https://kafka.apache.org/43/configuration/topic-configs/
Supports
- Minimum in-sync replicas with all acknowledgments
- Retention, cleanup, compaction, and record-size settings
- Per-topic configuration as an override of server defaults
- https://kafka.apache.org/43/operations/eligible-leader-replicas/
Supports
- Eligible Leader Replica behavior and election order
- Default enablement on new clusters from Kafka 4.1
- Interaction with minimum in-sync replica configuration
- https://kafka.apache.org/43/security/encryption-and-authentication-using-ssl/
Supports
- TLS transport encryption and certificate-based authentication
- Client and broker trust and key material
- Client authentication settings and encrypted listener configuration
- https://kafka.apache.org/43/security/authentication-using-sasl/
Supports
- Supported SASL mechanisms
- SASL over plaintext or encrypted transport
- Broker, inter-broker, and client authentication configuration
- https://kafka.apache.org/43/security/authorization-and-acls/
Supports
- KRaft StandardAuthorizer configuration
- ACL principals, hosts, operations, resources, and patterns
- ACL command-line listing, addition, and removal
- Default behavior for resources without matching ACLs
- https://kafka.apache.org/43/operations/hardware-and-os/
Supports
- Memory, disk-throughput, filesystem, and page-cache considerations
- Multiple data directories and disk-balancing tradeoffs
- Recovery and flush behavior
- https://kafka.apache.org/43/getting-started/upgrade/
Supports
- Kafka 4.3 rolling broker upgrade sequence
- Behavior and performance verification before feature finalization
- Release-version upgrade command and downgrade constraints
- https://github.com/apache/kafka/blob/trunk/docs/operations/basic-kafka-operations.md
Supports
- Primary source Markdown for operational commands and procedures
- https://github.com/apache/kafka/blob/trunk/docs/operations/monitoring.md
Supports
- Primary source Markdown for JMX metric names and meanings