Apache Kafka Operations

Operating Kafka means protecting three things at once: metadata, replicated logs, and client progress. A healthy process list is not enough. You need enough controllers for metadata decisions, enough in-sync replicas for safe writes, and enough consumer capacity to keep lag within the workload's objective.

Use this operational path as your mental model:

controllers decide → brokers store and serve → replicas copy → clients produce and consume → metrics expose pressure

Every routine change touches part of that path. Topic creation adds partition logs. A broker restart moves leadership. A reassignment copies data. An offset reset changes what a consumer group processes next.

The two planes

Modern Kafka uses KRaft for cluster metadata. Controllers form a metadata quorum. One controller is active, while the others remain ready to take over.

Brokers form the data plane. They store partition replicas and handle client requests. A Kafka process can have the broker role, controller role, or both roles. Combined roles suit small environments, but the official guidance does not recommend them for critical deployments.

The controller quorum needs a majority to stay available. A three-controller quorum tolerates one controller failure. A five-controller quorum tolerates two. Losing the quorum stops metadata decisions even if broker disks still hold partition data.

Kafka 4.3 supports static and dynamic controller quorums. Dynamic quorums use controller.quorum.bootstrap.servers. Static quorums use controller.quorum.voters. Treat quorum membership as its own maintenance procedure, not as a broker scaling side effect.

Replication defines the failure budget

Each partition has a leader and replica set. Clients use the leader. Followers copy the leader's log. Kafka tracks replicas that meet its current synchronization criteria in the in-sync replica set, or ISR.

Three settings shape write behavior:

• The replication factor sets the replica count.
• Producer acks=all waits for every current ISR member to acknowledge.
• min.insync.replicas rejects an all-acknowledgment write when the ISR is too small.

A common durability policy uses replication factor three, minimum in-sync replicas two, and all acknowledgments. It allows one replica to be unavailable while preserving the minimum for writes. A second replica loss makes the partition unavailable for those writes instead of silently weakening the policy.

Kafka 4.3 also includes Eligible Leader Replicas. This feature can preserve safe leadership choices beyond the current ISR under its defined rules. It is enabled by default on new clusters from Kafka 4.1. Do not treat it as permission to enable unclean leader election. Unclean election can select a replica outside the safe set and may lose data.

Topics are operational contracts

A topic combines partition count, replication, retention, cleanup, and per-topic limits. These settings encode workload behavior.

Create topics deliberately. Automatic creation can inherit defaults that do not match the data's durability or retention needs. Describe the topic after creation and record the intended values.

You can increase a topic's partition count, but Kafka cannot reduce it. Adding partitions can change key-to-partition mapping. Existing data also stays where it is; Kafka does not redistribute old records because you added partitions.

Topic deletion removes the topic and its data. Treat it as a destructive change with an owner, retention check, and recovery plan.

Movement is a data-copy operation

Adding a broker does not automatically move existing replicas onto it. Use kafka-reassign-partitions.sh to generate, review, execute, and verify a reassignment.

Save the current assignment before execution. It is your rollback input. Throttle replication when an unrestricted copy would compete with production traffic. After completion, run verification so the tool can clear the throttles it created.

Leadership and replica placement are different. A broker that returns after an outage may host healthy followers while other brokers still lead its partitions. Kafka can restore preferred leadership automatically, or you can run a preferred leader election.

Rack awareness spreads a partition's replicas across configured failure domains. It cannot create more distinct failure domains than either the replication factor or the number of available racks.

Consumer lag is an application signal

Kafka stores records and consumer-group offsets separately. The consumer-group tool shows current offsets, log-end offsets, lag, members, assignments, and group state.

Lag is not a complete service-level objective. A count of ten thousand records may mean seconds or hours, depending on production rate and processing cost. Monitor lag with throughput, processing latency, rebalance activity, and the age of the oldest required work.

Offset resets are controlled replay operations. Preview the proposed reset first. Stop active consumers before execution. Confirm that retained data still covers the target range, and confirm that repeated side effects are safe.

Monitor invariants before averages

Start with conditions that should normally be zero:

• Offline partition count
• Offline log directory count
• Under-minimum-ISR partition count
• Unclean leader election rate
• Metadata error count

Then watch pressure and balance:

• Under-replicated partitions
• Request latency and queue size
• Network processor idle percentage
• Bytes in and out
• Consumer lag
• Preferred replica imbalance
• Controller event queue time

An under-replicated partition has fewer in-sync replicas than its configured replica set. An under-minimum-ISR partition has crossed the write-availability threshold. The second condition is more urgent, but the first often gives you earlier warning.

Security has three separate jobs

Transport encryption protects traffic. Authentication establishes a principal. Authorization decides what that principal may do.

Kafka supports TLS and SASL mechanisms including Kerberos, PLAIN, SCRAM, and OAuth bearer tokens. Use encrypted transport for credentials and data. For KRaft authorization, Kafka provides StandardAuthorizer, which stores ACLs in the metadata log.

Grant the smallest resource and operation set a client needs. A consumer normally needs topic read access and group access. An operator needs administrative permissions that application identities should not have.

Test the denied path as well as the allowed path. A successful producer test proves little about an overbroad wildcard ACL.

Maintenance is a measured sequence

For a planned broker restart, use graceful shutdown. Kafka can flush logs and move leadership before the process exits when replicas are available.

Before each restart, check quorum health, offline partitions, ISR state, current reassignments, and available capacity. Restart one broker. Wait for recovery. Confirm the invariants again before moving to the next broker.

Kafka upgrades separate software rollout from feature finalization. The Kafka 4.3 guide directs you to upgrade brokers one at a time, verify behavior and performance, then finalize the release version with kafka-features.sh. Read the exact source-to-target upgrade notes before starting. Downgrade support depends on metadata changes.

Capacity and recovery

Kafka uses the filesystem and Linux page cache heavily. Size storage from retained bytes, replication, growth, and reassignment headroom. Keep Kafka data away from competing filesystem activity when predictable latency matters.

Recovery consumes the same disk and network paths as normal traffic. A cluster at steady-state saturation has no useful failure budget. Test a broker loss, replica rebuild, controller loss, and consumer replay under representative load.

Kafka replication is not a complete disaster-recovery plan. Operator error, credential compromise, and site loss can affect every replica in one cluster. Define cross-cluster recovery and validate the recovery point and recovery time against the workload.

An operating loop

1. Define availability, durability, latency, and recovery objectives.
2. Encode them in replication, acknowledgment, retention, and security policy.
3. Establish zero-state invariants and workload baselines.
4. Make one bounded change.
5. Observe replication, leadership, latency, and lag until stable.
6. Record the result and update the runbook.
7. Rehearse failure and restoration before production supplies the test.