Apache Pulsar Fundamentals

Apache Pulsar is a distributed messaging and streaming platform. Producers publish messages to topics. Consumers read those messages through named subscriptions.

That description sounds like many messaging systems. Pulsar's defining design choice sits below it. Brokers serve traffic, while Apache BookKeeper stores persistent messages. This separation lets you scale serving and storage for different pressures.

Pulsar also makes multi-tenancy part of every topic address. A persistent topic uses this structure:

persistent://tenant/namespace/topic

The tenant is the administrative boundary. The namespace groups related topics and carries shared policies. The topic is the named stream where applications exchange messages.

Why Pulsar exists

Messaging disconnects producers from consumers. A producer does not need to know which consumer is online. A consumer can process work at its own pace.

Pulsar adds durable storage, independent subscriptions, partitioning, and multi-cluster replication to that basic pattern. One topic can support several consumer groups without copying application logic. Each subscription keeps its own position.

This model supports event distribution, work queues, data pipelines, and streaming applications. Pulsar IO moves data between Pulsar and external systems. Pulsar Functions runs small consume, process, and publish transformations.

The three-part mental model

Use three views when you reason about Pulsar: address, data path, and cursor.

1. Address

Every topic belongs to a tenant and namespace. This hierarchy is more than naming.

• A tenant allocates an administrative boundary and allowed clusters.
• A namespace groups topics under common policies.
• A topic receives messages and exposes them to subscriptions.

Policies can cover retention, expiry, quotas, replication, and authorization. Some policies can also be set at topic level.

2. Data path

A producer connects to the broker that owns a topic. The broker accepts a message and writes persistent data to BookKeeper. A consumer connects through a broker and receives messages from its subscription.

The broker handles network traffic, topic ownership, dispatch, and coordination. A bookie is a BookKeeper storage node. Bookies store append-only ledgers that back topic data.

The broker can serve recent messages from cache. It reads older backlog from BookKeeper when needed. The optional Pulsar proxy gives clients one gateway when direct broker access is unsuitable.

The metadata store tracks coordination data, topic ownership, schemas, and load information. Do not treat it as the message store. BookKeeper holds persistent message data and subscription cursors.

3. Cursor

A subscription is a named delivery rule with a durable position. That position is its cursor.

Two subscriptions on one topic move independently. A slow analytics consumer does not hold back a separate operational consumer. Each subscription acknowledges its own progress.

An acknowledgment tells Pulsar that processing succeeded for that subscription. Unacknowledged messages form its backlog. A negative acknowledgment or timeout can cause redelivery.

This means duplicate processing remains possible in normal at-least-once designs. Make side effects idempotent unless you use a stronger end-to-end mechanism.

Subscription types shape delivery

The subscription name identifies a consumer group. The subscription type decides how consumers inside that group share messages.

Type	Delivery behavior	Typical fit
Exclusive	One consumer attaches	Simple ordered processing
Failover	One active consumer, with standbys	Ordered processing with standby takeover
Shared	Multiple consumers split messages	Parallel work where per-key order is unnecessary
Key_Shared	Multiple consumers split work by key	Parallel work with same-key ordering

Different subscription names create independent views of the topic. The same subscription name creates one shared cursor.

Partitioning and subscriptions solve different problems. Partitions raise a topic's throughput by spreading internal topics across brokers. Subscription types control delivery among consumers.

Persistence, backlog, and retention

Pulsar stores unacknowledged persistent messages in a backlog. By default, acknowledged messages become eligible for deletion.

A retention policy keeps acknowledged messages for a configured time or size. Message expiry sets a time to live for unacknowledged messages. These are different controls.

Tiered storage can move sealed, older ledgers from BookKeeper to external storage. Consumers still read the backlog through Pulsar. Older reads may have different latency and cost characteristics.

Schemas and message contracts

Pulsar stores message payloads as bytes. A Pulsar schema describes how clients translate those bytes into typed values.

The schema registry stores schema metadata at topic level. Brokers check compatibility when producers or consumers connect. A rejected schema is a contract failure, not a transient delivery error.

Schema management reduces accidental producer and consumer drift. It does not replace ownership, review, or a deliberate compatibility policy.

Clusters and geo-replication

A Pulsar instance can contain multiple clusters. Geo-replication copies messages between configured clusters for a namespace or topic.

Applications usually connect to a local cluster. Replicators move messages to remote clusters. Ordering remains per producer across geo-replication.

Geo-replication supports availability and disaster recovery. It is not a backup. A bad replicated policy or deletion can affect peer clusters.

Security and operations

Pulsar does not enable encryption, authentication, or authorization by default. A production design must configure each control explicitly.

Authentication establishes a client role. Authorization decides what that role can produce, consume, or administer. Encryption protects traffic and, when configured, message content.

Operate brokers, bookies, metadata stores, proxies, and function workers as separate failure domains. Watch message rates, storage, backlog, publish latency, dispatch latency, replication backlog, and component health.

Pulsar exposes broker and component metrics for Prometheus. It also provides administrative statistics through pulsar-admin. Namespace-level aggregation helps control metric cardinality.

Where Pulsar fits

Pulsar is a strong candidate when you need several of these properties together:

• durable publish and subscribe messaging;
• many independent subscriptions;
• built-in multi-tenancy and policy boundaries;
• separate scaling of serving and storage;
• multi-cluster replication;
• connectors or lightweight message functions near the broker platform.

It is not automatically the best choice for every queue. Its architecture adds brokers, BookKeeper, metadata services, and operational policy. A smaller broker may fit a modest workload with simpler requirements.

Pulsar is also not a general stream-processing engine. Functions handle focused transformations. Stateful joins, complex event-time logic, and large processing graphs may need a dedicated engine.

A useful learning path

1. Run standalone Pulsar and exchange messages on one persistent topic.
2. Create two subscriptions and observe their independent cursors.
3. Compare Exclusive, Shared, Failover, and Key_Shared delivery.
4. Learn topic partitions, routing, acknowledgments, and redelivery.
5. Add schemas, retention, quotas, authentication, and authorization.
6. Study BookKeeper, broker ownership, monitoring, and failure recovery.
7. Evaluate geo-replication, tiered storage, transactions, Functions, and Pulsar IO.