Apache Pulsar Fundamentals
Apache Pulsar is a distributed messaging and event-streaming platform that separates serving from storage. It supports multi-tenancy, geo-replication, and both queuing and streaming semantics in a single system, with tiered storage for cost-effective retention.
itDistributed systems, messaging, and integration | OpenSkills.info
Intro
Apache Pulsar Fundamentals
Apache Pulsar is a distributed messaging and streaming platform. Producers publish messages to topics. Consumers read those messages through named subscriptions.
That description sounds like many messaging systems. Pulsar's defining design choice sits below it. Brokers serve traffic, while Apache BookKeeper stores persistent messages. This separation lets you scale serving and storage for different pressures.
Pulsar also makes multi-tenancy part of every topic address. A persistent topic uses this structure:
persistent://tenant/namespace/topic
The tenant is the administrative boundary. The namespace groups related topics and carries shared policies. The topic is the named stream where applications exchange messages.
Why Pulsar exists
Messaging disconnects producers from consumers. A producer does not need to know which consumer is online. A consumer can process work at its own pace.
Pulsar adds durable storage, independent subscriptions, partitioning, and multi-cluster replication to that basic pattern. One topic can support several consumer groups without copying application logic. Each subscription keeps its own position.
This model supports event distribution, work queues, data pipelines, and streaming applications. Pulsar IO moves data between Pulsar and external systems. Pulsar Functions runs small consume, process, and publish transformations.
The three-part mental model
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://pulsar.apache.org/
Supports
- Apache Pulsar project identity and official documentation entry point
- https://pulsar.apache.org/download/
Supports
- Apache Pulsar 4.2.3 as the current stable release on the verification date
- Official binary and source distribution entry points
- https://pulsar.apache.org/docs/4.2.x/concepts-overview/
Supports
- Pulsar as a multi-tenant server-to-server messaging platform
- BookKeeper-backed persistent delivery and four subscription types
- Geo-replication, Pulsar Functions, Pulsar IO, tiered storage, transactions, and security capabilities
- https://pulsar.apache.org/docs/4.2.x/getting-started-standalone/
Supports
- Standalone mode for local development and testing
- Creating a persistent topic and exchanging messages with official command-line tools
- The public/default development namespace
- https://pulsar.apache.org/docs/4.2.x/concepts-messaging/
Supports
- Producer, topic, consumer, subscription, and acknowledgment relationships
- Message metadata, payload bytes, keys, schema versions, and message identifiers
- Individual, cumulative, and negative acknowledgments plus redelivery behavior
- Exclusive, Failover, Shared, and Key_Shared subscription behavior
- Independent subscriptions and shared consumer-group delivery
- Partitioned topics, routing, throughput, and ordering boundaries
- Backlog, retention, expiry, deduplication, compaction, and delayed delivery
- https://pulsar.apache.org/docs/4.2.x/concepts-architecture-overview/
Supports
- Cluster composition with brokers, BookKeeper bookies, and a metadata store
- Broker responsibilities for lookup, message traffic, dispatch, cache, and geo-replication
- BookKeeper ledgers, managed ledgers, durable message data, and persistent cursors
- Separation of broker ownership from durable storage
- Optional proxy and service discovery behavior
- https://pulsar.apache.org/docs/4.2.x/concepts-multi-tenancy/
Supports
- Tenant as an administrative and capacity boundary
- Namespace as a logical topic group and policy boundary
- Persistent topic name structure with tenant, namespace, and topic
- https://pulsar.apache.org/docs/4.2.x/schema-overview/
Supports
- Message payloads stored as bytes and translated through schemas
- Topic-level schema registry, versions, and compatibility checks
- Producer and consumer rejection when schemas are incompatible
- https://pulsar.apache.org/docs/4.2.x/security-overview/
Supports
- No encryption, authentication, or authorization enabled by default
- Authentication providers assigning role tokens
- Authorization controlling production, consumption, and administration
- Transport and end-to-end encryption as separate controls
- https://pulsar.apache.org/docs/4.2.x/deploy-monitoring/
Supports
- Broker statistics and namespace-aggregated monitoring metrics
- Prometheus endpoints for brokers and BookKeeper
- Function and connector worker metrics
- Metric cardinality guidance and OpenTelemetry status
- https://pulsar.apache.org/docs/4.2.x/administration-geo/
Supports
- Geo-replication configuration at namespace and topic level
- Local application access and remote message replication
- Per-producer ordering across replicated clusters
- Replication rates, backlogs, policy propagation, and deletion behavior
- Geo-replication for availability and disaster recovery, not backup
- https://pulsar.apache.org/docs/4.2.x/concepts-tiered-storage/
Supports
- Offloading sealed BookKeeper ledgers to external storage
- Transparent consumer access to offloaded backlog
- Storage-cost and older-read latency tradeoffs
- https://pulsar.apache.org/docs/4.2.x/txn-what/
Supports
- Atomic writes and acknowledgments across topics and partitions
- Commit and abort semantics for transactional messages
- Exactly-once consume, process, and produce operations under transaction semantics
- https://pulsar.apache.org/docs/4.2.x/functions-overview/
Supports
- Pulsar Functions as consume, process, and publish computations
- Functions for focused message transformations
- Java, Python, and Go function support
- https://pulsar.apache.org/docs/4.2.x/io-overview/
Supports
- Source connectors moving external data into Pulsar
- Sink connectors moving Pulsar data into external systems
- Connector processing guarantees depending on Pulsar and connector behavior
- https://pulsar.apache.org/docs/4.2.x/reference-landing/
Supports
- Official reference path for application programming interfaces, command-line tools, configuration, metrics, and terminology