eBPF Observability
eBPF observability runs small, verified programs at Linux kernel and application hook points to measure what a system is doing. It can expose network flows, system calls, latency, and stack samples without adding instrumentation to every application.
itObservability and performance | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic: eBPF Observability
eBPF observability puts a small program at a Linux event boundary, then lets a collector turn selected results into telemetry. It is close enough to see system activity where it happens, which is useful. It is not, despite its proximity to the machinery, a complete observability system in a trench coat.
The durable picture has five pieces: a hook chooses an event, a program performs bounded work, a verifier checks kernel safety, a map or buffer holds the result, and a collector enriches and exports it. Remember the first and last pieces. The hook decides what evidence exists; the collector decides whether that evidence becomes usable telemetry rather than an interesting kernel souvenir.
A tracepoint is often the sensible starting point because it is an intentional static event. Kprobes and uprobes can reveal more specialised details, but their functions, symbols, binaries, and libraries can change underneath them. The ambitious probe is not always wrong. It does, however, arrive with a maintenance bill wearing a false moustache.
Maps aggregate counts, histograms, configuration, and temporary correlation state. Buffers carry individual records to user space. Both have finite capacity. The application can continue serving while the observer loses events or the collector falls behind, so attachment, map pressure, loss, queues, symbol resolution, and export health belong beside the telemetry they produce.
There is another boundary: kernel context is not business context. A socket can reveal timing and bytes, while application instrumentation supplies customer, order, feature, and domain meaning. eBPF provides broad baseline coverage; explicit telemetry fills in the reasons. Neither needs to impersonate the other.
Read the Intro for the full path from hook to backend and its failure modes. Use Slides for the decision map, Cheatsheet for the comparison tables and triage signals, Practice for a bounded tracepoint experiment, and Field Notes for the operational consequences that do not fit neatly in a diagram. The rest is careful evidence selection, which is fortunate, because Linux already has enough exciting surprises.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://ebpf.io/what-is-ebpf/
Supports
- Event-driven eBPF programs and hook families
- Verification and JIT execution model
- Maps, helper functions, user-space loaders, and observability use cases
- BPF and eBPF naming convention
- https://docs.ebpf.io/
Supports
- Technical documentation index for concepts, program types, maps, helpers, system-call commands, and libraries
- Scope and progression of the reference path
- https://docs.ebpf.io/linux/concepts/
Supports
- Linux concepts including maps, verifier, concurrency, pinning, resource limits, and trampolines
- Compatibility dimensions beyond source code
- https://docs.ebpf.io/linux/program-type/
Supports
- Tracing program-type families and libbpf section conventions
- Program type and attachment as distinct parts of the execution model
- https://docs.ebpf.io/linux/program-type/BPF_PROG_TYPE_TRACEPOINT/
Supports
- Tracepoints as predefined kernel events
- Tracepoint context and attachment model
- https://docs.ebpf.io/linux/program-type/BPF_PROG_TYPE_TRACING/
Supports
- Fentry and fexit trampoline attachment
- Typed entry arguments and exit return values
- Relationship to kprobes and kretprobes
- https://docs.kernel.org/bpf/
Supports
- Primary Linux BPF subsystem documentation path
- Verifier, maps, libbpf, security, testing, and standardization references
- https://docs.kernel.org/bpf/verifier.html
Supports
- Control-flow and execution-path analysis
- Register, pointer, stack, and map-value state tracking
- Verifier acceptance as a kernel safety check
- https://docs.kernel.org/bpf/maps.html
Supports
- Maps as shared storage between BPF programs and user space
- Map creation, lookup, update, deletion, and lifetime
- Finite typed map families used by observability programs
- https://docs.kernel.org/bpf/ringbuf.html
Supports
- Shared cross-CPU event buffer and ordering motivation
- Reservation, commit, discard, and output APIs
- Non-blocking reservation failure and record-loss behavior
- Ring size as a fixed power-of-two capacity
- https://docs.kernel.org/6.10/bpf/libbpf/libbpf_overview.html
Supports
- BTF, compiler, and libbpf roles in BPF CO-RE
- Target-kernel relocation of types, fields, and offsets
- Kernel BTF exposure through the sysfs vmlinux object
- https://docs.ebpf.io/concepts/core/
Supports
- CO-RE definition and portability problem
- BTF and relocation responsibilities
- https://bpftrace.org/docs/release_026/language
Supports
- Tracepoint, kprobe, fentry, fexit, uprobe, USDT, and sampling provider roles
- Fentry and fexit BTF requirements and typed arguments
- Kprobe casting and function-coupling considerations
- https://bpftrace.org/docs/release_026/cli
Supports
- Probe discovery and verbose argument inspection
- Availability checks against the current host
- https://bpftrace.org/docs/release_025/language
Supports
- Scratch variables and BPF-backed maps
- Per-thread entry-to-return correlation pattern
- Bounded map declarations and LRU eviction behavior
- https://docs.kernel.org/bpf/bpf_design_QA.html
Supports
- Kernel internals not forming a stable ABI by default
- Tracing privilege and helper boundaries
- Portability limits for kernel functions and data structures
- https://kernelnewbies.org/Linux_5.8
Supports
- CAP_BPF and CAP_PERFMON capability introduction
- Capability combinations for tracing and networking programs
- BPF ring-buffer milestone in Linux 5.8
- https://opentelemetry.io/docs/zero-code/obi/
Supports
- eBPF application auto-instrumentation for HTTP and gRPC signals
- Zero-code baseline observability and compatibility limits
- Coexistence with application and language instrumentation
- https://grafana.com/docs/beyla/latest/
Supports
- Executable and network inspection for baseline RED metrics and traces
- OpenTelemetry and Prometheus export
- BTF, kernel, capability, language, and tracing limitations
- Application SDKs as the source of deeper spans and custom telemetry
- https://github.com/sindresorhus/awesome
Supports
- Discovery of the curated Awesome eBPF list
- https://github.com/qmonnet/awesome-ebpf
Supports
- Curated discovery of BCC, bpftrace, Inspektor Gadget, Pixie, Parca, Hubble, Caretta, DeepFlow, Coroot, and Kyanos
- Separation of reference documentation, workflow tools, and observability projects
- https://github.com/iovisor/bcc
Supports
- BCC framework, language front ends, examples, and ready-to-run tracing tools
- https://inspektor-gadget.io/docs/main/
Supports
- OCI-packaged eBPF gadgets for Linux and Kubernetes
- Container and Kubernetes enrichment of kernel data
- Collection, export, filtering, and extension roles
- https://docs.px.dev/about-pixie/what-is-pixie/
Supports
- Per-node eBPF collection and in-cluster data plane
- Protocol traces, resource metrics, network metrics, and application profiles
- Kubernetes-focused service and request visibility
- https://docs.px.dev/about-pixie/pixie-ebpf/
Supports
- Kernel probes for network system calls
- TLS-library uprobes for plaintext-side protocol visibility
- Sampling-based application CPU profiling
- https://docs.cilium.io/en/stable/observability/hubble/
Supports
- Node, cluster, and multi-cluster network observability
- Hubble command line, relay, user interface, metrics, and service map
- https://www.parca.dev/docs/overview/
Supports
- eBPF whole-system continuous profiling agent
- Profile storage, label-based queries, comparisons, and incident analysis
- https://github.com/groundcover-com/caretta
Supports
- eBPF-derived Kubernetes service dependency map
- Prometheus-compatible metrics and Grafana visualization
- https://deepflow.io/docs/
Supports
- eBPF service maps, distributed tracing, and continuous profiling
- Correlation of network, request, and stack evidence
- https://coroot.com/docs
Supports
- eBPF application and infrastructure observability scope
- Service maps, request signals, profiling, and operational workflows
- https://coroot.com/pricing
Supports
- Community and commercial editions
- Free community use and paid product tiers
- https://kyanos.io/
Supports
- eBPF network-request capture and troubleshooting
- HTTP, Redis, MySQL, Kafka, MongoDB, and DNS protocol analysis scope
- https://ebpf.io/applications/
Supports
- Community landscape of actively maintained open-source eBPF applications
- Cilium, Pixie, Pyroscope, and Coroot observability roles
- https://cilium.io/use-cases/network-observability/
Supports
- Hubble network-flow, service-dependency, and policy-verdict evidence
- Cilium-managed Kubernetes network scope
- https://docs.groundcover.com/
Supports
- Node-level eBPF sensor and automatic collection
- Customer-environment data plane and unified telemetry path
- https://www.groundcover.com/pricing
Supports
- Free entry option and paid per-node product tiers
- eBPF sensor as the monitored-node unit
- https://docs.datadoghq.com/universal_service_monitoring/
Supports
- eBPF-based service discovery, request metrics, and dependency visibility
- Baseline service monitoring without application code changes
- https://www.datadoghq.com/pricing/
Supports
- Universal Service Monitoring as a commercial Datadog product
- Paid platform packaging and trial path
- https://www.elastic.co/observability/universal-profiling
Supports
- eBPF whole-system continuous profiling
- Application, library, and kernel stack visibility
- Elastic Cloud trial and commercial backend relationship
- https://www.polarsignals.com/
Supports
- Cross-language eBPF continuous profiling
- Flame-graph queries, comparisons, and production profile history
- Commercial cloud trial and product path
- https://www.usenix.org/conference/usenix-winter-1993-conference/presentation/bsd-packet-filter-new-architecture-user-level-
Supports
- January 1993 publication of the BSD Packet Filter architecture
- Register-based filtering and reduced kernel-to-user copying
- https://www.kernel.org/doc/html/v6.0/bpf/bpf_licensing.html
Supports
- Significant extension of the classic BPF instruction set in 2014
- Instruction-set continuity and licensing context
- https://kernelnewbies.org/Linux_3.18
Supports
- December 2014 Linux 3.18 release date
- bpf system call, program loading, maps, helper calls, and JIT milestone
- https://lwn.net/Articles/742082/
Supports
- April 2015 BCC release
- Higher-level front ends and collection of tracing tools
- https://kernelnewbies.org/Linux_4.8
Supports
- October 2016 Linux 4.8 release date
- XDP as a programmable high-performance network data path
- https://kernelnewbies.org/Linux_4.18
Supports
- August 2018 Linux 4.18 release date
- Introduction and initial purpose of BPF Type Format
- https://facebookmicrosites.github.io/bpf/blog/
Supports
- February 2020 public BPF CO-RE portability documentation
- BTF, libbpf, and target-side compilation problem
- https://bpftrace.org/one-liners
Supports
- Tracepoint discovery and read-only syscall-count examples for the practice reference
- Map-based count aggregation and interval reporting examples