Software Profiling
Profiling measures where a running program spends its time and memory, attributing that cost to the specific functions and call paths responsible. It answers the question a log line or a request timing cannot: not that an operation was slow, but which code inside it was slow, and how much of the total that code accounts for.
itObservability and performance | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Intro
Software Profiling
Profiling measures how a running program consumes a resource — CPU time, wall-clock time, memory, lock waits — and attributes that consumption to the code responsible. The output is a profile: a set of call stacks, each carrying a measured cost. A profile does not say "this request took 800 ms". It says "43% of the CPU time in this process was spent under parseTimestamp, called from deserializeRow, called from the batch loader".
That attribution is the whole point, and it marks the boundary against the neighboring practices:
| Practice | Question it answers |
|---|---|
| Profiling | Where inside a process does the time or memory go? |
| Distributed tracing / APM | Where between services does a request spend its latency? |
| Load and performance testing | What does the system do under a defined applied workload? |
| Metrics and monitoring | Is the system currently within its targets? |
The four are complementary. A trace tells you the checkout service accounts for 700 ms of a 900 ms request. A profile of that service tells you which functions inside it burned the 700 ms. Neither substitutes for the other.
What a profiler actually collects
Every profiler, regardless of language or platform, runs the same four-stage path:
- Trigger. Something interrupts or notifies the profiler that a unit of cost occurred — a timer expiry, a hardware counter overflow, a memory allocation, a lock acquisition, a thread context switch.
- Stack capture. At that instant the profiler walks the call stack of the affected thread, producing an ordered list of return addresses.
- Symbolization. Raw addresses are resolved into function names, source files, and line numbers using debug information, symbol tables, or runtime-supplied maps.
- Aggregation. Identical stacks are merged and their costs summed, turning millions of individual observations into a few thousand distinct stacks with weights.
Stage 4 is why a profile is compact enough to read. It is also why a profile has no timeline: merging discards the order in which samples arrived. Tools that keep the ordering are tracers, not profilers, and they pay for it in data volume.
Sampling and instrumentation
There are two ways to reach stage 1, and the choice determines almost everything else about a profiler's behavior.
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://man7.org/linux/man-pages/man2/perf_event_open.2.html
Supports
- Counting events aggregate totals read with read(2); sampling events periodically write measurements to a buffer accessed via mmap(2)
- sample_period generates an overflow notification every N events; a sampling event has sample_period greater than zero
- sample_freq with the freq flag lets the kernel adjust the period to achieve a requested rate
- PERF_SAMPLE_CALLCHAIN records the callchain (stack backtrace) as a count followed by instruction pointers
- The kernel interface underlying Linux profilers
- https://man7.org/linux/man-pages/man1/perf-record.1.html
Supports
- -F sets the profiling frequency; max uses kernel.perf_event_max_sample_rate
- -c sets the event period to sample
- -g enables call-graph recording for kernel and user space
- --call-graph default is fp for user space
- fp unwinding produces bogus call graphs on binaries built with gcc --fomit-frame-pointer
- dwarf unwinding should be used instead when frame pointers are unavailable, with a default stack dump size of 8192 bytes customizable as dwarf,4096
- lbr requires no compiler options, is only available on newer Intel platforms such as Haswell, and can only get the user call chain
- -p records events on an existing process ID
- -a performs system-wide collection from all CPUs
- -e selects the PMU event
- https://man7.org/linux/man-pages/man1/perf-report.1.html
Supports
- --no-children disables accumulation of child function overhead into parent entries
- --sort orders histogram entries by keys including sym
- --stdio uses the stdio interface
- https://perf.wiki.kernel.org/
Supports
- Linux perf project home and documentation hub
- https://www.brendangregg.com/flamegraphs.html
Supports
- Flame graphs visualize hierarchical data to identify the most frequent code paths
- The y-axis shows stack depth counting from zero at the bottom
- The x-axis shows the stack profile population sorted alphabetically and is not the passage of time
- The wider a frame is, the more often it was present in the stacks
- Alphabetical ordering maximizes frame merging for a clearer overview than time-ordered layouts
- Variants include CPU, memory, off-CPU, hot/cold and differential flame graphs, with inverted versions called icicle charts
- Flame graphs were released in December 2011
- Origin in troubleshooting a MySQL performance issue
- https://www.brendangregg.com/perf.html
Supports
- Choosing 99 Hertz instead of 100 Hertz avoids accidentally sampling in lockstep with periodic activity, which would produce skewed results
- Counting increments a counter on events with minimal overhead reported by perf stat; sampling collects instruction pointer or stack details via perf record
- Frame pointers should be preserved by compiling with -fno-omit-frame-pointer
- perf_events JIT support requires the VM to maintain a /tmp/perf-PID.map file for symbol translation
- https://github.com/brendangregg/FlameGraph
Supports
- The perf record, perf script, stackcollapse-perf.pl, flamegraph.pl pipeline
- perf record -F 99 -a -g -- sleep 60 followed by perf script > out.perf
- stackcollapse-perf.pl folds stacks and flamegraph.pl renders an SVG
- --color=java and --hash options for flamegraph.pl
- Stack samples can be captured using Linux perf_events
- https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer
Supports
- Fedora adds -fno-omit-frame-pointer and -mno-omit-leaf-frame-pointer to default C/C++ compilation flags
- Targeted release Fedora Linux 38
- Traversing a stack using frame pointers is cheap
- DWARF unwinding requires copying the full stack from kernelspace to userspace and is relatively slow
- Frame pointers let BPF tools reliably access userspace stack traces without source modification
- Compiling the kernel with GCC is 2.4% slower with frame pointers; Blender rendering a frame is 2% slower
- Python benchmarks 1-10%; openssl, botan, zstd and Redis not significantly affected
- https://go.dev/blog/pprof
Supports
- When CPU profiling is enabled the Go program stops about 100 times per second and records a sample of the program counters on the currently executing goroutine's stack
- The first two columns show samples in which the function was running as opposed to waiting for a called function to return
- The -cum flag sorts by cumulative rather than flat samples
- The memory profiler only records information for approximately one block per half megabyte allocated
- Publication date 24 June 2011
- A worked iterative optimization session using go tool pprof
- https://pkg.go.dev/net/http/pprof
Supports
- Endpoints registered under /debug/pprof/ including profile, heap, allocs, goroutine, block, mutex, threadcreate, trace, cmdline and symbol
- go tool pprof http://localhost:6060/debug/pprof/heap and profile?seconds=30 example invocations
- seconds=N returns a delta profile for allocs, block, goroutine, heap, mutex and threadcreate profiles
- seconds=N profiles for the given duration for CPU and trace profiles; CPU profiling defaults to 30 seconds
- https://pkg.go.dev/runtime/pprof
Supports
- heap is a sampling of memory allocations of live objects
- allocs is a sampling of all past memory allocations
- block holds stack traces that led to blocking on synchronization primitives
- mutex holds stack traces of holders of contended mutexes
- goroutine holds stack traces of all current goroutines
- threadcreate holds stack traces that led to the creation of new OS threads
- Block profiling uses time-based sampling specified by runtime.SetBlockProfileRate
- Mutex profiling uses event-based sampling specified by runtime.SetMutexProfileFraction
- https://github.com/google/pprof
Supports
- pprof is a tool for visualization and analysis of profiling data
- It reads profiling samples in profile.proto format and generates reports
- It can read perf.data files from the Linux perf tool through a conversion utility
- Text and graphical reports via -top, -web and -http
- cum is the value of the location plus all its descendants
- -flat is the default sort and -cum sorts by cumulative value
- list annotates source lines matching a regex with flat and cum values for each source line
- -base subtracts one cumulative profile from another collected from the same program later
- https://docs.python.org/3/library/profile.html
Supports
- cProfile is a C extension with reasonable overhead suitable for profiling long-running programs
- profile is a pure Python module that adds significant overhead
- Deterministic profiling monitors all function call, function return and exception events with precise timings between them
- The profilers introduce overhead for Python code but not for C-level functions, so C code would seem faster than any Python one
- tottime is the total time spent in the given function excluding time made in calls to sub-functions
- cumtime is the cumulative time spent in this and all subfunctions
- The underlying clock ticks at roughly .001 seconds, bounding measurement accuracy
- Dispatch latency accumulates as error for functions called many times
- The profile module provides calibration and can produce negative numbers at low call counts
- https://docs.python.org/3/whatsnew/2.5.html
Supports
- Python 2.5 was released on September 19, 2006
- cProfile is a C implementation of the existing profile module with much lower overhead, contributed by Armin Rigo
- https://github.com/benfred/py-spy
Supports
- py-spy is a sampling profiler for Python programs that visualizes what a program spends time on without restarting it or modifying the code
- It is written in Rust and does not run in the same process as the profiled program, making it safe against production code
- It reads memory with process_vm_readv on Linux, vm_read on macOS and ReadProcessMemory on Windows
- It reads PyInterpreterState and iterates PyFrameObject structures per thread to build call stacks
- The record, top and dump subcommands
- --native profiles native extensions written in C, C++ or Cython
- --gil includes only traces for threads holding the Global Interpreter Lock
- --format changes the output to speedscope profiles or raw data
- https://github.com/async-profiler/async-profiler
Supports
- async-profiler does not suffer from the safepoint bias problem
- It uses AsyncGetCallTrace and perf_events to collect stack traces and track memory allocations
- It profiles CPU time, memory allocations in Java heap and native regions, lock contention, and hardware and software counters
- It includes native and kernel frames in stack traces
- It outputs interactive HTML flame graphs, JFR format and heatmaps
- https://github.com/async-profiler/async-profiler/blob/master/docs/ProfilerOptions.md
Supports
- asprof -d 30 <pid> profiles CPU for 30 seconds
- asprof -e alloc, -e lock and -e wall select allocation, lock and wall-clock events
- asprof -f profile.html writes an HTML flame graph
- The -agentpath form with start,event=cpu,timeout=30,file=profile.html profiles from JVM launch
- https://openjdk.org/jeps/328
Supports
- JEP 328 provides a low-overhead data collection framework for troubleshooting Java applications and the HotSpot JVM
- Delivered in Release 11
- Goal of at most 1% performance overhead out-of-the-box on SPECjbb2015
- No measurable performance overhead when not enabled
- Flight Recorder was previously a commercial feature of the Oracle JDK and this JEP moves the source to the open repository
- https://openjdk.org/projects/jdk/11/
Supports
- JDK 11 reached General Availability on 25 September 2018
- Release schedule with Rampdown Phase One 2018/06/28, Rampdown Phase Two 2018/07/26, Initial Release Candidate 2018/08/16, Final Release Candidate 2018/08/30 and General Availability 2018/09/25
- https://docs.oracle.com/en/java/javase/21/troubleshoot/diagnostic-tools.html
Supports
- Flight Recorder is a profiling and event collection framework built into the JDK
- JFR continuously saves profiling information including thread samples, lock profiles and garbage collection details
- Data collected covers memory allocation, GC, code execution, threads and synchronization, I/O and system information
- JMC monitors and manages Java applications with very small performance overhead, suitable for production
- jcmd JFR.start, JFR.check, JFR.stop and JFR.dump control a running recording
- -XX:StartFlightRecording.delay=20s,duration=60s,name,filename,settings=profile starts a time-fixed recording at launch
- -XX:StartFlightRecording.disk=true,maxage=6h,settings=default with -XX:FlightRecorderOptions=repository starts a continuous recording
- https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-trace
Supports
- dotnet-trace is a cross-platform .NET diagnostic tool enabling collection of traces without a native profiler
- It is built on EventPipe of the .NET runtime
- The dotnet-sampled-thread-time profile samples .NET thread stacks at approximately 100 Hz using the runtime sample profiler with managed stacks
- With no profile or providers specified, collect enables dotnet-common and dotnet-sampled-thread-time
- The gc-verbose profile tracks GC collections and samples object allocations
- --format supports Chromium, NetTrace and Speedscope, with NetTrace the default
- --duration uses dd:hh:mm:ss format
- dotnet-trace ps lists traceable .NET processes with their command lines
- dotnet-trace report <tracefile> topN with -n and --inclusive reports the top methods by inclusive or exclusive time
- Speedscope files can be opened at speedscope.app and conversion from nettrace is irreversible
- https://valgrind.org/docs/manual/cl-manual.html
Supports
- Callgrind records the call history among functions in a program's run as a call graph
- It measures the number of instructions executed, caller/callee relationships and numbers of calls
- valgrind --tool=callgrind [callgrind options] your-program [program options] starts a profile run
- callgrind_annotate summarizes callgrind.out.<pid>
- callgrind_control -e -b annotates the backtrace with event counts
- --cache-sim=yes and --branch-sim=yes add cache and branch simulation, expecting a further slowdown by approximately a factor of 2
- Callgrind propagates costs across function call boundaries, unlike Cachegrind's flat self-attribution
- https://sourceware.org/binutils/docs/gprof.html
Supports
- Profiling changes how every function is compiled so it stashes information about where it was called from
- Every function calls mcount, which records call graph data by examining the stack frame
- Profiling keeps a histogram of where the program counter happens to be, typically around 100 times per second of run time
- The flat profile shows the total amount of time spent executing each function
- The call graph shows how much time was spent in each function and its children
- https://dl.acm.org/doi/10.1145/872726.806987
Supports
- Gprof: A call graph execution profiler by Susan L. Graham, Peter B. Kessler and Marshall K. McKusick
- Presented at the SIGPLAN '82 Symposium on Compiler Construction, SIGPLAN Notices volume 17 number 6, June 1982
- It accounts for the running time of called routines in the running time of the routines that call them by gathering arcs in the program's call graph
- https://kernelnewbies.org/Linux_2_6_31
Supports
- Linux 2.6.31 was released on 9 September 2009
- The Performance Counter subsystem abstracts performance counter hardware registers available on most modern CPUs
- Those registers count events such as instructions executed, cache misses suffered and branches mispredicted
- The perf tool was included, providing perf top, perf record, perf report and perf annotate
- https://research.google/pubs/google-wide-profiling-a-continuous-profiling-infrastructure-for-data-centers/
Supports
- Google-Wide Profiling: A Continuous Profiling Infrastructure for Data Centers by Gang Ren, Eric Tune, Tipp Moseley, Yixin Shi, Silvius Rus and Robert Hundt
- Published in IEEE Micro in 2010, pages 65-79
- GWP provides performance insights for cloud applications with negligible overhead and stable, accurate profiles
- It introduces applications including application-platform affinity measurement and identification of platform-specific microarchitectural peculiarities
- https://docs.cloud.google.com/profiler/docs/concepts-profiling
Supports
- Cloud Profiler is a statistical, or sampling, profiler with low overhead suitable for production environments
- Profile types CPU time, heap, allocated heap, contention, threads and wall time with differing language support across Go, Java, Node.js and Python
- A single profile represents data collected for 10 seconds for most profile types
- Heap usage and thread profiles are collected instantaneously
- The backend instructs an agent to capture a profile approximately once per minute per deployment and profile type
- https://cloud.google.com/profiler
Supports
- Google Cloud Profiler product home
- https://grafana.com/docs/pyroscope/latest/introduction/profiling-types/
Supports
- CPU profiling measures the amount of CPU time consumed by different parts of application code
- Memory allocation profiling tracks the amount and frequency of memory allocations, reported as alloc objects and alloc space
- Goroutine profiling measures the usage and performance of goroutines
- Mutex profiling analyzes mutex locks with mutex count and mutex duration
- Block profiling measures the frequency and duration of blocking operations where a thread is paused or delayed
- https://grafana.com/oss/pyroscope/
Supports
- Grafana Pyroscope open-source continuous profiling database home
- https://grafana.com/blog/pyroscope-grafana-phlare-join-for-oss-continuous-profiling/
Supports
- Grafana Labs merges the Pyroscope project and Grafana Phlare under the new name Grafana Pyroscope
- Phlare was launched by Grafana Labs the previous year, 2022
- https://grafana.com/press/2023/03/15/grafana-labs-acquires-pyroscope-the-company-behind-the-popular-open-source-continuous-profiling-project/
Supports
- Grafana Labs announced the acquisition of Pyroscope on March 15, 2023
- Pyroscope was founded in 2021 and created the open source continuous profiling project of the same name
- https://www.parca.dev/docs/overview/
Supports
- Parca is a continuous profiling project; continuous profiling is taking profiles such as CPU, memory and I/O of programs in a systematic way
- A super low overhead profiler powered by eBPF
- A multi-dimensional data model with series of profiles identified by profile type and key/value pairs
- A label-selector based query language and a query engine designed for profiling data
- https://www.polarsignals.com/blog/posts/2021/10/08/introducing-parca-we-got-funded
Supports
- Parca was announced on October 8, 2021 as "Prometheus but for profiles"
- Parca is an umbrella project home to storage optimized for profiling data and a super low overhead eBPF based profiler
- The Parca Agent discovers targets using Kubernetes or systemd so an entire infrastructure can be profiled without restarts or deployment changes
- Storage allows slicing with label-selectors the same way as in Prometheus
- https://www.polarsignals.com/
Supports
- Polar Signals hosted continuous profiling product built by the Parca team
- https://opentelemetry.io/blog/2024/elastic-contributes-continuous-profiling-agent/
Supports
- OpenTelemetry accepted Elastic's contribution of its eBPF-based continuous profiling agent
- The agent observes code across different programming languages and runtimes, third-party libraries, kernel operations and system resources
- It eliminates the need for code instrumentation, recompilation or service restarts
- Low overhead with approximately 1% CPU usage in production environments
- Supported languages and runtimes include C/C++, Rust, Zig, Go, Java, Python, Ruby, PHP, Node.js/V8, Perl and .NET
- Post published 2024-06-07
- https://opentelemetry.io/blog/2024/profiling/
Supports
- The OpenTelemetry profiling signal connects profiles with other telemetry signals from applications and infrastructure
- The Profiling SIG was created in 2022 and a profiling data model OTEP was merged during 2024
- Elastic pledged to donate its eBPF-based profiling agent
- https://github.com/open-telemetry/opentelemetry-ebpf-profiler
Supports
- The OpenTelemetry eBPF profiler repository, the vendor-neutral continuous profiling agent
- https://www.elastic.co/observability/universal-profiling
Supports
- Elastic Universal Profiling uses eBPF and OpenTelemetry to profile every line of code running on the machine, including kernel and third-party libraries
- No intrusive code changes or instrumentation needed
- Less than 1% CPU overhead, able to run continuously on production systems
- Language support including PHP, Python, Java and JVM languages, Go, Rust, C/C++, Node.js/V8, Ruby, Perl and Zig
- https://www.datadoghq.com/product/code-profiling/
Supports
- Datadog Continuous Profiler provides always-on visibility into code behavior in production with low overhead and minimal setup
- Spans and traces are correlated with profiling data to reveal root causes of latency when traces lack detail
- Capabilities include continuous profiling across hosts and containers, thread-level visibility, memory leak and allocation profiling, and performance regression tracking across deployments
- https://perfetto.dev/docs/
Supports
- Perfetto is an open-source suite of SDKs, daemons and tools that use tracing to understand system behavior
- System probes on Android and Linux capture scheduling states, CPU frequencies, memory profiling and callstack sampling
- Callstack sampling profiles on Android for high CPU usage in C++, Java and Kotlin code
- Heap profiles on Linux for high memory usage of C, C++ and Rust applications
- A browser-based UI and an SQL-based analysis library
- https://perfetto.dev/
Supports
- Perfetto system tracing and profiling project home
- https://developer.chrome.com/docs/devtools/performance
Supports
- The Performance panel shows a flame chart of activity on the main thread over time, where the x-axis represents the recording over time
- Each bar represents an event, with a CPU chart and Summary tab breakdown
- Red triangles warn of potential issues with an event
- https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html
Supports
- Intel VTune Profiler attributes performance to code using processor performance events on Intel systems
- https://www.amd.com/en/developer/uprof.html
Supports
- AMD uProf provides counter-based performance analysis for AMD processors
- https://www.speedscope.app/
Supports
- speedscope is a fast interactive web-based viewer for performance profiles supporting import from many profilers and languages
- Time Order view orders call stacks left to right in the order they occurred in the input file
- Left Heavy view groups identical stacks and sorts the heaviest stack for each parent to the left
- Sandwich view is a table of all functions with self and total times, showing callers and callees of a selected row
- https://github.com/adriannovegil/awesome-profiling
Supports
- Discovery of Hotspot, Parca, Bytehound, gprof2dot, Flame Graph, FlameScope, Likwid, Pyroscope, Polar Signals, Arthas, Scalene, fgprof and flamebearer as profiling ecosystem resources
- Awesome Links curation decision for software profiling tooling
- https://github.com/kubo39/awesome-profiling
Supports
- Discovery of bcc, perf-tools, gperftools, oprofile, Coz and PAPI as profiling ecosystem resources
- https://github.com/KDAB/hotspot
Supports
- Hotspot is the Linux perf GUI for performance analysis
- Its main feature is graphical visualization of a perf.data file, including flame graph and caller-callee pages
- https://github.com/Netflix/flamescope
Supports
- FlameScope is a visualization tool for exploring different time ranges as flame graphs, for analyzing perturbations, variance and single-threaded execution
- It displays input data as an interactive subsecond-offset heat map, and a flame graph is generated for a selected time range
- https://github.com/jrfonseca/gprof2dot
Supports
- gprof2dot is a Python script that converts the output from many profilers into a Graphviz dot graph
- https://github.com/plasma-umass/scalene
Supports
- Scalene is a high-performance CPU, GPU and memory profiler for Python
- It profiles at the line level and per function, pointing to the lines responsible for execution time
- It separates out time spent in Python from time in native code including libraries
- It separates out system time, making I/O bottlenecks visible
- https://github.com/koute/bytehound
Supports
- Bytehound is a memory profiler for Linux
- It analyzes memory leaks, shows where memory is consumed, identifies temporary allocations and investigates fragmentation
- It gathers every allocation and deallocation along with full stack traces
- https://github.com/felixge/fgprof
Supports
- fgprof is a sampling Go profiler that analyzes On-CPU and Off-CPU time together
- Go's builtin sampling CPU profiler can only show On-CPU time
- This kind of profiling is also known as wall-clock profiling and suits mixed I/O and CPU workloads
- https://arthas.aliyun.com/en/
Supports
- Arthas is a Java diagnostic tool from the Alibaba Middleware Group
- Features include a real-time dashboard, viewing method parameters, return values and exceptions, online hotswap, class conflict resolution and flame graph generation for locating application hotspots
- https://github.com/mapbox/flamebearer
Supports
- flamebearer reads Chrome DevTools Performance recordings and Node .cpuprofile files and renders flame graphs
- flamebearer-node can profile a script directly
- https://github.com/RRZE-HPC/likwid
Supports
- likwid-perfctr configures and reads out hardware performance counters on Intel, AMD, ARM and POWER processors and Nvidia GPUs
- A Marker API activated with -m instruments specific code regions
- https://github.com/plasma-umass/coz
Supports
- Coz employs causal profiling, which measures optimization potential and predicts the impact of optimizing code
- A causal profiler uses performance experiments rather than instrumentation to establish that optimizing function X will have effect Y
- It measures optimization potential for serial, parallel and asynchronous programs
- https://icl.utk.edu/papi/
Supports
- PAPI is the Performance Application Programming Interface, a portable interface to hardware performance counters across processor architectures
- https://github.com/iovisor/bcc
Supports
- BCC is the BPF Compiler Collection, making eBPF programs easier to write for kernel instrumentation
- tools/profile profiles CPU usage by sampling stack traces at a timed interval
- tools/offcputime summarizes off-CPU time by kernel stack trace
