openskills.info
Course Preview

Software Profiling

Profiling measures where a running program spends its time and memory, attributing that cost to the specific functions and call paths responsible. It answers the question a log line or a request timing cannot: not that an operation was slow, but which code inside it was slow, and how much of the total that code accounts for.

itObservability and performance

Software Profiling

Profiling measures how a running program consumes a resource — CPU time, wall-clock time, memory, lock waits — and attributes that consumption to the code responsible. The output is a profile: a set of call stacks, each carrying a measured cost. A profile does not say "this request took 800 ms". It says "43% of the CPU time in this process was spent under parseTimestamp, called from deserializeRow, called from the batch loader".

That attribution is the whole point, and it marks the boundary against the neighboring practices:

PracticeQuestion it answers
ProfilingWhere inside a process does the time or memory go?
Distributed tracing / APMWhere between services does a request spend its latency?
Load and performance testingWhat does the system do under a defined applied workload?
Metrics and monitoringIs the system currently within its targets?

The four are complementary. A trace tells you the checkout service accounts for 700 ms of a 900 ms request. A profile of that service tells you which functions inside it burned the 700 ms. Neither substitutes for the other.

What a profiler actually collects

Every profiler, regardless of language or platform, runs the same four-stage path:

  1. Trigger. Something interrupts or notifies the profiler that a unit of cost occurred — a timer expiry, a hardware counter overflow, a memory allocation, a lock acquisition, a thread context switch.
  2. Stack capture. At that instant the profiler walks the call stack of the affected thread, producing an ordered list of return addresses.
  3. Symbolization. Raw addresses are resolved into function names, source files, and line numbers using debug information, symbol tables, or runtime-supplied maps.
  4. Aggregation. Identical stacks are merged and their costs summed, turning millions of individual observations into a few thousand distinct stacks with weights.

Stage 4 is why a profile is compact enough to read. It is also why a profile has no timeline: merging discards the order in which samples arrived. Tools that keep the ordering are tracers, not profilers, and they pay for it in data volume.

Sampling and instrumentation

There are two ways to reach stage 1, and the choice determines almost everything else about a profiler's behavior.

Continue the course

This section is part of the paid course.

See pricing to subscribe, or log in if you already have access.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Sources

  • https://man7.org/linux/man-pages/man2/perf_event_open.2.html
  • https://man7.org/linux/man-pages/man1/perf-record.1.html
  • https://man7.org/linux/man-pages/man1/perf-report.1.html
  • https://perf.wiki.kernel.org/
  • https://www.brendangregg.com/flamegraphs.html
  • https://www.brendangregg.com/perf.html
  • https://github.com/brendangregg/FlameGraph
  • https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer
  • https://go.dev/blog/pprof
  • https://pkg.go.dev/net/http/pprof
  • https://pkg.go.dev/runtime/pprof
  • https://github.com/google/pprof
  • https://docs.python.org/3/library/profile.html
  • https://docs.python.org/3/whatsnew/2.5.html
  • https://github.com/benfred/py-spy
  • https://github.com/async-profiler/async-profiler
  • https://github.com/async-profiler/async-profiler/blob/master/docs/ProfilerOptions.md
  • https://openjdk.org/jeps/328
  • https://openjdk.org/projects/jdk/11/
  • https://docs.oracle.com/en/java/javase/21/troubleshoot/diagnostic-tools.html
  • https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-trace
  • https://valgrind.org/docs/manual/cl-manual.html
  • https://sourceware.org/binutils/docs/gprof.html
  • https://dl.acm.org/doi/10.1145/872726.806987
  • https://kernelnewbies.org/Linux_2_6_31
  • https://research.google/pubs/google-wide-profiling-a-continuous-profiling-infrastructure-for-data-centers/
  • https://docs.cloud.google.com/profiler/docs/concepts-profiling
  • https://cloud.google.com/profiler
  • https://grafana.com/docs/pyroscope/latest/introduction/profiling-types/
  • https://grafana.com/oss/pyroscope/
  • https://grafana.com/blog/pyroscope-grafana-phlare-join-for-oss-continuous-profiling/
  • https://grafana.com/press/2023/03/15/grafana-labs-acquires-pyroscope-the-company-behind-the-popular-open-source-continuous-profiling-project/
  • https://www.parca.dev/docs/overview/
  • https://www.polarsignals.com/blog/posts/2021/10/08/introducing-parca-we-got-funded
  • https://www.polarsignals.com/
  • https://opentelemetry.io/blog/2024/elastic-contributes-continuous-profiling-agent/
  • https://opentelemetry.io/blog/2024/profiling/
  • https://github.com/open-telemetry/opentelemetry-ebpf-profiler
  • https://www.elastic.co/observability/universal-profiling
  • https://www.datadoghq.com/product/code-profiling/
  • https://perfetto.dev/docs/
  • https://perfetto.dev/
  • https://developer.chrome.com/docs/devtools/performance
  • https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html
  • https://www.amd.com/en/developer/uprof.html
  • https://www.speedscope.app/
  • https://github.com/adriannovegil/awesome-profiling
  • https://github.com/kubo39/awesome-profiling
  • https://github.com/KDAB/hotspot
  • https://github.com/Netflix/flamescope
  • https://github.com/jrfonseca/gprof2dot
  • https://github.com/plasma-umass/scalene
  • https://github.com/koute/bytehound
  • https://github.com/felixge/fgprof
  • https://arthas.aliyun.com/en/
  • https://github.com/mapbox/flamebearer
  • https://github.com/RRZE-HPC/likwid
  • https://github.com/plasma-umass/coz
  • https://icl.utk.edu/papi/
  • https://github.com/iovisor/bcc