openskills.info
Course Preview

Software Profiling

Profiling measures where a running program spends its time and memory, attributing that cost to the specific functions and call paths responsible. It answers the question a log line or a request timing cannot: not that an operation was slow, but which code inside it was slow, and how much of the total that code accounts for.

itObservability and performance

Don't Panic — Software Profiling

Software profiling is the practice of measuring where a running program spends a resource and attaching that cost to the call stacks responsible. It exists because “the service is slow” is a perfectly accurate statement with almost no useful steering information. A request timer can report eight hundred milliseconds. A profile can show that a particular path consumed a large share of the CPU beneath it. That is the difference between a weather report and a map.

The machinery is less mystical than its graphs suggest. A profiler notices an event, captures the stack, turns addresses into names through symbolization, and merges matching stacks into weighted totals. The merge is the useful bit and the annoying bit. It reduces millions of observations into something a human can read, but it also discards their order. A flame graph is not a tiny cinema of the program's day. Its width shows how often a frame appeared; its horizontal position is not time, despite looking extremely time-shaped.

The first fork in the road is sampling versus instrumentation. Sampling takes periodic observations, so its overhead follows the sample rate rather than the number of calls. That makes it the normal production choice. Instrumentation observes every call and can give exact counts, but its cost follows the work being measured and can alter the ratios under inspection. Exactness has a bill, as these things tend to do. Use it on a development machine when call counts matter.

Then choose the resource before admiring any output. A CPU profile sees threads that are running. A slow request waiting on a lock, disk, or database can leave that profile nearly empty, which is not a profiler sulking. It is evidence that the time is off-CPU and needs a wall-clock, block, mutex, or off-CPU profile. Likewise, allocation rate identifies garbage-collection pressure, while an in-use heap profile identifies memory still retained. Similar nouns have caused more than one long afternoon.

Read the Course tab for the collection path, the trade-offs in stack unwinding, and the limits of each profile type. The Cheatsheet is the compact reference when a graph contains mysterious main children or unknown addresses. The Practice tab supplies commands for perf, Go, Python, JVM, and .NET. Field Notes carries the operational traps that profiles politely decline to put in the title. The Quiz checks whether the vocabulary has become a working map rather than an attractive pile of rectangles.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Sources

  • https://man7.org/linux/man-pages/man2/perf_event_open.2.html
  • https://man7.org/linux/man-pages/man1/perf-record.1.html
  • https://man7.org/linux/man-pages/man1/perf-report.1.html
  • https://perf.wiki.kernel.org/
  • https://www.brendangregg.com/flamegraphs.html
  • https://www.brendangregg.com/perf.html
  • https://github.com/brendangregg/FlameGraph
  • https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer
  • https://go.dev/blog/pprof
  • https://pkg.go.dev/net/http/pprof
  • https://pkg.go.dev/runtime/pprof
  • https://github.com/google/pprof
  • https://docs.python.org/3/library/profile.html
  • https://docs.python.org/3/whatsnew/2.5.html
  • https://github.com/benfred/py-spy
  • https://github.com/async-profiler/async-profiler
  • https://github.com/async-profiler/async-profiler/blob/master/docs/ProfilerOptions.md
  • https://openjdk.org/jeps/328
  • https://openjdk.org/projects/jdk/11/
  • https://docs.oracle.com/en/java/javase/21/troubleshoot/diagnostic-tools.html
  • https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-trace
  • https://valgrind.org/docs/manual/cl-manual.html
  • https://sourceware.org/binutils/docs/gprof.html
  • https://dl.acm.org/doi/10.1145/872726.806987
  • https://kernelnewbies.org/Linux_2_6_31
  • https://research.google/pubs/google-wide-profiling-a-continuous-profiling-infrastructure-for-data-centers/
  • https://docs.cloud.google.com/profiler/docs/concepts-profiling
  • https://cloud.google.com/profiler
  • https://grafana.com/docs/pyroscope/latest/introduction/profiling-types/
  • https://grafana.com/oss/pyroscope/
  • https://grafana.com/blog/pyroscope-grafana-phlare-join-for-oss-continuous-profiling/
  • https://grafana.com/press/2023/03/15/grafana-labs-acquires-pyroscope-the-company-behind-the-popular-open-source-continuous-profiling-project/
  • https://www.parca.dev/docs/overview/
  • https://www.polarsignals.com/blog/posts/2021/10/08/introducing-parca-we-got-funded
  • https://www.polarsignals.com/
  • https://opentelemetry.io/blog/2024/elastic-contributes-continuous-profiling-agent/
  • https://opentelemetry.io/blog/2024/profiling/
  • https://github.com/open-telemetry/opentelemetry-ebpf-profiler
  • https://www.elastic.co/observability/universal-profiling
  • https://www.datadoghq.com/product/code-profiling/
  • https://perfetto.dev/docs/
  • https://perfetto.dev/
  • https://developer.chrome.com/docs/devtools/performance
  • https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html
  • https://www.amd.com/en/developer/uprof.html
  • https://www.speedscope.app/
  • https://github.com/adriannovegil/awesome-profiling
  • https://github.com/kubo39/awesome-profiling
  • https://github.com/KDAB/hotspot
  • https://github.com/Netflix/flamescope
  • https://github.com/jrfonseca/gprof2dot
  • https://github.com/plasma-umass/scalene
  • https://github.com/koute/bytehound
  • https://github.com/felixge/fgprof
  • https://arthas.aliyun.com/en/
  • https://github.com/mapbox/flamebearer
  • https://github.com/RRZE-HPC/likwid
  • https://github.com/plasma-umass/coz
  • https://icl.utk.edu/papi/
  • https://github.com/iovisor/bcc
  • https://blog.cloudflare.com/clickhouse-query-plan-contention/
  • https://engineering.fb.com/2009/04/02/production-engineering/xhproflive-continuous-function-level-performance-stats-from-production/
  • https://engineering.fb.com/2016/08/31/core-infra/apache-spark-scale-a-60-tb-production-use-case/