Linux Performance Tuning
Linux performance tuning is the disciplined work of measuring a Linux workload, finding the resource or code path that limits it, and changing the smallest relevant control. It covers CPU scheduling, memory pressure, storage and network behavior, application profiling, and validation under a repeatable workload.
itLinux | OpenSkills.info
Intro
Linux Performance Tuning
Linux performance tuning is a measurement-driven cycle. You define an outcome, reproduce the workload, observe where time and capacity go, form a hypothesis, change one relevant control, and repeat the same measurement. The goal is not to make every metric smaller. The goal is to improve a workload objective, such as request latency or batch throughput, without breaking correctness, stability, or another workload.
A Linux system sits between applications and hardware. Applications issue system calls and run instructions. The kernel schedules runnable tasks on CPUs, manages virtual memory, moves data through filesystems and block devices, and sends packets through the network stack. A delay visible to an application may originate in any of these layers. The first task is therefore localization, not configuration.
Begin with the workload
A performance claim needs a defined workload and a measured outcome. Throughput describes completed work per unit of time. Latency describes the time required for one unit of work. Utilization describes how busy a resource is. Saturation means demand is waiting because a resource cannot serve it immediately. Errors matter because a fast system that drops requests has not improved.
Record the workload shape, data set, concurrency, warm-up period, measurement interval, kernel and application configuration, and relevant hardware or virtual-machine placement. Compare like with like. Caches, background jobs, power management, noisy neighbors, and changing input can move a result even when the code and configuration remain unchanged.
Percentiles expose the distribution that an average hides. A median may remain stable while the ninety-ninth percentile grows. Use a percentile only when the sample population and interval are clear. A single benchmark score does not explain why a change helped, and a microbenchmark may not represent production behavior.
Follow demand through the system
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/monitoring_and_managing_system_status_and_performance/index
Supports
- Measurement and monitoring before optimization
- CPU, memory, storage, network, NUMA, TuneD, perf, BCC, vmstat, and iostat behavior
- Tuning tradeoffs and workload-specific validation
- https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/monitoring_and_managing_system_status_and_performance/overview-of-performance-monitoring-options
Supports
- Roles of top, ps, vmstat, sar, perf, BCC, iostat, ss, numastat, and related tools
- Reference-path rationale for the monitoring overview
- https://www.brendangregg.com/linuxperf.html
Supports
- Workload-first performance methodology
- Latency, utilization, saturation, profiling, tracing, and subsystem localization
- Reference-path rationale for Linux performance maps
- https://docs.kernel.org/6.10/accounting/psi.html
Supports
- CPU, memory, and I/O pressure stall information
- Some and full stall semantics
- PSI use for detecting resource contention
- https://docs.kernel.org/admin-guide/perf-security.html
Supports
- perf events data categories
- perf access controls and sensitive monitoring data
- https://docs.kernel.org/trace/index.html
Supports
- Linux tracing facility landscape
- Tracepoint, ftrace, kprobe, and event-tracing reference path
- https://docs.kernel.org/trace/kprobes.html
Supports
- Dynamic kernel function and return probes
- Kprobe capabilities and restrictions
- https://docs.kernel.org/admin-guide/sysctl/vm.html
Supports
- Swappiness, dirty-page, reclaim, compaction, and huge-page controls
- Memory tuning tradeoffs
- https://docs.kernel.org/scheduler/sched-design-CFS.html
Supports
- CFS virtual-runtime model
- Linux 2.6.23 merge and group scheduling
- https://www.man7.org/linux/man-pages/man5/proc_loadavg.5.html
Supports
- Runnable and uninterruptible tasks in load average
- One, five, and fifteen minute fields
- https://man7.org/linux/man-pages/man2/perf_event_open.2.html
Supports
- Hardware and software performance-event interfaces
- Sampling, counting, and event configuration
- https://www.rfc-editor.org/rfc/rfc6349.html
Supports
- TCP throughput testing methodology
- Bandwidth-delay product and window relationships
- https://github.com/sindresorhus/awesome
Supports
- Discovery of the Awesome eBPF list
- https://github.com/zoidbergwill/awesome-ebpf
Supports
- Discovery of BCC, bpftrace, Parca, Pixie, and Coroot
- eBPF observability and tracing-tool categorization
- https://github.com/iovisor/bcc/blob/master/docs/tutorial.md
Supports
- BCC tools for scheduler, syscall, filesystem, and block-I/O tracing
- Awesome Links rationale for BCC
- https://bpftrace.org/docs/release_024/guide
Supports
- High-level eBPF tracing language, probes, maps, and aggregation
- Awesome Links rationale for bpftrace
- https://www.parca.dev/docs/overview
Supports
- Continuous profiling and historical CPU-profile analysis
- Awesome Links rationale for Parca
- https://docs.px.dev/
Supports
- eBPF-based Kubernetes observability, protocol tracing, and profiling
- Awesome Links rationale for Pixie
- https://docs.coroot.com/
Supports
- Service and infrastructure observability across Linux resource signals
- Awesome Links rationale for Coroot
- https://oprofile.sourceforge.io/about/
Supports
- OProfile history and system-wide hardware-counter profiling
- https://www.brendangregg.com/Slides/LISA2014_LinuxPerfAnalysisNewTools/
Supports
- 2004 kprobes, 2008 ftrace, 2009 perf, 2009 tracepoints, and 2014 eBPF milestones
- https://www.brendangregg.com/Slides/SCALE2017_perf_analysis_eBPF.pdf
Supports
- 2012 uprobes milestone
- eBPF performance-analysis context
- https://lwn.net/Articles/742082/
Supports
- April 2015 BCC release and packaged tracing tools
- https://kernelnewbies.org/Linux_4.20
Supports
- December 2018 Linux 4.20 release and PSI inclusion
- https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html
Supports
- February 2020 BPF CO-RE model
- BTF, compiler relocation, and libbpf roles
- https://grafana.com/products/cloud/
Supports
- Grafana Cloud landscape placement for metrics, logs, traces, and profiles
- https://www.datadoghq.com/product/infrastructure-monitoring/
Supports
- Datadog landscape placement for host, process, and container monitoring
- https://www.dynatrace.com/platform/infrastructure-observability/
Supports
- Dynatrace landscape placement for infrastructure topology and observability
- https://newrelic.com/platform/infrastructure-monitoring
Supports
- New Relic landscape placement for host and process telemetry
- https://www.elastic.co/observability
Supports
- Elastic landscape placement for metrics, logs, traces, and profiling
- https://pcp.io/
Supports
- Performance Co-Pilot live and archived system measurement
- https://www.redhat.com/en/technologies/management/insights
Supports
- Red Hat Insights landscape placement for RHEL fleet analysis