Linux Performance Tuning
Linux performance tuning is the disciplined work of measuring a Linux workload, finding the resource or code path that limits it, and changing the smallest relevant control. It covers CPU scheduling, memory pressure, storage and network behavior, application profiling, and validation under a repeatable workload.
itLinux | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic — Linux Performance Tuning
Linux performance tuning is the disciplined refusal to poke a kernel setting because a graph looked grumpy. A Linux application runs instructions and makes system calls; the kernel then schedules work, manages memory, moves files, and sends packets. A delay can begin anywhere along that route, which is inconvenient but at least prevents boredom.
Start with the work that matters. Throughput is completed work per unit of time. Latency is the time for one operation. Utilization tells you how busy a resource was, while saturation tells you that demand had to wait. Those are not four names for the same anxious feeling. A system can be busy and healthy, or quiet and still make a request wait in a queue.
The first surprise is that Linux load average is not a CPU percentage. It counts runnable tasks and tasks in uninterruptible sleep, so high load with idle CPU can point at blocked work rather than exhausted processors. Pressure stall information, or PSI, adds another useful clue: it reports time lost while tasks cannot progress because CPU, memory, or input and output resources are contested. The machine is not being mysterious. It is being specific in several places at once.
The practical route is broad evidence first, detail later. Compare workload latency, throughput, and errors with CPU, memory, storage, network, and pressure signals. Then narrow the question to a process, control group, device, socket, or code path. Metrics tell you where to look; a profile estimates where execution time gathers; tracing records selected events with timing and context. Turning on everything at once is a fine way to produce a large collection of facts about your confusion.
A tuning control always moves a tradeoff. CPU affinity can improve locality and also strand capacity. Lower swappiness changes the balance between anonymous memory and filesystem cache. Larger dirty limits can absorb a write burst and later create a longer writeback stall. The useful change is one that improves the chosen workload outcome while errors, tail latency, pressure, and resource use remain acceptable.
Read the Intro for the complete path through CPU, memory, storage, and network behavior. Use Slides for the map of signals and decision points. Keep the Cheatsheet nearby when comparing metrics and controls. The Reference tab leads into the kernel and Red Hat documentation when a hypothesis needs exact semantics. That is the whole arrangement: measure, localize, change one thing, and make the result earn its keep.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/monitoring_and_managing_system_status_and_performance/index
Supports
- Measurement and monitoring before optimization
- CPU, memory, storage, network, NUMA, TuneD, perf, BCC, vmstat, and iostat behavior
- Tuning tradeoffs and workload-specific validation
- https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/monitoring_and_managing_system_status_and_performance/overview-of-performance-monitoring-options
Supports
- Roles of top, ps, vmstat, sar, perf, BCC, iostat, ss, numastat, and related tools
- Reference-path rationale for the monitoring overview
- https://www.brendangregg.com/linuxperf.html
Supports
- Workload-first performance methodology
- Latency, utilization, saturation, profiling, tracing, and subsystem localization
- Reference-path rationale for Linux performance maps
- https://docs.kernel.org/6.10/accounting/psi.html
Supports
- CPU, memory, and I/O pressure stall information
- Some and full stall semantics
- PSI use for detecting resource contention
- https://docs.kernel.org/admin-guide/perf-security.html
Supports
- perf events data categories
- perf access controls and sensitive monitoring data
- https://docs.kernel.org/trace/index.html
Supports
- Linux tracing facility landscape
- Tracepoint, ftrace, kprobe, and event-tracing reference path
- https://docs.kernel.org/trace/kprobes.html
Supports
- Dynamic kernel function and return probes
- Kprobe capabilities and restrictions
- https://docs.kernel.org/admin-guide/sysctl/vm.html
Supports
- Swappiness, dirty-page, reclaim, compaction, and huge-page controls
- Memory tuning tradeoffs
- https://docs.kernel.org/scheduler/sched-design-CFS.html
Supports
- CFS virtual-runtime model
- Linux 2.6.23 merge and group scheduling
- https://www.man7.org/linux/man-pages/man5/proc_loadavg.5.html
Supports
- Runnable and uninterruptible tasks in load average
- One, five, and fifteen minute fields
- https://man7.org/linux/man-pages/man2/perf_event_open.2.html
Supports
- Hardware and software performance-event interfaces
- Sampling, counting, and event configuration
- https://www.rfc-editor.org/rfc/rfc6349.html
Supports
- TCP throughput testing methodology
- Bandwidth-delay product and window relationships
- https://github.com/sindresorhus/awesome
Supports
- Discovery of the Awesome eBPF list
- https://github.com/zoidbergwill/awesome-ebpf
Supports
- Discovery of BCC, bpftrace, Parca, Pixie, and Coroot
- eBPF observability and tracing-tool categorization
- https://github.com/iovisor/bcc/blob/master/docs/tutorial.md
Supports
- BCC tools for scheduler, syscall, filesystem, and block-I/O tracing
- Awesome Links rationale for BCC
- https://bpftrace.org/docs/release_024/guide
Supports
- High-level eBPF tracing language, probes, maps, and aggregation
- Awesome Links rationale for bpftrace
- https://www.parca.dev/docs/overview
Supports
- Continuous profiling and historical CPU-profile analysis
- Awesome Links rationale for Parca
- https://docs.px.dev/
Supports
- eBPF-based Kubernetes observability, protocol tracing, and profiling
- Awesome Links rationale for Pixie
- https://docs.coroot.com/
Supports
- Service and infrastructure observability across Linux resource signals
- Awesome Links rationale for Coroot
- https://oprofile.sourceforge.io/about/
Supports
- OProfile history and system-wide hardware-counter profiling
- https://www.brendangregg.com/Slides/LISA2014_LinuxPerfAnalysisNewTools/
Supports
- 2004 kprobes, 2008 ftrace, 2009 perf, 2009 tracepoints, and 2014 eBPF milestones
- https://www.brendangregg.com/Slides/SCALE2017_perf_analysis_eBPF.pdf
Supports
- 2012 uprobes milestone
- eBPF performance-analysis context
- https://lwn.net/Articles/742082/
Supports
- April 2015 BCC release and packaged tracing tools
- https://kernelnewbies.org/Linux_4.20
Supports
- December 2018 Linux 4.20 release and PSI inclusion
- https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html
Supports
- February 2020 BPF CO-RE model
- BTF, compiler relocation, and libbpf roles
- https://grafana.com/products/cloud/
Supports
- Grafana Cloud landscape placement for metrics, logs, traces, and profiles
- https://www.datadoghq.com/product/infrastructure-monitoring/
Supports
- Datadog landscape placement for host, process, and container monitoring
- https://www.dynatrace.com/platform/infrastructure-observability/
Supports
- Dynatrace landscape placement for infrastructure topology and observability
- https://newrelic.com/platform/infrastructure-monitoring
Supports
- New Relic landscape placement for host and process telemetry
- https://www.elastic.co/observability
Supports
- Elastic landscape placement for metrics, logs, traces, and profiling
- https://pcp.io/
Supports
- Performance Co-Pilot live and archived system measurement
- https://www.redhat.com/en/technologies/management/insights
Supports
- Red Hat Insights landscape placement for RHEL fleet analysis
- https://www.brendangregg.com/blog/2015-03-03/performance-tuning-linux-instances-on-ec2.html
Supports
- Field Notes: tuning is a workload-specific process rather than a transferable setting set
- https://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html
Supports
- Field Notes: CPU utilization can conceal stalled execution and needs counter evidence for CPU tuning decisions
- https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html
Supports
- Field Notes: Linux load average combines runnable and uninterruptible tasks and is a clue rather than a resource diagnosis
- https://www.brendangregg.com/USEmethod/use-linux.html
Supports
- Field Notes: saturation and error metrics complement utilization when locating resource bottlenecks
- https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it/
Supports
- Field Notes: TCP receive-buffer growth can consume shared memory and affect unrelated connections
- https://man7.org/linux/man-pages/man1/taskset.1.html
Supports
- Practice reference and exercise: taskset constrains a process to a selected CPU affinity mask
