Performance Testing
Performance testing measures how quickly, reliably, and efficiently a system works under defined conditions. It applies controlled work, observes user-facing results and resource use, and compares the evidence with explicit targets.
itObservability and performance | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic — Performance Testing
Every performance test is trying to answer one nervous question: when the real users arrive, does the thing hold up? The honest answer is never a clean "yes". It is "under these conditions, measured this way, the system met these targets". This tab is about why that sentence needs so many clauses.
Start with what a performance test is not. It is not switching on a lot of traffic and watching a dashboard go red. It is an experiment: you decide the question first, such as whether a release meets a latency target or whether a service survives the forecast, and only then build conditions that let the numbers answer it.
Those conditions have four parts. The workload model is the operations, data, timing, and concurrency that stand in for real demand. The load generators are the machines that produce it. The system under test is your application plus every dependency inside the boundary you drew. Measurement is the clocks and counters on both sides, and both sides matter: if a generator runs out of processor or sockets, the result describes the generator, not the server.
The idea that trips people up is that a count of users is not a workload. Five hundred people acting once an hour and five hundred acting once a minute are very different loads. What matters is throughput, the work completed per second, and concurrency, the work in flight at once. Those are not the same number, and confusing them is how a test misses its target by an order of magnitude.
There are two ways to drive the load. A closed model keeps a fixed set of virtual users who wait for each response before continuing; when the system slows, they slow down with it, which quietly hides the pressure you were trying to create.
An open model schedules arrivals regardless of what came back, so when the system falls behind, queues grow, exactly as they would during a real incident. Choosing the wrong one is choosing whether your test can see the problem at all.
When you read results, respect the distribution. An average folds the fast and the slow into one number that describes neither. A 95th percentile of 400 milliseconds means 95 percent of responses were that quick or quicker; it says nothing about the slowest five percent, and those are the ones that wake someone up.
Then the humble part. A performance test supports a claim only for the workload, data, build, and hardware you actually tested. It does not prove behaviour at twice the scale, or under a dependency failure nobody injected. Recovery counts too: whether queues drain and instances come back once the load stops is part of the result, not an afterthought.
Where to go next: the Intro builds this experiment idea slowly and hands you the vocabulary. The Cheatsheet is the reference to keep open while you design a run. Field Notes is where the expensive lessons live, the ones that normally cost real traffic to learn.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://www.istqb.org/wp-content/uploads/2024/11/ISTQB-CT-PT_Syllabus_v1.0_2018.pdf
Supports
- Performance efficiency as time behavior, resource utilization, and capacity
- Load, stress, spike, endurance, scalability, capacity, and concurrency test vocabulary
- Operational profiles, load profiles, throughput, concurrency, and think time
- Open and closed system concepts and the closed-model throughput relationship
- Test planning, environment control, load generation, monitoring, analysis, and reporting
- Performance Testing syllabus general availability on 9 December 2018
- https://sre.google/sre-book/monitoring-distributed-systems/
Supports
- Latency, traffic, errors, and saturation as user-facing and system signals
- Separate latency treatment for successful and failed requests
- Implicit failures such as incorrect content despite a successful status
- https://sre.google/sre-book/addressing-cascading-failures/
Supports
- Queue growth, overload, retries, rejection, and load shedding
- Recovery and overload behavior as reliability concerns
- https://grafana.com/docs/k6/latest/testing-guides/api-load-testing/
Supports
- Test objectives before scenario configuration
- Smoke, average-load, stress, and spike test purposes
- Realistic scenario modeling
- https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/open-vs-closed/
Supports
- Open and closed workload scheduling
- Response-time coupling in a closed model
- Coordinated omission risk
- https://grafana.com/docs/k6/latest/using-k6/metrics/
Supports
- Counts, rates, trends, and latency percentiles
- Request duration, failure rate, and iteration metrics
- https://grafana.com/docs/k6/latest/testing-guides/running-large-tests/
Supports
- Generator CPU and memory limits
- Generator sizing and distributed execution
- Need to confirm that intended load was delivered
- https://jmeter.apache.org/usermanual/best-practices.html
Supports
- Generator sizing and non-GUI execution
- Parameterized data and distributed testing controls
- Coordinated omission warning
- https://jmeter.apache.org/
Supports
- Apache JMeter protocol-level load and performance testing
- Open-source licensing and supported protocol breadth
- Difference between protocol testing and browser rendering
- https://docs.gatling.io/
Supports
- Code-driven performance test scenarios
- Supported Java, JavaScript, TypeScript, Scala, and Kotlin SDKs
- Community and enterprise editions
- https://docs.locust.io/en/stable/
Supports
- Python-authored user behavior
- Command-line, web, and distributed execution
- Open-source load testing
- https://www.artillery.io/docs/get-started/load-testing
Supports
- JavaScript and TypeScript service and browser load testing
- Arrival ramps, open and closed models, and distributed execution
- https://www.w3.org/TR/navigation-timing-2/
Supports
- Browser navigation timing measurements
- High-resolution performance timeline entries
- https://github.com/sindresorhus/awesome
Supports
- Starting index used for required awesome-list discovery
- https://github.com/TheJambo/awesome-testing
Supports
- Discovery path to Awesome Performance Engineering and related testing resources
- https://github.com/be-next/awesome-performance-engineering
Supports
- Curated discovery of wrk2, Vegeta, Fortio, Hyperfoil, sitespeed.io, and WebPageTest
- Performance testing and observability ecosystem categorization
- https://github.com/giltene/wrk2
Supports
- Constant-throughput HTTP workload generation
- Latency recording designed to correct coordinated omission
- https://github.com/tsenart/vegeta
Supports
- Constant-rate HTTP load generation
- Reports and plotting for latency results
- https://fortio.org/
Supports
- Fixed-rate HTTP and gRPC load testing
- Latency histograms and command-line, server, and Kubernetes operation
- https://hyperfoil.io/
Supports
- Distributed benchmark phases and controlled request rates
- Latency collection for high-scale service tests
- https://www.sitespeed.io/documentation/
Supports
- Repeatable browser performance runs
- Performance budgets and time-series output
- https://docs.webpagetest.org/
Supports
- Browser tests from selected locations
- Waterfalls, filmstrips, and trace-based analysis
- https://docs.tricentis.com/neoload-latest/en-us/content/get_started/neoload_overview.htm
Supports
- Recorded interactions, virtual users, distributed execution, monitoring, and analysis
- Protocol and browser performance testing
- https://www.opentext.com/uk/products/professional-performance-engineering
Supports
- LoadRunner Professional for scalable enterprise performance testing
- Broad application and protocol coverage
- https://learn.microsoft.com/en-us/azure/app-testing/load-testing/overview-what-is-azure-load-testing
Supports
- Managed high-scale load generation
- Apache JMeter and Locust script execution
- Azure Monitor integration and automated fail criteria
- https://www.spec.org/30th/timeline.html
Supports
- SPEC incorporation on 14 November 1988
- Introduction of SPECmark and standardized comparative performance results
- https://www.tpc.org/information/about/history5.asp
Supports
- TPC formation on 10 August 1988
- TPC-A publication in November 1989 and benchmark review rules
- https://jmeter.apache.org/changes_history.html
Supports
- Apache JMeter 1.0 initial release on 15 December 1998
- https://jmeter.apache.org/usermanual/history_future.html
Supports
- JMeter becoming an Apache top-level project in November 2011
- https://www.w3.org/standards/history/navigation-timing/
Supports
- Navigation Timing becoming a W3C Recommendation on 17 December 2012
- https://grafana.com/blog/guest-blog-post-how-the-k6-load-testing-tool-is-leveraging-grafana/
Supports
- Open-source k6 introduction in February 2017
- Tests as JavaScript code and automation-oriented command-line operation
- https://grafana.com/blog/grafana-k6-one-year-later-lessons-learned-after-an-acquisition/
Supports
- Grafana Labs acquisition of k6 in June 2021
- https://grafana.com/docs/k6/latest/testing-guides/calculate-concurrent-users/
Supports
- Converting session or arrival rate and duration into a concurrent virtual-user count
- Sizing load from peak-hour traffic rather than long-run averages
- https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE/m/BfDekfBEs_sJ
Supports
- Coordinated omission definition and mechanism in a closed-loop load generator
- Documented case where reported latency was optimistic by roughly 35,000 times
- True 99.99th percentile of at least 582 milliseconds in that case
- Coordinated omission is a load-tester measurement problem, not a system-under-test problem
- https://qconlondon.com/london2018/london-2013/qconlondon.com/london-2013/presentation/How%20NOT%20to%20Measure%20Latency.html
Supports
- Strongly multi-modal nature of latency distributions
- Fallacy of using standard deviation for latency
- Back pressure and coordinated omission can skew results by orders of magnitude
- Reporting the full percentile spectrum out to the maximum
- https://www.youtube.com/watch?v=lJ8ydIuPFeU
Supports
- Recorded How NOT to Measure Latency talk on latency measurement and reporting pitfalls
- https://batey.info/percentiles-averages.html
Supports
- Averaging or aggregating precomputed percentiles is not a percentile of any population
- Worked example: 90th percentiles of 19 ms and 99 ms average to 59 ms while the true combined 90th percentile is about 98 ms
- Aggregate from raw response times or from histograms that preserve the distribution
