High-Concurrency Load & Stress Testing Services
Before go-live, step-load and full-journey mixed scenarios establish the maximum user load and the performance knee. A server performance monitoring stack is deployed during the test, and we deliver a module-level defect remediation recommendation, an API test report and a capacity plan with scaling and rate-limit advice.
High-Concurrency Load & Stress Testing Services
Overview
High-Concurrency Load & Stress Testing Services is built for IT business systems about to go live, undergo a major release, or face a traffic surge. We turn the vague question "how many users can this system actually handle" into a measurable, reproducible answer. In an isolated environment that mirrors production as closely as possible, we model full-journey mixed scenarios at real business ratios, ramp load up in steps from low concurrency and locate the performance knee. During the test we deploy a server performance monitoring stack, aligning CPU, memory, disk I/O, network and connection-pool metrics with response time, throughput and error rate on a single timeline. We deliver the maximum supported user load, the headroom of compute resources and software, an assessment of how much traffic pressure the system can absorb, plus two documents ready for remediation and acceptance: a Defect Remediation Recommendation by Business Module and an API Test Report. Go-live decisions are then based on evidence, not on estimates or "we will see after launch".
Test Types and Scenario Design
| Test type | How load is applied | Question it answers | Typical pass criteria |
|---|---|---|---|
| Single-endpoint baseline | One virtual user iterating repeatedly; take the steady-state value | Unloaded response time and TPS baseline per endpoint | A reference point for every later comparison |
| Step load test | Start at roughly 30% of the forecast peak, increase 10–20% per step, hold each step 3–5 minutes | How response time, throughput and resource levels change at each concurrency tier | P95 / P99 within limits, acceptable error rate, no sustained resource saturation |
| Capacity knee detection | Keep increasing load until throughput no longer grows linearly | Maximum effective concurrency and optimal concurrency | Throughput, latency and error knee signals (see below) |
| Stress / destructive test | Push past the knee until errors, cascading failure or outage occur | Failure mode, graceful degradation and recovery behaviour | Clear breaking point, protective mechanisms, recovery without manual rebuild |
| Peak burst test | Release high concurrency instantly with synchronisation points, simulating flash sales and on-the-hour rush | Queuing, overselling and rate limiting under an instantaneous spike | No dirty data, no duplicate submissions, core journey remains usable |
| Soak / endurance test | Hold 70–80% of target load for several hours up to 24 hours | Whether memory leaks, connection leaks, file handles and slow queries degrade over time | Stable metrics with no progressive deterioration |
| Full-journey mixed scenario | Combine endpoints at real business ratios with realistic think time | True capacity when all components work together | Scenario mix matches production traffic, without inflated load |
| Elasticity and degradation | Trigger HPA scale-out, rate limiting, circuit breaking and degradation while under load | Whether compute and software have real headroom and scaling reacts in time | Error rate and latency stay controlled during scale-out; scale-in causes no damage |
A script where every virtual user submits an order at once does not produce a valid conclusion. Real users browse, search and hesitate. Business ratios and think time are essential, otherwise connection pools are exhausted instantly and caches are never exercised.
Core Metrics
| Dimension | Key metrics | Why it matters |
|---|---|---|
| Business capacity | Concurrent users, active sessions, QPS / TPS | Directly answers "what is the maximum user load" |
| Response experience | Average response time, P50 / P95 / P99, tail latency | Averages hide tail degradation; percentiles reflect real user experience |
| Throughput | TPS / QPS curve and its slope as concurrency grows | Shows whether the system is still doing useful work and when it saturates |
| Correctness | Error rate, timeouts, 5xx and connection refusals, business success rate, data consistency | A red-line metric: crossing it fails that concurrency tier |
| Server resources | CPU (including sys and iowait), memory and swap, disk IOPS / await / util, NIC bandwidth, connection counts | Finds the first saturated bottleneck and separates capacity shortage from design defects |
| Application and middleware | Thread pools and queues, connection-pool usage, JVM GC and heap, Nginx workers, Redis hit rate, MQ backlog | Explains the common case of "CPU is idle but the system is slow" |
| Database | Slow queries, lock waits and deadlocks, connection count, buffer-pool hit rate, replication lag | Most high-concurrency bottlenecks eventually land in the data layer |
Pass criteria are agreed with the client and written into the plan before testing starts — for example "at target concurrency, P99 of core endpoints stays below one second, error rate below 0.1%, peak CPU sustained below 80%" — so acceptance is never settled by a vague "looks fine".
Finding the Knee and Converting It Into User Capacity
| Knee type | Signal | What it means |
|---|---|---|
| Throughput knee | A 20% concurrency increase yields less than half that growth in TPS; the curve flattens | The system is saturating; extra users only queue instead of producing work |
| Latency knee | P95 / P99 jump sharply beyond the business tolerance threshold | Queues are building; user experience collapses before the system does |
| Error knee | Sustained timeouts, 5xx or connection refusals push the error rate past the red line | The trust boundary is crossed; stop the load and protect test data |
We take the last concurrency tier before the knee as the maximum effective concurrency, then convert it to production capacity using "concurrency ≈ TPS × response time in seconds", apply a peak-to-average factor and a 1.5–2× safety margin, and derive recommended server sizes, instance counts and gateway rate limits. The conclusion is written as an actionable statement, for example "the current architecture sustains 4,000 concurrent users; provision for 6,000 and set ingress rate limiting at 1.2× peak TPS". After any scaling or optimisation we rerun the identical scenarios to confirm the bottleneck actually moved rather than being masked by bigger machines.
Server Performance Monitoring Deployed During the Test
| Layer | What we deploy | Role in the load test |
|---|---|---|
| System collection | Prometheus with node_exporter on every target host, plus nmon, sar, iostat -x, pidstat, PSI and eBPF sampling at second-level resolution | Leaves recoverable raw curves for the incident window instead of five-minute averages |
| Application and data | JVM / thread pools / connection pools, Nginx and gateways, Redis and MQ, database slow queries and lock metrics | Traces "the business is slow" down to a specific module and statement |
| Visualisation | Unified Grafana dashboards overlaying concurrency, TPS, P95 / P99, error rate and resource levels, aligned with the load-test timeline | Shows at a glance which load step and which layer saturated first |
| Alerting and stop protection | Thresholds derived from business tolerance; crossing an error-rate or resource threshold raises an alert and halts the load | Prevents dirty data and cascading failures, keeping the conclusion trustworthy |
| Load-generator self-monitoring | CPU, ports, network and GC of the JMeter / k6 / Locust injectors, scaled out to distributed load when needed | Ensures the bottleneck is not the test rig and the applied load is real |
The dashboards, collector configuration and raw metrics are handed over with the report. They remain usable as a production monitoring baseline and a source of alert thresholds after the engagement ends — nothing is torn down when the test finishes.
Deliverables: Remediation Recommendations and API Test Report
| Document | Contents | Primary audience |
|---|---|---|
| Defect Remediation Recommendation by Business Module | Per module and endpoint: observed defect, impact scope, severity, root cause, remediation advice, priority, suggested owner and re-test verification items | Development and product teams scheduling and tracking fixes |
| API Test Report | Per endpoint: scenario and concurrency tiers, TPS / QPS, P50 / P95 / P99, error rate, assertion results, throughput curves and SLA verdict | QA, project acceptance and delivery archives |
| Capacity Assessment and Go-Live Readiness Report | Maximum user load, knee concurrency, compute and software headroom, scaling and rate-limit recommendations, go-live risks and release gates | Operations, architecture and management making the launch decision |
- Server performance dashboards and complete raw metrics, with reusable collector configuration and alert rules.
- Load-test scripts, business scenario models, parameterised datasets and execution records so the client can rerun everything independently.
- A performance baseline library comparing this run with previous runs to detect regressions between releases.
- Concrete configuration advice and change steps for scaling, rate limiting, circuit breaking, connection pools and caching.
- Post-remediation re-test records, a go-live verdict and knowledge transfer for the operations team.
Both core documents are organised by module and endpoint ID, so developers can turn remediation items into tickets and acceptance teams can cite percentiles and verdicts directly, with no reformatting required.
Engagement Process
- Step 1 · Requirements and business model: agree core journeys, forecast peak, SLA red lines, acceptable response time and error rate, data volume and growth expectations, and sign off the pass criteria.
- Step 2 · Environment and data: build an isolated test environment equivalent to production, prepare masked parameterised datasets, and confirm no load is applied to production or shared staging.
- Step 3 · Monitoring and scripting: deploy the server performance monitoring stack and Grafana dashboards, build scripts with think time, parameterisation and assertions, and calibrate the load generators.
- Step 4 · Staged execution: baseline → step load → knee detection → peak burst → soak → elasticity and degradation, retaining metrics and logs per tier and halting immediately on anomalies.
- Step 5 · Bottleneck analysis: align the monitoring timeline, distinguish capacity shortages from design defects, and issue module-level remediation recommendations with an optimisation plan.
- Step 6 · Re-test and verdict: rerun the identical scenarios after remediation, deliver the API Test Report and capacity conclusion, and hand over the monitoring baseline, scaling configuration and knowledge transfer.
Technology Stack and Tooling
| Layer | Coverage |
|---|---|
| Protocols and ingress | HTTP / HTTPS, RESTful APIs, GraphQL, gRPC, WebSocket, long-lived TCP connections, file upload and download |
| Application architecture | Monoliths and microservices, Java / Spring Boot, Python, Node.js, PHP, plus Nginx, API gateways and load balancers |
| Data and middleware | MySQL, PostgreSQL, MariaDB, Redis, MongoDB, Kafka / RabbitMQ, Elasticsearch |
| Infrastructure | Bare metal, cloud instances, KVM / VMware, Kubernetes and containers, HPA autoscaling, and domestic platforms such as Galaxy Kylin, UnionTech UOS and openEuler |
| Load-testing tools | Apache JMeter, k6, Locust, Gatling, wrk / wrk2 — selected per scenario, with distributed injection and CI/CD pipeline integration |
Tools are a means, not the point: JMeter fits full business flows and distributed load, k6 integrates cleanly into pipelines, Locust defines user behaviour in code, Gatling produces detailed reports, and wrk drives a single endpoint to its limit. We choose by protocol, scenario complexity and reporting needs.
Representative Scenarios
The following are representative patterns grouped by common launch situations, illustrating test design and the form of the conclusion. They are not descriptions of specific customer engagements.
Scenario 1 · Full-journey capacity assessment before an e-commerce peak
- Challenge: The campaign peak is expected at three times normal traffic, and nobody can state how many users the current configuration supports.
- Approach: Mixed scenarios at real ratios — 60% browse, 20% search, 15% add-to-cart, 5% checkout — with 3–5 seconds of think time, followed by a flash-sale burst.
- Finding: At around 800 concurrent users the checkout endpoint exhausted the database connection pool; P99 jumped from 400 ms to 3 s while application CPU still had headroom.
- Outcome: After pool and slow-query fixes the knee moved significantly; the report gave maximum effective concurrency, recommended instance count and ingress rate limits as release gates.
Scenario 2 · Concentrated submissions on a public-service portal
- Challenge: Traffic concentrates the moment a submission window opens, with no historical baseline, and production may not be load-tested directly.
- Approach: Step load in an isolated environment to find the knee, then a 12-hour soak at 80% of target load with system- and gateway-level metrics collected together.
- Finding: CPU was far from saturated, but Nginx worker connections and file descriptors hit their limits first, causing connection refusals in the later phase.
- Outcome: Worker and descriptor limits were tuned and static assets moved to a CDN; maximum concurrent submissions was documented in the contingency plan and rate-limit configuration.
Scenario 3 · Acceptance testing of core financial transaction APIs
- Challenge: The error-rate red line is extremely strict, no dirty data or duplicate charges are acceptable, and the report must serve as project acceptance evidence.
- Approach: Constant-throughput load plus endurance testing, with destructive tests for degradation paths and automatic stop rules on error-rate breaches.
- Finding: After several hours a connection leak and periodic JVM GC pauses produced sawtooth spikes in P99.
- Outcome: Connection release was fixed and heap / GC parameters tuned; a per-endpoint API Test Report was issued with non-compliant concurrency tiers explicitly flagged.
Scenario 4 · Verifying elasticity on a multi-tenant SaaS platform
- Challenge: The team could not tell whether adding instances actually helps, or whether autoscaling reacts fast enough.
- Approach: Trigger Kubernetes HPA scale-out and gateway rate limiting during step load, and compare throughput, P99 and error-rate curves before and after scaling.
- Finding: New pods received traffic before warming up, causing a brief error spike; a short scale-in cooldown caused repeated oscillation.
- Outcome: Readiness probes and warm-up were added, thresholds and cooldown retuned; the engagement closed with an elasticity headroom conclusion and pay-as-you-go cost advice.
Why Work With Us
- Nearly 20 years of Linux and open-source technical services supporting tens of thousands of enterprises, with joint bottleneck analysis across system, application and data layers rather than a single response-time curve.
- More than raw test data: remediation recommendations, an API Test Report and a capacity plan are delivered together to support development scheduling, acceptance and go-live approval.
- A server performance monitoring stack is deployed during the test, and the dashboards, collector configuration and raw data are handed over for use as a production baseline afterwards.
- Coverage of mainstream web / API, microservices, databases and middleware, with load-testing and tuning experience on Galaxy Kylin, UnionTech UOS and openEuler domestic platforms.
- Standardised process, isolated environments, strict stop protection, transparent pricing, and 24/7 support from senior engineers.