November 21, 2026
A Magento 2 website we look after was running under high CPU load, the site serves nine countries from a single install, each country on its own domain, with Varnish in front.
On inspection Varnish’s hit ratio, excluding passes, was only around 26%. The main domain store was fine, but requests on the other country stores kept reaching PHP.
So we found out why, then fixed the Varnish configuration and picked up some quick wins along the way. After these changes the backend requests were about 70% lower.
Finding the cause
We worked from the outside in, ruling out one layer at a time.
CPU.
topshowedphp-fpmand MySQL doing the work, with Varnish idle. Varnish wasn’t absorbing enough of the stores traffic.Varnish hit rate.
varnishstatand a sample fromvarnishncsashowed hits, misses and passes. Most passes were static files, which nginx serves without PHP.Compare stores. We requested each store’s homepage straight from Varnish, twice, the second request should show cache HIT:
Bash · test each store through Varnish for i in 1 2; do curl -sI -H 'Host: www.example.fr' -H 'X-Forwarded-Proto: https' \ http://127.0.0.1:6081/ | grep -i x-magento-cache-debug doneThe main store returned
HIT. Every other store returnedMISS, both times.What Magento sends back. For the main store, Magento’s response said “cache me” (
Cache-Control: public, s-maxage=…). For the other stores it saidno-cache, no-store, and set anX-Magento-Varycookie.Ruling out the usual suspects. There was no
cacheable="false"block on shared layouts, no custom code on the other stores forcing no-cache headers, and no theme or config difference that explained the behaviour.Test the cookie. We replayed the same request with the
X-Magento-Varycookie the store had set. We gotMISS, thenHIT, thenHIT. These pages could be cached, so the requests without the cookie were clearly the problem.
Why it happens
In this installation, the country stores produced a non-default HTTP context, and as Magento represents page variations such as store, currency and customer group through X-Magento-Vary, this is what Varnish uses in its cache key. The exact defaults depend on how each storefront is selected.
Magento’s standard VCL prevents a response being cached when the origin sets a vary cookie that the request did not supply. That stops the page being stored under the wrong key.
On our affected stores the requests that did not retain the cookie and so kept reaching PHP. That included much of the Bot crawler and link-preview traffic. We also saw misses on ESI fragments such as the menu.
So real shoppers got cached pages from their second click onwards, but Bots arriving without the cookie continued to miss on every request and on this site bots were a large share of the traffic!
The fix
The principle is to add the store’s known default vary value before the cache lookup for eligible anonymous requests. This depends on the domain selecting a known default storefront context as a missing vary cookie alone does not establish that context. Existing shopper, store and currency selections must stay separate, as Adobe’s page-variation guidance explains.
To collect a store’s default value, request its homepage directly from the Magento origin, bypassing Varnish, with no cookies and the same Host and trusted proxy headers used in production. We then read X-Magento-Vary from the origin’s Set-Cookie header.
# Replace ORIGIN_HTTP_PORT with the Magento origin port, not Varnish.
curl -sS -D - -o /dev/null \
-H 'Host: www.example.fr' \
-H 'X-Forwarded-Proto: https' \
'http://127.0.0.1:ORIGIN_HTTP_PORT/'The illustrative block below limits injection to cookie-free, unauthenticated GET and HEAD requests on explicitly mapped domains. It leaves cookie-bearing requests unchanged.
sub vcl_recv {
std.collect(req.http.Cookie);
if ((req.method == "GET" || req.method == "HEAD")
&& !req.http.Authorization
&& !req.http.Cookie
&& req.url !~ "^/(graphql|rest|customer|checkout)(/|[?]|$)") {
if (req.http.host == "www.example.fr") {
set req.http.Cookie =
"X-Magento-Vary=<verified-fr-anonymous-default-hash>";
} elsif (req.http.host == "www.example.de") {
set req.http.Cookie =
"X-Magento-Vary=<verified-de-anonymous-default-hash>";
}
}
}The standard VCL already imports std. Varnish concatenates repeated subroutine definitions in source order, so this block runs first and falls through to the normal request handling. Keep Magento’s existing bypass and cache-key logic intact.
Cacheable fragments using the same verified context can benefit, but their hit/miss behaviour needs checking separately.
After testing the configuration changes, we load it and keep the old version ready to switch back to:
varnishadm vcl.load vary_fix /path/to/new.vcl
varnishadm vcl.use vary_fix
# Rollback: varnishadm vcl.use <previous_name>When a request reaches Magento, a mismatched vary value causes the response to be marked uncacheable. But now a cache hit bypasses that check, so this is not an automatic safeguard against stale mappings.
Refresh the hashes and invalidate affected cached pages when relevant store defaults or routing change. This follows from Magento’s origin-response check.
Other quick wins
Once the cache was working, the remaining load was easier to see.
Strip modern tracking parameters
The site’s older VCL only stripped utm_*, gclid and fbclid. Parameters such as _gl and srsltid were creating unnecessary cache-key variations. We added _gl, srsltid, gbraid, wbraid, gad_source, dclid and msclkid to the list. We also changed the value pattern to [^&]*, because _gl values contain *, which the old pattern didn’t match.
The visitor’s browser URL stays unchanged, so client-side analytics can still read it. Server-side attribution needs separate checking when parameters are removed upstream.
Redirect /favicon.ico/
Requests for the favicon with a trailing slash were going to Magento and rendering a full 404 page. A one-line nginx rule fixed that:
location = /favicon.ico/ { return 301 /favicon.ico; }Add two missing MySQL indexes
A 30-second sample of the MySQL process list helped identify two recurring queries. Their row counts showed the cost:
- The search indexer’s queue runner adds up sales per product from
sales_order_item. With no index onproduct_id, each run read all 1.36 million rows. A covering index on(product_id, qty_ordered, row_total)cut that to about 2,000. - An email marketing integration looks up customer spend by
customer_emailonsales_order, which also had no index. Adding one cut 275,000 rows to 2.
Both indexes were added online with ALGORITHM=INPLACE, LOCK=NONE, so checkout kept running. Declare them in a module’s db_schema.xml so deploys keep them.
Cache static files at Cloudflare
Cloudflare doesn’t cache HTML or JSON by default, and Magento serves its Knockout templates and js-translation.json as exactly those. A crawler running a real browser was requesting them hundreds of times. A Cache Rule on /static/* with a one-month edge TTL moved them to the edge.
That long TTL depends on versioned static URLs: static content signing must be enabled and deployments must update the version.
Results
These figures come from 60-second samples before and after the changes. They are short operational snapshots, not a controlled benchmark; traffic mix and cache warmth may differ.
| Metric | Before | After |
|---|---|---|
| Varnish hit ratiohits / (hits + misses) | ~26% | 78% |
| Requests reaching the backend | ~11.5/s | 3.1/s |
| CPU idle | 67% | 82% |
| Slowest recurring MySQL query | Full scan of 1.36M rows | ~2,000 rows from an index |
Takeaways
- Test every store, not just the default. A healthy hit rate on the main domain can hide the fact that the other stores aren’t cached.
- Test without cookies. Your browser already has them; requests from crawlers and link previews may not.
curlwith no cookie shows that path. - Compare origin headers. The difference between a working page and a broken one pointed to the cause faster than reading config.
- Look at the load that’s left. The tracking parameters, favicon redirect and missing indexes became clearer once PHP had quietened down.
- Keep the VCL in version control. Record any hashes it depends on, re-check them after store or currency changes, and invalidate affected cached pages.Af
Last updated: November 22, 2026
