Google logo
4.9 on Google
Get a free Magento audit

Recommendations on page speed, accessibility issues, and real user experience data

Skip to main article

Magento 2 multi-store wasn’t caching (and how we fixed it)

How we traced cookie-free cache misses across nine country stores, fixed the Varnish configuration and cut sampled backend requests by about 70%.

Magento 2 multi-store wasn’t caching (and how we fixed it)
7 mins

Founder of Develo the Specialist Mage-OS & Hyvä agency. With over 20 years in web development experience I've helped deliver projects both small and large. Now helping clients get value from A.I development and design.

Develo is a leading Magento agency and eCommerce web development company based in Birmingham, in the UK, serving clients globally since 2010.

A Magento 2 website we look after was running under high CPU load, the site serves nine countries from a single install, each country on its own domain, with Varnish in front.

On inspection Varnish’s hit ratio, excluding passes, was only around 26%. The main domain store was fine, but  requests on the other country stores kept reaching PHP.

So we found out why, then fixed the Varnish configuration and picked up some quick wins along the way. After these changes the backend requests were about 70% lower.

Finding the cause

We worked from the outside in, ruling out one layer at a time.

  1. CPU. top showed php-fpm and MySQL doing the work, with Varnish idle. Varnish wasn’t absorbing enough of the stores traffic.

  2. Varnish hit rate. varnishstat and a sample from varnishncsa showed hits, misses and passes. Most passes were static files, which nginx serves without PHP. 

  3. Compare stores. We requested each store’s homepage straight from Varnish, twice, the second request should show cache HIT:

    Bash · test each store through Varnish
    for i in 1 2; do
      curl -sI -H 'Host: www.example.fr' -H 'X-Forwarded-Proto: https' \
        http://127.0.0.1:6081/ | grep -i x-magento-cache-debug
    done

    The main store returned HIT. Every other store returned MISS, both times. 

  4. What Magento sends back. For the main store, Magento’s response said “cache me” (Cache-Control: public, s-maxage=…). For the other stores it said no-cache, no-store, and set an X-Magento-Vary cookie. 

  5. Ruling out the usual suspects. There was no cacheable="false" block on shared layouts, no custom code on the other stores forcing no-cache headers, and no theme or config difference that explained the behaviour.

  6. Test the cookie. We replayed the same request with the X-Magento-Vary cookie the store had set. We got MISS, then HIT, then HIT. These pages could be cached, so the requests without the cookie were clearly the problem.

Why it happens

In this installation, the country stores produced a non-default HTTP context, and as Magento represents page variations such as store, currency and customer group through X-Magento-Vary, this is what Varnish uses in its cache key. The exact defaults depend on how each storefront is selected.

Magento’s standard VCL prevents a response being cached when the origin sets a vary cookie that the request did not supply. That stops the page being stored under the wrong key. 

On our affected stores the requests that did not retain the cookie and so kept reaching PHP. That included much of the Bot crawler and link-preview traffic. We also saw misses on ESI fragments such as the menu.

So real shoppers got cached pages from their second click onwards, but Bots arriving without the cookie continued to miss on every request and on this site bots were a large share of the traffic!

The fix

The principle is to add the store’s known default vary value before the cache lookup for eligible anonymous requests. This depends on the domain selecting a known default storefront context as a missing vary cookie alone does not establish that context. Existing shopper, store and currency selections must stay separate, as Adobe’s page-variation guidance explains.

To collect a store’s default value, request its homepage directly from the Magento origin, bypassing Varnish, with no cookies and the same Host and trusted proxy headers used in production. We then read X-Magento-Vary from the origin’s Set-Cookie header. 

Bash · inspect the Magento origin response
# Replace ORIGIN_HTTP_PORT with the Magento origin port, not Varnish.
curl -sS -D - -o /dev/null \
  -H 'Host: www.example.fr' \
  -H 'X-Forwarded-Proto: https' \
  'http://127.0.0.1:ORIGIN_HTTP_PORT/'

The illustrative block below limits injection to cookie-free, unauthenticated GET and HEAD requests on explicitly mapped domains. It leaves cookie-bearing requests unchanged.

VCL · anonymous default-context example
sub vcl_recv {
    std.collect(req.http.Cookie);

    if ((req.method == "GET" || req.method == "HEAD")
        && !req.http.Authorization
        && !req.http.Cookie
        && req.url !~ "^/(graphql|rest|customer|checkout)(/|[?]|$)") {
        if (req.http.host == "www.example.fr") {
            set req.http.Cookie =
                "X-Magento-Vary=<verified-fr-anonymous-default-hash>";
        } elsif (req.http.host == "www.example.de") {
            set req.http.Cookie =
                "X-Magento-Vary=<verified-de-anonymous-default-hash>";
        }
    }
}

The standard VCL already imports std. Varnish concatenates repeated subroutine definitions in source order, so this block runs first and falls through to the normal request handling. Keep Magento’s existing bypass and cache-key logic intact.

Cacheable fragments using the same verified context can benefit, but their hit/miss behaviour needs checking separately.

After testing the configuration changes, we load it and keep the old version ready to switch back to:

Bash · activate the tested VCL and retain a rollback
varnishadm vcl.load vary_fix /path/to/new.vcl
varnishadm vcl.use vary_fix
# Rollback: varnishadm vcl.use <previous_name>

When a request reaches Magento, a mismatched vary value causes the response to be marked uncacheable. But now a cache hit bypasses that check, so this is not an automatic safeguard against stale mappings. 

Refresh the hashes and invalidate affected cached pages when relevant store defaults or routing change. This follows from Magento’s origin-response check.

Other quick wins

Once the cache was working, the remaining load was easier to see.

Strip modern tracking parameters

The site’s older VCL only stripped utm_*, gclid and fbclid. Parameters such as _gl and srsltid were creating unnecessary cache-key variations. We added _gl, srsltid, gbraid, wbraid, gad_source, dclid and msclkid to the list. We also changed the value pattern to [^&]*, because _gl values contain *, which the old pattern didn’t match.

The visitor’s browser URL stays unchanged, so client-side analytics can still read it. Server-side attribution needs separate checking when parameters are removed upstream.

Redirect /favicon.ico/

Requests for the favicon with a trailing slash were going to Magento and rendering a full 404 page. A one-line nginx rule fixed that:

nginx · avoid a Magento 404 for the favicon
location = /favicon.ico/ { return 301 /favicon.ico; }

Add two missing MySQL indexes

A 30-second sample of the MySQL process list helped identify two recurring queries. Their row counts showed the cost:

  • The search indexer’s queue runner adds up sales per product from sales_order_item. With no index on product_id, each run read all 1.36 million rows. A covering index on (product_id, qty_ordered, row_total) cut that to about 2,000.
  • An email marketing integration looks up customer spend by customer_email on sales_order, which also had no index. Adding one cut 275,000 rows to 2.

Both indexes were added online with ALGORITHM=INPLACE, LOCK=NONE, so checkout kept running. Declare them in a module’s db_schema.xml so deploys keep them.

Cache static files at Cloudflare

Cloudflare doesn’t cache HTML or JSON by default, and Magento serves its Knockout templates and js-translation.json as exactly those. A crawler running a real browser was requesting them hundreds of times. A Cache Rule on /static/* with a one-month edge TTL moved them to the edge.

That long TTL depends on versioned static URLs: static content signing must be enabled and deployments must update the version.

Results

These figures come from 60-second samples before and after the changes. They are short operational snapshots, not a controlled benchmark; traffic mix and cache warmth may differ.

Before and after the combined changes
MetricBeforeAfter
Varnish hit ratiohits / (hits + misses)~26%78%
Requests reaching the backend~11.5/s3.1/s
CPU idle67%82%
Slowest recurring MySQL queryFull scan of 1.36M rows~2,000 rows from an index

Takeaways

  • Test every store, not just the default. A healthy hit rate on the main domain can hide the fact that the other stores aren’t cached.
  • Test without cookies. Your browser already has them; requests from crawlers and link previews may not. curl with no cookie shows that path.
  • Compare origin headers. The difference between a working page and a broken one pointed to the cause faster than reading config.
  • Look at the load that’s left. The tracking parameters, favicon redirect and missing indexes became clearer once PHP had quietened down.
  • Keep the VCL in version control. Record any hashes it depends on, re-check them after store or currency changes, and invalidate affected cached pages.Af

Last updated: November 22, 2026

Need help with Magento performance?

Learn with us

Latest learning resources

Practical ideas, helpful platform tips, and smart strategies to help your eCommerce store thrive.

Loading articles...