Dispatcher, Caching and CDN#

#19 – Dispatcher, Caching and CDN

Target: 3–4 years experienced AEM Developer Supplementary topic 1 of the additional list: "Dispatcher in depth — dispatcher configuration, dispatcher caching, dispatcher flush, Apache, and CDN." Project domain: a global energy technology company's marketing site.

#Before we start — this is the topic your support experience is worth the most on

Every file so far has asked you to talk like a developer about things you may have only watched from the outside. This one is different.

If you have spent four years on AEM support tickets, you have already lived inside this file. "The page is published but the site shows the old version." "CSS is missing on production only." "The site went down when marketing published forty pages at once." "Someone can reach /system/console from the internet." Those are dispatcher tickets. You have raised them, escalated them, or watched somebody fix them.

The gap is not experience. The gap is vocabulary and mechanism — being able to say "the statfile was touched but the cached file underneath it had a newer timestamp" instead of "the cache didn't clear."

So this file is deliberately mechanism-first. By the end you should be able to explain, without hedging:

• Why the dispatcher deletes nothing when you flush it, and what it does instead.
• Why publishing one page can invalidate its siblings, and which single number controls that.
• Why ?page=2 costs you a cache hit and .html/2 does not.
• Which of the four caches — browser, CDN, dispatcher, publish — is actually serving a given request, and how to prove it in three commands.

And there is one framing decision worth making now, because it changes how the whole answer lands.

Almost everyone introduces the dispatcher as "AEM's caching layer." That is the weaker half of the truth. Adobe's own security checklist treats the dispatcher primarily as the filtering layer that stands between the internet and your publish instances. A slow site is a bad day. An open /system/console on a publish tier is an incident with a customer notification attached to it.

Lead with security, then caching. It is a genuinely better answer, and almost nobody gives it.

Where this file plugs into the rest of the repository. File 01 established that the dispatcher denies by default. File 04 left you with allowProxy and an unfinished second cause for the same 404. File 07 built a cacheable endpoint with a selector and a suffix and told you why without showing you the cache. File 14 said the dispatcher is a container in the publish pod. File 18 sent an invalidation to /dispatcher/invalidate.cache and told you a statfile gets touched. This file is where all five of those threads terminate.

#1. Introduction

#1.1 The problem the dispatcher solves

Start with what a publish instance actually does when a page is requested.

It resolves the URL to a resource, checks permissions, reads the resource type, finds the script, runs your Sling Models, executes the HTL, resolves every included component, renders the markup, and writes the response. On a product listing page that walks a page tree, that is a meaningful amount of CPU and a meaningful number of repository reads.

Now ask: how often does the answer actually change?

For a marketing page on an energy technology site — a transformer product page, an HVDC case study, a grid automation overview — the answer changes when an author publishes. Which might be once a week. Possibly once a quarter.

So you have a system doing expensive work to produce an identical answer thousands of times a day. That is the problem.

The obvious fix is to compute it once and keep the result. And because the result is HTML — a document identified by a path — the natural place to keep it is as a file on a disk, laid out in directories that mirror the URL.

That is the dispatcher's cache in one sentence. It is not a clever in-memory data structure. It is a folder full of HTML files, arranged so the URL is the file path.

And once you see it that way, several things that seem arbitrary become obvious:

• Why query parameters are a problem — ?page=2 is not part of a file path.
• Why a hit is so cheap — Apache is just serving a static file, exactly as it would serve an image.
• Why invalidation is awkward — you are managing a directory tree, not a key-value store.

#1.2 What the dispatcher actually is — and the misconception to avoid

This is the single most common wrong answer on the topic, and it is worth being precise about.

The dispatcher is not a standalone product. It is not a server you install and run. It is not a proxy application with its own process.

The dispatcher is a module for a web server — in almost every real deployment, an Apache HTTP Server module called mod_dispatcher.

Concretely, on a traditional AEM 6.5 deployment you have Apache installed, and in its configuration there is a line like:

LoadModule dispatcher_module modules/mod_dispatcher.so

That is it. From that point Apache has a new request handler, configured by a file conventionally called dispatcher.any.

Why does this distinction matter enough to be the first thing you say?

One — it tells the interviewer you have seen a real deployment. Someone who has only read about the dispatcher describes it as a box on a diagram. Someone who has configured one knows they were editing httpd.conf and vhosts alongside it.

Two — it explains what the dispatcher can and cannot do. It inherits everything Apache can do: rewrites via mod_rewrite, headers via mod_headers, SSL, mod_deflate for gzip, access control by IP. Half of what people call "dispatcher configuration" is actually Apache configuration sitting next to it. If someone asks how you'd add a redirect or a security header, the answer is often "that's mod_rewrite or mod_headers, not the dispatcher module."

Three — it explains the cache. The dispatcher's docroot is normally the same directory as Apache's DocumentRoot. That is not a coincidence; it is the design. Once the dispatcher has written a cached file there, subsequent hits can be served by Apache as an ordinary static file. That is why a cache hit costs almost nothing.

A vendor note worth knowing but not leading with: the module was historically also available for Microsoft IIS. In practice, and universally on Cloud Service, it is Apache.

A note on the name. "Dispatcher" is a caching name that sounds like a routing name, which confuses people. It is called that because its original job was to dispatch requests across several render instances — load balancing. Caching came to dominate, and the name stuck.

#1.3 The three jobs

The dispatcher does three things. Interviewers expect all three, and they expect you to order them by real-world importance rather than by fame.

Job one — caching. Store rendered responses on disk and serve them without touching AEM. This is the famous one and it is what most of the configuration is about.

Job two — security filtering. Decide which requests are even allowed to reach a publish instance, and reject everything else at the web server, before AEM sees it. This is the one that matters most.

Job three — load balancing. Distribute requests across several render instances, take a failing one out of rotation, and keep a visitor pinned to the same instance when that is needed.

Now the argument for putting security first, which is the differentiating part of this answer:

A publish instance is a full AEM installation. It contains the OSGi web console at /system/console. It contains CRXDE at /crx/de. It contains the QueryBuilder JSON endpoint at /bin/querybuilder.json, which will happily run an arbitrary repository query and return the results. It contains the DefaultGetServlet from file 07, which will serialise a whole subtree if you ask for .infinity.json.

None of those are bugs. They are the tools that make AEM usable on author. They exist on publish because publish runs the same software.

So the only thing standing between the public internet and all of it is the dispatcher's filter section. Not AEM's permissions — those help, but a wrongly-configured content ACL plus an open filter is a bad combination. The filter is the layer designed for this job.

If the cache misconfigures, the site is slow. If the filter misconfigures, the site is breached.

That sentence is worth having ready. It reframes the whole topic and it is true.

#1.4 When and where the dispatcher sits

Between the internet (or the CDN) and the publish instances:

Visitor → CDN → Apache + dispatcher module → publish instance(s) → Oak repository

There is normally also a dispatcher in front of author, and its job is different. It does far less caching — authors need to see their changes immediately — and its filter is configured for authenticated users rather than the public. Its main roles are load balancing across author instances in a cluster and terminating SSL. A common interview question is whether you cache on author, and the answer is "very little, and never HTML, because an author must see the change they just made."

#1.5 A real project example to adapt

"We run about twenty country sites off one AEM instance, with URLs like /us/en/ and /de/de/. The dispatcher config is one farm per environment with a shared filter section that's included, so a security rule can't be right in one place and wrong in another. Our filter is deny-by-default with roughly a dozen allow rules, and the two we had to add deliberately were /etc.clientlibs for the clientlib proxy and one exact /bin path for a CRM form handler — one path, one method, not /bin/*.

On caching, our hit ratio sits above ninety percent on the country sites, and the work to get there was mostly about URLs — moving listing pagination from a query parameter to a selector and suffix, and getting statfileslevel right so a translator publishing a German page doesn't invalidate the US cache. We use a grace period so a bulk publish doesn't stampede publish, and we have a replication event handler that dispatches a Sling Job to invalidate derived pages that a normal flush wouldn't touch — the classic case being pages that include an Experience Fragment.

On Cloud Service the config lives in Git under dispatcher/src, we validate it with the dispatcher SDK validator before pushing, and a rule change goes out through the web-tier pipeline, which is dramatically faster than a full-stack deploy."

That paragraph pre-empts about eight follow-ups: filter strategy, the /bin rule, the /etc.clientlibs rule, hit ratio, statfileslevel, grace period, derived-content invalidation, and the AEMaaCS workflow. It is worth learning the shape of it even if your numbers differ.

#2. Core Concepts

#2.1 The Apache side — what lives where

Because the dispatcher is an Apache module, its configuration is split across two kinds of file, and mixing them up is a common source of confusion.

Apache configuration — httpd.conf, and the virtual host files. This is where you load the module, point it at its config, set logging, and declare which requests Apache hands to the dispatcher handler.

Dispatcher configuration — dispatcher.any and whatever it includes. This is where farms, filters, cache rules and renders live.

The Apache side looks roughly like this:

# Load the module -- this is what makes "the dispatcher" exist at all
LoadModule dispatcher_module modules/mod_dispatcher.so

<IfModule disp_apache2.c>
    # Point at the dispatcher's own configuration file
    DispatcherConfig       conf/dispatcher.any

    # The dispatcher writes its own log, separate from Apache's
    DispatcherLog          logs/dispatcher.log

    # 0 = error, 1 = warn, 2 = info, 3 = debug
    # Debug is your main diagnostic tool. It is very verbose --
    # turn it on to answer a question, then turn it off.
    DispatcherLogLevel     3

    # Don't advertise the server version in responses
    DispatcherNoServerHeader 1

    # Let Apache handle "/" itself rather than passing it to AEM
    DispatcherDeclineRoot  0

    # Use the URL AFTER Apache's rewrites, not the original one.
    # If you use mod_rewrite, you almost always want this on --
    # otherwise the dispatcher filters and caches the pre-rewrite URL.
    DispatcherUseProcessedURL 1

    # Pass AEM's error responses through instead of letting Apache
    # substitute its own error page
    DispatcherPassError    0
</IfModule>

And in the virtual host, the block that actually routes requests into the module:

<VirtualHost *:80>
    ServerName www.example.com

    # The dispatcher docroot and Apache's DocumentRoot are the SAME
    # directory. That is the point: once a file is cached, Apache can
    # serve it as an ordinary static file.
    DocumentRoot /opt/dispatcher/cache

    <Directory /opt/dispatcher/cache>
        <IfModule disp_apache2.c>
            # This is the line that sends requests to the dispatcher
            SetHandler dispatcher-handler
        </IfModule>
        Options FollowSymLinks
        AllowOverride None
        Require all granted
    </Directory>
</VirtualHost>

Two details in there are worth knowing by name, because they come up:

DispatcherUseProcessedURL. If you rewrite URLs with mod_rewrite — say, mapping /us/en/products to /content/energy/us/en/products.html — you need the dispatcher to filter and cache the rewritten URL, not the incoming one. With this off, your filter rules are being tested against a URL that no longer exists by the time AEM sees it. That mismatch produces filter rules that look correct and do nothing.

DispatcherLogLevel 3. Debug level is the single most useful diagnostic in this whole file. It logs each filter check and the cache decision for every request. It is also extremely noisy, so it is a "turn on, reproduce, turn off" tool, not a standing setting.

#2.2 dispatcher.any — the anatomy

Now the dispatcher's own file. It uses a bracketed property format that looks unusual the first time. Every key starts with a slash, values are quoted, and blocks are wrapped in braces.

The top level is always /farms:

/farms
{
  /energysite
  {
    /clientheaders  { ... }    # which request headers to forward to AEM
    /virtualhosts   { ... }    # which hostnames this farm handles
    /renders        { ... }    # the publish instances behind it
    /filter         { ... }    # which requests are ALLOWED at all
    /cache          { ... }    # what gets cached, where, and how it expires
    /statistics     { ... }    # request categories, used for load balancing
    /vanity_urls    { ... }    # let publish's vanity URLs through the filter
  }
}

A farm is a complete configuration for a set of hostnames. You have more than one when different sites need genuinely different behaviour — the classic split being an author farm and a publish farm, or a public site and a partner portal with session management.

How the dispatcher picks a farm: it matches the request's Host header against each farm's /virtualhosts. When more than one farm could match, ordering decides — the documented behaviour is that the last matching farm wins. That is why, in the Cloud Service layout where farm files are included alphabetically, people name them with numeric prefixes: the file name controls the include order, and the include order controls which farm answers.

Getting the farm wrong is a nasty class of bug, because everything looks configured — you edited a filter rule and it had no effect, because the request is being handled by a different farm than the one you edited.

The include mechanism. Real configurations do not put everything in one file. They use $include:

/filter {
    $include "/etc/httpd/conf.dispatcher.d/filters/security_filters.any"
    $include "/etc/httpd/conf.dispatcher.d/filters/energy_filters.any"
}

Why that matters beyond tidiness: a shared, included security filter can be reviewed once and applied to every farm. A copy-pasted one gets fixed in one farm and forgotten in another — and the forgotten one is the one that gets found by a penetration test. On our project the security filter is one included file and nobody is allowed to add allow rules to it without a review; project-specific allows go in the second include.

#2.3 /clientheaders — which headers reach AEM

The dispatcher does not forward every request header to the publish instance. It forwards the ones you list.

/clientheaders
{
  "CSRF-Token"
  "referer"
  "user-agent"
  "authorization"
  "from"
  "content-type"
  "content-length"
  "accept-charset"
  "accept-encoding"
  "accept-language"
  "accept"
  "host"
  "if-match"
  "if-none-match"
  "max-forwards"
  "cookie"
  "X-Forwarded-For"
  "*"
}

Most shipped configurations end with "*", which forwards everything — but you will meet locked-down configurations that do not.

The bug this causes, and it is a genuinely confusing one: you write a servlet that reads a custom header — say X-Partner-Key — it works perfectly on author and locally, and on publish the header is always null. The dispatcher dropped it, because it was not in /clientheaders. Your code is correct, your test is correct, and the header simply never arrived.

Two more things worth knowing here:

X-Forwarded-For is what carries the visitor's real IP through to AEM, because from the publish instance's point of view every request comes from the dispatcher. If anything in your application logs or geolocates by IP, it depends on this header being forwarded — and on Apache being configured to set it.

authorization and cookie being forwarded is what makes authenticated requests work at all, and it is directly connected to the caching rules in 2.8.

#2.4 /virtualhosts — which hostnames this farm answers for

/virtualhosts
{
  "www.example.com"
  "*.example.com"
  "*"
}

Entries are matched against the request's Host header, and more specific entries win over wildcards. A farm with only "*" is a catch-all, which is fine as the last farm and dangerous as the first.

The practical reason to care: on our twenty-country setup, one farm serves all of them because the country is a path segment (/us/en/, /de/de/) rather than a hostname. If a market moved to its own domain with different caching rules, that would become a second farm — and then the ordering question in 2.2 becomes live.

#2.5 /renders — load balancing and sticky connections

This is job three, and it is where the name "dispatcher" comes from.

/renders
{
  /rend01
  {
    /hostname "publish1.internal"
    /port     "4503"
    /timeout  "10000"     # milliseconds to wait for a response
  }
  /rend02
  {
    /hostname "publish2.internal"
    /port     "4503"
    /timeout  "10000"
  }
}

With more than one render, the dispatcher distributes requests between them. It is not a simple round-robin — the dispatcher tracks response times per render (this is what the /statistics section is for) and prefers the faster one, which means a struggling instance naturally receives less traffic.

Failure handling is configured at farm level:

/failover           "1"     # retry a failed request on another render
/numberOfRetries    "5"
/retryDelay         "1"
/unavailablePenalty "1"     # weight applied to a render that just failed
/health_check { /url "/system/health" }

/failover is the one worth knowing: with it on, if a render returns an error or does not answer, the dispatcher retries the request on a different render rather than passing the failure to the visitor. Combined with /cache/serveStaleOnError, that is most of your resilience story.

Sticky connections are the part that gets asked about.

Normally it does not matter which publish instance answers, because they are identical. It matters when the response depends on state held on that specific instance — most often a server-side session, for example a multi-step quote form that keeps partially-entered data between steps.

/stickyConnectionsFor "/content/energy/global/en/quote-request"

Or, for several paths:

/stickyConnections
{
  /paths
  {
    "/content/energy/global/en/quote-request"
    "/content/energy/global/en/account"
  }
}

How it works: the dispatcher sets a renderid cookie identifying which render served the request, and subsequent requests under a sticky path go back to the same one.

And here is the answer that separates candidates. Sticky connections are a workaround, not a design. They mean you have server-side state, which means:

• You cannot cache those paths, because responses are visitor-specific.
• If that render dies, the visitor loses their session.
• Scaling is worse, because load cannot be spread freely.

"I'd use sticky connections only where there's genuinely instance-local session state, and I'd treat needing them as a signal. The better design is a stateless publish tier — keep the state in the browser, or in an external system the form posts to. On Cloud Service that's not really optional anyway, because pods come and go, so anything that assumes a stable instance is going to fail eventually."

#2.6 /filter — the security section

This is the most important part of the file.

The filter decides, for every request, whether it is even allowed to proceed. A rejected request never reaches AEM at all — Apache returns a 404 and the publish instance is unaware anything happened.

The pattern is deny everything, then whitelist.

/filter
{
  # RULE ZERO. Deny everything. Every subsequent rule is an exception.
  /0001 { /type "deny" /url "*" }

  # Then, deliberately, allow what the site actually needs.
  ...
}

Why deny-by-default rather than blocking known-bad paths?

Because a blocklist is a list of things you thought of. Every AEM version adds endpoints. Every project adds servlets. Every package you install may register something. A blocklist ages badly and fails silently — the day it stops being complete, nothing tells you.

An allowlist fails loudly. If you forget to allow something, that feature breaks in testing and someone raises a ticket. A wrong allowlist gives you a bug report; a wrong blocklist gives you a breach. That asymmetry is the whole argument, and it is worth stating in exactly those terms.

#The elements you can filter on

Older configurations use /glob, which matches against the whole request line as one string. Adobe now recommends the granular elements instead, because a glob is easy to get subtly wrong and hard to read.

Every table needs its reason, and here it is: these elements are the vocabulary of every filter rule you will ever write or review, and knowing which one to reach for is most of the skill.

Note the /path versus /url distinction, because it is exactly the kind of thing that gets asked. /path sees only the resource path. /url sees the path plus selectors, extension and suffix. So a rule written against /path cannot express "block the infinity selector" — you need /selectors, or /url with a pattern. Getting this wrong produces a rule that appears to block something and does not.

Values can be regular expressions when wrapped in single quotes: /extension '(html|json)'.

#What must be blocked on publish

This is the list to be able to recite. Each entry has a why, and the why is what earns the mark.

/system/console and everything under /system. The OSGi web console. Install a bundle, change a configuration, read every OSGi service, view configurations that contain credentials. This is administrative control of the application, reachable over HTTP. Nothing on a public publish tier needs it.

/crx, and /crx/de specifically. CRXDE Lite — a repository browser and editor. Reads and writes any node the session can reach. /crx/packmgr is Package Manager: build a package of /content and download the whole site, or upload a package containing code.

/bin/querybuilder.json. This one is the favourite of anyone doing an AEM security assessment, and it deserves explaining properly rather than just listing. It accepts query parameters and executes a repository query, returning JSON. So a request like ?path=/home/users&p.limit=-1 walks the user tree. It is not a vulnerability — it is a documented API doing exactly what it says. The vulnerability is exposing it. Even where anonymous permissions limit what comes back, it is an information-disclosure surface with no upside on a public site.

.infinity.json. File 07's DefaultGetServlet serialising an entire subtree in one response. /content.infinity.json is the demonstration everybody uses because it is dramatic.

.tidy.json, .query.json, .sysview.xml, .docview.xml, .-1.json, .2.json. Variants of the same problem. tidy is pretty-printed JSON. Numeric selectors control depth, so .10.json walks ten levels down. sysview and docview are the JCR XML export formats. The right way to handle these is a single rule denying the selector set, not one rule per selector — that way a new variant does not need a new rule.

POST to content paths. Because of the SlingPostServlet from file 07. A POST to a repository path creates or updates nodes. Anonymous cannot normally write, but this depends entirely on your ACLs being right, and it is exactly the kind of thing that drifts. Blocking POST except where a form genuinely needs it removes the dependency.

/apps, /libs, /home, /etc broadly. /apps is your source code. /libs is Adobe's, including tools. /home is users and groups. /etc historically held everything. Each gets narrow, deliberate exceptions — the important one being /etc.clientlibs, which is section 2.14.

/dispatcher/invalidate.cache from outside. The flush endpoint. If the public can reach it, the public can flush your cache repeatedly, which turns every request into a cache miss and puts your entire traffic load onto publish. That is a denial-of-service you configured yourself. This is guarded by /cache/allowedClients as well, and both matter.

#What a real filter section looks like

/filter
{
  # ---------------------------------------------------------------
  # 1. DENY EVERYTHING. Every rule below is a deliberate exception.
  # ---------------------------------------------------------------
  /0001 { /type "deny" /url "*" }

  # ---------------------------------------------------------------
  # 2. Allow the methods and extensions a public site actually uses
  # ---------------------------------------------------------------
  /0010 {
    /type "allow"
    /method '(GET|HEAD)'
    /extension '(css|eot|gif|ico|jpe?g|js|gif|pdf|png|svg|swf|ttf|woff2?|html)'
  }

  # The clientlib proxy from file 04. WITHOUT THIS, every stylesheet
  # and script on the published site returns 404 -- even when
  # allowProxy is set correctly on the clientlib.
  /0011 { /type "allow" /method "GET" /path "/etc.clientlibs" }

  # DAM assets referenced from pages
  /0012 { /type "allow" /method "GET" /path "/content/dam" }

  # ---------------------------------------------------------------
  # 3. Re-deny the dangerous things that the broad allows above
  #    would otherwise let through. Order matters: LAST MATCH WINS.
  # ---------------------------------------------------------------

  # Serialisation selectors -- one rule, not one per selector, so a
  # new variant doesn't need a new rule.
  /0030 {
    /type "deny"
    /selectors '(infinity|tidy|sysview|docview|query|childrenlist|ext|feed)'
    /extension '(json|xml|html|feed)'
  }

  # Numeric depth selectors: .1.json, .10.json, .-1.json
  /0031 { /type "deny" /selectors '([0-9-]+)' /extension "json" }

  # QueryBuilder: an arbitrary repository query over HTTP
  /0032 { /type "deny" /url "/bin/querybuilder.json*" }

  # Administrative surfaces
  /0033 { /type "deny" /path "/system" }
  /0034 { /type "deny" /path "/crx" }
  /0035 { /type "deny" /path "/bin" }
  /0036 { /type "deny" /path "/apps" }
  /0037 { /type "deny" /path "/libs" }
  /0038 { /type "deny" /path "/home" }
  /0039 { /type "deny" /path "/etc" }        # /etc.clientlibs is a
                                             # different path, not under /etc

  # The invalidation endpoint must never be reachable from outside
  /0040 { /type "deny" /url "/dispatcher/invalidate.cache" }

  # ---------------------------------------------------------------
  # 4. Narrow, deliberate exceptions -- each one justified in review
  # ---------------------------------------------------------------

  # The CRM quote form handler from file 07. ONE exact path, ONE
  # method. NEVER /bin/* -- that would re-expose querybuilder and
  # every other path-bound servlet in the system, including ones we
  # didn't write.
  /0100 { /type "allow" /method "POST" /path "/bin/energy/quote-request" }

  # The Sling Model Exporter JSON for the mobile app (file 17),
  # scoped to our content tree and one selector.
  /0101 {
    /type "allow"
    /method "GET"
    /path "/content/energy"
    /selectors "model"
    /extension "json"
  }
}

Three things to be able to defend about that file:

Last match wins. Rules are evaluated in order and the last one that matches decides. That is why the structure is broad-allow then re-deny: you cannot express "allow HTML but not .infinity.json " in one rule, so you allow the extension and then deny the selector.

Numbering is a naming convention, not a priority field. /0001 is just a label. It is the order in the file that matters. Numbering in gaps of ten or a hundred is convention so rules can be inserted without renumbering.

Every allow rule is as narrow as the requirement. One path, one method, one selector, one extension. /bin/energy/quote-request and not /bin/*. That single discipline is the difference between story 2 in section 7 happening to you and not.

#What breaks if you get the filter wrong

Too permissive: an administrative console or a data-dumping endpoint is reachable from the internet. This is the failure mode that ends up in a security report, and it is usually caused by someone widening a rule to unblock a feature under time pressure and never narrowing it again.

Too restrictive: something legitimate 404s on publish and works everywhere else. Stylesheets vanish. An AJAX endpoint fails. A JSON feed a partner depends on stops. The tell is always the same shape — it works on author, it works locally, it 404s on the published site — and the confirmation is always the same: check whether the request appears in the publish instance's request.log at all.

#2.7 /cache — the caching section

Now job one. The cache block has more parts than people expect.

/cache
{
  /docroot          "/opt/dispatcher/cache"
  /statfileslevel   "4"
  /gracePeriod      "2"
  /serveStaleOnError "1"
  /allowAuthorized  "0"
  /enableTTL        "1"

  /rules          { ... }   # which responses may be cached
  /invalidate     { ... }   # which cached files are auto-invalidated
  /allowedClients { ... }   # who may send an invalidation request
  /ignoreUrlParams{ ... }   # query params that don't defeat the cache
  /headers        { ... }   # which response headers to cache alongside
}

#/docroot — the cache is a directory tree

/docroot "/opt/dispatcher/cache"

And the mapping is the plainest thing in this entire file:

URL:   https://www.example.com/content/energy/us/en/products/transformers.html
File:  /opt/dispatcher/cache/content/energy/us/en/products/transformers.html

The URL path is the file path under the docroot. You can ls it. You can find it. You can rm it — and people do, though there are better ways.

Selectors and extensions are part of the filename, which is exactly why file 07's design worked:

/content/energy/us/en/products.cards.html
  → /opt/dispatcher/cache/content/energy/us/en/products.cards.html

A suffix becomes a nested path, which is worth understanding because it has a real consequence:

/content/energy/us/en/products.cards.html/2
  → /opt/dispatcher/cache/content/energy/us/en/products.cards.html/2

Notice what that requires: products.cards.html has to be a directory on disk, containing a file called 2. And a filesystem cannot hold a file and a directory with the same name. So if the same URL is requested both with and without a suffix, one of the two forms cannot be cached as a file at that location. In practice this means: pick one shape for an endpoint and stay with it. If the endpoint takes a suffix, always give it a suffix, including for the first batch — .cards.html/1 rather than a bare .cards.html.

That is a genuinely good detail to drop, because it shows you have looked at the docroot rather than only read about it.

#/rules — what may be cached

/rules
{
  # Cache everything by default...
  /0000 { /glob "*" /type "allow" }

  # ...except things that must always be fresh or are per-visitor.
  /0001 { /glob "/content/energy/*/*/search*" /type "deny" }
  /0002 { /glob "*.nocache.html" /type "deny" }   # SDI, section 2.18
}

/filter and /rules are different questions and people conflate them constantly.

/filter asks: is this request allowed to happen at all? /rules asks: may the response be stored?

A request can pass the filter and be uncacheable — a search results page, for instance. A request blocked by the filter never gets as far as the cache rules.

Being able to state that cleanly is a small, reliable mark.

#When is a request actually cacheable?

A response is cached only if all of these hold. This is the checklist to have memorised, because "why isn't my page caching" is one of the most common real questions on this topic.

One — the method is GET (or HEAD). POST, PUT and DELETE are never cached. They change state; caching them would be wrong.

Two — the URL has no query string, unless every parameter is covered by /ignoreUrlParams. Section 2.8.

Three — the request carries no authentication, unless /allowAuthorized is 1. Section 2.9.

Four — the path matches an allow in /cache/rules.

Five — the response status is cacheable. A 200 is. Errors and redirects are generally not stored as cached content.

Six — the response does not tell the dispatcher not to cache it. With /enableTTL "1", the dispatcher honours Cache-Control and Expires from AEM, so a response marked no-store or with max-age=0 will not sit in the cache.

And a seventh that is not a rule but a consequence: the URL has to be expressible as a file path. Which it always is, but it is why the query-parameter rule exists at all.

#2.8 Query parameters — "a dot is cached, a question mark is not"

This is the line that ties files 01, 02, 07, 15 and 17 together, and it is the single most commercially valuable idea in this file.

The mechanism, one more time, because it explains everything: the dispatcher stores a response as a file whose path is the URL path. A query string is not part of the URL path. There is nowhere to put it.

So:

/content/energy/us/en/products.cards.html/2     ← a file path. CACHED.
/bin/energy/cards?listing=/content/...&page=2   ← not a path. NOT CACHED.

Same data. Same JavaScript on the page. Entirely different behaviour on the publish tier. The first is served from disk by Apache. The second runs a page-tree traversal on a publish instance, every single time, for every visitor.

Selectors and suffixes are the path-based alternatives, which is precisely why file 07 chose them:

Now /ignoreUrlParams, which is the escape hatch and also the trap.

/ignoreUrlParams
{
  # Deny by default: any parameter defeats the cache...
  /0001 { /glob "*" /type "deny" }

  # ...except tracking parameters, which don't change the response.
  /0002 { /glob "utm_source"   /type "allow" }
  /0003 { /glob "utm_medium"   /type "allow" }
  /0004 { /glob "utm_campaign" /type "allow" }
  /0005 { /glob "gclid"        /type "allow" }
  /0006 { /glob "fbclid"       /type "allow" }
}

What it actually does: if every parameter in the query string is on the allow list, the dispatcher ignores the query string for caching purposes and serves — or stores — the file at the plain path. The parameters are still forwarded to publish; they are just not part of the cache decision.

If any parameter is not on the list, the whole request is uncached.

Why this is genuinely important on a marketing site. Campaign links carry utm_* parameters. Without this configuration, every visitor arriving from an email campaign or a paid ad gets an uncached request — which is to say, your entire campaign traffic bypasses the cache, which is exactly the traffic you most needed the cache for. This is one of the highest-value five-line changes in AEM.

And the trap, which is a good interview answer. /ignoreUrlParams says "this parameter does not change the response." If you list a parameter that does change the response, the dispatcher caches one visitor's version and serves it to everybody.

"The rule I use is that a parameter only goes in ignoreUrlParams if I can state confidently that two requests differing only in that parameter must return byte-identical HTML. Tracking parameters qualify. A ?lang= or ?region= parameter absolutely does not — and if someone adds one to fix a cache-miss problem, you get a much worse bug where visitors see the wrong country's content, and it's intermittent because it depends on who warmed the cache."

#2.9 Authenticated requests — /allowAuthorized

/allowAuthorized "0"

Zero is the default and it means: do not cache a request that carries authentication information — an Authorization header, or an AEM login token cookie.

Why that default exists is worth reasoning through rather than memorising.

A cached file has no concept of who it is for. It is a file on a disk, served to whoever asks for that path. If you cached an authenticated response, you would be storing a page rendered with that user's permissions and serving it to the next anonymous visitor.

On an energy technology site that could mean a partner portal page showing distributor pricing served to the public internet. That is a data breach caused by a caching setting.

So when would you ever set it to 1? Only in combination with the farm's /sessionmanagement block, which makes the dispatcher itself authenticate the request before serving anything from the cache. That is the closed-user-group pattern: everyone in the group sees the same content, and the dispatcher enforces that you are in the group before handing over the cached file.

" allowAuthorized is 0 by default and I'd leave it there unless there's a specific closed-user-group requirement, and then only alongside session management. Turning it on by itself to improve a hit ratio is how you serve a logged-in user's page to an anonymous visitor. It's a one-character change with a breach-shaped outcome."

This also explains something people find odd: author instances barely cache. Every author request is authenticated, so with allowAuthorized at 0 almost nothing is cacheable — which is exactly what you want, because an author must see the change they just made.

#2.10 Statfiles and invalidation — the mechanism to actually understand

This is the part of the topic that separates people who have read about the dispatcher from people who have operated one. Take it slowly, because the mental model most people arrive with is wrong.

#The wrong mental model

Most people assume: "you flush the dispatcher, and it deletes the cached files."

It does not delete anything.

#What actually happens

The dispatcher maintains small marker files in the cache directory tree, named .stat. They contain nothing. Only their modification timestamp matters.

When an invalidation arrives, the dispatcher touches the relevant .stat files — updating their timestamps to now. The cached HTML files are left exactly where they are, untouched.

Then, on the next request for a cached file, the dispatcher performs one comparison:

Is this cached file NEWER than the relevant .stat file above it?

• Newer → still valid. Serve it from disk.
• Older → stale. Fetch from publish, write the new file (which is now newer than the statfile), serve it.

That is the entire mechanism. Nothing is deleted; things are outdated by comparison.

A useful analogy. Think of the .stat file as a "clear the notice board" date pinned to a department's wall. Nobody walks around removing notices. Instead, when someone asks about a notice, you check whether it was posted before or after the date on the wall. Posted before? Ignore it and get a fresh one. Posted after? Still good.

#Why it is built this way

Because deleting is expensive and touching is free. Consider a bulk publish — a translation drop pushing four hundred German pages. Deleting the corresponding cached files means walking a directory tree and issuing hundreds of filesystem deletes while serving live traffic. Touching one or two marker files is a single fast operation regardless of how much content it invalidates.

It also means invalidation is lazy. Nothing is re-rendered at invalidation time. A page is only re-rendered when somebody actually asks for it. Pages nobody visits are never rebuilt — you do not pay for content nobody reads.

#statfileslevel — the number that controls granularity

/statfileslevel "4"

This controls how deep into the directory tree .stat files are maintained, counting from the docroot.

At level 0 there is exactly one .stat file, at the docroot. Touching it makes every cached file in the entire tree older than it. One page published invalidates the whole site.

At a higher level, .stat files exist in directories down to that depth. Invalidating a path touches the statfiles along that path down to the configured level, so the blast radius is confined to that branch.

Work it through on our structure. The cached path for a US product page is:

/content /energy /us /en /products /transformers.html
   1        2      3    4     5

With statfileslevel "4", statfiles are maintained down to /content/energy/us/en/. So publishing a US English page touches the statfile there — and everything cached under /content/energy/us/en/ becomes stale. The German tree under /content/energy/de/de/ is untouched, because its statfile was not modified.

At level 0, that same publish would have invalidated Germany, France, India and every other market.

#The consequence people get asked about: siblings

"Why does publishing one page invalidate its siblings?"

Because statfiles are per-directory, never per-file. There is no such thing as invalidating exactly one page. The finest granularity available is "everything under this directory", and how far down that directory sits is what statfileslevel controls.

That is a complete, mechanism-based answer, and it is a common question precisely because the answer sounds arbitrary until you know the statfile exists.

#Choosing the value — and the real trade-off

That last row is the sophisticated point, and it is the mistake in story 3.

A low statfileslevel invalidates broadly, which means it accidentally invalidates a lot of things you never explicitly told it about — including pages whose content is derived rather than direct. Raise the level and those hidden dependencies stop being covered.

The classic case: a page that includes an Experience Fragment (file 16). The XF lives under /content/experience-fragments/.... Publishing it touches statfiles in that branch. The pages that include it live under /content/energy/... — a different branch, whose statfile is not touched. So the consuming pages keep serving the old fragment, indefinitely.

At statfileslevel 0 you never noticed, because everything was being invalidated anyway. Raise the level and the latent bug surfaces.

"The honest framing is that a low statfileslevel is not really invalidating correctly — it's invalidating everything and getting away with it. Raising it is right, but it exposes every dependency you were relying on that accident for. So on our project raising the level came with an audit of what content is derived from what, and a replication event handler that issues targeted invalidations for the cases a path-based flush can't express."

#/gracePeriod — stopping the stampede

/gracePeriod "2"

The problem it solves. The moment an invalidation lands, every cached file in that branch becomes stale simultaneously. If a hundred visitors are on the site, the next hundred requests are all misses, and they all hit publish at once — each triggering a full render of a page that has not been rebuilt yet.

That is a thundering herd, and on a big invalidation it can be enough to take a publish instance down. Which then makes it worse, because the requests queue and retry.

What the grace period does: for the configured number of seconds after an invalidation, the dispatcher may still serve the stale cached file rather than passing every request through. It gives the rebuild time to happen and absorbs the initial spike.

The trade-off, stated honestly: for those seconds, some visitors see slightly old content. On a marketing site, two seconds of stale product copy is nothing. On something time-sensitive it might matter, and then you tune it down or handle that path separately.

Where this really earns its keep is a bulk activation — the translation drop, the campaign launch, the four hundred pages. File 18's advice about publishing large volumes out of hours and the grace period are two halves of the same concern.

#/serveStaleOnError

/serveStaleOnError "1"

Different problem, related idea. If a render returns an error, serve the stale cached copy rather than passing the error to the visitor.

This is one of the highest-value single lines in the config. It means a publish instance failing during a deployment, a restart, or an incident degrades into "the site shows slightly old content" rather than "the site shows 500 errors." For a site that changes weekly, that is very close to no impact at all.

#/invalidate — which files are auto-invalidated

/invalidate
{
  /0000 { /glob "*"      /type "deny"  }
  /0001 { /glob "*.html" /type "allow" }
}

This says which cached files participate in statfile-based invalidation. The convention is: only HTML.

Why exclude everything else? Because a lot of cached content is either immutable or managed differently. Clientlib files carry a hash in their filename (file 04) — a new build produces a new URL, so the old file never needs invalidating. Assets under /content/dam are usually handled by their own rules or TTLs.

What it means practically: a file not covered here is not removed by a normal flush. It goes when its TTL expires, or when something sends an explicit invalidation naming that resource. That is the cause of a specific and confusing production symptom — you publish, you flush, the HTML updates, and a cached JSON or feed keeps serving old data because it was never in /invalidate in the first place. If your project exposes JSON that must stay in step with page content (file 17's .model.json, for instance), it needs to be covered here deliberately.

#/allowedClients — who may flush

/allowedClients
{
  # Nobody, by default
  /0001 { /glob "*"           /type "deny"  }
  # Except the author instance that sends flush requests
  /0002 { /glob "10.0.1.15"   /type "allow" }
  /0003 { /glob "10.0.1.16"   /type "allow" }
}

This is a security control, not a convenience setting, and it is one of the most commonly left wide open.

If anyone can send an invalidation, anyone can invalidate your entire cache — repeatedly, in a loop. Every request then becomes a cache miss, your hit ratio goes to zero, and your full production traffic lands on the publish tier. That is a denial-of-service with a two-line curl command.

It is defence in depth alongside the filter rule denying /dispatcher/invalidate.cache: the filter stops the request arriving, and allowedClients stops it working if it somehow does.

#/headers and /enableTTL

/headers
{
  "Cache-Control"
  "Content-Disposition"
  "Content-Type"
  "Expires"
  "Last-Modified"
  "Link"
  "X-Content-Type-Options"
}

A cached file on disk has no headers — it is just bytes. So the dispatcher stores the response headers you list alongside it, and replays them on a cache hit.

The failure this causes when it is wrong: a PDF served with the wrong content type on the second request but the right one on the first. Or, more damagingly, Cache-Control not being in this list — so the first response carries the header that tells the CDN and browser how long to cache, and every subsequent response served from the dispatcher's cache does not. Downstream caching then behaves inconsistently in a way that is very hard to reproduce, because it depends on whether you happened to be the visitor who caused the miss.

/enableTTL "1"

With TTL enabled, the dispatcher honours Cache-Control: max-age and Expires from AEM and expires the cached file accordingly. This is what lets AEM control cache lifetime from application code rather than only from the dispatcher config — and it is how you give a news feed a five-minute life without touching dispatcher.any.

#2.11 The flush agent — how the invalidation arrives

File 18 built this from the replication side. Here is the dispatcher side of the same conversation.

The author instance has a flush agent — a replication agent whose transport URI points not at a publish instance but at the dispatcher:

transportUri = "http://dispatcher-host/dispatcher/invalidate.cache"
serializationType = "flush"

When content is activated, the agent sends a request that looks essentially like this:

POST /dispatcher/invalidate.cache HTTP/1.1
Host: dispatcher-host
CQ-Action: Activate
CQ-Handle: /content/energy/us/en/products/transformers
Content-Type: text/plain
Content-Length: 0

The headers are the whole message:

CQ-Action-Scope: ResourceOnly is the detail worth knowing, because it is the answer to "how would you invalidate exactly one page without invalidating its siblings?" It bypasses the statfile mechanism entirely and removes only the named resource's cached file. Tooling that does surgical invalidation — including the widely used ACS AEM Commons dispatcher flush rules — relies on it.

The dispatcher's side of the handshake:

1. The request arrives at Apache and hits the dispatcher module.
2. /cache/allowedClients is checked. Not allowed → rejected.
3. The CQ-Handle path is mapped into the docroot.
4. The relevant .stat files are touched.
5. A response goes back, and the agent's queue moves on.

And this is where the "content is published but the site shows the old page" ticket comes from. Those are two independent operations: content reaching publish, and the dispatcher being told. Either can succeed while the other fails. File 18's six-step investigation exists precisely because of that split.

#2.12 /statistics and /vanity_urls

Two smaller sections that get asked about mainly to see whether you actually know what they do — and /statistics has a name that misleads almost everyone.

/statistics
{
  /categories
  {
    /html   { /glob "*.html" }
    /others { /glob "*"      }
  }
}

/statistics is not about cache hit ratios. It is part of load balancing. The dispatcher measures render response times per category, so that a render which is slow at HTML but fine at everything else is judged on the right workload. Those measurements are what drive which render gets the next request.

Getting this right in an interview is a nice, cheap differentiator, because the name invites the wrong answer and most people give it.

/vanity_urls
{
  /url   "/libs/granite/dispatcher/content/vanityUrls.html"
  /file  "/tmp/vanity_urls"
  /delay 300
}

The problem this solves: a vanity URL like /hvdc is a property on a page in AEM. The dispatcher's filter knows nothing about it, so a deny-by-default filter blocks it, and authors cannot create working vanity URLs without a developer adding a filter rule each time.

The solution: the dispatcher periodically fetches the list of vanity URLs from publish, caches it in a local file, and allows those paths through the filter automatically. /delay is the refresh interval in seconds.

The trade-off, which is the interview-worthy part: it means authors can effectively punch holes in your filter by creating vanity URLs. That is usually acceptable — a vanity URL resolves to a real page — but it is worth being aware of, and it is one more reason your content ACLs must be right rather than relying on the filter alone.

#2.13 Cache hit ratio — measuring whether any of this works

Every conversation about dispatcher tuning ends here, so have a number and a method.

The definition:

hit ratio = requests served from the cache / total requests

What is good? Above 90% is the number to quote for a content site, and above 95% is achievable and normal for a marketing site with mostly static pages. Below 80% means something structural is wrong — almost always query parameters, uncacheable paths, or an invalidation that is far too broad.

How to actually measure it, since this is where vague answers get caught. Three methods, in increasing order of effort:

Method one — compare two logs. Every request appears in Apache's access_log. Only cache misses reach publish and appear in the publish instance's request.log. So:

hit ratio ≈ (Apache requests − publish requests) / Apache requests

Count over the same window, filter to the extensions you care about (usually just .html), and you have a defensible number in about two minutes. This is the answer to give, because it is simple, it uses logs you already have, and you can describe doing it.

Method two — the dispatcher log at debug level. With DispatcherLogLevel 3, the dispatcher logs its cache decision for each request. Counting those lines gives you a precise figure. It is verbose enough that this is a sampling exercise, not something you leave running.

Method three — log the outcome in Apache. Because a cache hit is served as a static file and a miss goes through the dispatcher handler, you can distinguish them in a custom LogFormat — response time (%D) is a decent proxy on its own, since a hit is served in microseconds and a miss carries a full render.

On Cloud Service, Adobe surfaces cache metrics for the CDN, and the same log-comparison approach works using the dispatcher and publish logs downloaded from Cloud Manager.

And the important framing: hit ratio is not a target in itself. A high ratio on a small number of pages is worth less than a slightly lower ratio across the pages people actually visit. If your ratio is 94% but the 6% is your homepage, you have a problem that the number is hiding. Say that — it shows you understand the metric rather than just quoting it.

#2.14 /etc.clientlibs — closing file 04's loop

File 04 ended with an unfinished story. This is the ending.

The recap. Clientlibs live under /apps, which the public cannot read. So AEM serves them through a proxy at /etc.clientlibs, and the clientlib must set allowProxy="{Boolean}true" to be eligible.

The part file 04 deferred: even with allowProxy set perfectly, the dispatcher filter must allow /etc.clientlibs, or the request is rejected at Apache and never reaches AEM.

/0011 { /type "allow" /method "GET" /path "/etc.clientlibs" }

So there are two independent causes for one identical symptom — the site renders unstyled on publish, and every CSS and JS request returns 404:

Cause A: allowProxy is missing, or was written as the string "true" without the {Boolean} type hint. AEM itself refuses to serve the file.

Cause B: the dispatcher filter has no /etc.clientlibs allow rule. The request never gets to AEM.

Telling them apart is the diagnostic worth memorising, and it is a two-command answer:

graph TD A["/etc.clientlibs/....css returns 404 on publish"] --> B{"Does the request appear in the<br/>PUBLISH instance's request.log?"} B -->|"No — it never arrived"| C["CAUSE B: the dispatcher filter.<br/>Add the /etc.clientlibs allow rule.<br/>Confirm in dispatcher.log at debug level:<br/>the filter check and the block are logged."] B -->|"Yes — AEM answered 404"| D["CAUSE A: allowProxy.<br/>Check the clientlib's.content.xml for<br/>allowProxy={Boolean}true — the type hint<br/>matters, the string 'true' does not work."] C --> E["Same symptom, two layers.<br/>The log tells you which."] D --> E

"It's the same 404 from two different layers, and the way I separate them is to ask whether the request reached AEM at all. If the publish instance's request.log has no entry, the dispatcher blocked it and I'm looking at a filter rule. If AEM logged the request and returned 404 itself, the request got through and it's allowProxy on the clientlib. Turning DispatcherLogLevel up to debug shows the filter evaluation directly, which confirms it either way.

The reason I check in that order is that it's one log grep and it eliminates half the problem space, and I've seen a team spend an afternoon re-checking allowProxy on a config that was already correct."

And there is a third, sneakier variant worth knowing: the filter allows /etc.clientlibs but the request is being rejected on the extension. If your broad allow rule lists extensions and someone adds a font format or a source map that is not in the list, those specific files 404 while the CSS and JS work. The symptom is then "the site is styled but the icon font is missing", which sends people looking at fonts rather than at the filter.

#2.15 CDN — the layer above the dispatcher

The dispatcher removes load from AEM. A CDN removes load from the dispatcher and removes distance from the visitor.

Why distance matters concretely on our project. The AEM infrastructure sits in one or two regions. Visitors are in Houston, Munich, Bangalore, São Paulo. A round trip from Brazil to a European data centre is a real, measurable delay, repeated for every asset on the page. A CDN puts a copy in a point of presence near the visitor, so most requests never cross an ocean.

The second thing a CDN gives you is absorption. It sits in front of everything, so a traffic spike — a campaign, a press mention, an attack — is absorbed at the edge instead of arriving at your Apache tier.

#On AEM 6.5 versus AEM as a Cloud Service

The AEMaaCS-included CDN is one of the more genuine benefits of the platform, and it is a better thing to cite in a "why Cloud Service" answer (file 14) than "auto-scaling", because it removes a whole vendor relationship rather than just some capacity planning.

On 6.5 you can still bring your own CDN in front of AEMaaCS if you have an existing contract or need specific WAF features, and Adobe supports that arrangement.

#The headers that control CDN behaviour

This is the vocabulary part, and the Cache-Control versus Surrogate-Control distinction is a good question because it separates people who have configured a CDN from people who have heard of one.

The pattern this vocabulary enables is the important bit, and it is the answer to "how do you cache HTML aggressively without visitors seeing stale content for hours":

Cache-Control:     max-age=60          ← browsers hold it for 1 minute
Surrogate-Control: max-age=86400       ← the CDN holds it for a day

Short in the browser, long at the edge. The CDN takes essentially all the load, but because you can purge the CDN and cannot purge a browser, a content change propagates in about a minute rather than a day. You keep the performance and keep control.

That combination is the single most useful CDN idea in this file, and it is a very good thing to be able to explain unprompted.

Age is your main debugging header. A response with Age: 0 on a repeat request means the CDN is not caching. A response with a growing Age means it is. Many CDNs also send a vendor-specific hit/miss header such as X-Cache: HIT, which is even more direct when it is available.

#Purging

Two strategies, and knowing when each applies is the answer:

Strategy one — versioned URLs, no purge needed. If the URL changes when the content changes, the old cached object is simply never requested again. This is exactly what clientlib hashes do (file 04): a new build produces clientlib-site.lc-<hash>-lc.min.css, a URL that has never been cached, so it cannot be stale. It is the most reliable cache invalidation there is, because it does not involve invalidation at all.

Strategy two — explicit purge. For HTML, where the URL must stay the same. You call the CDN's purge API for a URL, a set of URLs, or a tag.

Two things to say about purging that make the answer sound experienced:

Purge is not instant and it is not free. A global purge propagates across every point of presence. It is fine as part of a publish workflow; it is not something to fire on every content event.

Purging the CDN and flushing the dispatcher are different operations. Flushing the dispatcher does not touch the CDN. That is the cause of the classic escalation where a developer has flushed the dispatcher, verified the fix against the origin, and the visitor still sees the old page — because the layer above still holds it.

On AEMaaCS, Adobe exposes a purge capability configured through the CDN configuration in the config pipeline, using an API key you set up there. The recommended default posture is still short TTLs on HTML plus versioned URLs for assets, using purge as the exception for something urgent, rather than building a workflow that purges on every publish.

#2.16 The layered cache — and knowing which layer is lying to you

Four caches sit between a visitor and the repository. The most common time-waster in this whole topic is fixing the wrong one.

graph LR A["Browser cache<br/>per visitor<br/>you CANNOT purge it"] --> B["CDN<br/>global, at the edge<br/>purge API"] B --> C["Dispatcher cache<br/>files on disk<br/>statfile invalidation"] C --> D["Publish instance<br/>renders from Oak"] D --> E["Oak repository"]

Each layer answers if it can, so a stale page can be coming from any of them — and flushing the dispatcher does nothing at all if the CDN is the one holding it.

#The discipline: work inward, one layer per command

Step 0 — eliminate the browser. Private window, or a hard reload, or curl. curl is best because it has no cache at all, so anything it sees is genuinely coming from the network.

Step 1 — the CDN. Request the public URL and look at the response headers:

curl -sSI https://www.example.com/us/en/products/transformers.html

A non-zero and growing Age means the CDN is serving it. A vendor X-Cache: HIT says so directly.

Step 2 — the dispatcher. Request the dispatcher host directly, bypassing the CDN:

curl -sSI -H "Host: www.example.com" http://dispatcher-host/us/en/products/transformers.html

Fresh content here but stale from step 1 → the CDN is the problem; purge it.

Step 3 — the dispatcher's disk. If step 2 is also stale, look at the actual file:

ls -la /opt/dispatcher/cache/content/energy/us/en/products/transformers.html
ls -la /opt/dispatcher/cache/content/energy/us/en/.stat

Compare the two timestamps. The cached file newer than the statfile means the dispatcher considers it valid — which means the invalidation never arrived, or it touched a different statfile than you expected. That single comparison is the most direct answer to "why is the dispatcher still serving this."

Step 4 — publish. Go straight to the instance:

curl -sSI http://publish1.internal:4503/content/energy/us/en/products/transformers.html

Stale here too? It is not a caching problem at all — the content was never published, and you are now in file 18's investigation.

Why this order. Each step is one command, and each one eliminates an entire layer. It also stops you doing the thing everyone does under pressure, which is flush everything, watch it come right, and learn nothing about which layer was actually at fault — so it happens again next month.

"The mistake I try not to make is flushing every layer at once. It fixes the ticket and teaches you nothing, and the same ticket comes back. Going inward one layer at a time costs about two extra minutes and tells you which layer failed, which is the difference between fixing an incident and fixing a cause."

#2.17 The dispatcher on AEM as a Cloud Service

File 14 established the shape. Here is what it means day to day.

The dispatcher is a container inside each publish pod. Not a separate Apache VM you can SSH into. Each pod has its own Apache-plus-dispatcher container with its own local cache.

Three consequences that follow from that, and interviewers like all three:

One — each pod has an independent cache. New pod, empty cache. A scale-up event means new pods serving cold, so the first requests to them are misses. This is a genuine argument for having a CDN in front, because the CDN absorbs that while the pods warm.

Two — the cache is ephemeral. A pod replacement discards its cache. So you cannot treat the dispatcher cache as durable state, and anything that depends on a warm cache to be fast enough is fragile by design.

Three — you cannot log in and edit the config. It comes from Git.

The configuration lives in your Git repository under a dispatcher module, in a prescribed directory layout — conf.d for Apache configuration and conf.dispatcher.d for the dispatcher's own. The structure uses an "available / enabled" convention borrowed from Debian's Apache packaging:

dispatcher/src/
├── conf.d/
│   ├── available_vhosts/          ← all vhost definitions
│   ├── enabled_vhosts/            ← symlinks to the active ones
│   └── rewrites/
└── conf.dispatcher.d/
    ├── available_farms/           ← all farm definitions
    ├── enabled_farms/             ← symlinks to the active ones
    ├── cache/
    ├── clientheaders/
    ├── filters/
    └── virtualhosts/

Why "available and enabled" rather than just the active files: you can keep a farm definition in the repository without it being live, and enabling it is a symlink change rather than a file move — which is a much cleaner diff in a pull request.

The dispatcher SDK validator is the tool to know by name. It ships with the AEM SDK and checks your configuration before you push:

# Validate the configuration
./bin/validator full -d ./out ./dispatcher/src

# Run the whole thing locally in Docker against a local publish instance
./bin/docker_run ./out host.docker.internal:4503 8080

What it catches: directives that are not permitted on Cloud Service, structural mistakes, missing includes, broken symlinks. Why it matters practically: the Cloud Manager pipeline runs the same validation, so an invalid config fails your deployment. Running it locally turns a failed pipeline into a ten-second local error.

Deployment goes through the web-tier pipeline (file 14), and this is the operationally important fact:

A dispatcher config change does not need a full-stack build. The web-tier pipeline deploys only the dispatcher configuration and is dramatically faster.

Why that matters in an incident. File 14 was honest that Cloud Service has no emergency hotfix path. The web-tier pipeline is the closest thing to one — a filter rule, a cache rule, a block on an abusive path can go out far faster than a code change. If an interviewer asks "how would you respond quickly to something on Cloud Service", the web-tier pipeline and a CDN rule are the two real answers, and knowing that is a practical, senior-sounding detail.

Flexible mode. Adobe's Cloud Service dispatcher configuration originally restricted which directives you could use. "Flexible mode" relaxed that considerably, allowing your own includes and a wider set of Apache directives. If you meet a project on the older restricted layout, converting it is a known migration, and the Dispatcher Converter tool (file 14) exists for moving a 6.5 configuration to the Cloud Service structure.

Local development. The SDK's docker_run gives you an Apache-plus-dispatcher container pointed at your local AEM. This is the thing most developers skip and then regret, because dispatcher-only bugs — a filter blocking your new servlet, a URL that will not cache — are invisible when you develop against localhost:4502 directly. Testing behind a local dispatcher catches the entire "works locally, 404s on publish" class of bug before it leaves your machine.

#2.18 Sling Dynamic Include — dynamic parts of a cached page

The problem. A page is ninety-five percent identical for every visitor and five percent personalised — a "recently viewed products" strip, a regional stock indicator, a logged-in partner's name.

Caching the page caches the personalised part too, and everyone gets the first visitor's version. So the usual answer is: do not cache the page. Which throws away the ninety-five percent to protect the five.

Three ways out, and being able to compare them is the answer:

Option one — do it client-side. JavaScript fetches the dynamic part after load. The page stays fully cacheable. This is the default choice, and it is what file 02 and file 07 chose for our Load More. The costs are a flash of content appearing after load, and that the content is invisible to crawlers.

Option two — Sling Dynamic Include (SDI).

What it does: at render time, SDI replaces a configured component's markup with a server-side include directive instead of the component's actual output. The dispatcher caches the page containing the directive. Then Apache — via mod_include for SSI, or the CDN for ESI — resolves that directive on every request by fetching just that fragment.

So the page is cached and the fragment is not. You get both.

Roughly what the flow looks like:

1. Request arrives, page not cached → goes to publish
2. Publish renders the page, but SDI's filter swaps the configured
   component's output for:   <!--#include virtual="/content/....nocache.html" -->
3. The dispatcher caches THAT page, include directive and all
4. Apache's mod_include sees the directive and fetches
   /content/....nocache.html
5. That request is NOT cached (a cache rule denies *.nocache.html)
   so it goes to publish every time
6. Apache assembles the two and returns the response

Configuration is an OSGi factory configuration — "Apache Sling Dynamic Include - Configuration" — with a configuration per resource type set. The properties that matter:

And SDI needs three things configured outside AEM, all of which are common failure points:

1. Apache must have SSI enabled for the docroot — Options +Includes and SetOutputFilter INCLUDES, or the include directive is served to the browser as a literal HTML comment and the fragment simply never appears.
2. The dispatcher filter must allow the nocache selector, or the fragment request is blocked and you get an empty hole.
3. The dispatcher cache rules must deny caching *.nocache.html, or the fragment gets cached and you are back where you started — with the added confusion that it now looks like it is working.

Option three — the CDN's own edge personalisation, where the CDN assembles the page. Powerful, and it moves logic into a layer that is harder to test and version. Worth mentioning as an option; rarely the first choice.

The honest comparison, which is what an interviewer actually wants:

"My default is client-side, because it keeps the page fully cacheable and there's nothing extra to configure or go wrong. SDI is the right answer when the dynamic part has to be in the server-rendered HTML — usually for SEO, or when it would cause visible layout shift appearing late.

The reason I don't reach for SDI first is that it has three dependencies outside AEM — Apache's SSI configuration, a dispatcher filter rule for the include selector, and a cache rule that stops the fragment being cached. Any of those being wrong gives you a confusing failure, and the worst one is the cache rule, because a cached fragment looks like it's working right up until someone sees another visitor's data."

#3. Internal Working

#3.1 The full request path

This is the diagram to be able to draw on a whiteboard.

graph TD A["Request arrives at Apache"] --> B{"mod_rewrite<br/>rewrites the URL?"} B --> C["Dispatcher handler<br/>(DispatcherUseProcessedURL decides<br/>whether it sees the rewritten URL)"] C --> D{"Which FARM?<br/>match Host against /virtualhosts"} D --> E{"/filter<br/>is this request ALLOWED?"} E -->|"No"| F["404 from Apache.<br/>AEM never hears about it.<br/>Nothing in publish request.log."] E -->|"Yes"| G{"Is it cacheable?<br/>GET? no query string?<br/>no auth? matches /rules?"} G -->|"No"| H["Pass to a render.<br/>Response returned, NOT stored."] G -->|"Yes"| I{"Does the cached<br/>file exist on disk?"} I -->|"No — MISS"| J["Pass to a render"] I -->|"Yes"| K{"Is the file NEWER<br/>than the.stat file?"} K -->|"No — STALE"| J K -->|"Yes — HIT"| L["Serve from disk.<br/>AEM is not involved at all."] J --> M["/renders: pick an instance<br/>using /statistics timings"] M --> N{"Did it answer?"} N -->|"No"| O["/failover: retry on another.<br/>Or serveStaleOnError: serve<br/>the old file rather than an error."] N -->|"Yes"| P["Write the response into the docroot<br/>+ store the /headers alongside it"] P --> Q["Return it to the visitor"]

Four things to pull out of that diagram, because they are each a separate interview answer:

The filter runs before the cache. A blocked request is not a cache miss — it is not a cache anything. It never reaches that decision.

"Cacheable" and "cached" are different questions. A request can be perfectly allowed, go to publish, and never be stored, because it had a query string or carried authentication.

A hit costs nothing. It is a file read. No network call, no render, no repository access.

A miss writes as a side effect. The dispatcher does not have a separate cache-population step — serving a miss is how the cache gets populated. Which is also why the first visitor after an invalidation pays the render cost, and why the grace period exists.

#3.2 The statfile decision, precisely

graph TD A["Request for<br/>/content/energy/us/en/products.html"] --> B["Map to the docroot:<br/>{docroot}/content/energy/us/en/products.html"] B --> C{"Does that file exist?"} C -->|"No"| D["MISS → fetch from publish,<br/>write the file, serve it"] C -->|"Yes"| E["Find the relevant.stat files:<br/>this directory and its ancestors,<br/>down to statfileslevel"] E --> F{"Cached file mtime<br/>NEWER than the.stat mtime?"} F -->|"Yes"| G["HIT — serve from disk"] F -->|"No"| H{"Are we inside<br/>the gracePeriod after<br/>the last invalidation?"} H -->|"Yes"| I["Serve the STALE file anyway.<br/>This is what prevents<br/>a thundering herd."] H -->|"No"| J["STALE → fetch from publish,<br/>overwrite the file<br/>(it is now newer than the.stat),<br/>serve it"]

The whole mechanism is one timestamp comparison, and once you can say that sentence the rest of the topic follows from it:

• Why does flushing not free disk space? Nothing is deleted.
• Why does publishing one page invalidate siblings? The comparison is against a directory-level marker.
• Why does statfileslevel change the blast radius? It changes which directory holds the marker.
• Why does a grace period help? It is a temporary override of the comparison.

#3.3 An invalidation, end to end

sequenceDiagram participant AU as Author participant FA as Flush agent participant D as Dispatcher (Apache) participant FS as Cache on disk participant P as Publish participant V as Visitor AU->>AU: Author activates a page AU->>FA: Replication event FA->>D: POST /dispatcher/invalidate.cache<br/>CQ-Action: Activate<br/>CQ-Handle: /content/energy/us/en/products D->>D: Check /cache/allowedClients Note over D: Not allowed → rejected here,<br/>and the cache silently stays stale D->>FS: TOUCH the.stat files on that path,<br/>down to statfileslevel Note over FS: The cached.html files are NOT deleted.<br/>They are now older than the.stat file. D-->>FA: OK — the queue moves on V->>D: GET /us/en/products/transformers.html D->>FS: Compare file mtime vs.stat mtime FS-->>D: File is OLDER → stale D->>P: Fetch a fresh render P-->>D: Fresh HTML D->>FS: Write it (now newer than the.stat) D-->>V: Serve it Note over V: The NEXT visitor gets a cache hit.<br/>Only the first one pays the render cost.

The two failure points to name, because between them they account for most "published but not live" tickets:

allowedClients rejects the invalidation. Everything upstream looks perfect — the page is on publish, the replication queue is clean, the agent reported success — and the dispatcher never touched a statfile. From the author's point of view nothing failed. This is why the flush agent's own log is worth checking.

The statfile touched is not the one the stale file sits under. With a high statfileslevel and content whose relationship is not path-based — an Experience Fragment, a listing built from a page tree, a .model.json export — the invalidation lands in the wrong branch and the derived page never notices.

#3.4 The layered cache as a decision tree

graph TD A["A visitor reports stale content"] --> B{"Private window / curl:<br/>still stale?"} B -->|"No"| C["1. BROWSER CACHE.<br/>Nothing server-side is wrong.<br/>Look at your Cache-Control max-age."] B -->|"Yes"| D{"Is the CDN serving it?<br/>curl -I: is Age non-zero?"} D -->|"Yes and stale"| E["2. CDN.<br/>Purge it. Flushing the dispatcher<br/>would have done nothing."] D -->|"No / bypassed"| F{"Request the dispatcher directly.<br/>Still stale?"} F -->|"No"| E F -->|"Yes"| G{"On disk: is the cached file<br/>newer than the.stat file?"} G -->|"Yes — considered valid"| H["3. INVALIDATION NEVER ARRIVED.<br/>Check allowedClients, the filter rule<br/>on /dispatcher/invalidate.cache,<br/>and the flush agent's log."] G -->|"No — considered stale"| I{"Request publish directly.<br/>Is IT stale?"} I -->|"Yes"| J["4. NOT A CACHE PROBLEM.<br/>The content was never published.<br/>→ file 18's investigation."] I -->|"No"| K["5. Derived content:<br/>the flush touched a statfile that<br/>doesn't cover this page.<br/>Needs targeted invalidation."]

This decision tree is the single most reusable thing in this file, and it is worth being able to walk it out loud. It converts the vaguest ticket in AEM — "the site is showing old content" — into five distinct, testable causes.

#8. Configuration and Coding Examples

#8.1 A complete, commented publish farm

This is the reference configuration. Every block is commented with the reasoning, not just the syntax.

# ==================================================================
#  dispatcher.any  --  publish farm for the energy site
#
#  Read as: /farms holds one or more farms. A farm is a complete
#  configuration for a set of hostnames. The LAST farm whose
#  /virtualhosts matches the request's Host header wins -- which is
#  why the file (and, on Cloud Service, the FILE NAMES) control which
#  farm answers.
# ==================================================================

/farms
{
  /energy_publish
  {
    # --------------------------------------------------------------
    # 1. CLIENTHEADERS
    #    Request headers forwarded from the browser to AEM.
    #    A header NOT listed here never reaches your servlet.
    #    That is the cause of "my custom header is null on publish
    #    but works locally" -- the code is fine, the header was
    #    dropped one layer earlier.
    # --------------------------------------------------------------
    /clientheaders
    {
      "referer"
      "user-agent"
      "authorization"
      "from"
      "content-type"
      "content-length"
      "accept-charset"
      "accept-encoding"
      "accept-language"
      "accept"
      "host"
      "if-match"
      "if-none-match"
      "cookie"
      "CSRF-Token"
      # Carries the visitor's real IP. Without it every request looks
      # to AEM like it came from the dispatcher.
      "X-Forwarded-For"
      "*"
    }

    # --------------------------------------------------------------
    # 2. VIRTUALHOSTS
    #    Which Host headers this farm answers for.
    #    More specific entries beat wildcards.
    # --------------------------------------------------------------
    /virtualhosts
    {
      "www.example.com"
      "*.example.com"
    }

    # --------------------------------------------------------------
    # 3. RENDERS -- the publish instances behind this farm.
    #    With more than one, the dispatcher distributes requests and
    #    uses the /statistics timings to prefer the faster instance.
    # --------------------------------------------------------------
    /renders
    {
      /rend01
      {
        /hostname "publish1.internal"
        /port     "4503"
        # How long to wait before giving up on this render.
        # Too high and a hung publish holds Apache workers open.
        /timeout  "10000"
      }
      /rend02
      {
        /hostname "publish2.internal"
        /port     "4503"
        /timeout  "10000"
      }
    }

    # Retry a failed request on a DIFFERENT render rather than
    # passing the failure to the visitor.
    /failover           "1"
    /numberOfRetries    "5"
    /retryDelay         "1"
    /unavailablePenalty "1"

    # --------------------------------------------------------------
    # 4. FILTER -- the security section. The most important block
    #    in this file. See 8.2 for the full contents; in production
    #    it is an $include so the security rules are shared across
    #    every farm and reviewed in one place.
    # --------------------------------------------------------------
    /filter
    {
      $include "/etc/httpd/conf.dispatcher.d/filters/security_filters.any"
      $include "/etc/httpd/conf.dispatcher.d/filters/energy_filters.any"
    }

    # --------------------------------------------------------------
    # 5. CACHE
    # --------------------------------------------------------------
    /cache
    {
      # The cache is a DIRECTORY TREE. The URL path is the file path.
      #   /content/energy/us/en/products.html
      #     -> /opt/dispatcher/cache/content/energy/us/en/products.html
      # This is the SAME directory as Apache's DocumentRoot, so a
      # cache hit is Apache serving an ordinary static file.
      /docroot "/opt/dispatcher/cache"

      # HOW DEEP .stat FILES ARE MAINTAINED = the blast radius of an
      # invalidation.
      #   /content /energy /us /en
      #      1        2      3   4
      # At 4, publishing a US English page invalidates only the US
      # English branch. The German tree is untouched.
      # At 0 there is ONE statfile at the docroot, so publishing
      # anything invalidates EVERYTHING. That is the re-render storm.
      /statfileslevel "4"

      # After an invalidation, every file in the branch goes stale at
      # the same instant, so a burst of requests all miss together and
      # hit publish at once -- a thundering herd. For this many
      # seconds the dispatcher may still serve the stale file, which
      # absorbs the spike while the first rebuild happens.
      /gracePeriod "2"

      # If a render returns an error, serve the STALE cached copy
      # rather than passing the error on. One of the highest-value
      # lines in this file: a publish restart degrades to "slightly
      # old content" instead of "500 errors".
      /serveStaleOnError "1"

      # Do NOT cache requests carrying authentication.
      # A cached file has no notion of who it is for -- caching an
      # authenticated response means serving one user's page to the
      # next anonymous visitor. Only ever set this to 1 alongside
      # /sessionmanagement, for a closed user group.
      /allowAuthorized "0"

      # Honour Cache-Control / Expires coming FROM AEM, so cache
      # lifetime can be controlled from application code rather than
      # only from this file.
      /enableTTL "1"

      # ---- Which RESPONSES may be stored -------------------------
      # NOTE: this is a different question from /filter.
      #   /filter -> is the request ALLOWED at all?
      #   /rules  -> may the response be STORED?
      /rules
      {
        /0000 { /glob "*" /type "allow" }

        # Per-visitor or unbounded output must never be cached.
        /0001 { /glob "/content/energy/*/*/search*" /type "deny" }
        /0002 { /glob "/content/energy/*/*/account*" /type "deny" }
        # Sling Dynamic Include fragments. If this deny is missing,
        # the fragment gets cached and SDI silently serves one
        # visitor's personalised content to everybody.
        /0003 { /glob "*.nocache.html" /type "deny" }
      }

      # ---- Which cached files are AUTO-invalidated ---------------
      # Only HTML participates in statfile invalidation. Clientlibs
      # carry a hash in the filename (file 04) so a new build is a
      # new URL and the old file never needs invalidating.
      #
      # WATCH OUT: anything NOT listed here is never removed by a
      # normal flush. If you expose .model.json that must stay in
      # step with page content (file 17), add it deliberately.
      /invalidate
      {
        /0000 { /glob "*"      /type "deny"  }
        /0001 { /glob "*.html" /type "allow" }
      }

      # ---- WHO may send an invalidation --------------------------
      # A SECURITY control, not a convenience. If anyone can flush,
      # anyone can flush in a loop -- every request becomes a miss
      # and your full traffic lands on publish. That is a
      # denial-of-service with a two-line curl.
      /allowedClients
      {
        /0001 { /glob "*"         /type "deny"  }
        /0002 { /glob "10.0.1.15" /type "allow" }   # author
        /0003 { /glob "10.0.1.16" /type "allow" }   # author standby
      }

      # ---- Query parameters that do NOT defeat the cache ---------
      # THE TEST: two requests differing only in this parameter must
      # return byte-identical HTML.
      # Tracking parameters pass that test. ?page= and ?region= do
      # NOT -- listing those would cache one visitor's version and
      # serve it to everyone.
      #
      # Without this block, ALL campaign traffic bypasses the cache,
      # because campaign links carry utm_* parameters. That is the
      # traffic you most needed cached.
      /ignoreUrlParams
      {
        /0001 { /glob "*"            /type "deny"  }
        /0002 { /glob "utm_source"   /type "allow" }
        /0003 { /glob "utm_medium"   /type "allow" }
        /0004 { /glob "utm_campaign" /type "allow" }
        /0005 { /glob "utm_term"     /type "allow" }
        /0006 { /glob "utm_content"  /type "allow" }
        /0007 { /glob "gclid"        /type "allow" }
        /0008 { /glob "fbclid"       /type "allow" }
      }

      # ---- Response headers stored alongside the cached file -----
      # A cached file on disk has no headers -- it is just bytes.
      # These are stored next to it and replayed on a hit.
      # If Cache-Control is missing here, the FIRST response carries
      # it and every cached response does not, so downstream caching
      # behaves inconsistently depending on who caused the miss.
      /headers
      {
        "Cache-Control"
        "Content-Disposition"
        "Content-Type"
        "Expires"
        "Last-Modified"
        "Link"
        "X-Content-Type-Options"
      }
    }

    # --------------------------------------------------------------
    # 6. STATISTICS
    #    NOT about cache hit ratios -- this is LOAD BALANCING.
    #    The dispatcher measures render response times per category
    #    so a render that is slow at HTML but fine at everything else
    #    is judged on the right workload.
    # --------------------------------------------------------------
    /statistics
    {
      /categories
      {
        /html   { /glob "*.html" }
        /others { /glob "*"      }
      }
    }

    # --------------------------------------------------------------
    # 7. VANITY URLS
    #    Vanity URLs are page properties in AEM, so a deny-by-default
    #    filter blocks them and authors cannot create working ones
    #    without a developer adding a rule each time.
    #    The dispatcher periodically fetches the list from publish
    #    and lets those paths through automatically.
    #    TRADE-OFF: authors can effectively punch holes in the
    #    filter, so content ACLs must be right in their own terms.
    # --------------------------------------------------------------
    /vanity_urls
    {
      /url   "/libs/granite/dispatcher/content/vanityUrls.html"
      /file  "/tmp/vanity_urls"
      /delay 300
    }
  }
}

#8.2 The filter section, in full

Kept in its own file so it can be $include d by every farm and reviewed once.

# ==================================================================
#  security_filters.any
#
#  DENY EVERYTHING, then allow exactly what the site needs.
#  LAST MATCHING RULE WINS -- so the structure is
#  deny all -> broad allows -> re-deny the dangerous parts.
#
#  Rule numbers are LABELS, not priorities. Position in the file is
#  what decides. Numbering in gaps lets rules be inserted later.
# ==================================================================

# ---- 1. Deny everything ------------------------------------------
# Why an allowlist and not a blocklist: a blocklist is a list of the
# things you thought of. Every AEM version adds endpoints, every
# project adds servlets. A wrong allowlist gives you a bug report.
# A wrong blocklist gives you a breach.
/0001 { /type "deny" /url "*" }

# ---- 2. What a public site actually needs ------------------------
/0010 {
  /type "allow"
  /method '(GET|HEAD)'
  /extension '(css|eot|gif|ico|jpe?g|js|pdf|png|svg|swf|ttf|woff2?|html)'
}

# The clientlib proxy (file 04). WITHOUT THIS RULE every stylesheet
# and script on the published site returns 404 -- even when
# allowProxy is set perfectly on the clientlib. Same symptom, two
# different layers; the publish request.log tells you which.
/0011 { /type "allow" /method "GET" /path "/etc.clientlibs" }

# DAM assets referenced from pages
/0012 { /type "allow" /method "GET" /path "/content/dam" }

# Sling Dynamic Include fragments (section 2.18). Without this the
# include request is blocked and you get an empty hole in the page.
/0013 { /type "allow" /method "GET" /selectors "nocache" /extension "html" }

# ---- 3. Re-deny what the broad allows above would let through ----

# Serialisation selectors. ONE rule covering the set, so a new
# variant does not need a new rule.
#   .infinity.json  -> DefaultGetServlet dumps a whole subtree
#   .tidy.json      -> the same, pretty-printed
#   .sysview.xml    -> the JCR XML export format
#   .query.json     -> a query endpoint
/0030 {
  /type "deny"
  /selectors '(infinity|tidy|sysview|docview|query|childrenlist|ext|feed)'
  /extension '(json|xml|html|feed)'
}

# Numeric depth selectors: .1.json, .10.json, .-1.json
/0031 { /type "deny" /selectors '([0-9-]+)' /extension "json" }

# QueryBuilder: runs an ARBITRARY repository query over HTTP and
# returns JSON. Not a vulnerability -- a documented API. Exposing it
# is the vulnerability.
/0032 { /type "deny" /url "/bin/querybuilder.json*" }

# Administrative surfaces
/0033 { /type "deny" /path "/system" }          # OSGi web console
/0034 { /type "deny" /path "/crx" }             # CRXDE + Package Manager
/0035 { /type "deny" /path "/bin" }             # path-bound servlets
/0036 { /type "deny" /path "/apps" }            # our source code
/0037 { /type "deny" /path "/libs" }            # Adobe's, incl. tools
/0038 { /type "deny" /path "/home" }            # users and groups
/0039 { /type "deny" /path "/etc" }             # note: /etc.clientlibs
                                                # is a DIFFERENT path

# POST to content paths. The SlingPostServlet (file 07) creates and
# updates nodes from form fields. Anonymous cannot normally write,
# but that depends entirely on ACLs being right, and ACLs drift.
# Blocking POST removes the dependency.
/0040 { /type "deny" /method "POST" /path "/content" }

# The invalidation endpoint must never be reachable from outside.
# Defence in depth alongside /cache/allowedClients: this stops the
# request arriving, allowedClients stops it working if it does.
/0041 { /type "deny" /url "/dispatcher/invalidate.cache" }

# Debug and tracing selectors that leak internals
/0042 { /type "deny" /selectors "debug" }
/0043 { /type "deny" /query "debug=*" }
/0044 { /type "deny" /query "wcmmode=*" }

# ==================================================================
#  energy_filters.any  --  project-specific exceptions.
#
#  EVERY rule here is reviewed. An allow must name a PATH and a
#  METHOD, and ideally an extension. A wildcard branch is never an
#  acceptable allow rule -- "/bin/*" is how a penetration test finds
#  querybuilder on your production tier.
# ==================================================================

# The CRM quote form handler (file 07). Path-bound because there is
# genuinely no content context -- nothing is written to the
# repository. ONE exact path, ONE method.
/0100 { /type "allow" /method "POST" /path "/bin/energy/quote-request" }

# The Sling Model Exporter JSON for the mobile app (file 17).
# Scoped to our content tree, one selector, one extension.
/0101 {
  /type "allow"
  /method "GET"
  /path "/content/energy"
  /selectors "model"
  /extension "json"
}

# The cacheable Load More / tab endpoint (files 02 and 07).
# Resource-type bound, so it sits on a /content path and is already
# covered by the extension allow above. It needs no rule of its own
# -- which is itself an argument for resource-type registration.

#8.3 The Apache virtual host

# ==================================================================
#  Apache vhost for the energy publish tier.
#  Half of "dispatcher configuration" is actually Apache
#  configuration sitting next to the module.
# ==================================================================

<VirtualHost *:443>
    ServerName www.example.com

    SSLEngine on
    SSLCertificateFile      /etc/pki/tls/certs/example.crt
    SSLCertificateKeyFile   /etc/pki/tls/private/example.key

    # The dispatcher docroot and Apache's DocumentRoot are the SAME
    # directory. That is the design: once a file is cached, Apache
    # serves it as an ordinary static file with no module involved.
    DocumentRoot /opt/dispatcher/cache

    # ---- Security headers. mod_headers, NOT the dispatcher. -------
    Header always set X-Content-Type-Options    "nosniff"
    Header always set X-Frame-Options           "SAMEORIGIN"
    Header always set Referrer-Policy           "strict-origin-when-cross-origin"
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
    # Do not advertise what we are running
    Header always unset X-Powered-By
    Header always unset Server

    # ---- Compression. Also Apache, not the dispatcher. -----------
    <IfModule mod_deflate.c>
        AddOutputFilterByType DEFLATE text/html text/css text/plain \
                                      application/javascript application/json \
                                      image/svg+xml
    </IfModule>

    # ---- CDN cache lifetimes -------------------------------------
    # SHORT for the browser, LONG for the CDN. You can purge a CDN;
    # you cannot purge a browser. So the CDN absorbs the load and a
    # content change still propagates in about a minute.
    <LocationMatch "\.html$">
        Header set Cache-Control     "max-age=60"
        Header set Surrogate-Control "max-age=86400"
    </LocationMatch>

    # Clientlibs carry a content hash in the filename (file 04), so
    # the URL changes when the content changes. That means they can
    # be cached effectively forever and NEVER need purging -- the
    # most reliable invalidation there is, because it isn't
    # invalidation at all.
    <LocationMatch "^/etc\.clientlibs/.*\.lc-[0-9a-f]+-lc\.(css|js)$">
        Header set Cache-Control "max-age=31536000, immutable"
    </LocationMatch>

    # ---- Pretty URLs ---------------------------------------------
    # /us/en/products  ->  /content/energy/us/en/products.html
    RewriteEngine On
    RewriteRule ^/(us|de|fr|in|br|cn)/([a-z]{2})/(.*)$ \
                /content/energy/$1/$2/$3 [PT,L]

    # ---- Hand requests to the dispatcher module ------------------
    <Directory /opt/dispatcher/cache>
        <IfModule disp_apache2.c>
            SetHandler dispatcher-handler
        </IfModule>
        Options FollowSymLinks
        AllowOverride None
        Require all granted
    </Directory>

    # The invalidation endpoint, restricted at the Apache layer too.
    # Belt and braces with the filter rule and /cache/allowedClients.
    <Location /dispatcher/invalidate.cache>
        Require ip 10.0.1.15 10.0.1.16
    </Location>

    ErrorLog  logs/energy_error.log
    CustomLog logs/energy_access.log combined
</VirtualHost>

# ==================================================================
#  The module itself -- httpd.conf
# ==================================================================

LoadModule dispatcher_module modules/mod_dispatcher.so

<IfModule disp_apache2.c>
    DispatcherConfig  conf/dispatcher.any
    DispatcherLog     logs/dispatcher.log

    # 0=error 1=warn 2=info 3=debug.
    # Debug logs every filter check and cache decision. It is the
    # single most useful diagnostic here AND extremely noisy:
    # turn it on, reproduce, turn it off.
    DispatcherLogLevel 1

    DispatcherNoServerHeader 1
    DispatcherDeclineRoot    0

    # Filter and cache the REWRITTEN URL, not the incoming one.
    # With mod_rewrite in play and this off, your filter rules are
    # being tested against a URL that no longer exists by the time
    # AEM sees it -- rules that look correct and do nothing.
    DispatcherUseProcessedURL 1

    DispatcherPassError 0
</IfModule>

#8.4 A targeted invalidation for derived content

Story 3's fix, and the piece nothing does for you. This is file 18's event handler with the dispatcher-side reasoning made explicit.

package com.energy.core.listeners;

import org.apache.sling.event.jobs.JobManager;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.Reference;
import org.osgi.service.event.Event;
import org.osgi.service.event.EventConstants;
import org.osgi.service.event.EventHandler;
import com.day.cq.replication.ReplicationAction;
import com.day.cq.replication.ReplicationEvent;

import java.util.HashMap;
import java.util.Map;

/**
 * Invalidates the dispatcher cache for pages that CONSUME an
 * Experience Fragment when that fragment is published.
 *
 * WHY THIS HAS TO EXIST AT ALL:
 *
 *   Statfile invalidation is PATH-BASED. Publishing
 *     /content/experience-fragments/energy/global/banner
 *   touches statfiles in THAT branch.
 *
 *   The pages that include the fragment live at
 *     /content/energy/us/en/...      (and 19 other countries)
 *   -- a completely different branch, whose statfiles are never
 *   touched. So those pages are never considered stale and keep
 *   serving the old fragment INDEFINITELY.
 *
 *   The dependency is not expressible as a path, so the dispatcher
 *   cannot see it, and NOTHING tells it. Something has to.
 *
 * NOTE: at a low statfileslevel this appears to work, because every
 * publish invalidates the whole cache as collateral damage. Raising
 * statfileslevel correctly is exactly when this bug surfaces.
 */
@Component(
        service = EventHandler.class,
        property = {
                EventConstants.EVENT_TOPIC + "=" + ReplicationEvent.EVENT_TOPIC
        })
public class ExperienceFragmentFlushListener implements EventHandler {

    private static final String XF_ROOT = "/content/experience-fragments/energy";

    @Reference
    private JobManager jobManager;

    @Override
    public void handleEvent(Event event) {

        // THIS METHOD RUNS ON THE EVENT DISPATCH THREAD.
        // A bulk publish can fire hundreds of these. Anything slow
        // here blocks event delivery for the whole instance
        // (file 10). So: a cheap check, then hand off. Nothing else.

        ReplicationEvent replicationEvent = ReplicationEvent.fromEvent(event);
        if (replicationEvent == null) {
            return;
        }

        ReplicationAction action = replicationEvent.getReplicationAction();
        if (action == null) {
            return;
        }

        String path = action.getPath();

        // Cheap relevance check: is this even an Experience Fragment?
        // Cheapest possible test, because it runs for EVERY
        // replication event on the instance, not just ours.
        if (path == null || !path.startsWith(XF_ROOT)) {
            return;
        }

        // Only publishes and unpublishes change what visitors see.
        if (action.getType() != ReplicationActionType.ACTIVATE
                && action.getType() != ReplicationActionType.DEACTIVATE) {
            return;
        }

        // HAND OFF TO A SLING JOB and return immediately.
        //
        // Three reasons this is a job rather than inline work:
        //   1. Finding the consuming pages is a repository query --
        //      far too slow for this thread.
        //   2. The invalidation is an HTTP call to the dispatcher,
        //      which can fail. A job RETRIES; an event handler that
        //      throws just loses the work silently.
        //   3. Jobs survive a restart. Events do not.
        Map<String, Object> props = new HashMap<>();
        props.put("fragmentPath", path);
        props.put("actionType", action.getType().getName());

        jobManager.addJob("energy/dispatcher/invalidate-xf-consumers", props);
    }
}

And the job that does the work, showing the invalidation request itself:

/**
 * Sends targeted dispatcher invalidations for the pages that
 * include a published Experience Fragment.
 *
 * The request is exactly what a flush agent sends:
 *
 *   POST /dispatcher/invalidate.cache
 *   CQ-Action: Activate
 *   CQ-Handle: /content/energy/us/en/products
 *   CQ-Action-Scope: ResourceOnly     <-- see below
 *
 * CQ-Action-Scope: ResourceOnly removes ONLY that resource's cached
 * file and does NOT touch statfiles. That is what makes this
 * surgical: we invalidate the twenty consuming pages, not the
 * twenty country branches they sit in.
 *
 * Without it we would be invalidating far more than we changed,
 * which is the re-render storm from story 3 all over again --
 * just triggered by us instead of by statfileslevel.
 */

The five decisions to be able to defend in that code:

The relevance check is the first thing and it is cheap. This handler runs for every replication event on the instance. A path prefix comparison is the right cost; a repository lookup would not be.

It dispatches a job rather than doing the work. The event dispatch thread is shared, a bulk publish fires hundreds of events, and the actual work involves a query and an HTTP call.

A job because the work can fail. An HTTP call to the dispatcher can time out. A job retries; an event handler that throws loses the work with nothing to show for it.

CQ-Action-Scope: ResourceOnly keeps it surgical. Without it we would touch statfiles for twenty country branches to invalidate twenty pages — creating the exact re-render storm we removed.

It handles deactivation as well as activation. A fragment being unpublished has to remove it from the consuming pages too, and forgetting that half is a common and awkward bug.

#8.5 Diagnostic commands

# ==================================================================
#  Working INWARD one layer at a time. Each command eliminates one
#  layer. Never flush everything at once -- it fixes the ticket and
#  teaches you nothing, so the ticket comes back.
# ==================================================================

# ---- Layer 1: the CDN --------------------------------------------
# A non-zero, growing "Age" means the CDN is serving from its cache.
# Many CDNs also send X-Cache: HIT / MISS.
curl -sSI https://www.example.com/us/en/products/transformers.html \
  | grep -iE 'age|cache-control|x-cache|surrogate|etag|last-modified'

# ---- Layer 2: the dispatcher, bypassing the CDN ------------------
# Fresh here but stale above => it is the CDN. Purge it.
curl -sSI -H "Host: www.example.com" \
  http://dispatcher-host/us/en/products/transformers.html

# ---- Layer 3: the dispatcher's disk ------------------------------
# THE TWO TIMESTAMPS. This single comparison is the most direct
# answer to "why is the dispatcher still serving this".
ls -la /opt/dispatcher/cache/content/energy/us/en/products/transformers.html
ls -la /opt/dispatcher/cache/content/energy/us/en/.stat

#   cached file NEWER than .stat -> considered VALID
#      => the invalidation never arrived.
#         Check /cache/allowedClients, the filter rule on
#         /dispatcher/invalidate.cache, and the flush agent log.
#   cached file OLDER than .stat -> considered STALE
#      => it will be re-fetched on the next request.

# Every statfile in the tree, newest first -- shows what was
# actually invalidated and when.
find /opt/dispatcher/cache -name ".stat" -printf '%T+ %p\n' | sort -r | head -20

# ---- Layer 4: publish directly -----------------------------------
# Stale here too => not a caching problem. The content was never
# published. That is file 18's investigation, not this one.
curl -sSI http://publish1.internal:4503/content/energy/us/en/products/transformers.html

# ==================================================================
#  Cache hit ratio, from logs you already have.
#  Everything appears in Apache's access_log.
#  Only MISSES reach publish and appear in its request.log.
# ==================================================================
APACHE_HTML=$(grep -c '\.html' /var/log/httpd/energy_access.log)
PUBLISH_HTML=$(grep -c '\.html' /opt/aem/publish/crx-quickstart/logs/request.log)
echo "hit ratio: $(( (APACHE_HTML - PUBLISH_HTML) * 100 / APACHE_HTML ))%"

# What is actually missing matters more than how much. Group the
# publish request log by URL -- this is usually decisive in a minute.
grep '\.html' /opt/aem/publish/crx-quickstart/logs/request.log \
  | awk '{print $NF}' | sort | uniq -c | sort -rn | head -20

# ==================================================================
#  Filter verification -- what the pipeline smoke test should do.
#  Every one of these MUST return 404 from the public site.
# ==================================================================
for path in \
  "/system/console/bundles" \
  "/crx/de/index.jsp" \
  "/crx/packmgr/index.jsp" \
  "/bin/querybuilder.json?path=/home&p.limit=-1" \
  "/content.infinity.json" \
  "/content/energy.tidy.json" \
  "/content/energy.sysview.xml" \
  "/content/energy.-1.json" \
  "/dispatcher/invalidate.cache"
do
  code=$(curl -sS -o /dev/null -w '%{http_code}' "https://www.example.com${path}")
  # Anything other than 404 is a finding, not a warning.
  [ "$code" = "404" ] && echo "OK   $code  $path" || echo "FAIL $code  $path"
done

# ==================================================================
#  Sending an invalidation by hand -- to test the plumbing, or to
#  invalidate something a normal flush cannot reach.
# ==================================================================

# Statfile invalidation for a branch
curl -sS -H "CQ-Action: Activate" \
        -H "CQ-Handle: /content/energy/us/en/products" \
        -H "Content-Type: text/plain" \
        -H "Content-Length: 0" \
        http://dispatcher-host/dispatcher/invalidate.cache

# SURGICAL: remove only this resource's cached file, statfiles
# untouched. This is how you invalidate one page without
# invalidating its siblings.
curl -sS -H "CQ-Action: Activate" \
        -H "CQ-Action-Scope: ResourceOnly" \
        -H "CQ-Handle: /content/energy/us/en/products/transformers.html" \
        -H "Content-Type: text/plain" \
        -H "Content-Length: 0" \
        http://dispatcher-host/dispatcher/invalidate.cache

# ==================================================================
#  AEM as a Cloud Service
# ==================================================================

# Validate the config BEFORE pushing. The pipeline runs the same
# validation, so this turns a failed pipeline into a 10-second
# local error. This is the step people skip.
./bin/validator full -d ./out ./dispatcher/src

# Run Apache + dispatcher locally in Docker against a local publish.
# The only way filter and cache bugs are visible during development
# -- develop against localhost:4502 and you will not see them until
# production.
./bin/docker_run ./out host.docker.internal:4503 8080

#9. Common Mistakes

#10. Best Practices

On filters. Deny everything, then whitelist deliberately. Constrain path, extension, selector and method together rather than any one in isolation. Treat the filter section as a security artefact that deserves review, not configuration you copy from the last project.

On caching. Design for cacheability before you write the component — selector versus suffix versus query parameter is an architecture decision (file 01). Ignore tracking parameters so campaign links don't fragment the cache. Use a grace period so a flush doesn't stampede publish.

On invalidation. Set statfileslevel to match your actual content structure. Restrict /allowedClients to the instances that legitimately flush. And remember derived output — listings, cached GraphQL responses, pages that include an Experience Fragment — needs an explicit invalidation relationship (files 15, 16, 17, 18).

On personalisation. Never cache per-visitor content. Isolate it with SDI or a client-side fetch so the rest of the page stays cacheable.

On operations. Config in Git, validated by the pipeline. Run the validator locally before pushing. Measure the hit ratio and treat it as a real metric rather than a curiosity.

On debugging. Always be able to hit publish directly, bypassing the dispatcher. That single capability separates half the problems in this file from the other half.

#11. Debugging Tips

The move that answers most questions: hit publish directly. In a private window, against the publish instance rather than through the dispatcher. If the content is correct there and wrong through the dispatcher, it's a cache problem. If it's wrong in both, it's a content or publishing problem (file 18). One request eliminates half the possibilities.

Read the response headers. They tell you which layer answered:

Raise the dispatcher log level. DispatcherLogLevel 3 gives you the hit/miss decision and — importantly — which filter rule denied a request. That turns "it 404s" into "rule 0012 denied it."

Look at the cache on disk. The docroot mirrors URL paths, so you can see whether a file exists and compare its timestamp against the statfile. That's the statfile model made concrete.

Distinguish the two causes of a clientlib 404. Both allowProxy being unset (file 04) and a missing dispatcher rule produce an identical 404. Check dispatcher.log: if the request never reached publish, it's the filter; if it reached AEM and AEM returned 404, it's allowProxy.

#12. Performance Notes

The hit ratio is the number that matters. Above 90% is healthy on a content site. Below that, the question is always why — and it's almost always either query parameters defeating the cache, or caching rules that exclude more than intended.

Every request the filter denies is a request publish never sees. Security and performance are the same lever here, which is worth saying because people treat them as separate.

A grace period prevents the stampede. Without one, a flush means every in-flight request misses simultaneously and hits publish at once. A few seconds of serving stale content while the first request re-renders is almost always the right trade.

Tracking parameters fragment the cache badly. Every campaign link with a different utm_ value is a distinct cache entry for identical content. Ignoring them for cache-key purposes is often the single biggest hit-ratio improvement on a marketing site.

statfileslevel is a real tuning decision. Too shallow and one publish invalidates everything; too deep and invalidation misses pages it should reach. It should match how your content is actually structured.

A large activation means a large flush (file 18), and a large flush means a re-render burst. Plan for it rather than discovering it.

#15. Comparison Tables

What the dispatcher does

Cacheable or not

The cache layers

6.5 vs Cloud Service

Statfile behaviour

#Next topic

20-JCR-Oak-and-the-Repository — the storage layer underneath everything: nodes and properties, node types, the cq:Page and jcr:content split, node stores and why AEMaaCS uses different ones per tier, and the repository maintenance that keeps 6.5 alive.

Topic 19 of the AEM Interview Preparation repository. Teaching style, energy-sector project domain.