Replication and Content Distribution#

#18 – Replication and Content Distribution

Target: 3–4 years experienced AEM Developer Syllabus point covered (27): "What is replication, and what are its uses?" Project domain: a global energy technology company's marketing site.

#Before we start — this file closes a loop

This is the last of your 27 syllabus points, and it closes something that has run through the whole repository.

"It works on author but not on publish" has appeared in files 02, 04, 05, 08, 13, 15 and 16. Each time, one of the causes was "was it actually published?" — and each time I deferred the mechanism to this file.

This is that mechanism. And the reason it deserves its own file rather than a footnote is that "was it published" has several distinct failure points, and being able to walk them in order is the answer to the most common production question in AEM:

"The author says they published it and it's not on the live site."

Section 3.3 is that walk. It's the single most reusable thing here.

There is also a genuine architectural shift to understand. On AEM 6.5 replication pushes to a known list of publish instances. On Cloud Service that model is impossible, because publish pods are created and destroyed by autoscaling — so it was replaced by a publish/subscribe model. Explaining why it had to change is a much better answer than knowing that it did.

#1. Introduction

#1.1 What replication is

Replication is how content moves between AEM instances — primarily from author to publish.

Authors work on the author instance. Visitors see the publish instance. Nothing an author does is visible to the public until it has been replicated.

"Activate" and "publish" are the same operation. "Activate" is the technical term used in the API, the agents and the logs; "Publish" is what the button says. Knowing they're the same avoids a moment of confusion when an interviewer switches between them.

#1.2 The four actions

Deactivate versus delete is worth being precise about: deactivating removes content from publish while leaving it on author, so it can be republished. Deleting removes it from both.

#1.3 What it's used for — beyond the obvious

Your syllabus asks about uses, and the answer is broader than "publishing pages":

Publishing content — pages, assets, Content Fragments, Experience Fragments. Each is replicated independently, which is exactly the trap from files 15 and 16.

Dispatcher cache invalidation — a flush agent tells the dispatcher that content has changed, so it stops serving the cached copy. This is a replication agent too, and people often don't realise it.

Unpublishing — deactivation, for content that must come down.

Reverse replication — moving content created on publish back to author. Rare in modern AEM, but it exists.

Triggering downstream work — replication fires events (file 10), so other systems can react to something being published.

#1.4 A real project example to adapt

"We're on Cloud Service, so content moves through Sling Content Distribution rather than replication agents — the author publishes into a pipeline and publish pods subscribe and pull. That change was necessary rather than cosmetic: replication agents push to a configured list of publish instances, and on Cloud Service pods are created and destroyed by autoscaling, so the author can't know about a pod that doesn't exist yet. With pub/sub, a new pod subscribes and catches up on its own.

From an author's point of view nothing changed — they still press Publish. What changed for us is monitoring: instead of watching an agent queue we look at Cloud Manager.

The thing that still catches people is that references publish separately. Publishing a product page doesn't publish the Content Fragment it renders or the Experience Fragment in its header, and that's the most common cause of 'it works on author'."

That covers the mechanism, why it changed, the operational difference, and the recurring trap — four follow-ups pre-empted.

#2. Core Concepts — AEM 6.5

#2.1 How replication works on 6.5

The flow, which is worth knowing even if you're on Cloud Service, because interviewers ask and because it explains why the cloud model differs:

sequenceDiagram participant A as Author participant Q as Replication queue participant T as Transport participant P as Publish participant D as Dispatcher A->>A: author clicks Publish (activate) A->>A: serialise the content A->>Q: add to the agent's QUEUE Q->>T: agent picks it up T->>P: HTTP POST to /bin/receive P->>P: deserialise and write to the repository P-->>T: 200 OK T->>Q: remove from the queue A->>D: FLUSH AGENT sends an invalidation D->>D: touch the statfile — cached copies now stale

Four things to draw out:

It's a queue. Activation doesn't happen instantly; it's queued and processed. Which means it can back up, and one item can block the ones behind it.

It's an HTTP POST to /bin/receive on the publish instance, authenticated as a transport user. So it's a network operation with credentials — and both can fail.

Publish confirms before the item leaves the queue. If publish doesn't respond, the item stays.

Dispatcher invalidation is separate. A different agent, sending a different request. Content reaching publish and the dispatcher serving it are two different things — which is why a page can be on publish and still not visible.

#2.2 Replication agents

Agents are configured under /etc/replication/agents.author and /etc/replication/agents.publish.

The configuration tabs worth knowing:

Settings — enabled, serialisation type, retry delay, the agent user (who the agent runs as on author, reading the content) and log level.

Transport — the URI (http://publish-host:4503/bin/receive), and the user and password it authenticates with on publish.

Triggers — including "On Modification", which auto-replicates when content changes rather than waiting for an explicit publish.

Two user identities, and confusing them is a real source of bugs:

The agent user reads the content on author. If it can't read something, that content isn't replicated.

The transport user authenticates on publish and needs permission to write there.

A permissions failure on either side breaks replication, and the symptom differs — the first silently skips content, the second blocks the queue with a 401.

#2.3 The flush agent and dispatcher invalidation

This is the piece people forget, and it explains a whole class of "I published it and it's still old."

When content is activated, a flush agent sends an invalidation request to the dispatcher. The dispatcher touches its statfile — a file called .stat — and any cached file older than that statfile is treated as stale and refetched on the next request.

Note it doesn't delete anything. Files stay on disk; they're simply outdated by comparison with the statfile.

statfileslevel controls how deep in the directory tree statfiles are maintained. A higher level means invalidation is more targeted; level 0 means one flush effectively invalidates everything.

Which is why publishing one page can invalidate its siblings — a statfile sits at a directory level, not per file. That's a good detail from file 01 arriving here with its mechanism.

#2.4 Reverse replication

Content created on publish — form submissions, comments, user-generated content — sometimes needs to reach author.

How it works: publish maintains an outbox, and the author's reverse replication agent polls it and pulls content back.

Note the direction of initiative. Normal replication is author pushing to publish. Reverse replication is author pulling from publish — because publish shouldn't be able to initiate connections into the author tier, which sits behind a firewall.

That firewall reasoning is the point worth making, and it's why the design is asymmetric.

In practice it's rare in modern AEM. Form submissions typically go to an external system or a form service instead, which is simpler and doesn't couple your publish tier to your author tier.

#2.5 Replication queues — where things go wrong

The queue is where production problems show up.

The head-of-line blocking point matters: a single problematic item at the front of the queue stops everything behind it. So "nothing is publishing" often means "one thing failed," and clearing that item releases the rest.

Testing an agent: each agent has a test link that sends a test replication, which confirms connectivity and credentials without publishing anything real.

#2.6 The replication API

For triggering replication from code:

@Reference
private Replicator replicator;

Session session = resourceResolver.adaptTo(Session.class);
replicator.replicate(session, ReplicationActionType.ACTIVATE, path);

ReplicationActionType: ACTIVATE, DEACTIVATE, DELETE, TEST.

Two things to say about doing this in code:

The session's user needs crx:replicate — from file 13. Replication runs as somebody, and that somebody needs permission to publish.

Bulk replication from code is dangerous. Activating a thousand pages in a loop floods the queue and can effectively stall publishing for everything else. That belongs in a Sling Job with batching (file 10), not a tight loop.

#2.7 References publish separately — the recurring trap

This is the single most useful practical point in this file, because it explains a symptom that has appeared in six earlier files.

Publishing a page does not publish what the page references.

AEM's publish dialog does show references and offers to include them. But it's easy to skip, and programmatic activation doesn't offer anything — it publishes exactly what you tell it to.

Why the design is right, even though it surprises people: a Content Fragment might be referenced by fifty pages. Tying its publication to any one of them would be wrong. Each thing has its own lifecycle deliberately.

And the failure is silent. The page publishes successfully. The component that renders the missing fragment gets a null and renders nothing (file 05's OPTIONAL injection). No error anywhere.

#3. Core Concepts — AEM as a Cloud Service

#3.1 Why replication agents couldn't survive

This is the explanation that makes the answer good rather than memorised.

A replication agent has a transport URI — a specific publish instance to push to. On 6.5 you configure one agent per publish instance, because you know how many there are and where they live.

On Cloud Service, neither is true. Publish pods are created and destroyed by autoscaling. During a traffic spike new pods appear; afterwards they disappear.

Two things break:

The author can't push to a pod that doesn't exist yet. You can't configure an agent for it.

A newly created pod has no content. It needs to catch up on everything published so far, and nothing is going to re-push it.

So the model had to invert: instead of the author pushing to known instances, the author publishes into a pipeline and pods subscribe and pull.

#3.2 Sling Content Distribution

graph TD A["Author: publish"] --> B["Adobe-managed<br/>pub/sub pipeline"] B --> C["Publish pod 1<br/>subscribes and pulls"] B --> D["Publish pod 2<br/>subscribes and pulls"] B --> E["NEW pod<br/>subscribes and CATCHES UP"] B --> F["Preview service"]

The key property: a new pod subscribes and catches up by itself. Nothing has to know it exists in advance, which is exactly what autoscaling requires.

What this changes for you:

That last row matters. Authors see no difference — the button is the same. This is an infrastructure change, not a workflow one.

#3.3 The Preview tier

Cloud Service adds a Preview service (file 14) — content can be distributed there separately, so authors see a genuine publish-like rendering without exposing it publicly.

#3.4 "It's published and it's not live" — the diagnostic walk

The most reusable thing in this file. Six checks, in order, and the order matters because each one eliminates a layer.

graph TD A["Author says it's published,<br/>the live site shows otherwise"] --> B{"1. Was it ACTUALLY activated?<br/>page properties → replication status"} B -->|No| C["It was saved, not published"] B -->|Yes| D{"2. Is the REPLICATION QUEUE<br/>blocked or paused?"} D -->|Blocked| E["One stuck item blocks<br/>everything behind it"] D -->|Clear| F{"3. Does the node exist<br/>ON PUBLISH?"} F -->|No| G["Replication reported success<br/>but content didn't land"] F -->|Yes| H{"4. Were the REFERENCES published?<br/>fragments, assets, XFs"} H -->|No| I["The page is there;<br/>what it renders is not"] H -->|Yes| J{"5. Is the DISPATCHER<br/>serving a cached copy?"} J -->|Yes| K["Flush it — content is on publish,<br/>the cache doesn't know"] J -->|No| L{"6. Is the CDN caching it?"} L -->|Yes| M["Purge the CDN"]

Why this order: each step is cheaper to check than the next, and each eliminates a whole layer. Steps 1 and 5 are the two most common by a wide margin — "it wasn't actually published" and "the cache doesn't know yet".

And step 4 is the one that connects to every other file — the references trap.

#8. Configuration and Code

#8.1 A replication agent (6.5)

/etc/replication/agents.author/publish/jcr:content

<?xml version="1.0" encoding="UTF-8"?>
<jcr:root xmlns:jcr="http://www.jcp.org/jcr/1.0"
          xmlns:cq="http://www.day.com/jcr/cq/1.0"
    jcr:primaryType="nt:unstructured"
    jcr:title="Default Agent"
    sling:resourceType="cq/replication/components/agent"
    enabled="{Boolean}true"

    <!-- THE AGENT USER: reads content on AUTHOR.
         If it cannot read something, that content is silently NOT
         replicated. Under-privileging here fails quietly. -->
    userId="replication-agent-user"

    <!-- THE TRANSPORT: where to POST, and who to authenticate as
         on PUBLISH. This user needs to WRITE there.
         Wrong credentials → 401 → the QUEUE BLOCKS. -->
    transportUri="http://publish-host:4503/bin/receive?sling:authRequestLogin=1"
    transportUser="replication-receiver"
    transportPassword="{encrypted}"

    <!-- Triggers. "On Modification" would auto-replicate on every
         save, which is rarely what you want -- it publishes
         half-finished edits. -->
    triggerModified="{Boolean}false"
    triggerDistribute="{Boolean}false"
    triggerOnOffTime="{Boolean}true"

    retryDelay="{Long}60000"
    logLevel="info"

    serializationType="durbo"/>

The two user identities are the thing to be able to explain. The userId reads on author; the transportUser writes on publish. They fail differently — the first silently skips content, the second blocks the queue with a 401.

#8.2 A flush agent

/etc/replication/agents.author/flush/jcr:content

<?xml version="1.0" encoding="UTF-8"?>
<jcr:root jcr:primaryType="nt:unstructured"
    jcr:title="Dispatcher Flush"
    sling:resourceType="cq/replication/components/agent"
    enabled="{Boolean}true"

    <!-- Points at the DISPATCHER, not at publish.
         This is how the dispatcher learns that content changed --
         it touches its statfile, and any cached file older than
         the statfile is treated as stale. Files are NOT deleted;
         they are outdated by comparison. -->
    transportUri="http://dispatcher-host/dispatcher/invalidate.cache"

    serializationType="flush"
    protocolHTTPMethod="GET"
    protocolHTTPHeaders="[CQ-Action:{action},CQ-Handle:{path},CQ-Path:{path}]"/>

The point worth making: the flush agent is a replication agent, which surprises people. Content reaching publish and the dispatcher knowing about it are two separate operations, and this is the second one.

#8.3 Replicating from code

package com.energy.core.services.impl;

import com.day.cq.replication.ReplicationActionType;
import com.day.cq.replication.ReplicationException;
import com.day.cq.replication.Replicator;
import org.apache.sling.api.resource.ResourceResolver;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.Reference;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.jcr.Session;
import java.util.List;

@Component(service = PublishService.class)
public class PublishServiceImpl implements PublishService {

    private static final Logger LOG = LoggerFactory.getLogger(PublishServiceImpl.class);

    @Reference
    private Replicator replicator;

    @Override
    public void publish(ResourceResolver resolver, String path) {
        Session session = resolver.adaptTo(Session.class);
        if (session == null) {
            return;
        }
        try {
            // The session's USER needs crx:replicate (file 13).
            // A service user doing this needs it granted explicitly.
            replicator.replicate(session, ReplicationActionType.ACTIVATE, path);

        } catch (ReplicationException e) {
            LOG.error("Failed to activate {}", path, e);
        }
    }

    /**
     * DO NOT do this for large volumes.
     *
     * Activating hundreds of paths in a loop floods the replication
     * queue, which stalls publishing FOR EVERYONE ELSE while it
     * drains -- and then triggers a cache flush large enough to cause
     * a re-render burst on publish.
     *
     * For bulk work: Sling Jobs with batching and capped queue
     * concurrency (file 10), run out of hours, REFERENCES FIRST.
     */
    @Override
    public void publishMany(ResourceResolver resolver, List<String> paths) {
        throw new UnsupportedOperationException(
                "Use the batched job-based publisher — see BulkPublishJobConsumer");
    }
}

#8.4 Reacting to a publish event

From file 10, and it's the answer to the derived-cache problem:

@Component(service = EventHandler.class,
           property = { EventConstants.EVENT_TOPIC + "=" + ReplicationAction.EVENT_TOPIC })
public class ProductPublishHandler implements EventHandler {

    @Reference
    private JobManager jobManager;

    @Override
    public void handleEvent(Event event) {
        ReplicationAction action = ReplicationAction.fromEvent(event);
        if (action == null || action.getType() != ReplicationActionType.ACTIVATE) {
            return;
        }

        String path = action.getPath();
        if (path == null || !path.startsWith("/content/energy")) {
            return;
        }

        // DECIDE here, DO in the job.
        //
        // This runs SYNCHRONOUSLY on the event dispatch thread. During
        // a bulk publish there may be hundreds of these, so anything
        // slow here blocks event processing globally (file 10).
        //
        // Invalidating a derived listing may mean an HTTP call to the
        // dispatcher -- absolutely not on this thread.
        Map<String, Object> props = new HashMap<>();
        props.put("path", path);
        jobManager.addJob("energy/listing/invalidate", props);
    }
}

Why this exists at all: publishing content invalidates the page it's on. It does not invalidate anything derived from it — a listing built from the page tree, a cached GraphQL response, a page that includes an Experience Fragment. That connection has to be made explicitly, and nothing makes it for you.

#9. Common Mistakes

#10. Best Practices

On publishing. References first, then pages. Use the publish dialog's reference list rather than clicking past it. Keep crx:replicate in a separate group from editing.

On bulk. Never a loop. Batched Sling Jobs with capped concurrency, out of hours, and plan for the cache flush that follows.

On monitoring. Alert on queue depth, not just errors — it rises long before anything visibly breaks.

On verification. Check the way a visitor sees it — through the dispatcher and CDN — not just on publish. Especially for takedowns.

On derived content. Anything cached that's derived from content needs an explicit invalidation relationship. A replication event handler that dispatches a job, never doing the work inline.

On security. A dedicated, minimally-privileged transport user. /bin/receive blocked from outside. And an agent user that can read everything it needs, because under-privileging there fails silently.

#11. Debugging Tips

The six-step walk from section 3.4 is the main tool. Was it activated → is the queue blocked → does the node exist on publish → were references published → is the dispatcher caching it → is the CDN.

The key technique is checking publish directly, in a private window, bypassing the dispatcher. That single move separates a content problem from a caching problem, which otherwise present identically. It's the same technique as files 04, 13 and 15 — the recurring diagnostic in this repository.

Where to look:

A habit worth describing: "When content isn't live, my first question is always 'is this a content problem or a cache problem?' — and hitting publish directly answers it in one step. Everything after that is much more targeted."

#12. Performance Notes

Replication is queued, so it's throughput-limited. Flooding the queue doesn't make it faster; it makes everything else wait.

Large binaries are the usual cause of a stuck queue — a transfer that times out and retries at the head.

A large activation means a large flush, and a large flush means a re-render burst on publish as caches refill. A dispatcher grace period helps, serving stale content briefly rather than every request stampeding at once.

Replication runs on author, so bulk work competes with authoring. Out of hours for anything substantial.

Queue depth is the metric to watch, for the same reason as workflow instance counts in file 09 — it's a leading indicator that rises before anything visibly fails.

#14. Comparison Tables

6.5 replication vs Cloud Service distribution

Agent types

The two users

Replication actions

What publishes separately

#That completes your syllabus

All 27 points from your list are now covered across files 01–18.

Next comes the supplementary set from your additional list — Dispatcher in depth, Oak and Query Builder, JCR indexes, Core Components, the Style System, GraphQL, Cloud Manager and CI/CD, unit testing with JUnit and Mockito, SonarQube and code quality, security best practices, Java fundamentals for AEM developers, and system design basics.

Then the README with the roadmap and the 30, 60 and 90-day plans, written last so it reflects the actual file count.

Topic 18 of the AEM Interview Preparation repository. Teaching style, energy-sector project domain.