Skip to content

Manager Replication

This page describes how a manager replicates objects from the endpoints it manages. For the manager system as a whole, see Manager Architecture. For enabling and disabling replication, see Single Pane and Version Set Replication.

Lazy Replication

As of v4.17, Single Pane replication is lazy. When a client reads an object owned by a managed endpoint, the manager fetches it from that endpoint and holds it in a short-lived cache. The manager stores no downstream objects in its own database.

Earlier versions replicated eagerly: each managed endpoint streamed its database changes upstream, and the manager maintained a complete replica of every managed endpoint in its own datastore. Reads were answered entirely from those replicas.

The consequences run in both directions. A manager's memory footprint no longer scales with the size of the fleet beneath it. In exchange, reads depend on downstream reachability and on network latency between tiers, and they can be briefly stale, but systems of all scales behave much more predictably.

Writes are unchanged: they are forwarded to the endpoint that owns the object.

Reading a Single Object

A read for one object by key resolves in three stages, cheapest first:

  1. The manager's own store. Objects the manager owns are answered locally, exactly as on a non-manager endpoint.
  2. The per-endpoint object caches. A cache hit serves the complete object and requires no downstream request at all, with the tradeoff of potential short-term staleness.
  3. A REST fetch from the owning endpoint, whose result is then cached for subsequent reads.

Reading Lists

A list request is issued to every connected endpoint and merged with the manager's own matching objects. Filters expressible as URL parameters are forwarded so each endpoint does its own filtering; filters that cannot be forwarded are applied to the merged result.

Results are cached on the manager temporarily for future lookups. This behavior makes the UX experience of paginated tables and subsequent object inspection indistinguishably fast from pre-v4.17 streaming replication.

Counts

Object counts are replicated with count-only requests that return no object bodies, and the result is cached for 60 seconds. Counts therefore lag reality by up to a minute.

This applies to the machine, subnet, context, and user counts in /api/v3/info, to drpcli machines count and its equivalents, and to license counts. An endpoint with no managed endpoints does no remote counting and its counts are always exact.

Consistency

Reads through a manager are eventually consistent. Lifetimes for certain cached resources are heuristically determined, so some staleness is to be expected.

Endpoint Availability

The manager only replicates from endpoints it currently has a working connection to. An endpoint that is down, unreachable, or still reconnecting is skipped.

This is the behavior change most likely to surprise an operator. Under the replica model the manager kept serving a managed endpoint's objects during an outage; it now omits them.

Version Set Replication

Version Set replication is a distinct mechanism and is unchanged by any of this. It pushes content from the manager down to endpoints and is controlled independently. See Single Pane and Version Set Replication.

See Also