Skip to content

Work Order Mode

This document describes Work Orders running in the default task mode: one-shot, queued units of work executed by Machines, Clusters, and Resource Brokers that have been placed into WorkOrderMode. It covers the models, the lifecycle, the runner's role, cancellation and deletion, the API surface, and the usage patterns this mode enables.

For the other WorkOrder mode — multi-phase, restartable, pipeline-driven execution — see WorkOrder Workflow Mode. For a conceptual comparison of all three runner modes see Runner Modes.

The Models

WorkOrder (task mode)

A WorkOrder with Mode = "task" (the default) is a single, non-restartable unit of work targeted at one Machine (or Cluster, or Resource Broker). Its Tasks list is derived from a Blueprint at pickup time and executed once. When the WorkOrder terminates it is done — there is no automatic chaining, no successor concept, and no Predecessor / pipeline-uuid machinery.

Fields that matter in task mode:

  • Mode = "task" (default).
  • Blueprint: the source of the Tasks list, Meta, Profiles, and Params applied at pickup.
  • Machine (bound) or Filter (deferred). Exactly one must be set at creation time — see Machine Binding.
  • Tasks: normally left empty at creation and materialized from the Blueprint on the created → running transition. Late binding means editing the Blueprint before pickup changes what will run.
  • Params, Profiles: merged onto the Machine at pickup for the duration of the run.
  • State: created, running, finished, failed, or cancelled.

The workflow-mode-only fields (OnFailure, Predecessor, PipelinePhase, Meta["pipeline-uuid"]) are ignored in task mode.

Blueprint

A Blueprint used by a task-mode WorkOrder contributes:

  • Tasks: copied verbatim into the running WorkOrder.
  • Meta: merged under the WorkOrder's own Meta.
  • Params, Profiles: layered into the Machine's parameter resolution for the duration of the run.

Blueprint fields specific to workflow mode (Pipeline, Workflow, Context) are ignored when the Blueprint is applied to a task-mode WorkOrder.

Machine.WorkOrderMode

The boolean Machine.WorkOrderMode field is the switch that puts a Machine (or Cluster, or Resource Broker — both are Machines with role metadata) into WorkOrder processing. Two companion counters help operators and the runner observe queue depth:

  • Machine.PendingWorkOrders: count of created-state WorkOrders targeting this Machine, either bound directly or matched via Filter.
  • Machine.RunningWorkOrders: count of running-state WorkOrders for this Machine.

Both counters are read-only from the API; the server maintains them.

Machine Binding

A WorkOrder is targeted at a Machine in one of two ways, chosen at creation time. The frontend rejects a WorkOrder that sets both, or neither, of these fields:

Bound (Machine set). The WorkOrder is immediately associated with the named Machine. On create, the server increments the Machine's PendingWorkOrders counter and validates that the Machine has WorkOrderMode = true.

Deferred (Filter set, Machine empty). The WorkOrder is unbound at creation. The server walks the filter once to bump PendingWorkOrders on candidate Machines so their agents wake up. The WorkOrder remains created until the pick endpoint (see Runner Behavior) matches it to one of those Machines and stamps WorkOrder.Machine at that time.

Deferred WorkOrders are the primitive that Triggers with QueueMode and runner-pool patterns rely on.

Lifecycle

Creation

A WorkOrder is created via the WorkOrder API or spawned by other mechanisms (Trigger sinks, Batches). It starts in state created. If Mode is omitted it defaults to "task".

Pickup and Blueprint expansion

The created → running transition happens inside the pick endpoint (POST /machines/{uuid}/pick/agent01) when a Machine's agent claims the WorkOrder. On this transition the server:

  1. Resolves WorkOrder.Blueprint. The Blueprint must exist and be Available.
  2. Copies Blueprint.Tasks into the WorkOrder's Tasks list.
  3. Merges Blueprint.Meta into the WorkOrder's Meta.
  4. Inlines any stage:<name> entries into the corresponding Stage's Tasks (and adds bootenv: markers if the Stage sets one — see below).
  5. Prepends task prerequisites.
  6. Validates each entry resolves and — task-mode-specific — rejects any bootenv: entries. Task-mode WorkOrders are not allowed to change the Machine's BootEnv.

If any step fails, the WorkOrder is force-cancelled with a diagnostic message. On success the WorkOrder is set to running and returned to the agent.

Execution

The agent runs the WorkOrder's Tasks list one Job at a time, using the same task runner as Workflow mode. Two constraints apply:

  • Task-mode Jobs may not use exit_reboot, exit_shutdown, exit_stop, or the combined exit_incomplete_reboot / exit_incomplete_shutdown codes. See Reboots and Task Exits below.
  • Each Job is stamped with Job.WorkOrder = <uuid> so historical Jobs can be traced back to the WorkOrder that produced them.

Terminal state

A WorkOrder becomes terminal on transition to finished, failed, or cancelled. In task mode this is a plain state change — the server does no successor spawning, no OnFailure dispatch, and no pipeline-uuid clearing. The Machine's PendingWorkOrders or RunningWorkOrders counter is decremented as appropriate.

A terminal WorkOrder is inert: further state transitions are a no-op.

Runner Behavior

The Machine Agent's top-level dispatch branches on Machine.WorkOrderMode:

  • WorkOrderMode = false: the agent follows the Workflow state machine described in Workflow Mode.
  • WorkOrderMode = true: the agent enters the WorkOrder loop — AGENT_WAIT_FOR_WORKORDER → AGENT_PICK_WORKORDER → AGENT_RUN_TASK → AGENT_WAIT_FOR_RUNNABLE → repeat. The Workflow field on the Machine is ignored: if a Machine has both a Workflow set and WorkOrderMode = true, the runner takes the WorkOrder path and the Workflow never advances.

The pick call is POST /machines/{uuid}/pick/agent01 with an optional mode= query. Task-mode agents pass no mode (or mode=task), matching WorkOrders whose Mode != "workflow". Workflow-mode agents pass mode=workflow. This is how the two loops partition the queue.

The pick endpoint's matching order is:

  1. A running WorkOrder already on this Machine (idempotent re-fetch).
  2. A created WorkOrder bound to this Machine (Machine == mid).
  3. A created WorkOrder with Machine == "" and a Filter that matches this Machine. On match, the server sets WorkOrder.Machine = mid and transitions to running.

Between picks the agent waits on the Machine event stream for Available && Runnable && (WorkOrderMode == false || PendingWorkOrders > 0) so idle Machines do not spin.

Reboots and Task Exits

Task-mode WorkOrders are one-shot and cannot span reboots. The runner disallows these exit-code actions inside a WorkOrder:

  • exit_reboot (64)
  • exit_shutdown (32)
  • exit_stop (16)
  • exit_incomplete_reboot (192)
  • exit_incomplete_shutdown (160)

exit_incomplete (128) alone is allowed. Any of the above codes returned from a task inside a task-mode WorkOrder is treated as a failure. If a WorkOrder's work needs to reboot the Machine, that work belongs in a Workflow or in a workflow-mode WorkOrder — not a task-mode one.

Cancellation and Deletion

There is no dedicated cancel endpoint. Cancelling a WorkOrder is done by updating its State to cancelled via the normal WorkOrder PATCH or PUT endpoint. The transition is accepted from either created or running.

Deleting a live WorkOrder is blocked; the server rejects DELETE on any non-terminal WorkOrder. To remove one, cancel it first, then delete it.

Bulk deletion is available at DELETE /work_orders. It retains up to jobsToKeep-worth of terminal WorkOrders per Machine and force-removes orphaned rows whose Machine no longer exists. The two related prefs are:

  • defaultWorkOrdersToKeep: default number of terminal WorkOrders to retain per Machine.
  • Machine-scoped equivalents override the default.

Two additional server-initiated cancels can occur:

  • During Machine migration, all live WorkOrders on the migrating Machine are auto-cancelled.
  • During Blueprint-expansion failure at pickup, the WorkOrder is force-cancelled with a diagnostic in its log.

API Surface

WorkOrders

  • GET | POST | PATCH | PUT | DELETE /api/v3/work_orders[/{uuid}] — standard CRUD. POST requires exactly one of Machine or Filter. PATCH/PUT on State is how cancellation is performed. DELETE on a live WorkOrder is rejected.
  • DELETE /api/v3/work_orders — bulk retention-aware cleanup.

Machine pickup

  • POST /api/v3/machines/{uuid}/pick/agent01[?mode=task|workflow] — the runner's WorkOrder claim endpoint. mode=task (or omitted) is the task-mode caller; mode=workflow is documented in WorkOrder Workflow Mode.

Restart

POST /api/v3/work_orders/{uuid}/restart exists but is workflow-mode-only. Restarting a task-mode WorkOrder is not supported; if the same work needs to run again, create a new WorkOrder.

Usage Patterns

Task mode enables three broad usage patterns. All three can coexist in a single DRP environment.

Machine as Service

Once a Machine, Cluster, or Resource Broker has been provisioned, an operator can flip it into WorkOrderMode. From then on the system stops running Workflows and processes WorkOrders instead. The goal is to operate the provisioned object as a service: Blueprints represent named actions the operator can take against it, and WorkOrders are the individual invocations.

For example, a Machine that represents a switch might use WorkOrderMode to handle port-configuration changes as needed based on other systems' needs. A Cluster that represents a Kubernetes cluster might use WorkOrderMode to manage applications and other cluster-management functions after initial provisioning. Blueprints hold the tasks and initial parameters for those actions; WorkOrders are enqueued against the target when the action needs to run. The Machine / Cluster / Resource Broker object continues to represent an element in the data center that is being managed.

Orchestrated Tasks

A Trigger with QueueMode = true behaves as a deferred WorkOrder queue: its Filter defines which Machines are eligible workers, and its FilterCount sets how many deferred WorkOrders each event produces. Any Machine in the eligible set can claim the next one via its pick call.

This is subtly different from the Machine-as-Service model. The workers, grouped by Filter, are not addressed individually; they operate anonymously as a pool. This suits queues of homogeneous work where the operator does not need to name a specific worker up front.

CI/CD Pipelines

Triggers combined with task-mode WorkOrders let the DRP environment act as a general action-execution environment. A set of workers — Machines, Clusters, or Resource Brokers, backed by physical hosts, VMs, or containers — are started with tags. The workers run WorkOrders generated by Triggers to perform work that is not about managing DRP infrastructure directly.

Examples:

  • A Blueprint that repackages content on a GitLab push trigger, to rebuild and redeploy system components.
  • A Blueprint that runs heartbeat validation of a Machine on a cron-based Trigger, for monitoring components.

Interactions with Batches

Batches (backend) fan a Blueprint out across many Machines. A Batch may spawn either task-mode or workflow-mode WorkOrders depending on the Batch's WorkOrderTemplate.Mode:

  • Task-mode Batch fan-out: one one-shot WorkOrder per target Machine. On failure, the individual WorkOrder is terminal — the Batch does not restart it. If the Batch itself is restarted, all Machines are re-targeted (task mode has no per-Machine restart concept the way workflow mode does).
  • Workflow-mode Batch fan-out: covered in WorkOrder Workflow Mode, which documents per-Machine restart of only the failed subset.