Skip to content

WorkOrder Workflow Mode

This document describes the lifecycle and internals of a Work Order running in workflow mode: how it drives a machine through a Pipeline, how phase progress is recorded, how restart-via-successor works, and how Batches fan the same mechanism out across a fleet.

The two other modes of execution are described elsewhere:

The Models

Pipeline

A Pipeline is a declarative object that names a multi-phase execution chain and carries the Params and parent Pipeline references that drive it. A Pipeline has:

  • A Name (its key).
  • A Pipelines []string list of parent Pipelines. Parents are resolved leaf-to-root with cycle detection, bounded to a depth of
    1. The resolved list is flattened in declared order.
  • Standard ParamData (Params), ProfileData (Profiles), and the usual mixins (validation, access, metadata, ownership, bundling, documentation).

A Pipeline has no state, no timing, and no execution history. It is a parameterization vehicle. When a Machine has Machine.Pipeline set to a Pipeline's Name, that Pipeline and its resolved parents are inserted into the Machine's param aggregation stack as a distinct layer just above the Profiles layer, so a Pipeline's Params override Profiles but are overridden by Reservations, Stages, Blueprints, the Machine itself, and the WorkOrder.

At most one Pipeline may be active on a Machine at a time. Setting Machine.Pipeline to a name that does not resolve is an error.

WorkOrder (workflow mode)

A WorkOrder with Mode = "workflow" is a task-less run tracker. It carries the identity of a pipeline run and the operator's restart policy, but does not itself drive execution task-by-task the way a task-mode WorkOrder does. Instead, applying it to a Machine sets Machine.Pipeline, Machine.Workflow, and Machine.Context from the WorkOrder's Blueprint; the Machine then runs its chain workflow as it would for any other Workflow.

Workflow-mode-specific fields:

  • Mode = "workflow"
  • OnFailure (required): stop (default), pause, or the reserved values continue/retry. See Failure Handling.
  • Predecessor uuid.UUID (omitempty): UUID of the prior WorkOrder in a restart chain. Empty for the head of a pipeline run.
  • PipelinePhase []PipelinePhaseRecord: appended-to as the Machine crosses phase boundaries.
  • Meta["pipeline-uuid"]: the identity of the pipeline run, inherited by successors so that all restart-linked WorkOrders share a single pipeline-uuid.

The task-list validation forbids bootenv: and chroot: entries in a WorkOrder's own Tasks list; Machine BootEnv changes flow through the Machine's chain workflow instead.

PipelinePhaseRecord

The PipelinePhase list on a WorkOrder is composed of PipelinePhaseRecord entries with:

  • Phase: one of prepare, discover, hardware, os, or app. (The conceptual complete handoff and the pipeline bookend are not recorded as phase entries.)
  • Context: the Context the phase runs in, read from the resolved universal/<phase>-phase-context param. The Machine's own Context has not yet switched when the phase entry is written.
  • StartTime, EndTime: phase timing. An open (still-running) phase has EndTime omitted from the JSON representation.

Records are written by the server (not by tasks) whenever the Machine's Workflow field is set to a value matching the phase boundary pattern universal-{prepare,discover,hardware,os,app}-{start,complete}, and the Machine has an active workflow-mode WorkOrder. The bookend names universal-pipeline-start and universal-pipeline-complete are explicitly excluded from the pattern.

Blueprint (workflow-mode fields)

A Blueprint used by a workflow-mode WorkOrder sets one or more of:

  • Pipeline: the Pipeline name to apply to the Machine.
  • Workflow: the entry Workflow to run. Defaults to universal-pipeline-start if unset.
  • Context: the initial Machine Context.

A Blueprint is "pipeline-driven" for the purposes of this document if it has either Pipeline or Workflow set. A workflow-mode WorkOrder against a pipeline-driven Blueprint is what triggers the lifecycle described below.

Lifecycle

1. Creation

A workflow-mode WorkOrder is created via the normal WorkOrder API. It starts in state created and must specify a Machine, a Blueprint, Mode = "workflow", and an OnFailure policy. It may be created directly, spawned by machine creation (POST /machines?mode=<mode> with the appropriate default-blueprint pref), spawned by a Pool transition action, or spawned by a Batch.

2. Transition to running

On the first transition to running, the server calls applyPipelineBlueprint, which:

  • Sets Machine.Pipeline to the Blueprint's Pipeline.
  • Sets Machine.Workflow to the Blueprint's Workflow, or to universal-pipeline-start if the Blueprint's Workflow is empty. Setting the Workflow to the same value it already had is treated as a re-run: Machine.CurrentTask is reset to -1 to force re-materialization of the Tasks list from the entry Workflow.
  • Sets Machine.Context to the Blueprint's Context.
  • Sets Machine.Runnable = true.
  • Increments Machine.PendingWorkOrders and — at head of chain — calls ensurePipelineUUID to generate a fresh Meta["pipeline-uuid"] on the WorkOrder and stamp it onto the Machine as well.

Because the WorkOrder's Tasks list is empty in workflow mode, the server does not materialize it into Machine.Tasks. The Machine's chain workflow drives execution from here on.

3. Execution via the Machine's chain workflow

The Machine now behaves as if it were running a normal Workflow: the Agent's state machine (see Workflow Mode) walks Machine.Tasks, issuing POST /api/v3/jobs requests to obtain the next Job. Two things differ from a bare Workflow run:

  • TaskState pivots. The server's MakeJobState recognizes that the machine has an active workflow-mode WorkOrder and drives the Machine's TaskState directly, rather than the WorkOrder's. This is what allows BootEnv and Stage changes to flow through a WorkOrder run.
  • Jobs carry pipeline identity. Each new Job snapshots Machine.Pipeline into Job.Pipeline and Machine.Meta["pipeline-uuid"] into Job.Meta["pipeline-uuid"] at creation, so historical Jobs remain traceable to the pipeline run even if Machine.Pipeline is later changed.

4. Phase transitions

As the chain advances, content tasks (typically universal-chain-workflow) set Machine.Workflow to the next per-phase entry or exit workflow. See Phase-Based Chaining in the Universal Workflow doc for the content-side implementation. Every write to Machine.Workflow triggers recordPipelinePhaseTransition on the server:

  • If the new Workflow name matches the phase-boundary pattern, and the Machine has an active workflow-mode WorkOrder, a PipelinePhaseRecord is either appended (on a -start name) or sealed by setting its EndTime (on a -complete name).
  • The bookend names universal-pipeline-start and universal-pipeline-complete are recognized as pipeline boundaries for other purposes but do not themselves append records.
  • The Context recorded on each phase record is read from the resolved Param universal/<phase>-phase-context.

5. Terminal state

A WorkOrder becomes terminal on transition to finished, failed, or cancelled. The server's onWorkflowModeTerminal:

  • Decrements Machine.PendingWorkOrders.
  • On finished or cancelled, clears Machine.Meta["pipeline-uuid"].
  • On failed, keeps the pipeline-uuid in place so an operator can restart, and applies the OnFailure policy.

A workflow-mode Job failure additionally halts the Machine inline: Machine.Runnable is set to false and the Machine's TaskState is marked non-runnable. This prevents the Agent from continuing after a failure without an explicit operator action.

Failure Handling

OnFailure on a workflow-mode WorkOrder controls what happens when the Machine fails a Job:

  • stop (default): terminal. The WorkOrder stays failed; the Machine is left un-runnable. An operator may restart the WorkOrder (see Restart) or cancel it.
  • pause: the run is held until an operator triggers a restart. Behaves the same as stop at the point of failure — both leave the WorkOrder failed and the Machine un-runnable — but signals a different operational intent.
  • continue, retry: reserved for automatic successor spawning. Currently behave as stop.

WorkOrders owned by a Batch are forced to stop regardless of their declared OnFailure; a Batch is restarted at the Batch level rather than per-WorkOrder (see Batch Fan-out).

Restart

A failed workflow-mode WorkOrder is restarted by POST /work_orders/{uuid}/restart with an optional JSON body {"mode": "retry" | "continue" | "from-start"}. The default is retry. Batch-owned WorkOrders and non-workflow-mode WorkOrders cannot be restarted at the WorkOrder level.

Successor construction

Restart does not modify the failed WorkOrder in place. Instead the server calls BuildSuccessor(predecessor, mode), which constructs a new WorkOrder with:

  • A fresh UUID and CreateTime.
  • Predecessor set to the failed WorkOrder's UUID.
  • Blueprint, Machine, Profiles, and Params inherited from the predecessor. Params are deep-copied so successor mutations do not affect the predecessor.
  • Meta["pipeline-uuid"] inherited from the predecessor, so the entire restart chain shares a single pipeline-uuid.
  • PipelinePhase reset to empty (the successor starts a fresh phase log).
  • A Tasks list derived from the restart mode:
    • retry: tasks from predecessor.CurrentTask onward. The failed task will re-run.
    • continue: tasks from predecessor.CurrentTask + 1 onward. The failed task is skipped.
    • from-start: nil. The Tasks list is re-derived from the Blueprint the next time the WorkOrder is set Runnable, giving a clean run of the pipeline from its entry point.

The successor is persisted in created; from there it follows the normal Lifecycle steps above. The predecessor remains in place as failed for historical reference.

Restart chain identity

The set of WorkOrders linked by Predecessor from head to current represents one logical pipeline run across restarts. All members share the same Meta["pipeline-uuid"]. The /machines/{uuid}/pipeline-execution[/{pipeline-uuid}] endpoint (see API Surface) returns this set head-first.

Batch Fan-out

A Batch is the fan-out primitive for workflow-mode WorkOrders. A Batch owns three templates: a SetupWorkOrderTemplate, a WorkOrderTemplate, and a PostWorkOrderTemplate. When the Batch's setup WorkOrder completes, the server materializes the work template per target Machine.

Fresh batches

For a fresh Batch (no Predecessor), the server:

  • Runs SetupWorkOrderTemplate as an owned setup WorkOrder.
  • On setup completion, calls spawnBatchWorkOrders, which resolves the WorkOrderTemplate.Filter to a set of target Machines, excluding any Machine already mid-pipeline (non-empty Machine.Meta["pipeline-uuid"]), and creates one workflow-mode WorkOrder per remaining Machine.
  • Each spawned WorkOrder inherits Mode, OnFailure, and Blueprint from the template and is tagged in Meta as owned by the Batch.

Restarted batches

POST /batches/{uuid}/restart restarts a Batch. The original Batch must be in state failed or finished. The server:

  • Deep-copies the three template WorkOrders and links the new Batch via Predecessor to the original.
  • Stamps the UUIDs of the failed Machines from the predecessor Batch onto the setup template's Meta["batch/restart-machines"], so a setup task can consume that list.
  • When the new setup completes, spawnBatchWorkOrders walks the predecessor Batch's work WorkOrders and creates a successor WorkOrder for each failed one, targeting the same Machine and inheriting the failed WorkOrder's pipeline-uuid. Non-failed predecessors are not re-run.

This is the fan-out equivalent of WorkOrder-level restart: same Predecessor and pipeline-uuid inheritance semantics, but scoped to the subset of Machines that actually failed.

Param Aggregation

Adding Pipelines introduces a new layer to the Machine's param resolution stack. The full order (lowest priority first) is:

  1. Profiles
  2. Pipeline — the active Machine.Pipeline and its resolved parents, flattened in declared order
  3. Reservations / Leases / etc.
  4. Stage
  5. Blueprint
  6. Machine
  7. WorkOrder

A Pipeline's Params therefore override Profile Params but are overridden by anything from the Stage layer on up. Within the Pipeline layer, a child Pipeline's Params override its parents' Params.

Pipelines register their referenced Profiles with the DataTracker at save/load time, so a Pipeline that references a missing Profile fails validation the same way a Machine that references a missing Profile does.

API Surface

Pipelines

Pipelines are a first-class object type with standard CRUD at /api/v3/pipelines.

WorkOrder restart

  • POST /api/v3/work_orders/{uuid}/restart — restart a failed workflow-mode WorkOrder. Optional JSON body {"mode": "retry" | "continue" | "from-start"}; default retry. Rejects non-workflow-mode, non-failed, and Batch-owned WorkOrders. Returns the newly created successor WorkOrder.

Batch restart

  • POST /api/v3/batches/{uuid}/restart — restart a failed or finished Batch. Creates a new Batch linked via Predecessor and kicks off its setup WorkOrder; only the failed Machines from the predecessor Batch will be re-run when the work fan-out begins.

Machine creation with mode

  • POST /api/v3/machines?mode=<joinup|discover|manual|ipmi|unknown> — if the corresponding default-<mode>-pipeline and default-<mode>-blueprint prefs are set, the newly created Machine is stamped with the Pipeline and a workflow-mode WorkOrder is spawned against the Blueprint. Community-token requests apply these prefs unconditionally; user- or job-token requests use them only as fallback when the request itself does not specify a Pipeline or Blueprint.

Current-WorkOrder filtering

  • GET /api/v3/machines/{uuid}/work_orders/current?mode=<task|workflow> — restricts the returned "current WorkOrder" to one mode. The Machine Agent uses this to keep task-mode and workflow-mode WorkOrder queues disjoint.

Pipeline execution rollup

  • GET /api/v3/machines/{uuid}/pipeline-execution — returns the Machine along with all WorkOrders sharing the Machine's current Meta["pipeline-uuid"], ordered head-first by the Predecessor chain.
  • GET /api/v3/machines/{uuid}/pipeline-execution/{pipeline-uuid} — same rollup for a specific pipeline-uuid, useful for inspecting a completed or historical run.

Preferences

The pipeline machinery adds Preferences that control the POST /machines?mode= machine-creation shortcuts. For each supported mode there is a default-<mode>-pipeline and a default-<mode>-blueprint Preference, covering all five modes:

  • default-joinup-pipeline, default-joinup-blueprint
  • default-discover-pipeline, default-discover-blueprint
  • default-manual-pipeline, default-manual-blueprint
  • default-ipmi-pipeline, default-ipmi-blueprint
  • default-unknown-pipeline, default-unknown-blueprint

Each has a validator that ensures the referenced Pipeline or Blueprint exists at the time the Preference is set.

Interactions with Pool Transitions

A PoolTransitionAction may set Pipeline and Blueprint fields. When applied during a pool transition:

  • Pipeline (if set) is written to Machine.Pipeline.
  • Blueprint (if set) causes a workflow-mode WorkOrder to be spawned against that Blueprint for the transitioning Machine.

This is how a Pool can automatically start a pipeline run when a Machine joins it.