21.13. WorkOrders

dr-provision implements an extension to the Workflows system that allows for execution of sets of tasks as a unit after the initial provisioning of the machine. These WorkOrders can scheduled or triggered to start them.

To simplify building WorkOrders, a Blueprint can be created to allow for easy replication of WorkOrders across machines.

21.13.1. The Objects WorkOrders

The WorkOrder connects a set of Task, Param, and Profile to a machine.

When created, the WorkOrder is associated with a machine. If the machine is in Work Order Mode, the runner on that machine will pick up the WorkOrder and execute the task list on that work order. As the runner executes the tasks, the system accumulates Job objects to track the results.

If the a task fails, the WorkOrder fails. WorkOrders are not intended to be restarted. They can be cloned and rerun. These are intended to be trackable results oriented actions.

Currently, each runner will run one WorkOrder at a time, but this is not required and may change in the future.

The Task list on a WorkOrder can initially contain stages by prepending stage: and should NOT contain bootenv changes or context changes. WorkOrder State

The WorkOrder has a state and can only be operated based upon a state.

  • created - This is the starting state for a created WorkOrder. This means the WorkOrder is on the machine, but not started yet.

  • running - The WorkOrder has been picked up by a runner and is running.

  • failed - This WorkOrder has a failed task and has stopped.

  • finished - This WorkOrder has completed all tasks successfully.

  • cancelled - This WorkOrder was cancelled before completion of the tasks.

WorkOrders must me moved to one of the final three states to delete or remove the WorkOrder.

Additionally, the WorkOrder Status field contains information about what is going on with the WorkOrder. This is generally a failure message. Jobs

A Job is used to track the execution history of Tasks against a specific Machine. The job object contains a reference to the owning machine AND the owning WorkOrder. This allows for searching and filtering by Machine and WorkOrder. Blueprint

The Blueprint defines a set of tasks, parameters, and profiles that act as the basis for a WorkOrder.

When a WorkOrder is created that reference a Blueprint, the Params and Profiles fields are added to the Machine’s template render process. These are used before the Machine’s parameters, but after the WorkOrder’s own Params and Profiles.

The Blueprint has a task list that will get expanded when the WorkOrder is picked up to start execution. The Task list on a Blueprint may additionally include stages but prepending stage: to the name. The task list should not include bootenv changes. TriggerProvider

A TriggerProvider defines a Trigger type. The object defines the required and optional parameters for using the Trigger. The TriggerProvider defines is it is a URL-based trigger. If the NoURL field is set to true, this TriggerProvider is an internal DRP trigger. The Method field, if specified, defines the HTTP request type of this trigger. If unspecified, anything is allowed and the Trigger will decide if the Method is allowed.

The triggers plugin provides many webhook triggers. For a TriggerProvider that doesn’t have NoURL set to true, the system will create the following URL, https://IP:APIPORT/webhooks/<TriggerProviderName>. By default, all methods are allow. The URL-based trigger is assumed by handled by a plugin action on the system object. The plugin providing the TriggerProvider support should provide an Action on the system object named the same as the TriggerProvider.

The TriggerProvider has profiles and parameters associated with it. These are used to define the Trigger operations and customization. They are NOT applied to the WorkOrder.

There are two embedded TriggerProviders CronTrigger

This defines a trigger type that will generate a trigger bounded by a cron tab string.

The cron-trigger/time-string parameter defines a Crontab 5-element string that defines when the trigger will fire. EventTrigger

This defines a trigger type that will generate a trigger based upon a matching event.

The event-trigger/event-match parameter defines an event matcher that follows the scope, action, specific format of the event. This is the dot seperated with comma seperated elements. e.g. machines.update,create,save.* would match any machine event that is of action create, update, or save.

The event-trigger/event-object-match parameters optionally defines an await drpcli-style string that will allow for matching on the Object field of the event. await allows for And / Or / Not logical operators along with the Eq function to create a complex test structure. This example would match if the State of the event was finished, failed, or cancelled. Trigger

A Trigger defines the specific details of how the trigger operates and the resulting set of WorkOrders that result from that trigger. Additionally, the trigger can assign WorkOrders immediately to a specific set of machines or the machine assignment can be deferred until the machines are ready to accept new work.

There are three parts to a Trigger. Trigger Definition

The first part is the Trigger definition. The following fields define how the trigger operates:

  • Name - Defines the trigger’s key in the DRP data system.

  • TriggerProvider - Defines the type of Trigger

  • Enabled - Defines if the trigger is enabled

  • Params - Defines a set of parameters that alter what triggers. E.g. the crontab string for the CronTrigger

  • Profiles - Defines a list of profiles that also can be used to define parameters for the trigger

These parameters control how the trigger works. For example, the event-trigger provider uses the event-trigger/event-match parameter to define which events will invoke the trigger. These parameters are placed in the Params field or in a profile attached to the is Trigger.

Triggers are often delivered in content packs as Read-Only objects. This makes turning the on and off impossible with the Enabled field. The Enabled field represents the default state of the Trigger, Enabled or Disabled. This can be overridden by the global parameters, trigger/<name-of-trigger>-enabled or trigger/<name-of-trigger>-disabled. The disabled parameter is higher precedence than the enabled parameter. Either parameter overrides the Enabled field. These parameters can only be specified in the global profile and not in sub-profiles or parameter defaults. Trigger Scope

The second part is the Trigger scope. The following fields define what the trigger show act upon:

  • Filter - this is a DRP filter string to limit the scope of machines that should be applied. If the filter is empty, the machines list is Runnable=true and WorkOrderMode=true still applied.

  • AllInFilter - This boolean indicates if all the machines should receive a WorkOrder. Default is false.

  • FilterCount - This defaults to 1. If AllInFilter is not set to true, then this limits the WorkOrders to be created.

  • QueueMode - This boolean indiciates if the WorkOrders should be created without a Machine specifically assigned. This defers assignment until the Machines are ready to handle the work.

The Filter limits the machines to be triggered. Only machines that are Runnable and in WorkOrderMode are eligible for WorkOrders.

The Filter field can also use the rendering system to generate its value. The rendering of the field is restricted to functions that will work against the global profile. See Template for template functions. In general these should be restricted to parameter lookup functions. Here are some example strings.

Using the API or drpcli to test filter matching is useful to determine the scope of the trigger. See Filtering with Template Expansion. Trigger Work Orders

The third part is the Trigger Work Order definition. The following fields define what the WorkOrder will look like:

  • Blueprint - This defines the Blueprint object for the WorkOrder.

  • WorkOrderParams - These are parameters that should initialize the WorkOrder’s Params field.

  • WorkOrderProfiles - These are profiles that should initialize the WorkOrder’s Profiles field.

  • StoreDataInParameter - If this string parameter is defined, the data of the trigger will be stored on the WorkOrder in that parameter.

  • MergeDataIntoParams - If this boolean parameter is true, the data of the trigger will be merged into the WorkOrder’s Params field.

The combination of the fields will be used to create WorkOrder for each machine that machines the filter set of parameters.

21.13.2. Execution Models

With the addition of WorkOrders, there are two additional execution models to consider when building and planning infrastructure management.

The initial model revolved around provisioning and building up a machine. This is the traditional execution of DRP with extensions into Cloud infrastructure and logical components like clusters. Switch and storage provisioning through contexts fall into this model as well.

When used with Clusters v4.8+ and Resource Brokers, Machine as Service can allow for operating and maintaining specific services.

When used with Contexts and Trigger, the system can be used to operate a CI/CD pipeline like system.

All execution models can exist at the same time with a single DRP environment. Machine as Service

Once a Machine, Cluster, or ResourceBroker has been provisioned, that system can be placed in WorkOrderMode. This will cause the system not run workflows anymore and switch the WorkOrder execution mode, only WorkOrders will be processed. The goal is to operate those systems as service with Blueprints acting as defined actions to take through WorkOrders.

For example, a machine that represents a switch might use WorkOrderMode to handle port configuration changes as needed based upon other system’s needs. A cluster that represents a Kubernetes Cluster might use WorkOrderMode to manage applications and other cluster management functions after initial provisioning. Blueprints could be created with tasks and initial parameters needed to take those actions. WorkOrders would be created against these system that other WorkOrders or Workflows could use to handle configuration/state changes.

In this case, the Machine / Cluster / ResourceBroker objects continue to represent an element in the data center that is being managed. Orchestrated Tasks

When a Trigger is has QueueMode set to true, the Trigger operates as a deferred WorkOrder queue where the Filter field is the queue definition. This allows these triggers to operate like a pool of runners waiting to do jobs as they arrive. This is subtlely different than the Machine as Server model. The runners, grouped by the filter, are not specifically requested, but operate more anonymously. CI/CD Pipelines

With the introduction of Triggers and WorkOrders, the DRP environment can now act as a general action execution environment. This model matches a more worker / action model instead of a managed system environment.

In this execution model, a set of worker are started with tags. These worker could be Machines, Clusters, or ResourceBrokers that are backed by physical, virtual, or containers. These specified workers run WorkOrders generated by Triggers to manage the whole set of systems. This is NOT necessarily used to manage infrastructure elements directly.

Some examples:

  • A Blueprint could be written to handle repackaging content based upon a trigger from gitlab to rebuild and redeploy system components.

  • A Blueprint could be written to do heartbeat validation of a machine upon a cron-based trigger to monitor components.