Building Universal Applications for DRP Pipelines
Aka Universal Applications
What is a DRP Pipeline?¶
More generally, an Infrastructure Pipeline is a standardized end-to-end automation process composed of reusable components that drives infrastructure resources to a target state.
For the purpose of these instructions, a Pipeline is very narrowly defined as a special type of Digital Rebar Profile. The Pipeline Profile links to a Universal Application (also defined as a Profile) and has several key Params that define its configuration and expected sequence of operations.
By design, Universal Applications can also be used as a Pipeline because they have been created with safe defaults. When operators need to alter or extend those default settings, they create Pipeline. In this way, Universal Applications can be considered generalized templates while Pipelines have been extended to be operator specific.
Starting a Pipeline¶
Before we dig into the technical construction of a new Pipeline, it's important to understand that Pipelines should be
composed of modular components. An important part of that architecture is that Pipelines have consistent starting
points. Most pipelines start from either
universal-start for provisioned systems that should use the installed
operating system (aka local) or
universal-discover for new or redeployed infrastructure that needs to go through
Sledgehammer before installing a local operating system.
These instructions assume that you will always use
universal-discover to run a Pipeline. Advanced
users may choose other starting points for reasons that are outside of the scope of this document.
Creating a Minimal Universal Application¶
This documentation will walk you through creating a new Universal Application by building one up from the most basic required components and then expanding it to include all the components expected by the Universal Workflow pattern.
For clarity, the examples shown do not include
Meta values. These fields are
essential for Infrastructure as Code usability! When building your own pipeline, we strongly recommend including these
values in all components.
Choosing a Universal Application Name¶
Before starting any new Universal Application, we need to choose a name for the application. This will be used to connect together elements of your Universal Workflow.
For purposes of this documentation, we'll use the "myapp" for examples.
It is critical to understand that the Universal Workflow process relies heavily on naming conventions to connect components. This allows flexibility when building and extending Universal Applications within the Digital Rebar platform.
Creating a Universal Application Profile¶
The Universal Application Profile defines the critical configuration and component map needed to operate a Pipeline.
We'll be adding to this profile throughout this documentation. For now, we're going to create the most minimal Profile
possible with just a
It is important to follow the naming convention when creating the Profile name! The Profile should be
universal-application-[application]-[instance] and the
universal/application Param should be set to
[application] name. These patterns will be used extensively throughout building a Universal Application.
Advanced users: If you are creating a template application that will be used by other Universal Applications, then you will not create a Profile. Instead, you should define at least one implementation of that template for reference purposes.
For example, there is no
universal-application-linux-install Profile; Instead, a series of distribution specific
Universal Applications reference the
linux-install by including the
Creating a Chain Map Entry¶
A key feature of Pipelines is the Universal Workflow chaining process. The Stage named
universal-chain-workflow at the
end of each Universal Workflow allows Pipelines to be composed of modular pipeline segments based on your specific
environment and needs. We'll be creating your first Universal Workflow (
universal-myapp) segment next, but first we
will identify the standard Universal Workflow segments that you will include in your application's chain map.
The chain map is a simple "from: to" mapping which points from a Universal Workflow to another Universal Workflow. When
creating a map, each "from" must be unique, but you may have different "from" workflows pointing to the same "to"
workflow. In addition, there are a few common "start" workflows that should be included in your chain map. These
universal-linux-start. More advanced chains may also start with
universal-maintenance so you may choose to include those in your map.
Here's our updated Profile:
This map should be saved in your Universal Application profile in the
The above "chain-map" states that if the current workflow is either
universal-start, once that
workflow has completed, to next chain in to the
universal-myapp workflow. Since there is no left side entry point in
the map of
universal-myapp, the chaining process will stop when that workflow is completed.
Once you have tested the chain main, it can be added to the default
universal/workflow-chain-map Param. if you are
creating a generally available Universal Application. After you've promoted your map to the default, you should not
include it in the Universal Application Profile; instead, you will use the
Param to point to the template chain map.
Create Universal Workflow¶
Before we can test our new Universal Application, the chain map must include all the referenced Workflows. Since
universal-start already exist, we only need to create our application specific Workflow
universal-myapp. As usual, following the naming conventions is important.
The example shows the most minimal Universal Workflow. It literally does no work and has no standard extension points. Until we expand it, it's only purpose is to enable the chain mapping system; however, it's enough that we can actually run and test the Pipeline!
It's not technically required for a Universal Workflow to start with
start and end with
complete; however, providing
these standard entry and exit points ensure that the Workflow behaves predictably when used in isolation. Additionally,
it makes it easier for an operator to identify when the Workflow has started and completed successfully; particularly in
Here's the minimal myapp Workflow:
While many Universal Workflows include
discover as the first stage, this pattern should be approached with caution.
This is appropriate for Workflows that expect to run in Sledgehammer; however, most operator created Universal Workflows
will start in an installed system. For those Workflows, starting in
start is preferable.
Test your Universal Workflow¶
At this point, you've completed enough work to use your empty pipeline!
Using a provisioned machine, assign your Pipeline Profile to the machine and start your Pipeline by setting the machine
If your pipeline is working correctly, you should see it chain from
universal-start to your myapp Universal Workflow
Add WORK to Your Workflow¶
Now that you have an empty Pipeline working, you need to make it do something useful. That means actually building Stages and Tasks for the Workflow to implement.
If you are just doing this as part of a learning exercise, then consider including the
dev-library Content from the
catalog and including the
hello-world stages. These non-destructive stages are helpful when testing
Workflows because they provide feedback that work is being done without making any configuration changes.
For additional resources on how to create and manage Content Packs, which should be used to contain your Universal Application, please review the following examples (specifically the "Colordemo" content example):
Congratulations! You've built an Infrastructure Pipeline!
Adding Standardized Components¶
After your pipeline does something, it's time to add the rest of the standard Universal Workflow components. Don't be tempted to skip these next steps: these extra steps allow you and others to reuse and dynamically extend your Pipeline Segment.
Since your initial pipeline will perform exactly the work you need, these extra Stages and Params may seem unnecessary; however, they will become critical when you want to share or reuse your pipeline in different scenarios. Like other Infrastructure as Code patterns in Digital Rebar, following our proven automation patterns will help you build maintainable systems.
Note that Universal Applications have a very patterned workflow. While there are some inconsistencies in the pattern ( e.g.: sometimes pre vs start, etc), please keep and use the patterns because the underlying code relies on naming patterns.
Creating all these components from scratch can be time consuming, please see the
in https://gitlab.com/rackn/universal for examples that are designed as base templates for other pipelines. Search
REPLACEME to find places where your naming needs to be added and remember to update the "on disk" file names to
match the internal model name (typically the YAML or JSON "Name" field inside the file).
We're going to build our pipeline from the "top" down.
Update the Workflow to include Standard Stages¶
Pipeline segments, aka Universal Workflows, must include additional Pipeline segment specific stages to provide predictable extension points. Callback and Flexiflow are done before and after the workload. Classification and Validation are only done after.
The example below shows a fully standardized Universal Workflow.
In the above example, the
wait-time is the segment work in this Pipeline (eg. the "myapp" specific Stage).
Add Stages and Params¶
At this point, we're not trying to extend the pipeline beyond default behaviors so it's OK to simply copy and rename the
template Stages and Params with no modification except replacing
REPLACEMENT with your application name.
Clusters and Resource Brokers¶
This document explains how to use the v4.8 resources brokers and clusters to deploy batches of cloud machines. This assumes that you are using the UX Wizards to create and manage resource brokers and clusters.
To follow these instructions, you will need a cloud account and API token in which to create machines. Each cloud account has a unique API authentication strategy. Collecting that information is outside the scope of this document, but may be included in the param documentation for each cloud.
Create A Resource Broker¶
Before you can provision a cluster, you must create a resource broker that can provision machines on your behalf. For clouds, this process involves Selecting a cloud resource file (see below) matching your target cloud Set the name for the resource broker (must be unique from all brokers, machines and clusters) Filling in the required parameters for the resource broker (varies by cloud) Optionally setting the optional parameters for resource broker (varies by cloud) Optionally set the meta data for your resource broker
When the resource broker wizard starts, it automatically starts the "universal-application-broker-base" pipeline. This pipeline will do the minimum to start the context running the broker and then switch to Work Order mode and lock the resource broker. This is the normal operating state for a resource broker.
No additional configuration is required at this time; however, you may find that changes to Params are needed to correctly use the broker. In this case, you'll need to unlock the broker before you can make updates.
provision-content repo includes a tools/cloud-profiles.sh script that automatically creates resource brokers
for popular clouds by looking for local configuration information. This script is idempotent but requires to have
correct environment variables to use drpcli from the script location.
Behind the Scenes: Terraform¶
For the cloud provisioning process outlined here, the resource brokers use a Terraform process via the "Terraform Apply" task. That task does two important things in conjunction: builds and runs a terraform plan that includes a Join Up on cloud-init script pointed to Digital Rebar uses the Terraform create/destroy messages to synchronize with Digital Rebar machine entries.
There is a critical timing element to understand in this process because information needed for the machines to join Digital Rebar smoothly is only known by Terraform provision after the cloud responds to the provision request with the machine's unique identity in the cloud. Once Terraform has the cloud machine id, it will call the synchronization script. The synchronization script creates machines in Digital Rebar in which the fingerprint is set to match the assigned cloud identity.
When the machine joins up from the cloud, Digital Rebar matches the fingerprint and the machine immediately begins processing its assigned pipeline.
In addition to this per machine process, the Terraform Apply task manages the Terraform tfstate and tflock files by storing them on the Cluster machine entry. This allows Digital Rebar to transparently track the state of the resources assigned to the cluster.
Behind the Scenes: Cloud Resource Profile¶
The broker wizard leverages pre-defined cloud resource profiles from the cloud-wrappers context pack such as "resource-aws-cloud.yaml". These resource profiles contain the mappings needed by the broker and terraform-apply task to correctly build Terraform plans and read the Terraform event output. Operators can create new cloud mappings by creating their own version of these profiles.
Any profile that begins with "resource-" will be offered as a choice by the resource broker wizard.
As a helper, these resource profiles also specify the require and optional Params for the mapped cloud in their metadata.
Build a Cluster¶
When creating a Cluster, operators will choose the desired Cluster Pipeline (which is a universal application that
includes "cluster" in the title) and select the operating context (typically drpcli-runner). The reference cluster
pipelines also define required and optional params. All clusters will minimally require the
broker/name to be defined.
For the base cluster pipeline (aka universal-application-base-cluster), the params are: *
broker/name (required) -
this selects the resource broker to use to request resources via the
cluster/machines param. The pipeline will create
a work order targeting the selected broker.
cluster/count(required) - this tells the pipeline how many machines to add into the
cluster/machinesparam for the broker. Zero will create an empty cluster that can be modified later.
broker/set-pipeline(optional) - this determines the pipeline assigned to provisioned machines. By default, this is the universal-application-application-base which includes the Start and Runbook workflows.
broker/set-workflow(optional) - this determines the initial workflow for the provisioned machines. Generally, " start" is the correct workflow for cloud machines and "discover" for bare metal.
this determines the initial icon set for the provisioned machines. It can be helpful to set icons matching the cluster icon. The default is to match the broker's icon.
When Digital Rebar creates the cluster, it also creates a profile of the same name. This profile is assigned to the created machines as a shared resource. By design, information written to the cluster's profile is available to all the machines also. That is primarily shared SSH keys for the base-cluster pipeline.
When the cluster is created, it will start a pipeline that includes the "cluster-provision" task. That task will build
cluster/machines JSON structure that defines the cluster for the resource broker (see below). It will then pass
cluster/machines information to the resource broker in a work order and wait until the work order is complete. The
cluster UX has machines tab or you can watch for machines to appear on the machines list.
The base-cluster pipeline will complete as soon as the resource broker completes; however, cluster pipelines may also
wait for all provisioned machines to complete their workflow if the
cluster/wait-for-members param has been set to
If you want to modify information for a cluster before creating it, then set the
cluster/count to zero. This will allow the cluster to be created by the wizard but no machines will be provisioned until the
cluster/count is updated.
Behind the Scenes: Tracking Cluster Building¶
The Digital Rebar cluster provisioning process invokes a number of different actions behind the scenes to complete this operation. Here are some areas to review during the process:
- Resource Brokers - when the cluster pipeline creates the provisioning work order, you will be able to see it queued and running on the appropriate resource broker. These counts show up in the table and editing view of the UX.
- Work Orders - The Work Orders table is a good place to review pending, active, errored and completed work orders. Work Orders are read only and cannot be edited or restarted. To "restart" a work order, you will need to make a copy and resubmit it. When the Work Order is being processed, all work performed generates jobs just like a Workflow.
- Jobs - While the Work Order is being executed, it will create one or more job entries. This is a good place to observe the actual task logs and process performed by the Work Order. Remember that the job log includes both Work Order and Workflow output. Once the provisioned machines join they will start in Workflow mode and many job entries will be created.
Behind the Scenes: Building a Test Cluster with Context-Broker¶
If you just want to quickly test the Cluster capabilities, you can use the build-in "context-broker" that creates a cluster of "drpcli-runner" backed machines. Even though they are just running in a context, they can be handy for testing basic pipelines and cluster operations. They are also helpful to verify that the basic operations for clusters, resource brokers and machines are all working correctly.
Behind the Scenes: Multiple Machine Types and Named Machines¶
For simplicity, the base-cluster pipeline uses a simple cluster/count for input; however, the actual resource broker system uses a more robust cluster definition in the form of cluster/machine-types and cluster/machines. Operators can create complex clusters and control which machines are added or removed using these Params. If they have been defined for a cluster then the cluster/count will be ignored.
cluster/machine-types param is a string array used to create multiple types (aka tiers) of machines for a cluster
such as having leaders, storage and worker nodes in a Kubernetes cluster. Ideally, names of machine-types are
cluster/machines JSON structure expands upon the
cluster/machine-types and must have an entry for each type. At
a minimum, each machine-type must have a list of names for each machine in the cluster. The structure also allows
operators to specify Params, icon and metadata to be set on the machines provisioned. The Params are also used during
template expansion to influence Terraform plans.
Operators can control which machines are added or removed from a cluster by managing the names list in the
it is possible to use Param expansion in the
cluster/machines data so that information can be created based on other params.
Resize the Cluster¶
To resize the cluster, simply change the
cluster/count and restart the universal-cluster-provision workflow. This will
cluster/machines structure and pass it to the resource broker. The resource broker will adjust the
resources to match the cluster/machines requested.
If the cluster is set to zero then all machines will be released; however, the cluster will still exist. This is a handy way to turn off the resources for a cluster without losing the state of the cluster. It can be later "reconstituted" by simply requesting non-zero resources.
Destroy The Cluster¶
The cluster-destroy workflow will set the correct flags to destroy all cluster resources. For Terraform based clusters, that means calling "terraform destroy" in the Terraform Apply task and removing the state file.
By default, clusters are created with the
universal-cluster-destroy workflow assigned as the
Param. When you cleanup (instead of delete) a cluster, Digital Rebar will call the destroy workflow automatically before
deleting the cluster. This is very helpful for housekeeping because it ensures that all the cluster's allocated machines
are automatically removed when the cluster is removed.