Skip to content

Pipeline Architecture

Aka Universal Applications

What is a DRP Pipeline?

More generally, an Infrastructure Pipeline is a standardized end-to-end automation process composed of reusable components that drives infrastructure resources to a target state.

For the purpose of these instructions, a Pipeline is very narrowly defined as a special type of Digital Rebar Profile. The Pipeline Profile links to a Universal Application (also defined as a Profile) and has several key Params that define its configuration and expected sequence of operations.

By design, Universal Applications can also be used as a Pipeline because they have been created with safe defaults. When operators need to alter or extend those default settings, they create Pipeline. In this way, Universal Applications can be considered generalized templates while Pipelines have been extended to be operator specific.

Starting a Pipeline

Before we dig into the technical construction of a new Pipeline, it's important to understand that Pipelines should be composed of modular components. An important part of that architecture is that Pipelines have consistent starting points. Most pipelines start from either universal-start for provisioned systems that should use the installed operating system (aka local) or universal-discover for new or redeployed infrastructure that needs to go through Sledgehammer before installing a local operating system.

These instructions assume that you will always use universal-start or universal-discover to run a Pipeline. Advanced users may choose other starting points for reasons that are outside of the scope of this document.

Creating a Minimal Universal Application

This documentation will walk you through creating a new Universal Application by building one up from the most basic required components and then expanding it to include all the components expected by the Universal Workflow pattern.

For clarity, the examples shown do not include Description, Documentation or Meta values. These fields are essential for Infrastructure as Code usability! When building your own pipeline, we strongly recommend including these values in all components.

Choosing a Universal Application Name

Before starting any new Universal Application, we need to choose a name for the application. This will be used to connect together elements of your Universal Workflow.

For purposes of this documentation, we'll use the "myapp" for examples.

It is critical to understand that the Universal Workflow process relies heavily on naming conventions to connect components. This allows flexibility when building and extending Universal Applications within the Digital Rebar platform.

Creating a Universal Application Profile

The Universal Application Profile defines the critical configuration and component map needed to operate a Pipeline. We'll be adding to this profile throughout this documentation. For now, we're going to create the most minimal Profile possible with just a Name and universal/application Param.

---
Name: universal-application-myapp-v1
Params:
  universal/application: myapp

It is important to follow the naming convention when creating the Profile name! The Profile should be named universal-application-[application]-[instance] and the universal/application Param should be set to the [application] name. These patterns will be used extensively throughout building a Universal Application.

Advanced users: If you are creating a template application that will be used by other Universal Applications, then you will not create a Profile. Instead, you should define at least one implementation of that template for reference purposes.

For example, there is no universal-application-linux-install Profile; Instead, a series of distribution specific Universal Applications reference the linux-install by including the Param universal/workflow-chain-index-override:linux-install.

Creating a Chain Map Entry

A key feature of Pipelines is the Universal Workflow chaining process. The Stage named universal-chain-workflow at the end of each Universal Workflow allows Pipelines to be composed of modular pipeline segments based on your specific environment and needs. We'll be creating your first Universal Workflow (universal-myapp) segment next, but first we will identify the standard Universal Workflow segments that you will include in your application's chain map.

The chain map is a simple "from: to" mapping which points from a Universal Workflow to another Universal Workflow. When creating a map, each "from" must be unique, but you may have different "from" workflows pointing to the same "to" workflow. In addition, there are a few common "start" workflows that should be included in your chain map. These are: universal-discover and universal-linux-start. More advanced chains may also start with universal-decommission , universal-rebuild, and universal-maintenance so you may choose to include those in your map.

Here's our updated Profile:

---
Name: universal-application-myapp-v1
Params:
  universal/application: myapp
  universal/workflow-chain-map:
    myapp:
      universal-discover: universal-myapp
      universal-start: universal-myapp

This map should be saved in your Universal Application profile in the universal/workflow-chain-map Param.

The above "chain-map" states that if the current workflow is either universal-discover or universal-start, once that workflow has completed, to next chain in to the universal-myapp workflow. Since there is no left side entry point in the map of universal-myapp, the chaining process will stop when that workflow is completed.

Once you have tested the chain main, it can be added to the default universal/workflow-chain-map Param. if you are creating a generally available Universal Application. After you've promoted your map to the default, you should not include it in the Universal Application Profile; instead, you will use the universal/workflow-chain-index-override Param to point to the template chain map.

Create Universal Workflow

Before we can test our new Universal Application, the chain map must include all the referenced Workflows. Since universal-discover and universal-start already exist, we only need to create our application specific Workflow universal-myapp. As usual, following the naming conventions is important.

The example shows the most minimal Universal Workflow. It literally does no work and has no standard extension points. Until we expand it, it's only purpose is to enable the chain mapping system; however, it's enough that we can actually run and test the Pipeline!

It's not technically required for a Universal Workflow to start with start and end with complete; however, providing these standard entry and exit points ensure that the Workflow behaves predictably when used in isolation. Additionally, it makes it easier for an operator to identify when the Workflow has started and completed successfully; particularly in the Portal.

Here's the minimal myapp Workflow:

---
Name: universal-myapp
Stages:
  - start
  - universal-chain-workflow
  - complete

While many Universal Workflows include discover as the first stage, this pattern should be approached with caution. This is appropriate for Workflows that expect to run in Sledgehammer; however, most operator created Universal Workflows will start in an installed system. For those Workflows, starting in start is preferable.

Test your Universal Workflow

At this point, you've completed enough work to use your empty pipeline!

Using a provisioned machine, assign your Pipeline Profile to the machine and start your Pipeline by setting the machine to the universal-start workflow.

If your pipeline is working correctly, you should see it chain from universal-start to your myapp Universal Workflow automatically.

Add WORK to Your Workflow

Now that you have an empty Pipeline working, you need to make it do something useful. That means actually building Stages and Tasks for the Workflow to implement.

If you are just doing this as part of a learning exercise, then consider including the dev-library Content from the catalog and including the wait-time or hello-world stages. These non-destructive stages are helpful when testing Workflows because they provide feedback that work is being done without making any configuration changes.

For additional resources on how to create and manage Content Packs, which should be used to contain your Universal Application, please review the following examples (specifically the "Colordemo" content example):

https://gitlab.com/rackn/example-content

Congratulations! You've built an Infrastructure Pipeline!

Adding Standardized Components

After your pipeline does something, it's time to add the rest of the standard Universal Workflow components. Don't be tempted to skip these next steps: these extra steps allow you and others to reuse and dynamically extend your Pipeline Segment.

Since your initial pipeline will perform exactly the work you need, these extra Stages and Params may seem unnecessary; however, they will become critical when you want to share or reuse your pipeline in different scenarios. Like other Infrastructure as Code patterns in Digital Rebar, following our proven automation patterns will help you build maintainable systems.

Note that Universal Applications have a very patterned workflow. While there are some inconsistencies in the pattern ( e.g.: sometimes pre vs start, etc), please keep and use the patterns because the underlying code relies on naming patterns.

Creating all these components from scratch can be time consuming, please see the pipeline-template directory in https://gitlab.com/rackn/universal for examples that are designed as base templates for other pipelines. Search for REPLACEME to find places where your naming needs to be added and remember to update the "on disk" file names to match the internal model name (typically the YAML or JSON "Name" field inside the file).

We're going to build our pipeline from the "top" down.

Update the Workflow to include Standard Stages

Pipeline segments, aka Universal Workflows, must include additional Pipeline segment specific stages to provide predictable extension points. Callback and Flexiflow are done before and after the workload. Classification and Validation are only done after.

The example below shows a fully standardized Universal Workflow.

---
Name: universal-myapp
Stages:
  - start
  - universal-myapp-start-callback
  - universal-myapp-pre-flexiflow
  - wait-time
  - universal-myapp-post-flexiflow
  - universal-myapp-classification
  - universal-myapp-post-validation
  - universal-myapp-complete-callback
  - complete

In the above example, the wait-time is the segment work in this Pipeline (eg. the "myapp" specific Stage).

Add Stages and Params

At this point, we're not trying to extend the pipeline beyond default behaviors so it's OK to simply copy and rename the template Stages and Params with no modification except replacing REPLACEMENT with your application name.

Clusters and Resource Brokers

This document explains how to use the v4.8 resources brokers and clusters to deploy batches of cloud machines. This assumes that you are using the UX Wizards to create and manage resource brokers and clusters.

To follow these instructions, you will need a cloud account and API token in which to create machines. Each cloud account has a unique API authentication strategy. Collecting that information is outside the scope of this document, but may be included in the param documentation for each cloud.

Create A Resource Broker

Before you can provision a cluster, you must create a resource broker that can provision machines on your behalf. For clouds, this process involves Selecting a cloud resource file (see below) matching your target cloud Set the name for the resource broker (must be unique from all brokers, machines and clusters) Filling in the required parameters for the resource broker (varies by cloud) Optionally setting the optional parameters for resource broker (varies by cloud) Optionally set the meta data for your resource broker

When the resource broker wizard starts, it automatically starts the "universal-application-broker-base" pipeline. This pipeline will do the minimum to start the context running the broker and then switch to Work Order mode and lock the resource broker. This is the normal operating state for a resource broker.

No additional configuration is required at this time; however, you may find that changes to Params are needed to correctly use the broker. In this case, you'll need to unlock the broker before you can make updates.

Note: the provision-content repo includes a tools/cloud-profiles.sh script that automatically creates resource brokers for popular clouds by looking for local configuration information. This script is idempotent but requires to have correct environment variables to use drpcli from the script location.

Behind the Scenes: Terraform

For the cloud provisioning process outlined here, the resource brokers use a Terraform process via the "Terraform Apply" task. That task does two important things in conjunction: builds and runs a terraform plan that includes a Join Up on cloud-init script pointed to Digital Rebar uses the Terraform create/destroy messages to synchronize with Digital Rebar machine entries.

There is a critical timing element to understand in this process because information needed for the machines to join Digital Rebar smoothly is only known by Terraform provision after the cloud responds to the provision request with the machine's unique identity in the cloud. Once Terraform has the cloud machine id, it will call the synchronization script. The synchronization script creates machines in Digital Rebar in which the fingerprint is set to match the assigned cloud identity.

When the machine joins up from the cloud, Digital Rebar matches the fingerprint and the machine immediately begins processing its assigned pipeline.

In addition to this per machine process, the Terraform Apply task manages the Terraform tfstate and tflock files by storing them on the Cluster machine entry. This allows Digital Rebar to transparently track the state of the resources assigned to the cluster.

Behind the Scenes: Cloud Resource Profile

The broker wizard leverages pre-defined cloud resource profiles from the cloud-wrappers context pack such as "resource-aws-cloud.yaml". These resource profiles contain the mappings needed by the broker and terraform-apply task to correctly build Terraform plans and read the Terraform event output. Operators can create new cloud mappings by creating their own version of these profiles.

Any profile that begins with "resource-" will be offered as a choice by the resource broker wizard.

As a helper, these resource profiles also specify the require and optional Params for the mapped cloud in their metadata.

Build a Cluster

When creating a Cluster, operators will choose the desired Cluster Pipeline (which is a universal application that includes "cluster" in the title) and select the operating context (typically drpcli-runner). The reference cluster pipelines also define required and optional params. All clusters will minimally require the broker/name to be defined.

For the base cluster pipeline (aka universal-application-base-cluster), the params are: * broker/name (required) - this selects the resource broker to use to request resources via the cluster/machines param. The pipeline will create a work order targeting the selected broker.

  • cluster/count (required) - this tells the pipeline how many machines to add into the
  • cluster/machines param for the broker. Zero will create an empty cluster that can be modified later.
  • broker/set-pipeline (optional) - this determines the pipeline assigned to provisioned machines. By default, this is the universal-application-application-base which includes the Start and Runbook workflows.
  • broker/set-workflow (optional) - this determines the initial workflow for the provisioned machines. Generally, " start" is the correct workflow for cloud machines and "discover" for bare metal.

  • broker/set-icon (optional)

  • this determines the initial icon set for the provisioned machines. It can be helpful to set icons matching the cluster icon. The default is to match the broker's icon.

When Digital Rebar creates the cluster, it also creates a profile of the same name. This profile is assigned to the created machines as a shared resource. By design, information written to the cluster's profile is available to all the machines also. That is primarily shared SSH keys for the base-cluster pipeline.

When the cluster is created, it will start a pipeline that includes the "cluster-provision" task. That task will build the cluster/machines JSON structure that defines the cluster for the resource broker (see below). It will then pass the cluster/machines information to the resource broker in a work order and wait until the work order is complete. The cluster UX has machines tab or you can watch for machines to appear on the machines list.

The base-cluster pipeline will complete as soon as the resource broker completes; however, cluster pipelines may also wait for all provisioned machines to complete their workflow if the cluster/wait-for-members param has been set to true.

Tip

If you want to modify information for a cluster before creating it, then set the cluster/count to zero. This will allow the cluster to be created by the wizard but no machines will be provisioned until the cluster/count is updated.

Behind the Scenes: Tracking Cluster Building

The Digital Rebar cluster provisioning process invokes a number of different actions behind the scenes to complete this operation. Here are some areas to review during the process:

  • Resource Brokers - when the cluster pipeline creates the provisioning work order, you will be able to see it queued and running on the appropriate resource broker. These counts show up in the table and editing view of the UX.
  • Work Orders - The Work Orders table is a good place to review pending, active, errored and completed work orders. Work Orders are read only and cannot be edited or restarted. To "restart" a work order, you will need to make a copy and resubmit it. When the Work Order is being processed, all work performed generates jobs just like a Workflow.
  • Jobs - While the Work Order is being executed, it will create one or more job entries. This is a good place to observe the actual task logs and process performed by the Work Order. Remember that the job log includes both Work Order and Workflow output. Once the provisioned machines join they will start in Workflow mode and many job entries will be created.

Behind the Scenes: Building a Test Cluster with Context-Broker

If you just want to quickly test the Cluster capabilities, you can use the build-in "context-broker" that creates a cluster of "drpcli-runner" backed machines. Even though they are just running in a context, they can be handy for testing basic pipelines and cluster operations. They are also helpful to verify that the basic operations for clusters, resource brokers and machines are all working correctly.

Behind the Scenes: Multiple Machine Types and Named Machines

For simplicity, the base-cluster pipeline uses a simple cluster/count for input; however, the actual resource broker system uses a more robust cluster definition in the form of cluster/machine-types and cluster/machines. Operators can create complex clusters and control which machines are added or removed using these Params. If they have been defined for a cluster then the cluster/count will be ignored.

The cluster/machine-types param is a string array used to create multiple types (aka tiers) of machines for a cluster such as having leaders, storage and worker nodes in a Kubernetes cluster. Ideally, names of machine-types are descriptive.

The cluster/machines JSON structure expands upon the cluster/machine-types and must have an entry for each type. At a minimum, each machine-type must have a list of names for each machine in the cluster. The structure also allows operators to specify Params, icon and metadata to be set on the machines provisioned. The Params are also used during template expansion to influence Terraform plans.

Tip

Operators can control which machines are added or removed from a cluster by managing the names list in the cluster/machines schema.

Note

it is possible to use Param expansion in the cluster/machines data so that information can be created based on other params.

Resize the Cluster

To resize the cluster, simply change the cluster/count and restart the universal-cluster-provision workflow. This will rebuild the cluster/machines structure and pass it to the resource broker. The resource broker will adjust the resources to match the cluster/machines requested.

If the cluster is set to zero then all machines will be released; however, the cluster will still exist. This is a handy way to turn off the resources for a cluster without losing the state of the cluster. It can be later "reconstituted" by simply requesting non-zero resources.

Destroy The Cluster

The cluster-destroy workflow will set the correct flags to destroy all cluster resources. For Terraform based clusters, that means calling "terraform destroy" in the Terraform Apply task and removing the state file.

By default, clusters are created with the universal-cluster-destroy workflow assigned as the on-cleanup-workflow Param. When you cleanup (instead of delete) a cluster, Digital Rebar will call the destroy workflow automatically before deleting the cluster. This is very helpful for housekeeping because it ensures that all the cluster's allocated machines are automatically removed when the cluster is removed.