Skip to content

Pipelines

A Pipeline in DRP is a profile that links to a Universal Application and defines a target state for a machine through its parameters. Universal Applications provide standardized workflows with safe defaults, while Pipelines customize these defaults for specific use cases.

This reference guide covers the essential components and patterns used in Pipeline development, providing the foundational knowledge needed to build infrastructure automation workflows.

Requirements

  • DRP environment with administrative privileges
  • Knowledge of DRP concepts: Profiles, Workflows, and Stages

Architecture Overview

Pipelines use a naming convention-based system where components are linked through standardized patterns. The workflow chaining system enables modular composition of automation workflows, allowing for reusable and extensible infrastructure automation.

Creating a Universal Application Profile

The Universal Application Profile defines the critical configuration and component map needed to operate a Pipeline. We'll be adding to this profile throughout this documentation. For now, we're going to create the most minimal Profile possible with just a Name and universal/application Param.

---
Name: universal-application-myapp-v1
Params:
  universal/application: myapp

It is important to follow the naming convention when creating the Profile name. The Profile should be named universal-application-[application]-[instance] and the universal/application Param should be set to the [application] name. These patterns will be used extensively throughout building a Universal Application.

Advanced users: If you are creating a template application that will be used by other Universal Applications, then you will not create a Profile. Instead, you should define at least one implementation of that template for reference purposes.

For example, there is no universal-application-linux-install Profile; Instead, a series of distribution specific Universal Applications reference the linux-install by including the Param universal/workflow-chain-index-override:linux-install.

Creating a Chain Map Entry

A key feature of Pipelines is the Universal Workflow chaining process. The Stage named universal-chain-workflow at the end of each Universal Workflow allows Pipelines to be composed of modular pipeline segments based on your specific environment and needs. We'll be creating your first Universal Workflow (universal-myapp) segment next, but first we will identify the standard Universal Workflow segments that you will include in your application's chain map.

The chain map is a simple "from: to" mapping which points from a Universal Workflow to another Universal Workflow. When creating a map, each "from" must be unique, but you may have different "from" workflows pointing to the same "to" workflow. In addition, there are a few common "start" workflows that should be included in your chain map. These are: universal-discover and universal-linux-start. More advanced chains may also start with universal-decommission, universal-rebuild, and universal-maintenance so you may choose to include those in your map.

Here's our updated Profile:

---
Name: universal-application-myapp-v1
Params:
  universal/application: myapp
  universal/workflow-chain-map:
    myapp:
      universal-discover: universal-myapp
      universal-start: universal-myapp

This map should be saved in your Universal Application profile in the universal/workflow-chain-map Param.

The above "chain-map" states that if the current workflow is either universal-discover or universal-start, once that workflow has completed, to next chain in to the universal-myapp workflow. Since there is no left side entry point in the map of universal-myapp, the chaining process will stop when that workflow is completed.

Once you have tested the chain main, it can be added to the default universal/workflow-chain-map Param. if you are creating a generally available Universal Application. After you've promoted your map to the default, you should not include it in the Universal Application Profile; instead, you will use the universal/workflow-chain-index-override Param to point to the template chain map.

Create Universal Workflow

Before we can test our new Universal Application, the chain map must include all the referenced Workflows. Since universal-discover and universal-start already exist, we only need to create our application specific Workflow universal-myapp. As usual, following the naming conventions is important.

The example shows the most minimal Universal Workflow. It literally does no work and has no standard extension points. Until we expand it, it's only purpose is to enable the chain mapping system; however, it's enough that we can actually run and test the Pipeline!

It's not technically required for a Universal Workflow to start with start and end with complete; however, providing these standard entry and exit points ensure that the Workflow behaves predictably when used in isolation. Additionally, it makes it easier for an operator to identify when the Workflow has started and completed successfully; particularly in the Portal.

Here's the minimal myapp Workflow:

---
Name: universal-myapp
Stages:
  - start
  - universal-chain-workflow
  - complete

While many Universal Workflows include discover as the first stage, this pattern should be approached with caution. This is appropriate for Workflows that expect to run in Sledgehammer; however, most operator created Universal Workflows will start in an installed system. For those Workflows, starting in start is preferable.

Test Your Universal Workflow

At this point, you've completed enough work to use your empty pipeline!

Using a provisioned machine, assign your Pipeline Profile to the machine and start your Pipeline by setting the machine to the universal-start workflow.

If your pipeline is working correctly, you should see it chain from universal-start to your myapp Universal Workflow automatically.

Add Work to Your Workflow

Now that you have an empty Pipeline working, you need to make it do something useful. That means actually building Stages and Tasks for the Workflow to implement.

If you are just doing this as part of a learning exercise, then consider including the dev-library Content from the catalog and including the wait-time or hello-world stages. These non-destructive stages are helpful when testing Workflows because they provide feedback that work is being done without making any configuration changes.

For additional resources on how to create and manage Content Packs, which should be used to contain your Universal Application, please review the following examples (specifically the "Colordemo" content example):

https://gitlab.com/rackn/example-content

Congratulations! You've built an Infrastructure Pipeline!

Adding Standardized Components

After your pipeline does something, it's time to add the rest of the standard Universal Workflow components. Don't be tempted to skip these next steps: these extra steps allow you and others to reuse and dynamically extend your Pipeline Segment.

Since your initial pipeline will perform exactly the work you need, these extra Stages and Params may seem unnecessary; however, they will become critical when you want to share or reuse your pipeline in different scenarios. Like other Infrastructure as Code patterns in Digital Rebar, following our proven automation patterns will help you build maintainable systems.

Note that Universal Applications have a very patterned workflow. While there are some inconsistencies in the pattern (e.g.: sometimes pre vs start, etc), please keep and use the patterns because the underlying code relies on naming patterns.

Creating all these components from scratch can be time consuming, please see the pipeline-template directory in https://gitlab.com/rackn/universal for examples that are designed as base templates for other pipelines. Search for REPLACEME to find places where your naming needs to be added and remember to update the "on disk" file names to match the internal model name (typically the YAML or JSON "Name" field inside the file).

We're going to build our pipeline from the "top" down.

Update the Workflow to Include Standard Stages

Pipeline segments, aka Universal Workflows, must include additional Pipeline segment specific stages to provide predictable extension points. Callback and Flexiflow are done before and after the workload. Classification and Validation are only done after.

The example below shows a fully standardized Universal Workflow.

---
Name: universal-myapp
Stages:
  - start
  - universal-myapp-start-callback
  - universal-myapp-pre-flexiflow
  - wait-time
  - universal-myapp-post-flexiflow
  - universal-myapp-classification
  - universal-myapp-post-validation
  - universal-myapp-complete-callback
  - complete

In the above example, the wait-time is the segment work in this Pipeline (eg. the "myapp" specific Stage).

Add Stages and Params

At this point, we're not trying to extend the pipeline beyond default behaviors so it's OK to simply copy and rename the template Stages and Params with no modification except replacing REPLACEMENT with your application name.

References

Some additional documents that would assist with understanding DRP pipelines: