Upgrading Digital Rebar Provision

Digital Rebar Provision has the ability to upgrade itself and (optionally) a local copy of drpcli via the API. This is intended to be used as the primary method of keeping the running Digital Rebar Provision server up to date. There are two ways of doing this: the two-phase rolling upgrade process, and the single-step upgrade process.

Upgrade Prerequisites:

Digital Rebar Provision has a few requirements that must be met for either of the self-upgrade processes to be able to operate properly:

The dr-provision process must be able to locate its own binary

This is generally not a concern, but if something removed the running dr-provision binary then self-updating by any means will fail.

The the dr-provision binary must be writable by the user the dr-provision process is running as, and the directory the dr-provision binary is in must also be writable in the same way

The final part of the upgrade process is to replace the current running executable with the new one and re-execute dr-provision. In order for that to happen, the user that dr-provision is running as must have write access to the dr-provision binary and the directory that the dr-provision binary is in.

dr-provision must be able to fix up its own capabilities.

When running as a non-root user, dr-provision must be able to bind to some low ports to handle DHCP and TFTP traffic. On Linux systems, this is generally accomplished by using setcap on the dr-provision executable to give it the additional capabilities it needs. For the self-upgrade process to be able to copy over the capabilities properly, it must be able to run sudo setcap with the same capability list the current dr-provision executable has.

If any of these requirements are not able to be met, then API driven self-upgrade will not operate correctly, and you will have to resort to manual upgrades.

Two-phase Rolling Upgrade Process

The two-phase rolling upgrade process is available when thetwo-phase-rolling-upgrades feature flag is present. When this flag is present, the upgrade process is split into two phases: staging all the binaries to be upgraded, and copying binaries over and having dr-provision restart itself.

Staging the upgrade

Staging the upgrade involves uploading a dr-provision upgrade archive to the server, creating a rackn-system-upgrade-tracking profile to keep track of the upgrade progress, and (if in a consensus cluster) making sure that the upgrade archive and all of its components have replicated to all cluster members.

To stage an upgrade, you need an upgrade package. The easiest way to get one is to download it withdrpcli catalog item download drp to drp-upgrade.zip, although there are other options.

Once you have an upgrade package, you can stage the upgrade by issuing aPOST request to the API at /api/v3/system/upgrade/stage with the drp-upgrade.zip file contents as the body of the API request. If the request is accepted, you will get a StatusCreated response back, otherwise you will get an error back with information describing what is wrong.

You can run drpcli system upgrade stage drp-upgrade.zip from the commandline to perform this action as well.

Running the upgrade

Once an upgrade is staged, you can start the upgrade process by issuing a POST request to the API at /api/v3/system/upgrade/run with an empty request body. This will start the upgrade process in the background.

You can run drpcli system upgrade start on the commandline to perform this action as well.

The Upgrade Process Happening In The Background

Once the upgrade process has been started, it will run into the background until either all of the cluster members have upgraded or there is a failure. For upgrade purposes, a standalone dr-provision server is considered to be a cluster with one member that is (by definition) also the cluster leader.

The staged upgrade package will be applied to each cluster member one at a time with the cluster leader being the last one to be upgraded. After the upgrade package is applied to each individual member, it will restart and the leader will wait until the freshly-rebooted member re-synchronizes with the cluster and is back up before proceeding to the next cluster member. Once every cluster member besides the leader is upgraded, the leader will upgrade itself, hand leadership of the cluster off to another member, and then reboot. Each node gets up to an hour to reboot and rejoin the cluster, otherwise the upgrade on that node will be considered to have failed.

Monitoring the Upgrade Process

You can monitor the state of an upgrade using either GET /api/v3/profiles/rackn-system-upgrade-tracking or GET /api/v3/system/upgrade/status to get a point-in-time snapshot of the upgrade process. Both of these API requests will return the rackn-system-upgrade-tracking profile, which stores information in its Params about the ongoing state of the profile. The Params in the Profile are as follows:

status

A one-word description of the current status of the upgrade. Can be "staged" for a freshly-staged upgrade, "working" for an in-progress upgrade, "finished" when the upgrade is finished, and "failed" if the upgrade failed.

final-state

A one-line final summary of the upgrade process. It will only be present once the upgrade is finished or failed.

transcript

An array of one line status upgrades on the upgrade process. Can be used to get lower level information on how the upgrade is progressing.

working-node

The API URL of the node in the cluster that is currently being upgraded.

working-state

A one-line description of the current state of the upgrade on the working node.

Cleaning Up an Upgrade

To stop an in-progress upgrade or clean up a previous upgrade attempt, issue a DELETE request to /api/v3/system/upgrade/stage. This will remove the rackn-system-upgrade-tracking profile and remove the drp-upgrade.zip archive and all its components from the server. Any in-progress upgrade actions will exit gracefully.

You can do this via the CLI with drpcli system upgrade unstage.

Legacy Upgrade Process

The legacy process is present on all versions of dr-provision that have the inline-upgrade feature flag. It is deprecated and cannot be used on nodes that are participating in a consensus cluster.

Running a Legacy Upgrade

To run an upgrade, you need an upgrade package. The easiest way to get one is to download it with drpcli catalog item download drp to drp-upgrade.zip, although there are other options.

Once you have the upgrade package downloaded, you can run it by issuing a POST to /api/v3/system/upgrade with the contents of the drp-upgrade file as the body of the request. The system will then unpack the upgrade archive, swap out the binaries, and restart the system.

You can do this via the CLI with drpcli system upgrade drp-upgrade.zip.