Skip to content

Digital Rebar Platform: Remote API Integration — Developer Guide

Implementation Patterns & Code Reference

1. Overview

This guide provides working implementation patterns for DRP task developers building Pipeline automation that interacts with remote API services. It covers template rendering mechanics, practical curl-based bash examples, and idempotency patterns ready for adaptation into DRP task scripts.

For the architectural concepts, design decisions, and decision frameworks that govern remote API integration — including when to use each pattern and why — see the companion API Integration Architect Guide

Warning

All code examples use placeholder values (https://api.example.com, ${TOKEN}, ${JOB_ID}, etc.). Replace them with the actual endpoint URLs, credential mechanisms, and field names defined by the target API. Always store credentials as DRP Parameters or external secrets — never hardcode them in task scripts.

DRP does also integrate with external secrets/vault services to dynamically request secrets during render time without storing them in the Infrastructure-as-Code (Iac) at rest.

2. Template Rendering in Task Scripts

The bash script examples throughout this guide use Go template rendering constructs as processed by the DRP task engine at runtime. Before a task script is executed, DRP renders the script through Go's text/template engine, substituting template expressions with live values from Parameters, the Machine object, its associated Profiles, and the Global Profile. This means the script delivered to the runner already contains fully resolved values — no runtime drpcli calls are needed to retrieve Parameters.

In addition to standard Go template syntax, DRP exposes a library of RackN-provided template functions that simplify common provisioning patterns. The examples in this guide illustrate several of these features:

Feature Example Description
Param lookup {{ .Param "hostname" }} Returns the value of a Parameter resolved through the Machine's full Profile stack, or nil if not set
Param with expansion {{ .ParamExpand "api-token" }} Like .Param, but additionally expands any DRP object references embedded in the value
Conditional Param {{ if (.Param "hostname") }}...{{ end }} Tests whether a Parameter is set before using it, enabling safe fallback logic
Machine field access {{ .Machine.ShortName }} Accesses a built-in field directly from the Machine object
Param-or-field fallback {{ if (.Param "hostname") }}{{ .ParamExpand "hostname" }}{{ else }}{{ .Machine.ShortName }}{{ end }} Prefers an explicit Parameter when set, falls back to a Machine field — a common DRP idiom for flexible, reusable tasks

Note

Go template expressions are enclosed in {{ }} and are evaluated before the script runs. A script that contains TOKEN='{{ .ParamExpand "api-token" }}' will have the token value substituted inline at render time, producing a plain bash variable assignment by the time the runner executes it.

3. Practical Examples Using curl

The examples below translate each architectural concept into concrete bash code. They are written for clarity and portability rather than production completeness. Comments call out the specific pattern being illustrated in each block.

3.1 Synchronous API Call with Inline Response Handling

The simplest pattern: make a call, check the HTTP status code, parse the response body, and store a relevant value back into DRP.

Bash
#!/usr/bin/env bash
# Pattern: Synchronous call — read result immediately

API_BASE="https://api.example.com/v1"
TOKEN='{{ .ParamExpand "api-token" }}'

# -s  = silent (no progress bar)
# -o  = write body to file
# -w  = write HTTP status code to stdout
HTTP_CODE=$(curl -s \
  -H "Authorization: Bearer ${TOKEN}" \
  -o /tmp/api_response.json \
  -w '%{http_code}' \
  "${API_BASE}/servers/${RS_UUID}/details")

# Check for non-success before attempting to parse
if [[ "${HTTP_CODE}" != "200" ]]; then
  echo "API call failed with HTTP ${HTTP_CODE}"
  cat /tmp/api_response.json   # surface error detail to DRP task log
  exit 1
fi

# Parse the field we need using jq
ASSIGNED_IP=$(jq -r '.network.primaryIp' /tmp/api_response.json)

# Store state back on the DRP Machine object for downstream Stages
drpcli machines set $RS_UUID param assigned-ip to "\"${ASSIGNED_IP}\""
echo "Captured assigned IP: ${ASSIGNED_IP}"

3.2 Asynchronous API Call with Polling Loop

Many long-running operations follow the submit-then-poll pattern. The initial call returns a job identifier; subsequent calls check the job status until a terminal state is reached. See section 3.5 for the idempotent version of this pattern that handles task re-runs safely.

Bash
#!/usr/bin/env bash
# Pattern: Asynchronous call — submit job, then poll until complete

API_BASE="https://api.example.com/v1"
TOKEN='{{ .ParamExpand "api-token" }}'

# ── Step 1: Submit the asynchronous job ──────────────────────────────────────
HTTP_CODE=$(curl -s -X POST \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer ${TOKEN}" \
  -d '{ "action": "provision", "profile": "base-linux" }' \
  -o /tmp/submit_response.json \
  -w '%{http_code}' \
  "${API_BASE}/servers/${RS_UUID}/jobs")

if [[ "${HTTP_CODE}" != "202" ]]; then
  echo "Job submission failed with HTTP ${HTTP_CODE}"
  cat /tmp/submit_response.json
  exit 1
fi

JOB_ID=$(jq -r '.jobId' /tmp/submit_response.json)
echo "Job submitted: ${JOB_ID}"

# ── Step 2: Poll until terminal state ────────────────────────────────────────
MAX_WAIT=600   # seconds
INTERVAL=15    # poll every 15 seconds
ELAPSED=0

while [[ $ELAPSED -lt $MAX_WAIT ]]; do
  curl -s \
    -H "Authorization: Bearer ${TOKEN}" \
    -o /tmp/job_status.json \
    "${API_BASE}/jobs/${JOB_ID}"

  STATUS=$(jq -r '.status' /tmp/job_status.json)
  echo "Job ${JOB_ID} status: ${STATUS} (${ELAPSED}s elapsed)"

  case "${STATUS}" in
    "completed")  echo "Job completed successfully"; exit 0 ;;
    "failed")     echo "Job failed:"; jq '.errorMessage' /tmp/job_status.json; exit 1 ;;
    "cancelled")  echo "Job was cancelled remotely"; exit 1 ;;
    *)            : ;;   # still running — keep polling
  esac

  sleep $INTERVAL
  ELAPSED=$(( ELAPSED + INTERVAL ))
done

echo "Timed out waiting for job ${JOB_ID} after ${MAX_WAIT}s"
exit 1

3.3 Retry Logic with Exponential Backoff

Wrap any API call that may experience transient failures in a retry loop. The pattern below implements exponential backoff and distinguishes retryable (5xx) from non-retryable (4xx) errors.

Bash
#!/usr/bin/env bash
# Pattern: Retry with exponential backoff

API_BASE="https://api.example.com/v1"
TOKEN='{{ .ParamExpand "api-token" }}'
TRIES='{{ .ParamExpand "service-retries" }}'

DELAY=2
SUCCESS=false

for attempt in $(seq 1 $TRIES); do
  echo "Attempt ${attempt} of ${TRIES}"

  HTTP_CODE=$(curl -s \
    -H "Authorization: Bearer ${TOKEN}" \
    -o /tmp/api_response.json \
    -w '%{http_code}' \
    "${API_BASE}/resource/${RS_UUID}")

  if [[ "${HTTP_CODE}" == "200" ]]; then
    SUCCESS=true
    break
  fi

  # 4xx errors are client errors — retrying will not help
  if [[ "${HTTP_CODE}" -ge 400 && "${HTTP_CODE}" -lt 500 ]]; then
    echo "Non-retryable client error: HTTP ${HTTP_CODE}"
    cat /tmp/api_response.json
    exit 1
  fi

  # 5xx or network errors — back off and retry
  echo "Transient error HTTP ${HTTP_CODE}, retrying in ${DELAY}s..."
  sleep $DELAY
  DELAY=$(( DELAY * 2 ))       # exponential backoff
  [[ $DELAY -gt 60 ]] && DELAY=60   # cap at 60 seconds
done

if [[ "${SUCCESS}" != "true" ]]; then
  echo "All ${TRIES} attempts failed"
  exit 1
fi

# Process successful response
RESULT=$(jq -r '.result' /tmp/api_response.json)
drpcli machines set $RS_UUID param last-api-result to "\"${RESULT}\""
echo "Success: ${RESULT}"

3.4 Idempotent Synchronous Operation

Query the current state first, and only apply the change if the resource is not already in the desired state. Treats "already correct" as a success rather than an error.

Bash
#!/usr/bin/env bash
# Pattern: Idempotent synchronous operation — check before acting

API_BASE="https://api.example.com/v1"
TOKEN='{{ .ParamExpand "api-token" }}'
DESIRED_HOSTNAME='{{ if (.Param "hostname") }}{{ .ParamExpand "hostname" }}{{ else }}{{ .Machine.ShortName }}{{ end }}'

# Step 1: Query current state
curl -s \
  -H "Authorization: Bearer ${TOKEN}" \
  -o /tmp/current_state.json \
  "${API_BASE}/servers/${RS_UUID}"

CURRENT_HOSTNAME=$(jq -r '.hostname' /tmp/current_state.json)

# Step 2: Compare — skip the update if already in desired state
if [[ "${CURRENT_HOSTNAME}" == "${DESIRED_HOSTNAME}" ]]; then
  echo "Hostname already set to ${DESIRED_HOSTNAME} — no action required"
  exit 0
fi

# Step 3: Apply the change only if needed
PATCH_DOC=$(jq -n --arg h "${DESIRED_HOSTNAME}" \
  '[{ "op": "replace", "path": "/hostname", "value": $h }]')

HTTP_CODE=$(curl -s -X PATCH \
  -H 'Content-Type: application/json-patch+json' \
  -H "Authorization: Bearer ${TOKEN}" \
  -d "${PATCH_DOC}" \
  -o /tmp/patch_response.json \
  -w '%{http_code}' \
  "${API_BASE}/servers/${RS_UUID}")

if [[ "${HTTP_CODE}" != "200" && "${HTTP_CODE}" != "204" ]]; then
  echo "Patch failed with HTTP ${HTTP_CODE}"
  cat /tmp/patch_response.json
  exit 1
fi

echo "Hostname updated to ${DESIRED_HOSTNAME}"

3.5 Idempotent Asynchronous Operation

Persists the job ID as a DRP Parameter immediately after submission so that re-runs resume monitoring the existing job rather than creating a duplicate.

Bash
#!/usr/bin/env bash
# Pattern: Idempotent asynchronous operation — resume or submit

API_BASE="https://api.example.com/v1"
TOKEN='{{ .ParamExpand "api-token" }}'

# Step 1: Check for an existing job ID from a previous run
JOB_ID='{{ .ParamExpand "provision-job-id" }}'

if [[ -z "${JOB_ID}" || "${JOB_ID}" == "null" ]]; then
  # Step 2: No existing job — submit a new one
  echo "No existing job found, submitting..."
  HTTP_CODE=$(curl -s -X POST \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer ${TOKEN}" \
    -d '{ "action": "provision", "profile": "base-linux" }' \
    -o /tmp/submit_response.json \
    -w '%{http_code}' \
    "${API_BASE}/servers/${RS_UUID}/jobs")

  if [[ "${HTTP_CODE}" != "202" ]]; then
    echo "Job submission failed with HTTP ${HTTP_CODE}"
    cat /tmp/submit_response.json
    exit 1
  fi

  JOB_ID=$(jq -r '.jobId' /tmp/submit_response.json)

  # Step 3: Persist the job ID immediately — before polling begins
  drpcli machines set $RS_UUID param provision-job-id to "\"${JOB_ID}\""
  echo "Job submitted and recorded: ${JOB_ID}"
else
  echo "Resuming monitoring of existing job: ${JOB_ID}"
fi

# Step 4: Poll until terminal state (shared path for new and resumed jobs)
MAX_WAIT=600
INTERVAL=15
ELAPSED=0

while [[ $ELAPSED -lt $MAX_WAIT ]]; do
  curl -s \
    -H "Authorization: Bearer ${TOKEN}" \
    -o /tmp/job_status.json \
    "${API_BASE}/jobs/${JOB_ID}"

  STATUS=$(jq -r '.status' /tmp/job_status.json)
  echo "Job ${JOB_ID} status: ${STATUS} (${ELAPSED}s elapsed)"

  case "${STATUS}" in
    "completed")
      # Clear the job ID so an intentional future re-run starts fresh
      drpcli machines remove $RS_UUID param provision-job-id
      echo "Job completed successfully"
      exit 0
      ;;
    "failed")
      drpcli machines remove $RS_UUID param provision-job-id
      echo "Job failed:"
      jq '.errorMessage' /tmp/job_status.json
      exit 1
      ;;
    "cancelled")
      drpcli machines remove $RS_UUID param provision-job-id
      echo "Job was cancelled remotely"
      exit 1
      ;;
    *)
      : ;;   # still running — keep polling
  esac

  sleep $INTERVAL
  ELAPSED=$(( ELAPSED + INTERVAL ))
done

echo "Timed out waiting for job ${JOB_ID} after ${MAX_WAIT}s"
exit 1

3.6 JSON Patch Update

When modifying only specific fields of a remote resource, send a JSON Patch request rather than a full replacement payload.

Bash
#!/usr/bin/env bash
# Pattern: JSON Patch — targeted field update

API_BASE="https://api.example.com/v1"
TOKEN='{{ .ParamExpand "api-token" }}'

# Read the new value from a DRP Machine Parameter set by a prior Stage
NEW_HOSTNAME='{{ if (.Param "hostname") }}{{ .ParamExpand "hostname" }}{{ else }}{{ .Machine.ShortName }}{{ end }}'

# Build the JSON Patch document dynamically using jq
PATCH_DOC=$(jq -n \
  --arg hostname "${NEW_HOSTNAME}" \
  '[{ "op": "replace", "path": "/hostname", "value": $hostname }]')

HTTP_CODE=$(curl -s -X PATCH \
  -H 'Content-Type: application/json-patch+json' \
  -H "Authorization: Bearer ${TOKEN}" \
  -d "${PATCH_DOC}" \
  -o /tmp/patch_response.json \
  -w '%{http_code}' \
  "${API_BASE}/servers/${RS_UUID}")

if [[ "${HTTP_CODE}" != "200" && "${HTTP_CODE}" != "204" ]]; then
  echo "Patch failed with HTTP ${HTTP_CODE}"
  cat /tmp/patch_response.json
  exit 1
fi

echo "Hostname updated to ${NEW_HOSTNAME}"

3.7 Authentication Handling

Most production APIs require authentication. The most common mechanisms and their curl implementation patterns are described below.

Bearer Token (OAuth 2.0 / JWT)

Bash
# Token stored as a DRP Machine Parameter, retrieved at runtime
TOKEN='{{ .ParamExpand "api-bearer-token" }}'

curl -s \
  -H "Authorization: Bearer ${TOKEN}" \
  https://api.example.com/v1/resource

API Key in Header

Bash
API_KEY='{{ .ParamExpand "service-api-key" }}'

curl -s \
  -H "X-API-Key: ${API_KEY}" \
  https://api.example.com/v1/resource

Token Acquisition (OAuth 2.0 Client Credentials)

Bash
CLIENT_ID='{{ .ParamExpand "oauth-client-id" }}'
CLIENT_SECRET='{{ .ParamExpand "oauth-client-secret" }}'

# Obtain a short-lived access token
TOKEN=$(curl -s -X POST \
  -d "grant_type=client_credentials" \
  -d "client_id=${CLIENT_ID}" \
  -d "client_secret=${CLIENT_SECRET}" \
  https://auth.example.com/oauth2/token | jq -r '.access_token')

# Use the token for subsequent calls within this Stage
curl -s \
  -H "Authorization: Bearer ${TOKEN}" \
  https://api.example.com/v1/resource

3.8 Rate Limiting

Many APIs enforce rate limits. When a rate limit is hit, the service typically returns HTTP 429 Too Many Requests and may include a Retry-After header indicating how long to wait. Respect these signals rather than hammering the endpoint.

Bash
#!/usr/bin/env bash
# Pattern: Handle HTTP 429 rate limit with Retry-After

API_BASE="https://api.example.com/v1"
TOKEN='{{ .ParamExpand "api-token" }}'
TRIES='{{ .ParamExpand "service-retries" }}'

for attempt in $(seq 1 $TRIES); do
  # Capture response headers to extract Retry-After if present
  HTTP_CODE=$(curl -s \
    -H "Authorization: Bearer ${TOKEN}" \
    -D /tmp/response_headers.txt \
    -o /tmp/api_response.json \
    -w '%{http_code}' \
    "${API_BASE}/resource")

  if [[ "${HTTP_CODE}" == "200" ]]; then
    break
  fi

  if [[ "${HTTP_CODE}" == "429" ]]; then
    # Read Retry-After header (value is in seconds)
    RETRY_AFTER=$(grep -i '^Retry-After:' /tmp/response_headers.txt \
      | awk '{print $2}' | tr -d '\r')
    WAIT=${RETRY_AFTER:-30}   # default to 30s if header absent
    echo "Rate limited. Waiting ${WAIT}s before retry..."
    sleep $WAIT
    continue
  fi

  echo "Unexpected HTTP ${HTTP_CODE}"
  cat /tmp/api_response.json
  exit 1
done

RESULT=$(jq -r '.value' /tmp/api_response.json)
echo "Result: ${RESULT}"

Digital Rebar Platform | Remote API Integration — Developer Guide | For internal and customer use