Skip to content

Batch Operations System Environment

This article outlines the steps to construct a Batch Operations System.

A Batch Operations System is a framework that runs, manages, and reports on a set of batched actions.

To execute batch operations, an environment is required. This environment is represented by a set of runners that handle the operations.

The batch environment consists of a cluster of machines for task execution. While the default DRP Endpoint can be used for this purpose, it's also possible to utilize Context Worker Architecture.

When employing the Context Worker Architecture, the workers can be scaled across multiple machines, allowing for higher load capacity and concurrency.

While constructing the Batch Operations System's Environment, the underlying container context for the worker can be specified to manage specific workloads of the batched requests.

Once the environment setup is finalized, the workers should be put into Work Order mode.

Batch Operation

Upon completion of the Batch Operation System's Environment, it is prepared to handle batch operations, defined by a Batch object.

The Batch object outlines a setup phase, an execution phase, and an optional post phase. Each phase is defined by a Workorder Template in the Batch object.

The Batch Object comprises:

  • Id: Identifier of this batch operation
  • StartTime: Creation time of the batch
  • EndTime: Time when the batch completed all phases
  • Status: Description of the current batch status
  • State: Batch state (for example, created, setup, running, post, finished)
  • SetupWorkOrderTemplate: A Workorder Object that defines how the batch setup should occur
  • SetupWorkOrder: The actual ID of the Workorder created to handle batch setup
  • WorkOrderTemplate: A Workorder Object that defines the individual actions to be batched. The setup phase uses this to schedule work
  • WorkOrderCounts: A map of WorkOrder states containing the counts of the Workorders doing the batch work
  • PostWorkOrderTemplate: A Workorder Object that defines clean-up or post-processing actions for the batch once all Workorders are complete
  • PostWorkOrder: The actual ID of the Workorder performing the Post phase actions

The Batch system begins with the creation of a Batch Object in the DRP server.

Setup Phase

Upon creation of the Batch Object in the server, the Workorder template in the SetupWorkOrderTemplate field will be used to create a new workorder.

The template requires:

  • A Blueprint that will generate all the Workorders for the batch operation
  • A Filter targeting workers for the Batch
  • Additional data to customize the work distribution

The system provides the batch-run blueprint that will use a set of parameters to filter machines to create workers.

The batch-run blueprint requires the following parameters:

  • batch/filter-type: Object prefix to filter upon (e.g., machines)
  • batch/filter: A DRP CLI list string to filter objects (e.g., Params.scannable=Eq(true))
  • batch/iterator: Field on the object from the filter-type object to provide to the created batch workorders
  • batch/iterator-param: Parameter to set in the WorkOrder with the value of the batch/iterator
  • batch/limit: Maximum number of items to take from the list
  • batch/order: A DRP CLI order string to alter the order of machines (e.g., sort=scan/last-scan-time)

The batch-run task (called by the blueprint) will iterate over the batch/filter-type objects using the batch/filter, batch/sort, and batch/limit to get a list of objects to create new workorders for. The WorkOrderTemplate field will be used as the basis for creating a new work order. The batch/iterator field on the object will be stored in the new workorder as the batch/iterator-param in the new workorder. This workorder will be created.

If the setup workorder creation fails, it will be marked as failed. However, the batch will continue to function on the created work workorders.

Additional blueprints could be created that generate different methods for creating the work workorders.

The only requirement is that the blueprint creates workorders with the correct meta data or other work elements.

The Workorder has new meta data fields for tracking usage:

  • workorder-action - The action this workorder is fulling. e.g. setup
  • workorder-owner: 1ede8f64-3c0c-6dff-85cc-c5ccdaf70ae1
  • workorder-owner-type: batches

For the workorders, the workorder-action field is used to track the use of the workorder by the batch system. The values are: setup, work, post The workorder-owner field is used UUID of the owning batch. The workorder-owner-type field is the type of the owning object, e.g. batches.

Running Phase

The server, upon detection of the completion of the Setup order for the batch object, will monitor the completing workorders to update the WorkOrderCounts field, tracking the system state. Once all the work orders have completed, the server will create a Post Workorder from the template in the Batch object, if the Blueprint field is set.

Post Phase

If defined, the Post workorder is executed. This workorder can generate reports or events for failures, or perform any necessary tracking for the batch results. Upon completion of the Post Workorder, or if one is not present, the Batch object is marked complete, and the EndTime is set.