Batch Operations System Environment¶
This article outlines the steps to construct a Batch Operations System.
A Batch Operations System is a framework that runs, manages, and reports on a set of batched actions.
To execute batch operations, an environment is required. This environment is represented by a set of runners that handle the operations.
The batch environment consists of a cluster of machines for task execution. While the default DRP Endpoint can be used for this purpose, it's also possible to utilize Context Worker Architecture.
When employing the Context Worker Architecture, the workers can be scaled across multiple machines, allowing for higher load capacity and concurrency.
While constructing the Batch Operations System's Environment, the underlying container context for the worker can be specified to manage specific workloads of the batched requests.
Once the environment setup is finalized, the workers should be put into Work Order mode.
Upon completion of the Batch Operation System's Environment, it is prepared to handle batch operations, defined by a Batch object.
The Batch object outlines a setup phase, an execution phase, and an optional post phase. Each phase is defined by a Workorder Template in the Batch object.
The Batch Object comprises:
Id: Identifier of this batch operation
StartTime: Creation time of the batch
EndTime: Time when the batch completed all phases
Status: Description of the current batch status
State: Batch state (for example,
SetupWorkOrderTemplate: A Workorder Object that defines how the batch setup should occur
SetupWorkOrder: The actual ID of the Workorder created to handle batch setup
WorkOrderTemplate: A Workorder Object that defines the individual actions to be batched. The setup phase uses this to schedule work
WorkOrderCounts: A map of WorkOrder states containing the counts of the Workorders doing the batch work
PostWorkOrderTemplate: A Workorder Object that defines clean-up or post-processing actions for the batch once all Workorders are complete
PostWorkOrder: The actual ID of the Workorder performing the Post phase actions
The Batch system begins with the creation of a Batch Object in the DRP server.
Upon creation of the Batch Object in the server, the Workorder template in the
SetupWorkOrderTemplate field will be used to create a new workorder.
The template requires:
- A Blueprint that will generate all the Workorders for the batch operation
- A Filter targeting workers for the Batch
- Additional data to customize the work distribution
The system provides the
batch-run blueprint that will use a set of parameters to filter machines to create workers.
batch-run blueprint requires the following parameters:
batch/filter-type: Object prefix to filter upon (e.g.,
batch/filter: A DRP CLI list string to filter objects (e.g.,
batch/iterator: Field on the object from the filter-type object to provide to the created batch workorders
batch/iterator-param: Parameter to set in the WorkOrder with the value of the batch/iterator
batch/limit: Maximum number of items to take from the list
batch/order: A DRP CLI order string to alter the order of machines (e.g.,
batch-run task (called by the blueprint) will iterate over the
batch/filter-type objects using the
batch/limit to get
a list of objects to create new workorders for. The
WorkOrderTemplate field will be used as the basis for creating a new work order. The
on the object will be stored in the new workorder as the
batch/iterator-param in the new workorder. This workorder will be created.
If the setup workorder creation fails, it will be marked as failed. However, the batch will continue to function on the created work workorders.
Additional blueprints could be created that generate different methods for creating the work workorders.
The only requirement is that the blueprint creates workorders with the correct meta data or other work elements.
The Workorder has new meta data fields for tracking usage:
- workorder-action - The action this workorder is fulling. e.g. setup
- workorder-owner: 1ede8f64-3c0c-6dff-85cc-c5ccdaf70ae1
- workorder-owner-type: batches
For the workorders, the
workorder-action field is used to track the use of the workorder by the batch system. The values are:
workorder-owner field is used UUID of the owning batch. The
workorder-owner-type field is the type of the owning object, e.g.
The server, upon detection of the completion of the Setup order for the batch object, will monitor the completing workorders
to update the
WorkOrderCounts field, tracking the system state. Once all the work orders have completed, the server will
create a Post Workorder from the template in the Batch object, if the Blueprint field is set.
If defined, the Post workorder is executed. This workorder can generate reports or events for failures, or perform any
necessary tracking for the batch results. Upon completion of the Post Workorder, or if one is not present, the Batch object
is marked complete, and the
EndTime is set.