22.25. flexiflow - FlexiFlow¶
The following documentation is for FlexiFlow (flexiflow) content package at version v4.12.0-alpha00.78+gc037aaa40eb3ad853690ce178f9ab8a5bae4c436.
The FlexiFlow system allows an operator to dynamically inject new tasks in to an existing workflow based on a Parameter. This allows for other content to build up Tasks to add an existing workflow flexibly. Some examples may include adding specific tasks based on classification results from the Classify system, or other paths.
Stage Based Manipulation:
The Tasks that are defined in the parameter (specified by the flexiflow/list-parameter) will be composed and added to the task list. These will be dynamically inserted between the flexiflow-start and flexiflow-stop tasks.
Composed means the value of that parameter is the aggregation of all occurances of that parameter at all levels of the system, (e.g. From the parameters on the machine, then profiles on machines, then parameters on the stage, …). This allows multiple resources to provide the FlexiFlow tasks that will eventually be run on the system. Typically, Parameter orders of precedence would override earlier occurrences of the Parameter values. FlexiFlow does not follow that standard path.
The Param that is referenced by flexiflow/list-parameter should be a simple array type Param. It lists each Task that should be executed in the listed sequence by the FlexiFlow process at that Stage in the Workflow.
Workflow Based Manipulation
The Workflow that is defined in the Param named flexiflow-workflow will be run when the stage of the same name is run. This allows the operator to jump in to another Workflow in an existing workflow.
## Operating FlexiFlow Stage Based Manipulation
Here is a brief outline of example usage of the FlexiFlow system utilizing the RackN defined stage. In this example, we will define tasks to run in the flexiflow-stage.
### Using the Stage and Task Injection
In this example we will create a “my-flexiflow-discovery” workflow.
Note that the flexiflow-stage is run after the classify stage. A FlexiFlow stage can be added anywhere that conceptually makes sense.
### Using the FlexiFlow Stage
The FlexiFlow system is comprised of a single stage named flexiflow-stage. The stage must contain ONLY the flexiflow-start and flexiflow-stop Tasks. The flexiflow-start stage will dynamically inject the desired tasks in to the Workflow for the operator. DO NOT add any tasks between these start and stop tasks.
You can choose to use the provided flexiflow-stage, and customize it’s use by adding Params to the Machine that define the Tasks that will run, or you can create a custom Stage. The provided FlexiFlow stage is used the same way that a custom named stage would be used.
To define the custom list of tasks to execute, simply follow the below steps:
Add the flexiflow/list-parameter param to the Machine
The value of which is another Parameter that is an Array of Strings that lists tasks to add
An example is superflex (which would be the Array of Strings)
Insure you create a Param with Array of Strings as the type definition (eg superflex, defined as Array of Strings)
Add the Param (eg superflex) to the Machine with a list of tasks in the array
If you create a custom stage, you can optionally embed the Params that control defining the list of Tasks to insert and/or the actual task list to insert.
Here is an example Stage in YAML that defines which specific Param will contain the list of tasks to be executed. Note that we are not defining the actual list of task in this example. Presumably a previous Stage (possibly Classify) would build up the value of superflex tasks to execute dynamically.
```yaml — Name: “my-superflex-stage” Description: “Perform tasks defined by the ‘superflex’ Param.” Documentation: |
The Param ‘superflex’ is an array that will contain the list of flexiflow tasks to run.
- Params:
flexiflow/list-parameter: “superflex”
- Tasks:
“flexiflow-start”
“flexiflow-stop”
- Meta:
color: “orange” icon: “magic” title: “RackN Content”
- !!! note
Note that as the Documentation field says; the flexiflow/list-parameter for this Stage is defined as superflex. This Parameter must be defined as an array; which is a list of the Tasks to execute during this Stage. The Param can be defined on the Machine in all of the normal ways (directly as a Param, as part of a Profile, or as a Param in the Global Profile, dynamically by other content, etc).
Here is an example stage that defines the custom FlexiFlow stage, which sets the control Parameter name to superflex, and also defines the Tasks that superflex will reference and run.
```yaml — Name: “my-superflex-stage-v2” Description: “Perform tasks to run that are defined by the ‘superflex’ Param.” Documentation: |
The Param ‘superflex’ is an array that will contain the list of additional tasks to run. The actual tasks that are run are also listed in this Stage.
- Params:
flexiflow/list-parameter: superflex superflex:
task1
task2
task3
- Tasks:
“flexiflow-start”
“flexiflow-stop”
- Meta:
color: “orange” icon: “magic” title: “RackN Content”
- !!! warning
The Params that are used in FlexiFlow MUST BE TYPE DEFINED correctly and added to the system before they are used. This use case pattern does NOT support using adhoc Params.
### Create Parameter Definition
It is critical that you correctly type define the Parameter that defines the list of tasks to run in advance of using the Parameter. If you do not, you will receive a warning message and the task injection will fail.
An example type definition for the Parameter (using our superflex example above), would look like:
```yaml — Name: “superflex” Description: “Defines the flexiflow list parameter.” Documentation: |
This param defines the list of Tasks to execute for flexiflow during the stage run. If left empty, no tasks will be run.
- Schema:
type: “array” items:
type: “string”
default: []
- Meta:
color: “blue” icon: “magic” title: “RackN Content”
### Possible Errors
Here is a list of some of the possible errors you may encounter.
Failed to render actions: : template: :35:30: executing “flexiflow_start.sh” at <.ComposeParam>: error calling ComposeParam: Cannot compose freeform param superflex Updated job for flexiflow-stage:flexiflow:flexiflow-start to incomplete
In this case, the superflex list parameter must be defined in advance of using it, the operator has attempted to use the Parameter as an adhoc param. Create a Param type definition as specified in the Create Parameter Definition section above.
## Operating FlexiFlow Workflow Based Manipulation
The FlexiFlow content also allows you to add a Stage to a given workflow. At that Stage, the system will reference the Param named flexiflow/workflow. The value of the param should be the name of an existing Workflow. That workflow will then be executed on the system.
Please review the test-flexiflow-workflow-main Workflow which will dynamically add the test workflow named test-flexiflow-workflow-include. This provides a working example of using the include workflow capability.
22.25.1. Object Specific Documentation¶
22.25.1.1. params¶
The content package provides the following params.
22.25.1.1.1. flexiflow/list-parameter¶
This defines the parameter that the flexiflow-stage stage will use to reference the list of Tasks to dynamically execute. It essentially operates as a “control pointer” to the real list, allowing for flexible usage of a single defined Param to point to a variable list of tasks.
The Param that it points to must be type defined in the system, and it must be an Array of Strings.
22.25.1.1.2. flexiflow/maintenance-mode¶
Indicate that we should go back to local disk.
22.25.1.1.3. flexiflow/no-compose¶
Flexiflow by default will compose the task parameter and merge all values into a single list.
By setting this parameter to true, this will find the first value and stop merging.
22.25.1.1.4. test-flexiflow/parameter¶
Param for testing flexiflow system. This Param contains an Array of Strings that will define which tasks to dynamically add to the flexiflow-stage.
22.25.1.2. stages¶
The content package provides the following stages.
22.25.1.2.1. flexiflow-stage¶
This Stage implements the FlexiFlow capability. When the task flexiflow-start is run, it will reference the paramtger defined in the flexiflow/list-parameter Param. The Param that is referenced must be defined as an Array of Strings. Each array element must be an existing Task on the system. The list of tasks will be dynamically injected in to this stage and run.
Do not insert any tasks between the start and stop tasks.
This task requires that the Param flexiflow/list-parameter exists on the machine to run successfully.
22.25.1.2.2. flexiflow-workflow¶
Allows the operator to set the flexiflow/workflow Parameter to the name of an existing workflow. This will then be added dynamically to the system to be executed immediately at this position in the original workflow.
22.25.1.2.3. test-flexiflow-stage¶
This Stage tests the FlexiFlow capability. It defines that the flexiflow/list-parameter is set to test-flexiflow/parameter, and subsequently sets that Paramter to use the test tasks named test-flexiflow/task1 and test-flexiflow/task2.
22.25.1.2.4. test-flexiflow-start¶
Just provides a marker (decorator) that the flexiflow main stage has started.
22.25.1.2.5. test-flexiflow-stop¶
Just provides a marker (decorator) that the flexiflow main stage has stopped.
22.25.1.2.6. test-flexiflow-workflow¶
Sets the flexiflow/workflow Parameter to test-flexiflow-workflow-include which should then be dynamically included in the test-flexiflow-workflow-main.
22.25.1.3. tasks¶
The content package provides the following tasks.
22.25.1.3.1. flexiflow-start¶
This task begins the FlexiFlow stage. No other tasks should preceed it on the stage, and the only other task that should follow is the flexiflow-stop task.
Tasks will be dynamically injected in to the task list after this task, if they have been specified by the appropriate control Params.
22.25.1.3.2. flexiflow-stop¶
This task ends the FlexiFLow stage. No other tasks should follow it on the stage, and the only other task that should preceed is the flexiflow-start task.
Tasks will be dynamically injected in to the workflow if they have been specified by the appropriate control Params prior to this task.
22.25.1.3.3. test-flexiflow-start¶
Just markes the beginning of the test flexiflow workflow.
22.25.1.3.4. test-flexiflow-stop¶
Just markes the ending of the test flexiflow workflow.
22.25.1.3.5. test-flexiflow-task1¶
This is an example test task that just prints a message and exits successfully.
22.25.1.3.6. test-flexiflow-task2¶
This is an example test task that just prints a message and exits successfully.
