Runbooks
Runbooks are executable representations of complete or semi-complete conversations. When you want to test a prompt's behavior beyond a single message, runbooks make this effortless. Beyond message content, runbooks allow you to specify the variable data for each message, which enables true-to-life RAG validation, since the expected retrieval results can be injected to your prompt template at the point at which your conversation expects them!
Runbooks are JSON files, and they should be committed to your workspace files following the naming convention *.runbook.json
. Here is an example:
{
"name": "my-first-runbook",
"description": "A sample runbook.",
"steps": [
{
"utterance": "What time is it?",
"variables": {
"time" : "Eleven PM"
}
},
{
"utterance": "What time is it now?",
"variables": {
"time" : "Noon"
}
}
]
}
Each runbook JSON file should implement the following fields:
Field | Description |
---|---|
name | a descriptive name |
description | a description of the runbook and its goals |
steps | an array of Step objects - which will be executed in sequence, from first to last. |
A runbook needs at least one step to work. Step objects should provide these fields.
Field | Description |
---|---|
utterance | The user input that we want to send |
variables | A JSON object that defines the variables that we want to pass to the prompt template for this message |
Important Note: If your prompt expects variables, a value for each should be included in every step.
Importing a Runbook
Runbooks in your workspace that follow the naming convention *.runbook.json
will be collected and displayed in a treeview in the Promptscape Explorer:
Clicking on any runbook file in this treeview will load it immediately into the Studio, ready for execution against your prompt. Since prompts are not included in runbooks, they should be loaded or input separately.
You can also load runbooks by finding the 'Automation' option in the Chat Panel, clicking it, then selecting 'Load Runbook'. This will open a file explorer, which you can use to locate the runbook file. Once selected, the runbook will load.
Executing Runbook
Once you've loaded a runbook, the Chat Panel will look like this:
The three buttons at the bottom, in order from left to right, are:
- Play: Plays the runbook all the way through.
- Next Step: Sends the next message. This is useful for stepping through one message at a time.
- Reset: Resets the runbook and clears the chat, ready for running again.
The progress bar shows progress through the runbook:
Unloading a Runbook
To return to the regular chat mode, click the 'Unload Runbook' button, seen in the image here. You can do this at any time.