Runbooks

Runbooks are executable representations of complete or semi-complete conversations. When you want to test a prompt's behavior beyond a single message, runbooks make this effortless. Beyond message content, runbooks allow you to specify the variable data for each message, which enables true-to-life RAG validation, since the expected retrieval results can be injected to your prompt template at the point at which your conversation expects them!

Runbooks are JSON files, and they should be committed to your workspace files following the naming convention *.runbook.json. Here is an example:

{
    "name": "my-first-runbook",
    "description": "A sample runbook.",
    "steps": [
        {
            "utterance": "What time is it?",
            "variables": {
                "time" : "Eleven PM"
            }
        },
        {
            "utterance": "What time is it now?",
            "variables": {
                "time" : "Noon"
            }
        }

    ]
}

Each runbook JSON file should implement the following fields:

Field	Description
name	a descriptive name
description	a description of the runbook and its goals
steps	an array of Step objects - which will be executed in sequence, from first to last.

A runbook needs at least one step to work. Step objects should provide these fields.

Field	Description
utterance	The user input that we want to send
variables	A JSON object that defines the variables that we want to pass to the prompt template for this message

Important Note: If your prompt expects variables, a value for each should be included in every step.

Importing a Runbook

Runbooks in your workspace that follow the naming convention *.runbook.json will be collected and displayed in a treeview in the Promptscape Explorer:

Clicking on any runbook file in this treeview will load it immediately into the Studio, ready for execution against your prompt. Since prompts are not included in runbooks, they should be loaded or input separately.

You can also load runbooks by finding the 'Automation' option in the Chat Panel, clicking it, then selecting 'Load Runbook'. This will open a file explorer, which you can use to locate the runbook file. Once selected, the runbook will load.

Executing Runbook

Once you've loaded a runbook, the Chat Panel will look like this:

The three buttons at the bottom, in order from left to right, are:

1. Play: Plays the runbook all the way through.
2. Next Step: Sends the next message. This is useful for stepping through one message at a time.
3. Reset: Resets the runbook and clears the chat, ready for running again.

The progress bar shows progress through the runbook:

Unloading a Runbook

To return to the regular chat mode, click the 'Unload Runbook' button, seen in the image here. You can do this at any time.