The Testing Sandbox

There's nothing worse than deploying a tool and watching it fail on real data. The Testing Sandbox lets you run individual steps, inspect variables, and squash bugs—all without burning through API credits or waiting for full pipeline runs.

Why Test Steps Individually?

Imagine a 5-step tool that's failing. Is it step 1? Step 4? The output template? Running the whole thing over and over is slow and expensive. Step-by-step testing lets you:

Opening the Testing Sandbox

The Testing Sandbox is built into the Visual Builder. Here's how to access it:

1. Open the Visual Builder: cmdforge
2. Navigate to My Tools and select a tool (or create a new one)
3. Click Edit to open the tool builder
4. Find the step you want to test
5. Click the Test button on that step

The Testing Sandbox dialog opens with everything you need:

Testing a Prompt Step

Prompt steps call your AI provider. Here's how to test one:

1. Set Your Test Input

In the input text area, enter the text you want to process:

The quick brown fox jumps over the lazy dog.
This is a sample sentence for testing purposes.

2. Set Variables

If your prompt uses variables like {language} or {max_length}, fill them in the variables table:

3. Choose a Provider

Select which provider to use for this test. You can:

• Use the step's default provider
• Override with a different provider
• Use mock for testing without API calls

4. Run the Test

Click Run Test and watch the magic happen. You'll see:

• Output - What the AI returned
• Output Variable - The value stored in output_var
• Timing - How long the call took (in milliseconds)
• Status - Success or error with details

Testing a Code Step

Code steps run Python. The sandbox lets you verify your logic:

Set Up Variables

Code steps often depend on outputs from previous steps. Enter those values manually:

View All Output Variables

After running, you'll see every variable your code step creates:

emails = ['[email protected]', '[email protected]']
unique_count = 2
is_valid = True

Testing a Tool Step

Tool steps call other tools. Testing them works the same way:

1. Set the input that will be sent to the tool
2. Configure any arguments the tool needs
3. Optionally override the provider
4. Run and verify the output

This is incredibly useful for debugging composed tools—you can verify each tool in the chain is receiving and producing the right data.

Testing Without API Calls

Don't want to burn through API credits while iterating? Use the mock provider:

Option 1: Override in the Sandbox

Select mock from the provider dropdown before running your test.

Option 2: Create a Smart Mock

Add a mock provider that returns realistic test data:

# In ~/.cmdforge/providers.yaml
providers:
  - name: mock
    command: 'echo "This is a mock response for testing"'

  - name: mock-json
    command: 'echo "{"status": "ok", "count": 42}"'

  - name: mock-summary
    command: 'echo "This is a summary of the input text."'

- name: echo-mock
  command: 'cat'  # Just returns whatever is piped in

Common Testing Scenarios

Scenario: "My variable is empty"

Check these in order:

1. Is the variable name spelled correctly? (output_var matches what you're using)
2. Did the previous step actually set it? (Test that step first)
3. Are you using the right syntax? ({varname} not $varname)

Scenario: "Provider call failed"

Verify:

1. Is the provider configured in ~/.cmdforge/providers.yaml?
2. Does the provider CLI work on its own? Try: echo "test" | claude -p
3. Are API keys/auth set up correctly?

Scenario: "Code step error"

The error message shows the Python exception. Common causes:

• NameError - Variable from previous step not available (test that step)
• SyntaxError - Check your Python indentation and syntax
• TypeError - You're treating a string as a list, or vice versa

The Testing Workflow

Here's how the pros do it:

Next Steps

• Compose tools together - Build powerful pipelines
• Master code steps - Add Python processing
• Publish your tested tool - Share with confidence