forge-test

# forge-test

Analyze a CmdForge tool config, generate test cases, execute them, and report results as JSON. Classifies tools by step patterns to handle prompt-then-code chains that need synthetic AI responses for testing.

## Installation

This is a CmdForge tool. Install it using:

```bash
cmdforge install forge-test
```

Or publish it to your local registry and install from there.

## Usage

```bash
# Test a tool by name
echo "" | forge-test --name my-tool

# Generate up to 5 test cases
echo "" | forge-test --name summarize --max-cases 5

# Pipe output to another tool for processing
echo "" | forge-test --name code-review | jq '.test_results'
```

## Arguments

| Flag | Default | Description |
|------|---------|-------------|
| `--name` | _(required)_ | Tool name (required) |
| `--max-cases` | `3` | Maximum number of test cases to generate |

## How It Works

1. **analyze_config**: Reads the tool's YAML configuration from `~/.cmdforge/<name>/config.yaml`, analyzes its step structure (prompt vs code steps), and determines the appropriate testing strategy.

2. **generate_tests**: Uses Claude Sonnet to generate test cases based on the tool's purpose, arguments, and step pipeline. Test cases include sample inputs and expected output characteristics.

3. **run_tests**: Executes each generated test case against the tool. For tools with prompt-then-code patterns, injects synthetic AI responses to test the full pipeline without making actual API calls.

4. **format_output**: Formats test execution results as structured JSON, including pass/fail status, actual outputs, and any errors encountered.

The tool outputs a JSON report containing the test cases, execution results, and a summary of the test run.

## Dependencies

None - uses CmdForge's built-in execution engine and the Claude Sonnet provider.