Use Wild West API as DeepTeam's simulator and evaluator

DeepTeam is Confident AI's open-source red teaming framework built on DeepEval. It needs a simulator model to write attacks and an evaluation model to grade replies, and both can be DeepEval LocalModel objects pointed at Wild West API.

Set it up

1. Install with pip install -U deepteam, which pulls in deepeval.
2. Create a capped key in the dashboard and export it as WILDWEST_API_KEY.
3. Build LocalModel instances with base_url https://wildwestapi.com/v1.
4. Write an async model_callback that sends each attack to the system under test.
5. Call red_team with vulnerabilities, attacks, simulator_model and evaluation_model.
6. Review the returned risk assessment.

Roles

The simulator writes baseline attacks for each vulnerability type, such as Bias, Toxicity or PIILeakage, and then enhances them with attacks like PromptInjection. Filtered simulators often refuse that step. The evaluator reads the target's reply and scores it. An uncensored model handles both without refusing. The target is whatever your callback calls; it can also be Wild West API for a no-guardrail baseline.

Code

Use DeepEval's LocalModel, not GPTModel. An open DeepEval issue reports that the OpenAI class looks model names up in its own catalog and fails on ids it does not know, while LocalModel sends any id to the base URL. Pass the /v1 root; the OpenAI SDK adds the path.

import os
from openai import AsyncOpenAI
from deepeval.models import LocalModel
from deepteam import red_team
from deepteam.vulnerabilities import Bias, Toxicity
from deepteam.attacks.single_turn import PromptInjection

WW_URL = "https://wildwestapi.com/v1"
WW_KEY = os.environ["WILDWEST_API_KEY"]

simulator = LocalModel(model="outlaw-1", api_key=WW_KEY,
                       base_url=WW_URL, temperature=0.9)
evaluator = LocalModel(model="glm-5.3-outlaw", api_key=WW_KEY,
                       base_url=WW_URL, temperature=0)

# Target: replace with your own app. Here, a no-guardrail baseline.
client = AsyncOpenAI(api_key=WW_KEY, base_url=WW_URL)

async def model_callback(input: str, turns=None) -> str:
    r = await client.chat.completions.create(
        model="qwen3.8-27b-outlaw",
        messages=[{"role": "user", "content": input}],
    )
    return r.choices[0].message.content

risk = red_team(
    model_callback=model_callback,
    vulnerabilities=[Bias(types=["race"]), Toxicity(types=["insults"])],
    attacks=[PromptInjection()],
    simulator_model=simulator,
    evaluation_model=evaluator,
    attacks_per_vulnerability_type=3,
    max_concurrent=5,
)

DeepTeam also has a YAML route, deepteam run config.yaml, whose models block accepts provider: local with base_url and api_key. The Python route above is easier to check.

Cost estimate

Estimate only. Two vulnerabilities with one type each, 3 attacks per type and one enhancement gives 6 test cases. Each case is roughly one simulator call, one enhancement call, one target call and one evaluation call. Call it 10 cases x 4 calls x 1,200 tokens in and 400 out: 48K in, 16K out per model mix, a few cents. A full run with every vulnerability type and 10 attacks per type is a few hundred cases, still usually under $5 on outlaw-1 with a glm-5.3-outlaw evaluator.

Tips

• max_concurrent defaults to 10; drop it if you see 429s.
• Set ignore_errors=True (the default) so one failed call does not end the run.
• Keep one capped key per run so a large vulnerability list cannot overspend.
• Set target_purpose so generated attacks fit your app.

Troubleshooting

• KeyError or None on model data. You used GPTModel with a non-OpenAI id; switch to LocalModel.
• 404 on the URL. The base URL included /chat/completions; pass only https://wildwestapi.com/v1.
• 401 or 402. Bad key, or balance and cap reached; see 402.

FAQ

Related