Set up Spikee with Wild West API

Spikee is WithSecure Labs' toolkit for testing LLMs, guardrails and apps against prompt injection and jailbreaks. It reaches models through any-llm, and its custom provider takes any OpenAI-compatible URL.

Set it up

Install with pip install spikee.
Run spikee init to create a workspace.
Add CUSTOM_API_URL=https://wildwestapi.com/v1 and CUSTOM_API_KEY to the workspace .env.
Generate a dataset with spikee generate.
Run spikee test with --target llm_provider and --target-options custom/<model-id>.
Set --judge-options and --attack-options model= to Wild West models, then run spikee results analyze.

Roles

Target. The llm_provider target sends each test to a model. An uncensored model shows what an unguarded backend does, which is the baseline for measuring a guardrail in front of it.
Judge. Datasets that use llm_judge_harmful or llm_judge_output_criteria take their model from --judge-options.
Attacker. Dynamic attacks such as crescendo, goat, prompt_decomposition and llm_jailbreaker take model= in --attack-options.

Background on the attack class: LLM01 prompt injection.

Setup

pip install spikee
spikee init
# workspace/.env
CUSTOM_API_URL=https://wildwestapi.com/v1
CUSTOM_API_KEY=sk-ww-...

spikee generate --seed-folder datasets/seeds-cybersec-2026-01 --format full-prompt

spikee test --dataset datasets/cybersec-2026-01-full-prompt-dataset-*.jsonl \
  --target llm_provider \
  --target-options "custom/qwen3.8-27b-outlaw" \
  --judge-options "custom/glm-5.3-outlaw" \
  --threads 4

To add a multi-turn attack on entries that the static prompt did not break:

spikee test --dataset datasets/cybersec-2026-01-full-prompt-dataset-*.jsonl \
  --target llm_provider --target-options "custom/qwen3.8-27b-outlaw" \
  --attack crescendo --attack-options "max-turns=5,model=custom/outlaw-1"

Spikee only runs an attack when the original entry failed; add --attack-only to skip the static prompts.

Cost estimate

Estimate only. A generated dataset of 1,000 entries at about 600 tokens in and 200 out per target call is 0.6M in and 0.2M out: about $0.66 on qwen3.8-27b-outlaw. Datasets graded by an LLM judge add one judge call per entry, about $1.50 + $0.45 per 1,000 on glm-5.3-outlaw at similar sizes. A crescendo attack with 5 turns adds attacker and target calls for each failed entry.

Tips

--threads defaults to 4 and --throttle to 0 seconds; add a throttle if you hit 429s.
Set SPIKEE_API_TIMEOUT higher for long multi-turn runs.
Set PROVIDER_DEBUG=true to log every prompt and reply when wiring up.
Give each test run its own capped key.

Troubleshooting

Provider not found. Check the prefix is custom/ and run spikee list providers.
401. CUSTOM_API_KEY is missing from the workspace .env.
402. Cap or balance reached; see 402.
Unknown model. The part after custom/ must be an exact id from /models/.

FAQ

How does Spikee connect to an OpenAI-compatible API?

Through its custom provider: set CUSTOM_API_URL and CUSTOM_API_KEY, then use custom/<model-id> anywhere a model is accepted.

Can Spikee test a guardrail rather than a model?

Yes. That is one of its main uses; targets can wrap guardrail products, and an uncensored backend makes the guardrail's effect easy to isolate.

Which model should judge?

glm-5.3-outlaw for accuracy, or glm-5.3-flash-outlaw for cheaper iteration.

Test LLM Guard with Wild West API behind itLLM Guard is Protect AI's open-source library of input and output scanners for LLM apps. The scanners run locally; Wild West API supplies the model in between, and because it does not refuse, the output scanners are tested on their own merit. Run garak against Wild West API as target or judgegarak is NVIDIA's open-source LLM vulnerability scanner. It fires probe prompts at a model and runs detectors over the replies, and it talks to any OpenAI-compatible endpoint through its openai.OpenAICompatible generator. Configure Augustus with Wild West APIAugustus is Praetorian's open-source LLM security scanner, a single Go binary with garak-style probes and detectors plus multi-turn attacks. Its openai.OpenAI generator accepts a base_url, so it can target Wild West API or use it as the attacker and judge.