Skip to main content
Your Harmstack API key authenticates CLI commands and API requests. Pass it as the HARMSTACK_API_KEY environment variable or with the --harmstack-api-key flag on any command.
1

Download and install the CLI

Download the latest binary from:https://github.com/vettedmedical/harmstack-install-v0/releases/latestChoose the asset for your platform:
  • macOS (Apple Silicon)
  • Windows
  • Linux

​
macOS (Apple Silicon)

# Replace filename with the exact release asset name
curl -L -o harmstack.tar.gz "https://github.com/vettedmedical/harmstack-install-v0/releases/latest/download/harmstack-darwin-arm64.tar.gz"
tar -xzf harmstack.tar.gz
chmod +x harmstack
sudo mv harmstack /usr/local/bin/harmstack

​
Linux

# Replace filename with the exact release asset name
curl -L -o harmstack.tar.gz "https://github.com/vettedmedical/harmstack-install-v0/releases/latest/download/harmstack-linux-amd64.tar.gz"
tar -xzf harmstack.tar.gz
chmod +x harmstack
sudo mv harmstack /usr/local/bin/harmstack

​
Windows (PowerShell)

# Replace filename with the exact release asset name
Invoke-WebRequest -Uri "https://github.com/vettedmedical/harmstack-install-v0/releases/latest/download/harmstack-windows-amd64.zip" -OutFile "harmstack.zip"
Expand-Archive -Path "harmstack.zip" -DestinationPath ".\harmstack-cli"
# Add .\harmstack-cli to PATH or run .\harmstack-cli\harmstack.exe directly
2

Set your Harmstack API key

Export your Harmstack API key so the CLI can authenticate automatically:
export HARMSTACK_API_KEY="your_api_key_here"
If you prefer not to use an environment variable, pass the key directly with --harmstack-api-key on any command.
3

(Optional) Set model defaults

To avoid re-entering model settings in the wizard and commands, export:
export TARGET_MODEL_API_KEY="your_model_api_key_here"
export TARGET_MODEL_ENDPOINT_URL="https://api.openai.com/v1/responses"
The CLI can use these values as defaults.
4

Run the setup wizard

Run harmstack init to launch the interactive wizard:
harmstack init
The wizard prompts you for:
  • Benchmark - the benchmark ID you want to run
  • Model endpoint URL - the URL of your model API (for example https://api.openai.com/v1/responses)
  • Model provider - the API shape: openai, openai_responses, gemini, or raw
  • Model API key - the Bearer token used to call your model endpoint
5

Run non-interactively

Skip prompts entirely by passing all required flags with --consentandskip:
harmstack \
  --haystack \
  --target-model-endpoint="https://api.openai.com/v1/responses" \
  --target-model-api-key="your_model_api_key_here" \
  --provider="openai_responses" \
  --benchmark-id="1" \
  --unit-count="1" \
  --consentandskip
FlagDescription
--haystackRun in haystack mode
--target-model-endpointURL of the model API to benchmark, or set TARGET_MODEL_ENDPOINT_URL
--target-model-api-keyBearer token for your model endpoint, or set TARGET_MODEL_API_KEY
--providerAPI shape: openai, openai_responses, gemini, or raw
--benchmark-idBenchmark ID to run
--unit-countNumber of units to run
--consentandskipConsent to the disclaimer and skip prompts
For CI/CD, keep TARGET_MODEL_API_KEY as a secret environment variable and avoid hardcoding credentials in pipeline files.
6

View results

List recent benchmark jobs:
harmstack list-jobs
Inspect a specific job by UUID:
harmstack show-job --job-id="<uuid>"