Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.harmstack.ai/llms.txt

Use this file to discover all available pages before exploring further.

This guide gets you from zero to a submitted benchmark run. For command and flag details, see the CLI overview.

​
Prerequisites

  • A Harmstack API key (HARMSTACK_API_KEY) from Vetted Medical
  • macOS (Apple Silicon), Linux (amd64), or Windows (amd64)

​
Install the CLI

Binaries and checksums live on GitHub: vettedmedical/harmstack-install-v0 — latest release

​
macOS (Apple Silicon) and Linux (amd64) with Homebrew

brew tap vettedmedical/harmstack-install-v0 https://github.com/vettedmedical/harmstack-install-v0
brew install harmstack
harmstack --help

​
Manual install from a release archive

Download the asset that matches your OS (version numbers differ per release):
  • macOS Apple Silicon: harmstack_<version>_darwin_arm64.tar.gz
  • Linux amd64: harmstack_<version>_linux_amd64.tar.gz
  • Windows amd64: harmstack_<version>_windows_amd64.zip (contains harmstack.exe)
macOS or Linux — extract and put harmstack on your PATH: 
tar -xzf harmstack_*_darwin_arm64.tar.gz
# Linux amd64: use harmstack_*_linux_amd64.tar.gz
chmod +x ./harmstack
mkdir -p "$HOME/.local/bin"
mv ./harmstack "$HOME/.local/bin/harmstack"
export PATH="$HOME/.local/bin:$PATH"
harmstack --help
Windows — unzip the file, move harmstack.exe to a folder on your PATH, open a new terminal, then run harmstack --help. Optional: verify downloads using checksums.txt from the same release. More detail, upgrade paths, and platform notes: CLI overview — Installation.

​
Set your Harmstack API key

export HARMSTACK_API_KEY="your_api_key_here"
For a persistent shell setup (example: zsh): 
echo 'export HARMSTACK_API_KEY="your_api_key_here"' >> ~/.zshrc
source ~/.zshrc

​
Configure your model endpoint (non-interactive)

Export the URL and API key for the model you are benchmarking: 
export TARGET_MODEL_ENDPOINT_URL="https://api.openai.com/v1/responses"
export TARGET_MODEL_API_KEY="sk-proj-1234567"
Harmstack supports multiple provider shapes, including openai, openai_responses, gemini, and raw.

​
Run your first benchmark

This example skips the interactive wizard and submits a Haystack run from the root command: 
harmstack --haystack --consentandskip \
  --provider openai_responses \
  --benchmark-id 2 \
  --unit-count 1
Adjust --benchmark-id, --unit-count, and --provider to match your setup. See Flags for all options.

​
Next steps

CLI overview

Installation depth, authentication, and links to platform guides.

CLI commands

Command inventory and where to find generated metadata.