Find tools › Open-source projects › Test & monitor AI apps

promptfoo

Open-source project · by promptfoo

A free tool that tests your AI prompts and chatbots automatically and flags bad or unsafe answers.

Some setupPopular4,832,983 installs a month

What it costs

Free and open source; you may pay for the AI model it uses.

Use it to

  • Compare answers from several AI models side by side
  • Check a company chatbot for unsafe or leaky replies
  • Catch worse answers after you change a prompt

How to start

  1. Install Node.js, then in a terminal run npm install -g promptfoo.

  2. Create a starter project with promptfoo init --example getting-started.

  3. Add your AI provider key, then run promptfoo eval and promptfoo view to see results.

Good to know. The project claims evals run 100% locally, but you must still supply your own LLM API keys which send data to external providers unless using purely local models.

Before you install

  • Checked by a personWe read what it does before listing it.
  • License: MITYou are allowed to use it under these terms.
  • Last updated 2026-11-18

More detail

It is a command-line tool and library for testing prompts, agents, and RAG applications, featuring automated evaluations, red teaming, and model comparison.

It helps developers move away from trial-and-error by providing metrics, security vulnerability scans, and CI/CD checks for LLM applications. Reach for it when you need to rigorously test output quality or pentest an AI app before shipping.

Similar tools

Descriptions are written in plain English from each project's own documentation and refreshed by our agent; this one was last checked 2026-09-29. Spotted a mistake? · Open in Find tools · © 2026 Which AI Map