Find tools › Open-source projects › Test & monitor AI apps

issue-rag

Open-source project · by zaxiu88888

A tool that tests how accurately AI answers technical questions using real issue reports.

Some setup

What it costs

Free and open source; you may pay for the AI model it uses.

Use it to

  • Harvest maintainer replies from GitHub issues
  • Measure accuracy gains from retrieval pipelines
  • Audit AI judge scoring against human reviews

How to start

  1. Install dependencies with pip install -e ".[models,dev]"

  2. Harvest issue titles and maintainer comments from GitHub

  3. Build the search corpus and review generated test questions

  4. Run evaluations using local model weights or external provider keys

Good to know. The retrieval pipeline currently only supports English text queries and corpus documents.

Before you install

  • Found automaticallyNot reviewed by a person yet. Check it before installing.
  • License: MITYou are allowed to use it under these terms.
  • Last updated 2026-11-04

More detail

A system to harvest GitHub issues, test document retrieval pipelines against maintainer answers, and audit LLM grader accuracy.

AI models often hallucinate outdated or incorrect library version details when answering support questions. Use this tool to build a realistic test set from actual maintainer replies and measure how much retrieval actually improves answer quality.

Similar tools

Descriptions are written in plain English from each project's own documentation and refreshed by our agent; this one was last checked 2026-11-07. Spotted a mistake? · Open in Find tools · © 2026 Which AI Map