Software Testing Agents
This project demonstrates a multi-agent software testing system built using Coral Protocol, supporting agents from three different frameworks—LangChain, CAMEL-AI, and CrewAI. The system enables automatic understanding of codebases, pull request testing, and test coverage analysis in any compatible GitHub repository.
Key Features
The system currently supports four main functionalities:
- Comprehensive Repository Understanding
Automatically analyzes a GitHub repository and provides high-level summaries and usage instructions. - Unit Test Execution for New PRs
Automatically runs unit tests related to code changes in new pull requests. - Unit Test Coverage Evaluation for New PRs
Reviews code changes in a pull request, evaluates whether all necessary cases are covered by existing unit tests, and suggests improvements if needed. - GitHub Documentation Consistency Checking
Checks if documentation files (e.g., README, API docs) related to a PR have become out-of-date due to recent changes and suggests necessary updates.
Getting Started
1. Clone Repositories
First, clone both required repositories:
2. Set Up the Python Environment
It is highly recommended to use a virtual environment (venv
or conda
):
With venv
:
Or with conda
:
3. Install Dependencies
Install all required packages:
4. Configure Environment Variables
Export your OPENAI_API_KEY
and GITHUB_ACCESS_TOKEN
:
5. Start the MCP Server
Navigate to the coral-server
directory and run:
Note: Gradle may appear to stall at 83%, but the server is running. Check the terminal logs to confirm.
6. Launch Agents (in seven separate terminals)
Navigate to the software-testing-agents
directory and open seven terminals and run the following scripts in the software-testing-agents
directory:
Usage Examples
1. Repository Understanding
Ask the Interface Agent for a comprehensive summary of a repository:
Or specify a branch:
Or for other public repositories:
2. Unit Test Execution for New PRs
Ask the system to execute all relevant unit tests for a PR:
3. Unit Test Coverage Evaluation
Ask the system to evaluate whether a PR’s changes are fully covered by tests:
I created a new branch, `new-semantic-scholar-toolkit`, in the repository 'renxinxing123/camel-software-testing' and opened a new pull request (#3). For the changed files, could you please help me check whether the corresponding unit tests fully cover all necessary cases? Are there any additional tests that should be added?
4. Documentation Consistency Checking
Ask the system to automatically check whether relevant documentation files are up-to-date with PR changes:
I created a new branch 'repo-understanding+unit-test-advice' in the repo 'renxinxing123/software-testing-agents-test' and opened a new PR (#2), could you please help me check if the relevant doc covered all the changes from the PR?
Notes
- When running tests, the system identifies the relevant unit test files and executes all test cases within them for reliable coverage.
- Documentation consistency checking will analyze README, API docs, and other related files that may be affected by a PR and flag any outdated or missing documentation.
- The project is designed for easy extension—feel free to add new agent scripts or tools!
- Switching CodeDiffReviewAgent to Groq Llama 3 70B may help reduce cost and increase speed, but the quality of system coverage might be slightly affected compared to GPT-4.1.
👉 Check out the full implementation on GitHub: Coral-Protocol/software-testing-agents