IntelliJ IDEA, a JetBrains IDE
@intellijidea
Most Agent Skills focus on things like code style or commit messages, but they can do so much more. Here’s one that debugs flaky tests 👇 https://t.co/TymaYbdslT
4retweets16likes
View on XJetBrains Teaches AI Agents to Deterministically Debug Flaky Tests
IntelliJ IDEA, the professional development environment from JetBrains, released a specialized Agent Skill (modular AI instructions) to debug flaky tests. The update includes IntelliJ coverage agent v1.0.774, which generates text-based reports that allow AI agents to read and compare code execution paths directly.
- Tool version
- IntelliJ coverage agent 1.0.774
- Report format
- Plain text (hit-count enabled)
- Standard
- Agent Skills (agentskills.io)
- Availability
- Maven Central and GitHub
- Supported languages
- Java and Kotlin
This builds on the JetBrains Agent Skills implementation by giving models deterministic tools to verify reasoning. Flaky tests are hard to reproduce, but by automating the "diffing" of hit-count coverage data, agents can identify exactly where execution diverges—such as in a race condition—rather than just guessing based on code.
You can implement this by adding the SKILL.md file to your project, mirroring the Agent Skills open standard adopted by other platform providers. The updated coverage agent is on Maven Central, and the debugging Skill is on GitHub. This works with any compatible agentic tool, including Claude Code.
Still wondering? A few quick answers below.
Agent Skills are a modular standard for packaging AI instructions and domain-specific expertise. Instead of prompting an AI agent with the same instructions repeatedly, you define them once in a plain text document. This allows agents to reuse specific steps, conventions, or knowledge to solve complex tasks like debugging or code style enforcement.
The agent uses a specialized Skill to run a test repeatedly until it captures both passing and failing results. It then uses the IntelliJ coverage agent to generate hit-count reports for both states. By comparing these reports, the agent identifies exactly where the execution paths diverge, allowing it to pinpoint the root cause of the flakiness.
Yes, the updated IntelliJ coverage agent version 1.0.774 is available on Maven Central. This version includes the new text reporter required for AI agents to process coverage data. Additionally, the specific Skill for debugging flaky tests is open-source and available on GitHub for anyone to add to their project and use with compatible agents.
A flaky test is a test that returns both passes and failures despite no changes being made to the code or the test itself. These are notoriously difficult to debug because they often fail only once in several thousand runs. JetBrains uses code coverage analysis to make finding the root cause of these failures deterministic.
To use the Skill, you must include a SKILL.md file in your project that follows the Agent Skills specification. This file documents the debugging procedure step-by-step. When you ask a compatible AI agent to find the cause of flakiness, it discovers these instructions, runs the necessary coverage scripts, and analyzes the results to identify the bug.
