LangChain's LangSmith Engine now automatically proposes three resolution actions for every agent issue it identifies: opening a Pull Request (PR), creating a custom online evaluator, and adding failing traces to an offline evaluation suite. This aims to accelerate the agent development lifecycle by automating issue diagnosis and fix validation.
LangSmith Engine now proposes three specific resolution actions for every issue it surfaces in AI agents. This new capability automates the process of identifying, diagnosing, and fixing agent failures by drafting targeted code or prompt changes as a Pull Request, creating a custom online evaluator for the exact problem, and adding failing production traces to an offline evaluation dataset. The feature is available in public beta.
- Resolution Actions
- Opens a PR, Creates custom online evaluator, Adds to offline eval suite
- Availability
- Public beta
- Integration
- Existing LangSmith tracing projects, optional repository connection
- Issue Detection Signals
- Explicit errors, online evaluator failures, trace anomalies, negative user feedback, unusual behaviors
This update addresses the manual and time-consuming cycle of reviewing agent traces, identifying failure patterns, and creating fixes. By continuously monitoring production traces and clustering failures into named issues, LangSmith Engine diagnoses root causes against connected codebases and proposes solutions, aiming to prevent regressions and strengthen evaluation coverage over time.
LangSmith Engine is built on existing LangSmith tracing and evaluation infrastructure. Connect a tracing project and optionally a code repository, and the Engine will automatically begin surfacing issues from production traces. Every resolved issue also generates an evaluator to monitor performance, making future improvements more robust.
