LangSmith Adds Pytest and Vitest for LLM Evaluations

LangChain has released new Pytest and Vitest/Jest integrations for LangSmith Evaluations, bringing familiar software testing workflows to large language model development. Available in beta starting with version 0.3.0 of the LangSmith Python and TypeScript SDKs, these integrations allow developers to combine the runtime behavior of traditional testing frameworks with the observability and collaborative features of LangSmith. This makes it easier for teams to share experiment results and collaborate on prompt creation.

Unlike traditional evaluation libraries like OpenAI Evals, Hugging Face's Evaluate, or LangSmith's standard evaluate function—which require creating a dataset upfront—the new integrations let developers write specific evaluation logic for individual test cases. In Python, developers can use the @pytest.mark.langsmith decorator and built-in evaluation functions such as expect.edit_distance() to analyze outputs. In TypeScript, developers wrap their test cases in an ls.describe() block. The system logs inputs, outputs, and stack traces, and can even use models like gpt-4o-mini to grade semantic equivalence.

This approach addresses the non-deterministic nature of LLMs by logging nuanced metrics over time rather than relying solely on binary pass-or-fail results. By running evaluations directly inside Pytest or Vitest, practitioners receive real-time feedback during local iteration, making it easier to mock application components and debug issues. Additionally, the integration simplifies the process of running evaluations within continuous integration pipelines to prevent regressions before deployment, with LangChain planning to release a dedicated GitHub Action in the coming weeks.