LangChain Builds Better CSV Question-Answering Agent

LangChain has introduced a new benchmarking process and a custom agent architecture designed to optimize natural language question-answering over tabular CSV files. To address the historical difficulty of querying structured data containing fuzzy text, the company built a demo application using the classic Titanic passenger dataset. By gathering approximately 400 real-world user interactions and filtering those with feedback down to a high-quality evaluation dataset of 50 datapoints, the developers established a reliable testing ground for various LLM-based approaches.

Initial attempts to query the data relied on a parallel setup of a vector retriever and Kork, a library that restricts execution to a whitelist of predetermined functions. However, this initial solution suffered from formatting discrepancies between local environments and Streamlit cloud, which distorted how dataframes were presented to the language model. Developers used LangChain's LangSmith observability platform to diagnose these data engineering issues, ultimately resolving them with pandas display configurations and realizing that a rigid function whitelist could not handle the diverse questions users asked.

The final, optimized architecture utilizes an agent powered by OpenAI functions, specifically the OpenAIFunctionsAgent, equipped with two distinct tools: a Python REPL and a vector retriever. This hybrid approach allows the system to route analytical and mathematical queries to the Python environment while leveraging the retriever for unstructured text columns, such as passenger names. In testing against the 50-datapoint benchmark, this custom agent outperformed standard alternatives, failing on only four of the evaluated questions.

For AI practitioners, this development highlights the limitations of relying solely on generic code-writing agents like the standard Pandas Agent or PandasAI. While pure Python execution struggles with fuzzy textual matches, and pure retrieval fails at mathematical aggregations, combining both tools under a single agent with dataset-specific prompting yields superior accuracy. LangChain has open-sourced the entire pipeline, including the feedback-gathering application, the evaluation dataset, and the testing scripts, to help developers build more robust tabular QA systems.