Researchers presented the Vibe Patenting benchmark for evaluating AI agents

A group of researchers presented Vibe Patenting — a specialized testbed for evaluating the quality of work of AI agents in compiling patent documentation. The system uses the "judge-LLM" method, which analyzes patent drafts and provides structured feedback for iterative text improvement.

During testing, it was found that the use of iterative feedback allows less powerful models to achieve performance comparable to expensive high-performance systems. The researchers noted that without guided edits, the quality of generation quickly reaches a "plateau", while agent workflows involving an LLM judge provide a steady increase in metrics.

To verify the validity of the results, the authors compared AI evaluations with an independent analysis by a professional patent attorney. It was found that automated metrics demonstrate significant, but strongly criterion-dependent correlation with the expert's opinion, indicating the need to calibrate models to specific legal standards.

The work highlights both the potential and limitations of using LLMs as arbitrators for complex professional tasks. The study is available under the arXiv license, providing developers with a methodology for testing agents in conditions of strict requirements for accuracy and documentation structure.