MAESTRO: Multi-turn Agent Environment Synthesis for Training and Optimization

MAESTRO: Multi-turn Agent Environment Synthesis for Training and Optimization

Ionov T.R. (MWS AI ITMO, St. Petersburg, Russia)
Malykh V.A. (MWS AI ITMO, St. Petersburg, Russia)

Abstract

Manual construction of evaluation benchmarks for tool-calling conversational agents does not scale: each domain requires days of expert effort to design data models, implement tools, write policies, and create evaluation tasks. We present MAESTRO, a framework in which code-generating LLM agents produce complete benchmark domains – including executable tools, databases, policies, and evaluation tasks – through a structured pipeline with simulation-based quality filtering. To our knowledge, no prior system produces complete, human-validated evaluation benchmarks for multi-turn policy-following tool-calling agents. The framework evaluates agents using a multiplicative triple metric combining database state verification, tool call matching, and LLM-judged natural language assertions (95.6% judge-expert agreement with a group of 11 annotators). On a held-out test set (3 domains, 90 tasks), group relative policy optimization (GRPO) significantly outperforms supervised fine-tuning and direct preference optimization, and the resulting 9B model surpasses the 27B baseline, demonstrating that targeted RL on domain-specific tasks is more effective than parameter scaling. We release MAESTRO as open-source at the repository.

Keywords

large language models; benchmark generation; tool calling; conversational agents; reinforcement learning; code agents.

Edition

Proceedings of the Institute for System Programming, vol. 38, issue 6, part 2, 2026, pp. 275-284

ISSN 2220-6426 (Online), ISSN 2079-8156 (Print).

DOI: 10.15514/ISPRAS-2026-38(6)-37

For citation

Ionov T.R., Malykh V.A. MAESTRO: Multi-turn Agent Environment Synthesis for Training and Optimization. Proceedings of the Institute for System Programming, vol. 38, issue 6, part 2, 2026, pp. 275-284 DOI: 10.15514/ISPRAS-2026-38(6)-37.

Full text of the paper in pdf Back to the contents of the volume