LLM-Based Synthetic Data Generation for NLP Metric Validation

Abstract

Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist for English datasets. We propose Meta-Judge, a scalable framework that uses LLMs to generate synthetic evaluation datasets via controlled semantic degradation of reference texts, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from human-annotated data. We experiment across Machine Translation, Question Answering, and Summarization in eight languages using 4 open-source LLMs. Large models achieve meta-correlation above 0.9 on question-answering datasets. To reduce inference cost, we finetune a 1B-parameter model using GRPO with an unsupervised ensemble of metrics, recovering most of the performance of large models.