As large language models (LLMs) have become more capable, reasoning capability has frequently been treated as a measure of progress, with the expectation that more computational effort, more analysis and more complex models will produce better results. This blogpost examines whether that expectation holds for translation quality estimation (QE) performed at scale. 

LLM-based metrics such as GEMBA now outperform reference-based metrics such as BLEU and edit distance. System 1 decision models, such as TypeSafeAI’s Jev, are a further development intended as a fast, scalable approach. They are termed System 1 models because they have no reasoning option: the score is read directly from the model’s internal numerical state, which is the source of their speed. 

In this post we report results for Opal’s next-gen AIQE model on the WMT 2025 quality estimation task. Compared against the official submissions, the model obtained the highest correlation with human judgments. We also review how QE methods have developed and consider what this means for how QE systems are compared. One possibility is that future QE systems will be System 1 models that use no reasoning at all. 

What Is Translation Quality Estimation?

QE assigns quality scores to translations so that not every translation has to undergo the same level of review. The score determines the next step in the workflow. High-scoring translations may be delivered without further intervention; others may be passed through another round of automatic post-editing; and translations with lower scores can be routed to human reviewers. The approach is intended to maintain quality and workflow throughput while directing human review toward content that requires additional attention. 

QE has been a subject of academic and industry research for many years; the first WMT shared task dedicated to QE took place in 2012. Approaches have changed considerably since then, in particular following the introduction of LLMs. LLM-based metrics such as GEMBA now outperform traditional reference-based metrics, including BLEU and edit distance. These methods introduce their own difficulties with respect to processing time, computational complexity, and the reliability of numerical scores. 

Three Ways to Use an LLM for Quality Estimation 

There are at least three ways to use an LLM for QE: 

1. Error-span annotation: the model annotates error spans, which are then converted into a score. 

2. Direct score output: the model outputs a score directly, for example from 1 to 100, as a literal token. 

3. Internal-state readout: the score is read from the model’s internal numerical state. 

The third method has been difficult to apply until now, because the best-performing models have either had closed weights (from providers such as OpenAI or Anthropic) or been too large to self-host. Without this method, obtaining a fine-grained, well-calibrated, deterministic score from an LLM reliably and at scale has remained difficult. System 1 models hold the promise to address this gap. 

Results on the WMT25 Quality Estimation Task 

On the WMT25 Automated Metrics Task 1, a segment-level quality estimation task, the Opal next-gen AIQE System 1 model achieved the highest correlation with human judgments when compared against all official submissions, with a score of 60.4%. The next-best system, an LLM-as-a-judge system built on Gemini 2.5 Pro, scored 59.3%, a difference of 1.1 percentage points.

The improvement is marginal. The model does not use reasoning, so it is expected to be less costly and faster than the LLM-as-a-judge system; cost and runtime were not measured as part of this comparison. 

The result is also not uniform across languages. The model is the best system in 8 of 14 languages and is within the top three in 11 of 14. For the remaining languages, other systems score higher. Performance is lowest on English→Arabic (Egypt), where the model scores 40.3% compared with 53.7% for the LLM-as-a-judge system. The results nonetheless show that a System 1 model, without references, can achieve the highest overall correlation on this task. 

A note on the metric: scores are Kendall’s tau, computed with the variant implemented in Google’s mt-metrics-eval GitHub repository, the same variant used in the competition. The results are therefore directly comparable with Table 4 of the findings of the WMT25 shared task on automated translation evaluation systems. 

How Translation QE Has Developed: From COMET to System 1 

COMET and MetricX: Fast and Precise, but with Fixed Inputs 

The first version of COMET was built in 2020. It is a compact multilingual encoder, fine-tuned on human quality judgments, that takes the source and the translation as input and outputs a fine-grained decimal score directly. It is small, runs locally, and is trained specifically for localization. It is fast and deterministic and produces fine-grained scores. Its limitation is a fixed input format: it accepts the source, the translation, and optionally, a reference, and nothing else. Information such as the domain, a glossary, or a client’s preferences cannot be supplied. The model scores what it was built to score, in the format it was built to receive. 

MetricX followed the same approach with a different underlying language model and different training data and was then scaled to very large models. The increase in size did not produce proportionate gains. This suggests that a fine-tuned regressor with fixed inputs has a performance ceiling that scale alone does not raise. 

LLM-as-a-Judge: Flexible Inputs, but Uncalibrated Scores 

The MetricX result pointed toward a different approach. It did not indicate that large models were unsuitable, but rather that generative models were more effective than the encoder models used previously. In the LLM-as-a-judge approach, a large, closed, general-purpose model such as GPT-4 is prompted to assess a translation. No training is required, and the input is open-ended, so the same approach can be applied to any language pair, domain, or text type. The score is produced as a token: the model writes a number in the same way that it writes any other word. 

Early LLMs such as GPT-4 were less effective in this role, but with a sufficiently strong model the approach works. Reference-based metrics such as BLEU and edit distance compare a translation with a human reference translation. Historically, these outperformed QE metrics, which have access only to the source and the translation, because the reference provides a strong signal. In WMT25, LLM judges that output scores directly began to outperform reference-based metrics without access to a reference. 

A limitation of LLM-as-a-judge is that the sampled number is not a fine-grained, deterministic or well-calibrated score. Most implementations output a number between 1 and 100, which allows at most 100 unique ranking positions. We frequently observe a bias toward “stereotypical” values such as 90 or 95, which suggests that the model is not producing statistically calibrated scores. It is therefore difficult to determine whether a score of 95 corresponds to a translation that is 95% good. 

GEMBA and MQM: Scoring Translations by Annotating Errors 

GEMBA introduces a linguistic framework around the model. Rather than requesting a score directly, it has the model annotate a translation’s errors using MQM (Multidimensional Quality Metrics), an industry-standard approach to describing and categorizing errors by type and severity. The error spans are then converted into a score using MQM error-severity weighting. 

This design can be viewed as a response to early LLMs, which were often unreliable when asked to output a score, particularly in 2023 and 2024. Error-span annotation makes use of their generative capabilities instead. A second proposed benefit of the taxonomy is that asking about accuracy, fluency and terminology errors activates linguistic pathways in the LLM, directing it toward the criteria that linguists consider. 

GEMBA has several limitations. The typical severity-weighting policy used to convert error spans to scores can be considered arbitrary, and it is hard-coded. GEMBA is arguably closer to Language Quality Assurance (LQA) automation than to workflow-decision automation and may therefore be less suited to the original purpose of QE. Its additional cost and runtime also raise the question of whether the quality gains are large enough to justify the extra cost. 

The Effect of Reasoning 

Reasoning increases the latency and cost of both GEMBA and LLM-as-a-judge approaches, because the model generates a chain of thought before answering. Reasoning is now the default: the standard reasoning effort for current LLMs is medium, not none. For a QE system that must make thousands of workflow decisions quickly and at low cost, generating a chain of thought for every segment is poorly suited to the task. 

System 1 Models: Consistent Scores from the Internal Numerical State 

The transition from COMET-style models to LLMs involved a trade-off. COMET and MetricX produce their scores natively. With LLMs, whether the output requested is a number or a set of error spans, the quality signal must pass through generated text. The weights of frontier LLMs are highly capable, but they could not be accessed to extract quality scores. 

In recent years, the field has moved in three directions at once: from small local models to large closed models; from localization-specific fine-tuning to purely pretrained capability; and from fixed inputs to open-ended inputs. System 1 models do not follow this trend toward fewer constraints. They combine the flexible input of LLMs with the native numeric scoring of COMET. 

System 1 models are scoring models rather than error-span models. By reading scores directly from the model’s internal numerical state, they make use of frontier model weights while producing fine-grained, consistent output. The WMT25 result suggests that they are suited to use cases that require large numbers of decisions to be made quickly. LLM-as-a-judge remains competitive, but at a higher cost. 

Comparing QE Systems on Accuracy and Speed 

The number of available QE options is increasing, across both the LLMs and the harnesses built around them, ranging from a bare prompt to a full error-span pipeline with reasoning enabled. Under these conditions, accuracy alone is no longer a sufficient basis for comparison. We propose that systems be reported as points on an accuracy–speed plot rather than as a single rank, since the relevant question for a given workflow is which trade-off it requires. We expect System 1 models to occupy the fast, scalable end of that curve, and we expect their accuracy to increase as better LLMs are trained.