As AI becomes more deeply embedded in enterprise translation workflows, organizations are continually looking for ways to improve efficiency without sacrificing quality. One area receiving increased attention is translation quality estimation (QE), the ability to predict whether a machine-translated segment is good enough to publish without requiring human review. 

With today’s powerful embedding models capable of measuring semantic similarity across languages, it may seem reasonable to assume they could also determine translation quality. If a translation preserves the meaning of the source text, shouldn’t a high similarity score indicate a high-quality translation? 

That question is explored in “Embedding Similarity Is Not Quality Estimation: Lessons from Replacing a Dedicated QE Model,” presented at the 2026 European Association for Machine Translation (EAMT) Conference by Dimitrios Zaikis and Andrea Biondo, with contributions to the research by Matthew Dixon, Konstantinos Karageorgos, and Aaron Schliem of Welocalize. The research, which was published in Volume 2 of the proceedings, evaluates whether general-purpose embeddings can replace specialized quality estimation models in production localization workflows. The study draws on 10,000 translation segments from six enterprise accounts, covering 38 target languages, each carrying an independent human judgment of whether the translation was fit to publish. The findings reveal an important distinction between measuring semantic similarity and evaluating translation quality. 

Rather than confirming that one metric can replace another, the research demonstrates why translation quality is considerably more complex than semantic similarity alone. 

Why Quality Estimation Matters

Quality estimation plays an increasingly important role in modern localization pipelines. Instead of reviewing every machine-translated segment, organizations use QE models to predict which translations meet quality standards and which require human review. 

Accurate quality estimation directly affects both cost and turnaround time. Segments predicted to meet quality thresholds can move through production automatically, while only lower-confidence content is routed to professional linguists. The better the QE model performs, the more efficiently organizations can balance automation with human expertise. 

Traditionally, this task has relied on dedicated neural quality estimation models trained using human quality judgments. But as general-purpose embedding models have become widely available, researchers wanted to determine whether a simpler, lighter-weight approach could achieve comparable results. 

A Logical Question with an Unexpected Answer 

Embedding models excel at measuring semantic similarity. They can determine whether two pieces of text express roughly the same meaning, even when written differently or translated into another language. The research explored whether that capability could eliminate the need for dedicated QE models. 

The team evaluated a production system that replaced a specialized COMET-based quality estimation model with a scorer built on Gemini embeddings using cosine similarity. The motivation was practical. General-purpose embeddings reduce infrastructure complexity, support many languages, and may already be deployed elsewhere within enterprise AI systems. If they could accurately estimate translation quality, organizations could simplify their technology stack without compromising performance. Instead, the study uncovered a fundamental limitation. 

Meaning and Quality Are Not the Same Thing 

One of the central findings is that semantic similarity measures something different from translation quality. Even poor translations usually preserve the general subject matter of the original text. They reference the same products, people, concepts, or events. As a result, embedding similarity scores remain extremely high regardless of whether the translation is actually acceptable. 

Across the production dataset, cosine similarity scores clustered between 0.94 and 0.99 for both acceptable and unacceptable translations. Approved segments averaged 0.951 and those sent back for review averaged 0.947. A gap of 0.004 leaves almost no room to distinguish good translations from bad ones using similarity scores alone. 

This saturation effect explains why semantic similarity reaches a practical ceiling when applied to translation quality estimation. Measured by AUC, the standard way of scoring how well a method separates two classes, cosine similarity tops out at roughly 0.63 on this data. The dedicated COMET model reached only 0.631 on the same segments, so the ceiling was not an artifact of the embedding scorer alone. 

A translation may contain incorrect terminology, omit a negation, mistranslate a number, or introduce subtle grammatical errors while still remaining semantically close to the source text. Those issues matter enormously to professional translators but barely affect the embedding space. 

Testing Three Different Approaches 

Rather than stopping with that observation, the researchers explored whether the limitations could be overcome. The study evaluated three approaches designed to improve embedding-based quality estimation. 

The first investigated whether so-called “rogue dimensions” within embedding vectors were distorting similarity calculations, a phenomenon documented in earlier work on transformer representations. The dimensions were real and large: five of them accounted for nearly a quarter of the total similarity score. Removing them changed almost nothing. Segment rankings were preserved at an r-squared of 0.9991, because those dimensions act as a near-constant offset and accept-or-reject decisions depend on relative ranking rather than absolute score. 

Next, the researchers examined whether recalibrating similarity scores through normalization or isotonic regression could improve performance. Both methods left ranking performance essentially unchanged. Because these approaches preserve the ordering of similarity scores, they cannot solve the underlying discrimination problem. 

Finally, the team tested a learned calibration model using LightGBM that combined normalized cosine similarity with surface-level text features. On a held-out test set of 3,000 segments, the learned model reached an AUC of 0.751, well above both the cosine baseline of 0.627 and the dedicated model’s 0.631. In operational terms, the best per-account word acceptance rate rose from 14.4% to 34.8%, more than doubling the share of words that could bypass human review in the strongest account, and the number of accounts meeting the internal bar for automated deployment went from one of six to two of six. 

The Biggest Improvement Came from Outside the Embeddings

Perhaps the most interesting discovery was not simply that the LightGBM model performed better, but why it performed better. Feature importance analysis showed that the greatest improvements came from surface-level text features rather than embedding similarity itself. In other words, the additional performance resulted from signals that cosine similarity could not capture. 

This reinforces one of the paper’s primary conclusions. Embedding models are exceptionally good at measuring semantic relationships, but translation quality depends on many additional linguistic characteristics that semantic similarity alone cannot detect. 

For localization professionals, this distinction is significant. Two translations may communicate roughly the same meaning while differing substantially in terminology, grammar, style, or compliance with brand guidelines. Those differences often determine whether a translation is production-ready. 

What the Study Does Not Claim 

The research covers one embedding model, one production pipeline, and one content domain, and it uses binary accept-or-review labels rather than continuous quality scores. The authors expect the saturation effect to generalize, since it follows from what cosine similarity measures rather than from any particular model, but that remains to be tested elsewhere. 

A gain in AUC also does not convert proportionally into throughput. Whether a better model reduces review load depends on where each account sets its threshold and how its content is distributed. 

Practical Lessons for Enterprise AI 

The research carries broader implications for organizations evaluating AI architectures. As enterprises increasingly consolidate AI capabilities onto foundation models, there is understandable interest in using a single embedding model for multiple downstream tasks. In many cases, that strategy makes sense. However, this study illustrates that simplifying infrastructure should not come at the expense of selecting the right tool for the problem being solved. 

Semantic similarity works well for applications such as retrieval, search, clustering, or document comparison because those tasks focus primarily on meaning. Translation quality estimation is fundamentally different. It requires identifying subtle linguistic differences that may have little impact on overall semantic similarity but significant consequences for translation accuracy. 

The findings also reinforce the value of combining multiple AI techniques rather than expecting one model to solve every challenge. The recommendation is not to abandon embeddings but to stop asking a single similarity score to carry the whole decision. A lightweight learned model sitting on top of embedding similarity recovers much of what the scalar throws away, without the large-scale human annotation that training a dedicated QE model from scratch demands. 

Building Better Translation Quality Workflows 

As organizations continue to automate multilingual content production, quality estimation will remain a foundational capability for balancing efficiency with human oversight. The research demonstrates that semantic similarity is an important input, but not a sufficient measure of translation quality on its own. The most effective quality estimation systems combine semantic understanding with additional linguistic signals that reflect how professional translators evaluate real-world content. 

For enterprises investing in AI-powered localization, that distinction matters. Measuring whether two texts express similar ideas is valuable. Determining whether a translation is truly ready for publication requires a deeper understanding of language itself. 

Research Sources

Embedding Similarity Is Not Quality Estimation: Lessons from Replacing a Dedicated QE Model, by Dimitrios Zaikis, Andrea Biondo, Matthew Dixon, Konstantinos Karageorgos, and Aaron Schliem, presented at the 2026 European Association for Machine Translation (EAMT) Conference, June 2026. 

All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality, by William Timkey and Marten van Schijndel, which documents how a small number of outlier dimensions can dominate cosine similarity in transformer representations. 

COMET: A Neural Framework for MT Evaluation, by Ricardo Rei and colleagues, which introduces the dedicated neural quality estimation approach used as the comparison model in this study. 

LightGBM: A Highly Efficient Gradient Boosting Decision Tree, by Guolin Ke and colleagues, which describes the gradient boosting method used for the learned calibration head.