Machine translation (MT) has made major strides in recent years, especially with the rise of neural models and large language models (LLMs). Yet even the best MT systems still struggle with nuance—tone, context, and the subtle meaning that only a trained human linguist can consistently get right. Automatic post-editing (APE), the process of improving raw MT output, offers a promising solution to this gap.

But there’s a catch: evaluating APE systems—especially those using LLMs—requires realistic, multilingual datasets that reflect how professionals actually edit machine-translated content. Until now, most datasets have been limited in size, scope, or linguistic variety.

Enter LangMark, a new dataset introduced by researchers to address that exact gap.

What Is LangMark?

LangMark is a comprehensive, human-annotated benchmark built specifically to evaluate APE systems on neural MT outputs. The dataset contains over 206,000 “triplets”—each made up of an English source segment, a machine translation, and a human post-edit—spanning seven language pairs: Brazilian Portuguese, French, German, Italian, Japanese, Russian, and Spanish.

What sets LangMark apart is its focus on real-world, domain-specific content. The data comes from actual marketing materials, processed through a professional translation management system (TMS), and refined by experienced linguists. These aren’t artificial examples—they reflect the kind of post-edits linguists make every day to ensure clarity, cultural fit, and brand consistency.

Testing APE in the Age of LLMs

To evaluate how well today’s LLMs perform on realistic APE tasks, the researchers tested a range of models, including GPT-4o, Claude 3.5 (Sonnet and Haiku), Gemini 1.5, Qwen, and Llama 3. Each model was evaluated using a 20-shot prompting approach, where it was shown 20 examples of human-edited translations to guide its output.

The results? GPT-4o delivered the strongest performance across most language pairs, especially in complex languages like Japanese and Russian. Other models—such as Qwen and Claude—showed promise in certain areas but often failed to outperform the baseline neural MT engine used to generate the original translations.

One key takeaway: most LLMs made significantly fewer edits than human post-editors—and in many cases, this restraint led to better outcomes.

When Less Is More

A major insight from the LangMark study is that making fewer, more targeted edits often leads to higher-quality translations than attempting to fix everything. High-performing models like GPT-4o exhibited “conservative” behavior—editing only when necessary and generally aligning well with human judgments.

In contrast, more “aggressive” models, which flagged more segments as needing edits, frequently introduced unnecessary changes that hurt overall quality. This underscores a key challenge in APE: knowing when not to intervene.

As MT systems improve, the task of post-editing is evolving. It’s no longer just about catching obvious errors—it’s about fine-tuning already strong output while preserving accuracy, voice, and intent.

Rethinking Evaluation Metrics

LangMark also highlights the shortcomings of current evaluation methods. Standard metrics like CHRF and BLEU are helpful but limited—they measure similarity to a human reference but don’t assess whether a model correctly judged if an edit was needed in the first place.

To address this, the researchers used additional metrics for precision and recall in identifying whether a segment should be edited. They found that models with high precision (like GPT-4o) tended to perform better overall than those with high recall that made more frequent, and often unnecessary, changes.

This suggests the need for more nuanced evaluation frameworks—ones that reward models not just for accuracy, but for exercising good editorial judgment.

Built for the Real World

LangMark isn’t just a research dataset—it’s designed for practical impact. The post-edits in the dataset reflect typical changes made by professional linguists, such as adjustments to grammar, terminology, locale conventions, tone, and formatting.

Because the machine translations were generated by a proprietary, domain-trained MT engine, the benchmark sets a high bar. These aren’t generic translations; they’re already strong, which makes the task of post-editing—and evaluating models—more rigorous and realistic.

The study also reinforces the ongoing value of human expertise. Even as LLMs get better, expert post-editors remain essential for catching subtle mistakes, aligning with client expectations, and delivering content that resonates across markets.

Looking Ahead

LangMark offers a powerful new tool for anyone building multilingual AI systems, translation platforms, or content workflows. By focusing on in-domain, human-annotated data and real-world editing challenges, it pushes the field toward smarter, more human-aligned APE solutions.

The dataset is being released as a benchmark for further research and development in post-editing and multilingual NLP.

Read the full research paper here.

Ready to elevate your multilingual content with cutting-edge AI solutions? Contact Welocalize today to learn how we can help!