How to Move to Neural Machine Translation for Enterprise-Scale Programs

welocalize May 22, 2018

Challenges in relation to new translation technologies such as neural machine translation (NMT) are far more concrete than theoretical analyses or any potential “hype” in the localization and translation industry. Global businesses are looking for solutions and expert advice to put these technology solutions in place in a real commercial setting.

Olga Beregovaya is Welocalize VP of Language Services and Tanja Schmidt is Welocalize MT Program Manager. They share their insights and experience at migrating enterprise-scale programs to NMT for Dell Technologies and VMware.

Deploying new technology does involve theoretical analyses of related technology, systems, and providers. However, when it comes to commercial implementation, there are many other aspects to consider.

We need to think about the actual implementation, including technical prerequisites around existing API connectors, development time and cost for potential new API connectors, as well as deployment, adaptation and usage costs for the new systems—all of which must be in line with client requirements and budget.

Within three years from the first publications on neural networks for MT, NMT has become available for wider public consumption. Upcoming or already available customization options from large providers like Microsoft (currently in “preview” mode), Google, and Amazon, along with multiple open-source NMT projects and commercial offerings, add to the need for LSPs to work on concrete implementation plans, selecting the best option for their clients.

Moving an enterprise-scale program to NMT involves focus in key new areas including qualitative evaluation of NMT systems and comparing translation quality + productivity of NMT and incumbent SMT systems. Click To Tweet
NMT in Practice – Dell Technologies + VMware

Over the past months, Welocalize has worked on a major NMT migration and an evaluation project for two large enterprise clients—Dell Technologies and VMware.

Both projects involved multiple languages and various technologies.

For both projects, we initially started with an evaluation of several NMT engines and compared their performance to the existing customized statistical MT (SMT) engines. During this evaluation, we did autoscoring for common industry metrics like BLEU, GTM, Nist, Meteor, Precision, Recall, TER, and Edit Distance, using our proprietary scoring tool “WeScore” which combines all these evaluation methodologies and presents the results in an easy-to-read side-by-side dashboard. We proceeded with post-editing tests, as well as sophisticated human evaluations for both Adequacy, Fluency and Engine Ranking, as well as engine strengths and weaknesses within pre-defined categories like syntax, grammar, terminology, redundancy, localization, and more.

In the beginning, we analyzed generic, untrained NMT engines, readily available on the market. These untrained engines were already outperforming the existing customized SMT engines on various metrics, so we then moved to the creation and analysis of customized NMT engines.

Dell Technologies

For Dell, we selected a subset of four pilot languages (French, German, Russian, Japanese) out of the total 28—from different MT language complexity groups. This was to get representative results with a small subset of languages and at the same time, stay within a fixed budget.

Again, we compared the customized NMT to the existing SMT engines and set up a productivity test using the TAUS DQF Quality  Dashboard and the related SDL Trados Studio plug-in. We also scored the results with WeScore.

From all these tests, we derived a suitable language migration sequence, involving both generic and customized NMT engines, based on evaluation results.

VMware

As all industry publications and presentations, and also Welocalize’s own experimentation pointed at NMT engine performance being significantly superior to that of SMT, we wanted to be certain that we have the best set of engines implemented on the VMware MT program.

We conducted a bake-off between several MT engines for Japanese and Chinese, which have traditionally been more challenging languages for customized statistical MT deployments. The objective was both to compare the performance of SMT engines currently deployed in production against NMT technology, and also see how different commercially available customized NMT engines compare to each other for both languages. This bake-off test was followed by several other tests, all pointing at NMT technology as the best fit.

At Welocalize, when it comes to language technology such as translation management systems (TMS), machine translation (MT), natural language processing (NLP) or any other AI-driven technology, we are technology-agnostic. Instead of locking clients into one technology and/or selling this technology, we recommend the solution that is the best fit for the client and its global business objectives. To be able to do this, we maintain close relationships with various technology providers, open-source projects, and academic institutions.

 

Olga Beregovaya

Welocalize Vice President, Language Services, olga.beregovaya@welocalize.com

Tanja Schmidt

Welocalize MT Program Manager, tanja.schmidt@welocalize.com

 

Knowledge and findings from both these live projects will be presented during this year’s EAMT conference in Alicante, Spain, and LocWorld 37in Warsaw, Poland.

If you are planning to implement NMT or want to migrate from your existing solution, connect with us at marketing@welocalize.com.