TAUS Dynamic Quality Framework: Tackling Quality Evaluation

Evaluating quality is a big challenge for the localization industry. As Welocalize CEO Smith Yewell wrote in his blog on Time, Cost and Quality, in the past QA programs have typically not discerned between content types and business requirements. It is no great secret that one translation quality does not fit all.

Localization industry body, TAUS, started working with its enterprise members, like Welocalize, in 2011 to develop a platform for sharing best practices in translation quality evaluation. This work has resulted in the Dynamic Quality Framework (DQF). DQF was released for TAUS members in 2013. TAUS recently announced that the DQF is now available publicly.

Attila Görög TAUSWelocalize guest blogger and TAUS DQF Product Manager, Attila Görög, tells us a bit more about DQF and the benefits it will bring to the localization industry. Born in Hungary in a small town with a long name, Attila joined TAUS at the beginning of 2014, having spent over 10 years working with computational linguistics and translation technology for academic and commercial organizations. Attila is based in Amsterdam, Netherlands and is responsible for the TAUS DQF translation quality evaluation platform including its support, development and promotion. Attila also coordinates the development of the TAUS Post-Editing course.

We all know quality evaluation (QE) and related pricing in the translation industry can be problematic and it is becoming more widely understood that we don’t always need top quality. We need to approach QE in a flexible way. We need to go down as well as up in quality. The evaluation models used in the translation industry must reflect this. We need industry-level bench-marking to be able to compare quality scores across the industry. The TAUS DQF has been developed by industry stakeholders to measure quality objectively and follow a standardized workflow. The DQF platform contains a manual content profiling tool to help select the best QE model for the translation. It houses a knowledge base where users study the different models, get background information on QE types, download templates and read reports and best practices in QE.

It’s an incredible storehouse of information and we are planning to transfer that information into a wiki to encourage users to add content. The DQF tools enable users to evaluate the translations using 6 approaches: Error-typology, fluency, adequacy, comparison, ranking and post-editing productivity.

DQF can be used to assess the quality of MT engine output, to pinpoint the weaknesses and to try to improve the system. The DQF tools can also be used to assess human translations and to compare MT engines or translators and LSPs, based on the quality of their translations.

There are now also two online courses making use of DQF. The Quality Management course, offered by the Localization Institute, and the TAUS online Post-Editing course which Welocalize has contributed to and which has a whole module on QE.

There are many benefits of DQF to LSPs, vendors and clients, mainly related to the localization industry having a standardized way of assessing and bench-marking quality. In the very near future, there will be enough data to provide bench-marking functionality. When going through the process of QE, users can compare scores to other users. Scores don’t say much unless you can compare them.

Clients can compare different LSPs using samples before outsourcing a large project. And LSPs can do the same thing with MT engines or freelance translators. The error-typology approach helps pinpoint problems.

By making TAUS DQF publicly available and providing educational programs, like webinars on translation quality and integrating DQF into different tools, courses and workflows, we are trying to address the ongoing challenge of quality and evangelize a dynamic approach to QE.

I hope that DQF will raise awareness and flexibility in the approach to translation quality. We’re not there yet and I still think some approaches to QA remain static and subjective. We need industry matrices, made available publicly and DQF is a huge step towards achieving that goal.

Attila Görög, TAUS DQF Product Manager