Basque Government and TSA test automatic dubbing of audiovisual content into Basque

In cartoons, automatic dubbing has been able to reproduce with great fidelity the voice of the characters, performing original voice cloning, as well as the rhythm and expression of the dialogues. Being a cartoon, the synchronization of voices is more efficient than in real characters. In the case of this type of children’s content, automatic dubbing can be exploited quickly and fully automatically, provided that a final quality control is performed by a technician to correct possible errors. This new milestone opens a new dimension and monetization to all the children’s contents that exist in countless historical archives, both old and modern contents. Being able to provide them with the Basque language increases their value and brings contents closer to a young audience that otherwise would be lost, due to the high cost of a traditional manual dubbing.

A fundamental element of the project has been to be able to count on the voice synthesis service in Basque of the Itzuli project. The Basque Public Society has provided the project with access to three voices in Basque, two female and one male, with a very high quality, which has allowed us to use them as a basis for voice cloning. The high quality of the data with which these voices have been trained has allowed the result after cloning to be voices with a very natural prosody. Additionally, the translation service of the Itzuli project has been made available to the project, whose implementation based on neural networks has allowed a very good level of automatic translation.

In the case of the series and the documentary, the automatic dubbing has been less satisfactory. In the series, the automatic dubbing was able to translate, dub and faithfully reproduce the cloned voice of the main character, but it had difficulties in capturing the feeling, rhythm and expression of the dialogues, especially in short sentences. In the reality documentary, the automatic dubbing was less accurate, with errors in pronunciation and understanding of the context. This example used is even more complex than a fiction film or series. In a more classic documentary, with a narrator, the estimation of success is much higher than in a docu-reality.

In all cases, the results indicate that automatic dubbing technology is progressing. However, further development is needed to optimize its effectiveness for more complex audiovisual content, such as movies and series.

Factors influencing the results

The results of the proof-of-concept test have been influenced by a number of factors, including::

• Type of content: cartoons, which tend to have a more leisurely pace and simpler dialogue, are easier to dub automatically than series and movies, which tend to be more complex and require greater accuracy in pronunciation and understanding of context.
• Quality of the training data: The accuracy of automatic dubbing depends on the quality of the training data used to train the model. In this case, the training data used were of high quality, which contributed to the improved results.
• Technology applied: The specific artificial intelligence developments used for automatic dubbing also influence the results. In this case, state-of-the-art technology, based on transformers, has been used to obtain satisfactory results in the case of cartoons. These transformers include automatic transcription, translation, voice cloning and lip-sync.