UEA research to revolutionise animated characters' speech

Published by  News Archive

On 7th Aug 2017

disney animated characters

New research from the University of East Anglia (UK) could revolutionise the way that animated characters deliver their lines.

Animating the speech of family-favourite characters has been both time-consuming and costly. But now computer programmers have identified a way of creating natural-looking animated speech that can be generated in real-time as voice actors deliver their lines.

The discovery was unveiled in Los Angeles at the world’s largest computer graphics conference - Siggraph 2017. This work is a collaboration which includes UEA, Disney Research, Caltech and Carnegie Mellon University.

Researchers show how a ‘deep learning’ approach – using artificial neural networks – can generate natural-looking real-time animated speech.

As well as automatically generating lip sync for English speaking actors, the new software also animates singing and can be adapted for foreign languages. The online video games industry could also benefit from the research – with characters delivering their lines on-the-fly with much more realism than is currently possible – and it could also be it can be used to animate avatars in virtual reality.

A central focus for the work has been to develop software which can be seamlessly integrated into existing production pipelines, and which is easy to edit.

Lead researcher Dr Sarah Taylor, from UEA’s School of Computing Sciences, said: “Realistic speech animation is essential for effective character animation. Done badly, it can be distracting and lead to a box office flop.

“Doing it well however is both time consuming and costly as it has to be manually produced by a skilled animator. Our goal is to automatically generate production-quality animated speech for any style of character, given only audio speech as an input.”

The team’s approach involves ‘training’ a computer to take spoken words from a voice actor, predict the mouth shape needed, and animate a character to lip sync the speech.

This is done by first recording audio and video of a reference speaker reciting a collection of more than 2500 phonetically diverse sentences. Their face is tracked to create a ‘reference face’ animation model.

The audio is then transcribed into speech sounds using off-the-shelf speech recognition software.

This collected information can then be used to generate a model that is able to animate the reference face from a frame-by-frame sequence of phonemes. This animation can then be transferred to a CG character in real-time.

‘Training’ the model takes just a couple of hours. Dr Taylor said: “What we are doing is translating audio speech into a phonetic representation, and then into realistic animated speech.”

The method has so far been tested against sentences from a range of different speakers. The research team also undertook a subjective evaluation in which viewers rated how natural the animated speech looked.

Dr Taylor said: “Our approach only requires off-the-shelf speech recognition software, which automatically converts any spoken audio into the corresponding phonetic description. Our automatic speech animation therefore works for any input speaker, for any style of speech and can even work in other languages.

“Our results so far show that our approach achieves state-of-the-art performance in visual speech animation. The real beauty is that it is very straightforward to use, and easy to edit and stylise the animation using standard production editing software.”

Latest News

  News
A man at work smiles looking out of a window.
26 May 2022

New toolkit launched to help businesses boost staff wellbeing

Employers can find out the potential financial benefits of increasing employee wellbeing with an innovative cost effectiveness calculator, launched today.

Read more >
  News
Engineers work on an offshore wind farm
25 May 2022

New project launched to boost number of women working in offshore wind

The Offshore Wind Industry Council (OWIC) and the University of East Anglia (UEA) have announced a new joint research project which aims to tackle the gender...

Read more >
  News
A female protestor displays the
19 May 2022

USA slumbers, Europe leads in electoral integrity

The world’s leading democracy is falling behind on electoral integrity, according to new findings from the University of East Anglia (UEA) and the Royal Military...

Read more >
  News
Cranberries held in two hands.
19 May 2022

How cranberries could improve memory and ward off dementia

Adding cranberries to your diet could help improve memory and brain function, and lower ‘bad’ cholesterol – according to new research from the University of East...

Read more >
Are you searching for something?
  News
Cranberries held in two hands.
19 May 2022

How cranberries could improve memory and ward off dementia

Adding cranberries to your diet could help improve memory and brain function, and lower ‘bad’ cholesterol – according to new research from the University of East...

Read more >
  News
Surgeons perform heart surgery in an operating theatre.
18 May 2022

Timing of heart surgery crucial, research shows

Valve replacement heart surgery should be performed earlier than conventionally thought for people with aortic stenosis – according to new research from the...

Read more >
  News
16 May 2022

From testing for plastics in teabags to a Q&A with Countrywise’s Liz Bonnin: UEA’s Green Film Festival is back

Following a two-year pandemic hiatus, the Green Film Festival at the University of East Anglia (UEA) is back from Thursday 19 May - Saturday 21 May, offering...

Read more >
  News
Will Harris
26 May 2022

Will Harris appointed as new Visiting Poetry Fellow at The British Archive for Contemporary Writing

The British Archive for Contemporary Writing, based in UEA Library, has announced the appointment of a new Visiting Poetry Fellow, Will Harris.

Read more >
  News
World of lights with a really bright light shining from Norwich
12 May 2022

UEA’s research confirmed as ‘world-leading’ by national assessment

The global significance and real-world impact of the University of East Anglia’s (UEA’s) research has been confirmed with the Research Excellence Framework 2021...

Read more >
  News
11 May 2022

Innovation & Impact Awards 2022 winners

From saving the world’s animals through socks, improving animal nutrition to sequencing COVID-19 genomes and developing a diagnostic device for dizziness, there...

Read more >
  News
27 May 2022

Your Chem Magazine May issue

The Spring edition of our UEA Chemistry Magazine: yoUr chEm mAg is now available.

Read more >