UEA research to revolutionise animated characters' speech

Published by  News Archive

On 7th Aug 2017

disney animated characters

New research from the University of East Anglia (UK) could revolutionise the way that animated characters deliver their lines.

Animating the speech of family-favourite characters has been both time-consuming and costly. But now computer programmers have identified a way of creating natural-looking animated speech that can be generated in real-time as voice actors deliver their lines.

The discovery was unveiled in Los Angeles at the world’s largest computer graphics conference - Siggraph 2017. This work is a collaboration which includes UEA, Disney Research, Caltech and Carnegie Mellon University.

Researchers show how a ‘deep learning’ approach – using artificial neural networks – can generate natural-looking real-time animated speech.

As well as automatically generating lip sync for English speaking actors, the new software also animates singing and can be adapted for foreign languages. The online video games industry could also benefit from the research – with characters delivering their lines on-the-fly with much more realism than is currently possible – and it could also be it can be used to animate avatars in virtual reality.

A central focus for the work has been to develop software which can be seamlessly integrated into existing production pipelines, and which is easy to edit.

Lead researcher Dr Sarah Taylor, from UEA’s School of Computing Sciences, said: “Realistic speech animation is essential for effective character animation. Done badly, it can be distracting and lead to a box office flop.

“Doing it well however is both time consuming and costly as it has to be manually produced by a skilled animator. Our goal is to automatically generate production-quality animated speech for any style of character, given only audio speech as an input.”

The team’s approach involves ‘training’ a computer to take spoken words from a voice actor, predict the mouth shape needed, and animate a character to lip sync the speech.

This is done by first recording audio and video of a reference speaker reciting a collection of more than 2500 phonetically diverse sentences. Their face is tracked to create a ‘reference face’ animation model.

The audio is then transcribed into speech sounds using off-the-shelf speech recognition software.

This collected information can then be used to generate a model that is able to animate the reference face from a frame-by-frame sequence of phonemes. This animation can then be transferred to a CG character in real-time.

‘Training’ the model takes just a couple of hours. Dr Taylor said: “What we are doing is translating audio speech into a phonetic representation, and then into realistic animated speech.”

The method has so far been tested against sentences from a range of different speakers. The research team also undertook a subjective evaluation in which viewers rated how natural the animated speech looked.

Dr Taylor said: “Our approach only requires off-the-shelf speech recognition software, which automatically converts any spoken audio into the corresponding phonetic description. Our automatic speech animation therefore works for any input speaker, for any style of speech and can even work in other languages.

“Our results so far show that our approach achieves state-of-the-art performance in visual speech animation. The real beauty is that it is very straightforward to use, and easy to edit and stylise the animation using standard production editing software.”

Latest News

  News
Tall rainforest trees
18 Oct 2021

Dr Nem Vaughan awarded Philip Leverhulme prize

The prestigious award recognises outstanding researchers.

Read more >
  News
Ziggurats and trees on UEA campus
15 Oct 2021

UEA’s great outdoors celebrated for a fifth straight year with international Green Flag award

The last 18 months have placed more value on green spaces than ever, and students arriving at the University of East Anglia (UEA) for the new academic year can...

Read more >
  News
Power station chimney emitting smoke
12 Oct 2021

How recovery from COVID-19’s impact on energy demand could help meet climate targets

New research examines different scenarios of energy-related demand and how they could impact climate mitigation targets.

Read more >
  News
House in Brazilian Amazonia.
07 Oct 2021

Study reveals impact of wild meat consumption on greenhouse gas emissions

Consuming sustainably sourced wild meat instead of domesticated livestock reduces greenhouse gas emissions.

Read more >
Are you searching for something?
  News
House in Brazilian Amazonia.
07 Oct 2021

Study reveals impact of wild meat consumption on greenhouse gas emissions

Consuming sustainably sourced wild meat instead of domesticated livestock reduces greenhouse gas emissions.

Read more >
  News
29 Sep 2021

Could Vitamin-A bring back your sense of smell after Covid?

Researchers at the University of East Anglia and James Paget University Hospital are launching a new project to see whether Vitamin A could help people regain...

Read more >
  News
27 Sep 2021

Children who eat more fruit and veg have better mental health

Children who eat a better diet, packed with fruit and vegetables, have better mental wellbeing – according to new research from the University of East Anglia...

Read more >
  News
27 Sep 2021

Amazonian protected areas benefit both people and biodiversity

Highly positive social outcomes are linked to biodiversity efforts in Amazonian Sustainable-Use Protected Areas, according to new research from the University of...

Read more >
  News
25 Sep 2021

How rabbits help restore unique habitats for rare species

European wild rabbits are a ‘keystone species’ that hold together entire ecosystems – according to researchers at the University of East Anglia.

Read more >
  News
24 Sep 2021

Ageing the unageable: UEA researchers develop new way to age lobsters

Scientists at the University of East Anglia (UEA) have identified a way of determining the age of a lobster based on its DNA.

Read more >
  News
22 Sep 2021

New research reveals credit rating agencies responded too slowly to Covid-19

Sluggish response of credit rating agencies in assessing sovereign creditworthiness during the pandemic may have led to mispriced sovereign debt.

Read more >