New machine learning agent will address conflicts and facilitate interaction in extended reality environments such as the metaverse

The Eurecat technology centre is leading the European GuestXR project, which will develop a machine learning agent designed to facilitate the interactions between participants in virtual and augmented reality environments such as the metaverse, and help them achieve their objectives, while also making it possible to address conflict situations.

For this purpose, a machine learning system called the ‘Guest’ is to be designed as an agent that can examine participants’ individual and group behaviour by drawing on existing models from the neuroscience and social psychology standpoints. Moreover, Deep Reinforcement Learning will also give the ‘Guest’ the ability to learn over time and to be more effective in terms of social interaction, learning both from simulations as well as real meetings between people.

In this way, the agent ‘will be able to participate in the conversation in different ways, such as, for example, by providing multisensorial experiences through music or changes in the space, among others’, notes Adan Garriga, the director of Eurecat’s Audiovisual Technologies.

‘Every meeting between people in virtual and augmented reality has a purpose, whether explicit, such as reaching an agreement on a topic of discussion, or implicit, such as enjoyment. The fundamental idea of the ‘Guest’ is to help the group of people accomplish this purpose. This involves exciting research with a strong interdisciplinary component, opens new paths and hopefully contributes to the success of virtual and social augmented reality’, says Mel Slater, the scientific coordinator of the GuestXR project from the University of Barcelona and a member of the university’s Institute of Neurosciences (NeuroUB).

To modulate this social interaction, the ‘Guest’ will be able to carry out a series of multisensorial actions ‘using, for example, visual or auditory characteristics to create particular moods and stimulate a relaxed environment when it identifies a conflict’, explains Umut Sayin, an Audiovisual Technologies researcher at Eurecat.

This project ‘is designed to help offer solutions to existing challenges in online social interaction environments, such as cyberbullying, for which there is still no specific European legislation’, explains Aurora Sesé, Eurecat’s GuestXR project coordinator.

The machine learning system’s intervention will be tested initially in conflict resolution situations, interactions with participants with hearing difficulties and contexts that trigger polarised debates such as climate change.

Finally, there will be two open calls to include other innovative cases of use to confirm the system’s effectiveness, one of which will be aimed at society in general, including companies and associations, among others, while the other will be aimed specifically at people in the world of arts.

The GuestXR technology will be developed under the ‘Ethics by design’ approach. This means considering the potential ethical problems that might arise when artificial intelligence is used, such as the respect for human autonomy, damage prevention, fairness and explainability.

The GuestXR consortium is led by the Eurecat technology centre together with the University of Barcelona and comprises eight organisations from six countries with a multidisciplinary team with backgrounds in extended reality, machine learning, artificial Intelligence, social psychology, the neuroscience of emotions, multisensory integration, research ethics and technology transfer.

Alongside Eurecat, there are also four universities (the University of Maastricht, the University of Warsaw, the University of Barcelona and Reichman University), one research institute (Inria) and two businesses (Virtual Bodyworks and g.tec medical engineering GMBH) partnering the project.