MediaMonks Lab team trials use of AI in video conferencing
05-06-2020 13:20:00 | Editor: Bob Koigi | hits: 3170 | Tags:

MediaMonks Lab team has experimented how machine learning can augment video conferencing in a bid to break from the monotony of repetitive video calls that have become the new normal during the Covid-19 times.

The team has come up with a game that challenges users to quickly find a random object and show it to their webcam. Using an object-detecting machine learning model to identify whether the chosen object is shown, the app awards a point to whoever finds it first. The frantic game not only offers an opportunity to get together and have fun at home, but also gets people moving.

Using a lightweight machine learning model that enables object detection locally within a web browser, the game is compatible with any video conferencing platform that allows screen sharing. Connecting is merely a matter of loading up the web page and sharing your screen—and then the fun begins.

The game’s development began as a way to rethink the potential of video conferencing, breaking away from the buttoned-up approach and offering a space for play and fostering relationships.

Joe Mango, Creative Technologist on the MediaMonks Labs team: “The basis of the idea came out of the COVID situation that led people to work and socialize remotely. We wondered how can we use these tools that we have to build more fruitful and exciting interactions into mundane video calls?”

In challenging the notion of what a video call should look like, the team took inspiration from platforms that blur the boundaries between a standard call and a virtual space, like Online Town and Passtime—platforms that are almost more like games than just video calling.

Geert Eichhorn, Innovation Director, MediaMonks: “Mainstream videoconferencing platforms are enterprise tools, but now we’re discovering people want more out of meeting each other. We’re missing the spatial element, so you’re seeing gaming platforms competing with video calling platforms.”

The gamification element is an important piece of the web app’s development. It can be difficult to explain the potential of an emerging technology, as everyone approaches it with a different technical background and understanding. Upon coming up with the idea of using computer vision to augment video calls, the team found that gamifying the technology could help users easily wrap their heads around the feature.

A humorous video that captures the fast-paced energy of the game aims to not only demonstrate the app’s capabilities, but also kickstart new ideas in viewers’ minds.

Joe Mango: “Finding that human element that makes it fun is what made this project a success. We realized we could make ourselves move around and laugh. The hardest thing with these projects is portraying how it relates to us as functional human beings in time and space, and we need to be pioneers on how it evolves.”

Connecting digitally through collaboration across borders and video calls isn’t new to the MediaMonks team, though working from home has prompted people across the company to rethink the role that video calls play throughout the day, both inside and outside of working hours. There might be a happy hour call to keep teammates connected, but Monks have also been known to 24/7 virtual hangouts that serve no specific purpose—where people might not even talk to one another for the most part—simply to replicate the feeling of sharing a communal space with other people.

Beginning out of a necessity to make business calls more engaging, Eichhorn envisions other uses of the computer vision and machine learning as well.

Geert Eichhorn: “I think this is going to be another frontier. We’ve just used off-the-shelf models for this project, but in other iterations we can train the model for different purposes. What if you wanted to train an AI to rate your work from home outfit before jumping in a call?”

 He compares the technology to the popularity of Snapchat filters that have made their way out of the mobile space and into online business meetings.

On that note, the technology, and the scope of Labs’ thinking, isn’t limited to just the context of video conferencing. It could also be used to power live experiential activations at home by allowing users to engage with digital objects overlaying their webcam’s image, a bit like interacting with AR. But by starting with the humble video call—that technology that we all know far too well by now—the gamified web app offers a fun and accessible way to imagine how we can work to improve the tools that we use each day to enable closer connection.