Intro
I led a team that developed an AI-driven holodeck designed to introduce frog San Francisco visitors, clients and industry leaders to the current capabilities of multimodal Gen AI in an engaging way. The system leverages the most recent foundational LLMs, natural multi-lingual speech processing and multi-modal APIs to demonstrate some of the most recent innovations in AI capabilities. The goal is to immerse visitors across industries in the capabilities and potential applications of Gen AI while inspiring new ways of thinking about this technology.
Deep dive article on frog Designmind
As part of the build-out of the new frog studio, Capgemini, frog’s parent comapny, commissioned the installation of an Igloo Vision immersive space, an enclosed environment designed to deliver 360-degree visual experiences to facilitate collective immersion. In addition to enabling the development of the holodeck, the Igloo Vision space can be adapted for practical uses where virtual models of physical spaces or systems can be explored and manipulated alongside colleagues and is used for building immersive prototypes of new convergent experiences on behalf of clients.
Star Trek‘s Holodeck was an inspiration for the project. In the show this was presented as a virtual environment capable of creating highly realistic simulations. Characters could interact with these simulations as if they were real, immersing themselves in new worlds, historical moments or other spatial environments. The concept of a space that can respond dynamically to a user’s imagination or natural language input has inspired innovations across industries and was a reference for our AI-driven experience.
Process
As the concepting phase progressed, the visual design went through multiple iterations. Explorations included defining design and animations for how the AI entity would visually represent conversation states for speaking and listening, how the entity would handle latency while generative assets were being created and how the system would transition from the default state environment to the environments created by the user and back again.
The team needed to account for limitations of call and response, natural language processing (NLP) at the time of initial production and the need to detect presence of the user in an intuitive way. A physical platform and button system was designed to help identify and isolate the user. The platform invites the selected user to drive the experience and an embedded proximity sensor triggers the system to welcome the user. The AI begins to speak and prompts the user to imagine a place by holding down the glowing button as they speak. When the button is released, the experience begins to generate a new environment based on what the user said.
When a user speaks to the system, their voice is captured and transcribed using OpenAI’s Whisper model. This transcribed text then flows through our GPT-4 powered content generation system, which breaks down the input into three key components: scene descriptions for visual generation, narratives for voice-over and musical direction for ambient soundtracks.
The communication layer connects our physical and digital components through WebSocket and serial interfaces. It manages data flow between the platform sensors, control buttons and our React-based front-end application. This layer ensures that every user interaction— from stepping onto the platform to pressing the control button – receives an immediate response, maintaining fluid interactivity throughout the experience.
The AI Holodeck is an example of the type of forward-looking experiences a talented team can build on a short timeline when empowered by AI. It has delighted visitors to events and studio workshops. Participants have included executive and design team members from a huge range of Fortune 500 companies, an after party for Figma’s San Francisco-based Config conference, the leading Llama Lounge AI startup community, San Francisco Design Week participants and San Francisco Creative Mornings design community guests.
It is not just a glimpse into what technology can do today—it’s a vision of possible spatial interactions in the future.
Credits:
- Project Leads – Josh Baillon, Charles Yust, Ryan Starling, Antonello Crimi
- Digital Design – Will Schroeder, Antonello Crimi
- Physical Design/Build and Firmware – Adam Kingman, Julian Mickelson, Ryan Starling
- Graphics Software and AI Development – Kaz Saegusa, Anastasia Zimina, Charles Yust
More from this project
