Title Details
Role Lead Developer
Tools
Platform: Magic Leap 1 (spatial audio, digital objects)

Dynamic dialogue: Google Dialogflow v2

Pulse Sensor: Heart rate monitor
Length 25 hours
Features AI companion, Text-to-Speech, Dialogflow, Sound Design, & 3D Assets
Project Sixth Sense is an AR app to simulate schizophrenia symptoms in an authentic and respectful way to foster understanding and empathy.
Collaborators
Destiny Guzmán, Julia Scott, Tingru Lian, Lucas Wozniak

Demo

Motivation

Building empathy and awareness around psychosis is crucial to effective allyship and safe interaction with affected individuals. This project aims to immerse users in the experiences of someone with schizophrenia, bringing greater understanding to a wider audience.

Exhibitions

  • Games for Change Festival 2022 
  • XR Brain Jam 2022 Live Demo 

The App

To experience the app, download The Lab and use Device Bridge to load the application on your Magic Leap One device.

User Journey

In the app, the user embarks on a journey in the shoes of someone living with schizophrenia, guided by a supportive friend. Routine encounters of daily life spark auditory hallucinations, and the player must rely on their friend’s guidance to navigate these unsettling experiences.

Key Features

  • Spatial Audio for Auditory Hallucinations: We used spatial audio to simulate auditory hallucinations, making everyday sounds like an alarm, book, or TV feel immersive and disturbing. This feature mirrors common symptoms of schizophrenia, where auditory hallucinations affect over 70% of patients. source
  • Augmented Reality Elements: Simple AR objects placed in the real world—such as an alarm, book, and TV—add a layer of realism, enhancing the immersive experience.

Development Process

I led the app’s development using Magic Leap and Unity, which included troubleshooting and optimizing our workflow for AR on Magic Leap. This involved:

  • Testing Templates & Toolkits: Initially, I experimented with different example templates. I discovered that the original repository lacked the Magic Leap Toolkit (MLTK), which I then sourced from GitHub. I reviewed MLTK example scenes to find the most suitable fit and settled on the Controller Input scene, which was ideal for our experience.
  • Spatial Mapping Exploration: I explored spatial mapping features and plan to further refine this functionality to enhance adaptability across various physical spaces in future iterations.
  • Spatial Audio Implementation: Testing spatial audio was integral to crafting realistic auditory hallucinations that could anchor users in the experience. Successful Build: After around 20 hours of troubleshooting, I was thrilled to finally build the application successfully!

Storyboard + Content Creation

Event 1: Alarm

  • "Look at what the hell you did! You ruined everything."
  • "Run! Run!"
  • "You're always messing up. You need to leave before security finds you."

Event 2: Book

  • "THROW THE BOOK. THROW IT NOW."
  • "There is no hope for someone as meaningless as you."
  • "Everyone is watching you. Put it down right now, or else."

Event 3: Video on TV

  • "You’re so weak; they’re talking about you on TV!"
  • "If you weren’t such a snowflake, maybe you’d be better off."
  • [Laughter] "Did that hurt your feelings?"

Challenges & Solutions

  1. Mabu and manifest.xml Issues: Working with Mabu and manifest.xml proved challenging, as the documentation on the Magic Leap site was sometimes confusing. I found more effective support and templates on GitHub to address these issues.
  2. Device Registration Constraints: Although we had three Magic Leap devices, registration limitations restricted us to one device during development, which required careful planning and collaboration.
  3. Time Constraints: We agreed on the final storyboard just 15 hours before the end of the jam, necessitating quick implementation and prioritization to meet our goals.
  4. First-Time Use of Magic Leap: I familiarized myself with the Magic Leap device within 24 hours, quickly adapting to ensure efficient development.

Read more

Presentation

Github Special Thanks to Destiny Guzmán Julia Scott Tingru Lian Lucas Wozniak