assets/sixthsense/demo.png

Title Details
Role Lead Developer
Tools
Platform: Magic Leap 1 (spatial audio, digital objects)

Dynamic dialogue: Google Dialogflow v2

Pulse Sensor: Heart rate monitor

Length 25 hours
Features AI companion, Text-to-Speech, Dialogflow, Sound Design, & 3D Assets
Project Sixth Sense is an AR app to simulate schizophrenia symptoms in an authentic and respectful way to foster understanding and empathy.
Collaborators
Destiny Guzmán, Julia Scott, Tingru Lian, Lucas Wozniak

Teaser

Motivation

Understanding psychosis is critical to effective allyship and interfacing safely with affected persons.

Exhibitions

  • Games for Change Festival 2022 
  • XR Brain Jam 2022 Live Demo 

The App

To open the app, download The Lab and use Device Bridge to load the app on your Magic Leap One device.

User Journey

The player takes a walk in the shoes of someone with schizophrenia under the guidance of a benevolent friend. Typical encounters of daily life trigger auditory hallucinations. The player turns to their friend to help find their way through.

The Features

We delivered our first iteration of our product for the XR Brain Jam. We focused on spatial audio to create auditory hallucination with simple AR objects in the real world: alarm, book, TV.

“Auditory hallucinations are among the most common symptoms in schizophrenia, affecting more than 70% of the patients.” source

Development Process

Click to expand

I enjoyed developing for Magic Leap with Unity. Although, at first, I spent around 20 hours troubleshooting:

  • Trying out different example templates
  • The original repo didn't have MLTK tool embedded and I found the MLTK package on github
  • Explored the example scenes of the MLTK
  • Decided the Controller Input was the one that fits our experience the most
  • I also explored the spatial map and would love to explore more in the future



  • It is include in the Control Input scene and I would love to dive into it more to be more calibratable to different physical scenes 
  • I was so happy when I finally succeeded in building our application!



  • Testing out spatial audio


Storyboard + Content Creation

Click to expand

Storyboard

Script for Inner Voices

Event1: Alarm

  • Look at what the hell you did! You ruined everything.
  • Run! Run!
  • You're always messing up. You need to leave before security finds you.
  • Voice A: If you weren’t so stupid you wouldn’t get caught.
  • Voice B: Exactly! What a fucking idiot

Event2: Book

  • THROW THE BOOK. THROW IT NOW. (Repeat)
  • What’s the point of reading this
  • There is no hope for someone as meaningless as you.
  • Everyone is watching you, you look so pathetic!
  • Put it down right now, or else.

Event3: Video

The Video on TV
  • You’re so weak, you’re so sensitive - Look they’re talking about you on TV!
  • They’re watching you
  • If you weren’t such a snowflake maybe you’d be better off
  • _crazy laughter_
  • Did that hurt your feelings? I don’t give a shit


Challenges

Click to expand

Life without troubleshooting is dull. 

I have encountered the following challenges while working with Magic Leap:

  • Issue with Mabu, manifest.xml

  • I enjoyed the documentation on Magic Leap’s official website. Yet there is one website that has confusing most up to date templates. Luckily github comes to the rescue. Here is the image showing that zero iteration works with the Magic Leap Unity Exmamples repo.




  • Registering issues with Magic Leap. We have three Magic Leap devices to develop for yet due to the registering issue, we ended up working with one registered magic leap for our development.

  • First time using magic leap. Learned how to use it in less than 24 hours.

  • Limited developing time since we agreed with our storyboard 15 hours before the end of the Jam.


Read more

Presentation

Github

Special Thanks to Destiny Guzmán Julia Scott Tingru Lian Lucas Wozniak