BrainStory

Role

Designed and programmed a multi-modal application that changes the way people narrate memorable stories in a team of three. My role involved prototyping, ideation and programming.

Skills

Prototyping Multi-Modal Design Concept Design

Interactions Programming Video Making

What makes us human?

Imagine a statue waking up or even a scene from the Disney Pixar's movie, Toy Story. If you were given the power to instill senses in an inanimate object, how would do go about doing that?

This project was done at the University of Michigan as a part of graduate coursework in Dialogue of the Senses, under Prof. Sile o'Modhrain. The prompt given to us revolved around Condillac’s statue whose senses would be awakened one after the other, creating awareness of other objects in space. Condillac maintained that it is only through touch that we gain an understanding of space as an external continuum extending outwards beyond the bounds of our bodies, and gather an awareness of other objects in space.

We wanted to build beyond this and question what really makes us human. It is emotion and our ability to describe it in words that differentiates us from other living beings. Emotion, combined with all other senses is what makes us truly human. We decided to build off of this idea.

My interpretation

The numerous senses that we possess is what makes us human. In her book, A Natural History of Senses, Ackerman describes each of the human senses so beautifully with just words and weaves a magical story. It is a story that keeps humans going.

My interpretation of the design prompt falls in the space of enriching the storytelling experience for a person. Personally, I love stories.I believe that there is some rich context behind every story and it is the story that makes us so unique. Every story has a personal element attached to it. When asked to create an artifact, I wanted to see how multiple-modalities can be invoked to create a brilliant experience for the user.

Problem Scoping

Based on the initial prompt, each of us worked on it individually and then gathered for a design critique session. Since, we had different backgrounds, our insights were quite unique.

Our ideas ranged from creating an immersive experience that revolved around music to that of lights, colors. All these ideas had a common theme of capturing emotions in unique manner. When we looked at constraints such as time and resources, we narrowed down our scope.

Solution Exploration

After eliminating the idea of capturing heart rate using fitness trackers, we narrowed down brainwave signals. Capturing brainwaves or EEG (electroencephalogram) waves meant finding sensors for the same. Most of the sensor technology we found were either too bulky or inaccessible. We also noticed people did not like the idea of using bulky headsets.

Now the question was controlling EEG signals that are generated. During our brainstorming sessions, we decided to narrow it down to personal user stories instead of music or sound to control signals that were being generated. We got super excited as users would be able to preserve their stories as art pieces for life.

BrainStory in action

Simply put, whenever you concentrate, you generate brainwave activity. Based on your emotional state, the kind of brainwaves varies. Mindflex Duel uses a variation of EEG technology to "read" the intensity of these brainwaves via sensors positioned on your forehead and ear. These sensors don't generate or interfere with brain waves in any way. They only read what is already there. We decided to read these signals and feed it into another system using an Arduino to create beautiful visual patterns.

Technically speaking, we asked users to talk about something they cherished. We also asked them to talk about something that was painful. EEG signals with Attentive and Meditative values differ based on emotion. We captured these values to control the art piece that would be created by tracking a user’s EEG (electroencephalogram) signals.

Visual Art created by our application BrainStory

User learning about his own story recreated as an art piece

We used a particle generator to create art and used the signals from the EEG headset to control the hues from the color wheel and the rate at which the particles fall. By using multiple particle generators we were able to create abstract art. We use a person’s story to alter his/her brainwave signals to create random patterns that can be captured. The tool that we created captures frames in the art sequence that signify special moments for that person. These can be printed out on high quality photo paper, framed and preserved as a beautiful personal momento.

Future Scope

The future vision for this project is a wearable that captures a person’s brain signals and transmits it to our application tool that would create a personal artifact that can be 3D printed. In addition to this, we would ask both the narrator and the listener to wear the headset to create two art pieces for the same story. We would also aim to utilise all the brainwave signals to create different patterns and alter hues and their intensities.

This project is a proof-of-concept prototype to demonstrate how we can control applications using our thoughts and emotions. This is an exciting field and there is a lot to look forward to!