For my Research-by-Making these past 2 weeks, I made 2 storytelling projects in 2 different mediums. First, I made a video collage by using Motion Capture data retargeted to 3D models in Blender as well as archival footage of my grandfather, father, and uncles playing football. This was a final for my class, Mocap for the Archive, and I explored executing a personal storytelling concept with MoCap. Second, I made an immersive experience using projection and video collages to tell my partner’s dream as a story. This was a final for another class, Outside The Box: Site-Specific and Immersive Experiences. Both experiences taught me a lot about each technology (Motion Capture & Projection) as mediums for personal storytelling.
MoCap For the Archive
Above is my MoCap final: a video collage with footage of my family, MoCap footage of actors and myself performing football moves, and a voiceover of myself reading a poem written about my family, football, and grief. There is more written about my concept and process on my Process blogs for the class, starting here: [link to first MoCap blog]
I took away several lessons from the MoCap process. First, while Motion Capture sessions serve as rich opportunities to play and perform, editing this data is a real chore. Even though I made a Python script to automate some aspects of re-targeting and found a retargeting plug-in in Blender, the process of putting my MoCap data on a 3D character is tedious and repetitive. I had over 80 takes to re-target and about only 30 of them were usable, due to football having movements that can be too quick or obscure markers from the Infrared cameras. Also, instead of proper rendering, I screen-recorded each cleaned up take. I tried tackling Unreal Engine but did not have enough time to learn and accomplish a rough draft of my project.
Another main lesson I learned from my MoCap project is that the movement of the body does capture stories in a way words can’t express. Though I did write verse for this project, the words would not have been as effective if the viewer did not see the hits and difficult movements I was referencing. If my thesis does remain performance-based, I will have to factor in movement as a major storytelling component and build my experience with this major component in mind. This will then affect the space I design as well as all other interaction points.
However, the biggest lesson I learned from my MoCap project is that one way to make an audience vulnerable and empathetic is to make myself vulnerable with the story I’m telling. Since I told a story of my father and how he felt about my grandfather, I shared a lot of my personal history with my audience. The audience in my class said they were touched, and I felt that they had been affected in some way by what I shared. Not by the graphics of my 3D models (because they weren’t good), not by the flashiness of my editing, but people were touched by my story. In any case, if I am making an immersive experience where I’m asking people to share vulnerability, I need to share some myself and open up if I expect others to open up. What’s key going forward is what specific story I’m trying to share with others.
Outside The Box: Site-Specific and Immersive Experiences
Above is my Immersive final: a projection/video installation in my apartment based on a dream of my classmate, Akshita. There is more written about my concept on these slides here: [link to mood board slides]
There is also a technical/execution breakdown on these slides here: [link to final slides]
The first lesson I learned from this design experience is that every detail in an immersive experience matters. While my installation was visually focused with projections, my professor noted that the shelves that happen to be in my living room could have artifacts inside them. I had not considered the shelves as part of the experience (they just happened to be in my living room) but if I make a space, everything in that space will be analyzed as part of the Interactive Experience. The other lesson is that I have to make it very obvious the points of interaction that I really want the audience to find. I had a place underneath the table that I wanted the viewer to crawl to and see a steady mandala. I intended for it to be a shelter place, but I did have to directly tell my classmate to look under there after a few minutes. I have to really user test any experience I do because no one is going to immediately think of crawling under a table. How do I get them to think about doing such things as part of my Immersive Set design? What indicators (visually or otherwise – maybe smell?) nudge someone to check out certain areas of an experience?
Next Steps
For my thesis, I have a lot of experimenting to do over the next few months. Going forward, I need to nail down the specific story I want my concept to have. This will inform all aspects of my immersive experience, and there’s no point in doing anything else until I have some aspect of that figured out. Potential story ideas that I’m considering for an immersive experience: gambling and how it has overtaken our society; how technology affects the development of teenage/adult men; how the failure of our institutions has made us create a court of public opinion and how we can create other institutions together. All of these different ideas are huge and ambitious, and I will need to find my personal entrypoint in one of these in order to develop a meaningful immersive experience for my thesis.
Also, I will keep exploring other immersive or exhibit work that has great indicators and instructions built-in to an environment. I will make it a point to visit more museums and installations around New York in the next 2 weeks. With the narrative and instructions down, I feel that the rest of any experience will flow easily from there.
Ryan Rotella 
Leave a comment