Looking into Motion Capture

This week, a few members of the team and I spent some time in the motion capture studio. We wanted to test a few things and see how feasible it is for us to use motion capture in our final project. The most important question we had was whether we could capture the pose of the performer and track their movements without actually putting them in a full-body mocap suit.

An idea our team has been gravitating towards is "marionette on strings". We possibly want to make the connection between the performer on stage and the "puppet" on the projection screen. Putting the performer in a mocap suit would be the most accurate and reliable source to get real-time data, but it would limit the creativity we could use when creating an interactive costume that will go with the performance. Thus, I decided to test the idea of tracking props instead of full-body!

(Test prop and "Arm1" tracking in Vicon Shogun Live)

My idea was to place multiple marker clusters on the performer and track them as "props" in Vicon Shogun software. We could create at least 5 props (1 for each appendage and 1 for the general location and height of the performer) and get a good amount of real-time data. While we wouldn't be able to track a full skeleton with this method, the location and rotation of the props would still give us enough data to determine if the performer is raising their arm, leg, bending their body, etc.

After getting the test prop ready (named "Arm1" in Shogun Live), we tested streaming of the data to other software. Vicon already has native plug-ins for Motion Builder and Unreal Engine. This way, I was easily able to send tracking information to MoBu and see the location and rotation of the "Arm1" prop. In theory, with this method, we can easily send tracking data to Unreal Engine also, and create interactive effects on screen that change with the live information received.

(Getting live data from Vicon to Motion Builder.)

However, there were multiple set-backs with the mocap set up :(
  1. Vicon does not allow creating a prop with less than 4 markers. My initial idea was to create a prop with max 2 markers to allow the costume designers to have as much freedom designing as possible without worrying about the markers. Having 20+ markers on the performer may not be ideal for our project.
  2. Vicon does not have a plug-in for TouchDesigner. Most of my team's interactive art experience comes from TouchDesigner. While we are willing to try and create something in Unreal Engine, if we are to use TouchDesigner, we will have to find a method of reliable data streaming between the two software.
  3. To capture accurate, full-body poses, we will have to put the actor in a mocap suit, which is not ideal for costuming.
This is where our research for the week turned towards looking into different sources for capturing motion and body. For the upcoming week, we will dive deeper into new methods for body capture. I will also be looking into creating interactive experiences within Unreal Engine, in case we do not want to rely on TouchDesigner.

Comments

Post a Comment

Popular posts from this blog

DMX in Unreal Engine

Image
Though I dived into Unreal DMX setup without confirming if my lights were properly connected, I eventually figured out how to get them to work through the engine! My main sources were the Unreal documentation and this specific  tutorial on Youtube. There are many tutorials that show cool DMX stuff that can be done in Unreal, but I've realized that only a limited few actually show how to pipe in the equipment into Unreal Engine or set up the network connections. Here are the steps I followed to get the lighting to work with Unreal: Creating a project and enabling plug-ins. I chose to create a DMX template project, which already came with all necessary plugins enabled. Changing project settings. I created a new "Output Port" and adjusted the settings based on the hardware and network I used. (The first 3 ports are the pre-existing ones from the project template.) - Net3 Gateway uses sACN protocol instead of Art-Net. - "Network Interface IP Address" is the main ...
Read more

Final Tests on CRMX

Image
Putting together everything I've researched and learned so far, created a mini set with DMX lighting on finals week. I've started by gathering all the equipment and setting up all the connections. After a few iterations, my final set included a camera, 5 Aputure lights (controlled via CRMX), and a big screen displaying the Unreal Engine scene for background. For my first test, I wanted to use a scene of a city with multiple, moving light sources. Looking through the Epic store, I found the free Soul City sample and decided to use it for its night scene. I've set up my scene in Unreal Engine and parented a Cine Camera and Scene Capture 2D camera to a locator. These cameras were facing the opposite way from each other. This way, the Cine Camera would render the scene behind the actor and put it on the display screen, while the Scene Capture 2D camera would calculate lighting information from the scene in front of the actor and send it to Pixel Mapping. To make the world move...
Read more

Pixel Mapping

Image
Here is one of the ideas that got stuck with me since the first week of class: taking the light information from a camera or a specific point in space in Unreal and doing some calculations to send the average color into a DMX light in the studio. While looking for information or ideas on how to achieve this, I've come across an Unreal feature called Pixel Mapping. And yes... this feature was exactly what I had been thinking of creating all this time, so turns out I don't have to reinvent the wheel! Here is the Unreal Pixel Mapping documentation with more detailed information. Since this feature didn't seem too complicated to set up, I wanted to give it a try and see how customizable it can be. This process simply has 3 main actors... 2D Scene Capture Camera: This camera is how the artist can decide which area in the scene will be captured in view and used for color averaging. Render Target: This actor takes the output from the camera and turns it into a texture. Pixel Mapp...
Read more