Testing Kinect and Digital Puppet

Yesterday we met in the mocap lab to test our Kinect setup with TouchDesigner and Unreal Engine. We spent quite some time trying to get the motion and pose data from the iPi software to TouchDesigner. Unfortunately, this didn't work out because neither iPi nor Touch have their native plug-in to communicate with each other. The only solution seemed to be having a member of the team write a script to solve the communication issue. But since that might complicate things in our project, we decided to fall back into our backup plan and use Unreal Engine.

For this method, I'd say we were somewhat successful. At first, we set up the Kinect and iPi software as usual and made sure it was tracking the person in view. With tracking going, we made sure to turn on data streaming and choose the UE5 mannequin "Manny" as our test rig.

(Default rig and stream setup)

On the Unreal Engine side, I made sure that the Live Link plug-ins from iPi were added to the project and activated. Then I added the data stream to live link sources and set up a blueprint for the puppet to run based on the real-time data received from iPi. As soon as the puppets were connected, I realized that the data flow from iPi to UE was extremely slow and "Manny" in Unreal would lag almost a whole minute behind the actual person in front of Kinect. We realized this was probably due to hardware limitations. Despite having RTX3080 on the computer we were testing, the lag caused our real-time data stream to be basically useless.

(Live link setup and pose blueprint in UE)

To optimize our stream, I ended up excluding some data transfer from iPi to UE (all finger joints and detailed spine joints), reduced graphics settings in the Unreal project, and closed down everything else running on the laptop we were using. In the end, we managed to get much smoother animation in Unreal.
(Streaming from iPi to Unreal Engine in real-time)

As you might have noticed in the video, there are unfortunately still some jitter issues in the data. I think our only other option is to switch to hardware with a better GPU, so I suggested that next time, we test with the render box in the mocap room; it has an A6000 board. If we can set that machine up, we should be able to test whether the jitter issue can't be solved or not. If it can't, we'll have to look into different methods asap.

Right now, the backup options we have talked about are:
  • Removing jitter within Unreal Engine
  • Running one Kinect directly into UE (no iPi streaming)
  • Running one Kinect directly into TouchDesigner

Comments

Post a Comment

Popular posts from this blog

DMX in Unreal Engine

Image
Though I dived into Unreal DMX setup without confirming if my lights were properly connected, I eventually figured out how to get them to work through the engine! My main sources were the Unreal documentation and this specific  tutorial on Youtube. There are many tutorials that show cool DMX stuff that can be done in Unreal, but I've realized that only a limited few actually show how to pipe in the equipment into Unreal Engine or set up the network connections. Here are the steps I followed to get the lighting to work with Unreal: Creating a project and enabling plug-ins. I chose to create a DMX template project, which already came with all necessary plugins enabled. Changing project settings. I created a new "Output Port" and adjusted the settings based on the hardware and network I used. (The first 3 ports are the pre-existing ones from the project template.) - Net3 Gateway uses sACN protocol instead of Art-Net. - "Network Interface IP Address" is the main ...
Read more

Final Tests on CRMX

Image
Putting together everything I've researched and learned so far, created a mini set with DMX lighting on finals week. I've started by gathering all the equipment and setting up all the connections. After a few iterations, my final set included a camera, 5 Aputure lights (controlled via CRMX), and a big screen displaying the Unreal Engine scene for background. For my first test, I wanted to use a scene of a city with multiple, moving light sources. Looking through the Epic store, I found the free Soul City sample and decided to use it for its night scene. I've set up my scene in Unreal Engine and parented a Cine Camera and Scene Capture 2D camera to a locator. These cameras were facing the opposite way from each other. This way, the Cine Camera would render the scene behind the actor and put it on the display screen, while the Scene Capture 2D camera would calculate lighting information from the scene in front of the actor and send it to Pixel Mapping. To make the world move...
Read more

Pixel Mapping

Image
Here is one of the ideas that got stuck with me since the first week of class: taking the light information from a camera or a specific point in space in Unreal and doing some calculations to send the average color into a DMX light in the studio. While looking for information or ideas on how to achieve this, I've come across an Unreal feature called Pixel Mapping. And yes... this feature was exactly what I had been thinking of creating all this time, so turns out I don't have to reinvent the wheel! Here is the Unreal Pixel Mapping documentation with more detailed information. Since this feature didn't seem too complicated to set up, I wanted to give it a try and see how customizable it can be. This process simply has 3 main actors... 2D Scene Capture Camera: This camera is how the artist can decide which area in the scene will be captured in view and used for color averaging. Render Target: This actor takes the output from the camera and turns it into a texture. Pixel Mapp...
Read more