Some Setbacks with Kinect

Today was a testing day filled with setbacks. With a few members of my team, we have been trying to make the motion sensors, touch sensors, and synthetic music work together but every part is popping up with an issue.

With the Kinects, we've tried multiple configurations to get the most accurate and smooth motion data as possible. Unfortunately, none of them seemed to be the perfect choice for our project yet.

At first, we placed the Kinects behind the performer at a 45-degree angle from the origin. This configuration allows us to make sure that the audience can't get between the performer and the cameras. However, the biggest setback is that because both cameras are only seeing the back of the performer when she turns or spins around herself, the cameras cannot discern which direction she is facing. Thus we lose accurate tracking and even when the performer turns back to her original position, the tracking remains confused.

(45-degree configuration)

The second configuration we tried was 90 degrees from the origin. We placed the Kinects perpendicular to each other to check if having the side-view would help the tracking when the performer turns or spins. Unfortunately, this wasn't the case and the tracking still had issues when the performer faced a different direction. Despite this, we noted that the general jittery-ness of the tracked performer and the popping of arms and legs improved slightly in this configuration compared to the 45-degree one.

(90-degree configuration)

The third and last configuration we tried was with the cameras placed at the front and back at a 180-degree angle. Following the iPi documentation, we made sure to offset the cameras in opposite directions slightly so they weren't completely facing each other. For tracking spins and turns, this configuration was easily the best. Unfortunately, our main concern with this configuration is the physical locations of the cameras in our setup. The front camera can easily get blocked by the audience, while the back camera sits right in the middle of our screen and blocks the projection. This setup can be our latest resort but it's still not preferable.

(Front and back configuration)

Along with our issue with Kinect configurations, today we also tried to figure out the best way to calibrate our Kinects. We had to get very creative with our solutions, but that'll be a whole another blog post of its own :3

Though we weren't able to make motion-touch-music work together today as we had hoped, I still managed to get some triggers and effects in Unreal based on the tracking! The following first video showcases 2 interactions I've set up in Unreal: a sphere following basic physics and a light trigger based on collision, and the latter half of the video also shows issues we had with the performer spinning in the first two configurations. The second video is based on our final front-back configuration when the spin tracked more accurately.

 

I haven't worked with collisions and physics in Unreal before but setting up the sphere and light was much easier than I thought! By next week, I hope that we can include more effects into our scene to showcase in class.
(Sample light collision set-up)

Comments

Post a Comment

Popular posts from this blog

DMX in Unreal Engine

Image
Though I dived into Unreal DMX setup without confirming if my lights were properly connected, I eventually figured out how to get them to work through the engine! My main sources were the Unreal documentation and this specific  tutorial on Youtube. There are many tutorials that show cool DMX stuff that can be done in Unreal, but I've realized that only a limited few actually show how to pipe in the equipment into Unreal Engine or set up the network connections. Here are the steps I followed to get the lighting to work with Unreal: Creating a project and enabling plug-ins. I chose to create a DMX template project, which already came with all necessary plugins enabled. Changing project settings. I created a new "Output Port" and adjusted the settings based on the hardware and network I used. (The first 3 ports are the pre-existing ones from the project template.) - Net3 Gateway uses sACN protocol instead of Art-Net. - "Network Interface IP Address" is the main ...
Read more

Final Tests on CRMX

Image
Putting together everything I've researched and learned so far, created a mini set with DMX lighting on finals week. I've started by gathering all the equipment and setting up all the connections. After a few iterations, my final set included a camera, 5 Aputure lights (controlled via CRMX), and a big screen displaying the Unreal Engine scene for background. For my first test, I wanted to use a scene of a city with multiple, moving light sources. Looking through the Epic store, I found the free Soul City sample and decided to use it for its night scene. I've set up my scene in Unreal Engine and parented a Cine Camera and Scene Capture 2D camera to a locator. These cameras were facing the opposite way from each other. This way, the Cine Camera would render the scene behind the actor and put it on the display screen, while the Scene Capture 2D camera would calculate lighting information from the scene in front of the actor and send it to Pixel Mapping. To make the world move...
Read more

Pixel Mapping

Image
Here is one of the ideas that got stuck with me since the first week of class: taking the light information from a camera or a specific point in space in Unreal and doing some calculations to send the average color into a DMX light in the studio. While looking for information or ideas on how to achieve this, I've come across an Unreal feature called Pixel Mapping. And yes... this feature was exactly what I had been thinking of creating all this time, so turns out I don't have to reinvent the wheel! Here is the Unreal Pixel Mapping documentation with more detailed information. Since this feature didn't seem too complicated to set up, I wanted to give it a try and see how customizable it can be. This process simply has 3 main actors... 2D Scene Capture Camera: This camera is how the artist can decide which area in the scene will be captured in view and used for color averaging. Render Target: This actor takes the output from the camera and turns it into a texture. Pixel Mapp...
Read more