As an example project to show how Moviesandbox can be used for real-time digital Puppeteering, I created a scene with two fish telling each other a joke.

It’s two of my socks, outfitted with Hannah’s neoprene Bend Sensors and connected to an Arduino Microcontroller.

Here’s a picture of the Hardware setup:

These “augmented socks” serve as the main controllers for our two fish. Using them as sockpuppets, opening and closing the puppets’ mouth will be picked up by the bend sensors as change in resistance. I wrote a small processing application that reads out the sensor data and converts it to OSC messages that can be sent to Moviesandbox. There, I connect them to bone rotations for the mouths of our two fish. In the end, opening and closing the socks will open and close the fishes mouths.

Here’s a video explaining the setup:

Another three simple hardware buttons send trigger commands to switch cameras in moviesandbox, and there’s some key-inputs in the MSB setup that I am using to make the the fish swim to their designated places.

You could go all out and create sliders and joysticks for more controlled fish movement, but for the purpose of this example, I didn’t want to go too crazy. Also, I only spent three days on this (there’s so much other stuff to do and show!).

On the visual side, I implemented Depth of Field as a postprocessing effect into the MSB renderer to get a better aquarium feel into the scene. Here’s a screenshot to illustrate (the DV camera I used for the movie doesn’t quite do it justice):

Postprocessing does not mean extended rendering times. It’s still all nice and real-time. In fact, I was surprised to see how little of the framerate the effects chops off, even though I am using a rather unoptimised 5×5 gaussian blur kernel.

This example will be included in the V1.0 release. If you support Moviesandbox on Kickstarter and select the “Arduino” reward, you will get an Arduino plus bend-sensor (and some printed documentation) to test it out for yourself! Otherwise, you can build your own with the links provided above. Code for Arduino and Processing will be up soon.