Last semester, I had made models for the panorama presentation of the story bible for my thesis in Google Sketchup.
In the course of making the models, I fell into an infatuation with Google SketchUp, but using a mouse to model was like modeling with a bar of soap. So I began imagining a one-to-one correspondence of physical pointer position with a virtual cursor. My first sketch was for webcams, but then I remembered Ken's post.
Ken Leung posted a link to a novel Wii input device probject, by Johnny Lee. Johnny Chung Lee at Carnegie Mellon University prototyped a low-cost whiteboard and finger-controlled user interface using a Wii mote as the camera.
and so an idea began for a...
Wii Media Lab
Use infrared camera and infrared LED to intuitively control the cursor of a variety of applications while being free from the keyboard and mouse. These are input device systems for the Zemeckis Media Lab. My requirements for these input devices were:
There are four input setups for the Zemeckis Media Lab. Each one is illustrated in the Google SketchUp model. Please refer to the corresponding scene tabs for each proposal. SketchUp is free.
- utilize the room
- to present interactive media
- to navigate an virtual environment objects
- to model objects
- low cost
- available hardware
- bluetooth port
- infrared LED
- easy to recreate for future users
- easy to apply to other artists
2D pointer + 6 DOF device
Traditional Wii remote and sensor bar configuration. Application: Collaborate while developing interactive project, such as ZML Spaceship, and intuitively note item on screen. Most any motion mapped in WarioWare Smooth Moves could be programmed by a skilled engineer.
Wii mote, infared light pen, by Johnny Chung Lee. Application: Control multiple points for scaling or rotating on 2D plane.
2D multitouch whiteboard
Wii mote camera, infrared light pen, by Johnny Chung Lee. Applications: Draw on the wall. Control multiple points.
3D multipoint tracker
2 wii motes, 2 infrared light pens. Applications: Model 3D image. Navigate cursor in 3D environment.
- Engineers (such as I) are unfamiliar with programming input devices.
- Several Bluetooth connections interfere with each other and slow down data transfer.
- Cameras can be confused by multiple points of light from neighboring input stations.
- User may be confused by six degrees of freedom by moving in 3D space.
- User becomes fatigued from maintaining limbs away from body and supporting devices after several seconds.
- User may be confused by ubiquitous shape of wii mote.
- Adapting device drivers and user interface control maps to software applications may be time consuming for each application.
To lessen risks, the following precautions are intended:
- Test Wii remote on a Mac and Windows machine.
- Test operation of standard Wii system in Zemeckis Media Lab.
- Test previous implementations of Wii input device for PC that have similar objectives.
- Scan heat sources in Zemeckis Media Lab. Test range and field of view and field of emission of camera and infrared LED. Direct camera away from heat sources in Zemeckis Media Lab.
- Minimize number of bluetooth devices in use, by starting with a single wii remote.
- Employ a real-time introspective development language that I'm familiar with, Python.
- Start development in an OS that I have access to at school and home and the most experience with: Windows XP.
- Establish unit tests to quickly confirm operation of camera, wii remote, device drivers, and other units that are developed.
- Roleplay rehearsal of input device manipulation to detect fatigue, disorientation, confusion, and lack of affordances.
- Test simple cases of system operation and functionality for usability.
- Cite adaptation required to modify system to control novel software.
The system employs components and binds them with Python.
Hypothetical stages of signal flow on Windows XP for simple motion test:
- Raw image in physical space.
- Blobs of infrared light on photocell array of wii remote.
- Two dimensional points within wii remote.
- Bluetooth encoded 2D points from wii remote transmitter.
- Bluetooth encoded 2D points at Bluetooth-USB adapter.
- Bluetooth encoded 2D points in Windows PC.
- Wii 2D points decoded by pyBluez, Python binding to Bluetooth drivers.
- Wii data decoded by wii.py script, adapted from Wiiewer.
- Matrix transformation of 2D input to normalize plane of 2D points.
- Simple gesture recognition, as jestur.py.
- Inject input to mouse movement through windll.py.
- Display device shows mouse pointer moving.
Test case requirements
To evaluate the potential, a prototype is suggested as a test case. This test case:
- Renders simple 3D scene.
- The input device enables the user to translate, rotate, and scale an object in the environment.
- With 15 minutes training, a graduate designer is faster at performing these manipulations with the input device than with a mouse.
- The prototype can be developed by a couple full-time graduate designers within their technical knowledge set in a couple weeks.
- Prototype costs less than $200 in hardware and software.
Here are some test cases. At least one of these will be functionally prototypedduring the experimentation class.
Test case: Chopsticks
This prototype uses Python-Ogre to render a simple 3D scene. Python-Ogre is free, open-source, and interpreted, for real-time editing. PythonCard is used for tool user interface.
Test case: Model manipulator
Based on "Wii input to Blender" at Georgia Tech. Within a 3D modeler, Blender, script control of user interface by Python. Use pitch, yaw, and roll to rotate a model. Use buttons to select editing mode. Translate, rotate, and scale selection by wii remote manipulation.
Test case: Free paint
Paint on 3D environment by wii remote input by generating objects lights and particle systems along path of motion.
Minimum System Requirements:
1 GHz Pentium III or AMD Athalon
8x Speed DVD Drive
3 GB Hard Disk Space
64 MB Video Card
DirectX 9 compatible sound & video drivers
Wii remote with 2 charged AA batteries
Infrared light source with power supply
Each station costs:
- $30 wii remote
- $20? infrared LED with power
- $30 Bluetooth-USB adapter
- Open source software
- 1 hour? installation labor by skilled information technologist
- 1 day? adaptation labor by skilled information technologist
- A day? per month of service for maintenance and support per site
- 3 weeks of part-time research and development for test case by team of two (both are engineers and interaction designers)
Wii remote transmits point data for four points at 100 Hz, or 100 times per second. While an infrared camera may have a comparable price, the frame rate is lower and the image signal is raw, requiring further processing.
Research suggests that the prerequisite technology exists within the technical requirements, and that the objectives of the proposal are possible by skilled engineers.
Wii pointer, whiteboard, desktop VR, by Johnny Chung Lee.
Untested wii remote access exists for Python.
Wiimote communication test.
Wiimote input types.
Bluetooth for Python
How to make a webcam work in infrared
Python gesture recognition
Wiimote accelerometer for Vpython
OSC for media control
OSCulator for Mac
Wii input and device drivers
Mac Python mouse automation?
Wii input to Blender on Windows
Some users have had trouble making this Wii-Blender mashup work.
NNiimote (neural-net gesture recognition)
Wii to MIDI on Windows tutorial
GlovePIE for Windows
wxPython mouse control