Last semester, I had made models for the panorama presentation of the story bible for my thesis in Google Sketchup.

In the course of making the models, I fell into an infatuation with Google SketchUp, but using a mouse to model was like modeling with a bar of soap. So I began imagining a one-to-one correspondence of physical pointer position with a virtual cursor. My first sketch was for webcams, but then I remembered Ken's post.

Ken Leung posted a link to a novel Wii input device probject, by Johnny Lee. Johnny Chung Lee at Carnegie Mellon University prototyped a low-cost whiteboard and finger-controlled user interface using a Wii mote as the camera.

and so an idea began for a...

Wii Media Lab

Ethan Kennerly

Use infrared camera and infrared LED to intuitively control the cursor of a variety of applications while being free from the keyboard and mouse. These are input device systems for the Zemeckis Media Lab. My requirements for these input devices were: There are four input setups for the Zemeckis Media Lab. Each one is illustrated in the Google SketchUp model. Please refer to the corresponding scene tabs for each proposal. SketchUp is free.

2D pointer + 6 DOF device

Traditional Wii remote and sensor bar configuration. Application: Collaborate while developing interactive project, such as ZML Spaceship, and intuitively note item on screen. Most any motion mapped in WarioWare Smooth Moves could be programmed by a skilled engineer.

2D multipointer

Wii mote, infared light pen, by Johnny Chung Lee. Application: Control multiple points for scaling or rotating on 2D plane.

2D multitouch whiteboard

Wii mote camera, infrared light pen, by Johnny Chung Lee. Applications: Draw on the wall. Control multiple points.

3D multipoint tracker

2 wii motes, 2 infrared light pens. Applications: Model 3D image. Navigate cursor in 3D environment.



To lessen risks, the following precautions are intended:
  1. Test Wii remote on a Mac and Windows machine.
  2. Test operation of standard Wii system in Zemeckis Media Lab.
  3. Test previous implementations of Wii input device for PC that have similar objectives.
  4. Scan heat sources in Zemeckis Media Lab. Test range and field of view and field of emission of camera and infrared LED. Direct camera away from heat sources in Zemeckis Media Lab.
  5. Minimize number of bluetooth devices in use, by starting with a single wii remote.
  6. Employ a real-time introspective development language that I'm familiar with, Python.
  7. Start development in an OS that I have access to at school and home and the most experience with: Windows XP.
  8. Establish unit tests to quickly confirm operation of camera, wii remote, device drivers, and other units that are developed.
  9. Roleplay rehearsal of input device manipulation to detect fatigue, disorientation, confusion, and lack of affordances.
  10. Test simple cases of system operation and functionality for usability.
  11. Cite adaptation required to modify system to control novel software.

System Architecture

The system employs components and binds them with Python.

Hypothetical stages of signal flow on Windows XP for simple motion test:

  1. Raw image in physical space.
  2. Blobs of infrared light on photocell array of wii remote.
  3. Two dimensional points within wii remote.
  4. Bluetooth encoded 2D points from wii remote transmitter.
  5. Bluetooth encoded 2D points at Bluetooth-USB adapter.
  6. Bluetooth encoded 2D points in Windows PC.
  7. Wii 2D points decoded by pyBluez, Python binding to Bluetooth drivers.
  8. Wii data decoded by script, adapted from Wiiewer.
  9. Matrix transformation of 2D input to normalize plane of 2D points.
  10. Simple gesture recognition, as
  11. Inject input to mouse movement through
  12. Display device shows mouse pointer moving.

Test case requirements

To evaluate the potential, a prototype is suggested as a test case. This test case:

Here are some test cases. At least one of these will be functionally prototypedduring the experimentation class.

Test case: Chopsticks

This prototype uses Python-Ogre to render a simple 3D scene. Python-Ogre is free, open-source, and interpreted, for real-time editing. PythonCard is used for tool user interface.

Test case: Model manipulator

Based on "Wii input to Blender" at Georgia Tech. Within a 3D modeler, Blender, script control of user interface by Python. Use pitch, yaw, and roll to rotate a model. Use buttons to select editing mode. Translate, rotate, and scale selection by wii remote manipulation.

Test case: Free paint

Paint on 3D environment by wii remote input by generating objects lights and particle systems along path of motion.

Technical Requirements

Minimum System Requirements:
  • 1 GHz Pentium III or AMD Athalon
  • 8x Speed DVD Drive
  • 3 GB Hard Disk Space
  • 64 MB Video Card
  • DirectX 9 compatible sound & video drivers
  • Windows XP

  • Wii remote with 2 charged AA batteries
  • Infrared light source with power supply
  • Bluetooth adapter


  • Python 2.5


    Each station costs: Project costs:


    Wii remote transmits point data for four points at 100 Hz, or 100 times per second. While an infrared camera may have a comparable price, the frame rate is lower and the image signal is raw, requiring further processing.


    Research suggests that the prerequisite technology exists within the technical requirements, and that the objectives of the proposal are possible by skilled engineers.

    Wii pointer, whiteboard, desktop VR, by Johnny Chung Lee.

    Untested wii remote access exists for Python.

    Wiimote communication test.

    Wiimote input types.

    Bluetooth for Python

    How to make a webcam work in infrared

    Python gesture recognition

    Non-wii motion

    Wiimote accelerometer for Vpython

    Mouse control

    OSC for media control

    OSCulator for Mac Wii input and device drivers

    Mac Python mouse automation?

    Wii input to Blender on Windows

    Some users have had trouble making this Wii-Blender mashup work.

    NNiimote (neural-net gesture recognition)

    Wii to MIDI on Windows tutorial GlovePIE for Windows

    Python OSC

    wxPython mouse control*%3AIE-SearchBox&rlz=1I7GFRD&q=wxPython+mouse