Last semester, I had made models for the panorama presentation of the story bible for my thesis in Google Sketchup.

In the course of making the models, I fell into an infatuation with Google SketchUp, but using a mouse to model was like modeling with a bar of soap. So I began imagining a one-to-one correspondence of physical pointer position with a virtual cursor. My first sketch was for webcams, but then I remembered Ken's post.

Ken Leung posted a link to a novel Wii input device probject, by Johnny Lee. Johnny Chung Lee at Carnegie Mellon University prototyped a low-cost whiteboard and finger-controlled user interface using a Wii mote as the camera.

and so an idea began for a...

Wii Media Lab

Ethan Kennerly

Use infrared camera and infrared LED to intuitively control the cursor of a variety of applications while being free from the keyboard and mouse. These are input device systems for the Zemeckis Media Lab. My requirements for these input devices were: There are four input setups for the Zemeckis Media Lab. Each one is illustrated in the Google SketchUp model. Please refer to the corresponding scene tabs for each proposal. SketchUp is free.

2D pointer + 6 DOF device

Traditional Wii remote and sensor bar configuration. Application: Collaborate while developing interactive project, such as ZML Spaceship, and intuitively note item on screen. Most any motion mapped in WarioWare Smooth Moves could be programmed by a skilled engineer.

2D multipointer

Wii mote, infared light pen, by Johnny Chung Lee. Application: Control multiple points for scaling or rotating on 2D plane.

2D multitouch whiteboard

Wii mote camera, infrared light pen, by Johnny Chung Lee. Applications: Draw on the wall. Control multiple points.

3D multipoint tracker

2 wii motes, 2 infrared light pens. Applications: Model 3D image. Navigate cursor in 3D environment.

Risks

Precautions

To lessen risks, the following precautions are intended:
  1. Test Wii remote on a Mac and Windows machine.
  2. Test operation of standard Wii system in Zemeckis Media Lab.
  3. Test previous implementations of Wii input device for PC that have similar objectives.
  4. Scan heat sources in Zemeckis Media Lab. Test range and field of view and field of emission of camera and infrared LED. Direct camera away from heat sources in Zemeckis Media Lab.
  5. Minimize number of bluetooth devices in use, by starting with a single wii remote.
  6. Employ a real-time introspective development language that I'm familiar with, Python.
  7. Start development in an OS that I have access to at school and home and the most experience with: Windows XP.
  8. Establish unit tests to quickly confirm operation of camera, wii remote, device drivers, and other units that are developed.
  9. Roleplay rehearsal of input device manipulation to detect fatigue, disorientation, confusion, and lack of affordances.
  10. Test simple cases of system operation and functionality for usability.
  11. Cite adaptation required to modify system to control novel software.

System Architecture

The system employs components and binds them with Python.

Hypothetical stages of signal flow on Windows XP for simple motion test:

  1. Raw image in physical space.
  2. Blobs of infrared light on photocell array of wii remote.
  3. Two dimensional points within wii remote.
  4. Bluetooth encoded 2D points from wii remote transmitter.
  5. Bluetooth encoded 2D points at Bluetooth-USB adapter.
  6. Bluetooth encoded 2D points in Windows PC.
  7. Wii 2D points decoded by pyBluez, Python binding to Bluetooth drivers.
  8. Wii data decoded by wii.py script, adapted from Wiiewer.
  9. Matrix transformation of 2D input to normalize plane of 2D points.
  10. Simple gesture recognition, as jestur.py.
  11. Inject input to mouse movement through windll.py.
  12. Display device shows mouse pointer moving.

Test case requirements

To evaluate the potential, a prototype is suggested as a test case. This test case:

Here are some test cases. At least one of these will be functionally prototypedduring the experimentation class.

Test case: Chopsticks

This prototype uses Python-Ogre to render a simple 3D scene. Python-Ogre is free, open-source, and interpreted, for real-time editing. PythonCard is used for tool user interface.

Test case: Model manipulator

Based on "Wii input to Blender" at Georgia Tech. Within a 3D modeler, Blender, script control of user interface by Python. Use pitch, yaw, and roll to rotate a model. Use buttons to select editing mode. Translate, rotate, and scale selection by wii remote manipulation.

Test case: Free paint

Paint on 3D environment by wii remote input by generating objects lights and particle systems along path of motion.

Technical Requirements

Minimum System Requirements:
  • 1 GHz Pentium III or AMD Athalon
  • 8x Speed DVD Drive
  • 3 GB Hard Disk Space
  • 64 MB Video Card
  • DirectX 9 compatible sound & video drivers
  • Windows XP

  • Wii remote with 2 charged AA batteries
  • Infrared light source with power supply
  • Bluetooth adapter

    Software:

  • Python 2.5

    Costs

    Each station costs: Project costs:

    Alternatives

    Wii remote transmits point data for four points at 100 Hz, or 100 times per second. While an infrared camera may have a comparable price, the frame rate is lower and the image signal is raw, requiring further processing.

    Sources

    Research suggests that the prerequisite technology exists within the technical requirements, and that the objectives of the proposal are possible by skilled engineers.

    Wii pointer, whiteboard, desktop VR, by Johnny Chung Lee.
    http://www.cs.cmu.edu/~johnny/projects/wii/

    Untested wii remote access exists for Python.
    http://www.wiili.org/index.php/Wiiewer

    Wiimote communication test.
    http://www.wiili.org/index.php/Wiimotecomm

    Wiimote input types.
    http://www.wiili.org/index.php/GlovePIE

    Bluetooth for Python
    http://pypi.python.org/pypi/PyBluez/0.9.2


    http://code.google.com/p/pybluez/
    http://people.csail.mit.edu/albert/bluez-intro/

    How to make a webcam work in infrared
    http://www.hoagieshouse.com/IR/

    Python gesture recognition
    http://sourceforge.net/projects/jestur/
    http://projects.reichbier.de/gestikk/

    Non-wii motion
    http://www.youtube.com/watch?v=UGImvHEtUpE

    Wiimote accelerometer for Vpython
    http://www.youtube.com/watch?v=eZ9xKAFwPLg&NR=1

    Mouse control
    http://kvance.livejournal.com/985732.html
    http://mail.python.org/pipermail/python-list/2003-August/219221.html
    http://sourceforge.net/projects/pywinauto/

    OSC for media control
    http://createdigitalmotion.com/tag/windows/

    OSCulator for Mac Wii input and device drivers
    http://www.osculator.net/wiki/Main/HomePage

    Mac Python mouse automation?
    http://appscript.sourceforge.net/
    http://appscript.sourceforge.net/pyosa.html
    http://wiki.python.org/moin/MacPython/AppleScript

    Wii input to Blender on Windows
    http://www.selectparks.net/modules.php?file=article&name=News&sid=628

    Some users have had trouble making this Wii-Blender mashup work.
    http://blenderartists.org/forum/showthread.php?t=109517

    NNiimote (neural-net gesture recognition)
    http://jorgecardoso.eu/blog/index.php?/archives/92-NNiimote-update.html

    Wii to MIDI on Windows tutorial
    http://vjkungfu.tv/archive/wii-to-midi-walkthrough-for-windows/ GlovePIE for Windows
    http://carl.kenner.googlepages.com/glovepie

    Python OSC
    http://wiretap.stetson.edu/downloads/kits/pyKit.tar.gz
    http://opensoundcontrol.org/implementation/python-simple-osc
    http://www.nabble.com/simple-osc-on-python-to5170289.html

    wxPython mouse control
    http://www.google.com/search?hl=en&safe=off&rls=com.microsoft%3A*%3AIE-SearchBox&rlz=1I7GFRD&q=wxPython+mouse