Stories
Slash Boxes
Comments
Search

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

Wiimote as Multi-Touch Display Controller

Posted by Soulskill on Wed Dec 19, 2007 09:37 AM
from the amazing-home-projects dept.
Hardware Hacking Wii (Games) Hardware
Tmack writes "While hard-hacks with the Wiimote are somewhat old news, this particular implementation is quite interesting. Using the infrared camera on the Wiimote, pens with LEDs instead of ink, and an LCD projector, Johnny Chung Lee of Carnegie Mellon University has created software to use them as a (relatively) cheap multi-touch display. Any surface onto which you can project becomes an interactive multi-touch display, as demonstrated in the video at the link. He has the software available for download, along with some other neat projects. Lee has also documented another impressive Wiimote hack.
+ -
story

Related Stories

[+] Wiimote Hacking Goes Big-Time 51 comments
The Wall Street Journal is taking Wiimote hacking seriously. A front-page article from this past weekend discusses the many uses to which enterprising hackers have put the Wiimote, the motion sensing piece of the Wii console. Included is a video of a few of the projects in action. "Tim Groeneboom, who lives in the Netherlands, uses his Wii-mote to spice up his deejay act. He was inspired by a video on the Web of a California music student bobbing in front of the computer in his room and making jabbing motions with the Wii-mote to splice different tracks. During his second gig with the Wii-mote, Mr. Groeneboom, 22, says he was able to roam up to about 100 feet from his deejay booth and still be able to control how the music blended and do some sound effects ... Aaron Rasmussen has a sporting purpose for his Wii-mote. At his Garden Grove, Calif. software company, USMechatronics, he and his partner stuck a tennis racket in the 'hand' of a $40,000 industrial robot and then tweaked the Wii-mote to control the robot's arm so it can hit back tennis balls on the factory floor. 'This is what we do to relax,' he says."
[+] News: The Wiimote As Yoda Intended - A Lightsaber 268 comments
An anonymous reader writes "So what if the Wii can't handle the awesome 'next-generation' physics engine the other consoles will enjoy when Star Wars: The Force Unleashed is released? LucasArts announced today that Krome Studios is developing a version of the game for the Nintendo console, and players will finally get to use the Wiimote for its intended purpose — as a lightsaber. 'The sword-swinging action will be exclusive to the Wii version, and even then, it will only be available in an exclusive "duel mode." The description in the release says that this duel mode will be a multiplayer affair.'"
[+] Multitouch Without Touch Using Wiimote 94 comments
owlgorithm writes to mention that Gizmodo has a neat hack for the multitouch Holy Grail — multitouch without the touch. This hack turns the Wiimote into a receiver for IR light reflected from an emitter off of your fingers using reflective tape.
Submission: Wiimote as multi-touch display controller by Tmack (593755)
[+] Python + Motion detection = Fweemote 30 comments
jedie writes "After reading about different Wiimote hacks on Slashdot I decided to make a video with some demos of my motion-detection library. You can watch the video here. There's a link to the sourcecode (GPL) as well, but the demo is win32 only. It's basically a webcam and some software in python to track LEDs (preferrably IRs). In the demo video, you see the software (albeit badly because of the webcam's IR filter being removed) tracking two differently colored LEDs, so multiplayer is possible. The software can track multiple points easily, and when combined with IR-LEDs, it's easy to simulate one Wiimote (i.e. calculating the distance and angle between two IR-LEDs to determine where the remote is relative to the webcam). I want the code to get some publicity, because I don't have time to work on it (dissertation, blabla) but I don't want the code (however messy it is) to go to waste."
[+] Technology: Wiimote Turns TV into Touchless MS Surface 104 comments
RemyBR writes "User interface project allows you to control objects on a display using gestures, working like Microsoft's Surface but without touching the screen at all. Inspired by Johnny Chung Lee's work, the system requires you to wear Minority Report-style gloves equipped with infrared emitters on your fingertips. A Wiimote on top of the display keeps track of these IR LEDs, while the software can read the motion down to two-finger pinching gestures for image zooming."
[+] Modern Methods For Sharing Innovation 91 comments
The New York Times is running a story about Johnny Chung Lee, a hardware hacker made famous for his projects which modified the Nintendo Wiimote to do things like positional head tracking and multi-touch display control. The article focuses on the suggestion that Lee's use of YouTube to demonstrate his innovations has done a better job of communicating his ideas than more traditional methods could. Quoting: "He might have published a paper that only a few dozen specialists would have read. A talk at a conference would have brought a slightly larger audience. In either case, it would have taken months for his ideas to reach others. Small wonder, then, that he maintains that posting to YouTube has been an essential part of his success as an inventor. 'Sharing an idea the right way is just as important as doing the work itself,' he says. 'If you create something but nobody knows, it's as if it never happened.'"
[+] Head Tracking w/ the Wiimote 169 comments
mrneutron2003 writes "This guy just doesn't know when to stop. Johnny Chung Lee graces us with yet another one of his inventive Wiimote projects. This time it involves using the Wiimote and a pair of inexpensive LED safety goggles (with the standard LED's replaced with InfraRed ones) to allow positional head tracking , achieving an effect similar to what is experienced with three dimensional displays and CAVE systems. The video dramatically illustrates the effect. Game developers take note. This simple little variation on infrared tracking could allow for some seriously immersive gameplay in the future." This guy deserves a medal.
[+] Games: Why Natal Is a Big Deal 303 comments
Kikizo has an editorial piece evaluating the Xbox 360's upcoming motion-control scheme, Project Natal, and discussing why it's a bigger step forward for interactive gaming than many people think. Quoting: "[Natal] accurately perceives players in 3D space, simultaneously tracking over 48 joints on your body, enabling it to accurately redraw your skeleton in real time as you move about. On a separate 'debug screen' in the closed-doors session, we could witness for ourselves the 'mind's eye' of Natal, visually showing how it completely understands where we are, how we're moving, where we are in 3D space, how far in front of my face my hand is, whatever. It can supposedly even track individual hand and finger movement when it switches into this more finely-tuned mode. ... There is a surprising feeling of tactility and iPhone-like fluidity and precision to the way Natal works." Another interesting bit of news about Natal is that Wii-hacker Johnny Chung Lee is part of the development team. We've discussed some of his creations in the past.
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Wiimote as Multi-Touch Display Controller 50 Comments More /

 Full
 Abbreviated
 Hidden
More
Loading... please wait.

  • I can get one now? (Score:3, Funny)

    by FatAlb3rt (533682) on Wednesday December 19 2007, @09:40AM (#21751204) Homepage
    What's this...a way to pass time until the demand for the Wii consoles goes down enough that I don't have to crawl over people to get one? Sign me up!

    Oh...the Wiimotes are out of stock too? *walking away, hanging head*
    • There, there. On to the good news: we still have plenty of PS3's in stock, Mr. Alb3rt.

    • Re:I can get one now? (Score:4, Funny)

      by DrWho520 (655973) on Wednesday December 19 2007, @10:12AM (#21751526) Journal
      I have seen Wiimotes everywhere, Buy Buy, Circuit City, Target and Walmart. Maybe they should contract to the Wiimote manufacturer to also start cranking out Wii's.

    • Re:I can get one now? (Score:4, Informative)

      by Smidge204 (605297) on Wednesday December 19 2007, @11:30AM (#21752452)
      Just about any CMOS webcam can be easily modified to see only IR light. It involves removing the IR filter (usually a small glass plate with special coating inside the lens assembly) and adding a filter to block visible light (usually a fully exposed piece of film negative).

      =Smidge=

      • Re: (Score:3, Insightful)

        by ByOhTek (1181381)
        or Nintendo predicted a higher demand for the Wiimotes

        either
        (A) the Wiis are used less for multiplayer than Nintendo expected
        or
        (B) the Wiimotes have a higher survival rate than Nintendo expected
        or
        (C) A+B
        • I particularly like option

          (B) the Wiimotes have a higher survival rate than Nintendo expected

          Since this could imply that perhaps Nintendo expected more people to (inadvertently) send their wiimotes crashing through their TVs or sailing across the room...
          • inadvertantly, or intentionally.

            I have to admit, I'm quite annoyed with the detection on mine. I keep wanting to bash the thing against my coffee table because it won't detect that it is pointing at the screen.

            • Re: (Score:2, Informative)

              by gallwapa (909389)
              semi OT, but...

              A few notes if you're having trouble:

              First, make sure your IR sources don't confuse the wiimote. Example: Opening my drapes on the window behind the TV on a warm, sunny day, made the wiimote useless.

              Second, set the proper height for the "sensor" bar in the Wii console itself. It does make a difference, especially if you have a large TV

              Third, make sure your batteries are properly charged. Don't wait for the warning to go to red. When it gets to 1 bar, put the batteries in the recharger.

              T

            • I keep wanting to bash the thing against my coffee table because it won't detect that it is pointing at the screen.

              Check that there are no bright IR light sources nearby. Sunlight and Christmas lights are two common ones that confuse the Wii Remote. Also make sure that your Sensor Bar is far enough forward that it isn't blocked at certain angles.

              An easy way to figure things out is to go into the Wii Remote settings screen. There's a black and white image there that effectively shows you exactly what the Wii Remote "sees". You should see a few dots that do not flicker or go away as you move the remote around. If the dots disappear at any point during your tests, you may need to adjust your sensor bar.

              If the lights appear to be working okay, but the remote is still confused, try turning down the sensitivity of the remote. That will encourage the remote to ignore light sources other than the sensor bar.
              • That's useful, thanks.

                Actually, my problem is how close I have to be. They should make a smaller "nearsighted" bar.

                Since I got my replacement contact lenses (and can focus at larger distances), it's better, though still not perfect. They need to make a sensor-bar mini, that's half the width, designed for closer use.
                • You've probably already tried this, but if not, try going into your Wii's settings and adjusting the Wiimote's sensitivity.

                • Actually, my problem is how close I have to be.

                  Ah, I see how that could be a problem. The minimum distance is really about 4-5 feet from the screen. Even then, you pretty much need to be in a chair to play effectively.

                  Possibly the best advice I can give you is this: Get a bigger TV. :-)
                  • 47" is already particularly absurd for my living room... And about the top of my budget
                    • 47" is definitely a good size. Exactly how close are you to the TV? If you used your Wii Remote at the same distance I do during development (I have a 15" LCD Monitor), you're going to have a fully-immersive experience! :-P

                      If your TV is big enough and you're standing at least 4 feet away, then its probably your sensor bar. Either there is too back background IR light, or the bar is becoming invisible at certain angles. If it's the latter, you can try one of two things:

                      1. Move the sensor bar closer to the ed
                • The bar itself is nothing but two IR LEDs, so it shouldn't be difficult to build or modify an existing bar to be any length you wish. I've even seen someone who managed to get the Wii to work with two candles placed on top of the TV :)

  • Wiimote is a Wii-Detector (Score:5, Informative)

    by imstanny (722685) on Wednesday December 19 2007, @09:49AM (#21751294)
    The Wii-mote is actually a sensor that is used as a remote. It doesn't just send a signal, but rather it 'sees' the location of the 'motion detection' bar. In the traditional usage, it has only 1 point that it detects - the bar. But if you have 'many of these bars' the Wiimote is used as a detector of multi-inputs.

    • Re:Wiimote is a Wii-Detector (Score:5, Informative)

      by Paralizer (792155) on Wednesday December 19 2007, @10:01AM (#21751420) Homepage
      The sensor bar has two IR points on either end to help with calculating things like distance. The wiimote itself can see up to 4 IR sources at any given time. So the multitouch interface would be limited to 4 pens, not that you'd need more than that though.
      • ... you see how big/close they are as well, so it's not just position.

        • Re:they have size... (Score:5, Informative)

          by slim (1652) <john&hartnup,net> on Wednesday December 19 2007, @11:04AM (#21752114) Homepage

          ... you see how big/close they are as well, so it's not just position.
          No, the Wii software expects the Wiimote to see two IR dots (one on each end of the 'sensor' bar). Everything is calculated from these two points. X/Y position is the X/Y position of the midpoint between the two dots. Roll is the angle between the two dots. Z position (closeness) is the distance between the two dots.
        • That is an option you can enable if you want that data. Looking at it though, it seems pretty general and doesn't give very accurate results. For example, you may have to move backward a couple of feet for it to drop to the next smallest size interval. If you are using the two IR sources to estimate distance you will have much better precision.
          • Of course it's not precise, but still it does work if you want to track just one IR source. You need one that sends a constant amount if IR in all direction (lighter will work). It worked pretty well for me in the 30cm-90cm rangre..

            But using just x/y is just fine, and a lot more precise..

  • My project (Score:3, Interesting)

    by pkadd (1203286) on Wednesday December 19 2007, @10:00AM (#21751400) Homepage
    I am currently working on using a wii controller as primary pointing-device for my livingroom. The goal is to make it the only device needed to controll the projector + PC i use for DVD, TV, and Music playback in my home. I will post info when i've figured it all out.
    • You should know that the wiimote does not emit infrared like normal remotes - in case that was your angle. The wiimote is an infrared camera. The IR emitters are in the sensor bar on top of your TV.
      • That's nothing a little duct-tape can't fix!
      • You can have three paintings on the wall, a phone, a radio and a TV, which all have different IR patterns. Then when you point you Wiimote on the phone painting, the Wii mote will see the pattern and tell your computer what you are pointing at. The computer will then activate the Twinkle (sip) on your computer and you can interface with it.. :-)

        Using the motion sensors to answer calls [engadget.com] might be a bit awkward (some minutes into the clip).
    • I've done this. It was the first thing I thought of when I learned how the Wiimotes worked.

      It's actually rather simple...
      1. First you need a sensor bar that is no dependent of your Wii. You can make one [google.com] or buy one [ebgames.com].
      2. You need a bluetooth module
      3. A driver to handle requests for your Wiimote and translate them into mouse movements, etc. Linux, Mac and Windows drivers [wiili.org]
      4. (optional) I mapped my buttons to multimedia keys and bound some of them to commands
      5. You need some software. In my case I've written my own call

  • Detecting Parkinsons Tremors (Score:5, Interesting)

    by superid (46543) on Wednesday December 19 2007, @10:16AM (#21751552) Homepage
    My own trival "hack". I'm using Darwiinremote [sourceforge.net] to read xyz accelerometer data and Octave [octave.og] to make/graph ffts of Parkinson's tremors. It's remarkably sensitive!

  • Nice hack but that's pretty much how some existing surface computing devices work anyway. That a mass produced device is cheaper is nothing new.

    • Re:Nothing new... (Score:4, Insightful)

      by skorch (906936) on Wednesday December 19 2007, @11:10AM (#21752208)
      Some of the most important innovations are not about making something completely novel but about coming up with creative and more efficient ways of using what we already have. If this hack reproduces 75% of the performance of a commercial product at a fraction of the cost, then this is already a more cost-effective solution for simple touch-screen presentation software. It also puts it well within reach of the at-home user and not something that can only even be considered for large corporate presentations. All of a sudden an 8ft sq canvas for digital art is not out of the question, and it's scalable to a simple laptop as a tablet-pc replacement.

      And considering the application he offers is free this can only encourage more experimentation that can lead to even more innovative applications of a relatively cheap and abundant product which is (Nintendo sanctioned or not) becoming more and more of a multi-purpose tool.
      • If this hack reproduces 75% of the performance of a commercial product at a fraction of the cost,

        Actually it's alot better than most products on the market atm. Mainly because of the speed, it's insanely fast compared too some commercial options that only support one touch.. And if you look at many of the multi touch interfaces being done you will see that they almost always lag 0.1s or something...
    • You never did like the TV show MacGyver did you? I don't think you understand the joy of hacking something together to do something it wasn't designed to do. Whats good about this is that you could use it to replace a 2000+ dollar device for 50 bucks. It may not be new but it's defiantly interesting and cool. I mean I could go home tonight and have a touchscreen on my computer(I have a wii and bluetooth).

      I think your just a negative Nancy.

  • Better than the last one (Score:3, Insightful)

    by TheLostSamurai (1051736) on Wednesday December 19 2007, @10:19AM (#21751578)
    When Johnny Lee posted his last hack [slashdot.org], it seemed like a cool idea to track your fingers in the air, but not all that practical, as was discussed at length. This hack however could actually have very practical applications for discussion and presentation, without the $5000 price tag of many interactive white board products. I'd like to see if anyone ever attempts to implement this in a small business environment, such as a design studio.
  • I'm so going to have to try this. The software isn't complex and with some work I'll finally have a giant digital canvas with the infrared pen acting as pen/airbrush.
  • I've got to believe that using this with photoshop is the best use...
    • It will matter a lot on the resolution. He made a comment that the further away from the pen the remote is, the lower the resolution, so using this for Photoshop would work best with his personal setup where the remote was only ~3 feet away from the surface. Using an IR only LED would be a wise choice as well so you don't wind up with a red tinge to everything you try to click on.

      -Rick
    • Won't work for a long while, AFAIK photoshop doesn't have multitouch support.
  • The only problem, when you use this with a projector-based system, you're always casting shadows. It would be better as the presenter if you had a tablet PC replicating the screen display and you multi-touched on the tablet. Touch-sensitive screens are always ridiculously expensive so this might be a good way to keep it cheaper.

    I'm still waiting for the large-format flexy-screens that are as cheap as rolled paper. I love my LCD screens but they're just too damn expensive and fragile right now. I want 100" w
    • Rear projection... most projectors can be setup for this (by allowing them to project a mirror image).

      It looks like an insanely cool hack, and using things you have lying around the house too.
    • You ever hear of "rear projection"? Completely eliminates the shadow issue.

      -Rick
    • Rear projection has already been mentioned as a solution for the shadow casting. Another solution is to point the wiimote directly at your laptop screen and do the 4-point calibration on that, and just use your laptop as a tablet pc hooked up through the projector. Considerably cheaper than an actual tablet pc in both cases.
  • If you have two wiimotes, one located 45 degrees to your left, one 45 degrees to your right, wouldn't the appropriate math be able to locate the position of your pen in 3 dimensions?

    Normally I just nod and appreciate these hacking projects from a distance, but I think I'm going to jump in and do this for myself. Wish me luck, everybody.
    • Oh, and another way to track an object in 3D with IR is to track multiple points (at least 3)
      whose relative position is known. This what the TrackIR does.
  • People always joke about this, and I see that it doesn't. The c# gave me a little hope thinking maybe it could run with mono but it uses direct x. I'm wondering if there are any wii mote hacks for linux.
  • I made something similar in using python and a webcam. My hack was pretty simple:
    1) IR webcam
    2) IR-pen, similar to this guy's
    3) software in python to track the IR light
    4) demo-apps in pygame (tracking, swinging, 2 player pong)

    It worked pretty well to be honest. I also atached some LED's to a wireless mouse and controlled the mouse from distance (similar to but not as good as the wii-mote).I tried to add multiple inputs using one cam by using color video input and color LED's instead of IR LED's, but then sc
    • ### It seems to me humans are much more efficient at moving one hand at a time.

      That is only true when the motion of both hands are completly independent of each other. Human brain is doing quite fine with using both hands at once for a single task (trying typing with a single hand or opening a bottle with just a single hand).

      ### I think the usefulness of multi-touch displays is limited.

      Go and watch this demo video [youtube.com], now with a Wiimote you are limited to four points so many stuff seen there won't work, but al
    • I think it gets interesting to be able to have multiple people writing at once. Win, lose, or draw?

      It also might be interesting to use for zooming purposes. You can see the big picture as well as a box showing a portion of the big picture zoomed in to work on details...but moving that portion to where you want can be a pain in a lot of programs. While I couldn't write with two pens I think I could accurately move my one hand to the general area that I want to work on and hold it there or slide it occasi
    • He does mention near the beginning of the video I watched that when you set up the wiimote you have to be sure it has an unobstructed view of the pointers. It's near the same time he says the wiimote has a 40 degree (iirc) field of vision and the further away it is the less resolution you get.

      a fix to this could be to put the projector below a table and project upwards (inverted) on a glass surface so there is no change of LOSing any of the pointers.

      I actually thought that was where he'd go with it.

    • I'm getting that burning, trolling sensation, but I'll answer anyway: Don't the games on the Wii act as motion capture already? I suppose you want something you could merge into a 3-D environment of your own choosing, though.

All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2009 Geeknet, Inc.