I have yet to give up entirely on ShinyTouch, my experiment into creating a touch screen input system which requires virtually no setup. For people who haven’t read my posts from last year, it works because magically things look shinier when you look at it from an angle. And so if you mount a camera at an angle (It doesn’t need to be as extreme as the screenshot above), you end up seeing a reflection on the surface of the screen (this could be aided by a transparent layer of acrylic or by having a glossy display, but as you can see, mine are matte, but they still work). The other pretty obvious idea of ShinyTouch, is that on a reflective surface, especially observed from a non-direct angle, you can see that the distance from the reflection (I guess my eighth grade science teacher would say the “virtual image”) to the apparent finger, or “real image” is twice the distance from either to the surface of the display. In other words, the reflection gets closer to you when you get closer to the mirror. A webcam usually gives a two-dimensional bitmap of data (and one non-spatial dimension of time). This gives (after a perspective transform) the X and Y positions of the finger. But an important aspect of a touchscreen and what this technology is also capable of, a “zero-touch screen”, is a Z axis: the distance of the finger and the screen. A touchscreen has a binary Z-axis: touch or no touch. Since you can measure the distance between the apparent real finger and it’s reflection, you can get the Z-axis. That’s how ShinyTouch works.

Last year someone was interested and actually contributed some code. Eventually we both agreed that my code was crap so he decided to rewrite it, this time using less PIL and pixel manipulation, and instead, opting for more OpenCV so it’s faster. The project died a bit early this year, and with nothing more to do, I decided to revive it. His code had some neat features:

• Better perspective performs
• Faster
• Less Buggy
• Simpler configuration (track bars instead of key combinations and editing JSON files)
• Yellow square to indicate which corner to click when callibrating (actually, I wrote that feature)

It was left however, at a pretty unfinished state. It couldn’t do anything more than generate config files through a nice UI and doing a perspective transform on the raw video feed. So in the last few days, I added a few more features.

• Convert perspective-transformed code into grayscale
• Apply a 6×6 gaussian blur filter
• Apply a binary threshold filter
• Copy it over to PIL and shrink the canvas by 75% for performance reasons
• Hack a Python flood-fill function to do blob detection (because I couldn’t compile any python bindings for the opencv blob library)
• Filter those blobs (sort of)

Basically, it means ShinyTouch can now do multi-touch. Though the Z-axis processing, which is really what the project is all about still sucks. Like it sucks a lot. But when it does work (on a rare occasion), you get multitouch (yay). If TUIO gets ported (again), it’ll probably be able to interface with all the neat TUIO based apps.

Code here: http://github.com/antimatter15/shinytouch/ Please help, you probably don’t want to try it (yet).

Interestingly, if you don’t use the video and just do data from a raw image, you get upwards of 125fps on V8. Adding the video, it ceases to work on Chromium (maybe a linux thing? this tells me it’s just linux, but you can never be so sure).

//At this point, run away as the algorithm gets messy and hackish

So the thing just searches from right to left up to down within the quad. When it finds a column of something that fits the rgb range of the finger that is larger than a certain threshold, it checks for a reflection from the point. If it detects a reflection then yay! it throws the data at the perspective warper (based on a python one which is based on a C# one and though it would probably be easier to port from C# to JS making long chains of derivative work is fun). If there wasnt a reflection then it logs that and if that number is larger than some othe rthreshold then it kills the scanning and goes on with it’s life. The reflection algorithm just takes the point 5 pixels to the right and assumes that would be a reflection if there was one and a point 15px above and 5px to the left (nasty stuff) and takes the hue value from their RGB values. It takes the absolute value of the difference of the hue values multiplied by 100 (or 200 in the python version) and compares it with a preset configuration variable.

So now that that horrible algorithm which was just whatever came to my little totally untrained mind first. But it works semi-decently, at least for me. But you can hopefully see how nasty it’s inner workings are and it inspires people to clean it up. It’s quite a bit more readable than the Python version and only 200 lines of JS so it won’t be too hard to understand.

But since HTML5 has no Video capture thing for webcams, and my webcam doesn’t work with flash so I can’t use that canvas<-flash webcam bridge i built, uh, almost 2 years ago. So now you just get to gaze at my finger moving for like 20 seconds!

http://antimatter15.com/misc/shiny/shinytouch.html

The first one is shot from recordMyDesktop-gtk and the second one is from Cycorder on my iPhone.

One big issue with ShinyTouch is that it didn’t transform points correctly, but now it has been totally resolved by stealing this (MIT licensed) from the linux (python!) port of Wiimote Whiteboard (Johnny lee’s famous work). Now it works totally insanely awesomely, no elitists necessary.

So I totally super insanely over underestimated the complexity of the ShinyTouch project. It’s actually probably one of the most complicated projects I’ve ever attempted. And if all hardware related things are really this complicated, then I really applaud the efforts of all those people who work on this stuff. While ShinyTouch isn’t really large code-wise (currently less than 200 lines of Python), it’s not a very simple program.

And though the concept can be implemented as a simple thing, you would loose a lot of accuracy. The most basic possible implementation would be to search for a color matching the range for a finger (it does in fact stick out quite a bit) by searching pixel-by pixel from right-to-left (if the camera is mounted to the right of the monitor) and it will stop once it hits the fingertip, which is the first thing of skin that the camera sees since it’s mounted at an angle. After that, you would check if the color immediately next to the point and check if it fits in a range. Then if it matches, then you fix the distortion due to perception and send it to a app through something like TUIO.

With that being the most simple thing possible, everything else I’ve thought of builds on the second part: the reflection detection. This part is what I’ve discovered to be the most complicated. And the complexity of it isn’t totally unexpected. This detection is really all that is truly special about this project. Without it, it’s just finger tracking, something that’s been used for tons of projects.

Here is the auto-calibration process, it alternates between black and white

This is part of Auto-Calibration.

This is some stuff from the command line:

This is just hovering over the screen, notice it’s not touching, and the algorithm can distinctly recognize the lack of a touch because the reflection is seperated from the finger by a significant gap. (Compare the top red bar).

This is a actual touch, you can see that the red bar is far larger, and it’s very distinctly a touch event.

There’s a draw tool and, here is a primitive drawing of a smiley. The dots come from an issue with PIL/OpenCV or something that makes the image all chopped up and sends the point to an arbitary point on the screen.

This is the magical sensor the whole thing is powered by: An unmodified Playstation 3 Eye on a tissue box with a pink Office Depot eraser on the back (because the camera is made tilted and the script can’t handle those tilts very well)

It’s not too insanely slow either. This is 31 frames per second coming from a pure python app, all from a scripting language. It is nowhere as fast as the normal fast native apps.

So today, I added VideoCapture support, so it will now work better on Windows.

Auto calibration has been rewritten, and a few other small changes.

Then there is a function that makes cool stuff happen. It goes right to left to search for massive groups of consecutively marked hot pixel change areas. Ir searches for a general trapezoidal shape. And takes the lenghs and heights and positions. And right now, typing on my iPhone I wish the spell corrector was as awesome as the google wave contextual system.
So in the near future, shinytouch will be as simple as launchingthe app and hitting a calibrate button and the computer does the rest. Maybe it might ask you to touch the screen in a special magic box or click on your finger afterwards but overall, zero setup and almost no calibration.
So on another note, shinytouch is now almost a month old idea-wise. In my idea book, the references to it date back to June 28, though it may have originated a few days prior. For shinytouch the beginning window is quite a bit more broad. Anywhere from last year to march. I seem to recall late January experimentation with mirrors.
So now,
With shinytouch being the more promising (more acessible and radical) I have stalled development on mirrortouch. It’s quite annoying how fast time passes. There is so much that I really want to do but there is nost not enough time.

So what is reliable and practical? I think maybe just something which recognizes the general characteristics of the reflection (like looking like the actual image) shapewise could work. I dont know.

So i have a little notebook with tons of random sketches on ideas and equations and tiny pseudo-code algorithms to theoretically improve the detection rate.  One is basically analyzing more than teh single pixel it does now, by “measuring” the width of the reflection and matching it up with the image (and in all cases it should be equivalent). The problem is that it’s hard to measure, so I have this algorithm which I think may resolve it by comparing the value of  the pixel in question with the known point and the known non-reflection.

But it’s nowhere near complete, and ideas are welcome.