Saturday, May 2, 2009

Making a touch sensitive sidepanel

I love coming up with new ways of controlling my computer. But I'm also a cheap bastard. So, in joining my two aforementioned characteristics, I bring to you probably one of the cheapest ways of capturing finger presses on a non-touch display. It won't allow you to track touching on all of the display or tracking multiple fingers, but it will allow you to create a touch sensitive sidepanel. What's keeping you from doing the entire screen is the narrow field of view in most webcameras, so you could hypothetically get around this problem with some form of lensing.

It's also worth noting that it's a better idea to experiment with this on CRT displays instead of flat displays, since CRTs are easier to clean and harder to damage with your fingers.

All you require is 2 webcams with a decent frame rate (mostly anything in the 10-20 buck range will do the trick, so long as they can do 30 fps or so), some black cardboard and some adhesive tape.

Figure 1: The set-up. A. Vertical camera, B. Horizontal camera, C. Black screen. D. Section of screen where fingers can be tracked.

The black cardboard in opposite of the cameras is to make finger identification much simpler. Instead of comparing with some complicated idle state, you'll just have to check what portions of the view isn't black.

Figure 2: What the cameras see. A. The area to be ignored. B. The area to be scanned for fingers (with black cardboard in the background). C. A finger. D. Portion of the screen at a steep angle (to be ignored as well)

Once you've got your hardware set up, the actual interaction with the hardware can be a bit of an headache, but I got it working by reading the Video4Linux documentation. You can probably settle for a cargo cult implementation of the actual driver interaction. There's also some limitations in your computer that you may run into. The USB bus only has so much bandwidth, and if you're unlucky, that might not be enough for two video feeds. I had to hack the kernel module to get around that problem with my drivers.

The basic finger identification algorithm is pretty simple. You take an average of the positions of all the pixels which are bright. Simply scan region B in Figure 2 for pixels with a color intensity stronger than some threshold value, and for such regions, increment one counter by the brightness, and one counter by the vertical position of the pixel multiplied with it's brightness. When you've iterated over all points, divide by the first counter.

float sum = 0, pos = 0;
for(x = 0; x < x_max; x++) {
for(y = y_min; y < y_max; y++)
if(ispixel(x,y)) {

Besides the actual implementation, you'll need to do some form of calibration. It's mostly trial and error. First, figure out how much off the zero position is, then estimate how much off the scale is, and compensate for that.

You can cut down on some of the noise by giving the values some "inertia", which essentially the same as applying a low-pass filter to your data. Use the following algorithm:

X = X*(1-alpha) + X_new*alpha;
Y = Y*(1-alpha) + Y_new*alpha;

where alpha is a number in the 0 ... 1 range (I chose 0.25), and the position will stabilize a lot.

We can also use the fact that the position is relatively stable to identify a "press", i.e. when the difference in position between iterations is lower than some value, we decide that the user has pressed his/her finger. We may also want to wait a couple of frames before triggering a new press, since obviously it's not desirable to have the same action performed 30 times because you didn't have the reflexes of a ninja and accidentally pressing down for longer than 0.33 ms.

Putting it all together, I made a side-bar for my secondary screen that is touch sensitive, borrowing some icons from the crystal project.

The code is a working prototype, and a lot of stuff is hard coded, but it should be possible to salvage a lot of the tricky parts for use in whatever project you're working on.


It should on most x86 linux systems with SDL present, but requires a bunch of icons to be put in a subdirectory resources/ to actually run. I got mine from 64x64/apps and 32x32/actions in the crystal project icon tarball.

I'm actually surprised at how useful it is. It may not be 100% accurate, but it's still a really nice way of controlling your mp3 player or starting new programs.


  1. There are a couple techniques to get a more robust average in the face of outliers.

    First is to use the median instead of the mean. If your data is clustered around a central value, then a few outliers won't pull it far from that center. This method could have trouble if the density of samples near the center isn't consistent.

    The second is a way to mix median and mean to get the advantages of both. 1: Estimate the mean and standard deviation of the set. 2: Eliminate all points more than 2 standard deviations from the mean. 3: Recalculate the mean from the reduced set. 4: Don't tell any statisticians what you have done (we are making all sort of assumptions about the distribution that don't actually hold).

  2. Heh, thanks for the idea.

    Taking the median + lowpassing it made it extremely stable.

    At first I discarded the idea of using two cams side by side and then using parallax to determine the depth because of the instability of the values I got, but that might actually be doable now.