Cognotics Home  
Search Cognotics:
Home > OpenCV Resources > Seeing With OpenCV > Part 3 Pages:   1   2   3   Sidebar   Next
Seeing With OpenCV, Part 3: Follow that Face!
by Robin Hewitt
This article originally appeared in SERVO Magazine, March 2007. Reprinted by permission of T & L Publications, Inc.

The previous article in this series explained how to implement and configure face detection. In this article, I'll show you how to use OpenCV to track a face once you've detected it.

Face Tracking in OpenCV

Tracking a face is more difficult than tracking a strongly-colored object. Skin reflects the ambient light in subtle, changing ways as a person's head turns or tilts.

In principle, you could track a face by locating it over and over in every frame, using the Haar detector described in last month's article. To do that, however, you'd need to decide if the face you detected in each frame is the same face. If the detector finds more than one face in a frame, you'd need to decide which detection is the one you're tracking. Finally, if a person's head tilts towards one shoulder, or turns towards profile view, the frontal face detector will no longer detect it, so you'd need to handle that situation as well.

Fortunately, OpenCV includes specialized code for tracking a face efficiently, using continuity between frames to help find the best match for the face it's following.

The algorithm that OpenCV uses for face tracking is called Camshift. Camshift uses color information, but rather than relying on a single color, it tracks a combination of colors. Since it tracks by color, it can follow a face through orientation changes that the Haar detector can't handle. The sidebar, "How OpenCV's Face Tracker Works," explains this algorithm in more detail.

Camshift was originally developed for hands-free gaming. It's designed to be very fast and "lightweight" so the computer can do other tasks while tracking. Since it was developed as a gaming interface, Camshift also has a (limited) ability to detect changes in head position, such as tilting the head to one side. Could you use that ability to communicate with your robot? Maybe two fast head tilts means "Come here, robot!"

Figure 1 shows OpenCV's face tracker in action - following a face as it tilts to one side and during a turn to profile.

. Figure 1 .
Figure 1. OpenCV's face tracker in action. It's able to follow a face as it tilts to one side and during a turn to profile.


The Camshift Demo

The OpenCV samples directory contains a program called camshiftdemo. You can get some good hands-on experience and an intuitive feel for the Camshift algorithm with this demo program. Here are the steps for doing that:

  1. Plug in a webcam
  2. Launch the program called camshiftdemo in the samples directory.
  3. Use your mouse to select a rectangle centered tightly on your face.
  4. Click in the video-display window and type the letter b. (The display should change to look something like the view in Figure 2.)
  5. Adjust the sliders for Smin and Vmin until the ellipse is well positioned and the background is mostly black.
  6. Repeat Step 4 to toggle back to normal view, then use Camshift to track your face.
. Figure 2 .
Figure 2. To tune the Camshift parameters smin and vmin, run the camshiftdemo program in the samples directory. These parameters are easier to set if you toggle to the backprojection view by clicking in the view window, then typing b.

Tuning Camshift

As mentioned above, Camshift uses a combination of colors to track faces. In the representation that Camshift uses, color is undefined for pixels that have a neutral shade (white, gray, or black). Color can be computed for pixels that are almost neutral, but their color values are unstable, and these pixels contribute noise that interferes with tracking.

Camshift uses two parameters - smin and vmin - to screen out this noise. These parameters define thresholds for ignoring pixels that are too close to neutral. vmin sets the threshold for "almost black," and smin for "almost gray." These two threshold levels will need to be adjusted for your setup to get good results with Camshift.

Camshift also uses a third parameter, called vmax, to set a threshold for pixels that are too bright. But smin has the side effect of also eliminating pixels that are close to white, so you shouldn't need to tweak vmax to get good results.

The easiest way to select good values for your setup is with camshiftdemo. As suggested in the preceding section, it's easier to set these if you toggle the viewing mode by clicking the view window and typing b. (This alternative view is the called the "face-probability," or "backprojection" view. It's explained in the sidebar.)

Figure 2 shows the effect of adjusting smin and vmin. Initially, in the first frame, these were at their default values. At these levels, Camshift displayed a very large ellipse that included not only my face, but half the room as well! The reason for the oversized face detection is clearly visible in the face-probability view. Background pixels with a nearly neutral shade contributed too much noise when vmin and smin were at their default values.

The middle and right views in Figure 2 show the effect of increasing first smin, then vmin. In the righthand view, noisy pixels have been largely eliminated, but the face region still produces a strong signal. Tracking is now quite good, and the ellipse is well positioned.

Source Code for this Article CONTINUED   1   2   3   Sidebar   Next
. .
. .
Source Code for this Article
Related Resources:

Download OpenCV
Official OpenCV usergroup
OpenCV Wiki

Articles in this series:

Part 1: Introduction to OpenCV

Part 2: Finding Faces in Images

Part 3: Follow that Face!

Part 4: Face Recognition With Eigenface

Part 5: Implementing Eigenface
bottom margin
Home | OpenCV Resources | Seeing With OpenCV