Michael Friis' Blog

Some time ago I wrote a post on how to do face detection in C# using OpenCV. I’ve since begun using the Emgu CV wrapper instead of opencvdotnet. Emgu CV is much better, in active development and it even runs on Mono. Two gotchas:

1. You don’t have to install OpenCV, but instead have to copy the relevant dlls (included with the Emgu CV download) to the folder where you code executes.
2. Open CV and X64 are not friends. If you’re running X64 Windows (and unless you are up to recompiling OpenCV) you have to make sure your app is compiled to X86, instead of the usual “Any CPU”.
3. Remember to add PictureBox as per the original tutorial.

Here’s sample code:

using System;
using System.Windows.Forms;
using System.Drawing;
using Emgu.CV;
using Emgu.Util;
using Emgu.CV.Structure;
using Emgu.CV.CvEnum;

namespace opencvtut
{
    public partial class Form1 : Form
    {
		private Capture cap;
		private HaarCascade haar;

        public Form1()
        {
            InitializeComponent();
        }

        private void timer1_Tick(object sender, EventArgs e)
        {
		using (Image<Bgr, byte> nextFrame = cap.QueryFrame())
		{
			if (nextFrame != null)
			{
				// there's only one channel (greyscale), hence the zero index
				//var faces = nextFrame.DetectHaarCascade(haar)[0];
				Image<Gray, byte> grayframe = nextFrame.Convert<Gray, byte>();
				var faces =
					grayframe.DetectHaarCascade(
						haar, 1.4, 4,
						HAAR_DETECTION_TYPE.DO_CANNY_PRUNING,
						new Size(nextFrame.Width/8, nextFrame.Height/8)
						)[0];

				foreach (var face in faces)
				{
					nextFrame.Draw(face.rect, new Bgr(0,double.MaxValue,0), 3);
				}
				pictureBox1.Image = nextFrame.ToBitmap();
			}
		}
        }

        private void Form1_Load(object sender, EventArgs e)
        {
            // passing 0 gets zeroth webcam
			cap = new Capture(0);
            // adjust path to find your xml
			haar = new HaarCascade(
                "..\\..\\..\\..\\lib\\haarcascade_frontalface_alt2.xml");
        }
    }
}

This post will demonstrate the feasability of face detection in web cam feeds grabbed by a flash/flex application. It’s inspired by a prototype of Martin Speelmans, informed by my work with Flash and web cams and my experiments with OpenCV. The basic premise is that a flex application running in a users browser grabs web cam shots, compresses them and sends them to a server. The server decompresses the picture, does face detection on it, and sends the result back to the client. All fast enough to give the user a relatively smooth experience. A fairly impressive Actionscript 3 face detection implementation exists, but my guess is that — barring a quantum leap in Adobe compiler technology — it’s gonna be a few years before fast, reliable Actionscript 3 face detection is feasible in Flash Player. Silverlight may be a different story, but it doesn’t support web cams yet.

The first prototype was implemented in Flash since I refused to program in XML (which you tend to to in Flex) as a matter of principle. When it turned out that Flash doesn’t support web services, I relented and changed to Flex. The parameter marshalling code that Flex Builder generates turned out to be too slow for low latency operation tough and was dumped in favour of a HttpService. The server part is implemented in ASP.Net, with C# calling EmguCV wrapped OpenCV. The code is at the bottom of the post.

The client goes through the following steps:

1. Setting up the UI, getting the web cam running, setting up a timer to tick three times a second.
2. On timer ticks, a shot is grabbed from the web cam.
3. The picture is scaled to 320×240 pixels. The scaling is handled is by native library code and takes a few milliseconds.
4. The picture is converted to grayscale. OpenCV facedetection only operates on one colour channel anyway and grayscale images compress better. The colour transformation is handled by more native library code and also takes just a few milliseconds.
5. The picture is then PNG encoded. An encoder lives in the mx.graphics.codec namespace and at ~150ms for a 320×240 pic, it’s pretty fast. The resulting bytearray is around 60kb.
6. The bytearray is base64 encoded which takes around 100ms.
7. Finally the bytearray is sent to the server along with a sequence number. The server returns the sequence number so that the client can make sure it doesn’t use old, out-of-sequence results.

Upon receipt of a post, the server does the following:

1. The request string is base64 decoded to a bytearray.
2. A bitmap is created from the bytearray and made into a EmguCV grayscale image.
3. The faces are indentified with a Haar Cascade detector
4. The results, along with some performance information and the sequence number, are returned to the client as XML

Total server-side time is around 150ms with almost all of it spent in the face detection call. The client can do whatever it want’s with the face information — my demonstration projects a bitmap on top of detected  faces in the video feed. Total roundtrip time is less than half a second at three frames a second on a non-bandwith constrained connection in the same city as the server. This is probably enough to give relatively passive users the illusion of real-time face detection. I’ve also gotten positive reports from testers in the US where latency is higher.

I experimented with zlib compression instead of PNG encoding but the difference in both space and time was marginal (PNG uses zlib so that shouldn’t be too surprising). Instead of grayscaling the image, I also tried yanking out just one colour channel from the 32bit RGBA pixels, but iterating 320x240x4 bytearrays in Actionscript was dead-slow. Ditching http in favour of AMF and using AMF.Net or FluorineFx on the server may improve latency further.

What is point if all this you ask? A haven’t the foggiest idea really, but it’s good fun. You can certainly use it to create very silly facebook applications. You could conceivably use it to detect the mood of your users by determining whether they are smiling or not. Is this usable in a production scenario? Hard to say. The current server is a 3.5GHz Celeron D — a right riceboy machine. With one user hitting it, the CPU is at 20-30% utlization, suggesting it’ll tolerate a few concurrent users at most. Getting a proper machine and installing Intel Performance Primitives would probably help a lot. If you really meant it, you could run the OpenCV part on a PS3 or on a GPU.

Download the important parts of the code here.

I’ve been mucking around with face detection a bit lately. Here’s a short guide to getting face detection running in C# in a short amount of time.

UPDATE: Recommend EmguCV for C# wrapping OpenCV, read the updated guide.

I’ve identified two free computer vision libraries that do face detection:  Torch3vision and OpenCV(I’m sure there are plenty more, but these seem to be comprehensive, recently updated and freely available). Torch3vision claims to be better than OpenCV but on the other hand more people are building libraries and wrappers around OpenCV and there’s even a wrapper for .Net. While it doesn’t yet wrap the face detection components of OpenCV, it seems to be the most promising solution.

To get face detection in OpenCV to work with C#, do the following:

1. Install OpenCV and OpenCVDotNet as per the instructions
2. Get CVHaar.h from this discussion and place it in C:\Program Files\OpenCVDotNet\src\OpenCVDotNet UPDATE: Dud link, read the updated guide
3. Open OpenCVDotNet.sln, add CVHaar.h to the solution and include it in OpenCVDotNet.cpp
4. Rebuild the solution
5. Create a Windows forms application as described in the tutorial, but do something like the code below

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using OpenCVDotNet;

namespace opencvtut
{
    public partial class Form1 : Form
    {
        private CVCapture cap;
        private CVHaar haar;

        public Form1()
        {
            InitializeComponent();
        }

        private void timer1_Tick(object sender, EventArgs e)
        {
            using (CVImage nextFrame = cap.QueryFrame())
            {
                Rectangle[] faces = haar.detectPatterns(1.3, nextFrame, 20, 20, 1.1, 2, 1);
                for (int i = 0; i < faces.Length; i++)
                {
                    nextFrame.DrawRectangle(faces[i], Color.Yellow, 3);
                }
                pictureBox1.Image = nextFrame.ToBitmap();
            }
        }

        private void Form1_Load(object sender, EventArgs e)
        {
            // passing 0 gets zeroth webcam
            cap = new CVCapture(0);
            haar = new CVHaar(
                "C:Program FilesOpenCVdatahaarcascadeshaarcascade_frontalface_alt2.xml");
        }
    }
}

Happy detecting 🙂