ii. BEGINNING PEDESTRIAN DETECTION

Moving on to pedestrian detection, we realized that the Haar Cascade technique is quite the ancient algorithm (almost two decades old), so we moved onto more modern techniques: the HOG (Histogram of Oriented Gradients) detection technique, also implemented in OpenCV. We found that we were able to successfully detect pedestrians using these libraries, but again saw speed issues and accuracy problems. Ideas like running the program on a server, skipping frames between detections to approximately match the live speed were explored. This demo shows a 10 skip frame performance.

iv. INTRODUCING OBJECT TRACKING

Upon speaking to Austin Reiter, we learned the significance of the use of both detection and tracking in our project. Tracking is significantly faster due to the fact that there is context of where to look for the next object, while in detection, every frame is given new, hence the algorithm runs from scratch on every image slice. The key was to find a balance between the two. Also, by simply running detections per frame, we were not collecting any meaningful information across the frames – the detections were independent of each other, and we needed a way to recognize if a detection in one frame corresponded to a detection in the next frame to avoid double counting.

After researching many tracking techniques including TLD, BOOSTING, MIL, KCF, and MEDIANFLOW, we concluded that the KCF (Kernel Correlation Filter) tracking algorithm would work the best for us, with its speed and accuracy, and ability to notify tracking failures (which would hopefully represent a person exiting the screen).

We understood that the problem with object tracking alone was that the algorithms were not responsible for making any new detections. This meant that with the current code, only detection boxes in the first frame were able to be tracked. Hence introduced the idea of periodically running detections, that is, every nth frame, so that new pedestrians walking in would be able to be tracked as well. The gave rise to the same problem as before: how do we associate previous detections with new detections, ensuring that only new detections are added to the trackers, while old ones simply continue their tracking?

The first two demos show failed examples - in the first, we are using a simple centroid comparison technique, where if the centroids are close enough to each other, the objects are considered the same and does not create a new tracker, but it performs slowly and is not accurate enough. The second video shows an even worse case, where the trackers start accumulating and not disappearing. We also see the problem with hanging trackers. We were also having trouble with the objecct detection cycles, where they would run extremely slow.

Turns out, the Hungarian Algorithm is an association technique that involves matrix algebra in order to compare the centroids of detection boxes (or, ‘bounding boxes’) and find the highest likely pairings between the old and new detection boxes. Thus, using the Munkres package that implements this algorithm, we were able to eliminate many tracker duplications, rid of hanging boxes that resulted from a bad runaway tracker, and also successfully create new trackers for pedestrians that had newly entered the frame. Moreover, we fixed the detection bug in the code. The resulting significant improvement is visible in the following third demo.