Marcus Loo Vergara
All projects

Machine Learning ยท 2018

Keypoint R-CNN

Mask R-CNN extended with a keypoint head to estimate human poses, trained on MS COCO.

Source on GitHub
Diagram of the Keypoint R-CNN architecture with class, mask and keypoint heads
Ground truth skeletons drawn on snowboarders in a COCO image
Grid of predicted human keypoints on a variety of COCO images
A baseball pitcher next to the predicted heatmap for each of the 17 joints
Layer diagrams of the box, mask and keypoint prediction heads

1 / 5

As part of CSE 252C: Selected Topics in Vision and Learning at UCSD, I extended the Mask R-CNN model to detect human keypoints in images of people.

A keypoint prediction head is added in parallel to the bounding box, classification and mask heads. It predicts 17 probability masks, one for each joint, giving the probability of finding that joint at each location in the image. The model was trained on the MS COCO keypoint dataset.