278 — Fast animal pose estimation using deep neural networks

New from the lab!

Fast and easy pose tracking and kinematics using deep networks, by @talmop @shaevitz @SamWangPhD https://t.co/KAWimUaW5U pic.twitter.com/blXiwL41S7

— Adam J Calhoun (@neuroecology) May 25, 2018

LEAP (LEAP Estimates Animal Poses) is a recursive acronym for a new technology that tracks animals’ gaits with very little training data (the paper cites instances in which only a few frames of training are often enough for robust pose estimation). The input is a 2D video stream and the output is real-space coordinates of user-specified body parts.

LEAP comprises of three steps:

Registration and alignment, in which the animal’s bounding box is identified and the video feed is redrawn cropped to this area and rotated to face the primary axis of the animal;

Labeling and training, in which a user uses a GUI in order to manually label the positions of body parts along a rigid skeleton;

Pose estimation, in which the neural network trained in the previous step is used to determine the configuration and orientation of that same skeleton in the frames of the 2D video.

This technology enables far higher fidelity in tracking specific behaviors over time in freely behaving animals. As an example use case, the authors used this technology to analyze the gait dynamics of Drosophila, even on complex backgrounds and with multiple animals in the field of view of the camera.