268 — Unsupervised Intuitive Physics from Visual Observations

This new unsupervised learning model enables an agent to observe many trials of gravity and physics acting on a system, and then finally predict where it anticipates the system will track next.

This paper also introduces a new dataset called Roll4Real, a video dataset of balls rolling on both simple as well as complex terrain.

Using the Roll4Real dataset, this deep learning system builds an ‘intuition’ of real-world physics without an explicit simulator.

This is interesting for a whole variety of reasons, but I can see many really fascinating real-world applications: Perhaps we know what a pool table heighmap looks like… but we certainly don’t know what the surface of another planet looks like, and if we want to remotely navigate a rover across the surface of that planet, we need to have a top-view prediction of how the rover will respond. And due to undetectable winds or differing gravity, it’s possible that simple physics simulators won’t be adequate to model this scenario.

Separately, this type of technology could be immensely useful for ‘learning’ more complex worlds, like the patterns of quantum physics or other less obvious (?) fields. I would be particularly interested to see some sort of human-readable output that compares our human-derived models, like $g=9.8\frac{m}{s^2}$, to the output of the model’s predicted understanding of gravity, and see how they compare.