How are they able to change the video frame to show the motion amplification. In the last example, they are amplifying baby's breathing and showing it using some changes into the video frame itself. How are they able to stretch baby's cloths and body without knowing its 3D geometry, even wrinkles on the baby's cloths seems to shift. I am puzzled.
Human perception of 3D geometry comes from motion. This is an algorithm to amplify motion. And that's really all that is needed. See "Structure From Motion" and "Optical Flow" for more information about this. Note that this is technology that is used in some video codecs. You may have noticed in some corrupted videos, the corruption artefacts can sometimes move as if they are mapped on a detailed 3d model of the moving objects. But really all it is are 2D motion vectors moving blocks of color around on a predetermined optical flow path.
In this algorithm, exaggerating that same 2D optical flow path has the perceptual effect of somehow "knowing" something about the 3d geometry. you could more accurately say the original motion is constrained by the 3D geometry.
(more)
here is a kanye west video that uses the optical flow/video corruption effect intentionally for artistic purposes.
Thanks for the detailed explanation. The kanye video just looked like a corrupted low res video, I didn't understand that one. But i get the idea, its about extrapolation of moving pixels in 2D.
