Our method is a fully-differentiable convolutional layerinspired by optical flow algorithms. Unlike traditional optical flow methods, all the parameters of our method can belearned end-to-end, maximizing action recognition performance. Furthermore, our layer is designed to compute the‘flow’ of any representation channels, instead of limiting itsinput to be traditional RGB frames.