Schematic overview over the proposed method. (1) A variant of de-noising autoencoders learns the spatial configuration of the human skeleton via training with dropouts, removing entire joints at random which have to be reconstructed by the network. (2) We train a 3-layer LSTM recurrent neural network to predict skeletal configurations over time. (3) At inference time both components are stacked and the dropout autoencoder filters the noisy predictions of the LSTM layers, preventing accumulation of error and hence pose drift over time. </p>


We propose a new architecture for the learning of predictive spatio-temporal motion models from data alone. Our approach, dubbed the Dropout Autoencoder LSTM (DAE-LSTM), is capable of synthesizing natural looking motion sequences over long-time horizons1 without catastrophic drift or motion degradation. The model consists of two components, a 3-layer recurrent neural network to model temporal aspects and a novel autoencoder that is trained to implicitly recover the spatial structure of the human skeleton via randomly removing information about joints during training. This Dropout Autoencoder (DAE) is then used to filter each predicted pose by a 3-layer LSTM network, reducing accumulation of correlated error and hence drift over time. Furthermore to alleviate insufficiency of commonly used quality metric, we propose a new evaluation protocol using action classifiers to assess the quality of synthetic motion sequences. The proposed protocol can be used to assess quality of generated sequences of arbitrary length. Finally, we evaluate our proposed method on two of the largest motion-capture datasets available and show that our model outperforms the state-of-the-art techniques on a variety of actions, including cyclic and acyclic motion, and that it can produce natural looking sequences over longer time horizons than previous methods.

1 > 10s for periodic motions, e.g. walking, > 2s for aperiodic motion, e.g. eating


Published at

2017 International Conference on 3D Vision (3DV), 2017

Best Paper Award

Project Links


@inproceedings{ghosh2017learning, title={Learning human motion models for long-term predictions}, author={Ghosh, Partha and Song, Jie and Aksan, Emre and Hilliges, Otmar}, booktitle={2017 International Conference on 3D Vision (3DV)}, pages={458--466}, year={2017}, organization={IEEE} }