JoruriPuppet: Learning Tempo-Changing Mechanisms Beyond the Beat for Music-to-Motion Generation

Ran Dong1, Shaowen Ni2, Xi Yang3
1 Chukyo University, Japan · 2 Mie University, Japan · 3 Jilin University, China
📄 Paper 📎 Appendix 💻 Code (GitHub) 🗂 Dataset 📚 BibTeX 🎬 Video
SIGGRAPH Asia 2025 Technical Papers
Teaser — Music-to-motion with Jo–Ha–Kyu tempo-changing characteristics

Abstract

We study how tempo-changing mechanisms shape expressive music-to-motion generation. JoruriPuppet is a dataset of Japanese Ningyo Joruri puppet performances paired with music, enabling models to learn tempo-aware behavior. We introduce two tempo features (instantaneous BPM and its change) and three metrics for Jo–Ha–Kyu correlation, S-curve aesthetics, and head–hand theatrical contrast, revealing limitations of beat-only evaluations and improving state-of-the-art methods.

Highlights

  • Tempo-aware inputs: instantaneous BPM and ΔBPM derived from beat intervals.
  • New dataset: aligned puppet motions in SMPL with Japanese tempo-changing mechanism.
  • New metrics: Jo–Ha–Kyu score (tempo-change synchronization), S-curve motion aesthetics, head–hand contrast.
  • Generalization: adding tempo features boosts multiple SOTA models across datasets.

Demo Video

Data Download

Subject Motion/BVH Motion/SMPL-BVH Motion/SMPL‑NPZ Motion/SMPL-TRC Audio/WAV
Shamisen BVH BVH NPZ TRC WAV
Without Shamisen BVH BVH NPZ TRC WAV

Details and file structure are described in the Appendix. See Appendix.

BibTeX

@inproceedings{dong2025joruripuppet,
  title={JoruriPuppet: Learning Tempo-Changing Mechanisms Beyond the Beat for Music-to-Motion Generation with Expressive Metrics},
  author={Dong, Ran and Ni, Shaowen and Yang, Xi},
  booktitle={Proceedings of the SIGGRAPH Asia 2025 Conference Papers},
  pages={1--11},
  year={2025}
}