JoruriPuppet: Learning Tempo-Changing Mechanisms Beyond the Beat for Music-to-Motion Generation

Ran Dong1, Shaowen Ni2, Xi Yang3
1 Chukyo University, Japan · 2 Mie University, Japan · 3 Jilin University, China
📄 Paper (PDF) 📎 Appendix 💻 Code (GitHub) 🗂 Dataset 📚 BibTeX 🎬 Video
SIGGRAPH Asia 2025 Technical Papers (Coming soon)
Teaser — Music-to-motion with Jo–Ha–Kyu tempo-changing characteristics

Abstract

We study how tempo-changing mechanisms shape expressive music-to-motion generation. JoruriPuppet is a dataset of Japanese Ningyo Joruri puppet performances paired with music, enabling models to learn tempo-aware behavior. We introduce two tempo features (instantaneous BPM and its change) and three metrics for Jo–Ha–Kyu correlation, S-curve aesthetics, and head–hand theatrical contrast, revealing limitations of beat-only evaluations and improving state-of-the-art methods.

Highlights

  • Tempo-aware inputs: instantaneous BPM and ΔBPM derived from beat intervals.
  • New dataset: aligned puppet motions in SMPL with Japanese tempo-changing mechanism.
  • New metrics: Jo–Ha–Kyu score (tempo-change synchronization), S-curve motion aesthetics, head–hand contrast.
  • Generalization: adding tempo features boosts multiple SOTA models across datasets.

Demo Video

Data Download

Subject Motion/BVH Motion/BVH‑SMPL Motion/SMPL‑NPZ Motion/TRC Audio/WAV
Shamisen BVH BVH NPZ TRC WAV
Without Shamisen BVH BVH NPZ TRC WAV

Details and file structure are described in the Appendix. See Appendix (PDF).

BibTeX

@inproceedings{dong2025joruripuppet,
  title     = {JoruriPuppet: Learning Tempo-Changing Mechanisms Beyond the Beat for Music-to-Motion Generation},
  author    = {Dong, Ran and Ni, Shaowen and Yang, Xi},
  title = {Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers},
  pages     = {1--11},
  year      = {2025},
  doi       = {10.1145/3757377.3764006}
}