Visualization tools for learning to play a musical instrument

Learning to play an instrument requires dedication, hard work, and plenty of independent practice. Not everyone has access to a teacher, and even with one, a musician cannot expect to receive constant and appropriate feedback when practicing their instrument. To address this problem, this research project developed a collection of data-driven techniques and tools that use data taken from a musician's practice session to provide visual feedback. The resulting Visual Analytics web-tool enables musicians to detect errors and improve their style.

The visualization tools& helped make practicing more efficient. The project designed two methods that automatically assign colors to parts of the sheet music to make its structure easier to see, so that it is easier for the musician to see repeats and changes. Moreover, the visualization tools can show in real time how consistently the musician is playing, for example, a specific rhythm over multiple repetitions.

The research group also wanted to produce a practice support tool that did not take time away from actually practicing. Consequently, they designed a way to control the software running on a computer or similar device directly through the instrument. Using the electric guitar as a preliminary case, the group developed an on-screen menu that follows the grid-like guitar fretboard, so that the musician can use the guitar as a controller. This set-up did not require any modification of the guitar or additional hardware.

With some refinement and user-experience engineering, the tools developed by this project could be turned into smartphone or web apps that support musicians when practicing independently. The researchers do not conceive of the visualization tools developed by this project as replacing traditional music education, but rather as a support for musicians when practicing independently. In this respect, this project aligns with Cyber Valley's mission to translate cutting-edge research into real-world applications for a better future.

This project produced the following peer-reviewed publications:  

• Simeon Rau, Frank Heyen, Stefan Wagner, & Michael Sedlmair (2022). Visualization for AI-Assisted Composing. In Proceedings of the 23rd International Society for Music Information Retrieval Conference (ISMIR). https://doi.org/10.5281/zenodo.7316618 (open access) 

• Lucchas R Skreinig, Denis Kalkofen, Ana Stanescu, Peter Mohr, Frank Heyen, Shohei Mori, Michael Sedlmair, Dieter Schmalstieg, & Alexander Plopski (2023). guitARhero: Interactive Augmented Reality Guitar Tutorials. In IEEE Transactions on Visualization and Computer Graphics (TVCG). https://doi.org/10.1109/TVCG.2023.3320266 (open access) 

• Frank Heyen, Quynh Quang Ngo, & Michael Sedlmair (2023). Visual Overviews for Sheet Music Structure. In the 24th International Society for Music Information Retrieval Conference (ISMIR). https://doi.org/10.5281/zenodo.10265383 (open access) 

• Simeon Rau, Frank Heyen, Benedikt Brachtel & Michael Sedlmair (2025). MAICO: A Visualization Design Study on AI-Assisted Music Composition. In IEEE Transactions on Visualization and Computer Graphics (TVCG). https://doi.org/10.1109/TVCG.2025.3539779 

• Frank Heyen, Marius Labudda, Michael Sedlmair, & Andreas Fender (2025). GuitarPie: Using the Fretboard of an Electric Guitar for Audio-Based Pie Menu Interaction. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST). https://doi.org/10.1145/3746059.3747799 (open access) 

• Frank Heyen, Michael Gleicher, Michael Sedlmair (2026). Make the Unhearable Visible: Exploring Visualization for Musical Instrument Practice. In IEEE Transactions on Visualization and Computer Graphics (TVCG). https://doi.org/10.1109/TVCG.2026.3658216 

About the Cyber Valley Research Fund 

The Cyber Valley Research Fund was established to support Cyber Valley research groups undertake basic research in the fields of artificial intelligence and robotics. The fund totaled five million euros, including contributions from six of Cyber Valley’s founding corporate partners: Amazon, BMW, Bosch, IAV, Mercedes-Benz, Porsche, and ZF. It supported 20 research projects, the first of which began in 2020, and the final of which will conclude in 2026.