research philosophy

The current landscape

AI-generated music is broadly viewed by the public critically or straight up negatively, and the reasons are completely understandable. Humans want to listen to music written and performed by other humans, not algorithms. The music industry was extremely competitive already before the rise of generative AI, so we must be careful not to make artists’ lives even harder by crowding them out of this already oversaturated market. AI should primarily serve as a step in the toolchain that makes it easier for humans to learn and create music, rather than an end-to-end system that writes audio based on a simple text or audio prompt. There are many applications in Music Information Retrieval (MIR) that are genuinely useful, such as Neural Amp Modeling, AI stem separation, audio super-resolution, or vector-based similarity search. However, for me personally, the current state of offline generative systems remains more gimmicky than useful.

Recently, there have been a few innovative generative multimodal systems released, like Google Lyria and Magenta RealTime 2. The latter even allows for MIDI input in addition to text and audio input to steer the generative process. In particular, Magenta RealTime 2 is a major upgrade because it brings down control latency to just about 200 ms and can run on a consumer laptop. In my opinion, while these systems are a step in the right direction regarding user agency, they still function more like DJing tools. The next critical step for the future is to achieve even more fine-grained control over what is generated, alongside explanations for why specific choices are made.

A multimodal piano tutor

With this in mind, I envision two distinct research goals. The educational goal is a multimodal piano tutor system, equivalent to what chatbots can do in the text domain. The user could sit at the piano and communicate with the agent by talking or playing the keyboard. The agent could respond by voice, text, images, or by demonstrating how to play a passage directly on the keys. The difference from today’s state-of-the-art chatbots would be a MIDI integration. However, teaching the model how to truly understand MIDI, and thus piano music, is still an unsolved problem, mainly caused by data scarcity. There are simply orders of magnitude less MIDI data on the internet than text data. A recent work, jam_bot, proposes a human-in-the-loop approach where an expert musician fine-tunes a pre-trained model on their own playing, thus teaching the model their own style of playing. Yet, it remains unclear if this human-in-the-loop paradigm can be scaled up to a community level to gather sufficient data, or what alternative methods could solve this data-scarcity bottleneck. I believe that such a tutoring system could democratize instrument learning, as relatively few people can afford private lessons. Crucially, the goal is not to replace piano teachers, but to offer a viable alternative to those who cannot afford lessons, or to help students practice effectively between sessions with a human instructor. The system would help students overcome learning plateaus and practice material tailored to their current skill level. Based on discussions with my peers and hobbyist musicians, many people deeply desire to express their musical thoughts, but their journey toward proficiency is often hindered by factors like developing learned helplessness from negative experiences with a human tutor, or an inability to find practice material that suits both their taste and playing level.

A real-time piano cover model

The other system I envision is a real-time piano cover generator supporting various musical styles. The motivation for this stems from my personal fascination with Brad Mehldau’s covers of Radiohead, The Beatles, or popular songs like ‘Bitter Sweet Symphony’. It is fascinating how a song with vocals and multiple instruments can be translated to just one instrument while preserving the core characteristics of the original. The piano is physically constrained in many ways; for instance, there are no open strings, and there is no way to play a note between two semitones, whereas a vocalist can sing microtones. Yet, Brad Mehldau demonstrated that it is possible to create the illusion of guitar textures, or to imitate microtonality and blue notes on the piano through specialized techniques. I think that every now and then, a musician comes along who changes the way an instrument can sound, creating new genres and blending musical styles together. I believe that in the space where all possible music resides, there are still countless undiscovered pieces that humanity has yet to draw out. Pursuing this endeavour from the algorithmic side might be a promising and innovative research direction.