Diff4Steer: Steerable Diffusion Prior for Generative Music Retrieval with Semantic Guidance

Overview

Diff4Steer: Generative Music Retrieval via Steerable Diffusion Prior

Modern music retrieval systems often rely on fixed representations of user preferences, limiting their ability to capture users' diverse and uncertain retrieval needs. To address this limitation, we introduce Diff4Steer, a novel generative retrieval framework that employs lightweight diffusion models to synthesize diverse "oracle" seed embeddings representing potential directions for music exploration. Unlike deterministic methods that map user query to a single point in embedding space, Diff4Steer provides a statistical prior on the target modality (audio) for retrieval, effectively capturing the uncertainty and multi-faceted nature of user preferences. Furthermore, Diff4Steer can be steered by image or text inputs, enabling more flexible and controllable music discovery combined with nearest neighbor search. Our framework outperforms deterministic regression methods and LLM-based generative retrieval baseline in terms of retrieval and ranking metrics, demonstrating its effectiveness in capturing user preferences, leading to more diverse and relevant recommendations.

Image to Music: MelBench Examples

Diff4Steer can find matching music given an image input. For sample images from MelBench, we show that we can generate diverse seed embeddings, which is a vector representation of idealized user preference in the music embedding space. We invert the embedding to music using a embedding-based music generation model, so you can "hear" the embedding. It showcase that Diff4Steer can retrieve diverse music and music genres that match the image.

Index	Input Image	Generated Audio	Retrieved Audio	Groundtruth Genres
1		seed 0 seed 1	Classical, Compositional Ambient Electronic, Downtempo Electronic, Electronica	New Age, Color Noise
2		seed 0 seed 1	Electronic, Electro House Electronic, Stutter House Electronic, Brostep	Electronic, Uk dance
3		seed 0 seed 1	Classical, Orchestral Performance Classical, Italian orchestra World traditional, Entehno	Classical, Classical Performance
4		seed 0 seed 1	Rock, Classic Rock Classical, Neo-Classical Latin, Rock En Espanol	Rock, Country, Blues Rock, Metal, Country

Image + Text to Music

Diff4Steer allows combining the infulence from image and text, with customizable strength. Here we showcase the generated seed embedding with image style influence can be combined with the music style specified by text, to represent user's preference on combining artistic styles from different modalities.

Index	Input Image Query	Steering Text Prompt	No Steering	Steering at 0.04	Steering at 0.08
1		Chill-out	Environmental, Indie Folk, Ocean	Downtempo, Ocean, Indie Folk	Downtempo, Indietronica, Compositional Ambient
1		Chill-out	Environmental, Movie Tunes, Filmi	Environmental,Sleep,Healing Hz	Sleep, Downtempo, Environmental