In this work, we investigate the personalization of text-to-music diffusion models in a few-shot setting. Motivated by recent advances in the computer vision domain, we are the first to explore the combination of pre-trained text-to-audio diffusers with two established personalization methods. We experiment with the effect of audio-specific data augmentation on the overall system performance and assess different training strategies. For evaluation, we construct a novel dataset with prompts and music clips. We consider both embedding-based and music-specific metrics for quantitative evaluation, as well as a user study for qualitative evaluation. Our analysis shows that similarity metrics are in accordance with user preferences and that current personalization approaches tend to learn rhythmic music constructs more easily than melody. The code, dataset, and example material of this study are open to the research community.

We start from a pre-trained text-to-audio diffusion model, i.e. AudioLDM, and to the best of our knowledge are the first to investigate the ability to personalize its outputs for newly learned musical concepts in a few-shot manner. Motivated by the computer vision literature, we explore the application of two established methods, i.e. Textual Inversion and Dreambooth. We adapt these methods for music personalization by employing a set of audio-specific data augmentation methods. We evaluate the capacity of the model to learn new concepts, along two dimensions, reconstruction, i.e. the ability to faithfully reconstruct input examples, and editability, i.e. the ability to manipulate the generation through textual prompts. To this end, we construct a new dataset of various instruments and playing styles. Our evaluation protocol consists of a) embedding distance-based metrics, b) music-specific metrics, and c) an A/B testing user study comparing the two adaptation approaches. Finally, we adapt AudioLDM to perform text-guided style transfer for newly learned concepts. Our key contributions are a) the personalization of AudioLDM's generation and style-transfer abilities for new concepts, b) the exploration of audio-specific augmentations and evaluation metrics, and c) the construction of a new dataset for text-to-music personalization methods.