A Practical Guide to Generative Music AI for Developers

• Case study of the foundational tools behind the rise of Generative AI in music.
• Exploration of the main themes of research and development in AI Music: Audio vs Symbolic, Language Models vs Diffusion, Composition vs Live Performance, General Public vs Artists
• Outline of course objectives, final project, and portfolio goals.

• “What do you expect to get out of this course? What professional outcomes are you aiming for?”

• Installing Python 3.12 and required libraries. Working with multiple environments on a single machine (pyenv, anaconda, venv).
• Cloning the class repository from GitHub

The GitHub repository can be accessed at lancelotblanchard/ai_music_course.

• Using librosa and numpy to load audio, visualize waveforms, and extract simple features.
• Using mido and fluidsynth to load and play MIDI data.

• Go over significant differences between audio and symbolic music.
• Cover the statistical basics of generative modeling in artificial intelligence.
• Understand how Autoencoders are used in musical representation and generation.
• Learn how to legally source and curate music datasets (copyright, licensing).
• Industry Relevance: Ensuring compliance in data collection. 

• Downloading a subset of Lakh MIDI dataset and Free Music Archive and analyzing examples.
• Using and training RAVE models for new music generation.

• Explore Human-Computer Interaction (HCI) principles behind the development of tools with user feedback integration.
• Analyze continuous deployment strategies to ensure seamless updates and integration with artists’ workflows.
• Case study of Jordan Rudess’s project with the MIT Media Lab: ‘Jordan and the jam_bot’.

• “What are the most exciting applications of Generative AI in music creation?”
• “Which challenges need addressing to optimize Generative AI’s role in music making?”
• “How can we foster effective collaborations between artists and engineers in music technology advancements?”

• Overview of the different representations used for music (e.g., tokenization: MusicGen for audio and Music Transformer for MIDI, spectrograms, chromagrams, latent representations).
• Study of the trade-offs of each representation and the best situations to use them.

• Computing spectrograms from audio files
• Creating sequences of tokens for MIDI files.
• Using HiFi-GAN to synthesize spectrograms.
• Using Encodec or DAC to compress and resynthesize audio.

• Overview of Autoregressive Distribution Estimation and the Attention mechanism.
• Walkthrough of the Transformer architecture.
• Case study of Anticipatory Music Transformer for Symbolic Music Generation.

• Run a Music Transformer to generate new MIDI data.

• Case study of the different sampling and token modeling methodologies (Residual Vector Quantization and codebook patterns.
• Case study of the MusicGen architecture.

• Using MusicGen to generate audio autoregressively.

• Brief overview of the theory behind Denoising Diffusion Probabilistic Models (stochastic differential equations, score-matching)
• Overview of the modalities of diffusion models (Text Conditioning, Classifier-Free Guidance, Inference-Time Optimization).

• Using stable-audio-tools to generate spectrograms.
• Experimenting with different initial noises, infilling strategies, conditioning methods.
• Using Latent Diffusion to generate new latents for our RAVE models from Session 3.

• Overview of the current landscape of commercial solutions making use of Generative AI for music and audio.
• Main differences between academia and industry in the world of AI and music.
• Tips on developing a career in Generative AI and music (AI engineering, data science, product design)

• Try out Suno’s commercial product for music generation.

• Setting up a project specification, timeline, and scope—skills you can directly apply in music tech roles.
• Group or individual project proposals aligned with personal career objectives.

• Students pitch final project plans, receive structured critique for improvement.
• Emphasis on real-world communication styles used in professional music and tech teams.

• Students focus on building out their final AI-driven music projects: from automated composition, stem separation and AI mixing to real-time applications.
• Instructor mentorship on optimization and code review—mimicking real-world R&D or dev team environments.

• Verification that each project meets requirements: code structure, documentation, and creative/music quality.

• Students present completed projects, highlighting technical and creative achievements.
• Demonstration of functionalities and professional-grade code base.

• Discussion of relevant job roles, interview readiness, and resources for continued learning.