A Practical Guide to Generative Music AI for Developers hero

Creative Coding

A Practical Guide to Generative Music AI for Developers

A hands-on, code-first journey through modern music AI — from audio features and VAEs to transformers and diffusion models — building working generative systems in Python. Includes guest sessions with Jordan Rudess and Christian Steinmetz.

Level

Advanced

Duration

36h 9m of video content

Format

Self-paced video

Watch a preview

An Introduction to AI Music (50')

Course overview

Over 12 sessions this course takes developers from the fundamentals of machine learning for audio through to the architectures powering today's generative music tools. You'll set up a Python environment, work with MIDI and spectrogram representations, and get hands-on with RAVE, EnCodec, Anticipatory Music Transformers, MusicGen and Stable Audio. Later sessions cover real-time inference (TorchScript, quantization, ONNX) and the commercial landscape, before you scope, build and present your own final project. Designed for developers comfortable with Python who want to understand and build music-AI systems rather than just use them.

Course content

Session 1: An Introduction to AI Music

2 videos, 1 resource, 2 lessons

+
  • Session 1 Recording
    Checking access...
  • Main Topics: AI Music Case Studies, Course Roadmap
  • An Introduction to AI Music (50')
    Checking access...
  • Session 1 PDF handout
  • Open Discussion

Session 2: Setting up your environment

6 videos, 3 lessons

+
  • Session 2 Recording
    Checking access...
  • Main Topics: Environment Setup
  • GitHub Repository
  • Setting up your environment - Cloning the class repository (10')
    Checking access...
  • Hands On
  • Setting up your environment - Hands On 1.1: Loading, visualizing, playing audio (10')
    Checking access...
  • Setting up your environment - Hands On 1.2: Extracting Audio Features, RMS and ZCR (13')
    Checking access...
  • Setting up your environment - Hands On 1.2: Extracting Audio Features, Spectrograms (13')
    Checking access...
  • Setting up your environment - Hands On 2: Manipulating MIDI Data (20')
    Checking access...

Session 3: Core Machine Learning Concepts for Music and Audio

6 videos, 2 lessons

+
  • Session 3 Recording
    Checking access...
  • Main Topics: Audio vs Symbolic Music, Basics of Generative AI, Data Acquisition and Ethics
  • Statistical Basics of Generative Modeling in Artificial Intelligence (10')
    Checking access...
  • Variational Autoencoders (16')
    Checking access...
  • Hands On
  • Hands On 1.1: Lakh MIDI Dataset (12')
    Checking access...
  • Hands On 1.2: Free Music Archive (6')
    Checking access...
  • Hands On 2: Using RAVE
    Checking access...

Session 4: Real-Life Collaborations between Artists and Engineers with Guest Speaker Jordan Rudess

2 videos, 2 lessons

+
  • Session 4 Recording
    Checking access...
  • Main Topics: Human-Computer Interaction, Iterative Design, Continuous Deployment
  • Human-Computer Interaction & User-Centered Design
    Checking access...
  • Open Discussion with Jordan Rudess

Session 5: Representation Learning for Music

4 videos, 2 lessons

+
  • Session 5 Recording
    Checking access...
  • Deep Dive into MIDI & Spectrograms
  • Comparing Musical Representations & Encodec Deep Dive (14')
    Checking access...
  • Understanding RVQ in Encodec (12')
    Checking access...
  • Hands On
  • Hands On: Encodec (29')
    Checking access...

Session 6: Autoregressive Music Generation

5 videos, 2 lessons

+
  • Session 6 Recording
    Checking access...
  • Main Topics: Autoregressive modeling, the Transformer architecture, HuggingFace Hub
  • The Transformer architecture (15')
    Checking access...
  • Understanding Anticipatory Music Transformers (13')
    Checking access...
  • Hands On
  • Hands On: Using AMT to generate MIDI data (Part 1) (18')
    Checking access...
  • Hands On: Using AMT to generate MIDI data (Part 2) (17')
    Checking access...

Session 7: Autoregressive Music Generation (Part 2)

3 videos, 2 lessons

+
  • Session 7 Recording
    Checking access...
  • Main Topics: MusicGen & Audio Generation with Transformers
  • Understanding MusicGen (8')
    Checking access...
  • Hands On
  • Hands On: Using MusicGen to generate audio (38')
    Checking access...

Session 8: Diffusion Models for Music Generation

8 videos, 2 lessons

+
  • Session 8 Recording
    Checking access...
  • Main Topics: Diffusion Models, Latent Diffusion Models
  • Intro to Diffusion Models Part 1 (11')
    Checking access...
  • Intro to Diffusion Models Part 2 (14')
    Checking access...
  • Conditioning & Classifier-Free Guidance (10')
    Checking access...
  • The UNet Architecture (6')
    Checking access...
  • Inference-Time Optimization: DITTO (6')
    Checking access...
  • Hands On
  • Hands On: Using Stable Audio Part 1 (15')
    Checking access...
  • Hands On: Using Stable Audio Part 2 (18')
    Checking access...

Session 9: Real-Time Generative AI & Commercial Applications of Generative AI in Music [Guest: Christian Steinmetz]

4 videos, 2 lessons

+
  • Session 9 Recording
    Checking access...
  • Introduction to TorchScript (18')
    Checking access...
  • 8-bit Linear Quantization (9')
    Checking access...
  • ONNX and Graph Optimizations (15')
    Checking access...
  • Main Topics: Landscape of companies in AI and Music, Available Commercial Products
  • Demo

Session 10: Final Project Planning

6 videos, 2 lessons

+
  • Session 10 Recording 1
    Checking access...
  • Session 10 Recording 2
    Checking access...
  • Session 10 Recording 3
    Checking access...
  • Main Topics: Setting up a project specification, timeline, and scope
  • Training Part 1 (8')
    Checking access...
  • Training Part 2 (13')
    Checking access...
  • Training Part 3 (16')
    Checking access...
  • Peer Review & Feedback

Session 11: Final Project Lab

1 video, 2 lessons

+
  • Session 11
    Checking access...
  • Lab Session: Guided Coding & Troubleshooting
  • Milestone Check-Ins

Session 12: Project Showcase & Next Steps

1 video, 2 lessons

+
  • Session 12 Recording
    Checking access...
  • Final Presentations
  • Next Steps

Instructors

Lancelot Blanchard

Lancelot Blanchard

Instructor

Lancelot Blanchard is a musician, engineer, and AI researcher at the MIT Media Lab's Responsive Environments group. His research focuses on generative AI systems for human-AI co-created live musical performances, pioneering the concept of Symbiotic Virtuosity in AI-human music collaboration. He has collaborated with Grammy-winning keyboardist Jordan Rudess and brings eight years of software engineering experience to his teaching.

Ready to join?

One-off enrolment. Includes all course materials.