Interactive Sound and Visuals: A Tone.js and p5.js Beginner's Guide

Welcome to "Interactive Sound and Visuals: A Tone.js and p5.js Beginner's Guide"! In this course, we'll explore the exciting world of audio-visual programming by learning how to create interactive soundscapes and visuals using two powerful JavaScript libraries: Tone.js and p5.js.

Tone.js is a powerful audio library that makes it easy to create and manipulate sounds in your web applications. It provides a wide range of tools for synthesizing, sequencing, and controlling audio in the browser.

p5.js, on the other hand, is a versatile library for creating graphics and interactive experiences in the browser. It simplifies the process of working with the HTML5 canvas and provides a range of functions to create shapes, colors, and animations.

By the end of this course, you will be able to:

1. Understand the basics of Tone.js and p5.js libraries and their respective functionalities.
2. Create and control audio using Tone.js, including synthesizers and effects
3. Design and animate visuals using p5.js, including shapes, colours, and transformations.
4. Combine both libraries to create interactive audio-visual projects that respond to user input.
5. Implement user interface controls to modify audio and visual parameters in real-time.
6. Embed Tone.js and p5.js with HTML and CSSS into any webpage

Here's the final result of what we'll look at in this course. We will build this application step by step, and you will be able to modify it, and copy and paste the content into your own project. 

Throughout this course, we'll be using CodePen as our primary platform for creating and experimenting with the code for our audio-visual projects. CodePen is a popular online code editor and playground that enables you to write, test, and share HTML, CSS, and JavaScript code snippets easily, without the need to set up a local development environment. It's an excellent tool for learning, experimenting, and collaborating, particularly when working with libraries like Tone.js and p5.js.

Using CodePen has several advantages for this course:

1. Ease of use: CodePen provides an intuitive interface with separate panels for HTML, CSS, and JavaScript, making it simple to create and modify code in real-time. There's no need to worry about setting up a development environment, installing libraries, or configuring build processes.

2. Real-time feedback: As you write and edit code in CodePen, you can immediately see the results in the output panel. This instant feedback makes it easier to understand how your code is working and to quickly identify and fix any issues.

3. Library integration: CodePen makes it easy to include external libraries like Tone.js and p5.js in your projects. You can add libraries directly from the Pen Settings, and CodePen will automatically load them for you.

4. Sharing and collaboration: CodePen allows you to save and share your projects (called "Pens") with others through unique URLs. You can also fork and modify existing Pens to create new projects, making it an excellent platform for collaboration and learning from others.

Note that while CodePen is a great tool for learning and experimenting, it is not the best choice for building and deploying full-scale web applications. When working on a complete website, you'll typically want more control over your development environment, build processes, and hosting. However, for the purposes of this course, CodePen provides an accessible and powerful platform to learn and explore Tone.js and p5.js.

In the upcoming lessons, we'll share with you ready-to-go CodePen projects that break down and interactive audio-visual project step by step. Let's get started!

In this first step, we will create a simple sketch using p5.js to draw circles on a canvas. When the user clicks on the canvas, a new circle will be added to the display. We will also introduce comments within the code to help explain its functionality.

To edit the code, change the background colour or the size of the circles, click the "Edit on CodePen" button below. 

In this course, we will focus on the JavaScript code for your p5.js sketch, but you will also have HTML and CSS files to structure and style your project. Here's a brief overview of what to expect in these files:

HTML

The HTML file will include the basic structure for your project. It will load the necessary p5.js and Tone.js libraries, as well as your JavaScript file containing the sketch. 

CSS

The CSS file will contain the styling rules for your project. In this example, we will have a simple CSS file that sets the font family, centers the content, and adds some basic styling to the text. We've kept it fairly basic, but feel free to modify it!

Step 1: Drawing Circles with p5.js

In this step, we will learn the basics of using p5.js to create an interactive sketch. We will be able to draw circles on a canvas by clicking on it. Here's the code you need to write, with comments to help you understand what each part does:

In this step, we will integrate Tone.js into our project to add sound to our circles. We will create a simple synthesizer that plays a note every time a circle is drawn on the canvas. Let's go through the updated code with comments to explain the new additions.

In this stage, we have made a few changes to our original code:

1. We declared a new variable called synth to hold our Tone.js Synth object.
2. Inside the setup() function, we initialized the synth and connected it to the audio output using the toDestination() method.
3. We changed the fill() color to red (255, 0, 0) to make our circles more visually appealing.
4. Finally, we added a line in the mousePressed() function to play a "C4" note with an 8th note duration every time a circle is drawn.

Now, every time you draw a circle on the canvas, you will also hear a sound played by the synthesizer.

In this step, we will enhance our sound with audio effects using Tone.js. We will add reverb and delay to the sound produced by the synthesizer, making it more interesting. Let's review the updated code with comments explaining the new additions. 

Open the project on CodePen and change the delay time and the feedback % to see how it impact the delay. 

In this stage, we have made several changes to our previous code:

1. We declared two new variables called reverb and delay to hold our Tone.js Reverb and FeedbackDelay objects.
2. Inside the setup() function, we initialized the reverb and delay effects, and connected them to the audio output using the toDestination() method. We also connected the synth to both effects.
3. We modified the draw() function to use each circle's colour property when setting the fill() colour.
4. In the mousePressed() function, we generated a random shade of red for each new circle and stored it in the circle object.

With these changes, the sound produced by the synthesizer will now have added reverb and delay effects, and each circle will have a unique shade of red.

In this step, we'll add motion to our circles: they will start to move in a random direction as soon as the user clicks on the canvas. They will then bounce of the boundaries of the canvas, and bounce off of each other when they collide. 

To do this, we will  implement collision detection for both the canvas boundaries and other circles. Upon collisions, we'll play different notes to create a dynamic soundscape. Here's the updated code with comments explaining the additions.

Edit the code on CodePen to change the notes series (marked "C2", "G4" etc.), change the velocity of the circles and see if you can fix some issues! They will be solved in the next step. 

Let's break down the changes in Step 4 in more detail:

1. We added a new array of notes called `notes`. These notes will be assigned to circles when they are created and will be played when circles collide with walls or other circles.

2. We introduced two new variables: `circleSize` (to store the size of the circles) and `canvasScaleFactor` (to control the size of the canvas relative to the window dimensions).

3. In the `setup()` function, we now call the `createCircle()` function when the canvas is clicked. This separates the circle creation logic from the `mousePressed()` event.

4. We updated the `draw()` function to handle circle motion, collisions with walls, and collisions between circles. To achieve this:

- We added code to update the circle's x and y positions based on their velocities.
- We implemented collision detection with walls by checking if the circle's position goes beyond the canvas boundaries. If so, we reverse the circle's velocity and play the circle's note.
- We added a nested loop to check for collisions between circles. For each pair of circles, we call the `checkCollision()` function.

5. We created a new `createCircle()` function to handle the creation of circles when the canvas is clicked. In this function:

- We generate a random angle and speed.
- We calculate the circle's x and y velocities based on the angle and speed.
- We generate a random shade of red for the circle's color.
- We assign a note from the `notes` array based on the current number of circles.
- We create a new circle object with the properties `x`, `y`, `vx`, `vy`, `cColor`, and `note`, and then add it to the `circles` array.

6. We updated the `checkCollision()` function to handle circle-to-circle collisions. When two circles collide:

- We calculate the angle between the centers of the circles.
- We transform the positions and velocities of the circles into a rotated coordinate system where the x-axis aligns with the line connecting their centers.
- We swap the x-velocities of the circles to simulate the collision.
- We transform the positions and velocities back into the original coordinate system.
- We update the positions of the circles after the collision.
- We play a sound (the note "C5") when the circles collide.

These changes add motion to the circles and create a more dynamic, interactive experience for users, combining both visual and audio elements.
1. We added a new array of notes called `notes`. These notes will be assigned to circles when they are created and will be played when circles collide with walls or other circles.

2. We introduced two new variables: `circleSize` (to store the size of the circles) and `canvasScaleFactor` (to control the size of the canvas relative to the window dimensions).

3. In the `setup()` function, we now call the `createCircle()` function when the canvas is clicked. This separates the circle creation logic from the `mousePressed()` event.

4. We updated the `draw()` function to handle circle motion, collisions with walls, and collisions between circles. To achieve this:

- We added code to update the circle's x and y positions based on their velocities.
- We implemented collision detection with walls by checking if the circle's position goes beyond the canvas boundaries. If so, we reverse the circle's velocity and play the circle's note.
- We added a nested loop to check for collisions between circles. For each pair of circles, we call the `checkCollision()` function.

5. We created a new `createCircle()` function to handle the creation of circles when the canvas is clicked. In this function:

- We generate a random angle and speed.
- We calculate the circle's x and y velocities based on the angle and speed.
- We generate a random shade of red for the circle's color.
- We assign a note from the `notes` array based on the current number of circles.
- We create a new circle object with the properties `x`, `y`, `vx`, `vy`, `cColor`, and `note`, and then add it to the `circles` array.

6. We updated the `checkCollision()` function to handle circle-to-circle collisions. When two circles collide:

- We calculate the angle between the centers of the circles.
- We transform the positions and velocities of the circles into a rotated coordinate system where the x-axis aligns with the line connecting their centers.
- We swap the x-velocities of the circles to simulate the collision.
- We transform the positions and velocities back into the original coordinate system.
- We update the positions of the circles after the collision.
- We play a sound (the note "C5") when the circles collide.

These changes add motion to the circles and create a more dynamic, interactive experience for users, combining both visual and audio elements.

In Step 5, we will introduce some additional features to make the interactions more dynamic, and add controls to . Specifically, we'll add different sizes to the circles, a new way of triggering sounds, and a text indicator to show which note is currently playing.

In this step, we're going to enhance our interactive project by adding more variation to the circles, such as different sizes and improve our sound triggering mechanism. We will also add a text indicator to show which note is currently playing when a circle collides with a wall or another circle.

Now, let's look at the changes in the JavaScript code. Open the project in CodePen to modify the controls 

In step 5, a few modifications were made to the code. Here are the main changes and the relevant coding concepts:

1. Added a new variable called playSound and initialized it with true. This variable will control whether the sound should play or not.

let playSound = true;

2. Modified the draw() function to update the animation only if the playSound variable is true. This is done by wrapping the code inside the draw() function with an if (playSound) conditional block.

function draw() {
  if (playSound) {
    // ... original code ...
  }
}

3. Added an event listener for the pause button in the sketch.js file and created a function called togglePlaySound() to update the playSound variable when the button is clicked.

document.getElementById("pauseButton").addEventListener("click", togglePlaySound);

function togglePlaySound() {
  playSound = !playSound;
  document.getElementById("pauseButton").textContent = playSound ? "Pause" : "Play";
}

4. Updated the HTML code to include the pause button:

 
	<button id="pauseButton">Pause</button>

This button is added to the HTML code and is used to control pausing and resuming the sketch.

These changes allow you to control the sketch's sound and display updates using the pause button, which can be useful for user interaction and to save resources when the sketch is not in use.

5. Added random size for the circle in the createCircle() function:

// Add a random size for the circle
let size = random(15, 50);

This line of code generates a random size for each circle within a specified range (15 to 50). This creates more variety in the visual representation and sound of the sketch.

6. Calculated the duration based on the circle's size:

// Calculate the duration based on the circle's size
let durationFactor = map(size, 15, 50, 0, 1);
let minDuration = Tone.Time("16n").toSeconds();
let maxDuration = Tone.Time("2n").toSeconds();
let duration = lerp(minDuration, maxDuration, durationFactor);

These lines of code calculate the duration of the note based on the size of the circle. The map() function is used to map the size of the circle to a range between 0 and 1. The Tone.Time() function is used to convert note values (e.g., "16n" or "2n") into seconds. The lerp() function is then used to linearly interpolate between the minimum and maximum durations based on the duration factor.

7. Updated the checkCollision() function to include circle sizes in the collision condition:

if (distance < (circleA.size+circleB.size)/2) { // The sum of the radii (A + B)/2

This line of code is updated to account for the random sizes of the circles. It calculates the sum of the radii of both circles and checks if the distance between the centers of the two circles is less than the sum of their radii, indicating a collision.

8. Added a new variable noteIndicator:

let noteIndicator;

This variable is used to store the text element that will display the current note.

9. Created a text element and set its initial content in the setup() function:

// Create a text element and set its initial content
noteIndicator = createP("Current note: None");
noteIndicator.position(20, 20);

These lines of code create a paragraph element using the p5.js createP() function and set its initial content to "Current note: None". The position() function is used to position the element on the screen.

10. Updated the playCollisionSound() function to display the current notes:

// Update the note indicator text to display the current notes
noteIndicator.html("Current note: " + circleA.note + " / " + circleB.note);

This line of code updates the note indicator text to display the notes of the two colliding circles.

These additional changes make the sketch more visually dynamic and informative by adding random sizes for the circles, calculating note durations based on circle size, displaying the current notes being played, and taking into account the sizes of the circles when checking for collisions.

Now we are going to add a new functionality to our interactive music maker: the ability to choose whether or not we want to clear the canvas with every new frame.

By doing this, we'll be able to see the trails of where our circles have been, giving a nice visual effect.

Here are the instructions to accomplish this:

1. Modify the HTML code: We need to add a checkbox that will let us toggle whether or not the canvas gets cleared. Add this line to your HTML code:

 <input type="checkbox" id="redraw" checked> Redraw

2. Modify the JavaScript code: We're going to use the state of this checkbox to decide whether or not to clear the canvas each frame. To do this, modify your draw function in the JavaScript code like this:

 function draw() {
 // Get the checkbox
 var checkbox = document.getElementById("redraw");
 // Only clear the canvas if the checkbox is checked
 if (checkbox.checked) {
 noStroke();
 fill(0);
 rect(0, 0, windowWidth * canvasScaleFactor, windowHeight * canvasScaleFactor);
 } // rest of your draw code... } 

Now the draw function first checks the state of the checkbox. If it's checked, it clears the canvas before drawing the new frame. If it's not checked, it doesn't clear the canvas, which means the new frame is drawn on top of the old one, leaving a trail where the circles have moved.

3. Test it out: Go ahead and try running your program now. Play around with checking and unchecking the 'Redraw' checkbox and see how it changes the behavior of your music maker. Enjoy the new visual effect you've added! Now the Transparency slider makes more sense. 

This is a simple feature that will add great colour variety to our project. Try uploading images from paintings you like and see how the colours match together. I've had some great results with paintings from Turner especially, but Pollock or Kandinsky bring interesting chaos elements too. 

So, in this step, we can upload an image and pick random colours from the image to determine the colours of the circles. Here are the steps to achieve this:

1. Modify the HTML code: First, we need to add an image input field in our HTML file. This will allow users to upload an image from their device. Add this line to your HTML:

   <input type="file" id="imageUpload" accept="image/*">
   

2. Modify the JavaScript code: Next, we need to handle the image upload in our JavaScript code. Once an image is uploaded, we'll store the image and use it to pick colors for our circles. Add the following function to your JavaScript code:

   let img;
   let imgPixels = [];
   
   function preloadImage(event) {
       let reader = new FileReader();
       reader.onload = function(e) {
           img = loadImage(e.target.result, () => {
               img.loadPixels();
               for (let i = 0; i < img.pixels.length; i+=4) {
                   imgPixels.push(color(img.pixels[i], img.pixels[i+1], img.pixels[i+2]));
               }
           });
       };
       reader.readAsDataURL(event.target.files[0]);
   }
   
   document.getElementById('imageUpload').addEventListener('change', preloadImage, false);
   

   In this code, preloadImage is a function that runs when an image is uploaded. It reads the image file, loads the image and its pixels into the img and imgPixels variables.

3. Pick Colours from Image: Now, we need to modify the circle creation process to pick colours from our image. Change your createCircle function as follows:

   function createCircle(x, y, size, note) {
       let cColor;
       if (imgPixels.length > 0) {
           cColor = random(imgPixels);
       } else {
           cColor = color(random(100, 240), random(100, 240), random(100, 240));
       }
       
       // rest of your createCircle code...
   }
   

   This code picks a random color from the imgPixels array if an image has been uploaded, otherwise it picks a random RGB color as before.

4. Test It Out: Now try running your program again. You should be able to upload an image, and the circles should start appearing in colors from that image!

Remember that you will need to upload a new image or refresh the page if you want to change the image or go back to random RGB colors. Enjoy the new functionality you've added!

What should we add next? Leave a comment!

If you open the Javascript console (in Chrome, it's in View -> Developer Tools -> Javascript Console), you might see the warning message "Max polyphony exceeded. Note dropped.". This error message indicates that too many sounds are being triggered at once. In this context, polyphony refers to the number of simultaneous voices or notes it can produce. This will happen as soon as you create more than 10 circles, because the effects we've added use voices, and we quickly run out of the default number of voices. 

In Tone.js, the PolySynth object has a maxPolyphony property which by default is set to 32. This means if more than 32 notes are played at the same time, Tone.js will start dropping notes to prevent performance issues.

To optimise performance and handle this error, there are a few strategies we can apply:

1. Limit the number of circles: The more circles you have, the more potential collisions and thus the more simultaneous sounds. By limiting the number of circles you can indirectly limit the number of simultaneous sounds.

2. Limit the number of simultaneous sounds: Instead of playing a sound every time a circle collides with a wall or another circle, you could introduce a small delay or a cooldown period during which the circle cannot play another sound.

3. Increase Polyphony: If performance isn't a problem and you simply want to allow more simultaneous sounds, you could increase the maxPolyphony property when creating your PolySynth:

Let synth = new Tone.PolySynth(Tone.Synth, { maxPolyphony: 64 }).toDestination();

However, keep in mind that increasing the maximum polyphony could lead to performance issues, particularly in less powerful devices.

4. Only play a sound if a circle isn't already playing: One of the issue we're facing is that sounds might be triggered every frame, in particular if two circles are stuck together or near the border. We are going to add

Ten questions to check what you picked up. Have a go at each one before revealing the answer.

1. What is the purpose of the `circles` array in this code?

   • A. To store the note names
   • B. To store the circle objects created in the sketch
   • C. To store the color values for each circle
   • D. To store the duration of each note
   Show answer

   B. To store the circle objects created in the sketch

2. What is the purpose of the `canvasScaleFactor` variable?

   • A. To control the size of the circles
   • B. To control the maximum number of circles on the canvas
   • C. To control the size of the canvas relative to the window size
   • D. To control the transparency of the circles
   Show answer

   C. To control the size of the canvas relative to the window size

3. What does the `alphaLayer` variable represent?

   • A. The transparency level of the circles
   • B. The size of the circles
   • C. The duration of the note played when a circle is created
   • D. The maximum number of circles on the canvas
   Show answer

   A. The transparency level of the circles

4. What is the purpose of the `maxCircles` variable?

   • A. To control the maximum number of circles on the canvas
   • B. To control the size of the circles
   • C. To control the transparency of the circles
   • D. To control the duration of each note
   Show answer

   A. To control the maximum number of circles on the canvas

5. What does the `playSound` variable do in this code?

   • A. It plays a sound when a circle is created
   • B. It controls whether the sketch is paused or playing
   • C. It sets the volume of the sound played by the circles
   • D. It sets the note that will be played when a circle is created
   Show answer

   B. It controls whether the sketch is paused or playing

6. What does the `noteIndicator` variable store?

   • A. The current note being played
   • B. The text element that displays the current notes
   • C. The size of the circles
   • D. The transparency of the circles
   Show answer

   B. The text element that displays the current notes

7. What does the `createCircle()` function do?

   • A. It sets the size of the circles
   • B. It calculates the duration of the note played when a circle is created
   • C. It checks for collisions between circles
   • D. It creates a circle object with a random size, velocity, colour, and note
   Show answer

   D. It creates a circle object with a random size, velocity, colour, and note

8. What does the `checkCollision()` function do?

   • A. It creates a circle object with a random size, velocity, color, and note
   • B. It sets the size of the circles
   • C. It calculates the duration of the note played when a circle is created
   • D. It checks for collisions between circles and calculates new velocities if a collision occurs
   Show answer

   D. It checks for collisions between circles and calculates new velocities if a collision occurs

9. How is the note duration calculated in this code?

   • A. Based on the size of the circles
   • B. Based on the velocity of the circles
   • C. Based on the transparency of the circles
   • D. Based on the color of the circles
   Show answer

   A. Based on the size of the circles

10. What does the `playCollisionSound()` function do?

    • A. It calculates the average duration and updates the note indicator text
    • B. It sets the size of the circles,It checks for collisions between circles
    • C. It calculates the duration of the note played when a circle is created
    • D. It plays a sound based on the collision between circles, calculates the average duration, and updates the note indicator text
    Show answer

    D. It plays a sound based on the collision between circles, calculates the average duration, and updates the note indicator text